Blog Home

«
»

Is More Care Better? The Evidence Suggests No



December 10th, 2009

Editor’s Note: In the post below, Jonathan Skinner and Shannon Brownlee examine the relationship between health care spending and utilization in hospitals, on the one hand, and patient outcomes on the other. In an earlier post, John Wennberg and Brownlee rebutted claims that spending and utilization variations among academic medical centers are due to differences in patient income, race, and health status. 

Many Health Affairs articles and Health Affairs Blog posts have addressed the relationship between spending/utilization and quality. For a sampling, see articles by Yasaitis, Chandra, and coauthors and Jha and coauthors on the relationship between hospital spending and quality; blog posts by Jha and Chandra commenting on each other’s articles; and an article by Wennberg and coauthors on the relationship between inpatient intensity of care and patient ratings of care.

Is more care better? Three decades of research at Dartmouth suggests that on average the answer is no. Now a newer paper, published in the journal Circulation, Cadiovascular Quality and Outcomes, has garnered a lot of press and has been portrayed as the “anti-Dartmouth study” in at least one article. Even Betsy McCaughey, the Republican health care spoiler of “death panel” fame, has gotten into the act with a silly op-ed in the New York Post claiming that the paper demonstrates that Dartmouth has got it all wrong.

In contrast to the media portrayals, the paper itself was actually quite restrained in its claims, and the researchers (led by Dr. Michael Ong, a clinical scholar from UCLA) bring impeccable credentials and strong quantitative skills.   Briefly, they found a remarkably strong association between more spending and better risk-adjusted outcomes for heart failure patients in six California hospitals. But does the paper actually invalidate the Dartmouth findings and show that more is better?

Not exactly. The major focus of their paper is to contrast their “look-forward” measures of intensity with the “look-back” or end-of-life measures of health care intensity frequently used by Dartmouth to compare individual hospitals. In their look-forward approach, Ong and his team examined the care that 3,999 heart failure patients received at six California teaching hospitals over the course of 6 months. (Although the researchers don’t name the hospitals, their affiliations make it clear that they are the five University of California hospitals in Davis, San Francisco, Irvine, Los Angeles, and San Diego, plus Cedars-Sinai in Los Angeles.) The authors then looked at mortality rates, after adjusting for how likely patients were to die. The results appeared to show that patients who received more intense care – particularly more days in the hospital – were at lower risk of being dead by the end of the study period.

The authors then argued that they got a different result from the Dartmouth studies because Dartmouth researchers use a look-back method of assessing the relationship between outcomes and intensity of care. Briefly, many Dartmouth studies try to measure health care “intensity” by studying Medicare spending and utilization for people with one or more chronic illnesses in the last six months (or two years) of life.  This is a very simple way to risk-adjust for differences in health status across regions  — every person in the sample, after all, is deceased by the end of the study period. (For a more in-depth discussion of the evidence that suggests that more care is associated with equal or worse outcomes, see the sidebar below.)

As it turns out, even for the Circulation study’s sample of six hospitals, the look-forward approach yields a nearly identical result to the Dartmouth look-back approach.  When the six hospitals are ranked using the Circulation end-of-life expenditure measures, and compared to the rankings for 180-day mortality, similar results are obtained – end-of-life spending is strongly and negatively associated with 180-day mortality.

In other words, whether look-forward or look-back measures of intensity of care are used, the same results hold in this sample of six hospitals: more spending is associated with lower mortality and hence better outcomes.

Testing Whether More Care Is Better Or Worse

So what’s the answer – is more care better or worse?  To solve the puzzle, one of us used an entirely different data set described more fully here: heart attack (acute myocardial infarction) patients in the Medicare population.  Similar risk adjustment, and similar look-forward approaches were used for one-year mortality and one-year expenditures and utilization to compare spending and risk-adjusted mortality.  As the reader can see from the table below, this new analysis yields the same results for the six California hospitals: the correlation between more care and mortality is negative — which means more care is better. (The results are even larger when we use a “DRG-weight” utilization measure, which removes any distracting influence of price differences across regions.)

Now let’s consider a larger sample than just these six hospitals – hospitals treating AMI patients across the United States.   Using this much larger sample,  the correlation between mortality and expenditures, and between mortality and utilization, flips and becomes positive — more spending is associated with worse outcomes.

Table 1: Correlation Coefficient Between Part A Hospital Utilization and One-Year Risk-Adjusted Mortality Following a Heart Attack

 

 

Correlation

(Expenditures)

p-value

Correlation

(DRG Weight)

p-value

The six California hospitals used in Ong, et al

-0.67

0.14

-0.91

0.01

All U.S. hospitals

0.20

< 0.001

0.11

< 0.001

Source: Author’s (Skinner) calculations.  These calculations pool 2001-05 data (on one-year risk-adjusted mortality and one-year risk-adjusted expenditures or DRG weights. N = 1.1 million

In other words, the experience of six hospitals does not hold for the larger sample of all hospitals in the U.S.

Why would this be the case? Figure 1 shows a hypothetical scatter plot between expenditures and mortality, with each dot representing an individual hospital.  Sampling just the six hospitals corresponding to the six red dots, one finds a clear negative association between spending and mortality, and would naturally conclude that more spending was associated with lower mortality. However, the entire sample of hospitals (red and blue dots combined) shows a positive association between expenditures and mortality.

Figure 1: Hypothetical Graph Showing the Correlation between Hospital Spending and Quality for a Subgroup of Six Hospitals (Red Dots) and All Hospitals (Red and Blue Dots)

In sum, we believe that the differences in results obtained by Ong and his colleagues and by Dartmouth are explained by the different sample sizes: six hospitals versus thousands. 

Questions Raised: The Influence Of Risk Adjustment …

But the results also raise several additional questions, about which we can only speculate. For instance, a key question posed by these results is the influence of risk adjustment. As the Circulation study shows, the association between unadjusted look-forward” spending and unadjusted 180-day mortality is rather modest. It is only after risk adjustment that they find statistically significant differences among the six hospitals. In general, risk adjustment is the right thing to do, but it is also possible that differences in the way these six hospitals diagnose comorbidities could have a systematic impact on standard risk-adjustment measures.  (This is something that worries us, too, and in ongoing Dartmouth research we are trying to develop risk adjusters that are less sensitive to this type of bias.)

… And Interpreting The Association Between Spending And Outcomes

The second and more difficult question is how to interpret the association between spending and outcomes.  Does a positive association between spending and mortality really mean that it’s the spending that leads to worse outcomes?  Possibly. More spending generally indicates more time in the hospital, and that means more potentially risky procedures and more opportunities for patients to contract an infection or suffer an error, as described in an article in the Atlantic.

By the same token, it’s possible (though we think unlikely) that the negative association between spending and mortality found in the six California hospitals means that these specific hospitals are better solely because they spend more. One of the central problems for health care reform is that hospitals that spend more and get good results don’t necessarily know what it is they’re doing right. By the same token, hospitals that spend more and get worse outcomes may not be able to pinpoint what they’re doing wrong. For an excellent description of what this means, see the recent New York Times Magazine piece about Brent James and Intermountain HealthCare.

A new set of studies (for example, here and here) suggest a more nuanced story – that low-quality hospitals and physicians could end up spending more – additional tests or hospital admissions — to compensate for their initially worse quality.  Thus one might observe some hospitals spending more and getting worse (or no better) outcomes – but it’s not the spending that causes the poor outcomes.

Perhaps the key lesson to take home from the Circulation paper and the Dartmouth research is how little we know about the science of health care delivery.

SIDEBAR: Looking Forward and Back

In the most comprehensive study using national data and millions of people, Elliott Fisher, David Wennberg, and coauthors conducted a “look-forward” study of what happened to patients with one of three different conditions: heart attack, hip fracture, and colon surgery for colon cancer. But first, they divided the 306 U.S. hospital referral regions into five equally sized quintiles based on average end-of-life spending in the region.  (Their landmark study in the Annals of Internal Medicine is in two parts; here and here.)  In other words, the “look-back” end-of-life measures were used simply to assign regions to quintiles; New York City was a high-cost region and so anyone having a heart attack or hip fracture or colon cancer in New York City was assigned to that quintile.

They then looked forward at what happened to each of the three cohorts of patients starting on the day the patient was hospitalized for his heart attack, hip fracture, or colon surgery.  The most important conclusion to be drawn from this paper is they found that patients who were treated in regions where utilization (and spending) were higher did not experience better risk-adjusted outcomes. In other words, more was not necessarily better. (In fact, patients suffering from hip fracture and heart attack appeared to experience slightly higher mortality when treated in regions falling within the highest quintile compared to those in the lowest quintile.)

While their primary results were presented using quintiles created with the look-back measures, they also tested the sensitivity of their results to using look-forward measures of intensity of care.  Thus they used heart attack and colon cancer cohorts to create the assignment of hospital referral regions to the five quintiles, and then used those quintiles to study outcomes of hip fracture patients.  Whether they used look-forward or look-back approaches to creating the intensity quintiles, they got the same results – more spending did not yield better outcomes.

It is too bad that the critics of the Dartmouth research haven’t spent more time reading the Annals of Internal Medicine studies.  The statistical discussion can be difficult to follow without some level of statistical training, but is well worth the effort.

Email This Post Email This Post Print This Post Print This Post

 to the #1 source of health policy research.

2 Trackbacks for “Is More Care Better? The Evidence Suggests No”

  1. Twitted by hctransformed
    December 13th, 2009 at 5:10 pm
  2. Potpourri of Links (Mostly Healthcare Reform Related) « humani nil a me alienum puto
    December 13th, 2009 at 12:37 pm

5 Responses to “Is More Care Better? The Evidence Suggests No”

  1. bhshirts Says:

    This article shows how statistical differences in sampling can show different results. It is unfortunate that the null hypothesis is that increased costs do not cause worse outcomes. In a health system where there are justifications for each expenditure one would expect higher costs always correlate with better outcomes. I am not surprised that at least some of the expeditures at some of the six hospitals lead to good outcomes. There is still a long way to go to show exactly which treatments provide the greatest value and which should be used selectively or eliminated to reduce unnecessary costly treatments.

  2. Edward Donovan Says:

    International studies comparing spending and outcomes suggest that U.S. variation in both these measures may be so small that it is difficult to detect a relationship if one exists. Studies that focus on this relationship only in the U.S. likely are examining one end of a larger distribution in which all spending is high and all outcomes are relatively poor. Studies that include data from countries where costs are lower and outcomes better may better test the hypothesis that higher spending for care (especially relative to spending on prevention) is associated with worse health outcomes.

  3. mschorin Says:

    Frankly, we have a growing problem with obesity. Just keeping people out of hospital will be a major challenge. They suffer more heart disease, diabetes, joint problems.
    More care may not be better…but less care cannot be a benefit either
    .
    http://schorin-strategies.com/nutrition

  4. barryzajac Says:

    I look at the Ong, et al study with some reservations. The relationships that they identified are based largely on the results of one facility that had higher cost and better outcomes than the rest, which were all reasonably close together on the scatterplot. For this facility only (‘F’ in their analysis), approximately 32% of identified HF cases were removed from the ‘looking forward’ analysis due to missing cost data. No attempt was made in the article to demonstrate that this did not introduce bias into the analysis.
    In addition to this, I agree with the authors that generalizing from the experience of 5 facilities may not be wise in any case.

  5. MBMundorff Says:

    If the authors’ Figure 1 is a reasonable representation of the actual data, it would appear — despite the plotted regression line (which likely has a very low coefficient) — that there is essentially no relationship between mortality and spending. Which would certainly at a minumum not support the assertion by Ong et al. that more spending correlates with better mortality outcomes.

Leave a Reply

Comment moderation is in use. Please do not submit your comment twice -- it will appear shortly.

Authors: Click here to submit a post.