Editor’s Note: In the post below, Jonathan Skinner and Shannon Brownlee examine the relationship between health care spending and utilization in hospitals, on the one hand, and patient outcomes on the other. In an earlier post, John Wennberg and Brownlee rebutted claims that spending and utilization variations among academic medical centers are due to differences in patient income, race, and health status. 

Many Health Affairs articles and Health Affairs Blog posts have addressed the relationship between spending/utilization and quality. For a sampling, see articles by Yasaitis, Chandra, and coauthors and Jha and coauthors on the relationship between hospital spending and quality; blog posts by Jha and Chandra commenting on each other’s articles; and an article by Wennberg and coauthors on the relationship between inpatient intensity of care and patient ratings of care.

Is more care better? Three decades of research at Dartmouth suggests that on average the answer is no. Now a newer paper, published in the journal Circulation, Cadiovascular Quality and Outcomes, has garnered a lot of press and has been portrayed as the “anti-Dartmouth study” in at least one article. Even Betsy McCaughey, the Republican health care spoiler of “death panel” fame, has gotten into the act with a silly op-ed in the New York Post claiming that the paper demonstrates that Dartmouth has got it all wrong.

In contrast to the media portrayals, the paper itself was actually quite restrained in its claims, and the researchers (led by Dr. Michael Ong, a clinical scholar from UCLA) bring impeccable credentials and strong quantitative skills.   Briefly, they found a remarkably strong association between more spending and better risk-adjusted outcomes for heart failure patients in six California hospitals. But does the paper actually invalidate the Dartmouth findings and show that more is better?

Not exactly. The major focus of their paper is to contrast their “look-forward” measures of intensity with the “look-back” or end-of-life measures of health care intensity frequently used by Dartmouth to compare individual hospitals. In their look-forward approach, Ong and his team examined the care that 3,999 heart failure patients received at six California teaching hospitals over the course of 6 months. (Although the researchers don’t name the hospitals, their affiliations make it clear that they are the five University of California hospitals in Davis, San Francisco, Irvine, Los Angeles, and San Diego, plus Cedars-Sinai in Los Angeles.) The authors then looked at mortality rates, after adjusting for how likely patients were to die. The results appeared to show that patients who received more intense care – particularly more days in the hospital – were at lower risk of being dead by the end of the study period.

The authors then argued that they got a different result from the Dartmouth studies because Dartmouth researchers use a look-back method of assessing the relationship between outcomes and intensity of care. Briefly, many Dartmouth studies try to measure health care “intensity” by studying Medicare spending and utilization for people with one or more chronic illnesses in the last six months (or two years) of life.  This is a very simple way to risk-adjust for differences in health status across regions  — every person in the sample, after all, is deceased by the end of the study period. (For a more in-depth discussion of the evidence that suggests that more care is associated with equal or worse outcomes, see the sidebar below.)

As it turns out, even for the Circulation study’s sample of six hospitals, the look-forward approach yields a nearly identical result to the Dartmouth look-back approach.  When the six hospitals are ranked using the Circulation end-of-life expenditure measures, and compared to the rankings for 180-day mortality, similar results are obtained – end-of-life spending is strongly and negatively associated with 180-day mortality.

In other words, whether look-forward or look-back measures of intensity of care are used, the same results hold in this sample of six hospitals: more spending is associated with lower mortality and hence better outcomes.

Testing Whether More Care Is Better Or Worse

So what’s the answer – is more care better or worse?  To solve the puzzle, one of us used an entirely different data set described more fully here: heart attack (acute myocardial infarction) patients in the Medicare population.  Similar risk adjustment, and similar look-forward approaches were used for one-year mortality and one-year expenditures and utilization to compare spending and risk-adjusted mortality.  As the reader can see from the table below, this new analysis yields the same results for the six California hospitals: the correlation between more care and mortality is negative — which means more care is better. (The results are even larger when we use a “DRG-weight” utilization measure, which removes any distracting influence of price differences across regions.)

Now let’s consider a larger sample than just these six hospitals – hospitals treating AMI patients across the United States.   Using this much larger sample,  the correlation between mortality and expenditures, and between mortality and utilization, flips and becomes positive — more spending is associated with worse outcomes.

Table 1: Correlation Coefficient Between Part A Hospital Utilization and One-Year Risk-Adjusted Mortality Following a Heart Attack







(DRG Weight)


The six California hospitals used in Ong, et al





All U.S. hospitals


< 0.001


< 0.001

Source: Author’s (Skinner) calculations.  These calculations pool 2001-05 data (on one-year risk-adjusted mortality and one-year risk-adjusted expenditures or DRG weights. N = 1.1 million

In other words, the experience of six hospitals does not hold for the larger sample of all hospitals in the U.S.

Why would this be the case? Figure 1 shows a hypothetical scatter plot between expenditures and mortality, with each dot representing an individual hospital.  Sampling just the six hospitals corresponding to the six red dots, one finds a clear negative association between spending and mortality, and would naturally conclude that more spending was associated with lower mortality. However, the entire sample of hospitals (red and blue dots combined) shows a positive association between expenditures and mortality.

Figure 1: Hypothetical Graph Showing the Correlation between Hospital Spending and Quality for a Subgroup of Six Hospitals (Red Dots) and All Hospitals (Red and Blue Dots)

In sum, we believe that the differences in results obtained by Ong and his colleagues and by Dartmouth are explained by the different sample sizes: six hospitals versus thousands. 

Questions Raised: The Influence Of Risk Adjustment …

But the results also raise several additional questions, about which we can only speculate. For instance, a key question posed by these results is the influence of risk adjustment. As the Circulation study shows, the association between unadjusted look-forward” spending and unadjusted 180-day mortality is rather modest. It is only after risk adjustment that they find statistically significant differences among the six hospitals. In general, risk adjustment is the right thing to do, but it is also possible that differences in the way these six hospitals diagnose comorbidities could have a systematic impact on standard risk-adjustment measures.  (This is something that worries us, too, and in ongoing Dartmouth research we are trying to develop risk adjusters that are less sensitive to this type of bias.)

… And Interpreting The Association Between Spending And Outcomes

The second and more difficult question is how to interpret the association between spending and outcomes.  Does a positive association between spending and mortality really mean that it’s the spending that leads to worse outcomes?  Possibly. More spending generally indicates more time in the hospital, and that means more potentially risky procedures and more opportunities for patients to contract an infection or suffer an error, as described in an article in the Atlantic.

By the same token, it’s possible (though we think unlikely) that the negative association between spending and mortality found in the six California hospitals means that these specific hospitals are better solely because they spend more. One of the central problems for health care reform is that hospitals that spend more and get good results don’t necessarily know what it is they’re doing right. By the same token, hospitals that spend more and get worse outcomes may not be able to pinpoint what they’re doing wrong. For an excellent description of what this means, see the recent New York Times Magazine piece about Brent James and Intermountain HealthCare.

A new set of studies (for example, here and here) suggest a more nuanced story – that low-quality hospitals and physicians could end up spending more – additional tests or hospital admissions — to compensate for their initially worse quality.  Thus one might observe some hospitals spending more and getting worse (or no better) outcomes – but it’s not the spending that causes the poor outcomes.

Perhaps the key lesson to take home from the Circulation paper and the Dartmouth research is how little we know about the science of health care delivery.

SIDEBAR: Looking Forward and Back

In the most comprehensive study using national data and millions of people, Elliott Fisher, David Wennberg, and coauthors conducted a “look-forward” study of what happened to patients with one of three different conditions: heart attack, hip fracture, and colon surgery for colon cancer. But first, they divided the 306 U.S. hospital referral regions into five equally sized quintiles based on average end-of-life spending in the region.  (Their landmark study in the Annals of Internal Medicine is in two parts; here and here.)  In other words, the “look-back” end-of-life measures were used simply to assign regions to quintiles; New York City was a high-cost region and so anyone having a heart attack or hip fracture or colon cancer in New York City was assigned to that quintile.

They then looked forward at what happened to each of the three cohorts of patients starting on the day the patient was hospitalized for his heart attack, hip fracture, or colon surgery.  The most important conclusion to be drawn from this paper is they found that patients who were treated in regions where utilization (and spending) were higher did not experience better risk-adjusted outcomes. In other words, more was not necessarily better. (In fact, patients suffering from hip fracture and heart attack appeared to experience slightly higher mortality when treated in regions falling within the highest quintile compared to those in the lowest quintile.)

While their primary results were presented using quintiles created with the look-back measures, they also tested the sensitivity of their results to using look-forward measures of intensity of care.  Thus they used heart attack and colon cancer cohorts to create the assignment of hospital referral regions to the five quintiles, and then used those quintiles to study outcomes of hip fracture patients.  Whether they used look-forward or look-back approaches to creating the intensity quintiles, they got the same results – more spending did not yield better outcomes.

It is too bad that the critics of the Dartmouth research haven’t spent more time reading the Annals of Internal Medicine studies.  The statistical discussion can be difficult to follow without some level of statistical training, but is well worth the effort.