Note: Pratyusha Katikaneni and Carmen Diaz also contributed to this post. They are both research assistants at the Engelberg Center for Health Care Reform, The Brookings Institution.

January 23 update comparing this analysis to one by Scott Heiser, Carrie Colla, and Elliott Fisher: Yesterday, Health Affairs Blog published two analyses of Medicare ACO results, one by Scott Heiser and colleagues and one by us (below). These analyses should be viewed as complementary.

In their post, Heiser and colleagues note a greater likelihood of early financial success (i.e., qualifying for shared savings) in relatively high-cost areas of the country. They reach this conclusion by linking each ACO to their hospital referral region (HRR) and finding a significant relationship with early shared savings. They conclude that ACOs in higher cost regions are more likely to begin the program with a higher benchmark relative to costs, making it easier to attain shared savings.

Our analysis focuses on the specific benchmark for each ACO (not the regional cost) and also concludes that there is a statistically significant correlation between ACOs with higher financial benchmarks and their total savings. However, we note that the beginning financial benchmark explains less than 10 percent of the variation in early financial performance, and note that there are many successful ACOs in lower-cost areas as well.

The results of the two analyses are not contradictory, but rather reflect that while historical spending matters in early success, many other factors matter as well.

Original post: On December 1, CMS released a Notice of Proposed Rulemaking (NPRM) for the Medicare Shared Savings Program (MSSP), which requests feedback for changes CMS is considering for the Medicare accountable care organization (ACO) programs in 2016 and beyond. The proposal suggests significant potential alterations to the program, many of which we recently reviewed, that would address major issues that ACOs and others have raised: uncertainty and inexperience at transitioning to increasing levels of risk, lack of timely and accurate data, changes in attributed patient populations from year-to-year, and financial benchmarks that fail to account for regional variations and continue to reward high ACO performance over time.

The proposed rule raises more issues than it settles, but it clearly indicates that CMS is open to meaningful public comments and will make important revisions in the MSSP. However, the proposal also illustrates the challenges of resolving these issues in a way that both assures substantial ACO participation and improvement, as well as Medicare savings.

Ideally, big changes in key features in a major program like the MSSP would be based on extensive empirical evidence on what determines success in the program. Unfortunately, only limited evidence, including case studies and some comparative data, is available on the determinants of success for Medicare ACOs, and thus on the MSSP. Data released by CMS in September, which we previously reviewed, showed that the MSSP has generated over $700 million in savings to date relative to the spending benchmarks in the program. This is around 1 percent of the costs of care for beneficiaries affected by Medicare ACO initiatives.

Especially with the better quality performance of ACOs compared to fee-for-service Medicare and flat Medicare baseline spending, this is a notable achievement in a large program that is still at an early stage. However, as we described then, this average performance consists of a wide range of results across the Medicare ACOs, with some appearing to achieve substantial savings while providing high quality, and others performing well on quality but not better than the Medicare fee-for-service trend on costs. David Muhlestein and Chase Hall recently examined the variation in MSSP quality results reported in September, suggesting that ACO type and geographic region may have an impact on performance. In this post, we take a closer look at ACO financial performance data and examine some potential drivers of success so far.

In November, CMS released additional data on the quality and financial performance of all 220 MSSP ACOs that began in 2012 and 2013, which should enable more informative analyses than are possible by reviewing aggregate performance alone. The data include each ACO’s attributed population size, financial benchmark, actual spending, percent savings or losses relative to the benchmark, shared savings or shared losses, and performance on each of the 33 quality measures. The data can enable another step toward understanding how ACOs are performing and identifying significant trends among participant characteristics.

Below, we review some preliminary findings from the new data release that have not been highlighted to date, and describe some of the implications for possible changes in the program. In particular, the results suggest that while many ACOs are doing well and some factors like location may help predict success, other features and factors that are harder to measure are much more important. This suggests that Medicare ACOs will continue to benefit from experience and learning before most are willing to accept downside financial performance risk, a critical objective for the program.

At the same time, a wide range of organizations have been able to make progress on quality and cost. Thus, the results also suggest that allowing flexibility and focusing on results, including a clear transition path to two-sided risk coupled with other refinements that make the program more predictable for ACOs, could help more organizations succeed.

Significant Variation in Early Medicare ACO Financial Performance

We evaluated financial performance as the ACO’s actual performance against the benchmark spending projection for the ACO’s attributed beneficiaries. In turn, these calculations reflect the methods for attributing beneficiaries and calculating the benchmark implemented by CMS. These initial methods have been criticized and are likely to be revised; determining the sensitivity of MSSP performance measures to such revisions could provide valuable insights regarding the consequences of changes in the methods.

Financial performance varied significantly among MSSP ACOs, with many ACOs sharing in savings—over $300 million paid out so far—but with many other ACOs having insignificant impacts on spending relative to their benchmarks. As the table below shows, more than half of all program participants (118/220) were able to reduce spending relative to their benchmark in their first performance year, with roughly half of these (58) able to reduce spending enough to qualify for shared savings. However, only 52 of these ACOs earned shared savings, since six failed to satisfactorily report quality measures not captured by claims data. All six were relatively small ACOs, with a beneficiary population size ranging from roughly 5,700 to 17,000. Failing to meet quality reporting requirements had a substantial impact for these smaller ACOs: they left $941,236 to $10.62 million in shared savings on the table.

Table 1. Breakdown of MSSP ACOs Reducing and Increasing Spending

McClellan2-Table-1

Almost all early MSSP participants (215/220) opted for MSSP’s “track one” (upside-only risk) rather than “track two” (upside and downside risk). Of the five ACOs that assumed two-sided risk, two had significant enough reductions relative to their benchmark to qualify for shared savings. Three ACOs in track two had increased spending relative to their benchmark, and for one the higher trend was significant enough that it shared in losses of $4 million.

Thus, most MSSP ACOs achieved relatively small savings or losses, between 0 and 2.5 percent, as shown in Exhibit 1 below. But significantly more ACOs had large savings than had large cost increases relative to their benchmarks. Overall, a larger portion of ACOs saved rather than overspent compared to their benchmark (53 percent vs. 47 percent), but the shift was concentrated at the high and low ends of the spectrum of financial performance (i.e., 9 percent had losses of 5 percent or more, while 15 percent had gains of 5 percent or more).

Exhibit 1. Number of ACOs Attaining Different Levels of Spending Growth or Reduction

McClellan-Exhibit-1

Early Successes Spread Across Wide Range of ACOs

Large ACOs generally did not have an advantage in financial performance compared to smaller ACOs. Indeed, Exhibit 2 shows that MSSP ACOs with less than 8,000 attributed patients had a higher average savings rate than their larger counterparts. The ACOs with less than 8,000 beneficiaries, as a group, were the only ones that significantly (p=0.02) decreased spending. The overall positive early results for these ACOs provide some additional evidence for the view that smaller, physician-led ACOs may be better positioned to transform care delivery more effectively than their larger hospital-led or integrated delivery system ACO counterparts — or at least they may be able to implement changes more quickly.

On the other end of the spectrum, very large ACOs (more than 12,000 attributed patients) also had slightly better average performance relative to their benchmarks (although not statistically significant). Not surprisingly, outlier performances (high and low) in terms of savings were more common among the smaller ACOs.

Exhibit 2. Average Percent Savings by Number of Beneficiaries (with Standard Deviations)

McClellan-Exhibit-2

Some ACO experts have hypothesized that ACOs with a higher benchmark would be more able to attain savings, since they are likely to have more room to reduce spending. Exhibit 3 below shows the distribution of the percentage savings achieved by each ACO versus its per-capita benchmark (i.e., the measure of its expected per-capita spending). Consistent with this hypothesis, there is a significant correlation (p=0.01) between benchmarks and early savings, but the magnitude of the relationship is not that large (Spearman rho=0.206). Put another way, less than 10 percent of variation (r-squared=0.072) in savings rates can be attributed to the quintile of an ACO’s benchmark; there are many financial successes and failures in every benchmark range. For example, some ACOs with high benchmarks generated larger percentage savings (upwards of 15 percent) while other ACOs with similarly high benchmarks overspent by 10-15 percent.

Exhibit 3. Per Capita Expected Spending vs. Percent Savings

McClellan-Exhibit-3

We also note some variations across geographic region in performance of ACOs. Of states with more than 10 MSSP ACOs, Florida and Texas had the highest concentration of shared savings: more than a third (12/30) of Florida-based ACOs and nearly half (7/15) of Texas-based ACOs achieved shared savings. ACOs may be able to achieve better early financial performance in relatively high-cost areas (as in much of Texas and Florida).

However, ACOs in many other states also perform relatively well. Exhibit 4 shows average savings in states with at least five MSSP participants. Many other states also have had successful average ACO performance, including Michigan, Kentucky, Tennessee, Virginia, and Massachusetts. When clustered according to CMS Region, those in the South (including Texas) are performing better on average than other regions. While being in a high-cost area may help, ACOs have had early successes or failures in a wide variety of health care markets across the country.

Exhibit 4. Average ACO Savings in States with At Least 5 MSSP Participants

McClellan-Exhibit-4

Exhibit 5 shows that there is no meaningful correlation (p=0.83) between overall quality performance and an MSSP ACO’s initial financial performance. Many ACOs have attained high quality care as reflected in the MSSP performance measures both with and without notable improvements in financial performance, while others have reduced costs without attaining high quality. These results confirm that there are both high-cost and low-cost ways to achieve higher quality, and that most ACOs appear to have room to both improve quality and lower costs. Other recent results outside of the Medicare ACO program—such as those reported by Milstein and colleagues on primary care practices—similarly find that quality and spending are not closely related. Under traditional Medicare reimbursement, there is even less reward for improving quality while lowering overall spending than in Medicare’s shared-savings ACO program.

Providing a clearer pathway to greater financial rewards for ACOs that are making progress on improving quality and reducing costs could help strengthen the relationship between quality and cost. Moreover, the limited results to date only allow comparisons of baseline quality levels to trends in financial performance; more evidence is needed on how ACOs that improve quality while lowering cost trends are doing so.

Exhibit 5. ACO Quality Performance Score vs. Percent Savings

McClellan-Exhibit-5

These early MSSP results show that some factors like benchmark spending level and being in a high-cost region may have had some impact on the early successes of Medicare ACOs, but these factors explain only a small part of the variation in performance. Some ACOs are doing well in both quality performance and financial performance, while others have not achieved significant improvement in both, in every part of the country. Based on our work in the ACO Learning Network, as well as the findings of many case studies and other anecdotal reports, more important factors likely include clinical interventions, analytic capabilities, leadership and culture, and other up-front investments.

Next Steps

Succeeding in accountable care—achieving better care at a lower cost—is not easy, including for organizations that took the steps to become early MSSP participants. While the data released by CMS are a start, more data about ACO features and activities should be developed and shared so that we can better learn about what has worked for organizations so far. For example, can we learn in more detail about the activities of the minority of ACOs that continue to achieve favorable quality and cost trends? ACO surveys and other projects are underway that can help address this question, but they need the opportunity to link to more detailed CMS data on MSSP features and performance to be more informative.

While a better understanding of the factors determining ACO success will help promote success by reducing the uncertainty about what organizations can do to improve performance, the MSSP can also reduce uncertainty by providing a clearer long-term path to financial sustainability for organizations that are implementing accountable care. This could include more help for organizations that are committed to the program to move from early stages to broader reforms.

Revisions in MSSP policies around benchmarks, risk requirements, and other factors will also be important elements of this clearer pathway to success. This includes refinements around quality and cost measures that determine performance. Many physician and other health care providers continue to stress that quality measures are imperfect and could do a better job measuring what is really important.

Additionally, ACO financial measures may be noisier and less predictable as a result of issues with attribution and benchmark calculation, not real differences in actual performance for the Medicare beneficiaries being served. For example, Sharp Healthcare left the Pioneer ACO program, which uses the same benchmark methodology as the MSSP. Sharp contends that it lost money due to the benchmark formula, despite the fact that they did well compared to average costs in their region and apparently would have done well if a benchmark more like that used in the Medicare Advantage program was employed.

CMS discussed a number of possible approaches to adjusting the benchmark calculation, but did not identify a preferred solution in the proposed rule. Rather, CMS seems open to considering a number of different options for adjusting the benchmarks, including using regional expenditures, a hybrid between historical performance and regional factors, equally weighting historical spending, and accounting for previous shared savings in calculating a new benchmark or other approaches that may be proposed in comments.

The recently released NPRM also enables ACOs in “Track One” to be able to remain in one-sided risk for another performance cycle of three years, although with lower eligible shared savings and a demonstration that they have been able to reduce costs in at least one of the first two performance years. To reduce startup uncertainties and difficulties, this low-risk “starter” option could be accompanied by other steps (better data, and better support for implementing initial ACO reforms) that appear to have proven useful for commercial-insurance ACOs.

Further, since some ACOs are already beginning to succeed, it will be even more important to provide a clear road map to additional payment reforms that can enable further steps to improve care and lower costs. CMS has proposed creating a two-sided “Track Three” with higher percentage of savings and losses for those ACOs that are willing and able to assume greater levels of two-sided financial risk. In addition, CMS is seeking comment on coupling the model with the relaxation of some current payment restrictions, which include the requirement for a 3-day hospitalization stay for coverage of a skilled nursing facility (SNF) and limits on use of home health services and telemedicine. While these adjustments to the MSSP tracks could create incentives for organizations of varying size and sophistication to remain in or begin participation in an MSSP contract, they will likely not be enough to drive an immediate or dramatic increase in participation in two-sided risk models. CMS must do more to create a clearer path for more meaningful delivery reforms to pay off for MSSP ACOs willing to assume more risk.

Conclusion

These early results support the thoughtful examination taking place now on how MSSP can better achieve its key health care reform goals. While half of MSSP ACOs have been able to reduce spending in year one, and the program has achieved significant overall savings and even more notable quality results, further changes will be needed for MSSP to attract and sustain organizations that are effectively reforming care. Many ACOs are seeking more certainty about their financial performance and prospects for savings before committing to two-sided risk and making larger investments in redesigning care. Organizations that can improve care and lower costs need a clearer and more sustainable path beyond the current two tracks.

CMS has solicited comment on a broad range of improvements to the program. MSSP ACOs and those working with them must take this opportunity to submit formal comments to encourage CMS to address these issues in the development of its final rule. The future of the Medicare ACO program depends on it.

Editor’s note: Please note that the empirical analyses discussed in this post have not been subjected to external peer review.