On August 25, the Centers for Medicare and Medicaid Services (CMS) released the quality and financial results for the ACOs participating in the Medicare Shared Savings Program (MSSP) and the Pioneer ACO Model during 2015. Overall, 31 percent of the MSSP and Pioneer ACOs received shared savings bonuses for their 2015 performance, an increase over the 27 percent that earned a bonus in 2014.

While more ACOs are succeeding under the program, there continues to be substantial variation in financial performance and quality results. This variation underscores that payment reform alone is not enough to improve quality and reduce costs, but rather that organizations must also transform the way care is delivered. Care transformation is difficult and takes time to get right. Consistent with the 2014 results, this year’s data confirms that the most experienced ACOs have a greater likelihood of achieving shared savings.

Also consistent with last year, large, consolidated ACOs did not necessarily achieve the best performance. In fact, we found that the opposite was often true, as smaller, physician-led ACOs were more likely to improve quality and lower cost enough to earn shared savings. This result is a cautionary note given the trend toward mergers and consolidations among health systems; consolidation and larger size do not necessarily lead to the functional integration and efficiency needed to succeed under alternative payment models.

This blog post expands on these conclusions, provides additional analyses, and highlights some policy implications, focusing on the MSSP program.

Summary of the MSSP 2015 Results

In 2015, almost 400 ACOs participated in the MSSP, up from the approximately 330 in 2014. As illustrated in exhibit 1, these ACOs were located across the country and have been in the program for 1-4 years. As in previous years, the ACOs demonstrated relatively high quality, with an aggregate quality score (averaged over the 33 ACO measures) of 91 percent. This is an increase from an average quality score of 86 percent in 2014, although these numbers are not strictly comparable as the MSSP quality measure set changed between 2014 and 2015. The 2015 MSSP results are also similar to the Pioneer results from 2015, which had an average quality score of 92 percent.

Exhibit 1. Characteristics of ACOs in the 2015 Medicare Shared Savings Program

Number of ACOs392
Average Years in the Program (as of 12/31/15)2.4
Program Track
Track 1 (Shared Savings Only)389
Track 2 (Two-Sided Risk)3
Geographic Region
Average Quality Score91%

Relationship between Quality and Cost

Exhibit 2 shows the net savings (benchmark minus actual spending as a percent of the benchmark) versus aggregate quality scores for MSSP ACOs in 2015. (A few outliers are omitted for clarity.) ACOs that earned shared savings had modestly better quality scores (average aggregate quality score of 93 percent) than those that did not earn shared savings (average aggregate quality score of 91 percent). However, there was great variability among individual ACOs in their spending and quality performance. This high variation and the lack of any strong relationship between net spending and quality is consistent with the findings from previous years (see 2012-2013 and 2014). The finding suggests a continued need for better evidence of what quality improvement efforts will meaningfully improve care quality while also reducing costs.

Exhibit 2 Relationship of savings (benchmark spending minus actual spending as a percentage of benchmark) versus quality (aggregate quality score) for MSSP ACOs in 2015


Outliers have been omitted from this graph for clarity, as have ACOs that were in their pay-for-reporting year.

Exhibit 3 shows the difference in quality measure scores between the ACOs that earned shared savings and those that did not. Overall, the scores are relatively similar. However, ACOs that achieved savings did do better than those that did not with some measures, including screening for falls risks (ACO 13), weight screening and follow up (ACO 16), depression screening and follow up (ACO 18), screening for high blood pressure (ACO 21), and ACE/ARB therapy for patients with CAD and diabetes and/or LVSD (ACO 33). In contrast, these same ACOs tended to have lower (worse) scores on the percentage of primary care providers who qualify for the EHR incentive payment (ACO 11). (We intend to examine these quality measure scores in more depth in future studies to understand the cause of these variations).

Even though there were changes to the measures this year, only 7 ACOs failed to successfully report quality measures, in contrast to 11 in each of the first two years with fewer participants. It should be noted that ACOs in the first year of the program (those beginning January 2015) received full points if they successfully reported their measure results, independent of performance.

Exhibit 3. Average quality measure scores for ACOs that earned shared savings bonuses and those that did not


Statistically significant differences between the two groups are indicated with an asterisk in the measure title.

The Benchmark Matters

Consistent with previous analyses of Medicare ACO results, these findings indicate that ACOs that received shared savings had significantly higher financial benchmarks per beneficiary (average of $11,393) than those ACOs that did not share in savings (average of $9,968). As a result, those ACOs that earned shared savings bonuses had significantly higher average total expenditures per beneficiary ($10,555 on average) than those that did not ($10,187). Exhibit 4 summarizes these results.

Exhibit 4. Average benchmark per beneficiary and total expenditures per beneficiary for ACOs that received shared savings and those that did not


Standard errors were included for these values.

Exhibit 5 shows that progressively higher benchmarks were associated with a higher percentage of ACOs that qualified for shared savings. ACOs at all benchmark levels, though, were able to qualify for shared savings. Future work will examine how ACOs have been to achieve shared savings in different market environments and with different benchmarks.

Exhibit 5. Percentage of ACOs qualifying for shared savings based on their benchmark per beneficiary


Experience Matters

The longer an organization participates as an ACO, the more time it has to learn about how to be efficient, improve operations, and eventually transform care delivery. Exhibit 6 shows this trend. Fifty-five percent of the ACOs that joined in the first cohort in 2012 earned shared savings this year, compared to only 21 percent of those beginning in 2015. This result reinforces previous findings about the importance of experience.

The impact of experience on financial performance also suggests that the program will save more for Medicare as time goes on. That is, some short-term losses may occur from having a majority of ACOs in one-sided risk arrangements, but their experience may pay off in larger and more persistent spending reductions over time, particularly with a transition to a two-sided risk arrangement.

Exhibit 6. The percentage of ACOs earning shared savings bonuses as a function of years in the program


ACOs that remain in the program longer have a significantly higher probability of achieving shared savings.

Physician-Led and Integrated ACOs, and Smaller ACOs, Tended to Do Better

Exhibit 7 demonstrates the success of ACOs by provider configuration. As in previous years, physician-led and integrated (physician-hospital partnership) ACOs were more likely to achieve shared savings. In addition, as Exhibit 8 shows, ACOs that received shared savings tended to be smaller (our preliminary analyses suggest this association is not simply driven by physician-led ACOs being smaller). Exhibit 8 includes information on the net per-beneficiary impact to Medicare for all beneficiaries enrolled in each quintile of sizes. While these data need further analysis, the MSSP outcomes to date provide little evidence to support the notion that ACOs need to be large or hospital-led in order to achieve lower spending.

Exhibit 7. Rate of shared savings bonus for different types of ACOs (hospital systems, physician groups, or integrated)


Exhibit 8. Performance of ACOs by size

QuintileNumber of ACOsMean Number of BeneficiariesNet Savings/Loss Per Beneficiary

Geographic Variation

ACOs in the South had a higher probability of achieving shared savings than ACOs in other regions, as Exhibit 9 shows. We examined this effect further in a preliminary regression model adjusting for benchmark and population size and found that a significant regional effect persisted. Further analysis is needed to understand the causes of this “Southern effect.” The map in Exhibit 10 shows how the savings rate varies by state.

Exhibit 9. Rates of shared savings in different regions in the country


The South has higher shared savings rates than other geographic regions.

Exhibit 10. Map illustrating the average savings for ACOs in different states


Total Impact

In 2015, beneficiaries in the MSSP cost CMS $72.9 billion dollars against an estimated cost of $73.3 billion for a gross savings of $429 million. However, CMS paid $646 million in shared savings bonus payments to high performing ACOs, leading to a net loss of $216 million, or a loss of slightly less than 0.3 percent.. Contribution to the total impact varied considerably as seen in Exhibit 11. ACOs in the first round (start date of April 2012) saw average savings of over $200 per beneficiary for 2015, while rounds 3-5 collectively realized net losses for the year.

Exhibit 11. Financial impact of the Medicare Shared Savings Program by start date

RoundStart DateAssigned BeneficiariesPercent Savings/LosesNet Savings/Loss
1April 2012351,5851.89%$72,045,856
2July 20121,704,3410.13%$21,966,968
3January 20131,782,013-0.60%$-107,232,592
4January 20141,783,929-0.83%$-148,839,104
5January 20151,648,365-0.34%$-54,230,300

Policy Implications

Accountable care reforms have led to significant improvements in care quality without significantly raising costs. But like many payment reform models, the MSSP and Pioneer programs have been criticized for failing to realize meaningful and consistent short-term savings. With any program that requires significant operational and cultural changes to succeed, effects may grow over time. As noted earlier, ACOs that have been in the program for several years are more likely to succeed than new entrants. This is not a new effect for this year. As shown in Exhibit 12, in each year of the program more experienced ACOs have been able to improve their performance.

Exhibit 12. Percent of ACOs earning shared savings by start date for performance years 2013-2015

RoundStart Date201320142015
1April 201229.60%34.60%54.50%
2July 201232.20%37.60%38.50%
3January 201320.80%27.20%37.40%
4January 201419.30%22.30%
5January 201521.30%

While these are positive trends, there continues to be variation in performance. This reinforces the conclusion that payment reform alone is not sufficient to meaningfully improve quality and lower costs, but that changes in health care delivery are also required. However, reconfiguring care delivery is not simple. Delivery reform requires redesigning systems, infrastructure, and processes, investing in health information technology, creating new partnerships, substantive changes in clinician and administrative tasks, and many other significant challenges. One of the most difficult challenges is the cultural redesign required at all levels of the organization, from leadership to the individual clinicians. From interviews and surveys gathered from hundreds of ACOs, we have consistently heard that delivery reform is a monumental logistical, financial, and technical hurdle. The time to effect this change is measured in years, not months.

Importance of Delivery System Reform

Payment reform and delivery reform are like a pair of skis. Policymakers often want to see immediate results and try to push the policy ski as far forward as possible, but forget about the lagging delivery ski. If progress in payment reform is not matched by progress in delivery reform, the misaligned skis will cause a crash, or, in this case, it will cause ACOs to leave the voluntary program and revert to non-value-based models.

While further policy refinements could enhance the impact of ACOs and other value-based programs, perhaps the larger challenge today is providing effective support and tools for organizations to develop the care delivery competencies needed for success in new payment models. Organizations like the Accountable Care Learning Collaborative (which we are affiliated with), and others, are actively involved in identifying these competencies. Despite these initial efforts, much more work needs to be done to identify how different types of organizations can move more confidently toward better models of care.

The need to improve care delivery is also highlighted by the lack of correlation between quality scores and financial performance. While ACOs that qualified for bonus payments have slightly better average quality scores than those that did not qualify, the difference is minimal and the correlation between quality and financial performance is negligible. At the same time, ACOs perform much better than non-ACO providers on the majority of quality measures. It is important to understand the strategies employed by those ACOs that manage to have high quality and low cost, and to see how those could be spread broadly.

Updating the Benchmark

The results also suggest the need for additional program refinements. While the benchmark is the key bar that the ACOs must clear for bonus payments, the benchmark itself is a policy determination. In the MSSP, the benchmark is determined initially based on prior performance. While an ACO that spent $13,700 per beneficiary with a $14,000 benchmark is a success from this standpoint, it is not a long-term policy solution to count an ACO that spent $8,100 per beneficiary with an $8,000 benchmark as a failure. Potentially more important than comparing expected spending to actual spending is to create policies that both limit any short-term Medicare spending increases, while encouraging the longer-term development and sustainability of efficient, high-quality ACOs. The current CMS policy of transitioning ACOs who have remained in the program from historical to regional benchmarks may be one way to do this, especially with evidence suggesting that longer-term ACOs have significantly better financial performance. A potentially faster alternative might be the mandatory implementation of payment reform pilots, as CMS is undertaking in its recent bundled payment initiatives. But as we have noted, this is only likely to work if health care organizations can develop the needed competencies to have a higher probability of success in such payment models.

Preserving Small Physician Groups

It is important to emphasize the performance of smaller organizations. While ACOs have been described as being larger, integrated groups of providers that directly provide all aspects of care, the 2015 results suggest that consolidation and size do not imply functional integration and efficiency. Smaller groups, including small physician groups without hospitals, have performed better than larger groups with larger populations. It may be that smaller, less complicated organizations are able to make changes to their care delivery system process faster. Even after several years though, smaller and physician-led ACOs seem to retain their performance advantage. While some large integrated organizations have been successful, the results to date suggest that policymakers need not encourage the creation of large, consolidated health systems to achieve improvements in care. Because consolidation may have other adverse effects like higher prices outside of Medicare due to market power, Medicare and other payers should advance payment reforms that encourage changes in delivery without requiring very large scale or consolidation.

The 2015 results underscore the challenge in transitioning to a high-quality, high-value delivery system, but also show that progress is possible. The MSSP, to date, is neither a complete success nor a complete failure. Instead it represents a process where we can gain substantive learnings and insight that can be used to improve care delivery over time.