On September 23, 2014, the Centers for Medicare and Medicaid Services (CMS) released Year 1 Quality Performance results for Accountable Care Organizations (ACOs) that began participating in the Medicare Shared Savings Program (MSSP) in 2012 or 2013. Another report, released shortly before, outlined financial performance of the ACOs and showed that only 49 ACOs, or 22 percent of those ACOs, qualified for shared savings payments by successfully reducing total spending.

Opportunity for continued quality improvement aside, a troublesome snag for the program could be a very low correlation between improved quality and earned savings: our analysis shows that, in performance year one, improved quality and earned savings only correlate at 8.6 percent, so low that it is statistically insignificant (Figure 1).

In practice, this means that better quality is not associated with better financial results. Twenty-one of the 49 ACOs that did earn shared savings actually scored below the average quality of the group. For the first year, quality outcomes did not affect the size of shared savings payments, but in future years ACOs that perform poorly on quality measures will lose a portion of any shared savings.

Figure 1: Shared Savings Payments and Total Quality Score


To earn their first years’ Quality Incentive Payments (QIP), a payment to encourage meeting quality benchmarks, ACOs were only required to submit their performance on the 33 CMS quality metrics they agreed to be accountable for — actual performance on those measures was not taken into account. Most ACOs were successfully able to report quality data in performance year 1, with only 11 of 220 organizations—all of which were physician groups—failing to adequately report.

In future years, any earned savings will be dependent upon reaching specific performance benchmarks on those quality metrics. If first year quality data were evaluated using third year payment calculations, even the ACO with the highest quality performance metrics would have lost 8 percent of its share of savings because of missed benchmarks. At worst, one ACO would have lost nearly 60 percent of their shared savings earnings because of poor quality performance. On average, ACOs would have qualified for 74 percent of their share of total savings.

Reporting ACOs also show significant variability on quality performance, as seen in Figure 2. Not only was there a wide range of quality performance scores, but there was also considerable variability within individual quality measures: ACO scores ranged from the 30th to the 90th percentile of the comparison group for 23 of the 33 quality measures. Because of the huge variability from ACO to ACO in strengths and weaknesses, CMS and the ACOs themselves will need to be creative when looking to identify broad problem areas to target for performance within the program. Future work will need to identify different strategies that help lead to success with individual measures.

Figure 2: Histogram of Composite Quality Scores of MSSP ACOs


Results for the Group

Though the first year performance data indicates variety in overall quality performance scores, the group exhibited significant similarities in some measures. Overall performance was high for the group on readmissions (measure #8), with most of the participants earning the maximum points possible for reducing hospital readmissions within 30 days of a patient’s initial admission.

As hospital readmissions is an important cost-saver and is often touted as a focus of new CMS programs, it is no surprise that ACOs were most successful in this category. In contrast, the majority of reporting ACOs performed very poorly for first-time admissions measures, many earning no points at all for admissions measures like heart failure and asthma chronic obstructive pulmonary disease (measures #9 and #10).

ACO scores on measures tied to CMS’ Consumer Assessment of Healthcare Providers and Systems (CAHPS) patient satisfaction scores, like Health Promotion and Education and Shared Decision Making, were also not impressive. Such scores tended to vary considerably among ACOs, which had performance ranging across the board, with less than half of ACOs scoring at least in the 70th percentile.

Despite the low aggregate scores on all measures, CMS said in its release of the report that Shared Savings Program ACOs have improved on 30 of 33 measures.

Regardless of current measure improvement, CMS recently amended parts of its proposal to drop some existing quality measures and adopt new ones. This will require ACOs to shift priorities in their quality improvement initiatives, potentially adding difficulty for the ACOs to continue improvement and earn greater shares of performance payments in the future. In the new set of measures, CMS retired or replaced eight measures and added eight new measures, resulting in a total of 33 measures ACOs will need to meet for Quality Incentive Payments.

Based on our ongoing tracking of ACOs, we assigned provider types to each of the ACOs. We categorized each MSSP ACO as being managed by a physician group (119 ACOs), by a hospital system (26 ACOs), or by both a physician group and a hospital system (75 ACOs). We used these distinctions to evaluate differences by type of ACO.

The weighted total quality score of each type of ACO is available in Figure 3 and averages for individual quality measures is available in Figure 4. Physician group ACOs performed significantly worse than ACOs managed by both hospitals and physician groups (p<.001) and ACOs managed by hospitals (p=.046).

Figure 3: Average Composite Quality Score Per ACO Type


Figure 4: Average Quality Scores By ACO Type




Similar to their overall scores, ACOs that included both physician and hospital groups fared the best in 13 of 33 quality measures. Those ACOs did exceptionally well in several categories and earned 10 more points than the others in shared decision making (measure #6) and in qualifying for an EHR incentive payment (measure #11).

In contrast, physician group ACOs were generally poorer performers and received the lowest average score on 24 of 33 measures. For instance, vaccination and immunization rates were very low among all MSSP ACOs, with most groups barely meeting the 50th percentile of performance—which could be a warning sign that basic preventive care and education is an area for improvement. On these measures, physician-led groups performed even worse, reporting nearly 10 points below both the other groups. Physician-led ACOs scored in the 48th percentile for population receiving pneumococcal vaccinations (measure #15) and about the 41st percentile for their population receiving flu immunizations (measure #14).

Physician group-led ACOs also performed much worse than others for composite measures. For the diabetic management composite measure, physician group ACOs performed 22 percentile points lower than the leading group and for the coronary artery disease composite measure they performed nearly 15 percentile points lower than the leading group.

Physician group ACOs did perform better than the reporting cadre with well above-average rates of getting their patients in to screen for blood pressure (measure #21); however, the report also showed that their patients have a much higher rate of hypertension (measure #28) despite screening efforts. This measure comparison hints that this group of ACOs were more effective in getting their patients in the door for blood pressure screenings than their counterparts, but weren’t able to improve patients’ health with the same effectiveness or may have had sicker patients. Physician groups also reported moderately better in the percent of their beneficiaries with diabetes whose blood sugar was in poor control (measure #27).

Readmission rates were a consistent win with little disparity when comparing the average score of the ACOs, but first-time admission rates for heart failure was an issue for hospital systems, which performed worst and most erratically on the measure. Similarly, hospital ACOs on average performed on par with admissions for COPD and asthma but had wide variation among the group — while systems with both hospital and physician groups performed the most poorly on that measure. This could suggest that some hospital ACOs are having difficulty shifting away from an admissions focused revenue model.

We also looked at regional quality performance differences among the ACO group, illustrating moderate but noticeable differences. The 12 ACOs in the West North Central region fared the best, scoring an average of 79.58 points, while the 19 ACOs in the Pacific region reported the lowest score of 70.05 points (see Figure 5). There was a statistically significant difference for performance in different regions of the country (p=.005). The difference continued, even when adjusting for organization type. Differences among these regions could be helpful in future reporting years when identifying trends and opportunities for improvement.

Figure 5: Average Score by Geographic Region


Looking Forward

While ACOs, overall, did above average on quality performance, the results indicate that there is considerable room for improvement. Of the $296.8M in earned savings distributed this year, CMS would have withheld $71.1M based on failure to achieve full-standard quality benchmark, which is about 24 percent of the withholding pool based on the full pay-for-performance that begins in year 3.

Because of low quality performance, each individual type of ACO would have also lost out on shared savings. On average, a Physician group would have forfeited about 24 percent (about $32M total among that group of ACOs) of their earnings, Hospital-led groups would have lost about 22 percent (about $11M) of their earnings and ACOs with both Hospital and Physician Groups would have lost about 24 percent (about $28M). See Figure 6.

Figure 6: Total Actual Earned Savings Versus Quality-Reduced Shared Savings


In the next two years of the agreement, those losses of earned savings could become reality if significant improvements aren’t made. ACOs will be more at risk for performance; in year 2, pay for performance applies to 25 measures while 8 measures will remain on a pay-for-reporting basis. In year 3, pay for performance applies to 32 measures, meaning only one measure will be based on merely reporting.

Without significant improvement, many ACOs could lose a significant portion of their expected shared savings. While ACOs generally did well above average for quality, these early measures show that there is room for improvement.