Blog Home


Seven Policy Recommendations To Improve Quality Measurement

May 22nd, 2013

Performance measurement — if done right — can be a core activity to move the health care system to higher value for the American public, while rewarding health professionals and health care institutions for doing the right thing for their patients. Yet, policy makers, private and public, have a duty to the public, patients, and providers to get it right — to measure and report accurately and meaningfully.

Harlan Krumholz and Peter Pronovost have been among the most important contributors to the development of performance measures for quality and safety of health care.  At the same time, each has written powerful critiques of particular aspects of the current measurement enterprise with suggested improvements.  I work mostly inside the Beltway in a world of policy makers who, despite good intentions, by their actions often display a lack of understanding of the challenges associated with measures, measurement, public reporting, and pay-for-performance.  For example, the physician value-based modifier, which was mandated as part of the Affordable Care Act and now must be implemented by CMS, cannot produce a valid snapshot of an individual physician’s “value” but will be imposed nevertheless, unfortunately feeding those within the physician community who resist all efforts to improve accountability and transparency of performance.

With the encouragement of the Robert Wood Johnson Foundation, Harlan, Peter, and I joined in a collaborative endeavor to produce a comprehensive look at the state of play of performance measurement and public reporting — their conceptual underpinnings and limitations, successes and failures, and, perhaps most importantly, recommendations for major steps that are needed now to put the measurement enterprise on track to achieve its potential to improve the value of U.S. health care without doing harm.

Here are the seven major steps we propose:

1. Decisively move from measuring processes to outcomes.

The operational challenges of moving to producing accurate and reliable outcome measures, as opposed to the process measures that currently dominate performance measurement, are daunting but worth the commitment. Patients, payers, policymakers, and providers all care about the end results of care, not the technical approaches that providers may adopt to achieve desired outcomes. Simply, process measures are not strong predictors of outcomes that matter and may divert attention from work process improvements that would actually improve outcomes.

2. Use quality measures strategically, adopting other quality improvement approaches where measures fall short.

CMS’s current value-based purchasing efforts (“pay for performance” with a new name) require reporting on a raft of measures of varying usefulness and validity, which may divert providers’ attention from efforts to make culture and work process improvements that could produce larger improvements in outcomes. In their place, Congress should refocus its directives to CMS to emphasize using performance measures more strategically to improve specific quality deficiencies — relying more on promoting collaborative quality improvement activities and new payment approaches that incorporate performance measures than on public reporting and pay for performance per se.

3. Measure quality at the level of the organization, not the clinician.

Consistent with the broad movement toward population-based accountable care, performance measurement should move to the organizational or departmental level, allowing measures to assess and promote team-based care, while addressing many of the technical issues that can undermine measurement efforts at the level of the individual clinician.

4. Measure patient experience with care and patient-reported outcomes as ends in themselves.

Given the inevitable gaps in both process and outcome measures for specific areas of clinical care, it is important to realize that patient experience is ubiquitous and can be drawn upon to measure a broad range of health care dimensions. With the growing array of scientifically rigorous surveys of patient experiences with care, we now have the capacity to incorporate standardized assessments of that experience into the measurement enterprise.

5. Use measurement to promote the concept of the rapid-learning health care system.

The dissemination of quality measure data should be viewed as one prong in a multi-pronged strategy to improve health care quality. Accompanying strategies should include offering technical assistance to strengthen providers’ capacity to improve care and creating formal accountability systems.  Collaborative activities among institutions can produce insights that may elude them individually: measures can help identify top performers, and detailed analysis can then identify what distinguishes those who excel.

6. Invest in the “basic science” of measurement development.

An infrastructure is needed to gain national consensus on: what to measure; how to collect the data needed to calculate measures; the accuracy of EHR data for use in performance measurement; how to measure the cost-effectiveness of particular measures; how to reduce the costs of data collection; what thresholds to use to ensure measure accuracy; and how to prioritize which measures to collect. Establishing general standards for performance measures could help move the policy discussion from whether measures are good enough to use despite their flaws to a more fundamental discussion of how to design good measures, how to assess current measures, and whether the costs of producing better measures are worth the benefits.

7. Task a single entity with defining standards for measuring and reporting quality and cost data, similar to the role the Securities and Exchange Commission (SEC) serves for the reporting of corporate financial data, to improve the validity, and comparability, and transparency of publicly reported health care quality data.

The National Quality Forum does a good job of reviewing and approving proposed measures presented to it, but it lacks the authority to establish definitive quantitative standards that would apply broadly to purveyors of performance measures. As a result of this lack of standards, most quality measurement efforts struggle to find measures that are scientifically sound yet feasible to implement with the limited resources available; too often, feasibility trumps sound science. The field of quality measurement could advance significantly if providers and policymakers agreed on validity thresholds and voluntarily reported the validity of their quality measure data, all supervised by an entity modeled after the SEC. Policymakers will need to consider whether such an entity should be housed at the Agency for Healthcare Research and Quality (AHRQ), should be a public-private partnership such as NQF, or should be a new government entity. Such a commission could promote standardization, transparency, and auditing of the reporting of quality and cost measures.


Email This Post Email This Post Print This Post Print This Post

 to the #1 source of health policy research.

4 Trackbacks for “Seven Policy Recommendations To Improve Quality Measurement”

  1. Learning From Missed Opportunities To Diagnose US Ebola Patient Zero – Health Affairs Blog
    October 30th, 2014 at 2:40 pm
  2. Amitech Solutions » Seven Healthcare Policy Recommendations To Improve Quality Measurement
    November 22nd, 2013 at 10:31 am
  3. Measuring Health Care: Using Quality Data for Operational, Financial, and Clinical Improvement | WWW.DINFOSITE.COM
    August 23rd, 2013 at 7:18 am
  4. Policy recommendations on quality measurement: my reactions » Health Business Blog
    May 30th, 2013 at 8:03 am

11 Responses to “Seven Policy Recommendations To Improve Quality Measurement”

  1. cqueram Says:

    I would like to add my thanks to the authors for a timely and stimulating assessment of the evolving and dynamic field of performance measurement. Given the increasing importance of valid and reliable performance data as a foundation for the panoply of “levers” (public reporting, payment reform, quality improvement, population health, and consumer engagement) for achieving transformational change in health and health care, the recommendations represent an important contribution to the marketplace of ideas for improving our current work.

    That being said, I have a number of comments and reactions:

    1. While I agree with the recommendation to move “decisively” toward outcome measures, the authors ignore two other dimensions of the nation’s quality strategy — affordability and population health. Perhaps this is a reflection of “Health Care Performance Measures” as suggested in the title. However, for all of our gains in measuring performance over the past decade, the focus has been almost solely on process and intermediate outcome measures of clinical quality. Future investments in performance measurement development should strive to equitably allocate human and financial resources to make progress in addressing the other equally important dimensions of the National Quality Strategy,

    2. The authors offer an eloquent rationale, both philosophically and technically, of focusing measurement at the organizational level. Nonetheless, it overlooks the reality that what many consumers seek is information on “their” practitioner. Our organization — the Wisconsin Collaborative for Healthcare Quality — recently participated in a pilot project that featured a subset of the measures reported on our web site ( as an insert for Wisconsin subscribers to CR. In a survey designed to elicit feedback from the subscribers, we were surprised at the sheer number of verbatim comments requesting data on individual providers. The philosophical, technical and political challenges associated with measuring at this level are real, complex, and daunting; however, it is seems preferable to proactively engage with and shape the manner in which this is done versus ignoring it and hoping it will go away.

    3. The recommendation promoting the concept of rapid-learning health care systems rightly recognizes the importance of fostering learning collaboratives that facilitate the sharing of “best practices”. As noted by the authors as well as several other commentators, there exist today many examples of multi-stakeholder, regional health improvement collaboratives who share a common vision of accelerating achievement of the three aims of the National Quality Strategy. These organizations are ideally positioned to play a critical role in supporting the larger goals of CMS in engaging providers in generating valid and reliable measures of performance and using these as the foundation for focused improvement initiatives.

    4. Recommendation #7 is remarkably similar to one made in 1998 by President Clinton’s Advisory Commission on Consumer Protection and Quality in the Health Care Industry. (Author’s disclaimer — I was a member of the Commission.) The Commission saw the need for a combined public and private sector approach to establishing a national quality strategy and goals and adopting national standards for measurement; the public-sector agency was to be created by Congress and modeled after the Securities Exchange Commission and the private-sector entity was to function in a manner similar to the Financial Accounting Standards Board. The latter was created in the form of the National Quality Forum; the former was never acted upon by Congress. As a result, the NQF has assumed the role of “convener” in an attempt to accomplish much of what was to have been the province of the SEC-like entity. While laudable progress has been made through the ensuing development of the National Priorities Partnership, the authors renewed call for the SEC-like entity may well cause a healthy re-examination of the effectiveness of the current approach.

  2. Gail Amundson Says:

    Many thanks to Bob Berenson and colleagues for their paper on performance measurement, to RWJF for their commitment to advancing health care quality, and to the health care leaders who have added their insightful comments here. It is terrific to see this complex and important conversation developing. We all come at this so differently.

    It has been my privilege to lead innovative measurement work at HealthPartners, at Minnesota Community Measurement, and, most recently, at Quality Quest for Health of Illinois. I disagree with the authors recommendation that reporting be focused on healthcare systems and not individual physicians. BOTH are important.

    Quality Quest’s Colonoscopy Quality reporting at is an instructive case in point. Colonoscopy, like many elective procedures, is performed by individual clinicians, at a specific point in time, at a specific facility. It is fair and appropriate that individual results be made available to the public.

    In 2007 a quality team comprised of clinicians and patients, devised a strong quality measure that assesses whether ALL of the important quality aspects of colonoscopy were met for each individual patient. ‘Was my colonoscopy done correctly’ is a concept consumers understand. Local employers asked local endoscopy centers to participate. Quality Quest aggregates data submitted quarterly on all screening and surveillance colonoscopies. Quality results improved dramatically when individual physicians results were made available publicly. They had not changed significantly over the two years that results were shared confidentially.

    Public reporting provides recognition for those doing well and motivation for those that are not. It works at the system level and it works at the individual clinician level.

    There are terrific examples of cutting edge quality measurement and reporting. Regional Health Improvement Collaboratives are at the center of much of it. The biggest challege is spread.

  3. msepulv119 Says:

    This blog summarizing the key recommendations from the RWJ report by Berenson RA, Pronovost PJ, and Krumholz HM is a timely and most welcome addition to a complicated and complex subject. It is complicated because there are many parts contributing to clinical care outcomes including people, places and information systems, and it’s complex because the parts are semi autonomous, dynamic and the behavior of some causes changes in the others. This does not by any meansimply that performance assessment can’t be done well, just that it requires good multi-source data, good judgement, context consideration, and assessment of the measures themselves for continuous improvement and iteration.

    In this blog Bob makes subtle reference to two important concepts that are noteworthy for performance assessment and these involve the individual and the organizational level as pointed out by the comments by erShaffer and others. These concepts are essentially that it is problematic to assume that dynamics at the individual level transfer to the larger organization (so-called composition fallacy) or that dynamics at higher levels of aggregation like the organizational transfer to the individual level (sometimes referred to as ecological fallacy). Chris Cassel provides a spendid description of the importance of organizational level assessment as key to clinical outcomes improvement in a complex system like health care delivery, and others rightly note the importance of individual level performance assessment. Both are essential and should be linked unambiguously, transparent and clearly understood, timely and relevant to what people and systems really do day in and day out. In this way these measures promote acceptance, continuous learning and quality improvement for patients health and overall system performance.

    Kudos to RWJ and the senior authors for advancing thinking on a tough subject.

  4. ershaffer Says:

    Re: #3 Both systems and individual clinician performance can and must be evaluated and reported by users, Evaluation instruments must vary depending on the degree and level of organization within which the clinician practices. My letter from Kaiser querying the performance of a randomly assigned MD, contacted online after hours e.g., misses my ability to report the glitches in their pharmacy and call-back systems – because they have systems. The same queries re: the specialists I see regularly within that system would get very different answers. And posed to a user in a less organized or virtually unorganized setting – most of the U.S. – different still.

  5. David Williams Says:

    Provocative recommendations and lively discussion here. #4 (patient experience) is my favorite, but the authors miss the boat and undermine their recommendation on patient experience by insisting on measurement at the organizational level rather than at the level of the individual doctor.

    For a full review see the Health Business Blog

  6. Christine Cassel, M.D. Says:

    We should thank Berenson, Pronovost and Krumholz for this thoughtful and comprehensive analysis. Those of us focused on assessing quality of care and providers’ performance should deeply consider the paper and its recommendations.

    I want to specifically address recommendation #3: to measure quality at the level of the organization rather than the individual clinician. Given that quality measurement focuses on outcomes of clinical care and on patient experience, it makes great sense to recognize that in most situations a given patient’s clinical outcome relates to a number of different care providers. In medical training, much attention is focused on training physicians to be more effective members of clinical teams–a direct recognition of this reality. And yet consumers tell us they want to be able to know “how to find the good doctors”. Information on a physician’s licensure and certification status is generally available on public websites. Commercial “rating” sites are also prevalent, but none of them have systematic, comprehensive or rigorously defined assessment criteria.

    It is important for everyone involved in assessing and measuring care to work with consumer and patient advocates to make meaningful, reliable information available. At the same time, we must engage with payers–including Medicare and Medicaid–to recognize that some measures are potentially misleading and should not be used for assessing a given physician’s care, or determining his or her payment.

    Accountability is an important part of professionalism, and we will be best served by an accountability framework and measurement applications that provide consumers with meaningful information, and providers with information that is relevant to their practice and actionable for improvement.

  7. Elizabeth Mitchell Says:

    Many thanks to Bob Berenson and colleagues for their excellent paper on performance measurement. This is a timely and thoughtful article on the state of the art of performance measurement and our remaining challenges of improving care. As we increasingly- and rightly- demand transparency and accountability of our healthcare system, we must have fair and reliable standards by which to measure good care and good value.

    The authors make many critical points, most importantly:
    1) We must accelerate the move to outcome measures. This not only allows greater flexibility by providers to innovate to improve care, it meets the needs of patients whose primary interest is outcomes- not how they are achieved.
    2) Different measures have different uses. The authors rightly recognize the different uses of measures- process measures may be valuable for internal quality improvement to create actionable information for providers to change care for the better while outcome measures are appropriate for public reporting and payment.
    3) Patient reported outcomes and patient experience are critical measures of care. In a patient-centered system, this is the gold standard of information- can people function well in their daily lives and do they feel healthy. It is notable that we have highly regarded tools and instruments available to measure patient experience and outcomes- like CG-CAHPS-that to date have not been widely used. Patient experience and patient reported outcomes are particularly important as we transition to new financial incentives to contain costs. This change cannot be made at the expense of access, outcomes or patient experience and we need to protect against likely unintended consequences. In the fee for service system we predictably saw hazards of overuse, but in new systems that limit costs, we must protect against underuse, poor patient experience or inappropriate limits on care. Measures of patient experience and outcomes will be good bellwethers that care quality is protected as costs are contained.

    The key point that deserves even greater emphasis is the recognition that measure use in the field is what drives impact and improvement. No measure, regardless of its validity and significance will improve care without effective implementation. Each measure, once developed, requires a constellation of implementation activities to have impact. Even with the best data and measurement, care will only be improved if providers lead care transformation and are supported by reformed payment. Clinicians must accept measures as meaningful and work to change care to improve. Purchasers must tie payment and incentives to measures for there to be a business case for providers to make needed changes. And patients must be able to understand and ultimately use performance information on key measures to choose the best care and to effectively partner with providers to improve their own health outcomes.
    This work cannot be done solely from Washington and there is no ‘one size fits all’ national solution. Implementation and measure use can only happen locally and must reflect local priorities. Quality and costs vary dramatically across the country and across communities. This variation has been documented for over 30 years. Given regional variation, there is no single solution to improving care and reducing costs- improvement opportunities and priorities vary by region. Communities must identify their priorities, data must be collected and reported, and providers must use data to change practice – all of which is facilitated by local relationships and support. Care improvements require coordinated and aligned change from all parties so that payment can support optimal care delivery, incentives can support optimal utilization and that reliable information is available for all parties to make improvements. A multistakeholder forum where transparent data can be shared with all stakeholders is an important forum to both identify and understand opportunities for improvement, and to work together effectively for its achievement.

    Regional Health Improvement Collaboratives (RHICs) exist across the country to bring all stakeholders to a common table to improve care in their communities. A Regional Health Improvement Collaborative provides a neutral, trusted mechanism through which all of the key healthcare stakeholders in a state or region — physicians, hospitals, health plans, employers, and patients — can plan, facilitate, and coordinate the many different activities required for successful transformation of its healthcare system. There are over 40 Regional Health Improvement Collaboratives in the U.S. carrying out one or more of these functions in order to help their communities improve the quality of healthcare services while controlling skyrocketing costs. The leading Collaboratives are members of the Network for Regional Healthcare Improvement (NRHI). NRHI members provide programs to support improved healthcare for over 110 million Americans – 35% of the U.S. population. Many To learn more about NRHI and the important work of its members implementing improvement on the ground visit and watch videos from our February Summit featuring Collaborative leaders. As our nation takes on the challenge of care redesign, measurement and payment reform, Regional Health Improvement Collaboratives should be considered key to truly understanding which measures are meaningful to communities, to physicians and to improvement.

    The authors point to the important work of the National Quality Forum (NQF) as the organization that has ‘come closest’ to being the body of expertise with responsibility for addressing the science of performance measurement. NQF has played a vital role evaluating and identifying valid and reliable measures for use in the field. However they have not – for many good reasons- connected with the field to better understand implementation and measurement: what measure gaps exist, how measures are used and what barriers to measurement exist. The authors note the barriers, including cost, to extending NQF’s work, but I suggest that by partnering with RHICs who have expertise in measure use and relationships with community stakeholders, a natural feedback loop exists and a ready partnership to rapidly accelerate improvement in both measures and measurement is available. The challenges of measurement will not be eliminated but by having all stakeholders at the table, we may mitigate the greatest risk of inappropriate use, develop a culture of collaboration and learning, and accelerate the development of an empirical base of what measures are effective in what settings under what circumstances.
    NRHI members know which measures are most effective, which measures can be implemented and what their impact is. Potential national-regional partnership activities could include:

    • Advisory Group on Measure Gaps in Communities: NRHI members work day to day with those using measures and are well positioned to understand what users need. NRHI could regularly inform CMS and NQF about measurement priorities – by constituency- and community readiness for implementation. NRHI members represent a range of communities from Minnesota to Boston to New Mexico reflective of the range of practice in the field. In addition to needed measures, this group could identify what else is needed- data, guidance, resources, etc. to make measurement efforts successful and could serve as the front end of a performance improvement Feedback Loop.
    • Measure Use Technical Assistance Guide: Drawing on local experience, NRHI members could share lessons for working with providers on using measures for improvement, purchasers for measures of accountability, and consumers for measures of choice and engagement. NRHI could highlight and quantify which measures combined with which strategies have had measurable impact, what Quality Improvement, purchasing or engagement strategies are most effective for which measures. This should also include which measures are NOT working and don’t have the intended impact. These could be developed into a ‘Best Practice Catalogue’ for technical assistance to ensure measures come with ‘instructions’ or guidance for best use. Because care varies so dramatically, ultimately this could become a community improvement menu to meet local priorities- sharing what measures and improvement strategies are most effective to meet local priority needs.
    • A Guide to Understanding Barriers to Measure Use and How to Overcome Them: NRHI members have first hand experience obtaining and managing multipayer data, working with physicians and other stakeholders to understand and overcome resistance to measure use; and engaging patients in the importance of quality measurement. Compiling these lessons and success stories could inform and accelerate effective measure use.
    • Measure Test Beds: Many NRHI members have strong community partnerships with providers and other stakeholders and could pilot use of priority measures. This information could be systematically shared to accelerate filling measure gaps and measure evaluation.
    • Impact Evaluation: NRHI could evaluate measure use and effective implementation strategies in the field and their impact on quality, population health and cost, developing needed regional benchmarks and an important reference over time to document progress and demonstrate the importance of effective measure use. This could create a compelling case for change and the role for measurement in its achievement.

    These and other partnership ideas could be further developed tying national measure selection and approval to local implementation and impact. There are strong and effective collaboratives around the country using data with employers, patients and physicians and collectively we are able to partner with national policymakers to implement change on the ground. We have the tools, abilities and relationships with all stakeholders to do the hard work of transforming care. This ‘bridge’ from national policy to local implementation may reduce measure burden, accelerate improvement, and ultimately facilitate achievement of the Triple Aim. RHICs are the innovation infrastructure needed to transform US healthcare.

  8. Barbra Rabson Says:

    Bob Berenson and his colleagues did a great job outlining the challenges and opportunities for performance measurement, and they make very appropriate recommendations for creating a more effective set of measures. However I was struck by how “medical system” oriented his comments were, and how quickly he and his colleagues dismissed the importance of public reporting to inform consumer choice. It is because past and current reporting efforts have a “negligible impact on the selection of providers by patients and families” that we have a responsibility to do a better job improving our quality measurement programs and to make sure that not only the measures, but the unit of analysis, reporting mechanisms, and display techniques are more meaningful to the public. I do commend Dr. Berenson and colleagues for flagging the importance of measuring patient experience and patient reported outcomes. These are measures that patients universally care about, and they can help engage patients and the public to become more interested in other performance measures. But we have a long way to go to get measures in front of consumers in a way that is meaningful to them.

    Last summer, working jointly, Consumer Reports and Massachusetts Health Quality Partners (MHQP), the regional health improvement collaborative that I run, published a special Massachusetts report entitled “How Does Your Doctor Compare?” along with a 24-page insert that includes ratings of nearly 500 primary care physician practices from across the state. The ratings are based on data from a comprehensive patient experience survey conducted by MHQP that included 65,000 responses to a survey asking adult patients, and parents of pediatric patients about multiple, specific aspects of the care they received from their primary care doctor and their doctor’s office staff. The public response to this issue of Consumer Reports issue has been astounding – newsstand sales of the magazine where the results were published rose 110%; MHQP’s social media traffic, including Twitter and Facebook increased over 200%, and we estimate that the report was viewed 4.5 million times. In addition, the Consumer Reports user survey showed that 39% of those who read the report plan to change what they do or say during future visits and 25% said they felt better about the state of health care in Massachusetts after reading this report. The public use of and interest in the report was repeated in other markets, after Consumer Reports jointly released quality measurement results with Minnesota Community Measurement and Wisconsin Collaborative for Healthcare Quality. This is a good indication that it is possible for our quality measures to have an impact on patients and families, but it requires us, as an industry, to move closer to what consumers want and expect from this information, while we remain responsible evaluators of care. If we are to come close to realizing the hopes that “healthcare transparency” will help transform our health care system, we must acknowledge that we need fresh new approaches to quality measurement and public reporting. And we need to look to patients and communities for guidance – looking inward to our medical system will not get us there.

    I would add the following to Dr. Berenson’s recommendations:
    Invest in models that will make quality measures more meaningful to the public and that have a greater impact on patients and families. This could involve:
    • including patients and families in the selection of measures that are meaningful to them;
    • exploring which measures are reliable at the individual clinician level (e.g. patient experience measures) since this is what patients want, and when the unit of analysis is only reliable at the organizational level, in order to support consumer confidence in selecting clinicians, finding ways to share with the public how much variation occurs among the clinicians in an organization;
    • testing new reporting mechanisms and data display techniques for health care information by working with trusted consumer organizations that the public relies on to support other consumer decisions ;
    • acknowledging that there is clinician discomfort with some of the above recommendations to share performance data more widely, and working with physician leaders toward their better understanding of patient and family views about need, desire, and use for quality information. MHQP’s experience that 25% of the Consumer Reports readers felt better about the state of health care in Massachusetts after reading this report (only 3% felt worse and 67% felt the same) should be an important lesson that health care transparency can help improve consumer confidence in their health care system.

  9. Joanne Lynn Says:

    In the same spirit of building on a good report, let me suggest yet a few more additions. Perhaps #11 would be to measure, at least for children, elderly people, and disabled persons, at the community level. These people, when they are patients in health care, cannot move far, and much of the services needed are tied to residence. So, what we need is something like “the lifetime risk of pressure ulcers, given that you live in Pittsburgh, and whether it is improving and how it compares with other cities.”

    Perhaps #12 would be the need to measure, at least for people living with complicated situations associated with multiple chronic conditions and aging, the merits of their care plan in terms of its optimally advancing their priority concerns. This is a bit different than merely measuring outcomes, or patient satisfaction and patient-reported outcomes. In this endeavor, we need to have documented the patient’s situation, strengths and challenges, likely courses with different plans of care, and preferences among them. Then the outcome is actually the life lived, and the evaluation is how well that life served the patient’s goals and priorities, given the options. Since part of the life lived is the development of the plan of care, the process of care planning, evaluation and re-doing that process is a legitimate target of the evaluation itself. In many ways, this is the most missing part of health care for the seriously chronically ill at this time.

    And perhaps #13 could be data needs, e.g., for uniform assessment (e.g., the CARE instrument for the frail elderly), rapid data availability, and adequate historic data. One needs a time series to implement upper and lower control limits to guide quality improvement, one needs data quickly enough to guide interventions, and one needs assessment to be uniform across sites and time in order be able to see downstream effects.

  10. Karen Feinstein Says:

    Thanks to Berenson, Pronovost, and Krumholz, for surfacing the dilemma of often contradictory information – and misinformation – related to performance measurement. Their call for better science, standardization, transparency, and accountability in performance measurement could “reduce the noise” generated by multiple measurement development and reporting entities, which is highly confusing to patients, providers, and purchasers seeking information on which to act.

    Our mission at the Pittsburgh Regional Health Initiative, as with other regional quality improvement collaboratives, is to provide performance information that is useful to providers and to payers; data that “cause providers to engage in broader approaches to quality improvement activities.” Improvement collaboratives exist to surface and prove better ways to deliver health care and achieve better outcomes. But we can only sell our ideas for better care if we have credible data that substantiate our discoveries.

    There are a number of obstacles to applying credible, actionable measures and measurements for quality improvement and, hence, achieving convincing outcomes of experiments and demonstrations. What could slow progress is a shortage of healthcare professionals able to play in this brave new data-driven world. Not only are QI tools needed, but physicians and nurses, pharmacists, medical assistants, and others all need to master the fundamentals of data collection, analysis, application, and evaluation. They have to be able to recognize the limitations of any data, understand the implications or qualifications of findings, and know how to engage in substantive demonstrations and hypothesis-testing.

    PRHI and six other organizations have recently received the designation of Qualified Entities. We are pleased to engage in this new initiative to apply previously unavailable Medicare data to public reporting on cost and quality. Could we launch these activities in a “learning collaborative” mode to accelerate our understanding of what data we report to consumers and purchasers, as well as providers and plans, and how we can best report it, to stimulate better decision making, quality improvement initiatives, value based purchasing and payment?

  11. Harold Miller Says:

    The paper by Bob Berenson and colleagues on healthcare performance measurement should be a must-read for everyone interested in improving the quality and controlling the costs of healthcare, but particularly for policymakers in both federal and state governments, employers and other purchasers of healthcare, and health plans. The paper provides an excellent and much needed explanation of the challenges and opportunities in using measures of quality and cost in the most common types of public reporting and pay for performance programs, and it makes seven very appropriate recommendations for creating a more effective set of measures and using them more wisely that should receive high priority for implementation.

    However, while the paper states that its analysis and recommendations are focused only on how measures should be used for public reporting and pay for performance, I fear that the recommendations will be misinterpreted as being more comprehensive than they were intended to be, and so I offer three additional recommendations which I believe complement those in the paper and create a more comprehensive approach to using measurement to improve quality and control costs.

    #8: Encourage and support the ability of clinicians to use process measures as a tool for improving outcomes, while removing them from pay for performance programs.

    It is difficult, if not impossible, to improve outcomes and maintain high performance without implementing the appropriate care processes that will lead to good outcomes. As Berenson and colleagues point out, no one knows enough about what processes work best or the extent to which different processes are needed for different types of patients and circumstances. However, while that means we need to move away from using process measures in public reporting and pay for performance programs, we should not throw the baby out with the bathwater by not measuring processes at all. Those physicians and hospitals that have made the most dramatic improvements in outcomes are those that measured both what they were doing and what the outcomes were, and who continuously worked to determine which processes resulted in the best outcomes. National improvements in quality and cost will likely proceed at a much more rapid pace if providers can measure and compare their performance on various process measures as well as outcome measures, so having standardized measures of processes and mechanisms of sharing information on process measures will still be valuable. But that does not mean that it’s appropriate to use those process measures in simplistic and slow-to-change P4P programs.

    #9. Design payment systems that truly fix the problems with the current fee-for-service system.

    The most common payment “reforms,” particularly P4P and shared savings, are based on the flawed notion that physicians and other healthcare providers need financial incentives to improve their performance or that they’ll ignore measures unless they’re tied to payment. If you ask physicians, nurses, and others on the front line of healthcare, you’ll readily find that they want to improve quality for their patients and reduce unnecessary costs, and they don’t need financial incentives to do so. The reasons they often don’t implement more effective processes, however, is because the current payment system either fails to pay for the better processes or it actually penalizes them for improved outcomes. For example, there are dozens of examples of how patient-centered medical home programs have enabled primary care practices to hire nurse care managers to help patients with chronic disease better manage their conditions and reduce hospitalizations, and to be more proactive about preventive care through effective phone calls, emails, etc. rather than relying solely on office visits. Yet the fee-for-service system pays only for office visits, not for the nurse care managers or non-visit based care that can improve patient health and reduce costs, and a primary care practice that implements more effective care could easily go bankrupt in the process. P4P and shared savings systems don’t fix those underlying problems; they simply add a new, and fairly thin, layer of “incentives” on top of a fundamentally flawed payment system, so it’s no wonder the available evidence shows, as the Berenson/Pronovost/Krumholz paper points out, that P4P systems aren’t very effective.

    Hopefully, though, no one will interpret the Berenson/Pronovost/Krumholz paper as saying that all we need to do to improve payment is to insert outcome measures, team-based measures, etc. into current P4P systems. Instead, we need to totally restructure payment systems to give physicians and hospitals the flexibility they need to completely redesign care as well as the accountability to ensure that patient outcomes are improved and costs are reduced.

    #10. Identify the barriers to improvement and measure the extent to which they are being removed.

    Healthcare providers alone can’t fix the healthcare system. Patients also need to change the way they manage their health and choose healthcare services and providers. Purchasers need to change the way they pay providers and the benefit designs for patients they pay for. Health plans need to reduce the administrative burdens on providers. State and federal regulators need to remove regulatory barriers to change. All stakeholders need to do their share to completely redesign healthcare delivery, and yet most of what we measure today is only what healthcare providers do. Multistakeholder involvement in measure development, endorsement, and use shouldn’t mean just that all stakeholders decide on how healthcare providers should be measured, but rather that all stakeholders agree on what each stakeholder should be doing to change to support each other and then measuring the extent to which each stakeholder is actually doing its share.

    A simple way to start is to ask physicians, hospitals, and other healthcare providers what barriers they face in improving outcomes and in implementing the processes that are known to improve outcomes, and then develop and report measures of progress in removing those barriers. For example, although some progress is currently being made on reducing hospital readmissions, a lot faster progress would be made if Medicare and health plans used payment systems that paid for better primary care and helped hospitals maintain positive operating margins in the face of lower revenues from fewer admissions, and if patient benefit designs were changed to reduce or eliminate copayments for chronic disease maintenance medications and post-discharge physician visits. So in addition to measuring readmission rates, we should be measuring whether payers have implemented payment systems and benefit designs that actually overcome the problems providers face in changing care delivery to reduce readmissions. We can’t find that out by surveying payers as to whether they think their payment systems and benefit designs are effective, we can only find that out by asking the healthcare providers who are working to improve their performance. (For a more in-depth look at the barriers to more effective payment systems, see “10 Barriers to Payment Reform and How to Overcome Them” available at

    It’s absolutely true that you won’t improve what you don’t measure, so performance measurement of healthcare providers is a necessary step to transforming our healthcare system. The recommendations by Berenson, Pronovost, and Krumholz will help us develop more effective performance measurement and they should be a priority for implementation. But the fact that you can measure something doesn’t automatically mean you can improve it, so we also need to also make it a priority to remove the barriers providers face in improving performance.

Leave a Reply

Comment moderation is in use. Please do not submit your comment twice -- it will appear shortly.

Authors: Click here to submit a post.