Cost-effectiveness analysis (CEA) is an important tool for assessing and pointing the way toward better health care efficiency. The number of published CEAs on health care interventions has blossomed, averaging 34 per year in the 1990 to 1999 time frame and increasing to more than 500 per year in the 2010 to 2014 period. Advances in computing and data storage technology, along with a workforce with the appropriate statistical, health economic, modeling, and computational skills, will enable continued growth in the application of CEA, which can be used for the benefit of providers, payers, patients, and governments. The founding and rapid increase in membership of the International Society for Pharmacoeconomics and Outcomes Research (ISPOR), for example, is a testament to the growth and potential for the use of CEA and other quantitative assessments of the value of various health care technologies.

Researchers have noted, however, that the application of CEA across different types of health care interventions is imbalanced, with CEAs disproportionately conducted on pharmaceuticals. Data from the Tufts Medical Center Cost-Effectiveness Analysis Registry indicate that 46 percent of recent CEAs evaluated pharmaceuticals, yet less than 15 percent of personal health care expenditures in the United States are devoted to prescription drugs. In contrast, only 22 percent of CEAs evaluated surgical or medical procedures.

While CEAs of pharmaceutical interventions are vital, the disproportionate application of CEA to certain types of interventions can lead to less efficient use of health care dollars overall. Disproportionately less attention on medical and surgical procedures may mean that relatively more of those procedures with relatively high (that is, unfavorable) cost-effectiveness ratios will continue to be in widespread use. There are two sources of possible inefficiency here: 1) procedures may be used despite their being cost ineffective; and 2) opportunities for substituting more cost-effective drugs for less cost-effective procedures (and vice versa) are less likely to be taken because cost-effectiveness information is simply less likely to exist for procedures.

In this post, we address the question of why the imbalance may exist. We also discuss how both the methods and applications of CEA technologies and the policies of government and private entities might change to address this issue.

Why Are Medical And Surgical CEAs More Challenging Than Drug CEAs?

A central lesson from economics is that, other things being equal, more expensive items will be purchased less frequently than less expensive ones. Because it may be more difficult (and thus more expensive) for researchers to conduct CEAs on procedures than on pharmaceuticals, they may conduct fewer on procedures relative to their share of the overall health care market.

Two key ingredients likely make it more challenging to conduct CEAs on medical and surgical procedures compared to drugs: 1) data availability, and 2) methodological issues—both of which are relevant for measuring cost and effectiveness. Both the accessibility and nature of the data may favor drug application CEAs. There are more randomized clinical trial (RCT) data available for pharmaceuticals, for example. Moreover, when RCT data are lacking, more complicated, and possibly less reliable, statistical techniques must be used to produce measures of effectiveness from observational data.

Measuring Costs

At first blush, it may seem easier to measure costs for drug versus non-drug CEAs. The reality is not so clear. For the direct costs of the intervention, it is deceptively easy to obtain the nationwide price of a drug—either a published wholesale acquisition cost or an average wholesale price. A problem, however, is that the true net prices of brand-name drugs are seldom observed because of widespread and substantial manufacturer rebates to payers. Because these rebates are private transactions when commercial payers are involved, it may not be possible for analysts to know the true net price of a drug with any accuracy. (The Second Panel on Cost-Effectiveness in Health and Medicine recommends that, in the United States, analysts use the Federal Supply Schedule price of a drug as a proxy for social marginal costs, although the panel notes that more work is needed to determine the extent to which this price reflects the true social marginal cost.) Also, additional monitoring costs, including follow-up visits, require evidence from additional sources.

It may also appear straightforward to measure the cost of adverse events in pharmaceutical versus other applications of CEA. Because the RCTs used as part of the drug approval process often measure adverse events, safety data collected in trials are available for use in CEAs. However, estimating the future costs of those adverse events requires additional data and assumptions. Thus, while the frequencies of adverse events can be taken from available RCT evidence, the overall costs associated with them must be estimated from other sources, such as claims data. (There is also the concern that RCT outcomes may not accurately reflect real-world outcomes.)

In contrast, for CEAs on medical and surgical procedures, estimating direct costs generally requires an administrative claims database, and input prices may differ across payers. These costs may actually be more accurate than drug costs obtained from claims data because they are not confounded by the problem of rebates. However, if the observational data set contains charges but not actual payments or net reimbursement, then there is a similar problem to that of not knowing drug rebates, although sometimes analysts apply available cost-to-charge ratios to approximate various hospital and other costs. Both the frequency and costs of various adverse events from procedures will typically come from observational data sources, including claims data, unless an RCT is available.

Measuring Effectiveness

An essential ingredient in any CEA is evidence on the effectiveness of the therapy (or therapies) in question. This is where CEAs of pharmaceutical applications generally appear to have advantages over CEAs of non-drug interventions. The typical RCT requirement for Food and Drug Administration approval of a drug provides a built-in source of data for measuring its effectiveness. The analysis of these data is straightforward relative to the measurement of effectiveness with observational data, including claims data. We do not mean to imply that RCTs do not exist for medical and surgical procedures but that they are more common for pharmaceuticals. (The well-known study of arthroscopic knee surgery for osteoarthritis by J. Bruce Moseley and colleagues is an example of an RCT on a procedure, but it required special funding and was not part of a regulatory approval process.)

There are several problems facing analysts who attempt to measure effectiveness using observational data. One involves patient heterogeneity. Unless researchers can adequately control for differences in patient characteristics using statistical techniques, they may make incorrect inferences about effectiveness. Furthermore, statistical analysis is limited to the variables that appear in a data set, and while some aspects of heterogeneity (age, some co-morbidities) are recorded, others (frailty, will to live) are not. Heterogeneity can also be a problem, of course, in RCTs, but larger trials and randomization help mitigate the problem.

Another problem in using observational data is that, because doctors consider many aspects of a patient’s condition (some of which are not recorded in the data set) when making therapeutic decisions, there may be correlations between treatment and outcomes that do not reflect true causal influence (if any). For example, practitioners may hedge from performing a particular invasive treatment on frailer patients. As a result, simple comparisons from the observational data may show that patients who received the treatment live longer than those who didn’t. The challenge for statistical inference on effectiveness is to tease out how much of that greater longevity was due to the treatment as opposed to the greater robustness of patients on whom it was used. (We are assuming that frailty is not sufficiently measured in the database and thus cannot be adequately controlled for.)

Provider heterogeneity may also be a greater problem in assessing effectiveness of procedures compared to pharmaceuticals. Evidence of better outcomes among providers who perform sufficiently high volumes for some procedures is well documented. Of course, depending on the condition, pharmaceuticals may be more effectively used by some providers, but provider effects are likely less important for drugs than procedures.

Researchers require particular statistical techniques to correctly infer effectiveness from observational data. Those techniques—such as instrumental variables, regression discontinuity, propensity score matching, and multiple regression—are generally more complicated to implement and can be perceived as being less transparent or leaving a larger degree of uncertainty than comparisons made from well-designed RCTs. While the need to use these techniques to infer effectiveness from observational data may help explain why researchers conduct relatively fewer CEAs on procedures, that impediment could be overcome. If more analysts apply those statistical techniques to estimate the effectiveness of procedures, then more effectiveness evidence would be available for use in CEAs on medical and surgical procedures for which RCT data are less common. (Many of these techniques have been discussed in the literature, sometimes in the context of comparative effectiveness analysis.)

While we have argued that the relative ease of obtaining well-controlled data may be a substantial reason for the predominance of CEAs of pharmaceuticals, a cautionary note is in order. Although RCT data are considered a gold standard for measuring effectiveness, if one is trying to use CEA to improve the allocation of health care resources in the real world, one will have biased CEA results if real-world effectiveness differs from what is measured in RCTs. The same statistical tools we suggested to facilitate the application of CEA to non-drug interventions could also enhance our knowledge of effectiveness of pharmaceutical interventions in real-world settings using observational data.

Incentives To Conduct CEAs: The Role Of Property Rights

Besides the advantage of greater availability of RCT data to facilitate CEAs on pharmaceuticals, the existence of property rights creates an incentive for makers of drugs, devices, and some diagnostic tests to produce effectiveness research. (An additional factor may be that patent and exclusivity regulations, by design, reward innovative activity with the promise of a period of exclusivity and potentially high prices, which in turn increase the demand for CEAs.) In many, but not all, medical, surgical, and procedural settings, there are no such property rights. Exceptions exist: With robotic surgery, there are rights to the machine; with the use of an antibiotic-embedded cement for orthopedic applications, there are rights to the cement. But common procedures, such as C-sections or resections of the colon, do not have ownership rights, although specialty societies have some degree of incentive to promote information about procedures whose use is effectively limited to society members.

How Can Greater Balance Be Achieved?

The considerations above suggest several policy interventions that could lead to a greater balance of CEAs across the health care spectrum. Government subsidies emphasizing CEAs on important but understudied areas, including medical and surgical procedures, are one possible solution. Government grant-making organizations (for example, the National Institutes of Health, the Centers for Disease Control and Prevention, the Department of Veterans Affairs, and the Agency for Healthcare Research and Quality, among others) could encourage greater application of CEAs to areas that are understudied relative to their portion of national health care spending. In particular, those funds could be distributed to favor effectiveness studies for interventions for which there are no distinct property rights. For better allocation of CEA research dollars, interventions that account for larger amounts of overall health care spending would logically be prioritized.

Payer encouragement or requirements for CEAs of medical and surgical procedures would also act as a catalyst. While, especially for existing procedures, there may be ethical considerations that rule out RCTs in various circumstances, a coverage incentive for RCT evidence for new interventions (or for effectiveness analysis using observational data) would encourage the building of an evidence base for use in CEAs. If, for example, health plans were to adopt copayment policies that favored interventions that had undergone CEA (for both pharmaceutical and non-pharmaceutical interventions), that would encourage greater application of CEAs to under-represented areas. Journal editors can also contribute to a greater balance. Further embracing analyses that use the statistical tools needed to measure effectiveness in observational settings would encourage researchers to take on those challenges, ultimately leading to a greater balance of CEA applications across all aspects of health care.

Authors’ Note

This research was funded through the Innovation and Value Initiative, a multistakeholder research initiative that is part of Precision Health Economics (PHE). In addition to his academic appointment at Tufts Medical Center, Peter J. Neumann is a consultant for PHE, a health economics consultancy to the life science industry.