Blog Home


New Drug And Device Approval: What Is Sufficient Evidence?

July 1st, 2014

Editor’s note: In addition to Jonathan Darrow, this post is also coauthored by Aaron Kesselheim. 

The federal Food, Drug, and Cosmetic Act gives the Food and Drug Administration (FDA) the authority to evaluate all prescription drugs and high-risk medical devices before they can be marketed to physicians and patients to ensure that they are safe and effective.

However, there is growing pressure to lessen the traditional standards for defining “safe and effective” for particularly promising therapies and accelerate patient access to these products.

A recent national health policy conference in Washington, D.C., explored the nature of the evidence needed for the regulatory approval of new therapeutics and the implications for patient care. The conference was organized by the Program On Regulation, Therapeutics, And Law (PORTAL) at Brigham and Women’s Hospital/Harvard Medical School, the National Center for Health Research, and the American Association for the Advancement of Science (AAAS).

It provided a forum for reviewing the data that exist about the FDA’s current medical product approval criteria, and the implications of providing patients with expedited access to investigational prescription drugs and medical devices.

Conflicting Pressures

Among the themes explored during the day-long event were the various and sometimes conflicting pressures that the FDA faces as it engages in policymaking to promote the public interest. In the face of rising development costs, large pharmaceutical manufacturers and their lobbying organizations frequently appeal to the FDA and Congress to develop more efficient drug approval programs.

Patients, some desperately ill, urge the FDA to approve new drugs more quickly and can seek pre-approval access to investigational drugs on a compassionate use basis. Yet when FDA-approved products are linked to deadly adverse events, all eyes turn to the FDA for answers and its leaders may be brought before Congress to address why the public did not know about these outcomes sooner.

There are currently a number of regulatory pathways through which medical products that appear in very early testing to have an important impact on public health can be brought to market. For example, the accelerated approval pathway, formalized in 1992, allows certain drugs to be approved on the basis of meaningful surrogate endpoints such as cholesterol level or blood pressure rather than clinical endpoints such as stroke or death.

Originally described as a “highly specialized mechanism for speeding the approval of drugs or biologics that promise significant benefit over existing therapy for serious or life-threatening illnesses,” accelerated approval has accounted for approximately 10% of all new molecular entities in the past decade.

Using surrogate endpoints can shorten trials, but drugs approved under accelerated approval tend to be tested on fewer patients in their pivotal trials, averaging only 142 patients in the intervention group. This can lead to dilemmas for regulators and clinical conundrums for physicians and patients.

Shorter or smaller trials combined with the use of surrogate endpoints can also lead to evidence that is difficult to interpret: In a small trial, bedaquiline (Sirturo), a drug intended to treat multi-drug resistant tuberculosis, showed efficacy in converting sputum from M. tuberculosis positive to negative (a surrogate endpoint), but 2.5 times as many people died from tuberculosis in the bedaquiline group than in the control group. By contrast, sipuleucel-T (Provenge), a prostate cancer drug, failed to establish efficacy on the basis of time to disease progression (a surrogate endpoint) but demonstrated a 4.1 month survival benefit. Both drugs were approved.

Imperfect Knowledge

So how confident should physicians and patients be in using these drugs? Patients with multi-drug resistant tuberculosis or castration resistant prostate cancer may be willing to accept increased risks in return for the possibility of effective treatment. However, despite the legal requirement that patients consent to treatment, patients and even their physicians may not sufficiently comprehend the uncertainties inherent in the use of surrogate measures.

Imperfect knowledge arises not only from the complexities inherent in clinical trial data interpretation, but from inadequate communication of this information to physicians and patients. Draft FDA guidance promulgated in March 2014 addressed both problems — uncertainty and inadequate communication — by suggesting (but not necessarily requiring) that drug companies use the following statement when labeling drugs approved via the accelerated approval pathway:

“An improvement in {identify the specific clinical benefit that remains to be established} has not been established. Continued approval for this indication may be contingent upon…confirmatory trials.”

Although this language is intended to increase consumer awareness of the uncertainty inherent in relying on less-than-well-established surrogate endpoints, PhRMA, the industry trade group, expressed concern that the language “could be misinterpreted to mean that products approved under the Accelerated Approval pathway have not met the statutory requirements of safety and efficacy required of all FDA-approved drugs.”

Evidentiary gaps in the approval of new medical products may be even more pronounced with respect to medical devices. One study found that of 78 approval decisions related to the highest-risk cardiovascular devices, only 24 (31 percent) were supported by at least 1 randomized study, 10 (13 percent) by at least 1 blinded study, and 4 (5 percent) by at least 2 blinded, randomized trials.

In some cases, promising results from randomized but uncontrolled or non-blinded trials of medical devices have been called into question after larger and more rigorous controlled trials were conducted. For example, a catheter-based renal denervation treatment of resistant hypertension showed positive early results in unblinded comparisons between the device and medical therapy. These results led to cost reimbursement and widespread adoption.

However, a subsequent larger sham placebo-controlled trial failed to confirm the expected benefit. Other cases are even more troubling: A 45-person nonrandomized study for the treatment of intracranial atherosclerotic stenosis appeared to “provide compelling evidence supporting the safety of…the Wingspan stent system for stroke prevention,” but 6 years after FDA approval, a 451-patient randomized trial was stopped after higher rates of stroke or death were observed in the Wingspan group.

Extended follow-up reported in 2014 confirmed that the stroke or death risks associated with the Wingspan device persisted over time. Following an FDA-convened expert panel meeting in 2012 in which panel members agreed that there was no evidence of benefit for most patients, the FDA decided to narrow the indication that appeared on the Wingspan’s label but did not require withdrawal.

Premature approval of drugs or devices that may later be shown to be harmful or ineffective is a so-called Type I error. These errors are cause for concern, but it is also important to consider the harm that could result by unnecessarily withholding approval of drugs or devices that are safe and effective (a Type II error), particularly for patients with no treatment options.

Frequency of error, therefore, is a critical piece of data. One study examined 47 new indications associated with 35 oncology drugs approved between 1992 and 2010 on the basis of surrogate endpoints via the accelerated approval pathway: it found that 26 (55 percent) of the 47 were subsequently validated, while 3 (6 percent) were either withdrawn from the market or had their distribution or indications restricted (the remaining 18 (38 percent) were pending).

Ethical Questions

It is also important to consider the ethical questions associated with evidence sufficiency. Should the government restrict the ability of fully-informed patients to choose whether to accept treatment based upon available, if incomplete, evidence? Most people would likely say that expedited programs, such as accelerated approval, are appropriate for patients with terminal cancer or other seriously diseases who have no other options.

However, certain patient groups — even among those groups representing patients in the most dire situations — understand the complicated risks inherent in seeking access to unproven therapies.

While advocacy by AIDS activists in the 1980s helped spearhead the creation of the accelerated approval pathway and other expedited FDA review programs, some of those same activists later witnessed the trouble that can arise from widespread availability of poorly tested drugs of unknown effectiveness. In 1994, one Treatment Action Group member was quoted as saying, “We don’t just need drugs, we need information.”

Even worse, accelerated approval can impose substantial additional burdens on those already vulnerable from illness. In the 1990s, 41,000 breast cancer patients notoriously endured high-dose chemotherapy with autologous bone marrow transplantation based on the promise shown in early disease models; unfortunately, the therapy was later shown to confer no survival advantage.

Considering the FDA Brand

The medical community and the public rely on the FDA to keep ineffective and unsafe drugs from the market while efficiently approving important new medicines. Industry also benefits from the confidence and credibility bestowed on a product by the “FDA-approved” label. However, just as a car manufacturer’s brand can be undermined by having to recall a faulty ignition switch, so too can the FDA’s reputation be called into question by frequent withdrawals of approved drugs and devices across multiple companies.

The stakes are particularly high for the FDA not only because its products are intended to address matters of life and health, but also because Type I approval errors in this marketplace are unusually resistant to correction. The inability to conclusively establish or disprove efficacy at the individual patient level — combined with placebo effect and optimism bias — has driven significant company and patient resistance to FDA attempts at withdrawing products.

The difficulty of removing drugs from the market is not new, but, in the midst of intensifying public pressure for ever faster access, the number and variety of expedited approval programs has increased in recent years, and with them the potential for premature approval.

While timely regulatory review is a useful goal and expedited pathways may be appropriate in certain circumstances, the FDA’s default position should not change. Continued insistence on robust evidence development prior to approval is essential to maintain the high quality standards the public has come to expect from FDA-approved products.

Email This Post Email This Post Print This Post Print This Post

 to the #1 source of health policy research.

No Trackbacks for “New Drug And Device Approval: What Is Sufficient Evidence?”

2 Responses to “New Drug And Device Approval: What Is Sufficient Evidence?”

  1. Jonathan Schwartz Says:

    I traveled to Canada seven years ago to undergo high frequency focused ultrasound (“HIFU”) treatment for prostate cancer, as it was unavailable in the United States. Based on my excellent results and minimal side effects, I expected HIFU would soon be approved domestically so that others could benefit from this non-invasive treatment without traveling abroad.

    The recent negative decision of an FDA panel evaluating HIFU makes that unlikely.

    Over 15 years, 40,000 men have been treated with HIFU in Switzerland, Germany, France, Britain, South Korea and Canada. In fact, highly regarded Cleveland Clinic offers HIFU treatments in Canada. So, I am puzzled at how the FDA’s panel of nine American urologists concluded that the procedure is unsafe and ineffective.

    Results reported in peer reviewed journals and by North American urologists, who have used HIFU, are comparable or superior to those achieved by conventional prostate treatments – without the cutting and or radiation that often permanently damages healthy tissue.

    To the authors’ point, perhaps this is a case where an expedited FDA process may be appropriate. Or, perhaps further investigation is warranted to discover how a treatment that is both embraced internationally and that offers a lower cost, can be denied.

    I am surprised the FDA process overlooks the possibility that a conflict of interest (conscious or subconscious) may be a factor here. A vote in favor of allowing Ablatherm HIFU treatment in the United States would likely reduce the income of urologists performing prostate surgeries, as HIFU treatment costs substantially less than the $30,000 to $40,000 that those surgeries command.

    If we are serious about reducing health care costs and offering effective new treatments in the states, we must find ways to make the FDA’s decision-making process more effective, responsive, and free from potential bias.

  2. Steven Walker Says:

    This article is a reasonably complete, albeit 10,000 foot, discussion of the issue regarding the various positions taken by stakeholders on what constitutes sufficient evidence for approval, but without some of the details needed to fully understand whether, for example, the 3 cancer drugs removed or labeled off the market were good decisions by FDA. For example, when FDA labeled Iressa, a lung cancer drug, off the market in 2005, every other country in which it had been approved kept it on the market looking at exactly the same data. Since that time, Iressa has been approved in nearly every other country in the world, is considered a useful drug for the treatment of lung cancer (and some other cancers) and can be targeted to patients most likely to respond to it – yet it remains unapproved in the US. Another example provided in the article also lacks important detail. The use of high-dose chemotherapy followed by a bone marrow transplant for advanced breast cancer patients was never approved by the FDA, and therefore has no relevance to a discussion of the utility of accelerated approval. It was a procedure applied by doctors using a combination of approved chemotherapy drugs off-label to try to wipe out the cancer cells, followed by an infusion of the patient’s own bone marrow stem cells (harvested before the chemotherapy treatments), to restore the patients bone marrow and associated immune systems (which were also killed by the chemo treatments). It was never proposed for approval, and never received approval. It also worked for approximately the same number of women as the standard treatments of the day, but turned out to be more dangerous, and fell from favor as a result. It was off label use, not a fast-tracked approved therapy, and nearly all oncologists both then and today would assert strongly that off label use of cancer drugs should not be curtailed because it would cripple their ability to help their patients. Also missing from the discussion is that high dose chemo – stem cell rescue is widely used as the only potentially-curative procedure for a variety of blood cancers, for which its use is also dangerous for the patient. It is nonetheless used because it is the patient’s only chance of avoiding death from their cancer. So its OK in those equally deadly blood cancers, but not in breast cancer? The details vary about whether a treatment is appropriate from cancer to cancer, and from patient to patient, but there were quite a few research docs who thought the high-dose chemo – stem cell transplant procedure could be refined into a useful, safer treatment, as it has been in blood cancers, but the scandal that was manufactured around its use in breast cancer pretty much wiped its potential off the cancer research map. After the scandal broke, research using it couldn’t attract funding and oncologists abandoned it. An opportunity lost? We’ll never know.

    There are always a lot of details missing from discussions like the one in this article that need to be understood to be able to make an informed judgment about the utility of accelerated approvals, and to even understand what “accelerated” means. It often does not mean fast at all. The truth is that accelerated approval was never really implemented as intended by FDA, and as a result has severely under-performed for patients. The current issues and debate on this subject aren’t about whether approving drugs based on surrogates is appropriate. They are about how to get the FDA to use them more often, with more flexibility and efficiency. By any measure, to the extent FDA has used them, the result has been an overwhelming success. A lot of people (millions) gained access to medicines that helped them live longer, better lives as a result, but because FDA never really embraced the concept, many millions more waited, and died waiting for drugs that represented best medical care for them, remained stuck in FDA’s process. Even FDA admits it could be doing a better job expediting the most needed, new safe and effective treatments to patients, but so far they continue to struggle in doing it.

Leave a Reply

Comment moderation is in use. Please do not submit your comment twice -- it will appear shortly.

Authors: Click here to submit a post.