After all, two years earlier, their findings had prompted a standing ovation at an American Society of Clinical Oncology conference. Their case was clear: Women whose breast cancer had spread to other parts of their body had double the time without the disease progressing — 5.5 extra months — when given Avastin in conjunction with the first round of chemotherapy, compared with those who received only the chemotherapy.

Yet on that snowy day in December, several of the nine members of the FDA’s Oncologic Drug Advisory Committee were perplexed. Although Avastin produced more time without tumor growth — what is known as “progression-free survival,” or PFS — the women who took Avastin ultimately lived no longer than the women who hadn’t.

“It’s not just that the pipeline hasn’t produced a gusher, but that it’s falling to a trickle. We can’t continue on the road we’re traveling. It’s scary — the system is heading toward some sort of cusp.”

True, the trial had not been designed to show whether Avastin prolonged women’s lives, but to measure PFS as its primary endpoint. Such an approach has started to gain wider acceptance at the FDA — it allows for shorter studies with fewer subjects — though never to assess a first-line metastatic breast cancer treatment. More than 250 people had braved the icy roads to see whether the FDA’s advisory committee would push the boundaries of what was viewed as acceptable evidence.

Would PFS alone be sufficient to approve a therapy for the earliest phase of a fatal disease, for which the efficacy and safety standards tend to be higher than for therapies treating later-stage cancers?

“It’s a philosophical issue,” says Richard Pazdur, MD, director of the FDA Office of Oncologic Drug Products. “Some people believe that delayed progression is a direct clinical benefit in itself, while others believe PFS should be a surrogate for overall survival.”

The deliberations over Avastin underscore a crisis in the evaluation of new therapies: Randomized, controlled trials, the gold standard in the hierarchy of medical evidence, are not delivering the authoritative answers about drug’s efficacy that we have come to expect. More and more researchers are devising innovative trial designs, but their efforts are bedeviled by such problems as skyrocketing costs, increased trial lengths, fraudulent reporting and confusion about the results’ implications.

These problems have led to a worrisome trend. The number of new molecular entities approved by the FDA dropped from an average of 35.5 per year from 1994 through 1997 to 23.3 per year from 2001 through 2004, according to The Journal of the American Medical Association. In 2007, the FDA approved 19 new drugs, the fewest in 24 years, according to the financial research firm Washington Analysis.

“It’s not just that the pipeline hasn’t produced a gusher, but that it’s falling to a trickle,” says Philip Lavori, PhD, professor of health research and policy at the Stanford School of Medicine. Some experts emphasize that the decline is part of a cyclical process, but Lavori, as well as others, sees it as a bellwether. “We can’t continue on the road we’re traveling,” he says. “It’s scary — the system is heading toward some sort of cusp.”

Pazdur brushes aside such dire predictions, noting that the FDA has approved dozens of new uses for already-approved drugs and that the agency is taking steps to accelerate the process of bringing drugs to market. The Avastin trial reflects how some researchers are responding to this challenge, but it also shows how such efforts face resistance.

In the last few years, for instance, the FDA has used evidence of progression-free survival to approve a dozen new indications for cancer therapies. That is a marked departure from a decade ago, when the agency rarely, if ever, considered such evidence. Indeed, in June 1999, the Oncologic Drug Advisory Committee had voted unanimously against PFS being used as a surrogate endpoint, let alone a primary one, for trials of first-line metastatic breast cancer therapies.

The Avastin study was taking the FDA into the uncharted territory of using PFS to judge a therapy for the early stage of a fatal cancer instead of the later ones. As opposed to prior decisions using PFS, this one involved patients who still had years to live and a host of other treatment options.

The study enrolled 722 women from 2001 through 2004 and was unusual in that it turned on the subjective evaluation of X-rays. These scans were the basis for the judgments as to which group went for a longer period without the disease progressing.

Researchers had not expected Avastin to cure breast cancer, but hoped it could help a subset of those with the disease. A study to measure whether Avastin prolongs overall survival in a relatively small percentage of patients would have required roughly three times as many women in the early stage of metastatic breast cancer, and the trial could have spanned the decade. What’s more, participants in that trial would have had to adhere to randomly assigned sequences of treatment, foregoing other options.

Why bother with such headaches when measuring progression-free survival could be a valid — and quicker — way of evaluating the therapy? Thousands of patients, argue breast cancer oncologists, would welcome more time without the disease progressing, even if it did not extend their lives.

But critics, including some patient advocacy groups, say that statisticians could have designed a trial that wouldn’t unrealistically restrict a woman’s options. The study could adjust the way randomization is achieved as women pursue different treatments. The bigger hurdle, critics say, is the drug company’s reluctance to pay for a comprehensive study that might not have a big return.

Is it worth “lowering the standard for drug approval,” as the National Breast Cancer Coalition Fund describes it, for a treatment that costs more than $55,000 a year without offering a cure?

The hearing ended with a 5-4 vote against approval. Weeks passed as Pazdur weighed what the FDA’s final decision would be. The maker of the drug, Genentech Inc., vowed to continue discussing the matter with the agency. Yet again, a clinical trial had sown confusion about a drug.

Trials, triumphs and trip-ups

Randomized, controlled trials are supposed to deliver definitive results that dispel doubts, not create them.

In its most basic form, researchers divide their study sample by chance, or randomly, into two groups. While one group receives a new drug, the other receives a placebo, or perhaps the standard therapy. The randomization guarantees that each group is a rough reflection of the other, while ensuring that physicians’ bias does not predispose the selection of particular subjects for the new therapy. To make certain that doctors’ and patients’ perceptions do not influence the findings, the trial may be “double-blind,” meaning neither doctor nor participant knows whether the participant is receiving the new therapy.

As a result, when one group has a substantially different health outcome than the other, the researchers can attribute the result to the treatment being tested.

Modern medicine traces its first randomized, controlled trial to 1946, when such a study proved the benefit of using streptomycin to treat pulmonary tuberculosis. Their pre eminence was cemented in practice in the United States in the 1970s when the FDA required that new claims about the benefits and safety of drugs be backed by “adequate and well-controlled clinical investigations.”

Over the years, these trials led to many, if not most, of the advances in medical practice. They caused doctors to abandon the Halsted radical mastectomy, first for the less-intrusive total mastectomy and later the lumpectomy, with radiation when appropriate. They revealed that healthy, postmenopausal women who take estrogen combined with progestin have a greater risk of breast cancer, heart attack, stroke and blood clots. They established that beta-blockers prolong survival in patients with heart failure.

But recent snafus have led to skepticism about clinical trials. In the last decade, pharmaceutical companies have trumpeted results about new drugs that later appeared, at best, incomplete and, in some instances, fraudulent. After Vioxx and other Cox-2 inhibitors became best sellers, for instance, reports emerged that these drugs upped the risk of heart attack while offering little more pain relief than aspirin — points that the drugs’ manufacturers had, to put it gently, obscured.

The controversies surrounding the trials behind these and other drugs might involve deceit, but they also turn on the studies’ increased complexities. In the past, these studies were expected to yield a yes or a no. This dynamic can be seen with Vioxx, which went from nearly 20 million prescriptions in 2003 to being withdrawn from the market in 2004. The middle ground was lost: This drug might be appropriate for select patients in need of a pain reliever who can’t tolerate aspirin or naproxen and are at low risk of heart attack.

“It’s either pedal to the metal or slam on the brakes,” says Lavori. Unfortunately, he adds, as new drugs become one of a sequence of treatments for chronic conditions, the public’s desire for black-and-white pronouncements, encouraged by the drug company’s marketing, crashes against a new reality. Randomized, controlled trials increasingly detect subtle benefits for subsets of patients rather than for everyone with the disorder.

The FDA has responded with measures intended to ferret out faulty data, identify adverse events in trials that had previously gone unreported and produce better labeling about which sets of patients could benefit or should avoid a particular drug. It now requires that the middle and final clinical trial phase results of FDA-regulated drugs be posted online at a designated federal Web site: clinicaltrials.gov. It requires monitors and auditors, independent from the study investigators, to guarantee that the data collection is as precise as clockwork. And for every trial at every site, an institutional review board, an independent panel of experts, must oversee the trial’s conduct and receive immediate notice whenever an adverse event occurs. The IRB can suspend or shut down the trial.

According to research from the Tufts Center for the Study of Drug Development, such good intentions have had unintended consequences:

The price of getting a new drug approved, when adjusted for inflation, has more than doubled since the 1980s, surpassing $800 million.

Trials take longer, up to 780 days during 2003-06 from 460 days during 1999-2002. Recruiting participants is more difficult, with the rate of those proceeding from screening through randomization down to 59 percent from 75 percent during the same time spans.

“It’s a lot more expensive today — the sheer amount of paperwork involved, the time and the personnel necessary to do this correctly,” remarks Mark Genovese, MD, a Stanford rheumatologist who has been conducting trials for more than a decade. “We may be missing some wonderful opportunities.”

Testing patients, testing patience

Genovese, associate professor of immunology and rheumatology, is currently involved in 10 trials of drugs for rheumatoid arthritis and other autoimmune diseases, but he would be doing even more if it weren’t so difficult to find and keep trial participants.

On a recent Monday morning, he examines the puffy knuckles of a 52-year-old man with rheumatoid arthritis in one such trial. “Two of my fingers have been frozen together all week,” the patient says. “That’s never happened before.”

Genovese moves his hands from the patient’s knuckles to his shoulders, knees, ankles and feet, keeping a tally of how many are swollen and tender — data that is added to the patient’s trial binder, which after 12 weeks is already the size of a Manhattan phonebook. In response to Genovese’s questions, the patient says the pain is no better since the study began.

“Well, it’s one of two things,” Genovese says. “Either the drug is not working or you’re on the placebo.”

The man is one of three participants at Stanford in a 26-week, double-blind, randomized-control trial, which overall aims to enroll roughly 120 patients at sites in the United States, Canada, the United Kingdom and Belgium. The study is designed to evaluate the efficacy and safety of a recently discovered molecule that blocks the cell signals that may trigger rheumatoid arthritis.

To enroll in the trial, patients must have had active rheumatoid arthritis for at least six months; taken between 10 milligrams and 25 milligrams weekly of methotrexate (a drug that addresses a different mechanism of the disease) for more than three months, and had an inadequate response to treatment with anti-tumor necrosis factor therapy. There’s a checklist of other criteria, and sometimes new ones emerge as the study progresses: A fourth participant had to withdraw because the trial designers decided that no one under 120 pounds should be in the study.

“The biggest challenge we face is recruiting patients, given the recent advances in the field and the way clinical trials have been portrayed in the media,” Genovese says. “Because of the difficulty recruiting, we are constantly trying to come up with shorter studies requiring fewer patients.”

The organization needed to carry out such studies is a stark contrast to the early 1970s when Lavori began doing statistics for randomized, controlled trials. “All you needed then was a hockey team — five MDs and a statistician as goalie,” Lavori says. He alone would be the trial coordinator.

Today, the global management of trials is often contracted out to clinical research organizations, or CROs, which comprise a multibillion-dollar industry in their own right. Stanford’s Immunology and Rheumatology Clinic has three full-time research coordinators to handle the collection of data; the recruitment and retention of trial participants; and the constant multiparty communication among investigators, sponsors, external auditors, the IRB and the CRO.

Genovese acknowledges that the system is growing more cumbersome, but he has seen it produce results. He was the lead investigator on a series of successful trials of abatacept, now marketed as Orencia, the first in a new class of drugs known as co-stimulation blockers. The drug now is used by tens of thousands of people with rheumatoid arthritis for whom other treatments offered little relief.

This kind of breakthrough — and desperation — is what brings the 52-year-old man to Stanford to participate in the trial. “You want to know why I’m doing this?” he says. “Because nothing else is working.”

Given the degree of the man’s pain, Genovese promises to move him to the open-label portion of the trial at his next visit. At that point, they would check whether he received the drug or placebo, and provide him the test drug, perhaps at higher dose, for free. Genovese reminds the patient that he can leave the trial at any time.

“I’m not griping or anything,” the man says. “It is worth the wait. I want you to cure me.”

Trials in the age of chronic ills

Find the “cure” — that’s what randomized, controlled trials delivered for much of the latter half of the 20th century. But a plethora of scientists and policy makers say that the old system is, at best, outmoded or, at worst, broken.

“What has changed is that we have converted many diseases that once were uniformly fatal to chronic conditions,” says Lavori. “There are now opportunities for second and third lines of care. The value of the initial treatment is defined by how it interacts with the subsequent treatments.”

Newfound understanding of the human genome has advanced the ability to tailor treatments using the particulars of a patient’s genetic profile. Instead of one pill cures all, physicians have a menu of treatments. Researchers now have to compare a host of drugs in different sequences and combinations.

Ralph Horwitz, MD, chair of Stanford’s Department of Medicine, is urging colleagues to rethink the rigid hierarchy that has randomized, controlled trials at the top of the pyramid and has issued a call “to break the hegemony of the randomized trial design.”

“The popular belief that only randomized, controlled trials produce trustworthy results and that all observational studies are misleading does a disservice to patients, clinical investigations and the education of health-care professionals,” he wrote, with co-author John Concato, MD, in The New England Journal of Medicine in 2000.

Horwitz and Concato compared 55 randomized, controlled trials with 44 cohort or case-controlled trials, known as observational studies, which assessed one of five separate treatments. In these and other observational studies, the assigned treatment comes from physicians’ and patients’ decisions rather than from the trial investigators’ designs.

Although many statisticians contend that observational studies’ results will be biased — physicians might tend to assign particular patients to new therapies — Horwitz and Concato found that these particular well-designed observational studies had results that were largely in line with those of comparable randomized, controlled trials.

In an interview, Horwitz notes that as data analysis and data mining grows more sophisticated, it will become more of an option to do studies in which randomization and blinders are no longer needed. “The availability of large databases is changing — and will continue to change — the nature of clinical research,” he says.

Pazdur, the FDA official, acknowledges that observational studies have a larger role to play, as last year Congress gave the FDA more authority to require companies to continue to study a drug’s effects after it enters the market. As part of an approval, the FDA can demand a risk evaluation and mitigation strategy to assess the drug over the long run — and impose heavy fines if it isn’t followed.

Randomized and controlled: it’s still gold

At the same time, the FDA is encouraging researchers to design more effective randomized clinical trials. The agency, for instance, is looking into requiring pre-approval trials to collect evidence of certain biomarkers that flag the potential for serious side effects, a move that could avoid nasty situations like the one with Vioxx. More important, it is pushing for researchers to explore “adaptive clinical designs,” which allow for trials to be modified as they proceed.

“The research and development process in the pharmaceutical industry hasn’t changed in 40-plus years. It isn’t feasible for companies to continue to develop drugs in this way.”

Lavori, for instance, says statisticians must take into account “adaptive treatment strategies” in chronic diseases, assessing how the different sequences of therapies stack up against each other. Pulling off such assessments is challenging, he admits, but the objective has to be evaluating the complete treatment strategy, not just one stage. With the growth in electronic health records and other databases of patient information and the increased deployment of decision-support software for clinicians, he believes that researchers can get a large enough sample size to do such analyses — without sacrificing randomization.

“The statistician Fred Mosteller once said, ‘While it is easy to lie with statistics, it is even easier to lie without them,’” Lavori remarks. “So I would say, ‘While it is easy to get it wrong with a randomized trial, it is even easier without one.’”

But new statistical methods alone won’t solve the problem. The FDA can consider only the trials submitted to the agency for review. Even then, it cannot weigh whether the benefits of a therapy merit its cost. The incentives are for pharmaceutical companies to pursue approval for expensive blockbuster drugs, then get those drugs approved for as many diseases as possible.

What exacerbates this situation is the difficulty of funding trials independent of the pharmaceutical industry, says George Fisher Jr., MD, associate professor of oncology, who oversees more than 250 clinical trials at the Stanford Cancer Center. Despite promises made by leading insurers to cover some of the costs of clinical trials, the difficulties in getting insurance reimbursements cause many patients to opt out of studies of experimental drugs that could benefit them. “It’s infuriating,” says Fisher, an oncologist who specializes in gastrointestinal cancers.

Who, Fisher asks, is going to pay for trials that show that a drug already found to be effective for a relatively common cancer might also have benefits for a rare cancer that afflicts some 10,000 patients? “Those patients are left out in the cold,” he says. “The companies tend to pick the low-hanging fruit.”

Some health-care experts say that the pharmaceutical industry is rising to these new challenges. The Tufts Center for the Study of Drug Development reported in May 2006 that the number of new drugs entering the earliest stages of clinical testing by the top 10 pharmaceutical firms jumped 52 percent for 2003-05 over 1998-2002. If these companies want to get these drugs approved and reap their profits, they have to overhaul their business model. “The research and development process in the pharmaceutical industry hasn’t changed in 40-plus years,” says Kenneth Kaitin, PhD, the center’s director. “It isn’t feasible for companies to continue to develop drugs in this way.”

Indeed, the big pharma companies are looking for partnerships to facilitate new types of research. And that brings us back to the case of Avastin, which is emblematic of the tectonic shift in the conduct of randomized, controlled trials.

The biotech firm Genentech, which holds the Avastin patent, had already won FDA approval for its efficacy in treating both colon cancer and lung cancer. But the bid to extend its use as a treatment to patients who have not received chemotherapy for early-stage, metastatic, HER2-negative breast cancer was risky.

The unprecedented trial was actually launched — independent of the company — by the Eastern Oncology Cooperative Group, with funding from the National Cancer Institute. It was not intended to be submitted to the FDA. Only after the results came in did Genentech register the trial, now known as E2100, with the agency.

While the Dec. 5 vote appeared to be a setback, Pazdur says he was more interested in the discussion than in which side had a majority. He also took into account the results of a similar study in Europe, prompted by the E2100 results, that the agency received shortly after the hearing. Those findings supported the E2100 conclusions. The study was conducted by the Swiss pharmaceutical company Roche Holding AG, which is Genentech’s largest shareholder and owns the rights to market Avastin outside the United States.

On Feb. 22, the FDA granted Avastin “accelerated” approval, which permits drugs for life-threatening diseases onto the market even when doubts persist. While the drug is now approved for use in treating these particular cases of breast cancer, Pazdur adds, Genentech must conduct several more trials to evaluate whether Avastin prolongs the lives of women with breast cancer.

Pazdur grants that trials using PFS can be, as one member of the advisory committee put it, “fuzzy.” Nonetheless, Pazdur warns that relying on overall survival as the ultimate yardstick can be “overly rigid.”

“We have to demonstrate flexibility in providing drugs to patients with advanced malignancies,” Pazdur says in the publication Cancer Letter. “Avastin may be an appropriate choice for some patients; it may not be for others.”

The FDA still considers results provided by randomized, controlled trials as the gold standard of medical evidence, but those results don’t necessarily translate into easy conclusions.

Speaking of the FDA’s guidance to physicians in the use of Avastin for breast cancer patients, Pazdur adds, “There is a choice.”