Assessing the effectiveness of medical treatments

Healthfacts, April, 1993

Which medical treatments work? Is a new treatment always an improvement over the old one? Do published research results have much impact on the day-to-day practice of medicine? How well do physicians keep up with the ever-escalating amount of medical information?

These questions were discussed by the 100 researchers from North America and Europe who gathered in New York City for a conference entitled "Doing More Good Than Harm: The Evaluation of Interventions." The unofficial consensus from the parade of speakers was bleak, for many were exposing the fact that medical practice is not as scientifically grounded as it is widely perceived to be.

The topics of discussion at this three-day event, sponsored by the New York Academy of Sciences, have reached critical importance as the U.S. and virtually all other developed countries grapple with staggering medical care costs. The need for more hard evidence of medical treatments' effectiveness ("outcomes research") is now the rallying cry of health care reformers, and the goal of this conference was to discuss the best ways to produce and disseminate this vital information.

While most of the discussions centered on what constitutes the most reliable research methods, several speakers readily acknowledged that practicing physicians are not greatly influenced by research results. Two speakers noted a gap between what physicians say they do and what they actually do. For example, most doctors say they keep their information up to date by reading medical journals, yet studies show that few of them read medical journals.

Consequently, there is also a wide gap between the evidence produced by clinical trials and the day-to-day practice of medicine, according to Canadian epidemiologist Dr. R. Brian Haynes of McMaster University, Ontario. "And the gap is getting wider."

MDs Inherently Conservative

At a press briefing held midway through the conference, the following question was put to the assembled experts: Why do research results have so little impact on practicing physicians, especially when they show that a common, very expensive procedure like coronary bypass surgery should be performed less frequently?

The best they could offer in the way of explanations were: 1) most physicians are inherently conservative [about making changes in medical practice]; 2) doctors are not taught in medical school how to "digest" clinical trial results; 3) many doctors don't know about the research results, and the problem lies in information dissemination.

"Never underestimate the powerful forces of the status quo," said Jonathan Lomas, expanding on the reasons why research results don't usually have an impact on the practice of medicine. A behaviorist from the Centre for Health Economics and Policy Analysis at McMaster University, Mr. Lomas said, "Since few doctors read medical journals, we must turn elsewhere for behavior change and see it as a dynamic process," Mr. Lomas continued. "[Acceptance of research results] doesn't happen overnight. Research should be synthesized for them by a credible person in their work environment, and there must be repeated reinforcement, or the physician is likely to go back to the old way."

The relatively new statistical method called the meta-analysis is seen by many at the conference as the ultimate tool for deciding the value of medical treatment. But this too is not accepted by the average practicing physician. "A doctor ten years out of medical school is not likely to know what a meta-analysis is," said one medical journal editor, "and when they have it explained, they find it boring."

A description of a meta-analysis can indeed make many people's eyes glaze over. But an understanding of the method is every bit as important for medical consumers as it is for practicing physicians. The best designed studies exploring the same issue frequently produce' contradictory results, and the idea behind the meta-analysis is to combine the results of all in order to come to one conclusion. The best study design is the randomized, controlled clinical trial (RCT), which randomly allocates participants to receive the new treatment or the old one and follows them for comparison purposes.

The RCT was extolled by Richard Peto, M.Sc., who, with his mentor and colleague at the University of Oxford, Sir Richard Doll, was treated almost reverentially by most of the conference speakers, who seemed honored to be on the same podium with Britain's most famous epidemiologists. (In the 1950s, Sir Richard Doll pioneered the field of epidemiology--the observational study of disease and risk factors in large population groups-by being the first to detect the link between smoking and lung cancer.)

Small Benefit

Mr. Peto stressed the fact that only by randomly assigning very large numbers of people to one treatment or another can the "objective" worth of medical interventions be determined. The large numbers are important because treatments under study usually benefit only a small minority of those who receive them. Mr. Peto observed, in his flippant style, "Doctors aren't much good, but they're some good." For example, a meta-analysis of all 133 clinical trials shows that the women with early-stage breast cancer who were given the hormone drug tamoxifen have a 17% rate of recurrence, as compared to the 25% rate of cancer recurrence among those not taking tamoxifen.