How Can the Public Be Misled with Seemingly Solid Research?

Many people believe that scientific research is, by definition, reliable. When people read or hear statements such as “RESEARCH SHOWS…” or “RECENT RESEARCH HAS PROVEN…”, they usually trust what follows. But how justified is this trust in research and researchers, and what should we watch for?

Traditional, complementary, and alternative medicine has its foremost global figure in Prof. Dr. Edzard Ernst, who uses striking examples in the article below to explain that not all scientific studies have the same evidential strength or quality, and how societies are deceived by emphasis on scientific research:

Dr. Ernst begins his article with these words: "During my 25 years of involvement with so-called alternative medicine (SCAM), I have encountered countless examples that made me skeptical. In this article, I will briefly discuss the many ways consumers can be misled with solid evidence."

Note: Edzard Ernst considers “traditional, complementary, and alternative medicine” practices under the heading “so-called alternative medicine (SCAM)”. However, in the rest of this article, we have used the term “alternative medicine” instead of SCAM.

Absence of evidence

I have just finished reading a German Heilpraktiker book devoted entirely to alternative medicine. The author makes hundreds of statements and presents them as evidence-based facts. I am sure this will seem convincing to many laypeople or consumers. Yet it has a fatal flaw: the author fails to provide any actual evidence to support the statements. The references provided come only from other books that are equally unsupported. This popular technique of making unsupported assertions allows the author to make unchecked and unbalanced claims. A layperson is often unable or unwilling to distinguish such stories from evidence, making this an easy and popular way to mislead us about alternative medicine.

False evidence

In my earlier articles, we have encountered this phenomenon ad nauseam: a commentator makes a claim and supports it with some seemingly solid evidence, often from reputable sources. Those few of us who bother to read the cited papers quickly realize that they do not say what the commentator claims. This method relies on readers being easily persuaded by some false evidence. Most consumers do not bother to look beyond the smokescreen provided by such claimants, so this method often works surprisingly well.

An example: Vidatox is a homeopathic cancer “treatment” from Cuba (see our article on blue scorpion venom). The Vidatox website claims it is effective for most cancers. Given how exciting such a claim would be, one would expect many published papers on Vidatox. Yet a Medline search yielded one paper on the topic. Its authors concluded: "Our results show that the concentration of Vidatox used in the present study has no anticancer effects, and caution should be exercised in using Vidatox in patients with liver cancer."

An often necessary question is: where is the fine line between misleading research and fraud?

Surveys

No area of healthcare produces more surveys than alternative medicine!

Approximately 500 survey studies are published in this field every year! This “survey frenzy” has a purpose: to support a positive message about alternative medicine, something experimental research rarely does.

For a typical alternative medicine survey, a team of enthusiastic researchers puts together a few questions to find out what percentage of a group has tried alternative medicine in the past. They conduct a survey and may receive one hundred to two hundred responses. Simple descriptive statistics then show what percentage used alternative medicine. This finding is eventually published in one of the many third-rate alternative medicine journals.

The inference is: if alternative medicine is so popular, it must be good; if it is good, the government should pay for it. Few consumers will realize that this conclusion is little more than a misleading appeal to popularity.

Avoiding the question

Another popular way alternative medicine researchers mislead the public is by avoiding important research questions; if you do not ask the real questions, you will not appear to fail!

For example, chiropractic specialists (manual therapy applied to the spinal cord and joints) will deny that one of the practice's most important adverse effects is the risk of spinal manipulation (spinal cord injury). One might therefore expect much of the currently published chiropractic research to address it. Yet the opposite is true. Medline currently lists more than 3000 papers on “chiropractic”, but only 17 on “chiropractic and harm”.

Pilot studies

A pilot study is conducted before a full-scale research project to assess feasibility, time, cost, and adverse effects, and to improve study design; it is a small-scale preliminary study. However, the many alternative medicine pilot studies currently being published do not meet these basic prerequisites. Genuine alternative medicine pilot studies are very rare. The reason for the abundance of sham pilot studies is clear: they can easily be interpreted as showing encouragingly positive results for whatever alternative therapy is being tested. Advocates can then mislead the public by claiming many positive studies exist and alternative medicine practices are therefore supported by solid evidence.

"Safe" study designs

There are several ways to design a study so that the risk of a negative result is minimal:

The most popular in alternative medicine research is the “A + B versus B” design. For example, in this study, cancer patients suffering from fatigue were randomly assigned to receive usual care or usual care plus regular acupuncture. The researchers then monitored patients' fatigue and found that the acupuncture group did better than controls. The effect was statistically significant, and an editorial in the journal where it was published called the evidence “compelling”. Because of a cleverly overstated press release, the news spread quickly, and the study was celebrated worldwide as a major breakthrough in cancer treatment.

Imagine that you have an amount of money A, while your friend has the same amount plus an amount B. Who has more money? Simple: your friend, of course. A+B is always greater than A (unless B is negative). For the same reason, such “pragmatic” trials will always yield positive results (unless the treatment causes actual harm). As always, treatment plus acupuncture is more than treatment alone and is therefore more than likely to produce a good result. This would be true even if acupuncture were a pure placebo: after all, a placebo is more than nothing, and the placebo effect will influence the outcome, especially with a highly subjective symptom such as fatigue.

A more obvious way to produce false-positive results is by not including blinding in the study design. Clinical trials blind the patient, treating practitioner, and group-allocation assessor to ensure that “expectation” does not contribute to the outcome. Expectation may not move mountains, but it can certainly influence a clinical trial's result. Patients hoping to improve may experience fewer symptoms even if their regular treatment is useless. If doctors and outcome assessors have preconceived ideas about an experimental treatment, they also tend to view the results through rose-colored glasses.

Failure to allocate participants randomly to study groups is another source of bias that can mislead us. If patients or researchers are allowed to choose who receives experimental treatment and who receives control treatment, the two groups are likely to differ in various variables (see the story of high-dose vitamin C therapy in cancer). Some of these variables may affect the outcome. For example, if doctors assign their patients to experimental and control groups, they may choose responders for the former and non-responders for the latter. This may not be deliberate, but a matter of intuition or instinct: responsible health professionals want patients who, in their experience, have the best chance of benefiting from a treatment to receive it. Proper randomization lets us ensure that comparable patient groups are compared. Non-random assignment can easily produce false-positive findings.

Studies that do not test whether one experimental treatment is superior to another (often called superiority studies), but instead assess whether it is equivalent to a therapy generally considered effective, can also mislead people [in medicine, this is called the non-inferiority hypothesis (non-inferiority)]. The idea is that, if both treatments produce similarly positive results, both must be effective. Such trials are called non-superiority or equivalence trials and offer many opportunities to mislead us. For example, without enough patients, such a study may show no difference regardless of actual effectiveness. Consider a simple theoretical example: someone proposes comparing antibiotics with acupuncture for bacterial pneumonia in older patients. Researchers treat 10 patients in each group, and the results show 2 deaths in one group and 3 in the other. Statistical tests show that a difference of only one patient is not statistically significant, so the authors conclude that acupuncture is as good as antibiotics for bacterial infections.

Even more misleading in an equivalence study is the option of giving the control group low-dose treatment. In the example above, researchers could then recruit hundreds of patients to overcome criticism of their initial study, but decide to give controls a low antibiotic dose. The results would seemingly confirm their initial finding that acupuncture is as good as antibiotics for pneumonia. Acupuncture practitioners could then claim that a very large randomized clinical trial proved their treatment effective for this condition. People unfamiliar with the correct antibiotic dose could easily be deceived.

If controls in an equivalence study receive a treatment that is not only ineffective but harmful, the results will obviously be even more impressive. In such a scenario, even the most useless alternative treatment will appear effective simply because it is less harmful than the comparator.

Another version of this idea is the many controlled clinical trials in alternative medicine that compare one unproven treatment with another unproven treatment. Results generally show no difference in the clinical outcomes experienced by the two groups. Enthusiastic alternative medicine researchers tend to conclude that this proves both treatments equally effective. Yet the more likely conclusion is that both are equally useless.

Another way to mislead the public is to draw conclusions unsupported by the data. Imagine a homeopathy (diluted medicine) trial producing outright negative data. As a homeopathy enthusiast, you are far from happy with your findings; a sponsor may also be pressuring you. What can you do? The solution is simple: emphasize at least one positive message in the published paper. In homeopathy, for example, you can make much of the treatment being quite safe and inexpensive: not a single patient died, and most were very satisfied with this treatment, which was not even very expensive.

Omission

Another popular way to mislead the public is simply to omit findings that alternative medicine researchers dislike. If the aim is for the public to believe the myth that all alternative medicine practices have no side effects, researchers merely need to neglect to report them in clinical trials. I have repeatedly warned readers about this widespread phenomenon. We even evaluated it in a systematic review including 60 randomized controlled chiropractic and acupuncture trials. We found that 29 randomized controlled trials did not mention adverse effects at all. Complete information on the method, frequency, duration, severity, and incidence of adverse-effect reporting appeared in only one randomized controlled trial. 

Most studies have many outcome measures. For example, an acupuncture study for pain control may measure pain in half a dozen different ways, as well as treatment duration until pain relief, the amount of medication taken alongside acupuncture, days off work because of pain, the spouse's impression of the patient's health, quality of life, pain-disrupted sleep, and so on. If researchers then assess all outcomes, one or two are likely to change in the desired direction, particularly if variables are rated at half a dozen different time points. This may be a chance finding: with typical statistical tests, one of 20 outcome measures may yield a significant result purely by chance. To mislead us, researchers need only “forget” all negative results and focus the publication on those that happened to turn out as hoped.

Fraud

When it comes to fraud, there is far more to choose from than one might wish. We and others have shown, for example, that Chinese acupuncture studies almost never produce negative findings. In other words, one need not read the paper: one already knows it is positive. More extremely, one need not conduct the study: its result is known before research begins. This strange phenomenon suggests something is wrong in Chinese acupuncture research. Even a team of Chinese scientists confirmed this suspicion. In this systematic review, a team of Chinese scientists identified all randomized controlled trials (RKA) published in Chinese journals. They found 840 RKA in total, including 727 comparing acupuncture with modern medical treatment, 51 with no-treatment controls, and 62 with sham-acupuncture controls. Of these 840 RKA, 838 studies (99.8%) reported positive primary outcomes and two trials (0.2%) negative outcomes. The percentages of RKA concerning publication withdrawal or concealment of sample-size calculations were 43.7%, 5.9%, 4.9%, 9.9%, and 1.7%, respectively. The authors concluded that there may be a major problem with acupuncture RKA reported in Chinese journals, related to a high risk of bias. We recommend that all studies be registered prospectively in international trial registries in future.

A survey of clinical trials in China revealed large-scale fraudulent practices. China's Food and Drug Regulatory Board conducted a year-long review of clinical trials. They concluded that more than 80 percent of clinical data were “fabricated”. The review assessed data from 1,622 clinical research programs for new drugs awaiting regulatory approval for mass production. Officials now warn that further evidence of malpractice may emerge in this scandal.

I would like to add that fraud in alternative medicine research is certainly not restricted to China. In our complementary and alternative medicine section, you will find plenty of evidence for this statement, I am sure.

Conclusion

Research is clearly necessary if we want satisfactory answers to many questions about alternative medicine. Unfortunately, not all research is reliable, and much alternative medicine research is misleading. We must therefore remain vigilant and apply all the critical appraisal skills we can muster.

*

RELATED TOPICS

• No, high-dose vitamin C does not treat coronavirus! There is no FDA approval except for scurvy
• Edzard Ernst – The foremost global figure in traditional, complementary, and alternative medicine
• What is the difference between complementary and alternative therapies?
• Why do ineffective treatments appear effective?