Online reviews can make over-the-counter drugs look way too effective

Here’s one good reason why people often take medications and use health products that don’t live up to expectations or just don’t work — digital word of mouth.

The reviews can be glowing. Take this scuttlebutt about a cholesterol treatment: “I have been using this product for 2 years. Within the first 3 – 4 months my cholesterol was down 30 points. Just got cholesterol tested last week: down from 245 to 196.”

That’s incredible. It might even be true. But the big picture is alarming: Patients’ online reviews of three over-the-counter drugs — two for lowering cholesterol and one for losing weight — greatly exaggerate how well these substances actually work for most people, says psychologist Mícheál de Barra of the University of Aberdeen in Scotland. Online reviewers portray these medications as working three to six times better than they do in clinical trials that randomly assign drugs or placebos to broad samples of volunteers, he says.

“To learn what works in medicine, you need systematically collected data, usually randomized clinical trials,” de Barra says. “It’s very risky to rely on observations and word of mouth, whether it’s electronic or in person.”

Online reviewers probably aren’t lying, at least not in most cases. Neither are online review readers unrealistically optimistic about health treatments. The problem is that people who benefit most from a medication are especially apt to post their experiences online. Review readers get bamboozled by a wave of positive recommendations that don’t accurately represent how a drug works, or doesn’t work, for people in general.

The duping doesn’t end there. Direct word of mouth about medical treatments, as well as glowing testimonials about health products in advertisements, also traffic in positively skewed information.

In some cases, science does, too. Researchers have recently raised alarms about the “file-drawer problem,” in which studies that find no statistical effects are not published (SN: 5/19/12, p. 26). Published studies reporting positive effects then look unjustifiably bulletproof.

Clinical trials can tilt results in other ways as well. Authors of published trials often fail to preregister details of what they plan to study and how they’ll measure it, enabling data manipulation, selective reporting of results and self-serving interpretations of findings. Clinical trials report patients’ “average” responses to, say, an anticholesterol drug, but do a poor job of predicting which individuals will benefit from that treatment. “Published evidence from randomized trials is already an amalgam of evidence-based medicine and hearsay,” notes Stanford University epidemiologist John Ioannidis.
Online reviews are just old-fashioned word of mouth with a global bullhorn. But they offer one way to quantify the positive bias inherent in so much communication about medical treatments. Patients’ medication reviews that include numerical information, such as cholesterol levels before and after taking an anticholesterol drug, represent what Ioannidis calls “evidence-based hearsay.” Even if patients’ reviews rate treatments with a star system rather than including quantified measures of some kind, those opinions become powerful hearsay evidence for other patients, Ioannidis suspects.

De Barra evaluated 908 online reviews of two cholesterol reduction products, Benecol caramel smart chews and CholestOff, written on or before March 18, 2015. He also assessed 767 reviews of a weight-loss drug, Alli (orlistat), written on or before February 28, 2015.

Benecol reviewers reported a not-too-shabby average cholesterol decline of 45 milligrams per deciliter, versus a small average decline of 14 milligrams per deciliter in nine clinical trials. Respective numbers for CholestOff were 31 milligrams per deciliter and 13 milligrams per deciliter. Alli reviewers reported an average weight loss of about 10 kilograms after taking the drug for three months, versus an average of about two or three kilograms in two clinical trials. That disparity widened slightly after taking Alli for seven months.

Each drug garnered mainly positive reviews, some reporting effects far larger than the average, peppered with a few pans, De Barra reports in the March Social Science & Medicine.

What’s particularly concerning is that these three medications displayed small or negligible effects in clinical trials that probably already inflated how well the drugs work, Ioannidis says. So people searching for help in lowering cholesterol or shedding pounds may instead be getting a massive dose of disappointment from online reviews.

Online reviews can also exaggerate a treatment’s worst effects, whether real or alleged. Evidence-based hearsay may be particularly catchy online and in person when it’s frightening, de Barra suspects.

The anti-vaccination movement provides a potent example, due in part to the success of vaccines. When people face no immediate threat from measles or other infectious diseases, anti-vaccine stories may go viral, so to speak, thanks to misapplied but deeply felt intuitions. Two cognitive scientists, Helena Miton of Central European University in Budapest and Hugo Mercier of the Cognitive Sciences Institute in Bron, France, proposed this scenario in the November 2015 Trends in Cognitive Sciences.

However medical treatments acquire halos or horns, only well-conducted, transparent clinical trials can identify effective medical treatments, such as childhood vaccinations, and their potential side effects, Ioannidis says. But an ominous research trend is gathering steam, he warns. Researchers, professional societies, funders and lobbyists increasingly push to replace clinical trials of randomly selected volunteers with studies of interventions for select samples of people who eat certain foods, take certain drugs or behave in certain ways. The latter investigations may inflate a treatment’s effects even more than online patient reviews do, Ioannidis asserts.

Perhaps a new wave of studies showing how evidence-based hearsay misrepresents many medications will be just what the doctor ordered to resuscitate evidence-based medicine, he muses.