Nine Essential Questions to Evaluate Medical News
WRITTEN BY: Marie Ennis; POSTED: May 23, 2021
Ben Goldacre writing in “Bad Science” classified science reporting as falling into three categories — wacky stories, scare stories, and breakthrough stories; the last of which he views as ”a more subtly destructive category of science story”.
Whether you get your news through digital or traditional means, you can’t fail to notice the regularity with which journalists report on the latest medical breakthroughs.
Some of these reports are sensationalist (“coffee causes cancer”) and fairly easy to dismiss; but do you know how to separate fact from fiction when it comes to less sensationalist headlines?
The following questions are designed to help sort the signal from the noise next time you read the latest news story heralding a medical breakthrough.
1. Does the article support its claims with scientific research?
2. What is the original source of the article?
If the article cites scientific research you should still treat the findings with caution. Always consider the source. Find out where the study was done. Who paid for and conducted the study? Is there a potential conflict of interest?
3. Does the article contain expert commentary to back up claims?
Look for expert independent commentary from doctors or other healthcare providers to explain the findings (there should be an independent expert source quoted — someone not directly connected with the research).
4. Is this a conference presentation?
Journalists frequently report on research presented at large scientific meetings. It’s important to realize that this research may only be at a preliminary stage and may not fulfill its early promise.
5. What kind of clinical trial is being reported on?
If the news relates to results from a clinical trial, it’s important you understand how, or even if, the results apply to you. Quite often, news publications report on trials that have not yet been conducted on humans. Many drugs that show promising results in animals don’t work in humans.
6. What stage is the trial at?
Research studies must go through several phases before a treatment can be considered safe and effective, but many times journalists report on early phase trials as if these hold all the answers. The testing process in humans is divided into several phases:
- Phase I trials: Researchers test a new drug or treatment in a small group of people for the first time to evaluate its safety, determine a safe dosage range, and identify side effects.
- Phase II trials: The drug or treatment is given to a larger group of people to see if it is effective and to further evaluate its safety.
- Phase III trials: The drug or treatment is given to large groups of people to confirm its effectiveness, monitor side effects, compare it to commonly used treatments, and collect information that will allow the drug or treatment to be used safely.
7. How many people did the research study include?
In general, the larger a study the more you can trust its results. Small studies may miss important differences because they lack statistical power.
8. Did the study include a control group?
A control group allows researchers to compare outcomes in those who receive treatment with those who don’t. The gold standard is a “randomized controlled trial”, a study in which participants are randomly allocated to receive (or not receive) a particular intervention (e.g. a treatment or a placebo).
9. What are the study’s limitations?
Many news stories fail to point out the limitations of the evidence. The limitations of a study are the shortcomings, conditions or influences that cannot be controlled by the researcher. Any limitations that might influence the results should be mentioned in the study’s findings, so always read the original study where possible.
- Sense about Scienceis an independent charity that champions the public interest in sound science and ensures evidence is recognized in public life and policymaking.
- Trust It or Trashis a tool to help you think critically about the quality of health information (including websites, handouts, booklets, etc.).
- Understanding Health Research(UHR) is a free service created with the intention of helping people better understand health research in context. It gives clear and understandable explanations of important considerations like sampling, bias, uncertainty, and replicability.