Flawed studies, agenda-driven research, and open-door journals are putting the public at risk. PM investigates who creates it, who promotes it – and how you can fight back. By Sarah Fecht
In 2012, radiation expert Robert Emery examined a research paper in his office at the University of Texas Health Science Centre that declared a health crisis was at his door. Authors Joseph Mangano and Janette Sherman claimed they could link 14 000 American deaths to fallout from Japan’s March 2011 Fukushima nuclear accident. Mangano, the executive director of the Radiation and Public Health Project (RPHP), holds a master’s degree in public health; Sherman is an adjunct professor of environmental studies at Western Michigan University in Kalamazoo. “Public health means prevention,” Mangano told Popular Mechanics. “It means removing any potential harm from our world.”
Mangano and Sherman tallied the number of deaths in about 100 US cities in the 14 weeks after the Fukushima accident, compared with the same time period from the year before. Projected across the entire country, the difference amounted to thousands of “excess” deaths. The study was titled “An Unexpected Mortality Increase in the United States Follows Arrival of the Radioactive Plume from Fukushima: Is There a Correlation?”
“I read the thing and was taken aback,” says Emery, who has a doctorate in public health and is a licensed medical health physicist. The study implied that fallout from Fukushima caused 484 deaths in the city where his university is located, Houston. If there had been radiation-related deaths in Texas, Emery was well positioned to know about them. Following the disaster in Japan, he supervised the effort to set up extra air-sampling stations and Geiger counters throughout Houston to monitor any increase in radioactivity; elevated levels were not found.
Radiation from Fukushima did reach some parts of the US, but Emery says the doses were so low that there is no way they could have caused the immediate fatalities the study claimed. Even at Fukushima, where workers absorbed radiation doses thousands of times higher than Americans on the other side of the Pacific Ocean, no one died from acute exposure.
The paper’s conclusions were based on a remarkably simplistic correlation that didn’t stand up to further scrutiny; at least one scientist pointed out that RPHP saw an increase only because it counted deaths from 119 cities in the year after Fukushima versus 104 cities in the year before. (The authors say the disparity was due to incomplete data for some cities.)
The Mangano and Sherman paper is a prime example of a troubling new trend in which junk science is becoming harder to distinguish from rigorous research. It is an example of activists using the trappings of science to influence public opinion and policy.
Today there are cottage industries that produce and disseminate skewed research in publications that masquerade as legitimate science journals. Celebrities and mainstream media outlets then tout the results, so that even retracted or clearly biased research can reach larger audiences than ever before. These studies cause real harm – for instance, by denouncing lifesaving vaccines and vilifying foods that could ease famine in developing countries.
People who produce junk science often come from outside the scientific mainstream, and they bend the rules of research in an attempt to legitimise their personal beliefs, says Mark Hoofnagle, a surgery resident at the University of Maryland who runs the science-monitoring blog Denialism.com. “What if your ideology is simply not supported by the evidence?” he says. “You can change your mind or you can hijack the system.”
Joe Mangano is admittedly fixated on the risks of radiation. Whereas low-level radiation can increase a person’s chances of developing cancer over the course of a lifetime, Mangano hypothesises that it also has short-term effects by exacerbating problems in people who are already sick.
“Saying low-dose radiation is harmless is like saying that, if you smoke five cigarettes a day or less, that’s a low dose so you don’t need to do the studies,” he told PM. But the lack of evidence linking low-dose radiation to immediate damage was lost in the headline of the RPHP press release: “Medical Journal Article: 14 000 US Deaths Tied to Fukushima Reactor Disaster Fallout”.
“This is the ‑rst peer-reviewed study published in a medical journal documenting the health hazards of Fukushima,” the press release announced.
Ten days after the paper had been published, United Press International ran a story about the study, quoting Mangano as saying the “findings are important to the current debate of whether to build new reactors, and how long to keep ageing ones in operation”. A San Diego ABC affiliate later ran a news segment describing the dangers of fallout from Fukushima, interviewing Mangano and describing “which children may be vulnerable”.
On-air reporter Michael Chen noted that “critics say there is no direct evidence” that the low radiation levels could cause harm, but there were no quotes from or on-air interviews with any of those critics.
When a journal publishes a paper outside its area of expertise it should raise a red flag, according to Ivan Oransky, a medical doctor and the cofounder of the Retraction Watch blog, which reports on the retractions of scientific papers. The Fukushima study deals with epidemiology. It appeared, however, in the International Journal of Health Services (IJHS), which specialises in public and social policy, including “the articulation of science and ideology in the pursuit of health”.
The IJHS is peer reviewed. Peer review alone, though an important step in establishing scientific credibility, is no guarantee of accuracy. During the typical review process, a scientist submits his or her research paper to a journal. If the journal editors want to publish it, they usually ask two experts to assess anonymously whether the findings are novel, important and supported by the data. But journal reviewers don’t typically scrutinise raw data, re-run the statistical analyses, or look for evidence of fraud. “What they’re reviewing are mostly advertisements of research rather than the research itself,” says Stanford epidemiologist John Ioannidis.
With new Web-publishing platforms, it’s easy to set up a journal that publishes papers indiscriminately and calls them peer-reviewed. There are journals dedicated to sequencing Bigfoot’s genome or supporting creationism, for example. All it takes is a Web site and an e-mail address.
Open-access journals make their articles available free – instead of charging for subscriptions, they charge authors. “There’s an inherent conflict of interest,” says Jeffrey Beall, a librarian at the University of Denver who maintains a list of questionable open-access journals. “The more papers they accept, the more money they make.” He says some open-access journals, such as PLOS One, manage that conflict well, but others will publish almost anything to collect the fee.
In 2013, scientist-turned-journalist John Bohannon conducted a sting operation to determine just how easy it is to publish bogus research in open-access journals. Under a false name, he wrote a flawed paper about a fake drug and sent it to 304 open-access journals. Of the 255 journals that responded, 98 rejected the article and 157 accepted it. Once erroneous information is rubber-stamped as peer-reviewed, it can be futile for mainstream scientists to try to counter it.
Expert push-back did not banish Mangano and Sherman’s work to obscurity. In a paper that came out in the open-access journal Biomedicine International in 2013, Mangano and Sherman link the 1989 shutdown of California’s Rancho Seco nuclear reactor to 4 000 fewer deaths from cancer in Sacramento County. The study is based on the same sort of speculative correlations and selective use of data as the Fukushima paper. “I think these individuals have a bias towards what they believe to be happening,” Emery says. “They’re drawing conclusions that support that bias. Have you ever heard of the Texas sharpshooter? It’s where a guy goes out in the field, shoots bullet holes in a barn, and then paints the target around the bullet holes.”
Despite his work’s scientific flaws, Mangano is popular with antinuclear activists. Actor Alec Baldwin, a supporter of RPHP, wrote an introduction to Mangano’s 2012 book on the risks of nuclear power plants, Mad Science: The Nuclear Power Experiment. In it Baldwin calls him “one of the most dedicated, intelligent, and even-handed public activists I have ever known”.
During his rounds at the children’s hospital of Philadelphia, paediatrician Paul Of‑t often sees patients with vaccine-preventable illnesses. In a recent case, he treated a 4-month-old baby with type B meningitis – an infection of the membranes that surround the brain and spinal cord. The inflammation can result in brain damage, hearing and vision impairment, even death. Vaccinations can prevent type B meningitis, but after the baby recovered, the child’s mother still refused to consider vaccination.
“Although her child suffered and could have died from a vaccine-preventable disease,” Of‑t says, “she still believed that she was doing the right thing.”
The modern antivaccine movement can be traced to a paper published 16 years ago in The Lancet, a respected medical journal. In 1998 a team led by surgeon Andrew Wakefield studied 12 children with developmental delays, and found that eight children with autism had developed their first symptoms shortly after they received the MMR vaccine (for measles, mumps and rubella). But the correlation was weak. “I could go out and produce a paper that says that eight children developed leukaemia within a few months of eating their first peanut butter and jelly sandwich,” Of‑t says, “but it doesn’t mean anything.”
Although scores of scientists denounced Wakefield’s claim, Rolling Stone, Salon, and other media outlets issued dramatic headlines linking MMR to autism, calling the vaccine a lethal injection, and telling the sad stories of children who developed autism after receiving it.
Numerous follow-up studies failed to find evidence of a link between vaccines and autism. And the Wakefield study turned out to be not just awed, but fraudulent.
Journalist Brian Deer, writing in the British Medical Journal and The Sunday Times, revealed in 2004 that Wakefield had been hired by a lawyer to find evidence against the vaccine to support a lawsuit and had falsified data in the 1998 study. The study was retracted in 2010, and medical authorities in the UK stripped Wakefield of his medical licence.
Yet Wakefield’s legacy lives on. One of his chief boosters is former model Jenny McCarthy, who became the unofficial spokeswoman for the antivaccine movement after she announced in 2007 that her son, Evan, had been diagnosed with autism. She blames vaccines, which she has called “injected toxins”. She said in an interview with PBS, “If you ask 99,9 per cent of parents who have children with autism if we’d rather have the measles versus autism, we’d sign up for the measles.”
Before the MMR vaccine was developed, in 1963, nearly all children contracted measles, and one in 1 000 died from the disease. By the late 1990s, when Wakefield’s paper came out, measles had been all but eliminated. The US is now seeing a resurgence: the Centres for Disease Control and Prevention recorded 189 cases in 2013, compared with an annual average of 60 cases in previous years. The CDC cites lack of vaccinations in America as a primary cause of the increase; David Elliman, a doctor with the UK’s Royal College of Paediatrics and
Child Health, calls these kinds of increases the Andrew Wakefield legacy effect. The antivaccine movement doesn’t stop at measles. Parents who refuse to have their children vaccinated for MMR also refuse vaccines for diphtheria, tetanus and whooping cough; the US is seeing a resurgence of these diseases as well. A Web site named Jenny McCarthy’s Body Count uses data from the CDC to tally the number of Americans who have died from vaccine-preventable illnesses since June 2007. As of February 2014, the number had reached 1 336.
Jenny McCarthy now co-stars on The View, a show on the ABC network with more than 3 million viewers. She claims she cured Evan’s autism using diet and detoxification. Meanwhile, Offit, who invented a rotavirus vaccine that has saved the lives of thousands of children, receives hate mail accusing him of being an industry shill; he denies taking money from pharmaceutical companies.
“A parent has a right to make a terrible decision based on misinformation that could hurt or kill their child,” Offit says. “But it’s hard to stand back and watch.”
The foods you eat everyday are killing you, according to a September 2012 study by French researchers. The experiment tested the long-term effects of eating genetically modified corn, and the results were startling. After feeding lab rats genetically modified corn for two years, the researchers reported that some of the animals sprouted tumours the size of kiwifruit and died earlier than rats that ate non-modified corn.
The results, published in the peer-reviewed journal Food and Chemical Toxicology under the title “Long Term Toxicity of a Roundup Herbicide and a Roundup-Tolerant Genetically Modified Maize,” run contrary to previous findings.
In fact, up to 88 per cent of corn and 94 per cent of soy grown in the US is genetically modified, and researchers from the World Health Organisation, the US National Academy of Sciences, and the Royal Society of Medicine have found no ill effects. In 2012, the American Medical Association determined that “bioengineered foods have been consumed for close to 20 years, and during that time, no overt consequences on human health have been reported and/or substantiated in the peer-reviewed literature”.
The press release accompanying the French study was sensational. It announced the “severe toxic effects” of a genetically modified organism (GMO), “even when ingested at extremely low levels”. Led by activist and molecular biologist Gilles-Eric Séralini from the University of Caen, the paper was timed to match the launch of Séralini’s latest anti-GMO book, All of Us Guinea Pigs!, and a film of the same name. The study was released six weeks before a California vote to label genetically modified foods, and those in favour of the proposition forwarded the report to media outlets. Voters rejected the measure by a razor-thin majority (51,4 per cent). But consumers around the world panicked. Food-safety authorities in France launched an emergency investigation. Russia and Kazakhstan suspended imports of GM corn from the United States.
The Séralini paper caused an immediate backlash. Legions of scientists denounced the paper as critically flawed and some accused Séralini of fraud. Among the chief concerns was that he failed to use a large enough number of rats to tell whether the results were real or merely due to chance. (He used 200 rats, divided into 20 treatment groups of 10 rats apiece. A significantly sized study would have used 20 groups of 50 rats apiece.) The low numbers are particularly problematic because he used a rat breed that spontaneously develops tumours up to 87 per cent of the time even when eating standard lab rat food. “It would fail as a lab project from students in an undergraduate class,” says Michael Eisen, a geneticist at the University of California, Berkeley.
“Anybody who was a critical statistician would look at this paper and say that it shouldn’t be published.” When PM interviewed Séralini, his first line of defence was to suggest that his critics have financial ties with Monsanto, the world’s biggest producer of GMOs. He stands by his use of the tumour-prone breed because it is typically used in long-term cancer studies; he dismisses the statistic that 87 per cent develop tumours, since standard laboratory food may contain GMOs and pesticides.
The Séralini paper was retracted by Food and Chemical Toxicology in November 2013, causing more controversy. Papers are typically retracted because of fraudulent data or because the results can’t be reproduced. The journal found no evidence of fraud, and no one has yet tried to reproduce the study, prompting speculations that public outcry led to the retraction.
This set a bad precedent, further eroding public trust in the scientific process and becoming a lightning rod of paranoia for those who believe mainstream science and media are suppressing self-anointed truth tellers. In January 2014, a pair of Georgetown University professors wrote on the Bioethics Forum blog of the Hastings Centre think tank that the retraction “reeks of industry pressure” and is a “black mark on medical publishing”.
By the time the paper was retracted it had been cited in 28 other studies. The discredited paper still provides anti-GMO ammunition for groups such as Greenpeace, the Organic Consumers Association, and GMWatch, which described it in January 2014 as a “pioneering study”.
The anti-GMO campaign has had real-world ramifications. In the Philippines, a potentially life-saving strain of golden rice has been trapped in regulatory limbo for years. The rice, engineered to produce beta carotene, would help fight vitamin A deficiencies. During the 12-year delay, an estimated 3 million children worldwide have died from vitamin A deficiency and many more have gone blind, according to World Health Organisation statistics.
In November 2012, Kenya banned imports of genetically modified food as a result of the Séralini study. Ten million Kenyans already suffer from chronic food shortages and poor nutrition, and now they’ll have one less tool with which to battle worsening hunger.
Institutional changes could help the media and the public differentiate good science from agenda-driven efforts. Psychologist Brian Nosek of the University of Virginia is a founding member of the Centre for Open Science, which aims to make it easier for scientists to check each other’s work by providing a platform on which to share data and research protocols. Promoting openness deters fraud, Nosek says. “If everyone is transparent about their research process and data, then faking becomes really inconvenient,” he says.
A new site called PubPeer allows readers to leave comments, which anyone can view, on published journal articles. According to the Web site, its anonymous founders want to foster post-publication discussion – its founding principle is that the entire scientific community is much better at evaluating a paper than just two reviewers.
It’s easy to blame the impact of junk science on sloppy experiments, irresponsible reporters, or a failure of peer review. But even after it’s debunked, junk science sticks because it preys on the public’s fear and distrust. Ultimately, junk science can be dispelled only if individuals think like scientists: evaluate all the evidence and try to disprove your own preconceptions.
DIY Debunking: 6 junk science warning signs
Was the paper published in a peer-reviewed journal?
“If it wasn’t, you have no reason to trust it,” says Ivan Oransky, former executive editor at Reuters and cofounder of the blog Retraction Watch. “The peer-review system, as flawed as it is, stands between us and really poor science.” Also, find out if the journal or its publisher is on Jeffrey Beall’s list of questionable open-access journals, at scholarlyoa.com.
What is the journal’s impact factor?
The impact factor is the average number of times a journal’s papers are cited by other researchers. You can usually find this information on the journal’s home page or by searching “impact factor” along with its name. Check out the impact factor of other journals in that field of research to see how they compare.
Do the researchers mostly cite their own papers?
If so, this is a red flag that they are promoting views that fall outside the scientific consensus. Citations are listed at the end of a paper.
How many test subjects were used?
A large number of test subjects makes a study more robust and reduces the likelihood that the results are random. In general, the more questions a paper asks, the greater its sample size should be. Most reliable papers contain something called a p-value, which measures the probability (p) that a study’s results occurred by random chance. In science a p-value of 0,05 suggests the study’s conclusions may be meaningful. Smaller p-values are better.
Does it rely on correlation?
Cigarette smoking has declined dramatically in the US in the past few decades, and so has the national homicide rate. But just because two events occur at the same time doesn’t mean that one caused the other.
Have the results been reproduced?
To find out, search the paper’s name on Google Scholar and click on the Cited By link beneath the name. This will list other researchers who mention the paper in their own publications, and may also give you a clearer view of how other researchers critiqued the paper.