Again, the Internet contends with another negative take on GMOs, like Seralini’s rat-cancer study from last year. This “study” by Judy Carman involves following pigs fed GM and non-GM feed over 22.7 weeks and trying to find something, anything, wrong at all with the GM-fed pigs while ignoring everything that showed no effect or a positive effect. I don’t have time enough to go through the study, so I’ll briefly summarize the findings of Mark Lynas’ take on the study, as well as another from Weed Control Freaks to show you the pseudoscience indicators:
1st Warning Sign: The results were published in a journal not indexed by PubMed with a low-impact factor.
What this means: Scientists don’t take the journal seriously, it has no credibility, or both.
2nd Warning Sign: The study was funded by The Organic Federation of Australia; i.e., they’re potentially biased.
What this means: Lynas articulates the position well in his own take – imagine if Monsanto sponsored a study that showed GMOs are safe. The Antis would lose their minds and completely discard it. That they don’t do the observe shows the hypocrisy of their position. Bad science is bad science regardless of whose, in this case, attempting to do science.
3rd Warning Sign: The study’s protagonist has a history of anti-GMO activism, as well as being a cohort of Gille Seralini whose history of conducting flawed science in service to his anti-GMO activism makes him suspect.
What this means: A scientist’s biases color the interpretation of their results. This has a corollary in the 1st Warning Sign, as such a lowly journal’s filters might not be quite as stringent as a scientist would expect. Carman went into the study with an expectation that GMOs are bad, and was, therefore, more likely to interpret any evidence through that lens, which is, after all, exactly what happened. Also, survey the furore over the overlap between the US gov’t and Monsanto. There are thousands, perhaps millions, of people who think that Monsanto controls the US gov’t, and as a result of that belief, discount public efforts to soothe anti-GMO fears. Again, that the observe is not even a talking point reveals the hypocrisy of the position.
4th Warning Sign: They claim they have no commercial interests, despite many involved with the study standing to gain from increased organic sales if GMOs were shown to be unsafe.
What this means: The study was funded by The Organic Federation of Australia. They receive dues from organic farmers and the more popular organic farming is, for example, through swayed consumer opinion influencing planted acres, would increase their financial stakes.
5th Warning Sign: Funding came from Verity Farms (natural product outfit) and The Institute of Health and Environmental Research, which seems to be entirely dedicated to anti-GMO activism. Their funding sources are not disclosed, though they solicit donations. Jeffrey Smith and Arpad Putzai are both listed as acknowledgments; the former has no relevant science experience and the latter’s GM potato study that supposedly showed harm from GM potatoes is not accepted by the scientific community for methodological errors.
What this means: Imagine a biologist listed Lysonky as an acknowledgement on an evolutionary biology paper. His version of evolutionary biology was a causative factor in the Soviet famines
6th Warning Sign: The study showed pictures of severely inflamed pigs stomachs from the GM-fed group against non or mild inflammation from the non-GM fed pigs. Very similar to what Seralini did in his rat study last year. Showing big, fat disgusting tumors from the GM-fed mice, but neglect to show pictures of the control rats who also had tumors.
What this means: media bait. Juicy, clearly selected photos intended to pack an emotional punch and sway consumer confidence. This is not the sign of an objective analysis intended to impart facts.
However, the above are just indicators. In and of themselves, they don’t automatically mean that it is pseudoscience (therefore, false), but, the indicators taken together paint a picture, and it would be irresponsible for anyone to accept such claims at face value. Moreover, in science, the data speaks first and foremost, so let’s dig in. I’ll start by summarizing some of the findings.
Fifteen-percent of the non-GM fed pigs had heart abnormalities, compared to six-percent of GM-fed pigs. Hmm, that wasn’t anywhere in the press release, but there’s more. Liver problems were twice as prevalent in non-GM pigs as GM-fed pigs. One will note, that in all credulous mentions of this study on the Internet, there is no mention anywhere of these mildly positive findings. This data strikes to the heart of the one-note tone-deaf claims reverbarating around the Internet in support of this study.
In the four categories of stomach inflammation that was, apparently, the subject of the study: None, Mild, Moderate, Severe. In all but the severe section, the GM-fed pigs were better off than their conventionally fed brethren. That tells us that there is no dose-dependent mechanism between the GM feed and the stomach inflammation, so the likelihood that there is a correlation between GM-fed pigs and inflammation is next to none. I happen to agree with Lynas’ conclusion (emphasis mine):
“My judgement is that, as with Seralini, this study subjects animals to inhumanely poor conditions resulting in health impacts which can then be data-mined to present ‘evidence’ against GMO feeds. Most damning of all, close to 60% of both sets of pigs were suffering from pneumonia at the time of slaughter – another classic indicator of bad husbandry. Had they not been slaughtered, all these pigs might well have died quickly anyway. No conclusions can be drawn from this study, except for one – that there should be tighter controls on experiments performed on animals by anti-biotech campaigners, for the sake of animal welfare.”
I highlight “data-mined” for a very specific purpose. In any scientific study worth its salt, the hypothesis is present before the experiment begins. This is to guard against scope-creep (or, in science, hypothesis-creep, though that doesn’t sound so good). What that means is that the scientist can’t change the purpose of the study when the results don’t match the hypothesis. However, when the conclusion is mined from the data after-the-fact, it allows fluke readings to seem to have a causative effect. In science, statistical significance (a non-random effect) is what all studies strive for. A statistically significant effect is defined as any event that falls below P < 0.05, which means, in layman speak, a 5% or less chance of the result being due to chance. (However, deriving a result at or below 0.05 does not actually tell you if it is due to chance or causative, only the likelihood that it was. That is, with a statistical significance of 0.05, if you repeated the experiment 20 times, you should get 19 consistent results and one fluke result.) Bringing that understanding to bear upon this study, Carman et al. analyzed some 20 factors, and found that severe stomach inflammation in GM-fed pigs was statistically-significant compared to the non-GM-fed pigs. Yet, since that hypothesis was absent prior to the experiment, it neatly falls under the more probable explanation that since this result was absent a dose-dependent result, that it is was a fluke result. Without a follow-up study, it is the only rational position to take.
However, it gets worse. Andrew Kniss over at Weed Control Freaks has shown that their P-values were sneakily derived to ensure that they passed the statistical significance test. So not only did it not pass statistical significance which would obligate the scientific community to replicate the results, they performed funky mathematics just to hit their biased after-the-fact hypothesis!
Here’s how they did it. They separated the four groups of inflammation, and ran separate statistical tests on each so that their P-values could limbo under the P < 0.05. That is, they had to obfuscate the mathematics just to get the right values to validate their bias (I refuse to call it a hypothesis), which, as I mentioned before, was chosen after the experiment run. When all the results were properly subjected to statistical analysis by Kniss, the P-values came out to be 0.2142 if running a t-test analysis, and 0.2081 running a Wilcox test. That is, roughly a 20% chance that the results are consistent with chance. Twenty-percent is a very poor marker for a causal effect. Surprisingly, there is more: hen the males and females were separated and analyzed again, the results were even more stark:
Male t-test = 0.5667 (56.7%) – Male wilcox test = 0.5669 (56.7%) – Female t-test = 0.2564 (25.6%) – Female wilcox test = 0.2408 (24.1%)
Andrew Kniss concludes appropriately:
“If I were to have analyzed these data, using the statistical techniques that I was taught were appropriate for the type of data, I would have concluded there was no statistical difference in stomach inflammation between the pigs fed the two different diets. To analyze these data the way the authors did makes it seem like they’re trying to find a difference, where none really exist.”
Prof. David Spiegelhalter wrote to Lynas thatthere were so many statistical tests that one was bound to come up positive, and he is right. Twenty variables in total were run, and only one, that they had to fish for after-the-fact, showed a result they wanted. (Which, as above, if the much wanted P < 0.05 is aimed at, would have shown up anyway as every twenty tests, one would turn out random.) Then again, it’s not about the science; I very much doubt that Carman cares about how debunked, criticized, and wrong her study is, she and her cohorts only need to create the media attention with the facade of science to fool the public, which Reuters and dozens of others have already bought hook, line, and sinker.
Cami Ryan has summarized many more rebuttals from around the web by scientists of this nonsense study. David Tribe, aka The GMO Pundit, also has a detailed breakdown on his site, and I encourage you to read both Mark Lynas’ review (which has been updated) and Andrew Kniss’ statistical analysis in full to get a flavour of peer-review works.