The title of this post: “Not all scientific statements have equal weight” was written by Carl Sagan in his brilliant book Broca’s Brain. It is a statement you should write on a post-it to keep by your monitor as you browse, if that is your cup of tea, the online intellectual fight on such nerve touching issues as the safety of genetically modified organisms (GMO), evolution vs. creationism, climate change, and many other topics that are, at the end of the day, empirically verifiable. It should sound in your brain after each and every scientific claim you read on the Internet. (In Carl Sagan’s voice too.)
In these intellectual disputes of which there are many positions, you’ll see many studies thrown this way and that as authoritative evidence that one’s own position is the correct one. (In fact, this just occurred on my Q&A with a scientist on GM food; a tirade by a commenter about how Dr. Folta is incorrect despite studying in the field, that scientific labs are ego-driven, and to top it off, using a Reuters article as the coup d’etat, as if the media’s representation of an incredibly complicated topic as is molecular biology means anything more than nothing.)
So, to do what little I can to reverse the entropy of reason, I wrote this post. As I’ve mentioned in quite a few debates (the informal kind), both on this blog and off, it matters little what a single scientific study says, peer-reviewed or otherwise. Why, you might ask? Well, because it hasn’t been replicated, and replication is the hallmark of good science (as well as scientific consensus, which means the data is likely to have been interpreted correctly). Many scientific fields use P < 0.05 as the barometer of accepting a study as evidence, but this is only the first step in any first of its kind study. (Slightly over simplified, but for the most part, true.) That means there was less than a 5% chance of the result being consistent with chance. But being “consistent with chance” is not the same thing as “occurred by chance,” so how do you tell if the data is due to chance or not, authoritatively, if it has only been done once? Obviously, you replicate, or hope that other scientists do. A better way of expressing the P-value is to say if you repeated the experiment 100 times, then random results should occur 5 times (if P = 0.05). As Dick Taverne wrote in The March of Unreason: “The great virtue of science is that its truths must be reproducible and are independent of time, place, and personality.” Until such time as they’re replicated, the results can be interesting, fruitful, hopeful, or mind-boggling, but short of extraordinary circumstances, never final; just preliminary, the start of the scientific journey along the yellow brick road.
While it does matter what a single study says, albeit quite little (it serves more to inform the hypothesis, methodology, and expected outcomes of future related experiments to validate or falsify), it matters more what the scientific literature says, and, just as importantly, what the scientific consensus is of that literature. As Jerry Coyne said in my interview with him: “a scientific consensus is more than an opinion.” Think of it this way: the scientific literature of a particular field is an engine, full of gears and cogs. Most of them spin forward, though a few spin backwards. So, if you want the whole picture, you must look at the engine, while the consensus is the mechanic, not at a single study, to properly absorb and hopefully understand, the field. For while most gears spin forward, some spin backwards, and unless you know what you’re looking at, you may confuse a single backwards spinning gear for reality.
Let’s see it play out in a few examples.
Organic vs. Conventional
Consider the nutritional difference between organic and conventional foods in the online tug of war between proponents of either/or methodology. (I consider the differences somewhat of a false dichotomy, but it is easier to write and read about it like this, so bear with me.) In the one corner, we have a Stanford meta-analysis that took 237 separate studies that attempted to understand the nutritional difference, if any, over 50 years, between organic and conventional produce. (A meta-analysis treats them all as one single study.) In the other corner, a sprightly high school student who recently fed fruit flies both organic and conventional food, and came to the conclusion that fruit flies who eat of the organic produce fared better than those fed conventional produce, which bought it wide circulation on the Internet.
Which has more scientific weight? Obviously, the Stanford study. The why is obvious; it had a wider reach, tested more parameters, had many studies testing the same, or similar, things, and some of the studies actually looked at the results in people, instead of only in fruit flies.
Though, to be fair, both studies had exceptions; after all, science is messy being as it reflects the messy world we inhabit. Organic milk seems to fare better than conventional milk in healthy fatty acids and organic produce seems to have a little less pesticides—though not in an amount all that relevant to our biology, especially if you wash your produce—from the Stanford study; and not all fruit flies that ate organic food fared better than their conventional brethren. Although the insignificant—to us—amounts of pesticide found on conventional produce are known to affect the fruit flies negatively, especially since they’re an insect that pesticides are deliberately attuned to attack, so it’s safer to say that organic food doesn’t help fruit flies, but, as Stephen Propatier wrote on the Skeptoid website: “It is a far more plausible conclusion that conventional produce is more toxic to fruit flies.” And, lastly, all the flies on this experiment died sooner than flies fed processed fly food—so, you know, go figure.
The latter study—fruit fly experiment—did not outweigh the Stanford study. We have no solid reason to do away with the Stanford meta-analysis, no matter which way you interpret the fruit fly experiment.
The safety of GM food is another hotly contested debate: on one side we have activists who, along with a vocal minority of society, are adamant that those who eat of the GM apple tree are giving up their health one bite at a time. In the other corner, the scientific consensus assures us that foods of a GM origin are functionally equivalent and safe to eat. Who to trust? That is to say, whose scientific statement is more equal ?
Let’s dive in and find out. What does the evidence say? The evidence says a lot, actually, but it all lines up one side. There are 600 studies, all peer-reviewed, that show that GM produce is safe to eat. “Funding bias!” some will say, and that indeed could be an influencing factor, but of those 600, 25% are independently funded, and the independently funded studies line up with the industry-funded studies. “Conspiracy!” others will say; and while that is possible, it is unlikely. So unlikely in fact, that no doubt that if it were true, thousands of molecular biologists would have broken rank to alert the world, as climate scientists have done in their own sphere of influence. That has not occurred. After all, scientists are people too. They have wives, families, brothers, sisters, and friends like the rest of us who partake in GM food, either directly in purchasing produce from the supermarket, or indirectly, by eating meat grown having fed on GM produce.
But, what about the other side? The activist side; surely, they mean well, and if they are so screaming from their online rooftops there must be some problem. There is the study from the Entropy journal that links glyposhate to all manner of health problems such as autism and Alzheimer’s disease. Then there are Putzai’s potatoes, and don’t forget Seralini’s tumor-stricken mice from last year. They surely must count for something, no? Unfortunately, for activists, they do not. They’ve all been rejected by the greater scientific community, and not for ideological purposes (science is, after all, without ideology; it tries its best to be objective, hence peer-review, bias-removal, P-values, replication, publishing of all results and methodologies etc.). Scientists simply have a very high standard of evidence, and the studies just mentioned, as well as others, fall well short of that standard. Let’s quickly go through those three just mentioned, however.
Seralini et al.
Seralini’s mice were predisposed to tumors; the study lacked proper controls, as well as far too few mice; the full results and methodology of the experiment weren’t published, nor divulged upon request by the European Food and Safety Authority; Seralini et al. published to the media first; and Seralini and the organization that funded him have a long history of anti-GMO activism putting in doubt the integrity of their experiment.
The linking of glyphosate and a 101 dalmatians (diseases) in the Entropy journal was done by a computer scientist and an independent researcher who specializes in chemical engineering; both were untrained, un-specialized, and therefore unable to properly understand the nuances of molecular biology and immunology, and notwithstanding their ‘expertise,’ or lack thereof, the ‘study’ had no data to speak of; it just linked glyphosate and (pick your favourite disease) willy nilly with neither rhyme nor reason (well, unless you consider linking glyphosate to disease based upon public myths as science). One might say that it was the two authors opinions and assertions that glyphosate was so bad. Good thing that in science, opinions count for—in the wise words of Darian Fawkes—”a little less than nothing.“ They also based their ‘assertions’ on Seralini’s et al discredited study, Wakefield’s discredited vaccine study, among others. This study actually regresses from the scientific starting point, it is merely storytelling and myth-making, and adequately plays the part of attention grabber. (The rhyme and reasoning of these studies, it seems to be, goes something like this: as long as you grab the publics attention, it doesn’t matter how bad the science is, they’ll fall for it hook, line, and sinker; clueless activists will take it from there. After all, we’re still dealing with the fallout from Wakefield’s bogus study on MMR vaccinations.)
Lastly, what becomes of Putzai’s potatoes? Surely, that study prevented the rise of GM potatoes that would have harmed society and children? Again, unfortunately, no. Academic’s Review did a great job distilling the problems with his study, which I will summarize here: (1) No conclusions could be made from the study’s results; (2) No differences were seen between the two groups of animals, GM-fed and non-GM fed; (3) the study had a flawed design, such as feeding potatoes to rats (potato is toxic to rats); (4) Lancet, the medical journal, published the paper despite objections from the reviewers.
(Now, this is not to say there aren’t problems with GM technology, nothing is perfect, and there are always niggles here, there, and yonder. But to distort the debate, especially with such nonsensical studies neither supported by its own data nor by the greater scientific community, is folly and serves only to poison the well.)
Callan Bentley expressed the sentiment I wish to impress upon you with this post by way of the organic / conventional dichotomy, safety of GMOs, and many other false debates out there, albeit with a slight modification in square brackets:* “Why is [it] so difficult to show unambiguously? Why can’t we convince people who are the toughest to convince? Where’s the really compelling evidence we seek?”
When you can’t convince the people whose full-time job it is to know their field inside and out in an occupation where you get famous and climb the ladder to success by proving other scientists wrong, it’s pretty safe to say you’re not onto something. Calling out conspiracy, funding bias, and ego-driven lab fabrications only further the point home to those on the other side and those who observe from the sidelines trying to make up their minds.
Not all scientific statements have equal weight, some aren’t even science, and some almost seem to be made up. Hopefully, this post made that obvious. If you have any questions, shoot, and if you have a friend (or friends) that would benefit from knowing this, share this article so that they might have their foundations shaken just a little, or a lot. As Lawrence Krauss is so fond of saying over and over again:** “I hope that every [person] at one point in their life has the opportunity to have something that is at the heart of their being, something so central to their being that if they lose it they won’t feel they’re human anymore, to be proved wrong because that’s the liberation that science provides. The realization that to assume the truth, to assume the answer before you ask the questions leads you nowhere.” I’ve had many such occurrences, it reminds me I’m human, and therefore, an idiot. For everything I know, there’s a thousand things I could never dream to know. You too.
So, out there on the world wide wild web when reading controversial science stories, especially from the media, keep in mind Carl Sagan’s truthful quip: “Not all scientific statements have equal weight.”
I’d like to conclude this post as it began, with a quote by Carl Sagan: “Our common-sense intuitions do not count. What does count is experiment.”