Not All Scientific Statements Have Equal Weight

The title of this post: “Not all scientific statements have equal weight” was written by Carl Sagan in his brilliant book Broca’s Brain. It is a statement you should write on a post-it to keep by your monitor as you browse, if that is your cup of tea, the online intellectual fight on such nerve touching issues as the safety of genetically modified organisms (GMO), evolution vs. creationism, climate change, and many other topics that are, at the end of the day, empirically verifiable. It should sound in your brain after each and every scientific claim you read on the Internet. (In Carl Sagan’s voice too.)

In these intellectual disputes of which there are many positions, you’ll see many studies thrown this way and that as authoritative evidence that one’s own position is the correct one. (In fact, this just occurred on my Q&A with a scientist on GM food; a tirade by a commenter about how Dr. Folta is incorrect despite studying in the field, that scientific labs are ego-driven, and to top it off, using a Reuters article as the coup d’etat, as if the media’s representation of an incredibly complicated topic as is molecular biology means anything more than nothing.)

So, to do what little I can to reverse the entropy of reason, I wrote this post. As I’ve mentioned in quite a few debates (the informal kind), both on this blog and off, it matters little what a single scientific study says, peer-reviewed or otherwise. Why, you might ask? Well, because it hasn’t been replicated, and replication is the hallmark of good science (as well as scientific consensus, which means the data is likely to have been interpreted correctly). Many scientific fields use P < 0.05 as the barometer of accepting a study as evidence, but this is only the first step in any first of its kind study. (Slightly over simplified, but for the most part, true.) That means there was less than a 5% chance of the result being consistent with chance. But being “consistent with chance” is not the same thing as “occurred by chance,” so how do you tell if the data is due to chance or not, authoritatively, if it has only been done once? Obviously, you replicate, or hope that other scientists do. A better way of expressing the P-value is to say if you repeated the experiment 100 times, then random results should occur 5 times (if P = 0.05). As Dick Taverne wrote in The March of Unreason: “The great virtue of science is that its truths must be reproducible and are independent of time, place, and personality.” Until such time as they’re replicated, the results can be interesting, fruitful, hopeful, or mind-boggling, but short of extraordinary circumstances, never final; just preliminary, the start of the scientific journey along the yellow brick road.

While it does matter what a single study says, albeit quite little (it serves more to inform the hypothesis, methodology, and expected outcomes of future related experiments to validate or falsify), it matters more what the scientific literature says, and, just as importantly, what the scientific consensus is of that literature. As Jerry Coyne said in my interview with him: “a scientific consensus is more than an opinion.” Think of it this way: the scientific literature of a particular field is an engine, full of gears and cogs. Most of them spin forward, though a few spin backwards. So, if you want the whole picture, you must look at the engine, while the consensus is the mechanic, not at a single study, to properly absorb and hopefully understand, the field. For while most gears spin forward, some spin backwards, and unless you know what you’re looking at, you may confuse a single backwards spinning gear for reality.

Let’s see it play out in a few examples.

Organic vs. Conventional

Consider the nutritional difference between organic and conventional foods in the online tug of war between proponents of either/or methodology. (I consider the differences somewhat of a false dichotomy, but it is easier to write and read about it like this, so bear with me.) In the one corner, we have a Stanford meta-analysis that took 237 separate studies that attempted to understand the nutritional difference, if any, over 50 years, between organic and conventional produce. (A meta-analysis treats them all as one single study.) In the other corner, a sprightly high school student who recently fed fruit flies both organic and conventional food, and came to the conclusion that fruit flies who eat of the organic produce fared better than those fed conventional produce, which bought it wide circulation on the Internet.

Which has more scientific weight? Obviously, the Stanford study. The why is obvious; it had a wider reach, tested more parameters, had many studies testing the same, or similar, things, and some of the studies actually looked at the results in people, instead of only in fruit flies.

Though, to be fair, both studies had exceptions; after all, science is messy being as it reflects the messy world we inhabit. Organic milk seems to fare better than conventional milk in healthy fatty acids and organic produce seems to have a little less pesticides—though not in an amount all that relevant to our biology, especially if you wash your produce—from the Stanford study; and not all fruit flies that ate organic food fared better than their conventional brethren. Although the insignificant—to us—amounts of pesticide found on conventional produce are known to affect the fruit flies negatively, especially since they’re an insect that pesticides are deliberately attuned to attack, so it’s safer to say that organic food doesn’t help fruit flies, but, as Stephen Propatier wrote on the Skeptoid website: “It is a far more plausible conclusion that conventional produce is more toxic to fruit flies.” And, lastly, all the flies on this experiment died sooner than flies fed processed fly food—so, you know, go figure.

The latter study—fruit fly experiment—did not outweigh the Stanford study. We have no solid reason to do away with the Stanford meta-analysis, no matter which way you interpret the fruit fly experiment.

Another example!

GMO Safety

The safety of GM food is another hotly contested debate: on one side we have activists who, along with a vocal minority of society, are adamant that those who eat of the GM apple tree are giving up their health one bite at a time. In the other corner, the scientific consensus assures us that foods of a GM origin are functionally equivalent and safe to eat. Who to trust? That is to say, whose scientific statement is more equal ?

Let’s dive in and find out. What does the evidence say? The evidence says a lot, actually, but it all lines up one side. There are 600 studies, all peer-reviewed, that show that GM produce is safe to eat. “Funding bias!” some will say, and that indeed could be an influencing factor, but of those 600, 25% are independently funded, and the independently funded studies line up with the industry-funded studies. “Conspiracy!” others will say; and while that is possible, it is unlikely. So unlikely in fact, that no doubt that if it were true, thousands of molecular biologists would have broken rank to alert the world, as climate scientists have done in their own sphere of influence. That has not occurred. After all, scientists are people too. They have wives, families, brothers, sisters, and friends like the rest of us who partake in GM food, either directly in purchasing produce from the supermarket, or indirectly, by eating meat grown having fed on GM produce.

But, what about the other side? The activist side; surely, they mean well, and if they are so screaming from their online rooftops there must be some problem. There is the study from the Entropy journal that links glyposhate to all manner of health problems such as autism and Alzheimer’s disease. Then there are Putzai’s potatoes, and don’t forget Seralini’s tumor-stricken mice from last year. They surely must count for something, no? Unfortunately, for activists, they do not. They’ve all been rejected by the greater scientific community, and not for ideological purposes (science is, after all, without ideology; it tries its best to be objective, hence peer-review, bias-removal, P-values, replication, publishing of all results and methodologies etc.). Scientists simply have a very high standard of evidence, and the studies just mentioned, as well as others, fall well short of that standard. Let’s quickly go through those three just mentioned, however.

Seralini et al.

Seralini’s mice were predisposed to tumors; the study lacked proper controls, as well as far too few mice; the full results and methodology of the experiment weren’t published, nor divulged upon request by the European Food and Safety Authority; Seralini et al. published to the media first; and Seralini and the organization that funded him have a long history of anti-GMO activism putting in doubt the integrity of their experiment.

Glyphosate Suppression

The linking of glyphosate and a 101 dalmatians (diseases) in the Entropy journal was done by a computer scientist and an independent researcher who specializes in chemical engineering; both were untrained, un-specialized, and therefore unable to properly understand the nuances of molecular biology and immunology, and notwithstanding their ‘expertise,’ or lack thereof, the ‘study’ had no data to speak of; it just linked glyphosate and (pick your favourite disease) willy nilly with neither rhyme nor reason (well, unless you consider linking glyphosate to disease based upon public myths as science). One might say that it was the two authors opinions and assertions that glyphosate was so bad. Good thing that in science, opinions count for—in the wise words of Darian Fawkes—”a little less than nothing. They also based their ‘assertions’ on Seralini’s et al discredited study, Wakefield’s discredited vaccine study, among others. This study actually regresses from the scientific starting point, it is merely storytelling and myth-making, and adequately plays the part of attention grabber. (The rhyme and reasoning of these studies, it seems to be, goes something like this: as long as you grab the publics attention, it doesn’t matter how bad the science is, they’ll fall for it hook, line, and sinker; clueless activists will take it from there. After all, we’re still dealing with the fallout from Wakefield’s bogus study on MMR vaccinations.)

Putzai’s Potatoes

Lastly, what becomes of Putzai’s potatoes? Surely, that study prevented the rise of GM potatoes that would have harmed society and children? Again, unfortunately, no. Academic’s Review did a great job distilling the problems with his study, which I will summarize here: (1) No conclusions could be made from the study’s results; (2) No differences were seen between the two groups of animals, GM-fed and non-GM fed; (3) the study had a flawed design, such as feeding potatoes to rats (potato is toxic to rats); (4) Lancet, the medical journal, published the paper despite objections from the reviewers.

(Now, this is not to say there aren’t problems with GM technology, nothing is perfect, and there are always niggles here, there, and yonder. But to distort the debate, especially with such nonsensical studies neither supported by its own data nor by the greater scientific community, is folly and serves only to poison the well.)


Callan Bentley expressed the sentiment I wish to impress upon you with this post by way of the organic / conventional dichotomy, safety of GMOs, and many other false debates out there, albeit with a slight modification in square brackets:* “Why is [it] so difficult to show unambiguously? Why can’t we convince people who are the toughest to convince? Where’s the really compelling evidence we seek?” 

When you can’t convince the people whose full-time job it is to know their field inside and out in an occupation where you get famous and climb the ladder to success by proving other scientists wrong, it’s pretty safe to say you’re not onto something. Calling out conspiracy, funding bias, and ego-driven lab fabrications only further the point home to those on the other side and those who observe from the sidelines trying to make up their minds.

Not all scientific statements have equal weight, some aren’t even science, and some almost seem to be made up. Hopefully, this post made that obvious. If you have any questions, shoot, and if you have a friend (or friends) that would benefit from knowing this, share this article so that they might have their foundations shaken just a little, or a lot. As Lawrence Krauss is so fond of saying over and over again:** “I hope that every [person] at one point in their life has the opportunity to have something that is at the heart of their being, something so central to their being that if they lose it they won’t feel they’re human anymore, to be proved wrong because that’s the liberation that science provides. The realization that to assume the truth, to assume the answer before you ask the questions leads you nowhere.” I’ve had many such occurrences, it reminds me I’m human, and therefore, an idiot. For everything I know, there’s a thousand things I could never dream to know. You too.

So, out there on the world wide wild web when reading controversial science stories, especially from the media, keep in mind Carl Sagan’s truthful quip: “Not all scientific statements have equal weight.”

I’d like to conclude this post as it began, with a quote by Carl Sagan: “Our common-sense intuitions do not count. What does count is experiment.”

* – Callan’s original quote had [this] instead of [it], and is aimed at ‘evidence’ for Intelligent Design (ID). The meaning of the quote applies to almost any scientific field, but it made more sense with an [it] instead of a [this] considering that I applied it to many such fields, instead of narrowly to ID.
** I slightly modified his quote to make more sense. His original quote is this: “I hope that every student who ever goes to university at one point in their life has the opportunity to have something that is at the heart of their being, something so central to their being that if they lose it they won’t feel they’re human anymore to be proved wrong because that’s the liberation that science provides. The realization that to assume the truth, to assume the answer before you ask the questions leads you nowhere.” I broadened the quote to include all people, not only those lucky enough to get a university education.

15 thoughts on “Not All Scientific Statements Have Equal Weight”

  1. Excellent post! I have had to explain many times to people who start their arguments with “This peer reviewed study proves…” that single studies do not prove *anything*. I explain that peer review simply means a few people (3-4) thought the work deserved a wider audience and scrutiny from the scientific community at large. I also have to explain that extraordinary results require extraordinary proof (like claiming that 70% of the U.S. corn crop is tumorigenic!)

  2. I was pleasantly surprised yesterday: a page i follow on FB, The Cult Of Dusty, called out the anti-GMO crowd as anti-science. Dusty is a loud mouthed southerner, a vocal atheist, and he has a pretty big following. Against the tide he came out and simply called BS on all the anti-gmo stuff. The discussion this generated was heated, but interesting.

  3. Not all scientific statements are as important as they seem. There may be many studies supporting the theory that GM food is as nutritious as unmodified. More importantly, however, we don’t know what the unintended, long-range consequences are. Ordinarily, evolution is slow and balances between species are maintained. GM is a shock to the eco-system. Nobody knows how it will play out, nor can we “experiment” to get to an answer, so we have to rely on common sense. Mine tells me that we should be more scared of GM than we are.

  4. […] Where do we go from here?  With all the doom and gloom, the fighting and the dissent, is civil debate even possible?  I believe it is, but we must make a concerted effort.  We must read things that make us uncomfortable and that challenge our views, and we must do so with a critical eye.  We must ask ourselves at every turn, “Does this make sense?  Does this meet the criteria for robust scientific inquiry?” […]

  5. Okay, I stumbled across your blog, which I’m enjoying immensely, but may I ask you to use standard English? It’s like nails on a blackboard to read “their” (which is a possessive) when what you really mean is “they’re” (which is a contraction). Your punctuation could use a little help too.
    Yes, I understand your intent, but it’s jarring to see the wrong word then have to interpret that in addition to the content that you write.
    Other than that request, I think you’re brilliant, and thanks for the great information!

  6. Science and technology are instinctive processes of evolution, but don’t people have valid reasons to be afraid of science given the way it is often used ? The ‘P’ value merely proves the fact that something is repeatable, a kind of truth, not an absolute. The truth of whether or not an outcome is actually good as opposed to simply repeatable is another kettle of fish, or flying spaghetti monster, isn’t it ? Perhaps scientists today become desensitised to the emotional concerns of less skilful people, doing both parties a dis-service in the process.

    People feel disempowered when individuals and corporations appear to pursue the profit margin, regardless of any harm the product they’re selling may bring. Isn’t it natural that some such people might want to choose as simple a life as possible, even if it means living shorter lives, if they feel they will live a more autonomous and fulfilling life in the process ? Wouldn’t people feel a need to defend such a way of life if they felt their liberties were being over regulated and those of large corporations under regulated in light of the evident harms that corporations, using science and technology, inflict on the environment and many people’s lives ? Wouldn’t such fearful people react hastily out of ignorance ?

    A lot of remarks made by people in defence of the scientific community appear insensitive and ill thought out, deepening distrust.

    Anyway, enjoying your blog. I enjoy anything that allows people to experiment in how to reach a consensus on the goodness of human nature.

    Perhaps the word ‘God’ is just a derivation of the word ‘Good’, something we all need to define more accurately if we really want to reach the epitome of human evolution. I believe religion is also an instinctive process necessary for our evolution that has been misrepresented by shoddy practice and hype.

Hit me where it hurts...

Please log in using one of these methods to post your comment: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s