Understanding the Importance of Validating Non-Published Health Research and Its Impact on Outcomes: Why Nullius in Verba is Significant

Science is under assault world-wide. Thus, the underlying principle of empirical science, the Latin term cited in the title ”Nullius in Verba” — which means: “Don’t take anybody’s word for it”, is probably under question too.  Many believe that the operative slogan now is this; “Take anybody’s word for it” (assuming that that particular “anyone” is a recognized leader to you personally).

Nullius in Verba is the founding slogan of Royal Society of London for the Improvement of Natural Knowledge (“Royal Society”), an organization founded in 1660 (see image).   They define the term like this: “It is an expression of the determination of Fellows to withstand the domination of authority and to verify all statements by an appeal to facts determined by experiment.”

This tried and true principle –  truly revolutionary then and now – is the reason we have “peer review” and the reason scientists stick to the belief the replicability of their experiment is part of the process.   Trust, but verify is the name of the game.

How well does it work?  A highly cited article, published over 10 years ago, by Dr. John Ioannidis, entitled, “Why Most Published Research Findings Are False,” suggested not very well.   He concluded this because most studies are poorly designed and do not stand up to statistical scrutiny.  Other research shows that many of their results (e.g. the relationship between anti-oxidants and cancer) are not borne out in subsequent studies.  

Now to me, his paper (and many more like it) is one of best reminders we have that peer-review works!   Yes, there may be some scientists with egg on their faces, but that is the game to which they signed up. The failure of a well-designed experiment is at least as important as it’s success. Seeking truth is the ultimate driver, and a truth must be verifiable and verified by peers.

But what do we do with research findings that are not published?  This type of research is very common. These “trade secrets” are often used as a foundation for a private company’s products and services. In these cases, independent peer review is not demanded (as it is by any reputable scientific journal), not required, and often not done.  These research outcomes are often blessed by some authority in a private company; the only opportunity in these cases for public review is when the product is released to the public.  Failure to succeed in the marketplace is really the only method that is similar to the process of “peer review.”  But that leaves a lot to be desired.

Think about this:  If over 50% of the published findings later turn out to be wrong; what percent of non-published findings are wrong?   A conservative estimate would be same percentage, but a realistic estimate has to be much higher: 60%, 80%. More?   We will never know.

But those inside a company should know – at minimum to avoid unnecessary risk, to protect against massive product failure, to build an excellent product that has gone through a series of iterative (and independent) research steps that lead to its improvement –a private and independent peer review – or better “reviews”– of  method is a prudent strategy for private, unpublished, research. 

This can help avoid group think, cover-ups and more.

But remember, a single peer-review is still not equated with “truth.”  Not only may the internal validity of the work hold muster (as discussed above); the external validity (replicability) may still be problematic, as discussed in a recent Washington Post article.   But peer-review is a still a terribly, wonderfully worthwhile endeavor – like Winston Churchill’s adage about democracy:  [It’s] the worst form of government, except for all the others.”

So think about Horace (the Roman poet from whom the phase is extracted), Sir Isaac Newton (an early fellow of the Royal Society), and Ronald Reagan (not a fellow of the Royal Society, but a former US President) who uttered, in our time, the famous quote:  “Trust, but Verify.”

The point of this blog is this:  Even if the attempt at verification of research is done outside public scrutiny, it is better than no independent look at the research methods at all. And as the ancient adage proclaims, “two [+] heads are better than one”, means in this context that an independent replication of findings is even better, in public and private research.

Just Do It!

References:

  • Royal Society: https://royalsociety.org/ (accessed July 26, 2019)
  • Richard Smith. Peer review: a flawed process at the heart of science and journals. J R Soc Med. 2006 Apr; 99(4): 178–182. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1420798/
  • John P.A. Ioannidis. Why Most Published Research Findings Are False. PLOS Medicine, August 30, 2005 https://doi.org/10.1371/journal.pmed.0020124
  • National Cancer Institute. Antioxidants and Cancer Prevention. https://www.cancer.gov/about-cancer/causes-prevention/risk/diet/antioxidants-fact-sheet
  • P.D. Thacker & J. Tennant. Why we shouldn’t take peer review as the ‘gold standard’ Washington Post.  August 1, 2019. https://www.washingtonpost.com/outlook/why-we-shouldnt-take-peer-review-as-the-gold-standard/2019/08/01/fd90749a-b229-11e9-8949-5f36ff92706e_story.html.