"Why Most Published Research Findings are False" Part I

"Why Most Published Research Findings Are False"

John P. A. Ioannidis

John P. A. Ioannidis is in the Department of Hygiene and Epidemiology, University of Ioannina School of Medicine, Ioannina, Greece, and Institute for Clinical Research and Health Policy Studies, Department of Medicine, Tufts-New England Medical Center, Tufts University School of Medicine, Boston, Massachusetts, United States of America. E-mail: jioannid@cc.uoi.gr

Published research findings are sometimes refuted by subsequent evidence, with ensuing confusion and disappointment. Refutation and controversy is seen across the range of research designs, from clinical trials and traditional epidemiological studies [1-3] to the most modern molecular research [4,5].

There is increasing concern that in modern research, false findings may be the majority or even the vast majority of published research claims [6-8]. However, this should not be surprising. It can be proven that most claimed research findings are false. Here I will examine the key factors that influence this problem and some corollaries thereof.

Modeling the Framework for False Positive Findings

Several methodologists have pointed out [9-11] that the high rate of nonreplication (lack of confirmation) of research discoveries is a consequence of the convenient, yet ill-founded strategy of claiming conclusive research findings solely on the basis of a single study assessed by formal statistical significance, typically for a p-value less than 0.05.

Research is not most appropriately represented and summarized by p-values, but, unfortunately, there is a widespread notion that medical research articles should be interpreted based only on p-values. Research findings are defined here as any relationship reaching formal statistical significance, e.g., effective interventions, informative predictors, risk factors, or associations.

"Negative" research is also very useful. "Negative" is actually a misnomer, and the misinterpretation is widespread. However, here we will target relationships that investigators claim exist, rather than null findings.

It can be proven that most claimed research findings are false.

As has been shown previously, the probability that a research finding is indeed true depends on the prior probability of it being true (before doing the study), the statistical power of the study, and the level of statistical significance [10,11].

Consider a 2 X 2 table in which research findings are compared against the gold standard of true relationships in a scientific field. In a research field both true and false hypotheses can be made about the presence of relationships. Let R be the ratio of the number of "true relationships" to "no relationships" among those tested in the field. R is characteristic of the field and can vary a lot depending on whether the field targets highly likely relationships or searches for only one or a few true relationships among thousands and millions of hypotheses that may be postulated.

Let us also consider, for computational simplicity, circumscribed fields where either there is only one true relationship (among many that can be hypothesized) or the power is similar to find any of the several existing true relationships. The pre-study probability of a relationship being true is R/(R + 1).

The probability of a study finding a true relationship reflects the power 1 - Beta (one minus the Type II error rate). The probability of claiming a relationship when none truly exists reflects the Type I error rate, a.

After a research finding has been claimed based on achieving formal statistical significance, the post-study probability that it is true is the positive predictive value, PPV. The PPV is also the complementary probability of what Wacholder et al. have called the false positive report probability [10].

According to the 2 X 2 table, one gets PPV = (1 -Beta ) R/(R -BetaR + a). (Apologies, we cannot post tables)

A research finding is thus more likely true than false if (1 -Beta)R > a. Since usually the vast majority of investigators depend on a = 0.05, this means that a research finding is more likely true than false if (1 -Beta)R >0.05.

Table 1. (sorry we cannot show tables) Research Findings and True Relationships

What is less well appreciated is that bias and the extent of repeated independent testing by different teams of investigators around the globe may further distort this picture and may lead to even smaller probabilities of the research findings being indeed true. We will try to model these two factors in the context of similar 2 X 2 tables.

Public Library of Science Volume 2 Issue 8 AUGUST 2005


References Part I

1-11

Part III - "Why Research Findings Are False"

Public Library of Science, 8/05

References Part III
Part II - "Why Research Findings Are False"

Public Library of Science, 8/05

References Part II

Remember we are NOT Doctors and have NO medical training.

This site is like an Encyclopedia - there are many pages, many links on many topics.

Support our work with any size DONATION - see left side of any page - for how to donate. You can help raise awareness of CAM.