Replication Crisis

How the crisis began

Ioannidis (2005) made waves with a model showing that the smaller the samples and effects and the greater the analytic flexibility, the more likely a published finding is to be false. In 2011, Bem published experiments in a leading journal claiming evidence for precognition, heightening concern about how easily standard statistical practice could make implausible results "significant." That same year, Simmons, Nelson and Simonsohn (2011) showed by simulation that freely choosing sample size, dependent measures and covariates could push false-positive rates above 60%. In a survey by John, Loewenstein and Prelec (2012), many psychologists admitted to questionable research practices such as stopping data collection after seeing results or reporting only some conditions.

Results of large-scale replications

The Open Science Collaboration (2015) repeated 100 studies from three major psychology journals: while 97% of the originals had significant results, only about 36% of the replications did, and effect sizes were on average about half as large. In medicine, Begley and Ellis (2012) reported that only 6 of 53 landmark preclinical cancer studies were confirmed. Camerer and colleagues found that 11 of 18 experimental economics studies (2016) and 13 of 21 social science experiments published in Nature and Science (2018) replicated. Button and colleagues (2013) highlighted very low statistical power in neuroscience, and in Baker's (2016) survey more than 70% of researchers said they had failed to reproduce another scientist's experiment.

Causes and reforms

Causes cited include small samples and low power, p-hacking to cross the 0.05 threshold, forming hypotheses after seeing results, and publication bias favoring significant findings. These link to Goodhart's law, under which a measure that becomes a target is distorted, and to confirmation bias, the preference for evidence that fits expectations. Munafò and colleagues (2017) published a manifesto calling for preregistration, registered reports, open data and code, and collaborative research, and Benjamin and colleagues (2018) proposed lowering the significance threshold for new discoveries to 0.005. Nosek and colleagues (2022) reviewed how, following these reforms, the concepts of replicability, robustness and reproducibility were clarified and research practices began to change. The crisis renewed the point that the organized skepticism of Merton's norms of science must work institutionally.

FAQ

What is the replication crisis in simple terms?

When other researchers repeated experiments published in prominent journals in the same way, many did not produce the same results. This set off a range of reforms to make research more reliable.

Does the replication crisis mean science is wrong?

No. It can be seen as science finding and correcting its own errors. Efforts such as replication studies, preregistration and data sharing aim to make research more trustworthy.

Sources reviewed: 2026-10-26

References

1. Ioannidis, J. P. A. (2005). Why most published research findings are false. PLoS Medicine, 2(8), e124.
2. Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22(11), 1359–1366.
3. Bem, D. J. (2011). Feeling the future: Experimental evidence for anomalous retroactive influences on cognition and affect. Journal of Personality and Social Psychology, 100(3), 407–425.
4. John, L. K., Loewenstein, G., & Prelec, D. (2012). Measuring the prevalence of questionable research practices with incentives for truth telling. Psychological Science, 23(5), 524–532.
5. Begley, C. G., & Ellis, L. M. (2012). Raise standards for preclinical cancer research. Nature, 483(7391), 531–533.
6. Button, K. S., et al. (2013). Power failure: Why small sample size undermines the reliability of neuroscience. Nature Reviews Neuroscience, 14(5), 365–376.
7. Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349(6251), aac4716.
8. Camerer, C. F., et al. (2016). Evaluating replicability of laboratory experiments in economics. Science, 351(6280), 1433–1436.
9. Baker, M. (2016). 1,500 scientists lift the lid on reproducibility. Nature, 533(7604), 452–454.
10. Munafò, M. R., et al. (2017). A manifesto for reproducible science. Nature Human Behaviour, 1, 0021.
11. Benjamin, D. J., et al. (2018). Redefine statistical significance. Nature Human Behaviour, 2(1), 6–10.
12. Camerer, C. F., et al. (2018). Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015. Nature Human Behaviour, 2(9), 637–644.
13. Nosek, B. A., et al. (2022). Replicability, robustness, and reproducibility in psychological science. Annual Review of Psychology, 73, 719–748.