History
Ronald Fisher treated the p-value as a measure of evidence against the null hypothesis. Jerzy Neyman and Egon Pearson (Karl Pearson's son) recast testing as a decision rule in 1933. They defined two kinds of wrong decision: rejecting a true null (Type I) and failing to reject a false one (Type II). From the 1960s to the 1980s, Jacob Cohen warned that most psychology studies were too underpowered, so Type II errors were common. John Ioannidis's 2005 paper argued that Type I errors plus publication bias make many published findings false, which helped spark the replication crisis.
Concepts and the trade-off
The Type I error rate is the significance level α, usually 5%. The Type II error rate is β, and 1−β is power, typically targeted at 80% or more. In a courtroom, a Type I error convicts an innocent person and a Type II error acquits a guilty one. With COVID tests, an uninfected person testing positive is Type I, and an infected person testing negative is Type II. At a fixed sample size, lowering α raises β. To cut both, increase the sample size.
Practical pitfalls
Run 20 tests with no real effects and about one will come out "significant" by chance. Multiple comparisons therefore call for corrections such as Bonferroni's. P-hacking, tweaking analyses until something is significant, inflates Type I errors badly. So does peeking at an A/B test every day and stopping the moment it crosses the line. From a Bayesian perspective, the less plausible a hypothesis is beforehand, the more likely a "significant" result is a false positive.
FAQ
- Which error is worse?
- It depends: in cancer screening, missing a case (Type II) is worse; in drug approval, approving a useless drug (Type I) is worse.
- How do I increase power?
- Use a larger sample, reduce measurement noise, or study conditions where the effect is larger.
Sources reviewed: 2026-10-26
References
- Neyman, J., & Pearson, E. S. (1933). On the problem of the most efficient tests of statistical hypotheses. Phil. Trans. R. Soc. A, 231, 289–337.
- Cohen, J. (1988). Statistical Power Analysis for the Behavioral Sciences (2nd ed.). Erlbaum.
- Ioannidis, J. P. A. (2005). Why most published research findings are false. PLoS Medicine, 2(8), e124.