What it means
When testing a new drug, the null hypothesis is "no effect." A p-value of 0.03 means that if the drug truly did nothing, a difference this large would show up by chance only 3% of the time. If p falls below a preset significance level (often 0.05), researchers reject the null and call the result statistically significant. The 0.05 cutoff is a convention Fisher suggested, not a law of nature.
Common misconceptions
A p-value is not the probability the null is true, and it says nothing about effect size or importance. Huge samples make trivial differences significant, and p-hacking—running many analyses and reporting only the significant ones—fuels the replication crisis. In 2016 the ASA warned against basing conclusions on p-values alone. For prior probabilities, see Bayes' theorem; for why significance isn't causation, see correlation vs. causation.
FAQ
- Does p = 0.06 mean no effect?
- No—the evidence just falls short of the cutoff; check effect sizes and confidence intervals.
- Does a smaller p mean a bigger effect?
- No; it depends heavily on sample size.
Sources reviewed: 2026-10-26
References
- Wasserstein, R. L., & Lazar, N. A. (2016). The ASA statement on p-values: Context, process, and purpose. The American Statistician, 70(2), 129–133.
- Fisher, R. A. (1925). Statistical Methods for Research Workers. Oliver & Boyd.
- Ioannidis, J. P. A. (2005). Why most published research findings are false. PLoS Medicine, 2(8), e124.