History: representativeness beats size
In the 1936 US election, the Literary Digest polled 2.4 million people and predicted a landslide for Republican Alf Landon. Roosevelt won 46 states. The sample was skewed toward wealthier telephone and car owners. George Gallup called the race correctly with a representative sample of about 50,000 and made his name. The lesson was that randomness and representativeness matter more than raw size. William Cochran later systematized sampling theory. In 1988 Jacob Cohen popularized sizing experiments by effect size and power. A 2013 analysis found that the median neuroscience study had only about 20% power.
Survey sample sizes
For estimating a proportion, n = z² × p(1−p) ÷ e². At 95% confidence (z = 1.96) and the most conservative p = 0.5, a ±5-point margin needs about 385 people, and ±3 points needs about 1,068. That's why national polls usually survey around 1,000 people. Halving the margin of error takes four times the sample. For large populations, the population size barely matters. This calculation sets the width of the confidence interval.
Experiments and A/B tests
For an experiment comparing two means with a t-test, sample size depends on α (usually 0.05), power (usually 0.8), and the expected effect size (Cohen's d). A medium effect (d = 0.5) needs about 64 per group, and a small one (d = 0.2) about 394. In A/B testing, detecting a lift from 5% to 5.5% conversion can take over 30,000 users per variant. Underpowered studies raise the Type II error rate, and the significant results they do produce tend to overstate the effect. Free tools such as G*Power and online calculators make the math easy.
FAQ
- Does a bigger population need a bigger sample?
- Not much—above a few thousand, the required sample is nearly the same. Only small populations need a finite population correction.
- What if the response rate is low?
- Divide your target by the expected response rate to decide how many to invite, and check for nonresponse bias.
Sources reviewed: 2026-10-26
References
- Cohen, J. (1992). A power primer. Psychological Bulletin, 112(1), 155–159.
- Cochran, W. G. (1977). Sampling Techniques (3rd ed.). Wiley.
- Button, K. S., et al. (2013). Power failure: why small sample size undermines the reliability of neuroscience. Nat Rev Neurosci, 14, 365–376.