What if traffic is low?

Test bolder changes, run longer, or consider Bayesian methods.

What's A/B/n testing?

Comparing three or more versions at once, which requires multiple-comparison corrections.

A/B testing

💼 Business By CampusDown Wiki Editorial Team Last updated October 26, 2026 Views 2

A/B testing is an experiment that randomly shows users one of two versions (A or B) and compares which one better improves a target metric.

Contents

1. How it works
2. Common mistakes
3. FAQ
4. References

How it works

Start with a hypothesis—say, a green checkout button will lift conversions—and a primary metric. Calculate the needed sample size up front, then randomly assign users to A (control) or B (variant). Randomization means differences can be attributed to the change, establishing causation rather than mere correlation. Results are judged with p-values and confidence intervals.

Common mistakes

Peeking daily and stopping once results look significant, checking many metrics and cherry-picking (multiple comparisons), ignoring seasonality, and novelty effects all mislead. Google famously tested 41 shades of blue for links, and companies like Amazon and Netflix run thousands of experiments a year. A/B tests are especially useful for fixing bottlenecks in a marketing funnel.

FAQ

What if traffic is low?: Test bolder changes, run longer, or consider Bayesian methods.
What's A/B/n testing?: Comparing three or more versions at once, which requires multiple-comparison corrections.

Sources reviewed: 2026-10-26

References

Kohavi, R., Tang, D., & Xu, Y. (2020). Trustworthy Online Controlled Experiments. Cambridge University Press.
Fisher, R. A. (1935). The Design of Experiments. Oliver & Boyd.
Kohavi, R., Longbotham, R., Sommerfield, D., & Henne, R. M. (2009). Controlled experiments on the web: Survey and practical guide. Data Mining and Knowledge Discovery, 18(1), 140–181.

Reports and materials that use this theory (Korean)

There are no materials dedicated to this theory yet, but you can search related reports in the CampusDown library (Korean).

Find more reports · Search 'A/B 테스트' on CampusDown →

Marketing funnelWhat the marketing funnel is, how it maps the path from awareness to purchase, the AIDA model and TOFU/MOFU/BOFU stages, calculating stage-by-stage conversion rates, AARRR pirate metrics, and McKinsey's critique.💼 Business P-valueWhat a p-value actually measures, where the 0.05 cutoff came from, why it isn't the probability a hypothesis is true, the 2016 American Statistical Association statement, and the problem of p-hacking.➗ Mathematics & Statistics Correlation vs. causationWhy two variables moving together doesn't mean one causes the other: confounding variables, reverse causation, coincidence, the correlation coefficient r, randomized controlled trials, and the Bradford Hill criteria.➗ Mathematics & Statistics t-testThe t-test, from William Gosset's 1908 'Student' paper at the Guinness brewery to today: how the t-distribution works, one-sample, independent, paired, and Welch's t-tests, assumptions, and how to report p-values and effect sizes.➗ Mathematics & Statistics Chi-squared testThe chi-squared test, from Karl Pearson's 1900 paper and Fisher's 1922 correction to today: comparing observed and expected counts, goodness-of-fit vs independence vs homogeneity tests, degrees of freedom, and when to use Fisher's exact test.➗ Mathematics & Statistics Type I and Type II errorsType I and Type II errors, from Neyman and Pearson's 1933 framework: false positives vs false negatives, significance level α and power 1−β, the trade-off between them, examples from courts, COVID tests, and A/B tests, and multiple comparisons and p-hacking.➗ Mathematics & Statistics

Latest ArticlesView all ›

New guides for college study and assignments (Korean)

This article was last updated on October 26, 2026. It is based on widely recognized original works and textbooks; when citing it in a paper, please check the original sources listed in the references.