A/B Test Sample Size Calculator: How Many Visitors Do You Need?
The most common A/B testing mistake isn't bad traffic or wrong goals — it's stopping the experiment too early.
Without calculating the required sample size before you start, you're likely to call a winner based on random noise and ship a change that has no real effect, or worse, a negative one.
This guide walks through the exact formula, a real worked example, and what to do when your traffic is too low.
The Formula
A/B test sample size is calculated using the two-proportion z-test:
n = (z_α + z_β)² × (p₁(1-p₁) + p₂(1-p₂)) / (p₂ - p₁)²
Where:
- n = required visitors per variant
- z_α = 1.96 — critical value for 95% confidence (two-sided)
- z_β = 0.84 — critical value for 80% statistical power
- p₁ = your current (baseline) conversion rate
- p₂ = the conversion rate you want to be able to detect (baseline × MDE)
This is the same formula used by Optimizely's sample size calculator, Statsig's sample size planner, and Evan Miller's A/B testing calculator.
What is MDE?
Minimum Detectable Effect (MDE) is the smallest relative improvement you'd care about. If your baseline CVR is 2%:
- MDE = 10% → you want to detect ≥ 2.2% CVR (absolute +0.2pp)
- MDE = 20% → you want to detect ≥ 2.4% CVR (absolute +0.4pp)
- MDE = 50% → you want to detect ≥ 3.0% CVR (absolute +1.0pp)
Smaller MDEs require proportionally more data. This is the most important input to get right.
Worked Example
You're testing a new CTA button on your pricing page.
Inputs:
- Baseline CVR: 3% (0.03)
- MDE: 20% (want to detect ≥ 3.6% CVR)
- Confidence: 95%, Power: 80%
Calculation:
p₁ = 0.030
p₂ = 0.030 × 1.20 = 0.036
z_α + z_β = 1.96 + 0.84 = 2.80
numerator: (2.80)² × (0.030 × 0.970 + 0.036 × 0.964)
= 7.84 × (0.0291 + 0.03470)
= 7.84 × 0.06380
= 0.500
denominator: (0.036 - 0.030)² = (0.006)² = 0.000036
n = 0.500 / 0.000036 = 13,889 visitors per variant
Total visitors needed: 27,778
If you get 1,000 visitors/day: ~28 days
The Effect of MDE on Sample Size
| MDE | Baseline 2% CVR | Baseline 5% CVR |
|---|---|---|
| 10% | 22,240/variant | 8,500/variant |
| 15% | 9,920/variant | 3,800/variant |
| 20% | 5,600/variant | 2,140/variant |
| 30% | 2,480/variant | 950/variant |
| 50% | 900/variant | 345/variant |
If you're a low-traffic site, you can only reliably detect larger effects. That's not a bug — it's the statistics. Set your MDE to match the minimum effect you'd actually act on, not the minimum you'd like to be able to detect.
Common Mistakes
Stopping when P(win) looks good
Even Bayesian P(win) can hit 95% by chance after 200 visitors with a 3% CVR (that's 6 conversions). Early spikes in P(win) are noise. Run to your pre-calculated sample size, then evaluate.
Using the same experiment for multiple goals
If you calculated sample size for a "click on pricing" goal but you're also watching "signups", you've run multiple comparisons and inflated your false positive rate. Decide on your primary goal before you start.
Forgetting novelty effect
The first week of any experiment captures users' reactions to something new. If possible, let the experiment run 2+ weeks to let novelty settle.
Calculating days wrong
Sample size calculators give you visitors per variant. If you're splitting 50/50, you need 2 × n total visitors. If you're running 3 variants (Control + A + B), you need 3 × n total visitors — and your experiment takes 3× as long.
What to Do With Low Traffic
If your calculated experiment duration is more than 60–90 days, you have options:
- Increase MDE — only test changes you expect to move the needle by 25%+
- Target a higher-traffic page — a homepage test with 5k visitors/day vs a checkout page with 200 visitors/day
- Run an 80/20 split — 80% control, 20% variant — lets you observe effect size with less risk
- Use GhostSnip's sample size endpoint — automatically estimates your daily visitors and tells you estimated days before you launch
Use GhostSnip's Built-in Calculator
GhostSnip calculates the required sample size automatically when you create an experiment. Enter your baseline CVR and MDE, and it shows:
- Visitors per variant
- Total visitors needed
- Your estimated daily visitors (from the last 30 days of traffic)
- Estimated days to completion
The results page shows a progress bar tracking your current visitors against the required amount.