Bayesian vs Frequentist A/B Testing: Which Should You Use?
Every major A/B testing platform is now wrestling with the same question: Bayesian or frequentist?
Optimizely built their original tool on frequentist statistics. Then switched to "Stats Accelerator" (Bayesian). VWO calls their approach "SmartStats" (also Bayesian). Statsig uses frequentist for feature flags, Bayesian for experiments. The research teams at Microsoft and Netflix have published papers on both.
Here's the practical guide to what actually matters for your team.
The Core Difference
Frequentist (p-values, confidence intervals):
"Assuming the null hypothesis is true, the probability of seeing data at least this extreme is 3.2%."
Bayesian (probability of winning):
"There is an 87% probability that Variant A is better than Control."
The Bayesian statement answers the question you actually want to ask. The frequentist statement answers a hypothetical about a world you're trying to disprove.
Why P-values Are Misunderstood
A study by Hoekstra et al. surveyed 442 psychology researchers who used frequentist statistics daily. They found that 97% of them — including experienced statisticians — misinterpreted the 95% confidence interval.
The common misinterpretation: "There's a 95% probability the true effect is in this range."
The actual meaning: "If we ran this experiment 100 times with fresh samples each time, 95 of those confidence intervals would contain the true effect."
This is a subtle but critical difference. The frequentist CI says nothing about the probability for any given experiment run — it's a property of the procedure over many runs.
For A/B testing, you're running one experiment, not 100. You want to know the probability for this experiment.
The Peeking Problem
The most common frequentist A/B testing mistake: checking results every day and stopping when p < 0.05.
This is called "peeking" and it dramatically inflates your false positive rate:
| Checks before stopping | False positive rate (if checking at p < 0.05) |
|---|---|
| 1 (no peeking) | 5% |
| 2 | 8.2% |
| 5 | 14.2% |
| 10 | 19.3% |
| 100 | 40%+ |
With Bayesian testing, this problem is less severe. P(win) at any point in time is a valid probability statement about current data. Checking in early doesn't mathematically inflate your error rate in the same way.
However, Bayesian P(win) is still noisy with small samples. We still recommend running to your pre-calculated sample size before making decisions.
The Expected Loss Metric
The most useful metric that Bayesian testing gives you that frequentist doesn't: Expected Loss.
Expected Loss is the average conversion rate you'd sacrifice if you made the wrong decision.
Example:
- Variant A has P(win) = 0.93
- Expected Loss = 0.0002 (you'd sacrifice 0.02% CVR if wrong)
vs:
- Variant B has P(win) = 0.94
- Expected Loss = 0.008 (you'd sacrifice 0.8% CVR if wrong)
Variant B has higher P(win) but dramatically higher Expected Loss. The smart decision is to ship Variant A.
GhostSnip requires P(win) ≥ 0.95 and Expected Loss < 0.001 before marking a result as significant. This combination eliminates variants that technically win but have tiny absolute effects.
How GhostSnip's Bayesian Engine Works
GhostSnip uses a Beta-Binomial conjugate model:
- Prior: Beta(1, 1) — uniform, no prior belief about conversion rates
- Posterior update: After observing n conversions and m non-conversions:
Beta(1 + n, 1 + m) - P(win): Monte Carlo simulation — draw 10,000 pairs of samples from each posterior, count how often the variant's sample exceeds the control's
The Beta-Binomial is the correct model for conversion rate data (binary outcomes: converted or didn't convert). It's the same model used by VWO's SmartStats and Google's internal experimentation platform.
When Frequentist Is Still Right
Bayesian is better for most business teams, but frequentist has its place:
- Regulatory requirements: Some industries require p-values for compliance documentation
- Multi-armed bandit algorithms: UCB and Thompson sampling are frequentist algorithms that work well for continuous optimization (exploration vs exploitation)
- When you need to exactly replicate another tool's results: If you're comparing to a published paper or another tool's report, use the same statistical model
Practical Recommendations
For most product and growth teams running conversion experiments:
- ✅ Use Bayesian P(win) as your primary metric
- ✅ Use Expected Loss as your decision threshold (< 0.001)
- ✅ Pre-calculate sample size and run to completion
- ✅ Report "87% probability Variant A wins" to stakeholders — they'll understand it
- ❌ Don't peek and stop when p < 0.05
- ❌ Don't confuse "statistically significant" with "practically significant"
GhostSnip implements all of this by default. See how results are calculated →