2 posts tagged with "statistics"

Every major A/B testing platform is now wrestling with the same question: Bayesian or frequentist?

Optimizely built their original tool on frequentist statistics. Then switched to "Stats Accelerator" (Bayesian). VWO calls their approach "SmartStats" (also Bayesian). Statsig uses frequentist for feature flags, Bayesian for experiments. The research teams at Microsoft and Netflix have published papers on both.

Here's the practical guide to what actually matters for your team.

The most common A/B testing mistake isn't bad traffic or wrong goals — it's stopping the experiment too early.

Without calculating the required sample size before you start, you're likely to call a winner based on random noise and ship a change that has no real effect, or worse, a negative one.

This guide walks through the exact formula, a real worked example, and what to do when your traffic is too low.