Reading Your Results
Datsnip shows you results in plain English — not confusing statistics jargon. Here's what each number means.
P(win) — The main number to watch
"There's an 87% chance Variant A is better than your original."
This is the probability your variant is genuinely better. Think of it like a confidence reading.
| P(win) | What it means |
|---|---|
| Under 70% | Too early to tell — keep running |
| 70–90% | Looking good — but wait for more data |
| 90–95% | Strong signal — almost there |
| 95%+ | Winner — safe to ship |
| 99%+ | Very strong win — act on it |
You're waiting for P(win) to reach 95% or higher before calling a winner.
Uplift — How much better?
"+18.3% relative improvement"
This is the percentage increase in your conversion rate. A 2% baseline going to 2.4% is a +20% uplift.
Uplift tells you how much you're leaving on the table by not shipping the winner.
Sample size progress bar
The progress bar shows how far along you are toward a statistically reliable result. It fills up as more visitors enter the experiment.
Don't stop the experiment before the bar reaches 100%. Early results are noisy — a 95% P(win) after 50 visitors is often just luck. After the required sample size is reached, a 95% P(win) is meaningful.
⚠ SRM warning (red banner)
If you see a red "Traffic allocation mismatch" warning, your two variants received unexpectedly different amounts of traffic. This can invalidate your results.
Common causes:
- Your site's caching layer served old cached pages to more visitors on one variant
- A redirect is cached in visitors' browsers (use "back" button behavior to diagnose)
- Bots or scrapers are being counted as visitors
What to do: Pause the experiment, identify the cause, fix it, and start fresh.
⚡ "Run for at least 7 days" warning
If you see an amber warning saying the experiment hasn't run long enough — trust it.
Traffic patterns differ dramatically between weekdays and weekends. If you only capture Mon–Wed data, you're missing how weekend visitors behave on your site. SaaS products see ~40% less weekend traffic. E-commerce sees the opposite. Always run a minimum of one full week.
Revenue per visitor
If you're tracking purchase events, you'll see a Revenue Impact card showing how much each variant earned per visitor — not just conversion rate.
This matters when a higher-converting variant actually sells lower-value items. Revenue per visitor catches that.
When to ship
Ship when both of these are true:
- P(win) ≥ 95%
- The sample size progress bar has reached 100%
If neither variant reaches 95% P(win) after the required sample size, it means there is no meaningful difference. That's valuable information — it means your original is already pretty good for that element. Pick a bolder change to test next.