Glossary
Every term you may meet in Datsnip, in plain English. Each entry says what it means and why it exists. Terms are grouped by the question they help answer.
For how to act on these numbers, see Reading Your Results.
The basics
Experiment (A/B test) Showing different versions of a page to different visitors at the same time, then comparing which does better. Running them at the same time matters: it makes sure a holiday or an ad campaign affects every version equally.
Control The original, untouched version. Everything else is compared to it.
Variant A changed version you are testing. Datsnip supports more than one (A/B/n): Variant A, Variant B, and so on.
Visitor / exposure An exposure is one visitor being shown one version. Datsnip counts each visitor once per experiment and keeps them on the same version for 90 days, so they don't flip between versions.
Goal
What counts as a win: a click, a page visit, or a custom event such as purchase. An experiment has one goal.
Conversion A visitor who completed the goal.
Conversion rate Conversions divided by exposures. 40 conversions out of 1,000 visitors is a 4% conversion rate.
Traffic % The share of visitors on the matching pages who enter the experiment. Everyone else sees the original and is not counted.
Traffic split (weights) How visitors are divided between versions, for example 50/50, or 80/20 to limit risk on a bold idea.
Custom event
An action you report yourself with one line of code, such as purchase or signup_completed. See Tracking Events and Goals.
Is this version actually better?
Uplift How much better or worse a version converts than the control, as a percentage. If control converts at 5% and a variant at 6%, the uplift is +20% (one fifth more). It is the size of the win, and it swings wildly early on.
Bayesian A way of doing the math that works like updating a belief. You start cautious ("probably no difference"), and every visitor nudges that belief. The more data, the more the data wins over the starting assumption. It ends in a direct answer: "87% likely this version is better." Datsnip uses it because that is the question you are actually asking.
Prior The cautious starting assumption in a Bayesian model. Datsnip's is deliberately weak, so a handful of lucky visitors cannot produce a dramatic result, but real data quickly takes over.
Chance to beat control (P(win)) How likely it is that a version is truly better than the control, not just lucky so far. About 50% is a coin flip. High 90s means very likely real. This is the number Datsnip uses to pick a winner.
Expected loss If you ship a version and it turns out to be worse, how much conversion you would give up on average. Datsnip only calls a winner when this is tiny, so a close call that could cost you is never crowned.
Winner A version whose chance to beat control is high enough and whose expected loss is tiny. Both must hold.
Frequentist / p-value / "statistical significance" The older method. It answers a backwards question: "if the versions were identical, how surprising would this data be?" Most people misread it. Datsnip shows a plain probability instead.
Confidence A number from the older method, checking how unlikely a difference is under pure chance. Datsnip's headline figure is chance to beat control instead, because it is easier to read.
Can I trust these numbers?
Sample ratio mismatch (SRM) The visitors were not divided the way you set. A 50/50 test showing 60/40 is not luck; something is wrong. Usually a version is broken on some devices or browsers, or a redirect loses visitors. Datsnip warns you, and you should fix the cause before trusting the results.
Multiple comparisons (Bonferroni correction) Test four ideas at once and, by chance alone, one will look good. Datsnip raises the bar for a winner as you add versions, so a lucky one does not win. The more versions, the stricter the check.
Peeking Checking results every day and stopping the moment one version is ahead. It makes false winners much more likely, because early leads are often luck. Datsnip's careful winner rule helps, but the best habit is to let tests run.
Minimum run time Tests shorter than a week are flagged. Weekdays and weekends behave differently, so a test must cover at least one full cycle.
False positive Declaring a winner that is not actually better. Most of the features above exist to prevent this.
How long will it take?
Sample size How many visitors each version needs before its result can be trusted. Smaller improvements and lower conversion rates need more visitors.
Minimum detectable effect (MDE) The smallest improvement you care about finding. Detecting a 5% improvement takes roughly four times as many visitors as detecting a 10% one. In the experiment wizard this is labelled Min. improvement to detect.
Power and confidence (80% / 95%) Inputs to the sample-size estimate: how sure you want to be of not being fooled (95%), and how likely you are to catch a real win if there is one (80%). You can ignore them; Datsnip sets them.
Sample-size estimator In the experiment wizard. Enter your current conversion rate and the smallest improvement worth finding, and it estimates visitors needed and roughly how many days that takes on your traffic.
Did it make money?
Revenue
The total purchase value credited to a version, from the revenue number you send with your purchase event.
Revenue per visitor Revenue divided by exposures. It catches wins that lose money: a discount can raise conversion rate while lowering revenue per visitor.
Average order value (AOV) Revenue divided by orders. A version that gets more orders can still earn less if each order is smaller.
Funnels and tracking
Funnel
A series of steps visitors take, such as /product → add_to_cart → /checkout → purchase. It shows how many visitors reach each step and where they drop off.
Drop-off The share of visitors who reached one step but not the next.
Conversion window How long visitors have to complete the whole funnel, counted from their first step. The default is 24 hours.
Funnel by variant The same funnel shown separately for each experiment version, so you can see where a winner wins, not only that it does.
Page view A visit to a page path. Recorded automatically, including navigation inside single-page apps. Only the path is stored, never the query string.
Event queue (window._sfq)
A safe way to report an event before Datsnip has finished loading, used on thank-you pages. See Tracking Events and Goals.
Targeting and delivery
Audience targeting Limiting an experiment to certain visitors: device, browser, operating system, traffic source or campaign, referrer, or new versus returning. See Audience Targeting.
URL match
Which pages an experiment runs on: one path, a prefix such as /blog/*, or everything.
Redirect (split URL) test Sending one version's visitors to a different URL, to compare two entirely different pages. See Redirect Testing.
Anti-flicker An optional snippet that briefly holds the page while a version applies, so visitors never see the original flash before the change. See Anti-Flicker.
Roll out the winner One click that turns the winning version's changes into live Page Fixes for everyone and ends the experiment.
Quick lookup: what should I do?
| You see | Meaning | Do this |
|---|---|---|
| Collecting data | Not enough visitors yet | Leave it running |
| Winner found | Very likely a real improvement | Roll it out |
| No significant difference | Neither version is clearly better | Keep the original or test something bolder |
| Traffic split warning | The numbers cannot be trusted | Pause, fix the broken version, restart |