Skip to main content

Testing Multiple Variants

Most experiments are A vs B. But sometimes you have two or three completely different ideas and want to test them all at once rather than sequentially. Datsnip supports up to 5 variants in a single experiment.

When to test multiple variants​

Good reasons:

  • You have 3 different headline approaches and no strong hypothesis about which is best
  • You're testing multiple pricing page layouts from scratch
  • You want to test Control + bold change + moderate change in one run

When to stick with A/B:

  • You have a clear hypothesis ("this specific change will help")
  • Your traffic is limited — splitting to 4 groups means 4× the time to results

How to add variants​

In the experiment wizard, Step 2 lists all your variants. Click + Add variant to add Variant B, Variant C, and so on (up to 4 variants plus Control = 5 total).

Each variant shows its own Edit button that opens a visual editor session for that specific variant. You can make completely different changes for each — different headlines, different layouts, different CTAs.

To remove a variant, click the × on its row. The control variant cannot be removed.

Traffic is automatically split equally across all variants. Adding Variant B to a Control/Variant A experiment gives each ~33% of traffic. The rebalancing happens immediately when you add or remove a variant.

Reading results with multiple variants​

Results show all variants side by side. The P(win) for each variant is measured against Control.

A few things to keep in mind:

  • Don't declare a winner just because one variant beats the others — wait for P(win) ≥ 95% against Control
  • Running many variants increases the chance of a false positive. Be stricter — aim for 98%+ P(win) if you have 4+ variants.
  • If none reach significance, the clearest outcome is "none of these directions works — try something more different"

Traffic and timing​

Each additional variant multiplies the time to results. A 3-variant experiment (Control + A + B) takes 1.5× as long as a 2-variant experiment.

If your site gets 500 visitors/day and a 2-variant experiment would take 14 days, a 3-variant experiment takes about 21 days.

Use the sample size estimator in the experiment wizard to plan ahead.

Ideas for multi-variant experiments​

3 pricing page versions — Current page / simplified 2-plan version / premium-first layout

3 headline framings — Feature ("Automate your email") / Outcome ("Close deals faster") / Social proof ("Join 12,000 teams")

3 CTA styles — Current / High-contrast filled button / Ghost button with urgency text

3 hero images — Product screenshot / Team photo / Customer using the product