Login Create Account

Sample Size Planner

Multi-arm A/B tests

Plan per-arm and total sample size for experiments with one control and multiple variants (A/B/C/…) on either conversion rates or means. Choose whether you care most about detecting a minimum lift vs. control for each arm or about any meaningful difference across all arms.

👨‍🏫 Professor Mode: Guided Learning Experience

New to multi-arm testing? Enable Professor Mode for step-by-step guidance through planning sample sizes for tests with multiple treatment groups!

OVERVIEW & WHAT THIS TOOL SOLVES

Testing three or four creative variants against a Control group in a single experiment is faster than running back-to-back A/B tests, but it creates two hidden traps: traffic is split across more buckets, and testing multiple variants inflates your risk of declaring a false-positive winner by pure chance. The Multi-Arm A/B Sample Size Planner calculates the exact per-arm and total traffic required across k experimental arms—with built-in Bonferroni multiplicity protection—for either conversion rates or continuous means.

🎯 Questions It Answers

  • How much traffic do we need per arm—and in total—when testing 1 Control group against 2, 3, or more Variants?
  • How much extra sample size is required when we apply a Bonferroni adjustment to control false alarms across multiple comparisons?
  • How does sizing for the hardest-to-detect variant compare against sizing only for the largest expected lift?

📂 What Data You Need

  • Control & Variant Targets: Expected conversion rate (Proportions mode) or expected mean and common SD (Means mode) for Control and each Variant arm.
  • Test Goal Choice: Minimum Lift vs. Control (Bonferroni-adjusted across all arms) or Largest Single Contrast (nominal alpha on biggest lift).
  • Confidence & Power: Overall family-wise alpha and desired statistical power.

💡 What You'll Discover

  • Per-Arm & Total Experiment N: Required sample size per bucket and across the entire multi-arm study.
  • Pairwise Contrast Breakdown: Which specific Variant-vs.-Control comparison is the bottleneck driving the sample size.
  • Multiplicity Penalty Visualization: How adding extra arms affects adjusted alpha and required traffic.
🔍 Key Nuances: Multiplicity Adjustment & Choosing Your Test Goal
  • Why Multi-Arm Tests Need Alpha Adjustment: If you compare 4 variants against Control at α = 0.05 without adjustment, your chance of at least one false alarm jumps to nearly 19%. Under the Minimum Lift vs. Control goal, this planner divides alpha by the number of variant–control comparisons (m = k − 1) so your family-wise false-positive rate stays protected.
  • Minimum Lift vs. Largest Single Contrast: Minimum Lift vs. Control sizes every arm so that even the smallest variant lift you entered is adequately powered after Bonferroni adjustment. Largest Single Contrast sizes the study for the single biggest variant–control gap at nominal alpha (a permissive planning floor that does not control family-wise error).
  • Total Traffic Grows Two Ways: Adding a 4th or 5th variant increases total sample size both because you have more buckets to fill (k × nper arm) and because the stricter Bonferroni threshold increases nper arm.
📐 Under the Hood: Model Equations & Variable Definitions
1. Bonferroni-Adjusted Pairwise Sample Size Across k Arms (m = k − 1 Variants vs. Control)
\[ \alpha_{\text{adj}} = \frac{\alpha}{m} \quad (m = k - 1), \qquad n_{j,\,\text{mean}} = \frac{2\,(z_{1-\alpha_{\text{adj}}^*} + z_{1-\beta})^2 \sigma^2}{(\mu_j - \mu_0)^2} \] \[ n_{j,\,\text{prop}} = \frac{\left[z_{1-\alpha_{\text{adj}}^*}\sqrt{2\bar{p}_j(1-\bar{p}_j)} + z_{1-\beta}\sqrt{p_0(1-p_0)+p_j(1-p_j)}\right]^2}{(p_j - p_0)^2}, \qquad N_{\text{total}} = k \cdot \max_{j} n_j \]
💬 Read it in plain English: We divide your significance threshold α by the number of variant comparisons (m = k − 1), compute the required per-arm sample size nj for each Variant j against Control (0), and set the per-arm requirement so the hardest-to-detect variant (or largest contrast) hits your target power.
k, m, αadj
Total Arms, Variant Comparisons & Adjusted Alpha YOUR DATA ⓘ
Total experimental groups (k), number of variant-vs.-control tests (m = k − 1), and Bonferroni per-comparison alpha (αadj).
p0, pj, μ0, μj, σ
Control & Variant Rates / Means & Spread YOUR DATA ⓘ
Control baseline (p0 or μ0), Variant j target (pj or μj), pooled proportion p̄j, and common standard deviation σ.
z1−α*, z1−β
Critical Normal Multipliers INFERENCE ⓘ
Standard normal critical values for the per-contrast false-positive threshold and statistical power.
nj, Ntotal
Per-Arm & Total Study Sample Size OUTPUT ⓘ
Required sample size per arm (nj) and total traffic across all k arms (Ntotal = k · nper arm).

INPUTS & SETTINGS

Design the multi-arm test

Arms & target conversion rates

Specify the expected conversion or response rate for the control and each variant. The planner will compute the minimum per-arm sample size to detect the differences you have entered at your chosen confidence and power.

Arm Label Target rate
Control (A)
Variant B
Variant C
Variant D

Arms, target means, and variability

Specify the expected mean outcome for the control and each variant, along with a common standard deviation. The planner assumes roughly equal variability across arms.

Arm Label Target mean
Control (A)
Variant B
Variant C
Variant D

Rough measure of spread around each arm’s mean (for example, dollars or points).

Lift vs. control powers every variant–control comparison and adjusts alpha for the number of them. Largest single contrast powers only the biggest difference you specified, at the nominal alpha.

Confidence = 1 - alpha for a two-sided test.

Common choices are 80% or 90% power.

Advanced settings

Two-sided tests are standard when any increase or decrease matters. One-sided tests can be used when you only care about improvements relative to control, but they ignore strong evidence of a decrease.

MARKETING SCENARIOS

Use presets to explore common marketing multi-arm tests, such as three competing subject lines or four hero images on a landing page. Each scenario sets control and variant targets, a goal, and default confidence/power.

VISUAL OUTPUT

Required per-arm sample vs. effect size

This chart shows how the required per-arm sample size changes as the smallest lift vs. control you care about becomes larger. Stronger lifts reduce the required per-arm sample size for a fixed confidence and power.

Required per-arm sample vs. power

This chart plots the required per-arm sample size against desired power, holding the effect pattern and alpha fixed. Higher power always requires a larger per-arm n.

DESIGN SUMMARY

Required per-arm sample size (n): –
Total required sample size (N total): –
Outcome type: –
Goal: –
Design alpha / confidence: –
Design power: –

Per-arm summary vs. control

Enter control and variant targets above to see a summary of lifts and required n per arm.

Statistical Planning Statement

Provide control and variant targets, choose a goal (lift vs. control or largest single contrast), and set alpha and power to generate a planning statement for this multi-arm design.

Managerial Interpretation

This panel translates the design into plain language: how many observations per arm you need, how that depends on the minimum lift you care about, and how often the experiment would succeed in flagging a real winner vs. looking inconclusive just because of noise.