Group Inputs
Slide values or type directly into number boxes. Every result reads the number boxes, so a typed value such as 0.036 or 16342 is used exactly.
A/B Proportion Testing Tool
Decision-First Design
Determine if your A/B test shows a real winner or just random noiseβwith instant decision guidance, business impact calculations, and visual evidence.
QUICK START: Choose Your Path
DATA SOURCE
π
Use a Case Study
πΌ Real Marketing A/B Tests
Select a preset scenario to explore real-world A/B testing situations with authentic marketing metrics and business context.
π€
Load Data
Upload raw data
Required columns: group (two labels) and conversion (0 or 1).
Drag & Drop raw data file (.csv, .tsv, .txt, .xls, .xlsx)
Two columns: group label and conversion (0 or 1). Up to 2,000 rows.
No raw file uploaded yet.
ENTER YOUR DATA
Confidence Level & Reporting
Advanced settings
Apply FPC when population size is known or approximately known. Used for sampling without replacement from a finite population. FPC reduces standard error when sample size is >5% of total population.
The share your test plan assigned to this group (50 for an even split). The diagnostics check the observed arm sizes against it.
YOUR DECISION
Enter data to see your result
We'll calculate significance and show you what to do next
BUSINESS IMPACT CALCULATOR (OPTIONAL)
π° Calculate potential revenue impact (optional - only if you're testing a monetized action)
When to use this calculator:
- β You're testing a change that directly affects revenue (e.g., checkout flow, pricing display, purchase button)
- β You know the average transaction value
- β You want to estimate the financial impact of implementing the winning variant
When NOT to use this calculator:
- β You're testing brand awareness, engagement, or other non-monetized actions
- β The test doesn't directly connect to revenue (e.g., newsletter signups, social shares)
- β You don't have reliable transaction value data
Many valuable A/B tests don't have explicit dollar values attachedβthat's perfectly fine! Skip this section if your test measures engagement, awareness, or other non-revenue metrics.
E.g., visitors, customers, trials, units sold
π Show calculation breakdown
VISUAL EVIDENCE
The fan charts show the uncertainty around your estimates. Narrower fans = more precision.
Proportions Fan Chart
Difference Fan Chart
Chart Settings
Range (%):
Range:
DETAILED STATISTICAL RESULTS
β οΈ Small Sample Warning
Show detailed statistical output
APA-Style Statistical Reporting
Managerial Interpretation
Summary Table
| Measure | Label | Proportion / Ξ | CI Lower | CI Upper | n |
|---|
LEARNING RESOURCES
π When to use this test
Use the two-proportion z-test when:
- You have two independent groups (Control vs. Variant)
- You're measuring a binary outcome (converted/didn't convert)
- Sample sizes are reasonably large (n β₯ 30 per group, with at least 5 successes and 5 failures)
- Random assignment was used
β οΈ Common mistakes to avoid
- Peeking repeatedly: Checking results multiple times inflates your false positive rate
- Stopping too early: Small samples have low statistical power
- Ignoring practical significance: Statistical significance β business relevance
- Not pre-registering sample size: Decide your stopping rule before you start
π How we calculate this (equations)
Core test, default \(\Delta_0 = 0\) (pooled standard error):
$$z = \frac{P_2 - P_1}{\sqrt{\bar{P}(1-\bar{P})\left(\dfrac{1}{n_1} + \dfrac{1}{n_2}\right)}}, \qquad \bar{P} = \frac{x_1 + x_2}{n_1 + n_2}$$
Under \(H_0:\ \Delta = 0\) both groups share one conversion rate, so the test pools them. With this SE, \(z^2\) equals the Pearson chi-square for the same 2Γ2 table.
Test against a non-zero benchmark \(\Delta_0\) (unpooled standard error):
$$z = \frac{P_2 - P_1 - \Delta_0}{\sqrt{\dfrac{P_1(1-P_1)}{n_1} + \dfrac{P_2(1-P_2)}{n_2}}}$$
Null hypothesis: \(H_0:\ \Delta = \Delta_0\). The groups no longer share a rate under \(H_0\), so nothing is pooled.
Confidence interval for the difference (\(\Delta = P_2 - P_1\)), always unpooled: $$\Delta \pm z_{1-\alpha/2}\sqrt{\frac{P_1(1-P_1)}{n_1}+\frac{P_2(1-P_2)}{n_2}}$$
π Related tools
- Independent t-test - For comparing continuous metrics (not proportions)
- Chi-Square test - For categorical data with more than 2 groups
- McNemar test - For paired/matched data
DIAGNOSTICS & ASSUMPTIONS
View diagnostic checks
Diagnostics will appear after you enter data.