Group Inputs
Slide values or type directly into number boxes.
A/B Proportion Testing Tool
Decision-First Design
Determine if your A/B test shows a real winner or just random noiseβwith instant decision guidance, business impact calculations, and visual evidence.
QUICK START: Choose Your Path
DATA SOURCE
π
Use a Case Study
πΌ Real Marketing A/B Tests
Select a preset scenario to explore real-world A/B testing situations with authentic marketing metrics and business context.
π€
Load Data
Upload raw data
Required columns: group (two labels) and conversion (0 or 1).
Drag & Drop raw data file (.csv, .tsv, .txt, .xls, .xlsx)
Two columns: group label and conversion (0 or 1). Up to 2,000 rows.
No raw file uploaded yet.
ENTER YOUR DATA
Confidence Level & Reporting
Advanced settings
Apply FPC when population size is known or approximately known. Used for sampling without replacement from a finite population. FPC reduces standard error when sample size is >5% of total population.
YOUR DECISION
Enter data to see your result
We'll calculate significance and show you what to do next
BUSINESS IMPACT CALCULATOR (OPTIONAL)
π° Calculate potential revenue impact (optional - only if you're testing a monetized action)
When to use this calculator:
- β You're testing a change that directly affects revenue (e.g., checkout flow, pricing display, purchase button)
- β You know the average transaction value
- β You want to estimate the financial impact of implementing the winning variant
When NOT to use this calculator:
- β You're testing brand awareness, engagement, or other non-monetized actions
- β The test doesn't directly connect to revenue (e.g., newsletter signups, social shares)
- β You don't have reliable transaction value data
Many valuable A/B tests don't have explicit dollar values attachedβthat's perfectly fine! Skip this section if your test measures engagement, awareness, or other non-revenue metrics.
E.g., visitors, customers, trials, units sold
π Show calculation breakdown
VISUAL EVIDENCE
The fan charts show the uncertainty around your estimates. Narrower fans = more precision.
Proportions Fan Chart
Difference Fan Chart
Chart Settings
Range (%):
Range:
DETAILED STATISTICAL RESULTS
β οΈ Small Sample Warning
Show detailed statistical output
APA-Style Statistical Reporting
Managerial Interpretation
Summary Table
| Measure | Label | Proportion / Ξ | CI Lower | CI Upper | n |
|---|
LEARNING RESOURCES
π When to use this test
Use the two-proportion z-test when:
- You have two independent groups (Control vs. Variant)
- You're measuring a binary outcome (converted/didn't convert)
- Sample sizes are reasonably large (n β₯ 30 per group, with at least 5 successes and 5 failures)
- Random assignment was used
β οΈ Common mistakes to avoid
- Peeking repeatedly: Checking results multiple times inflates your false positive rate
- Stopping too early: Small samples have low statistical power
- Ignoring practical significance: Statistical significance β business relevance
- Not pre-registering sample size: Decide your stopping rule before you start
π How we calculate this (equations)
Core test (two-proportion z-test with hypothesized difference \(\Delta_0\)):
$$z = \frac{P_2 - P_1 - \Delta_0}{\sqrt{\dfrac{P_1(1-P_1)}{n_1} + \dfrac{P_2(1-P_2)}{n_2}}} \quad\text{(Wald / unpooled)}$$
Null hypothesis: \(H_0:\ \Delta = \Delta_0\). By default \(\Delta_0 = 0\) (no difference).
Confidence interval for the difference (\(\Delta = P_2 - P_1\)): $$\Delta \pm z_{1-\alpha/2}\sqrt{\frac{P_1(1-P_1)}{n_1}+\frac{P_2(1-P_2)}{n_2}}$$
π Related tools
- Independent t-test - For comparing continuous metrics (not proportions)
- Chi-Square test - For categorical data with more than 2 groups
- McNemar test - For paired/matched data
DIAGNOSTICS & ASSUMPTIONS
View diagnostic checks
Diagnostics will appear after you enter data.