A/B Test Calculator — Statistical Significance for Your Split Test

This free A/B test calculator takes the raw numbers from your split test — visitors and conversions for both variants — and tells you whether the difference you're seeing is a genuine, statistically reliable result or just noise. Enter Variant A's traffic and conversions, Variant B's traffic and conversions, pick the confidence level you need, and you get an instant verdict: a winner with a stated confidence, or "not significant yet." Every calculation happens in your browser using a standard two-proportion z-test, the same math behind the significance checks in most professional testing platforms.

We built this A/B test calculator because so many teams stop tests the moment a variant "looks" ahead, without ever checking whether the gap could plausibly be random chance. At Arb Digital we run split tests on landing pages and ad creative constantly, and the single habit that separates teams who genuinely grow from teams who just reshuffle traffic every month is this: they check significance before they ship a decision, not after.

What This A/B Test Calculator Does

You're running a classic split test: two versions of a page, an email, an ad, or a checkout flow, each shown to a separate slice of traffic. Variant A is usually your control — the existing version. Variant B is the challenger — your new headline, layout, price, or button color. Each variant racks up its own visitor count and its own conversion count, and those four numbers are all this calculator needs.

From them it computes the conversion rate of each variant, the relative lift of B over A, a standard error for the difference between the two rates, a z-score, and a p-value — then compares that p-value against the confidence level you chose. If the result clears your bar, the tool declares a winner and states how confident you can be in that call. If it doesn't, it tells you plainly that you don't have a result yet, which is just as valuable to know as a win.

How to Use It

1. Enter Variant A's visitors and conversions. This is almost always your existing control — the page or version you're testing against.
2. Enter Variant B's visitors and conversions. Your challenger — the new version you're hoping beats the control.
3. Choose your confidence level. 95% is the standard default for most marketing decisions; use 99% for high-stakes changes like pricing or checkout, where a false positive is expensive.
4. Read the verdict. The big result tells you plainly whether you have a statistically significant winner, and which variant it is.
5. Check the lift, not just the stars. A significant result with a tiny relative lift may not be worth the engineering effort to ship. Significance answers "is this real?" — it doesn't answer "is this worth doing?"

The Formula / How It's Calculated

This calculator runs a standard two-proportion z-test, the textbook approach for comparing two conversion rates. First it computes each variant's rate: conversions divided by visitors. Then it pools the two samples into a combined conversion rate and uses that pooled rate to calculate the standard error of the difference between A and B. Dividing the observed difference in rates by that standard error gives a z-score, and converting the z-score through the normal distribution's cumulative density function gives a p-value — the probability of seeing a gap this large (or larger) purely by chance if there were actually no real difference between the variants. This is the same underlying statistical approach documented by Evan Miller's widely cited guide to A/B testing statistics and used across the industry's major testing tools, including those referenced in Google Analytics' experiment reporting documentation.

Why "Peeking" Ships Losers

Here's the mistake that quietly ruins more A/B tests than any other: checking the results every day and stopping the instant the numbers look good. It feels responsible — you're monitoring the test, after all — but it's statistically dangerous. Every time you peek at a test and have the option to stop it, you're giving randomness another chance to produce a false positive. Run that check daily over a two-week test and the chance of falsely declaring a winner at some point along the way can climb far past your stated 5% significance threshold — sometimes to 20% or higher, even when the two variants are truly identical.

The discipline that fixes this is simple to state and hard to follow: decide your sample size before you launch the test, using a tool like our sample size calculator, and don't call a winner until you hit that number. Checking progress along the way is fine — stopping early because the interim numbers look promising is the trap. Treat the pre-committed sample size as a finish line, not a suggestion.

Statistical Significance Is Not Business Significance

A result can be perfectly, rigorously statistically significant and still be a business non-event. Run a test with enough traffic — hundreds of thousands of visitors — and you can detect a 0.2% relative lift with total statistical confidence. It's real. It's also probably not worth a single engineering hour to implement, because the revenue impact rounds to nothing. Significance tells you the difference is probably not random chance; it says nothing about whether the difference is big enough to justify the cost of shipping it. Always read the relative lift percentage alongside the p-value, and ask honestly whether that size of change moves a number anyone in the business actually cares about.

Never Run a Test Through a Holiday

Traffic and buyer intent on Black Friday, Christmas week, or a major holiday weekend behave nothing like a normal Tuesday. Visitors are more price-sensitive, more impulsive, or entirely absent depending on your industry, and whichever variant happens to be running during that window gets a distorted read that has nothing to do with which page design is actually better. If a holiday, a major sale, or a big press mention falls inside your test window, either pause the test and resume it in normal conditions, or extend the run so the anomaly gets diluted across a full, representative sample. A "winner" declared on holiday data rarely holds up once traffic returns to normal.

Reading the Result Correctly

• "Significant — B wins" means the observed lift cleared your chosen confidence bar. Ship it, but sanity-check the lift size first.
• "Not significant yet" means you don't have proof either way. It does not mean the variants are equal — it means you don't yet have enough data to say.
• A negative lift that's significant is just as valuable as a positive one — it tells you to keep the control and stop wasting traffic on the challenger.
• Small differences at small sample sizes will almost never reach significance. That's not a bug in the maths — it's the maths correctly telling you the sample is too small to trust.

Common Mistakes to Avoid

• Peeking and stopping early. Set your sample size in advance and stick to it — see the section above.
• Testing too many things at once. Change one significant element per test, or you won't know which change actually moved the number.
• Ignoring practical significance. A statistically real 0.1% lift usually isn't worth shipping. Weigh the lift size against the cost of change.
• Running through unrepresentative traffic. Holidays, viral spikes, and outages all distort a test window — pause or extend around them.
• Declaring a winner from a tiny sample. Use the sample size calculator before you start so you know how much data you actually need.
• Confusing "not significant" with "no difference." It usually just means you need more data, not that the variants perform identically.

Related Free Tools From Arb Digital

Before you launch your next split test, plan it properly with our sample size calculator — it tells you how many visitors you need before you start, so you're not peeking at an underpowered test. Once you have a winner, model the revenue impact with the landing page conversion calculator, check your baseline rate with the conversion rate calculator, and see how a better-converting page affects your customer acquisition cost and click-through rate. Browse everything at our free online tools hub.

Frequently Asked Questions

Figures produced by this calculator are statistical estimates for planning purposes, based on the numbers you enter.