Sample Size Calculator — How Long to Run Your A/B Test

This free sample size calculator answers the question every A/B test needs answered before it starts, not after: how many visitors does each variant actually need before the result can be trusted? Enter your baseline conversion rate, the smallest lift you care about detecting, your target power and confidence level, and optionally your daily traffic, and you get the exact per-variant visitor count required — plus, if you add your traffic, a realistic estimate of how many days the test will take to reach it.

We built this sample size calculator because it's the step most teams skip. They launch a test, check it daily, and stop whenever the numbers look good — which is exactly backwards. At Arb Digital, before any client's landing page test goes live, we run this calculation first, because a test designed without a target sample size isn't really a test. It's a coin flip dressed up in a dashboard.

What This Sample Size Tool Does

Sample size planning is the missing half of A/B testing. Our A/B test calculator tells you, after a test has run, whether the result you got is statistically real. This tool does the opposite job — it tells you, before a test starts, how much data you'll need to collect in order to trust that result once it arrives. The two tools are meant to be used together: plan with this one, then verify with that one.

Feed it four numbers — your current conversion rate, the smallest relative improvement worth detecting, how confident you want to be (confidence level), and how often you're willing to miss a real effect (statistical power) — and it returns the number of visitors each variant needs to see before you can trust a significant result.

How to Use It

1. Enter your baseline conversion rate. Pull this from your analytics for the page or flow you're about to test — the more accurate this number, the more accurate your plan.
2. Set your minimum detectable effect (MDE). This is the smallest relative lift that would actually be worth shipping. Don't ask the test to detect a lift smaller than you'd bother acting on.
3. Choose your statistical power. 80% is the standard default — it means you'll correctly catch a real effect 80% of the time it exists. 90% is more rigorous but needs more traffic.
4. Choose your confidence level. 95% is standard for most marketing decisions; 99% for high-stakes changes.
5. Add your daily visitors (optional). This converts the raw sample size into a practical number of days, so you know upfront whether the test is realistic to run.

The Formula / How It's Calculated

This calculator uses the standard sample-size formula for comparing two proportions, the same approach behind the well-known reference calculator at Evan Miller's A/B test sample size tool. It converts your confidence level and power into their corresponding z-scores (for example, 95% confidence corresponds to a z-score of about 1.96, and 80% power corresponds to about 0.84), combines those with the baseline rate and the target rate implied by your minimum detectable effect, and solves for the number of observations per group needed so that a true difference of that size would be detected with your chosen power, at your chosen confidence level. It's a closed-form statistical formula, not a simulation — the same underlying maths documented in Google Analytics' guidance on experiment design.

This Tool Tells You Whether a Test Is Even Possible

The most useful thing this calculator does isn't the number it produces — it's the reality check that number forces. Plug in a site with 500 visitors a day, a 2% baseline conversion rate, and a target of detecting a 5% relative lift, and the answer will be a sample size that, at your traffic, takes months to reach — sometimes past next Christmas. Run the test anyway, without knowing this, and you'll spend that time staring at daily numbers that bounce around meaninglessly, "peeking" at a test that was never adequately powered to give you an answer, and eventually calling a winner based on noise.

This is the single biggest reason A/B tests fail teams: not bad ideas, but tests designed to answer a question the traffic simply cannot answer in a reasonable time frame. Running this calculator first turns that invisible failure into a visible number, before you've spent a month of traffic finding out the hard way.

Smaller Effects Need Dramatically More Traffic

The relationship between the effect size you're trying to detect and the sample size you need is not linear — it's closer to inverse-square. Halving your minimum detectable effect roughly quadruples the sample size required. Detecting a 20% relative lift might need a few thousand visitors per variant; detecting a 5% relative lift on the same baseline can need over ten times as many. This is why so many "inconclusive" tests on small or mid-sized sites aren't actually inconclusive — they were simply asked to detect an effect too small for the traffic available, and no result was ever going to reach significance in a sensible time frame.

The practical takeaway: on lower-traffic sites, design tests around bold, high-conviction changes rather than small tweaks. A dramatically different headline or offer produces a bigger MDE, which this calculator will show you needs a far smaller — and far more achievable — sample size than testing a button color.

If the Answer Is "97 Days," Change the Test, Not the Maths

When this calculator returns a timeline that's clearly unworkable, resist the temptation to shrink the sample size by lowering your confidence level or power just to make the number look friendlier — that just means you'll trust a result that's more likely to be wrong. Instead, change what you're testing:

• Raise the minimum detectable effect. Test a bigger, bolder change instead of a marginal tweak — bigger changes are both easier to detect and often more impactful anyway.
• Widen where the test runs. Testing across your whole funnel or site instead of a single low-traffic page increases the visitor pool feeding the test.
• Extend the timeline honestly. If the change truly matters, it may simply be worth the longer run — just go in knowing the real number, not hoping it finishes early.
• Switch to a different method. For very low-traffic pages, sequential or Bayesian testing approaches, or simply qualitative research and user testing, can be more efficient than a classic split test.

Common Mistakes to Avoid

• Skipping this step entirely. Launching a test with no target sample size means you have no way to know when — or whether — it's actually finished.
• Setting an unrealistically small MDE. Asking to detect a 2% relative lift on low traffic will produce an unworkable sample size — be honest about what's worth detecting.
• Lowering confidence just to shrink the number. This doesn't fix the underlying traffic problem, it just makes you more likely to trust a false result.
• Stopping before reaching the calculated sample size. Use this number together with the A/B test calculator — plan here, verify there, and don't call it early.
• Ignoring weekly traffic cycles. Make sure your test window covers full weeks, since weekday and weekend behavior often differs substantially.

Related Free Tools From Arb Digital

Once you know your required sample size, run the finished test through our A/B test calculator to confirm a real winner. Model what a winning variant is worth with the landing page conversion calculator, check your current baseline with the conversion rate calculator, and see the bigger economic picture with the marketing ROI calculator and cost per lead calculator. Browse everything at our free online tools hub.

Frequently Asked Questions

Figures produced by this calculator are statistical planning estimates based on the numbers you enter.