A p-value is the probability of obtaining a result at least as extreme as the one you observed, assuming the null hypothesis is true. This calculator converts a test statistic into that probability for the four distributions that cover nearly all routine hypothesis testing: the normal, Student t, chi-square, and F distributions.
Arb Digital evaluates each distribution numerically rather than reading a table. The normal tail comes from the error function, the t and F tails from an incomplete beta function computed by continued fraction, and the chi-square tail from an incomplete gamma function. The result is an exact probability at any degrees of freedom, including fractional ones from Welch-corrected tests.
What This P-Value Calculator Does
Enter a test statistic, choose its distribution, supply degrees of freedom where the distribution requires them, and the tool returns the p-value for the tail you specify. Alongside it you get the critical value at your significance level — the threshold the statistic would have to cross — the cumulative area to the left of your statistic, and a clear reject or do-not-reject decision.
The tail selector applies to the z and t distributions, which are symmetric and can be tested in either direction. Chi-square and F are right-tailed by construction: both statistics are built from squared quantities, so evidence against the null always appears as a large value, and the tail selector is fixed accordingly.
How to Use It
- Pick the distribution that matches the test you ran — z for large-sample proportion and mean tests, t for sample-based mean comparisons, chi-square for categorical counts, F for analysis of variance.
- Enter the test statistic exactly as your analysis produced it, including the sign for z and t.
- Supply the degrees of freedom. The t and chi-square distributions need one value, F needs two, and the normal needs none.
- Choose the tail based on your hypothesis, which should have been fixed before you saw the data.
- Compare the p-value to your significance level, and use the critical value as a sanity check that the two agree.
The Formula and Four Worked Examples You Can Check
Each p-value is a tail area under a probability density curve. For a two-tailed z test, p = 2 × (1 − Φ(|z|)), where Φ is the standard normal cumulative distribution function. For a two-tailed t test, p = Idf/(df+t²)(df/2, 1/2) using the regularised incomplete beta function. For chi-square, p = Q(df/2, χ²/2) using the upper incomplete gamma function. For F, p = Id₂/(d₂+d₁F)(d₂/2, d₁/2).
The neatest way to verify a p-value calculator is to feed it the classic 5% critical values, all of which should return 0.05 to four decimal places. Enter z = 1.9600 two-tailed and the tool returns 0.0500. Enter t = 2.0639 with 24 degrees of freedom, two-tailed, and it returns 0.0500. Enter chi-square = 3.8415 with 1 degree of freedom and it returns 0.0500. Enter F = 3.3258 with 5 and 10 degrees of freedom and it returns 0.0500 again. The small deviations in the sixth decimal come from the rounding in those published critical values, not from the tool.
Those four numbers are the standard 5% critical values printed in the statistical tables at the back of every textbook, and they appear throughout the NIST/SEMATECH e-Handbook of Statistical Methods. If a calculator returns 0.05 for all four, its tail routines are working; if it drifts at the third decimal, it is approximating.
What the p-Value Is Not
The p-value is a conditional probability that runs in one direction: the probability of data this extreme given that the null hypothesis holds. It is not the probability that the null hypothesis is true given your data. Those two quantities can differ enormously, and converting between them requires a prior probability the p-value does not contain.
Three further clarifications. A p-value of 0.03 does not mean there is a 3% chance the result is a fluke. It does not mean there is a 97% chance the effect will replicate. And a large p-value is not evidence that the null hypothesis is true — it means your data was insufficient to distinguish the effect from noise, which is a statement about the study rather than about reality. Absence of evidence is not evidence of absence, and with a small sample the two look identical from the outside. Penn State's STAT 500 course notes set out the same interpretation rules for hypothesis tests.
Why the Threshold Is Arbitrary
The 5% convention has no mathematical basis. It comes from a 1920s suggestion that one in twenty was a convenient standard of surprise, and it stuck because printed tables were expensive to produce at many levels. Nothing distinguishes p = 0.049 from p = 0.051 except which side of a historical convention they land on.
The practical implication is to report the actual p-value rather than a bare "significant" label, and to state the threshold in advance. Fields where a false positive is expensive use far stricter levels: particle physics requires the equivalent of about p = 0.0000003, and genome-wide association studies apply corrections that push thresholds below one in a million because they test millions of hypotheses at once. Your threshold should reflect the cost of being wrong in your own context, not the default in a spreadsheet.
Multiple Testing: The Fastest Way to a False Positive
Every test at the 5% level carries a 5% chance of a false positive when nothing is happening. Run 20 independent tests and the probability of at least one false positive is 1 − 0.95²⁰ ≈ 64%. Run 100 and it is 99.4%. This is why scanning a dashboard for any segment showing a significant difference reliably produces significant differences.
The standard defences are the Bonferroni correction, which divides the threshold by the number of tests (0.05 ÷ 20 = 0.0025), and false discovery rate procedures, which are less conservative and better suited to large-scale screening. Both require you to count every test performed, including the ones you decided not to report. A p-value from an analysis where 30 variants were examined and the best one selected is not comparable to a p-value from a single pre-specified test, however identical the two numbers look. The probability calculator is useful for working through the compounding arithmetic.
Statistical Significance and Practical Importance
The p-value depends on both the size of an effect and the amount of data. With a large enough sample, an effect too small to matter will produce a very small p-value. A conversion rate difference of 0.02 percentage points across ten million sessions is statistically overwhelming and commercially irrelevant.
The remedy is to report an effect size and a confidence interval alongside every p-value. The interval shows the range of effects the data is consistent with, which is directly actionable in a way a probability is not; the confidence interval calculator builds it from your summary statistics. Effect size measures such as Cohen's d, available in the t-test calculator, express the magnitude in units that do not grow with n. The pairing of an effect size with an interval answers the question a business actually has, which is how big the difference is, not merely whether it exists.
One Tail or Two, and When Each Applies
A two-tailed test splits the significance level between both ends of the distribution, asking whether the statistic is extreme in either direction. A one-tailed test puts the whole level in one end, which makes it more sensitive to effects in that direction and completely blind to effects in the other.
That blindness is the reason one-tailed tests are hard to justify. If a result in the opposite direction would change your decision — and it usually would — then a one-tailed test is inappropriate, because it cannot detect one. The direction must also be fixed before the data is examined; choosing a tail after seeing which way the numbers went halves the p-value without any corresponding gain in evidence, which is why the practice is treated as a reporting error rather than an analytical choice. Note that chi-square and F tests are right-tailed but not one-tailed in this sense: their statistics cannot be negative, so all evidence against the null necessarily lies in the upper tail.
Arb Digital publishes a growing library of free calculators for statistics, maths, and analysis — no sign-up, no limits, and every formula documented on the page.
Browse All Free Tools Contact Arb DigitalCommon Mistakes to Avoid
- Reading the p-value as the probability the null is true — it is the probability of the data given the null, which is a different quantity entirely.
- Treating a large p-value as proof of no effect — it usually means the study lacked the power to detect one.
- Switching to a one-tailed test after seeing the direction — this halves the p-value without adding any evidence.
- Running many tests and reporting the smallest p-value — with 20 tests, a false positive is more likely than not.
- Confusing statistical and practical significance — a very large sample makes trivial effects significant.
Related Free Tools From Arb Digital
Produce the statistic first with the t-test calculator or the chi-square calculator, standardise an observation with the z-score calculator, and pair every p-value with a range from the confidence interval calculator. Plan a study with enough power using the sample size calculator, and check compounding error rates with the probability calculator. Everything else lives in the free online tools hub.
Frequently Asked Questions
It means that if the null hypothesis were true, data at least as extreme as yours would occur about 5% of the time. It does not mean there is a 5% chance the null hypothesis is true, and it says nothing about the size of the effect.
Use z for large-sample tests of means or proportions with a known standard deviation, t for mean comparisons where the spread was estimated from the sample, chi-square for tests on categorical counts, and F for comparing variances or running analysis of variance.
Not mathematically, though it can be small enough that floating-point arithmetic reports zero. This calculator switches to scientific notation for very small values, so a result such as 3.2e-12 remains readable rather than collapsing to nothing.
Both statistics are built from squared quantities and cannot be negative. Any departure from the null hypothesis, in either direction of the underlying data, makes the statistic larger, so all the evidence against the null sits in the upper tail.
They are two views of the same comparison. The p-value is the tail probability of your statistic; the critical value is the statistic that would produce exactly your significance level. If the statistic exceeds the critical value, the p-value is below the threshold.
No. The p-value reflects both effect size and sample size, so a tiny effect measured across a huge sample yields a very small p-value. Judge magnitude with an effect size and a confidence interval instead.
That depends on the cost of a false positive in your context. The 5% convention is historical rather than principled, and fields where errors are expensive routinely use far stricter levels. Whatever you choose, fix it before running the analysis.
This tool is provided for education and general analysis. Results depend on your test meeting the assumptions of the chosen distribution, and it is not a substitute for professional statistical advice on a specific study design.
