A t-test asks whether a difference between averages is bigger than the noise in the data can comfortably explain. This calculator runs all three standard forms — one-sample, two-sample independent, and paired — directly on numbers you paste in, and returns the t statistic, the exact p-value, the degrees of freedom, the critical value at your chosen significance level, and Cohen's d as a measure of effect size.
Arb Digital built the tool around exact distribution maths rather than table lookups. The p-value comes from an incomplete beta function evaluated by continued fraction, which is the same routine used inside professional statistics packages, so fractional Welch degrees of freedom such as 12.61 are handled properly instead of being rounded down to the nearest whole number.
What This T-Test Calculator Does
The two-sample test compares two independent groups — a control and a variant, two suppliers, two cohorts. The one-sample test compares a single group against a fixed reference value, such as a specification, a target, or a published benchmark. The paired test handles measurements taken twice on the same subjects, where each pair is linked and the analysis works on the differences.
For two-sample tests you also choose the variance assumption. Welch's version does not assume the two groups have equal spread and adjusts the degrees of freedom accordingly. The pooled version assumes equal variances and combines both samples into one estimate of spread. Welch is the safer default in nearly every practical situation, and it is offered first for that reason.
How to Use It
- Choose the test type. If each number in sample 1 has a natural partner in sample 2 — the same person, machine, or page measured twice — you want the paired test, not the two-sample test.
- Paste your data. Values can be separated by commas, spaces, tabs, or line breaks, so a column copied straight out of a spreadsheet works without editing.
- Set the alternative hypothesis. Two-tailed asks whether the means differ at all; one-tailed asks about one specific direction and must be chosen before you look at the data.
- Pick the significance level. This sets the critical value shown in the results grid. It does not change the p-value, which is a property of the data alone.
- Read the result. Compare the p-value to your significance level, then check Cohen's d to judge whether the difference is large enough to matter.
The Formula and a Worked Example You Can Check
For two independent samples with pooled variance, the statistic is t = (x̄₁ − x̄₂) ÷ √(s²p × (1/n₁ + 1/n₂)), where the pooled variance s²p = ((n₁ − 1)s₁² + (n₂ − 1)s₂²) ÷ (n₁ + n₂ − 2) and the degrees of freedom are n₁ + n₂ − 2. Welch's version replaces the pooled term with √(s₁²/n₁ + s₂²/n₂) and uses the Welch-Satterthwaite approximation for degrees of freedom.
Check the default data by hand. Sample 1 is 24, 28, 32, 29, 35, 26, 31, 30 — eight values with a mean of 29.375 and a sample variance of 11.9821. Sample 2 is 20, 22, 27, 25, 23, 21, 26, 24, with a mean of 23.5 and a variance of 6.0. The pooled variance is (7 × 11.9821 + 7 × 6.0) ÷ 14 = 8.9911. The standard error is √(8.9911 × (1/8 + 1/8)) = √2.2478 = 1.4993. The t statistic is (29.375 − 23.5) ÷ 1.4993 = 3.9186 on 14 degrees of freedom, giving a two-tailed p-value of 0.00154 against a critical value of 2.1448.
Switch the same data to Welch and the t statistic is unchanged at 3.9186, because the sample sizes are equal, but the degrees of freedom fall to 12.61 and the p-value rises slightly to 0.00186. That is Welch's correction visible in a single click. The procedure follows the standard two-sample method documented in the NIST/SEMATECH e-Handbook section on the two-sample t-test.
Why Welch Should Be Your Default
Statistics courses traditionally teach the pooled test first and present Welch as a special case for when variances are unequal. Practice has moved the other way. Welch's test performs almost identically to the pooled test when the variances really are equal, and far better when they are not, so the cost of always using it is close to zero and the cost of wrongly using the pooled test can be substantial.
The failure mode is worst when unequal variances are paired with unequal sample sizes. If the smaller group also happens to have the larger spread, the pooled test produces p-values that are too small — it reports significance that is not there. The traditional remedy, running a preliminary F-test for equal variances and choosing the t-test based on the outcome, makes things worse, because the two-stage procedure distorts the error rate of the test that follows it. Choosing Welch unconditionally avoids the whole problem. Penn State's STAT 500 course notes set out both versions with worked comparisons.
Paired Data Analysed as Independent: The Expensive Mistake
When the same subjects are measured twice, treating the two columns as independent groups throws away the pairing and usually destroys the test's power. The paired test works on the within-subject differences, so all the variation between subjects — the fact that some people are simply faster, heavier, or higher-converting than others — cancels out.
The practical effect is large. Ten people whose scores each improve by roughly 3 points will produce a highly significant paired result, because every difference points the same way. Run the same numbers as two independent samples and the individual differences between people swamp the 3-point shift, often leaving a p-value nowhere near significance. If your data has a natural pairing, using it is not an optional refinement — it is the difference between detecting the effect and missing it.
Reading the p-Value Without Overreading It
The p-value is the probability of obtaining a t statistic at least this extreme if the null hypothesis were true. It is not the probability that the null hypothesis is true, and it is not the probability that your result would replicate. A p-value of 0.03 does not mean there is a 97% chance the effect is real.
Two consequences follow. First, the difference between p = 0.049 and p = 0.051 is meaningless; treating one as a discovery and the other as a null result is an artefact of a threshold chosen for convenience in the 1920s. Second, a non-significant result is not evidence of no difference — it means this data was not sufficient to distinguish the difference from noise, which is a statement about the study, not about reality. Use the confidence interval calculator on the observed difference to see what magnitudes remain plausible, and the p-value calculator if you already have a test statistic and only need the probability.
Effect Size: The Number the p-Value Hides
Cohen's d expresses the difference between two means in standard deviation units: d = (x̄₁ − x̄₂) ÷ spooled. It is unaffected by sample size, which is precisely why it belongs next to every p-value. The conventional reference points are 0.2 for a small effect, 0.5 for medium, and 0.8 for large, though these are rough conventions rather than measured constants and vary by field.
The default data gives d ≈ 1.96, an unusually large separation — the two group means are nearly two standard deviations apart. In real work you will more often see a p-value of 0.001 attached to a d of 0.08, which happens whenever the sample is very large. That combination means the difference is real and almost certainly too small to act on. Reporting both numbers together makes that distinction visible immediately.
Assumptions, and Which Ones Actually Matter
The t-test assumes independent observations, approximate normality of the sampling distribution, and (for the pooled version) equal variances. Independence is the assumption that matters most and is the hardest to repair. Repeated measurements on the same users, values collected in a time series, or observations clustered within teams all violate it, and no variance correction will fix the resulting understatement of uncertainty.
Normality matters far less than most people expect. Thanks to the central limit theorem the test is robust to moderate departures once each group has more than roughly 15 to 20 observations. What does still cause trouble is severe skew combined with small samples, and outliers — a single extreme value can move a mean and inflate a variance enough to flip a conclusion. Screen for that first with the z-score calculator, and check the spread of each group with the standard deviation calculator before trusting the test.
Arb Digital publishes a growing library of free calculators for statistics, maths, and analysis — no sign-up, no limits, and every formula documented on the page.
Browse All Free Tools Contact Arb DigitalCommon Mistakes to Avoid
- Running a two-sample test on paired data — you discard the pairing that makes the comparison sensitive, and usually lose the effect entirely.
- Choosing a one-tailed test after seeing the direction — the alternative hypothesis has to be fixed before the data is examined.
- Testing repeatedly as data accumulates — checking a running experiment daily until it reaches significance inflates the false positive rate well beyond the stated level.
- Comparing more than two groups with a series of t-tests — three pairwise comparisons at 5% each give roughly a 14% chance of at least one false positive.
- Reporting significance without effect size — with thousands of observations, a difference too small to matter will still produce a very small p-value.
Related Free Tools From Arb Digital
Plan the sample before collecting it with the sample size calculator, describe the spread with the standard deviation calculator, and place an interval on the difference with the confidence interval calculator. For categorical outcomes use the chi-square calculator instead. Everything else lives in the free online tools hub.
Frequently Asked Questions
Use the one-sample test to compare a group against a fixed reference value, the paired test when the same subjects are measured twice, and the two-sample test for two genuinely independent groups. The pairing question is the one people most often get wrong.
The pooled test assumes both groups have the same variance and combines them into one estimate. Welch's test allows different variances and adjusts the degrees of freedom instead. Welch is safer in almost every real situation and costs almost nothing when variances are in fact equal.
It is the probability of seeing a difference at least this large if there were no real difference between the groups. It is not the probability that the null hypothesis is true, and it says nothing about how big or important the difference is.
There is no fixed minimum, but very small samples give the test little power to detect anything but a huge effect. Below roughly 15 per group the normality assumption also starts to matter, so check for skew and outliers before relying on the result.
Welch's test uses the Welch-Satterthwaite approximation, which produces a fractional value such as 12.61. This calculator evaluates the t distribution at that fractional value directly rather than rounding it to a whole number.
Not with a series of pairwise tests, because each comparison carries its own false positive risk and they accumulate. Analysis of variance handles multiple groups in a single test, with post-hoc procedures for the individual comparisons.
It measures the size of the difference in standard deviation units, independent of sample size. A significant p-value with a tiny d means the effect is real but probably too small to be worth acting on.
This tool is provided for education and general analysis. Results depend on your data meeting the assumptions of the chosen test, and it is not a substitute for professional statistical advice on a specific study design.
