Z-Test Calculator - A/B Test, One Proportion and Mean

Z-test for an A/B split, a single rate or a mean with known sigma

Two thousand visitors saw version A and 200 of them bought. Another two thousand saw version B and 250 bought. Is a jump from 10% to 12.5% real? This z-test calculator answers with z = 2.502 and a two-sided p = 0.0124, then goes further than most A/B tools: it prints the continuity-corrected p that R's prop.test shows by default (0.0142), a 95% interval for the lift, Cohen's h and the group size you would need for 80% power at the same gap, 2,507 per arm. The same form runs a one-proportion test against a benchmark, with an exact binomial p beside the normal one, and a z-test for a mean when the population standard deviation is known from outside the sample.

A dashboard reading versus a tested difference

Reading the rates by eye

a +25% lift on 1,000 visitors per arm looks like a win, yet p is 0.0769
the test gets stopped the first day the dashboard turns green
3 successes out of 10 get judged with a formula built for large samples
nobody knows how much traffic the next test needs

What the z-test adds

z, the p-value and the critical z for the alpha you picked
the continuity-corrected p, and a warning when it falls on the other side of alpha
an exact binomial p for one proportion, which matters below about 10 expected successes
the sample size for 80% power at the gap you observed

What to type for each of the three tests

Kind of z-test - two proportions for an A/B test, one proportion against a benchmark rate, or one mean with a known population standard deviation. Only the boxes for that test appear.
Group sizes and successes - for A/B, visitors (or users, or emails sent) and conversions in each group, as whole numbers. Write 1000, not 1,000: a comma is refused rather than guessed.
Benchmark rate - for one proportion, the rate you compare with, in percent, strictly between 0 and 100, for example 50 for a coin or 3.5 for last year's conversion rate.
Sample mean, population sigma, n and reference mean - for the mean test. Sigma has to come from outside these measurements, such as a machine specification or long-run records.
Alternative hypothesis - two-sided for any difference, or one-sided when the direction was fixed before the data came in (B above A, or the sample below the reference).
Significance level - 0.05, 0.01 or 0.10. It also sets the confidence level of the interval.
Read the result: z and the verdict first, then the tiles, the step table and any note about small counts or a one-sided test pointing the wrong way.

How the z statistic is built, three cases by hand

    two proportions: z = (pB - pA) / sqrt(p(1 - p)(1/nA + 1/nB)), p = pooled rate

    one proportion: z = (p - p0) / sqrt(p0(1 - p0) / n)

    one mean: z = (mean - mu0) / (sigma / sqrt(n))

The A/B test. The pooled rate is 450 / 4,000 = 11.25%, so the standard error is sqrt(0.1125 × 0.8875 × (1/2,000 + 1/2,000)) = 0.00999. The gap of 0.025 divided by it gives z = 2.502 and p = 0.0124. Squared, z is 6.26, the chi-square that prop.test reports with correct = FALSE. The 95% interval for B - A runs from 0.54% to 4.46%.

Three out of ten against 50%. The standard error under the benchmark is sqrt(0.25 / 10) = 0.158, so z = -0.2 / 0.158 = -1.265 and the normal p is 0.2059. The exact binomial p is 0.3438, almost 70% larger. With only 5 expected successes the calculator flags the gap and tells you which p to keep.

A mean with known sigma. A class averages 103 on a test scaled to mean 100 and sigma 15, with 36 students. The standard error is 15 / 6 = 2.5, z = 3 / 2.5 = 1.200, p = 0.2301. Cohen's d is 0.200, and detecting a 3-point gap with 80% power takes 197 students.

The same 10% versus 12.5%, at five amounts of traffic

Rates fixed, only the visitors per arm change. The lift is identical in every row, and the verdict flips somewhere between 1,200 and 2,000 per arm. The last column is what you gain or lose by stopping at that point.

Visitors per arm	z	p, two-sided	95% interval for B - A	Verdict at 0.05
800	1.582	0.1136	-0.59% to 5.59%	not significant
1,200	1.938	0.0526	-0.03% to 5.03%	not significant, by a hair
2,000	2.502	0.0124	0.54% to 4.46%	significant
2,600	2.853	0.0043	0.78% to 4.22%	significant
4,000	3.538	0.0004	1.12% to 3.88%	significant
Planned in advance	2,507 per arm gives an 80% chance of catching this gap			decided before launch

The 1,200 row is the trap. A p of 0.0526 invites one more day of traffic "to get it over the line", and repeated peeking like that inflates the false positive rate well above the 5% you think you are running at. Fix the sample size first, then test once.

Critical z values worth memorizing

Alpha	Two-sided	One-sided	Confidence level
0.10	1.645	1.282	90%
0.05	1.960	1.645	95%
0.01	2.576	2.326	99%

For any other cutoff, or for t, chi-square and F statistics, the p-value calculator returns the critical value for an alpha of your choice.

Short answers before you ship variant B

Is a z-test for two proportions the same as a chi-square test?

For a 2x2 table, yes: the chi-square statistic equals z squared and the two-sided p-values match. In the example above z = 2.502 and chi-square = 6.26. The one-sided test exists only in the z form.

Why does my result differ from R or another calculator?

R's prop.test applies Yates' continuity correction by default, which shrinks the gap by half a count and raises p, here from 0.0124 to 0.0142. Most online A/B calculators skip it. This tool prints both, so you can match either one.

When should I use a t-test instead of a z-test for a mean?

Whenever the standard deviation comes from the same sample, which is almost always. The z-test needs sigma known in advance. With a sample of 30 or more the two give close answers, below that the one-sample t-test is the honest choice.

How small is too small for a z-test on counts?

A common rule asks for at least 10 expected successes and 10 expected failures for one proportion, and at least 5 in every cell for two. Below that, read the exact binomial p here or use Fisher's exact test for two groups.

Can I run a one-sided test to reach significance faster?

Only if the direction was written down before launch. Switching after seeing the data halves the p-value on paper and doubles the real false positive risk. At 2,000 visitors per arm the example gives 0.0062 one-sided and 0.0124 two-sided.

Related tools

Chi-Square Calculator

The same two-group comparison as a 2 x 2 table, plus tables with more rows and columns - See calculator

Sample Size Calculator

How many visitors each variant needs before the test starts, with dropout and power - See calculator

P-Value Calculator

Turns a z, t, chi-square or F statistic into a p-value and gives critical values for any alpha - See calculator

Z-Score Calculator

Standardizes a single value, finds its percentile and spots outliers in a list - See calculator

Fisher's Exact Test Calculator

The exact alternative for a 2x2 table when some counts are small - See calculator

One-Sample T-Test Calculator

Tests a mean against a target when the standard deviation comes from the sample itself - See calculator

Two-Sample T-Test Calculator

Compares the means of two groups with Welch, Student or a paired test - See calculator