Twenty numbers. Run them through a t-test one way and you get p = 0.0794: no evidence that the two drugs differ. Run the same twenty numbers another way and you get p = 0.0028: strong evidence that they do. Nothing was added, nothing removed, no outlier dropped. The only thing that changed was a single choice in the form, and one of the two answers is simply wrong.
The numbers are not made up for the occasion. They come from the paper that introduced the t-test, written by William Sealy Gosset under the pen name Student while he worked at the Guinness brewery in Dublin. Ten patients, two sleeping drugs, and for each patient the extra hours of sleep each drug gave compared with a night without it. The same table ships with every copy of R as the sleep data set, so anyone can check every figure below.
This article walks through the wrong reading first, then the reason it goes wrong, then four more mistakes that the same twenty numbers make easy. Each one is a real way people misuse t-tests and p-values, and each one moves a verdict.
The twenty numbers
Every patient took both drugs on different nights. That detail decides everything that follows. The table lists the extra hours of sleep for each drug and, in the last column, the difference for the same patient.
| Patient | Drug 1 (extra hours) | Drug 2 (extra hours) | Drug 2 minus drug 1 |
|---|---|---|---|
| 1 | 0.7 | 1.9 | 1.2 |
| 2 | -1.6 | 0.8 | 2.4 |
| 3 | -0.2 | 1.1 | 1.3 |
| 4 | -1.2 | 0.1 | 1.3 |
| 5 | -0.1 | -0.1 | 0.0 |
| 6 | 3.4 | 4.4 | 1.0 |
| 7 | 3.7 | 5.5 | 1.8 |
| 8 | 0.8 | 1.6 | 0.8 |
| 9 | 0.0 | 4.6 | 4.6 |
| 10 | 2.0 | 3.4 | 1.4 |
Look at the last column before any test. Nine differences are positive and one is zero. Not a single patient slept less on drug 2. The mean difference is 1.58 hours, with a standard deviation of 1.23. Now look at the two drug columns on their own: they swing from -1.6 to 5.5 hours, because some patients simply sleep much more than others whatever they take. Patient 7 gained 3.7 and 5.5 hours; patient 4 gained -1.2 and 0.1. That swing between people is real, but it has nothing to do with the question of which drug works better.
Mistake 1: analyzing pairs as two separate groups
Paste the two columns into the two-sample t-test calculator, choose "Two independent groups" and the Welch version, and it returns t = -1.861 on 17.78 degrees of freedom, p = 0.0794. Switch the design to "Paired measurements" and, with the very same columns, it returns t = -4.062 on 9 degrees of freedom, p = 0.0028. The first result is what you get if you forget that the columns belong to the same ten people.
Why does the pairing matter so much? Both tests put the same mean difference, -1.58 hours (drug 1 minus drug 2), on top of the fraction. What changes is the noise underneath.
The independent test builds its noise from the spread of each column: standard deviations of 1.789 and 2.002, which give a standard error of the difference of 0.849 hours. That spread is mostly the difference between people. The paired test builds its noise from the ten differences only, with a standard deviation of 1.23 and a standard error of 0.389. The heavy sleepers and the light sleepers cancel out, because each patient is compared only with their own other night.
The calculator prints the number that explains the gap: the correlation between the two columns is 0.795. A patient who sleeps a lot on one drug sleeps a lot on the other. The variance of a difference is the sum of the two variances minus twice their covariance, so a strong positive correlation shrinks it. Here it shrinks the standard error to less than half, and t grows from 1.861 to 4.062. Ignore the pairing and you throw away exactly the information the design was built to collect. In the original paper Student worked with the ten differences, not with two separate groups.
One data set, five analyses
The table puts the tempting analyses of these twenty numbers side by side. Every row is a result from the calculators, and every row is a test someone could defend at first glance. Only one of them answers the question the study asked.
| Analysis | t | df | p (two-sided) | Verdict at 0.05 |
|---|---|---|---|---|
| Welch, two independent groups | -1.861 | 17.78 | 0.0794 | not significant |
| Student's pooled, two independent groups | -1.861 | 18 | 0.0792 | not significant |
| Paired, ten differences | -4.062 | 9 | 0.0028 | significant |
| Drug 1 alone against 0 hours | 1.326 | 9 | 0.2176 | not significant |
| Drug 2 alone against 0 hours | 3.680 | 9 | 0.0051 | significant |
Welch and Student agree to three decimals because the two groups are the same size and have similar spreads. That is worth knowing: when the sizes are equal, the choice between them barely matters. The choice between independent and paired matters enormously. The two one-sample rows answer a different question altogether, "does this drug add sleep at all?", and they set up mistake number four.
Mistake 2: choosing one tail after seeing the data
Suppose you ran the wrong, independent test and got 0.0794. It is so close to 0.05. The idea arrives on its own: drug 2 was expected to be better anyway, so why not a one-sided test? Type t = -1.861 and 17.78 degrees of freedom into the p-value calculator, pick the left tail, and the p-value halves to 0.0397. Significant.
That is not a new result, it is the same result with the threshold moved after the race. A one-sided test is legitimate only when the direction was fixed before anyone saw the numbers, written into the protocol, and when an effect in the other direction would be treated exactly like no effect. If the halving happens because the two-sided p missed, the real false-alarm rate is no longer 5% but up to 10%. The p-value calculator gives both tails for exactly this reason: so that the one you report is visibly a choice.
Mistake 3: reading "not significant" as "no difference"
Stay with the wrong analysis a moment longer, because it holds a second trap. Welch says p = 0.0794, not significant. It is tempting to write "the drugs did not differ". But the same printout gives a 95% confidence interval for the difference of -3.3655 to 0.2055 hours. The data are compatible with drug 2 being almost no better than drug 1, and equally with it adding more than three extra hours of sleep a night. That is not evidence of no difference. It is evidence of not enough information.
The calculator says so in words: Cohen's d is -0.83, a large effect, and still not significant, and it estimates that an independent design would need about 24 people per group to detect an effect of that size 80% of the time. Ten per group was never going to be enough. The paired design, by contrast, needs about 7 pairs for the same power, because it removes the between-patient noise. That is the practical argument for pairing, before any statistics are run: the same question costs 14 measurements instead of 48.
Mistake 4: comparing two verdicts instead of testing the difference
Here is a subtler one. Test each drug separately against zero in the one-sample t-test calculator. Drug 2 gives t = 3.680, p = 0.0051: it clearly adds sleep. Drug 1 gives t = 1.326, p = 0.2176: no evidence that it does. So drug 2 works and drug 1 does not, which means drug 2 is better. Right?
The conclusion happens to be true here, but the reasoning is broken. "Significant" and "not significant" are two verdicts on two separate questions, and the difference between them is not itself a test. A p of 0.04 next to a p of 0.06 can come from two nearly identical effects. The only way to say whether drug 2 beats drug 1 is to test the difference directly, which here means the paired test and its p = 0.0028. Had the design been two independent groups, that direct test would have said p = 0.0794, while the two one-sample verdicts would still have read "works" and "does not work".
The one-sample test has plenty of honest uses. A cereal box labeled 12 ounces, a pipette that should deliver 100 microliters, a class average against a national norm: one sample against one fixed value. It is the wrong tool only when the real question is a comparison.
Mistake 5: using z where t belongs
Ten differences is a small sample, and small samples need the t distribution, not the normal one. The difference is not cosmetic. With 9 degrees of freedom the two-sided 5% cut-off is 2.262, not the familiar 1.960, so a confidence interval built with 1.96 comes out about 13% too narrow.
The p-values drift even further apart in the tail. Type t = -4.062 with 9 degrees of freedom into the p-value calculator and you get p = 0.0028. Read the same number as a z-score and you get 4.87 × 10-5, more than fifty times smaller. For comparison, the p-value for a z-score of 4 is 6.33 × 10-5, while the p-value for a z-score of 1.96 sits right at 0.05 and the p-value for a z-score of 2 just under it. Those z figures are right for large samples or a known population SD. With ten patients they overstate the evidence.
Which test fits which design
Most of the mistakes above start before any number is typed, at the moment the design is matched to a test. The table is a short map from the shape of the data to the test and the tool that runs it.
| Your data | Test | What goes in | Tool |
|---|---|---|---|
| One sample against a fixed value (a label, a norm, zero) | One-sample t-test | the raw values or mean, SD and n, plus the reference value | one sample t test |
| The same people or items measured twice | Paired t-test | both columns in the same order | two sample t test |
| Two unrelated groups | Welch's t-test (Student's only by choice) | both columns, any sizes | two sample t test |
| A report that prints only means and SDs | Welch from summaries | mean, SD and n of each group | two sample t test |
| A statistic already printed by software | P-value from t, z, chi-square or F | the statistic and its degrees of freedom | p value |
| Paired differences with a wild outlier | Wilcoxon signed-rank | both columns in the same order | wilcoxon test |
Two rows need a word. Welch is the sensible default for unrelated groups because it does not assume equal spreads, and when the spreads are equal it gives nearly the same answer as Student's version, as the sleep data showed. And the Wilcoxon row exists because the paired t-test assumes the differences are roughly normal. With ten differences one value like patient 9's 4.6 hours can pull the mean. The two-sample calculator flags exactly that on the sleep data: the differences (A minus B) have a skewness of -1.75, strong for ten pairs, so it suggests a signed-rank test as a second look. If your own differences look lopsided, the skewness calculator will say so, and a rank test is the safer check.
Five rules that keep a t-test honest
- Decide the design from how the data were collected, not from which p looks better. Same people or items twice means paired. Different people means independent. If the rows share an ID, they are pairs.
- Fix the tail before seeing the data. Two-sided unless a one-sided hypothesis was written down in advance, and an effect in the other direction would count as nothing.
- Report the interval next to the p-value. -3.37 to 0.21 hours tells a reader far more than "not significant", and it makes the difference between "no effect" and "too few people" visible.
- Test the difference you care about directly. Two separate verdicts against zero do not add up to a comparison.
- Use t, not z, whenever the SD comes from the sample. With 10 observations the cut-off is 2.262, with 30 it is 2.045, and only in large samples does it settle near 1.96.
None of these rules is new. Student's own analysis of these twenty numbers followed the first one, and the paired result he found has held up for well over a century. The mistakes survive because each of them produces a clean, confident number. The tools will compute whatever design you choose. Choosing it is still the analyst's job.
Tools discussed in this article
- Two-sample t-test calculator: Welch, Student's pooled and paired tests from raw values or from means and SDs, with the other versions printed side by side, the confidence interval, Cohen's d, Hedges' g and the group size for 80% power.
- One-sample t-test calculator: one sample against a reference value from raw data or a summary, one- or two-sided, with the interval, effect size and the sample size a follow-up would need.
- P-value calculator: the exact p-value for a z, t, chi-square or F statistic in any tail, fractional degrees of freedom for Welch, and critical values for any alpha, including a Bonferroni-adjusted one.
More statistics tools
Wilcoxon test (the signed-rank test when the differences have heavy tails) · Z-score (one value against a mean and SD) · Pearson correlation (the r between two columns, here 0.795) · Linear regression (a line through paired data) · R squared (how much of the variation a model explains) · Skewness (how lopsided the differences are) · Kurtosis (how heavy their tails are) · Interquartile range (Tukey fences for outliers) · Coefficient of variation (spread as a share of the mean) · Percentile (where one value sits in a list) · Mode (the most frequent value) · Fisher's exact test (small 2x2 tables of counts) · Standard deviation (the spread that goes into every t) · Average (the mean of a list) · Median (the middle value)