A complication rate of 5% on a new protocol and 30% on the old one is a difference nobody needs a statistician to see. Twenty-five percentage points. Six times the risk. If a gap that size could come from chance, what couldn't?
Except it can. With 20 patients in each group, that is 1 complication against 6, and Fisher's exact test puts the two-sided p-value at 0.0915. Not significant at 0.05. The odds ratio is 0.13, which sounds decisive until you read the 95% interval next to it: 0.0025 to 1.25. The data cannot rule out that the new protocol is no better at all.
Small samples are where statistics stops matching intuition, and where most of the popular rules of thumb quietly fail. Below are six beliefs about small samples, each checked with real numbers, one that turns out to be right, and a single rule that survives all of them. Every figure was run through Fisher's exact test, the Wilcoxon test or the interquartile range calculator, and the t-tests and chi-square values were checked against their textbook formulas.
Six beliefs that small samples break
Can a 25-point gap still be chance?
The easiest way to see why it is not: count the possibilities. Seven people had a complication, forty were treated, twenty in each group. Keep those totals fixed and there are only 8 ways the seven complications can split between the groups, from 0 and 7 to 7 and 0. Each split has an exact probability if the protocols are equally safe.
| Complications on the new protocol | Probability | As extreme as 1 of 20? |
|---|---|---|
| 0 | 0.0042 | yes |
| 1 (observed) | 0.0416 | yes |
| 2 | 0.1580 | no |
| 3 | 0.2963 | no |
| 4 | 0.2963 | no |
| 5 | 0.1580 | no |
| 6 | 0.0416 | yes |
| 7 | 0.0042 | yes |
The observed split, and every split at least as lopsided in either direction, adds up to 0.0915. Roughly one time in eleven, two equally safe protocols would produce a table this uneven. That is the whole test, with no approximation anywhere.
And it is not a quirk of this table. An email A/B test with 14 sign-ups out of 100 against 5 out of 100, almost three times the rate, lands at p = 0.0513. Size of the gap and strength of the evidence are two different things.
When does chi-square go wrong on a 2x2 table?
Run a Pearson chi-square on the same 1 of 20 against 6 of 20 and you get χ² = 4.33 and p = 0.0375. Significant. Same four numbers, opposite verdict.
The chi-square p-value comes from a smooth curve that only fits well when every expected count in the table is around 5 or more. Here the smallest expected count is 3.50. The table below puts three small tables through all three methods: plain chi-square, chi-square with Yates' continuity correction, and Fisher's exact test.
| Table | Smallest expected | Chi-square p | With Yates | Fisher exact p |
|---|---|---|---|---|
| 1 of 20 vs 6 of 20 | 3.50 | 0.0375 | 0.0960 | 0.0915 |
| 0 of 12 vs 4 of 12 | 2.00 | 0.0285 | 0.1003 | 0.0932 |
| 7 of 8 vs 2 of 10 | 4.00 | 0.0044 | 0.0177 | 0.0152 |
Plain chi-square is too eager every time. Yates pulls it back and overshoots, landing a little above the exact answer in all three rows. The calculator prints the smallest expected count under every result, so the question "did I need the exact test?" answers itself.
You can also check it by hand before opening any tool. The expected count of a cell is its row total times its column total, divided by the grand total. For the complication table that is 7 × 20 / 40 = 3.5 for each group's complication cell and 33 × 20 / 40 = 16.5 for the cells without one. The usual guidance allows at most one cell in five below 5, and a 2x2 table only has four cells, so in practice a single expected count under 5 is already one too many.
Is one-sided p really half of two-sided?
Sometimes it is exactly half. The A/B test above has a one-sided p of 0.0257 against 0.0513 two-sided. But take a pass rate of 9 of 12 after tutoring against 4 of 15 without it: one-sided p = 0.0165, two-sided p = 0.0213. That is 77%, not 50%.
The reason is that the distribution of possible tables is lopsided whenever the group sizes or the totals are, so the two tails do not match. The bigger problem is behavioral, though. Choosing one-sided after seeing which way the data lean is the most common way to turn 0.0513 into 0.0257. The direction has to be written down before the data exist, or the one-sided number means nothing.
The checkout time that sank a t-test
Checkout times in seconds for two page designs, eight people each. Design A: 48, 52, 61, 45, 58, 56, 50, 55. Design B: 38, 41, 44, 36, 47, 40, 43, 39. Welch's t-test gives p = 0.0002. Clear.
Now one participant on design A gets distracted and takes 190 seconds instead of 56. Welch's t-test drops to p = 0.1387. Not significant. One number out of sixteen erased the effect, because it dragged the mean of design A up by 16.75 seconds and inflated the standard deviation along with it.
The Wilcoxon rank-sum test, also called the Mann-Whitney U test, gives p = 0.0003 with or without the outlier. To a rank test, 190 seconds is simply the slowest time, the same rank 70 seconds would get. It also reports the typical shift, -12.5 seconds with the outlier and -12 without, and how often a random design B time is slower than a random design A time: 1.6%, one pair out of 64.
Why Excel, a TI-84 and R give different quartiles
Type the eight numbers 3, 5, 7, 8, 9, 11, 13, 15 into four different tools and you get three different interquartile ranges.
| Method | Q1 | Q3 | IQR |
|---|---|---|---|
| Linear interpolation (Excel QUARTILE.INC, Google Sheets, R) | 6.5 | 11.5 | 5 |
| Excel QUARTILE.EXC (Minitab, SPSS) | 5.5 | 12.5 | 7 |
| Halves without the median (TI-84) | 6 | 12 | 6 |
| Tukey's hinges (R fivenum) | 6 | 12 | 6 |
None of them is wrong. There are several published definitions of a quartile, and software picked different ones. The classic homework example 1, 2, 3, 4, 5 shows it in the smallest possible form: Excel's QUARTILE.INC gives an IQR of 2, a TI-84 gives 3. A student checking a calculator answer against a spreadsheet is not making a mistake, just comparing two rules.
Which one to report, then? The one your reader will check against. A lab report whose numbers will be redone in R or Excel should use linear interpolation. An AP Statistics answer key expects the TI-84 halves. A box plot drawn by hand usually follows Tukey's hinges. Whatever you pick, name it next to the result, the same way you would name the units.
Flagged by the 1.5 × IQR rule is not the same as wrong
Eleven delivery times: 22, 25, 27, 28, 30, 31, 33, 35, 38, 41 and 95 minutes. Q1 is 27.5, Q3 is 36.5, the IQR is 9, and the upper fence at Q3 + 1.5 × IQR sits at 50. The 95-minute delivery is outside it, beyond 3 × IQR even, so it is flagged as extreme.
Flagged is not the same as wrong. The rule catches about 0.7% of values even in perfectly normal data, roughly seven in every thousand, none of them errors. A 95-minute delivery might be a typo for 59, or it might be the one order that went to the wrong address, which is exactly the one worth knowing about. What the IQR does well is refuse to be pushed around: setting the 95 aside cuts the standard deviation from 20.09 to 5.89, a drop of 70.7%, while the IQR only moves from 9 to 7.25.
Where the skeptics are right: rank tests do cost something
Of the seven beliefs, this is the one that holds up: rank tests are weaker than t-tests on clean data.
Ten patients score their pain before and after physical therapy: 7, 6, 8, 5, 7, 9, 6, 8, 7, 6, then 4, 5, 6, 5, 3, 7, 5, 6, 4, 6. Eight improved, two stayed the same, nobody got worse. A paired t-test gives p = 0.0019. The Wilcoxon signed-rank test gives 0.0078, about four times larger, with a typical change of -2 points.
Part of the gap is structural. The two unchanged patients drop out of the signed-rank test, leaving 8 pairs, and with 8 pairs the smallest two-sided p the test can ever produce is 2 divided by 28, which is 0.0078. This data already sits on that floor. With 4 changed pairs the floor is 0.1250, so no result, however consistent, can reach 0.05. Fisher's test has the same kind of floor: with 3 people per group even a perfect 3 of 3 against 0 of 3 only reaches p = 0.1000, and it takes 4 per group to get below 0.05.
So the trade is real. On small, clean, symmetric data a t-test squeezes a little more out of the same numbers. On anything with an outlier, a skew or a rating scale, the rank test gives an answer that one bad measurement cannot overturn.
What to read before the p-value
Every belief that failed above has the same root: judging a result by how it looks rather than by how much data stands behind it. A 25-point gap, a threefold rate, a flagged value, a significant chi-square. Before reading the p-value, read the sample size and the smallest expected count, and pick the test that fits them, not the one that gives the nicer number. Then report the effect with its interval, because an odds ratio of 0.13 means very little next to an interval that runs to 1.25.
Tools discussed in this article
- Fisher's exact test calculator: the exact p-value for a 2x2 table, one- or two-sided, with the odds ratio and its exact interval, the risk ratio and the smallest expected count.
- Wilcoxon test calculator: signed-rank for before-and-after pairs and rank-sum (Mann-Whitney U) for two groups, with exact p-values, the typical shift and the effect size.
- Interquartile range calculator: Q1, Q3 and the IQR under four quartile conventions, outlier fences and a box plot.
More tools for working with data
Standard deviation (spread of a measured variable) · Median (the middle value of a list) · Average (mean, median and mode side by side) · Probability (single, combined and conditional events) · Combinations and permutations (the counting behind exact tests) · Percentage (rates and percentage points)