If the ANOVA is significant, at least one pair of groups differs
It is the usual way to read the result. The one-way ANOVA tests whether all group means are equal; a small p-value rejects that; so at least one mean is different from another, and a post hoc test will tell you which pair. You run the ANOVA, it comes back significant, you run Tukey HSD, and the pair appears. That is the whole workflow.
Except it does not always work that way. Take three groups of 8, 15 and 11 with means of 21.4, 24.9 and 25.1 and standard deviations of 3.2, 3.6 and 2.9. The ANOVA Calculator returns F(2, 31) = 3.6248, p = 0.0385. Significant at 0.05. The Tukey HSD Calculator, fed the same three lines, finds 0 of 3 pairs that differ. The two closest come in at p = 0.0542 and 0.0554.
Nobody made a mistake here. The F-test pools the evidence from all three groups into one question. Tukey has to answer a harder one for each pair separately, while keeping the risk of a single false alarm across all pairs at 5%. When the overall result sits just under the line, the pairwise results can sit just over it. A significant ANOVA says the means are not all equal. It does not promise a pair you can name.
That belief is the first of seven below. Six of them do not survive the numbers. One does.
Six beliefs that the numbers do not back
Myth: with several groups, just run a t-test on every pair
It feels thorough. Three groups, three t-tests, done. The problem is arithmetic. Each test at α = 0.05 carries a 5% chance of a false alarm, and the chances add up across tests. If the tests were independent, the probability of at least one false alarm would be 1 - 0.95 raised to the number of pairs:
| Groups | Pairs | Chance of at least one false alarm |
|---|---|---|
| 3 | 3 | 14.3% |
| 4 | 6 | 26.5% |
| 5 | 10 | 40.1% |
| 6 | 15 | 53.7% |
| 10 | 45 | 90.1% |
Pairwise tests on the same groups are not independent, so treat these figures as a guide rather than exact values; the direction does not change. With ten groups you are close to certain to "find" something.
Fact: test the overall question once with ANOVA, then compare pairs with a method that controls the error across all of them. Tukey HSD does that for every pair at once.
Myth: unequal variances? Switch to Kruskal-Wallis
This one sounds like good practice. The NCSS documentation for one-way ANOVA says otherwise in plain words: the Kruskal-Wallis test "assumes that the variances are equal". Its job is non-normal data, not unequal spreads. For normal data with very different spreads, the tool is the Welch version of the ANOVA, which does not pool the variances.
Here is the difference it makes. Three groups: 10 11 9 10 10, then 2 30 15 1 25, then 12 13 11 14 12. The middle group is all over the place. The classic F gives p = 0.6443: nothing to see. The Brown-Forsythe test of equal variances flags the problem with p = 0.0011, and Welch's ANOVA gives F = 7.478, p = 0.0197. The ANOVA Calculator prints all three and adds a warning when the variances look unequal.
Fact: check the spreads first. If the Brown-Forsythe p is under 0.05, or the largest variance is more than four times the smallest (a rule of thumb NCSS quotes), report Welch.
Myth: ANOVA is always the stronger test, ranks throw information away
With clean, normal data, the ANOVA does make better use of the numbers. With one bad value it can fall apart. Three groups of six, and then the same data with a single 14 mistyped as 95:
| Data | One-way ANOVA | Kruskal-Wallis |
|---|---|---|
| Three groups of six, as measured | F = 13, p = 0.0005 | H = 10.7126, p = 0.0047 |
| The same, one 14 typed as 95 | F = 0.9181, p = 0.4206 | H = 8.4941, p = 0.0143 |
One keystroke moved the ANOVA from p = 0.0005 to p = 0.4206 and flipped the verdict. The Kruskal-Wallis Test Calculator moved from 0.0047 to 0.0143 and kept it. Why so different? To the ANOVA, 95 is 81 units away from where 14 used to be, and it multiplies the variance of that group from 2 to 1,095.5. To a rank test, 95 is simply the largest value: rank 18 instead of rank 8.
Fact: the rank test trades a little power on perfect data for a conclusion one wild value cannot rewrite. When an outlier is a typo, fix it. When it is real, a rank test is the honest choice.
Myth: the chi-square p-value for H is fine at any sample size
Most textbooks and spreadsheets read the p-value of the Kruskal-Wallis H from a chi-square distribution with k - 1 degrees of freedom. It is an approximation, and with small groups it can be far off, in either direction. The exact alternative counts every way the observed ranks could be split into groups of the same sizes and asks how many give an H at least as large.
| Data | Chi-square p | Exact p |
|---|---|---|
| 1 2 / 3 4 / 5 6 / 7 8 | 0.0833 | 0.0095 |
| Ranks 1-9 in three groups of three | 0.0273 | 0.0036 |
| Groups of 5, 5 and 4 | 0.0110 | 0.0028 |
| Two groups: 1 3 5 / 2 4 6 | 0.5127 | 0.7000 |
In the first row the shortcut says "not significant at 0.05" and the exact count says "significant at 0.01", almost nine times smaller. In the last row it errs the other way. The Kruskal-Wallis calculator does the exact count whenever the number of splits stays under 300,000, and shows both values side by side.
Scroll further down the same result, past the ranks, to the pairwise table. The exact p is 0.0095, yet the Dunn test finds no pair that differs: the most distant pair, groups 1 and 4, reaches an adjusted p of only 0.0858. It is the opening myth again, this time on ranks. With two values per group, no single pair has enough evidence on its own.
Fact: with fewer than five values in a group, trust the exact p. The calculator warns when the groups are small and the exact count is out of reach.
Myth: Tukey's p-values still need a Bonferroni correction
The habit is understandable: Bonferroni is the correction most people learn first, and multiplying every p by the number of pairs feels safe. Applied to Tukey, it counts the same thing twice. Tukey's p comes from the studentized range distribution, the distribution of the largest gap among all k means. The protection for all pairs is built into that distribution, which is why the critical value climbs with the number of groups: at 30 error degrees of freedom, q is 2.89 for two groups and 4.82 for ten.
With unequal group sizes the calculator switches to the Tukey-Kramer version, and the NIST/SEMATECH e-Handbook notes that the method is then conservative: the real error rate is below α, not above it.
Fact: report Tukey's p as it comes. If you want a Bonferroni, use it instead of Tukey, not on top of it.
Myth: the overall test must agree with the pairwise one
This is the opening belief turned around, and it deserves its own line because it comes back as a complaint: "my ANOVA is significant but Tukey shows nothing, the software is broken." In the example from the start, ANOVA p = 0.0385 and the best Tukey pair p = 0.0542. Both are right. The overall F weighs all the group means together, while each Tukey row has to clear a bar set for the largest of all the gaps.
Fact: when the two disagree near the threshold, report both and say so. That is a result, not an error.
The one belief that is true
With two groups, ANOVA is just a t-test
Run the ANOVA on two groups, 1 2 3 and 4 5 6. It returns F(1, 4) = 13.5 and p = 0.0213. The pooled two-sample t-test on the same data gives t = 3.674, and 3.674 squared is 13.5. The p-value is identical, because an F with one numerator degree of freedom is a squared t. Tukey agrees too: with two groups its critical q is the t value times the square root of 2.
So the folk wisdom holds. If you have two groups, the Two-Sample T-Test Calculator tells you everything the ANOVA would, plus a Welch and a paired version.
All seven at a glance
The table puts every belief next to the single number that decides it.
| Belief | Verdict | The number that settles it |
|---|---|---|
| A significant F means some pair differs | False | F p = 0.0385, closest Tukey pair p = 0.0542 |
| Test every pair at 0.05 | False | 45 pairs: 90.1% chance of a false alarm |
| Unequal variances, so switch to ranks | False | classic p 0.6443, Welch p 0.0197 |
| ANOVA is always the stronger test | False | one typo: 0.0005 to 0.4206, ranks 0.0047 to 0.0143 |
| Chi-square p is fine for H at any size | False | 0.0833 against an exact 0.0095 |
| Tukey p still needs a Bonferroni | False | Tukey p already covers all pairs |
| Two groups: ANOVA is a t-test | True | F = 13.5 = t², same p = 0.0213 |
The rule that survives all seven
Look at the data before choosing the test: the spread of each group, the shape, the outliers, the group sizes. Then let the three tests do three different jobs. The ANOVA, or its Welch version, asks whether anything differs at all. Kruskal-Wallis asks the same on ranks when the data are skewed or ordinal. Tukey asks which pairs differ, and it is allowed to say none. A significant overall test and a pairwise test that finds nothing are not a contradiction; they are two different questions answered honestly.
Tools discussed in this article
- ANOVA Calculator - one-way ANOVA from raw values or from sizes, means and SDs, with the full ANOVA table, η² and ω², Welch's test and a Brown-Forsythe check of equal variances.
- Kruskal-Wallis Test Calculator - the rank-based comparison of 2 to 10 groups with the tie correction, an exact p for small samples and Dunn's pairwise comparisons.
- Tukey HSD Calculator - every pair of group means at once, with the critical q, HSD, simultaneous intervals and the Tukey-Kramer version for unequal groups.