Corrected exercises: one-way analysis of variance (360-301)

The number of pairs is $\dfrac{k(k-1)}{2}$. With 3 groups: $\dfrac{3\times 2}{2}=3$ comparisons. With 5 groups: $\dfrac{5\times 4}{2}=10$ comparisons.

If the five groups came from the same population, each test would have a 95% chance of not rejecting, so the probability of making no mistake in any of the ten is $0.95^{10}\approx 0.5987$. The risk of at least one wrong conclusion is therefore $1-0.5987\approx 0.4013$, that is 40%. We think we are working at a 5% risk and we are actually working at 40%: this is the inflation of the Type I error rate, and it is the reason analysis of variance exists.

NULL hypothesis: $\mu_{A}=\mu_{B}=\mu_{C}$, the three programs have the same mean in the population. ALTERNATIVE hypothesis: at least one mean differs from the others. Note the cautious wording of the alternative: it says neither which one, nor how many, nor in which direction. This is what is called an OMNIBUS test, which answers a global question with a yes or a no.

$\overline{x}_{A}=\dfrac{72+75+78+80+85}{5}=\dfrac{390}{5}=78$. $\overline{x}_{B}=\dfrac{350}{5}=70$. $\overline{x}_{C}=\dfrac{330}{5}=66$. Grand mean: $\dfrac{390+350+330}{15}=\dfrac{1{,}070}{15}\approx 71.333$. Since the groups have equal sizes, the grand mean is also the mean of the three means, which is no longer true when the sizes differ.

We know that the three means are not all equal in the population: at least one pair differs. We do NOT know which pair, how many pairs, or in which direction. Answering those questions takes a second step, post hoc multiple comparisons, with a corrected threshold. And above all, a rejection says nothing about the SIZE of the differences: a measure such as $\eta^{2}$ is needed, which gives the share of the variance in the marks explained by the program.

Deviations of the group means from the grand mean: $78-71.333=6.667$; $70-71.333=-1.333$; $66-71.333=-5.333$. Their squares: $44.444$; $1.778$; $28.444$, with sum $74.667$. Each group having 5 students: $SS_{between}=5\times 74.667\approx 373.33$.

Group A, deviations from 78: $-6$, $-3$, 0, 2, 7; squares 36, 9, 0, 4, 49, sum 98. Group B, deviations from 70: $-5$, $-2$, 0, 2, 5; squares 25, 4, 0, 4, 25, sum 58. Group C, deviations from 66: $-6$, $-2$, 0, 3, 5; squares 36, 4, 0, 9, 25, sum 74. Total: $98+58+74=230$.

$373.33+230=603.33$. Direct check: computing the fifteen deviations from the grand mean 71.333 and adding their squares, we do get 603.33. The equality $SS_{total}=SS_{between}+SS_{within}$ is exact, not approximate: it is a theorem, and it holds whatever the group sizes.

Degrees of freedom: between, $k-1=3-1=2$; within, $N-k=15-3=12$; total, $N-1=15-1=14$. We check that $2+12=14$. The degrees of freedom therefore split exactly like the sums of squares, which is the second half of the theorem.

The sum of squares BETWEEN measures what the groups have that is DIFFERENT: it is zero if the three means coincide and grows as they move apart. The sum of squares WITHIN measures the RESIDUAL variability, the variability that remains between students of the same program, and which serves as the reference: it is the noise against which the signal will be judged.

Mean square BETWEEN: $\dfrac{373.33}{2}\approx 186.67$. Mean square WITHIN: $\dfrac{230}{12}\approx 19.167$. Table: source between, $SS=373.33$, $df=2$, $MS=186.67$; source within, $SS=230.00$, $df=12$, $MS=19.167$; total, $SS=603.33$, $df=14$. Careful: the mean squares do NOT add up, unlike the sums of squares and the degrees of freedom.

$F=\dfrac{186.67}{19.167}\approx 9.739$. Since $9.739>6.927$, we reject the hypothesis of equal means at the 1% level, and a fortiori at the 5% level. The p-value is below 0.01; software would give about 0.003. The conclusion is written: the three programs do not have the same mean, $F(2, 12)=9.74$, p < 0.01.

Because the numerator can only GROW when the means move apart, never shrink: the sum of squares between is a sum of squares, hence positive or zero. An $F$ close to zero means that the group means resemble each other more than chance would make them, which is no argument against the null hypothesis. The whole rejection region therefore lies on the right, and there is no two-tailed version of the test.

If the three means were identical, the sum of squares between would be zero and $F=0$. If the within-group variability were zero, every student in a program having exactly the same mark, the denominator would be zero and $F$ would tend to infinity: any gap between groups would then be infinitely significant, which is consistent, since without internal noise the slightest gap is certain.

With two groups, $F=t^{2}$, exactly. The F test and the two-sided t-test then give strictly the same decision, and the critical values correspond: $1.96^{2}=3.84$, which is indeed the critical value of $F$ with 1 and infinitely many degrees of freedom. Analysis of variance is therefore a GENERALIZATION of the t-test, not a competing tool.

$\eta^{2}=\dfrac{373.33}{603.33}\approx 0.6188$. The program therefore explains about 61.9% of the total variation in the marks, which is considerable. In other words, knowing a student's program already accounts for three fifths of the spread in the results; only 38% of the variation remains to be explained by everything else.

$\eta^{2}$ ranges from 0 to 1. It is 0 when all the group means coincide, so when the factor explains nothing. It is 1 when the within-group variability is zero, that is, when knowing the group lets you predict the mark exactly. These two bounds are the same as those of the coefficient of determination of a regression, and this is no accident: $\eta^{2}$ IS a coefficient of determination, that of predicting a mark by the mean of its group.

At 0.619, we are far beyond the 0.14 threshold that marks a large effect: the effect here is massive. It must be tempered, however: these data have five students per group, chosen for the exercise, and such ratios are rare in real social science data, where an $\eta^{2}$ of 0.10 is already a fine result.

With 15 students per group instead of 5, all the sums of squares are multiplied by 3: $SS_{between}=1{,}120$, $SS_{within}=690$. The degrees of freedom become 2 and 42, so $MS_{between}=560$ and $MS_{within}\approx 16.43$, whence $F\approx 34.09$: $F$ has more than tripled. On the other hand $\eta^{2}=\dfrac{1120}{1810}\approx 0.6188$, strictly UNCHANGED. This is exactly the contrast of the previous chapter between significance and strength.

$\eta^{2}$ is BIASED UPWARD on small samples: even under the null hypothesis, the group means never coincide perfectly, so $SS_{between}$ is strictly positive and so is $\eta^{2}$. Its expected value under the null hypothesis is about $\dfrac{k-1}{N-1}$, here $\dfrac{2}{14}\approx 0.143$, which is not negligible. The usual correction is omega squared, $\omega^{2}=\dfrac{SS_{between}-\left(k-1\right)MS_{within}}{SS_{total}+MS_{within}}$, which here equals $\dfrac{373.33-2\times 19.167}{603.33+19.167}=\dfrac{335.0}{622.5}\approx 0.538$, eight points lower.

First assumption, INDEPENDENCE of the observations: each student appears in one group only, and the results are not linked to each other. Second assumption, NORMALITY of the variable in each population. Third assumption, HOMOGENEITY of the variances, called homoscedasticity: the $k$ populations have the same variance.

Variances: $4.95^{2}=24.50$; $3.81^{2}=14.52$; $4.30^{2}=18.49$. Ratio of the extremes: $\dfrac{24.50}{14.52}\approx 1.688$. The most widely used rule of thumb tolerates a ratio of up to 4 when the sample sizes are equal; with 1.69, homogeneity is amply acceptable. Levene's test, which formalizes the check, would certainly not reject.

NORMALITY is the most forgiving, for two reasons. First, the test is about group MEANS, whose sampling distribution tends toward the normal by the central limit theorem even if the raw variable is not normal. Second, $F$ is known to be robust to moderate departures from normality, especially when the sample sizes are equal. Marked skewness on very small groups remains a problem, however, and in that case we look at the residuals rather than at the raw variable.

INDEPENDENCE is never forgiving. Example of a violation: measuring the same students three times under three conditions and treating the fifteen measurements as fifteen independent observations. The variability between individuals is then counted in the denominator when it should be removed, exactly as in the paired case of the previous chapter, and the conclusion can be wrong in either direction. Another frequent violation: students from the same class, who resemble each other, treated as an independent draw.

Three routes. If only homogeneity fails, use WELCH's analysis of variance, which adjusts the degrees of freedom the way Welch's t-test does. If normality fails badly on small groups, switch to the nonparametric KRUSKAL-WALLIS test, which works on ranks. If independence fails because the same subjects are measured several times, the model must change: REPEATED-MEASURES analysis of variance, which isolates the variability between subjects exactly as pairing did for two groups.

$SE=\sqrt{19.167\left(\dfrac{1}{5}+\dfrac{1}{5}\right)}=\sqrt{19.167\times 0.4}=\sqrt{7.6667}\approx 2.7689$. This standard error is the same for the three pairs, since the sample sizes are equal and the common variance is estimated on the whole.

A minus C: $\dfrac{78-66}{2.7689}=\dfrac{12}{2.7689}\approx 4.334$. A minus B: $\dfrac{8}{2.7689}\approx 2.889$. B minus C: $\dfrac{4}{2.7689}\approx 1.445$.

With the corrected critical value of 2.779: A against C, $4.334>2.779$, SIGNIFICANT difference. A against B, $2.889>2.779$, also significant, but only just. B against C, $1.445<2.779$, NOT significant. We conclude that program A stands apart from the other two, and that B and C cannot be told apart. Without the correction, the critical value would have been 2.179: A against B would have passed with room to spare instead of only just, and B against C would still not have been significant. This shows the real effect of the correction.

The minimum detectable difference is $2.779\times 2.7689\approx 7.694$ points. Any difference between means below 7.7 points will be declared not significant in this study, whatever the pair. This is very useful information to publish: it says right away that the study, with five students per group, is blind to any gap of less than eight points.

Because $MS_{within}$ estimates the common variance on all FIFTEEN observations, hence with 12 degrees of freedom, whereas a single pair would provide only 8. The estimate is more stable, the critical value lower, and the test more powerful. This is one of the advantages of analysis of variance: it pools the information from all the groups to estimate the noise, even for comparisons that do not directly involve them. The price of this gain is that it assumes homogeneity of the variances, an assumption that therefore had to be checked beforehand.

Grand mean: $\dfrac{40\times 74.0+25\times 70.4+15\times 66.0}{80}=\dfrac{2{,}960+1{,}760+990}{80}=\dfrac{5{,}710}{80}=71.375$. This is not the mean of the three means, which would be $\dfrac{74.0+70.4+66.0}{3}\approx 70.133$, because the grand mean WEIGHTS each group by its size: the group of 40 students counts two and a half times as much as the group of 15. The gap of 1.24 points between the two calculations is exactly the kind of error that throws off a whole ANOVA table.

Deviations from 71.375: $2.625$; $-0.975$; $-5.375$. $SS_{between}=40\times 2.625^{2}+25\times 0.975^{2}+15\times 5.375^{2}=40\times 6.8906+25\times 0.950625+15\times 28.8906=275.625+23.7656+433.359\approx 732.75$.

Degrees of freedom: between $3-1=2$; within $80-3=77$; total 79. Mean squares: $\dfrac{732.75}{2}=366.375$ and $\dfrac{5{,}940}{77}\approx 77.143$. $F=\dfrac{366.375}{77.143}\approx 4.749$. Since $4.749>3.12$, we reject: the three programs do not have the same mean, $F\left(2, 77\right)=4.75$, p < 0.05.

$SS_{total}=732.75+5{,}940=6{,}672.75$, so $\eta^{2}=\dfrac{732.75}{6{,}672.75}\approx 0.1098$. The program therefore explains only 11% of the variation in the marks, which is a medium to large effect by the benchmarks but leaves 89% of the spread to be explained by something else. The contrast with the example of five students per group is instructive: there, $F$ was 9.74 for an $\eta^{2}$ of 0.62; here, $F$ is 4.75 for an $\eta^{2}$ six times smaller. It is the total of 80 students that makes a far more modest effect significant.

First trap, the GRAND MEAN: it must be weighted by the sample sizes, and taking the mean of the means throws off every sum of squares that follows from it. Second trap, SENSITIVITY TO THE ASSUMPTIONS: when the sample sizes are equal, analysis of variance tolerates unequal variances well; when they are unequal, it becomes seriously misleading, in one direction or the other depending on whether the large group carries the large variance. With unbalanced sample sizes, checking homogeneity stops being a formality, and Welch's version becomes the cautious choice.

FALSE. The test is OMNIBUS: it indicates that at least one pair of means differs, without saying which or how many. In the example of the chapter, A stands apart from B and from C, but B and C cannot be told apart. Correct statement: at least one mean differs from the others; post hoc multiple comparisons, with a corrected threshold, identify which.

FALSE. Three t-tests at the 5% level bring the overall risk to about $1-0.95^{3}\approx 0.143$, that is 14% instead of 5%, and the problem quickly gets worse: ten comparisons give 40%. Moreover, each t-test would use only two groups to estimate the residual variance, with 8 degrees of freedom instead of 12, hence less power. Correct statement: analysis of variance controls the overall risk and estimates the noise on all the groups at once.

FALSE. The SUMS OF SQUARES add up, and so do the DEGREES OF FREEDOM, but the mean squares are quotients and do not add up. Here, $186.67+19.17=205.83$, whereas the total mean square is $\dfrac{603.33}{14}\approx 43.1$. Correct statement: only the sums of squares and the degrees of freedom split additively.

FALSE. $F$ depends on the sample size as much as on the size of the gap: in the example of the chapter, tripling the sample sizes takes $F$ from 9.74 to 34.09 although nothing has changed in the data. The strength measure is $\eta^{2}$, which stayed at 0.619. Correct statement: the effect is very clearly established; its strength is read from $\eta^{2}$, which is 0.62.

FALSE. The three measurements are on the same people: they are not independent, and the first assumption of analysis of variance is violated. The variability between individuals, which should be removed, ends up in the denominator and drowns the signal. Correct statement: a REPEATED-MEASURES analysis of variance is needed, which isolates the variability between subjects, exactly as the paired t-test does for two conditions.

$N=18+20+22=60$. Grand mean: $\dfrac{18\times 68.5+20\times 74.1+22\times 71.8}{60}=\dfrac{1{,}233+1{,}482+1{,}579.6}{60}=\dfrac{4{,}294.6}{60}\approx 71.577$.

Deviations from 71.577: $-3.077$; $2.523$; $0.223$. $SS_{between}=18\times 9.4661+20\times 6.3665+22\times 0.0498=170.39+127.33+1.10\approx 298.82$.

$SS_{1}=17\times 9.2^{2}=17\times 84.64=1{,}438.88$. $SS_{2}=19\times 8.7^{2}=19\times 75.69=1{,}438.11$. $SS_{3}=21\times 9.6^{2}=21\times 92.16=1{,}935.36$. Total: $SS_{within}\approx 4{,}812.35$. This is the formula to remember: it lets you rebuild an analysis of variance from published statistics alone, without the raw data.

Degrees of freedom: 2 and $60-3=57$. Mean squares: $\dfrac{298.82}{2}\approx 149.41$ and $\dfrac{4{,}812.35}{57}\approx 84.427$. $F=\dfrac{149.41}{84.427}\approx 1.770$. Since $1.770<3.16$, we DO NOT REJECT: the data do not allow us to conclude that the three methods differ at the 5% level.

$SS_{total}=298.82+4{,}812.35=5{,}111.17$, so $\eta^{2}=\dfrac{298.82}{5{,}111.17}\approx 0.0585$, which is close to a MEDIUM effect against the benchmarks (0.06). Conclusion: “The means of the three methods are 68.5, 74.1 and 71.8. The analysis of variance does not allow us to conclude that they differ, $F\left(2, 57\right)=1.77$; p > 0.05. The share of the variance associated with the method, $\eta^{2}=0.059$, nevertheless comes close to a medium effect, and the gap of 5.6 points between methods 1 and 2 remains appreciable: with 20 students per group, the study did not have the power to detect it. A replication with about 50 students per group would be needed before concluding that the methods are equivalent.” This shows the whole value of reporting the effect size even when the test does not reject: it turns a null result into an informative one.

The factor has $df=k-1=4$, so $k=5$ regions. The residual degrees of freedom are $N-k=295$, so $N=295+5=300$ respondents. Reconstructing these two numbers from the degrees of freedom alone is the first reflex when reading an ANOVA table.

Mean square of the factor: $\dfrac{1{,}842}{4}=460.5$. Residual mean square: $\dfrac{27{,}630}{295}\approx 93.66$. $F=\dfrac{460.5}{93.66}\approx 4.917$. Since $4.917>3.38$, the result is indeed significant at the 1% level; the p-value is about 0.0008, which justifies the “p < 0.001” announced.

$SS_{total}=1{,}842+27{,}630=29{,}472$, so $\eta^{2}=\dfrac{1{,}842}{29{,}472}\approx 0.0625$. Region therefore explains 6.25% of the variation in the sense of belonging, and 93.75% escapes it. The expression “highly significant” is accurate in the technical sense and misleading in the ordinary sense: it describes how solid the detection is, not how large the phenomenon is. Speaking of “the importance of territorial roots” on the basis of one sixteenth of the variance is an extrapolation the table does not support.

No, not in the causal sense. Region was not randomly assigned: the inhabitants of a region differ in age, schooling, origin and rurality, all variables related to the sense of belonging. One-way analysis of variance controls none of this, and the word “effect”, which is a technical term for the factor in the model, reads in the text as a claim of causation. This is an ambiguity of vocabulary that careful writers avoid by writing “difference associated with region”.

“The sense of belonging differs significantly among the five regions, $F\left(4, 295\right)=4.92$; p < 0.001. Region nevertheless accounts for only 6.3% of the total variation, $\eta^{2}=0.063$, which leaves most of the individual differences unexplained. Since region was not assigned, this association cannot be interpreted as a causal effect, and age, schooling and place of birth would have to be controlled before concluding anything about territorial roots.” Three corrections: the effect size is reported, the word effect disappears, and the missing control variables are named.