Advanced Quantitative Methods 360-301 • CEGEP, Montreal

Revision sheet: one-way analysis of variance (360-301)

Q: Why not run several t-tests instead of an analysis of variance?

Because each test carries its own five percent risk. Three groups give three comparisons and an overall risk of more than fourteen percent, five groups give ten comparisons and a risk of forty percent. Analysis of variance tests everything at once at the announced level, and only after a rejection do you compare the pairs, with a corrected threshold.

Q: What does a significant F mean in analysis of variance?

That at least one of the group means differs from the others, without the test saying which one. That is why it is called omnibus. To identify the groups involved, you then need post hoc comparisons, run with the standard error taken from the within-groups mean square and a corrected threshold.

Q: How do you compute the degrees of freedom of an analysis of variance?

The factor has the number of groups minus one, the residual has the total number of observations minus the number of groups, and the total has the number of observations minus one. For three groups of ten, that gives two, twenty-seven and twenty-nine. The first two must always add up to the third.

Q: What is the correlation ratio eta squared?

It is the share of the total variance explained by the factor, that is, the between-groups sum of squares divided by the total sum of squares. The usual benchmarks are one hundredth for a small effect, six hundredths for a medium effect and fourteen hundredths for a large one. Unlike the p-value, it does not depend on the sample size.

Q: Can you run an analysis of variance on the same people measured several times?

Not with the one-way kind, which assumes independent observations. Repeated measures on the same subjects are correlated with each other, and counting them as independent inflates the residual degrees of freedom and manufactures significant results. You need a repeated-measures analysis of variance.

This sheet covers the central chapter of advanced quantitative methods: one-way analysis of variance. It does not repeat the formulas, it gives the eight places where a correct table leads to a wrong conclusion, starting with the most frequent one, believing that F points to the group that stands out.

It is written for Social Science CEGEP students taking course 360-301. The corrected exercise set on the same chapter goes through every step, from the partition of the variation to reading a table published in a journal.

Mark this sheet as read or add it to your favourites: a free account, no password, keeps your read sheets and favourites from one visit to the next, tells you which chapter to tackle next and lets you ask for the method or exercise you are missing. Create your space, an email is enough.

The thread of the chapter

Analysis of variance compares MEANS by comparing VARIANCES: the spread between the groups against the spread inside them. The test is OMNIBUS, it says that at least one mean differs and never which one.

This chapter is part of 360-301, Advanced Quantitative Methods

The essentials

Two sources of variation, and a single question

• $SS_{between}=\sum n_{j}\left(\overline{x}_{j}-\overline{x}\right)^{2}$ : what separates the group means.
• $SS_{within}=\sum \left(n_{j}-1\right)s_{j}^{2}$ : what spreads the individuals inside their group.
• $SS_{total}=SS_{between}+SS_{within}$ , exactly: this is the partition that gives the method its name.
• $H_{0}$ : all the means are equal. $H_{1}$ : at least one differs. The test is OMNIBUS and does not say which one.

The three short bars are the group means and the dashed line is the grand mean: the between part measures the gap of the bars from the dashed line, the within part that of the points from their own bar.

Analysis of variance does not compare variances for their own sake: it compares means, by measuring whether they move apart more than the internal noise can explain.

The table, and what adds up inside it

• Degrees of freedom: $k-1$ for the factor, $N-k$ for the residual, $N-1$ in total. They ADD UP, like the sums of squares.
• $MS=\dfrac{SS}{df}$ : the mean squares, on the other hand, do NOT add up. There is no useful “total MS” row.
• $F=\dfrac{MS_{between}}{MS_{within}}$ , and the test is always RIGHT-TAILED: only a large $F$ contradicts $H_{0}$ .
• With two groups, $F=t^{2}$ : analysis of variance generalizes the t-test exactly.

The first two columns add up row by row, the fourth does not:

203.3+42.4

is worth nothing at all.

Under $H_{0}$ , the two mean squares estimate the same variance, so $F$ hovers around $1$ . An $F$ below $1$ is never significant.

Strength, and the assumptions

• $\eta^{2}=\dfrac{SS_{between}}{SS_{total}}$ : the share of the variance explained by the factor. Benchmarks: $0.01$ small, $0.06$ medium, $0.14$ large.
• $\eta^{2}$ is biased upward on small samples; $\omega^{2}=\dfrac{SS_{between}-\left(k-1\right)MS_{within}}{SS_{total}+MS_{within}}$ corrects this bias.
• Assumptions: INDEPENDENCE, never negotiable; normality, forgiving; homogeneity of the variances, forgiving if the sample sizes are equal, up to a ratio of $4$ between the extremes.
• Unequal sample sizes: the grand mean must be WEIGHTED by the sample sizes, and the assumptions stop being forgiving.

Same means, hence the same between variation. On the left, the groups are tight: the factor explains almost everything,

\eta^{2}

is large. On the right, the within spread drowns the gap.

Repeated measures on the same subjects do NOT fall under one-way analysis of variance: a repeated-measures analysis is needed.

The mistakes that cost marks

These are the errors I correct most often in session. Each one costs marks on a paper, even when the reasoning behind it is right.

1. Running three t-tests instead of an analysis of variance

2 marks, and the real risk is no longer the one declared

What not to write

“Three groups, so I compare A with B, B with C and A with C, each at the 5% level.”

What to write

“Three comparisons at the 5% level push the overall risk up to $1-0.95^{3}\approx 14.3$ %. I start with an analysis of variance, which tests everything at once at the announced level.”

With five groups, ten comparisons are enough to push the risk of at least one wrong conclusion beyond four chances in ten.

Why: The number of pairwise comparisons is $\dfrac{k(k-1)}{2}$ : it explodes with $k$ , and each comparison adds its own chance of a false positive.

2. Believing that F points to the group that differs

2 marks, and the conclusion goes beyond what the test allows

What not to write

“ $F=4.79$ is significant: method C is therefore better than the other two.”

What to write

“The test is OMNIBUS: it establishes that at least one mean differs. To find out which one, post hoc comparisons with a corrected threshold are needed.”

Why: $F$ lumps the three gaps into a single number: it keeps no information about how they are distributed. A rejection can come from one group far out as well as from three medium gaps.

3. Adding up the mean squares

1 mark, and a wrong F if the error carries through

What not to write

“ $MS_{between}+MS_{within}=203.3+42.4=245.7$ , that is the total mean square.”

What to write

“The SUMS OF SQUARES add up and so do the DEGREES OF FREEDOM, but the mean squares are quotients: $MS_{total}=\dfrac{1551.9}{29}=53.5$ , which is not the sum of the two.”

Why: An average of averages is not the average of the whole, unless the denominators are equal. Here they are $2$ and $27$ : nothing adds up.

4. Confusing eta squared with the p-value

2 marks, and the question on strength

What not to write

“ $p<0.01$ : the factor therefore explains 99% of the variance.”

What to write

“The $p$ -value says nothing about the share explained. That share is $\eta^{2}=\dfrac{406.7}{1551.9}\approx 0.26$ , that is 26% of the total variance.”

Why: $p$ depends on the sample size, $\eta^{2}$ does not. These are the same two roles as in the t-test: the evidence on one side, the size on the other.

5. Computing the grand mean without weighting by the sample sizes

2 marks, with the whole

SS_{between}

thrown off

What not to write

“The three means are $68$ , $72$ and $77$ , so the grand mean is $72.33$ .”

What to write

“That is true only if the three groups have the same size. With unequal sample sizes, $\overline{x}=\dfrac{\sum n_{j}\overline{x}_{j}}{N}$ .”

Why: The grand mean is the centre around which the gap of the groups is measured. Off-centre, it inflates or crushes the between part without anything else giving it away.

6. Looking for a two-tailed threshold for the F statistic

1 mark, and a threshold read in the wrong place in the table

What not to write

“I split the 5% risk between the two sides of the F distribution.”

What to write

“The test is RIGHT-TAILED: only a large between variation compared with the within variation contradicts $H_{0}$ . The whole 5% risk is in the right tail.”

The whole

5\%

risk lies to the right of

3.35

, in green. A small value of

F

, on the left in red, says that the means resemble each other: nothing to reject.

Why: A small $F$ means that the means resemble each other even more than chance would have made them: that is compatible with $H_{0}$ , never against it.

7. Treating repeated measures as a single factor

the whole question, since the test does not apply

What not to write

“I have the same thirty students measured at three times: I run a one-way analysis of variance with $N=90$ .”

What to write

“The three series are on the SAME subjects: this is a repeated-measures analysis. Independence, a non-negotiable assumption, is not met here.”

On the right, each line joins the three measurements of one subject: they are not independent, unlike the three groups on the left.

Why: Counting $90$ independent observations when there are thirty people triples the residual degrees of freedom artificially, so it crushes the standard error and manufactures significant results.

8. Comparing the groups two at a time at the ordinary level after the rejection

1.5 marks, and the very problem that analysis of variance had avoided comes back through the back door

What not to write

“ $F$ is significant, so now I compare the pairs at 5% each.”

What to write

“I use the standard error from $MS_{within}$ and a CORRECTED threshold, for example $\alpha'=\dfrac{0.05}{3}=0.0167$ with Bonferroni, or Tukey's test.”

Why: Running the analysis of variance and then three uncorrected tests amounts to running three uncorrected tests. The correction is what protects the overall level.

Which method to choose

Analysis of variance, and which one?

Count the groups, check whether they are the same individuals, then check independence.

If only two groups → t-test; the analysis of variance would give $F=t^{2}$ and the same decision

Example: workshop against control
If three groups or more, DIFFERENT individuals in each → ONE-WAY analysis of variance

Example: three teaching methods, three separate classes
If the same individuals measured at several times or under several conditions → REPEATED-MEASURES analysis of variance, never the one-way kind

Example: the same students assessed three times
If two crossed factors in the same design → TWO-WAY analysis of variance, with an interaction term

Example: method and gender crossed
If the assumptions fail badly, or the samples are very small → KRUSKAL-WALLIS test, the equivalent on ranks

Example: three groups of six, very skewed distributions

Independence is the only assumption that cannot be repaired: if it is missing, no test in this family is suitable.

What to do after rejecting?

Look at what the question still asks for, and how many comparisons will be needed.

If the question asks WHICH means differ → post hoc comparisons with $SE=\sqrt{MS_{within}\left(\dfrac{1}{n_{1}}+\dfrac{1}{n_{2}}\right)}$ and a corrected threshold

Example: three pairs, Bonferroni threshold at $0.0167$
If all the pairs must be compared and the sample sizes are equal → TUKEY's test, more powerful than Bonferroni in this particular case

Example: four groups of twenty
If only a few comparisons were planned in advance → PLANNED contrasts, less costly in power

Example: each new method against the classical method
If the question asks for the strength of the effect → $\eta^{2}$ , or $\omega^{2}$ if the sample is small

Example: $\eta^{2}=0.26$ , large effect

If the analysis of variance does not reject, you stop: looking for a significant pair anyway is the very definition of fishing for results.

How the answer is expected to be written

A marker ticks steps. Here they are in order, with the concluding sentence expected word for word.

Writing up an analysis of variance table

When to use it: As soon as the question gives three means or more, with their sample sizes and standard deviations.

1 Write $H_{0}$ and $H_{1}$ in words, naming the groups of the question.
2 Compute the WEIGHTED grand mean, then $SS_{between}$ and $SS_{within}$ , and check their sum.
3 Set out the degrees of freedom $k-1$ and $N-k$ , and check that they add up to $N-1$ .
4 Divide to get the mean squares, then $F$ , and read the critical value at $(k-1, N-k)$ degrees of freedom.
5 Decide, then give $\eta^{2}$ and announce the post hoc comparisons if you reject.

Concluding sentence

“At the 5% level, $F=4.79$ with $(2, 27)$ degrees of freedom exceeds the critical value $3.35$ : we reject the hypothesis that the three means are equal. The factor explains $\eta^{2}\approx 26$ % of the total variance.”

The trap: Forgetting to check that $SS_{between}+SS_{within}=SS_{total}$ : this is the check that catches almost every calculation error in the table.

Marking: 1 mark for the hypotheses, 2 marks for the sums of squares, 1 mark for the degrees of freedom, 1 mark for $F$ , 1 mark for the decision, 1 mark for $\eta^{2}$ .

Carrying out the post hoc comparisons

When to use it: After a rejection, when the question asks which groups differ.

1 Count the comparisons: $\dfrac{k(k-1)}{2}$ , and write it down.
2 Compute the common standard error from $MS_{within}$ , and not from the variances of each pair.
3 Compute one $t$ per pair, with the RESIDUAL degrees of freedom $N-k$ .
4 Compare each $t$ with the corrected threshold, Bonferroni or Tukey, saying which one you use.
5 Conclude pair by pair, naming the groups.

Concluding sentence

“With three comparisons, the Bonferroni threshold is $\dfrac{0.05}{3}=0.0167$ , a critical value of $2.55$ with $27$ degrees of freedom: only the difference between the first and the third method, $t=3.09$ , is significant.”

The trap: Recomputing a standard error specific to each pair: you lose the power brought by the whole set of observations, whereas $MS_{within}$ uses them all.

Marking: 1 mark for the number of comparisons, 1 mark for the common standard error, 1 mark per pair correctly concluded.

Check before you hand in

Five minutes of checking recover more marks than one more problem started in a hurry.

The sums of squares add up

Add $SS_{between}$ and $SS_{within}$ : you must get $SS_{total}$ to within rounding.

$406.67+1145.25=1551.92$ : the partition is right (the table rounds the total to 1551.9), with no error in the grand mean.
The degrees of freedom too

$(k-1)+(N-k)$ must equal $N-1$ . It is an instant check of the number of groups and the number of observations.

$2+27=29$ for three groups of ten: correct.
F hovers around 1 under the null hypothesis

An $F$ below $1$ can never be significant: no need to open the table.

An $F$ of $0.4$ announced as significant signals that the numerator and the denominator have been swapped.
Eta squared stays between 0 and 1

$\eta^{2}$ is a part of a sum divided by that same sum: beyond $1$ , one of the two sums of squares is wrong.

An $\eta^{2}$ of $1.4$ comes from an $SS_{total}$ computed without the within part.

The typical problem, taken apart

Three teaching methods, from the table to the comparisons

Three groups of ten students follow three different methods. Means: $68$ , $72$ and $77$ . Standard deviations: $6$ , $7$ and $6.5$ .

The question asks whether the three methods give different results, how strong the effect is, and which ones really differ.

The three methods, mean plus or minus one standard deviation: the intervals overlap widely, and only the test will tell whether the means differ more than the noise.

Step 1

Grand mean: the sample sizes being equal, $\overline{x}=\dfrac{68+72+77}{3}\approx 72.33$ .

Why

We note explicitly WHY the simple mean is fine here: with unequal sample sizes it would have had to be weighted, and that is where the most marks are lost.

Step 2

$SS_{between}=10\left[(68-72.33)^{2}+(72-72.33)^{2}+(77-72.33)^{2}\right]\approx 406.7$ .

Why

Each gap is multiplied by the size of the group: a large group that moves away weighs more than a small group that moves away as much.

Step 3

$SS_{within}=9(36)+9(49)+9(42.25)=1145.25$ , so $SS_{total}\approx 1551.9$ .

Why

We go through $\left(n_{j}-1\right)s_{j}^{2}$ so that we never need the raw data: this is what makes the exercise doable from a simple summary.

Step 4

$MS_{between}=\dfrac{406.7}{2}=203.3$ and $MS_{within}=\dfrac{1145.25}{27}=42.4$ , so $F=\dfrac{203.3}{42.4}\approx 4.79$ . Critical value $F(2, 27)$ at 5%: $3.35$ . We reject.

Why

The two mean squares estimate the same variance under $H_{0}$ : a ratio of nearly five is hard to put down to chance.

Step 5

$\eta^{2}=\dfrac{406.7}{1551.9}\approx 0.26$ , and $\omega^{2}=\dfrac{406.7-2(42.4)}{1551.9+42.4}\approx 0.20$ .

Why

Both exceed the benchmark of $0.14$ , so the effect is large. The gap between the two quantifies the optimistic bias of $\eta^{2}$ on a sample of thirty.

Step 6

$SE=\sqrt{42.4\left(\dfrac{1}{10}+\dfrac{1}{10}\right)}\approx 2.91$ ; the three $t$ values are $1.37$ , $1.72$ and $3.09$ . Bonferroni threshold at $27$ degrees of freedom: $2.55$ .

Why

Only the comparison between the first and the third method exceeds the corrected threshold. The overall rejection therefore came from a single pair, which $F$ did not say.

The conclusion, written out

“At the 5% level, $F=4.79$ with $(2, 27)$ degrees of freedom exceeds $3.35$ : the three methods do not give the same results. The factor explains about $26$ % of the variance, and the post hoc comparisons at the Bonferroni threshold distinguish only the first method from the third.”

The classic mistake on this problem: Concluding that the third method is better than the other two: the gap with the second, $t=1.72$ , does not even exceed the uncorrected threshold.

Learn by heart

• $SS_{total}=SS_{between}+SS_{within}$ , and the degrees of freedom add up too. The MEAN SQUARES, never.
• $F=\dfrac{MS_{between}}{MS_{within}}$ , RIGHT-TAILED test. Under $H_{0}$ , $F$ hovers around $1$ .
• With two groups, $F=t^{2}$ : the same decision by another route.
• The test is OMNIBUS: at least one mean differs, and it never says which.
• $\eta^{2}=\dfrac{SS_{between}}{SS_{total}}$ : $0.01$ small, $0.06$ medium, $0.14$ large, and it does not depend on $n$ .
• Post hoc comparisons: standard error taken from $MS_{within}$ , and a CORRECTED threshold, Bonferroni or Tukey.
• Unequal sample sizes: the grand mean is weighted, and the assumptions stop being forgiving.

Frequently asked questions

Why not run several t-tests instead of an analysis of variance?

Because each test carries its own five percent risk. Three groups give three comparisons and an overall risk of more than fourteen percent, five groups give ten comparisons and a risk of forty percent. Analysis of variance tests everything at once at the announced level, and only after a rejection do you compare the pairs, with a corrected threshold.

What does a significant F mean in analysis of variance?

That at least one of the group means differs from the others, without the test saying which one. That is why it is called omnibus. To identify the groups involved, you then need post hoc comparisons, run with the standard error taken from the within-groups mean square and a corrected threshold.

How do you compute the degrees of freedom of an analysis of variance?

The factor has the number of groups minus one, the residual has the total number of observations minus the number of groups, and the total has the number of observations minus one. For three groups of ten, that gives two, twenty-seven and twenty-nine. The first two must always add up to the third.

What is the correlation ratio eta squared?

It is the share of the total variance explained by the factor, that is, the between-groups sum of squares divided by the total sum of squares. The usual benchmarks are one hundredth for a small effect, six hundredths for a medium effect and fourteen hundredths for a large one. Unlike the p-value, it does not depend on the sample size.

Can you run an analysis of variance on the same people measured several times?

Not with the one-way kind, which assumes independent observations. Repeated measures on the same subjects are correlated with each other, and counting them as independent inflates the residual degrees of freedom and manufactures significant results. You need a repeated-measures analysis of variance.

Practise it

Corrected exercises: One-way analysis of variance, 360-301

A method is proved on a paper, not on a sheet. The set for the same chapter takes each of these traps into a problem, with the solution written out step by step.

10 corrected exercises
100 points
150 minutes

Do the exercises

Something missing from this sheet? A method, a trap, an exercise to practise it, a mistake you spotted: tell me, I read every request.

Sending goes through your free space: that is how I can answer you and tell you when the exercise is added. An email is enough, no password. Create your space. Rather not create an account? Write to me directly.

Previous sheet Comparing two groups: t-test and effect size Next sheet Multiple regression

© Ahmed Squalli Houssaini. Revision sheet published at www.letuteurscientifique.ca/en/fiches/cegep-aqm-analysis-of-variance. Free for personal and classroom use; republishing it elsewhere requires written permission (legal notice).

Looking for a 360-301 tutor in Montreal?

Get in touch for a first session. 360-301 arrives at the time of the end-of-program research project, and it is the course where a mistake of method costs the whole analysis chapter.

Book a session WhatsApp

Site by Studio Squalli

Revision sheet: one-way analysis of variance (360-301)

The essentials

Two sources of variation, and a single question

The table, and what adds up inside it

Strength, and the assumptions

The mistakes that cost marks

1. Running three t-tests instead of an analysis of variance

2. Believing that F points to the group that differs

3. Adding up the mean squares

4. Confusing eta squared with the p-value

5. Computing the grand mean without weighting by the sample sizes

6. Looking for a two-tailed threshold for the F statistic

7. Treating repeated measures as a single factor

8. Comparing the groups two at a time at the ordinary level after the rejection

Which method to choose

Analysis of variance, and which one?

What to do after rejecting?

How the answer is expected to be written

Writing up an analysis of variance table

Carrying out the post hoc comparisons

Check before you hand in

The typical problem, taken apart

Three teaching methods, from the table to the comparisons

Learn by heart

Frequently asked questions

Why not run several t-tests instead of an analysis of variance?

What does a significant F mean in analysis of variance?

How do you compute the degrees of freedom of an analysis of variance?

What is the correlation ratio eta squared?

Can you run an analysis of variance on the same people measured several times?

Corrected exercises: One-way analysis of variance, 360-301

See also

Looking for a 360-301 tutor in Montreal?