Advanced Quantitative Methods 360-301 • CEGEP, Montreal

Revision sheet: comparing two groups, t-test and effect size (360-301)

Q: When should you use a paired t-test rather than an independent one?

Whenever the two series are on the same individuals, or on pairs formed by the researcher. You then work on the individual differences, with a number of degrees of freedom equal to the number of pairs minus one. Pairing removes the variability between people, which makes the test far more sensitive on before-and-after measurements.

Q: What does a p-value of 0.03 really mean?

That if the two groups were truly identical, we would observe a gap at least this large in three percent of possible samples. It is not the probability that the null hypothesis is true, nor the probability of being wrong in rejecting it: the p-value is computed by assuming the null hypothesis is true.

Q: How do you compute Cohen's d?

You divide the difference between the two means by the pooled standard deviation of the two groups, never by the standard error. A difference of six points with a pooled standard deviation of ten gives a d of 0.60. The usual benchmarks are 0.2 for a small effect, 0.5 for a medium effect and 0.8 for a large one.

Q: How many subjects do you need to detect an effect?

About fifteen point seven divided by the square of the expected effect size, per group, for eighty percent power at the five percent level. A medium effect of 0.5 therefore calls for about sixty subjects per group, and a small effect of 0.2 for nearly four hundred.

Q: What do you do when the two groups do not have the same variance?

You use the Welch correction, which adds the two variances divided by their respective sample sizes instead of pooling them, and which reduces the degrees of freedom. It is the default choice today: when the variances are similar, it gives almost the same result as the classical formula, and it protects you in the opposite case.

This sheet deals with the most used test of the course in advanced quantitative methods, and the one whose conclusions are most often over-interpreted. It does not restate the formulas, it gives the eight places where a correct calculation ends in a false sentence, and the exact sentence that repairs each one.

It is written for CEGEP Social Science students taking 360-301, Advanced Quantitative Methods. The corrected exercise set on the same chapter goes through every step, from choosing the test to reading a published results section.

Mark this sheet as read or add it to your favourites: a free account, no password, keeps your read sheets and favourites from one visit to the next, tells you which chapter to tackle next and lets you ask for the method or exercise you are missing. Create your space, an email is enough.

The thread of the chapter

A t-test answers a single question: would this gap be credible if there were no effect at all? It says neither how large the gap is nor whether it matters, and that is what the effect size answers, without depending on the sample size.

This chapter is part of 360-301, Advanced Quantitative Methods

The essentials

The four outcomes of a test, and the two risks

• Type I error $\alpha$ : rejecting $H_{0}$ when it is true. It is the significance level, set in advance, almost always $0.05$ .
• Type II error $\beta$ : failing to detect an effect that exists. POWER equals $1-\beta$ , and we aim for $0.80$ .
• The $p$ -value is the probability of observing a gap at least this large IF $H_{0}$ is true. It is never the probability that $H_{0}$ is true.
• A 95% confidence interval for the difference contains zero IF AND ONLY IF the two-tailed test at the 5% level fails to reject.

The right-hand column is the one where an effect exists: the Type II error is the failure to find it, and power is the success in finding it.

The two risks move in opposite directions: lowering $\alpha$ to $0.01$ protects against false positives and misses more real effects.

The two risks in a picture

• Under $H_{0}$ , the statistic follows a curve centred at $0$ ; if there is an effect, it follows a shifted curve.
• $\alpha$ is the area of the $H_{0}$ curve beyond the cutoff; $\beta$ is the area of the shifted curve below the cutoff; the power $1-\beta$ is what remains beyond it.
• Moving the cutoff back reduces $\alpha$ and enlarges $\beta$ . Only a larger sample size, which pulls the two curves apart, reduces both risks at once.

In dark blue,

\alpha

under

H_{0}

; in orange,

\beta

, the missed effect; in green, the power. The three areas are read on either side of the same cutoff.

Three t-tests, three denominators

• INDEPENDENT, comparable variances: $s_{p}^{2}=\dfrac{\left(n_{1}-1\right)s_{1}^{2}+\left(n_{2}-1\right)s_{2}^{2}}{n_{1}+n_{2}-2}$ , then $SE=s_{p}\sqrt{\dfrac{1}{n_{1}}+\dfrac{1}{n_{2}}}$ , $df=n_{1}+n_{2}-2$ .
• INDEPENDENT, very unequal variances, WELCH correction: $SE=\sqrt{\dfrac{s_{1}^{2}}{n_{1}}+\dfrac{s_{2}^{2}}{n_{2}}}$ , with reduced degrees of freedom. It is the default choice today.
• PAIRED: we work on the $n$ individual DIFFERENCES, $t=\dfrac{\overline{d}}{s_{d}/\sqrt{n}}$ , $df=n-1$ .
• When the two sample sizes are equal, the pooled formula and the Welch formula give exactly the same standard error.

Pairing removes the variability between individuals and costs degrees of freedom: it pays off as soon as the two measurements are correlated, which is almost always the case for a before-and-after.

What the test does not say, and what does

• Cohen's $d=\dfrac{\overline{x}_{1}-\overline{x}_{2}}{s_{p}}$ : $0.2$ small, $0.5$ medium, $0.8$ large. It does NOT depend on the sample size.
• The confidence INTERVAL of the difference gives the range in the original unit: that is what interests the reader, much more than $p$ .
• Sample size for 80% power: $n\approx \dfrac{2\left(1.96+0.84\right)^{2}}{d^{2}}$ per group, that is about $\dfrac{15.7}{d^{2}}$ .
• Halving $d$ QUADRUPLES the required sample size: that is why small effects call for very large studies.

Two curves shifted by

d

standard deviations: at

d=0.2

they nearly coincide, at

d=0.8

they clearly separate. The sample size changes nothing here.

A complete conclusion cites three things: the decision of the test, the effect size, and the confidence interval in the unit of the problem.

The mistakes that cost marks

These are the errors I correct most often in session. Each one costs marks on a paper, even when the reasoning behind it is right.

1. Reading the p-value as the probability that the null hypothesis is true

1.5 points, deducted from every badly worded conclusion

What not to write

“ $p=0.03$ : there is only a $3$ % chance that there is no difference at all.”

What to write

“ $p=0.03$ : IF the two groups were identical, we would observe a gap at least this large in $3$ % of samples.”

The

p

-value is the blue area, under the curve drawn ASSUMING

H_{0}

is true, beyond the observed

t

: a probability about the data, not about

H_{0}

Why: The $p$ -value is a probability computed UNDER $H_{0}$ , that is, assuming $H_{0}$ is true. It cannot, in the same sentence, measure the probability of that assumption.

2. Treating paired measurements as two independent groups

2 points, and the test loses almost all its power

What not to write

“I have the before and after grades for the same $25$ students: I compare the two means with an independent-samples t-test, $df=48$ .”

What to write

“These are the SAME people: I work on the $25$ individual differences, $t=\dfrac{\overline{d}}{s_{d}/\sqrt{25}}$ with $df=24$ .”

Each line joins the two grades of the same student: almost all of them go up. The paired test works on these slopes, the independent test ignores them.

Why: The independent-samples test puts the variability between individuals in the denominator, whereas pairing removes it. On correlated measurements, it often takes three times as many people to detect the same effect.

3. Pooling the variances when they are very different

1.5 points, and a distorted Type I risk

What not to write

“ $s_{1}=4$ and $s_{2}=16$ : I compute the pooled variance anyway.”

What to write

“The variance ratio is $16$ , well beyond what the homogeneity assumption tolerates: I apply the WELCH correction, which does not require that assumption.”

Standard deviations

4

and

16

: the variances are in a ratio of

16

. Pooling them would lend the tight group the spread of the wide group.

Why: The pooled variance is a weighted average: it lends the small group the spread of the large one. When the sample sizes are as unequal as the variances, the actual level is no longer $5$ %.

4. Concluding there is no difference when we fail to reject

2 points, and the conclusion is reversed

What not to write

“ $p=0.31$ : the two teaching methods give the same results.”

What to write

“We fail to reject $H_{0}$ : the data do not allow us to conclude that there is a difference. With $n=12$ per group, the study only had the power to detect a very large effect.”

The orange interval contains

0

, so we fail to reject; but it also contains large gaps: the data do not settle the question, they do not prove absence.

Why: Not finding something and showing that there is nothing are two different things. Demonstrating equivalence requires an equivalence design and a margin set in advance.

5. Taking a small p-value for an important effect

2 points, and it is the most frequent mistake in the course

What not to write

“ $p<0.001$ : the workshop has a considerable effect on the results.”

What to write

“The effect is very hard to reconcile with luck. Its size is read from $d=0.20$ , a small effect, obtained here thanks to a thousand participants per group.”

The three rows are ranked by increasing effect size and by increasing p-value: it is the sample size, and it alone, that produces this reversed ranking.

Why: $t$ is about $d\sqrt{\dfrac{n}{2}}$ : the $p$ -value therefore depends as much on the sample size as on the effect. The $d$ does not.

6. Getting the degrees of freedom wrong

1 point, and sometimes a reversed decision near the threshold

What not to write

“Two groups of $30$ : $df=30-1=29$ .”

What to write

“Independent test: $df=n_{1}+n_{2}-2=58$ . The $n-1$ belongs to the PAIRED test, where there is only one series of differences.”

Why: Each estimated mean costs one degree of freedom: two means cost two. The error shows up immediately if you ask how many means the calculation required.

7. Choosing a one-tailed test after seeing the data

1.5 points of method, and the actual level is no longer the announced one

What not to write

“The difference goes in the expected direction, so I divide my $p$ -value by two.”

What to write

“The one-tailed choice is made BEFORE looking at the data, and only if a gap in the opposite direction would be of no interest. Otherwise, we stay two-tailed.”

Why: Choosing the side after the fact amounts to giving yourself two chances to reject at the $5$ % level, hence an actual risk of $10$ %. It is the same logic as multiple comparisons.

8. Believing a small effect can be offset by a few more people

1 point, and a study designed without power

What not to write

“The expected effect is half what was planned: I double the sample.”

What to write

“ $n\approx\dfrac{15.7}{d^{2}}$ : halving $d$ QUADRUPLES the required sample size. Going from $d=0.8$ to $d=0.4$ takes us from $25$ to $98$ subjects per group.”

The curve explodes on the left: below

d=0.3

, the required sample size already comes to about one hundred and seventy-five subjects per group.

Why: The sample size enters the test statistic through its square root: you must therefore quadruple the sample size to double the sensitivity. It is the same law as the margin of error of a poll.

Which method to choose

Which test should you choose?

Four questions, in this order: are the groups independent or paired (the same individuals measured twice), what kind of dependent variable, how many groups are compared, do the variances and the distribution allow a parametric test.

If two different groups, quantitative variable, comparable variances → INDEPENDENT-samples t-test, pooled variance, $df=n_{1}+n_{2}-2$

Example: workshop against control, 30 students each
If two different groups but very unequal variances, or very unbalanced sample sizes → t-test with the WELCH correction, reduced degrees of freedom

Example: a very homogeneous group of 12 against a group of 80
If the same individuals measured twice, or matched pairs → PAIRED t-test on the individual differences, $df=n-1$

Example: grades before and after the same workshop
If more than two groups → analysis of VARIANCE, never a series of pairwise t-tests

Example: three teaching methods
If the dependent variable is categorical, or the samples are very small and the distribution very skewed → chi-square, or the Mann-Whitney rank test

Example: a Likert scale on two groups of eight

Today, the Welch correction is the default choice for two independent groups: it costs almost nothing when the variances are similar, and it rescues the test when they are not.

What do you say once the calculation is done?

Look at the decision, then at what else the problem asks for.

If we reject and the problem asks for the magnitude → give Cohen's $d$ and place it on the benchmarks $0.2$ , $0.5$ , $0.8$

Example: $d=0.60$ , medium effect
If we reject and the problem asks for the practical significance → give the confidence INTERVAL of the difference, in the unit of the problem

Example: between $0.8$ and $11.2$ points
If we fail to reject → speak of a lack of evidence, and compute the sample size that would have given 80% power

Example: it would have taken 44 subjects per group
If the problem asks whether the result is important → answer with the effect size and the context, never with the $p$ -value

Example: six points out of a hundred, on a final exam

A complete conclusion fits in one sentence: decision, effect size, interval. All three are in the marking scheme.

How the answer is expected to be written

A marker ticks steps. Here they are in order, with the concluding sentence expected word for word.

Writing up a complete t-test

When to use it: As soon as the problem compares the means of two groups.

1 Name the chosen test and JUSTIFY it: independent or paired, pooled or Welch.
2 Write $H_{0}$ and $H_{1}$ in words, and state whether the test is two-tailed or one-tailed, decided in advance.
3 Compute the standard error, then $t$ , then the degrees of freedom.
4 Compare with the critical value read from the table, and decide.
5 Conclude in context, giving the effect size AND the confidence interval of the difference as well.

Concluding sentence

“At the 5% level, $t=2.31$ with $58$ degrees of freedom exceeds the critical value $2.00$ : we reject the hypothesis that the means are equal. The workshop students score $6$ points higher on average, a $d$ of $0.60$ , and the 95% confidence interval of this gap runs from $0.8$ to $11.2$ points.”

The trap: Writing “the difference is significant” without saying by how much: half the marks of the conclusion are in the $d$ and the interval.

Marking: 1 point for the justified test, 1 point for the hypotheses, 2 points for the calculation, 1 point for the decision, 2 points for the effect size and the interval.

Handling a paired data set

When to use it: When the two series are on the same individuals, or on pairs matched by the researcher.

1 Build the column of individual differences $d_{i}$ , fixing the direction once and for all.
2 Compute $\overline{d}$ and $s_{d}$ on this column, never on the two series separately.
3 Apply $t=\dfrac{\overline{d}}{s_{d}/\sqrt{n}}$ with $df=n-1$ .
4 Give the effect size on the differences, $\dfrac{\overline{d}}{s_{d}}$ , and the confidence interval of $\overline{d}$ .

Concluding sentence

“Across the $25$ students, the mean improvement is $\overline{d}$ points, with $t$ on $24$ degrees of freedom: pairing removes the variability between individuals and gives the test its full power.”

The trap: Switching the direction of subtraction partway through the table: the sign of $\overline{d}$ becomes uninterpretable and so does the $t$ .

Marking: 1 point for the column of differences, 2 points for the calculation, 1 point for the degrees of freedom.

Check before you hand in

Five minutes of checking recover more marks than one more problem started in a hurry.

The interval and the test say the same thing

If the two-tailed test at 5% rejects, the 95% confidence interval of the difference does NOT contain zero. A contradiction signals a calculation error.

A rejection announced with an interval from $-2$ to $+9$ : one of the two results is wrong.
The order of magnitude of t

$t\approx d\sqrt{\dfrac{n}{2}}$ for two groups of the same size: a $t$ far from this estimate betrays an error in the standard error.

With $d=0.6$ and $n=30$ , we expect $t\approx 2.3$ : a $t$ of $12$ comes from a standard deviation confused with a variance.
The sign of t follows the sign of the difference

Check that the sign of $t$ matches the higher group: it is the quickest mistake to spot on a paper.

A negative $t$ with a larger first mean signals a reversed subtraction.
The degrees of freedom match the test

Independent, $n_{1}+n_{2}-2$ ; paired, $n-1$ where $n$ is the number of PAIRS. Recount before reading the table.

Two groups of $30$ give $58$ ; twenty-five pairs give $24$ .

The typical problem, taken apart

Evaluating a study-methods workshop, from the test to the power

Thirty students take a study-methods workshop: mean $72$ , standard deviation $9$ .

Thirty other comparable students do not take it: mean $66$ , standard deviation $11$ .

We ask whether the workshop makes a difference, by how much, and whether the study had the necessary power.

The two workshop groups, with means

72

and

66

: the curves overlap heavily, and it is the standard error of the DIFFERENCE that will decide.

Step 1

Two different groups, quantitative variable, similar variances: INDEPENDENT-samples t-test, two-tailed at the 5% level.

Why

The choice is justified BEFORE the calculation and earns its point. Since the sample sizes are equal, the pooled formula and the Welch formula will give the same standard error anyway.

Step 2

$s_{p}^{2}=\dfrac{29(81)+29(121)}{58}=101$ , so $s_{p}=10.05$ and $SE=10.05\sqrt{\dfrac{1}{30}+\dfrac{1}{30}}=2.59$ .

Why

The pooled variance is an average of the variances weighted by their degrees of freedom: here it is exactly the mean of the two, since the sample sizes are equal.

Step 3

$t=\dfrac{72-66}{2.59}=2.31$ with $df=30+30-2=58$ . Two-tailed critical value at 5%: $2.00$ . We reject $H_{0}$ .

Why

Write the critical value you read, not only the decision: it shows that you entered the table with the right degrees of freedom.

Step 4

$d=\dfrac{72-66}{10.05}=0.60$ , a MEDIUM effect by Cohen's benchmarks.

Why

The $d$ is computed on the pooled standard deviation, not on the standard error: dividing by $2.59$ would give $2.31$ , that is, the $t$ , and not an effect size.

Step 5

95% confidence interval of the difference: $6\pm 2.00\times 2.59$ , that is from $0.81$ to $11.19$ points.

Why

The interval does not contain zero, which confirms the rejection. It also says that the real effect could be as little as one point: it is the most honest piece of information in the problem.

Step 6

Power: for $d=0.60$ , we would need $n\approx\dfrac{2(2.80)^{2}}{0.60^{2}}\approx 44$ subjects per group.

Why

With thirty per group, the study was slightly underpowered: it detected the effect, but it had less than four chances in five of doing so.

The conclusion, written out

“At the 5% level, $t=2.31$ for $58$ degrees of freedom exceeds the critical value $2.00$ : the workshop students obtain a significantly higher mean. The gap is $6$ points, that is $d=0.60$ , a medium effect, and the 95% confidence interval runs from $0.8$ to $11.2$ points.”

The classic mistake on this problem: Dividing the gap of $6$ points by the standard error $2.59$ to get the effect size: we recover the $t$ , and the announced magnitude is four times too large.

Learn by heart

• The $p$ -value is computed UNDER $H_{0}$ : it is never the probability that $H_{0}$ is true.
• INDEPENDENT: $df=n_{1}+n_{2}-2$ . PAIRED: $df=n-1$ , where $n$ is the number of PAIRS.
• Welch when the variances or the sample sizes are very unequal; with equal sample sizes, the two formulas coincide.
• $d=\dfrac{\overline{x}_{1}-\overline{x}_{2}}{s_{p}}$ : $0.2$ small, $0.5$ medium, $0.8$ large, and it does not depend on $n$ .
• The 95% confidence interval contains zero if and only if the two-tailed test at 5% fails to reject.
• $n\approx\dfrac{15.7}{d^{2}}$ per group for 80% power: halving $d$ QUADRUPLES the sample size.
• Failing to reject is not demonstrating equality: it is often a lack of power.

Frequently asked questions

When should you use a paired t-test rather than an independent one?

Whenever the two series are on the same individuals, or on pairs formed by the researcher. You then work on the individual differences, with a number of degrees of freedom equal to the number of pairs minus one. Pairing removes the variability between people, which makes the test far more sensitive on before-and-after measurements.

What does a p-value of 0.03 really mean?

That if the two groups were truly identical, we would observe a gap at least this large in three percent of possible samples. It is not the probability that the null hypothesis is true, nor the probability of being wrong in rejecting it: the p-value is computed by assuming the null hypothesis is true.

How do you compute Cohen's d?

You divide the difference between the two means by the pooled standard deviation of the two groups, never by the standard error. A difference of six points with a pooled standard deviation of ten gives a d of 0.60. The usual benchmarks are 0.2 for a small effect, 0.5 for a medium effect and 0.8 for a large one.

How many subjects do you need to detect an effect?

About fifteen point seven divided by the square of the expected effect size, per group, for eighty percent power at the five percent level. A medium effect of 0.5 therefore calls for about sixty subjects per group, and a small effect of 0.2 for nearly four hundred.

What do you do when the two groups do not have the same variance?

You use the Welch correction, which adds the two variances divided by their respective sample sizes instead of pooling them, and which reduces the degrees of freedom. It is the default choice today: when the variances are similar, it gives almost the same result as the classical formula, and it protects you in the opposite case.

Practise it

Corrected exercises: Comparing two groups: t-test and effect size, 360-301

A method is proved on a paper, not on a sheet. The set for the same chapter takes each of these traps into a problem, with the solution written out step by step.

10 corrected exercises
100 points
150 minutes

Do the exercises

Something missing from this sheet? A method, a trap, an exercise to practise it, a mistake you spotted: tell me, I read every request.

Sending goes through your free space: that is how I can answer you and tell you when the exercise is added. An email is enough, no password. Create your space. Rather not create an account? Write to me directly.

Next sheet One-way analysis of variance

© Ahmed Squalli Houssaini. Revision sheet published at www.letuteurscientifique.ca/en/fiches/cegep-aqm-comparing-two-groups. Free for personal and classroom use; republishing it elsewhere requires written permission (legal notice).

Looking for a 360-301 tutor in Montreal?

Get in touch for a first session. The 360-301 comes at the time of the end-of-program research project, and it is the course where a mistake in method costs the whole analysis chapter.

Book a session WhatsApp

Site by Studio Squalli

Revision sheet: comparing two groups, t-test and effect size (360-301)

The essentials

The four outcomes of a test, and the two risks

The two risks in a picture

Three t-tests, three denominators

What the test does not say, and what does

The mistakes that cost marks

1. Reading the p-value as the probability that the null hypothesis is true

2. Treating paired measurements as two independent groups

3. Pooling the variances when they are very different

4. Concluding there is no difference when we fail to reject

5. Taking a small p-value for an important effect

6. Getting the degrees of freedom wrong

7. Choosing a one-tailed test after seeing the data

8. Believing a small effect can be offset by a few more people

Which method to choose

Which test should you choose?

What do you say once the calculation is done?

How the answer is expected to be written

Writing up a complete t-test

Handling a paired data set

Check before you hand in

The typical problem, taken apart

Evaluating a study-methods workshop, from the test to the power

Learn by heart

Frequently asked questions

When should you use a paired t-test rather than an independent one?

What does a p-value of 0.03 really mean?

How do you compute Cohen's d?

How many subjects do you need to detect an effect?

What do you do when the two groups do not have the same variance?

Corrected exercises: Comparing two groups: t-test and effect size, 360-301

See also

Looking for a 360-301 tutor in Montreal?