Corrected exercises: comparing two groups (360-301)

NULL hypothesis: $\mu_{1}=\mu_{2}$, the two groups come from populations with the same mean, and the observed gap is due only to the luck of sampling. Two-tailed ALTERNATIVE hypothesis: $\mu_{1}\neq \mu_{2}$. We choose the two-tailed version because nothing was announced in advance about the DIRECTION of the effect; adopting a one-tailed test after seeing that the treated group is ahead would quietly double the significance level.

$s_{p}^{2}=\dfrac{24\times 8.2^{2}+27\times 9.6^{2}}{51}=\dfrac{24\times 67.24+27\times 92.16}{51}=\dfrac{1{,}613.76+2{,}488.32}{51}=\dfrac{4{,}102.08}{51}\approx 80.433$, so $s_{p}\approx 8.968$. Standard error of the difference: $s_{p}\sqrt{\dfrac{1}{25}+\dfrac{1}{28}}=8.968\sqrt{0.04+0.035714}=8.968\times 0.275162\approx 2.468$. The pooled variance is an average WEIGHTED by the degrees of freedom: the larger group counts for more, which makes sense since it estimates the common variance better.

$t=\dfrac{72.4-67.9}{2.468}=\dfrac{4.5}{2.468}\approx 1.824$. Degrees of freedom: $n_{1}+n_{2}-2=25+28-2=51$. The statistic reads as a ratio: the observed gap is 1.82 times the standard error of what luck alone would produce.

$\left|t\right|=1.824<2.008$: we FAIL TO REJECT the null hypothesis. The data therefore do not allow us to conclude that the means differ at the 5% level. We must be careful not to write that the two groups are equal: the p-value is about 0.074 here, which leaves the hypothesis of a difference very much alive, and a larger sample might bring it out.

No, it is not contradictory, and the graph helps explain the result. The test is NOT about the overlap of the individual distributions but about the precision of the MEANS, which is much better than that of individuals, since it is divided by the square root of the number of subjects. Two distributions can overlap almost entirely and have very significantly different means if the samples are large. Conversely, as here, a huge overlap signals that even if the difference were real, its PRACTICAL SIGNIFICANCE would be limited: a student drawn from the treated group has a little under seven chances in ten of beating the median of the control group.

$F=\dfrac{9.6^{2}}{8.2^{2}}=\dfrac{92.16}{67.24}\approx 1.371$. The larger variance always goes in the numerator, so that $F\ge 1$ and a single table is enough.

$1.371<2.20$: we fail to reject the equality of variances. The homogeneity assumption is therefore acceptable, and the classical t-test can be used. Note that this F test is itself weak on small samples and very sensitive to non-normality: many statisticians now recommend using the Welch correction from the start and not testing the equality of variances at all.

Welch standard error: $\sqrt{\dfrac{67.24}{25}+\dfrac{92.16}{28}}=\sqrt{2.6896+3.29143}=\sqrt{5.98103}\approx 2.44562$. Statistic: $t=\dfrac{4.5}{2.44562}\approx 1.84005$. The difference from the classical version is tiny, 1.840 against 1.824, because the variances are close and the sample sizes comparable.

Welch gives 50.9 degrees of freedom against 51 for the classical version: the loss is negligible here. The critical value stays close to 2.008, and the conclusion is the same, we fail to reject. The Welch formula lowers the degrees of freedom all the more as the variances and the sample sizes are unbalanced; in the worst case, it brings them down to the size of the smaller group minus one.

The correction really changes the conclusion when the two imbalances combine: very different variances AND very different sample sizes. The dangerous case is the one where the SMALL group has the LARGE variance: the classical test, by weighting the pooled variance by the degrees of freedom, then underestimates the standard error and rejects too often, with an actual risk well above the announced 5%. When, on the contrary, the large group carries the large variance, the classical test becomes too conservative. In both cases, Welch corrects, and its cost when the homogeneity assumption is true is limited to a few tenths of a degree of freedom: that is why it has become the default choice.

Sum: $4-1+6+3+0+5+2+7+1+3=30$, so $\overline{d}=\dfrac{30}{10}=3.0$. Deviations from the mean: $1$, $-4$, $3$, $0$, $-3$, $2$, $-1$, $4$, $-2$, $0$; their squares: 1, 16, 9, 0, 9, 4, 1, 16, 4, 0, with sum 60. Sample variance: $\dfrac{60}{9}\approx 6.667$, so $s_{d}\approx 2.582$.

Standard error: $\dfrac{2.582}{\sqrt{10}}=\dfrac{2.582}{3.1623}\approx 0.8165$. Statistic: $t=\dfrac{3.0}{0.8165}\approx 3.674$, with $n-1=9$ degrees of freedom. Note that the sample here is made of the DIFFERENCES, ten numbers and not twenty: that is what explains the 9 degrees of freedom.

$3.674>2.262$: we REJECT the null hypothesis. The mean improvement of 3.0 points is significant at the 5% level; the p-value is about 0.005, so the result also holds at the 1% level. The conclusion is about the existence of an improvement, not its cause: without a control group, maturation, testing effects and regression to the mean remain competing explanations.

Standard error of the difference between two independent means: $9\sqrt{\dfrac{1}{10}+\dfrac{1}{10}}=9\times 0.4472\approx 4.025$. Statistic: $t=\dfrac{3.0}{4.025}\approx 0.745$, with 18 degrees of freedom and a critical value of 2.101: we would NOT reject. The same 3-point gap therefore goes from highly significant to perfectly harmless depending on the treatment chosen.

The gain comes from ELIMINATING the variability between individuals. By working on the differences, everything specific to each student, starting level, motivation, study method, disappears by subtraction, and only the variability of the CHANGE remains, here much smaller: the standard deviation drops from 9 to 2.58. What pairing costs is twofold. First, degrees of freedom: 9 instead of 18, which raises the critical value from 2.101 to 2.262. Second, a design constraint: the same people must be measured twice, which exposes the study to testing effects and attrition. The gain wins out by far as soon as the two measurements are strongly correlated, which is almost always the case for a school before-and-after.

$d=\dfrac{72.4-67.9}{8.968}=\dfrac{4.5}{8.968}\approx 0.502$. The difference is therefore about half a standard deviation.

Usual benchmarks: 0.2 small, 0.5 medium, 0.8 large. The effect is therefore MEDIUM. In terms of overlap, a $d$ of 0.5 means that a student drawn at random from the treated group beats the median of the control group in about 69% of cases, against 50% if there were no effect, and beats a student drawn at random from the control group in about 64% of cases. These statements say far more than an abstract number, and they remind us that a medium effect leaves an immense zone of overlap.

$\left(\overline{x}_{1}-\overline{x}_{2}\right)\pm t_{crit}\times SE=4.5\pm 2.008\times 2.46776=4.5\pm 4.955$, that is $\left[-0.455,\ 9.455\right]$ points.

The link is exact, not approximate: a 95% confidence interval for the difference contains zero IF AND ONLY IF the two-tailed test at the 5% level fails to reject. The two procedures use the same statistic and the same critical value; they say the same thing in two forms. Checking this consistency is an excellent control against computational errors.

Because the interval contains all the information of the p-value and more. Here, it says that the true difference could be a drop of half a point as well as a rise of nine and a half points: in other words, the study rules out neither the absence of an effect nor a considerable effect, and this is a finding of IMPRECISION more than of absence. A p-value of 0.074 reads as a failure, the interval reads as an invitation to repeat the study on a larger scale. That is why the recommendations of professional statistical associations have for years asked for intervals and effect sizes to be published, and for the mere crossing of the 5% threshold to stop being commented on.

$n=\dfrac{2\left(2.80\right)^{2}}{0.25}=\dfrac{2\times 7.84}{0.25}=\dfrac{15.68}{0.25}=62.72$, that is 63 subjects per group, so 126 in all.

For $d=0.2$: $\dfrac{15.68}{0.04}=392$ subjects per group, that is 784 in all. For $d=0.8$: $\dfrac{15.68}{0.64}=24.5$, that is 25 per group. The sample size varies as $\dfrac{1}{d^{2}}$: halving the effect size to be detected QUADRUPLES the required sample. That is why modest effects, which are the rule in the social sciences, call for studies that few teams can fund.

With about 26 subjects per group, we solve $26=\dfrac{15.68}{d^{2}}$, so $d^{2}=\dfrac{15.68}{26}\approx 0.6031$ and $d\approx 0.777$. The study therefore had an 80% chance of detecting only a LARGE effect, close to 0.8. Yet the effect actually observed was 0.50: the study was undersized by a factor of about 2.4 for what it was looking for, and its inconclusive result was largely predictable BEFORE the data were collected.

The SECOND type of error, denoted $\beta$, is failing to reject $H_{0}$ when it is false: missing a real effect. Power equals $1-\beta$. With 80% power, we therefore accept a one-in-five chance of missing an effect of the targeted size. The parallel with the Type I error is worth drawing: $\alpha$ is the risk of crying wolf, $\beta$ that of not seeing the wolf coming, and the two can only be reduced together by increasing the sample size.

No. With 26 subjects per group, the study was unable to detect anything but a very large effect: finding nothing was the most likely outcome even if a medium effect existed. A failure to reject in an underpowered study therefore carries almost no information. What must be published in that case is the confidence INTERVAL of the difference, which will show that appreciable effects remain compatible with the data, and post hoc power should be avoided: it is merely a restatement of the p-value. The honest conclusion is “our data do not settle the question”, and the next study must be sized on the observed effect size.

First question: are the groups INDEPENDENT or PAIRED, that is, the same units measured twice? Second question: is the dependent variable QUANTITATIVE or categorical? Third question: how many groups are being compared, TWO or more than two? Fourth question: does the distribution allow a parametric test, that is, is the variable approximately normal, or are the sample sizes large enough for the central limit theorem to apply? The order matters: the first question determines the family, the following ones refine it.

Two independent groups, an ORDINAL variable and samples of 12, too small to invoke the central limit theorem: the appropriate test is the nonparametric MANN-WHITNEY test, which compares ranks rather than means. The t-test would be defensible if we accepted the convention that treats a Likert score as an interval variable, but with 12 subjects per group and a five-level variable, caution calls for ranks.

Two independent groups, a quantitative variable, large samples: the independent-samples t-test is appropriate, with the Welch correction. The non-normality of income, which is strongly right-skewed, is not an obstacle because the test is about MEANS, and the sampling distribution of a mean tends toward the normal as $n$ grows, whatever the shape of the population: that is the central limit theorem. With nearly 400 observations per group, the approximation is excellent. One reservation remains: on such a skewed distribution, the mean is not necessarily the quantity of interest, and comparing the MEDIANS with a Mann-Whitney test would answer the question asked better.

Thirty students measured twice: paired-samples t-test on the individual differences. With a control group, the question changes: we no longer test an improvement but a GAP between improvements, and we compare the individual differences of the treated group with those of the control group by an independent-samples t-test. This is the statistical version of the difference in differences, and it is the correct design for evaluating a workshop.

The dependent variable is CATEGORICAL, passed or failed, and we compare three independent groups: no t-test applies. We build a 3 by 2 contingency table and use the chi-square test of independence, with $df=\left(3-1\right)\left(2-1\right)=2$, then a Cramér's $V$ for the strength. If an expected frequency fell below 5, we would merge programs or switch to Fisher's exact test.

Standard error: $9\sqrt{\dfrac{2}{40}}=9\times 0.22361\approx 2.0125$. Statistic: $t=\dfrac{1.5}{2.0125}\approx 0.745$, very far from the critical value close to 1.99: we fail to reject.

With 400 per group: standard error $9\sqrt{\dfrac{2}{400}}=9\times 0.070711\approx 0.6364$, so $t=\dfrac{1.5}{0.6364}\approx 2.357$, greater than 1.96: we reject. With 1,000 per group: standard error $9\times 0.044721\approx 0.40249$, so $t\approx 3.727$, far beyond the threshold.

$d=\dfrac{1.5}{9}\approx 0.167$ in ALL THREE cases. The effect size does not depend on the sample size: it describes the separation of the two distributions, not the amount of data collected. That is exactly the role we ask of it, and that is why it must accompany every test.

We look for $n$ such that $\dfrac{1.5}{9\sqrt{2/n}}\ge 1.96$, so $\sqrt{\dfrac{2}{n}}\le \dfrac{1.5}{9\times 1.96}=0.085034$, hence $\dfrac{2}{n}\le 0.0072308$ and $n\ge 276.6$, that is 277 subjects per group. Beyond that, a gap of 1.5 points out of 100, which interests nobody, will be declared significant every time.

“The treated group scores on average 1.5 points higher than the control group, on a scale of 100, a statistically significant difference, $t\left(1{,}998\right)=3.73$, p below 0.001, but with a very small effect size, $d=0.17$. This gap is about one sixth of a standard deviation, and it has no identifiable practical significance for an individual student.” The sentence gives the three required elements, the raw gap with its unit, the test with its degrees of freedom, the effect size, and it explicitly refuses to turn significance into importance.

FALSE. The p-value is the probability of observing a gap at least this large IF the null hypothesis were true, not the probability that the null hypothesis is true. The two quantities are inverted conditional probabilities, like the probability of being wet given that it rains and the probability that it rains given that we are wet. Correct statement: if the two groups came from identical populations, we would observe a gap at least this large in 7.4% of repetitions of the study.

FALSE. The test is about the MEANS, whose standard error is $\dfrac{s}{\sqrt{n}}$ and decreases without limit as the sample size grows, whereas the overlap of the individual distributions does not depend on it. With 1,000 subjects per group, a $d$ of 0.17, hence an enormous overlap, gives $t\approx 3.7$ and a p-value below 0.001. Correct statement: a heavy overlap indicates a small effect size, not the impossibility of a significant result.

FALSE. The two measurements are on the same people: they are not independent, and treating the sixty measurements as two separate samples inflates the standard error by feeding all the variability between individuals back into it. On the example of the chapter, the $t$ fell from 3.67 to 0.75 for the same data. Correct statement: we compute the 30 individual differences and test whether their mean is zero, with a PAIRED-samples t-test with 29 degrees of freedom.

FALSE. With 26 subjects per group, the study had an 80% chance of detecting only an effect of size 0.78, that is, a big effect. Finding nothing was the expected outcome even in the presence of a medium effect. Correct statement: our data do not allow a conclusion; the confidence interval of the difference runs from $-0.5$ to $+9.5$ points, which leaves open the possibility of an appreciable effect, and a study of about 63 subjects per group would be needed.

FALSE, the conclusion mixes up the two notions. A tiny p-value only indicates that the gap is hard to attribute to luck, which, with large samples, happens for any gap whatsoever. Importance is judged on the $d$ and on the unit of measurement, and a $d$ of 0.17 is small. Correct statement: the effect is very clearly established and very small; it is real and without obvious practical significance, which is in no way contradictory.

This is a WAITLIST EXPERIMENT with random assignment among the applicants. It provides INTERNAL validity: the two groups differ on average only by the treatment, including on unmeasured variables, so the observed difference can be attributed to the workshop. EXTERNAL validity remains limited to the students who ASK for such a workshop, a population that is not that of the CEGEP as a whole.

The sample sizes being equal, $s_{p}^{2}=\dfrac{47\times 10.4^{2}+47\times 11.1^{2}}{94}=\dfrac{10.4^{2}+11.1^{2}}{2}=\dfrac{108.16+123.21}{2}=\dfrac{231.37}{2}=115.685$, so $s_{p}\approx 10.7557$. Standard error: $10.7557\sqrt{\dfrac{2}{48}}=10.7557\times 0.204124\approx 2.19549$. Statistic: $t=\dfrac{74.2-70.1}{2.19549}=\dfrac{4.1}{2.19549}\approx 1.8675$, with $48+48-2=94$ degrees of freedom.

$1.868<1.986$: we fail to reject at the 5% level. The p-value is about 0.065, so the result is close to the threshold without reaching it. The correct wording is that the study does not establish an effect, not that the workshop has none.

$d=\dfrac{4.1}{10.7557}\approx 0.381$, a SMALL TO MEDIUM effect. Confidence interval: $4.1\pm 1.986\times 2.19549=4.1\pm 4.3602$, that is $\left[-0.260,\ 8.460\right]$ points. It contains zero, in exact agreement with the test, and above all it shows that the true difference could reach eight and a half points: the study is imprecise rather than negative. A replication with about 110 students per group, the sample size required to detect $d=0.38$ with 80% power, would settle the question.

They must be KEPT in the treated group, and this analysis presented as the main one: it is the INTENTION-TO-TREAT analysis, and it is what preserves the equivalence produced by the random draw. Removing the nine absentees would amount to comparing a subgroup selected by its own behaviour, probably the least engaged, with a complete control group, which would inflate the effect. A complementary analysis restricted to the 39 students who actually took part can be published, labelled as such and noting that it gives an optimistic upper bound. The question that intention to treat answers is the decision maker's: what does OFFERING the workshop produce, knowing that some of those enrolled will not come.

With $df=n_{1}+n_{2}-2=1{,}198$, the total sample size is 1,200, that is about 600 per group if the groups are balanced. The standard error follows from the $t$: $SE=\dfrac{73.8-70.2}{4.12}=\dfrac{3.6}{4.12}\approx 0.874$. We can check consistency: with $s=15$ and 600 per group, $SE=15\sqrt{\dfrac{2}{600}}\approx 0.866$, which agrees. This recalculation is a good reading reflex: it catches inconsistencies and lets you reconstruct what the article did not publish.

$d=\dfrac{3.6}{15}=0.24$, a SMALL effect. In other words, the distributions overlap by more than 90%, and knowing a student's gender only marginally improves the prediction of the grade. The p-value below 0.001 comes almost entirely from the 1,200 subjects, not from the size of the gap.

First objection, the word DEMONSTRATED applied to an effect of 0.24: the study establishes a small difference between means, and has demonstrated nothing more. Second objection, the word EFFECT, which asserts causality: gender was not randomly assigned, and the gap may reflect differences in chosen program, hours of paid work, relationship to school, all correlated with gender. The correct wording speaks of a difference ASSOCIATED with gender, not an effect of gender.

A failure to reject is not proof of absence. With $p=0.21$, the data are COMPATIBLE with the null hypothesis, and just as compatible with differences of several points, as the confidence interval that the article does not publish would show. We would also need to know the number of programs compared and the size of each: a comparison among six programs of 200 students does not have the same power as one between two programs of 600.

“Female students obtain an average of 73.8 against 70.2 for male students, a gap of 3.6 points, a statistically significant difference, $t\left(1{,}198\right)=4.12$; $p<0.001$, but of small magnitude, $d=0.24$, with a confidence interval of 1.9 to 5.3 points. Since gender was not assigned, this difference cannot be interpreted as a causal effect. The comparison between programs does not allow a conclusion, $p=0.21$; the confidence interval of the gap between the extreme programs remains nevertheless compatible with differences of several points, and the study therefore cannot rule out a link.” The paragraph gives the raw gaps, the effect sizes, the intervals, and replaces two overreaching claims with two findings of indecision.