Corrected practice final exam: advanced quantitative methods (360-301)

$H_{0}:\ \mu_{1}=\mu_{2}$; $H_{1}:\ \mu_{1}\neq \mu_{2}$. Pooled variance: $s_{p}^{2}=\dfrac{31\times 7.4^{2}+29\times 8.9^{2}}{60}=\dfrac{31\times 54.76+29\times 79.21}{60}=\dfrac{1{,}697.56+2{,}297.09}{60}=\dfrac{3{,}994.65}{60}\approx 66.578$, so $s_{p}\approx 8.160$.

$SE=8.160\sqrt{\dfrac{1}{32}+\dfrac{1}{30}}=8.160\sqrt{0.03125+0.033333}=8.160\times 0.254131\approx 2.0738$. $t=\dfrac{76.8-72.1}{2.0738}=\dfrac{4.7}{2.0738}\approx 2.267$, with $32+30-2=60$ degrees of freedom.

$2.267>2.000$: we REJECT the null hypothesis. The difference of 4.7 points is significant at the 5% level; the p-value is about 0.027.

$d=\dfrac{4.7}{8.160}\approx 0.576$, a MEDIUM effect. Confidence interval: $4.7\pm 2.000\times 2.0738=4.7\pm 4.148$, that is $\left[0.552,\ 8.848\right]$ points. The interval excludes zero, which agrees with the rejection, and it shows that the true difference could be as small as half a point or as large as nearly nine points: the result is established but imprecise.

$F=\dfrac{8.9^{2}}{7.4^{2}}=\dfrac{79.21}{54.76}\approx 1.447$. The ratio stays well below 4, the usual cutoff when the sample sizes are similar: the homogeneity of variances is acceptable and the classic t-test can be used. With more unbalanced variances, we would have switched to the Welch correction.

$SE=\dfrac{s_{d}}{\sqrt{n}}=\dfrac{3.6}{\sqrt{12}}=\dfrac{3.6}{3.4641}\approx 1.0392$. $t=\dfrac{2.4}{1.0392}\approx 2.309$, with $n-1=11$ degrees of freedom.

$2.309>2.201$: we REJECT. The mean improvement of 2.4 points is significant at the 5% level, the p-value being about 0.041. The result is therefore established, but only just.

$d=\dfrac{\overline{d}}{s_{d}}=\dfrac{2.4}{3.6}\approx 0.667$, which corresponds to a MEDIUM to LARGE effect by the usual benchmarks. In other words, the typical improvement is worth two thirds of the spread of the individual improvements, which is appreciable.

$n\approx \dfrac{\left(1.96+0.84\right)^{2}}{d^{2}}=\dfrac{7.84}{0.4449}\approx 17.6$, that is 18 pairs. With 12 pairs, the study was therefore slightly undersized, and the narrow rejection was to be expected.

From the standpoint of the design, nothing is shown about the workshop. A before-and-after study without a control group leaves maturation, testing, history and, if the participants were chosen for a poor initial result, regression toward the mean all open. The test establishes that there was an improvement; it does not establish that the improvement comes from the workshop. A waitlist experiment would be needed, in which the individual improvements of the students drawn by lot would be compared with those of the students not drawn.

Degrees of freedom: between, $k-1=4-1=3$; within, $N-k=40-4=36$; total, $N-1=39$. Total sum of squares: $486.0+2{,}340.0=2{,}826.0$. We check that $3+36=39$.

$MS_{between}=\dfrac{486.0}{3}=162.0$ and $MS_{within}=\dfrac{2{,}340.0}{36}=65.0$. $F=\dfrac{162.0}{65.0}\approx 2.492$.

$2.492<2.87$: we FAIL TO REJECT. The data do not allow us to conclude that the four methods differ at the 5% level. The p-value is about 0.076, so the result is close to the threshold without reaching it.

$\eta^{2}=\dfrac{486.0}{2{,}826.0}\approx 0.172$. The mentoring method therefore accounts for 17.2% of the variance in the results, which is well above the 0.14 benchmark for a LARGE effect. So there is a substantial effect that the test does not detect: this is an underpowered study, and simply reporting “no significant difference” would be downright misleading here.

More subjects are needed. For an effect of this size, expressed as $f=\sqrt{\dfrac{\eta^{2}}{1-\eta^{2}}}=\sqrt{\dfrac{0.172}{0.828}}\approx 0.456$, 80% power with four groups requires about 15 to 16 subjects per group, against 10 here. In other words, about twenty to twenty-five more students would probably be enough to settle it, and that is exactly what the conclusion of the report should recommend rather than concluding that the methods are equivalent.

$\dfrac{4\times 3}{2}=6$ comparisons. Without correction, the risk of wrongly concluding at least once is $1-0.95^{6}\approx 0.2649$, close to 27% instead of 5%.

$SE=\sqrt{MS_{within}\left(\dfrac{1}{10}+\dfrac{1}{10}\right)}=\sqrt{65.0\times 0.2}=\sqrt{13.0}\approx 3.6056$. The standard error is the same for all six pairs since the sample sizes are equal.

Minimum detectable difference: $2.79\times 3.6056\approx 10.06$ points. Any difference between means smaller than 10.1 points will be declared not significant after correction.

Observed gaps: $C-A=74.8-68.2=6.6$; $D-A=4.7$; $B-A=3.3$; $C-B=3.3$; $D-B=1.4$; $C-D=1.9$. NO pair reaches 10.1 points. The largest gap, 6.6 points between C and A, stays well below the threshold.

“Since the overall test rejected the hypothesis that the means are equal, six pairwise comparisons were carried out with a Bonferroni correction. No pair reaches the threshold: the minimum detectable difference is 10.1 points, and the largest observed gap, between methods C and A, is 6.6 points. The study therefore establishes that at least one method stands out, without being able to say which, which is a direct consequence of the sample size of ten students per group.” This situation, in which the omnibus test rejects and no comparison follows, is common and must be reported as it stands rather than hidden.

$t_{1}=\dfrac{0.42}{0.058}\approx 7.241$; $t_{2}=\dfrac{0.91}{0.240}\approx 3.792$; $t_{3}=\dfrac{-0.28}{0.090}\approx -3.111$. All three exceed 1.974 in absolute value: the three coefficients are significantly different from zero at the 5% level.

“Holding the high school average and the hours of paid work constant, each additional weekly hour of study is associated with 0.91 more points in the term average.” The holding-constant clause and the word “associated” are both mandatory: since the data are cross-sectional, no causal link is established.

$R_{adj}^{2}=1-\left(1-0.47\right)\dfrac{179}{176}=1-0.53\times 1.017045=1-0.539034\approx 0.461$. With 180 observations for three variables, the correction costs only about one percentage point.

$\hat{y}=38.5+0.42\times 80+0.91\times 10-0.28\times 15=38.5+33.6+9.1-4.2=77.0$. The prediction concerns the mean of the students with this profile, not the individual.

$-0.28\pm 1.974\times 0.090=-0.28\pm 0.17766$, that is $\left[-0.458,\ -0.102\right]$. The interval excludes zero, consistent with the test, and it indicates that ten weekly hours of work cost between 1.0 and 4.6 points of average, other variables held constant: a wide range, but entirely on the same side.

The scatter plot traces a U: the residuals are positive at both ends and negative in the middle. The problem is NONLINEARITY: the true relationship is curved, and the fitted line systematically underestimates at the extremes while overestimating at the center. The coefficients are then perfectly calculated and biased over the whole domain.

First remedy, add a QUADRATIC TERM $x^{2}$ to the model, which lets the fit curve. Second remedy, TRANSFORM the variable, with a logarithm if the relationship shows diminishing returns, or with a square root. A third remedy exists: cut the variable into classes and introduce them through indicator variables, which imposes no particular form but uses up degrees of freedom.

$VIF=\dfrac{1}{1-0.78}=\dfrac{1}{0.22}\approx 4.545$. The standard error is multiplied by $\sqrt{4.545}\approx 2.132$: that of $x_{1}$ would go from 0.058 to 0.1236, and its $t$ statistic from 7.24 to 3.40. The coefficient would remain significant, but we are already close to the warning threshold of 5 for the variance inflation factor.

We need $4-1=3$ indicator variables, the fourth category serving as the REFERENCE. Each coefficient is read as the mean gap between its field of study and the reference field, other variables held constant. Comparing two fields other than the reference requires an additional test, or rerunning the model with a different reference.

No. $R_{adj}^{2}$ goes from $1-0.53\times \dfrac{179}{176}\approx 0.461$ to $1-0.514\times \dfrac{179}{171}\approx 0.462$, a gain of one thousandth for five variables. In other words, the five variables add practically nothing beyond what the degrees of freedom they consume cost, and they complicate the interpretation while aggravating the risk of collinearity. The three-variable model remains preferable.

Three reasons add up. The variable is ORDINAL, a scale from 1 to 10 whose gaps are not guaranteed to be comparable. The sample sizes are very small, five per group, which rules out appealing to the central limit theorem. And a bounded scale produces a mechanical skew as soon as the answers pile up toward one end.

$62+40+18=120$, which must equal $\dfrac{N(N+1)}{2}=\dfrac{15\times 16}{2}=120$. The check passes: no rank was forgotten or counted twice.

$\sum \dfrac{R_{j}^{2}}{n_{j}}=\dfrac{62^{2}+40^{2}+18^{2}}{5}=\dfrac{3{,}844+1{,}600+324}{5}=\dfrac{5{,}768}{5}=1{,}153.6$. Then $H=\dfrac{12}{15\times 16}\times 1{,}153.6-3\times 16=0.05\times 1{,}153.6-48=57.68-48=9.68$.

$9.68>5.991$: we REJECT. At least one of the three groups stands out from the others, with a p-value of about 0.008. A customary reservation: with five observations per group, the chi-square approximation is at the edge of its domain, and software would use the exact distribution of $H$.

The logical next step is a series of pairwise comparisons by Mann-Whitney tests, with a corrected threshold, $\dfrac{0.05}{3}\approx 0.0167$ for three comparisons. The difficulty to expect is that with five observations per group, the smallest possible $U$ is 0 and the smallest p-value attainable is about 0.008: only a perfect separation, where one group lies entirely above the other ($U=0$), or a near-perfect one ($U=1$, p-value about 0.016), will clear the corrected threshold. It is therefore likely that no comparison will be significant even though the overall test was, a situation that has to be reported explicitly.

$1.12+0.85+1.20+0.83=4.00$: the sum is exact, no renormalization is needed. The index of 0.85 means that the second quarter sits structurally 15% BELOW the average level of its year, year after year.

$\overline{x}=2$, $\overline{y}=\dfrac{620+668+710}{3}=\dfrac{1{,}998}{3}=666$. Sum of cross-products of deviations: $\left(-1\right)\left(620-666\right)+\left(1\right)\left(710-666\right)=46+44=90$. Sum of squares of the deviations of $x$: 2. Slope: $\dfrac{90}{2}=45$. Intercept: $666-45\times 2=576$. Line: $\hat{y}=576+45x$.

Year 4: $576+45\times 4=576+180=756$ visits per quarter on average.

$756\times 1.12=846.7$; $756\times 0.85=642.6$; $756\times 1.20=907.2$; $756\times 0.83=627.5$. Sum: $3{,}024.0$, and $4\times 756=3{,}024$. The check passes exactly, which confirms both the indices and the trend.

RAW change: $\dfrac{852-568}{568}=0.5$, that is 50%. Deseasonalized values: $\dfrac{568}{0.85}\approx 668.2$ and $\dfrac{852}{1.20}=710.0$. Deseasonalized change: $\dfrac{710.0-668.2}{668.2}\approx 0.0625$, that is 6.25%. The real movement is therefore eight times smaller than the apparent one: the announced 50% describes the calendar, the 6.3% describes the service.

FALSE. Failing to reject means that the data are compatible with equality, not that equality has been demonstrated. Here $\eta^{2}=0.172$ corresponds to a large effect that ten students per group are not enough to detect. Correct statement: the study does not allow a conclusion; the share of variance associated with the method is nevertheless substantial, and a sample of about 16 per group would be needed to settle it.

FALSE. The two measurements concern the same people: they are not independent. We must work on the twelve individual DIFFERENCES, with $n-1=11$ degrees of freedom and not 22. Treating the measurements as independent puts the variability between individuals back into the denominator and makes the statistic collapse. Correct statement: paired-samples t-test, $t=2.31$ at 11 degrees of freedom.

FALSE. The two coefficients are expressed in different units, points per hour of study and points per hour of work, and their contribution also depends on the spread of each variable. Correct statement: to compare relative importance, we need the STANDARDIZED coefficients, or failing that we can compare the $t$ statistics, here 3.79 against $-3.11$, which measure reliability and not magnitude.

FALSE. The second quarter has a seasonal index of 0.85 and the third quarter 1.20: the rise is largely a calendar effect. After deseasonalizing, the real increase is 6.3%. Correct statement: with seasonal variation removed, attendance rises by 6.3% over the previous quarter.

FALSE, the rule is reversed for $U$. We reject when the statistic is LESS THAN OR EQUAL TO the critical value, because a small $U$ means that one group monopolizes the high ranks. With $U=3$ against 6, we REJECT. Correct statement: the difference between the two groups is significant at the 5% level.

No. The design is a before-and-after study WITHOUT A CONTROL GROUP, the weakest there is. Three competing explanations: MATURATION, students improve on their own over the course of the term; HISTORY, a different instructor, exam or calendar at the second measurement; TESTING, simple familiarity with the format of the test. The t-test establishes that there was an improvement, it establishes nothing about its cause.

REGRESSION TOWARD THE MEAN. Selecting on a failure guarantees that part of the poor initial result was due to passing circumstances, and the next measurement rises mechanically. It is quantified by $z_{2}=r\,z_{1}$: if the selected students were 1.5 standard deviations below the mean and the correlation between the two exams is 0.6, the prediction with no intervention at all is $-0.9$ standard deviations, a rise of $-0.9-\left(-1.5\right)=0.6$ standard deviations. Beware of the trap: the rise is the DIFFERENCE of the two Z scores, not the value $0.9$ of the second one. With a standard deviation of 10 points, this is 6 points of expected rise, against 6.8 points observed: the program therefore adds only eight tenths of a point to what doing nothing would have produced.

Two criticisms. First, with $p=0.076$, the test FAILS TO REJECT at the usual level: the report presents as a demonstration a result that does not reach the threshold it set for itself. Second, an analysis of variance would compare fields of study with one another, not treatment EFFECTS: showing that the effect varies by field of study would require an interaction term between treatment and field of study, in a design that includes a control group. The word “confirms” is, moreover, the opposite of what a test does, which can only fail to refute.

A failure to reject is not proof of absence. With $p=0.22$, the data are simply compatible with zero, and the confidence interval, which is not reported, probably contains appreciable values. It must also be added that the income coefficient is PARTIAL: it is controlled for the other variables in the model, several of which, high school average and hours of paid work, are themselves consequences of income. A good part of the effect of income has therefore been absorbed upstream, which makes the absence of a coefficient of its own expected rather than informative.

“Participants improve by 6.8 points on average, $t=3.42$; $p=0.002$; $n=40$. The design being a before-and-after study with no control group, and the participants having been selected on a failure, this improvement is compatible with regression toward the mean alone, whose expected size is of the order of 6 points: the data therefore do not allow us to conclude that the program has an effect. For the next term, we recommend a waitlist experiment: among the students who failed the midterm, half, drawn by lot, would receive the program immediately and the other half the following term, and the analysis would compare the individual improvements of the two groups on an intention-to-treat basis. A sample of about 60 students per group would detect a medium effect with 80% power.” The paragraph keeps the facts, names the threat, quantifies what would be needed, and replaces a recommendation to extend the program with a protocol.