Corrected exercises: multiple regression (360-301)

In model 1, the coefficient 1.30 is a RAW slope: each additional weekly hour of study is associated with 1.30 more points, with everything else left as it is observed. In model 2, the coefficient 1.05 is a PARTIAL slope: each additional hour of study is associated with 1.05 more points HOLDING HOURS OF PAID WORK CONSTANT. The phrase “holding the other variables constant” is not a writing formality, it is the entire content of the coefficient.

Because the two explanatory variables are CORRELATED with each other: students who work a lot study less. In model 1, the slope of 1.30 mixes the own effect of study with the opposite effect of the paid work that goes with it. Introducing $x_{2}$ removes that borrowed share, and 1.05 is what remains. About a fifth of the raw association was therefore due to something other than the hours of study themselves: this is exactly the logic of elaboration seen with contingency tables, carried over to continuous variables.

Holding hours of study constant, each additional weekly hour of paid work is associated with 0.42 points LESS. Stress “holding hours of study constant”: the coefficient does not measure the total effect of paid work, much of which operates through reduced study time, but what remains beyond that: fatigue, absences, disengagement.

$\hat{y}=55.4+1.05\times 12-0.42\times 18=55.4+12.6-7.56=60.44$. The prediction applies to the MEAN of students with this profile, not to the individual, whose grade will differ from this value by a residual.

The student would be at 18 hours of study and 12 hours of work: $\hat{y}=55.4+1.05\times 18-0.42\times 12=55.4+18.9-5.04=69.26$, that is 8.82 points more. The same number can be found directly: moving 6 hours is worth $6\times \left(1.05+0.42\right)=6\times 1.47=8.82$. Note, however, that this is a model PREDICTION, not a causal promise. Since the data are cross-sectional, nothing guarantees that a student who changes behaviour actually obtains this gain.

$t=\dfrac{b}{SE}$. Hours of study: $\dfrac{1.05}{0.21}=5.00$. Hours of work: $\dfrac{-0.42}{0.11}\approx -3.818$. Constant: $\dfrac{55.40}{3.12}\approx 17.76$. This recalculation takes ten seconds and catches transcription errors: do it every time you read a published table.

Each row tests $H_{0}\ :\ \beta_{j}=0$, that is, that variable $j$ contributes nothing HOLDING THE OTHER VARIABLES CONSTANT. The test is two-sided. Stress the constancy clause: a variable can be strongly related to $y$ taken alone and perfectly nonsignificant in the model, simply because another variable already carries the same information.

$1.05\pm 1.976\times 0.21=1.05\pm 0.415$, that is $\left[0.635, 1.465\right]$. The interval excludes zero, consistent with $p<0.001$, and above all it shows that the plausible effect of one hour of study ranges from a gain of six tenths of a point to nearly a point and a half.

Different from zero: yes, $\left|t\right|=3.82$ is well beyond 1.976. Different from $-1$: a new test is needed, $t=\dfrac{-0.42-\left(-1\right)}{0.11}=\dfrac{0.58}{0.11}\approx 5.27$, so yes again, and very clearly. The question is worth asking because the table tests ONLY the zero hypothesis, whereas the interesting question is often another value, for example a threshold considered important in practice.

No. The two coefficients are expressed in different units, points per hour of study and points per hour of work, and the size of their effect also depends on the SPREAD of each variable: a variable whose values barely vary cannot produce much difference, even with a large coefficient. To compare, use STANDARDIZED coefficients, obtained by multiplying each $b$ by $\dfrac{s_{x_{j}}}{s_{y}}$: they are all expressed in standard deviations and become comparable. The ratio of the $t$ statistics, here 5.00 against 3.82, is an acceptable shortcut for saying which one is more solidly established, but it does not measure the size of the effect.

Model 1 accounts for 28% of the variance of the grades, model 2 for 41%. Adding hours of paid work therefore explains 13 more percentage points of variance. Put differently, knowing both variables reduces by 41% the sum of squared deviations relative to a prediction by the mean alone. 59% of the variation in grades remains unexplained by the model, which is the ordinary situation in the social sciences.

$R_{adj}^{2}=1-\left(1-0.41\right)\dfrac{149}{147}=1-0.59\times 1.013605=1-0.598027\approx 0.402$. The correction costs eight thousandths: it is light because 150 observations for 2 variables leave plenty of degrees of freedom.

With $k=7$: $R_{adj}^{2}=1-\left(1-0.425\right)\dfrac{149}{142}=1-0.575\times 1.049296=1-0.603345\approx 0.397$. The raw $R^{2}$ ROSE, from 0.410 to 0.425, and the adjusted $R^{2}$ FELL, from 0.402 to 0.397. This is exactly the signal you must learn to read: the five added variables do not bring in more than the degrees of freedom they use up, so they contribute nothing. The two-variable model is preferable.

Because the method of least squares MINIMIZES the sum of squared residuals over all possible coefficients. Adding a variable enlarges the set of available models: in the worst case, the software gives it a zero coefficient and recovers exactly the previous model. The minimum can therefore only fall or stay equal, and $R^{2}$ can only rise or stay equal. In practice, with real data, a random variable always gains a little, and with $n$ variables for $n$ observations you reach $R^{2}=1$ having learned nothing.

In a context of individual PREDICTION, 0.41 is weak: a considerable residual standard deviation remains, and the grade of a given student will still be poorly predicted. In a context of EXPLANATION in the social sciences, where behaviour depends on dozens of unmeasured factors, explaining 41% of the variance of grades with two administrative variables is on the contrary a very good result. The same number would be catastrophic in physics and remarkable in sociology: an $R^{2}$ is never judged in absolute terms, but against what the literature in the field usually obtains.

Holding hours of study constant, a student who has a job obtains on average 2.9 points LESS than a student without a job. The coefficient of a dummy variable is not a slope but a SHIFT: it moves the whole line down or up, without changing its incline.

Without a job, $e=0$: $\hat{y}=57.1+1.02x_{1}$. With a job, $e=1$: $\hat{y}=57.1-2.9+1.02x_{1}=54.2+1.02x_{1}$. Two PARALLEL lines, vertically 2.9 points apart. Parallelism is an assumption of the model, not an observed fact: lifting it requires an interaction term.

You need $3-1=2$ dummy variables. With three, their sum would equal 1 for everyone, so they would be perfectly redundant with the constant, and the system would no longer have a unique solution: this is the DUMMY VARIABLE TRAP, or perfect collinearity. The category left out becomes the REFERENCE, and all the coefficients are read relative to it.

The third program, the reference, serves as zero. The first is $+3.4$ points from the reference, the second $-1.8$. Ranking from highest to lowest: program 1, then program 3, then program 2, with 5.2 points between the two extremes. Note: the table tests the difference of each program FROM THE REFERENCE, not the programs against each other; concluding that 1 differs significantly from 2 requires an additional test, or changing the reference and rerunning.

You would be imposing on the model that going from program 1 to program 2 has exactly the same effect as going from 2 to 3, since a single coefficient multiplies the code. But these codes are NOMINAL labels whose order is arbitrary: swapping two programs would completely change the result. The model would be both constrained for no reason and dependent on a meaningless coding choice. This is the most common beginner error, and the software never complains.

The residual is the difference between the observed value and the predicted value: $e_{i}=y_{i}-\hat{y}_{i}$. Here $67-60.44=6.56$ points: the model underestimates this student by 6.56 points, in other words the student does better than their hours of study and work would suggest.

First, their SUM is zero, and so is their mean: this is a direct consequence of having a constant in the model. Second, they are UNCORRELATED with each of the explanatory variables and with the predicted values: all the linear information available in these variables has been used. These two properties are automatic and so prove nothing about the quality of the model; what you examine is the SHAPE of the cloud.

This is HETEROSCEDASTICITY: the variance of the residuals grows with the predicted value. It does NOT bias the coefficients, which remain correct estimates, but it distorts their STANDARD ERRORS, and therefore the $t$ statistics, the p values and the confidence intervals, generally toward excessive confidence. It is remedied with robust standard errors, known as White's, or with a transformation of the dependent variable, often the logarithm when the variable is an amount of money.

Three checks before any decision. First, look for a DATA ENTRY ERROR: a grade of 97 typed as 79, a duplicate, a different unit. Next, measure the observation's actual INFLUENCE by rerunning the regression without it: if the coefficients barely move, the point is an outlier without being influential, and you keep it. Finally, examine the CASE: a student repeating the course, an athlete in competition, a profile the model was not meant to cover. You never delete an observation because it is inconvenient, and any deletion must be justified and reported.

A U-shaped cloud signals NONLINEARITY: the relationship between the variables is not a straight line, and the model underestimates at both ends while overestimating in the middle, or the reverse. This is common with hours of study, whose returns diminish: the first ten hours bring a lot, the next ten less. It is remedied by adding a quadratic term $x_{1}^{2}$, by transforming the variable with a logarithm, or by cutting it into classes with dummy variables. A linear regression on a curved relationship gives coefficients that are perfectly calculated and systematically wrong at each end of the range.

The paradox is only apparent. The two variables carry almost the same information: the model can therefore attribute most of the effect to one, to the other, or split it in a thousand ways, without the quality of the fit suffering. The $R^{2}$, which measures the OVERALL fit, therefore stays excellent. But the standard error of EACH coefficient explodes, since the data cannot settle between the possible splits, and each $t$ collapses. The characteristic sign is therefore this: a model that is significant overall in which no coefficient is individually significant.

$VIF=\dfrac{1}{1-0.85}=\dfrac{1}{0.15}\approx 6.667$. The variance of the coefficient is therefore 6.67 times larger than it would be if the variable were independent of the others.

The standard error is the square root of the variance, so it is multiplied by $\sqrt{6.667}\approx 2.582$. A coefficient that would have had a standard error of 0.21 now has one of 0.54, and its $t$ statistic is divided by 2.58: it would go from 5.00 to 1.94, below the threshold.

The usual thresholds are 5 and 10: a $VIF$ above 5 signals a serious problem, and beyond 10 the situation is generally judged unacceptable. Three remedies exist. DROP one of the two variables, which is legitimate when they measure the same concept, here the time devoted to study. COMBINE them into a single index, a mean or a score. Or INCREASE the sample size, which reduces all the standard errors and can be enough when the collinearity is moderate. What you must not do is conclude that there is no effect because the coefficients are not significant.

No, and this is a point that is often misunderstood. The predictions and the $R^{2}$ remain perfectly valid: the model predicts just as well with collinear variables. What is lost is the INTERPRETATION of the coefficients taken separately, that is, the ability to say which variable does what. If the goal is to predict a grade, multicollinearity is harmless; if the goal is to know whether it is study time or library attendance that matters, it makes the question unanswerable with these data.

Without a job, $e=0$: $\hat{y}=54.8+1.30x_{1}$. With a job, $e=1$: $\hat{y}=54.8+1.30x_{1}-3.0-0.80x_{1}=51.8+0.50x_{1}$. Two lines with neither the same intercept nor the same SLOPE: this is exactly what the interaction allows.

The coefficient $-0.80$ is the DIFFERENCE IN SLOPES between the two groups. Among students without a job, one more hour of study brings 1.30 points; among those who work, it brings only $1.30-0.80=0.50$ points. The return on an hour of study is therefore more than twice as low for a student who works, which is readily interpreted as fatigue and fragmented time.

At 10 hours: without a job, $54.8+13.0=67.8$; with a job, $51.8+5.0=56.8$. Gap: 11.0 points. At 20 hours: without a job, $54.8+26.0=80.8$; with a job, $51.8+10.0=61.8$. Gap: 19.0 points. The gap between the two groups is therefore not constant, it GROWS with hours of study, and that is the signature of an interaction.

The gap is $\left(54.8+1.30x_{1}\right)-\left(51.8+0.50x_{1}\right)=3.0+0.80x_{1}$. It exceeds 10 points when $0.80x_{1}>7.0$, that is $x_{1}>8.75$ hours. Beyond nine hours of study per week, the difference between the two groups therefore exceeds ten points, which is considerable.

Its scope changes. In the absence of an interaction, $-3.0$ would be the gap between the two groups, valid everywhere. With the interaction, it holds only for $x_{1}=0$, that is for a student who would not study at all: it is a SIMPLE EFFECT, conditional on a particular value of the other variable, and often on a value that makes no practical sense. This is why continuous variables are usually CENTRED before building an interaction: by subtracting the mean from $x_{1}$, the coefficient of $e$ becomes again the gap between groups for a student with AVERAGE study time, which is interpretable.

FALSE on two counts. First, the coefficient is an ASSOCIATION and not an effect: the data are cross-sectional, nobody was assigned to study one more hour, and the student who studies more may differ in motivation. Second, the constancy clause is missing. Correct statement: holding hours of paid work constant, one more weekly hour of study is associated with 1.05 more points on average.

FALSE. $R^{2}$ can only rise when variables are added, even pure noise: its increase proves nothing. The adjusted $R^{2}$, for its part, goes from 0.402 to 0.397, so it FALLS: the five variables cost more degrees of freedom than they bring in explanation. Correct statement: the addition degrades the model, and the two-variable model is preferable.

FALSE. The two coefficients are not in the same unit, points per hour of study and points per hour of work, and the size of their contribution also depends on the spread of each variable. Correct statement: to compare relative importance, you need the STANDARDIZED coefficients, obtained by multiplying each $b$ by $\dfrac{s_{x_{j}}}{s_{y}}$.

FALSE, there is no contradiction: this is the classic signature of MULTICOLLINEARITY. The explanatory variables resemble each other too much for the model to assign each a distinct share, so the standard errors of all the coefficients inflate while the overall fit stays good. Correct statement: the model predicts well and does not allow the effect to be attributed to one variable rather than another; you must examine the variance inflation factors and remove or combine the redundant variables.

FALSE. With three dummy variables, their sum equals 1 for every observation, so they exactly reproduce the constant: the system becomes undetermined. Correct statement: $k-1=2$ dummy variables are needed, the omitted category serving as the REFERENCE to which the two coefficients are compared.

$t$ for the high school average: $\dfrac{0.38}{0.061}\approx 6.230$. Hours of study: $\dfrac{0.84}{0.190}\approx 4.421$. Hours of work: $\dfrac{-0.31}{0.092}\approx -3.370$. Dummy variable: $\dfrac{-1.70}{1.45}\approx -1.172$. The first three exceed 1.971 in absolute value and are therefore significant; the fourth is not.

With $n=220$ and $k=4$: $R_{adj}^{2}=1-\left(1-0.52\right)\dfrac{219}{215}=1-0.48\times 1.018605=1-0.488930\approx 0.511$. The correction costs less than one point: with 220 observations for 4 variables, it is almost painless.

$\hat{y}=41.2+0.38\times 78+0.84\times 14-0.31\times 12-1.70\times 1=41.2+29.64+11.76-3.72-1.70=77.18$. The prediction applies to the mean of students with this profile; the residual standard deviation, which would be needed to give an individual prediction interval, is not provided.

You can NOT conclude that being first in the family to attend postsecondary education is unrelated to the grade. You conclude that the data do not allow it to be distinguished from zero HOLDING THE OTHER VARIABLES CONSTANT. Interval: $-1.70\pm 1.971\times 1.45=-1.70\pm 2.858$, that is $\left[-4.558, 1.158\right]$. The interval leaves open a penalty of more than four and a half points, which is far from negligible: the result is imprecise, not null. Note also that the coefficient is controlled for the high school average, which is itself already affected by social origin: a good part of the raw gap has therefore been absorbed upstream.

“The regression of the term average on four variables explains 52% of the variance, $R_{adj}^{2}=0.51$, on 220 students. Holding the other variables constant, each point of high school average is associated with 0.38 more points, $t=6.23$, p less than 0.001; each weekly hour of study with 0.84 more points, $t=4.42$, p less than 0.001; each hour of paid work with 0.31 fewer points, $t=-3.37$, p less than 0.01. Being first in the family to attend postsecondary education is not associated with a significant difference once these variables are controlled, $b=-1.70$; confidence interval from $-4.56$ to $1.16$, which does not rule out an appreciable penalty. Since the data are cross-sectional, these coefficients describe associations and not causal effects.”

Each additional dollar of annual income is associated with 0.0004 more points, holding the other variables constant. Brought back to a readable unit, 10,000 more dollars of annual income correspond to $10\,000\times 0.0004=4.0$ points of average. The gap between a household at 40,000 dollars and a household at 100,000 dollars is therefore 24 points, which is enormous. The writing rule is always to express a coefficient for a REALISTIC change in the variable, never in its raw unit when that unit is a dollar or a second.

No, not at all. The size of a coefficient depends entirely on the unit chosen: expressing income in thousands of dollars would give $b=0.4$, in tens of thousands $b=4$, with nothing else changing. A coefficient is therefore never judged by its apparent size, but relative to the standard deviation of the variable, or converted into an interpretable change as in the previous question.

First criticism, failure to reject read as proof of absence: with $p=0.31$, the data are simply compatible with zero, and the confidence interval, not published, probably contains appreciable values. Second criticism, the forgotten constancy clause: the number of books has no OWN effect once income and parents' education are controlled, which is very different from having no relationship with achievement. Taken alone, it almost certainly would have one.

These three variables are strongly correlated with each other: income, parents' education and the number of books largely measure the same socioeconomic capital. This is therefore a situation of MULTICOLLINEARITY, which inflates the standard errors and makes the split of the effect among the three arbitrary. That income stands out and books do not probably has no substantive meaning: another specification, or another sample, could reverse the roles. This is precisely the case where the variance inflation factors should be published and, no doubt, a single index built.

“Each 10,000 dollars of annual household income is associated with 4.0 more points of average, holding parents' education and the number of books constant, $b=0.0004$; $p=0.004$. The number of books at home is not associated with a significant difference once these variables are controlled, $b=0.011$; $p=0.31$, which does not mean that it is unrelated to achievement: since the three variables largely measure the same socioeconomic capital, the model cannot assign them distinct shares, and the cross-sectional design rules out any causal interpretation.” Two decisive corrections: the coefficient is brought back to a readable change, and the lack of significance is presented as an indeterminacy of the model, not as a result.