Advanced Quantitative Methods 360-301 • CEGEP, Montreal

Revision sheet: multiple regression (360-301)

This sheet covers the richest chapter of the Advanced Quantitative Methods course, the one you meet again in every economics, sociology and psychology article. It does not repeat the algebra, it gives the eight places where a correct software output produces a false sentence, and the exact wording that repairs each one.

It is written for Social Science students at CEGEP taking the 360-301 course. The corrected exercise set on the same chapter goes through each step, from reading an output to diagnosing with residuals.

Mark this sheet as read or add it to your favourites: a free account, no password, keeps your read sheets and favourites from one visit to the next, tells you which chapter to tackle next and lets you ask for the method or exercise you are missing. Create your space, an email is enough.

The thread of the chapter

A multiple regression coefficient is PARTIAL: it holds “with the other variables held constant”. That clause is not decoration, it is the definition of the number, and half the mistakes in this chapter come from forgetting it.

This chapter is part of 360-301, Advanced Quantitative Methods

What is on this sheet

1. The essentials
2. The mistakes that cost marks
3. Which method to choose
4. How the answer is expected to be written
5. Check before you hand in
6. The typical problem, taken apart
7. Learn by heart
8. Frequently asked questions

Topics covered

Partial regression coefficient
Standardized coefficient
Coefficient of determination
Adjusted R squared
Dummy variable
Reference category
Interaction term
Residual
Heteroscedasticity
Multicollinearity
Variance inflation factor
Standard error of a coefficient

The essentials

What each coefficient says, and what it does not

• A multiple regression coefficient is PARTIAL: it gives the change in $y$ for one unit of $x_{j}$ , THE OTHER VARIABLES STAYING CONSTANT.
• The raw slope and the partial slope differ as soon as the explanatory variables are correlated with each other.
• Each row of an output tests $H_{0}:\beta_{j}=0$ with $t=\dfrac{b}{SE}$ , on $n-k-1$ degrees of freedom. Interval: $b\pm t_{crit}\times SE$ .
• STANDARDIZED coefficients: $\beta_{j}=b_{j}\times \dfrac{s_{x_{j}}}{s_{y}}$ . They are the only ones comparable across variables with different units.

Within each paid-work group, the slope is

1.05

: that is the partial coefficient. The raw line, in purple, mixes the two groups and climbs faster.

The clause “holding the other variables constant” must appear VERBATIM in every interpretation: it is a marking point for each coefficient.

Model quality, and the temptation to add

• $R^{2}$ can NEVER decrease when a variable is added, even a totally useless one: this is an algebraic property, not a sign of quality.
• $R_{adj}^{2}=1-\left(1-R^{2}\right)\dfrac{n-1}{n-k-1}$ penalizes each added variable. It can FALL, and that fall is the useful signal.
• A variable whose coefficient is not significant almost always makes the adjusted $R^{2}$ fall.
• $R^{2}$ measures how well the model fits the data, never how correct its specification is nor the existence of a cause.

The best model is the two-variable one:

R^{2}

keeps rising beyond it, but the adjusted value falls back, which says that the later variables bring only noise.

Reporting the adjusted $R^{2}$ next to $R^{2}$ is expected as soon as there is more than one explanatory variable.

Dummy variables, and interaction

• A qualitative variable with $k$ categories takes $k-1$ DUMMY variables, the omitted category serving as the REFERENCE. Three dummy variables for three categories make the model undetermined.
• The coefficient of a dummy variable is a vertical SHIFT from the reference, never a slope: the lines stay PARALLEL.
• An INTERACTION term $x_{1}\times e$ changes this: its coefficient is a DIFFERENCE IN SLOPES.
• With an interaction in the model, the simple coefficients become conditional on the zero value of the other variable.

On the left, the dummy variable only translates the line; on the right, the interaction term makes the two slopes diverge, and the gap between A and B then depends on

x

Naming the reference category in the interpretation is mandatory: “ $2.4$ points more THAN Natural Science students”.

Residuals, and what they reveal

• $e_{i}=y_{i}-\hat{y}_{i}$ . Their sum is zero and so is their correlation with each predictor: these two facts are AUTOMATIC and teach nothing.
• It is the SHAPE of the cloud you examine, by plotting the residuals against the predicted values.
• FUNNEL-shaped cloud: heteroscedasticity, which distorts the standard errors without biasing the coefficients.
• U-shaped cloud: nonlinearity, to be fixed with a quadratic term or a logarithm.

The healthy cloud: a horizontal band, with no funnel and no curvature, centred on zero. This is the shape to compare with the two defects of the pitfall.

$VIF=\dfrac{1}{1-R_{j}^{2}}$ : above $5$ , multicollinearity inflates the standard errors, leaves predictions intact and ruins the interpretation of the separate coefficients.

The mistakes that cost marks

These are the errors I correct most often in session. Each one costs marks on a paper, even when the reasoning behind it is right.

1. Reading a partial coefficient as a raw slope

1 point per interpreted coefficient, and the marker counts them

What not to write

“ $b=1.80$ : each additional hour of study adds $1.8$ points to the grade.”

What to write

“HOLDING hours of paid work and program CONSTANT, one more hour of study is associated with $1.8$ more points.”

Why: Without the clause, the sentence describes the raw correlation, which is worth something else as soon as the explanatory variables are correlated with each other. The model never measured that.

2. Comparing two raw coefficients with different units

1.5 points, and a false ranking of the predictors

What not to write

“ $1.80$ against $-0.35$ : study weighs five times more than paid work.”

What to write

“The two variables have neither the same unit nor the same spread. Standardized, the coefficients are $0.63$ and $-0.26$ : study weighs about $2.4$ times more.”

Why: A coefficient is a change in $y$ per unit of $x$ . Changing the unit of $x$ , from hours to minutes for example, divides the coefficient by sixty without changing the model at all.

3. Adding variables to push the coefficient of determination up

2 points, and an overloaded final model

What not to write

“By adding six variables, $R^{2}$ goes from $0.46$ to $0.49$ : the model improves.”

What to write

“ $R^{2}$ cannot fall, it is an algebraic property. The ADJUSTED $R^{2}$ goes from $0.437$ to $0.390$ : the model got worse.”

R^{2}

, in blue, rises with every added variable; the adjusted

R^{2}

, in orange, peaks at two variables and then falls back: the rest is only noise.

Why: Each added variable picks up a bit of noise, which always improves the fit on the sample and never the prediction outside it.

4. Using as many dummy variables as categories

2 points, and the software refuses to estimate the model

What not to write

“Three programs, so three dummy variables in the model.”

What to write

“Only two dummy variables: the third category serves as the REFERENCE. With three dummy variables, their sum is always $1$ and the model becomes undetermined.”

Why: The constant is already a column of $1$ s: three dummy variables reproduce it exactly, so the inverse needed for the computation no longer exists. This is the dummy variable trap.

5. Reading the coefficient of a dummy variable as a slope

1.5 points, and a sentence that makes no sense

What not to write

“ $b_{3}=2.40$ : the grade rises by $2.4$ points per unit of program.”

What to write

“Social Science students obtain on average $2.4$ points more THAN Natural Science students, the reference category, other variables held constant. It is a shift, not a slope.”

Two parallel lines: the dummy variable shifts the line by

2.4

points, in purple, without touching the slope.

Why: A dummy variable takes only the values $0$ and $1$ : “one more unit” means “moving from one program to the other”. Without an interaction term, the two lines stay parallel.

6. Checking the correlation of the residuals instead of their shape

1.5 points, and a diagnosis with no content

What not to write

“The residuals are correlated with no predictor: the model is validated.”

What to write

“This lack of correlation is AUTOMATIC, like the zero sum of the residuals. I examine the SHAPE of the cloud: funnel or curvature.”

Both clouds have zero correlation with the predictor and a zero mean: it is their SHAPE, funnel on the left and curvature on the right, that exposes the problem.

Why: Least squares builds the residuals orthogonal to the predictors: checking it amounts to checking that the software did its job.

7. Dropping a variable because of multicollinearity when the goal is prediction

1 point, and an unjustified modelling decision

What not to write

“The inflation factor is $6$ : I drop the variable, the model will be better.”

What to write

“Multicollinearity inflates the standard errors and ruins the interpretation of the SEPARATE coefficients. It leaves the PREDICTIONS intact: if the goal is to predict, we keep the variable.”

The two predictors are almost aligned: the model knows what they contribute TOGETHER, not how to split it between them.

Why: Two strongly correlated variables carry almost the same information: the model knows how much they contribute TOGETHER and can no longer tell them apart. It is a problem only if you want to tell them apart.

8. Interpreting the simple coefficients in the presence of an interaction

2 points, and the interpretation of the whole model

What not to write

“The model contains $x_{1}$ , the dummy variable and their product: the coefficient of $x_{1}$ gives the effect of hours of study.”

What to write

“With an interaction, the coefficient of $x_{1}$ is the effect of hours of study FOR THE REFERENCE CATEGORY only. For the other group, the interaction coefficient must be added.”

Why: The interaction coefficient is a DIFFERENCE IN SLOPES: as soon as it is present, each simple coefficient becomes conditional on the zero value of the other variable.

Which method to choose

How to read a regression output, row by row

Go through the output in this order: overall quality, then each coefficient, then the diagnostics.

If the question is about the overall quality of the model → read $R^{2}$ AND adjusted $R^{2}$ , and comment on the gap between them

Example: $0.42$ against $0.389$ with three variables
If the question is about a specific coefficient → compute $t=\dfrac{b}{SE}$ , compare it with the critical value on $n-k-1$ degrees of freedom, then interpret with the clause “holding the other variables constant”

Example: $b=1.80$ , $SE=0.45$ give $t=4.00$
If the question asks which variable matters most → switch to the STANDARDIZED coefficients, never the raw coefficients

Example: $0.63$ against $-0.26$
If the variable is qualitative → identify the REFERENCE category and read the coefficient as a shift from it

Example: $+2.4$ points relative to Natural Science
If the question asks for a prediction → substitute the values of all the variables, dummy variables included, and stay within the range of the data

Example: 12 hours of study, 15 hours of work, dummy variable at 1

A nonsignificant variable is not dropped automatically: if it serves as a control for a confounding variable, it stays in the model.

Which diagnosis does this residual cloud call for?

Plot the residuals against the predicted values and look at the SHAPE, not the correlation.

If the spread grows from left to right → HETEROSCEDASTICITY: the coefficients stay good, the standard errors are wrong, so the tests are too

Example: incomes, whose variability grows with the level
If the cloud draws a curve, U-shaped or bell-shaped → NONLINEARITY: add a quadratic term or switch to a logarithm

Example: diminishing returns on hours of study
If one point lies very far out in $x$ → LEVERAGE POINT: redo the analysis with and without it, and publish both

Example: a student at 45 hours of study per week
If the residuals follow one another over time → AUTOCORRELATION: the standard errors are underestimated, a time series model is needed

Example: monthly data
If the cloud is a rectangle with no structure → nothing to report: this is exactly what you want to see

Example: a horizontal band of constant thickness

None of these diagnoses can be read from $R^{2}$ : a misspecified model can easily show a high $R^{2}$ .

How the answer is expected to be written

A marker ticks steps. Here they are in order, with the concluding sentence expected word for word.

Interpreting a multiple regression coefficient

When to use it: Whenever a question asks what a coefficient in an output means.

1 Name the variable, with its unit.
2 Give the direction and the value: “one more unit is associated with so many points more, or fewer”.
3 Add the clause “holding the other variables constant”, naming at least one of those variables.
4 Give the test: $t=\dfrac{b}{SE}$ , the comparison with the threshold, and the decision.
5 Finish with the confidence interval, in the unit of the variable.

Concluding sentence

“Holding hours of paid work and program constant, one additional weekly hour of study is associated with $1.80$ more points on the term grade. With $t=4.00$ on $56$ degrees of freedom, this coefficient is significant at the 5% level, and its confidence interval runs from $0.90$ to $2.70$ points.”

The trap: Writing “increases the grade”: the model measures an association, and the causal verb costs the point.

Marking: 1 mark for the value and direction, 1 mark for the clause, 1 mark for the test, 1 mark for the interval.

Deciding whether a variable deserves to stay in the model

When to use it: When the statement compares two models, or asks whether a variable is useful.

1 Compare the ADJUSTED $R^{2}$ of the two models, never the $R^{2}$ .
2 Look at the significance of the coefficient of the variable in question.
3 Check its variance inflation factor: above $5$ , point out multicollinearity.
4 Say what the model is for: to PREDICT, you keep it; to INTERPRET separate effects, you decide.

Concluding sentence

“The adjusted $R^{2}$ goes from $0.437$ to $0.390$ when the six variables are added, and none of their coefficients is significant: the two-variable model is preferable.”

The trap: Dropping a nonsignificant control variable: it is in the model to neutralize a confounding variable, not to be significant.

Marking: 1 mark for the comparison of the adjusted $R^{2}$ values, 1 mark for the significance, 1 mark for the conclusion justified by the purpose of the model.

Check before you hand in

Five minutes of checking recover more marks than one more problem started in a hurry.

Adjusted R squared never exceeds R squared

$R_{adj}^{2}\le R^{2}$ always, and the gap grows with the number of variables. An adjusted value that is higher signals a formula error.

An $R^{2}$ of $0.42$ with an adjusted value of $0.45$ comes from an $n-k-1$ written upside down.
The t is recomputed in one division

Divide the coefficient by its standard error and compare with the printed $t$ : it is the fastest check of a badly copied software output.

$\dfrac{1.80}{0.45}=4.00$ : the row is consistent.
The interval and the significance agree

A coefficient significant at 5% has a 95% confidence interval that does not contain zero. A contradiction exposes an error.

$b=2.40$ with $t=1.50$ : the interval MUST contain zero, and it does.
The prediction stays within the range of the data

Check that the substituted values look like those of the sample: a prediction outside the range is not a prediction.

Predicting the grade of a student at $60$ hours of study when the observed maximum is $25$ : the model has learned nothing about that zone.

The typical problem, taken apart

Modelling the term grade, from a software output to the prediction

On $60$ students, the term grade is regressed on weekly hours of study, hours of paid work and a dummy variable equal to $1$ for Social Science students.

Output: constant $52.00$ ; hours of study $b=1.80$ and $SE=0.45$ ; paid work $b=-0.35$ and $SE=0.12$ ; dummy variable $b=2.40$ and $SE=1.60$ . $R^{2}=0.42$ .

Standard deviations: hours of study $4.2$ ; paid work $9.0$ ; grade $12.0$ .

Step 1

$t_{1}=\dfrac{1.80}{0.45}=4.00$ ; $t_{2}=\dfrac{-0.35}{0.12}=-2.92$ ; $t_{3}=\dfrac{2.40}{1.60}=1.50$ . Degrees of freedom: $60-3-1=56$ , critical value $2.00$ .

Why

The first two variables are significant, the third is not. Say so before interpreting anything: a nonsignificant coefficient is not interpreted as an established effect.

Step 2

Interpretation: holding paid work and program constant, one more hour of study is associated with $1.80$ more points; holding hours of study and program constant, one more hour of paid work is associated with $0.35$ fewer points.

Why

The clause changes the meaning of the sentence: without it, you would assert a raw correlation that the model never estimated.

Step 3

The dummy variable: Social Science students obtain $2.40$ more points than the reference category, but $t=1.50$ does not exceed $2.00$ : the gap is not significant.

Why

It is a SHIFT, not a slope: without an interaction term, the lines of the two programs stay parallel.

Step 4

$R_{adj}^{2}=1-\left(1-0.42\right)\dfrac{59}{56}\approx 0.389$ .

Why

The gap of three hundredths puts a price on the three variables. It is this number, not $R^{2}$ , that is compared from one model to another.

Step 5

Standardized coefficients: $\beta_{1}=1.80\times\dfrac{4.2}{12.0}=0.63$ and $\beta_{2}=-0.35\times\dfrac{9.0}{12.0}=-0.26$ .

Why

The raw coefficients suggested a ratio of $5$ , the standardized ones give $2.4$ : paid work varies much more from one student to another, which offsets its small coefficient.

Step 6

Prediction for $12$ hours of study, $15$ hours of work, dummy variable at $1$ : $52+1.80(12)-0.35(15)+2.40=70.75$ .

Why

You substitute ALL the variables, dummy variable included. The check is immediate: the predicted value must fall within the range of the observed grades.

The conclusion, written out

“The model explains $42$ % of the variance of the grades, $38.9$ % after adjustment. Hours of study and hours of paid work are both significantly associated with the grade, study weighing about $2.4$ times more once the coefficients are standardized; the gap between programs, $2.4$ points, is not significant.”

The classic mistake on this problem: Concluding that study counts five times more than paid work, from the raw coefficients: the two variables have neither the same unit nor the same spread.

Learn by heart

• A multiple regression coefficient is PARTIAL: “holding the other variables constant” is part of its definition.
• $t=\dfrac{b}{SE}$ on $n-k-1$ degrees of freedom, and the interval $b\pm t_{crit}\times SE$ .
• $R^{2}$ never falls when a variable is added. $R_{adj}^{2}$ can fall, and it is the one you compare.
• $k$ categories, $k-1$ dummy variables, one REFERENCE category. The coefficient is a SHIFT, not a slope.
• Standardized coefficients $\beta=b\dfrac{s_{x}}{s_{y}}$ : the only ones comparable across variables with different units.
• Residuals are examined by their SHAPE: funnel for heteroscedasticity, U for nonlinearity.
• $VIF=\dfrac{1}{1-R_{j}^{2}}$ , serious above $5$ : it inflates the standard errors and leaves predictions intact.

Frequently asked questions

What does a partial coefficient mean in multiple regression?

It gives the change in the response variable for one more unit of the variable concerned, with all the other variables in the model held constant. That clause is part of the definition of the number: without it, you are describing the raw correlation, which is worth something else as soon as the explanatory variables are correlated with each other.

Why use adjusted R squared rather than R squared?

Because R squared can never decrease when a variable is added, even a totally useless one: it is an algebraic property, not a sign of quality. The adjusted R squared penalizes each added variable, so it falls when the new variable brings only noise. That is why it is the one you compare from one model to another.

How many dummy variables does a qualitative variable need?

One fewer than the number of categories. Three programs need two dummy variables, the third category serving as the reference. With as many dummy variables as categories, their sum reproduces the constant of the model exactly, which then becomes impossible to estimate. The coefficient you get reads as a gap from the reference category.

How can you tell whether a regression model has a problem?

Plot the residuals against the predicted values and look at the shape of the cloud. A horizontal band of constant thickness is what you want to see. A funnel signals heteroscedasticity, which distorts the tests. A U-shaped curve signals a nonlinear relationship, to be fixed with a quadratic term or a logarithm.

What is multicollinearity and when should you worry about it?

It means that two explanatory variables carry almost the same information. It is measured by the variance inflation factor, serious above five. It inflates the standard errors and makes the separate coefficients unreliable, but it leaves the predictions intact. It is therefore a problem only if you want to interpret each variable on its own.

Practise it

Corrected exercises: Multiple regression, 360-301

A method is proved on a paper, not on a sheet. The set for the same chapter takes each of these traps into a problem, with the solution written out step by step.

10 corrected exercises
100 points
150 minutes

Do the exercises

Something missing from this sheet? A method, a trap, an exercise to practise it, a mistake you spotted: tell me, I read every request.

Sending goes through your free space: that is how I can answer you and tell you when the exercise is added. An email is enough, no password. Create your space. Rather not create an account? Write to me directly.

Previous sheet One-way analysis of variance Next sheet Nonparametric tests

© Ahmed Squalli Houssaini. Revision sheet published at www.letuteurscientifique.ca/en/fiches/cegep-aqm-multiple-regression. Free for personal and classroom use; republishing it elsewhere requires written permission (legal notice).

Looking for a 360-301 tutor in Montreal?

Get in touch for a first session. 360-301 arrives at the time of the end-of-program research project, and it is the course where a method error costs you the whole analysis chapter.

Book a session WhatsApp

Site by Studio Squalli

Revision sheet: multiple regression (360-301)

The essentials

What each coefficient says, and what it does not

Model quality, and the temptation to add

Dummy variables, and interaction

Residuals, and what they reveal

The mistakes that cost marks

1. Reading a partial coefficient as a raw slope

2. Comparing two raw coefficients with different units

3. Adding variables to push the coefficient of determination up

4. Using as many dummy variables as categories

5. Reading the coefficient of a dummy variable as a slope

6. Checking the correlation of the residuals instead of their shape

7. Dropping a variable because of multicollinearity when the goal is prediction

8. Interpreting the simple coefficients in the presence of an interaction

Which method to choose

How to read a regression output, row by row

Which diagnosis does this residual cloud call for?

How the answer is expected to be written

Interpreting a multiple regression coefficient

Deciding whether a variable deserves to stay in the model

Check before you hand in

The typical problem, taken apart

Modelling the term grade, from a software output to the prediction

Learn by heart

Frequently asked questions

What does a partial coefficient mean in multiple regression?

Why use adjusted R squared rather than R squared?

How many dummy variables does a qualitative variable need?

How can you tell whether a regression model has a problem?

What is multicollinearity and when should you worry about it?

Corrected exercises: Multiple regression, 360-301

See also

Looking for a 360-301 tutor in Montreal?