Advanced Statistics 201-337 • CEGEP, Montreal

Revision sheet: correlation and linear regression (201-337)

Regression is the chapter where the calculator always gives an answer, even when the question makes no sense. It fits a line to a curved scatter plot, it computes a coefficient on eleven points of which a single one decides everything, and it flags nothing.

This sheet gathers what the machine does not do: look at the scatter plot before, read the residuals after, refuse to extrapolate, and tell an association from a cause. These are exactly the questions that earn marks on the 201-337 exam. It is written for CEGEP students; once it is read, the corrected exercise set on the same chapter puts every reflex to the test.

Mark this sheet as read or add it to your favourites: a free account, no password, keeps your read sheets and favourites from one visit to the next, tells you which chapter to tackle next and lets you ask for the method or exercise you are missing. Create your space, an email is enough.

The thread of the chapter

A correlation coefficient is never read on its own: you need the scatter plot BEFORE computing it, the residuals AFTER computing it, and the question of causation remains wide open in every case.

This chapter is part of 201-337, Advanced Statistics

What is on this sheet

1. The essentials
2. The mistakes that cost marks
3. Which method to choose
4. How the answer is expected to be written
5. Check before you hand in
6. The typical problem, taken apart
7. Learn by heart
8. Frequently asked questions

Topics covered

Scatter plot
Linear correlation coefficient
Least-squares line
Mean point
Coefficient of determination
Residuals
Leverage point
Outlier
Extrapolation
Confounding variable
Semi-logarithmic linearization
Standard error of the estimate

The essentials

The scatter plot first, the calculation second

• Always look at the SHAPE of the scatter plot before computing: linearity, curvature, isolated values, constant spread or not.
• A coefficient computed on a nonlinear scatter plot has no meaning, even when it is high.
• Two scatter plots of completely different shapes can give exactly the same $r$ and the same line.

Two scatter plots with the same coefficient

r=0.82

and the same line: on the left the relationship is linear, on the right it is clearly curved and the model does not fit.

This is the one rule of the chapter that cannot be sidestepped: the calculator returns a number in every case, including where linear regression is out of place.

The working sums and the line

• $S_{xy}=\sum xy-\dfrac{(\sum x)(\sum y)}{n}$ , $S_{xx}=\sum x^{2}-\dfrac{(\sum x)^{2}}{n}$ , $S_{yy}=\sum y^{2}-\dfrac{(\sum y)^{2}}{n}$ .
• $r=\dfrac{S_{xy}}{\sqrt{S_{xx}S_{yy}}}$ , always between $-1$ and $1$ : the SIGN gives the direction, the ABSOLUTE VALUE the strength.
• Least-squares line $y=a+bx$ : $b=\dfrac{S_{xy}}{S_{xx}}$ , then $a=\overline{y}-b\overline{x}$ .
• It ALWAYS passes through the mean point $(\overline{x}, \overline{y})$ , which gives a free check.

The orange segments are the VERTICAL deviations

y_{i}-\hat{y_{i}}

that least squares makes as small as possible; the line passes through the mean point

(3.5, 5.5)

(green).

“Least squares” refers to the sum of the squares of the VERTICAL deviations. That is why the line of $y$ on $x$ is not the line of $x$ on $y$ : the deviations minimized are not the same.

r, r squared, and what each one measures

• $r^{2}$ is the proportion of the variation in $y$ explained by the linear relationship with $x$ , expressed as a percentage.
• $r=0.80$ gives $r^{2}=0.64$ : $64\%$ of the variation explained, not $80\%$ .
• Residuals $e_{i}=y_{i}-\hat{y_{i}}$ : their sum is zero by construction, so it proves nothing. It is their PLOT that informs.
• Standard error of the estimate: $s_{e}=\sqrt{\dfrac{\sum e_{i}^{2}}{n-2}}$ , with $\sum e_{i}^{2}=S_{yy}(1-r^{2})$ .

The sign of

r

gives the direction, its absolute value the strength;

r^{2}

tells the share of the variation in

y

explained. A scatter plot with no slope gives

r

close to

0

The significance of $r$ is tested with $t=\dfrac{r\sqrt{n-2}}{\sqrt{1-r^{2}}}$ on $n-2$ degrees of freedom. The same $r=0.50$ is not significant for $10$ individuals and is significant for $30$ .

The four explanations of a correlation

• $x$ causes $y$ ; or $y$ causes $x$ ; or a third, CONFOUNDING variable causes both; or it is chance on a small sample.
• None of these four is ruled out by the calculation: only an experiment with random assignment makes it possible to conclude that there is a cause.
• EXTRAPOLATING, that is, predicting outside the range of the observed $x$ values, is never justified by the model.

The same correlation between

x

and

y

admits four readings; the calculation does not decide between them, only an experiment with random assignment can.

The causation question is often worth as many marks as the whole regression. Naming a plausible, concrete confounding variable earns more than a general sentence about caution.

The mistakes that cost marks

These are the errors I correct most often in session. Each one costs marks on a paper, even when the reasoning behind it is right.

1. Computing a coefficient without having looked at the scatter plot

3 points, the whole question rests on this diagnosis

What not to write

“ $r=0.82$ , so the relationship between the two variables is strong and linear.”

What to write

“The scatter plot is clearly curved: the coefficient $0.82$ does not describe the relationship; the data must be linearized or another model chosen.”

Seven points exactly on

y=x^{2}

: the link is total, and yet

r=0

. The coefficient measures ALIGNMENT, not the existence of a relationship.

Why: The coefficient measures alignment, not the existence of a relationship. A perfect parabola can give a zero $r$ even though the link is total.

2. Reading r as a percentage

1.5 points, every time an interpretation is asked for

What not to write

“ $r=0.80$ , so $80\%$ of the variation in $y$ is explained by $x$ .”

What to write

“It is $r^{2}=0.64$ that is read as a percentage: $64\%$ of the variation in $y$ is explained.”

With

r=0.80

, the share explained is

r^{2}=0.64

: the blue band stops at

64\%

, not at the red mark at

80\%

Why: Only the square of the coefficient can be read as a proportion of variance. The gap is all the larger when $r$ is middling: $0.5$ explains only $25\%$ .

3. Predicting outside the observed range

1.5 points, and it is often the last question of the problem

What not to write

“The line gives $\hat{y}=2x$ , so for $x=50$ we predict $100$ .”

What to write

“The observed $x$ values run from $1$ to $6$ : beyond that, the model has nothing to say, and the prediction is not justified.”

The line

\hat{y}=2x

describes the observed range

[1, 6]

(green); extended in red, it rests on no data at all.

Why: Nothing guarantees that the relationship continues: fitted lines describe a domain, they do not predict the world. Linear growth always ends up hitting a physical limit.

4. Drawing a cause from a correlation

2 points, and it is the most heavily weighted question of the chapter

What not to write

“Ice cream sales and drownings are correlated, so ice cream promotes drowning.”

What to write

“A third variable, temperature, explains both: it is a confounding variable, and the correlation establishes no cause.”

Temperature pushes both up (green arrows); between ice cream and drownings there is only a correlation (red), no arrow.

Why: Four explanations remain possible after the calculation, and the regression does not decide between them. Only an experiment with random assignment could.

5. Deleting the inconvenient point

2 points for method, and the whole answer loses its credibility

What not to write

“This point makes $r^{2}$ drop from $0.99$ to $0.65$ : I remove it and recompute.”

What to write

“I investigate where it comes from, then I publish the analysis WITH and WITHOUT this point, pointing out the difference.”

The eight aligned points give a slope close to

1

; the single point far away in

x

cuts it by half, although the central cloud has not changed.

Why: A point far away in $x$ , called a leverage point, can halve the slope while leaving $r$ high. It may be a data-entry error just as well as the most informative observation in the data set.

6. Swapping the roles of x and y

1.5 points, and a wrong prediction in the reverse direction

What not to write

“The line of $y$ on $x$ is $y=2x$ , so the line of $x$ on $y$ is $x=\dfrac{y}{2}$ .”

What to write

“These are two different regressions: the first minimizes the vertical deviations, the second the horizontal deviations, and their slopes are reciprocals only if $r^{2}=1$ .”

The line of

y

x

(blue) minimizes the vertical deviations, the line of

x

y

(orange) the horizontal deviations: two distinct lines, crossing at the mean point.

Why: The response variable is the one whose deviations are minimized. Choosing which one goes on the $x$ axis is therefore not a writing convention, it is a modelling decision.

7. Trusting the coefficient without looking at the residuals

2 points, and the model proposed is the wrong one

What not to write

“ $r^{2}=0.92$ , so the linear model is good.”

What to write

“The residual plot forms a clear curve: the relationship is not linear, despite a high $r^{2}$ .”

The residuals form a parabola instead of a shapeless cloud: the linear model systematically misses the ends and the centre, in opposite directions.

Why: A residual cloud must have no structure. A curvature signals a nonlinear relationship, a funnel a spread that changes with $x$ : two defects that no coefficient reveals.

8. Forgetting that the line passes through the mean point

2 points when the error slips through, none when the check is made

What not to write

“ $b=2$ and $a=1.5$ , on to the next question.”

What to write

“Check: $\hat{y}(\overline{x})=a+b\overline{x}$ must equal $\overline{y}$ . Here $1.5+2\times 3.5=8.5\neq 7$ , so $a$ is wrong.”

The correct line

\hat{y}=2x

passes through

(3.5, 7)

; with

a=1.5

, it passes at

8.5

, above the mean point (red): the error shows up in one line.

Why: The intercept is derived from the mean point: the check is therefore exactly the reverse of its calculation, and it catches every sign error.

9. Confusing the strength of a relationship with how solid the evidence is

2 points on the test question

What not to write

“ $r=0.50$ for $10$ individuals: the relationship is significant, it is moderate.”

What to write

“For $10$ individuals, $t=\dfrac{0.50\sqrt{8}}{\sqrt{0.75}}\approx 1.63$ : not significant. For $30$ , the same $r$ becomes significant.”

Why: Significance depends on $n$ as much as on $r$ . It tells whether the relationship is established, never whether it is strong, and even less whether it is useful.

Which method to choose

What to do depending on the shape of the scatter plot

Describe the scatter plot in one sentence before touching the calculator.

The scatter plots the tree decides between: alignment, growth that accelerates (exponential), a curve that bends over (power), an isolated point far away in

x

(red).

If the points line up with no residual structure → ordinary linear regression, then $r$ , $r^{2}$ and the residuals

Example: $\hat{y}=2x$ with $r^{2}=0.95$
If the growth accelerates and the gaps widen as $y$ grows → exponential model $y=ae^{bx}$ : regress $\ln y$ on $x$ , on a semi-logarithmic scale

Example: population, invested capital, radioactive decay
If the relationship looks like a power law, with no preferred scale → model $y=ax^{b}$ : regress $\ln y$ on $\ln x$ , the slope giving the exponent directly

Example: scaling law between body mass and metabolism
If one point is isolated, very far away in $x$ → recompute with and without it, compare the two slopes, investigate where it comes from

Example: a slope that goes from $1.0$ to $0.5$
If the points form several distinct groups → do not regress on the whole set: the overall line describes none of the groups

Example: two workshops with different machine settings

Every linearization transforms $y$ , so the $r^{2}$ obtained refers to the TRANSFORMED data. Comparing a semi-log $r^{2}$ with a raw $r^{2}$ makes no sense.

Reading a residual plot

Plot the residuals against $x$ and look for any remaining shape.

Residuals against

x

: the first plot has no shape and validates the model; a curve, a funnel or an isolated residual (red) signal a defect.

If a cloud with no structure, spread evenly on both sides of zero → the linear model fits, $r^{2}$ can be interpreted

Example: residuals scattered at random on both sides of zero
If the residuals form a curve → the relationship is not linear: linearize or change the model

Example: negative residuals in the centre, positive at the ends
If the residuals open out into a funnel as $x$ grows → the spread is not constant: the prediction intervals no longer hold

Example: incomes whose spread grows with their level
If one residual dwarfs all the others → outlier: investigate, then publish with and without it

Example: a residual of $8$ when the others stay within $\pm 1$

The sum of the residuals is zero by construction: presenting it as proof that the model is good is a classic error; it only proves that $a$ was computed correctly.

How the answer is expected to be written

A marker ticks steps. Here they are in order, with the concluding sentence expected word for word.

Writing up a complete regression

When to use it: The question gives a table of pairs and asks for the line, the coefficient and a prediction.

1 Describe the scatter plot in one sentence: shape, direction, whether or not there are isolated points.
2 Write the three working sums $S_{xy}$ , $S_{xx}$ , $S_{yy}$ with their numerical values.
3 Compute $b$ , then $a$ from the mean point, and write the equation with the units of the context.
4 Give $r$ , then $r^{2}$ as a percentage, stating that it is the square that is interpreted.
5 Predict only inside the observed range, and write down that reservation.

Concluding sentence

“ $b=\dfrac{35}{17.5}=2$ and $a=7-2\times 3.5=0$ , so $\hat{y}=2x$ ; $r=0.97$ and $r^{2}=0.95$ : $95\%$ of the variation in $y$ is explained by the linear relationship.”

The trap: Giving the prediction before checking that the requested $x$ lies within the observed range.

Marking: 1 point for describing the scatter plot, 2 points for the sums, 1 point for the equation, 1 point for interpreting $r^{2}$ , 1 point for the reservation about the prediction.

Writing a conclusion about causation

When to use it: The question asks whether one variable “causes” the other, or asks you to comment on a newspaper claim.

1 Restate what the calculation establishes: a linear association, with its strength.
2 List the competing explanations, NAMING a plausible confounding variable from the context.
3 Say what it would take to conclude that there is a cause: an experiment with random assignment.
4 Conclude without denying the relationship or turning it into a cause.

Concluding sentence

“The data establish a strong association, but temperature may explain both the sales and the drownings: only a controlled experiment would allow us to conclude that there is a cause-and-effect link.”

The trap: Writing “correlation is not causation” without naming a confounding variable: the general sentence does not earn the mark.

Marking: 1 point for the association, 1 point for the confounding variable named, 1 point for the condition for a causal conclusion.

Check before you hand in

Five minutes of checking recover more marks than one more problem started in a hurry.

The line passes through the mean point

Replace $x$ by $\overline{x}$ in the equation found: you must get exactly $\overline{y}$ .

$\hat{y}(3.5)=2\times 3.5=7=\overline{y}$ : the intercept is correct.
The sign of r and the sign of the slope

Both always have the same sign. A positive $r$ with a negative slope signals an error in $S_{xy}$ .

$r=0.97$ and $b=2$ : consistent.
The sum of the residuals is zero

Add up the $y_{i}-\hat{y_{i}}$ : the total must be zero up to rounding. This validates the calculation, never the model.

$0-1+1+1-1+0=0$ : the predicted values are computed correctly.
r stays between minus one and one

Any value outside this interval comes from an error in $S_{xx}$ or $S_{yy}$ , usually a forgotten square.

An $r$ reported as $1.4$ : the square root in the denominator was left out.

The typical problem, taken apart

A complete regression, from the scatter plot to the prediction

Six pairs are recorded: $(1, 2)$ , $(2, 3)$ , $(3, 7)$ , $(4, 9)$ , $(5, 9)$ and $(6, 12)$ .

Find the least-squares line, interpret the coefficient of determination and predict the value for $x=4.5$ .

The six pairs of the question: the scatter plot is clearly linear and increasing, with no isolated point, which allows a linear regression.

Step 1

$\sum x=21$ , $\sum y=42$ , $\sum xy=182$ , $\sum x^{2}=91$ , $\sum y^{2}=368$ , so $\overline{x}=3.5$ and $\overline{y}=7$ .

Why

The five sums are computed once and for all: everything else follows from them, and an error here spreads everywhere. Copying them out neatly earns the method mark.

Step 2

$S_{xy}=182-\dfrac{21\times 42}{6}=35$ , $S_{xx}=91-\dfrac{21^{2}}{6}=17.5$ , $S_{yy}=368-\dfrac{42^{2}}{6}=74$ .

Why

These three quantities are variations, so $S_{xx}$ and $S_{yy}$ are always positive. A negative $S_{xx}$ signals a badly computed sum of squares.

Step 3

$b=\dfrac{35}{17.5}=2$ , then $a=7-2\times 3.5=0$ , so $\hat{y}=2x$ .

Why

The intercept is derived from the mean point: that is what guarantees the line passes through it, and it is the check to make right away.

Step 4

$r=\dfrac{35}{\sqrt{17.5\times 74}}\approx 0.973$ , so $r^{2}\approx 0.946$ .

Why

Give both, but interpret only the square: $94.6\%$ of the variation in $y$ is explained by the linear relationship with $x$ .

Step 5

Residuals: $0,\,-1,\,1,\,1,\,-1,\,0$ . Sum zero, $\sum e^{2}=4$ , and $s_{e}=\sqrt{\dfrac{4}{6-2}}=1$ .

Why

The residuals show no structure and stay within $\pm 1$ : the linear model holds, which no coefficient alone could establish.

Step 6

Prediction: $\hat{y}(4.5)=9$ , a value inside the range $[1, 6]$ of the observed $x$ values.

Why

The reservation about the range must be written BEFORE the number. For $x=50$ , the same line would say nothing reliable, and the marker expects that sentence.

Step 7

Check: $S_{yy}(1-r^{2})=74\times 0.054=4=\sum e^{2}$ .

Why

This identity links the coefficient and the residuals: the two routes of calculation validate each other in ten seconds.

The conclusion, written out

“The least-squares line is $\hat{y}=2x$ , with $r=0.97$ and $r^{2}=0.95$ : $95\%$ of the variation in $y$ is explained by the linear relationship with $x$ . For $x=4.5$ , a value within the observed range, we predict $\hat{y}=9$ .”

The classic mistake on this problem: Announcing that $97\%$ of the variation is explained: it is $r$ , not $r^{2}$ , that equals $0.97$ .

Learn by heart

• LOOK at the scatter plot before computing: an $r$ on a curved scatter plot means nothing.
• $b=\dfrac{S_{xy}}{S_{xx}}$ , then $a=\overline{y}-b\overline{x}$ : the line ALWAYS passes through the mean point.
• $r$ gives the direction and the strength; only $r^{2}$ is read as a percentage of variation explained.
• Least squares minimizes the VERTICAL deviations: the line of $y$ on $x$ is not the line of $x$ on $y$ .
• The sum of the residuals is zero by construction: it is their PLOT that judges the model.
• A leverage point can halve the slope while leaving $r$ high. Investigate, do not delete.
• Exponential: $\ln y$ against $x$ . Power: $\ln y$ against $\ln x$ , the slope giving the exponent.
• Correlation is not causation: name the confounding variable, otherwise the sentence earns nothing.

Frequently asked questions

What is the difference between r and r squared?

The correlation coefficient gives the direction of the relationship through its sign and its strength through its absolute value, between minus one and one. Its square, the coefficient of determination, is read as a percentage: it is the share of the variation in the variable explained by the linear relationship. A coefficient of zero point eight thus corresponds to sixty-four percent, not to eighty.

Why look at the scatter plot before computing?

Because the coefficient measures alignment and nothing else. A perfectly deterministic but curved relationship can give a low coefficient, and two scatter plots of very different shapes can give exactly the same one. The calculator always returns a number, even where a straight line makes no sense at all.

Can an outlier be deleted?

No, not without an investigation. A distant point may be a data-entry error, but it may also be the most informative observation in the data set. The expected approach is to look for where it comes from, then to present the analysis with and without that point while pointing out the difference, rather than making it disappear.

What does the residual plot show?

What the coefficient hides. A shapeless cloud of residuals validates the linear model. A curvature signals that the relationship is not a straight line. A funnel-shaped widening signals a spread that grows with the explanatory variable, which invalidates the prediction intervals even if the fit looks good.

Does a strong correlation prove a cause?

Never. Four explanations remain possible: the first variable causes the second, the second causes the first, a third variable causes both, or chance on a small sample. Only an experiment in which individuals are assigned to the conditions at random makes it possible to conclude that there is a cause-and-effect link.

Practise it

Corrected exercises: Correlation and linear regression, 201-337

A method is proved on a paper, not on a sheet. The set for the same chapter takes each of these traps into a problem, with the solution written out step by step.

11 corrected exercises
110 points
150 minutes

Do the exercises

Something missing from this sheet? A method, a trap, an exercise to practise it, a mistake you spotted: tell me, I read every request.

Sending goes through your free space: that is how I can answer you and tell you when the exercise is added. An email is enough, no password. Create your space. Rather not create an account? Write to me directly.

Previous sheet Estimation and hypothesis testing

© Ahmed Squalli Houssaini. Revision sheet published at www.letuteurscientifique.ca/en/fiches/cegep-337-correlation-regression. Free for personal and classroom use; republishing it elsewhere requires written permission (legal notice).

Looking for a 201-337 tutor in Montreal?

Get in touch for a first session. Regression is the most useful chapter of the course for the rest of your studies: we work on the calculations and on the interpretation in equal measure.

Book a session WhatsApp

Site by Studio Squalli

Revision sheet: correlation and linear regression (201-337)

The essentials

The scatter plot first, the calculation second

The working sums and the line

r, r squared, and what each one measures

The four explanations of a correlation

The mistakes that cost marks

1. Computing a coefficient without having looked at the scatter plot

2. Reading r as a percentage

3. Predicting outside the observed range

4. Drawing a cause from a correlation

5. Deleting the inconvenient point

6. Swapping the roles of x and y

7. Trusting the coefficient without looking at the residuals

8. Forgetting that the line passes through the mean point

9. Confusing the strength of a relationship with how solid the evidence is

Which method to choose

What to do depending on the shape of the scatter plot

Reading a residual plot

How the answer is expected to be written

Writing up a complete regression

Writing a conclusion about causation

Check before you hand in

The typical problem, taken apart

A complete regression, from the scatter plot to the prediction

Learn by heart

Frequently asked questions

What is the difference between r and r squared?

Why look at the scatter plot before computing?

Can an outlier be deleted?

What does the residual plot show?

Does a strong correlation prove a cause?

Corrected exercises: Correlation and linear regression, 201-337

See also

Looking for a 201-337 tutor in Montreal?