Corrected exercises: correlation and linear regression (201-337)

$S_{xy}=564-\frac{30\times 85}{5}=564-\frac{2550}{5}=564-510=54$. The positive sign announces a positive slope.

$S_{xx}=380-\frac{52^{2}}{8}=380-\frac{2704}{8}=380-338=42$. Pitfall: writing $\frac{\sum x^{2}}{n}$ instead of $\frac{(\sum x)^{2}}{n}$; it is the sum that gets squared.

$S_{yy}=572-\frac{60^{2}}{7}=572-\frac{3600}{7}=572-514.29=57.71$. Pitfall: rounding $\frac{3600}{7}$ to the nearest unit too early, which gives $58$; keep at least two decimals in the intermediate calculation.

$r=\frac{-84}{\sqrt{60\times 150}}=\frac{-84}{\sqrt{9000}}=\frac{-84}{94.868}\approx-0.885$. The sign of $r$ is always that of $S_{xy}$, since the square root is positive. Pitfall: forgetting the square root, $\frac{-84}{9000}$, which gives a tiny $r$.

$r^{2}=(-0.74)^{2}=0.5476\approx 0.548$. Pitfall: keying in $-0.74^{2}$ without parentheses, which the calculator reads as $-(0.74^{2})=-0.5476$; a coefficient of determination is never negative.

$b=\frac{S_{xy}}{S_{xx}}=\frac{45}{18}=2.5$. Pitfall: dividing by $S_{yy}$ or by $\sqrt{S_{xx}S_{yy}}$, which is the denominator of $r$, not that of the slope.

$a=\overline{y}-b\overline{x}=40-(-1.6)(12)=40+19.2=59.2$. Pitfall: losing the double minus sign and writing $40-19.2=20.8$.

$\hat{y}=14.2+3.5\times 7.4=14.2+25.9=40.1$.

$\hat{y}(15)=88-2.4\times 15=88-36=52$, so $e=y-\hat{y}=49-52=-3$: the point lies below the line. Pitfall: computing $\hat{y}-y=+3$, the residual the wrong way round.

$S_{xy}=186-\frac{15\times 55}{5}=186-165=21$ and $S_{xx}=55-\frac{15^{2}}{5}=55-45=10$, so $b=\frac{21}{10}=2.1$. The means are $\overline{x}=\frac{15}{5}=3$ and $\overline{y}=\frac{55}{5}=11$, so $a=11-2.1\times 3=11-6.3=4.7$: the line is $\hat{y}=4.7+2.1x$.

The scatter plot is clearly INCREASING: the grades go up with the number of hours of study. The points lie very close to a line, with no visible curvature and no outlier: the relationship is linear and appears very strong. This preliminary reading is essential, because it is what justifies using the linear correlation coefficient.

$\sum x=2+4+5+7+9+11=38$. $\sum y=35+48+52+68+80+95=378$. $\sum xy=70+192+260+476+720+1045=2763$. $\sum x^{2}=4+16+25+49+81+121=296$. $\sum y^{2}=1225+2304+2704+4624+6400+9025=26\,282$.

$S_{xy}=2763-\dfrac{38\times 378}{6}=2763-2394=369$. $S_{xx}=296-\dfrac{38^{2}}{6}=296-240.667=55.333$. $S_{yy}=26\,282-\dfrac{378^{2}}{6}=26\,282-23\,814=2468$.

$r=\dfrac{S_{xy}}{\sqrt{S_{xx}S_{yy}}}=\dfrac{369}{\sqrt{55.333\times 2468}}=\dfrac{369}{369.54}=0.9985$. The correlation is POSITIVE, so the two variables move in the same direction, and its absolute value is extremely close to 1: the linear relationship is almost perfect. Such a coefficient is rare on real data; here it points to constructed data or to a very regular phenomenon.

NO, and this is the classic trap. An $r$ close to zero only means that there is no LINEAR relationship. There may very well be a strong but curved relationship: on a scatter plot shaped like a symmetric parabola, for example $y=x^{2}$ with $x$ values spread symmetrically around zero, the coefficient is exactly 0 even though $y$ is completely determined by $x$. That is precisely why you DRAW the scatter plot before computing: the coefficient alone cannot tell “no relationship” apart from “a nonlinear relationship”.

$b=\dfrac{S_{xy}}{S_{xx}}=\dfrac{369}{55.333}=6.669$. Then $a=\overline{y}-b\overline{x}=63.0-6.669\times 6.333=63.0-42.235=20.765$. The equation is therefore $y=20.77+6.67x$.

The SLOPE means that one extra hour of study per week is associated with an average increase of 6.67 points in the grade. The INTERCEPT, 20.77, would be the grade of a student who studies zero hours. Handle this number with care: $x=0$ lies outside the observed range, which runs from 2 to 11 hours, so it is an extrapolation. It may be read as a plausible order of magnitude, but the model does not guarantee it.

The mean point is $\left(6.333, 63.0\right)$. Substituting into the equation: $20.765+6.669\times 6.333=20.765+42.235=63.0$. The line does pass through the mean point, and this is no coincidence: it follows directly from the formula $a=\overline{y}-b\overline{x}$, which is built to make it happen. It is a quick check to make systematically on every regression.

“Least squares” means that the line chosen is the one that makes the sum $\sum\left(y_{i}-\hat{y}_{i}\right)^{2}$ as SMALL as possible, where $\hat{y}_{i}$ is the predicted value. The raw deviations are not minimized because their sum is ZERO for infinitely many lines, the positive deviations cancelling the negative ones: the criterion would select nothing. Squaring prevents this cancelling, penalizes large deviations more than small ones, and leads to simple explicit formulas, which is not the case if you minimize the absolute values.

The predicted value for $x=5$ is $\hat{y}=20.765+6.669\times 5=54.11$. The residual is $y-\hat{y}=52-54.11=-2.11$. Its NEGATIVE sign means that this student obtained a grade LOWER than the one the model predicted for that amount of study: the point lies below the line. It is in fact the largest residual in absolute value of the six, hence the worst-fitted point of the scatter plot.

$r^{2}=0.9985^{2}=0.9971$, that is, 99.7%. Interpretation: 99.7% of the observed variation in the grades is EXPLAINED by the linear relationship with the number of hours of study. This is an exceptionally good fit.

The remaining 0.3% is the RESIDUAL variation, that is, everything study time does not account for: individual aptitude, the quality of the studying and not just its length, how the student feels on exam day, the luck of the questions chosen, as well as measurement errors. No statistical model claims to explain 100% of a human phenomenon, and an $r^{2}$ of 1.000 on real data would in fact be suspicious.

$\hat{y}=20.765+6.669\times 8=20.765+53.35=74.1$. The prediction is about 74. It is LEGITIMATE because 8 hours lies inside the range of the observed data, between 2 and 11: this is an interpolation.

$\hat{y}=20.765+6.669\times 20=20.765+133.37=154.1$. The prediction exceeds 100: it is IMPOSSIBLE, since the grade cannot go above 100. The prediction is therefore not legitimate, and for a fundamental reason: $x=20$ is far beyond the observed range, so we are EXTRAPOLATING. Nothing in the data guarantees that the relationship stays linear beyond 11 hours; in reality it necessarily levels off.

$\hat{y}=20.765+6.669\times 30=220.8$, more than twice the maximum possible. The absurdity of the result is actually good news here: it makes the error VISIBLE. The real danger of extrapolation is the case where it produces a plausible number that nothing flags as wrong. The rule is therefore never to predict outside the range of the observed $x$ values, and always to state that range of validity next to the equation. A linear model is only a LOCAL approximation of a relationship that, in reality, almost always saturates.

With the point $(5, 90)$: $n=7$, $\sum x=43$, $\sum y=468$, $\sum xy=3213$, $\sum x^{2}=321$, $\sum y^{2}=34\,382$. So $\overline{x}\approx 6.143$ and $\overline{y}\approx 66.86$. Working sums: $S_{xy}=3213-\frac{(43)(468)}{7}\approx 338.1$, $S_{xx}=321-\frac{43^{2}}{7}\approx 56.86$, $S_{yy}=34\,382-\frac{468^{2}}{7}\approx 3092.9$. Coefficient: $r=\frac{338.1}{\sqrt{(56.86)(3092.9)}}\approx 0.806$.

$r^{2}$ goes from $0.997$ to $0.650$: the proportion of variation explained collapses from 99.7% to 65%. One point out of seven, that is 14% of the data, has destroyed most of the relationship. This is the most useful reminder of the chapter: the correlation coefficient is not a robust measure, it is extremely sensitive to extreme values, and reporting it without having looked at the scatter plot amounts to betting blindfolded.

With $(20, 100)$: $\sum x=58$, $\sum y=478$, $\sum xy=4763$, $\sum x^{2}=696$, $\sum y^{2}=36\,282$. This gives $S_{xy}\approx 802.4$, $S_{xx}\approx 215.4$ and $S_{yy}\approx 3641.4$, hence a slope $b=\frac{802.4}{215.4}\approx 3.72$ and $r\approx 0.906$. The contrast with a) is instructive: here the coefficient stays high, apparently reassuring, while the SLOPE has almost been halved, going from 6.67 to 3.72. The model now predicts that one extra hour of study brings only 3.7 points instead of 6.7: the practical conclusion is turned upside down without $r$ sounding the alarm.

The slope is $b=\frac{S_{xy}}{S_{xx}}=\frac{\sum(x_{i}-\overline{x})(y_{i}-\overline{y})}{\sum(x_{i}-\overline{x})^{2}}$. Each point enters through its deviation $x_{i}-\overline{x}$, and a point very far away in $x$ contributes a huge term to the numerator as well as to the denominator: it weighs much more heavily than the others in determining the line. A point far away in $x$ literally acts as a LEVER, with the mean point as the pivot. So it is the second point, $(20, 100)$, that is the more dangerous: the first one visibly lowers the coefficient and alerts the analyst, while the second moves the line silently, leaving an $r$ of 0.906 that would go unquestioned in any report.

Above all, do not delete it because it is inconvenient. The correct approach has three stages. First INVESTIGATE: is it a data-entry error, a badly converted unit, an individual who does not belong to the population under study? A value that is demonstrably wrong is corrected or set aside, with a written justification. Next, if the point is genuine, PUBLISH both analyses, with and without it, and let the reader judge its impact: this is standard practice in research. Finally, consider a ROBUST method, less sensitive to extremes than least squares, or a transformation that brings the $x$ values closer together. An outlier is often the richest observation in the data set: it signals a measurement error, a heterogeneous population, or a phenomenon the model ignores.

NO. The coefficient only measures how closely the two series vary together; it contains no information about the mechanism linking them. Inferring causation from a high $r$ is a reasoning error, whatever the value of the coefficient.

Four explanations compete, and the data alone cannot decide between them. First, $x$ causes $y$. Second, $y$ causes $x$, that is, reverse causation. Third, a THIRD VARIABLE, called a confounding variable, causes both at the same time. Fourth, it may be a coincidence due to sampling chance, all the more plausible when the sample is small or when a large number of series have been compared until two were found that match.

The confounding variable is TEMPERATURE, or more broadly the season. When it is hot, people buy more ice cream; and when it is hot, they swim more, which mechanically increases the number of occasions for drowning. The two series rise and fall together over the year because both are driven by the same cause, without any direct link between them. Removing all ice cream from the city would change nothing in the number of drownings.

Through a CONTROLLED EXPERIMENT with RANDOM assignment: subjects are allocated at random to an exposed group and a control group, then the groups are compared. Randomization guarantees that the two groups differ, on average, only in the factor manipulated, including for the confounding variables nobody has identified, and that is its decisive advantage. When the experiment is impossible or unethical, as here, one falls back on observational studies in which the known variables are CONTROLLED statistically, for example by examining the correlation at constant temperature, which in our case would vanish.

The analysis is of the same kind, but with an instructive twist. It would be absurd to conclude that the firefighters cause the damage. The confounding variable here is the INITIAL SIZE of the fire: a big fire causes both the dispatch of many firefighters and heavy damage. One can even add that there is indeed a causal link, but in the OPPOSITE direction to the naive reading: it is the seriousness of the fire that causes the firefighters to be sent. And the real effect of the firefighters on the damage is surely to REDUCE it, an effect the raw correlation completely hides. This is the textbook example of a positive correlation coexisting with a negative causal effect.

$\overline{x}=\dfrac{440}{8}=55.0$ m² and $\overline{y}=\dfrac{10\,400}{8}=1300$ dollars. Next $S_{xy}=608\,000-\dfrac{440\times 10\,400}{8}=608\,000-572\,000=36\,000$. $S_{xx}=25\,800-\dfrac{440^{2}}{8}=25\,800-24\,200=1600$. $S_{yy}=14\,000\,000-\dfrac{10\,400^{2}}{8}=14\,000\,000-13\,520\,000=480\,000$.

$r=\dfrac{36\,000}{\sqrt{1600\times 480\,000}}=\dfrac{36\,000}{\sqrt{768\,000\,000}}=\dfrac{36\,000}{27\,713}=1.299$... which is IMPOSSIBLE, since a correlation coefficient can never exceed 1. Such a result necessarily signals an error in the calculation or in the data. Going over it again: $\sqrt{1600\times 480\,000}=\sqrt{7.68\times 10^{8}}=27\,713$, and $\dfrac{36\,000}{27\,713}=1.299$. The sums given are therefore mutually incompatible, and the right reflex on an exam is to POINT THIS OUT rather than carry on. This check, $\left|r\right|\leq 1$, must be made systematically: it catches the vast majority of data-entry errors in the working sums.

The line can nevertheless be computed, since it depends only on $S_{xy}$ and $S_{xx}$: $b=\dfrac{36\,000}{1600}=22.5$ and $a=1300-22.5\times 55.0=1300-1237.5=62.5$. The equation is $y=62.5+22.5x$. These two coefficients are consistent and plausible, which points to $S_{yy}$, hence to $\sum y^{2}$, as the source of the inconsistency.

The slope is 22.5 dollars per square metre: each additional square metre is associated with an average increase in rent of 22.50 dollars a month. The intercept, 62.50 dollars, would be the rent of an apartment with zero floor area: it obviously has no concrete meaning and must certainly not be interpreted. It is only a fitting parameter, needed so that the line passes through the mean point.

$\hat{y}=62.5+22.5\times 70=62.5+1575=1637.50$ dollars. Limits: on the one hand, this estimate rests on only 8 apartments and on sums shown to be inconsistent, which rules out any quantitative confidence until the data have been checked; on the other hand, floor area explains only part of the rent, which also depends on the neighbourhood, the condition, the floor and the year, so that a point prediction without a confidence interval gives a false impression of precision.

Predicted values: $\hat{y}(2)=34.10$, $\hat{y}(4)=47.44$, $\hat{y}(5)=54.11$, $\hat{y}(7)=67.45$, $\hat{y}(9)=80.78$ and $\hat{y}(11)=94.12$. Corresponding residuals: $+0.90$, $+0.56$, $-2.11$, $+0.55$, $-0.78$ and $+0.88$. Their sum is 0, up to rounding.

The least-squares line always passes through the mean point: $\overline{y}=a+b\overline{x}$. Now $\sum e_{i}=\sum(y_{i}-a-bx_{i})=\sum y_{i}-na-b\sum x_{i}=n\overline{y}-na-nb\overline{x}=n(\overline{y}-a-b\overline{x})=0$. The sum is therefore zero by CONSTRUCTION, whatever the data, even absurd ones. It says strictly nothing about the quality of the model: it is a check on the calculation, not a diagnosis. To measure the fit, you must look at the sum of the SQUARES of the residuals, which does not vanish.

A satisfactory residual plot shows NO structure: a horizontal cloud, scattered at random on both sides of zero, of constant thickness, with no trend and no pattern. Any visible shape is a defect of the model. First anomaly, a U-shaped or bell-shaped CURVATURE: the relationship is not linear, and the line passes systematically above and then below the points. Second anomaly, a FUNNEL that widens: the spread increases with $x$, which is called heteroscedasticity; the line remains acceptable but the prediction intervals are worthless, because they assume a constant spread. Third anomaly, an ISOLATED point very far from the others: an outlier or a leverage point, whose influence must be examined separately.

$\sum e_{i}^{2}\approx(0.90)^{2}+(0.56)^{2}+(2.11)^{2}+(0.55)^{2}+(0.78)^{2}+(0.88)^{2}\approx 7.26$. So $s_{e}=\sqrt{\frac{7.26}{6-2}}=\sqrt{1.815}\approx 1.35$ points. Interpretation: the actual grades typically deviate from the line by about 1.3 points, which gives the order of magnitude of the error to expect from a prediction. We divide by $n-2$ and not by $n$ because two parameters, the slope and the intercept, have already been estimated from these same data. Check: $S_{yy}(1-r^{2})=(2468)(1-0.99706)=(2468)(0.00294)\approx 7.26$, identical. This relation also sheds light on the meaning of $r^{2}$: it is exactly the share of $S_{yy}$ that the line has managed to absorb.