Regression and chi-square 201-SN1-RE: revision sheet

Q: What is the difference between the correlation coefficient and the coefficient of determination?

The correlation coefficient r, between minus 1 and 1, says how closely the points are aligned on a line and in which direction. The coefficient of determination, its square, gives the share of the variation in y explained by the linear relationship: r equal to 0.9 gives 81 percent, not 90. So it is the square that is interpreted as a percentage.

Q: How do you compute the expected counts for a chi-square test?

For each cell, multiply the total of its row by the total of its column, then divide by the overall total. This is the count you would expect if the two variables were independent. The expected counts of a row have the same total as the observed counts, which lets you check the calculation, and they must all be at least 5 for the test to apply.

Q: How many degrees of freedom does a chi-square test of independence have?

Multiply the number of rows minus one by the number of columns minus one, without counting the totals. A table with 3 rows and 4 columns therefore has 6, and a 2 by 2 table has 1. The sample size plays no role: using n minus 1 is a common mistake, and it makes you read the wrong critical value.

Q: Can you predict with a regression line outside the data?

Only with great caution. Within the range of the observed x values, the prediction is an interpolation supported by the data. Beyond it, it is an extrapolation: nothing guarantees that the relationship stays linear, and a spring, an absorbance or a population often changes behaviour. A coefficient close to 1 does not change this rule.

Q: Does a significant chi-square test prove a cause-and-effect relationship?

No, not on its own. It shows that the two variables are associated. You can speak of an effect only if the study is an experiment in which the factor was assigned at random, for example seeds divided among treatments. In an observational study, a third variable can explain the relationship, and the conclusion must stick to the word associated.

The thread of the chapter

A measure of association answers the question of its own model and no other: $r$ and $r^{2}$ measure alignment on a line, $\chi^{2}$ the departure from independence. We look at the data before the number, and we conclude without going beyond the study design or the measured range.

Topics covered

Scatter plot
Covariance
Linear correlation coefficient
Coefficient of determination
Least-squares line
Residuals
Interpolation and extrapolation
Influential point
Contingency table
Expected counts
Chi-square test of independence
Degrees of freedom

The chapter at a glance

Two variables: which tool, which question, which conclusion

Choose the column from the TYPE of the two variables, before any calculation; each row says what the tool measures and how far you can conclude.

Criterion	Linear regression two quantitative variables	Chi-square test of independence two qualitative variables
Data	measured pairs $(x, y)$ , shown in a scatter plot	cross-classified counts in a contingency table
Before computing	look at the shape of the scatter plot and at isolated points	check that every expected count is at least $5$
The number	$r$ : alignment; $r^{2}$ : share explained; $a$ : change in $y$ per unit of $x$	$\chi^{2}=\sum\frac{(O-E)^{2}}{E}$ , with $(\ell-1)(c-1)$ degrees of freedom
Question settled	are the points close to a line, and to which one?	do the departures from independence go beyond chance?
Limit of the conclusion	only within the range of the observed $x$ values (interpolation)	an association; a cause only if the factor was assigned at random

Linear regression two quantitative variables

Data: measured pairs $(x, y)$ , shown in a scatter plot
Before computing: look at the shape of the scatter plot and at isolated points
The number: $r$ : alignment; $r^{2}$ : share explained; $a$ : change in $y$ per unit of $x$
Question settled: are the points close to a line, and to which one?
Limit of the conclusion: only within the range of the observed $x$ values (interpolation)

Chi-square test of independence two qualitative variables

Data: cross-classified counts in a contingency table
Before computing: check that every expected count is at least $5$
The number: $\chi^{2}=\sum\frac{(O-E)^{2}}{E}$ , with $(\ell-1)(c-1)$ degrees of freedom
Question settled: do the departures from independence go beyond chance?
Limit of the conclusion: an association; a cause only if the factor was assigned at random

Key: the words in the table

Scatter plot: The graph of the measured pairs, one variable per axis.
Contingency table: The table that counts the individuals in each combination of categories of the two variables.
Expected count: The count you would expect in a cell if the two variables were independent: row total times column total, divided by n.
Degrees of freedom: The number of cells that can be filled in freely once the totals are fixed.
Interpolation: A prediction made inside the range of the observed x values; beyond it, it is an extrapolation.

A quantitative variable and a qualitative one (a mass under three treatments) belong to neither column: that is a comparison of means, covered with hypothesis tests.

The essentials

The least-squares line and its three properties

• It minimizes $\sum e_{i}^{2}$ , the sum of the squared VERTICAL deviations $e_{i}=y_{i}-(ax_{i}+b)$ .
• $a=\dfrac{S_{xy}}{S_{xx}}$ and $b=\overline{y}-a\overline{x}$ : the line passes through the mean point $(\overline{x}, \overline{y})$ .
• The sum of the residuals is zero, and the minimum sum of squares equals $S_{yy}(1-r^{2})$ : this is the proof that $-1\le r\le 1$ .
• For the six points of the figure: $a\approx 1.335$ , $b\approx 1.94$ , mean point $(5.67, 9.5)$ .

The orange deviations are vertical; the blue line minimizes the sum of their squares and passes through the green mean point

(5.67, 9.5)

These formulas are proved by minimizing first in b, then in a: a classic exam question, which only requires the derivative of a quadratic polynomial.

Three numbers, three different questions

• The slope $a$ : by how much $y$ changes on average when $x$ increases by one unit. It has a unit.
• The coefficient $r=\dfrac{S_{xy}}{\sqrt{S_{xx}S_{yy}}}$ : how closely the points are aligned. No unit, between $-1$ and $1$ , with the sign of $a$ .
• The coefficient of determination $r^{2}$ : the share of the variation in $y$ explained by the linear relationship. It is THIS one that is read as a percentage.
• The covariance $s_{xy}=\dfrac{S_{xy}}{n-1}$ gives the direction of the relationship, never its strength: it changes with the units.

The chi-square test compares profiles

• $H_{0}$ : the two variables are independent, that is, each column has the same profile as the whole table.
• $E_{ij}=\dfrac{L_{i}C_{j}}{n}$ comes from $P(A\cap B)=P(A)P(B)$ under independence (L for the row total, C for the column total).
• $\chi^{2}=\sum\dfrac{(O-E)^{2}}{E}$ is computed on COUNTS; $H_{0}$ is rejected when it exceeds the critical value from the table, read at $(\ell-1)(c-1)$ degrees of freedom.
• Conditions: independent observations (each individual counted once) and every $E\ge 5$ .

On the left, the three columns have the overall profile, 40% yes: that is independence. On the right, the profiles depart from 40%: that is what the chi-square statistic measures.

The mistakes that cost marks

These are the errors I correct most often in session. Each one costs marks on a paper, even when the reasoning behind it is right.

1. Reading the correlation coefficient as a percentage

1 point per interpretation

What not to write

“ $r=0.90$ , so the line explains $90\%$ of the variation in $y$ .”

What to write

“ $r^{2}=0.81$ : the linear relationship explains $81\%$ of the variation in $y$ .”

Why: The proof for the line gives $1-r^{2}=\frac{S_{\min}}{S_{yy}}$ : it is the square that measures a share of the variation.

2. Inverting the line of y on x to predict x

1 to 2 points

What not to write

“ $\hat{y}=0.821x+1.571$ , so for $y=8$ we predict $x=\frac{8-1.571}{0.821}\approx 7.83$ .”

What to write

“To predict $x$ from $y$ , I use the line of $x$ on $y$ : $\hat{x}=0.856y-0.160$ , that is, $\hat{x}\approx 6.69$ for $y=8$ .”

Same seven points, two lines: the blue one minimizes the vertical deviations, the orange one the horizontal deviations; they cross at the mean point

(4, 4.86)

Why: Each line minimizes its own deviations, vertical or horizontal; their slopes satisfy $a\,a'=r^{2}$ and the two lines coincide only if $|r|=1$ . Exception: in a calibration, $x$ is set without error, and the line of $y$ on $x$ is inverted.

3. Predicting outside the observed range

1 point, often the whole question

What not to write

“The spring follows $\hat{L}=1.98F+12.0$ with $r^{2}=0.999$ ; under $40$ N, it will be $91$ cm long.”

What to write

“The line describes the spring between $0$ and $5$ N, the range of the measurements; at $40$ N, it is an extrapolation that nothing guarantees.”

Why: An r² close to 1 judges the alignment of the measured points, never what happens beyond them; a spring, an absorbance or a population changes regime outside the range studied.

4. Judging a suspicious point on r alone

2 points

What not to write

“With the Arctic station, r goes from −0.996 to −0.984: this point changes nothing.”

What to write

“r barely moves, but the slope goes from −0.90 to −0.70: this point, far from the others in latitude, rotates the line.”

Why: The slope is a weighted average in which the weight of a point is $\frac{(x_{i}-\overline{x})^{2}}{S_{xx}}$ : a point far from $\overline{x}$ has high leverage, and $r$ does not see it.

5. Concluding that there is no relationship because r is almost zero

2 points

What not to write

“ $r=0.01$ between temperature and enzyme activity: temperature has no effect.”

What to write

“ $r$ is almost zero: there is no LINEAR relationship. The scatter plot shows a bell with an optimum: the relationship exists, it is just not linear.”

Why: r measures alignment on a line; in a bell, the rise and the fall cancel each other out. Always look at the scatter plot before computing.

6. Getting the degrees of freedom wrong

2 points: the whole decision is wrong

What not to write

“Table with $2$ rows and $3$ columns, $n=150$ : $149$ degrees of freedom.” or “ $2\times 3-1=5$ .”

What to write

“ $(2-1)(3-1)=2$ degrees of freedom, so the critical value is $5.991$ at the $5\%$ level.”

5\%

, the critical value is

5.991

with

2

degrees of freedom and

9.488

with

4

: with

\chi^{2}=7

H_{0}

is rejected at

2

degrees of freedom, not at

4

Why: The row and column totals are fixed: only the cells of a sub-table with one row and one column fewer are free. The critical value changes a lot with the number of degrees of freedom.

7. Computing chi-square on percentages

the whole question

What not to write

“I convert the table to percentages, it is easier to read, then I compute the chi-square statistic.”

What to write

“ $\chi^{2}$ is computed on the observed counts; percentages only serve to describe the profiles.”

Why: Multiplying all the counts by $k$ multiplies $\chi^{2}$ by $k$ : with percentages, the sample is brought down to $100$ individuals. With $150$ seeds, $10.67$ becomes $7.11$ .

8. Taking a failure to reject for a proof of independence

1 conclusion point

What not to write

“H0 is not rejected at the 5% level: the two variables are independent.”

What to write

“At the 5% level, the data do not allow us to conclude that there is a relationship between the two variables.”

Why: A weak relationship can remain invisible in a small sample: with the same proportions and four times as many individuals, the chi-square statistic is four times larger and the test may reject.

9. Drawing a cause from an observational study

1 to 2 points

What not to write

“The test rejects independence: soil type causes flower colour.”

What to write

“In these gardens, colour and soil type are associated; the survey, with no random assignment, does not by itself establish the mechanism.”

Why: Only an experiment in which the factor is assigned at random, such as seeds divided among treatments, allows the word “effect”. In an observational study, a third variable can explain the relationship.

Which method to choose

Which tool, depending on the type of the two variables

Look at what the problem gives you: measurements with units, or counted categories.

If two measured variables (a length, a temperature, an absorbance) → scatter plot, then linear regression if the scatter plot is straight

Example: length of a spring and force: $\hat{L}=1.98F+12.0$
If two categorical variables, given as cross-classified counts → contingency table, expected counts, chi-square test of independence

Example: flower colour and soil type: $\chi^{2}\approx 73.2$ with $4$ degrees of freedom
If one measurement and one category → neither: compare means with a hypothesis test

Example: mass of plants under three fertilizers

A quantitative variable cut into classes can go into a contingency table, but information is lost: do it only if the problem asks for it.

What to do depending on the shape of the scatter plot

Draw the scatter plot before any calculation, then read its shape.

If points aligned, no isolated point → least-squares line, r and r², prediction within the range

Example: speed of sound: $r\approx 0.9987$
If curved scatter plot, or U-shaped residuals → change the model: take logarithms for a power or exponential law

Example: mass and length: slope $3.02$ between $\ln m$ and $\ln L$
If a point far from the others in y, in the middle of the x values → outlier: it makes r drop; look for the cause before any decision

Example: mountaintop station: $r$ goes from $-0.996$ to $-0.786$
If a point far from the others in x → high-leverage point: redo the line without it and compare the SLOPES

Example: Arctic station: slope from $-0.90$ to $-0.70$

A point is never set aside without saying which one and why: the reason comes from the phenomenon (another population, a measurement error), never from the wish to improve r.

How the answer is expected to be written

A marker ticks steps. Here they are in order, with the concluding sentence expected word for word.

Writing up a chi-square test of independence

When to use it: The problem gives a table of cross-classified counts and asks whether the two variables are related.

1 Write H0 (independence) and H1 (dependence) with the names of the variables in context.
2 Compute the expected counts $E=\frac{L\times C}{n}$ and check that they are all at least $5$ , saying so.
3 Compute $\chi^{2}$ by writing out the contributions, then the number of degrees of freedom $(\ell-1)(c-1)$ .
4 Compare with the critical value given for this level and this number of degrees of freedom, and decide.
5 Conclude in one sentence in context, then name the cells that contribute the most.

Concluding sentence

“ $\chi^{2}\approx 10.67>5.991$ ( $2$ degrees of freedom): at the $5\%$ level, $H_{0}$ is rejected; germination depends on the treatment received.”

The trap: Writing “H0 is accepted” or “the variables are independent” when H0 is not rejected.

Marking: 1 point for the hypotheses, 2 points for the expected counts and the condition, 3 points for the statistic and the degrees of freedom, 1 point for the decision, 3 points for the conclusion in context.

Writing up a linear regression

When to use it: The problem gives pairs of measurements and asks for a model, its interpretation and a prediction.

1 Describe the scatter plot: direction, form, strength, isolated points.
2 Write $S_{xx}$ , $S_{xy}$ , $S_{yy}$ with their values, then $a=\frac{S_{xy}}{S_{xx}}$ and $b=\overline{y}-a\overline{x}$ .
3 Interpret the slope with its units, and the intercept only if x = 0 makes sense.
4 Give $r$ , then $r^{2}$ as a percentage of variation explained.
5 Predict, saying whether it is an interpolation or an extrapolation.

Concluding sentence

“ $\hat{L}\approx 1.98F+12.0$ : each newton stretches the spring by about $1.98$ cm; $r^{2}\approx 0.999$ , the linear relationship explains $99.9\%$ of the variation in length between $0$ and $5$ N.”

The trap: Interpreting the intercept when x = 0 is outside the range (a latitude of 0, a snake of zero length).

Marking: 1 point for the scatter plot, 3 points for the sums and the line, 2 points for the interpretations, 2 points for r and r², 2 points for the prediction and its caveat.

Check before you hand in

Five minutes of checking recover more marks than one more problem started in a hurry.

The line passes through the mean point

Replace $x$ by $\overline{x}$ in your equation: you must get $\overline{y}$ back.

$1.983\times 2.5+12.04\approx 17.0=\overline{L}$ : the line of the spring is right. A result of $15.2$ says that a sum is wrong.
The sign of r and that of the slope

$r$ and $a$ always have the same sign, that of $S_{xy}$ , and the scatter plot must rise or fall accordingly.

A scatter plot that falls with $r=+0.93$ betrays a sign error in $S_{xy}$ .
The expected counts give back the totals

Add up the expected counts of a row: you must get the observed row total back, to the nearest hundredth.

$17.4+17.4+23.2=58$ for the blue hydrangeas: correct. A total of $60$ signals a cell computed with the wrong total.
Orders of magnitude

$-1\le r\le 1$ , $0\le r^{2}\le 1$ , $\chi^{2}\ge 0$ , and each $(O-E)$ in a row of a two-column table has the opposite value in the other column.

An $r^{2}$ of $1.04$ or a negative $\chi^{2}$ means you must redo the calculation, not interpret it.

The typical problem, taken apart

A chi-square test on a two-by-two table

A mycologist visits $120$ plots, $60$ in coniferous forest and $60$ in deciduous forest, and records whether a species of mushroom is present. She finds it in $18$ coniferous plots and in $30$ deciduous plots.

At the $5\%$ level, does the presence of the mushroom depend on the type of forest? Critical value with $1$ degree of freedom: $3.841$ .

Step 1

Table: coniferous, $18$ present and $42$ absent; deciduous, $30$ and $30$ . Totals: $48$ present, $72$ absent, $60$ per type, $n=120$ .

Why

First put the data back into a contingency table with its totals: all the following calculations come from it, and a wrong total shows up immediately.

Step 2

$H_{0}$ : presence is independent of the type of forest. $E_{\text{present}}=\dfrac{48\times 60}{120}=24$ in each column, $E_{\text{absent}}=\dfrac{72\times 60}{120}=36$ .

Why

The expected counts describe the world in which H0 is true; they all exceed 5, and saying so earns the point for the condition.

Step 3

$\chi^{2}=\dfrac{(18-24)^{2}}{24}+\dfrac{(30-24)^{2}}{24}+\dfrac{(42-36)^{2}}{36}+\dfrac{(30-36)^{2}}{36}=1.5+1.5+1+1=5$

Why

Writing out the four contributions shows the method to the grader and reveals where the deviation comes from; in a two-by-two table, the four deviations have the same absolute value, here 6.

Step 4

Degrees of freedom: $(2-1)(2-1)=1$ . Since $5>3.841$ , $H_{0}$ is rejected at the $5\%$ level.

Why

The right number of degrees of freedom picks the right critical value: with 3 instead of 1, you would read 7.815 and would no longer reject.

Step 5

Check: with $1$ degree of freedom, $\chi^{2}=Z^{2}$ , so the P-value is $P(|Z|>\sqrt{5})=2\left(1-\Phi(2.24)\right)\approx 0.025<0.05$ .

Why

An independent check of the chi-square table: the distribution with 1 degree of freedom is that of the square of a standard normal variable, and the P-value confirms the decision.

The conclusion, written out

At the $5\%$ level, the presence of this mushroom is associated with the type of forest: it is found in $50\%$ of the deciduous plots versus $30\%$ of the coniferous plots. The study is observational: it does not establish that the trees themselves are the cause.

The classic mistake on this problem: Computing the expected counts with $n=60$ instead of $120$ , or writing $2\times 2-1=3$ degrees of freedom: the critical value becomes $7.815$ and the conclusion is reversed.

Learn by heart

• $a=\dfrac{S_{xy}}{S_{xx}}$ , $b=\overline{y}-a\overline{x}$ : the least-squares line passes through $(\overline{x}, \overline{y})$ .
• $r=\dfrac{S_{xy}}{\sqrt{S_{xx}S_{yy}}}$ measures ALIGNMENT; $r^{2}$ is the share of variation explained, the one that is read as a percentage.
• Interpolation within the range of the observed x values; extrapolation beyond it, never guaranteed.
• $E=\dfrac{L\times C}{n}$ , $\chi^{2}=\sum\dfrac{(O-E)^{2}}{E}$ on COUNTS, $(\ell-1)(c-1)$ degrees of freedom, every $E\ge 5$ .
• Reject H0: association. Do not reject: no relationship shown, which is not a proof of independence.
• The word “effect” only if the factor was assigned at random.

Frequently asked questions

What is the difference between the correlation coefficient and the coefficient of determination?

The correlation coefficient r, between minus 1 and 1, says how closely the points are aligned on a line and in which direction. The coefficient of determination, its square, gives the share of the variation in y explained by the linear relationship: r equal to 0.9 gives 81 percent, not 90. So it is the square that is interpreted as a percentage.

How do you compute the expected counts for a chi-square test?

For each cell, multiply the total of its row by the total of its column, then divide by the overall total. This is the count you would expect if the two variables were independent. The expected counts of a row have the same total as the observed counts, which lets you check the calculation, and they must all be at least 5 for the test to apply.

How many degrees of freedom does a chi-square test of independence have?

Multiply the number of rows minus one by the number of columns minus one, without counting the totals. A table with 3 rows and 4 columns therefore has 6, and a 2 by 2 table has 1. The sample size plays no role: using n minus 1 is a common mistake, and it makes you read the wrong critical value.

Can you predict with a regression line outside the data?

Only with great caution. Within the range of the observed x values, the prediction is an interpolation supported by the data. Beyond it, it is an extrapolation: nothing guarantees that the relationship stays linear, and a spring, an absorbance or a population often changes behaviour. A coefficient close to 1 does not change this rule.

Does a significant chi-square test prove a cause-and-effect relationship?

No, not on its own. It shows that the two variables are associated. You can speak of an effect only if the study is an experiment in which the factor was assigned at random, for example seeds divided among treatments. In an observational study, a third variable can explain the relationship, and the conclusion must stick to the word associated.

Mark this sheet as read or add it to your favourites: a free account, no password, keeps your read sheets and favourites from one visit to the next, tells you which chapter to tackle next and lets you ask for the method or exercise you are missing. Create your space, an email is enough.

Practise it

Corrected exercises: Linear regression and the chi-square test of independence, 201-SN1-RE

A method is proved on a paper, not on a sheet. The set for the same chapter takes each of these traps into a problem, with the solution written out step by step.

11 corrected exercises
110 points
180 minutes

Do the exercises

Something missing from this sheet? A method, a trap, an exercise to practise it, a mistake you spotted: tell me, I read every request.

Sending goes through your free space: that is how I can answer you and tell you when the exercise is added. An email is enough, no password. Create your space. Rather not create an account? Write to me directly.

Previous sheet Hypothesis tests

© Ahmed Squalli Houssaini. Revision sheet published at www.letuteurscientifique.ca/en/fiches/cegep-sn1-regression-chi-square. Free for personal and classroom use; republishing it elsewhere requires written permission (legal notice).

Revision sheet: linear regression and the chi-square test of independence (201-SN1-RE)

Two variables: which tool, which question, which conclusion

Key: the words in the table

The essentials

The least-squares line and its three properties

Three numbers, three different questions

The chi-square test compares profiles

The mistakes that cost marks

1. Reading the correlation coefficient as a percentage

2. Inverting the line of y on x to predict x

3. Predicting outside the observed range

4. Judging a suspicious point on r alone

5. Concluding that there is no relationship because r is almost zero

6. Getting the degrees of freedom wrong

7. Computing chi-square on percentages

8. Taking a failure to reject for a proof of independence

9. Drawing a cause from an observational study

Which method to choose

Which tool, depending on the type of the two variables

What to do depending on the shape of the scatter plot

How the answer is expected to be written

Writing up a chi-square test of independence

Writing up a linear regression

Check before you hand in

The typical problem, taken apart

A chi-square test on a two-by-two table

Learn by heart

Frequently asked questions

What is the difference between the correlation coefficient and the coefficient of determination?

How do you compute the expected counts for a chi-square test?

How many degrees of freedom does a chi-square test of independence have?

Can you predict with a regression line outside the data?

Does a significant chi-square test prove a cause-and-effect relationship?

Corrected exercises: Linear regression and the chi-square test of independence, 201-SN1-RE

See also

Looking for a 201-SN1-RE tutor in Montreal?