Advanced Statistics 201-337 • CEGEP, Montreal

Revision sheet: estimation and hypothesis testing (201-337)

Q: What does a 95% confidence interval really mean?

That the method used produces, over all possible samples, intervals of which ninety-five percent contain the true parameter. The parameter is a fixed number: it is either in the interval obtained or it is not. The probability therefore describes the procedure, not the position of the parameter.

Q: Is the p-value the probability that the null hypothesis is true?

No. It is the probability of observing a result at least as extreme as the one obtained, assuming the null hypothesis is true. It is computed under that hypothesis, so it cannot measure how likely the hypothesis is. A small p-value means the data would be surprising if the null hypothesis held.

Q: Why do we never say that we accept the null hypothesis?

Because a test only looks for evidence against it. Not finding any may mean that it is true, but also that the sample was too small to detect a real gap. So we say that we do not reject it, which leaves the second possibility open.

Q: When should Student's t distribution be used instead of the normal distribution?

Whenever the population standard deviation is unknown and is estimated from the sample. That estimate adds uncertainty, which Student's t distribution handles with heavier tails and therefore larger critical values. Beyond thirty or so observations, the two distributions become almost identical.

Q: Is a significant result necessarily important?

No. Significance measures the strength of the evidence, and it depends mostly on the sample size. With ten thousand observations, a tiny gap becomes significant without having the slightest practical consequence. Always look at the estimated size of the effect, which the confidence interval gives directly.

Inference is the only statistics chapter where a correct calculation and a wrong sentence sit side by side on the same paper without anyone noticing. The number $95\%$ is right, the sentence “there is a $95\%$ chance that the mean is in this interval” is not, and it is the sentence that gets marked.

This sheet gathers the moves that decide the mark: checking the conditions before choosing $z$ or $t$ , putting the right proportion in the denominator, concluding without ever “accepting” the null hypothesis, and separating what is significant from what is important. It is written for CEGEP students taking 201-337; once it is read, the corrected exercise set on the same chapter puts every reflex to the test.

Mark this sheet as read or add it to your favourites: a free account, no password, keeps your read sheets and favourites from one visit to the next, tells you which chapter to tackle next and lets you ask for the method or exercise you are missing. Create your space, an email is enough.

The thread of the chapter

The parameter does not vary: it is the SAMPLE that varies, and the whole chapter comes down to never giving the parameter the probability that describes the method.

This chapter is part of 201-337, Advanced Statistics

What is on this sheet

1. The essentials
2. The mistakes that cost marks
3. Which method to choose
4. How the answer is expected to be written
5. Check before you hand in
6. The typical problem, taken apart
7. Learn by heart
8. Frequently asked questions

Topics covered

Sampling distribution
Standard error
Central limit theorem
Confidence interval
Margin of error
Null hypothesis
P-value
Significance level
Type I error
Student's t distribution
Test on a proportion
Chi-square test of independence

The essentials

The sampling distribution and the standard error

• $\mu_{\overline{x}}=\mu$ and $\sigma_{\overline{x}}=\dfrac{\sigma}{\sqrt{n}}$ : the latter quantity is called the STANDARD ERROR.
• Central limit theorem: for $n$ large enough, in practice $n\geq 30$ , the distribution of $\overline{x}$ is approximately normal, whatever the shape of the population.
• The decrease goes as $\sqrt{n}$ : making the sample four times larger divides the standard error by $2$ , not by $4$ .

The three distributions have the same centre; only their width changes, as

\dfrac{1}{\sqrt{n}}

: going from

n=9

n=36

only tightens by a factor of

2

The standard error describes the spread of sample MEANS, not that of individuals. Confusing $\sigma$ with $\sigma_{\overline{x}}$ is the silent error of the chapter: the calculation still looks plausible.

What a confidence interval says, and what it does not say

• Mean, $\sigma$ known: $\overline{x}\pm z_{\alpha/2}\dfrac{\sigma}{\sqrt{n}}$ . Usual values: $1.645$ at $90\%$ , $1.96$ at $95\%$ , $2.576$ at $99\%$ .
• Proportion: $\hat{p}\pm z_{\alpha/2}\sqrt{\dfrac{\hat{p}(1-\hat{p})}{n}}$ .
• CORRECT interpretation: if the sampling were repeated a large number of times, $95\%$ of the intervals built would contain the true parameter.
• Required margin of error: $n\geq\left(\dfrac{z\sigma}{E}\right)^{2}$ for a mean, $n\geq\left(\dfrac{z}{2E}\right)^{2}$ for a proportion in the worst case $\hat{p}=0.5$ .

The central

95\%

of the normal distribution lies between

-1.96

and

1.96

: the interval is the estimate plus or minus

1.96

standard errors.

The parameter is a fixed number: it is either in the interval or it is not. The randomness is in the method, not in the truth, and that is exactly what the expected sentence must say.

The logic of a test

• $H_{0}$ is the default hypothesis, the status quo; $H_{1}$ is what we are trying to establish.
• The P-VALUE is the probability of observing a result at least as extreme IF $H_{0}$ is true. Reject $H_{0}$ when p-value $<\alpha$ .
• Type I error: rejecting a true $H_{0}$ , with probability $\alpha$ . Type II error: failing to reject a false $H_{0}$ , with probability $\beta$ . For a fixed $n$ , reducing one increases the other.
• Test on a proportion: $z=\dfrac{\hat{p}-p_{0}}{\sqrt{p_{0}(1-p_{0})/n}}$ , with $p_{0}$ IN THE DENOMINATOR, since we reason under $H_{0}$ .

H_{0}

is true (blue), we wrongly reject it with probability

\alpha

; if

H_{1}

is true (orange), we miss it with

\beta

. Moving the cut-off to the right reduces

\alpha

and increases

\beta

We never prove $H_{0}$ : for lack of evidence against it, we simply do not reject it. The difference between “accepting” and “not rejecting” is worth a mark in every written conclusion.

Student, and the threshold of thirty

• If $\sigma$ is UNKNOWN, it is estimated by $s$ and $\dfrac{\overline{x}-\mu}{s/\sqrt{n}}$ follows a Student t distribution with $n-1$ degrees of freedom.
• Student's t distribution is more spread out than the normal: its critical values are always LARGER, so its intervals are wider.
• It converges to the normal as $n$ grows, which explains the usual threshold of $30$ .
• Difference of two independent means: $\sigma_{\overline{x_{1}}-\overline{x_{2}}}=\sqrt{\dfrac{s_{1}^{2}}{n_{1}}+\dfrac{s_{2}^{2}}{n_{2}}}$ : the VARIANCES add up, even for a difference.

Student's t distribution (orange) has heavier tails than the normal (blue): with

3

degrees of freedom, the two-sided critical value goes from

1.96

3.18

A $95\%$ interval for a difference that excludes zero is always equivalent to a two-sided p-value below $0.05$ : the two approaches give the same decision.

The mistakes that cost marks

These are the errors I correct most often in session. Each one costs marks on a paper, even when the reasoning behind it is right.

1. Giving the parameter the probability of the method

2 marks, it is the most corrected sentence in the whole course

What not to write

“There is a $95\%$ chance that the population mean lies between $12.3$ and $14.7$ .”

What to write

“If the sampling were repeated, $95\%$ of the intervals built this way would contain the true mean; this one may be one of them.”

Twenty samples, twenty different intervals around the same fixed parameter: only one misses it, and nothing in its appearance sets it apart from the others.

Why: The population mean is a fixed number, not a random variable: it is inside or outside. What varies from one sample to the next is the interval.

2. Dividing by n instead of the square root of n

2 marks, and an interval six times too narrow

What not to write

“ $\sigma=12$ and $n=36$ , so the standard error is $\dfrac{12}{36}=0.33$ .”

What to write

“The standard error is $\dfrac{\sigma}{\sqrt{n}}=\dfrac{12}{6}=2$ .”

Why: The mean of a sample of $36$ is not $36$ times more precise than a single observation, only $6$ times. All the slowness of polling lies in that square root.

3. Reading the p-value as the probability that the null hypothesis is true

2 marks on the interpretation question

What not to write

“The p-value is $0.03$ , so there is a $3\%$ chance that $H_{0}$ is true.”

What to write

“If $H_{0}$ were true, a result at least as extreme would be observed in $3\%$ of samples; that is rare enough to reject $H_{0}$ at the $5\%$ level.”

The p-value is the area beyond the observed

z

UNDER the curve of

H_{0}

: it assumes

H_{0}

is true, so it cannot measure its probability.

Why: The p-value is a probability computed UNDER $H_{0}$ , that is, assuming $H_{0}$ is true. It cannot, in the same breath, measure the probability of that hypothesis.

4. Accepting the null hypothesis

1 mark for each conclusion, and there are several per exam

What not to write

“The p-value is $0.22$ , so we accept $H_{0}$ : the mean is indeed equal to $50$ .”

What to write

“We do not reject $H_{0}$ : the data do not provide enough evidence against it, which does not prove that it is true.”

Why: A test can only reject or not reject. A large p-value may just as well signal a real effect with a sample too small to detect it.

5. Putting the observed proportion in the denominator of the test

1.5 marks, and the decision can flip near the cut-off

What not to write

“ $z=\dfrac{0.55-0.50}{\sqrt{0.55\times 0.45/400}}$ .”

What to write

“Under $H_{0}$ , the proportion is $p_{0}=0.50$ : $z=\dfrac{0.55-0.50}{\sqrt{0.50\times 0.50/400}}=2.00$ .”

Why: A test reasons under $H_{0}$ , so every theoretical quantity comes from $H_{0}$ . In a confidence interval, on the other hand, there is no hypothesis and $\hat{p}$ is what is used.

6. Using z when the population standard deviation is unknown

1.5 marks, and an interval that is too narrow, so too assertive

What not to write

“ $n=12$ , $s=4$ : I take $z=1.96$ for the $95\%$ interval.”

What to write

“ $\sigma$ is unknown and $n<30$ , so I use $t$ with $11$ degrees of freedom, that is $2.201$ .”

Why: Estimating $\sigma$ by $s$ adds a source of uncertainty, which Student's t distribution handles with heavier tails. Its critical values are therefore always greater than those of $z$ .

7. Confusing significant with important

2 marks on the critical reading question

What not to write

“The difference is significant at $p=0.01$ , so the effect of the treatment is large.”

What to write

“The difference is significant, but its estimated size is $0.25$ points out of $100$ : statistically detectable, practically negligible.”

Both intervals exclude zero, so both effects are significant; only the second, wide and far from zero, describes an effect of useful size.

Why: Significance measures the strength of the EVIDENCE, which depends mostly on $n$ . A huge sample makes any gap significant, however tiny.

8. Adding the standard deviations of a difference

2 marks, and a standard error that is always too large

What not to write

“ $\sigma_{d}=\dfrac{s_{1}}{\sqrt{n_{1}}}+\dfrac{s_{2}}{\sqrt{n_{2}}}$ .”

What to write

“The VARIANCES add up: $\sigma_{d}=\sqrt{\dfrac{s_{1}^{2}}{n_{1}}+\dfrac{s_{2}^{2}}{n_{2}}}$ .”

Variances add like the squares of the sides of a right triangle:

3

and

4

give

\sqrt{3^{2}+4^{2}}=5

, never

3+4=7

Why: The variance of a sum or a difference of independent variables is the sum of the variances, never the sum of the standard deviations. The minus sign takes nothing away from the uncertainty, it adds to it.

9. Doubling the sample to halve the margin

1.5 marks, and it is the sample size question

What not to write

“The margin is $4\%$ with $600$ people; with $1200$ it will drop to $2\%$ .”

What to write

“The margin decreases as $\dfrac{1}{\sqrt{n}}$ : the sample must be four times larger, that is $2400$ people, to divide it by $2$ .”

The

95\%

margin decreases as

\dfrac{1}{\sqrt{n}}

: from

4\%

with

600

people, it only drops to

2.8\%

with

1200

(red); it takes

2400

to reach

2\%

(green).

Why: The formula $n\geq\left(\dfrac{z}{2E}\right)^{2}$ squares: dividing $E$ by two multiplies $n$ by four. It is also the economic reason why polls stop at a thousand people.

10. Drawing causation from a significant chi-square

2 marks, and it is the conclusion question of the test

What not to write

“The chi-square is significant, so the type of program influences success.”

What to write

“The test establishes an ASSOCIATION between the two variables; it says nothing about the direction of the influence or about a possible confounding variable.”

Why: No statistical test produces causation without randomization. The chi-square test compares observed counts with expected counts, nothing more.

Which method to choose

Which test statistic to choose

Look at what is being estimated, and at what the question gives as the spread.

The distributions the tree chooses between: the symmetric normal, Student's t more spread out than the normal (dashed), and the skewed chi-square, whose rejection region is on the right only.

If a mean, population $\sigma$ known → $z$ -score, with $\dfrac{\sigma}{\sqrt{n}}$ in the denominator

Example: $\sigma=12$ given by the manufacturer
If a mean, $\sigma$ unknown and estimated by $s$ → Student's t distribution with $n-1$ degrees of freedom

Example: $n=12$ , $s=4$ : critical value $2.201$

beyond $n=30$ , $t$ and $z$ become almost identical
If a proportion → $z$ -score, with $p_{0}$ in the denominator for a TEST, $\hat{p}$ for an INTERVAL

Example: $\sqrt{0.5\times 0.5/400}$ under $H_{0}$
If two independent means → two-sample Student test, with the variances adding up

Example: $\sqrt{\dfrac{s_{1}^{2}}{n_{1}}+\dfrac{s_{2}^{2}}{n_{2}}}$
If a two-way table of two categorical variables → chi-square test of independence, with $(r-1)(c-1)$ degrees of freedom

Example: expected count $=\dfrac{\text{row total}\times\text{column total}}{n}$

condition: all EXPECTED counts at least $5$

None of these choices is made after the calculation: each one depends on a condition that must be checked and written down. An unchecked condition costs the mark even when the statistic is right.

Sizing the sample for a required margin

Read the required margin of error and what is being estimated.

The product

\hat{p}(1-\hat{p})

is largest at

\hat{p}=0.5

: with no information, it is the worst case that guarantees the margin; a prior estimate reduces

n

If a mean, with $\sigma$ known or estimated → $n\geq\left(\dfrac{z\sigma}{E}\right)^{2}$ , rounded UP to the next integer

Example: $z=1.96$ , $\sigma=12$ , $E=2$ give $n\geq 139$
If a proportion with no prior information → $n\geq\left(\dfrac{z}{2E}\right)^{2}$ , worst case $\hat{p}=0.5$

Example: margin of $3\%$ at $95\%$ : $n\geq 1068$
If a proportion with a prior estimate of $\hat{p}$ → $n\geq\hat{p}(1-\hat{p})\left(\dfrac{z}{E}\right)^{2}$ , always smaller

Example: $\hat{p}=0.2$ cuts the requirement by about a third
If the question asks for the effect of a change in size → reason in $\sqrt{n}$ : the margin is divided by $\sqrt{k}$ when $n$ is multiplied by $k$

Example: making $n$ four times larger divides the margin by $2$

Rounding is always UP: $138.3$ people give $139$ people, because $138$ would not be enough to keep the promised margin.

How the answer is expected to be written

A marker ticks steps. Here they are in order, with the concluding sentence expected word for word.

Writing up a complete hypothesis test

When to use it: The question asks you to test a claim, to check a manufacturer's claim or to draw a conclusion about a majority.

1 State $H_{0}$ and $H_{1}$ with the parameter as a symbol, and say whether the test is one-sided or two-sided.
2 Check the conditions and write them down: sample size, normality, $np_{0}\geq 5$ .
3 Compute the statistic, stating the value used in the denominator, $p_{0}$ or $\sigma$ .
4 Compare with the critical value, or give the p-value and compare it with $\alpha$ .
5 Conclude in the CONTEXT of the question, without ever “accepting” $H_{0}$ .

Concluding sentence

“ $z=2.00>1.645$ , so we reject $H_{0}$ at the $5\%$ level: the data support the hypothesis of a majority in favour in the population.”

The trap: Concluding on the statistic and not on the context: “we reject $H_{0}$ ” without saying what it means for the poll.

Marking: 1 mark for the hypotheses, 1 mark for the conditions, 1 mark for the statistic, 1 mark for the decision, 1 mark for the conclusion in context.

Building and interpreting a confidence interval

When to use it: The question asks you to estimate a mean or a proportion with a given confidence level.

1 Choose $z$ or $t$ depending on whether $\sigma$ is known, and write down the critical value used.
2 Compute the standard error, telling $\dfrac{\sigma}{\sqrt{n}}$ apart from $\sigma$ itself.
3 Write the interval as estimate plus or minus margin, then as bounds.
4 Interpret with a sentence about the METHOD, never about the probability of the parameter.

Concluding sentence

“The $95\%$ interval is $[0.501, 0.599]$ : if the sampling were repeated, $95\%$ of the intervals built this way would contain the true proportion.”

The trap: Using $p_{0}$ instead of $\hat{p}$ in the margin: a confidence interval assumes no hypothesis.

Marking: 1 mark for the critical value, 1 mark for the margin, 1 mark for the bounds, 1 mark for the correct interpretation.

Check before you hand in

Five minutes of checking recover more marks than one more problem started in a hurry.

The point estimate is at the centre of the interval

Check that the average of the two bounds gives back exactly $\overline{x}$ or $\hat{p}$ .

An interval $[0.501, 0.599]$ has its centre at $0.550$ : consistent with the observed proportion.
The interval and the test say the same thing

A two-sided $95\%$ interval that excludes the tested value is equivalent to a p-value below $0.05$ . If the two answers disagree, one of them is wrong.

Lower bound $0.501$ above $0.50$ , and $z=2.00$ significant: the two agree.
The order of magnitude of the standard error

The standard error is always SMALLER than the standard deviation of individuals, by a factor of $\sqrt{n}$ .

A standard error larger than $\sigma$ signals a multiplication instead of a division.
The conditions, before the conclusion

Check again that $n\geq 30$ or normality holds, that $np_{0}\geq 5$ , and that the expected counts of the chi-square test reach $5$ .

A two-way table with an expected count of $3$ rules out the test as it stands: categories must be merged.

The typical problem, taken apart

A poll reports 55 percent: can we conclude there is a majority?

Out of $400$ people surveyed at random, $220$ say they are in favour, that is $\hat{p}=0.55$ .

At the $5\%$ level, can we conclude that the proportion in favour exceeds $50\%$ in the population?

220

in favour out of

400

: the observed proportion

0.55

is above the

50\%

mark, and the whole question is whether the gap goes beyond chance.

Step 1

$H_{0}:p=0.50$ against $H_{1}:p>0.50$ , right-tailed test.

Why

The question says “exceeds”, so only one direction matters. Setting up a two-sided test would double the p-value and could reverse the conclusion.

Step 2

Conditions: $np_{0}=200\geq 5$ and $n(1-p_{0})=200\geq 5$ .

Why

They justify the normal approximation to the underlying binomial distribution. Writing them is worth a mark and takes ten seconds.

Step 3

$z=\dfrac{0.55-0.50}{\sqrt{0.50\times 0.50/400}}=\dfrac{0.05}{0.025}=2.00$ .

Why

The denominator comes from $H_{0}$ , not from the data: that is what distinguishes a test from a confidence interval, and it is the most frequent error on this question.

Step 4

One-sided critical value at $5\%$ : $1.645$ . p-value: $P(Z>2.00)=0.0228$ .

Why

Both readings lead to the same decision, and giving both secures the mark: $2.00>1.645$ and $0.0228<0.05$ .

Step 5

$95\%$ interval: $0.55\pm 1.96\sqrt{\dfrac{0.55\times 0.45}{400}}=[0.501, 0.599]$ .

Why

Here $\hat{p}$ is what is used, since there is no hypothesis. The interval excludes $0.50$ , which confirms the test by an independent route.

Step 6

Check: the centre of the interval is $0.550$ , and $220/400=0.55$ .

Why

The centre must give back the point estimate. A gap would signal a margin wrongly subtracted on one side.

The conclusion, written out

“ $z=2.00>1.645$ and the p-value is $0.0228<0.05$ : we reject $H_{0}$ . The data support the hypothesis of a majority in favour in the population, with a $95\%$ interval of $[50.1\%, 59.9\%]$ .”

The classic mistake on this problem: Writing the denominator with $0.55\times 0.45$ : this gives $z=2.01$ , almost the same here, but the method is wrong and the gap becomes decisive when $\hat{p}$ moves away from $0.5$ .

Learn by heart

• Standard error $=\dfrac{\sigma}{\sqrt{n}}$ : making $n$ four times larger only divides the margin by $2$ .
• A $95\%$ interval describes the METHOD, never the probability of the parameter, which is fixed.
• p-value: the probability of a result at least as extreme IF $H_{0}$ is true. It is not $P(H_{0})$ .
• We reject or we DO NOT REJECT $H_{0}$ ; we never accept it.
• Test on a proportion: $p_{0}$ in the denominator. Interval for a proportion: $\hat{p}$ in the denominator.
• $\sigma$ unknown: Student's t distribution with $n-1$ degrees of freedom, critical values larger than those of $z$ .
• For a difference, the VARIANCES add up: $\sqrt{\dfrac{s_{1}^{2}}{n_{1}}+\dfrac{s_{2}^{2}}{n_{2}}}$ .
• Chi-square: expected counts all greater than or equal to $5$ , and association, never causation.

Frequently asked questions

What does a 95% confidence interval really mean?

That the method used produces, over all possible samples, intervals of which ninety-five percent contain the true parameter. The parameter is a fixed number: it is either in the interval obtained or it is not. The probability therefore describes the procedure, not the position of the parameter.

Is the p-value the probability that the null hypothesis is true?

No. It is the probability of observing a result at least as extreme as the one obtained, assuming the null hypothesis is true. It is computed under that hypothesis, so it cannot measure how likely the hypothesis is. A small p-value means the data would be surprising if the null hypothesis held.

Why do we never say that we accept the null hypothesis?

Because a test only looks for evidence against it. Not finding any may mean that it is true, but also that the sample was too small to detect a real gap. So we say that we do not reject it, which leaves the second possibility open.

When should Student's t distribution be used instead of the normal distribution?

Whenever the population standard deviation is unknown and is estimated from the sample. That estimate adds uncertainty, which Student's t distribution handles with heavier tails and therefore larger critical values. Beyond thirty or so observations, the two distributions become almost identical.

Is a significant result necessarily important?

No. Significance measures the strength of the evidence, and it depends mostly on the sample size. With ten thousand observations, a tiny gap becomes significant without having the slightest practical consequence. Always look at the estimated size of the effect, which the confidence interval gives directly.

Practise it

Corrected exercises: Estimation and hypothesis testing, 201-337

A method is proved on a paper, not on a sheet. The set for the same chapter takes each of these traps into a problem, with the solution written out step by step.

11 corrected exercises
110 points
150 minutes

Do the exercises

Something missing from this sheet? A method, a trap, an exercise to practise it, a mistake you spotted: tell me, I read every request.

Sending goes through your free space: that is how I can answer you and tell you when the exercise is added. An email is enough, no password. Create your space. Rather not create an account? Write to me directly.

Previous sheet Continuous random variables and density Next sheet Correlation and linear regression

© Ahmed Squalli Houssaini. Revision sheet published at www.letuteurscientifique.ca/en/fiches/cegep-337-estimation-hypothesis-tests. Free for personal and classroom use; republishing it elsewhere requires written permission (legal notice).

Looking for a 201-337 tutor in Montreal?

Get in touch for a first session. Inference is the chapter where method matters more than calculation: five steps that never change, and half of the marks come from the write-up.

Book a session WhatsApp

Site by Studio Squalli

Revision sheet: estimation and hypothesis testing (201-337)

The essentials

The sampling distribution and the standard error

What a confidence interval says, and what it does not say

The logic of a test

Student, and the threshold of thirty

The mistakes that cost marks

1. Giving the parameter the probability of the method

2. Dividing by n instead of the square root of n

3. Reading the p-value as the probability that the null hypothesis is true

4. Accepting the null hypothesis

5. Putting the observed proportion in the denominator of the test

6. Using z when the population standard deviation is unknown

7. Confusing significant with important

8. Adding the standard deviations of a difference

9. Doubling the sample to halve the margin

10. Drawing causation from a significant chi-square

Which method to choose

Which test statistic to choose

Sizing the sample for a required margin

How the answer is expected to be written

Writing up a complete hypothesis test

Building and interpreting a confidence interval

Check before you hand in

The typical problem, taken apart

A poll reports 55 percent: can we conclude there is a majority?

Learn by heart

Frequently asked questions

What does a 95% confidence interval really mean?

Is the p-value the probability that the null hypothesis is true?

Why do we never say that we accept the null hypothesis?

When should Student's t distribution be used instead of the normal distribution?

Is a significant result necessarily important?

Corrected exercises: Estimation and hypothesis testing, 201-337

See also

Looking for a 201-337 tutor in Montreal?