My account Contact me

Quantitative Methods in Social Science 360-300 • CEGEP, Montreal

Revision sheet: the margin of error of a poll (360-300)

This sheet deals with the most quoted and most misunderstood figure of the whole course: the margin of error of a poll. It does not repeat the formula, it gives the eight places where this figure is applied to something other than what it measures, and the exact sentence that repairs each one.

It is meant for Social Science students at CEGEP taking the compulsory Quantitative Methods course, 360-300, and it is useful to anyone who reads an election night. The corrected exercise set on the same chapter goes back over every computation, from the subgroup to the aggregation of several firms.

Mark this sheet as read or add it to your favourites: a free account, no password, keeps your read sheets and favourites from one visit to the next, tells you which chapter to tackle next and lets you ask for the method or exercise you are missing. Create your space, an email is enough.

The thread of the chapter

The margin of error covers ONLY sampling error, and it applies to ONE figure at a time. Almost every mistake in the chapter comes from applying it to a subgroup, a gap or a change over time without recomputing it.

This chapter is part of 360-300, Quantitative Methods in Social Science

The essentials

What the margin measures, and what it ignores

  • • E=zp(1−p)nE=z\sqrt{\dfrac{p(1-p)}{n}}, with z=1.96z=1.96 at 95%, the level known as “19 times out of 20”.
  • • The INTERVAL is the random quantity, the true proportion is FIXED: 95% of the intervals built this way contain the target.
  • • The margin covers ONLY sampling error. Coverage, nonresponse, mode of collection and wording come on top of it and often exceed it.
  • • Handy approximation: margin in points ≈100n\approx \dfrac{100}{\sqrt{n}} for pp near 0.50.5.
the curve is almost flat between 0.34 and 0.66which is why a single margin is announced0.20.40.60.81proportion p
Between p=0.34p=0.34 and p=0.66p=0.66, the curve barely drops, from 3.103.10 to 2.942.94 points: this is why a single margin, computed at 5050%, is enough for the whole poll.

No margin can be computed without a PROBABILITY sample: on a panel of volunteers the formula runs and the result means nothing.

One margin per figure, never one margin for everything

  • • SUBGROUP: redo the computation with the size of the subgroup. A group NINE times smaller has a margin THREE times larger.
  • • GAP between two candidates of the SAME poll: Var=p1(1−p1)+p2(1−p2)+2p1p2n\text{Var}=\dfrac{p_{1}(1-p_{1})+p_{2}(1-p_{2})+2p_{1}p_{2}}{n}, and the margin on the gap is about DOUBLE the ordinary margin.
  • • Difference between two INDEPENDENT samples: the variances add, with no covariance term.
  • • The OVERLAP of two intervals is too severe a criterion: build the interval of the DIFFERENCE and check whether it contains zero.
the two intervals overlapand the gap still contains zeroA: 42%B: 38%gap0
The intervals of A and B overlap widely, and the interval of their gap still contains zero: the four-point lead is not established.

The term 2p1p22p_{1}p_{2} comes from the two proportions being NEGATIVELY correlated: a respondent won by one is lost by the other.

The two corrections and the two reading traps

  • • FINITE POPULATION correction: multiply the margin by N−nN−1\sqrt{\dfrac{N-n}{N-1}}, useful as soon as nn exceeds about 5% of NN.
  • • Crossing a THRESHOLD fixed in advance, 5050% or two thirds: the LOWER BOUND of the interval must exceed the threshold.
  • • MULTIPLE COMPARISONS: at the 5% level, one comparison in twenty reaches a wrong conclusion. Over 6666 comparisons, 3.33.3 false ones are expected.
  • • HOUSE EFFECT: each firm has its own systematic bias; aggregating several polls reduces sampling error, never that bias.
3638404244464850firm 1firm 2firm 3firm 4true valuemean of the 16
Illustration of the house effect: each firm draws its polls around its own bias. The mean of the sixteen stays away from the true value: aggregating does not remove a shared bias.

Aggregating ten polls of a thousand people does not give the precision of a poll of ten thousand: shared biases do not average out.

The mistakes that cost marks

These are the errors I correct most often in session. Each one costs marks on a paper, even when the reasoning behind it is right.

1. Saying that the true value has a 95% chance of being in the interval

1 point of wording, taken off every conclusion

What not to write

“There is a 9595% chance that the real proportion is between 38.938.9 and 45.145.1%.”

What to write

“If the poll were repeated a large number of times, 9595% of the intervals built this way would contain the real proportion.”

36384042444648true value, fixed
Twenty simulated polls of 1,000 people: the real proportion, 42%, does not move, it is the intervals that change; here one in twenty misses it (red).

Why: The real proportion is a FIXED number, unknown but not random. What varies from one poll to the next is the interval, so it is the interval that the probability describes.

2. Believing that the margin covers every flaw of the poll

1.5 points, and it is the critique the exam expects

What not to write

“The margin is 3.13.1 points: the total error therefore does not exceed three points.”

What to write

“The margin covers only SAMPLING error. Nonresponse, coverage and wording add an error that is not quantified and often exceeds it.”

303438424650true value 40poll: 46 ± 1.6nonresponse bias
Illustration: a precise poll, at 1.6 points, but shifted by nonresponse. Its narrow interval does not contain the true value: the margin ignores biases.

Why: The formula knows only nn and pp: it can only describe the variability of the draw. Everything else is a bias, and a bias does not shrink when the sample grows.

3. Applying the overall margin to a subgroup

2 points, and a conclusion about young people that does not hold

What not to write

“Among 18 to 24 year olds, 5555% are in favour, give or take 3.13.1 points.”

What to write

“This subgroup has only 110110 respondents: its margin is 1.960.55×0.45110≈9.31.96\sqrt{\dfrac{0.55\times 0.45}{110}}\approx 9.3 points, so from 45.745.7 to 64.364.3%.”

subgroup sizemargin in points1,0003.15004.42506.21109.39 times smaller, 3 times the margin
From 1,0001{,}000 to 110110 respondents, the margin goes from 3.13.1 to 9.39.3 points: the ratio of the margins is the square root of the ratio of the sizes.

Why: The margin depends on n\sqrt{n}: nine times fewer people triples the margin. This is the most frequent mistake in poll commentary.

4. Comparing the gap between two candidates with the ordinary margin

the whole conclusion, and it is the standard election night question

What not to write

“A leads 4242 to 3838: the 44-point gap exceeds the margin of 3.13.1, so the lead is real.”

What to write

“The gap has its OWN margin, about double: 5.55.5 points here. The interval of the gap runs from −1.5-1.5 to 9.59.5 points, so it contains zero.”

Why: A respondent won by A is often lost by B: the two proportions are negatively correlated, which the term 2p1p22p_{1}p_{2} expresses in the variance of the gap.

5. Concluding a tie because the intervals overlap

1.5 points, and sometimes a reversed conclusion

What not to write

“The intervals of A and B overlap: the two candidates are tied.”

What to write

“Overlap is a criterion that is TOO SEVERE. You must build the interval of the DIFFERENCE and check whether it contains zero: it is not the same thing.”

3438424650wave 2: 45%wave 1: 40%−404812difference: 0.7 to 9.3
Two independent waves of 1,000, at 40% and 45%: the intervals overlap, and yet the interval of the difference, from 0.7 to 9.3 points, excludes zero.

Why: Two intervals can overlap slightly while the interval of the difference excludes zero. Overlap therefore makes you miss real gaps.

6. Announcing a majority at 51%

2 points, and the question is often exactly about this

What not to write

“The YES side gets 5151%: it has a majority.”

What to write

“With a margin of 3.13.1 points, the interval runs from 47.947.9 to 54.154.1%: the LOWER BOUND does not exceed 5050%, so the majority is not established.”

4446485052545658threshold 50%47.954.1
51% from 1,000 respondents: the interval runs from 47.9 to 54.1%. Its lower bound stays below the 50% threshold, so the majority is not established.

Why: Crossing a threshold fixed in advance is judged on the bound on the right side, never on the point estimate. It is the same logic as an interval that must exclude zero.

7. Forgetting the finite population correction

1 point, and an underestimated precision

What not to write

“1,0001{,}000 respondents out of the 4,0004{,}000 students of the CEGEP: the margin is 3.13.1 points.”

What to write

“The sample is 2525% of the population: we multiply by 4000−10003999≈0.866\sqrt{\dfrac{4000-1000}{3999}}\approx 0.866, so the margin drops to 2.72.7 points.”

0.250.50.7510.250.50.75125%: × 0.866share sampled n / Nfactor
The correction factor as a function of the share sampled: at 25% of the population, the margin is multiplied by 0.866; at 100%, it drops to zero.

Why: Every person interviewed removes uncertainty about those who remain. If the whole population were interviewed, the margin would drop to zero, which the uncorrected formula cannot do.

8. Drawing a conclusion from the largest difference in a table of comparisons

the whole regional conclusion

What not to write

“Among the 6666 regional comparisons, this one is significant at 5%: the region stands out.”

What to write

“Out of 6666 comparisons at the 5% level, 3.33.3 are expected to be significant by pure chance. The threshold must be corrected, or the analysis announced as exploratory.”

66 comparisons with no real difference:about 3 “significant” (red)
Sixty-six comparisons at the 5% level with no real difference: 3.3 false positives are expected, here three red squares.

Why: Picking the largest difference AFTER looking at all sixty-six amounts to running sixty-six tests and reporting only one: the real threshold no longer has anything to do with 5%.

Which method to choose

Which margin should you compute?

We look at WHAT the claim to check is about, and how many people it rests on.

  • If a single percentage, on the whole sample → ordinary margin E=zp(1−p)nE=z\sqrt{\dfrac{p(1-p)}{n}}, with the total nn

    Example: 4242% from 1,0001{,}000 respondents, that is 3.13.1 points

  • If a percentage within a SUBGROUP → same formula, but with the size of the SUBGROUP

    Example: 5555% among 110110 young people, that is 9.39.3 points

  • If the gap between TWO candidates of the same poll → variance with the term 2p1p22p_{1}p_{2}, margin about DOUBLE

    Example: 44 points of gap, margin of 5.55.5 points

  • If the difference between two WAVES or two independent samples → add the two variances, with no covariance term

    Example: two polls of a thousand people a month apart

  • If we want to know whether a THRESHOLD is crossed → compare the BOUND of the interval with the threshold, never the estimate

    Example: 5151% with a lower bound at 47.947.9%

If the sample is not probability-based, no branch applies: saying so IS the expected answer.

Is this conclusion allowed?

We check the claim of the text against what the interval really allows.

  • If the interval of the gap contains zero → we can NOT conclude to a lead: say “the two candidates are statistically tied”

    Example: gap of 44 points, interval from −1.5-1.5 to 9.59.5

  • If the interval of the gap excludes zero → the lead is established, and we give the INTERVAL of that lead

    Example: gap of 88 points, interval from 2.52.5 to 13.513.5

  • If the lower bound exceeds the target threshold → the threshold is crossed, and we can announce it

    Example: 5656% with a lower bound at 52.952.9%

  • If the result comes from a subgroup not planned at the start → flag the exploration and the margin specific to the subgroup

    Example: one region out of twelve

A lead that is not significant is not a proven tie: it means the poll does not settle the question, which is a third answer and not a draw.

How the answer is expected to be written

A marker ticks steps. Here they are in order, with the concluding sentence expected word for word.

Writing up a confidence interval on a proportion

When to use it: As soon as the problem gives a percentage and a sample size, and asks for the precision or the scope of the result.

  1. 1 Check and say that the sample is PROBABILITY-BASED, without which the computation has no place.
  2. 2 Write pp, nn and the zz of the announced confidence level.
  3. 3 Compute E=zp(1−p)nE=z\sqrt{\dfrac{p(1-p)}{n}} and express it in percentage POINTS.
  4. 4 Give the interval [p−E,p+E][p-E, p+E] with both bounds.
  5. 5 State the conclusion in terms of the INTERVALS and not of the true value.

Concluding sentence

“With p=0.42p=0.42 from 1,0001{,}000 randomly drawn respondents, the margin is 3.13.1 points at the 95% confidence level: the interval runs from 38.938.9 to 45.145.1%. If the poll were repeated, 9595% of the intervals built this way would contain the real proportion.”

The trap: Writing “there is a 95% chance that the true value is in the interval”: this is the wording the marker penalizes.

Marking: 1 mark for the probability-based nature, 1 mark for the formula, 1 mark for the bounds, 1 mark for the wording.

Settling a gap between two candidates

When to use it: When the problem asks whether one candidate's lead is real.

  1. 1 Write the two proportions and the size of the poll, and recall that they come from the SAME sample.
  2. 2 Compute the variance of the gap with the term 2p1p22p_{1}p_{2}.
  3. 3 Deduce the margin on the gap, and compare it with the ordinary margin to show that it is about double.
  4. 4 Build the interval of the gap and check whether it contains ZERO.
  5. 5 Conclude in three exact words: lead established, or poll that does not settle it.

Concluding sentence

“The observed gap is 44 points and its margin 5.55.5 points: the interval runs from −1.5-1.5 to 9.59.5 points and contains zero. The poll therefore does not allow us to conclude that A leads B.”

The trap: Concluding “the two candidates are tied”: the poll does not settle it, which is not the same as proving equality.

Marking: 1 mark for the variance of the gap, 1 mark for the margin, 1 mark for the interval, 1 mark for the nuanced conclusion.

Check before you hand in

Five minutes of checking recover more marks than one more problem started in a hurry.

The typical problem, taken apart

Election night, from the overall poll to the subgroup

A probability poll of 1,0001{,}000 respondents gives 4242% to candidate A, 3838% to candidate B and 2020% to candidate C.

Among the 110110 respondents aged 18 to 24, 5555% support A.

We are asked for the margin on each figure, whether A's lead is established, and whether A has a majority among young people.

42%A38%B20%C55%A, ages 18-241,000 respondents110 respondents
The figures of the problem: three candidates from 1,000 respondents, and A's score among the 110 respondents aged 18 to 24.

Step 1

Announced margin, computed at p=0.5p=0.5: 1.960.251000≈3.101.96\sqrt{\dfrac{0.25}{1000}}\approx 3.10 points. On A precisely: 3.063.06 points.

Why

The two numbers differ by four hundredths of a point: this is why firms announce a single margin computed in the worst case, that of p=0.5p=0.5.

Step 2

Interval for A: from 38.938.9 to 45.145.1%. Interval for B: from 34.934.9 to 41.141.1%.

Why

The two intervals overlap, but that observation is not enough: overlap is too severe a criterion and does not replace the interval of the gap.

Step 3

Variance of the gap: 0.42(0.58)+0.38(0.62)+2(0.42)(0.38)1000=0.000798\dfrac{0.42(0.58)+0.38(0.62)+2(0.42)(0.38)}{1000}=0.000798, so a margin of 5.545.54 points.

Why

The term 2p1p22p_{1}p_{2} accounts for 4040% of the variance: without it, the margin would be underestimated, at 4.34.3 points instead of 5.55.5.

Step 4

Interval of the gap: 4±5.544\pm 5.54, that is from −1.5-1.5 to 9.59.5 points. It contains zero.

Why

The four-point lead is therefore not established. That is the expected answer, and it contradicts the spontaneous reading of the table.

Step 5

Subgroup of the 18 to 24 year olds: 1.960.55(0.45)110≈9.301.96\sqrt{\dfrac{0.55(0.45)}{110}}\approx 9.30 points, that is from 45.745.7 to 64.364.3%.

4045505560657050%45.764.3

Why

Nine times fewer respondents triple the margin. The interval still contains 5050%, so a majority among young people is not established either.

Step 6

If the population were only 4,0004{,}000 people, the overall margin would be multiplied by 30003999≈0.866\sqrt{\dfrac{3000}{3999}}\approx 0.866, that is 2.682.68 points.

Why

This correction does not apply to an electorate of several million, but it changes everything in a survey carried out in a single CEGEP.

The conclusion, written out

“With a margin of 3.13.1 points, A is estimated between 38.938.9 and 45.145.1%. The four-point gap over B has its own margin of 5.55.5 points and its interval contains zero: the poll does not allow us to conclude that A leads. Among 18 to 24 year olds, the margin reaches 9.39.3 points and a majority is no more established.”

The classic mistake on this problem: Comparing the four-point gap with the 3.13.1-point announced margin: that is the margin of ONE percentage, not of a gap, and it is almost twice too small here.

Learn by heart

  • • E=zp(1−p)nE=z\sqrt{\dfrac{p(1-p)}{n}}, with z=1.96z=1.96 at 95%. Approximation: 100n\dfrac{100}{\sqrt{n}} points for pp near 0.50.5.
  • • It is the INTERVAL that is random, never the true proportion: 9595% of the intervals built this way contain the target.
  • • The margin covers ONLY sampling error. Nonresponse, coverage and wording come on top of it without being quantified.
  • • SUBGROUP: recompute with ITS size. Nine times smaller, three times the margin.
  • • GAP between two candidates of the same poll: margin about DOUBLE, because of the term 2p1p22p_{1}p_{2}.
  • • The OVERLAP of two intervals is too severe: it is the interval of the DIFFERENCE that decides, according to whether it contains zero.
  • • Crossing a threshold is judged on the BOUND of the interval, never on the point estimate.

Frequently asked questions

What does a margin of error of 3 points, 19 times out of 20, mean?

That if the same poll were repeated a large number of times, in nineteen cases out of twenty the interval built around the result would contain the real proportion. The real proportion is a fixed number: what varies from one poll to the next is the interval. The wording matters, and it is in the marking scheme.

Why is the margin larger for a subgroup?

Because it depends on the square root of the size. A subgroup of one hundred and ten people in a poll of a thousand has a margin three times larger, about nine points instead of three. That is why comments about young people, a region or a profession are almost always more fragile than they look.

How can you tell whether a candidate's lead is real?

You compute the margin specific to the gap, which is about double the ordinary margin because the two proportions are negatively correlated. You then build the interval of that gap: if it contains zero, the poll does not allow us to conclude that there is a lead. Comparing the gap with the margin of a single percentage is a mistake.

Do two overlapping intervals mean a tie?

No. Overlap is a criterion that is too severe: two intervals can overlap slightly while the interval of their difference excludes zero, so the gap is well established. It is always the interval of the difference that decides, and it alone.

Does a poll at 51 percent announce a majority?

Not necessarily. With a thousand respondents, the margin is about three points, so the interval runs from 47.9 to 54.1 percent. The lower bound does not exceed fifty, so the majority is not established. Crossing a threshold fixed in advance is always judged on the bound, never on the estimate.

Practise it

Corrected exercises: The margin of error of a poll, 360-300

A method is proved on a paper, not on a sheet. The set for the same chapter takes each of these traps into a problem, with the solution written out step by step.

  • 10 corrected exercises
  • 100 points
  • 150 minutes
Do the exercises
Something missing from this sheet? A method, a trap, an exercise to practise it, a mistake you spotted: tell me, I read every request.

Sending goes through your free space: that is how I can answer you and tell you when the exercise is added. An email is enough, no password. Create your space. Rather not create an account? Write to me directly.

What is it about?
I read every request myself, nobody else sees it. No need to write your name or your school.
When your request is dealt with, you hear about it by email and in your space.

By sending, you agree that I read your request to answer it. It is kept with your account, erased with it, and never published: privacy policy.

Previous sheet Cross-tabulations, chi-square and association Next sheet Research design and critical reading

© Ahmed Squalli Houssaini. Revision sheet published at www.letuteurscientifique.ca/en/fiches/cegep-qm-poll-margin-of-error. Free for personal and classroom use; republishing it elsewhere requires written permission (legal notice).

See also

All revision sheets All exercise sets Fiche de révision: la marge d'erreur d'un sondage (in French)

Looking for a 360-300 tutor in Montreal?

Get in touch for a first session. We go back over the mistakes that cost marks on the margin of error, from the subgroup to the majority threshold, then test them on exercises at the level of real CEGEP tests.

Book a session WhatsApp
Site by Studio Squalli