Corrected exercises: the margin of error of a poll (360-300)

$E=1.96\sqrt{\dfrac{0.42\times 0.58}{1{,}067}}=1.96\sqrt{\dfrac{0.2436}{1{,}067}}=1.96\sqrt{0.00022831}=1.96\times 0.015110\approx 0.0296$, that is 3.0 points. The announced margin is therefore exact, and note that it is computed with $p=0.42$; the firm could have used $p=0.5$, which would also have given 3.0 points, the difference being invisible to the nearest tenth.

Interval: $\left[42.0-3.0, 42.0+3.0\right]=\left[39.0\%, 45.0\%\right]$. What it means: if this poll were repeated a very large number of times, with the same method and the same size, 95% of the intervals built this way would contain the true proportion of the population. The property is about the PROCEDURE, not about this particular interval.

The probability is about the INTERVAL, which is the random quantity, and not about the true value, which is a fixed, unknown number. In the frequentist framework, the true proportion does not “land” anywhere: it is where it is. It is the interval that varies from one sample to the next, and we guarantee that 19 times out of 20 it will catch its target. The image of the ring toss helps: the ring moves, the peg is fixed.

The sentence reverses the subject of the probability, attributing a 95% chance to the position of the true value when that value is fixed. In practice the mistake is minor: the numerical result is the same, and Bayesian statisticians, starting from a neutral prior, would obtain an almost identical credible interval whose interpretation would be precisely the reader's. The nuance is still worth knowing, because it explains why an interval says nothing about the probability that a party exceeds 45%: that question needs a different framework.

No, and this is the most important limit of the whole chapter. The margin of error measures ONLY the variability due to the random draw of the sample from a perfect sampling frame, with a 100% response rate. It ignores undercoverage, nonresponse, the effect of the mode of collection, the wording of the question and the order of the choices. Post-mortems of failed elections regularly show that these sources exceed the sampling margin, which is why a gap of four or five points between a poll and the result remains possible even when everyone has done their job correctly.

$p=0.50$: $1.96\sqrt{\dfrac{0.25}{1000}}=1.96\times 0.015811\approx 0.0310$, that is 3.10 points. $p=0.30$: $1.96\sqrt{\dfrac{0.21}{1000}}\approx 0.0284$, that is 2.84 points. $p=0.10$: $1.96\sqrt{\dfrac{0.09}{1000}}\approx 0.0186$, that is 1.86 points. $p=0.03$: $1.96\sqrt{\dfrac{0.0291}{1000}}\approx 0.0106$, that is 1.06 points.

The margin is proportional to $\sqrt{p(1-p)}$, a function that is maximal at 0.5 and very flat around it. To stay within a tenth of a point of 3.10, we need $\sqrt{p(1-p)}\ge \dfrac{3.00}{3.10}\times 0.5=0.4839$, so $p(1-p)\ge 0.23415$, which holds for $p$ between about 0.374 and 0.626. In other words, for any proportion between 37% and 63%, the margin announced at 50% is correct to the nearest tenth of a point: this plateau is what justifies the use of a single margin.

Real margin for 3%: 1.06 points, against the 3.10 announced, nearly three times smaller. Expressed relative to the proportion itself, however, the margin remains enormous: $\dfrac{1.06}{3.0}\approx 35\%$ of the estimated value, against $\dfrac{3.10}{50}\approx 6\%$ for a party at 50%. That is the useful paradox of the question: in points, small parties are measured more precisely; relative to what they are worth, they are measured far worse.

Because announcing one margin per row of a table would assume that the reader knows which estimate each margin belongs to, and above all because the margin at 50% is the LARGEST: announcing it guarantees that no result in the poll is presented as more precise than it is. It is a conservative convention, hence defensible. What is not defensible is applying this single margin to a subgroup or to a small party when the question is specifically about them.

The wording is cautious and correct, and it is better than a number. With $n=1{,}000$ and a point estimate of 0.5%, the margin computed by the normal formula is about 0.44 points, but the formula itself is no longer valid: it requires $np\ge 5$ and $n(1-p)\ge 5$, and here $np=5$ just barely. We are leaving the domain where the normal approximation applies, and the correct interval comes from an exact method, which would give something like 0.1% to 1.2%. Writing “less than 1%” honestly sums up this imprecision.

Size of the subgroup: $0.11\times 1{,}000=110$ respondents. Margin: $1.96\sqrt{\dfrac{0.58\times 0.42}{110}}=1.96\sqrt{\dfrac{0.2436}{110}}=1.96\times 0.047056\approx 0.0922$, that is 9.2 points. The real interval therefore runs from 48.8% to 67.2%, which leaves very different conclusions open.

The margin goes from 3.1 to 9.2 points, a factor of about 3.0. There is nothing mysterious about this factor: it equals $\sqrt{\dfrac{1000}{110}}=\sqrt{9.09}\approx 3.0$, since the margin decreases as $\dfrac{1}{\sqrt{n}}$. This shortcut is worth remembering over the full formula: a subgroup that makes up one ninth of the sample has a margin three times larger.

We need $1.96\sqrt{\dfrac{0.2436}{n}}\le 0.05$, so $n\ge \dfrac{1.96^{2}\times 0.2436}{0.05^{2}}=\dfrac{0.93602}{0.0025}\approx 375$ respondents in the 18 to 24 subgroup alone. To get this without changing the proportions, the total sample would have to be about $\dfrac{375}{0.11}\approx 3{,}400$ people, which triples the cost of the survey. That is why polling firms deliberately OVERSAMPLE the subgroups the client cares about, then reweight.

No. The margin on the difference of two proportions from independent groups is $1.96\sqrt{\dfrac{0.58\times 0.42}{110}+\dfrac{0.47\times 0.53}{260}}=1.96\sqrt{0.0022145+0.0009581}=1.96\sqrt{0.0031726}=1.96\times 0.056327\approx 0.1104$, that is 11.0 points. The observed gap is 11 points, so right at the exact limit of the threshold: we cannot conclude. Note that the margin on the gap is governed by the SMALLER of the two groups, whose variance dominates the sum.

“Before quoting a subgroup result, ask for the size of that subgroup, and redo the margin with it.” In practice, most firms refuse to publish a result below 100 respondents, and many require 200. A complementary rule of thumb: dividing 100 by the square root of the size gives a good approximation of the margin in points, here $\dfrac{100}{\sqrt{110}}\approx 9.5$, which is more than enough to decide whether a headline holds up.

Candidate A: $\left[38.9, 45.1\right]$. Candidate B: $\left[34.9, 41.1\right]$. They overlap over the whole zone from 38.9 to 41.1, a common width of 2.2 points. A hurried reader would conclude that nothing can be said.

No, the overlap of intervals is a criterion that is TOO SEVERE. Two intervals can overlap slightly while the difference is significant: the quantity to test is the gap itself, with its own uncertainty, and that uncertainty is not the sum of the two widths. Comparing two intervals therefore leads us to miss real differences, and it is one of the most widespread reading errors in scientific reports and in the press alike.

Variance of the gap: $\dfrac{0.42\times 0.58+0.38\times 0.62+2\times 0.42\times 0.38}{1000}=\dfrac{0.2436+0.2356+0.3192}{1000}=\dfrac{0.7984}{1000}=0.0007984$. Standard deviation: $\sqrt{0.0007984}\approx 0.028256$. Margin: $1.96\times 0.028256\approx 0.05538$, that is 5.5 points. The term $2p_{1}p_{2}$ comes from the negative correlation between the two proportions in a single poll, and it is far from negligible: on its own it contributes 40% of the variance.

The observed lead is 4.0 points and the margin on the gap is 5.5 points: the interval of the lead runs from $-1.5$ to $+9.5$ points. It contains zero, so equality remains plausible and the lead is not established. To conclude, we would need a lead above about 5.5 points, that is a gap of at least 6 points given the published rounding. This is exactly the rule of thumb that serious commentators apply before talking about a lead.

Because the ratio between the two margins is stable around the usual values. Here, $\dfrac{5.54}{3.06}\approx 1.8$. In the limiting case of two candidates at 50% each, the variance of the gap is $\dfrac{0.25+0.25+0.5}{n}=\dfrac{1}{n}$, against $\dfrac{0.25}{n}$ for a single proportion: the ratio of the standard deviations is then exactly 2. Hence the mnemonic: double the displayed margin before commenting on a lead.

$E=1.96\sqrt{\dfrac{0.52\times 0.48}{1000}}=1.96\sqrt{0.00024960}=1.96\times 0.015799\approx 0.03097$, that is 3.1 points. Interval: $\left[48.9\%, 55.1\%\right]$. It CONTAINS 50%, so we cannot conclude to a majority: the NO side remains plausible. The correct wording is “YES is ahead, without a majority being established.”

The lower bound must exceed 50%, so $p-1.96\sqrt{\dfrac{p(1-p)}{1000}}>0.50$. Approximating with $\sqrt{p(1-p)}\approx 0.5$, we need $p>0.50+0.031=0.531$, that is about 53.1%. An exact computation with $p=0.531$ gives $E=0.03094$ and a lower bound of 50.01%: the threshold value is indeed a little over 53%. Remember the order of magnitude: with a thousand respondents, you need a three-point lead over the bar to declare it crossed.

We need $0.02>1.96\sqrt{\dfrac{0.2496}{n}}$, so $n>\dfrac{1.96^{2}\times 0.2496}{0.02^{2}}=\dfrac{0.958816}{0.0004}\approx 2{,}397$ respondents. The sample would therefore have to be more than doubled to settle a two-point gap above the threshold, which illustrates the square law once more: dividing the margin by 1.55 requires multiplying $n$ by 2.4.

$E=1.96\sqrt{\dfrac{0.69\times 0.31}{400}}=1.96\sqrt{0.00053475}=1.96\times 0.023125\approx 0.04533$, that is 4.5 points. Interval: $\left[64.5\%, 73.5\%\right]$. The two-thirds threshold is 66.67%, and it is CONTAINED in the interval: the qualified majority is therefore not assured. A union announcing the strike as won would be taking a real risk, which the computation quantifies precisely.

No, not necessarily. The real value of 64% falls just below the lower bound of the interval, by 0.5 points, which is compatible with the 5% of cases where the interval misses its target. But two other explanations are even more likely: the population may have CHANGED its mind in ten days, a strike vote being preceded by an intense campaign; and the actual voters are not the poll respondents, turnout itself being selective. A poll estimates opinion at the moment it is taken, in the population it interviewed; it does not predict a future vote, and confusing the two is the source of nearly all the accusations of error levelled at polling firms.

Wave 1: $1.96\sqrt{\dfrac{0.38\times 0.62}{1000}}=1.96\times 0.015346\approx 0.03008$, that is 3.0 points. Wave 2: $1.96\sqrt{\dfrac{0.43\times 0.57}{1000}}=1.96\times 0.015657\approx 0.03069$, that is 3.1 points. The two intervals, $\left[35.0, 41.0\right]$ and $\left[39.9, 46.1\right]$, overlap over a little more than one point.

Variance of the difference: $\dfrac{0.38\times 0.62}{1000}+\dfrac{0.43\times 0.57}{1000}=0.00023560+0.00024510=0.00048070$. Standard deviation: $\sqrt{0.00048070}\approx 0.021925$. Margin on the change: $1.96\times 0.021925\approx 0.04297$, that is 4.3 points. Note that this margin is smaller than the one on the gap between two candidates of the same poll, computed in exercise 4: here the two measurements are independent, so there is no covariance term to add.

The observed change is 5.0 points, and the margin on this change is 4.3 points: the interval runs from $+0.7$ to $+9.3$ points, and it does not contain zero. The increase is therefore established at the 5% level, barely. Note that simply checking the overlap of the two individual intervals would have led to the opposite conclusion: this directly illustrates that this criterion is too severe.

It would be SMALLER, often much smaller. With a panel in which the same people are interviewed again, the two measurements are positively correlated, since most respondents do not change their minds. The variance of the difference becomes $\sigma_{1}^{2}+\sigma_{2}^{2}-2\rho\sigma_{1}\sigma_{2}$, and the subtracted term can be considerable. This is the statistical argument for longitudinal panels: they detect changes far better than two independent samples of the same size, at the price of attrition and of conditioning the respondents.

Twelve waves give eleven successive comparisons. At the 5% level, each comparison has a one-in-twenty chance of wrongly concluding that something changed, so on average $11\times 0.05=0.55$ false detection a year. If the firm also compares each wave with all the previous ones, the number of comparisons rises to $\dfrac{12\times 11}{2}=66$, so $66\times 0.05=3.3$ false detections a year. This is the problem of MULTIPLE COMPARISONS, and it explains a good part of the “trends” commented on in the media: if you look long enough, you always find something.

Sum: $41.0+43.5+39.5+42.0+44.5+40.0+42.5+41.5+43.0+40.5=418.0$. Mean: $\dfrac{418.0}{10}=41.8\%$.

Ten polls of 1,000 amount to a sample of 10,000, so $E=1.96\sqrt{\dfrac{0.418\times 0.582}{10{,}000}}=1.96\times 0.0049325\approx 0.00967$, that is about 1.0 point instead of 3.1. This is the gain advertised by aggregators, and it is real to the extent that the independence assumption holds.

Deviations from the mean: $-0.8$; $1.7$; $-2.3$; $0.2$; $2.7$; $-1.8$; $0.7$; $-0.3$; $1.2$; $-1.3$. Sum of squares: $0.64+2.89+5.29+0.04+7.29+3.24+0.49+0.09+1.44+1.69=23.10$. Sample variance: $\dfrac{23.10}{9}\approx 2.567$, so a standard deviation of $\approx 1.60$ points. For a poll of 1,000 at 41.8%, theory predicts a standard deviation of $\sqrt{\dfrac{0.418\times 0.582}{1000}}\approx 0.0156$, that is 1.56 points. The agreement is excellent, which suggests that here the observed dispersion is entirely explained by sampling.

A HOUSE EFFECT is a SYSTEMATIC and persistent gap between the results of a polling firm and the average of the others, due to its method: mode of collection, wording, treatment of undecided voters, weighting model. It is not random noise but a bias specific to the firm, often stable over years and of the order of one to three points. Aggregation does not correct it because it rests on the idea that errors AVERAGE OUT, which assumes that they are centred on zero; a bias shared by several firms, for example a general underrepresentation of a hard-to-reach electorate, survives the averaging intact and can even be reinforced if the firms watch one another.

In three ways. It ESTIMATES the house effect of each firm, by comparing its repeated gaps from the average of the others over a long series, then corrects for it. It WEIGHTS the polls by their size, their recency and the methodological quality of their firm. Finally, it publishes an interval wider than that of pure sampling, calibrated on the HISTORICAL errors of polls against real results, which are typically two to three times the theoretical margin. In other words, an honest aggregator uses aggregation to reduce noise and refuses to announce the precision of a sample of ten thousand people.

FALSE. Nothing is “necessarily” in the interval: the procedure catches the true value 19 times out of 20, which leaves one time in twenty when it fails. And the margin covers only sampling error, not coverage and nonresponse biases. Correct statement: the method used produces, in 95% of repetitions, an interval containing the true proportion; here that interval runs from 39% to 45%, provided that the data collection introduced no bias.

FALSE. The overlap of two intervals is a criterion that is too severe: two intervals can overlap while the gap is significant, as the case of the two waves at 38% and 43% shows. Correct statement: build the interval of the DIFFERENCE, with its own variance, and check whether it contains zero.

FALSE. The margin of 3 points is that of the full sample. The 18 to 24 year olds are only 110 respondents, and their own margin is about 9 points. Correct statement: among 18 to 24 year olds, support is estimated at 58%, plus or minus 9 points, from 110 respondents.

FALSE. The margin of 3 points applies to each proportion taken separately, not to their gap. The margin on the gap is 5.5 points here, and the interval of the lead, from $-1.5$ to $+9.5$, contains zero. Correct statement: candidate A is 4 points ahead of candidate B, a gap that remains within the limits of uncertainty; about 6 points would be needed to speak of an established lead.

FALSE, at least as an automatic conclusion. Three reasons can explain the gap without any methodological fault: the poll's interval, $\left[48.9, 55.1\right]$, already contained 49%; opinion may have changed between the poll and the vote; and the voters are not the respondents, turnout being selective. Correct statement: the result of 49% was within the poll's interval, which therefore did not allow a conclusion of a YES majority; the poll did not predict the result, but it did not contradict it either.

Margins with $n=1{,}200$: for A, $1.96\sqrt{\dfrac{0.41\times 0.59}{1200}}\approx 0.0278$, that is 2.8 points, interval $\left[38.2, 43.8\right]$; for B, $1.96\sqrt{\dfrac{0.37\times 0.63}{1200}}\approx 0.0273$, that is 2.7 points, interval $\left[34.3, 39.7\right]$; for C, $1.96\sqrt{\dfrac{0.22\times 0.78}{1200}}\approx 0.0234$, that is 2.3 points, interval $\left[19.7, 24.3\right]$. The final result for A, 38%, falls just BELOW its lower bound of 38.2. That of B, 40%, falls just ABOVE its upper bound of 39.7. That of C, 22%, is exactly at the centre.

Variance of the gap: $\dfrac{0.41\times 0.59+0.37\times 0.63+2\times 0.41\times 0.37}{1200}=\dfrac{0.2419+0.2331+0.3034}{1200}=\dfrac{0.7784}{1200}=0.00064867$. Margin: $1.96\sqrt{0.00064867}=1.96\times 0.025469\approx 0.0499$, that is 5.0 points. The observed lead was 4 points, therefore SMALLER than the margin on the gap: the poll announced no established lead, and a reversal was perfectly compatible with its own figures.

First explanation, SAMPLING CHANCE: two of the three estimates miss their target by two or three tenths of a point, which happens in the five percent of cases that are expected, and nothing else is needed to explain the reversal. Second explanation, a LATE SHIFT in opinion between the poll's fieldwork and election day, a documented and frequent phenomenon in the last 48 hours. Third explanation, a TURNOUT BIAS: polls measure intentions in the population, the result counts the votes of those who actually show up, and if B's electorate is more mobilized, the gap appears without anyone's opinion having changed. The three can in fact act together, and only a post-election analysis can tell them apart.

“Party A leads party B by 4 points, less than the 5-point margin on leads: the two parties are in a statistical tie.” The correct headline states the gap, states the RELEVANT margin, that of the gap and not that of the proportions, and concludes that the question is undecided. Note that this headline sells less, which explains why it is rare, but it would have spared the press the trial for incompetence the next day.

For C, the claim is beyond dispute: 22% observed against 22% estimated, zero gap. For A, the gap is 3.0 points and the margin 2.8: the result is OUTSIDE the interval, narrowly. For B, the gap is 3.0 points and the margin 2.7: also outside the interval. The firm's claim is therefore false for two parties out of three, though only just. The honest wording would be: “our estimates miss their target by about three points for the two main parties, slightly more than our margin; this happens in about one poll in twenty for each party, and we are checking whether a systematic bias is involved.”

Without correction: $E=1.96\sqrt{\dfrac{0.54\times 0.46}{340}}=1.96\sqrt{0.00073059}=1.96\times 0.027029\approx 0.05298$, that is 5.3 points. With the finite population correction, which applies because the sample represents $\dfrac{340}{6200}=5.5\%$ of the population, we multiply by $\sqrt{\dfrac{N-n}{N-1}}=\sqrt{\dfrac{5860}{6199}}\approx 0.97227$, which gives 5.15 points. The correction gains about 0.15 points, which is marginal but legitimate here since the 5% threshold is crossed.

Interval: $\left[54.0-5.2, 54.0+5.2\right]=\left[48.8\%, 59.2\%\right]$. It contains 50%, so YES is NOT assured: on these data, a victory for NO remains possible. The association must not announce the result as won, and it has every reason to campaign.

Excluding the undecided, the base becomes $54+39=93$ percent of the sample, so about $0.93\times 340\approx 316$ respondents who expressed a choice, and YES is $\dfrac{54}{93}\approx 58.1\%$. The margin on this new base: $1.96\sqrt{\dfrac{0.581\times 0.419}{316}}\approx 0.05438$, that is 5.4 points, and the interval becomes $\left[52.7\%, 63.5\%\right]$, which excludes 50%. The conclusion CHANGES: among those who take a side, YES is significantly in the majority. Both computations are correct and answer two different questions, depending on whether or not we assume that the undecided will split like the others.

$n=\dfrac{1.96^{2}\times 0.25}{0.04^{2}}=\dfrac{0.9604}{0.0016}\approx 601$ respondents, brought down by the finite population correction to $\dfrac{601}{1+\dfrac{600}{6200}}\approx 548$. About 550 respondents are therefore needed, 9% of the members, which remains attainable for a student association with a well-followed-up online survey.

Yes, for two reasons. First, 58% falls inside the poll's interval, $\left[48.8, 59.2\right]$, and even coincides almost exactly with the estimate made on the votes expressed alone, 58.1%, which is the relevant computation since a referendum does not count the undecided. Second, the poll supplied the useful information: YES is ahead among those who take a side, without victory being assured if the undecided swing. One reservation remains, and it is a big one: with a 14% turnout, the real result describes the 868 voters and not the 6,200 members, and the selection of voters is probably correlated with their opinion. The referendum therefore settled the matter legally without measuring the association's opinion any better than the poll did.