Corrected exercises: sampling and questionnaires (360-300)

The TARGET POPULATION is the set of units about which we want to conclude, here decided by the research question and not by the means available. The SAMPLING FRAME is the concrete list in which the draw takes place. UNDERCOVERAGE refers to the members of the target population that are missing from the frame: they have a ZERO probability of being chosen, whatever we do afterwards. OVERCOVERAGE refers to the elements of the frame that do not belong to the target population: duplicates, outdated units, people outside the scope. A survey strictly speaks only about the intersection, the middle zone of the diagram, and anything said beyond it rests on a wager.

Undercoverage: adults without a landline, which today means the majority of people under 45, recently moved households, some tenants and people on low incomes. Overcoverage: numbers assigned to businesses, lines that are out of service and residents under 18 who pick up the phone. The worst represented group is by far that of YOUNG ADULTS, whose exclusion is almost systematic: since their opinions often differ from those of older people on political and social subjects, the resulting error is directional and not random. That is why serious polling firms now combine landlines, cell phones and panels, with known inclusion rates for each source.

The fall list ignores the students enrolled for the first time in the winter term, and keeps those who dropped out between the two terms. We therefore get undercoverage, the newcomers, and overcoverage, the leavers, at the same time. The defect is especially troublesome if the survey is about persistence: the dropouts, overrepresented in the frame, are precisely the hardest to reach, so coverage error and nonresponse push in the same direction.

No, they are not equal: the household with two lines has two chances of being drawn, so roughly double the probability. A probability-based plan does not require the probabilities to be EQUAL, it requires them to be KNOWN and nonzero. We therefore correct at the analysis stage, by giving each unit a weight inversely proportional to its probability of inclusion: the household with two lines counts for one half. The same principle governs stratified plans with unequal sampling rates.

No. A sentence to remember: enlarging the sample multiplies the respondents we already had the means to reach, and does not make anyone appear in a list where they are not listed. A zero probability of inclusion stays zero however many draws we make. Increasing $n$ reduces the RANDOM error, as $\dfrac{1}{\sqrt{n}}$; coverage error is SYSTEMATIC and depends only on the frame. The only remedy is to change the frame or to combine several.

SIMPLE RANDOM: every member of the frame has the same probability of being drawn and every group of size $n$ is equally possible. SYSTEMATIC: we draw a starting point at random among the first $k$, then take one unit out of every $k$ in the list. STRATIFIED: we first split the population into homogeneous and exhaustive groups, the strata, then draw within EACH one. CLUSTER: we split the population into natural blocks, we draw BLOCKS, and we survey everyone inside the selected blocks. The decisive difference between the last two is in the diagram: stratified sampling draws within all the groups, cluster sampling draws groups.

Step of the systematic sample: $k=\dfrac{6{,}200}{300}\approx 20.67$, rounded down to 20, which will give a sample slightly larger than 300 that we adjust at the end. Proportional allocation: $0.38\times 300=114$, $0.24\times 300=72$, $0.18\times 300=54$, $0.12\times 300=36$, $0.08\times 300=24$. The sum is indeed 300, and that is the check to make every time, because a badly placed rounding is the most frequent error on this type of question.

Stratified sampling wins because the total variance splits into WITHIN-stratum variance and BETWEEN-strata variance: by imposing the right sizes in each stratum, we completely eliminate the part of the error due to the chance of the sample's composition, and only the variability inside the strata remains. The gain is larger the more homogeneous the strata are internally and the more they differ from one another. Cluster sampling suffers exactly the opposite mechanism: the members of one cluster resemble one another, a class, a building, a neighbourhood, so that the twelfth student of a class already surveyed brings much less new information than a student drawn at random from the whole CEGEP. We pay in precision for what we save in travel.

The ratio 1.9 is the DESIGN EFFECT. The effective size is $n_{eff}=\dfrac{336}{1.9}\approx 177$ students: the 336 students grouped into 12 classes are worth, in precision, about 177 students drawn at random one by one. This is the number to use in any margin-of-error formula, otherwise we announce a precision the survey does not have. Conversely, to obtain the precision of a simple random sample of 336, we would need here $336\times 1.9\approx 638$ students in clusters.

Systematic sampling becomes dangerous when the list contains a PERIODICITY whose period is a multiple or a divisor of $k$. A specific example: a list of students arranged class by class, each class having exactly 30 names and the first name of each class being the class representative; with a step of 30, we survey only representatives. Another classic example: a daily sales record sampled one day in seven, which always falls on the same day of the week. The remedy is to check the order of the list before drawing, or to shuffle it deliberately.

QUOTA: the interviewer is required to fill cells, for example 40 women aged 18 to 34, and recruits whoever they like to fill them. CONVENIENCE, or accidental: we interview the people who are available, at the exit of a shopping mall, in our own class, on a social network. PURPOSIVE, or judgment sampling: the researcher deliberately selects cases judged informative, typical cases, extreme cases, contrasting cases, which is common in qualitative research. SNOWBALL: the first participants recruit the next ones from their network, a procedure reserved for rare or hard-to-reach populations.

The only difference is the mode of selection INSIDE the groups: stratified sampling draws AT RANDOM in each stratum, quota sampling lets the interviewer CHOOSE who fills the cell. The consequence is total: without a random draw, no inclusion probability is known, so no sampling theory applies, so neither a margin of error nor a confidence interval can be computed. And the interviewer's choice is never neutral: with identical quotas, they will approach people who are available, pleasant and present during business hours. The structure will be right and the selection biased.

Snowball sampling is justified here because there is no sampling frame of people experiencing homelessness: no list, no address, no telephone line, and a legitimate distrust of institutions. The participants' network is then the only way in. What it risks missing: precisely the ISOLATED people, with no network, who are often the most vulnerable, as well as the segments that never come into contact with the first people recruited, for example women housed by acquaintances rather than on the street. The method explores the network, not the population, and the report must say so.

“Convenience sample of 96 students enrolled in three quantitative methods groups at the CEGEP, in the winter term.” What changes is not cosmetic: the wording names the plan, so it announces the absence of a margin of error, and it names the population actually reached, which is not the CEGEP but the students enrolled in one given course. The results are still publishable and instructive; they describe these 96 students and suggest hypotheses, they do not estimate anything.

You must answer that weighting corrects the observable COMPOSITION and never the SELECTION. Reweighting by age, sex, region and education aligns the proportions of the sample with those of the census, which is useful; but inside each cell, the volunteers of a panel remain volunteers, that is, people more interested in surveys, more available and often more politicized than those they are supposed to represent. No weighting variable measures this difference, and weighting on more variables only moves the problem. A well-run panel is a good tool for comparison over time, with a constant mode; it does not become a probability sample for that.

For $E=0.03$: $n=\dfrac{1.96^{2}\times 0.25}{0.03^{2}}=\dfrac{0.9604}{0.0009}\approx 1{,}067$. For $E=0.02$: $n=\dfrac{0.9604}{0.0004}\approx 2{,}401$. The ratio is $\dfrac{2401}{1067}\approx 2.25=\left(\dfrac{3}{2}\right)^{2}$: dividing the margin by 1.5 multiplies the size by 1.5 squared. The margin decreases as $\dfrac{1}{\sqrt{n}}$, so the size grows as the square of the precision demanded, and that is what makes very precise surveys prohibitively expensive.

Because $p(1-p)$ is at its MAXIMUM at $p=0.5$. Indeed $0.5\times 0.5=0.25$ whereas $0.2\times 0.8=0.16$. The function $p\mapsto p(1-p)$ is a downward-opening parabola with roots 0 and 1, so its vertex is halfway. Choosing $p=0.5$ therefore means placing ourselves in the worst case: the size obtained guarantees the margin wanted whatever the true proportion. It is a prudent decision, not an estimate.

$n=\dfrac{1.96^{2}\times 0.16}{0.0009}=\dfrac{0.614656}{0.0009}\approx 683$. The prior information brings the requirement down from 1,067 to 683 respondents, about 384 questionnaires saved, nearly a third of the budget. Beware of a calculation trap: the 0.9604 of question a) already contains $p(1-p)=0.25$, so we do not multiply it by 0.16, we start again from $1.96^{2}=3.8416$. Beware of the substance as well: if the true proportion turned out to be close to 0.5, the real margin would be much wider than 3 points. We take this risk only when the prior estimate is solid.

$n_{c}=\dfrac{1067}{1+\dfrac{1066}{6200}}=\dfrac{1067}{1.17194}\approx 910$. The correction saves about 157 respondents, because a draw without replacement in a population of 6,200 uses up a non-negligible share of the information available. The usual rule is to apply it as soon as the sample exceeds about 5% of the population, which is largely the case here with 17%.

The 10% rule has no basis. At the CEGEP, 10% of 6,200 is 620 respondents, fewer than the 910 that a margin of 3 points requires (it would give a margin of about 3.7 points); at the scale of Quebec, 10% would mean about 900,000 people for a precision that 1,067 already reach. The point to get across is that the margin of error depends on $n$ and practically not on $N$ once the population is large: it is the absolute size of the sample that counts, not its share. One spoonful is enough to taste the soup, whether the pot is small or large, as long as it has been stirred.

The weights are $\dfrac{3720}{6200}=0.60$, $\dfrac{1860}{6200}=0.30$ and $\dfrac{620}{6200}=0.10$. Proportional allocation therefore gives $180$, $90$ and $30$ questionnaires. The sum is 300, and each stratum is represented in the sample as it is in the population.

Products $N_{h}s_{h}$: $3720\times 4=14{,}880$, $1860\times 6=11{,}160$, $620\times 9=5{,}580$. Their sum is $31{,}620$. Neyman allocation: $300\times \dfrac{14880}{31620}\approx 141$, $300\times \dfrac{11160}{31620}\approx 106$, $300\times \dfrac{5580}{31620}\approx 53$. The sum is indeed 300, up to rounding.

Tremplin DEC receives 30 questionnaires under proportional allocation and 53 under Neyman, nearly double. Proportional logic looks only at the WEIGHT of the stratum: small stratum, small size. Neyman logic looks at the weight AND the SPREAD: a very heterogeneous stratum, here $s=9$ hours against 4 elsewhere, needs more observations to be estimated with the same precision, otherwise it degrades the overall estimate. In other words, we invest the questionnaires where the uncertainty is greatest, exactly as we devote more revision time to the chapter we master least.

We need $n=\dfrac{1.96^{2}\times 0.25}{0.08^{2}}=\dfrac{0.9604}{0.0064}\approx 150$ respondents in Tremplin DEC alone, and the finite population correction on $N_{h}=620$ brings this number down to $\dfrac{150}{1+\dfrac{149}{620}}\approx 121$. Neither of the previous allocations is enough: we then move to what is called an optimal allocation under constraint, first fixing the 121 for Tremplin and then splitting the remaining 179 between the other two sectors. The stratum is from then on OVERREPRESENTED relative to its real weight, and the respondents will have to be weighted at the analysis stage to recover a correct overall picture.

Because the two objectives ask for opposite things. Estimating the CEGEP as a whole with the least possible error pushes toward allocating by weight and spread, so toward giving few questionnaires to small strata, whose contribution to the overall mean is small. COMPARING the sectors demands on the contrary a sufficient size IN EACH one, including the smallest, since the precision of a comparison is governed by the smaller of the two groups. A survey that pursues both objectives must therefore rank them explicitly, then restore the weights at the analysis stage: that is the role of sampling weights.

This is a DOUBLE-BARRELLED question: it asks two, one about the location and one about the equipment. A respondent who finds the CEGEP well located but poorly equipped can answer neither yes nor no, and the analyst will never know which of the two the “yes” refers to. Rewrite: two separate questions, “Is the CEGEP well located?” and “Is the CEGEP's equipment adequate?”, each on a scale of four or five levels. The general rule can be spotted by ear: as soon as an “and” or an “or” links two objects of evaluation, you have to split.

First problem, the question is LEADING: “don't you think” and “reasonable” suggest the expected answer, and some respondents follow the suggestion out of politeness. Second problem, it is LOADED: it ties the fee increase to an improvement presented as certain, which makes it impossible to oppose the increase without appearing to oppose the services. Neutral rewrite: “Would you be in favour of or opposed to a fee increase of 60 dollars per term?”, with a symmetric scale from in favour to opposed and a “don't know” option. The stated amount usefully replaces the adjective, which does not carry the same value for everyone.

First defect, the categories are not MUTUALLY EXCLUSIVE: a student who studies exactly 5 hours can tick two boxes, and the total of the percentages no longer means anything. Second defect, they are not EXHAUSTIVE: nobody can answer for 15 to 20 hours, an interval that is simply absent. Fix: “0 to under 5”, “5 to under 10”, “10 to under 15”, “15 to under 20”, “20 and over”. The bounds now do not overlap and cover the whole possible range, zero included.

We should fear an ORDER EFFECT, or contamination effect: the first question activates the memory of difficulties and lowers the satisfaction reported right after. We remedy it in three ways: place the general questions BEFORE the specific ones, separate the two blocks with unrelated questions, and above all rotate the order of the blocks from one respondent to the next, which an online questionnaire allows at no cost. Rotation does not remove the effect, it makes it measurable by spreading it evenly.

This is SOCIAL DESIRABILITY bias: admitting to cheating costs something, even anonymously, and the proportion reported will underestimate reality. Three techniques reduce it. Real and visible anonymity, with a self-administered mode rather than an interview. Wording that normalizes, of the type “Many students sometimes... have you ever...”. And the RANDOMIZED RESPONSE technique: the respondent secretly flips a coin and answers the sensitive question only if it lands on heads, otherwise answers yes automatically; since the proportion of tails is known, we recover the proportion we are after by calculation without ever knowing what a given person answered.

Response rate: $\dfrac{420}{1200}=0.35$, that is 35%. On its own, it does not allow you to conclude to ANY bias: it measures the risk, not the error. A rate of 35% is perfectly harmless if the 780 absentees think like the 420 present, and catastrophic if they differ from them. What the rate announces is the size of the lever: the lower it is, the smaller the difference between respondents and nonrespondents needed to shift the result.

The sample of 80 followed up estimates the nonrespondents at 38% satisfaction. The overall proportion is obtained by a weighted mean: $0.35\times 0.62+0.65\times 0.38=0.217+0.247=0.464$, that is about 46.4%. Note that the weight of 0.65 given to the nonrespondents nearly crushes the respondents' result: when two thirds of the sample is missing, they are the ones who decide.

The naive estimate announces 62% against 46.4% in reality, an overestimate of 15.6 points. The general formula is $\text{bias}=\left(1-t\right)\left(\overline{y}_{r}-\overline{y}_{nr}\right)$, where $t$ is the response rate: here $0.65\times \left(0.62-0.38\right)=0.65\times 0.24=0.156$. The formula says it all: the bias is the product of the missing share and the gap between the two groups, and it vanishes if either factor vanishes. That is why a survey with a 95% response rate is robust even if the absentees differ a lot, and a survey with 20% never is.

Yes, and exactly in the ratio of the missing shares: the bias becomes $0.30\times 0.24=0.072$, that is 7.2 points instead of 15.6. The reduction is $\dfrac{0.072}{0.156}\approx 0.46$, so more than half of the bias has disappeared, only because the missing share went from 65% to 30%. Following up is thus the only action that acts directly on the bias, whereas sending out more initial invitations without improving the rate would change nothing at all.

It does not correct it, except by coincidence. Weighting by age and program restores the STRUCTURE of the sample; it would correct the bias only if nonresponse depended solely on age and program, that is, if, at a given age and program, respondents and nonrespondents had the same satisfaction. But dissatisfaction is very probably a direct CAUSE of nonresponse, and disengagement is not a variable in the administrative file. Weighting would thus treat an observable symptom while leaving the cause intact. The only serious correction is the one this exercise made: go and measure a random subsample of nonrespondents, then recombine the two parts.

FALSE. The margin of error depends on $n$, and on $N$ only through the finite population correction, which is negligible as soon as the sample represents less than 5% of the population. With 1,000 people, the margin is about 3.1 points in Quebec, in Canada and in the United States alike. Correct statement: the precision of a poll depends on the number of people surveyed, not on the proportion of the population they represent, as long as that proportion remains small.

FALSE, or more exactly not demonstrated. A rate of 78% is excellent and strongly reduces the risk, since nonresponse bias is written $\left(1-t\right)\left(\overline{y}_{r}-\overline{y}_{nr}\right)$ and the first factor is now only 0.22. But representativeness cannot be deduced from any rate: it is checked by comparing the sample with what we know of the population, and above all by going to measure a subsample of nonrespondents. Correct statement: a rate of 78% makes nonresponse bias unlikely and small; it does not measure it.

FALSE. Quotas impose the COMPOSITION of the sample, stratification imposes the composition AND draws at random inside each group. It is the random draw, and it alone, that provides the inclusion probabilities without which no margin of error is defined. Correct statement: quota sampling is NONPROBABILITY sampling; it produces a sample of conforming structure, with no computable margin of error.

FALSE. Students of the same class resemble one another, so each brings less new information than a student drawn on their own: that is the design effect. With a design effect of 1.9, the effective size falls to about 177, and the margin of error is that of 177, not of 336. Correct statement: the size must be divided by the design effect before any margin calculation, otherwise we announce a precision of about 5.3 points where the survey offers only 7.4.

FALSE. Open questions have their uses, in exploration or at the end of a questionnaire, but they do not remove influence: they shift it toward whatever the respondent has in mind at the moment, they favour the more educated, they make the response rate drop and they force a coding after the fact that reintroduces the researcher's categories, this time without anyone being able to criticize them. Correct statement: we build response categories that are exhaustive, mutually exclusive and balanced, with an “other, please specify” option and a “don't know” option, and we reserve the open format for the small number of questions that really require it.

The best frame available is the file of students registered in the current term, kept by the registrar's office: it is exhaustive, up to date, and it already carries age, program, sector and study status, which will serve as stratification and weighting variables. Residual defect: students who are registered but dropped out during the term are still listed, and those who have just arrived by transfer may be missing for a few weeks. Overcoverage is the more troublesome here, since dropouts are at once overrepresented in the frame and nearly unreachable, which will feed nonresponse.

A STRATIFIED plan with proportional allocation, possibly adjusted for small strata. Strata selected: the sector, pre-university, technical and Tremplin, crossed with study status, full-time and part-time. Justification: these variables are known for the whole frame, they are strongly related to the three themes of the survey, and stratification guarantees that no sector will be accidentally absent. A cluster plan by class groups would be cheaper if we went into classrooms, but its design effect would waste half of the sample, which is pointless for an online survey where the marginal cost of one mailing is zero.

We need $\dfrac{400}{0.32}=1{,}250$ invitations. Overall margin with $n=400$ and $p=0.5$: $E=1.96\sqrt{\dfrac{0.25}{400}}=1.96\times 0.025=0.049$, that is 4.9 points, brought down to about 4.7 points by the finite population correction on 6,200. This is an honest precision for an overall picture. Note that the rate of 32% leaves 68% absent: the real risk of this survey is not its margin, it is its nonresponse, and the protocol must provide from the start for two follow-ups and a measurement on a subsample of nonrespondents.

The subgroup represents 18% of the sample, about $0.18\times 400=72$ respondents. The margin on this subgroup is $1.96\sqrt{\dfrac{0.25}{72}}\approx 0.1155$, that is 11.6 points: clearly beyond the 8 points aimed at. To reach 8 points, we would need $\dfrac{0.9604}{0.0064}\approx 150$ respondents in this subgroup, so either raise the total sample to about 833 respondents, or deliberately oversample the heavy workers, which requires knowing their status before the mailing. Since this information is not in the registrar's file, the only practicable route is to increase the overall sample, or to accept a margin of 11.6 points on this subgroup and say so.

“Survey conducted from March 3 to 21 among students registered for the winter term at the CEGEP; sampling frame: registrar's file, 6,200 registered students. Plan stratified by sector and study status, proportional allocation, 1,250 invitations, 402 usable questionnaires, response rate 32%. Data weighted by sector, status and age; overall margin of error of plus or minus 4.7 points, 19 times out of 20; margin raised to about 11.6 points for subgroups of about 70 respondents.” Three lines that give the frame, the plan and the precision, with its limits: this is exactly what a reader must find in order to judge the rest of the report.

No, the gap is too large for chance alone. The intervals are $\left[51.0, 57.0\right]$ for A and $\left[38.0, 44.0\right]$ for B: they do not overlap, and seven points remain between them. More finely, the difference of two proportions from independent samples has a standard error of about $\sqrt{0.0153^{2}+0.0153^{2}}\approx 0.0217$, so a margin of about 4.2 points on the gap: a gap of 13 points is worth more than six standard errors. The conclusion is unavoidable: the difference does not come from sampling, it comes from the method.

It probably explains most of the gap. B's question contains an additional piece of information, and a costly one: mentioning a price makes support drop for almost every measure, a phenomenon so consistent that sponsors use it deliberately. The two firms are therefore not measuring the same thing: A measures agreement with the principle, B measures agreement with the principle once the price is known. Both figures can be accurate at the same time, and it is the headline of the press release, if it speaks simply of “support for the measure”, that is wrong in both cases.

The mode acts through three channels. COVERAGE: an online panel excludes the least connected, a telephone survey with cell phones reaches them better. SOCIAL DESIRABILITY: facing a human interviewer, a respondent softens their positions on sensitive subjects, whereas a self-administered questionnaire frees speech; the gap commonly reaches five points on questions of immigration or voting. Finally PACE and ATTENTION: online, the respondent sees all the categories, including the middle of the scale, whereas by telephone we mostly retain the first and the last read out. A serious firm therefore never compares two waves run in different modes without saying so.

Out of 100 respondents of B, 41 are in favour, 14 undecided, so 45 opposed. Excluding the undecided, the share in favour becomes $\dfrac{41}{86}\approx 0.477$, that is 47.7%. The gap with A falls from 13 to about 6.3 points, a reduction by half. The treatment of the undecided is thus a third factor of discrepancy, purely conventional: forcing a choice distributes them between the two camps, offering the way out removes them from the denominator or leaves them in it, and each decision shifts the result by several points without a single opinion having changed.

We must demand from each the EXACT wording of the question, the order of the preceding questions, the mode of collection, the sampling frame and the plan, the field dates, the response rate, the weighting variables and the treatment of the undecided. That is the content of a complete technical note, which professional codes require be published with any result made public. The general principle fits in one sentence: two polls can be compared only if everything, except what we want to compare, has been held constant. Failing that, the gap measures the method, not the opinion.