Quantitative Methods in Social Science 360-300 • CEGEP, Montreal

Revision sheet: sampling and questionnaires (360-300)

This sheet does not rewrite the sampling chapter: it deals with the six or seven specific places where papers lose marks. All of them come down to the same confusion, between the NUMBER of people surveyed and the WAY they were chosen. The first shows in the report, the second decides what the report is entitled to conclude.

It is meant for Social Science students at CEGEP taking the compulsory Quantitative Methods course, 360-300. Once the sheet is read, the corrected exercise set on the same chapter puts each reflex to the test, from computing a size to criticizing a published technical note.

Mark this sheet as read or add it to your favourites: a free account, no password, keeps your read sheets and favourites from one visit to the next, tells you which chapter to tackle next and lets you ask for the method or exercise you are missing. Create your space, an email is enough.

The thread of the chapter

A sample is worth not its SIZE but the way it was drawn. A thousand people who volunteered are worth less than four hundred drawn at random, and no weighting makes up the difference.

This chapter is part of 360-300, Quantitative Methods in Social Science

What is on this sheet

1. The essentials
2. The mistakes that cost marks
3. Which method to choose
4. How the answer is expected to be written
5. Check before you hand in
6. The typical problem, taken apart
7. Learn by heart
8. Frequently asked questions

Topics covered

Sampling frame
Undercoverage and overcoverage
Simple random sampling
Systematic sampling
Stratified sampling
Cluster sampling
Nonprobability plans
Design effect
Required sample size
Neyman allocation
Finite population correction
Nonresponse bias

The essentials

Three sets, and it is between them that biases are born

• The TARGET POPULATION is the one about which we want to conclude. The SAMPLING FRAME is the list in which we actually draw.
• UNDERCOVERAGE: in the population, missing from the frame. The probability of being chosen is then zero, and no sample size repairs that.
• OVERCOVERAGE: present in the frame, outside the population. It is less serious, we set those units aside at the time of contact.
• The SAMPLE comes out of the frame, never out of the population: every critique of a survey starts by comparing the two.

The zone on the left is lost in advance: these people belong to the population and have no chance of being drawn. The sample lives entirely in the intersection.

Naming the sampling frame almost always earns a mark: “the list of residential telephone numbers” is not “the population of Quebec”.

Probability-based or not: the boundary that decides everything

• A plan is PROBABILITY-BASED if every unit has a KNOWN and NONZERO probability of being drawn. Known, not necessarily equal: weights correct for it afterwards.
• Four probability plans: simple random, systematic (step $k=\dfrac{N}{n}$ ), stratified (we draw within every stratum), cluster (we draw entire blocks).
• Four nonprobability plans: quota, convenience, purposive, snowball. None gives a margin of error.
• At equal size, stratified sampling is MORE precise than simple random sampling, and cluster sampling LESS precise, because the members of a cluster resemble one another.

Illustration: at equal size, the stratified plan gives the narrowest interval, clusters the widest, because the members of a cluster resemble one another.

The question “is this plan probability-based?” is settled on one thing only: do we know, for each person, what their chance of being chosen was?

The three formulas to write from memory

• Size for a proportion: $n=\dfrac{z^{2}p(1-p)}{E^{2}}$ , with $z=1.96$ at 95%. With no information on $p$ , we take $p=0.5$ , where $p(1-p)$ is largest.
• Finite population correction: $n_{c}=\dfrac{n}{1+\dfrac{n-1}{N}}$ , useful as soon as the sample exceeds about 5% of the population.
• Design effect: $n_{eff}=\dfrac{n}{deff}$ . It is $n_{eff}$ , and never $n$ , that goes into a margin-of-error formula.
• PROPORTIONAL allocation: according to $N_{h}$ . NEYMAN allocation: according to $N_{h}s_{h}$ , so more questionnaires where the stratum is spread out.

The 526 questionnaires of the example: proportional to the sizes (blue), or by Neyman to the products size times standard deviation (orange), which sends more to Arts, the spread-out stratum.

The $p=0.5$ is not an estimate, it is a conservative choice: it gives the largest size, hence the one that protects in every case.

The mistakes that cost marks

These are the errors I correct most often in session. Each one costs marks on a paper, even when the reasoning behind it is right.

1. Mistaking size for representativeness

2 marks, and it is the first sentence the marker looks for

What not to write

“We have $5{,}000$ respondents, so the sample is very representative.”

What to write

“Representativeness comes from the WAY THE SAMPLE IS DRAWN. These $5{,}000$ respondents volunteered themselves: the sample is large and biased.”

As n grows, the estimate tightens (dotted: 95%); that of the volunteer panel tightens too, but around 62% instead of 50%.

Why: A selection bias does not shrink as the size grows: it repeats identically. Enlarging a biased sample means measuring the wrong thing more precisely.

2. Believing that doubling the sample halves the margin

1 mark, and a survey budget computed wrongly

What not to write

“We go from $400$ to $800$ respondents, so the margin will drop from $4.9$ to $2.45$ points.”

What to write

“The margin varies as $\dfrac{1}{\sqrt{n}}$ : doubling $n$ divides it by $\sqrt{2}$ , that is $3.5$ points. To halve it, we must QUADRUPLE the size.”

The curve flattens very fast: between

n=400

and

n=1600

we gain

2.45

points, and it would take

6{,}400

respondents to gain as much again.

Why: The square root in the denominator is what makes large polls so expensive: each point of precision gained costs more and more.

3. Calling a quota sample stratified

the whole question on the precision of the survey

What not to write

“We respected the sex and age proportions of the population: the sample is stratified, so it is probability-based.”

What to write

“The proportions are respected, but the respondents were recruited by judgment inside each cell: it is a QUOTA plan, so nonprobability, and without a margin of error.”

Same cells, same number per cell: on the left we draw at random in each stratum, on the right the interviewer takes the most accessible. Only the first is probability-based.

Why: Stratified and quota sampling produce the same final structure. The difference is invisible in the table and decisive in the method: in a stratified plan, we DRAW AT RANDOM inside each stratum.

4. Putting the systematic step on a periodic list

1 mark for the starting point not drawn, more if the periodicity is pointed out in the problem

What not to write

“The list has $700$ names and I want $100$ : I take one name in $7$ starting from the first.”

What to write

“The step is $k=\dfrac{700}{100}=7$ , and the starting point is DRAWN AT RANDOM between $1$ and $7$ . I first check that the list has no cycle of length $7$ .”

A duty roster arranged by week has a cycle of 7: with a step of 7 and a starting point not drawn at random, the sample contains only Sundays.

Why: A duty roster organized by week has a cycle of seven: a step of seven would select only Sundays. Systematic sampling is the only plan whose validity depends on the ORDER of the list.

5. Computing the margin of a cluster plan with the raw size

2 marks, and a precision announced wrongly

What not to write

“ $1{,}000$ students spread over $40$ randomly drawn classes: the margin is $3.1$ points.”

What to write

“The design effect is $2.0$ , so $n_{eff}=\dfrac{1000}{2}=500$ and the margin is $4.4$ points. It is $n_{eff}$ that goes into the formula.”

On the left, the eight drawn are spread over the four strata. On the right, all eight come from the same cluster: the same size, much less diversity.

Why: Students of the same class resemble one another: the thousandth student of a cluster brings much less new information than the thousandth student of a simple random draw.

6. Taking the response rate for a measure of bias

1 mark of method, more if the problem supplies what is needed to compare the two groups

What not to write

“The response rate is 72%: nonresponse bias is therefore small.”

What to write

“The response rate measures the RISK of bias, not the bias. The bias is $(1-t)(\overline{y}_{r}-\overline{y}_{nr})$ : it is zero if the nonrespondents resemble the respondents, enormous otherwise.”

Rectangles whose widths equal the shares: 72% respondents at 60, 28% nonrespondents at 40. The whole is 54.4; the respondents alone overestimate by 5.6.

Why: A rate of 95% with nonrespondents all of the same profile biases more than a rate of 40% with nonresponse that is indifferent to the subject of the survey.

7. Believing that weighting repairs a volunteer panel

the whole conclusion on the scope of the survey

What not to write

“We weighted by sex, age and region: the sample is now representative.”

What to write

“Weighting straightens the OBSERVABLE structure. It corrects nothing of what made these people sign up, and it creates no inclusion probability where there was none.”

Illustration: weighting brings the share of 18-34-year-olds back to that of the population (green), but interest in the subject, absent from censuses, stays inflated (red).

Why: We can weight only on variables that we know in the population. Interest in the subject of the survey is in no census.

8. The double-barrelled question

1 mark per faulty question, and the item becomes unusable

What not to write

“Do you find the premises clean and well lit? Yes / No”

What to write

“Do you find the premises clean?” then “Do you find the premises well lit?”: two ideas, two questions.

Why: A “no” does not say which of the two properties is lacking. No statistical treatment rescues a question that asked two things at once: the data never existed.

Which method to choose

Which plan does the problem describe?

We look in the problem for the SENTENCE that says how people were reached, and we compare it with four forms.

The four probability plans: at random in the list, one in k, at random in each stratum (rows), entire blocks drawn (columns).

If “drawn at random from the complete list” → simple random: probability-based, the margin formula applies directly

Example: 300 names drawn from the 4,200 on the student list
If “one in k starting from a randomly drawn start” → systematic: probability-based, provided the list has no cycle of length $k$

Example: one file in 14 in an alphabetical cabinet
If “we drew in EACH group” → stratified: probability-based and more precise than simple random; the allocation must be specified

Example: 40 students drawn in each of the 3 programs
If “we drew ENTIRE groups, then surveyed everyone in them” → cluster: probability-based but LESS precise; the design effect must be applied

Example: 12 classes drawn at random, all students surveyed
If “we recruited until the proportions were respected” or “volunteers answered online” → NONPROBABILITY: quota or convenience. No margin of error can be announced

Example: an open poll on the student newspaper's page

If no branch applies, the problem does not say how the respondents were reached: pointing that out IS the expected answer.

Which wording fault does this question carry?

We read the questionnaire item under our breath and look for which of these five forms it takes.

If an “and” or an “or” links two ideas that can be evaluated separately → DOUBLE-BARRELLED: split into two questions

Example: “are the premises clean and well lit?”
If the question contains a judgment, an alarming figure or an authority → LEADING QUESTION: remove the premise and ask the bare question

Example: “given the rise in fees, do you find...”
If a possible answer is in no category, or two categories overlap → CATEGORIES NOT EXHAUSTIVE OR NOT MUTUALLY EXCLUSIVE: add “other” and recut the bounds

Example: the bands “20 to 30” and “30 to 40”
If the answer exposes the respondent, on income, grades or frowned-upon opinions → SOCIAL DESIRABILITY: guarantee anonymity, switch to self-administered, word in bands

Example: “have you ever cheated on an exam?”
If the question immediately follows another that steers it → ORDER EFFECT: separate the two questions or rotate the order from one questionnaire to the next

Example: a question on cost right before a question on satisfaction

One question can carry two faults: name them both, each is worth its mark.

How the answer is expected to be written

A marker ticks steps. Here they are in order, with the concluding sentence expected word for word.

Justifying a sample size

When to use it: As soon as the problem asks “how many people must be surveyed” or supplies a target margin of error.

1 Write the three inputs of the formula: the confidence level and its $z$ , the target margin $E$ , the value kept for $p$ .
2 Justify the choice of $p$ : $p=0.5$ for lack of information, because $p(1-p)$ is largest there and the size obtained protects in every case.
3 Compute $n=\dfrac{z^{2}p(1-p)}{E^{2}}$ and ROUND UP, never to the nearest.
4 Compare $n$ with the population size $N$ : if $n$ exceeds about 5% of $N$ , apply the finite population correction.
5 Conclude with a sentence that restates the margin and the confidence level, without which the number means nothing.

Concluding sentence

“For a margin of $4$ points at the 95% confidence level, and taking $p=0.5$ for lack of prior information, $601$ respondents are needed; the population having only $4{,}200$ people, the finite population correction brings this number down to $526$ respondents actually reached.”

The trap: Rounding $600.25$ to $600$ : the target margin is then no longer reached. We always round up.

Marking: 1 mark for the formula set out, 1 mark for the justified choice of $p$ , 1 mark for the calculation, 1 mark for the finite population correction when it is needed.

Describing a stratified plan and its allocation

When to use it: When the problem gives strata with their sizes, and sometimes their standard deviations.

1 Name the stratification variable and say why it is relevant: the strata must be homogeneous inside and different from one another.
2 Choose the allocation and announce it: proportional if we have only the $N_{h}$ , Neyman if the $s_{h}$ are supplied.
3 Compute the $n_{h}$ , then check that their sum gives back $n$ after rounding.
4 Flag the strata whose drawn size falls below about thirty units: no conclusion by stratum will be possible there.

Concluding sentence

“Neyman allocation assigns $231$ , $103$ and $192$ questionnaires to the three programs, against $226$ , $150$ and $150$ under proportional allocation: it moves the effort toward the most spread-out stratum.”

The trap: Splitting into equal shares among the strata: this is neither proportional nor Neyman, and it overrepresents the small strata without saying so.

Marking: 1 mark for the allocation named, 2 marks for the calculations, 1 mark for the check of the sum.

Check before you hand in

Five minutes of checking recover more marks than one more problem started in a hurry.

The sum of the subsamples gives back the total

After any allocation, add up the $n_{h}$ : rounding often loses or gains one unit, which we make up on the largest stratum.

$225+150+150=525$ instead of $526$ : we add the missing unit to the largest stratum.
The margin approximation at a glance

For $p$ close to $0.5$ , the margin in points is about $\dfrac{100}{\sqrt{n}}$ . It confirms or contradicts any calculation in three seconds.

For $n=1000$ , we expect about $3.2$ points: a result announced at $1.5$ points signals a formula error.
Does the plan allow a margin of error?

Before writing any interval, reread the sentence of the problem that describes the recruitment and check that it contains a random draw.

“site visitors were able to answer”: no draw, so no margin, and the question is often aimed exactly at that.
Was the finite population correction necessary?

Compute $\dfrac{n}{N}$ : below $0.05$ the correction barely changes the result, above it, it is expected.

$\dfrac{600}{4200}\approx 0.14$ : the correction is required and saves 75 questionnaires.

The typical problem, taken apart

Designing the student life survey, from size to allocation

A CEGEP has $4{,}200$ students: $1{,}800$ in Social Science, $1{,}200$ in Science, $1{,}200$ in Arts and Letters.

We want to estimate the proportion of students who hold a paid job, with a margin of $4$ points at the 95% confidence level.

The standard deviations of hours worked, known from a previous survey, are $12$ , $8$ and $15$ hours respectively.

The three strata of the CEGEP: sizes 1,800, 1,200 and 1,200, standard deviations of hours worked 12, 8 and 15.

Step 1

$n=\dfrac{1.96^{2}\times 0.5\times 0.5}{0.04^{2}}=\dfrac{0.9604}{0.0016}=600.25$ , that is $601$ respondents.

Why

We take $p=0.5$ because no prior estimate is given and $p(1-p)$ is largest there: the size obtained is the one that protects whatever the true $p$ .

Step 2

$\dfrac{600}{4200}\approx 0.14 > 0.05$ , so $n_{c}=\dfrac{600.25}{1+\dfrac{599.25}{4200}}\approx 525.3$ , that is $526$ respondents.

Why

In a small population, each person surveyed removes uncertainty from those who remain. Skipping this step pays for 75 questionnaires for nothing.

Step 3

Proportional allocation: $526\times\dfrac{1800}{4200}\approx 225.4$ , then $150.3$ and $150.3$ , that is $226$ , $150$ and $150$ .

Why

The unit lost in rounding goes back to the largest stratum: that is where it changes the relative precision least, and the sum must give back $526$ .

Step 4

Neyman allocation: the $N_{h}s_{h}$ are $21{,}600$ , $9{,}600$ and $18{,}000$ , summing to $49{,}200$ ; hence $231$ , $103$ and $192$ questionnaires.

Why

Neyman sends the questionnaires where the answers vary most: Arts and Letters is the most spread-out stratum and receives $192$ questionnaires instead of $150$ .

Step 5

If the plan became a cluster plan with $deff=1.6$ , then $n_{eff}=\dfrac{526}{1.6}\approx 328.8$ and the margin would rise to $1.96\sqrt{\dfrac{0.25}{328.8}}\approx 5.4$ points.

Why

This is the figure that justifies the cost of the stratified plan: at the same size, switching to clusters would lose $1.4$ points of precision.

Step 6

Check: $226+150+150=526$ and $231+103+192=526$ .

Why

An allocation sum that does not land on $n$ signals a badly distributed rounding, before a single questionnaire has been printed.

The conclusion, written out

“We need $526$ respondents drawn at random, split into $231$ , $103$ and $192$ according to Neyman allocation, to estimate the proportion of students with a job with a margin of $4$ points, 19 times out of 20.”

The classic mistake on this problem: Computing $601$ and then splitting these $601$ questionnaires directly without applying the finite population correction: the survey costs 14% more for the same precision.

Learn by heart

• A plan is PROBABILITY-BASED if every unit has a KNOWN and NONZERO probability of being drawn. Otherwise, no margin of error.
• The margin varies as $\dfrac{1}{\sqrt{n}}$ : to halve it, we must QUADRUPLE the size.
• $n=\dfrac{z^{2}p(1-p)}{E^{2}}$ , with $p=0.5$ for lack of information, and we always round UP.
• Stratified is more precise than simple random, cluster is LESS: it is $n_{eff}=\dfrac{n}{deff}$ that goes into the formula.
• Proportional allocation according to $N_{h}$ , Neyman allocation according to $N_{h}s_{h}$ : the second sends the effort where things vary.
• The response rate measures the RISK of bias, never the bias: the bias is $(1-t)(\overline{y}_{r}-\overline{y}_{nr})$ .
• One question per idea. The double-barrelled question cannot be repaired in processing.

Frequently asked questions

What sample size do you need for a poll?

It is computed from three choices: the confidence level, almost always 95 percent, the target margin of error, and the expected proportion. With no information on that proportion, we take 0.5, which gives the largest size. For a margin of 4 points at 95 percent, about 601 people are needed, and fewer if the population is small.

What is the difference between a stratified plan and quotas?

Both respect the proportions of the population, but a stratified plan draws at random inside each stratum whereas quotas let the interviewer recruit whoever they like until the cell is full. The first is probability-based and gives a margin of error, the second does not. The difference is invisible in the final table and decisive in the method.

Why does an online poll have no margin of error?

Because the respondents volunteered themselves: nobody's probability of inclusion is known, and the margin formula assumes precisely that it is known and nonzero. The calculation still gives a number, but that number describes nothing. Weighting the results does not create the probability that was missing.

What is the design effect in a poll?

It is the factor that says how much precision a plan loses compared with a simple random sample of the same size. We divide the size by this factor to get the effective size, and that is what goes into the margin formula. A cluster plan has a design effect above 1 because the members of a cluster resemble one another.

Does a high response rate guarantee a reliable poll?

No. The response rate measures the risk of bias, not the bias itself. If the nonrespondents resemble the respondents on the question asked, a rate of 40 percent biases nothing. If they differ systematically, a rate of 90 percent can still distort the result. What matters is the gap between the two groups, not their proportion.

Practise it

Corrected exercises: Sampling and questionnaires, 360-300

A method is proved on a paper, not on a sheet. The set for the same chapter takes each of these traps into a problem, with the solution written out step by step.

10 corrected exercises
100 points
150 minutes

Do the exercises

Something missing from this sheet? A method, a trap, an exercise to practise it, a mistake you spotted: tell me, I read every request.

Sending goes through your free space: that is how I can answer you and tell you when the exercise is added. An email is enough, no password. Create your space. Rather not create an account? Write to me directly.

Previous sheet Concepts, variables and measurement scales Next sheet Frequency distributions and graphs

© Ahmed Squalli Houssaini. Revision sheet published at www.letuteurscientifique.ca/en/fiches/cegep-qm-sampling-questionnaires. Free for personal and classroom use; republishing it elsewhere requires written permission (legal notice).

Looking for a 360-300 tutor in Montreal?

Get in touch for a first session. We go back over the choices that cost marks on sampling, from the plan to the sample size, then test them on exercises at the level of real CEGEP tests.

Book a session WhatsApp

Site by Studio Squalli

Revision sheet: sampling and questionnaires (360-300)

The essentials

Three sets, and it is between them that biases are born

Probability-based or not: the boundary that decides everything

The three formulas to write from memory

The mistakes that cost marks

1. Mistaking size for representativeness

2. Believing that doubling the sample halves the margin

3. Calling a quota sample stratified

4. Putting the systematic step on a periodic list

5. Computing the margin of a cluster plan with the raw size

6. Taking the response rate for a measure of bias

7. Believing that weighting repairs a volunteer panel

8. The double-barrelled question

Which method to choose

Which plan does the problem describe?

Which wording fault does this question carry?

How the answer is expected to be written

Justifying a sample size

Describing a stratified plan and its allocation

Check before you hand in

The typical problem, taken apart

Designing the student life survey, from size to allocation

Learn by heart

Frequently asked questions

What sample size do you need for a poll?

What is the difference between a stratified plan and quotas?

Why does an online poll have no margin of error?

What is the design effect in a poll?

Does a high response rate guarantee a reliable poll?

Corrected exercises: Sampling and questionnaires, 360-300

See also

Looking for a 360-300 tutor in Montreal?