Quantitative Methods in Social Science 360-300 • CEGEP, Montreal
This sheet covers the chapter that decides the mark on the synthesis exam: the one where you must read a study and say what it really allows you to conclude. It does not repeat the list of designs, it gives the eight places where an honest study produces a false conclusion, and the sentence that repairs each one.
It is written for Social Science students at CEGEP taking the compulsory Quantitative Methods course, 360-300, and it is also useful for reading articles in psychology, sociology and economics. The corrected exercise set on the same chapter tests each reflex on real cases.
The thread of the chapter
A result is never worth more than the DESIGN that produced it. The whole chapter comes back to a single question: what would have happened without the intervention, and what makes it possible to know?
A test conclusion is always written in the conditional of proof: “the data allow us to reject”, never “it is proven that”.
A laboratory experiment on thirty volunteers has excellent internal validity and dismal external validity: the two qualities are independent.
Bonferroni correction: . It restores the overall risk at the price of power, so it is decided BEFORE looking at the results.
These are the errors I correct most often in session. Each one costs marks on a paper, even when the reasoning behind it is right.
What not to write
“The test confirms my hypothesis: tutoring improves results.”
What to write
“The test rejects the null hypothesis of no effect at the 5% level: the data are hardly compatible with the idea that tutoring changes nothing.”
Why: A test never accepts a hypothesis, it rejects one. The nuance is not hair-splitting: it is a reminder that another mechanism can produce the same gap.
What not to write
“The participants were drawn at random from the population: the study is therefore experimental and allows a conclusion about a cause.”
What to write
“The draw concerns WHO is studied, hence external validity. To speak of a cause, the draw must also concern WHO receives the treatment.”
Why: The two draws have the same name and opposite roles. Generalizing and attributing a cause are two different operations, protected by two different devices.
What not to write
“Students who play sports get better grades: sports improve school results.”
What to write
“The two variables are associated at a given moment. Nothing says which comes first, nor that a third, such as personal organization, does not explain both.”
Why: A cross-sectional design measures everything at the same time: it can establish neither temporal precedence nor non-spuriousness, two of the three conditions of causation.
What not to write
“The forty struggling students went from to after tutoring: the program works.”
What to write
“Selected on an extreme result, these students would have moved back up anyway: with , the prediction with no intervention at all is , about .”
Why: Any selection on an extreme value catches a share of bad luck, which does not repeat. The phenomenon is purely statistical and requires no mechanism.
What not to write
“The group went from to : the program brought points.”
What to write
“The control group went from to with no program. The program's own effect is the DIFFERENCE IN DIFFERENCES: points.”
Why: Between the two measurements, everyone got older, studied, took other courses. The control group is there exactly to measure what these factors would have produced alone.
What not to write
“Out of the twenty crossings tested, the one between program and sex is significant at 5%: that is our result.”
What to write
“Over twenty independent tests at the 5% level, ONE significant result is expected by pure chance. The threshold must be corrected, for example , or the analysis announced as exploratory.”
Why: The 5% threshold is a risk PER TEST. Multiplying the tests multiplies the chances to be wrong: at fourteen tests, the risk of at least one false positive already exceeds one chance in two.
What not to write
“We remove the twelve participants who dropped out of the tutoring along the way: they would distort the results.”
What to write
“We analyse by INTENTION TO TREAT: everyone stays in the group to which he or she was assigned. Dropping out is an effect of the program, not a flaw in the data.”
Why: Those who drop out do not resemble those who persist. Removing them brings back through the back door the selection bias that random assignment had eliminated.
What not to write
“: the program's effect is considerable.”
What to write
“ says that the effect is hardly compatible with chance. Its importance is read on the EFFECT SIZE, here , a modest effect.”
Why: The -value depends on the sample size; the effect size does not. With ten thousand participants, a gap with no practical significance becomes highly significant.
Look in the description for two things: are there several MOMENTS of measurement, and is there an ASSIGNMENT to the treatment?
If a single case, described in depth → CASE STUDY: it generates hypotheses, it tests none
Example: the portrait of a single CEGEP
If a single measurement, at a single moment, on several units → CROSS-SECTIONAL: association only, neither temporal precedence nor non-spuriousness
Example: a survey on sports and grades
If the same units measured at several moments, with no assigned treatment → LONGITUDINAL: temporal precedence is established, non-spuriousness still is not
Example: a cohort followed over three semesters
If one group receives the treatment, another does not, but WITHOUT a random draw → QUASI-EXPERIMENTAL: difference in differences possible, selection to be discussed
Example: tutoring volunteers compared with the others
If assignment to the treatment is DRAWN AT RANDOM → EXPERIMENTAL: causation is established on the population studied
Example: a hundred students split at random into two groups
The word “random” in a problem is not enough: you must read WHAT the draw concerns, recruitment or assignment.
Faced with an observed improvement, go through the list of threats before crediting the intervention.
If the participants were chosen because they were extreme → REGRESSION TOWARD THE MEAN: compute and compare
Example: the forty weakest on the first exam
If something else happened between the two measurements, for everyone → HISTORY: only a control group can neutralize it
Example: a change of instructor mid-semester
If time alone would be enough to explain the progress → MATURATION: the participants simply moved ahead in their program
Example: first-semester students compared six months later
If the first test taught the participants something → TESTING: use different versions of the two tests
Example: the same questionnaire taken twice
If the two groups did not lose the same people along the way → ATTRITION: analyse by intention to treat and compare the profiles of those lost to follow-up
Example: the weakest drop out of the treated group
If no threat applies and assignment was drawn at random, then, and only then, you may write that the program is the cause.
A marker ticks steps. Here they are in order, with the concluding sentence expected word for word.
When to use it: As soon as the problem offers an intervention and asks whether it worked.
Concluding sentence
“The design is quasi-experimental: the participants volunteered, so assignment is not random. The difference in differences is points, an effect size of . Selection on a weak result requires taking regression toward the mean into account, which the control group does.”
The trap: Concluding on the raw gain of the treated group: it is the number in the press release, and it is the one that measures nothing.
Marking: 1 mark for the named design, 1 mark for the hypotheses, 2 marks for the difference in differences, 1 mark per threat ruled out, 1 mark for the effect size.
When to use it: When the problem supplies a newspaper article or a research summary to comment on.
Concluding sentence
“The study establishes an association on a sample of volunteers, with neither random assignment nor correction for multiple comparisons: it allows a hypothesis, not a causal conclusion.”
The trap: Criticizing the sample size first: it is almost always the least serious flaw, and the marker is expecting the design.
Marking: 1 mark per question addressed, with the justification drawn from the text provided.
Five minutes of checking recover more marks than one more problem started in a hurry.
Did the control group move too?
Before attributing a gain to the program, look in the problem for the control group's figure. If there is none, point it out.
A gain of points that becomes once the control group is taken into account: two thirds of the announced effect did not exist.
Was the selection made on an extreme value?
Reread the sentence that says how the participants were chosen. “The weakest”, “those struggling”, “the worst results” are signals of regression toward the mean.
Selecting at with guarantees a rebound of about standard deviation with no action at all.
How many tests were run?
Count the comparisons reported, then compute . Beyond three or four, demand a correction.
Five subgroups tested already give a % risk of at least one false positive.
Does the effect size accompany the p-value?
A conclusion that cites only is incomplete: look for the , the gap in real units, or the proportion of variance explained.
A of on a -point exam corresponds to four points: significant does not mean spectacular.
Forty students who failed the first exam, with a mean of , take part in a peer tutoring program. On the next exam, their mean is .
A control group of forty students, also selected on their failure, with a mean of , received nothing; its mean goes to .
The cohort mean is , the standard deviation is , and the correlation between the two exams is .
Step 1
: tutoring does not change the mean mark. : it raises it. QUASI-EXPERIMENTAL design: the participants are volunteers, assignment is not drawn at random.
Why
Naming the design first sets from the start what the conclusion will be entitled to say, and is worth a mark by itself.
Step 2
Raw gain of the treated group: points. Gain of the control group: points.
Why
The second figure is the one the press release never cites, and it is the only one that gives a point of comparison.
Step 3
Regression toward the mean: , so , an expected mark of .
Why
With no intervention at all, is predicted: that is almost exactly the observed mean of the control group (). The phenomenon is computed, not invoked.
Step 4
Difference in differences: points.
Why
It is what tutoring added to what would have happened anyway. The announced effect goes from to , one third.
Step 5
Effect size: . Required sample size for a power of 80%: per group.
Why
With forty participants per group, the study is underpowered: it has less than one chance in two of detecting the effect it is looking for, even if it exists.
Step 6
Check: if five subgroups had also been tested, the risk of at least one false positive would be %.
Why
This figure decides whether a subgroup result deserves to be reported. Without a correction, it is an exploration, not a test.
The conclusion, written out
“Tutoring is associated with a gain of points beyond the spontaneous progress of the control group, an effect size of . Since the design is quasi-experimental and the study is underpowered with forty participants per group, this result justifies an experiment with random assignment, not a causal conclusion.”
The classic mistake on this problem: Announcing a gain of points: the control group gained with nothing received, and regression toward the mean alone predicted .
It is the return toward the center observed when a group is selected on an extreme value and then measured again. Part of the extreme result came from bad luck, which does not repeat. The prediction with no intervention at all is the starting Z score multiplied by the correlation between the two measurements. It is a statistical fact, not a program effect.
Internal validity is the right to attribute the observed effect to the treatment rather than to something else: it is protected by random assignment. External validity is the right to generalize to a wider population: it is protected by random sampling. A laboratory experiment excels at the first and often fails at the second.
By the difference in differences: compute the gain of the group that received the program, the gain of the control group over the same period, then subtract the second from the first. The result is what the program added to what would have happened anyway. Without a control group, the raw gain mixes the effect of the program with everything else.
Because the five percent threshold is a risk per test. Over twenty independent tests, one significant result is expected by pure chance, and the risk of getting at least one exceeds fifty percent from fourteen tests on. The Bonferroni correction divides the threshold by the number of tests, which restores the overall risk at the price of power.
No. The p-value measures how improbable the result would be if the treatment changed nothing, and it shrinks mechanically as the sample grows. Importance is read on the effect size, which does not depend on the sample size. With ten thousand participants, a gap with no practical significance becomes highly significant.
© Ahmed Squalli Houssaini. Revision sheet published at www.letuteurscientifique.ca/en/fiches/cegep-qm-research-design-critique. Free for personal and classroom use; republishing it elsewhere requires written permission (legal notice).
