Corrected exercises: research design and critical reading (360-300)

The PROBLEM is the general observation that motivates the study: a growing share of CEGEP students hold a job during the semester, and graduation rates are stagnating. The QUESTION is what the study sets out to answer: is the number of hours of paid work associated with academic success at the college level? The HYPOTHESIS is the anticipated answer, stated in a verifiable way: beyond 20 hours a week, the pass rate across all courses is lower than that of students who work less. The three differ in their degree of precision, and the usual mistake is to jump straight from the problem to the conclusion.

Three things must be made precise: the population, the operationalized variables and the expected direction. For example: “among full-time college students, the pass rate across all courses of the semester decreases as the number of reported hours of paid work increases, the drop being marked beyond a threshold of 20 hours a week”. The word “hurts”, which asserts causation, has disappeared in favour of a measurable association: that is the price to pay for a cross-sectional design.

Because a claim that no result could contradict teaches nothing: it is compatible with everything, so it is not informative. Example of a non-falsifiable statement: “academic difficulties have many complex causes”. No conceivable data would refute it. Another common case is a hypothesis formulated AFTER seeing the results, which has nothing left to risk: it has the look of a prediction and the function of a description.

NULL hypothesis: the pass rate is the same for students who work 20 hours or more and for the others, in other words any observed difference is due only to the luck of sampling. ALTERNATIVE hypothesis: the two rates differ. Remember that the test is about $H_{0}$, which we try to REJECT, and that the researcher's research hypothesis corresponds to the alternative: we never prove directly what we believe, we show that the opposite is hard to defend.

Yes: a hypothesis is directional when it announces the DIRECTION of the difference, here “lower” rather than “different”. The test then becomes ONE-TAILED, and the whole 5% rejection region sits on one side, which lowers the critical value from 1.96 to 1.645 for a normal distribution and makes rejection easier. The counterpart is strict: the direction must be announced BEFORE seeing the data, and a spectacular result in the other direction cannot be declared significant. Choosing the one-tailed test after the fact, according to the observed direction, amounts to quietly doubling your threshold.

CASE STUDY: in-depth analysis of one or a few units, rich in detail, with no claim to generalize. CROSS-SECTIONAL: one measurement of several variables at a single moment on a sample; it is the design of the survey. LONGITUDINAL: the same units measured repeatedly over time. QUASI-EXPERIMENTAL: comparison between a treated group and a control group without random assignment, the groups being formed by circumstances. EXPERIMENTAL: comparison between groups formed by random draw, the only design in which the researcher controls who receives what.

Case study: sometimes, if the chronological reconstruction is solid, but on a single unit. Cross-sectional: NO, that is its fundamental weakness, everything is measured at the same time and nothing says what came first. Longitudinal: YES, that is precisely its contribution, repeated measurement establishes the time order. Quasi-experimental: yes for the treatment, whose date is known, no for the preexisting differences between the groups. Experimental: YES, the treatment is given after the groups are formed, on a date known to the researcher.

It is a QUASI-EXPERIMENTAL design, and its main flaw is SELECTION BIAS: students who voluntarily enrol in a methods workshop differ from the others in everything that led them to enrol, motivation, awareness of their difficulties, availability, family support. A difference in pass rates between the two groups mixes the effect of the workshop with the effect of these characteristics, and no calculation can separate them. Even the direction of the bias cannot be decided in advance: the volunteers may be the most motivated, hence better anyway, or the ones struggling most, hence weaker at the start.

You would have to DRAW AT RANDOM who receives the workshop among the interested candidates, and compare those drawn with those not drawn. The random draw balances on average all the characteristics, including those that were not measured, which no statistical control can do. It is often impossible for three reasons: ethics, since refusing a service thought to be beneficial to students who ask for it is hard to justify; administration, since an institution cannot always restrict access to a service; and contamination, since students in both groups talk to each other, swap documents, and the effect of the treatment spreads. An intermediate solution is to draw the ORDER of access at random, offering the workshop to the control group the following semester.

The specific problem is ATTRITION, or experimental mortality: participants disappear along the way, and they do not disappear at random. The students who drop out are precisely the ones whose trajectory we would like to know, and their departure empties the sample of its most informative cases, which makes the cohort look better and better as time goes by. It is measured by comparing, on the variables measured AT BASELINE, those who stay with those who left: if the two groups look alike, attrition is probably harmless; if they differ, this must be stated and weighting applied.

Random SAMPLING draws at random WHO IS STUDIED, in a defined population. It protects EXTERNAL validity, that is, the ability to generalize the results to that population. Random ASSIGNMENT draws at random WHO RECEIVES THE TREATMENT, among the people already selected. It protects INTERNAL validity, that is, the right to attribute the observed difference to the treatment rather than to a preexisting difference between the groups. The first draw is used to speak about the population, the second to speak about the cause.

INTERNAL validity is strong: thanks to random assignment, the two groups are comparable on average on everything, measured or not, and a difference in results can reasonably be attributed to the treatment. EXTERNAL validity is weak: 60 volunteers from a single CEGEP represent neither the non-volunteers nor the other institutions. We therefore know that it works, on these people, and we do not know for whom else.

It is the opposite situation. EXTERNAL validity is strong: the national probability sample makes it possible to generalize the proportions and associations to the whole target population. INTERNAL validity is weak: nobody was assigned to use the app, students chose, and that choice is linked to their motivation, their comfort with technology and their earlier results. We therefore know that the association exists in the population, and we do not know whether the app has anything to do with it.

Yes, it is the RANDOMIZED TRIAL ON A PROBABILITY SAMPLE: first a representative sample of the population is drawn at random, then the treatment is assigned at random within that sample. It is the strongest design that exists, and it is rare because it adds up the costs and constraints of both: you need a sampling frame, a high participation rate, and the ethical possibility of treating some and not others. It is found mostly in large public-policy evaluations.

The researcher confuses the two draws. Correction: “since assignment was random, the observed difference between the two groups can reasonably be attributed to the treatment, which ensures the internal validity of the study. However, since our participants are volunteers from a single institution, nothing guarantees that the effect would be of the same size elsewhere; generalizing would require a replication in other CEGEPs, with different student populations.” The corrected sentence names the validity obtained, refuses the one that is not secured, and indicates what it would take to obtain it.

First condition, ASSOCIATION: $X$ and $Y$ vary together. Second, TEMPORAL PRECEDENCE: $X$ comes before $Y$ in time. Third, NON-SPURIOUSNESS: the association is not explained by a third variable that comes before both. A crosstab only establishes the FIRST. Temporal precedence comes from the design, non-spuriousness comes from the control of third variables, and neither can be read in a coefficient, however high.

Association: yes, it is well documented and robust. Temporal precedence: plausible, since breakfast comes before the school day, but if the measurement is cross-sectional nothing establishes it formally, and one can imagine that a child who does well sleeps better and gets up earlier. Non-spuriousness: this is where everything is decided, and the condition is NOT satisfied. Household income and organization determine both the regularity of breakfast and a host of other school advantages, from a quiet home to the vocabulary used there. The honest conclusion is that breakfast is an INDICATOR of favourable conditions, whose own effect remains to be shown by an intervention.

The GRADIENT, or dose-response relationship: the more the cause increases, the more the effect increases, which makes chance less plausible; for example, if the dropout rate rose steadily from 15% to 20% and then to 35% as the weekly hours of paid work increased. And the PLAUSIBILITY OF A MECHANISM: you must be able to tell by which path the cause produces the effect, here fatigue, study time cut short and absence from college activities. These criteria, popularized by Bradford Hill in epidemiology, prove nothing on their own, but their absence should make you wary.

It is a SPURIOUS CORRELATION due to a common prior variable, the TEMPERATURE, or more generally the season: heat makes people buy ice cream and go swimming. The control that would reveal it is elaboration: at constant temperature, by comparing for instance only days between 25 and 30 degrees, the association between sales and drownings would collapse. A coefficient of 0.93 on twelve monthly points illustrates the danger of short time series well: two series that follow the same season are correlated almost perfectly without having the slightest link.

Yes, by approximating the experiment rather than replacing it. Three strategies are common. The NATURAL EXPERIMENT: you exploit an outside event that changed $X$ for reasons unrelated to $Y$, for example a reform applied to some regions and not others. The BEFORE-AND-AFTER COMPARISON WITH A CONTROL GROUP, which subtracts what would have happened anyway. And MATCHING, where each treated individual is compared with an untreated individual who resembles him or her on all the measured variables. None is worth a random draw, because all remain blind to unmeasured variables, and a causal conclusion drawn from these designs must always be announced as conditional.

The treated group gains $70-61=9$ points. A study without a control group would conclude that the effect is 9 points and would issue an enthusiastic press release. This is the most widespread design in program evaluation, and the most misleading, because it credits the program with everything that changed.

The control group gains $67-62=5$ points with no treatment. The NET gain is therefore $9-5=4$ points. This difference in differences measures what the program added TO WHAT WOULD HAVE HAPPENED ANYWAY, under an explicit assumption: without treatment, the two groups would have evolved in parallel. The true effect is therefore 4 points, less than half of the raw figure.

MATURATION: between the two measurements, students simply progressed, got used to the pace of CEGEP, learned how to study. HISTORY: an outside event affected everyone, for example a clearer instructor in the second term, or an easier exam. TESTING: taking the test a first time improves performance on the second, through simple familiarity with the format. One could add INSTRUMENTATION, if the second exam was not exactly as difficult as the first, which is enough to create an artificial gain in both groups.

The effect is to INFLATE the observed gain, often by a lot. Those who leave a tutoring program are typically those for whom it does not work, or those whose situation is getting worse: measuring them would have pulled the mean down. By measuring only the remaining 66.7%, you compare the end of a selected group with the start of a complete group. The remedy is called INTENTION-TO-TREAT analysis: all students initially enrolled in the program are kept in the treated group, whether or not they followed it. The result is smaller and it is honest.

It introduces REGRESSION TOWARD THE MEAN. Selecting students on an EXTREME performance, here a failure, guarantees that part of their bad result is due to temporary circumstances, illness, a bad night, a misunderstood question. At the next measurement, these circumstances do not repeat, and the group mechanically moves back up, without any intervention being responsible. It is the most insidious threat because it produces exactly the hoped-for result, and it alone justifies demanding a control group selected in the same way.

$z_{1}=\dfrac{48-65}{10}=-1.70$. The group sits 1.7 standard deviations below the mean, a very low position, held by about 4.5% of the distribution under a normal model.

$z_{2}=0.60\times \left(-1.70\right)=-1.02$. In points: $65+\left(-1.02\right)\times 10=65-10.2=54.8$. The prediction assumes no intervention: it follows only from the fact that the correlation between the two exams is 0.60 and not 1.

The group goes from 48 to 54.8, a PREDICTED gain of 6.8 points with nothing done. A remedial program that obtained a 7-point gain on this group would therefore have demonstrated strictly no effect. To conclude that there is an effect, you would have to clearly exceed 6.8 points, or better, compare with a control group selected in exactly the same way.

For the five best students: $z_{1}=\dfrac{82-65}{10}=1.70$, so $z_{2}=0.60\times 1.70=1.02$ and the predicted mark is $65+10.2=75.2$. The strong group LOSES 6.8 points, symmetrically, and again without any intervention being responsible. The phenomenon is perfectly symmetric: what lifts the weak back up brings the strong back down.

Tell him that his two conclusions are produced by the same artifact, and that neither rests on his programs. Selecting on an extreme performance guarantees a move toward the mean at the next measurement, whatever is done in between; this is why programs aimed at the weakest always seem to work and those aimed at the strongest always seem to disappoint. This reasoning also explains a stubborn belief among instructors: praise seems to be followed by a decline, reprimand seems to be followed by an improvement, whereas both are only the return to the mean after an exceptional performance. The only remedy is the control group selected on the same criterion.

Exactly 5%, that is the very definition of the threshold: $\alpha$ is the probability of rejecting the null hypothesis when it is true, that is, of making a Type I error. This risk is accepted in advance and it never goes away.

The probability that a given test is NOT significant is 0.95. For 20 independent tests, the probability that none is significant is $0.95^{20}\approx 0.3585$. So the probability of at least one significant result is $1-0.3585\approx 0.6415$, close to two chances in three. By looking for twenty links that do not exist, you find one in almost two cases out of three.

You need $1-0.95^{k}>0.5$, so $0.95^{k}<0.5$ and $k>\dfrac{\ln 0.5}{\ln 0.95}=\dfrac{-0.6931}{-0.051293}\approx 13.5$. From 14 tests on, there is more than one chance in two of finding at least one significant result on pure noise. Fourteen tests is an ordinary questionnaire crossed with a single dependent variable.

Bonferroni requires $\alpha'=\dfrac{0.05}{20}=0.0025$. The overall risk then drops back to about $1-\left(1-0.0025\right)^{20}\approx 0.0488$, the 5% wanted. The cost is a loss of POWER: with a threshold twenty times stricter, the critical value goes from 1.96 to about 3.02 standard deviations, and a real but moderate effect will no longer be detected. The correction is therefore conservative, and other methods, such as controlling the false discovery rate, offer a less brutal compromise.

First practice, the CHOICE OF THRESHOLD AFTER THE FACT, moving to 10% when 5% gives nothing, or declaring a test one-tailed when it was planned as two-tailed. Second practice, OPTIMISTIC STOPPING: you collect, you test, and you keep collecting as long as the result is not significant, which guarantees you will get there one day. Third practice, the RESEARCHER DEGREES OF FREEDOM: cutting the age classes differently, excluding a few outliers, changing the control variable, until the table speaks. None of these three leaves a trace in the article, and that is precisely why journals increasingly demand PREREGISTRATION of the analysis plan.

FALSE. Random sampling ensures EXTERNAL validity, generalization to the population, and nothing more. The strength of a CAUSAL conclusion comes from random assignment of the treatment, which is another draw altogether. Correct statement: since our sample is random, our estimates generalize to the population; the absence of random assignment, however, forbids us to attribute the observed differences to the treatment.

FALSE, or at least not shown. Since the group was selected on a low performance, it mechanically moves back toward the mean, and that return is here about 6.8 points, almost all of the observed gain. Correct statement: the group improved by 7 points, a gain consistent with regression toward the mean alone; concluding would require a control group selected on the same criterion.

FALSE. With 18 independent tests on pure noise, the probability of getting at least one significant result is $1-0.95^{18}\approx 0.603$, six chances in ten. A result at 4% is nothing remarkable in this context. Correct statement: out of 18 tests, this isolated result must be presented as exploratory, with the total number of tests performed, and confirmed on an independent sample before being asserted.

FALSE twice over. First, a high correlation never establishes causation, only an association. Second, two monthly series that follow the same seasonal cycle are correlated almost perfectly with no link at all: twelve annual points are dominated by the season. Correct statement: the two series vary together over the months; the season would have to be controlled, or the work done on seasonally adjusted variations, before considering any link.

FALSE. Finding no difference means the data are consistent with no effect, not that its absence is demonstrated. With 30 participants per group, the power to detect a modest effect is low. Correct statement: our data do not allow us to conclude to an effect; the confidence interval on the difference runs from $-3$ to $+7$ points, which leaves open the possibility of an appreciable effect, and a larger study would be needed.

The strongest practicable design is the WAITLIST EXPERIMENT: among the students who request tutoring, half are drawn at random to start right away, the other half being served the following semester. This gives genuine random assignment, hence internal validity, without refusing the service to anyone, which removes the ethical objection. What prevents the ideal, the randomized trial on a probability sample, is that the participants remain VOLUNTEERS: the conclusion will hold for students who ask for tutoring, not for the whole CEGEP.

Precisely by the random draw among the applicants. This point is decisive: a control group made of students who did NOT request tutoring would be unusable, since it would differ in motivation, awareness of difficulties and availability, exactly the variables that determine success. The control group must resemble the treated group on everything except the treatment, and the only way to ensure this is to draw it from the same population of applicants.

Standard error of the difference: $\sqrt{\dfrac{9^{2}}{60}+\dfrac{9^{2}}{60}}=\sqrt{1.35+1.35}=\sqrt{2.70}\approx 1.6432$. Margin: $1.96\times 1.6432\approx 3.22$ points. The observed gap is $71.4-67.9=3.5$ points, so the interval runs from $0.28$ to $6.72$: it does not contain zero, the effect is significant at the 5% level, but only just. The lower bound, barely above zero, must be reported: it says the effect could be only three tenths of a point.

$d=\dfrac{3.5}{9}\approx 0.389$. By the usual benchmarks, 0.2 small, 0.5 medium, 0.8 large, this is a SMALL TO MEDIUM effect. In other words, the average tutored student stands at about the 65th percentile of the control group distribution, against the 50th if there were no effect. It is a respectable result for a light intervention, and it must be reported with the effect size rather than with the p-value alone, because it is the effect size that will allow this program to be compared with others.

They must be KEPT in the tutored group, and this result presented as the main analysis: this is INTENTION-TO-TREAT analysis. The reason is that random assignment protects only the groups as they were drawn; as soon as you remove the 14 non-participants, you remove a subgroup selected by its own behaviour, probably the least engaged, and the equivalence of the groups disappears. You may report IN ADDITION an analysis restricted to actual participants, naming it as such and pointing out that it probably overestimates the effect. Intention to treat answers the decision maker's question, which is what OFFERING the program produces, not what it would produce if everyone followed it.

It is a CROSS-SECTIONAL design by questionnaire: social media use and depressive symptoms are measured at the same moment, on the same people. It allows you to establish an ASSOCIATION in the population surveyed, and nothing more. In particular, it establishes neither temporal precedence, since we do not know which came first, nor the absence of a confounding variable. No causal conclusion follows, whatever the sample size.

The ratio is $\dfrac{26}{13}=2.0$: “twice as” is exact in the sense of a ratio of proportions. Two other exact presentations: the ABSOLUTE gap, $26-13=13$ percentage points, and the complement, 74% of heavy users do NOT have depressive symptoms against 87% of light users, which gives a ratio of 0.85, a far less spectacular gap. The three formulations describe the same data, and the choice among them is the main dramatization lever available to a journalist. The reading rule is always to ask for the raw proportions behind a ratio.

First explanation, REVERSE CAUSATION: a teenager who is doing badly withdraws into his or her phone, mood coming before use. Second explanation, a CONFOUNDING VARIABLE: social isolation, a family conflict or insomnia increase both screen time and symptoms. Third explanation, a COMMON MEASUREMENT BIAS: both variables are self-reported in the same questionnaire, and a teenager in distress probably overstates both screen time and symptoms, which creates a correlation with no real link. This third explanation is specific to self-administered surveys and is too rarely mentioned.

An online questionnaire recruits through social media or panels, so it mechanically OVERREPRESENTS heavy users, and it excludes teenagers who are little connected, precisely the comparison group. The problem is more serious here than usual because the independent variable of the study, social media use, is also the recruitment mechanism: the sampling frame is correlated with what is being measured. Such a design can manufacture an association by itself, and at the very least the recruitment method would have to be described precisely, which the article does not do.

Headline: “Heavy social media use and depressive symptoms go hand in hand among teenagers, with no way to tell which comes first”. First sentence: “In a cross-sectional online survey of 4,500 teenagers aged 13 to 17, 26% of those who report more than three hours of daily use have depressive symptoms, against 13% of light users, a gap of 13 points. The design cannot say whether use comes before the symptoms or the reverse, and both measures rest on the reports of the same respondents.” Three decisive corrections: the verb “go hand in hand” replaces any idea of cause, the gap is given in points in addition to the ratio, and the limit of the design appears in the first sentence rather than in the last paragraph.