Corrected exercises: nonparametric tests (360-301)

First situation, an ORDINAL variable: a satisfaction rating, a ranking, a coded level of education. Second situation, SMALL SAMPLES combined with a clearly nonnormal distribution: below about thirty observations per group, the central limit theorem no longer saves you. Third situation, the presence of EXTREME VALUES: a single outlier can move a mean by several standard deviations, while it moves a rank by only one place.

Ranks of 12, 15, 9, 40, 11: $3$, $4$, $1$, $5$, $2$. Changing 40 to 400 gives exactly the same ranks: $3$, $4$, $1$, $5$, $2$. The mean, on the other hand, goes from 17.4 to 89.4. This is the central property of ranks: they are INSENSITIVE to the size of the gaps and keep only the order, which makes them robust and, at the same time, less informative.

They keep the assumption of INDEPENDENCE of the observations, which is never negotiable, and most of them also assume that the distributions being compared have the SAME SHAPE, differing only in location. Without that last assumption, a rejection by Mann-Whitney does not necessarily translate into a difference of medians: it only indicates that an observation from one group tends to exceed an observation from the other.

On an ORDINAL scale. The t test compares means, and a mean assumes that the gaps between successive values are comparable, which an ordinal scale does not guarantee: nothing says that the gap between “disagree” and “neutral” equals the one between “neutral” and “agree”. Computing a mean there is therefore a convention, and testing it as if it were a quantity is an abuse. Ranks use only the order, which is exactly the information the scale contains.

Each tied value receives the AVERAGE RANK of the positions it occupies. Three equal observations in positions 4, 5 and 6 each receive the rank $\dfrac{4+5+6}{3}=5$. The total of the ranks therefore stays $\dfrac{N(N+1)}{2}$, which is the check to make. If there are many ties, the variance of the ranks decreases and a correction must be applied to the formulas, otherwise the test becomes too conservative.

In increasing order, with the group in parentheses: 9 (B) rank 1, 11 (B) rank 2, 12 (A) rank 3, 14 (B) rank 4, 15 (A) rank 5, 16 (B) rank 6, 17 (A) rank 7, 19 (B) rank 8, 21 (A) rank 9, 22 (B) rank 10, 24 (A) rank 11, 26 (B) rank 12, 30 (A) rank 13.

$R_{A}=3+5+7+9+11+13=48$. $R_{B}=1+2+4+6+8+10+12=43$. Total: $48+43=91$, which must equal $\dfrac{N(N+1)}{2}=\dfrac{13\times 14}{2}=91$. The check passes, which guarantees that no rank was forgotten or counted twice.

$U_{A}=48-\dfrac{6\times 7}{2}=48-21=27$. $U_{B}=43-\dfrac{7\times 8}{2}=43-28=15$. Check: $27+15=42=6\times 7=n_{A}n_{B}$. This identity is automatic and serves as a second safeguard.

Keep the SMALLER of the two, $U=15$. The decision rule of rank tests is reversed compared with parametric tests: you reject when the statistic is LESS THAN OR EQUAL TO the critical value. Since $15>6$, you do not reject: the data do not allow us to conclude that the two groups differ at the 5% level.

$U_{B}=15$ means that, among the 42 possible pairs made of one member of A and one member of B, there are 15 in which the member of B exceeds the member of A. Under the null hypothesis you would expect half of them, that is 21. So we observe a slight advantage for group A, 27 pairs out of 42, or 64%: real but far too weak for thirteen observations. It is this reading in terms of pairs that gives the test its meaning, and it is directly tied to the probability that a member drawn at random from A exceeds a member drawn at random from B.

Absolute values: 4, 1, 6, 3, 5, 2, 7, 9. Sorted: 1, 2, 3, 4, 5, 6, 7, 9, which gives the ranks 1 to 8 in that order. So rank 1 for the difference $-1$, rank 2 for $-2$, rank 3 for $+3$, rank 4 for $+4$, rank 5 for $+5$, rank 6 for $+6$, rank 7 for $+7$, rank 8 for $+9$.

$W^{+}=3+4+5+6+7+8=33$. $W^{-}=1+2=3$. Total: $33+3=36$, which must equal $\dfrac{n(n+1)}{2}=\dfrac{8\times 9}{2}=36$. The check passes.

Keep the smaller one, $W=3$. The rule is to reject if $W\le W_{crit}$: since $3\le 3$, you only just REJECT the null hypothesis at the 5% level. The observed improvement is therefore significant, by the narrowest of margins. Note that strict equality is enough to reject, which is not obvious and is read from the definition of the critical value: it is the largest $W$ whose cumulative probability does not exceed 2.5% on each side.

A difference of zero can receive neither a sign nor a useful rank: the dominant practice is to EXCLUDE it and reduce $n$ accordingly, which changes the critical value. A more conservative variant splits the zeros between the two camps. What matters is to say what you did: with small samples, excluding two zeros out of ten noticeably changes the threshold.

The SIGN test looks only at the direction of each difference, plus or minus, and ignores their size: on our data it would see six pluses and two minuses, without knowing that the two negative differences are the smallest of all. Wilcoxon, by ranking the absolute values, takes this extra information into account and becomes markedly more powerful. The price is one more assumption: the distribution of the differences must be approximately SYMMETRIC about its median, otherwise the ranks of the positive and negative differences are not comparable.

Increasing order: 12 (C) 1, 15 (C) 2, 17 (C) 3, 18 (B) 4, 20 (B) 5, 21 (C) 6, 22 (A) 7, 24 (B) 8, 25 (A) 9, 27 (B) 10, 28 (A) 11, 31 (A) 12. Sums: $R_{A}=7+9+11+12=39$; $R_{B}=4+5+8+10=27$; $R_{C}=1+2+3+6=12$.

$39+27+12=78$, and $\dfrac{12\times 13}{2}=78$. The check passes. Notice the order of the sums: A occupies the high ranks, C the low ranks, which the test is going to formalize.

$\sum \dfrac{R_{j}^{2}}{n_{j}}=\dfrac{39^{2}}{4}+\dfrac{27^{2}}{4}+\dfrac{12^{2}}{4}=\dfrac{1{,}521+729+144}{4}=\dfrac{2{,}394}{4}=598.5$. Then $H=\dfrac{12}{12\times 13}\times 598.5-3\times 13=\dfrac{12}{156}\times 598.5-39=0.076923\times 598.5-39=46.038-39\approx 7.038$.

Degrees of freedom: $k-1=2$. The chi-square critical value is 5.991 at the 5% level. Since $7.038>5.991$, you REJECT: at least one of the three groups differs from the others. The p-value is about 0.030. A customary caveat: the chi-square approximation assumes at least five observations per group, and we have only four; software would use the exact distribution of $H$ here, and the conclusion might be a little less clear-cut.

Exactly as after an analysis of variance: by pairwise comparisons, here Mann-Whitney tests, with a correction of the threshold for the number of comparisons. With three groups, three comparisons, hence a Bonferroni threshold of $\dfrac{0.05}{3}\approx 0.0167$. With such small samples, no pairwise comparison will reach that threshold: the overall test detects a general trend that the individual comparisons lack the power to locate, a common situation that you must know how to report rather than hide.

$d_{i}$ is the difference between the two ranks of the same observation: its rank on the first variable minus its rank on the second. The formula assumes there are no ties; with ties, you compute Pearson's coefficient directly ON THE RANKS, which gives the same result when there are no ties and remains correct when there are.

Differences: $1-2=-1$; $2-1=1$; $3-4=-1$; $4-3=1$; $5-6=-1$; $6-5=1$; $7-8=-1$; $8-7=1$. Squares: eight times 1, so $\sum d_{i}^{2}=8$. Then $r_{s}=1-\dfrac{6\times 8}{8\times \left(64-1\right)}=1-\dfrac{48}{504}=1-0.095238\approx 0.905$. The two rankings are therefore in very strong agreement, which you can see by eye since each student moves by only one rank.

If they coincided exactly, all the $d_{i}$ would be zero and $r_{s}=1$. If they were exactly reversed, we would have $\sum d_{i}^{2}=\dfrac{n\left(n^{2}-1\right)}{3}$, here $\dfrac{8\times 63}{3}=168$, so $r_{s}=1-\dfrac{6\times 168}{504}=1-2=-1$. The coefficient therefore does range between $-1$ and $+1$, like Pearson's.

Since the relationship is strictly increasing, the ranks match perfectly and $r_{s}=1$ exactly. Pearson's coefficient, for its part, is only about 0.90, because it measures the fit to a LINE and the relationship is exponential. That is the essential difference between the two: Spearman reaches 1 as soon as the relationship is monotonic, whatever its shape, whereas Pearson requires linearity.

First case, when the relationship is monotonic but NONLINEAR: Spearman recognizes it for what it is, Pearson underestimates it. Second case, in the presence of EXTREME VALUES: a single point far away can take a Pearson coefficient from 0.2 to 0.8 or the reverse, while it moves a rank by only one place. We add a third case, a mandatory one: when at least one of the two variables is ORDINAL, Pearson's coefficient simply makes no sense.

$\dfrac{3}{\pi}=\dfrac{3}{3.14159}\approx 0.955$. Translated into sample size: to reach the same power as the t test, you need about $\dfrac{1}{0.955}\approx 1.047$ times as many observations, that is 4.7% more. With 100 subjects per group, you would need 105: the cost is almost nil, while the rank test requires no assumption of normality. That is remarkable value for money, and it is why many methodologists now recommend ranks by default.

It EXCEEDS 1, often by a wide margin: for heavy-tailed distributions, where extreme values are frequent, the efficiency can reach 1.5 and more. In other words, the rank test becomes MORE powerful than the t test, because the mean and the standard deviation are themselves strongly disturbed by extreme values, whereas ranks absorb them. The parametric test is optimal only in the precise case for which it was built.

That caution has a low cost but it is not free, and above all that it changes the question being asked. A t test compares MEANS; Mann-Whitney compares the tendency of one group to exceed the other, which amounts to comparing medians only if the two distributions have the same shape. On data where the mean is the quantity the decision maker cares about, a total income, an expense, you have to test the mean itself. The reasonable rule is: ranks when the variable is ordinal, when the samples are small, or when extremes dominate; otherwise, a parametric test with a check of the residuals.

Generally not, if the only worry is normality: with 200 observations per group, the sampling distribution of the mean is normal by the central limit theorem, whatever the shape of the variable. The t test is then perfectly valid. A rank test remains useful for another reason: if the distribution is very skewed, the MEDIAN may be a much better description than the mean, and you then choose ranks out of relevance and not out of caution.

A rank test never provides an ESTIMATE of the size of the effect in the original unit: it does not say by how many points the groups differ. You make up for it by reporting alongside a readable measure, the difference of medians with its confidence interval, obtained by the Hodges-Lehmann method, or a rank-based effect size, such as the probability that an observation from one group exceeds an observation from the other, which is read directly from $U$. Without that, you publish a decision with no quantity, which is the same flaw as publishing a p-value alone.

$H_{0}$: the probability that a student improves is $0.5$, in other words the median of the individual differences is zero and the direction of the change is a pure coin toss. The two-sided alternative hypothesis is that this probability differs from 0.5.

The number of improvements follows a BINOMIAL distribution with parameters $n=15$ and $p=0.5$, since the fifteen students are independent and each has the same probability of improving under the null hypothesis.

Total number of possible outcomes: $2^{15}=32{,}768$. Number of outcomes giving 12 improvements or more: $455+105+15+1=576$. One-sided probability: $\dfrac{576}{32{,}768}\approx 0.017578$. By symmetry of the distribution for $p=0.5$, the two-sided p-value is twice that: $2\times 0.017578\approx 0.035156$.

$0.0352<0.05$: you REJECT the null hypothesis. The number of improvements is too high to be attributed to the luck of the direction of the changes, and you conclude that the median of the differences is positive. It must be made clear that the conclusion concerns the DIRECTION and not the size: the test completely ignores by how much each student improved.

The sign test uses only twelve pluses and three minuses. Wilcoxon would also use the ranks of the sizes, extra information that makes it more powerful: on data where the improvements are large and the declines tiny, Wilcoxon will reject where the sign test hesitates. In return, Wilcoxon assumes SYMMETRY of the distribution of the differences, whereas the sign test assumes nothing. The choice therefore rests on that assumption: if symmetry is plausible, take Wilcoxon; if the distribution is clearly skewed, fall back on the sign test and accept the loss of power.

FALSE. They all assume INDEPENDENCE of the observations, an assumption that is never negotiable. Most also assume equal SHAPES of the distributions compared, and Wilcoxon requires symmetry of the differences. Correct statement: they do without the assumption of NORMALITY, which is already a lot, but they do not do without every assumption.

FALSE, the rule is reversed for the statistic $U$. You reject when $U$ is LESS THAN OR EQUAL TO the critical value, because a small $U$ means that one group monopolizes the high ranks. With $U=15$ against a critical value of 6, you do not reject. Correct statement: the data do not allow us to conclude that the two groups differ.

FALSE. Spearman measures MONOTONICITY, not linearity: it equals 1 as soon as the relationship is strictly increasing, whether it is a line, an exponential or a logarithm. On the exponential scatter plot of the chapter, Spearman is 1 and Pearson about 0.90. Correct statement: the two rankings coincide perfectly, so the relationship is strictly increasing; its shape remains to be examined on the scatter plot.

FALSE. Under a normal distribution, the efficiency of Mann-Whitney relative to the t test is $\dfrac{3}{\pi}\approx 0.955$, that is 4.7% more observations for the same power. And for heavy-tailed distributions, it EXCEEDS 1. Correct statement: the loss of power is very small in the case most favourable to the parametric test, and it becomes a gain as soon as extreme values are frequent.

FALSE, exactly as for analysis of variance. Kruskal-Wallis is an OMNIBUS test: it indicates that at least one group stands out, without saying which. Correct statement: you must follow up with pairwise comparisons, Mann-Whitney tests with a corrected threshold, and it often happens that none is significant even though the overall test was.

Three reasons add up. The variable is ORDINAL, a seven-level Likert scale: nothing guarantees that the gap between 4 and 5 equals the one between 6 and 7. The SAMPLE SIZES are very small, 7 and 6, so the central limit theorem does not apply. Finally the variable is BOUNDED at 7, which produces a mechanical skew as soon as responses bunch up toward the top, as they do in the guided group.

Values sorted with their groups: 2 (S), 3 (S), 4 (G), 4 (S), 4 (S), 5 (G), 5 (G), 5 (S), 6 (G), 6 (G), 6 (S), 7 (G), 7 (G). The three 4s occupy positions 3, 4 and 5, so each receives the average rank 4. The three 5s occupy 6, 7 and 8, average rank 7. The three 6s occupy 9, 10 and 11, average rank 10. The two 7s occupy 12 and 13, average rank 12.5. Final ranks: 2 gets 1; 3 gets 2; each 4 gets 4; each 5 gets 7; each 6 gets 10; each 7 gets 12.5.

Guided group: one 4, two 5s, two 6s, two 7s, that is $4+7+7+10+10+12.5+12.5=63$. Self-directed group: one 2, one 3, two 4s, one 5, one 6, that is $1+2+4+4+7+10=28$. Check: $63+28=91=\dfrac{13\times 14}{2}$. Then $U_{G}=63-\dfrac{7\times 8}{2}=63-28=35$ and $U_{S}=28-\dfrac{6\times 7}{2}=28-21=7$. Check: $35+7=42=7\times 6$.

Keep $U=7$. Since $7>6$, you do NOT reject at the 5% level, but only just. The data strongly suggest that the guided group finds the course clearer, without thirteen observations being enough to establish it. The honest write-up is to give the exact p-value, here about 0.062 (exact permutation of the ranks, ties included), rather than flatly concluding there is no difference. The normal approximation corrected for ties gives 0.042 and would reject, but it is unreliable with 7 and 6 observations: the exact value is the one that counts.

The MEDIAN of each group, here 6 for the guided group and 4 for the self-directed one, and above all the probability that an observation from the guided group exceeds an observation from the self-directed group, which is read directly from $U$: $\dfrac{U_{G}}{n_{G}n_{S}}=\dfrac{35}{42}\approx 0.833$. In other words, in five pairs out of six, the guided student finds the course clearer. This measure, called the probability of superiority, is far more telling than the raw $U$ statistic and it does not depend on the sample sizes.

No, not automatically. With 60 subjects per group, the sampling distribution of the mean is normal by the central limit theorem, whatever the shape of the variable: the t test would be valid. Moreover, a normality test becomes very powerful as $n$ grows and then detects departures of no practical importance: on 118 observations, a $p$ of 0.03 may correspond to a tiny deviation. The decision should therefore rest on a plot of the residuals and on the question being asked, not on a normality test. That said, if the variable is ordinal or very skewed, the choice of ranks remains defensible, for reasons other than the one given.

The product $n_{1}n_{2}=60\times 58=3{,}480$, and $U$ must lie between 0 and 3,480: the value 812 is therefore plausible. The probability of superiority is $\dfrac{812}{3{,}480}\approx 0.233$, or its complement $\dfrac{2{,}668}{3{,}480}\approx 0.767$ depending on the direction chosen: in about three pairs out of four, a member of the second group exceeds a member of the first. This reading is far more informative than the p-value.

Two criticisms. Mann-Whitney does NOT test means: it tests the tendency of one group to produce higher values, which translates into medians only if the two distributions have the same shape. And the author, who set the t test aside precisely because the distribution was not normal, comes back to conclude about the mean, which is contradictory. Correct statement: the values of the second group tend to exceed those of the first.

It lacks the MEDIANS and quartiles of each group, so the reader knows the order of magnitude of the gap; an effect size, the probability of superiority or the difference of medians with its interval; the real justification for the choice of ranks, the nature of the variable rather than the result of a normality test; and the direction of the test, two-sided or one-sided. A p-value and a $U$ statistic alone allow neither judging nor reproducing.

“Because the variable is an ordinal scale and its distribution is strongly skewed, we compared the two groups with a two-sided Mann-Whitney test. The values of group 2 tend to exceed those of group 1, $U=812$; $n_{1}=60$; $n_{2}=58$; $p=0.021$. The medians are 4 and 5 respectively, with interquartile ranges of 2 in both groups, and the probability that an observation from group 2 exceeds an observation from group 1 is 0.77.” The paragraph names the real reason for the choice, states the conclusion in the terms the test allows, and finally gives the reader a quantity.