Advanced Quantitative Methods 360-301 • CEGEP, Montreal

Revision sheet: nonparametric tests (360-301)

This sheet covers the family of tests you use when the distribution refuses to cooperate: small samples, ordinal scales, extreme values. It does not repeat the formulas; it gives the eight places where a correct rank calculation leads to a wrong conclusion, starting with the direction of rejection, which is reversed compared with every other test in the course.

It is written for Social Science students at CEGEP taking 360-301. The set of corrected exercises on the same chapter goes through each test, from Mann-Whitney to reading a methods section.

Mark this sheet as read or add it to your favourites: a free account, no password, keeps your read sheets and favourites from one visit to the next, tells you which chapter to tackle next and lets you ask for the method or exercise you are missing. Create your space, an email is enough.

The thread of the chapter

Switching to ranks means trading information for robustness: you no longer know BY HOW MUCH the values differ, only in what order they arrive. The whole chapter is judged on that trade.

This chapter is part of 360-301, Advanced Quantitative Methods

What is on this sheet

1. The essentials
2. The mistakes that cost marks
3. Which method to choose
4. How the answer is expected to be written
5. Check before you hand in
6. The typical problem, taken apart
7. Learn by heart
8. Frequently asked questions

Topics covered

Rank statistic
Average ranks for ties
Mann-Whitney U test
Probability of superiority
Wilcoxon signed-rank test
Kruskal-Wallis test
Spearman's rank correlation
Monotonic relationship
Sign test
Relative efficiency
Heavy-tailed distributions
Independence assumption

The essentials

What ranks keep and what they destroy

• Ranks are insensitive to the SIZE of the gaps: replacing $40$ with $400$ changes no rank, hence no result.
• That is both the strength of the method, an extreme value no longer distorts anything, and its limit: you can no longer estimate by how much the groups differ.
• TIES: the AVERAGE rank of the positions occupied. Three equal values in positions $5$ , $6$ and $7$ each receive the rank $6$ .
• Universal check: the sum of all the ranks equals $\dfrac{N(N+1)}{2}$ , ties included.

Above, the values in their true places; below, the same ones spread at equal intervals: all the distance information has vanished between the two lines.

A rank test gives no estimate in the original unit: you must report medians, quartiles and a probability of superiority.

The four tests, and their statistics

• MANN-WHITNEY, two independent groups: $U_{j}=R_{j}-\dfrac{n_{j}\left(n_{j}+1\right)}{2}$ , with $U_{1}+U_{2}=n_{1}n_{2}$ . Reject if the SMALLER $U$ is LESS THAN OR EQUAL TO the critical value.
• WILCOXON, paired samples: rank the $\left|d_{i}\right|$ , then add up the ranks of the positive and of the negative differences; $W^{+}+W^{-}=\dfrac{n(n+1)}{2}$ .
• KRUSKAL-WALLIS, $k$ groups: $H=\dfrac{12}{N(N+1)}\sum \dfrac{R_{j}^{2}}{n_{j}}-3\left(N+1\right)$ , compared with a chi-square distribution with $k-1$ degrees of freedom. An OMNIBUS test.
• SPEARMAN: $r_{s}=1-\dfrac{6\sum d_{i}^{2}}{n\left(n^{2}-1\right)}$ where $d_{i}$ is the difference between the TWO RANKS. It measures MONOTONICITY, not linearity.

Mann-Whitney is the only test in the course where you reject for a SMALL statistic: it is the most frequent mechanical mistake.

What they still assume, and what they cost

• All assume INDEPENDENCE of the observations, and most assume equal SHAPES of the distributions compared.
• Without equal shapes, Mann-Whitney tests a general difference in location, never specifically the medians.
• WILCOXON also assumes SYMMETRY of the distribution of the differences; without it, the SIGN test remains.
• Efficiency of Mann-Whitney relative to the t test: $\dfrac{3}{\pi}\approx 0.955$ under the normal distribution, that is $4.7$ % more observations; it EXCEEDS $1$ for heavy-tailed distributions.

Ranks go on the ABSOLUTE DIFFERENCES and the sign is restored afterwards: rank

5

goes to the largest difference,

\left|+8\right|

, whatever its direction.

“Nonparametric” does not mean “assumption-free”: the term only means that the shape of the distribution has no parameter to estimate.

The mistakes that cost marks

These are the errors I correct most often in session. Each one costs marks on a paper, even when the reasoning behind it is right.

1. Ranking ties in order of arrival

1.5 marks, and the result then depends on the order of data entry

What not to write

“Three equal values: I give them the ranks $5$ , $6$ and $7$ in the order of the table.”

What to write

“They each receive the AVERAGE rank of the positions occupied, that is $\dfrac{5+6+7}{3}=6$ . The total of the ranks stays $\dfrac{N(N+1)}{2}$ .”

The three values

3

occupy positions

5

6

and

7

, in red: each receives the average rank

6

, in green. The total of the ranks stays

78

Why: Two identical values must receive the same rank, otherwise the ranking encodes the order of the file instead of the data. The check by the sum catches the error immediately.

2. Comparing the larger U with the critical value

the whole question, since the decision is reversed

What not to write

“ $U_{A}=26$ and the critical value is $5$ : $26$ exceeds $5$ , so I reject.”

What to write

“You compare the SMALLER of the two, here $U_{B}=10$ , and reject only if it is LESS THAN OR EQUAL TO the critical value. $10>5$ : do not reject.”

You reject if the SMALLER

U

falls in the red zone, from

0

5

. Here

U_{B}=10

, in green, is outside it: do not reject, despite

U_{A}=26

, in grey.

Why: A small $U$ means that one group dominates the other by a wide margin. That is the opposite of every other test in the course, and it is exactly why the mistake is so frequent.

3. Announcing a test of medians without checking the shapes

1.5 marks, and a conclusion stronger than the test

What not to write

“Mann-Whitney is significant: the medians of the two groups differ.”

What to write

“Mann-Whitney tests a difference in GENERAL LOCATION. It concerns the medians only if the two distributions have the same SHAPE, which I must check before saying so.”

Same median, in purple, but two shapes: one symmetric, the other stretched to the right. A rank test can separate them although no median differs.

Why: Two distributions of different shapes can have the same median and still give a highly significant test, because the probability of superiority departs from one half.

4. Applying Wilcoxon to skewed differences

1 mark, and an actual significance level different from the one announced

What not to write

“The differences are spread out mostly on one side, but Wilcoxon is nonparametric, so it applies.”

What to write

“Wilcoxon assumes SYMMETRY of the distribution of the differences. Without it, I switch to the SIGN test, which assumes only independence.”

Why: Wilcoxon uses the SIZE of the differences through their ranks: if the large differences all lie on the same side, the statistic is biased even though the median of the differences is zero.

5. Confusing Spearman and Pearson

1.5 marks, and the question on the shape of the relationship

What not to write

“ $r_{s}=1.00$ : the relationship is perfectly linear.”

What to write

“ $r_{s}=1.00$ says that the relationship is perfectly MONOTONIC: whenever $x$ goes up, $y$ goes up. The curve may be an exponential, and Pearson's $r$ is then only $0.91$ .”

The eight points rise without exception, so the ranks coincide and Spearman is

1

; the curvature, for its part, costs Pearson almost 0.1.

Why: Spearman is Pearson computed ON THE RANKS: it sees only the order. A monotonic curvature is perfectly invisible to it, whereas it makes Pearson's coefficient drop.

6. Publishing means after a rank test

1 mark of consistency, and the marker always spots it

What not to write

“The Mann-Whitney test rejects: the mean of group A, $3.67$ , exceeds that of group B, $2.67$ .”

What to write

“The test worked on the ranks: I report MEDIANS, $3.5$ against $2.5$ , the quartiles, and the probability of superiority, $0.72$ .”

Why: We chose ranks precisely because the mean was not reliable here. Reporting it in the conclusion cancels the reason for the choice.

7. Believing a nonparametric test is assumption-free

1 mark of method, more if the data are paired

What not to write

“The data are not normal, so I take a rank test: there is no condition left to check.”

What to write

“INDEPENDENCE remains mandatory, and most of these tests assume equal shapes. ‘Nonparametric’ means that the distribution has no parameter to estimate, not that it can be anything.”

Why: A violated independence assumption distorts every test in the same way, parametric or not: it is the design of the study that guarantees it, never the choice of the statistic.

8. Switching to ranks out of caution when the t test applied

1 mark, and power lost for nothing

What not to write

“When in doubt, I always run a rank test: it is safer.”

What to write

“Under the normal distribution, Mann-Whitney has an efficiency of $\dfrac{3}{\pi}\approx 95.5$ %: you need about $4.7$ % more observations for the same power. It is free only if the conditions of the t test really fail.”

Why: The price is low but real, and above all a rank test produces neither a difference of means nor a confidence interval in the original unit, which the conclusion often calls for.

Which method to choose

Which rank test?

Count the groups, see whether they are the same individuals, and what the question is trying to establish.

If two INDEPENDENT groups → MANN-WHITNEY: reject if the smaller $U$ is less than or equal to the critical value

Example: two classes of six, answers on a Likert scale
If the same individuals measured twice, SYMMETRIC differences → WILCOXON signed-rank test, with no $df$ , only its own table of critical values

Example: before and after, fifteen participants
If the same individuals, ASYMMETRIC differences or only the direction known → SIGN TEST, binomial distribution with parameters $n$ and $0.5$

Example: each judge prefers A or B
If three or more independent groups → KRUSKAL-WALLIS, compared with a chi-square distribution with $k-1$ degrees of freedom, then corrected pairwise comparisons

Example: three study programs
If looking for a link between two ordinal variables → SPEARMAN, which measures monotonicity

Example: rank in high school and rank at CEGEP

If none of these cases applies and the conditions of the t test are met, the t test is the one to run: it is more powerful and gives an interpretable interval.

What to report after a rank test?

Look at what the question asks, then choose measures that are CONSISTENT with ranks.

If the groups must be described → MEDIANS and QUARTILES, never means and standard deviations

Example: $3.5$ versus $2.5$ on a Likert scale
If the effect size must be given after Mann-Whitney → PROBABILITY OF SUPERIORITY $\dfrac{U}{n_{1}n_{2}}$ , between $0$ and $1$

Example: $\dfrac{26}{36}\approx 0.72$
If the effect size must be given after Spearman → $r_{s}$ itself, stating that it measures MONOTONICITY

Example: $r_{s}=0.84$
If the test does not reject and the samples are small → point out the LACK OF POWER, and still give the observed effect size

Example: probability of superiority of $0.72$ without significance

A large effect without significance, on small groups, is a result worth reporting: it justifies a larger study.

How the answer is expected to be written

A marker ticks steps. Here they are in order, with the concluding sentence expected word for word.

Writing up a Mann-Whitney test

When to use it: As soon as the question compares two independent groups on an ordinal variable, or with very small samples.

1 Justify the choice: ordinal scale, small samples or a strongly skewed distribution.
2 Rank ALL the observations together, give ties their average ranks, and check that the sum equals $\dfrac{N(N+1)}{2}$ .
3 Compute $R_{1}$ and $R_{2}$ , then the two $U$ values, and check that $U_{1}+U_{2}=n_{1}n_{2}$ .
4 Compare the SMALLER $U$ with the critical value, and reject if it is less than or equal to it.
5 Conclude with the MEDIANS and the probability of superiority.

Concluding sentence

“The smaller $U$ is $10$ , above the critical value $5$ for two groups of six at the 5% level: we do not reject the hypothesis that the distributions are equal. The probability of superiority is nevertheless $0.72$ , which would justify a study on larger samples.”

The trap: Subtracting $\dfrac{n(n+1)}{2}$ with the wrong $n$ : it is the size of the GROUP whose ranks you are summing, not the total size.

Marking: 1 mark for the justified choice, 2 marks for the ranks, 1 mark for the $U$ values and their check, 1 mark for the decision, 1 mark for the effect size.

Computing a Spearman coefficient

When to use it: When the question gives two rankings, or two ordinal variables.

1 Rank each variable separately, with average ranks for ties.
2 Compute the rank differences $d_{i}$ , then their squares, and check that $\sum d_{i}=0$ .
3 Apply $r_{s}=1-\dfrac{6\sum d_{i}^{2}}{n\left(n^{2}-1\right)}$ .
4 Interpret in terms of MONOTONICITY, and not of linearity or proportionality.

Concluding sentence

“Spearman's coefficient is $0.84$ : the CEGEP ranking closely follows the high school one, which describes a monotonic relationship and not necessarily a linear one.”

The trap: Using the short formula when there are many ties: it is exact only without ties, otherwise you must go through Pearson's formula applied to the ranks.

Marking: 1 mark for the two rankings, 1 mark for the $d_{i}^{2}$ , 1 mark for the calculation, 1 mark for the interpretation as monotonicity.

Check before you hand in

Five minutes of checking recover more marks than one more problem started in a hurry.

The sum of all the ranks

Add up all the ranks assigned: the total must equal $\dfrac{N(N+1)}{2}$ , ties included.

For $N=12$ , you expect $78$ : a total of $76$ signals a badly computed average rank.
The sum of the two U values

$U_{1}+U_{2}$ must equal exactly $n_{1}n_{2}$ . It is the instant check on the two rank sums.

$26+10=36=6\times 6$ : the two $U$ values are right.
The direction of rejection

On Mann-Whitney and Wilcoxon, you reject for a SMALL statistic. Telling yourself so before concluding avoids the costliest mistake of the chapter.

A $U$ of $30$ for two groups of six is close to the maximum $36$ : the groups look alike, do not reject.
The probability of superiority stays between 0 and 1

$\dfrac{U}{n_{1}n_{2}}$ is a proportion of pairs: above $1$ , you divided by the wrong product.

$\dfrac{26}{36}=0.72$ : in 72% of the pairs, a member of group A exceeds a member of group B.

The typical problem, taken apart

A Likert scale on two small groups, from start to finish

Group A, six students, answers on a scale from $1$ to $5$ : $2$ , $3$ , $3$ , $4$ , $5$ , $5$ .

Group B, six students: $1$ , $2$ , $2$ , $3$ , $4$ , $4$ .

We ask whether the two groups differ, and by how much.

The twelve answers on the scale from

1

5

, A on top and B at the bottom: many ties, and group A sits one step higher.

Step 1

ORDINAL scale and six observations per group: neither the mean nor normality is defensible. MANN-WHITNEY test.

Why

The justification of the test comes before the calculation and earns its own mark. These are the same two reasons in almost every question of the chapter: ordinal scale and small sample.

Step 2

Ranks over the twelve pooled values: the $1$ takes rank $1$ ; the three $2$ s take the average rank $3$ ; the three $3$ s rank $6$ ; the three $4$ s rank $9$ ; the two $5$ s rank $11.5$ . Sum: $78=\dfrac{12\times 13}{2}$ .

Why

The check by the sum is done BEFORE separating the groups: an error in an average rank shows up there and nowhere else.

Step 3

$R_{A}=3+6+6+9+11.5+11.5=47$ and $R_{B}=1+3+3+6+9+9=31$ , summing to $78$ .

Why

You add the two back up: if the total does not give $78$ again, an observation was ranked twice or not at all.

Step 4

$U_{A}=47-\dfrac{6\times 7}{2}=26$ and $U_{B}=31-21=10$ ; check: $26+10=36=6\times 6$ .

Why

The term subtracted is that of the GROUP whose ranks you are summing, not of the total. The check $U_{1}+U_{2}=n_{1}n_{2}$ catches the error immediately.

Step 5

The smaller $U$ is $10$ . Critical value for $n_{1}=n_{2}=6$ at the two-sided 5% level: $5$ . Since $10>5$ , you do NOT reject.

Why

This is where half the papers go wrong: you reject for a SMALL $U$ . Writing the rule next to the calculation avoids the reversal.

Step 6

Effect size: $\dfrac{U_{A}}{n_{1}n_{2}}=\dfrac{26}{36}\approx 0.72$ ; medians $3.5$ for A and $2.5$ for B.

Why

The effect is notable and the test does not reject: with six observations per group, the power is very low. Reporting both is the honest conclusion.

The conclusion, written out

“The smaller $U$ is $10$ , above the critical value $5$ : at the 5% level, the data do not allow us to conclude that the two groups differ. The median is nevertheless $3.5$ against $2.5$ and the probability of superiority reaches $0.72$ , which justifies repeating the study on larger samples.”

The classic mistake on this problem: Comparing $U_{A}=26$ with the critical value $5$ and concluding to reject: it is the SMALLER of the two $U$ values that decides, and rejection requires it to be below the threshold.

Learn by heart

• Ranks ignore the SIZE: replacing $40$ with $400$ changes nothing.
• TIES: AVERAGE rank of the positions occupied, and the sum of all the ranks equals $\dfrac{N(N+1)}{2}$ .
• Mann-Whitney: $U_{j}=R_{j}-\dfrac{n_{j}(n_{j}+1)}{2}$ , $U_{1}+U_{2}=n_{1}n_{2}$ , and you reject for the SMALLER $U$ .
• Wilcoxon ranks the ABSOLUTE DIFFERENCES and assumes SYMMETRY of the differences; without it, the sign test.
• Spearman measures MONOTONICITY, not linearity: an increasing exponential gives it $1$ .
• After a rank test, you publish MEDIANS and a probability of superiority, never means.
• Efficiency $\dfrac{3}{\pi}\approx 95.5$ % under the normal distribution, and above $1$ for heavy-tailed distributions.

Frequently asked questions

When should you use a nonparametric test?

When the variable is ordinal, when the samples are very small, or when the distribution is strongly skewed or contains extreme values. In other cases the t test remains preferable: it is slightly more powerful and it gives a difference of means with its confidence interval, which a rank test never does.

How do you assign ranks when there are ties?

Each repeated value receives the average of the positions it occupies. Three identical values placed in positions five, six and seven therefore all receive rank six. The check is immediate: the sum of all the ranks assigned must equal the number of observations times that number plus one, all divided by two.

Which way do you reject with the Mann-Whitney test?

You compare the smaller of the two U values with the critical value and reject if it is less than or equal to it. This is the opposite of most tests, where you reject for a large statistic. A small U means that one group dominates the other in the pairwise comparisons.

What is the difference between Spearman and Pearson?

Pearson measures the strength of a linear relationship on the values themselves, while Spearman does the same calculation on the ranks and therefore measures monotonicity. A curve that rises without being straight gives a Spearman coefficient of one and a clearly lower Pearson coefficient. Spearman is also much less sensitive to extreme values.

How do you measure the effect size after a rank test?

With the probability of superiority, that is U divided by the product of the two sample sizes. It gives the proportion of pairs in which a member of the first group exceeds a member of the second, and it equals one half when the two groups are alike. Report it alongside the medians and quartiles, never the means.

Practise it

Corrected exercises: Nonparametric tests, 360-301

A method is proved on a paper, not on a sheet. The set for the same chapter takes each of these traps into a problem, with the solution written out step by step.

10 corrected exercises
100 points
150 minutes

Do the exercises

Something missing from this sheet? A method, a trap, an exercise to practise it, a mistake you spotted: tell me, I read every request.

Sending goes through your free space: that is how I can answer you and tell you when the exercise is added. An email is enough, no password. Create your space. Rather not create an account? Write to me directly.

Previous sheet Multiple regression Next sheet Time series and forecasting

© Ahmed Squalli Houssaini. Revision sheet published at www.letuteurscientifique.ca/en/fiches/cegep-aqm-nonparametric-tests. Free for personal and classroom use; republishing it elsewhere requires written permission (legal notice).

Looking for a 360-301 tutor in Montreal?

Get in touch for a first session. 360-301 arrives just as the end-of-program research project begins, and it is the course where a method error costs you the whole analysis chapter.

Book a session WhatsApp

Site by Studio Squalli

Revision sheet: nonparametric tests (360-301)

The essentials

What ranks keep and what they destroy

The four tests, and their statistics

What they still assume, and what they cost

The mistakes that cost marks

1. Ranking ties in order of arrival

2. Comparing the larger U with the critical value

3. Announcing a test of medians without checking the shapes

4. Applying Wilcoxon to skewed differences

5. Confusing Spearman and Pearson

6. Publishing means after a rank test

7. Believing a nonparametric test is assumption-free

8. Switching to ranks out of caution when the t test applied

Which method to choose

Which rank test?

What to report after a rank test?

How the answer is expected to be written

Writing up a Mann-Whitney test

Computing a Spearman coefficient

Check before you hand in

The typical problem, taken apart

A Likert scale on two small groups, from start to finish

Learn by heart

Frequently asked questions

When should you use a nonparametric test?

How do you assign ranks when there are ties?

Which way do you reject with the Mann-Whitney test?

What is the difference between Spearman and Pearson?

How do you measure the effect size after a rank test?

Corrected exercises: Nonparametric tests, 360-301

See also

Looking for a 360-301 tutor in Montreal?