Choosing the right test

Comparing groups

For differences between conditions on a continuous outcome:

• Two independent groups — independent t-test; Mann-Whitney U if assumptions fail.
• Two related conditions — paired t-test; Wilcoxon signed-rank if assumptions fail.
• Three or more independent groups — one-way ANOVA; Kruskal-Wallis if assumptions fail.
• Three or more related conditions — repeated-measures ANOVA; Friedman if assumptions fail.
• Categorical outcome and categorical predictor — chi-square test of independence.

Examining relationships

For association rather than difference:

• Two continuous variables, linear — Pearson’s r.
• Ordinal data or a monotonic but non-linear relationship — Spearman’s rho.
• Predicting a continuous outcome from one or more predictors — linear regression.
• Predicting a binary outcome — logistic regression.

Parametric assumptions

Parametric tests assume interval or ratio data, approximately normal sampling distributions, and — for between-groups comparisons — homogeneity of variance. Independence of observations is assumed by all of them and is the one that cannot be worked around.

Where marks get lost

• Running multiple t-tests instead of an ANOVA, inflating the Type I error rate.
• Using an independent t-test on repeated measures, which discards the pairing and loses power.
• Reaching for a non-parametric test at the first sign of skew, when the sample is large enough for the CLT to apply.