Statistics · Core

Effect size and confidence intervals

Effect size says how big a difference is; the p-value only says how surprising it would be under the null. With a large enough sample, a clinically meaningless difference becomes statistically significant, which is why effect sizes and confidence intervals are reported alongside p.

Common effect sizes

Which one to report follows from the test:

Cohen's d — standardised mean difference. Roughly: .2 small, .5 medium, .8 large.
Pearson's r — .1 small, .3 medium, .5 large. r² is the proportion of variance explained.
η² and partial η² — proportion of variance explained in ANOVA.
Odds ratio and risk ratio — for categorical outcomes.

Confidence intervals

A 95% confidence interval is the range of values compatible with the data at that confidence level. Strictly, it means that 95% of intervals constructed this way across repeated samples would contain the true population value — not that there is a 95% probability the true value lies in this particular interval.

A confidence interval carries more information than a p-value because it shows both the estimate and its precision. A wide interval spanning zero tells you the study was uninformative; a narrow one close to zero tells you the effect is genuinely small.

Clinical against statistical significance

A statistically significant change of two points on a depression measure may be well below the minimal clinically important difference. In clinical psychology this distinction matters more than in most fields, and selection panels ask about it.

Where marks get lost

Reporting significance without effect size.
Interpreting Cohen’s benchmarks as fixed rather than field-dependent.
Saying a confidence interval has a 95% chance of containing the true value.

4 questions on this topic.

Test yourself