2Research methods
Effect size and confidence intervals
Effect size says how big a difference is; the p-value only says how surprising it would be under the null. With a large enough sample a clinically meaningless difference becomes statistically significant, so effect sizes and confidence intervals are reported alongside p.
Practice
11 questions
Level
Core
Reading time
3 min
Common effect sizes
Which one to report follows from the test:
- Cohen’s d: the standardised mean difference. Roughly .2 small, .5 medium, .8 large (Lakens, 2013).
- Pearson’s r: .1 small, .3 medium, .5 large. r² is the proportion of variance explained.
- η² (eta squared), in ANOVA: the share of all the variance in the outcome that a factor explains.
- Partial η²: the factor’s share once the variance other factors explain is set aside. It is never smaller than η², and compares only across similar designs (Lakens, 2013).
- Odds ratio, risk ratio and number needed to treat: for categorical outcomes. The NNT is 1 ÷ the absolute risk reduction (Laupacis, Sackett and Roberts, 1988).
Worked example:From two means to Cohen’s d
- After a social anxiety group, members score a mean of 27 on an anxiety measure and people still on the waiting list 33. The pooled standard deviation is 12.
- Cohen’s d puts the gap between the two means in standard deviations:d = (33 − 27) ÷ 12 = 0.5
- Half a standard deviation: medium by Cohen’s rough benchmarks, though what counts as large depends on the field and the outcome.
Source: Lakens (2013), checked .
Confidence intervals
A 95% confidence interval is the range of values compatible with the data at that confidence level. Strictly, 95% of intervals constructed this way across repeated samples would contain the true population value. That is different from a 95% probability that the true value lies in this particular interval.
A confidence interval carries more information than a p-value because it shows both the estimate and its precision. A wide interval spanning zero tells you the study was uninformative; a narrow one close to zero tells you the effect is small.
Clinical against statistical significance
A statistically significant change of two points on a depression measure may be well below the minimal clinically important difference.
Worked example:Significant, and too small to matter
- A trial with 1,000 people in each arm finds a difference of 2 points on a depression measure. The standard deviation is 8.
- With so many people, the estimate is precise:SE = 8 × √(2 ÷ 1,000) = 0.36t = 2 ÷ 0.36 = 5.6, p < .001
- The 95% confidence interval:2 ± 1.96 × 0.36 = 1.3 to 2.7
- Suppose patients notice a change only from 5 points. The whole interval lies below that: statistically significant, and on average too small to matter to the people treated.
Source: Cochrane Handbook, chapter 15, checked .
Common mistakes
- Reporting significance without effect size.
- Interpreting Cohen’s benchmarks as fixed rather than field-dependent.
- Saying a confidence interval has a 95% chance of containing the true value.
Sources
- Cochrane Handbook for Systematic Reviews of Interventions, version 6.5 (2024), chapter 15: Interpreting results and drawing conclusions. Schünemann, Vist, Higgins, Santesso, Deeks, Glasziou and colleagues; last updated August 2023. Checked .
- Field (2024), Discovering Statistics Using IBM SPSS Statistics, sixth edition. SAGE. Checked .
- Greenland, Senn, Rothman, Carlin, Poole, Goodman and Altman (2016), Statistical tests, P values, confidence intervals, and power: a guide to misinterpretations. European Journal of Epidemiology, 31(4), 337–350. Checked .
- Lakens (2013), Calculating and reporting effect sizes to facilitate cumulative science: a practical primer for t-tests and ANOVAs. Frontiers in Psychology, 4, 863. Checked .
- Laupacis, Sackett and Roberts (1988), An assessment of clinically useful measures of the consequences of treatment. New England Journal of Medicine, 318(26), 1728–1733. Checked .