2Research methods
Interpreting results
Interpreting a result means saying what a reported statistic shows, what it does not, and whether it matters for the people it describes.
Practice
35 questions
Level
Advanced
Reading time
7 min
Reading reported statistics
The figures in brackets after a test statistic are its degrees of freedom. With the test, they let you check the design and the sample size the method section reports.
- t(58): an independent t-test on 60 people in all (df = n − 2), or a paired t-test on 59 pairs (df = n − 1).
- F(2, 87) in a one-way between-groups ANOVA: groups minus one is 2, so three groups; total minus groups is 87, so 90 people.
- b: the unstandardised regression coefficient, meaning the change in the outcome, in its own units, for each unit of the predictor.
- β: the standardised coefficient, in standard deviations, so predictors measured on different scales can be compared.
- R²: the proportion of variance in the outcome the model explains. Adjusted R² penalises predictors that add little.
Regression, including predictors entered in steps
Multiple regression predicts one outcome from several predictors. Each coefficient is that predictor’s contribution with the others held constant, so it can shrink when a correlated predictor joins the model.
In hierarchical regression the researcher sets the order before seeing the results. Predictors go in as blocks, ordered by theory or earlier research, and the change in R² at each step is the variance a block explains over and above the blocks before it. Entering baseline severity first controls for it statistically; it does not do what randomisation does. This is standard teaching (Field, 2024; Cohen and colleagues, 2003).
In stepwise regression the software chooses, adding or dropping predictors by statistical criteria. Thompson (1995) names three problems: packages use the wrong degrees of freedom, the method does not find the best set of predictors of a given size, and it capitalises on sampling error, so the results tend not to replicate.
- Hierarchical: the researcher fixes the order in advance. Report the R² change, and its F test, at each step.
- Stepwise (forward or backward): the software picks the order from the data. Exploratory at best.
- ΔR²: the extra proportion of variance explained by the block just added.
- The order changes how the explained variance is shared out between steps. The coefficients in the final model stay the same.
Interactions, moderation and mediation
A significant interaction means the effect of one factor depends on the level of another. Once there is one, the main effects can mislead, because an average across levels may describe nobody; the next step is to look at simple effects. On a plot, lines that are not parallel suggest an interaction.
A moderator changes the strength or direction of a relationship: it answers for whom, or when. A mediator is the route by which a predictor has its effect: it answers how, or why. Baron and Kenny set out the distinction in 1986.
Mediation is a claim about a causal sequence. When predictor, mediator and outcome are all measured at the same moment, the data cannot show that order, and Maxwell and Cole (2007) showed that cross-sectional estimates of mediation can be badly biased.
Source: Field (2024), checked .
Reliable and clinically significant change
For one client, two questions matter. Is the change bigger than measurement error alone would produce? And has the person moved into the range of people without the problem? Jacobson and Truax (1991) turned these into the reliable change index and the criteria for clinically significant change.
- Standard error of measurement: SE = SD × √(1 − r), where r is the measure’s reliability.
- Standard error of the difference: Sdiff = √2 × SE.
- Reliable change index: RCI = (post − pre) / Sdiff. Beyond ±1.96, the change is unlikely (p < .05) to be measurement error alone.
- Clinically significant change: reliable change, plus a move across a cut-off into the functional range.
Worked example:Did this client change?
- A depression measure has a standard deviation of 7.5 and a reliability of .85. A client scores 28 at the start of therapy and 18 at the end.
- Standard error of measurement:SE = 7.5 × √(1 − .85) = 2.90
- Standard error of the difference:Sdiff = √2 × 2.90 = 4.11
- Reliable change index:RCI = (18 − 28) ÷ 4.11 = −2.43
- Beyond −1.96, so the 10-point fall is unlikely to be measurement error alone. On this measure a change has to be more than about 8 points to count as reliable.1.96 × 4.11 = 8.1
- Whether it is also clinically significant depends on the measure’s cut-off for the functional range.
Source: Jacobson and Truax (1991), checked .
Screening tests
Sensitivity and specificity describe a test (Altman and Bland, 1994a). Predictive values describe what a result means for the person in front of you, and they depend on how common the condition is: where it is rare, even a good test can produce more false positives than true ones (Altman and Bland, 1994b).
- Sensitivity: the proportion of people with the condition who test positive.
- Specificity: the proportion of people without it who test negative.
- Positive predictive value: the proportion of positive results that are true. It falls as prevalence falls.
Worked example:Screening 1,000 people
- A screening questionnaire picks up 90% of people with a condition (its sensitivity) and clears 90% of people without it (its specificity). Of 1,000 people screened, 1% have the condition: 10 people.
- True positives, from the 10 with the condition:.90 × 10 = 9
- False positives, from the 990 without it:.10 × 990 = 99
- Positive predictive value:9 ÷ (9 + 99) = 9 ÷ 108 = 8%
- About 1 in 12 people who screen positive have the condition, from a test that is right for 9 in 10 people with it and 9 in 10 without.
Source: Altman and Bland (1994b), checked .
Risk and treatment effects
When an outcome either happens or does not, such as recovery or relapse, a trial compares how often it happens in each arm.
- Number needed to treat: 1 ÷ the absolute risk reduction (Laupacis, Sackett and Roberts, 1988).
- Odds ratio: close to the risk ratio when the outcome is rare, and further from 1 than the risk ratio when it is common (Bland and Altman, 2000).
Worked example:From recovery rates to the NNT
- In a trial, 45% of clients recover with the therapy and 20% with usual care.
- The absolute difference between the arms:.45 − .20 = .25
- The number needed to treat:NNT = 1 ÷ .25 = 4
- In this trial, about one more client recovered for every four given the therapy instead of usual care.
Source: Laupacis, Sackett and Roberts (1988), checked .
Reading a meta-analysis
A forest plot shows each study’s estimate with its confidence interval, and a diamond for the pooled estimate. The vertical line marks no effect: zero for a difference, one for a ratio. A study whose interval crosses it is, on its own, compatible with no effect.
I² estimates the percentage of the variation between study results that reflects real differences rather than chance. Higgins and colleagues (2003) tentatively described 25%, 50% and 75% as low, moderate and high. A funnel plot that is not symmetrical can signal small-study effects such as publication bias, though asymmetry has other causes too (Sterne and colleagues, 2011).
Source: Cochrane Handbook, chapter 10, checked .
Common mistakes
- Interpreting main effects as if a significant interaction were not there.
- Calling a variable a mediator on the strength of a cross-sectional correlation.
- Treating any improvement on a measure as meaningful without asking whether it exceeds measurement error.
- Quoting a test’s sensitivity as if it were the chance that someone who screens positive has the condition.
- Reading an odds ratio as “times as likely” when the outcome is common.
- Calling a model hierarchical when the software chose the order: that is stepwise.
- Treating the predictors a stepwise model kept as the important ones, when another sample could keep different ones.
- Reading “controlling for” a covariate entered first as if the groups had been matched or randomised.
Sources
- Altman and Bland (1994a), Diagnostic tests 1: sensitivity and specificity. BMJ, 308, 1552. Checked .
- Altman and Bland (1994b), Diagnostic tests 2: predictive values. BMJ, 309, 102. Checked .
- Baron and Kenny (1986), The moderator–mediator variable distinction in social psychological research. Journal of Personality and Social Psychology, 51(6), 1173–1182. Checked .
- Bland and Altman (2000), The odds ratio. BMJ, 320, 1468. Checked .
- Cochrane Handbook for Systematic Reviews of Interventions, version 6.5 (2024), chapter 10: Analysing data and undertaking meta-analyses. Deeks, Higgins, Altman, McKenzie and Veroniki; last updated November 2024. Checked .
- Cohen, Cohen, West and Aiken (2003), Applied Multiple Regression/Correlation Analysis for the Behavioral Sciences (3rd edition). Routledge. Checked .
- Field (2024), Discovering Statistics Using IBM SPSS Statistics, sixth edition. SAGE. Checked .
- Higgins, Thompson, Deeks and Altman (2003), Measuring inconsistency in meta-analyses. BMJ, 327, 557–560. Checked .
- Jacobson and Truax (1991), Clinical significance: a statistical approach to defining meaningful change in psychotherapy research. Journal of Consulting and Clinical Psychology, 59(1), 12–19. Checked .
- Laupacis, Sackett and Roberts (1988), An assessment of clinically useful measures of the consequences of treatment. New England Journal of Medicine, 318(26), 1728–1733. Checked .
- Maxwell and Cole (2007), Bias in cross-sectional analyses of longitudinal mediation. Psychological Methods, 12(1), 23–44. Checked .
- Sterne, Sutton, Ioannidis, Terrin, Jones, Lau and colleagues (2011), Recommendations for examining and interpreting funnel plot asymmetry in meta-analyses of randomised controlled trials. BMJ, 343, d4002. Checked .
- Thompson (1995), Stepwise regression and stepwise discriminant analysis need not apply here: a guidelines editorial. Educational and Psychological Measurement, 55(4), 525–534. Checked .