2Research methods
Hypothesis testing and p-values
A p-value is the probability of obtaining data at least as extreme as the observed data, if the null hypothesis were true.
Practice
12 questions
Level
Core
Reading time
4 min
What a p-value means
A p-value is calculated by assuming that the null hypothesis, and every other assumption of the test, is true. It says how surprising the data would be under that model, not how likely the model is (Wasserstein and Lazar, 2016).
- p is not the probability the null hypothesis is true.
- p is not the probability the finding will replicate.
- p is not a measure of how large or important the effect is.
- A non-significant result means no effect was detected, which is not evidence that there is none (Altman and Bland, 1995).
Worked example:Two therapies, 40 people in each group
- After treatment, mean depression scores differ by 3 points. Each group’s standard deviation is 8.
- The standard error of the difference isSE = SD × √(1/n₁ + 1/n₂) = 8 × √(1/40 + 1/40) = 1.79
- The test statistic, on 78 degrees of freedom, gives a two-tailed p of about .10.t = 3 ÷ 1.79 = 1.68
- The 95% confidence interval for the difference, in points, uses the critical t, 1.99:3 ± 1.99 × 1.79 = −0.6 to 6.6
- Not significant at .05, and not evidence of no difference either. The interval holds a small difference the other way and one of more than 6 points, so the study settles nothing.
Sources: Altman and Bland (1995), checked ; Field (2024), checked .
Type I and Type II errors
A Type I error is rejecting a true null: a false positive, with probability α, conventionally .05. A Type II error is failing to reject a false null: a false negative, with probability β.
The two trade off: lowering α to reduce false positives increases β unless you compensate with a larger sample.
| The test | The null is true | The null is false |
|---|---|---|
| Rejects the null | Type I error, a false positive. Probability α | Correct, a true positive. Probability 1 − β, the power |
| Does not reject it | Correct, a true negative. Probability 1 − α | Type II error, a false negative. Probability β |
- Rejects the null
- The null is true: Type I error, a false positive. Probability α
- The null is false: Correct, a true positive. Probability 1 − β, the power
- Does not reject it
- The null is true: Correct, a true negative. Probability 1 − α
- The null is false: Type II error, a false negative. Probability β
Source: Field (2024), checked .
Power
Power is the probability of detecting a real effect. It rises with sample size, effect size and α, and falls with measurement error.
In an underpowered study, the significant results that do appear have systematically inflated effect sizes, because only large observed effects clear the threshold (Button and colleagues, 2013).
- Power = 1 − β. The generally accepted minimum is .80 (Lakens, 2013).
- Calculate power before data collection, not after.
Worked example:Planning the same trial
- Power belongs to planning. Had 3 points been the smallest difference worth finding, with a standard deviation of 8, the standardised difference would bed = 3 ÷ 8 = 0.375
- With 40 people in each group, a two-tailed test at α = .05 would find a difference that size less than 40% of the time.power = .38
- For power of .80, the usual approximation gives, for each group,n = 2 × (1.96 + 0.84)² ÷ 0.375² ≈ 112
- The exact calculation for a t-test, as software runs it, gives 113. Working power out afterwards from the 3 points observed adds nothing the p-value does not already say.
Sources: Field (2024), checked ; Hoenig and Heisey (2001), checked .
Multiple comparisons
Running many tests at α = .05 inflates the family-wise error rate, the chance of at least one false positive across them all. Bonferroni correction divides α by the number of comparisons. It is simple, and conservative enough to cost power.
Worked example:Twenty tests at α = .05
- A study runs 20 independent tests, and every null hypothesis is true. Each test avoids a false positive with probability .95, so all 20 avoid one with probability.95²⁰ = .36
- So the probability of one false positive or more is about 64%.1 − .95²⁰ = .64
- Bonferroni tests each comparison at α divided by the number of tests:.05 ÷ 20 = .0025
- At that threshold, the chance of any false positive across the 20 is back under .05.1 − .9975²⁰ = .049
Source: Field (2024), checked .
Common mistakes
- Interpreting p = .06 as “a trend towards significance”. It is a non-significant result.
- Reporting p without an effect size and confidence interval.
- Treating p = .049 and p = .051 as categorically different findings.
- Calculating power after the fact from the observed effect, which is circular.
Sources
- Altman and Bland (1995), Absence of evidence is not evidence of absence. BMJ, 311, 485. Checked .
- Button, Ioannidis, Mokrysz, Nosek, Flint, Robinson and Munafò (2013), Power failure: why small sample size undermines the reliability of neuroscience. Nature Reviews Neuroscience, 14(5), 365–376. Checked .
- Field (2024), Discovering Statistics Using IBM SPSS Statistics, sixth edition. SAGE. Checked .
- Gelman and Stern (2006), The difference between ‘significant’ and ‘not significant’ is not itself statistically significant. The American Statistician, 60(4), 328–331. Checked .
- Greenland, Senn, Rothman, Carlin, Poole, Goodman and Altman (2016), Statistical tests, P values, confidence intervals, and power: a guide to misinterpretations. European Journal of Epidemiology, 31(4), 337–350. Checked .
- Hoenig and Heisey (2001), The abuse of power: the pervasive fallacy of power calculations for data analysis. The American Statistician, 55(1), 19–24. Checked .
- Lakens (2013), Calculating and reporting effect sizes to facilitate cumulative science: a practical primer for t-tests and ANOVAs. Frontiers in Psychology, 4, 863. Checked .
- Wasserstein and Lazar (2016), The ASA’s statement on p-values: context, process, and purpose. The American Statistician, 70(2), 129–133. Checked .