Hypothesis testing and p-values

What a p-value is and is not

Formally: p is the probability of the observed result, or one more extreme, assuming the null hypothesis is true. Everything follows from the fact that the null is assumed, not tested.

• p is NOT the probability the null hypothesis is true.
• p is NOT the probability the finding will replicate.
• p is NOT a measure of how large or important the effect is.
• A non-significant result is not evidence of no effect — it is a failure to detect one.

Type I and Type II errors

A Type I error is rejecting a true null: a false positive, with probability α, conventionally .05. A Type II error is failing to reject a false null: a false negative, with probability β.

The two trade off. Lowering α to reduce false positives increases β unless you compensate with a larger sample.

Power

Power is 1 − β: the probability of detecting an effect that genuinely exists. It rises with sample size, with effect size, and with α; it falls with measurement error. Convention is to aim for .80.

An underpowered study is not merely inconclusive. Among the significant results it does produce, effect sizes are systematically inflated, because only large observed effects clear the threshold.

• Power = 1 − β, conventionally .80.
• Determined by sample size, effect size, α and measurement precision.
• Power should be calculated before data collection, not after.

Multiple comparisons

Running many tests at α = .05 inflates the family-wise error rate: with 20 independent tests, the chance of at least one false positive is about 64%. Bonferroni correction divides α by the number of comparisons — simple, and conservative to the point of costing power.

Where marks get lost

• Interpreting p = .06 as "a trend towards significance". It is a non-significant result.
• Reporting p without an effect size and confidence interval.
• Treating p = .049 and p = .051 as categorically different findings.
• Calculating power after the fact from the observed effect, which is circular.