Research methods, revisited

Descriptive statistics summarise a sample without generalising beyond it. The three things to be able to choose between are a measure of central tendency, a measure of spread, and the shape of the distribution — and the choice depends on the level of measurement and the skew.

Inferential statistics work because sample means behave predictably even when the underlying data does not. The central limit theorem is the reason a t-test is valid on non-normal data given a reasonable sample size, and the standard error is the quantity that makes it work.

A p-value is the probability of obtaining data at least as extreme as the observed data, if the null hypothesis were true. It is not the probability that the null hypothesis is true, and it is not a measure of effect size. Almost every exam question on this topic tests that distinction.

Test choice follows from four questions: what is the level of measurement, how many groups or conditions, are they independent or related, and are parametric assumptions met. Answer those in order and the test chooses itself.

Effect size says how big a difference is; the p-value only says how surprising it would be under the null. With a large enough sample, a clinically meaningless difference becomes statistically significant, which is why effect sizes and confidence intervals are reported alongside p.

Reliability is consistency; validity is whether the measure captures what it claims to. A measure can be highly reliable and completely invalid — consistently measuring the wrong thing — but it cannot be valid without being reliable.

Design determines what a study can legitimately claim. Randomisation supports causal inference; correlation does not. Most critical appraisal comes down to asking what else could explain this result, and whether the design rules it out.