2Research methods
Reliability and validity
Reliability is consistency; validity is whether a measure captures what it claims to. A measure can be reliable and invalid, consistently measuring the wrong thing, but it cannot be valid without being reliable.
Practice
19 questions
Level
Core
Reading time
3 min
Types of reliability
- Internal consistency: do the items measure the same construct? Reports of an acceptable Cronbach’s alpha range from .70 to .95. Tavakol and Dennick (2011) note a recommended maximum of .90, since a very high alpha can mean some items repeat each other.
- Test-retest: stability over time, assuming the construct itself is stable.
- Inter-rater: agreement between raters. Cohen’s kappa corrects for chance agreement; raw percentage agreement does not (McHugh, 2012).
- Parallel forms: agreement between equivalent versions.
Worked example:Two assessors, 50 interviews
- Two assessors each decide whether 50 recorded interviews meet the criteria for a diagnosis. Both say yes for 14 and no for 24, and they disagree on 12. Each says yes 20 times.
- Observed agreement:(14 + 24) ÷ 50 = .76
- Agreement expected by chance, from how often each says yes (.40) and no (.60):.40 × .40 + .60 × .60 = .52
- Kappa takes chance agreement out:κ = (.76 − .52) ÷ (1 − .52) = .50
- Agreement of 76% looks high, but more than two-thirds of it would happen by chance.
Source: McHugh (2012), checked .
Types of validity
- Face: does it look appropriate? The weakest form.
- Content: does it cover the whole construct?
- Criterion: does it agree with an external standard? Concurrent (now) or predictive (later).
- Construct: does it behave as the theory says it should? Convergent validity with related measures, discriminant validity with unrelated ones (Campbell and Fiske, 1959).
- Ecological: do findings generalise to real settings?
Idiographic and nomothetic measurement
A nomothetic measure is the same for everyone and scored against norms, so people can be compared with each other. An idiographic measure is chosen or built for one person, to be as relevant as possible to them (Haynes, Mumma and Pinson, 2009).
The case for idiographic measurement is that people differ in how a problem shows itself and in what drives it. A standard questionnaire can therefore pick up variance that has nothing to do with the problem as it affects this person (Haynes and colleagues, 2009). The cost is comparison: a target built for one client has no norms to set them against.
Idiographic research studies one person closely over time, as a single-case experimental design does, instead of averaging across a group (Barlow and Nock, 2009). Clinicians often use both kinds side by side: a standard questionnaire and a measure built round the client’s own goals.
- Nomothetic: standardised and normed. How does this person compare with others?
- Idiographic: chosen or built for the individual. How is this person changing on what matters to them?
- Idiographic examples: personalised outcome measures, goal-based measures, and a target rated repeatedly in a single-case design.
Threats to internal validity
Internal validity is whether the study supports a causal claim. Common threats are history, maturation, regression to the mean, selection bias, attrition and testing effects.
Regression to the mean matters clinically: people typically present at their worst, so improvement follows even without treatment (Barnett, van der Pols and Dobson, 2005). That is a large part of why control groups exist.
Common mistakes
- Assuming a reliable measure is therefore valid.
- Reporting percentage agreement rather than kappa for inter-rater reliability.
- Attributing improvement in an uncontrolled study to the intervention when regression to the mean would predict it anyway.
- Taking idiographic to mean qualitative: a weekly rating on a client’s own target is an idiographic number.
- Comparing a score on a client’s personal target with population norms, which it does not have.
Sources
- Barlow and Nock (2009), Why can’t we be more idiographic in our research? Perspectives on Psychological Science, 4(1), 19–21. Checked .
- Barnett, van der Pols and Dobson (2005), Regression to the mean: what it is and how to deal with it. International Journal of Epidemiology, 34(1), 215–220. Checked .
- Campbell and Fiske (1959), Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin, 56(2), 81–105. Checked .
- Field (2024), Discovering Statistics Using IBM SPSS Statistics, sixth edition. SAGE. Checked .
- Haynes, Mumma and Pinson (2009), Idiographic assessment: conceptual and psychometric foundations of individualized behavioral assessment. Clinical Psychology Review, 29(2), 179–191. Checked .
- McHugh (2012), Interrater reliability: the kappa statistic. Biochemia Medica, 22(3), 276–282. Checked .
- Tavakol and Dennick (2011), Making sense of Cronbach’s alpha. International Journal of Medical Education, 2, 53–55. Checked .