2Research methods
Research design and critical appraisal
Design determines what a study can legitimately claim. Most critical appraisal comes down to asking what else could explain the result, and whether the design rules it out.
Practice
14 questions
Level
Advanced
Reading time
3 min
The evidence hierarchy and its limits
The usual pyramid puts systematic reviews and meta-analyses above randomised controlled trials, then cohort and case-control studies, then cross-sectional studies, then case series. It ranks designs for questions about cause and effect. Petticrew and Roberts (2003) argue for matching each question to the designs that suit it instead: what an experience is like is a question an RCT cannot answer.
Even for questions of effect, the pyramid is a starting point. GRADE rates the certainty of the evidence for each outcome, from high to very low: trials start high and non-randomised studies usually low, and any body of evidence can be rated down for risk of bias, inconsistency, indirectness, imprecision or publication bias (Cochrane Handbook, chapter 14).
What to look for in a trial
- Randomisation: balances known and unknown confounders alike, on average, which nothing else does (Cochrane Handbook, chapter 8).
- Allocation concealment: prevents the person recruiting from influencing which arm someone enters.
- Blinding: say who was blinded (participants, care providers, outcome assessors, data analysts), as CONSORT 2025 asks (item 20a); “single” or “double” does not say who. Participants and therapists often cannot be blinded in therapy trials, but outcome assessors often can.
- Intention-to-treat analysis: analyse participants in the arm they were randomised to, whatever they actually received. A per-protocol analysis can reintroduce selection bias.
- Control condition: waiting list, treatment as usual or an active comparator. Waiting-list controls tend to overstate effects (Furukawa and colleagues, 2014; Michopoulos and colleagues, 2021).
Worked example:Intention to treat against per protocol
- A trial randomises 100 clients to therapy and 100 to usual care, and follows everyone up. 70 complete the therapy and 42 of them recover; 6 of the 30 who stop early recover too. In usual care, 30 recover.
- Intention to treat compares everyone in the arm they were randomised to, therapy and then usual care:(42 + 6) ÷ 100 = 48%30 ÷ 100 = 30%
- Per protocol keeps only those who completed the therapy, against the same 30% in usual care:42 ÷ 70 = 60%
- Here those who stopped early did worse, so leaving them out makes the therapy look 30 points better instead of 18. Who completes is not randomised.
Source: Cochrane Handbook, chapter 8, checked .
Qualitative approaches
Clinical psychology uses qualitative methods routinely. Thematic analysis identifies patterns of meaning across accounts (Braun and Clarke, 2006). Interpretative phenomenological analysis examines how individuals make sense of experience, with small homogeneous samples (Smith, Flowers and Larkin, 2022). Grounded theory builds theory from data, sampling until saturation (Charmaz and Thornberg, 2021).
Quality is judged by qualitative standards, such as transparency, owning one’s perspective and credibility checks, rather than by sample size or generalisability (Yardley, 2000; Elliott, Fischer and Rennie, 1999).
Common mistakes
- Inferring causation from a cross-sectional correlation.
- Treating a per-protocol result as equivalent to intention-to-treat.
- Dismissing qualitative work for lacking generalisability, which is not what it is for.
- Ignoring that a waiting-list control inflates apparent effect size relative to an active comparator.
Sources
- Braun and Clarke (2006), Using thematic analysis in psychology. Qualitative Research in Psychology, 3(2), 77–101. Checked .
- Charmaz and Thornberg (2021), The pursuit of quality in grounded theory. Qualitative Research in Psychology, 18(3), 305–327. Checked .
- Cochrane Handbook for Systematic Reviews of Interventions, version 6.5 (2024), chapter 8: Assessing risk of bias in a randomized trial. Higgins, Savović, Page, Elbers and Sterne; last updated October 2019. Checked .
- Cochrane Handbook for Systematic Reviews of Interventions, version 6.5 (2024), chapter 14: Completing ‘Summary of findings’ tables and grading the certainty of the evidence. Schünemann, Higgins, Vist, Glasziou, Akl, Skoetz and Guyatt; last updated August 2023. Checked .
- Elliott, Fischer and Rennie (1999), Evolving guidelines for publication of qualitative research studies in psychology and related fields. British Journal of Clinical Psychology, 38(3), 215–229. Checked .
- Furukawa, Noma, Caldwell, Honyashiki, Shinohara, Imai and colleagues (2014), Waiting list may be a nocebo condition in psychotherapy trials: a contribution from network meta-analysis. Acta Psychiatrica Scandinavica, 130(3), 181–192. Checked .
- Hopewell, Chan, Collins, Hróbjartsson, Moher, Schulz and colleagues (2025), CONSORT 2025 statement: updated guideline for reporting randomised trials. BMJ, 389, e081123. Checked .
- Michopoulos, Furukawa, Noma, Kishimoto, Onishi, Ostinelli and colleagues (2021), Different control conditions can produce different effect estimates in psychotherapy trials for depression. Journal of Clinical Epidemiology, 132, 59–70. Checked .
- Petticrew and Roberts (2003), Evidence, hierarchies, and typologies: horses for courses. Journal of Epidemiology and Community Health, 57(7), 527–529. Checked .
- Smith, Flowers and Larkin (2022), Interpretative Phenomenological Analysis: Theory, Method and Research, second edition. SAGE. Checked .
- Yardley (2000), Dilemmas in qualitative health research. Psychology and Health, 15(2), 215–228. Checked .