Grading the evidence behind a claim: from anecdote to controlled comparison

What it is

Evidence-based medicine formalised the idea that not all evidence deserves the same weight. The Oxford CEBM levels, first produced in 1998 and revised since (the page links the 2009 and 2011 tables), rank sources by question type. For treatment benefit, the 2011 table runs from systematic reviews of randomised trials, through individual randomised trials, non-randomised cohort studies and case series, down to mechanism-based reasoning; the 2009 table put expert opinion without explicit critical appraisal at the bottom. The CEBM page itself notes that hierarchies of evidence have been used inflexibly and criticised, and asks that the table be read with its introductory document. Two further ideas complete it. Risk of bias: the Cochrane Handbook's RoB 2 tool asks signalling questions in five domains (randomisation process, deviations from the intended intervention, missing outcome data, outcome measurement, selection of the reported result); a result is at low risk only if every domain is, and at high risk if any domain is or if concerns in several domains together lower confidence. Conflicts of interest: the ICMJE recommendations make authors responsible for disclosing all relationships and activities that might bias or be seen to bias their work.

Why it matters

Technical writing mixes evidence of different strength: a vendor benchmark, one team's blog post, a self-selected survey, a controlled comparison. Presenting them with equal confidence misleads readers and the agents reusing the text. A grade next to each claim shifts the argument from "is it true" to "what would raise the grade".

How to apply

• Classify the design first: opinion or single anecdote; several cases without comparison; comparison with a non-randomised control; randomised comparison; several independent comparisons summarised systematically.
• Then downgrade: small sample (wide interval), outcome measured by the people who chose the intervention, missing data, outcome chosen after the results were seen, the author's stake in the result. A randomised comparison with a selectively reported outcome can rank below a careful observational study.
• Check whether the population and setting resemble the one the claim is applied to; a result about large organisations may not transfer to a three-person team.
• Write the grade next to the claim in plain words: "one team's report", "vendor measurement, method not published", "replicated controlled comparison".
• Prefer the primary source; a secondary summary inherits the primary's grade at best and adds transcription risk.

Pitfalls

Treating the hierarchy as a ladder regardless of question type: for mechanisms or rare harms, observational data and case reports can be the best available. Counting citations as evidence. Confusing "peer reviewed" with "high grade". Dismissing low-grade evidence instead of labelling it: an anecdote is a good reason for a better study.