Levels of Evidence

Levels of evidence rank study designs in a hierarchy so that a clinician can appraise the strength of the evidence behind a clinical decision. The higher the level, the greater the confidence the findings deserve.

The hierarchy. Systematic reviews and meta-analyses sit at the top, randomised controlled trials next, cohort and case-control studies in the middle, and case series, case reports and expert opinion at the bottom.

View larger

Why the label is only a start. Study design alone does not determine the evidence level. The quality of the randomisation, the blinding, the power, the attrition and the risk of bias must all be assessed, and a poorly conducted RCT may be downgraded to Level II or III. Different question types have different hierarchies, and the context of the decision determines which level of evidence is appropriate for it.

Why design matters. The designs rank as they do because of what each feature guards against. Randomisation controls for known and unknown confounders and eliminates selection bias, which is why it is critical for therapeutic questions. Blinding prevents performance and detection bias, a control group allows the effect of the intervention to be compared, and a prospective design avoids recall and selection bias.

The hierarchy is not one table but several, because the design that answers a question best depends on the question. For a therapeutic question the RCT is the gold standard, for a prognostic question the cohort study is best, and for a diagnostic question it is a cross-sectional study against a reference standard.

What GRADE is. GRADE (Grading of Recommendations Assessment, Development and Evaluation) is the most widely used system for assessing evidence quality and recommendation strength. It produces two separate outputs:

1. Quality of evidence: High, Moderate, Low or Very Low, which answers how confident we are in the effect estimate
2. Strength of recommendation: Strong or Weak, for or against, which answers whether we should do this

Why the separation matters. Quality and strength are separate judgements. A strong recommendation can arise from lower-quality evidence when the effect is large and the harms are minimal.

Rating the quality. Start with the study design, then apply the modifiers. An RCT starts as high quality and an observational study as low; the rating is then downgraded for risk of bias, inconsistency, indirectness, imprecision and publication bias, and upgraded for a large effect, a dose-response relationship, or residual confounding that favours the null.

Worked examples. An RCT with a high risk of bias (-1) and wide confidence intervals (-1) drops two levels, from High to Low quality evidence. A cohort study with a very large effect (+2) rises two levels, from Low to High quality evidence.

The evidence level is a starting label. The next step is to appraise the individual study with the validated instrument matched to its design, keeping two kinds of instrument apart: a risk-of-bias (appraisal) tool appraises the study itself, while a reporting checklist governs how it is reported.

A common exam task is to be handed a study description and asked to name the design, its level and its dominant bias. The direction of the study and the presence of a comparison group tell the designs apart quickly.

Because the systematic review and meta-analysis sit at the apex of the pyramid, the examiner expects you to interpret one: the forest plot, the heterogeneity statistic, the choice of model and the funnel plot.

The forest plot. Each row is one study. The box is its point estimate, sized in proportion to the study's weight, and the horizontal whiskers are its confidence interval. The diamond is the pooled estimate: its centre is the value and its width the confidence interval.

The line of no effect. The vertical line marks the null value, a relative risk or odds ratio of 1 or a mean difference of 0. A confidence interval that crosses it is non-significant, and a diamond touching it is non-significant too.

View larger

Heterogeneity. I-squared is the proportion of variability that is due to between-study differences rather than chance: roughly under 25% is low, about 50% moderate and over 75% high. High heterogeneity means the studies disagree, and a single pooled estimate may mislead.

Fixed versus random effects. A fixed-effects model assumes one common true effect and suits low heterogeneity. A random-effects model assumes a distribution of true effects, gives a wider confidence interval, and is the model to prefer when heterogeneity is high.

The funnel plot. Plotting each effect estimate against its precision screens for small-study effects and publication bias. A symmetric inverted funnel is reassuring; asymmetry suggests missing small negative studies.

Applying evidence to the patient. Level I evidence is ideal but not always applicable to the person in front of you. Before acting on a trial, ask:

• Does the patient match the RCT inclusion criteria?
• Were the exclusion criteria too strict?
• Do the patient's values align with the outcomes studied?

When lower evidence is acceptable. Level III-IV evidence may suffice when:

• The disease is rare and no RCT is feasible
• The clinical need is urgent and cannot wait for an RCT
• Ethical constraints prevent randomisation
• Observational data are consistent and show large effects

Reading guidelines critically. A guideline should cite the evidence level for each recommendation. A strong recommendation based on weak evidence is a prompt to question the rationale.

Communicating uncertainty. Be honest with patients: if the evidence is Level IV, explain the uncertainty. Shared decision-making is crucial when the evidence is weak.

Evidence-Grading Systems Used Worldwide

Different bodies grade evidence and recommendations differently. Knowing which system a guideline uses is essential to interpret its recommendations correctly across examination jurisdictions.

Side-by-Side Society Approaches

• AAOS (US) publishes CPGs and Appropriate Use Criteria, rating each recommendation Strong, Moderate, Limited, or Consensus based on the quality and consistency of the underlying evidence.
• BOA / BOAST (UK) standards are pragmatic, consensus-and-evidence based, and increasingly cite GRADE-rated NICE guidance where available.
• AO Foundation education and guidance are largely expert-consensus and principle-based, explicitly acknowledging limited Level I evidence for many fracture-fixation decisions.
• EFORT / European national societies generally follow GRADE methodology for formal guidelines while recognising registry data as key observational evidence.

Registry Evidence as High-Quality Observational Data

Large arthroplasty registries are the prime real-world example of observational evidence that can be upgraded under GRADE (very large sample, consistent effects):

• AOANJRR (Australia), NJR (England, Wales, NI and IoM), AJRR (US), Swedish (SHAR) and Norwegian registries provide implant-survival and revision-rate data that no RCT could feasibly generate.
• Registry signals (for example, early failure of specific implant designs) have changed practice faster than trials could, illustrating when robust observational data legitimately drives strong recommendations.
• Limitations remain: confounding by indication, surgeon and patient selection, and outcome restricted largely to revision rather than patient-reported outcomes.

High- vs Limited-Resource Practice Variation

• In high-resource settings, guideline-concordant care can rely on RCTs, meta-analyses, and registry feedback loops.
• In limited-resource settings, Level I evidence may be unavailable or non-applicable (different implants, case-mix, and follow-up capacity); well-conducted local cohorts and pragmatic adaptation of global guidelines are appropriate.
• The principle is constant worldwide: integrate the best available external evidence with clinical expertise and patient values rather than apply a single hierarchy mechanically.

Why This Matters in the Exam

• Levels of evidence and GRADE are core research-methodology topics across all major fellowship exams.
• Vivas commonly test the ability to assign a level to a described study, identify its dominant bias, and apply the RIIIP downgrade factors.
• Examiners expect candidates to translate an evidence level into a defensible treatment recommendation, acknowledging uncertainty when evidence is weak.

Is the design hierarchy too rigid? Concato and colleagues (NEJM 2000) showed that well-designed observational studies did not systematically overestimate effects versus RCTs. GRADE responded by allowing observational data to be upgraded, but how large an effect justifies upgrading remains a judgement call.

Does a Level I label mean high quality? Poolman and Bhandari (2006) found that Level I and Level II orthopaedic RCTs had similar, often low, reporting-quality scores. The label is a starting point; the individual methodological safeguards must still be appraised.

External validity. Strict inclusion criteria, expert centres and protocolised follow-up can make trial populations unrepresentative. Efficacy in a trial is not always effectiveness in routine practice, which is where pragmatic trials and registries add value.

Feasibility of the surgical RCT. Blinding surgeons is impossible, sham surgery is ethically fraught, learning curves bias early results, and equipoise is often lacking. This is why much high-quality orthopaedic evidence is necessarily observational.