Reliability and Validity (Kappa, ICC, Bland-Altman)

Reliability (reproducibility, precision) asks whether repeated measurements agree with each other; validity (accuracy) asks whether the measurement reflects the true value. The classic dartboard analogy makes the relationship clear: tight grouping = reliable, hitting the centre = valid. A measurement can be reliable but not valid - tightly clustered but systematically off-target (a bias) - so reliability is necessary but not sufficient for validity. Reliability comes in forms - intra-observer (same rater repeated), inter-observer (different raters) and test-retest (over time) - all of which matter when validating a classification system or an outcome measure.

• Categorical data (e.g. a fracture classification): Cohen's KAPPA for two raters (corrects for chance agreement), Fleiss' kappa for more than two raters, and weighted kappa for ordinal categories. Interpret with Landis-Koch: below 0.20 slight, 0.21-0.40 fair, 0.41-0.60 moderate, 0.61-0.80 substantial, 0.81-1.00 almost perfect.
• Continuous data (e.g. angle/length measurements): the INTRACLASS CORRELATION COEFFICIENT (ICC) (0-1; broadly above 0.75 good, above 0.9 excellent) for reliability.
• Method comparison: the BLAND-ALTMAN plot - plot the difference between two methods against their mean, revealing systematic bias (the mean difference) and the limits of agreement (mean difference plus or minus 1.96 standard deviations).
• Do NOT use correlation (Pearson's r) to claim agreement - two methods can correlate strongly yet disagree systematically; use ICC or Bland-Altman.

Validity has several forms: face (does it look reasonable), content (does it cover the construct), construct (does it behave as theory predicts), and criterion validity - concurrent (agrees with a gold standard now) and predictive (predicts a future outcome); sensitivity/specificity against a gold standard are criterion validity (see our Diagnostic Test Statistics topic). In orthopaedics, classification systems (e.g. fracture classifications) are tested for inter- and intra-observer reliability using kappa/ ICC; the evidence shows their stand-alone reliability is often only fair-to-moderate, that increasing granularity (more categories) lowers kappa/ICC, and that rater calibration improves agreement - which is why classifications are best used as a shared language and research scaffold alongside specific radiographic thresholds and patient factors, rather than as the sole basis for decisions.