Kappa, ICC and Bland-Altman
- RELIABILITY is reproducibility or PRECISION - how consistently a measurement gives the same result; VALIDITY is ACCURACY - whether it measures what it is supposed to. They are independent: a measurement can be RELIABLE WITHOUT being VALID (consistently wrong, a systematic bias), so reliability is necessary but NOT sufficient for validity (the dartboard analogy - tight grouping vs hitting the bullseye).
- Reliability has forms: INTRA-OBSERVER (the same rater repeating a measurement), INTER-OBSERVER (different raters), and TEST-RETEST (the same instrument over time); orthopaedic classification systems and outcome measures must be tested for these.
- For CATEGORICAL data (e.g. a fracture classification), agreement is measured by COHEN'S KAPPA between two raters - which corrects for the agreement expected by CHANCE - with FLEISS' kappa for more than two raters and WEIGHTED kappa for ORDINAL categories; the conventional LANDIS-KOCH interpretation is: below 0.20 slight, 0.21-0.40 fair, 0.41-0.60 moderate, 0.61-0.80 substantial, and 0.81-1.00 almost perfect agreement.
- For CONTINUOUS data (e.g. an angle or a length measurement), reliability is measured by the INTRACLASS CORRELATION COEFFICIENT (ICC), which runs from 0 to 1 (broadly above 0.75 is good and above 0.9 excellent), and method comparison is best shown with a BLAND-ALTMAN PLOT, which plots the DIFFERENCE between two methods against their MEAN to reveal systematic BIAS and the LIMITS OF AGREEMENT (mean difference +/- 1.96 standard deviations).
- A key trap is that CORRELATION (e.g. Pearson's r) is NOT agreement: two methods can correlate almost perfectly yet disagree systematically (one always reads higher), so r should NOT be used to claim two methods/raters agree - use the ICC or a Bland-Altman plot instead.
- Applied to orthopaedics: classification systems are valuable as a SHARED LANGUAGE but have LIMITED stand-alone reliability - increasing the number of categories/subcategories consistently REDUCES kappa/ICC, while a brief rater CALIBRATION session improves agreement; report agreement statistics with confidence intervals and choose the statistic that matches the data type.
- “Reliability = precision (reproducible); Validity = accuracy (true). Reliable can be invalid (systematic bias) - dartboard analogy.
- “Categorical agreement = kappa (Cohen's 2 raters, Fleiss' for more than 2, weighted for ordinal); Landis-Koch: 0.21-0.40 fair, 0.41-0.60 moderate, 0.61-0.80 substantial, 0.81-1.0 almost perfect.
- “Continuous: ICC for reliability; Bland-Altman (difference vs mean; bias + limits of agreement) for method comparison. Correlation is NOT agreement. More categories → lower kappa.
Reliability vs Validity
Two separate questions. Reliability (reproducibility, precision) asks whether repeated measurements agree with each other. Validity (accuracy) asks whether the measurement reflects the true value. The classic dartboard analogy makes the relationship clear: tight grouping is reliable, hitting the centre is valid.
Reliable is not the same as valid. A measurement can be reliable but not valid, tightly clustered but systematically off-target, which is a bias. Reliability is therefore necessary but not sufficient for validity.
Forms of reliability. Reliability comes in forms: intra-observer (the same rater repeating a measurement), inter-observer (different raters) and test-retest (the same instrument over time). Each matters when validating a classification system or an outcome measure.
Agreement Statistics
Categorical data: kappa. For categorical data such as a fracture classification, agreement between two raters is measured with Cohen's kappa, which corrects for the agreement expected by chance. Fleiss' kappa extends it to more than two raters, and weighted kappa is used for ordinal categories, penalising bigger disagreements more. The conventional interpretation is that of Landis and Koch:
- Below 0.20: slight
- 0.21-0.40: fair
- 0.41-0.60: moderate
- 0.61-0.80: substantial
- 0.81-1.00: almost perfect
Continuous data: the ICC. For continuous data such as an angle or a length measurement, reliability is measured with the intraclass correlation coefficient (ICC). It runs from 0 to 1; broadly, above 0.75 is good and above 0.9 excellent.
Comparing two methods: Bland-Altman. The Bland-Altman plot puts the difference between two methods against their mean. The mean difference is the systematic bias, and the limits of agreement are the mean difference plus or minus 1.96 standard deviations.
Correlation is not agreement. Two methods can correlate strongly yet disagree systematically, one always reading higher than the other. Do not use correlation (Pearson's r) to claim that two methods or raters agree; use the ICC or a Bland-Altman plot.
- Statistic
- Cohen's kappa
- Notes
- Corrects for chance; Landis-Koch grades
- Statistic
- Fleiss' kappa
- Notes
- Extension of kappa
- Statistic
- Weighted kappa
- Notes
- Penalises bigger disagreements more
- Statistic
- Intraclass correlation (ICC)
- Notes
- 0-1; above 0.75 good, above 0.9 excellent
- Statistic
- Bland-Altman plot
- Notes
- Bias + limits of agreement (NOT correlation)
Validity Types & Classification Reliability
Validity has several forms. Each asks a different question of the measure:
- Face validity: does it look reasonable?
- Content validity: does it cover the construct?
- Construct validity: does it behave as theory predicts?
- Criterion validity, in two kinds: concurrent (agrees with a gold standard now) and predictive (predicts a future outcome)
Sensitivity and specificity measured against a gold standard are criterion validity, covered in our Diagnostic Test Statistics topic.
How reliable are classifications? Orthopaedic classification systems, fracture classifications among them, are tested for inter- and intra-observer reliability using kappa or the ICC. The evidence shows their stand-alone reliability is often only fair-to-moderate. Increasing granularity, with more categories and subcategories, lowers kappa and ICC, while a brief rater calibration session improves agreement.
What they are for. Classifications are best used as a shared language and research scaffold, alongside specific radiographic thresholds and patient factors, rather than as the sole basis for decisions. Report their agreement statistics with confidence intervals.
Patient-reported outcome measures. Everything above concerns whether observers agree about an image. A patient-reported outcome measure is judged on a wider set of properties, and these are standard psychometric teaching rather than claims resting on any one paper:
- Internal consistency: do the items in a scale measure the same underlying construct? It is reported as Cronbach's alpha, conventionally acceptable around 0.70 to 0.95. Alpha above about 0.95 is not a triumph but a warning: the items are probably redundant and the questionnaire could be shorter.
- Test-retest reliability: the same patient, unchanged, measured twice; reported as an ICC.
- Responsiveness: can the instrument detect change when change has genuinely occurred? A score can be perfectly reliable and completely unresponsive, which makes it useless for measuring treatment effect.
- Floor and ceiling effects: if a large proportion of patients score the minimum or maximum, the instrument cannot detect deterioration or improvement beyond that point. This is why several classic hip and knee scores are poor at discriminating among well-functioning patients.
- Minimal clinically important difference (MCID): the smallest change a patient notices, and the bridge from statistics to meaning. A difference can be highly statistically significant and smaller than the MCID, which is a finding of no clinical importance however small the p value.
The organising idea. Reliability asks whether the measurement is reproducible, validity whether it measures what it claims, and responsiveness whether it can detect change. An instrument can pass any one and fail the others, and a classification that raters apply consistently may still predict nothing: agreement is not correctness.
The Kappa Paradox: Why a High Agreement Can Give a Low Kappa
The prevalence paradox. When one category is very common (or very rare), the agreement expected by chance is high. Even with high observed agreement, kappa then comes out paradoxically low; it is "punished" for an unbalanced category mix.
The bias paradox. When the two raters use the categories at different rates (asymmetric marginals, that is rater bias), kappa can be paradoxically raised.
What to report. Always report the raw (observed) percent agreement alongside kappa, and consider the prevalence-and-bias-adjusted kappa (PABAK = 2 × observed agreement − 1) when the categories are unbalanced. The Landis-Koch bands are arbitrary, a convention and not an absolute, so interpret kappa with its confidence interval and in the context of prevalence and bias rather than as a hard grade.
ICC in Practice: Which Model, and Turning It Into SEM and MDC
There is no single ICC. You must state the model: one-way or two-way, random or fixed effects, single-measure or average-measure, and consistency or absolute agreement. These give different values for the same data, so a reported ICC is uninterpretable without its form.
From reliability to measurement error. The clinically useful quantities derived from the ICC are the standard error of measurement (SEM = SD × √(1 − ICC)), the typical error in the units of the measure, and the minimal detectable change (MDC; e.g. MDC95 = 1.96 × √2 × SEM). The MDC is the smallest change that exceeds measurement error, a change you can be confident is real and not noise.
Real versus meaningful. Pair the MDC with the MCID: an observed change should ideally exceed both the MDC, to be real, and the MCID, to be meaningful. The MCID and responsiveness of outcome scores are developed in our Outcome Measures (PROMs) topic.
Mnemonics & Memory Aids
PRECISE vs TRUE
Hook:Reliability = PRECISE; Validity = TRUE; you need both.
KIB
Hook:KIB: Kappa (categorical), ICC (continuous), Bland-Altman (method comparison).
Clinical Decision Scenarios
Practise clinical reasoning and management decisions out loud
“What is the difference between reliability and validity, and which statistics would you use to assess the reliability of a fracture classification and of an angle measurement?”
“Why do more detailed (granular) classification systems tend to be less reliable, and how can agreement be improved?”
Concepts
- Reliability = precision/reproducibility; validity = accuracy
- Reliable can be invalid (systematic bias) - dartboard analogy
- Reliability forms: intra-observer, inter-observer, test-retest
Categorical agreement
- Cohen's kappa (2 raters), Fleiss' kappa (more than 2), weighted kappa (ordinal)
- Corrects for chance agreement
- Landis-Koch: 0.21-0.40 fair, 0.41-0.60 moderate, 0.61-0.80 substantial, 0.81-1.0 almost perfect
Continuous agreement
- ICC for reliability (0-1; above 0.75 good, above 0.9 excellent)
- Bland-Altman for method comparison (difference vs mean; bias + limits of agreement)
- Correlation (Pearson r) is NOT agreement
Validity & classifications
- Validity: face, content, construct, criterion (concurrent/predictive)
- More categories → lower kappa; rater calibration improves agreement
- Classifications = shared language; report kappa/ICC with CIs
Evidence & Key Studies
Distal radius fracture classifications in real life: reliability and how they change treatment
- Interobserver agreement for distal radius fracture classifications was typically fair-to-moderate on radiographs, with only modest improvement on CT.
- Increasing granularity (more categories/subcategories) consistently REDUCED kappa/ICC, whereas a brief rater calibration session improved agreement.
- Classifications remain valuable as a shared language but have limited stand-alone reliability and prognostic power, best combined with instability thresholds and patient factors.
Zero echo time MRI vs CT in intra-articular distal radius fractures: inter/intraobserver agreement
- Inter- and intraobserver agreement were quantified with Cohen's and Fleiss' kappa and intraclass correlation coefficients.
- Classification agreement was 'good' (kappa about 0.68-0.78), with surgeons agreeing more than radiologists; continuous measures showed good ICC for angulation (about 0.76-0.86) but lower for inclination.
- Illustrates the practical use and interpretation of kappa (categorical) and ICC (continuous) for reliability.
The fair-to-moderate reliability of fracture classifications, the reduction of kappa/ICC with greater granularity and the benefit of rater calibration come from the cited Nguyen review, and the worked use of Cohen's/Fleiss' kappa and ICC (with interpretive values) from the cited Kaymakoglu study. The reliability-versus-validity distinction, the Landis-Koch kappa grades, the ICC and the Bland-Altman method- comparison approach are standard, well-established statistical teaching. (See also our Diagnostic Test Statistics, Measures of Effect and Study Design topics.)