P-Values and Confidence Intervals

View larger

What Are P-Values and Confidence Intervals?

P-values and confidence intervals (CIs) are the two primary tools for statistical inference in orthopaedic research. They address different but complementary questions:

• P-value: Tests whether observed data are compatible with the null hypothesis (no effect/difference)
• Confidence interval: Estimates the range of plausible values for the true effect

Why This Matters: Misinterpretation of p-values is pervasive in medical literature. Understanding these concepts prevents overconfident claims from underpowered studies and helps distinguish statistically significant but clinically trivial findings from truly meaningful results.

Historical Context: Ronald Fisher introduced p-values in the 1920s as a continuous measure of evidence against the null hypothesis. The 0.05 threshold became convention, not scientific law. Jerzy Neyman and Egon Pearson later developed confidence intervals in the 1930s as a complementary approach to estimation.

Current Emphasis: Modern statistical guidelines (ASA 2016, CONSORT, STROBE) emphasize reporting effect sizes and confidence intervals over dichotomous p-value thresholds. Journals increasingly require CIs alongside or instead of p-values.

Clinical Relevance in Orthopaedics:

• Distinguishing statistical significance from clinical importance (MCID)
• Interpreting RCT results for treatment decisions
• Evaluating diagnostic test accuracy studies
• Assessing prognostic factor analyses
• Critical appraisal for exam vivas and clinical practice

Understanding p-values and CIs is essential for evidence-based orthopaedic practice and exam success.

Fundamental Definitions

P-Value (Probability Value):

• Definition: The probability of observing data as extreme as, or more extreme than, what was observed, assuming the null hypothesis is true
• Formula: p = P(Data | H₀ is true)
• NOT: p ≠ P(H₀ is true | Data) - this is the most common error
• Range: 0 to 1 (often expressed as 0 to 100%)

Confidence Interval (CI):

• Definition: A range of values that, if the study were repeated many times, would contain the true population parameter in 95% of studies (for 95% CI)
• Components: Point estimate (observed effect) ± margin of error
• Interpretation: Provides effect size, direction, precision, and significance simultaneously
• NOT: There is NOT a 95% probability the true value is in this specific CI (frequentist interpretation forbids this)

Null Hypothesis (H₀):

• Definition: Statement of no effect, no difference, or no association
• Examples:
  • Mean difference = 0
  • Risk ratio = 1
  • Correlation coefficient = 0

Alternative Hypothesis (H₁):

• Definition: Statement that there IS an effect, difference, or association
• Can be: Two-sided (any difference) or one-sided (specific direction)

Relationship Between P-Values and Confidence Intervals

Key Connection: P-values and confidence intervals are mathematically related:

• For 95% CI: If the CI excludes the null value (0 for differences, 1 for ratios), then p less than 0.05
• For 99% CI: If the CI excludes the null value, then p less than 0.01
• For 90% CI: If the CI excludes the null value, then p less than 0.10

Why This Matters: You can determine statistical significance directly from the confidence interval without needing the p-value. This is why modern guidelines emphasize CIs over p-values - they provide MORE information (effect size, precision, AND significance).

What is a P-Value?

Definition: The probability of observing data as extreme as, or more extreme than, the observed data, assuming the null hypothesis is true.

Null Hypothesis (H₀): There is no difference between groups or no effect.

Formula: p = P(Data | H₀ is true)

Interpreting P-Values

Key Point: p = 0.051 is NOT fundamentally different from p = 0.049. The 0.05 threshold is arbitrary convention, not natural boundary.

Common P-Value Misconceptions

What is a Confidence Interval?

Definition: A range of values that likely contains the true population parameter.

95% CI Interpretation: If we repeated the study many times, 95% of the confidence intervals calculated would contain the true effect.

NOT: There is a 95% probability the true value is in this CI (frequentist interpretation).

Relationship Between CI and P-Value

CI Components

Examiners frequently probe whether candidates can distinguish closely related statistical terms. The table below contrasts the concepts most often confused in vivas and the literature.

The p-value and confidence interval are frequentist tools: they describe long-run behaviour and cannot give the direct probability statement clinicians actually want ("how likely is it that this treatment works?"). The Bayesian framework answers that question directly, at the cost of having to specify a prior.

By Bayes' theorem, the posterior is proportional to the likelihood times the prior: the observed data update a prior belief into a posterior. A credible interval is the Bayesian analogue of the confidence interval but permits the natural statement the frequentist CI forbids - a 95% probability the true value lies inside it. A Bayes factor quantifies how much the data shift the odds between two hypotheses and, unlike a p-value, can provide positive evidence in favour of the null. Bayesian methods are increasingly used in adaptive trials and evidence synthesis, but the dependence on the prior remains the central criticism.

Statistical vs Clinical Significance

Four Possible Scenarios:

Key Principle: Always check if effect size (point estimate) and CI bounds exceed MCID, not just if p less than 0.05.

Worked Example: THA Study

Study: Compares cemented vs uncemented THA on WOMAC score at 1 year.

Results:

• Mean difference = 8 points (cemented better)
• 95% CI: 1 to 15 points
• p = 0.02
• MCID for WOMAC = 10 points

Interpretation:

1. Statistically Significant: p = 0.02 less than 0.05, CI excludes 0 → Yes
2. Point Estimate: 8 points less than MCID of 10 → Not clinically meaningful
3. CI Upper Bound: 15 points greater than MCID → Could be meaningful
4. CI Lower Bound: 1 point much less than MCID → Could be trivial

Conclusion: Result is statistically significant but clinically uncertain. The CI is wide and crosses the MCID threshold. Point estimate suggests effect may not be clinically important. Need larger study to narrow CI and determine if true effect exceeds 10 points.

Understanding this nuanced interpretation prevents overconfidence in borderline results.

For binary (yes/no) outcomes the same result can be expressed in absolute or relative terms, and the choice strongly affects how impressive it looks. Confidence intervals and significance apply to all of these measures, but clinical decisions depend on the absolute ones.

View larger

The key trap is the relative-versus-absolute distinction: a treatment that cuts risk from 2% to 1% has an impressive relative risk reduction of 50% but an ARR of only 1% and an NNT of 100. The RRR is identical whether the baseline risk is 2% or 50%, so a relative measure detached from baseline risk can badly mislead. The number needed to harm (NNH) is the analogous 1 divided by the absolute risk increase for adverse effects, and the NNT should always be quoted with its confidence interval.

Reporting Standards Across the Global Literature

Statistical reporting expectations are remarkably consistent worldwide because they are driven by international journal and methodology guidelines rather than national bodies. The candidate should know which guideline governs which study type.

Registries and Effect Estimation

Large national arthroplasty registries (NJR for England, Wales, Northern Ireland and the Isle of Man; AOANJRR Australia; SHAR Sweden; the Norwegian and New Zealand registries; AJRR USA) illustrate the practical primacy of confidence intervals over p-values. Because registry sample sizes run into hundreds of thousands, almost any difference reaches statistical significance. Registries therefore report hazard ratios and revision rates with narrow confidence intervals and interpret them against clinically meaningful thresholds, not against p = 0.05. This is the real-world embodiment of the large-sample problem: with very large n, statistical significance is near-guaranteed and effect size plus CI width become the only meaningful discriminators.

High- vs Limited-Resource Practice Variation

• Well-resourced settings: Routine access to statistical software and biostatistician support; journals enforce CI reporting and pre-registration, reducing selective p-value reporting.
• Limited-resource settings: Smaller single-centre studies predominate, raising the risk of underpowered analyses and Type II errors; wide confidence intervals are common and should be interpreted as inconclusive rather than negative.
• Universal principle: The interpretation of p-values and CIs does not change by country; what varies is study size, access to methodological support, and exposure to publication and reporting bias.

Statistical inference is an area of genuine, ongoing debate. Candidates who can articulate the controversy (not just recite rules) demonstrate consultant-level understanding.