P-Values and Confidence Intervals

P-values and confidence intervals (CIs) are the two primary tools for statistical inference in orthopaedic research, and they answer different but complementary questions. The p-value tests whether the observed data are compatible with the null hypothesis of no effect or difference. The confidence interval estimates the range of plausible values for the true effect.

View larger

Why it matters. Misinterpretation of p-values is pervasive in the medical literature. Understanding these tools prevents overconfident claims from underpowered studies, and separates statistically significant but clinically trivial findings from truly meaningful ones. In orthopaedics they matter for:

• Distinguishing statistical significance from clinical importance (MCID)
• Interpreting RCT results for treatment decisions
• Evaluating diagnostic test accuracy studies
• Assessing prognostic factor analyses
• Critical appraisal, in the exam viva and in clinical practice

History. Ronald Fisher introduced p-values in the 1920s as a continuous measure of evidence against the null hypothesis. The 0.05 cut-off attributed to him became a historical convention, not a scientific law or constant. Jerzy Neyman and Egon Pearson later developed confidence intervals in the 1930s as a complementary approach to estimation.

Current emphasis. Modern statistical guidelines (ASA 2016, CONSORT, STROBE) emphasise reporting effect sizes and confidence intervals over dichotomous p-value thresholds. Journals increasingly require CIs alongside or instead of p-values.

The null hypothesis (H₀) is a statement of no effect, no difference or no association. Its value depends on the measure being tested:

• Mean difference = 0
• Risk ratio = 1
• Correlation coefficient = 0

The alternative hypothesis (H₁) states that there is an effect, difference or association. It can be two-sided (any difference) or one-sided (a specific direction).

P-values and CIs are mathematically related. When a confidence interval excludes the null value (0 for differences, 1 for ratios), the result is statistically significant at the level that matches the interval:

• 95% CI excludes the null: p less than 0.05
• 99% CI excludes the null: p less than 0.01
• 90% CI excludes the null: p less than 0.10

So statistical significance can be read directly from the confidence interval without needing the p-value. This is why modern guidelines emphasise CIs over p-values.

Definition. The p-value is the probability of observing data as extreme as, or more extreme than, what was observed, assuming the null hypothesis is true. Formally, p = P(Data | H₀ is true). It ranges from 0 to 1, and is often expressed as 0 to 100%.

Saying it correctly. "Assuming there is no true difference between groups, the probability of observing a difference as large as or larger than what we observed, purely by chance, is [p-value]." For a comparison of two surgical techniques with p = 0.03, that becomes: if the two techniques are truly equivalent, there is a 3% probability of observing a difference this large or larger by random chance alone.

The threshold is arbitrary. p = 0.051 is not fundamentally different from p = 0.049, and the 0.05 line is not a natural boundary. Treating the two as categorically different is statistically indefensible, yet remains common in practice and in journal decisions.

What a p-value is not. Each of these readings is wrong:

1. The probability that the null is true. The p-value assumes the null is true and then calculates the probability of the data. It is P(Data | H₀), not P(H₀ | Data), and confusing the two is the most common error.
2. The probability of a Type I error. The Type I error rate is alpha, set before the study (usually 0.05). The p-value is calculated from the observed data; alpha is the pre-set threshold.
3. The size or clinical importance of the effect. The p-value reflects both effect size and sample size, so a large sample can yield p less than 0.05 for a trivial effect.
4. Proof of the null when p is greater than 0.05. Failure to reject the null does not prove it is true. The study may be underpowered (Type II error), or the null may be true.
5. A measure of power. The p-value does not indicate whether the study was adequately powered.

For the p = 0.03 example, then, "there is a 3% chance the null hypothesis is true", "there is a 3% chance this result is a false positive" and "the techniques differ by 3%" are all incorrect statements.

Definition. A 95% confidence interval is a range of values that, if the study were repeated many times, would contain the true population parameter in 95% of those studies. It is built as the point estimate (the observed effect) ± a margin of error. The frequentist interpretation forbids the tempting reading: there is not a 95% probability that the true value lies in this particular interval.

What it tells you. The point estimate (mean or median) is the best guess of the true effect, and the interval is the range of plausible values within which the true effect likely lies. A CI therefore shows effect size, direction, precision and statistical significance at once, whereas a p-value shows only significance, not magnitude.

Examiners frequently probe whether candidates can distinguish closely related statistical terms. The table contrasts the concepts most often confused in vivas and the literature.

The p-value and confidence interval are frequentist tools: they describe long-run behaviour and cannot give the direct probability statement clinicians actually want ("how likely is it that this treatment works?"). The Bayesian framework answers that question directly, at the cost of having to specify a prior.

Bayes' theorem. The posterior is proportional to the likelihood times the prior, so the observed data update a prior belief into a posterior. The credible interval is the Bayesian analogue of the confidence interval, and the Bayes factor quantifies how much the data shift the odds between two hypotheses.

In practice. Frequentist CIs and p-values dominate the orthopaedic literature. Bayesian methods are increasingly used in adaptive trials and evidence synthesis, and adoption is growing but uneven. The dependence on the prior remains the central criticism.

Two kinds of significance. Statistical significance (p less than 0.05, CI excludes the null) and clinical significance (the effect exceeds the MCID) are separate judgements, and a result can have one without the other. Always check whether the point estimate and the CI bounds exceed the MCID, not just whether p is less than 0.05. The four possible scenarios:

Worked Example: THA Study

The study. Cemented versus uncemented THA, compared on WOMAC score at 1 year. The results:

• Mean difference = 8 points (cemented better)
• 95% CI: 1 to 15 points
• p = 0.02
• MCID for WOMAC = 10 points

Reading it. Take each number in turn against the MCID:

1. Statistically significant. p = 0.02 is less than 0.05 and the CI excludes 0.
2. Point estimate. 8 points is less than the MCID of 10, so not clinically meaningful.
3. CI upper bound. 15 points is greater than the MCID, so the effect could be meaningful.
4. CI lower bound. 1 point is much less than the MCID, so the effect could be trivial.

The verdict. The result is statistically significant but clinically uncertain. The CI is wide and crosses the MCID threshold, and the point estimate suggests the effect may not be clinically important. A larger study is needed to narrow the CI and determine whether the true effect exceeds 10 points.

For binary (yes/no) outcomes the same result can be expressed in absolute or relative terms, and the choice strongly affects how impressive it looks. Confidence intervals and significance apply to all of these measures, but clinical decisions depend on the absolute ones.

Relative versus absolute. A treatment that cuts risk from 2% to 1% has an impressive relative risk reduction of 50%, but an ARR of only 1% and an NNT of 100. The RRR is identical whether the baseline risk is 2% or 50%, so a relative measure detached from baseline risk can badly mislead: the same RRR gives a very different NNT depending on baseline risk.

View larger

Harm and precision. The number needed to harm (NNH) is the analogous 1 divided by the absolute risk increase for adverse effects. The NNT should always be quoted with its confidence interval.

Reporting Standards Across the Global Literature

Statistical reporting expectations are remarkably consistent worldwide because they are driven by international journal and methodology guidelines rather than national bodies. The candidate should know which guideline governs which study type.

Registries and Effect Estimation

Large national arthroplasty registries (NJR for England, Wales, Northern Ireland and the Isle of Man; AOANJRR Australia; SHAR Sweden; the Norwegian and New Zealand registries; AJRR USA) illustrate the practical primacy of confidence intervals over p-values. Because registry sample sizes run into hundreds of thousands, almost any difference reaches statistical significance. Registries therefore report hazard ratios and revision rates with narrow confidence intervals and interpret them against clinically meaningful thresholds, not against p = 0.05. This is the real-world embodiment of the large-sample problem: with very large n, statistical significance is near-guaranteed and effect size plus CI width become the only meaningful discriminators.

High- vs Limited-Resource Practice Variation

• Well-resourced settings: Routine access to statistical software and biostatistician support; journals enforce CI reporting and pre-registration, reducing selective p-value reporting.
• Limited-resource settings: Smaller single-centre studies predominate, raising the risk of underpowered analyses and Type II errors; wide confidence intervals are common and should be interpreted as inconclusive rather than negative.
• Universal principle: The interpretation of p-values and CIs does not change by country; what varies is study size, access to methodological support, and exposure to publication and reporting bias.

Statistical inference is an area of genuine, ongoing debate. Candidates who can articulate the controversy, as well as recite the rules, demonstrate consultant-level understanding.

Redefine or abandon significance. Some authors propose lowering the threshold for new discoveries to p less than 0.005 to reduce false positives. Others argue for abandoning fixed thresholds entirely and reporting p-values as continuous evidence. No global consensus exists.

Which MCID. There is no universally agreed MCID for many orthopaedic PROMs. Anchor-based and distribution-based methods can give different values for the same instrument, which complicates the judgement of clinical importance.