Type I and Type II Errors

Every hypothesis test ends in a decision taken without knowing the truth, so it can go wrong in two ways. A Type I error finds a difference that does not exist; a Type II error misses one that does.

Type I error, the false positive. Rejecting the null hypothesis when the null is actually true: concluding, for example, that a new surgical technique is superior when it has no benefit. Its probability is alpha, set before the study and usually 0.05, which accepts a 5% risk. Acting on a false positive means:

• Adopting an ineffective or harmful treatment
• Wasting resources implementing the change
• Potential harm to patients
• False confidence in the intervention

Type II error, the false negative. Failing to reject the null hypothesis when the alternative is actually true: concluding that two treatments are equivalent when one is superior. Its probability is beta, conventionally 0.20, and power is 1 minus beta, so the conventional 80% power accepts a 20% risk. Acting on a false negative means:

• Discarding an effective treatment
• Delaying progress in patient care
• Wasted research effort in a failed trial
• Missing a therapeutic opportunity

A memory aid. The Boy Who Cried Wolf. A Type I error is crying wolf falsely, a false alarm; a Type II error is missing the real wolf.

Alpha is not the p-value. Alpha is set before the study; the p-value is calculated after, from the data. If p is less than alpha the null is rejected, and if the null was in fact true, that rejection is a Type I error.

View larger

Four outcomes, two of them errors. Crossing the decision against the truth gives four cells. A Type I error sits at rate alpha and a Type II error at rate beta; the correct decisions are the true negative, at 1 minus alpha, and the true positive, which is power.

We never know which column we are in. The true state of nature is unknown. What the investigator sets is alpha and beta, and they control the error rates.

A worked example. A trial tests whether a new implant reduces the revision rate compared with a standard implant. The null hypothesis (H₀) is no difference in revision rates; the alternative (H₁) is that the new implant has a lower revision rate. Each conclusion carries its own cost if the truth lies in the other column.

View larger

The trade-off. With a fixed sample the two errors cannot both be minimised. Lowering alpha, say to 0.01, reduces Type I error but increases the risk of Type II error unless the sample size increases, and reducing Type II error by raising power increases the sample size needed. A larger sample reduces both errors, though it mainly acts on Type II.

Choosing alpha. The threshold moves with the setting, and each choice is paid for in sample size or in the other error.

Controlling each error. Type I error is controlled by pre-specifying alpha and by appropriate correction for multiple testing. Type II error is controlled by an adequate sample size built on appropriate effect-size assumptions.

Which error is worse? It depends on context: the severity of the disease and the risks of the treatment. A Type I error is often considered worse, because it adopts a harmful treatment, but a Type II error can be worse if it misses a life-saving one. Type I is the more serious when the treatment is invasive, the decision irreversible or the intervention expensive; Type II is the more serious when a life-saving treatment is missed, or in a rare disease with few options.

Screening tests. A false positive brings unnecessary workup and anxiety; a false negative brings a missed diagnosis and delayed treatment. For serious diseases such as cancer, screening prioritises minimising Type II error, which means high sensitivity.

Power. Power is 1 minus beta, the chance of detecting a real effect. The conventional target is 80%, and every step above it costs sample size.

Underpowered orthopaedic trials. Many orthopaedic trials are underpowered, with power under 80% and beta greater than 0.20, so their negative results may be Type II errors. Always check the a priori power calculation before accepting a negative result.

Low power also undermines the significant results. Power is usually taught as protection against missing a true effect, and that is the smaller half of it. Alpha fixes how often a false positive arises among true nulls, but the proportion of significant findings that are real depends on power as well.

Why. With power at 0.20 a true effect is detected one time in five, while false positives keep arriving at their usual rate, so a larger share of the significant results in that literature are wrong. A p value below 0.05 from an underpowered study is weaker evidence than the same p from a well-powered one, which is not something the p-value itself tells you.

The winner's curse. In an underpowered study only an unusually large observed effect will clear the significance threshold, so the effects that reach print are systematically exaggerated. This is why small early trials so often fail to replicate and why meta-analyses shrink toward smaller effects over time. The same mechanism explains why trials stopped early for benefit overstate the treatment effect.

Who pays. Both consequences fall on the reader, not the investigator. The investigator who runs an underpowered study risks a null result; the literature inherits exaggerated positives.

Post-hoc power. Power calculated after a non-significant result, using the observed effect, is meaningless: it is statistically circular, merely re-expresses the p-value, and can never explain a null result. Judge underpowering from the a priori calculation and the width of the confidence interval instead, and to know whether a negative trial excluded a worthwhile effect, read the confidence interval.

Meta-analysis. Pooling data from multiple studies increases power, which reduces the risk of Type II error and gives a more precise estimate of the effect.

Four moving parts. Power is fixed by four interrelated determinants, and understanding them is how a sample-size calculation is actually performed. Fix any three and the fourth is determined; conventionally alpha, power and the target effect size are specified to solve for the required sample size.

Effect size. Effect size expresses the magnitude of a difference independently of sample size. For continuous outcomes it is the standardised mean difference (Cohen's d), the difference in means divided by the pooled standard deviation, conventionally about 0.2 small, 0.5 medium and 0.8 large. For proportions or risks it is the relative risk, odds ratio or absolute risk difference.

The MCID. The minimal clinically important difference is the smallest change a patient perceives as worthwhile. A trial should be powered to detect the MCID, not a trivial difference, and this is what links the statistics to clinical relevance.

The flip side, in registries. With a very large sample, a difference far smaller than the MCID can still reach p less than 0.05, so a "statistically significant" registry finding may be clinically meaningless. Always read the effect size and its confidence interval, never the p-value alone.

The multiple testing problem. Testing multiple hypotheses inflates the overall Type I error rate. Test 20 different outcomes at alpha 0.05 each and you expect 20 × 0.05 = one false positive on average, and the family-wise error rate (FWER), the probability of at least one Type I error, increases with each test.

The arithmetic. FWER = 1 minus (1 minus alpha)^n. For 20 tests at alpha 0.05 that is 1 minus 0.95^20 = 0.64, a 64% chance of at least one false positive.

Bonferroni correction. Divide alpha by the number of tests, so adjusted alpha = 0.05 / n, to maintain the overall Type I error. Testing five outcomes gives 0.05 / 5 = 0.01, and each test must then reach p less than 0.01 to hold the overall rate at 0.05. It is conservative, and may increase Type II error by reducing power.

How far to correct is contested. Whether and how to adjust is genuinely disputed, between Perneger and proponents of strict family-wise control. Perneger's paper is titled What's wrong with Bonferroni adjustments: blanket correction of everything is over-conservative, buys its Type I reduction with Type II error, and rests on an arbitrary decision about what counts as the "family" of tests.

The defensible middle ground. Pre-specify one primary outcome, which needs no adjustment, and treat everything else as hypothesis-generating, reported with effect sizes. Where correction is needed, correct thoughtfully (Holm, false discovery rate) rather than applying Bonferroni to everything.

Less conservative corrections. Bonferroni is the simplest multiplicity correction but also the most conservative, and examiners frequently ask what else is available, because the alternatives sacrifice less power. Bonferroni, Holm and Hochberg control the family-wise error rate, the chance of even one false positive; Benjamini-Hochberg controls the false discovery rate instead.

Repeated looks over time: alpha-spending. Multiple interim analyses inflate the Type I error in the same way as multiple outcomes, so the total alpha must be spent across the looks using group-sequential boundaries:

• O'Brien-Fleming boundaries are very stringent early, making it hard to stop early and preserving alpha and power for the final analysis
• Pocock boundaries apply a constant, less stringent threshold at every look

These terms are repeatedly conflated in vivas and MCQs, and each has its classic trap.

Global Reporting Standards

Error control is enforced internationally through reporting and regulatory frameworks rather than country-specific rules - the concepts are universal across fellowship curricula.

Registries and Large Datasets

National joint replacement registries (NJR for England/Wales, AOANJRR Australia, SHAR Sweden, the Norwegian and New Zealand registries, and AJRR in the US) hold hundreds of thousands of procedures. Their value for this topic is power: rare events such as implant revision are detectable with adequate precision, dramatically reducing Type II error compared with single-centre series. The trade-off is that with such large samples, trivial differences become statistically significant, so the emphasis shifts to clinical significance and effect size (e.g. hazard ratios for revision) rather than the p-value alone.

High- vs Limited-Resource Practice Variation

The teaching point is universal: interpret a "negative" study in the light of its power, and a "positive" study in the light of multiplicity and effect size - independent of country.

Should alpha stay at 0.05? A 2017 proposal argued for lowering the default threshold for new claims to 0.005 to curb false positives. Critics countered that this simply trades a higher Type I rate for a higher Type II rate and demands much larger samples. No global consensus exists, and 0.05 remains the working convention.

Abandon significance testing? Some statisticians advocate retiring the word "significant" altogether in favour of estimation (effect sizes with confidence intervals) and Bayesian reasoning. Exam answers should still command the classical framework but can acknowledge this debate.