Study Planning | Adequate Sampling | Effect Detection
- Power: Probability of detecting a true effect (1 minus Beta). Conventional target is 80 percent.
- Sample Size Calculation Requires: Effect size (MCID), Alpha (usually 0.05), Power (usually 0.80), Variability (SD)
- Effect Size: The magnitude of difference you want to detect - must be clinically meaningful (MCID), not just statistically significant
- Underpowered Studies: Risk Type II error (false negative) - failing to detect real treatment effect
- Factors Increasing Sample Size: Smaller effect size, lower power, higher variability, lower alpha
- “Power = 80% means 20% chance of Type II error (missing a true effect)
- “MCID (Minimal Clinically Important Difference) defines what effect size matters to patients
- “Larger sample size increases power but also increases cost and time
- “Pilot studies help estimate variability (SD) for sample size calculations
Power = 1 minus Beta. Probability of correctly rejecting null hypothesis when alternative is true. Power = 80% means 80% chance of detecting real effect if it exists.
Four key inputs: (1) Alpha (Type I error, usually 0.05), (2) Power (1-Beta, usually 0.80), (3) Effect Size (MCID), (4) Variability (Standard Deviation).
Risk: Type II error (false negative). Study may fail to detect real treatment benefit. Common in orthopaedic trials with small sample sizes.
Statistical Significance: p less than 0.05. Clinical Significance: Difference exceeds MCID. Large studies detect trivial differences; small studies miss important ones.
APESSample Size Calculation Inputs
Hook:APES calculate sample size - Alpha, Power, Effect, SD are the four essentials!
SHAPEFactors that Increase Required Sample Size
Hook:SHAPE your sample size - these five factors determine how many participants you need!
Overview/Introduction
What is Power?
Definition: Statistical power is the probability that a study will detect an effect when there truly is an effect to detect.
Formula: Power = 1 minus Beta (Type II error rate)
Interpretation:
- Power = 80%: 80% chance of detecting true effect, 20% chance of missing it (Type II error)
- Power = 50%: Coin flip - as likely to miss effect as to find it (underpowered)
- Power = 95%: 95% chance of detecting true effect, but requires much larger sample
- Meaning
- Very high chance of detecting true effect
- Adequacy
- Excellent but may be excessive
- Sample Size
- Very large sample needed
- Meaning
- High chance of detecting true effect
- Adequacy
- Conventional and adequate
- Sample Size
- Moderate sample size
- Meaning
- Moderate chance, meaningful risk of missing effect
- Adequacy
- Underpowered - risky
- Sample Size
- Smaller sample
- Meaning
- More likely to miss effect than find it
- Adequacy
- Severely underpowered
- Sample Size
- Very small sample
Understanding power is essential for designing adequately powered studies.
Principles of Power Analysis
Core Principles
- Larger sample size increases power
- Doubling sample size does NOT double power (diminishing returns)
- Power increases steeply initially, then plateaus
- Higher power requires larger sample (more cost, time)
- Smaller effect size (more clinically conservative) requires larger sample
- Lower alpha (more statistically conservative) requires larger sample
- Power ∝ Sample Size
- Power ∝ Effect Size
- Power ∝ Alpha
- Power inversely proportional to Variability (SD)
Understanding these principles allows rational study design decisions.
Sample Size Calculation
Four Essential Inputs
Every sample size calculation requires four inputs:
Alpha: Type I Error Rate
Definition: Probability of falsely rejecting null hypothesis (false positive).
Conventional Choice: Alpha = 0.05 (5%)
Meaning: Willing to accept 5% chance of finding difference when none exists.
Trade-off: Lower alpha (e.g., 0.01) reduces false positives but requires larger sample size.
Bonferroni Correction: When testing multiple outcomes, divide alpha by number of tests to maintain overall Type I error rate.
Understanding alpha is critical for interpreting p-values and planning studies.
Performing Sample Size Calculation
Sample Size Formula (Continuous Outcome, Two Groups)
For comparing means between two groups:
n = 2 × (Zα + Zβ)² × SD² / MCID²
Where:
- n = sample size per group
- Zα = Z-score for alpha (1.96 for alpha = 0.05 two-tailed)
- Zβ = Z-score for beta (0.84 for power = 0.80)
- SD = standard deviation
- MCID = effect size (minimal clinically important difference)
Worked Example
Question: How many patients needed per group to detect 10-point improvement in WOMAC score?
Given:
- MCID = 10 points
- SD = 20 points (from literature)
- Alpha = 0.05 (Zα = 1.96)
- Power = 0.80 (Zβ = 0.84)
Calculation:
- n = 2 × (1.96 + 0.84)² × 20² / 10²
- n = 2 × 7.84 × 400 / 100
- n = 2 × 31.36
- n = 63 patients per group
Accounting for Dropout:
- If expecting 15% dropout: n = 63 / 0.85 = 74 patients per group
- Total enrollment: 148 patients
Understanding sample size calculation ensures adequately powered studies.
Standardised Effect Size (Cohen's d)
When the effect size is expressed in the units of the outcome (for example a 10-point WOMAC difference) it is an absolute effect size. Dividing it by the standard deviation gives the standardised effect size, or Cohen's d, which is dimensionless and lets effects be compared across different scales.
Cohen's d = (mean difference) divided by the (pooled standard deviation)
- Cohen's d
- 0.2
- Approx. n per group (80% power, alpha 0.05)
- About 394
- Cohen's d
- 0.5
- Approx. n per group (80% power, alpha 0.05)
- About 64
- Cohen's d
- 0.8
- Approx. n per group (80% power, alpha 0.05)
- About 26
Because the required sample size is roughly proportional to 1 divided by the square of the standardised effect size, halving the effect you wish to detect roughly quadruples the patients needed - which is why a clinically tiny difference demands an enormous trial. When no validated MCID exists, a distribution-based estimate of 0.5 standard deviation (a medium effect) is sometimes used as a transparent fallback, but an anchor-based MCID is preferred.
Cohen's d is the mean difference divided by the standard deviation: 0.2 small, 0.5 medium, 0.8 large. Sample size scales with 1 over d squared, so detecting a small standardised effect needs roughly an order of magnitude more patients than a large one.
Types of Power Analysis
A Priori vs Post Hoc Power Analysis
- When Performed
- Before study begins
- Purpose
- Calculate required sample size
- Validity
- Valid and recommended
- When Performed
- After study completed
- Purpose
- Calculate achieved power
- Validity
- Controversial - often misleading
- When Performed
- During planning
- Purpose
- Assess power across range of assumptions
- Validity
- Useful for uncertainty
- Calculate sample size BEFORE enrolling patients
- Uses estimated effect size and SD from literature or pilot
- Ensures study designed with adequate power
- Calculating power AFTER study complete using observed data
- Often done to explain non-significant results
- Mathematically redundant - p-value and post hoc power are directly related
Clinical Application
Common Problem: Many orthopaedic RCTs are underpowered. Small sample sizes fail to detect clinically meaningful differences. Results are inconclusive, not negative.
Clinical Relevance: A statistically significant finding (p less than 0.05) may not be clinically important. Always check if difference exceeds MCID.
Purpose: Estimate variability (SD) and feasibility before full trial. Helps refine sample size calculation. Do NOT use pilot for hypothesis testing.
Solution: When single center cannot recruit adequate sample, multicenter collaboration achieves power. AOANJRR and international registries provide large samples.
Superiority, Equivalence and Non-inferiority Trials
The hypothesis a trial is designed to test determines its sample size and how a "negative" result may be read - and explains why a failed superiority trial can never prove equivalence.
- Question
- Is the new treatment better than the comparator?
- Statistical approach
- Two-sided test of no difference; reject the null to claim superiority
- Question
- Is the new treatment neither better nor worse, within a margin?
- Statistical approach
- Two one-sided tests; the confidence interval must lie entirely within the equivalence margin (plus or minus delta)
- Question
- Is the new treatment not unacceptably worse than the standard?
- Statistical approach
- One-sided; the relevant confidence-interval bound must not cross the non-inferiority margin (delta)
A non-inferiority trial accepts that the new treatment may be marginally worse, in exchange for advantages such as lower cost, toxicity or convenience, provided it stays within the pre-specified margin (delta). The margin must be clinically justified: too wide allows a genuinely inferior treatment to pass, too narrow demands an impractically large sample. Such trials require assay sensitivity (the trial could have detected a true difference if one existed) and, unlike superiority trials, the per-protocol analysis is at least as important as intention-to-treat, because dropouts and non-compliance bias toward apparent non-inferiority.
A non-significant superiority trial is inconclusive, not proof of equivalence. To show two treatments are equivalent or non-inferior you must design the trial for that purpose, with a pre-specified margin and (for non-inferiority) a one-sided comparison and a per-protocol analysis.
Software and Calculation Tools
Common Power Analysis Software
- Cost
- Free
- Features
- Wide range of tests, user-friendly
- Best For
- Academic researchers, most designs
- Cost
- Free
- Features
- Simple interface, basic designs
- Best For
- Quick calculations, beginners
- Cost
- Commercial
- Features
- Comprehensive, regulatory accepted
- Best For
- Industry trials, complex designs
- Cost
- Commercial
- Features
- Extensive documentation, FDA submissions
- Best For
- Regulatory submissions
Online Calculators:
- ClinCalc sample size calculator (free online)
- OpenEpi power calculation (epidemiological studies)
- Sealed Envelope (clinical trial tools)
Manual Calculation Reference
- Formula Component
- Z-score for alpha
- Value (Common)
- 1.96
- Formula Component
- Z-score for alpha
- Value (Common)
- 1.645
- Formula Component
- Z-score for beta
- Value (Common)
- 0.84
- Formula Component
- Z-score for beta
- Value (Common)
- 1.28
Addressing Underpowered Studies
Strategies to Increase Power
- Approach
- Enroll more participants
- Advantages
- Direct power increase
- Disadvantages
- More cost, time, resources
- Approach
- Pool recruitment across sites
- Advantages
- Achieves larger sample
- Disadvantages
- Heterogeneity, logistics complexity
- Approach
- Stricter inclusion criteria, standardized protocols
- Advantages
- Increases precision
- Disadvantages
- Reduces generalizability
- Approach
- Choose outcome with lower SD
- Advantages
- More precise measurement
- Disadvantages
- May not be clinically preferred
When Power Cannot Be Achieved
- Rare conditions: May need registry-based or multi-national studies
- Ethical constraints: Cannot enroll more for safety reasons
- Resource limitations: Accept lower power with pre-specified disclosure
Alternative Approaches:
- Bayesian analysis (can provide evidence even with small samples)
- Meta-analysis (combine with existing studies)
- Confidence interval interpretation (focus on precision)
Common Pitfalls and Errors
Errors in Sample Size Calculation
- Problem
- Sample shrinks below powered size
- Consequence
- Underpowered final analysis
- Solution
- Inflate by 15-25% for attrition
- Problem
- MCID too large or optimistic
- Consequence
- Study underpowered for true effect
- Solution
- Use conservative, validated MCID
- Problem
- Variability higher than expected
- Consequence
- Lower power than calculated
- Solution
- Use upper bound of SD estimate
- Problem
- Many outcomes without correction
- Consequence
- Inflated Type I error
- Solution
- Adjust alpha (Bonferroni) or define primary outcome
Interpretation Errors
Common Mistakes:
- Concluding treatments are "equivalent" from underpowered negative study
- Using post hoc power to justify non-significant results
- Ignoring confidence intervals when assessing clinical relevance
- Confusing statistical significance with clinical importance
Guidelines, Registries & Global Practice
Why Power and Sample Size Are a Global Concern
Underpowering is not confined to any one country. The landmark survey of orthopaedic trauma RCTs found a mean primary-outcome power of roughly 25 percent and a Type-II error rate over 90 percent, and fragility analyses of sports-surgery trials worldwide report a median Fragility Index of only two patients. Adequately powered, registry-linked and multinational collaborations are the international response.
- Region
- Global (journals worldwide)
- Core Requirement
- Items 7a/7b: report how sample size was determined and any interim analyses
- Emphasis
- Transparency of a priori justification
- Region
- Global regulatory (FDA, EMA, PMDA)
- Core Requirement
- Pre-specify primary outcome, effect size, alpha, power and analysis set
- Emphasis
- Confirmatory trial rigour
- Region
- UK
- Core Requirement
- Funded trials must show a robust, MCID-anchored sample-size calculation
- Emphasis
- Value for public research funding
- Region
- Global
- Core Requirement
- Protocols must state sample-size assumptions before recruitment
- Emphasis
- Pre-registration of design
- Region
- Global trauma
- Core Requirement
- Promote multicentre recruitment to reach powered samples
- Emphasis
- Overcoming single-centre limits
Registries as a Power Solution
Large national arthroplasty and trauma registries supply the sample sizes single centres cannot. The AOANJRR (Australia), NJR (England, Wales, Northern Ireland and Isle of Man), AJRR (USA), SHAR (Sweden), the Norwegian Arthroplasty Register and the NZJR pool hundreds of thousands of procedures, giving the statistical power to detect small but clinically important differences in implant survival and revision rates that no individual RCT could achieve.
High- versus Limited-Resource Practice Variation
Access to dedicated trial units, biostatisticians, multicentre networks and mature registries makes adequately powered RCTs and registry studies feasible. Power and MCID-anchored calculations are an expected norm for grant funding and publication.
Smaller catchments, fewer statisticians and funding constraints make large RCTs difficult. Pragmatic responses include international collaboration, registry-based studies, Bayesian designs that extract more information from small samples, and honest pre-specified disclosure of limited power.
Controversies and Areas of Uncertainty
Where Experts Still Disagree
- One View
- 80% is the long-standing convention and keeps trials feasible
- Opposing View
- 20% chance of missing a true effect is too high for definitive surgical questions; 90% should be standard
- Pragmatic Position
- Use 90% for pivotal or hard-to-repeat trials; justify the choice explicitly
- One View
- Convenient when no anchor-based MCID exists
- Opposing View
- Detached from the patient's perspective; may not reflect what patients value
- Pragmatic Position
- Prefer validated anchor-based MCID; use 0.5 SD only as a transparent fallback
- One View
- Estimate SD, recruitment and feasibility before a full trial
- Opposing View
- Pilots estimate SD imprecisely and are misused for hypothesis testing
- Pragmatic Position
- Use adequately sized pilots for feasibility only, never for efficacy claims
- One View
- Intuitive measure of how robust a significant result is
- Opposing View
- Conflates significance with sample size and ignores effect magnitude
- Pragmatic Position
- Report alongside, not instead of, effect size and confidence intervals
- One View
- Familiar, regulator-accepted default
- Opposing View
- Arbitrary; some call for 0.005 or abandoning thresholds for estimation
- Pragmatic Position
- Pre-specify alpha; emphasise estimation with confidence intervals over dichotomising at 0.05
A non-significant result in a small trial means the study was inconclusive, not that the treatments are equivalent. Proving equivalence requires a purpose-designed equivalence or non-inferiority trial with a pre-specified margin - it can never be inferred from a failure to reach p less than 0.05 in an underpowered superiority trial.
MCQ Practice Points
Q: What is statistical power? A: The probability of detecting a true effect when it exists, calculated as 1 minus Beta (Type II error rate). Power = 80% means 80% chance of finding real difference if present, 20% risk of missing it.
Q: Which factor does NOT increase required sample size? A: Higher alpha (e.g., 0.10 vs 0.05) actually decreases required sample size. Factors that increase sample size: smaller effect size, higher power, higher variability (SD), lower alpha.
Q: Why is MCID important for sample size calculation? A: MCID defines the clinically meaningful effect size - the smallest difference that matters to patients. Using MCID ensures study is powered to detect differences that are clinically relevant, not just statistically significant. Without MCID, large studies may detect trivial differences.
Exam Viva Scenarios
Practise clinical reasoning and management decisions out loud
“You are planning an RCT to compare two surgical approaches for rotator cuff repair. What information do you need to calculate the required sample size?”
“You read an RCT comparing two implants for THA. The study found no significant difference (p = 0.15) with 40 patients per group. The power calculation shows the study had 35 percent power. How do you interpret this result?”
“A reviewer asks you to add a post hoc power calculation to your non-significant orthopaedic RCT to show it was adequately powered. The trial had 50 patients per arm. How do you respond, and how else might you convey the robustness of your findings?”
Core Concepts
- Power = 1 minus Beta = Probability of detecting true effect
- Conventional power = 80% (20% risk of Type II error)
- Sample size needs 4 inputs: Alpha, Power, Effect Size (MCID), SD
- Underpowered study = High risk of missing real effect (Type II error)
Sample Size Calculation Inputs
- Alpha = Type I error (usually 0.05) - false positive rate
- Power = 1 minus Beta (usually 0.80) - true positive rate
- Effect Size = MCID (clinically meaningful difference)
- SD = Variability (from literature or pilot study)
- Inflate by 15-20% for anticipated dropout
Factors Increasing Sample Size
- Smaller effect size (harder to detect)
- Higher power (90% vs 80%)
- Lower alpha (0.01 vs 0.05)
- Higher variability (larger SD)
- Expected dropout or loss to follow-up
Interpreting Power
- Power greater than 90% = Excellent, may be excessive
- Power 80-90% = Adequate and conventional
- Power 50-80% = Underpowered, risky
- Power under 50% = Severely underpowered, likely to fail
- Negative result from underpowered study = Inconclusive
Clinical Application
- MCID defines clinical relevance, not just statistical significance
- Many orthopaedic RCTs are underpowered (power under 80%)
- Pilot studies estimate SD and feasibility, NOT for hypothesis testing
- Absence of evidence is NOT evidence of absence (underpowered studies)
- Wide confidence intervals indicate insufficient precision
Evidence Base
Type-II Error Rates in Orthopaedic Trauma RCTs
- Systematic survey of 117 randomised fracture-care trials (1968-1999) including 19,942 patients
- Mean study power for the primary outcome was only 24.65% (range 2% to 99%)
- The Type-II error rate (beta) for primary outcomes was 90.52% - far above the accepted 20% threshold
- Most trials were grossly underpowered; performing a priori power and sample-size calculations is the corrective
Understanding the MCID: Concepts and Methods
- Defines the MCID as the smallest improvement a patient considers worthwhile - the effect size that should anchor sample-size calculations
- MCID is derived by anchor-based methods (linked to an external patient-reported criterion) or distribution-based methods (e.g. 0.5 standard deviation, standard error of measurement)
- Three core limitations: multiplicity of MCID estimates, loss of the patient's perspective, and dependence on the baseline score
- No single MCID is universal; the value depends on the instrument, population and method used
CONSORT 2010: Reporting Standard for Sample Size
- CONSORT Item 7a requires authors to report how the sample size was determined
- CONSORT Item 7b requires reporting of interim analyses and stopping guidelines where applicable
- Sample-size justification should state the target effect size, power, alpha and the assumed standard deviation
- Adopted worldwide by leading journals to make trial adequacy transparent to readers