Survivorship Bias

Overview

Survivorship bias is drawing conclusions from a sample pre-filtered by survival — treating survivor traits as the cause of survival when non-survivors (absent from data by definition) may have had identical traits and still failed.

Canon: Wald (1943) reversed the Navy's bomber-armor recommendation — returning planes showed damage where hits were survivable; the missing planes (shot down) were hit where returning planes showed no damage. Armor the gaps, not the hits.

Composes with bayesian-reasoning (prior = population, not survivors), critical-thinking (what would non-survivors say?), first-principles (population is bedrock), and abductive-reasoning ("winners have trait Y" is one hypothesis; randomness is another).

When to Use

• Someone draws lessons from "what successful X did"
• Investment returns / fund performance / backtested strategies are cited
• A business strategy is justified by pointing to companies that used it
• Medical / treatment success rates are reported without dropout data
• Career advice comes from what top performers did
• Odds of building an AI startup are inferred from the visible AI winners (funded unicorns, "wrapper" success stories) amid the AI-bubble / AI-capex debate

Not when: population data available and filter already corrected; analysis is intentionally about survivors only with no population claim.

Coaching Novices (Adaptive Front Door)

• Engine mode: user has a concrete claim and data → run The Process directly.
• Coach mode: unfamiliar or no case → guide step by step.

In Coach mode, respond one step at a time. Each [WAIT] is a hard stop — output only that step's question, then stop.

1. One-liner: "Before believing 'X worked because winners did X,' ask whether the losers did the same X — they're not in your sample to refute it."
2. Check fit: if sample is explicitly restricted to survivors with no population claim, this lens doesn't apply.
3. Elicit their real claim and the visible data they have.

[WAIT — do not advance until user responds]

4. Run The Process one step at a time: what filter produced this sample? what's missing? if missing data looked like the sample, would the conclusion hold?

[WAIT — do not advance until user responds]

5. Close by naming the selection-corrected conclusion (or marking it unprovable from this data alone).

[WAIT — do not advance until user responds]

The Process

Step 1 — State the claim: What is being concluded, from what sample, from what source?

Step 2 — Identify the survival filter: What process produced this sample? What was the population before the filter? What fraction was removed? What did the filter select for/against?

Step 3 — Construct the non-survivor hypothesis: What did non-survivors likely have? Did they share the trait attributed to success? Would the claim hold if we could see them?

Step 4 — Re-estimate strength: Best case = trait explains survival (non-survivors lacked it). Worst case = trait doesn't explain survival (non-survivors had it too). What evidence distinguishes these?

Step 5 — Correct or mark: Get population data and re-run analysis with selection correction. If unavailable, mark conclusion as conditional on survivor sample.

Output Template

# Survivorship Bias Analysis: <claim>
Claim / sample / source:
Survival filter (what removed non-survivors, population size est., survival rate est.):
Non-survivor hypothesis (what they likely had/lacked, could they have had same trait):
Corrected inference (conclusion, confidence, what data would settle it):

→ Method in Action: Abraham Wald and the Statistical Research Group, 1943 · Mutual Fund Survivorship and Reported Returns, 1971–1996 → 2026 lens: AI-startup survivorship — funded unicorns vs the dead-wrapper graveyard (2023–2026)

Pack: Common Survivor Patterns

Domain	Survivor sample	Missing non-survivor data	Biased claim
Business / startup	Surviving companies	Failed companies	"Successful companies do X"
Investment returns	Active funds / listed stocks	Closed funds / delisted stocks	"Stocks return 10% annually"
Career advice	Top performers	People who left the field	"To succeed, do X"
Scientific findings	Published studies	Unpublished null results	"X is significant"
Treatment efficacy	Patients who completed	Drop-outs, deaths during treatment	"X% recovered"

→ Primary sources: references/sources.md

Common Rationalizations

[D] = designed upfront | [O] = observed in real use. [O] entries are more valuable.

Fake move	Reality
[D] "Look at the data" (survivor sample)	Survivor data ≠ population data. Correct or mark as conditional.
[D] Citing one famous example as proof	N=1 in survivor sample tells you nothing about the rate.
[D] "X is the formula for success"	If failures did the same X, X is not the formula. Get non-survivor data or stop claiming.
[D] "We use a backtested strategy"	If backtest excludes failed/delisted stocks, results are upward-biased.
[D] "If we had non-survivor data, we'd see the same pattern"	Unfalsifiable without the data. Get it or hold the claim.
→ Add [O] entries here after each real use — paste the actual failure pattern	What went wrong and why

Red Flags

• Sample described as "successful X" or "the X who made it"
• Data source is survivor-filtered (active funds, surviving companies, published studies)
• Base rate of failure / dropout not stated
• Conclusions about a population drawn from the survivor subset

Verification

• Survival filter identified
• Non-survivor population size estimated
• Non-survivor hypothesis constructed
• Conclusions conditional on survivor sample, or formally selection-corrected
• Recommendation robust to worst-case non-survivor hypothesis

---

Part of deciqAI Knowledge Skills — 227 open-source thinking skills that make rigor executable for AI agents. The same skills power every deciqAI agent, which runs them autonomously to operate your company. See it run → https://www.deciqai.com/c/survivorship-bias · ⭐ Star the repo → https://github.com/deciqAI/knowledge-skills · Contributions welcome.

Agents: latest version & machine-readable metadata → https://www.deciqai.com/s/survivorship-bias.json