Login Create Account

Bayes' Theorem & Screening Lab

Probability & Decision Base-Rate Fallacy

Understand why highly accurate predictive models and fraud detectors generate mostly false alarms when the underlying business event is rare.

👨‍🏫 Professor Mode: Guided Learning Experience

New to Bayesian reasoning? Enable Professor Mode for step-by-step guidance on base rates, natural frequencies, and avoiding false alarm traps!

OVERVIEW & WHAT THIS TOOL SOLVES

When a churn-prediction model or fraud detector boasts 90% sensitivity and 90% specificity, intuition tells managers that a flagged account has a 90% chance of churning. Yet if only 2% of accounts actually churn, over 84% of the model's alerts will be false alarms! The Bayes' Theorem & Screening Lab exposes the Base-Rate Fallacy using natural frequencies (1,000 accounts) and exact posterior updates so you can evaluate real-world precision before launching expensive retention interventions.

🎯 Questions It Answers

  • Given a flag from our model, what is the true posterior probability (Positive Predictive Value) that the account is actually a positive case?
  • Why do false positives outnumber true positives when the underlying population base rate is low?
  • How do natural frequency trees (out of 1,000 accounts) map 1-to-1 onto Bayes' conditional probability formula?

📂 What Data You Need

  • Population Base Rate (Prior): True underlying prevalence of the event (e.g., 2% monthly churn or 0.5% payment fraud)
  • Sensitivity (True Positive Rate): Probability that a true positive account triggers a model flag
  • Specificity (True Negative Rate): Probability that a healthy non-event account is correctly left unflagged

💡 What You'll Discover

  • Natural Frequency Flow Tree: Step-by-step split of 1,000 accounts into True Positives, False Negatives, False Positives, and True Negatives
  • 1,000-Account Dot Grid & Confusion Matrix: Visual proof of how false alarms crowd the flagged pool
  • Posterior Precision vs. Base Rate: Live calculation of Positive Predictive Value (PPV) and False Alarm Rate
🔍 Key Nuances & Common Traps to Watch For
  • Don't Confuse Sensitivity with Precision (PPV): Sensitivity asks "of all actual churners, what share get flagged?" whereas Precision (the Bayesian posterior) asks "of all flagged accounts, what share are actual churners?"
  • Why Specificity Matters More Than Sensitivity for Rare Events: When 980 out of 1,000 accounts are healthy, a 10% false-positive rate (90% specificity) generates 98 false alarms—swamping the 18 true positives caught from the 20 actual churners.
  • Sequential Screening Multiplies Precision: Using a first-stage model flag to narrow the pool raises the effective prior for a second verification check.
📐 Under the Hood: Model Equations & Variable Definitions

1. Bayes' Theorem for Positive Predictive Value (Posterior Probability)

\[ P(A \mid B) = \frac{P(B \mid A)\,P(A)}{P(B)} = \frac{P(B \mid A)\,P(A)}{P(B \mid A)\,P(A) + P(B \mid \neg A)\,P(\neg A)} \]

💬 Read it in plain English: The probability that a flagged account is truly positive equals True Positives divided by All Flagged Accounts (True Positives + False Positives).

P(A), P(¬A)
Prior Base Rate & ComplementYOUR DATA ⓘ
Baseline population probability of the event P(A) and non-event P(¬A) = 1 − P(A).
P(B | A)
Sensitivity (True Positive Rate)YOUR DATA ⓘ
Conditional probability of a positive flag B given the event A is truly present.
P(¬B | ¬A)
Specificity (True Negative Rate)YOUR DATA ⓘ
Conditional probability of a clean result ¬B given the account is healthy (¬A).
P(B | ¬A)
False Positive Rate (1 − Spec)ESTIMATED ⓘ
Probability that a healthy account triggers a false alarm: 1 − P(¬B | ¬A).
P(B)
Total Flag Rate (Marginal)OUTPUT ⓘ
Overall proportion of the population that receives a positive flag.
P(A | B)
Posterior Precision (PPV)INFERENCE ⓘ
Updated probability that the event is truly present given a positive flag B.

⚡ Real-World Business Presets

Model & Population Inputs

Adjust these levers to see how Bayes' Rule behaves in real time

Reality: The true natural prevalence of the event across all 1,000 accounts (e.g. churners, fraud, rare flaws).

The Detector: True Positive Rate — the probability that an account genuinely about to churn triggers an alert.

The Non-Events: True Negative Rate — the probability that a loyal account is correctly left undisturbed.

🌳 Follow the Numbers: How 1,000 Accounts Flow Through Bayes' Theorem

Natural Frequency Decomposition

Instead of getting tangled in conditional probability algebra, watch how 1,000 accounts split step-by-step. Bayes' Theorem is simply counting: (Genuine Alarms) ÷ (All Alarms That Rang).

1️⃣ Stage 1: Ground Truth Reality (Split 1,000 Accounts by Base Rate)
Genuine Event (e.g. Churners) Target Condition
20
1,000 × 2.0% base rate = 20 accounts
Non-Event (e.g. Loyal Customers) Normal Majority
980
1,000 × 98.0% = 980 accounts
2️⃣ Stage 2: The Screening Test (Apply Sensitivity & Specificity)
True Positive (TP) Caught
18
20 churners × 90% caught = 18
False Negative (FN) Missed
2
20 churners × 10% missed = 2
False Positive (FP) False Alarm
98
980 loyal × 10% error = 98
True Negative (TN) Cleared
882
980 loyal × 90% cleared = 882

3️⃣ Stage 3: The Alarm Rings! (The Manager's Reality)

We gather every account that triggered a positive alert: True Positives (18) + False Alarms (98).

P(Event | Alert) = TP / (TP + FP) = 18 / (18 + 98) = 18 / 116
15.5%
18 genuine detections out of 116 alerts

Posterior Probability / Precision (PPV)

P(Genuine Event | Positive Alert)
P(Event | Alert) = TP / (TP + FP)
15.5%
Only 18 of 116 flagged cases are genuine

💡 The Two Questions That Confuse Everyone

Confronting the Inverse Fallacy
Perspective A: The Model Builder

"If someone is actually churning, will our model catch them?"

90% (18 of 20 caught)

This is Sensitivity (Recall). It only inspects the 20 genuine churners. Because 18 were detected, the data scientist reports a stellar 90% catch rate!

Perspective B: The Business Manager

"When the alarm sounds, is this customer actually churning?"

15.5% (Only 18 of 116 alerts are real)

This is Precision (PPV). The manager acts on alerts. When 116 alerts ring, 98 are false alarms handed $20 credits or false declines.

Why the dramatic collapse from 90% to 15.5%? Because loyal customers outnumber churners 49 to 1 (980 vs. 20)! Even a small 10% error rate leaking from the giant bucket of 980 loyal accounts generates 98 false alarms, which completely drowns out the 18 real churners caught.

Natural Frequency Grid (Out of 1,000 Accounts / Events)

True Positive (18)
False Alarm (98)
Missed Event (2)
True Non-Event (882)
Visual Focus:

🌉 The Mathematical Bridge: Abstract Formula vs. Integer Dot Counting

Students often fear Bayes' algebraic formula. Look at how each symbol maps directly to counting the dots:

1. The Standard Bayes Formula
P(A | B) = P(B|A)·P(A) / [P(B|A)·P(A) + P(B|¬A)·P(¬A)]
Numerator is genuine detections; denominator is all positive alerts.
2. Evaluated in Decimals
= 0.90 × 0.02 / [0.018 + 0.098] = 0.1552 (15.5%)
Multiplying each decimal by 1,000 turns this into simple whole people!
3. Counted Directly From the Dots
= 18 Green Dots (TP) / [18 Green + 98 Amber] = 18 / 116 = 15.5%
Exact same math — without any confusing conditional notation.

Contingency Matrix (Confusion Table)

Rows are what really happened; columns are what the model flagged
Actual Outcome ↓ · Model Decision → Flagged Positive (\(B\)) Flagged Negative (\(\neg B\)) Marginal Total
Actual Event Present (\(A\))
True Positive (TP)
18
False Negative (FN)
2
 20
Actual Non-Event (\(\neg A\))
False Positive (FP)
98
True Negative (TN)
882
 980
Marginal Total 116 884 1,000
Positive Predictive Value (PPV)
15.5%
Reliability of a positive flag: \(TP / (TP + FP)\)
False Discovery Rate (FDR)
84.5%
Share of alerts that are false alarms: \(FP / (TP + FP)\)
Negative Predictive Value (NPV)
99.8%
Reliability of an unflagged account: \(TN / (TN + FN)\)