Rare Event Calibration Lab

Rare eventsProbability calibration

Changing the class balance changes what a model's percentages mean. Explore the difference between ranking customers, estimating their risk, and deciding whom to contact.

OVERVIEW & WHAT THIS TOOL SOLVES

When customer churn or fraud happens in only 2% to 5% of accounts, analysts often rebalance the training set to 50/50 by undersampling non-churners or oversampling churners—and then forget that the model's predicted probabilities are now wildly inflated on the 50/50 training scale. The Rare Event Calibration Lab separates customer ranking (AUC), probability calibration (King & Zeng prior correction), and action cutoffs across natural and rebalanced training samples.

🎯 Questions It Answers

Does rebalancing a 5% churn training set to 20%, 35%, or 50% improve how well the model ranks holdout customers (AUC), or just inflate its raw probability scores?
How do we mathematically recalibrate oversampled/undersampled probabilities back to the true population base rate (τ)?
Why does a 12% churn probability justify a retention call when the population base rate is 5%—even though 12% is far below the software default 50% cutoff?

📂 What Data You Need

Population Base Rate (τ): True rare-event rate in the customer population (2%, 3%, 5%, or 10%)
Sample Size & Seed: Total simulated customers (split into training and untouched holdout) and reproducible RNG seed
Scale & Cutoff Controls: Toggle raw vs. prior-corrected probabilities and adjust the contact cutoff percentage

💡 What You'll Discover

4-Condition Holdout Comparison: Natural vs. 20%, 35%, and 50% rebalanced training sets evaluated on the exact same holdout
Reliability (Calibration) Curves: Decile-by-decile predicted risk vs. observed holdout churn before and after prior correction
Contact Volume & Equivalent Cutoffs: Exact mapping showing how a 50% raw cutoff equals a 5% corrected cutoff

🔍 Key Nuances & Common Traps to Watch For

Rebalancing Changes the Intercept, Not the Ranking: In a single-signal logistic regression, undersampling non-churners shifts the log-odds upward by a constant while leaving customer ordering (and holdout AUC) identical.
Why Uncorrected Rebalanced Scores Are Dangerous for ROI: If a model trained on a 50/50 sample predicts 50% churn for an average customer whose true risk is 5%, multiplying that 50% score by customer LTV overstates expected campaign value by 10×!
Never Rebalance the Holdout Set: Rebalancing is a training-only intervention. Evaluating a model on an artificially balanced test set fabricates fake precision and hides false-positive outreach volume.

📐 Under the Hood: Model Equations & Variable Definitions

1. King & Zeng (2001) Prior-Correction Intercept Shift & Probability Mapping

\[ \tilde{\beta}_0 = \hat{\beta}_0 - \ln\!\left[\left(\frac{1 - \tau}{\tau}\right)\left(\frac{s}{1 - s}\right)\right], \qquad \tilde{p}_i = \frac{\hat{p}_i}{\hat{p}_i + (1 - \hat{p}_i)\,\frac{s(1 - \tau)}{(1 - s)\tau}} \]

💬 Read it in plain English: Because training had positive fraction s instead of the true population rate τ, we subtract the log-odds ratio of training-to-population priors from the intercept (or rescale raw probability p̂_i by the odds factor) to put risks back on the real population scale.

True Population Event RateYOUR DATA ⓘ

Actual rare-event base rate in the deployment population (e.g., τ = 0.05).

Rebalanced Training Positive RateYOUR DATA ⓘ

Actual fraction of positive cases in the rebalanced training sample (e.g., s = 0.50).

β̂₀, β̃₀

Raw & Prior-Corrected InterceptESTIMATED ⓘ

Fitted logistic intercept on the rebalanced sample and its prior-corrected population intercept.

p̂_i, p̃_i

Raw & Calibrated ProbabilitiesOUTPUT ⓘ

Uncorrected training-scale score p̂_i and recalibrated population probability p̃_i for customer i.

2. Brier Score & Expected Calibration Error (ECE)

\[ \text{Brier} = \frac{1}{N_{\text{hold}}}\sum_{i=1}^{N_{\text{hold}}} (y_i - p_i)^2, \qquad \text{ECE} = \sum_{b=1}^{B} \frac{n_b}{N_{\text{hold}}} \left|\bar{y}_b - \bar{p}_b\right| \]

💬 Read it in plain English: The Brier score averages squared probability errors across all holdout customers, while ECE averages the absolute gap between mean predicted risk p̄_b and actual observed churn ȳ_b across B score groups.

y_i, N_hold

Holdout Outcome & Sample SizeYOUR DATA ⓘ

Actual 0/1 outcome for holdout customer i and total untouched holdout count.

Brier

Brier Probability Error ScoreINFERENCE ⓘ

Mean squared error of probability predictions (lower indicates better calibration + discrimination).

ECE

Expected Calibration ErrorINFERENCE ⓘ

Sample-weighted average calibration gap across B binned risk groups.

Run a population

Each simulated customer has one continuous risk signal from 0 to 100 and a binary outcome: churn (1) or stay (0). Split first, rebalance training only, and evaluate every model on the same untouched holdout.

Population churn rate

Total customers

1. Compare the rankings

AUC summarizes ordering across cutoffs. The contact counts below depend on the cutoff you choose in step 3.

Apply the prior correction

The selected probability scale applies to every condition card, the calibration curve, and the decision chart. The raw/corrected comparison chart and the detailed table always show both scales.

2. Put the probabilities back on the population scale

Read the curve: horizontal = predicted risk; vertical = observed churn. Points below the diagonal mean predictions are too high. Points above it mean predictions are too low.

Customers are sorted into about ten similar-sized score groups, keeping tied scores together. Whiskers show 95% Wilson intervals for each group's observed churn rate, conditional on the fitted model. They do not include model-fitting uncertainty or provide a simultaneous test of calibration.

Read the calibration groups as a table

Condition	Customers	Churners	Predicted	Observed	95% interval

Check the overall average too

The dashed line is the population setting. Raw and corrected means are always shown together here. A matching average alone does not establish calibration across customers.

3. Choose whom to contact

For churn outreach, 50% is a software default rather than a business rule. Compare the churners you catch with the non-churners you would also contact.

Contact cutoff (% on the displayed scale)

Changing the cutoff does not refit the model or improve its ranking. Correcting the scores preserves ranking, but keeping the same numeric cutoff can change the contact list.

Inspect the evidence

Detailed metrics: both probability scales

Brier measures overall probability prediction error, combining calibration and discrimination. ECE measures grouped calibration gaps; it depends on grouping and is noisy with rare events. A lower value on one run is not a guarantee for another run.

Condition	Train N	Train churn	AUC	Brier raw	Brier corrected	ECE raw	ECE corrected

Why the correction works

Formula, assumptions, and limits

The prior correction adjusts the odds by the ratio of population odds to training odds. Let s be the actual positive fraction in the rebalanced training set and τ the population event rate:

p_corrected = p_raw / [p_raw + (1 − p_raw) × s(1 − τ) / ((1 − s)τ)]

Equivalently, add logit(τ) − logit(s) to the fitted intercept. The transformation is increasing, so it preserves customer ranking. The natural model is left unchanged because it was sampled from the population without class rebalancing.

These simulations fit an unpenalized, correctly specified logistic model using the shared converged fitter. Training and holdout observations still vary from sample to sample. A correction removes the sampling-prior shift; it cannot remove finite-sample error, misspecification, or overfitting.

The formula assumes random selection within each outcome class. It does not provide a general guarantee for SMOTE, feature-dependent sampling, every model family, or every regularization scheme. Validate on data that represent the deployment population.

Read King and Zeng (2001), section 4.1, and the guide to choosing decision thresholds.

Professor Mode: Guided Learning