Classification Threshold Lab

Model Fitting Decision Thresholds

Discover why the default 50% decision cutoff almost never maximises profit in marketing. Set error costs and revenue, drag the threshold, and watch net profit rise — or fall — in real time.

OVERVIEW & WHAT THIS TOOL SOLVES

Every churn, lead-scoring, or fraud model outputs a continuous probability between 0 and 1, yet turning that score into an action list requires picking a decision threshold (T). Leaving the cutoff at the software default of 0.50 implicitly assumes a false positive costs the exact same dollar amount as a false negative—which is almost never true in marketing. The Classification Threshold Lab links confusion-matrix trade-offs directly to unit economics to find the exact profit-maximizing cutoff T*.

🎯 Questions It Answers

What probability cutoff T* maximizes net campaign profit given our outreach cost, save rate, and customer lifetime value?
How much money do we leave on the table if we use the default T = 0.50 or optimize a purely statistical metric like F1 or Youden's J?
What is the best threshold when our call center or sales team has a hard capacity cap (e.g., can only contact the top 15% of leads)?

📂 What Data You Need

Scored Holdout Dataset: Preset marketing cases (SaaS churn, B2B lead scoring, promo targeting) or uploaded CSV of actual (0/1) and predicted_prob
Payoff Matrix (Unit Economics): Net value of a True Positive (V_{TP}), cost of a False Positive (C_{FP}), and cost of a False Negative
Capacity Constraint (Optional): Maximum percentage or count of customers your team can contact

💡 What You'll Discover

Profit Curve & Optimal Cutoff T*: Expected net dollar payoff across every threshold from 0.01 to 0.99
Live Confusion Matrix & Metric Trade-Offs: How Precision, Recall, Specificity, and F1 shift as you drag T
Multi-Rule Comparison Table: Side-by-side profit and outreach volume at T* (Max Profit), Max F1, Max Youden's J, and T = 0.50

🔍 Key Nuances & Common Traps to Watch For

Why T = 0.50 Is Almost Always Wrong: If sending a $10 retention offer has a 30% chance of saving a $600 subscriber (expected TP benefit = $170) while wasting the offer on a loyal customer costs $10, you should contact anyone with churn risk above 10 / (170 + 10) ≈ 5.6%—not 50%!
Changing the Threshold Does Not Refit the Model: Dragging T moves your operating point along the exact same ROC and Precision–Recall curves without changing customer rankings or AUC.
Calibration Prerequisite: The theoretical break-even threshold formula assumes your model's probabilities are calibrated to the true population rate (if you oversampled training data, recalibrate first in the Rare Event Calibration Lab).

📐 Under the Hood: Model Equations & Variable Definitions

1. Expected Campaign Profit Curve & Theoretical Break-Even Threshold (T*)

\[ \Pi(T) = N_{\text{TP}}(T)\cdot V_{\text{TP}} - N_{\text{FP}}(T)\cdot C_{\text{FP}} - N_{\text{FN}}(T)\cdot C_{\text{FN}}, \qquad T^* = \frac{C_{\text{FP}}}{V_{\text{TP}} + C_{\text{FP}}} \]

💬 Read it in plain English: Total campaign profit Π(T) sums the net gain from true positives minus the wasted outreach cost on false positives, and the marginal customer is worth contacting whenever their calibrated probability exceeds the cost-to-payoff ratio C_FP / (V_TP + C_FP).

T, T^*

Decision Cutoff & Profit-Optimal T*YOUR DATA ⓘ

Probability threshold above which a customer is flagged positive (ŷ = 1), and the profit-maximizing cutoff.

N_TP(T), N_FP(T), N_FN(T)

Confusion Matrix Counts at TOUTPUT ⓘ

Counts of True Positives, False Positives, and False Negatives produced at cutoff T.

V_TP, C_FP, C_FN

Unit Economic Payoffs & CostsYOUR DATA ⓘ

Net dollar benefit of catching a true positive and unit costs of false-positive and false-negative errors.

Π(T)

Net Campaign Profit at Cutoff TOUTPUT ⓘ

Total expected dollar payoff across the scored cohort when operating at threshold T.

2. Precision, Recall (Sensitivity), F1 Score & Youden's J Statistic

\[ \text{Precision}(T) = \frac{\text{TP}}{\text{TP} + \text{FP}}, \qquad \text{Recall}(T) = \frac{\text{TP}}{\text{TP} + \text{FN}}, \qquad F_1(T) = \frac{2\cdot\text{Prec}\cdot\text{Rec}}{\text{Prec} + \text{Rec}}, \qquad J(T) = \text{TPR}(T) - \text{FPR}(T) \]

💬 Read it in plain English: Lowering T catches more true positives (higher Recall/TPR) at the cost of flagging more false alarms (lower Precision, higher FPR); F1 and Youden's J balance these rates without knowing dollar costs.

Precision(T)

Positive Predictive ValueOUTPUT ⓘ

Share of contacted customers at threshold T who are actual positive cases.

Recall(T) / TPR

Sensitivity / True Positive RateOUTPUT ⓘ

Share of all actual positive cases successfully caught at threshold T.

F₁(T), J(T)

Harmonic Mean & Youden’s IndexINFERENCE ⓘ

Cost-agnostic statistical criteria balancing Precision/Recall (F₁) or Sensitivity/Specificity (J).

Load a marketing use case:

Select a case study above to load data and see the business context.

Drag & Drop scored data file (.csv, .tsv, .txt, .xls, .xlsx)

Needs columns: predicted_prob (0–1 score) and actual_outcome (0 = negative, 1 = positive).

No file uploaded.

👆 Load a case study or upload your own scored dataset to begin the analysis.

Step 1

SEE YOUR DATA

Each dot is one customer. Top half = actual positives (the people you want to catch). Bottom half = actual negatives (the people you don't want to waste budget on). The model has given every customer a score between 0 and 1. Your job is to decide which score is the cutpoint for action.

Dots change colour as you move the threshold: Green TP = correctly flagged, Amber FN = missed, Red FP = false alarm, Grey TN = correctly ignored. Drag the blue handle to move the threshold. Once you press ⚡ Jump to Optimal T*, a gold dashed line marks the profit-optimal T*.

📊 How to Read This Chart

Top half (Positives ↑) — actual positives stacked upward. Green = TP (caught), Amber = FN (missed). You want as many green as possible.
Bottom half (Negatives ↓) — actual negatives stacked downward. Grey = TN (ignored, good), Red = FP (false alarm, costs budget). Minimise red.
Drag threshold left → wider net → more green TPs but more red FPs. Drag right → fewer red FPs but more amber FNs missed.
The gold dashed line is T* — the profit-maximising threshold. It stays hidden until you click ⚡ Jump to Optimal T*, so you can predict it first.
When N > 120, a representative sample is shown. Every dot stands for several real customers — the colour pattern still reflects the full dataset.

Step 2

MOVE THE LINE

The threshold is your decision rule. Drag it left and right — watch how the four quadrants of the confusion matrix fill and empty in real time. Every threshold is a different policy: how much risk are you willing to take on each type of error?

DECISION THRESHOLD

Customers with score ≥ threshold are flagged for action.

0.00
Flag everyone 1.00
Flag no-one

Current threshold: 0.50

🤔 Try this: Drag to T = 0.20. Notice the confusion matrix — many more Green TPs, but also many more Red FPs. Now drag to T = 0.80. What happens? This trade-off is the whole game.

CONFUSION MATRIX

At your current threshold, predictions fall into four categories:

Predicted Positive
(Flagged) Predicted Negative
(Ignored)

Actual
Positive

True Positive (TP)

—

Correctly flagged — earns R, pays C_FP ✅

False Negative (FN)

—

Missed positive — FN cost applies

Actual
Negative

False Positive (FP)

—

Wasted action — pays C_FP, no R earned

True Negative (TN)

—

Correctly ignored ✅

Precision —

Recall —

Step 3

SET THE STAKES

Not all errors cost the same. An unnecessary email to the wrong customer costs you $8 in send cost. Missing a churner you could have saved might cost $200 in lost lifetime value. Enter your real costs — and optionally the gross revenue per successful intervention — and watch the economics update live.

COST & REVENUE INPUTS

Action Cost per Contact (C_FP) Intervention cost charged on every flagged customer — TPs and FPs alike. In profit mode this reduces each TP's net value (net per TP = R − C_FP). In cost-only mode it serves as the false-alarm penalty.

False Negative Cost (C_FN) Cost of missing someone who needed the intervention — lost revenue, undetected risk, or missed opportunity.

Gross Revenue per True Positive (R) Gross value earned from each correctly-flagged positive, before deducting the action cost. Net value per TP = R − C_FP. Set to 0 for cost-only mode. Assumes 100% intervention success — use the Advanced toggle below to model a partial flip rate (α).

⚙️ Advanced: Intervention Flip Rate (α)

Intervention Flip Rate (α) What % of contacted true positives actually change behaviour due to your intervention? Default = 100% (every flagged churner is saved). At 50%, only half your TPs generate the full revenue R — effective revenue per TP = α × R. This raises the profit-optimal threshold. Formula: T* = C_FP ÷ (α·R + C_FN).

COST & PROFIT SUMMARY

Economic impact at your current threshold:

False Positive Costs

— flagged incorrectly

—

False Negatives

— missed

—

Total Costs

—

💡 How the Economics Work

Cost-only (R = 0): C_FP penalises every false alarm; C_FN penalises every miss. The tool minimises total cost.
With revenue (R > 0): R is gross revenue per true positive. C_FP is charged on every flagged customer — TP and FP alike. Net value per TP = R − C_FP. The tool maximises net profit.
Flip rate (α): When only a fraction of your TPs actually change behaviour, effective revenue per TP = α × R. Lower α → T* rises.

Step 4

FIND THE PEAK

The chart below plots your outcome metric at every possible threshold. When R = 0 it shows total cost — seek the minimum. When R > 0 it switches to net profit — seek the peak. The jump button lands you exactly on the optimal point.

After the jump, the green diamond ◆ marks T* (the optimal point). The purple dot ● is your current threshold.

📊 Reading the Curve

Left end (T → 0): Flag everyone — zero FN but maximum FP waste.
Right end (T → 1): Flag no one — zero FP but maximum FN losses.
Cost mode (R = 0): U-shaped curve; minimum at T* = C_FP / (C_FP + C_FN).
Profit mode (R > 0): Inverted-U; maximum at T* = C_FP / (R + C_FN). Adding revenue pushes T* left — be more aggressive.
💡 Case 3 insight: Cost-only gives T* ≈ 0.91. With $80K deal revenue, profit-max T* ≈ 0.10. Revenue transforms the strategy.

Step 5

THE FORMULA REVEALED

Now that you've seen the peak on real data — here's why it's there. The optimal threshold follows directly from a simple expected-value calculation. And if the formula disagrees with your data, that disagreement is itself a lesson.

Predict where the optimum sits, then press ⚡ Jump to Optimal T* above: the optimal threshold, the formula and the calibration check appear here.

📐 The Derivation — Where T* Comes From

At any threshold T, a customer with true probability p generates either a correct decision or an error:

Expected cost of flagging (acting):

$$\text{E}[\text{cost} \mid \text{act}] = (1-p) \cdot C_{FP}$$

Expected cost of not flagging (ignoring):

$$\text{E}[\text{cost} \mid \text{ignore}] = p \cdot C_{FN}$$

Set them equal. The threshold where acting and ignoring cost exactly the same in expectation is T*:

Cost-only:

$$T^* = \frac{C_{FP}}{C_{FP} + C_{FN}}$$

With Revenue (Profit-max):

$$T^* = \frac{C_{FP}}{R + C_{FN}}$$

Act when: $p \cdot R - C_{FP} > -p \cdot C_{FN}$ ⇒ $p(R + C_{FN}) > C_{FP}$ ⇒ $p > \dfrac{C_{FP}}{R + C_{FN}}$

Generalised — Partial Flip Rate (α):

$$T^* = \frac{C_{FP}}{\alpha \cdot R + C_{FN}}$$

When only fraction α of flagged TPs actually change behaviour, effective revenue per TP = α·R. As α decreases, T* rises.

If $C_{FP} \ll C_{FN}$ or R is large → $T^*$ is small → cast a wide net.
If $C_{FP} \gg R + C_{FN}$ → $T^*$ is large → be very selective.

⚠️ Critical caveat — calibration: This formula assumes scores are perfectly calibrated (the model's score = the true probability the customer is positive). In real marketing data, this is almost never fully true. Always prefer the empirical T* (scanned from the net-profit curve, or the total-cost curve when revenue is $0) over the formula when they differ.

Error Type	Who is Flagged?	Consequence	Cost
False Positive (FP)	Flagged as positive; actually negative	Wasted intervention on someone who didn't need it	$C_{FP}$
False Negative (FN)	Not flagged; actually positive	Missed opportunity, lost revenue, or undetected risk	$C_{FN}$

EXPLORE FURTHER

🤔 Work Through These

The Default Trap: Load Case 2 (StreamFlow streaming service). What is net profit at the default threshold of 0.50? Now click ⚡ Jump to Optimal T*. How much additional profit do you capture? Why isn't the default threshold optimal?
Cost Sensitivity: Load Case 1 (email retention — cost-only mode). Double the FP cost from $8 to $16. What happens to the optimal threshold? In which direction does it shift? Why?
Comparing Cases: Load all three case studies one at a time. Note the optimal threshold in each. What's the pattern? How do the cost and revenue parameters together determine where the profit-maximising threshold ends up?
The Profit Curve: In Case 3 (B2B sales with $80K deal revenue), look at the net profit curve. Why does profit rise as the threshold lowers, then fall off? Where exactly is the peak, and does it match the profit formula? Now set Revenue = 0 — how does the curve shape and optimal threshold change?
Calibration Gap: Does the formula T* agree with the empirical T* on your data? If they differ significantly, what does that tell you about model calibration? Why does miscalibration matter more when the formula T* is very low (e.g., 0.05)?
Operational Reality: In Case 3, the profit-optimal threshold is ~0.10. How many leads would you pitch at 0.10 vs. 0.50 vs. the cost-only threshold (~0.91)? If your team can only handle 20 pitches per quarter, does that constraint change your threshold choice?

📚 Connecting to the Bigger Picture

⚖️ Calibration in Marketing

Models trained on imbalanced marketing data (churn, response, conversion) are routinely miscalibrated. Always verify calibration before trusting the formula threshold directly.

📈 Precision-Recall Tradeoff

Every threshold shift moves precision and recall in opposite directions. Higher recall = more positives caught = lower threshold. Higher precision = fewer false alarms = higher threshold.

🔄 Re-evaluate Over Time

Costs change. An email that cost $8 to send today may cost $5 next year. The optimal threshold should be recalculated whenever cost assumptions change materially.

🎯 Beyond Binary Decisions

When you have multiple intervention tiers (low-cost email, medium-cost call, high-cost discount), you're no longer choosing one threshold but two or more. Expected-value optimisation extends naturally.