Classification Threshold Lab
Discover why the default 50% decision cutoff almost never maximises profit in marketing. Set error costs and revenue, drag the threshold, and watch net profit rise — or fall — in real time.
OVERVIEW & LEARNING OBJECTIVES
Most classification models output a probability score — the likelihood a customer will churn, a lead will convert, or a transaction is fraudulent. Turning that score into a decision requires picking a threshold. By default, analysts use 0.50. That is almost always wrong.
When acting on a false alarm costs budget and missing a real event costs revenue, the profit-maximising threshold shifts — sometimes dramatically. This lab shows you where it goes and exactly why.
🎯 What You'll Learn
- The 50/50 myth: See how net profit at T = 0.50 compares to the profit-maximising threshold.
- Concrete payoffs: Understand how false-positive costs, false-negative costs, and revenue each pull T* in different directions.
- The profit curve: Visualise net profit across all possible thresholds — the peak reveals the optimal threshold.
- The formula — and its limits: Learn the simple derivation, and discover when real data disagrees with it (calibration).
- The marketing reality: Customer models are routinely miscalibrated — using the empirical peak is almost always safer than the formula alone.
💡 Why This Matters: Churn models, lead-scoring systems, fraud detectors — every binary classifier in marketing has asymmetric error costs. Defaulting to 0.50 leaves money on the table in every single one.
Use a Case Study
Select a case study above to load data and see the business context.
Upload Your Data
Drag & Drop scored data file (.csv, .tsv, .txt, .xls, .xlsx)
Needs columns: predicted_prob (0–1 score) and
actual_outcome (0 = negative, 1 = positive).
👆 Load a case study or upload your own scored dataset to begin the analysis.
SEE YOUR DATA
Each dot is one customer. Top half = actual positives (the people you want to catch). Bottom half = actual negatives (the people you don't want to waste budget on). The model has given every customer a score between 0 and 1. Your job is to decide which score is the cutpoint for action.
Dots change colour as you move the threshold: Green TP = correctly flagged, Amber FN = missed, Red FP = false alarm, Grey TN = correctly ignored. Drag the blue handle to move the threshold. Once you press ⚡ Jump to Optimal T*, a gold dashed line marks the profit-optimal T*.
📊 How to Read This Chart
- Top half (Positives ↑) — actual positives stacked upward. Green = TP (caught), Amber = FN (missed). You want as many green as possible.
- Bottom half (Negatives ↓) — actual negatives stacked downward. Grey = TN (ignored, good), Red = FP (false alarm, costs budget). Minimise red.
- Drag threshold left → wider net → more green TPs but more red FPs. Drag right → fewer red FPs but more amber FNs missed.
- The gold dashed line is T* — the profit-maximising threshold. It stays hidden until you click ⚡ Jump to Optimal T*, so you can predict it first.
- When N > 120, a representative sample is shown. Every dot stands for several real customers — the colour pattern still reflects the full dataset.
MOVE THE LINE
The threshold is your decision rule. Drag it left and right — watch how the four quadrants of the confusion matrix fill and empty in real time. Every threshold is a different policy: how much risk are you willing to take on each type of error?
DECISION THRESHOLD
Customers with score ≥ threshold are flagged for action.
🤔 Try this: Drag to T = 0.20. Notice the confusion matrix — many more Green TPs, but also many more Red FPs. Now drag to T = 0.80. What happens? This trade-off is the whole game.
CONFUSION MATRIX
At your current threshold, predictions fall into four categories:
(Flagged) Predicted Negative
(Ignored)
Positive
Negative
SET THE STAKES
Not all errors cost the same. An unnecessary email to the wrong customer costs you $8 in send cost. Missing a churner you could have saved might cost $200 in lost lifetime value. Enter your real costs — and optionally the gross revenue per successful intervention — and watch the economics update live.
COST & REVENUE INPUTS
⚙️ Advanced: Intervention Flip Rate (α)
COST & PROFIT SUMMARY
Economic impact at your current threshold:
💡 How the Economics Work
- Cost-only (R = 0): CFP penalises every false alarm; CFN penalises every miss. The tool minimises total cost.
- With revenue (R > 0): R is gross revenue per true positive. CFP is charged on every flagged customer — TP and FP alike. Net value per TP = R − CFP. The tool maximises net profit.
- Flip rate (α): When only a fraction of your TPs actually change behaviour, effective revenue per TP = α × R. Lower α → T* rises.
FIND THE PEAK
The chart below plots your outcome metric at every possible threshold. When R = 0 it shows total cost — seek the minimum. When R > 0 it switches to net profit — seek the peak. The jump button lands you exactly on the optimal point.
After the jump, the green diamond ◆ marks T* (the optimal point). The purple dot ● is your current threshold.
📊 Reading the Curve
- Left end (T → 0): Flag everyone — zero FN but maximum FP waste.
- Right end (T → 1): Flag no one — zero FP but maximum FN losses.
- Cost mode (R = 0): U-shaped curve; minimum at T* = CFP / (CFP + CFN).
- Profit mode (R > 0): Inverted-U; maximum at T* = CFP / (R + CFN). Adding revenue pushes T* left — be more aggressive.
- 💡 Case 3 insight: Cost-only gives T* ≈ 0.91. With $80K deal revenue, profit-max T* ≈ 0.10. Revenue transforms the strategy.
THE FORMULA REVEALED
Now that you've seen the peak on real data — here's why it's there. The optimal threshold follows directly from a simple expected-value calculation. And if the formula disagrees with your data, that disagreement is itself a lesson.
Predict where the optimum sits, then press ⚡ Jump to Optimal T* above: the optimal threshold, the formula and the calibration check appear here.
📐 The Derivation — Where T* Comes From
At any threshold T, a customer with true probability p generates either a correct decision or an error:
Expected cost of flagging (acting):
$$\text{E}[\text{cost} \mid \text{act}] = (1-p) \cdot C_{FP}$$Expected cost of not flagging (ignoring):
$$\text{E}[\text{cost} \mid \text{ignore}] = p \cdot C_{FN}$$Set them equal. The threshold where acting and ignoring cost exactly the same in expectation is T*:
Cost-only:
$$T^* = \frac{C_{FP}}{C_{FP} + C_{FN}}$$With Revenue (Profit-max):
$$T^* = \frac{C_{FP}}{R + C_{FN}}$$Act when: \(p \cdot R - C_{FP} > -p \cdot C_{FN}\) ⇒ \(p(R + C_{FN}) > C_{FP}\) ⇒ \(p > \dfrac{C_{FP}}{R + C_{FN}}\)
Generalised — Partial Flip Rate (α):
$$T^* = \frac{C_{FP}}{\alpha \cdot R + C_{FN}}$$When only fraction α of flagged TPs actually change behaviour, effective revenue per TP = α·R. As α decreases, T* rises.
If \(C_{FP} \ll C_{FN}\) or R is large → \(T^*\) is small → cast a wide net.
If \(C_{FP} \gg R + C_{FN}\) → \(T^*\) is large → be very selective.
⚠️ Critical caveat — calibration: This formula assumes scores are perfectly calibrated (the model's score = the true probability the customer is positive). In real marketing data, this is almost never fully true. Always prefer the empirical T* (scanned from the profit curve) over the formula when they differ.
| Error Type | Who is Flagged? | Consequence | Cost |
|---|---|---|---|
| False Positive (FP) | Flagged as positive; actually negative | Wasted intervention on someone who didn't need it | \(C_{FP}\) |
| False Negative (FN) | Not flagged; actually positive | Missed opportunity, lost revenue, or undetected risk | \(C_{FN}\) |
EXPLORE FURTHER
🤔 Work Through These
- The Default Trap: Load Case 2 (StreamFlow streaming service). What is net profit at the default threshold of 0.50? Now click ⚡ Jump to Optimal T*. How much additional profit do you capture? Why isn't the default threshold optimal?
- Cost Sensitivity: Load Case 1 (email retention — cost-only mode). Double the FP cost from $8 to $16. What happens to the optimal threshold? In which direction does it shift? Why?
- Comparing Cases: Load all three case studies one at a time. Note the optimal threshold in each. What's the pattern? How do the cost and revenue parameters together determine where the profit-maximising threshold ends up?
- The Profit Curve: In Case 3 (B2B sales with $80K deal revenue), look at the net profit curve. Why does profit rise as the threshold lowers, then fall off? Where exactly is the peak, and does it match the profit formula? Now set Revenue = 0 — how does the curve shape and optimal threshold change?
- Calibration Gap: Does the formula T* agree with the empirical T* on your data? If they differ significantly, what does that tell you about model calibration? Why does miscalibration matter more when the formula T* is very low (e.g., 0.05)?
- Operational Reality: In Case 3, the profit-optimal threshold is ~0.10. How many leads would you pitch at 0.10 vs. 0.50 vs. the cost-only threshold (~0.91)? If your team can only handle 20 pitches per quarter, does that constraint change your threshold choice?
📚 Connecting to the Bigger Picture
⚖️ Calibration in Marketing
Models trained on imbalanced marketing data (churn, response, conversion) are routinely miscalibrated. Always verify calibration before trusting the formula threshold directly.
📈 Precision-Recall Tradeoff
Every threshold shift moves precision and recall in opposite directions. Higher recall = more positives caught = lower threshold. Higher precision = fewer false alarms = higher threshold.
🔄 Re-evaluate Over Time
Costs change. An email that cost $8 to send today may cost $5 next year. The optimal threshold should be recalculated whenever cost assumptions change materially.
🎯 Beyond Binary Decisions
When you have multiple intervention tiers (low-cost email, medium-cost call, high-cost discount), you're no longer choosing one threshold but two or more. Expected-value optimisation extends naturally.