Three models all pass the holdout test. Now which one ships? Accuracy collapses on rare outcomes, AUC compares rankings while precision–recall describes the campaign, and a model can rank beautifully while lying about magnitudes. Learn to compute the baseline first, compare fairly, check calibration before multiplying probabilities by money, and choose the metric from the decision.
Topics and managerial frameworks covered in Model Comparison and Judgment:
Students manipulate these interactive visual simulations directly inside the chapter reading before answering gated check-in questions:
Students set the base rate and the model's per-class detection ability, then watch the majority-class baseline, the model's accuracy, and balanced accuracy move against each other. The point being made: as the outcome gets rarer, the do-nothing baseline ris...
Holds the model's discriminative ability FIXED and varies only the base rate, so students see the central asymmetry: the ROC curve does not move at all, while the precision-recall curve collapses as the positive class gets rarer.
Applies a distortion to a set of well-calibrated probabilities and draws the resulting reliability diagram. The distortion is MONOTONIC, so the ranking — and therefore AUC — is mathematically unchanged no matter how far the slider moves.
This chapter connects directly to these full MktPraxis analytical apps (with pre-loaded business case scenarios, CSV upload, and reproducible Python export):
Compare binary classification algorithms side-by-side on training and holdout data. Select from Logistic Regression, Decision Tree, k-NN, Naive Bayes, and Linear SVM.
Pre-loaded Business Scenarios: CloudMetrics SaaS — Customer Churn Prediction · NexaFlow — B2B Lead Conversion Scoring · PixelReach Ads — Click Prediction for Programmatic Bidding
Manually calibrate logistic regression by minimizing log-loss.
Pre-loaded Business Scenarios: Email Campaign Conversion - StyleBox Fashion · Customer Churn Analysis - StreamFlix · Display Ad Click-Through - TechDeals · Premium Upgrade Prediction - CloudTools SaaS · Luxury EV Test Drive Booking - PeakAuto
Discover how oversampling the rare class during model training inflates predicted probabilities, and apply the King-Zeng correction to restore calibration.
Watch a prediction model degrade over time as customer data shifts. Diagnose feature drift with PSI and KS statistics, retrain to recover, and learn when to set monitoring alerts.
Pre-loaded Business Scenarios: BrightCart — E-Commerce Customer Churn · AdVantage — Digital Ad Click-Through Rate · FreshBrew — Coffee Subscription Retention · MediaMax — B2B SaaS Lead Scoring
Interactively explore how the classification threshold affects profit vs. cost trade-offs in binary ML models. Find the optimal T* that maximises net profit.
Pre-loaded Business Scenarios: FreshBox — DTC Snack Subscription · StreamFlow — Streaming Service · Kiteline Analytics — Enterprise-Tier B2B Analytics SaaS