70 / 30
Feature Selection
Select continuous columns to use as predictors. All selected features must be numeric.
Classifier Comparison Lab
Model Fitting
Binary Classification
Compare five binary classification algorithms side-by-side on the same data. See how algorithm choice alone changes accuracy, precision, recall, and ROC curves — then examine the holdout set to spot which models overfit.
OVERVIEW & WHAT THIS TOOL SOLVES
No single classification algorithm wins on every marketing problem: Logistic Regression excels when log-odds are linear, Decision Trees capture rectangular interaction thresholds, and k-Nearest Neighbors adapts to local pockets—yet flexible models easily overfit training data. The Classifier Comparison Lab trains five classic classifiers side-by-side on the exact same train/holdout split so you can compare their decision boundaries, ROC curves, and out-of-sample generalization.
🎯 Questions It Answers
- Which classifier (Logistic Regression, Decision Tree, k-NN, Gaussian Naive Bayes, or Linear SVM) generalizes best on unseen holdout data?
- How do the five algorithms' 2D decision boundaries differ when the true pattern is linear, curved, XOR-interacted, or noisy?
- Which model shows a dangerous gap between training accuracy/AUC and holdout test accuracy/AUC (overfitting)?
📂 What Data You Need
- Dataset Selection: Built-in marketing classification scenarios (varying boundary geometry and noise) or custom CSV upload
- Model Selection & Hyperparameters: Toggle up to 5 classifiers and tune Tree max depth, k-NN neighbors
k, and SVM penaltyC - Evaluation Controls: Train/Holdout split ratio and random seed (every model classifies at a fixed 0.50 probability cutoff)
💡 What You'll Discover
- Side-by-Side Decision Boundary Maps: Visual probability contours showing how each model carves up predictor space
- Train vs. Holdout Leaderboard: Accuracy, Precision, Recall, F1, Specificity, ROC-AUC, and Log-Loss
- Overlaid ROC Curves: Direct visual comparison of each model's ranking discrimination on the holdout set
🔍 Key Nuances & Common Traps to Watch For
- The No-Free-Lunch Principle: A 1-nearest-neighbor model or deep decision tree can score 100% on training rows while failing on holdout customers because its jagged boundary memorized random noise. Always rank models by Holdout AUC / Log-Loss.
- Geometry Matches Assumptions: Logistic Regression and Linear SVM draw straight hyperplanes; Decision Trees draw axis-aligned boxes; Naive Bayes draws elliptical contours; k-NN follows local neighborhood density.
- Threshold Sensitivity vs. AUC: Accuracy, Precision, and F1 depend on the probability threshold (fixed at 0.50 in this lab), whereas ROC-AUC evaluates ranking quality across all possible cutoffs.
📐 Under the Hood: Model Equations & Variable Definitions
1. Parametric & Margin Classifiers (Logistic Regression, Naive Bayes, Linear SVM)
\[ P_{\text{Logit}}(y=1\mid\mathbf{x}) = \sigma(\mathbf{w}^\top\mathbf{x} + b), \qquad P_{\text{NB}}(y=1\mid\mathbf{x}) \propto \pi_1 \prod_{j=1}^{P} \mathcal{N}(x_j \mid \mu_{1j}, \sigma_{1j}^2), \qquad f_{\text{SVM}}(\mathbf{x}) = \mathbf{w}^\top\mathbf{x} + b \]
💬 Read it in plain English: Logistic Regression and Linear SVM separate classes with a weighted linear score wᵀx + b, while Gaussian Naive Bayes multiplies the baseline class prior π₁ by each feature's independent normal density.
x, y
Feature Vector & Binary ClassYOUR DATA ⓘ
Predictor vector x = (x₁, …, x_P) and binary target label y ∈ {0, 1}.
w, b
Hyperplane Weights & InterceptESTIMATED ⓘ
Learned linear boundary coefficients separating positive and negative classes.
π1, μ1j, σ1j2
Naive Bayes Prior & Class MomentsESTIMATED ⓘ
Class-1 base rate and feature-specific mean and variance assuming conditional independence.
2. Non-Parametric Classifiers (k-Nearest Neighbors & Decision Tree Gini Split)
\[ \hat{P}_{\text{kNN}}(y=1\mid\mathbf{x}) = \frac{1}{k}\sum_{i \in \mathcal{N}_k(\mathbf{x})} y_i, \qquad G(t) = 1 - \sum_{c \in \{0,1\}} p_{tc}^2 = 2\,p_{t1}(1 - p_{t1}) \]
💬 Read it in plain English: k-NN predicts the positive class probability from the fraction of positive cases among the k closest training points N_k(x), while a Decision Tree splits nodes to minimize Gini impurity G(t).
k, Nk(x)
Neighbor Count & NeighborhoodYOUR DATA ⓘ
Hyperparameter k and the set of k nearest training observations in Euclidean distance.
pt1, G(t)
Node Class Proportion & GiniESTIMATED ⓘ
Positive class fraction in tree node t and its Gini impurity index G(t).
AUC, Log Loss
Holdout Ranking & Probability PenaltyINFERENCE ⓘ
Area Under the ROC Curve and the average log penalty −1/N ∑ [y_i ln p̂_i + (1 − y_i) ln(1 − p̂_i)] on holdout data.
DATA SOURCE
📚
Use a Case Study
Each scenario has different distributional characteristics designed to show that different algorithms excel on different problems. Try all three!
📤
Upload Your Data
Upload numeric CSV data
Provide a header row with numeric feature columns and a binary target column (0/1). Up to 2,000 rows recommended for responsive performance.
Drag & Drop raw data file (.csv, .tsv, .txt, .xls, .xlsx)
Include headers. All feature columns must be numeric. Target column must be binary (0/1).
No file uploaded.
RESULTS
Training Set Performance
These metrics reflect how well each model fits the data it was trained on.
| Metric |
|---|
CONFUSION MATRICES (Holdout Set)
Compare true/false positive and negative counts across all models side-by-side on holdout data.
ROC CURVES (Holdout Set)
The ROC curve plots the trade-off between true positive rate and false positive rate across all possible classification thresholds. A curve closer to the top-left corner indicates better discrimination.
DECISION BOUNDARIES
Pick any two of your selected features below. Each algorithm is refit on just those two features so its decision regions can be drawn on a 2D grid. Actual data points are overlaid (circles = class 0, triangles = class 1).
KEY INSIGHTS
Auto-generated observations based on your comparison results. These highlight the most important patterns to notice.
INTERPRETATION GUIDE
What the Metrics Mean
| Metric | Meaning | Range |
|---|---|---|
| Accuracy | Proportion of all predictions that are correct | 0 – 1 (higher is better) |
| Precision | Of predicted positives, how many are truly positive | 0 – 1 (higher = fewer false alarms) |
| Recall | Of actual positives, how many were detected | 0 – 1 (higher = fewer missed cases) |
| F1 Score | Harmonic mean of precision and recall | 0 – 1 (balanced measure) |
| Specificity | Of actual negatives, how many were correctly identified | 0 – 1 |
| AUC | Area under the ROC curve; overall discrimination ability | 0.5 (random) – 1.0 (perfect) |
| Log Loss | Penalizes confident wrong predictions heavily | 0+ (lower is better) |
Spotting Overfitting
If a model scores much higher on the training set than the holdout set, it has memorized training noise rather than learned generalizable patterns. The gap indicator beside each holdout value quantifies this in percentage points, printed as pts (5 pts means 5 points, not 5% of the training score).
- ≤ 5 pts: Healthy, minimal overfitting
- 5 to 10 pts: Moderate, consider simplifying the model
- > 10 pts: Severe, model is likely overfitting
Only a drop is graded. An up arrow (↑) means the holdout scored higher than training, which is not overfitting, so it stays green whatever its size. A large one usually means a small or lucky holdout: rerun with another random seed before trusting it.
Log Loss has no percent scale, so its indicator shows the raw rise in log loss from training to holdout (+0.042 means the average penalty per customer grew by 0.042):
- up to +0.050: Healthy, about what a simple model shows from which rows happen to land in the holdout
- +0.050 to +0.100: Moderate, rerun with another random seed before reading much into it
- above +0.100: Severe, typical of a model that hands out 0% or 100% probabilities and is sometimes wrong
Decision Trees are particularly prone to overfitting (try lowering max depth). Logistic Regression and Linear SVM typically show smaller gaps because their linear boundary can’t memorize complex noise.
When Each Algorithm Excels
- Logistic Regression — Best for linearly separable data; outputs well-calibrated probabilities; highly interpretable.
- Decision Tree — Captures non-linear patterns and interactions; easy to visualize; prone to overfitting without depth limits.
- k-NN — Adapts to any boundary shape; no training phase; slow at prediction time on large datasets.
- Naive Bayes — Fast and effective when class-conditional features are roughly Gaussian; handles high-dimensional data well.
- Linear SVM — Finds the maximum-margin boundary; robust to outliers near the boundary; needs a kernel for non-linear patterns.