Overfitting Explorer

Model Fitting Bias-Variance Tradeoff

Discover how adding complexity improves training fit while destroying predictive power. Build polynomial models, reveal holdout data, and see overfitting happen in real-time.

OVERVIEW & WHAT THIS TOOL SOLVES

It is dangerously easy to build a marketing model that looks brilliant on historical data but fails miserably when deployed on next month's campaigns. As you add polynomial terms (X², X³, …, X⁶) to a regression, training error always goes down because the curve bends to chase random noise in your sample. Overfitting Explorer lets you incrementally increase model complexity, reveal unseen holdout data, and watch the Bias–Variance Tradeoff unfold in real time so you can pick the degree that actually generalizes.

🎯 Questions It Answers

Why does adding more predictor terms always improve Training R² even when the model is getting worse?
At what polynomial degree does our model switch from learning the true marketing signal to memorizing random noise?
How do Adjusted R² and Holdout RMSE help us select the right model complexity before deployment?

📂 What Data You Need

Built-In Marketing Scenarios: Social Media Ad Spend, Coffee Price Elasticity, and Email Frequency Optimization
Or Custom CSV Upload: Any paired X (predictor) and Y (outcome) dataset
Train / Holdout Split: 50 training observations used to fit the polynomial and 100 unseen holdout points to test generalization

💡 What You'll Discover

Diverging Error Curves: Classic U-shaped Holdout RMSE curve vs. monotonically falling Training RMSE
Side-by-Side Complexity Comparison: Simultaneous overlay of Degree 1 through Degree 6 polynomial curves
Bias–Variance Diagnosis: Automated classification of each model as Underfit (High Bias), Just Right, or Overfit (High Variance)

🔍 Key Nuances & Common Traps to Watch For

Never Judge a Model by Training R² Alone: Raw Training R² cannot decrease when you add a variable—even a column of random lottery numbers will nudge R² upward. Always check Holdout RMSE or Adjusted R².
Wild Extrapolation at the Edges: High-degree polynomials (Degree 4–6) oscillate violently near the minimum and maximum values of X (Runge's phenomenon), producing absurd predictions just outside your historical range.
Adjusted R² Is a Penalty, Not a Holdout Test: Adjusted R² penalizes each extra parameter k using degrees of freedom, which helps flag bloat on training data, but true out-of-sample holdout validation remains the gold standard for predictive accuracy.

📐 Under the Hood: Model Equations & Variable Definitions

1. Degree-p Polynomial Regression Model

\[ \hat{Y}_i = \beta_0 + \beta_1 X_i + \beta_2 X_i^2 + \cdots + \beta_p X_i^p = \beta_0 + \sum_{m=1}^{p} \beta_m X_i^m \]

💬 Read it in plain English: A degree-p polynomial predicts outcome Y using a weighted sum of X raised to powers 1 through p—where p = 1 is a straight line, p = 2 is a single parabola bend, and higher degrees allow p − 1 directional turns.

Ŷ_i

Predicted OutcomeOUTPUT ⓘ

Model prediction for observation i evaluated on either the training or holdout set.

Polynomial Degree (Complexity)YOUR DATA ⓘ

Highest exponent of X included in the regression (selected from Degree 1 to Degree 6).

β₀, β_m

Fitted Polynomial CoefficientsESTIMATED ⓘ

Intercept β₀ and slope weights β_m estimated via Ordinary Least Squares on the training sample only.

X_i^m

m-th Power of Predictor XYOUR DATA ⓘ

Predictor value for observation i raised to exponent m (standardized internally for numerical stability).

2. Root Mean Squared Error (RMSE), R² & Adjusted R²

\[ \text{RMSE} = \sqrt{\frac{1}{n}\sum_{i=1}^{n}(Y_i - \hat{Y}_i)^2}, \qquad R^2 = 1 - \frac{\text{RSS}}{\text{TSS}}, \qquad \bar{R}^2 = 1 - \frac{(1 - R^2)(n - 1)}{n - k - 1} \]

💬 Read it in plain English: RMSE measures the typical prediction miss in Y's original units (squaring errors before averaging so big misses hurt more), while Adjusted R² shrinks raw R² whenever an extra parameter k fails to reduce error enough to justify losing a degree of freedom.

RMSE

Root Mean Squared ErrorOUTPUT ⓘ

Square root of mean squared residuals; computed separately on Training data and unseen Holdout data.

R², R̄²

Raw & Adjusted R-SquaredINFERENCE ⓘ

Proportion of variance explained (R²) and its complexity-penalized counterpart (R̄²).

RSS, TSS

Residual & Total Sum of SquaresOUTPUT ⓘ

Unexplained squared error RSS = ∑(Y_i − Ŷ_i)² and total outcome spread TSS = ∑(Y_i − Ȳ)².

n, k

Sample Size & Parameter CountYOUR DATA ⓘ

Number of observations n in the evaluation split and number of non-intercept terms k = p.

MARKETING SCENARIOS

Load a marketing use case:

Select a case study above to load data and see the business context.

BUILD YOUR MODEL

🔧 Model Terms

Select transformations to include in your model. Intercept is always included.

Intercept (β₀) X (β₁) X² (β₂) X³ (β₃) X⁴ (β₄) X⁵ (β₅) 1/X √X ln(X)

Parameters: 2 / 8

Model: Ŷ = β₀ + β₁X

📊 Model Metrics

Training Sample

R² --

RMSE --

k 2

📈 Information Criteria

AIC --

BIC --

VISUALIZATION

Training Data Holdout Data Current Model Saved Models

MODEL COMPARISON

Save up to 3 models to compare their performance side-by-side.

Model 1 Not saved

Model 2 Not saved

Model 3 Not saved

💡 Comparing Models

When comparing models:

Training R² will almost always favor more complex models — it literally cannot decrease when you add a term.
Adjusted R² penalizes for each additional parameter using degrees of freedom. A large gap between R² and Adj R² is a warning sign of overfitting — revealed here alongside the holdout results.
Holdout RMSE reveals true predictive performance on unseen data.
Red holdout values mean holdout R² fell below 0.8 × training R², or holdout RMSE rose above 1.5 × training RMSE. A red flag marks a gap, not its cause: a model that is too simple can trip the R² rule too. If training and holdout RMSE are both high and close together, the model is underfitting; overfitting looks like low training error with much higher holdout error.
AIC/BIC balance fit and complexity without needing holdout data.
The "best" model often isn't the most complex one.

THE MOMENT OF TRUTH

🔒

Save at least 2 models to unlock

Commit to your model choices before seeing how they perform on unseen data. This simulates real-world model deployment where you can't peek at future data.

0 / 2 models saved

EXPLORATORY QUESTIONS

🤔 Think About These Questions

The Training Trap: Start with just X. Then add X², X³, X⁴ one at a time. What happens to training R² each time? Does it ever go down?
The Holdout Truth: Now reveal the holdout data. Look at holdout RMSE as you add terms. Does it follow the same pattern as training RMSE?
Finding the Sweet Spot: At what complexity level does holdout error reach its minimum? Is this the same model that has the highest training R²?
The Wiggly Line: With 4+ polynomial terms, look at the fitted curve. Does it capture the "true" relationship, or is it chasing individual data points?
Real-World Implications: Imagine you built the highest-R² model and deployed it for next quarter's marketing budget allocation. What might happen?
The Adjusted R² Signal: After revealing the holdout data, look at the Adj R² values in your saved model cards. Does the model with the largest gap between R² and Adj R² also perform worst on the holdout? Could you have spotted the overfit without holdout data using Adj R² alone?

📚 Connecting to Broader Concepts

⚖️ Bias vs. Variance

Simple models have high bias (underfit)—they miss patterns. Complex models have high variance (overfit)—they memorize noise. The goal is the sweet spot in between.

🔄 Cross-Validation

In practice, we use k-fold cross-validation rather than a single train/test split. This gives more reliable estimates of out-of-sample performance. Same principle, more robust execution.

📉 Regularization

Methods like Ridge and Lasso regression add penalty terms that shrink coefficients, effectively preventing overfitting without removing terms entirely. They automate what you're doing manually here.

📏 Adjusted R²

Adj R² penalizes for each parameter added: \(\bar{R}^2 = 1 - \frac{(1-R^2)(n-1)}{n-k}\). It decreases if a new term doesn't carry its weight. A large R² vs. Adj R² gap is a complexity red flag — no holdout data required.

🎯 Marketing Applications

Every marketing model—response curves, CLV prediction, churn scoring—faces this tradeoff. A model that "fit perfectly" on last quarter's data may fail catastrophically on next quarter's.

👨‍🏫 Professor Mode: Guided Learning Experience