Missing Data Lab
Your survey came back with 30% of income fields blank. Do you drop those rows? Fill them with the average? Use a smarter prediction? Each choice leads to a different reality in your downstream analysis — some close to truth, some dangerously wrong. This lab puts you in the driver's seat: inject missing data, choose your fix, and see what you present to your CMO.
OVERVIEW & LEARNING OBJECTIVES
The real problem with missing data isn't that rows are gone. It's that the rows which are missing are almost never a random sample of your customers. High earners skip income questions. Unhappy customers ignore NPS surveys. The moment your missing pattern is non-random, every imputation shortcut quietly bends your conclusions — sometimes by 50% or more — and you won't know unless you think carefully about why the data is missing.
Imputation is the process of replacing missing values with estimated substitutes so that downstream analyses can run on a complete dataset. The four methods in this lab differ in sophistication — from "plug in the column average" to "predict each missing value from a regression on other columns." The critical question is not how clever the algorithm is, but whether its assumptions match why the data went missing in the first place. A method that is perfect for random missingness can make systematically missing data worse.
🎯 Three Mechanisms — What They Mean in Plain English
Missing Completely at Random (MCAR)
The probability of a value being missing has nothing to do with any variable in your dataset — observed or unobserved. A server glitch that randomly drops 1-in-10 responses is MCAR. Who's missing is truly unpredictable. Result: complete-case analysis is unbiased (just smaller n). Most imputation methods recover the truth.
Missing at Random (MAR)
Missingness depends on other observed variables, but not on the missing value itself. Gold-tier customers ignoring email surveys is MAR — you can predict who's missing from tier, which you do have. Result: listwise deletion causes bias. Conditional imputation (using tier) can fully recover the truth because the mechanism is explainable.
Missing NOT at Random (MNAR)
Missingness depends on the unobserved value itself. High earners hiding their income, unhappy customers skipping NPS — the missing value predicts its own absence. You cannot observe the pattern and you cannot fully correct it. Result: every imputation method is biased to some degree. MNAR requires domain knowledge and sensitivity analysis, not just a better algorithm.
💡 The hard truth: You usually cannot test from data alone which mechanism applies. It requires thinking carefully about why someone might not answer — behavioral reasoning, not statistical testing.
📋 How to Use This Tool — 6-Step Flow
What: Choose one of four marketing datasets, each with a different missingness mechanism (MCAR, MAR, or MNAR).
Why: The mechanism determines whether imputation can recover the truth — or whether all roads lead to bias.
Watch for: Start with Scenario 1 (MCAR) to build intuition, then try Scenario 3 or 4 for the real gut-punch.
What: Choose what % of targeted column values to make missing (10–70%) and click Inject. The seed controls which specific rows go missing.
Why: Different missing rates change the severity of bias — 10% MCAR is survivable; 40% MNAR can be catastrophic.
Watch for: Keep seed at 67 for class discussions so everyone sees the same missing rows. A preview table below shows you exactly what changed.
What: Two charts show the distribution of what went missing vs. what you still have.
Why: Under MCAR the missing pattern is flat (random). Under MNAR a specific part of the distribution vanishes — you can see the wound.
Watch for: In MNAR scenarios, notice how the highest-value rows disappear. The "listwise warning" shows you how many complete customer records you'd discard just to avoid one blank column.
What: Pick one of four imputation strategies: listwise deletion, mean/mode replacement, conditional imputation, or predictive regression imputation.
Why: Each method makes a different assumption about why data is missing. Picking the wrong one for the wrong mechanism amplifies bias rather than fixing it.
Watch for: Commit to a choice before seeing ground truth — that’s the point. See your imputed distribution and downstream analysis first.
What: Click Reveal to expose the true parameter values from the complete synthetic data.
Why: You only know how wrong you are if you have something to compare against. In real data there is no ground truth — we can only do this because the data is simulated.
Watch for: The β bias badge. A 30%+ bias means your CMO would draw a materially wrong conclusion from your report.
What: The final panel runs all four methods + ground truth and shows % bias for each in a single table.
Why: This is the screenshot worth sharing. It shows which methods survive each mechanism — and which fail spectacularly.
Watch for: Under MNAR, notice that even the best method has meaningful bias — there is no algorithmic escape from a fundamentally flawed data collection process.
📐 The Four Treatment Tiers
| Tier | Method | What It Does | Key Failure Mode |
|---|---|---|---|
| 0 | Listwise Deletion | Drop every row containing a missing value in the focal column | Catastrophic under MNAR — removes exactly the customers you need to see |
| 1 | Mean / Mode Imputation | Replace each missing value with the column mean or most common category | Collapses variance and weakens correlations and R². A mean-filled predictor keeps the listwise slope in a one-predictor regression; a mean-filled outcome flattens it, and slopes shift when filled predictors are correlated with others |
| 2 | Conditional Imputation | Impute using within-group means/modes (e.g., mean open rate by loyalty tier) | Only works if you condition on the variable that explains the missingness |
| 3 | Predictive Imputation | Fit a regression on complete rows; predict missing values from other columns | Only as good as its auxiliary column: a weak predictor collapses it to mean imputation, and a fit learned from responders can move the slope. Still biased under MNAR; compare it in the Step 6 table |
STEP 1 — CHOOSE A SCENARIO
Each scenario has a distinct dataset, missingness mechanism, and downstream analysis. Start with Scenario 1 to calibrate; Scenarios 3 & 4 deliver the real surprises.
Email Survey Glitch
Missing: satisfaction_score
Outcome: repurchase probability
A server error corrupted survey responses at random. Some high scorers are missing, some low scorers — no pattern. The forgiving baseline.
Loyalty Tier Non-Response
Missing: email_open_rate
Outcome: conversion rate
Gold-tier customers ignore survey emails. Bronze customers always respond. Missingness is explained by a variable you do have.
Income Disclosure Trap
Missing: annual_income
Outcome: avg order value
High earners systematically refused to disclose income. The right tail of your distribution simply vanishes. Every method gets it wrong.
Silent Detractors
Missing: nps_category
Outcome: churn flag
Unhappy customers skip NPS surveys. Your collected data over-represents people who like you. Your NPS score will look fantastic — and be wrong.
STEP 2 — SET MISSINGNESS & INJECT
STEP 3 — EXAMINE THE WOUND
STEP 4 — CHOOSE YOUR TREATMENT METHOD
Select the strategy you believe will best recover the true relationship. You’ll see your downstream results before ground truth is revealed — so choose carefully.
Listwise Deletion
Drop every row where the focal column is blank. Use only complete cases.
Best when: Data is MCAR and missing rate is low (< 10%). Unbiased under MCAR but wasteful.
- ✅ Unbiased under MCAR
- ❌ Destroys rows and causes bias under MAR/MNAR
- ❌ Throws away all other columns for dropped rows
Mean / Mode Imputation
Fill missing values with the column average (continuous) or most frequent category.
Best when: Missing rate is very low (< 5%) and you just need a quick fix that won’t collapse variance too badly.
- ✅ Simple, preserves sample size
- ❌ Shrinks the column’s spread and R² — yet a mean-filled predictor keeps the listwise slope, so the slope won’t warn you
- ❌ Creates a spike at the mean in distributions
Conditional Imputation
Impute using within-group means or modes, conditioned on a related observed variable (e.g., loyalty tier, age group, churn flag).
Best when: Data is MAR and you know which variable explains the missingness pattern. Conditions on that variable to undo the systematic bias.
- ✅ Correct fix for MAR when conditioning variable is right
- ❌ Fails if the conditioning variable doesn't explain missingness
Predictive Imputation
Fit a regression (continuous) or probabilistic model (categorical) using all available complete rows; predict each missing value from other columns.
Best when: Other columns carry real predictive signal for the missing variable and the responders it learns from resemble the non-responders. Still biased under MNAR; whether it beats simpler fills depends on the data, so check it against the others in the Step 6 table.
- ✅ Keeps more of the column’s spread than one global mean when the auxiliary column predicts well
- ❌ Collapses to mean imputation when the auxiliary column predicts poorly
- ❌ Still biased under MNAR — the missing pattern is unobservable
STEP 5 — DOWNSTREAM IMPACT
Your Results (ground truth hidden)
💡 Why does this happen?
Under MCAR, missing values are a random sample. Any imputation strategy gets an unbiased estimate of the mean, and the slope stays close to the truth because nothing pushes it in a systematic direction. What imputation costs you is spread: filled-in values pile up at the centre, so the column's standard deviation and R² both fall.
Under MAR, the systematic pattern (e.g., Gold tier never responds) distorts the sample if you don't account for it. Conditional imputation essentially re-weights back to the correct group distributions — which is why it brings the mean open rate back close to the truth here. The slope is not where this damage shows: listwise deletion and mean imputation both leave it near the true value while the mean falls.
Under MNAR, the imputed value is always a guess about a biased sample. High earners' income is systematically absent from your data. Every imputation method plugs in a value from the observable lower part of the distribution — so the mean stays low and the high end of the scatter disappears no matter what. The slope does not collapse with it: because the gaps are in the predictor, it stays close to the truth (slightly steeper, not flatter, at the default seed), which is exactly what makes the damage easy to miss. The only real fix is to design the data collection process differently (e.g., bracket income ranges, verify through linked data, use Bayesian sensitivity analysis).
STEP 6 — ALL METHODS SIDE BY SIDE
Ground truth pinned as reference (the dark bar in the chart, the top row of the table). Mean Bias and β Bias give each method's percentage distance from it.
⚠️ What you'd report to a CMO: The table below is the most important thing to study. Every number that drifts from the ground-truth row (the mean and its Mean Bias, the β Bias, the R²) represents a question your CMO could ask (“What does our typical customer earn?”) that you'd answer wrongly because of how you handled missing data upstream.
📘 Key Takeaways
- MCAR: All four methods recover the truth reasonably. Listwise deletion loses rows (lower n) but doesn't introduce directional bias. Mean imputation leaves the slope exactly where listwise deletion put it, but it shrinks the column's spread and R²; it is acceptable at low % missing.
- MAR: Listwise deletion and mean imputation both drag the mean open rate well below the truth. Mean imputation does not flatten the slope (it reports the listwise slope exactly), but it collapses the within-tier spread and cuts R² sharply. Conditional imputation (using the correct conditioning variable) is the right tool — it brings the mean back close to the truth.
- MNAR (continuous): Every method is biased, and by almost the same amount: each one reports an average income well below the truth because each one learns only from the customers who answered. The slope stays close to the truth, so it will not warn you. The only real fix is better data collection.
- MNAR (categorical): Mode imputation amplifies the over-representation of Promoters. Listwise deletion removes most of the Detractors. The chi-square statistic weakens dramatically, hiding the churn-NPS link. Conditional imputation (using churn flag) can bring Cramér's V back near the truth while the category shares stay wrong; at the default settings it fills no one in as a Detractor. A recovered V does not mean recovered shares.
- Rule of thumb: At 10–15% MCAR, almost any approach is fine. Above 20% under MAR or any % under MNAR, your methodology section needs to explicitly justify your choice — or your results aren't trustworthy.