Difference-in-Differences
Estimate whether a campaign, launch or price change really moved results, by comparing the markets that got it with similar markets that didn’t.
OVERVIEW & LEARNING OBJECTIVES
Most marketing interventions cannot be randomized at the individual customer level: a regional TV flight airs in specific DMAs, store renovations happen in selected locations, or a mobile app rollout debuts in test cities first. When stakeholders ask “did it work?”, the two most common answers are almost always wrong:
1. Before vs. After
Wrongly credits everything else that happened.
Comparing test stores before and after launch credits the campaign with seasonality, macroeconomic growth, competitor supply disruptions, and general category momentum.
2. Test vs. Control After
Wrongly credits differences that were always there.
Comparing test stores to control stores after launch credits the campaign with pre-existing market size, demographic advantages, and brand maturity that existed long before the launch.
3. Difference-in-Differences
Subtracts both to isolate the true incremental lift.
DiD takes the test group’s change and subtracts the control group’s change. The remainder is the lift that occurred above and beyond what test stores would have done anyway.
The Core Parallel Trends Assumption
The entire validity of DiD rests on one unprovable assumption: without the campaign, the test and control groups would have moved along parallel paths. You can never prove this would have held into the future, but you can check whether they moved together in the pre-period. The pre-period check is the single most important diagnostic on this page.
🎯 What You’ll Learn
- How to subtract the counterfactual trend: Decompose a $45K sales jump into $15K of market drift and $30K of incremental campaign lift.
- Why non-significant pre-trends do not prove parallel trends: With few markets, statistical power is low. You must inspect both the joint lead test and the lead magnitude ratio.
- Why small market counts require honest inference: With 4 to 8 test cities, standard errors are noisy; randomization inference provides exact, truthful confidence bounds.
- How anticipation and transition periods distort findings: Detect pre-launch disruption (e.g. renovation dips) and adjust the transition window.
Step 1 CHOOSE YOUR DATA & MODE
Work with panel data (markets tracked over time) or start with a textbook Quick 2×2 calculation.
Use a Case Study
Upload Your Data
Upload panel CSV data
Provide a long-format CSV with at least four columns: market, period,
outcome, and a market-level treated indicator (0 or 1).
Drag & Drop CSV file (.csv, .tsv, .txt, .xls, .xlsx)
Include headers with market, period, outcome, and treated columns.
- Outcome: Continuous numerical scale (e.g. sales revenue, orders, app sessions, visits). Do not use nominal categories or qualitative choices as the outcome.
- Market: Categorical cluster identifier (store ID, city, DMA, territory) tracked over time.
- Period: Ordered integer time steps (1, 2, 3...) or consecutive calendar intervals.
- Treated: Binary indicator (1 = test market receiving treatment, 0 = control market).
Enter the average outcome for test and control groups before and after the campaign. Defaults match the textbook app launch figure exactly ($420K → $465K vs $400K → $415K).
Test Markets (Treated)
Control Markets (Comparison)
Step 2 CONFIGURE COLUMN ROLES & SETTINGS
Verify column mappings and declare the campaign launch period.
The first period when the campaign is active in test markets.
Shown on charts and in managerial reports.
Advanced Diagnostics: Placebo & Transition Periods
In-time falsification test: a real campaign cannot have worked before it started.
Exclude disruption/anticipation period immediately prior to launch (e.g. construction).
🤔 Try This: Three Revealing Experiments
- Decompose the Textbook Case: In Quick 2×2 mode, see how the $45K test gain ($420K → $465K) separates into $15K of market trend ($400K → $415K) and exactly $30K of true DiD lift.
- Catch the Selection Trap in Pinnacle Fitness: Load Pinnacle Fitness and click Estimate the Effect. Naive DiD claims a lift of +$2,377. Now open the Pre-period check tab: see the pre-trend test fail completely ($p < .001$), proving the lift was an illusion.
- Diagnose Anticipation in Oakline Furniture: Load Oakline Furniture. Notice the sharp dip at $r = -1$ (month 9) from renovation closures. Then open Advanced Diagnostics and exclude month 9.
Step 3 START HERE: PLAIN-LANGUAGE READOUT
Step 4 VISUAL EVIDENCE & ESTIMATES
The 2×2 Difference Table
How the arithmetic works: market-level averages ensure every store and city counts equally. The counterfactual shows where test markets would have landed if they had simply followed the control group.
| Group | Before Launch | After Launch | Change (Δ) |
|---|---|---|---|
| Test Markets (Treated) | — | — | — |
| Control Markets (Comparison) | — | — | — |
| Counterfactual (Test Without Campaign) | — | — | — |
| Difference-in-Differences Lift | (Test Δ) − (Control Δ) | — | |
Explore the Visual Evidence
Solid blue line: average outcome for test markets. Solid grey line: control markets. Dashed blue line: the counterfactual — what test markets would have done if they had followed control trends. The shaded gap after launch is the true incremental DiD effect.
📊 How to Read This Chart
- Before the vertical launch line: The test and control lines should move in parallel. Divergence here is a warning sign.
- The dashed counterfactual line: Takes the test group’s baseline and adds the control group’s growth.
- The shaded gap: If test markets surge above the dashed line after launch, the campaign generated positive lift.
Relative periods ($r = t - t^*$). The period immediately prior to launch ($r = -1$) is normalized to 0. Leads ($r < -1$) hugging the zero line confirms the parallel trends assumption. Post-launch lags ($r \ge 0$) show the evolving impact over time.
📊 How to Read the Event Study
- Leads near zero: If the 95% confidence bars for pre-periods cross zero, test and control were moving together before launch.
- Upward pre-trend: If leads slope upward into launch, the test group was already pulling ahead, meaning the DiD estimate is inflated.
- Anticipation dip ($r = -1$): If results drop right before launch, customers or stores anticipated the rollout or suffered renovation downtime.
Comparing the three ways to evaluate the campaign. Horizontal bars show point estimates and confidence intervals.
📊 Why the Three Estimates Disagree
- Post-Period Gap (Test vs. Control after): Biased by pre-existing level differences between groups.
- Before vs. After (Test group alone): Biased by macro seasonality and organic market growth.
- Difference-in-Differences: Subtracts both biases to isolate genuine causal lift.
Every market as an individual trajectory. Blue lines are test markets; grey lines are control markets. Hover over any line to see the market name and trajectory.
Randomization Inference: Shuffles treatment labels across markets to see how often a random assignment produces a DiD lift as large as what we observed. The red vertical line marks our real estimate.
Step 5 APA & MANAGERIAL INTERPRETATION
APA Statistical Reporting
Managerial Interpretation
Step 6 CHECK THE PARALLEL TRENDS ASSUMPTION
Causal inference requires both statistical validation and domain judgement. Your answers here update the managerial readout. (Answers only move the verdict toward caution; stored locally in your browser).
1. Statistical Pre-Trend Test
2. In-Time Placebo Test
3. Were test markets selected because they were growing?
If management chose test stores because they had momentum or were “ready”, parallel trends is violated by selection.
4. Could the campaign have reached control markets?
Shared broadcast media or customer cross-shopping dilutes control performance, making the true lift appear smaller.
Step 7 REVEAL THE PLANTED TRUTH
Known-Truth Simulation Benchmarking
In this case study, the ground truth was deliberately planted. Test whether your estimate and assumptions caught the reality or fell into the trap.
Reproducible Python Export & Data Pipeline
Export a standalone, executable Python script using pandas and statsmodels
with Liang-Zeger cluster-robust standard errors, or download the full augmented dataset with fitted values,
counterfactuals, and estimated causal lifts appended to every observation.
Exploratory Questions for Marketing Analysts
- Why does the dashed counterfactual line matter more than the raw control line? Because test stores often operate at a different baseline scale. Shifting the control trajectory to the test group’s pre-period level shows what test stores would have produced under the null hypothesis of no campaign effect.
- When does a Difference-in-Differences model lie? Whenever test markets are selected because they are accelerating, when nationwide macro events disproportionately affect one region, or when media spillover bleeds into control stores.
- What should an executive do if pre-trends fail? Do not present the DiD number as causal. Instead, explore synthetic control methods, find a more comparable control subset with matched pre-slopes, or execute a designed geo-experiment with randomized market assignment.