Difference-in-Differences
Estimate whether a campaign, launch or price change really moved results, by comparing the markets that got it with similar markets that didn’t.
OVERVIEW & WHAT THIS TOOL SOLVES
Most geo-experiments, store remodels, and regional TV flights cannot be randomized at the individual shopper level. Comparing test markets before vs. after launch wrongly credits macro seasonality, while comparing test vs. control markets after launch wrongly credits pre-existing city size differences. Difference-in-Differences (DiD) subtracts the control group's pre-to-post trend from the treated group's pre-to-post jump to isolate the true incremental lift above the counterfactual trajectory.
🎯 Questions It Answers
- How much of the post-launch sales jump in our treated markets was truly caused by the intervention vs. background market drift?
- Did Treated and Control markets move along parallel trajectories during the pre-launch periods?
- How does the treatment effect evolve week-by-week after launch (immediate spike, gradual ramp, or habituation)?
📂 What Data You Need
- Panel Dataset: Repeated observations across multiple units (stores, DMAs, states) over pre- and post-treatment periods
- Treatment Assignment: A binary indicator marking which units received the intervention and the exact launch period
- Optional Covariates: Time-varying local controls (e.g., weather, local promos) or transition-window exclusions
💡 What You'll Discover
- 2×2 & Two-Way Fixed Effects Lift: Point estimate of causal lift (
ATT), cluster-robust SEs, and randomization-inference p-values - Event-Study Lead/Lag Plot: Period-by-period dynamic coefficients testing pre-trends and post-launch dynamics
- Pre-Trend Diagnostics: Joint Wald test on pre-launch leads and MDE sensitivity for small market counts
🔍 Key Nuances & Common Traps to Watch For
- The Unprovable Parallel Trends Assumption: DiD assumes that without the campaign, Treated and Control units would have changed by the exact same amount. Always inspect the Event-Study pre-launch leads to verify they tracked together before launch.
- Non-Significant Pre-Trends ≠ Proven Parallel Trends: With only 6 to 10 DMAs, a pre-trend test has low statistical power. Check both the joint lead p-value and the visual magnitude of pre-launch drift relative to your post-launch effect.
- Cluster Standard Errors at the Unit Level: Weekly sales within the same store are serially correlated; failing to cluster by unit (or use Randomization Inference when cluster counts are small) severely understates uncertainty.
📐 Under the Hood: Model Equations & Variable Definitions
1. Canonical 2×2 Difference-in-Differences Estimator
💬 Read it in plain English: Take the average pre-to-post change in the treated markets and subtract the average pre-to-post change in the untreated control markets to strip out both baseline level differences and shared time drift.
2. Two-Way Fixed Effects (TWFE) & Dynamic Event-Study Specification
💬 Read it in plain English: Unit fixed effects (α_i) absorb permanent store/city differences and time fixed effects (λ_t) absorb common period shocks, while event-study coefficients γ_k trace out pre-launch parallel trends (k < 0) and post-launch lift dynamics (k ≥ 0).
Step 1 CHOOSE YOUR DATA & MODE
Work with panel data (markets tracked over time) or start with a textbook Quick 2×2 calculation.
Use a Case Study
Upload Your Data
Upload panel CSV data
Provide a long-format CSV with at least four columns: market, period,
outcome, and a market-level treated indicator (0 or 1).
Drag & Drop CSV file (.csv, .tsv, .txt, .xls, .xlsx)
Include headers with market, period, outcome, and treated columns.
- Outcome: Continuous numerical scale (e.g. sales revenue, orders, app sessions, visits). Do not use nominal categories or qualitative choices as the outcome.
- Market: Categorical cluster identifier (store ID, city, DMA, territory) tracked over time.
- Period: Ordered integer time steps (1, 2, 3...) or consecutive calendar intervals.
- Treated: Binary indicator (1 = test market receiving treatment, 0 = control market).
Enter the average outcome for test and control groups before and after the campaign. Defaults match the textbook app launch figure exactly ($420K → $465K vs $400K → $415K).
Test Markets (Treated)
Control Markets (Comparison)
Step 2 CONFIGURE COLUMN ROLES & SETTINGS
Verify column mappings and declare the campaign launch period.
The first period when the campaign is active in test markets.
Shown on charts and in managerial reports.
Advanced Diagnostics: Placebo & Transition Periods
In-time falsification test: a real campaign cannot have worked before it started.
Exclude disruption/anticipation period immediately prior to launch (e.g. construction).
🤔 Try This: Three Revealing Experiments
- Decompose the Textbook Case: In Quick 2×2 mode, see how the $45K test gain ($420K → $465K) separates into $15K of market trend ($400K → $415K) and exactly $30K of true DiD lift.
- Catch the Selection Trap in Pinnacle Fitness: Load Pinnacle Fitness and click Estimate the Effect. Naive DiD claims a lift of +$2,377. Now open the Pre-period check tab: see the pre-trend test fail completely (\(p < .001\)), proving the lift was an illusion.
- Diagnose Anticipation in Oakline Furniture: Load Oakline Furniture. Notice the sharp dip at \(r = -1\) (month 9) from renovation closures. Then open Advanced Diagnostics and exclude month 9.
Step 3 START HERE: PLAIN-LANGUAGE READOUT
Step 4 VISUAL EVIDENCE & ESTIMATES
The 2×2 Difference Table
How the arithmetic works: market-level averages ensure every store and city counts equally. The counterfactual shows where test markets would have landed if they had simply followed the control group.
| Group | Before Launch | After Launch | Change (Δ) |
|---|---|---|---|
| Test Markets (Treated) | — | — | — |
| Control Markets (Comparison) | — | — | — |
| Counterfactual (Test Without Campaign) | — | — | — |
| Difference-in-Differences Lift | (Test Δ) − (Control Δ) | — | |
Explore the Visual Evidence
Solid blue line: average outcome for test markets. Solid grey line: control markets. Dashed blue line: the counterfactual — what test markets would have done if they had followed control trends. The shaded gap after launch is the true incremental DiD effect.
📊 How to Read This Chart
- Before the vertical launch line: The test and control lines should move in parallel. Divergence here is a warning sign.
- The dashed counterfactual line: Takes the test group’s baseline and adds the control group’s growth.
- The shaded gap: If test markets surge above the dashed line after launch, the campaign generated positive lift.
Relative periods (\(r = t - t^*\)). The period immediately prior to launch (\(r = -1\)) is normalized to 0. Leads (\(r < -1\)) hugging the zero line confirms the parallel trends assumption. Post-launch lags (\(r \ge 0\)) show the evolving impact over time.
📊 How to Read the Event Study
- Leads near zero: If the 95% confidence bars for pre-periods cross zero, test and control were moving together before launch.
- Upward pre-trend: If leads slope upward into launch, the test group was already pulling ahead, meaning the DiD estimate is inflated.
- Anticipation dip (\(r = -1\)): If results drop right before launch, customers or stores anticipated the rollout or suffered renovation downtime.
Comparing the three ways to evaluate the campaign. Horizontal bars show point estimates and confidence intervals.
📊 Why the Three Estimates Disagree
- Post-Period Gap (Test vs. Control after): Biased by pre-existing level differences between groups.
- Before vs. After (Test group alone): Biased by macro seasonality and organic market growth.
- Difference-in-Differences: Subtracts both biases to isolate genuine causal lift.
Every market as an individual trajectory. Blue lines are test markets; grey lines are control markets. Hover over any line to see the market name and trajectory.
Randomization Inference: Shuffles treatment labels across markets to see how often a random assignment produces a DiD lift as large as what we observed. The red vertical line marks our real estimate.
Step 5 APA & MANAGERIAL INTERPRETATION
APA Statistical Reporting
Managerial Interpretation
Step 6 CHECK THE PARALLEL TRENDS ASSUMPTION
Causal inference requires both statistical validation and domain judgement. Your answers here update the managerial readout. (Answers only move the verdict toward caution; stored locally in your browser).
1. Statistical Pre-Trend Test
2. In-Time Placebo Test
3. Were test markets selected because they were growing?
If management chose test stores because they had momentum or were “ready”, parallel trends is violated by selection.
4. Could the campaign have reached control markets?
Shared broadcast media or customer cross-shopping dilutes control performance, making the true lift appear smaller.
Step 7 REVEAL THE PLANTED TRUTH
Known-Truth Simulation Benchmarking
In this case study, the ground truth was deliberately planted. Test whether your estimate and assumptions caught the reality or fell into the trap.
Augmented Data Pipeline Export
Download the full augmented dataset with fitted values, counterfactuals, and estimated causal lifts appended to every observation, or use the code export bar below to download or copy the standalone Python script.
Exploratory Questions for Marketing Analysts
- Why does the dashed counterfactual line matter more than the raw control line? Because test stores often operate at a different baseline scale. Shifting the control trajectory to the test group’s pre-period level shows what test stores would have produced under the null hypothesis of no campaign effect.
- When does a Difference-in-Differences model lie? Whenever test markets are selected because they are accelerating, when nationwide macro events disproportionately affect one region, or when media spillover bleeds into control stores.
- What should an executive do if pre-trends fail? Do not present the DiD number as causal. Instead, explore synthetic control methods, find a more comparable control subset with matched pre-slopes, or execute a designed geo-experiment with randomized market assignment.