Understanding Regression

Where does R² actually come from?

You have almost certainly seen R² reported as a single number — something like 0.61 — alongside a claim that the model is doing well, or badly. That is a lot to take on faith, and taking statistics on faith is exactly how it starts feeling like magic.

So let us not take it on faith. Over the next ten pictures we are going to build that number from nothing. By the end you will be able to say precisely what R² counts, because you will have watched it being counted. There is no formula to memorise here.

Go slowly, and use the play buttons. Each picture is the same chart as the one above it with a single new thing added, so nothing ever jumps or reshuffles on you.

THE SETUP — You work for a retail chain with thirty stores. For each store you know how much revenue it brought in last quarter, and revenue is the thing you would like to be able to predict. More information arrives as we go: in a few steps you will find out what each store spent on advertising, and later still whether it sits inside a mall or stands on its own. For now, though, assume you know nothing at all except the revenue figures. That blank slate is where every model starts.

Why are there two lines through one cloud?

Correlation and regression get taught as close cousins, so you would expect one line through a cloud of points. Look at the chart below and you will find two. That is not a mistake, and it is not the software being fussy.

The two lines answer two different questions. They look almost like the same question, which is exactly why this trips people up. Once you can see what separates them, correlation stops being an abstract coefficient you report and becomes something you can point at on a chart.

Nothing here is graded and nothing can break. Drag the slider, change the units, and watch which numbers follow you and which refuse to budge.

THE SETUP — You are a few weeks into a digital marketing internship. Your team has run 40 different YouTube pre-roll versions of the same product ad. For each version you have two numbers: average watch time, meaning how many seconds people watched before skipping or finishing, and conversion rate, meaning the percentage who clicked through and bought. Your manager asks how strongly the two are related. Then she asks a follow-up that sounds like the same question but quietly is not, and that second question is what this page is really about.

Does it matter which group you compare against?

When you put a category into a regression — ad style, region, package size — the software makes you leave one of the groups out. The group you leave out is called the reference category, and every other group gets described as a distance from it.

Students worry about this, and the worry is a fair one. You had to pick a group, the choice felt arbitrary, and now every coefficient in your output depends on which one you picked. So does that choice change what you found?

Click through the four options below. Watch carefully: some numbers move every time you click, and others do not budge at all. Which is which turns out to be the whole lesson.

THE SETUP — Forty people each saw one of four ad styles — a humorous opening, a fear appeal, a rational claim, or a customer testimonial — with ten people per style. Afterwards each of them rated how likely they were to buy, on a scale from 1 to 7. That is the whole dataset, and every number on this page is worked out from it.

Why do these three lines stay parallel?

Have a look at the three fitted lines in the chart below. They are exactly parallel, and it is worth stopping to ask why that would be.

It is not because the three placements behave identically — they plainly do not. It is because the model was never given permission to let them differ. Parallel is what you get by default, and defaults are easy to walk straight past without noticing you made a choice.

An interaction term is simply that permission, written into the equation. The switch below adds it. Flip it on and off a few times and watch what the lines are suddenly allowed to do.

THE SETUP — Your team ran the same creative across three YouTube placements: pre-roll, the skippable ad before a video; in-feed, a thumbnail on the browse page; and Shorts, the vertical feed. That gives sixty campaign-days in total, twenty per placement, each with a daily ad spend and the conversions that followed. What everyone in the room actually wants to know is whether a dollar of spend works harder on some placements than others. That is an interaction question, even though nobody will use the word.

Is ANOVA really a different analysis?

ANOVA and regression are taught in separate weeks, they print different-looking tables, and they use different words for nearly everything. So it is completely reasonable to have filed them away as two separate methods you have to learn twice.

They are not two methods. They are one analysis wearing two sets of labels, and the table below shows you that directly: the same numbers, with the ANOVA name on the left and the regression name on the right.

If that sounds like a technicality, it is not. It means anything you already understand about regression, you already understand about ANOVA. That is one less thing to learn, not one more.

THE SETUP — These are the same forty respondents from the reference-category demo, just described the way an agency would describe them. Four hook styles for the opening three seconds of a YouTube pre-roll — humour, fear appeal, rational claim, customer testimonial — each shown to ten viewers, who then rated their purchase intention from 1 to 7. Your manager wants to know which hook wins. Worth holding on to as you read: this is exactly the same data as the earlier demo. Not one number has changed. Only the vocabulary has.

When the numbers cannot tell you what happened

You have just spent ten steps building R² from nothing, so here is the honest next question: once you have it, is it enough? Can you read a regression table and know what your data looks like?

Below are five sets of sixty ad campaigns. Every single one produces the same slope, the same intercept, the same standard error, the same t, the same p-value, the same correlation and the same R². Not similar. Identical, to every decimal place shown.

Click between them and keep one eye on the table on the right. It will not move. Then look at the chart. This is the single best argument anyone has ever made for plotting your data before you trust your output.

THE SETUP — Sixty ad campaigns. For each one you know the budget and the revenue that followed. You have run the regression, the output looks respectable — a solid R² and a p-value comfortably under .001 — and you are about to put it in a deck. The only step left is the one people skip.

The variable you left out

Everything so far has been about reading a model correctly. This last one is about a model that is computed perfectly, reported honestly, and still points you the wrong way. It is the most expensive mistake on this page, and unlike the others it does not announce itself: the output looks great.

Here is the situation. You have 300 customers. For each one you know how many emails they get each month and how much revenue they bring in. You regress revenue on emails, and the answer is wonderful — each extra email is worth about ten more dollars, and the p-value is under .001. The obvious move is to email everybody more.

There is one thing the model does not know: these customers sit in three loyalty tiers. Premium customers spend far more than Basic customers, and they also get emailed far more, because that is how loyalty programmes work. Press colour by loyalty tier and watch what the three groups have been hiding.

THE SETUP — 300 customers in three loyalty tiers. The data was generated with a rule we know for certain, because we wrote it: inside any single tier, one more email per month is worth about −$1.80. Email fatigue is real and it is negative. Nothing below changes that rule. The only question is whether your model can find it.

Drag the tiers together until they are identical. With nothing left to confuse it, the naive model lands on about −$1.64 all by itself — the very same answer the tier-aware model gives. Confounding is not a flaw in the arithmetic, it is a flaw in what you handed the arithmetic.

“Significant” does not mean big

Of everything on this page, this is the misunderstanding that costs the most money, and it hides inside a word we borrowed from ordinary English. In conversation, significant means important, sizeable, worth caring about. In statistics it means only one thing: probably not a fluke. Those are completely different claims, and a p-value only ever makes the second one.

Here is why that matters so much in marketing, where samples are enormous. A p-value gets smaller for two quite separate reasons: because the effect is genuinely large, or simply because you collected a lot of data. With enough visitors, an effect far too small to care about will still land at p < .001 — and it will look, in a slide deck, exactly like a real discovery.

The chart below makes the separation visible. Move the sample-size slider and watch carefully: the dot does not move. Your measured lift is whatever it is. All that changes is the interval around it, tightening as evidence accumulates, until it finally stops touching zero and the result is declared significant — without the effect ever getting any bigger.

THE SETUP — An A/B test on your checkout button. The current page converts at 3.00%. Before running anything, your team agreed that a lift below +0.50 percentage points would not repay the engineering time it takes to ship. Write that number down first, because deciding what counts as “big enough” after you have seen the p-value is how teams talk themselves into shipping noise.

Try this: leave the lift at +0.20, well under the shipping bar, and press collect more data. The result becomes statistically significant without ever becoming worth shipping. Then set the lift to +1.20 and drag the visitors right down — a lift that clearly would matter gets reported as “no significant difference”.

Why your worst stores “improve”

This one is different from the others. It is not a mistake in how you read a model — it is a mistake in how you chose who to look at, and it happens before any model is fitted at all. It is also, quietly, one of the most common ways a business convinces itself that something worked.

Any store's sales in a given month are two things added together: how good the store genuinely is, and how the month happened to go. Good weather, a roadworks closure, a local event, a competitor stumbling. Call the first part quality and the second part luck. Quality sticks around from month to month. Luck does not: next month deals a fresh hand.

Now think about what you actually selected when you picked the ten worst stores. You did not pick the ten worst-run stores. You picked the ten with the lowest total, which tends to catch stores that are genuinely weak and stores that simply had a rotten month. The second group was always going to look better next month, with or without you. Watch it happen below, and keep one eye on the ten best stores while you do.

THE SETUP — Forty stores. You rank them on last month's sales, take the worst ten, and put them through a turnaround programme. Be clear about what that programme does here: nothing. Not a little. Its effect is set to exactly zero, which is why anything you see afterwards cannot possibly have been caused by it.

Drag the luck slider to zero and run it again. With nothing but quality in the numbers, the worst stores stay exactly where they are and the bounce disappears completely. The size of the fake improvement is set entirely by how much luck is in your measure.

When two predictors are twins

Demo 7 showed you what it costs to leave something out of a model. This is the opposite complaint: what it costs to put two things in when they are near-copies of each other. It is the single most common frustration in marketing mix modelling, and it produces a very particular kind of confusion, because the model does not look broken at all.

Here is the situation. TV and digital budgets rise and fall together, because both are set as a share of one overall budget. In a good quarter both go up; in a lean one both get cut. Your data has therefore almost never seen a week with heavy TV and light digital, or the reverse — and those are exactly the weeks that would tell you which of the two was doing the work.

Slide the two channels from unrelated towards perfectly locked together and watch the chart. The individual estimates start swinging wildly, and their intervals grow until they comfortably include zero. Then look at the bottom bar, which is what the pair is worth. It barely moves. That contrast is the whole lesson: the data has not stopped being informative, it has just stopped being able to answer the particular question you asked.

THE SETUP — Sixty weeks of TV spend, digital spend and sales. When this data was generated, each channel was genuinely worth 2.00 in extra sales per dollar — so the honest answers are 2.00, 2.00 and 4.00 for the pair. Those are marked on the chart, and they never change no matter where you put the slider.

Push the slider up near the top and then press collect another sixty weeks a few times. The two individual estimates leap around from sample to sample — one can easily come out negative — while the bar for the pair stays put. That jumpiness is what a large standard error actually looks like.