Login Create Account

Bandit vs. A/B Race

Experimentation Simulation Lab

You have three email subject lines and one send. To find out which works best you have to send some people the weaker ones — and every one of those is a sale you probably gave up. This lab lets you watch three different ways of handling that trade-off compete for the same customers, and shows you exactly what each one costs.

Never heard of a “bandit” before? Turn on Professor Mode and it will walk you through one race from start to finish — what the chart is showing, why the winner wins, and the catch that nobody mentions until it is too late.

OVERVIEW & LEARNING OBJECTIVES

Imagine you run the homepage for an online store. You have two versions of the hero banner and you genuinely do not know which one sells more. So you run a test: half the visitors see version A, half see version B. After a few weeks you look at the numbers and keep the winner.

That is an A/B test, and it is the standard answer. But notice what it costs. If version B really is better, then for the whole length of the test you were sending half of your customers to the worse banner on purpose. You paid for that information in lost sales.

Here is what that looks like in actual money. Say the good banner converts 6% of visitors and the weak one converts 2%. You sent 1,000 people to the weak one, so you got about 20 orders instead of the 60 you would have got from the good one. Those 40 missing orders are called regret — the sales you gave up, compared to what a know-it-all oracle would have earned from exactly the same visitors.

A bandit algorithm tries to be cleverer. Instead of a fixed 50/50 split for a fixed length of time, it checks in as it goes and starts steering more traffic toward whichever option is winning. Less money wasted on the loser. The name comes from slot machines — “one-armed bandits” — where each machine pays out at a different rate and you have to work out which is which while you are still spending real coins.

🎯 What You’ll Learn
  • Why testing costs money, not just time. Every visitor sent to a worse option is a sale you probably lost. You will be able to put a number on it.
  • The explore / exploit trade-off. Exploring means trying options you are unsure about, which is how you learn. Exploiting means going with whatever looks best right now, which is how you earn. You cannot do both at once.
  • Why the highest earner is not automatically the right choice. Bandits usually make more money during the test. They also come out the other side barely able to tell you anything about the options they stopped showing.
  • When to use which. A one-week holiday banner and a permanent checkout redesign are not the same kind of decision, and they should not get the same kind of test.
  • Why one result proves nothing. Run the identical situation again with different luck and you can get a different winner. You will see this happen.
So which one should I actually use?
  • Use a bandit for a short-lived decision you will never revisit — a one-day flash sale, a seasonal banner, an ad creative in rotation. You want the money, you do not need a defensible number afterwards.
  • Use an A/B test for a change you have to live with, or one you will have to justify to a boss who asks “better by how much, exactly?” A bandit cannot answer that question well, for reasons you will see below.
  • Either is fine when the options are genuinely identical. Nobody loses money. But only the A/B test can actually tell you “we found no difference” — which is itself a useful, reportable finding.

Step 1 PICK A SITUATION

Each of these is a real marketing decision with real money attached. Start with Nordvik Travel if this is your first time — it is the simplest version, just two homepage images and a clear winner. The others get more interesting: one where the options are nearly identical, one where the world changes halfway through, and one where there is genuinely nothing to find.

Step 2 SET UP THE RACE

Because this is a simulation, you get to play god: you set the true conversion rate of each option, and then watch whether the three approaches can figure out what you already know. In a real test you would never have this information — that is the entire reason testing exists. If a preset is loaded, these are already filled in and you can go straight to Step 3.

The options

Set the true conversion rate of each option. In a real test you would not know these — that is the point of testing.

Option True conversion rate

With this on, the rates above are masked until the race ends, so you face the same uncertainty the algorithms do.

The race

Contestants

Three different answers to the same question: “where should I send the next visitor?” Leave all three ticked so you can compare them.

%

At 10%, one visitor in ten is sent to a random option regardless of how badly it has been doing. That is the price epsilon-greedy pays to avoid getting stuck on a bad early guess.

Advanced: let the world change mid-test

Step 3 RUN IT

Run one race sends every visitor through, one at a time, and shows you what happened on this particular occasion. Repeat the race does the whole thing over and over with different luck each time, which is the only way to tell a real pattern from a fluke.

More repeats give a steadier answer and take longer. You can stop a run at any point and nothing is lost.

🤔 Try this:

  1. Load Nordvik Travel and click Run one race. Look at the Scoreboard. Thompson sampling almost certainly earned the most conversions. Case closed?
  2. Now look further down at What each contestant can tell you at the end. Find the row for the option that lost. Compare how many visitors the A/B test sent there against how many Thompson sent, and look at how much wider Thompson's interval is.
  3. That width is the catch. Thompson made more money and learned less. If your boss asks “how much worse was the old banner?”, the A/B test can answer and Thompson has to shrug.

EXPLORATORY QUESTIONS

Each of these is answerable with the controls on this page. They are in rough order of difficulty, and the last two are the ones worth arguing about.

  1. How expensive is patience? Run Nordvik Travel. Note the A/B test's regret, then note Thompson's. The difference is what the A/B test paid for a defensible answer. Would you pay it?
  2. Does more exploring help? Drag ε from 10% up to 40% and re-run. Epsilon-greedy now explores four times as hard. Does its regret go up or down, and does it get better at picking the right option?
  3. What if there is nothing to find? Load Arcadia Software, where all four pricing pages convert identically. Everyone's regret is zero — so has everyone done equally well? Look at what each one would report.
  4. Can a bandit be fooled? Load Lumen Skincare, where the three subject lines are only a few tenths of a percentage point apart. Repeat the race 250 times and look at “Picked the right one”. Would you trust any of these to choose your Black Friday subject line?
  5. What happens when the world moves? Load Kettle & Co., where the winning banner changes in December. Which approach notices? Before you run it, write down your prediction — most people guess wrong.
  6. Why does exploring not rescue it? In that same scenario, push ε to 30% so epsilon-greedy is constantly sampling everything. It still gets the answer wrong. What is actually broken — is it not looking enough, or is it the way it is averaging what it sees?