Train a small neural network on your customer data, watch it learn, and find out honestly whether it beats a
regression you could explain.
π¨βπ« Professor Mode: Guided Learning Experience
New to neural networks? Professor Mode walks you through training one, judging it against a
regression on customers it never saw, and spotting when it memorizes.
QUICK START: Choose Your Path
OVERVIEW & LEARNING OBJECTIVES
A vendor says their AI predicts which subscribers will cancel better than your analyst's regression.
Ridgeline's save offer costs $14 a bag, so a better list is worth real money, but only if the gain
shows up on customers neither model has seen. This lab trains a small neural network on a customer
file, scores it against a regression on held-back customers, and shows you what it learned.
π― What You'll Learn
A network is a flexible regression, not magic. It can bend where a regression is
straight. When there is nothing to bend, it does no better.
Compare on held-back customers, against the simplest model that could work. How well
a model does on the rows it learned from says nothing. If it cannot beat a regression on new rows, it
should not ship.
When the network wins, ask why. A win points at a curve or an interaction. Give the
regression that shape and it usually wins back, and you can still explain it.
Watch the validation curve, not the training curve. Training loss always falls. Stop
where validation loss is lowest; on small data, more epochs and more units make things worse.
Importance is not effect, and not cause. It says how much the predictions lean on an
input, not which way the input pushes or what would happen if you changed it.
Never trust a network outside the data it saw. Past the training range its curve
flattens or shoots off because of how it was built, not because of the market.
πΈοΈ What a neural network is, in one paragraph
A neural network turns each customer's columns into a prediction in stages. The first stage makes a
handful of weighted mixes of the inputs (the "hidden units"), bends each one with a fixed curve (the
"activation"), and the last stage mixes those into a probability or a number. Training nudges every
weight, over and over, to make the predictions on the training customers a little less wrong; one full
pass over them is an "epoch". With enough units it can draw almost any shape, which is exactly why it
can also memorize noise. To see a network bend a boundary with two inputs, try the
Neural Network Playground; this lab is where you judge one on a
real customer file.
π Key concepts
Three parts of the data: the network learns from the training rows, the validation
rows decide when to stop, and the holdout rows are scored once, at the end, for every number you
report.
Baseline: the regression the network has to beat: logistic for a yes/no outcome,
multinomial logistic for a category, multiple regression for a number.
Early stopping: keep the network from the epoch where validation loss was lowest,
not the last one.
Paired bootstrap interval: how much the gap between the two models would move with a
different set of held-back customers. If it includes zero, there is no real difference.
π§ When to use it, and when not to
Use it on a table with one row per customer (or account, or market) and a dozen or so columns, when you
want to know whether a flexible model finds something a regression misses.
Fewer than about 300 rows: a network will mostly memorize. Start with a regression.
Rows over time for one series: use the time-series tools.
Step 1DATA SOURCE
Start with a case: each one was generated from a known formula, so at the end you can check what the network
found against the truth. Or bring a customer file of your own.
π
Use a Case Study
Five cases: a coffee subscription deciding who gets a save offer, a catalog mailer told to "use AI",
a software trial with a network far too big for its data, a streaming service choosing which plan to
lead with, and a mattress brand asking what more social spend would buy.
πUse a Case Studyclick to switch
π€
Upload Your Data
Drag & Drop raw data file (.csv, .tsv, .txt)
One row per customer: an outcome column (yes/no, a category with 3 to 8
levels, or a number) and the columns to predict it from. Numbers can be on any scale; text columns
become 0/1 columns automatically. Leave out IDs, dates, free text, and anything recorded after the
outcome. At least 150 rows.
No file uploaded.
π€Upload Your Dataclick to switch
INPUTS & SETTINGS
Step 2Tell the model what to predict, and from what
Pick the outcome, then tick the columns to predict it from. Both models (the network and the regression)
use exactly the same columns and the same customers, so the comparison is fair.
Predictors
No data loaded.
Step 3Set up the network, the split and the baseline
The defaults are a sensible small network. Change one thing at a time and train again, so you can see what
that one change did.
How the customers are divided
The network learns from the training rows. The validation rows only decide when to stop. The
holdout rows are kept back until training is over, and every number in the results comes from them.
The network
Training
The baseline to beat
The regression gives every chosen column one straight-line effect of its own.
Up to four. A squared term lets the regression curve; a product lets one input's effect
depend on another's.
Advanced
π€ Try this:
Load Ridgeline Roasters and train. Note the network's log loss and the logistic regression's, and
read the verdict.
Add two terms to the baseline: emails_per_month squared, and
save_offer Γ lifecycle_stage. The network does not need to train again.
Now who wins? Look at what the network's emails curve looked like, and ask why those two terms were
the right ones to add.
Step 4TRAINING
Press Train and watch both curves. The network only ever learns from the training customers; the validation
line shows how it is doing on customers it is not learning from. That second line is the one that matters.
Load a case or upload a
file to begin.
Parameters per training row:–
Estimated training time:–
Loss on the training and validation customers, epoch by epoch
Down is good: each line is the average penalty for the network's predictions, where a confident miss costs
a lot. The solid line is the customers it learns from, the dashed line the validation customers it does
not. The marker shows the epoch where the dashed line was lowest.
π How to Read This Chart
The training line almost always keeps falling: the network is getting better at the customers it is
memorizing.
When the dashed line turns up while the solid one keeps falling, the network has started learning the
training customers' noise. Everything after that point makes it worse on new customers.
The gap between the two lines at the best epoch is normal. A gap that keeps widening is the warning.
Step 5IS IT WORTH IT?
Training is over, so the holdout customers are finally used, once, for every number below. Neither model
has seen any of them.
The network against the regression, on customers neither saw
Shorter bars are better. Each bar is the average penalty on the held-back customers. The no-model guess is
the floor: a model that cannot beat it has learned nothing.
π How to Read This Chart
The interval under the chart is the one to read. It says how much the gap would move with a
different set of held-back customers.
If the interval includes zero, the two models are not really different here: keep the one you can
explain.
A gap that is real but tiny may still not be worth what the network costs you in explanation.
Step 6INSIDE THE MODEL
A network has no coefficients to read, so we ask it questions instead. Which inputs does it lean on? And
when one input changes, which way do its predictions move?
Which inputs the predictions lean on
Longer bars mean the predictions depend more on that input: scramble its values across the held-back
customers and the error rises by this much. A bar near zero means the model barely uses it. The thin lines
show how much the answer moved between shuffles.
π How to Read This Chart
Importance says how much an input matters to the predictions. It does not say whether more of it
raises or lowers the outcome; the next chart does.
It does not say the input causes anything. An input can stand in for something the data does not
have.
With few held-back customers the bars move a lot from one seed to the next.
What happens to the prediction as one input changes
Each line sets the chosen input to every value along the bottom for every held-back customer, keeps their
other columns as they are, and averages the predictions. A flat line means the model ignores the input;
a bend means it found a curve.
π How to Read This Chart
The line is an average over customers. Two groups moving in opposite directions can average out to
flat.
The shaded stretch is past the largest value in the training data. Neither model has any evidence
there; whatever its line does comes from how the model was built.
Compare the shapes: a straight regression line next to a bent network line is the structure the
network found.
Step 7WHAT THIS MEANS
Start with the plain reading, then the two write-ups. The numbers below are the ones to quote.
Did the network beat the regression on new customers?
–
Baseline minus network
Train the network to see what this means here.
Where should training have stopped?
–
Best epoch
What does it lean on most?
–
Top input by importance
Start here
Managerial Interpretation
Train the network to see the managerial interpretation.
APA-Style Report
Train the network to see the APA-style report.
REVEAL THE TRUTH
This case was generated from a known formula. Decide what you think the network found first, then compare
it with the planted drivers and shapes.
Step 8SCORE A NEW FILE
Upload customers you want predictions for. Each row is prepared exactly the way the training rows were (same categories, same scaling). A
row with a number beyond anything the network trained on is still scored, but flagged: there, the network
is guessing.
DIAGNOSTICS & ASSUMPTIONS
Each check says what it found and what to do about it. A failed check is a next step, not a verdict on the
data.
How the network's held-back predictions line up with what happened
For a yes/no outcome: predictions grouped by how confident they were, against how often the outcome
happened. For a category: which category was predicted against which happened. For a number: each
customer's miss against the prediction.
π How to Read This Chart
Yes/no: points on the diagonal mean a 30% prediction happens about 30% of the time.
Category: the diagonal holds the customers predicted correctly; read across a row to see where
the rest went.
Number: misses scattered evenly around zero are good; a fan or a curve means the model is wrong
in a pattern.
EXPLORATORY QUESTIONS
On Ridgeline, the network wins and emails is its top input. Does that mean sending more emails causes
cancellations? What does the emails curve say instead?
On Ridgeline, add the two terms from the Try this box. Why is it good news when the regression wins
back?
On Copperline, change the seed a few times and train again. Would you switch the mailing list to the
network? What would it cost you to explain it?
On Tallyhop, where should training have stopped? Turn early stopping on, then try weight decay alone.
Which one rescues the network, and why?
On Kinoscope, accuracy barely moves between the two models. Which score shows the network's gain, and
what is it measuring that accuracy is not?
On Wrenfield, show the curve beyond the data, then switch tanh to ReLU and train again. Which model
would you trust at $60k, and what would you do instead of trusting either?
LEARNING RESOURCES
π Going further
Bigger models: gradient-boosted trees usually beat both a small network and a
regression on tables like these. The honest comparison is the same one: held-back rows, against the
simplest model.
Uncertainty: ensembles or conformal prediction give a network prediction
intervals. This lab does not.
Single customers: SHAP values extend permutation importance to one customer at a
time, with the same caveat: importance is not cause.