How a machine learning model is trained
The real pipeline behind a working model — collecting and cleaning data, splitting it honestly, training, tuning, testing, deploying and watching for drift.
A question worth predicting
Machine learning does not begin with code. It begins with a question you can actually score. Will this customer cancel? Is this scan showing a tumour? If nobody can say whether an answer was right, there is nothing here to learn from.
Collect the raw data
Now go and gather examples, thousands of them. Teams routinely spend most of a project right here, and the uncomfortable truth of the field is that more and better data usually beats a cleverer algorithm.
Label the examples
Someone has to say what the correct answer was. Labelling a hundred thousand images costs real money and months of human attention, and every disagreement between two labellers quietly sets a ceiling on how good the model can ever be.
Clean out the mess
Real data is filthy. Missing values, duplicated rows, dates written three different ways, a body temperature recorded as minus nine hundred and ninety nine. Left alone, every one of those teaches the model something confidently and permanently wrong.
Build the features
Feature engineering turns raw columns into signal. A date of birth is nearly useless to a model; the age it implies is not. One well chosen feature often lifts accuracy more than a whole month of fiddling with the model.
Split into three sets
Before any training, carve the data into three parts, commonly eighty, ten and ten. The training set teaches, the validation set guides your choices, and the test set is locked in a drawer and not opened until the very end.
Choose a model
Pick the simplest thing that might work. For tables of numbers, gradient boosted trees still win most competitions; for images and language, neural networks. Starting simple gives you a baseline that everything fancier has to beat.
Train on the training set
Training is a loop. Predict, measure the error, nudge the parameters, repeat. It might run for two minutes on a laptop or for six weeks across thousands of graphics cards, but the loop itself is exactly the same shape.
Score the validation set
Now score the model on data it has never seen. The gap between training accuracy and validation accuracy is the single most informative number in the entire project, and experienced people look at it before anything else.
Overfitting?
If it is brilliant on training data and merely average on validation data, it memorised rather than learned. Researchers demonstrating this trained a husky and wolf classifier that scored well by ignoring the animals entirely and simply checking whether there was snow in the background.
Tune the hyperparameters
Hyperparameters are the dials you set before training starts. Tree depth, learning rate, how hard to penalise complexity. Search them properly with cross validation rather than by hand, then train again and see whether the gap has closed.
One run on the test set
The test set is used exactly once. Peek at it repeatedly and you begin tuning to it as well, and your honest estimate of real world performance quietly evaporates without anybody noticing it has gone.
Deploy behind an API
Now the genuinely hard part. The model has to answer in milliseconds, compute its features exactly the way the training pipeline did, and fail gracefully. Most models that die in production die of a mismatch between those two pipelines.
Watch it in production
Watch latency, watch errors, and watch the shape of the incoming data. Watch what the predictions do to the world as well, because a model that recommends products starts changing the very behaviour it is trying to predict.
Has the data drifted?
The world keeps moving. Prices rise, fashions turn, a pandemic arrives, and a model trained on last year's reality slowly stops working. That slide is called drift, and it is completely silent unless you measure it deliberately.
The loop starts again
So a trained model is never really finished. It is a loop that keeps turning, with fresh data, fresh labels and fresh training runs, for as long as anyone out there still depends on the answer.
Watch it explain itself
Every step above is narrated aloud. Play it, or open it in the editor and make it yours — no account needed.