How a neural network learns
From a single neuron to backpropagation and epochs — the ideas behind machine learning, narrated one concept at a time.
Neural network
A neural network is not a brain. It is a very large pile of multiplication, and millions of tiny numbers inside it get nudged over and over until the pile happens to turn photographs into the word cat. Here is how that nudging works.
Input layer
Everything begins as numbers. A small grey image of a handwritten digit, twenty eight pixels by twenty eight pixels, becomes seven hundred and eighty four inputs, one per pixel, each a brightness somewhere between zero and one.
One neuron
A single neuron is embarrassingly simple. It takes every incoming number, scales each one, adds the results together, and pushes the total through one small function. Everything a network can ever do is built out of that one move.
Weights
The weights are those scales, one for every connection, and they are the part that actually gets learned. A modern image model can carry tens of millions of them, and at the start they are just small random numbers.
Bias
The bias is one extra number added after the sum. It lets a neuron speak up even when its inputs are quiet, rather like a door that swings open with the lightest push because the hinge is already loaded.
Weighted sum
Multiply every input by its weight, add the bias, and you have one number. A layer of a thousand neurons fed by a thousand inputs needs a million multiplications for a single pass, which is exactly why these models are trained on graphics cards.
Activation function
Now the crucial twist. The sum runs through a rectifier, a function that throws away negatives and passes positives through unchanged. Without that little bend, stacking a hundred layers would collapse into a single straight line and learn nothing interesting.
Hidden layers
Stack those layers and something strange emerges. The first hidden layer finds edges, the next finds corners and curves, deeper ones respond to eyes and wheels and faces. Nobody programs those features; they appear on their own.
Output layer
The final layer holds one neuron per possible answer. For handwritten digits that is ten neurons, and a step called softmax turns their raw scores into probabilities that always add up to exactly one.
The prediction
Now the network commits. It might report an eighty seven percent chance this digit is a seven and a nine percent chance it is a one. Early in training that confident guess is complete nonsense, and that is the point.
Loss function
The loss function scores exactly how wrong the guess was, boiling it down to a single number. Confident and right costs almost nothing; confident and wrong costs a great deal. All of training is driving that one number down.
Backpropagation
Backpropagation runs the network in reverse, using the chain rule from calculus to ask every single weight the same question. If I nudged you upward a little, would the loss go up or down, and by how much?
Gradient descent
The answer is a gradient, an arrow pointing straight uphill. Every weight then takes one small step the opposite way, downhill. It is walking off a mountain in thick fog by feeling the slope under your feet, and it genuinely works.
Learning rate
The learning rate sets how big that step is, often a value like zero point zero zero one. Too large and the model bounces straight back out of the valley; too small and training drags on for weeks instead of hours.
Epochs
One epoch is one complete pass over the training data. Real models run for dozens of epochs across millions of examples, so a single weight can be nudged billions of times before it finally settles somewhere useful.
A trained model
What remains at the end is just an enormous table of numbers. No rules were written and no cat was ever described. The network saw only examples and its own mistakes, and it turns out that is enough.
Watch it explain itself
Every step above is narrated aloud. Play it, or open it in the editor and make it yours — no account needed.