How websites handle millions of users
One server, then a bigger server, then a load balancer, health checks, stateless replicas, queues, replicas, a CDN and finally several continents — the whole scaling ladder, one rung at a time.
Your first users
At the beginning there is one server and a few hundred visitors, and it is glorious. The web app, the database and the uploaded files all live on a single box. This works beautifully right up until the day it very suddenly does not.
One server
Watch what actually breaks first, because it is rarely the processor. It is memory, or the number of database connections, or one slow query holding a lock while eight hundred requests pile up behind it. Find the real bottleneck before you buy anything.
A bigger server
The first instinct is to scale up: more cores, more memory, faster disks. It needs no code changes and it is the right answer far more often than engineers like to admit. But price climbs faster than power, and there is always a largest machine.
Load balancer
So instead you scale out. A load balancer sits in front and spreads requests across many identical servers. A single modern balancer can handle millions of connections, and it has now become the one thing in your system that must never fall over.
App server one
The servers behind it are clones, and this discipline is what makes the whole thing work. Any request must be servable by any server. The moment machine four knows something machine five does not, the plan quietly starts falling apart.
App server two
Round robin simply deals requests out in turn. Least connections sends the next one to whoever is least busy, which is much better when some requests take a millisecond and others take two seconds. Sticky sessions are a last resort, never a design.
Health checks
Every few seconds the balancer asks each server a simple question and expects a fast answer. A server that fails twice is pulled from rotation and nobody notices. Keep that check shallow, though: if it also tests the database, one slow database fails every server at once and the whole pool empties.
Shared session store
Statelessness is the real unlock. Sessions, uploads and caches move off the machines into shared storage, so any server can serve anyone. Only then does a server become disposable, and only disposable servers can be created and destroyed at will.
Autoscaler
Now capacity can follow demand on its own. A metric crosses a line and new instances appear, usually within one to three minutes. That lag is why you scale on a leading signal like the depth of the waiting queue, rather than waiting until processor use is already pinned at its limit.
Background job queue
Anything slow gets off the request path entirely. Sending email, resizing images, building reports: drop a job on a queue and answer the user immediately. Workers drain it at their own pace, so a traffic spike becomes a longer line rather than an outage.
Object cache
The cheapest request is the one you never compute. Keep results in memory and a page that took two hundred milliseconds takes two. The hard part was never caching, it is deciding the moment a cached answer quietly turns into a lie.
Read replicas
Web servers multiply easily. The database does not. Read replicas copy the data so hundreds of readers can spread out, and since most applications read far more than they write, this single move buys years of headroom for very little effort.
Primary database
Writes still land in exactly one place. When that place fills up you shard, splitting rows across machines by customer or region, and every question that crosses a shard gets harder from then on. Delay this step as long as you honestly can.
Edge network
Meanwhile most of what a page weighs is not your code at all. Images, scripts and video get copied to hundreds of edge locations, so the bytes travel fifty kilometres instead of eight thousand and your own servers never see that traffic.
More than one region
The last rung is geography. Run the whole stack in several regions and send each visitor to the closest one, cutting a round trip from three hundred milliseconds to thirty. That is how a site serves millions of people and still feels like it is next door.
Watch it explain itself
Every step above is narrated aloud. Play it, or open it in the editor and make it yours — no account needed.