We keep saying "just add more servers." This post is about the part nobody explains when they say that: how a request actually finds its way to one of those servers, and how the traffic gets spread so no single machine drowns. It's the plumbing of every scalable system, and understanding it removes a lot of hand-waving.
This is part three of my system design fundamentals series, following the posts on scaling and on estimation.
A request's journey, briefly
When a client talks to your server, the data passes through a stack of layers — physical, network, transport, application — usually described by the OSI or TCP/IP models. You don't need to recite all seven layers, but two matter constantly in design conversations. The transport layer (TCP) is where connections and reliability live. The application layer (HTTP) is where your actual API lives. Most of the choices you'll make happen at these two levels, and knowing which layer you're operating at clears up a surprising amount of confusion later.
HTTP is the workhorse — a request/response protocol built on TCP. HTTPS is the same thing wrapped in TLS encryption, which is non-negotiable in practice; there's no good reason to ship plaintext HTTP anymore.
Choosing how your services talk: API styles
How your clients and services communicate is an early, load-bearing decision. The main options each have a personality.
REST is the default for a reason: simple, stateless, cacheable, and everyone understands it. You model resources and act on them with HTTP verbs. It's a great fit for public APIs and the vast majority of CRUD-style services. The downside is over- and under-fetching — you often get more data than you need, or have to make several calls to assemble one screen.
GraphQL answers that by letting the client ask for exactly the fields it wants in a single query. It's lovely for complex, nested data and mobile clients on slow networks. The cost is server-side complexity and caching that's harder than REST's, since every query can be different.
gRPC is the choice for fast internal service-to-service communication. It uses a compact binary format and HTTP/2, supports streaming, and is strongly typed via schemas. It's excellent between microservices and awkward directly from browsers, so it usually lives inside your system rather than at the edge.
WebSockets are different in kind: a persistent, two-way connection for real-time features — chat, live dashboards, multiplayer. When the server needs to push to the client rather than wait to be asked, this is the tool. The trade-off is that long-lived connections are stateful and harder to scale than stateless HTTP.
There's no universal winner. REST at the edge, gRPC between services, and WebSockets for anything real-time is a combination I reach for often.
A word on authentication
Once services talk over the network, you have to prove who's calling. Two terms come up endlessly. OAuth 2.0 is an authorization framework — it's how "log in with Google" works, handing your app a scoped token instead of a user's password. JWT (JSON Web Token) is a compact, signed token that carries claims (who you are, what you can do) and can be verified without a database lookup, which makes it popular for stateless auth across services. They solve related but distinct problems, and you'll frequently see them used together.
Load balancing: the traffic cop
Now the main event. A load balancer sits in front of your servers and distributes incoming requests across them. It's what makes "add more servers" actually mean something — without it, clients wouldn't know the extra machines exist. It also improves availability: if a server goes unhealthy, the balancer stops sending traffic there, and users never notice.
How it decides where to send each request
The distribution algorithm matters more than people expect.
- Round robin — hand requests to servers in rotation. Dead simple, works well when servers are equal and requests are cheap.
- Weighted round robin — same idea, but bigger servers get a larger share. Useful when your fleet is a mix of machine sizes.
- Least connections — send the next request to whichever server currently has the fewest active connections. Better when request durations vary a lot, since it naturally avoids piling work on a busy box.
- IP hash — route based on a hash of the client's IP, so a given client keeps landing on the same server. Handy when some state lives on the server.
- Consistent hashing — a smarter hashing scheme that, when you add or remove a server, remaps only a small fraction of traffic instead of reshuffling everything. It's important enough that it deserves its own discussion, but for now: it's how you scale a hashed fleet without a stampede every time the fleet changes size.
Layer 4 vs layer 7
Load balancers come in two flavors. A layer 4 balancer works at the transport level — it forwards packets based on IP and port without looking inside them. It's fast and cheap because it doesn't inspect content. A layer 7 balancer works at the application level — it can read the HTTP request and route based on the URL path, headers, or cookies. That's more powerful (send /api traffic here, /images there) but does more work per request. Most modern setups use layer 7 at the edge for its routing smarts, sometimes with layer 4 underneath for raw throughput.
Health checks and sticky sessions
Two features make load balancers trustworthy. Health checks are periodic pings to each server; when one stops responding correctly, the balancer pulls it out of rotation until it recovers. This is the mechanism behind "users never notice a server died."
Session persistence (sticky sessions) pins a user to the same server for their session, which you need if that server holds session state in memory. It works, but it undercuts even load distribution and makes failures more disruptive — if that server dies, the user's session goes with it. The cleaner long-term answer is to make your servers stateless and push session state into a shared store like Redis, so any server can handle any request and you don't need stickiness at all.
The mental model to keep
Networking and load balancing are how a system stays reachable and even under load. Pick API styles that fit each boundary — REST at the edge, gRPC inside, WebSockets for real-time. Put a smart load balancer in front, choose a distribution algorithm that matches your traffic, lean on health checks, and design your servers to be stateless so you can add and remove them freely. Get this layer right and horizontal scaling stops being a slogan and starts being something you can actually do.
Next up in the series: caching and CDNs — the single highest-leverage way to make a system faster, and the one place a small change can cut your database load by an order of magnitude.
