Every previous post in this series has been quietly leading here. The moment you spread a system across more than one machine — replicas, shards, multiple services — you're doing distributed systems, and distributed systems have their own laws of physics. This final part is about those laws: why the network will betray you, what the CAP theorem really forces you to choose, and how message queues and microservices help you cope.
This wraps up my system design fundamentals series.
The fallacies that get everyone
Decades ago, engineers at Sun compiled a list of false assumptions that architects new to distributed systems almost always make. They're still true, and still routinely ignored. The gist: the network is not reliable, latency is not zero, bandwidth is not infinite, the network is not secure, and topology, admins, and transport cost are not constants you can forget about.
Why does this matter? Because a huge share of distributed-systems bugs come from writing code that pretends a network call is a local function call. It isn't. It can be slow, it can fail halfway, it can succeed but never tell you. Internalizing "the network is a hostile place where anything can fail at any time" is the mental shift that separates people who design robust distributed systems from people who build demos that fall over in production.
The CAP theorem: pick two, but really you pick one
The CAP theorem is the most cited idea in distributed systems and the most misunderstood. It says that when your data is spread across machines, you can have at most two of three properties: Consistency (every read sees the most recent write), Availability (every request gets a response), and Partition tolerance (the system keeps working even when the network between nodes breaks).
Here's the part people miss: in a real distributed system, network partitions will happen — cables break, nodes get isolated. So partition tolerance isn't optional; you must have it. Which means CAP is really a choice between two options during a partition:
- CP (consistency over availability) — when nodes can't talk to each other, refuse to serve possibly-stale data. Some requests fail, but you never return wrong data. Right for banking, inventory, anything where a wrong answer is worse than no answer.
- AP (availability over consistency) — when nodes can't talk, keep serving from whatever data you have, even if it might be slightly stale, and reconcile later. Right for social feeds, likes, product catalogs — where staying up matters more than being perfectly current.
The theorem isn't telling you which to pick; it's telling you the choice is unavoidable. Pretending you can have all three is how systems get designed on a lie.
Consistency models: it's a spectrum, not a switch
"Consistency" isn't binary. There's a range, and where you land is a deliberate choice.
Strong consistency means everyone always sees the latest write immediately. Simplest to reason about, but it costs latency and availability, because nodes must coordinate before answering.
Eventual consistency means if writes stop, all replicas eventually converge to the same value — but for a little while they may disagree. This sounds scary and is actually fine for a lot of data. Your follower count being off by one for two seconds harms no one. Eventual consistency is what lets AP systems stay fast and available.
Causal consistency is a useful middle ground: operations that are causally related (a reply must come after the message it replies to) are seen in the right order everywhere, while unrelated operations can be seen in any order. It preserves the ordering that actually matters to users without paying for full strong consistency.
Related is the ACID vs BASE framing. Traditional databases give you ACID (strongly consistent, transactional). Many distributed systems instead offer BASE — Basically Available, Soft state, Eventually consistent — trading strict consistency for availability and scale. Neither is "better"; they're different points on the same trade-off you keep meeting throughout this series.
Message queues: decoupling in time
Once you have many moving parts, making them all talk synchronously — each waiting on the next — is fragile. If one service is slow or down, everything upstream stalls. Message queues fix this by letting services communicate asynchronously: a producer drops a message on the queue and moves on; a consumer picks it up when it's ready.
This buys you three big things. Decoupling — the producer doesn't need to know or wait for the consumer. Buffering — a traffic spike piles up in the queue instead of crushing a downstream service; it drains at its own pace. Resilience — if a consumer crashes, messages wait safely in the queue until it recovers, instead of being lost.
The two names you'll hear: Kafka, a distributed log built for enormous throughput and for keeping a replayable history of events, and RabbitMQ, a traditional message broker great for flexible routing and per-message workflows. Rough rule: Kafka for high-volume event streaming and event sourcing; RabbitMQ for classic task queues and complex routing.
This leads naturally to event-driven architecture, where services react to events ("order placed," "payment received") rather than calling each other directly. It scales beautifully and decouples teams — but the flip side is that the flow of logic is now spread across many services and a queue, which is genuinely harder to trace and debug. Asynchronous power comes with asynchronous confusion.
Microservices: a solution and a tax
Finally, how do you structure a large system? The old default was a monolith — one codebase, one deployable. Monoliths get a bad rap they don't fully deserve: they're simple to build, test, and deploy, and for a small team they're often the right call. Their pain shows up at scale — one huge codebase many teams trip over, and you can't scale one hot piece independently of the rest.
Microservices split the system into small, independently deployable services, each owning one capability and often its own database. The wins are real: teams work and deploy independently, you scale each service to its own load, and one service failing needn't take down the rest. But the tax is equally real, and it's easy to underestimate. You've traded in-process function calls for network calls (see: the fallacies), simple local transactions for hard distributed ones, and one thing to operate for dozens. You've made an organizational problem into a distributed-systems problem.
My honest take: most teams should start with a monolith and extract microservices when a specific part genuinely needs independent scaling or ownership. Splitting too early buys you all of the operational cost and none of the benefit. Microservices are a scaling tool for organizations as much as systems — reach for them when the org actually feels the pain a monolith causes.
Two supporting pieces you'll meet in that world: an API gateway is the single front door to your services, handling routing, authentication, and rate limiting so each service doesn't reimplement them. A service mesh manages the messy service-to-service communication — retries, load balancing, observability — as infrastructure rather than code in every service. Both exist to tame the complexity microservices create; neither is a reason to adopt microservices in the first place.
Wrapping up the series
Distributed systems don't hand you new powers so much as new constraints. The network is unreliable, so design for failure. You can't have perfect consistency and availability during a partition, so choose deliberately. Loosen consistency where it's safe, use queues to decouple in time, and reach for microservices only when the pain of not having them is real.
If there's one thread running through all six parts of this series, it's this: system design is the discipline of choosing trade-offs on purpose. Scaling, latency, caching, databases, distribution — none of them have a universally right answer. The engineers who are good at this aren't the ones who memorized the answers. They're the ones who can look at a problem, see the trade-offs clearly, and pick the ones that fit. That's the whole craft, and it's a genuinely fun one to keep getting better at.
