I've been meaning to write down how I actually think about system design for a while. Not the interview-flashcard version, but the mental models I reach for when someone asks "will this hold up when we have a million users?" So I'm turning it into a short series. This first post is about the foundations — scaling, latency, throughput, and availability — because almost every design decision downstream comes back to these four ideas.
If you've ever nodded along to "just add more servers" without being totally sure what that buys you, this one's for you.
What system design actually is
System design is the art of arranging components — servers, databases, caches, queues, networks — so that a system does what it's supposed to do and keeps doing it as load, data, and complexity grow. That's the whole job. Anyone can make something work for ten users. The interesting problems start when you need it to work for ten million, survive a server catching fire, and still be something a human can reason about six months later.
There's no single "correct" design, which trips people up at first. Every choice is a trade-off — faster usually costs more, simpler usually scales less far, more consistent usually means less available. Good system design isn't about knowing the one right answer; it's about understanding the trade-offs well enough to pick the right one for your situation.
Scaling: up vs out
When your system gets slow under load, you have two fundamentally different moves.
Vertical scaling (scaling up) means making your existing machine bigger — more CPU, more RAM, faster disks. It's the easy button: no code changes, no distributed-systems headaches. The catch is that there's a ceiling. You can only buy so much machine, it gets expensive fast at the top end, and that one big box is a single point of failure. If it dies, everything dies.
Horizontal scaling (scaling out) means adding more machines and spreading the work across them. This is how every large system on the planet actually scales, because there's effectively no ceiling — need more capacity, add more boxes. The cost is complexity: now you have to distribute load, keep data consistent across machines, and handle the reality that any individual node can fail at any moment.
My rule of thumb: scale up first because it's cheap and simple, and design so you can scale out before you're forced to. The worst time to discover your architecture can't go horizontal is the day you desperately need it to.
Performance vs scalability
People use these interchangeably, and they're not the same thing.
Performance is how fast the system is for a single user right now. Scalability is what happens to that performance as you pile on more users or more data. A system can be fast and not scalable — blazing for one user, falling over at a thousand. Or slow but scalable — never great, but it degrades gracefully forever.
The clean way to say it: if you have a performance problem, your system is slow for a single user. If you have a scalability problem, it's fast for one user and slow under load. They have completely different fixes, so it's worth knowing which one you actually have before you start optimizing.
Latency vs throughput
Another pair that gets muddled. Latency is how long one operation takes — the time from request to response. Throughput is how many operations you can handle per unit of time.
The classic analogy is a highway. Latency is how long it takes one car to drive from A to B. Throughput is how many cars pass through per hour. Widening the road (more lanes) increases throughput without making any single car faster. And here's the counterintuitive part: optimizing one can hurt the other. Batching requests together raises throughput but adds latency to each individual request, because now it waits for the batch.
Generally you want to maximize throughput while keeping latency at an acceptable level. "Acceptable" is the operative word — users don't need instant, they need fast enough, and knowing your real latency budget keeps you from over-engineering.
The latency numbers worth memorizing
Every engineer should have a rough feel for how long things take, because it shapes what's expensive. Reading from memory is on the order of nanoseconds. Reading from an SSD is microseconds. A spinning disk seek is milliseconds. And a network round trip across the world is hundreds of milliseconds.
The takeaway isn't the exact figures — it's the orders of magnitude between them. Memory is dramatically faster than disk, which is dramatically faster than the network. This single fact is why caching exists, why we dread chatty cross-region calls, and why "just add a database query" in a hot loop can quietly wreck your latency. When a design feels slow, I mentally trace where the data is coming from, and the answer is almost always "somewhere too far away, too many times."
Availability and redundancy
Availability is the percentage of time your system is up and working. It gets talked about in "nines" — 99.9% availability ("three nines") allows about 8.7 hours of downtime a year; 99.99% ("four nines") cuts that to under an hour. Each extra nine is exponentially harder and more expensive to reach, so be honest about how many you actually need. A hobby project does not need the uptime of a payment processor.
The way you buy availability is redundancy — eliminating single points of failure so that when (not if) something breaks, something else takes over. Two servers instead of one, replicated databases, multiple data centers. The core mindset shift is designing for failure as the normal case, not the exception. In a big enough system, something is always broken; the goal is that users never notice.
There's a related idea worth naming: fail-over. When a component dies, traffic automatically shifts to a healthy standby. Whether that standby sits idle waiting (active-passive) or already shares the load (active-active) is a cost-vs-complexity call, but the principle is the same — no single thing should be able to take the whole system down.
Where this leaves us
Everything else in system design is built on these four pillars. Scaling decisions determine your architecture's shape. Latency and throughput determine what "fast enough" means and what you optimize. Availability and redundancy determine how you survive the inevitable failures. Keep them in mind and most design conversations get a lot less mysterious — you start seeing the trade-offs instead of memorizing answers.
In the next part of this series, I'll get practical about back-of-the-envelope estimation — how to quickly sanity-check whether a design can actually handle the numbers — and walk through the framework I use to structure a system design discussion from a blank page.
