Scaling: Vertical vs Horizontal, and the Path to a Million Users

Every system starts on one box — one server, one database, serving a few hundred users. Traffic grows, the CPU pegs at 100%, requests queue, and you face the oldest question in backend engineering: buy a bigger box, or buy more boxes?

That fork — scale up or scale out — is what "scaling" means, and picking wrong is expensive: scale up too long and you hit a hardware ceiling with a single point of failure underneath; scale out too early and you inherit distributed-systems problems — consistency, partitioning, coordination — before the traffic justifies them.

This guide covers both axes, why stateless services make scale-out cheap, how databases scale once the app tier maxes out, and the failure modes that only appear with many nodes.

Vertical vs horizontal scaling#

Vertical scaling (scale-up) means giving one machine more resources — more CPU, RAM, faster NVMe disks. Nothing about your code changes: no load balancer, no partitioning, no distributed transactions. You reboot onto a bigger instance and move on.

The catch is a hard ceiling: instance sizes are finite, price scales worse than linearly near the top, and that one big box is a single point of failure. Cloud vendors sell some genuinely enormous machines, but you rarely reach the absolute maximum — a stateful primary usually saturates on IOPS or lock contention well before then, in practice around 64–128 cores or ~1TB of hot working set. That's the point where scaling up stops paying off and you have to scale out.

Horizontal scaling (scale-out) adds more machines behind a load balancer, each handling a slice of traffic. Throughput grows nearly linearly — double the nodes, roughly double the capacity — and losing one node degrades capacity instead of causing an outage. The price is complexity: routing requests, partitioning or replicating data, and coordinating any state shared between nodes.

The practical rule: scale stateless tiers horizontally from day one since it is nearly free, and scale stateful components (the primary database, an in-memory cache) vertically first, partitioning only once a single box genuinely saturates around 64–128 cores or 1TB of hot data.

Why stateless services scale so easily#

A service is stateless when it keeps no request-specific data in local process memory. Session tokens, auth context, and cart contents live in an external store — Redis, the database, or a signed JWT — not in a variable on one particular server.

That's what makes horizontal scaling trivial: any instance can serve any request, so the load balancer sends work wherever there's spare capacity. You can add nodes during a spike, kill them during a deploy, and autoscale on CPU or RPS without users noticing.

The opposite — a stateful service holding session data locally — forces sticky sessions, pinning each user to one server. Sticky routing wrecks elasticity: you cannot rebalance load freely, a node death loses live sessions, and deploys get harder. The fix is to push the state into a shared store, making the service stateless again.

Scaling the database: replicas vs sharding#

The app tier is easy. The database holds the state everything else offloaded, and two strategies dominate there.

Read replicas stream the primary's write-ahead log to N followers. Reads fan out across them; writes still go only to the primary — scaling reads close to linearly, ideal for read-heavy traffic. The cost is replication lag: followers trail by anywhere from a few milliseconds to seconds, so a read right after a write can return stale data. Replicas do nothing for write throughput.

Sharding partitions the keyspace across independent primaries — by hash of a key, by range, or by a lookup directory — so each shard owns a slice of the data and takes its own writes. This is the only strategy that scales writes, since you now have many primaries instead of one. The price is steep: cross-shard joins become application work, cross-shard transactions need distributed coordination, and rebalancing when you add a shard is genuinely painful.

The rough rule: reach for read replicas first when read:write is above roughly 5:1, and only shard once write throughput or dataset size exceeds a single primary — practically, north of ~10,000–50,000 writes/sec or ~1–2TB of hot data.

Consistent hashing: why hash-mod-N breaks#

Once you shard or run a distributed cache, you need to map each key to a node. The naive hash(key) % N works fine until N changes — then it detonates: going from N to N+1 remaps roughly (N-1)/N of all keys. Add one cache server and ~90% of keys point to the wrong place, causing a stampede of data movement.

Consistent hashing fixes this. Picture a ring of 2³² positions: each node sits at one or more points, and each key is routed to the first node found walking clockwise from its hash. Adding or removing a node now only reshuffles the keys between it and its neighbor — about 1/N of the keyspace instead of nearly all of it.

Each physical node also gets many virtual nodes (typically 100–200 points), smoothing out the clustering random placement would otherwise cause. This is the routing scheme behind DynamoDB, Cassandra, the ketama Memcached clients, and Envoy's ring-hash load balancer.

Walkthrough: sizing the path from 200k to 2M writes/sec#

Numbers turn vague architecture into concrete decisions. Take a real one: a service doing 200,000 writes/sec against a single Postgres primary now saturating disk IOPS, aiming for a 10x jump to 2,000,000 writes/sec. Here is the decision trace.

The load-bearing line is step 2's arithmetic — but it only closes because of step 3. Taken alone, 62,500 writes/sec per shard sits above the ~10,000–50,000 writes/sec trigger where an un-batched primary would itself need further sharding, so the naive number does not clear the bar. Batching is what reconciles it: issuing writes via COPY and multi-row INSERT amortizes the WAL fsync cost across many rows, lifting a single NVMe-backed primary's effective envelope to roughly 60,000–120,000 writes/sec. Against that batched envelope, 62,500 lands comfortably inside, so the plan is credible rather than aspirational. The discipline — divide the target by shard count, then check the result against the envelope you'll actually be running (batched, not per-statement) — is exactly the reasoning an interviewer wants to hear out loud.

The failure modes that only appear at scale#

Horizontal scaling introduces problems that don't exist on one box.

Throughput does not buy you latency. Adding workers raises aggregate requests-per-second, but it cannot shorten the critical path of a single request. Worse, fan-out amplifies tail latency — the tail-at-scale effect described by Dean and Barroso. A request that hits 100 shards, each with a p99 of 10ms, sees an effective p99 closer to the p99.99 of a single shard, often 50–100ms, because it waits on the slowest of the hundred. Mitigations are hedged requests, less fan-out, and caching hot results — not more capacity, which can worsen p99 via queueing variance.

Overload needs back-pressure. A saturated component must signal upstream to slow down — via bounded queues, 429/503 rejections, or shrinking TCP windows. Without it, queue depth grows unbounded, latency explodes, timeouts fire, and clients retry, piling more load onto the thing already drowning — a retry storm that takes down an entire fleet. Bounded queues with load shedding, token-bucket rate limits, and circuit breakers keep a partial failure from becoming a total one.

How this shows up in interviews#

Scaling questions are rarely "draw the architecture" — they're "you're at X, get to 10X," and the interviewer probes whether your diagram survives real numbers. Expect to be pushed on the partition key (does it spread load evenly, or create a hot shard?), on replication lag (can this read tolerate stale data?), and on what breaks first as traffic multiplies.

The strongest answers start from requirements: state the read:write ratio, estimate peak requests-per-second, then choose replicas, shards, caches, or queues — each justified by a number. Say which trade-off you would reverse if the workload changed; that's what separates memorizing an architecture from being able to operate one.

Further learning#

• System Design BASICS: Horizontal vs. Vertical Scaling — Gaurav Sen's visual intro to the two scaling axes.
• From 0 to Millions: A Guide to Scaling Your App — ByteByteGo's staged walkthrough from a single server to a multi-region, sharded system.
• Practice the full roadmap: SQL vs NoSQL, CAP Theorem, Caching, and Load Balancing.
• Step through it interactively on YeetCode's scaling roadmap.

FAQ#

When should I scale vertically instead of horizontally?#

Scale vertically while state is expensive to distribute and you have not hit the hardware ceiling — a primary database or in-memory cache is usually simpler on one large machine than sharded, until a single box saturates around 64–128 cores or 1TB of hot data. Scale stateless app and web tiers horizontally from day one instead: adding nodes behind a load balancer is nearly free and removes the single point of failure a big box represents.

What actually breaks when I add read replicas?#

Replicas scale reads but introduce staleness. A follower applies the primary's changes asynchronously, lagging by milliseconds to seconds, so a user who writes then immediately reads may see the old value if that read lands on a replica. Flows that need read-your-own-writes consistency should route those reads to the primary or use session pinning. Replicas also do nothing for write throughput — that's the signal it's time to shard.

Why can adding servers make my p99 latency worse?#

Because latency is bounded by a single request's critical path, which more servers widen via fan-out rather than shorten. A request waiting on 100 backends is only as fast as the slowest of them, so even a tidy 10ms-p99 backend can push the combined request to 50–100ms. Added capacity also adds queueing variance and coordination overhead. The fixes are architectural: touch fewer backends per request, send hedged requests to duck slow tails, and cache hot results so the fan-out never happens.

What is back-pressure and why does it matter at scale?#

Think of it as flow control between services: when a downstream component runs out of headroom, back-pressure is the signal that stops upstream callers from pushing more work than it can absorb. The reason it becomes non-negotiable across many nodes is arithmetic — one slow dependency, multiplied across a fleet of retrying clients, turns a local hiccup into a self-reinforcing overload, where each failed attempt piles on load that makes the next attempt more likely to fail too. The tempting non-fixes — uncapped retries, unbounded in-memory buffers — actively make this worse, because they hide saturation until the component tips over all at once. The durable answer is to make limits explicit and, ideally, adaptive: cap in-flight work rather than guessing a static queue size, and let a controller like Netflix's adaptive-concurrency library probe the real throughput sweet spot via Little's Law as conditions shift. Done well, an overloaded system degrades to "slower, and rejecting some requests" instead of falling over entirely.