Caching: Strategies, Eviction, and the Failure Modes That Bite
How caching actually works in a real system — cache-aside vs write-through, LRU vs LFU eviction, thundering herd, and the stale-data bugs that bite in production.
This article has an interactive companion
Don't just read it — step through it interactively, one state change at a time.
A cache is a bet. You wager that reading a value is far more common than changing it, and that a slightly-out-of-date answer is cheaper than a slow one. When the bet pays off, a single Redis node absorbs traffic that would flatten a database. When it goes wrong, you serve stale data to users, take the backend down on a cache expiry, or spend a debugging afternoon chasing a value that "should have been invalidated."
Caching is the highest-leverage lever in system design and the one with the most sharp edges. The concept is trivial — keep a copy of expensive-to-compute data somewhere fast. Everything hard lives in the follow-up questions: where do you put the copy, when do you write it, how do you throw the old one away, and what happens when two requests race. Getting those four decisions right is the whole game.
Where does the copy live? The latency hierarchy
"Add a cache" is never one decision, because a request passes through half a dozen places where a copy could sit. Each layer trades a different amount of latency for a different blast radius — how many users a single cached entry serves.
| Layer | Typical latency | Shared across | Best for |
|---|---|---|---|
| Browser + service worker | ~0 ms | one user | static assets, last API response |
| CDN edge | ~10–50 ms | a region | images, public pages |
| Reverse proxy (Varnish/nginx) | ~1 ms | a data center | rendered HTML fragments |
| In-process (Caffeine, LRU-cache) | ~100 ns | one app instance | feature flags, config, hot lookups |
| Distributed (Redis/Memcached) | ~0.5–2 ms | all instances | sessions, hot DB queries |
| Materialized view in the DB | ~5–20 ms | everyone | precomputed aggregates |
The in-process cache is a thousand times faster than Redis because it never touches the network — but each instance has its own copy, so a 20-server fleet means 20 caches that can disagree. Redis is slower but shared, so every instance sees the same value and the same invalidation. The rule of thumb: cache at the layer that gives the best hit rate for the smallest blast radius. Static assets belong at the CDN. Session data and hot query results belong in Redis. Config that changes on deploy belongs in-process, where a little staleness is harmless.
When do you write it? Cache-aside vs write policies
The default pattern in almost every read-heavy web app is cache-aside (also called lazy loading). The application, not the cache, owns the logic:
read(key):
value = cache.get(key)
if value is null: # miss
value = db.query(key)
cache.set(key, value, ttl) # populate with an expiry
return value
write(key, newValue):
db.update(key, newValue)
cache.delete(key) # invalidate, don't updateCache-aside is the default for three reasons. It only ever caches data someone actually asked for, so cold or rarely-read rows never waste memory. It is failure-tolerant: if Redis goes down, every read becomes a slower DB read, but nothing breaks. And the app stays in control, so the cache never needs to know how to talk to the database. The price is a cold-start penalty — the first request for any key pays the full miss cost — and a race window on writes, which we'll get to.
When the write path itself matters, you pick a write policy that decides how the cache and database stay in sync:
| Policy | What happens on write | Wins | Loses |
|---|---|---|---|
| Write-through | Write cache and DB synchronously | Cache always fresh, simple reads | Every write pays DB latency |
| Write-around | Write straight to DB, cache fills on next read miss | Won't pollute cache with write-once data (logs) | First read after a write is cold |
| Write-back | Write cache now, flush to DB asynchronously | Lowest write latency, highest throughput | Data loss if the cache dies before flush |
Write-back is tempting because it makes writes feel instant, but it turns your cache into a source of truth that can vanish. Most teams reach for it only for tolerant workloads like metrics counters. For everything with real consistency needs, cache-aside with delete-on-write is the boring, correct default.
How do you throw the old copy away? Eviction policies
A cache is finite, so something has to leave when it fills. The eviction policy decides what, and the choice quietly determines your hit rate.
- LRU (least recently used) evicts whatever was touched longest ago. It's cheap — a doubly-linked list plus a hashmap gives O(1) updates — and it matches how most access patterns behave (recent things get re-read). Its weakness: one big sequential scan of cold data, like a nightly batch job reading every row once, evicts all your hot keys and trashes the cache.
- LFU (least frequently used) counts accesses and evicts the rarest. It shrugs off scans, because a one-time read never builds up a count. But it adapts slowly — an item that was popular last week keeps its high count and squats in memory long after interest died.
- ARC (Adaptive Replacement Cache) keeps two LRU lists, one for recently-seen and one for frequently-seen keys, plus "ghost" lists of recently-evicted keys, and shifts the boundary based on which list is producing hits. It gets LRU's recency-awareness and LFU's scan-resistance at once.
Modern high-performance caches like Caffeine use W-TinyLFU: a candidate for admission has to beat the entry it would evict on a compact frequency sketch before it's even allowed in. That admission filter beats ARC on most real workloads and is why Caffeine is the default JVM cache today. For most systems, though, plain LRU with a sensible TTL is entirely adequate — reach for the fancier policies only when profiling shows your hit rate collapsing under scans.
Walkthrough: a hot key under cache-aside
Numbers make the trade-offs concrete. Take a product page keyed product:42, backed by a DB read that costs 20 ms, cached in Redis (1 ms) with a 60-second TTL. Watch four requests hit it:
| t (s) | Request | cache.get | Result | Latency | Cache state after |
|---|---|---|---|---|---|
| 0.0 | R1 read | miss | query DB (20 ms), set product:42 ttl=60 | 21 ms | product:42 fresh, expires at 60 |
| 0.4 | R2 read | hit | serve from Redis | 1 ms | unchanged |
| 12.0 | Write: price change | — | db.update then cache.delete(product:42) | — | key gone |
| 12.1 | R3 read | miss | query DB (20 ms), repopulate | 21 ms | product:42 fresh, expires at 72 |
| 60.0 | (no write) key expires on its own | — | — | — | key gone |
R1 pays the full 20 ms miss and warms the cache. R2, 400 ms later, is a clean hit at 1 ms — a 20x speedup, and the entire point of the cache. The write at t=12 doesn't update the entry, it deletes it: delete-on-write is safer than update-on-write because you never have to compute the new cached shape, and the next read simply reloads the truth from the DB. R3 pays a fresh miss and warms it again. This is cache-aside's rhythm: one cold read per key per TTL window (or per write), and cheap hits for everything in between.
Now imagine product:42 is on the homepage and getting 5,000 requests/second when it expires at t=60. That's the next problem.
What happens when requests race? Herds and stale reads
Two failure modes separate people who've read about caching from people who've run one.
Thundering herd (cache stampede). A hot key expires. In the same millisecond, thousands of concurrent requests all miss, all decide to recompute, and all hammer the database with the identical query. The DB, sized for the 5% of traffic that normally misses, now takes 100% of it and falls over. Four standard defenses:
- Request coalescing (single-flight) — the first miss grabs a lock (
SET key val NXin Redis), recomputes, and everyone else waits or serves the stale value. Only one DB query happens. - Probabilistic early refresh (XFetch) — refresh a key before it expires, with a probability that rises as expiry approaches and scales with recompute cost, so the herd is spread out in time.
- Stale-while-revalidate — keep serving the expired value for a short grace window while one background request refreshes it. HTTP
Cache-Controland SWR-style client libraries do exactly this. - No TTL on the hottest keys — refresh them explicitly on write instead of letting them expire at all.
Stale data despite invalidation. You delete the cache on every write and still see old values. The classic culprit is a race: a reader loads the DB value at t=0, a writer commits and deletes the cache at t=1, and then the reader — still holding its now-stale value — populates the cache at t=2, resurrecting the old data past the invalidation. Fixes include versioned keys, compare-and-set with WATCH/MULTI, or Facebook's lease pattern, where a miss hands out a token and the cache only accepts a write carrying a valid lease. Replication lag is the other frequent cause: you invalidate, then a read hits a lagging replica, loads the pre-write value, and caches that. Reading from the primary right after a write, or waiting for replication, closes the gap.
How this shows up in interviews
Caching is where a system-design interview stops being a diagram and starts being an interrogation. The moment you write "add Redis here," expect the follow-ups: What's the TTL, and why that number? What happens when this key expires under load? How do you invalidate on write? What's your hit rate, and what does the DB see when it's zero?
The strongest answers quantify. If the read path takes 100,000 requests/second at a 95% hit rate, only 5,000 reach storage — say that out loud, because it shows you understand the cache as a load shield, not a speed trick. For a personalized feed, split the problem: cache the shared global candidate pool in Redis with a short TTL, precompute per-user feeds with fan-out-on-write for normal accounts and fan-out-on-read for celebrities to dodge write amplification, and let a CDN carry the static assets. Naming that hybrid — the Twitter/Instagram model — signals you've seen the real thing, not just a textbook diagram.
Further learning
- Cache Systems Every Developer Should Know — ByteByteGo's visual tour of where caches sit and why.
- Top Caching Strategies — ByteByteGo's written breakdown of the read and write patterns.
- Related roadmap topics: Load Balancing, CDN, Message Queues, and Consistent Hashing — the last of which is how a distributed cache decides which node owns each key.
- Practice caching on the YeetCode roadmap to drill these trade-offs interactively.
FAQ
What is the difference between cache-aside and write-through caching?
They differ in who writes the cache and when. In cache-aside, the application controls everything: it reads from the cache, falls back to the database on a miss, and on a write it updates the DB and deletes (or updates) the cache entry. The cache only ever holds data that was actually requested. Write-through instead writes to the cache and the database together on every write, so the cache is always warm and consistent, but each write pays the database's latency and you cache data that may never be read again. Cache-aside is the more common default because it's failure-tolerant and doesn't waste memory on cold data.
How do I prevent a thundering herd when a hot cache key expires?
Stop every request from recomputing the same value at once. The simplest fix is request coalescing: the first request to miss acquires a short lock (a Redis SET ... NX), recomputes the value, and every other request either waits for it or serves the last known value. Beyond that, stale-while-revalidate keeps serving the expired entry for a brief grace period while one background job refreshes it, and probabilistic early expiration refreshes hot keys slightly before their TTL so the reload is spread across time instead of a single cliff. For the very hottest keys, drop the TTL entirely and refresh them explicitly whenever the underlying data changes.
When should I choose Redis over Memcached?
Choose Memcached when you want a plain, fast, multi-threaded key-value store and nothing more — session blobs and query-result strings are its sweet spot, and its simplicity makes it extremely fast. Choose Redis when you need data structures or durability. Redis gives you sorted sets (leaderboards), hashes, streams, HyperLogLog, and bitmaps, plus optional persistence, replication, Sentinel/Cluster for high availability, Lua scripting, and pub/sub. That feature set lets one Redis handle rate limiting, queues, dedup, and leaderboards that Memcached simply can't express. Redis is the default choice today because those capabilities usually outweigh its slightly lower raw throughput.
Why does deleting the cache entry beat updating it on a write?
Deleting is safer because it doesn't require you to know the cache's exact shape. If your cached value is a joined, formatted, or aggregated view of several rows, computing the new cached version on every write means duplicating query logic in the write path — and any bug there writes a wrong value straight into the cache. Deleting sidesteps all of that: the next read simply misses and reloads the authoritative value from the database. Delete-on-write also shrinks the window for a stale-write race, since there's no partially-updated entry sitting around between the DB commit and the cache refresh.
What eviction policy should I use for a general-purpose cache?
Start with LRU. It's cheap to maintain, matches the common pattern where recently-used data gets re-read, and every major cache supports it. Move off it only when profiling shows a specific problem: if a periodic full scan of cold data is evicting your hot keys, LFU or a frequency-admission policy like W-TinyLFU (used by Caffeine) resists that by refusing to admit one-time reads. ARC adapts between recency and frequency automatically but is more complex to reason about. For the vast majority of systems, LRU plus a sensible TTL gets you the hit rate you need without the extra machinery.
Make it stick: run this one yourself
Don't just read it — step through it interactively, one state change at a time.