Cache Invalidation & Cache Stampedes

"There are only two hard things in computer science…"#

"…cache invalidation and naming things." The joke is popular because it's true. A cache is a copy, and keeping copies in sync with the source of truth across many servers is genuinely hard.

Invalidation strategies#

Production systems usually combine them: explicit or event-based invalidation for freshness, plus a TTL as a safety net.

The race condition you need to know#

Cache-aside has a classic race between a reader and a writer:

1. Reader: cache miss for course:1
2. Reader: reads OLD value from the database
3. Writer: updates the database to NEW value
4. Writer: deletes cache key course:1
5. Reader: writes the OLD value into the cache   ← stale until TTL

Mitigations:

• Short TTLs for data that changes often, so the window is bounded.
• Delayed double-delete — delete the key again shortly after the write.
• Versioned values — store a version with each value and refuse to cache a value older than the latest known version.
• For critical reads, skip the cache entirely.

Cache stampede (thundering herd)#

A popular key expires. In the next instant, hundreds of requests miss at once — and all of them query the database to rebuild the same value. The database collapses exactly when traffic is highest.

Fix 1: single-flight (request coalescing)#

Only one request rebuilds a missing key; the others wait for its result.

import threading
import time

db_calls = 0
cache: dict[str, str] = {}
locks: dict[str, threading.Lock] = {}
locks_guard = threading.Lock()

def load_from_db(key: str) -> str:
    global db_calls
    db_calls += 1
    time.sleep(0.2)                          # a slow query
    return f"value-for-{key}"

def get(key: str) -> str:
    if key in cache:
        return cache[key]
    with locks_guard:                        # one lock per key
        lock = locks.setdefault(key, threading.Lock())
    with lock:
        if key not in cache:                 # double-check after acquiring the lock
            cache[key] = load_from_db(key)
    return cache[key]

threads = [threading.Thread(target=get, args=("trending",)) for _ in range(50)]
for t in threads:
    t.start()
for t in threads:
    t.join()

print("50 concurrent requests → database calls:", db_calls)

Output:

50 concurrent requests → database calls: 1

Across multiple servers, the same idea uses a distributed lock — for example Redis SET key value NX PX 5000 — so only one server rebuilds the value.

Fix 2: jittered TTLs#

If thousands of keys are cached at the same moment with the same TTL, they also expire together. Add randomness:

import random

def ttl_with_jitter(base_seconds: int, jitter: float = 0.1) -> int:
    return int(base_seconds * random.uniform(1 - jitter, 1 + jitter))

Fix 3: refresh before expiry#

Refresh hot keys in the background shortly before they expire (sometimes called refresh-ahead or stale-while-revalidate): users keep getting the slightly old value while a single worker fetches the new one. Probabilistic early expiration does this without a scheduler — each read has a small chance of refreshing early, growing as expiry approaches.

Fix 4: serve stale on error#

If the database is down, returning a slightly stale cached value is often better than an error. Keep a longer "stale" copy alongside the fresh one.

Other cache failure modes#

• Cache penetration — repeated requests for keys that don't exist always miss. Cache "not found" results briefly, or use a Bloom filter to reject impossible keys.
• Hot keys — one key receives a huge share of traffic (a viral post). Replicate it across several keys or keep a small in-process copy.
• Cache avalanche — the whole cache restarts empty. Warm critical keys on startup and protect the database with rate limits.

Common mistakes#

• Relying only on TTLs for data users just edited — they refresh the page and see their old data.
• Identical TTLs for everything, causing synchronised expiry.
• Global locks that serialise all cache rebuilds instead of per-key locks.
• No monitoring — track hit ratio, evictions and rebuild latency.

Key takeaways#

• Combine explicit or event-based invalidation with TTLs as a safety net.
• Know the read/write race in cache-aside and bound it with short TTLs or versions.
• Prevent stampedes with single-flight locking, jittered TTLs and refresh-ahead.
• Handle penetration, hot keys and cold starts before they become outages.