NOA Labs

Load Balancing, Reverse Proxies & Caching

Distributing traffic across servers, and speeding up reads with caches and CDNs.

Load balancers

A load balancer sits in front of a pool of servers and distributes incoming requests across them, so no single server is overwhelmed and the system can scale horizontally.

Plaintext
                 ┌──────────┐
     requests -> │   Load   │ ---> Server 1
                 │ Balancer │ ---> Server 2
                 └──────────┘ ---> Server 3

Common strategies:

  • Round robin — requests cycle through servers in order. Simple, works well when servers and requests are roughly uniform.
  • Least connections — routes to whichever server currently has the fewest active connections. Better when request costs vary a lot.
  • IP hash / consistent hashing — routes the same client (or key) to the same server consistently, useful when a server holds session state or a local cache.
  • Weighted — some servers get proportionally more traffic (e.g., newer, more powerful machines).

Layer 4 (transport) load balancers route based on IP/port without inspecting content (fast, simple). Layer 7 (application) load balancers can route based on the HTTP path, headers or cookies (more flexible, slightly more overhead) — e.g., routing /api/* to one service and /images/* to another.

Health checks

A load balancer periodically pings each backend (GET /health) and automatically removes unhealthy instances from rotation — this is what makes horizontal scaling resilient to individual server failures, not just capacity growth.

Reverse proxies

A reverse proxy (Nginx, HAProxy, Envoy) sits between clients and backend servers, forwarding requests on their behalf. It's often the same physical component as a load balancer, but the term emphasizes different responsibilities: TLS termination, request buffering, compression, rewriting URLs, and shielding backend servers from being directly exposed to the internet.

Plaintext
Client --> Reverse Proxy (TLS termination, compression) --> App servers (plain HTTP internally)

Caching

Caching stores a copy of expensive-to-compute or expensive-to-fetch data somewhere faster to access, dramatically reducing load on the database and improving latency for read-heavy workloads (recall: most real systems are read-heavy).

Where caches live, from closest to the user to farthest:

  1. Client-side cache (browser cache, mobile app cache) — zero network round-trip if hit.
  2. CDN (Content Delivery Network) — caches static (and sometimes dynamic) content at edge locations physically close to users, cutting latency and origin server load dramatically for images, video, JS/CSS bundles.
  3. Application-layer cache (Redis / Memcached) — an in-memory key-value store between your app servers and the database.
  4. Database query cache / buffer pool — the database's own internal caching of frequently accessed pages.

Cache invalidation strategies

The famous quote — "there are only two hard things in computer science: cache invalidation and naming things" — exists because stale cache data is a real, common source of bugs.

  • TTL (time-to-live) — simplest approach; data expires automatically after N seconds. Accepts brief staleness in exchange for simplicity.
  • Write-through — every write updates the cache and the database together, keeping them in sync at write time (adds write latency).
  • Write-behind (write-back) — writes go to the cache first and are flushed to the database asynchronously (faster writes, risk of data loss if the cache fails before flushing).
  • Cache-aside (lazy loading) — the application checks the cache first; on a miss, it reads from the database and populates the cache for next time. The most common pattern in practice.
Plaintext
Cache-aside read:
  1. app checks Redis for key "user:42"
  2. miss -> app queries the database
  3. app writes the result into Redis with a TTL
  4. app returns the result

Redis as a caching layer

Redis is the most common choice for an application-layer cache — it's an in-memory data store supporting strings, hashes, lists, sets and sorted sets, with sub-millisecond typical latency.

Plaintext
SET user:42 '{"name":"Ali"}' EX 300   # cache for 300 seconds
GET user:42

Beyond caching, Redis is also commonly used for session storage, rate limiting counters, leaderboards (sorted sets), and as a lightweight pub/sub message bus.

Common mistakes

  • Caching data with no invalidation strategy at all — "cache it and forget it" eventually serves stale or wrong data.
  • Putting a cache in front of infrequently-read or rapidly-changing data, where the cache hit rate is too low to justify the added complexity.
  • Forgetting that a CDN cache and an application cache need different invalidation approaches — a CDN purge is a different operation than clearing a Redis key.

Interview questions

Q: What's the difference between a load balancer and a reverse proxy? They're often the same software instance, but the terms emphasize different jobs: a load balancer's core job is distributing traffic across multiple backend instances; a reverse proxy's core job is intercepting and forwarding client requests on behalf of backend servers (TLS termination, compression, hiding backend topology). In practice, most production reverse proxies (Nginx, Envoy) do both.

Q: Why is cache-aside (lazy loading) the most commonly used caching pattern? It only caches data that's actually requested (no wasted cache space on unused data), degrades gracefully if the cache is unavailable (falls back to the database), and is simple to reason about — at the cost of a guaranteed cache miss (and slightly higher latency) on the very first request for any given key.

Q: When would you choose a CDN over an application-layer cache like Redis? For static or rarely-changing content (images, videos, JS/CSS bundles, sometimes full HTML pages) that benefits from being physically close to users worldwide. Redis is better suited for dynamic, frequently-changing, per-user or per-query data close to your application servers.