Rate Limiting & Throttling

Why an API needs rate limiting#

Without a limit, a single misbehaving client — a buggy retry loop, a scraper, or an outright abusive caller — can consume enough of an API's capacity to degrade service for every other client. Rate limiting caps how many requests a given client (identified by API key, user ID, or IP address) can make in a given time window, and throttling is the broader practice of slowing or rejecting excess traffic once that cap is hit, keeping the system stable for everyone else.

The token bucket algorithm, recapped#

The most common rate-limiting algorithm models each client as having a bucket that holds up to some maximum number of tokens. Every request costs one token; tokens refill continuously at a fixed rate up to the bucket's capacity. A request is allowed if a token is available (and one is deducted); it's rejected if the bucket is empty.

Bucket capacity: 10 tokens        Refill rate: 1 token/second

t=0s   Bucket: [██████████] 10/10   -- client hasn't made any requests yet
t=0s   5 requests arrive at once -> all allowed (burst absorbed)
t=0s   Bucket: [█████     ] 5/10
t=1s   Bucket: [██████    ] 6/10    -- refilled by 1 token
t=1s   8 more requests arrive     -> only 6 allowed, 2 rejected (bucket empty)
t=1s   Bucket: [          ] 0/10

The key property that makes token bucket the standard choice over a naive fixed-window counter: it allows a reasonable burst up to the bucket's capacity, while still enforcing a steady average rate over time — a client that's been idle can make a quick flurry of requests without being unfairly capped the instant it starts, but can't sustain a rate faster than the refill rate indefinitely.

Implementing it in an Express middleware#

A minimal, per-client token bucket, backed by an in-memory Map for a single-instance API (a real multi-instance deployment needs a shared store — see Common mistakes):

const buckets = new Map();

const CAPACITY = 10;       // max burst size
const REFILL_RATE = 1;     // tokens added per second

function getBucket(clientId) {
  const now = Date.now();
  let bucket = buckets.get(clientId);

  if (!bucket) {
    bucket = { tokens: CAPACITY, lastRefill: now };
    buckets.set(clientId, bucket);
  }

  // Refill based on elapsed time since the last check — no separate timer needed
  const elapsedSeconds = (now - bucket.lastRefill) / 1000;
  bucket.tokens = Math.min(CAPACITY, bucket.tokens + elapsedSeconds * REFILL_RATE);
  bucket.lastRefill = now;

  return bucket;
}

function rateLimiter(req, res, next) {
  const clientId = req.headers['x-api-key'] || req.ip;
  const bucket = getBucket(clientId);

  if (bucket.tokens < 1) {
    const retryAfterSeconds = Math.ceil((1 - bucket.tokens) / REFILL_RATE);
    res.set('Retry-After', String(retryAfterSeconds));
    return res.status(429).json({
      error: 'Too many requests',
      retryAfterSeconds,
    });
  }

  bucket.tokens -= 1;
  next();
}

app.use('/api/', rateLimiter);

The 429 response and Retry-After#

429 Too Many Requests is the status code built specifically for this — a client (or a well-behaved HTTP library) can recognize it programmatically and back off, rather than treating it as a generic failure to retry immediately and make the problem worse.

HTTP/1.1 429 Too Many Requests
Retry-After: 4
Content-Type: application/json

{"error": "Too many requests", "retryAfterSeconds": 4}

The Retry-After header tells the client exactly how long to wait before trying again — either as a number of seconds (as above) or an HTTP date. Omitting it leaves the client guessing, which tends to produce exactly the wrong behavior: immediate retries that hit the rate limit again, over and over, in a tight loop that never actually backs off.

Many production APIs also expose the current state proactively, on every response, not just 429s — so well-behaved clients can throttle themselves before ever actually hitting the limit:

HTTP/1.1 200 OK
X-RateLimit-Limit: 100
X-RateLimit-Remaining: 37
X-RateLimit-Reset: 1798329600

A quick comparison of rate-limiting algorithms#

Token bucket wins out for most APIs because it matches how real traffic actually behaves — legitimate clients are often bursty (a user opens an app and it fires off several requests at once), and a strictly smoothed leaky bucket would reject a perfectly reasonable burst just as readily as it would reject actual abuse.

Common mistakes#

• Storing rate-limit state in a plain in-memory object on an API that runs more than one instance (behind a load balancer, or clustered) — each instance enforces its own separate limit with no visibility into the others, so the effective limit becomes (configured limit) × (number of instances). A shared store like Redis is required the moment there's more than one process.
• Returning 429 with no Retry-After header — clients are left guessing how long to wait, and poorly written ones often respond by retrying immediately, hammering the same limit again instead of backing off.
• Rate limiting only by IP address when clients sit behind a shared corporate NAT or proxy — many genuinely distinct users end up sharing one effective limit. Rate limiting by an authenticated API key or user ID, where available, is more precise.
• Setting limits so strict that normal, legitimate usage patterns (a page load firing five parallel requests) get throttled — token bucket's burst capacity exists specifically to accommodate this; a capacity of 1 defeats that benefit entirely.

Interview questions#

Q: Why is token bucket generally preferred over a fixed-window counter for API rate limiting? A fixed window resets sharply at a boundary, which lets a client send close to double the intended rate by timing requests around that edge (a burst right before the window ends, another right after it resets). Token bucket refills continuously and allows a bounded burst up to its capacity while still enforcing a steady average rate over time, which better matches how legitimate traffic actually behaves.

Q: What problem does the Retry-After header solve on a 429 response? Without it, a client only knows it was rejected, not how long to wait before trying again — which often leads to immediate retries that hit the same limit repeatedly. Retry-After gives the client an explicit wait time (in seconds or as a date), letting well-behaved clients back off correctly instead of guessing.