Rate Limiting
Token bucket vs sliding window vs fixed window, and a complete per-client rate limiter in Spring Boot, ASP.NET Core, and Go.
Why rate limiting is a microservices concern, not just an edge concern
A single misbehaving client — a buggy retry loop, a scraper, a partner integration that decided to poll every 100ms — can consume enough capacity to degrade a service for everyone else calling it. Rate limiting rejects requests past a defined budget (typically per client, per API key, or per IP) before they can do that damage, and it's worth understanding both the different algorithms behind it and where in a microservices system it actually belongs.
Fixed window vs sliding window vs token bucket
| Fixed window | Sliding window | Token bucket | |
|---|---|---|---|
| How it works | Count requests in a fixed-size window (e.g., every clock-aligned 60 seconds), reset the counter to zero at each boundary | Weight the count between the previous and current window (or track a rolling log of request timestamps) | A bucket holds tokens that refill continuously at a steady rate; each request consumes one token, requests are rejected once the bucket is empty |
| Boundary burst problem | Yes — a client can send the full limit right before a window resets, then the full limit again right after, doubling the effective rate briefly | Smoothed out — no hard reset boundary for a burst to exploit | No hard boundary at all; naturally allows a burst up to the bucket's capacity, then throttles to the steady refill rate |
| State per client | One counter + one reset timestamp | A counter plus the previous window's count, or a full timestamp log (more memory) | A token count plus a last-refill timestamp |
| Implementation cost | Lowest | Higher — either extra math or more storage | Low, and the standard choice in most modern rate-limiting libraries |
| Typical backing store | A single counter (Redis INCR + EXPIRE) |
Redis sorted sets, or an approximation using two fixed windows | An in-memory or Redis-backed bucket per client |
None of the three is strictly "best" — fixed window is simplest to reason about and cheap to run at huge scale, sliding window is the most accurate but costs more to track, and token bucket is the most common default because it naturally tolerates a reasonable burst (a client that's been quiet for a while can send a quick flurry of requests) while still enforcing a hard steady-state rate. All three examples below implement token bucket, since it's what each ecosystem's mainstream library defaults to.
Where to enforce it: gateway vs per-service
The API Gateway Patterns page covers the gateway as the natural place for cross-cutting concerns, and rate limiting is a textbook example: enforcing a client's overall request budget once, at the edge, is simpler to operate than repeating the same logic in every service. But gateway-level limiting alone isn't always enough — a single expensive endpoint deep inside one service (a report generator, a search query) may need its own, much stricter limit regardless of the client's overall budget, and an internal caller that reaches a service directly (bypassing the gateway entirely, which happens more often than teams expect) isn't covered by a gateway limit at all. The common real-world answer is both: a coarse, per-client budget enforced at the gateway, and a finer, endpoint-specific limit enforced by the owning service itself as defense in depth.
A complete implementation: N requests per client per window
The same scenario in all three stacks — limiting an endpoint to 100 requests per client per minute, identified by an X-Client-Id header, returning 429 Too Many Requests with a Retry-After header once the budget is exhausted.
Spring Boot — Bucket4j
@Component
public class RateLimitingFilter extends OncePerRequestFilter {
private final Map<String, Bucket> bucketsByClient = new ConcurrentHashMap<>();
private Bucket newBucket() {
Bandwidth limit = Bandwidth.classic(100, Refill.greedy(100, Duration.ofMinutes(1)));
return Bucket.builder().addLimit(limit).build();
}
@Override
protected void doFilterInternal(HttpServletRequest request, HttpServletResponse response, FilterChain chain)
throws ServletException, IOException {
String clientId = request.getHeader("X-Client-Id");
if (clientId == null) {
response.sendError(HttpStatus.BAD_REQUEST.value(), "X-Client-Id header is required");
return;
}
Bucket bucket = bucketsByClient.computeIfAbsent(clientId, id -> newBucket());
ConsumptionProbe probe = bucket.tryConsumeAndReturnRemaining(1);
if (probe.isConsumed()) {
response.setHeader("X-RateLimit-Remaining", String.valueOf(probe.getRemainingTokens()));
chain.doFilter(request, response);
} else {
long retryAfterSeconds = probe.getNanosToWaitForRefill() / 1_000_000_000;
response.setStatus(429);
response.setHeader("Retry-After", String.valueOf(retryAfterSeconds));
response.setContentType("application/json");
response.getWriter().write("{\"error\":\"rate limit exceeded\"}");
}
}
}
Bandwidth.classic(100, Refill.greedy(100, Duration.ofMinutes(1))) is the token bucket itself: capacity 100, refilling 100 tokens every minute. tryConsumeAndReturnRemaining is the atomic check-and-decrement, and getNanosToWaitForRefill() is exactly the number that belongs in Retry-After — how long until at least one token is available again. (Resilience4j's own RateLimiter module solves the same problem and is worth knowing about too, but Bucket4j's ConsumptionProbe maps more directly onto returning an accurate Retry-After value.)
ASP.NET Core — Microsoft.AspNetCore.RateLimiting
builder.Services.AddRateLimiter(options =>
{
options.OnRejected = async (context, cancellationToken) =>
{
context.HttpContext.Response.StatusCode = StatusCodes.Status429TooManyRequests;
if (context.Lease.TryGetMetadata(MetadataName.RetryAfter, out var retryAfter))
{
context.HttpContext.Response.Headers.RetryAfter = ((int)retryAfter.TotalSeconds).ToString();
}
await context.HttpContext.Response.WriteAsJsonAsync(new { error = "rate limit exceeded" }, cancellationToken);
};
options.AddPolicy("PerClientTokenBucket", httpContext =>
{
var clientId = httpContext.Request.Headers["X-Client-Id"].ToString();
return RateLimitPartition.GetTokenBucketLimiter(clientId, _ => new TokenBucketRateLimiterOptions
{
TokenLimit = 100,
TokensPerPeriod = 100,
ReplenishmentPeriod = TimeSpan.FromMinutes(1),
QueueLimit = 0,
AutoReplenishment = true,
});
});
});
var app = builder.Build();
app.UseRateLimiter();
app.MapGet("/api/orders/{id}", GetOrderHandler)
.RequireRateLimiting("PerClientTokenBucket");
RateLimitPartition.GetTokenBucketLimiter creates one independent bucket per partition key — here, per X-Client-Id — so each client gets its own 100-tokens-per-minute budget rather than sharing one global bucket. QueueLimit = 0 means a request that finds the bucket empty is rejected immediately rather than queued, which is what keeps the 429 timely instead of the caller hanging.
Go — golang.org/x/time/rate as HTTP middleware
var (
limitersByClient = make(map[string]*rate.Limiter)
limitersMu sync.Mutex
)
func getLimiterFor(clientID string) *rate.Limiter {
limitersMu.Lock()
defer limitersMu.Unlock()
limiter, exists := limitersByClient[clientID]
if !exists {
// 100 requests/minute steady rate, with a burst allowance of 20
limiter = rate.NewLimiter(rate.Every(time.Minute/100), 20)
limitersByClient[clientID] = limiter
}
return limiter
}
func rateLimitMiddleware(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
clientID := r.Header.Get("X-Client-Id")
if clientID == "" {
http.Error(w, "X-Client-Id header is required", http.StatusBadRequest)
return
}
if !getLimiterFor(clientID).Allow() {
w.Header().Set("Retry-After", "1")
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(http.StatusTooManyRequests)
w.Write([]byte(`{"error":"rate limit exceeded"}`))
return
}
next.ServeHTTP(w, r)
})
}
rate.NewLimiter(rate.Every(time.Minute/100), 20) is Go's token bucket: rate.Every(time.Minute/100) sets the steady refill rate (one token roughly every 0.6s, i.e., 100/minute), and 20 is the burst size — how many requests can fire back-to-back before the steady rate takes over. Allow() is the non-blocking check-and-consume; it never blocks the request goroutine waiting for a token, which matters for an HTTP middleware that needs to respond immediately either way.
Common mistakes
- Rate limiting purely by client IP address — a shared corporate NAT gateway or mobile carrier can put thousands of genuine users behind one IP, so an IP-based limit either blocks all of them together or has to be set so loose it stops protecting anything. Key by an authenticated client identity (API key, client ID, user ID) whenever one is available.
- Not returning
Retry-Afteron a429— without it, a well-behaved client has no signal for how long to back off, and either retries immediately (making things worse) or gives up entirely instead of waiting the right amount of time. - Enforcing the limit only after expensive work has already run (a database query, a call to another service) instead of as the very first check in the request pipeline — the limiter should reject before any real cost is incurred, not after.
- Relying on a gateway-level limit alone and assuming a backend service is therefore protected — any caller that can reach the service directly (another internal service, a misconfigured client, a mistake in routing) bypasses the gateway's limiter entirely.
- Choosing fixed window with a short window purely because it's the simplest to implement, then being surprised that clients can burst to roughly double the intended rate right across a window boundary.