NOA Labs

Advanced Design Principles

Bulkheads, circuit breakers, jittered retries with backoff, and idempotency keys for safe retries — with Spring Boot, ASP.NET Core, and Go examples.

Beyond "it compiles and the happy path works"

Service Communication shows how OrdersService calls InventoryService, either synchronously or via an event. What it doesn't cover is what happens when that call is slow, or fails outright, or fails repeatedly — and in a system with a dozen services calling each other, that isn't an edge case, it's Tuesday. This page covers four patterns that turn "a call to another service" into something that survives production: the bulkhead pattern, the circuit breaker pattern, jittered retries with backoff, and idempotency keys. A fifth core principle, database-per-service, is deliberately not re-explained here — it's covered in full, along with the Saga pattern it motivates, on Saga Pattern & Distributed Transactions.

Pattern Problem it solves Failure mode without it
Bulkhead One slow/failing dependency exhausts the threads or connections the whole service needs Cascading failure — a single misbehaving dependency starves capacity needed by requests that don't even touch it
Circuit breaker Repeatedly calling a dependency that's already down Wasted latency and resources retrying a call that's almost certain to fail, which also slows the failing dependency's own recovery
Jittered retry with backoff A transient failure needs a retry, without making the outage worse A retry storm — many clients retrying in lockstep multiplies load on an already-struggling service at the worst possible moment
Idempotency key A retried request must not be applied a second time A duplicate charge, a duplicate shipment, or some other unsafe side effect triggered by a retry that looked perfectly safe

The bulkhead pattern

The name comes from a ship's bulkheads — physical partitions that keep one flooded compartment from sinking the entire vessel. Applied to a service, a bulkhead isolates the resources (usually threads or connection-pool slots) used to call one dependency from the resources used for everything else, so a single dependency grinding to a halt can only exhaust its own allocation, not the whole service's capacity.

Without one, a common shared thread pool means: PaymentService starts timing out, every thread handling a payment call blocks waiting on it, and within seconds every thread in the pool is stuck waiting on PaymentService — including the ones that were only trying to handle an unrelated GET /orders/482 request that never touches payments at all. The entire service goes down because of a dependency most incoming requests didn't even need.

Spring Boot — Resilience4j's Bulkhead

YAML
resilience4j:
  bulkhead:
    instances:
      paymentService:
        max-concurrent-calls: 10
        max-wait-duration: 0
Java
@Service
public class PaymentClient {

    private final RestClient restClient;

    public PaymentClient(RestClient.Builder builder) {
        this.restClient = builder.baseUrl("http://payment-service").build();
    }

    @Bulkhead(name = "paymentService", fallbackMethod = "chargeFallback")
    public ChargeResponse charge(String orderId, BigDecimal amount) {
        return restClient.post()
            .uri("/api/charges")
            .body(new ChargeRequest(orderId, amount))
            .retrieve()
            .body(ChargeResponse.class);
    }

    private ChargeResponse chargeFallback(String orderId, BigDecimal amount, BulkheadFullException ex) {
        // payment-service's bulkhead is saturated — fail fast instead of queuing behind it
        return ChargeResponse.rejected(orderId, "payment-service is at capacity, try again shortly");
    }
}

At most 10 concurrent calls to payment-service can be in flight at once, no matter how many other threads the rest of the application is using — a slow payment-service can only ever tie up those 10.

ASP.NET Core — Polly's bulkhead policy

C#
using Polly;
using Polly.Bulkhead;

var bulkheadPolicy = Policy.BulkheadAsync<HttpResponseMessage>(
    maxParallelization: 10,
    maxQueuingActions: 5,
    onBulkheadRejectedAsync: _ =>
    {
        logger.LogWarning("Bulkhead rejected a call to payment-service — at capacity");
        return Task.CompletedTask;
    });

builder.Services.AddHttpClient("PaymentService", client =>
{
    client.BaseAddress = new Uri("http://payment-service");
})
.AddPolicyHandler(bulkheadPolicy);

Up to 10 calls run concurrently, up to 5 more are allowed to queue briefly, and anything beyond that is rejected immediately with a BulkheadRejectedException rather than being allowed to pile up indefinitely.

Go — a buffered channel as a semaphore

Go has no dedicated bulkhead library because the language's own primitives already express it directly: a buffered channel used purely for its capacity, not to carry data.

Go
// Capacity of 10 — at most 10 concurrent calls to payment-service are allowed
var paymentSemaphore = make(chan struct{}, 10)

var ErrBulkheadFull = errors.New("payment-service bulkhead is at capacity")

func chargeCard(ctx context.Context, orderID string, amount float64) (*ChargeResponse, error) {
    select {
    case paymentSemaphore <- struct{}{}:
        defer func() { <-paymentSemaphore }() // release the slot when this call finishes
    default:
        return nil, ErrBulkheadFull // don't block waiting for a slot — fail fast instead
    }

    return doChargeRequest(ctx, orderID, amount)
}

The select/default pair is what makes this non-blocking: if the channel's buffer is already full, the default case fires immediately instead of waiting for a slot to free up — exactly the "fail fast rather than queue behind a struggling dependency" behavior a bulkhead is for.

The circuit breaker pattern, properly implemented

A circuit breaker tracks the recent failure rate of calls to a dependency and moves through three states: closed (calls go through normally), open (calls fail immediately without even attempting the network call, once failures cross a threshold), and half-open (after a cooldown, a limited number of test calls are let through to check whether the dependency has recovered). The point isn't just "stop calling a dead service" — it's protecting the failing dependency too, by cutting off the flood of retry traffic that would otherwise slow its recovery down further.

Spring Boot — Resilience4j's CircuitBreaker

YAML
resilience4j:
  circuitbreaker:
    instances:
      inventoryService:
        sliding-window-type: COUNT_BASED
        sliding-window-size: 20
        failure-rate-threshold: 50
        wait-duration-in-open-state: 10s
        permitted-number-of-calls-in-half-open-state: 5
        automatic-transition-from-open-to-half-open-enabled: true
Java
@CircuitBreaker(name = "inventoryService", fallbackMethod = "reserveStockFallback")
public InventoryResponse reserveStock(String orderId, List<Item> items) {
    return restClient.post()
        .uri("http://inventory-service/api/reservations")
        .body(new ReservationRequest(orderId, items))
        .retrieve()
        .body(InventoryResponse.class);
}

private InventoryResponse reserveStockFallback(String orderId, List<Item> items, Throwable ex) {
    // circuit is open — inventory-service is failing enough that we stop calling it for a while
    return InventoryResponse.degraded(orderId, "inventory-service unavailable, reservation queued for retry");
}

Over any rolling window of 20 calls, once 50% have failed the circuit trips to open for 10 seconds, then lets 5 test calls through in half-open state before deciding whether to close again or reopen.

ASP.NET Core — Polly's CircuitBreakerPolicy

C#
var circuitBreakerPolicy = Policy<HttpResponseMessage>
    .Handle<HttpRequestException>()
    .OrResult(response => (int)response.StatusCode >= 500)
    .CircuitBreakerAsync(
        handledEventsAllowedBeforeBreaking: 5,
        durationOfBreak: TimeSpan.FromSeconds(30),
        onBreak: (outcome, breakDelay) =>
            logger.LogWarning("Circuit opened for {Seconds}s calling inventory-service", breakDelay.TotalSeconds),
        onReset: () => logger.LogInformation("Circuit closed — inventory-service recovered"),
        onHalfOpen: () => logger.LogInformation("Circuit half-open — testing inventory-service"));

builder.Services.AddHttpClient("InventoryService", client =>
{
    client.BaseAddress = new Uri("http://inventory-service");
})
.AddPolicyHandler(circuitBreakerPolicy);

Five consecutive handled failures (a thrown exception, or a 5xx response) trip the circuit for 30 seconds, after which Polly automatically allows one probing call through (half-open) to decide whether to reset or re-break.

Go — sony/gobreaker

Go
import (
    "time"

    "github.com/sony/gobreaker"
)

var inventoryBreaker = gobreaker.NewCircuitBreaker(gobreaker.Settings{
    Name:        "inventory-service",
    MaxRequests: 5,                // requests allowed through while half-open
    Interval:    60 * time.Second, // closed-state failure count resets every 60s
    Timeout:     30 * time.Second, // how long the circuit stays open before trying half-open
    ReadyToTrip: func(counts gobreaker.Counts) bool {
        failureRatio := float64(counts.TotalFailures) / float64(counts.Requests)
        return counts.Requests >= 10 && failureRatio >= 0.5
    },
})

func reserveStock(orderID string, items []Item) (*InventoryResponse, error) {
    result, err := inventoryBreaker.Execute(func() (interface{}, error) {
        return doReservationRequest(orderID, items)
    })
    if err != nil {
        return nil, err // ErrOpenState if the circuit is currently open
    }
    return result.(*InventoryResponse), nil
}

ReadyToTrip is called after every request and decides, from the rolling Counts, whether to flip the circuit open — here, once at least 10 requests have been seen and at least half failed.

Timeouts with jittered retries

A naive retry — "if it fails, try again immediately, up to 3 times" — is one of the most common ways teams make an outage worse. When a service starts timing out under load, every client hitting it retries at roughly the same moment, multiplying the request volume the already-struggling service receives right when it can least handle it — a retry storm. Two fixes, always used together: exponential backoff (each retry waits longer than the last, giving the dependency room to recover) and jitter (randomizing that wait slightly, so thousands of clients that all failed at the same instant don't all retry at the same instant too).

Java
IntervalFunction intervalFn = IntervalFunction.ofExponentialRandomBackoff(
    Duration.ofMillis(200), // initial interval
    2.0,                    // multiplier — each attempt roughly doubles the previous wait
    Duration.ofSeconds(5)); // cap — never wait longer than this, however many attempts have failed

RetryConfig retryConfig = RetryConfig.custom()
    .maxAttempts(4)
    .intervalFunction(intervalFn)
    .retryExceptions(IOException.class, java.util.concurrent.TimeoutException.class)
    .build();

Retry retry = Retry.of("inventoryService", retryConfig);

Supplier<InventoryResponse> decorated = Retry.decorateSupplier(retry,
    () -> reserveStockOverHttp(orderId, items));

InventoryResponse response = decorated.get();

ofExponentialRandomBackoff is the part that matters: instead of a fixed doubling sequence (200ms, 400ms, 800ms, 1600ms — identical for every client), it randomizes each wait around that exponential curve, so a thousand clients that all just saw inventory-service fail spread their retries out across a window instead of hammering it in one synchronized burst four times.

Idempotency keys: making retries safe

Backoff and jitter make retries survivable for the system as a whole, but they don't make an individual retried request safe — if the first attempt at POST /payments actually succeeded and only the response got lost (a timeout on the way back, not on the way there), a naive retry charges the customer twice. An idempotency key fixes this at the API contract level: the client generates a unique key once per logical operation and sends it on every attempt (including retries) of that same operation; the server uses it to recognize "I've already processed this exact request" and returns the original result instead of processing it again.

Go
func chargeHandler(w http.ResponseWriter, r *http.Request) {
    idempotencyKey := r.Header.Get("Idempotency-Key")
    if idempotencyKey == "" {
        http.Error(w, "Idempotency-Key header is required", http.StatusBadRequest)
        return
    }

    ctx := r.Context()
    dedupKey := "idempotency:" + idempotencyKey

    // SETNX: only the FIRST request with this key wins the race and proceeds to charge the card
    acquired, err := redisClient.SetNX(ctx, dedupKey, "processing", 24*time.Hour).Result()
    if err != nil {
        http.Error(w, "internal error", http.StatusInternalServerError)
        return
    }

    if !acquired {
        cached, err := redisClient.Get(ctx, dedupKey).Result()
        if err == nil && cached != "processing" {
            // a previous attempt with this exact key already completed — replay its response, don't charge again
            w.Header().Set("Content-Type", "application/json")
            w.Write([]byte(cached))
            return
        }
        // an attempt with this key is still in flight right now
        http.Error(w, "a request with this Idempotency-Key is already being processed", http.StatusConflict)
        return
    }

    result, err := chargeCard(r)
    if err != nil {
        redisClient.Del(ctx, dedupKey) // don't leave a poisoned "processing" marker behind on failure
        http.Error(w, "payment failed", http.StatusPaymentRequired)
        return
    }

    responseBody, _ := json.Marshal(result)
    redisClient.Set(ctx, dedupKey, string(responseBody), 24*time.Hour) // store the real response for any future retry to replay

    w.Header().Set("Content-Type", "application/json")
    w.Write(responseBody)
}
Bash
# The client sends the SAME key on the original attempt and every retry of it
curl -X POST https://api.example.com/payments \
  -H "Idempotency-Key: 7c1e2a90-4b3f-4e91-9c3a-b6e2f1a9d001" \
  -d '{"orderId":"order-482","amount":49.99}'

SETNX (set-if-not-exists) is what makes the check-and-reserve atomic — without it, two retries arriving milliseconds apart could both read "no entry yet" and both proceed to charge the card, which is exactly the race an idempotency key exists to close.

Database-per-service, briefly

The fourth pillar of resilient service design is that each service owns its data exclusively, which is why a failure partway through a multi-service operation needs a compensating action rather than a database rollback in the first place. That's covered in full — including the trade-offs against two-phase commit and a complete choreography/orchestration example — on Saga Pattern & Distributed Transactions; it isn't repeated here.

Common mistakes

  • Sizing a bulkhead's pool the same as (or larger than) the shared pool it's meant to protect — a bulkhead that can consume just as many resources as "everything else" combined isn't actually isolating anything.
  • Retrying without backoff, or with backoff but no jitter — a fixed exponential sequence still lets every failing client retry in lockstep, reproducing a retry storm on a predictable schedule instead of preventing one.
  • Wrapping a circuit breaker around a call but never actually implementing the fallback path — a fallback method that just rethrows the original exception means the circuit breaker only added latency and complexity without changing the failure behavior at all.
  • Treating an idempotency key as optional for "internal" calls between services — a retried inter-service call is exactly as capable of double-charging or double-shipping as a retried client request, and often more likely to happen unnoticed.
  • Storing an idempotency dedup entry with no expiry — the store grows forever, and a key legitimately reused (by a buggy client, or after a long delay) can be rejected or replayed incorrectly years later.