NOA Labs

Distributed Tracing

Micrometer Tracing and Zipkin, and propagating trace IDs across services with a concrete example.

Why a single request needs a trace ID across services

A single user-facing request in a microservices system might touch four or five separate services before a response comes back. When something is slow or fails, each service's own logs only show its own piece of the story — there's no way to tell, from an individual service's logs alone, which of its log lines belong to the same originating request as a slow call three services downstream.

Distributed tracing solves this by attaching one shared identifier — a trace ID — to a request the moment it enters the system, and propagating it through every subsequent internal call so every service's logs and spans can be correlated back to that same original request.

Plaintext
Client request
     |  trace-id: abc123 (generated here)
     v
+-------------+   trace-id: abc123    +-------------+   trace-id: abc123    +--------------+
|  Gateway      | ---------------------> |  Orders Service | ---------------------> | Payments Service |
+-------------+                       +-------------+                       +--------------+
     |                                       |                                      |
     v                                       v                                      v
  span: gateway-handling                span: fetch-order                    span: charge-card
     (all reported to Zipkin, correlated under the same trace-id abc123)

A span is one unit of work within the trace (one service's handling of the request, or one specific operation like a database call) — a trace is the tree of every span that happened as part of one logical request.

Micrometer Tracing (the modern replacement for Sleuth)

Spring Cloud Sleuth — the older tracing library — is no longer developed as a separate project; its functionality moved into Micrometer Tracing, the same metrics facade Actuator//metrics is built on (see the Spring Boot Actuator page), extended to also handle trace propagation:

HTML
<dependency>
    <groupId>io.micrometer</groupId>
    <artifactId>micrometer-tracing-bridge-brave</artifactId>
</dependency>
<dependency>
    <groupId>io.zipkin.reporter2</groupId>
    <artifactId>zipkin-reporter-brave</artifactId>
</dependency>
YAML
management:
  tracing:
    sampling:
      probability: 1.0   # trace every request in dev; a smaller fraction (e.g. 0.1) in high-traffic production
  zipkin:
    tracing:
      endpoint: http://localhost:9411/api/v2/spans

With this on the classpath and configured, Spring Boot automatically instruments incoming/outgoing HTTP calls, RestClient/WebClient calls, and scheduled tasks — attaching and propagating trace/span IDs without any manual code in most cases.

A concrete example: propagation across two services

Java
@RestController
@RequestMapping("/api/orders")
public class OrderController {

    private final RestClient restClient;

    public OrderController(RestClient.Builder builder) {
        this.restClient = builder.baseUrl("http://payments-service").build();
    }

    @PostMapping
    public Order placeOrder(@RequestBody PlaceOrderRequest request) {
        // Micrometer Tracing automatically attaches the current trace/span headers to this outgoing call --
        // no manual header code needed here at all
        PaymentResult result = restClient.post()
            .uri("/api/payments")
            .body(new ChargeRequest(request.amount()))
            .retrieve()
            .body(PaymentResult.class);

        return new Order(request.sku(), result.transactionId());
    }
}

On the wire, this looks like an ordinary outgoing HTTP call with a couple of extra headers Micrometer Tracing adds automatically:

Plaintext
POST /api/payments HTTP/1.1
traceparent: 00-abc123def456...-789xyz...-01

payments-service, if it also has Micrometer Tracing configured, automatically recognizes the incoming traceparent header, continues the same trace (rather than starting a new one), and reports its own span back to Zipkin tagged with the same trace ID — which is what lets Zipkin's UI show the entire request, across both services, as one connected timeline.

What you get in Zipkin

Once spans from multiple services report to the same Zipkin instance, its UI shows a single trace as a waterfall: how long the gateway took, how long orders-service took before calling payments-service, and how long payments-service itself took — making it immediately visible which hop in a multi-service call chain was actually slow, instead of guessing from separate, uncorrelated log files.

Common mistakes

  • Sampling every request (probability: 1.0) in a high-traffic production environment — tracing every single request adds real overhead and storage cost at scale; a sampled fraction (often 1-10%) is the normal production setting, with 100% reserved for lower-traffic environments or targeted debugging.
  • Assuming trace propagation "just happens" across a call made with a raw HttpURLConnection or an unconfigured HTTP client — automatic propagation depends on Micrometer's instrumentation actually wrapping the client in use; an uninstrumented client silently breaks the chain.
  • Logging without including the trace ID in the log line itself — correlating logs across services depends on the trace ID actually appearing in each service's log output (Micrometer Tracing integrates with common logging patterns/MDC for exactly this), not just existing on the wire.
  • Standing up tracing infrastructure only after a production incident makes the need obvious — it's cheap to add early and genuinely difficult to reconstruct a request's cross-service path after the fact without it.