Observability for Microservices

Why it's harder with many services#

In a monolith, a single stack trace tells the whole story — every frame ran in the same process, and one log file has the full picture of a request from start to finish. Once a single user request fans out across five services, the bug or the slowdown could be in any one of them, none of them individually has the complete picture, and the network hops between them are themselves a new source of failure. Observability tooling exists specifically to stitch that story back together across process and network boundaries.

Structured logging with a correlation ID#

Every request gets a unique ID the moment it enters the system — usually at the API gateway, or the first service that receives it — and every service in the resulting call chain must carry that ID forward and stamp it onto every log line it writes.

// At the edge (API Gateway, or the first service in the chain):
// reuse the caller's request ID if one was sent, otherwise mint a new one
String requestId = Optional.ofNullable(request.getHeader("X-Request-Id"))
    .orElseGet(() -> UUID.randomUUID().toString());

MDC.put("requestId", requestId); // SLF4J's Mapped Diagnostic Context — attaches to every log line on this thread

// Every outgoing call this service makes — sync or async — carries the ID onward
httpRequestBuilder.header("X-Request-Id", MDC.get("requestId"));

kafkaTemplate.send(MessageBuilder
    .withPayload(event)
    .setHeader("requestId", MDC.get("requestId"))
    .build());

With logging configured to emit JSON (structured logging, rather than free-text lines), the requestId field becomes queryable once logs are shipped to a central store:

{"timestamp":"2026-08-25T10:15:03Z","service":"orders-service","level":"INFO","requestId":"6f2a1c9e","message":"Order created"}
{"timestamp":"2026-08-25T10:15:03Z","service":"inventory-service","level":"INFO","requestId":"6f2a1c9e","message":"Stock reserved"}
{"timestamp":"2026-08-25T10:15:04Z","service":"notification-service","level":"ERROR","requestId":"6f2a1c9e","message":"Email send failed: SMTP timeout"}

Filtering a log aggregator (Elasticsearch, Loki, or similar) for requestId=6f2a1c9e reconstructs the entire cross-service timeline of that one request — this is the whole reason structured logging matters: fields like requestId, service, and level need to be queryable, not buried inside a sentence.

Distributed tracing: spans and traces#

A trace represents one end-to-end request. It's composed of spans, each representing one unit of work — a single service handling a call, a database query, a message being published. Every span carries a start/end timestamp, a traceId shared by every span in the whole trace, its own spanId, and a parentSpanId pointing at whoever called it. That parent/child structure is exactly what lets a tracing UI (Jaeger, Zipkin, or a vendor APM tool) draw the request as a waterfall or flame graph:

Trace: 6f2a1c9e
├─ Span: api-gateway              (12ms)
│  └─ Span: orders-service        (9ms)
│     ├─ Span: orders DB query    (2ms)
│     └─ Span: inventory-service  (5ms)   <- an HTTP call, shows up as a child span
│        └─ Span: inventory DB query (3ms)

This is where "correlation ID" and "tracing" stop being separate ideas in practice: most tracing instrumentation (OpenTelemetry included) uses the trace/span IDs themselves as the correlation mechanism, and many teams simply log the traceId as their correlation ID rather than minting a second, unrelated one.

The three pillars#

None of the three replaces the others, and they're most useful chained together: a metric (p99 latency on inventory-service just spiked) tells you something is wrong; a trace tells you where, in a specific slow request, the time actually went; a log tells you exactly why it happened on that instance, at that moment.

Implementation in Spring Boot, ASP.NET Core and Go#

Instrumenting the same request — orders-service handling POST /orders, which calls inventory-service — with real distributed tracing via OpenTelemetry in each stack.

Spring Boot — Micrometer Tracing bridged to OpenTelemetry#

# application.yml
management:
  tracing:
    sampling:
      probability: 1.0
  otlp:
    tracing:
      endpoint: http://otel-collector:4318/v1/traces

With micrometer-tracing-bridge-otel and opentelemetry-exporter-otlp on the classpath, Spring's own HTTP clients (RestClient, WebClient) are automatically instrumented — every span they create carries the trace onward with no extra code. A manual span is for the parts of the request that aren't an outgoing HTTP call on their own, like a specific business operation worth seeing on its own line in a trace:

@RestController
public class OrdersController {

    private final Tracer tracer;
    private final RestClient inventoryClient;

    public OrdersController(Tracer tracer, RestClient.Builder builder) {
        this.tracer = tracer;
        this.inventoryClient = builder.baseUrl("http://inventory-service").build();
    }

    @PostMapping("/orders")
    public OrderResponse placeOrder(@RequestBody OrderRequest request) {
        Span span = tracer.nextSpan().name("reserve-stock").start();
        try (Tracer.SpanInScope ws = tracer.withSpan(span)) {
            span.tag("order.id", request.orderId());

            InventoryResponse response = inventoryClient.post()
                .uri("/api/reservations")
                .body(request)
                .retrieve()
                .body(InventoryResponse.class); // the outgoing call above becomes a child span automatically

            return OrderResponse.confirmed(response);
        } finally {
            span.end();
        }
    }
}

ASP.NET Core — ActivitySource and the OpenTelemetry .NET SDK#

var serviceName = "orders-service";
var activitySource = new ActivitySource(serviceName);

builder.Services.AddOpenTelemetry()
    .ConfigureResource(resource => resource.AddService(serviceName))
    .WithTracing(tracing => tracing
        .AddSource(serviceName)
        .AddAspNetCoreInstrumentation()  // spans for every incoming request
        .AddHttpClientInstrumentation()  // spans for every outgoing HttpClient call, with trace context propagated automatically
        .AddOtlpExporter(otlp => otlp.Endpoint = new Uri("http://otel-collector:4317")));

app.MapPost("/orders", async (OrderRequest request, HttpClient inventoryClient) =>
{
    using var activity = activitySource.StartActivity("reserve-stock");
    activity?.SetTag("order.id", request.OrderId);

    var response = await inventoryClient.PostAsJsonAsync(
        "http://inventory-service/api/reservations", request);

    return Results.Ok(await response.Content.ReadFromJsonAsync<InventoryResponse>());
});

AddSource(serviceName) is what tells the SDK to actually export spans created by the manual activitySource.StartActivity(...) call above, alongside the automatic ones AddAspNetCoreInstrumentation()/AddHttpClientInstrumentation() create for the incoming request and outgoing call.

Go — the OpenTelemetry Go SDK#

var tracer = otel.Tracer("orders-service")

func placeOrderHandler(w http.ResponseWriter, r *http.Request) {
    ctx, span := tracer.Start(r.Context(), "place-order")
    defer span.End()

    var order OrderRequest
    json.NewDecoder(r.Body).Decode(&order)
    span.SetAttributes(attribute.String("order.id", order.OrderID))

    if err := reserveStock(ctx, order); err != nil {
        span.RecordError(err)
        http.Error(w, "reservation failed", http.StatusBadGateway)
        return
    }

    w.WriteHeader(http.StatusCreated)
}

func reserveStock(ctx context.Context, order OrderRequest) error {
    ctx, span := tracer.Start(ctx, "reserve-stock")
    defer span.End()

    body, _ := json.Marshal(order)
    req, _ := http.NewRequestWithContext(ctx, http.MethodPost,
        "http://inventory-service/api/reservations", bytes.NewReader(body))

    // propagate the trace context (the traceparent header) onto the outgoing call
    otel.GetTextMapPropagator().Inject(ctx, propagation.HeaderCarrier(req.Header))

    resp, err := http.DefaultClient.Do(req)
    if err != nil {
        return err
    }
    defer resp.Body.Close()
    return nil
}

tracer.Start(ctx, ...) returns a new context.Context carrying the span — passing that context (not the original one) into reserveStock, and from there into otel.GetTextMapPropagator().Inject(...), is exactly what keeps reserve-stock correctly nested under place-order as a child span, and what puts a valid traceparent header onto the call to inventory-service so its own span joins the same trace.

Common mistakes#

• Logging without a correlation/trace ID — in a multi-service system, one isolated log line with no way to connect it to the rest of the request it belongs to is close to useless for debugging.
• Instrumenting tracing only at the edge (the API gateway) and skipping it for internal service-to-service calls and async event processing — a trace that stops at the first hop misses exactly the cross-service problems tracing exists to catch.
• Treating a metrics dashboard as sufficient on its own — it tells you that p99 latency spiked at 10:15, not which request was slow or why; you still need traces and logs to finish the investigation.
• Generating a new correlation ID at every service instead of propagating the one from upstream — this silently breaks the entire premise of being able to follow one request across the system.