Observability and Logging

Why observability needs its own approach in Kubernetes#

A Pod's filesystem — including any log file an application writes to disk — disappears the moment that Pod is deleted, and Pods are deleted constantly: a rollout replaces them, a node failure replaces them, the Horizontal Pod Autoscaler (see the ConfigMaps/Secrets/Scaling page) scales them down under lighter load. Logging to local disk the way you might on a single long-lived server means losing that history the instant the Pod that wrote it is gone. Kubernetes' answer is that containers should log to stdout/stderr rather than a file, and something outside the Pod's own lifecycle collects and centralizes those streams before they disappear.

Centralized logging, conceptually#

The standard pattern runs a log-collecting agent on every node — most commonly Fluentd or Fluent Bit — as a DaemonSet (a controller that guarantees exactly one copy runs on every node, covered briefly here since it's the mechanism this pattern depends on). Each agent tails the container logs Kubernetes already writes to disk on its node (under /var/log/containers/ on the node itself, populated automatically from every Pod's stdout/stderr) and ships them to a central store — Elasticsearch, Loki, or a cloud provider's managed logging service (CloudWatch Logs, covered in the AWS track, is a common destination for clusters running on EKS).

Node 1                          Node 2
┌─────────────────────┐         ┌─────────────────────┐
│ Pod A -> stdout      │         │ Pod C -> stdout      │
│ Pod B -> stdout      │         │ Pod D -> stdout      │
│         |            │         │         |            │
│  Fluent Bit (DaemonSet)│        │  Fluent Bit (DaemonSet)│
└─────────┼────────────┘         └─────────┼────────────┘
          │                                 │
          └──────────────┬──────────────────┘
                          ▼
              Centralized log store
           (Elasticsearch / Loki / CloudWatch)

The payoff: logs survive Pod deletion, can be searched and correlated across every Pod and node at once (essential once an application runs as more than a handful of replicas), and a single dashboard shows a request's path across several microservices instead of requiring someone to kubectl logs each one individually and stitch the timeline together by hand.

kubectl get daemonset -n logging

NAME          DESIRED   CURRENT   READY   AGE
fluent-bit    3         3         3       12d

Liveness and readiness probes, revisited#

The Pods/Deployments/Services page introduced readinessProbe for keeping a still-starting Pod out of Service rotation. The other half of health checking is livenessProbe, and the difference between the two is exactly what makes each one useful for a distinct failure mode:

spec:
  containers:
    - name: app
      image: my-app:1.2
      ports:
        - containerPort: 3000
      readinessProbe:
        httpGet:
          path: /healthz/ready
          port: 3000
        initialDelaySeconds: 5
        periodSeconds: 10
      livenessProbe:
        httpGet:
          path: /healthz/live
          port: 3000
        initialDelaySeconds: 15
        periodSeconds: 20
        failureThreshold: 3

A common, genuinely useful pattern is to make readiness stricter than liveness: readiness might check that a downstream dependency (a database connection pool, a cache) is currently reachable, while liveness only checks that the process itself is still responsive at all. That way, a temporary database blip pulls a Pod out of rotation (readiness fails) without triggering a pointless restart of a container that was never actually broken (liveness still passes) — restarting it wouldn't fix a downstream outage anyway, and would just add churn on top of an already-degraded dependency.

kubectl logs and kubectl describe for debugging#

Centralized logging is the right long-term answer, but the fastest way to debug a single misbehaving Pod right now is still direct kubectl commands:

# Stream a Pod's current logs
kubectl logs my-app-7d8f9c-x2j4k

# Logs from the *previous* container instance — essential after a crash/restart,
# since a fresh container's logs start empty
kubectl logs my-app-7d8f9c-x2j4k --previous

# Follow logs live, like tail -f
kubectl logs -f my-app-7d8f9c-x2j4k

# A Pod with multiple containers needs the container named explicitly
kubectl logs my-app-7d8f9c-x2j4k -c sidecar

# Full detail on a Pod: current status, resource requests/limits, recent events
kubectl describe pod my-app-7d8f9c-x2j4k

kubectl describe pod is often more useful than the logs themselves for a Pod that never even reached a running state — its Events section at the bottom surfaces exactly why scheduling or startup failed, in plain language:

Events:
  Type     Reason     Age                From               Message
  ----     ------     ----               ----               -------
  Warning  Failed     2m (x4 over 5m)    kubelet            Failed to pull image "my-app:1.2": not found
  Warning  BackOff    1m (x6 over 4m)    kubelet            Back-off restarting failed container

This particular output — a bad image tag — never produces any application logs at all, since the container never actually started; kubectl logs would return nothing useful, while kubectl describe's Events section names the exact problem directly.

Common mistakes#

• Configuring an application to log to a file on the container's local disk instead of stdout/stderr — the log-shipping agent pattern above depends on stdout/stderr; file-based logs need extra sidecar configuration to be collected at all, and are lost entirely once the Pod is deleted.
• Using the same endpoint (or the exact same check) for both readinessProbe and livenessProbe — a transient downstream dependency failure then triggers an unnecessary container restart instead of just a brief pause in traffic.
• Reaching straight for kubectl logs on a Pod that's stuck in Pending or CrashLoopBackOff — for a Pod that never successfully started, kubectl describe pod and its Events section is almost always the faster path to the actual cause.
• Forgetting --previous when debugging a Pod that already restarted — plain kubectl logs only shows the current container instance's output, which is empty or unhelpful if the crash happened in the one before it.