Production Best Practices

Getting a cluster running versus getting it production-ready#

Everything so far in this track is enough to get an application running on Kubernetes. Running it reliably in production, under real traffic and real failures, needs a further set of practices that don't come from any single resource type — they're about how deliberately you configure the resources you already know: resource requests/limits, pod disruption budgets, and rollout tuning. Each targets a specific, common way a cluster degrades gracefully in theory but badly in practice.

Resource requests and limits, done deliberately#

The Pods/Deployments/Services page introduced resources.requests/resources.limits as fields on a container spec. In production, getting their actual values right matters far more than knowing the syntax:

resources:
  requests:
    cpu: "250m"
    memory: "256Mi"
  limits:
    cpu: "500m"
    memory: "256Mi"

• requests is what the scheduler reserves — a node needs at least this much unreserved capacity to place the Pod at all. Set too low, and the scheduler happily over-packs a node with Pods that all end up fighting for real resources under load, since the guarantee they were scheduled against wasn't the resources they actually need.
• limits is the hard ceiling. A CPU limit just throttles the container once it's hit — it keeps running, only slower. A memory limit is different and stricter: exceeding it gets the container killed outright (OOMKilled), not throttled, because unlike CPU there's no way to "slow down" memory usage after the fact.

Notice the example sets memory.requests equal to memory.limits — this is a deliberate, common production pattern (giving a Pod Guaranteed QoS when CPU is matched too) that avoids a Pod being killed for exceeding a limit it was never actually guaranteed protection up to. A CPU limit set well above the request, by contrast, is normal and intentional — it lets a Pod burst above its baseline for a short spike without being throttled immediately, as long as the node has spare CPU capacity available at that moment.

kubectl top pods

NAME                     CPU(cores)   MEMORY(bytes)
my-app-7d8f9c-4k2j9      180m         210Mi
my-app-7d8f9c-x2j4k      310m         245Mi

kubectl top (requires the metrics-server add-on running in the cluster) shows real observed usage — the same starting point as EC2 right-sizing covered in the AWS track: watch real usage for a representative period, then set requests/limits based on that, not a guess made before the workload ever ran.

Pod Disruption Budgets#

Kubernetes sometimes needs to evict Pods for reasons that have nothing to do with your application's own rollouts — draining a node for a cluster upgrade, or the cluster autoscaler consolidating underused nodes. Left unconstrained, this voluntary disruption could evict every replica of a Deployment at once if they all happen to land on the node(s) being drained simultaneously. A PodDisruptionBudget (PDB) caps how much of that can happen at once:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: my-app-pdb
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: my-app

With minAvailable: 2 and, say, 3 replicas running, Kubernetes will not voluntarily evict a Pod if doing so would drop the available count below 2 — a node drain touching this Deployment's Pods proceeds one at a time, always keeping at least 2 serving traffic, rather than potentially taking all 3 down together. maxUnavailable is the equivalent expressed as a cap instead of a floor (maxUnavailable: 1 given 3 replicas means the same practical guarantee here).

It's worth being precise about what a PDB does and doesn't cover: it governs voluntary disruptions initiated by the cluster (drains, autoscaling), not involuntary ones (a node crashing outright) — nothing can guarantee availability against a sudden hardware failure, only against disruptions the cluster chooses to schedule.

Tuning the rolling update strategy#

The Pods/Deployments/Services page showed RollingUpdate with maxUnavailable/maxSurge at their common defaults. In production, the right values genuinely depend on the workload:

spec:
  replicas: 10
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 0
      maxSurge: 2

A latency-sensitive, high-traffic service typically wants maxUnavailable: 0 with a modest maxSurge (capacity never dips, cost is briefly running a couple of extra Pods during rollout); a low-traffic internal tool might not care and can use the cheaper default that dips capacity briefly instead. This is the same rolling-deployment trade-off covered generally in the CI/CD track's deployment strategies page, expressed here as concrete Deployment fields.

Common mistakes#

• Setting no resources.requests/limits at all in production — the scheduler has no real basis for placement decisions, and a single misbehaving Pod can starve every other Pod sharing its node.
• Setting a memory limit without understanding it means the container gets killed (not throttled) the moment it's exceeded — an application with an occasional legitimate memory spike needs that spike accounted for in the limit, not discovered via repeated OOMKilled restarts.
• Running a multi-replica Deployment with no PodDisruptionBudget at all — a routine node drain during a cluster upgrade can then take down more replicas simultaneously than the application can actually tolerate.
• Leaving maxUnavailable/maxSurge at whatever the default happened to be instead of deliberately choosing values that match the workload's actual tolerance for reduced capacity during a rollout.