PodDisruptionBudgets for MCP Servers on EKS

What PodDisruptionBudgets actually do (and don't do)

A PodDisruptionBudget (PDB) is a Kubernetes policy object that constrains voluntary disruptions — situations where the cluster itself decides to evict a pod. The Eviction API (used by kubectl drain, the cluster autoscaler, and EKS managed node group rollovers) checks the PDB before evicting any pod. If eviction would push the number of available pods below minAvailable (or above maxUnavailable), the API returns HTTP 429 and the drain retries later.

PDBs do not protect against involuntary disruptions: if a node loses power, the AWS hypervisor reclaims the instance, or the kernel OOM-killer fires, your pods die regardless of any PDB. For those scenarios, you need multiple replicas spread across availability zones — which a PDB alone cannot enforce (see node affinity and topology spread).

Key resource fields:

• spec.minAvailable — integer or percentage; minimum pods that must remain available during voluntary disruption. Mutually exclusive with maxUnavailable.
• spec.maxUnavailable — integer or percentage; maximum pods that may be unavailable at once. Use this when you want drain speed to scale with replica count.
• spec.selector.matchLabels — must match the pod labels on the target Deployment or StatefulSet exactly. A mismatch means the PDB silently governs zero pods.

Check how many evictions are currently allowed with:

kubectl get pdb mcp-server-pdb -n prod -o wide
# NAME             MIN-AVAILABLE   MAX-UNAVAILABLE   ALLOWED-DISRUPTIONS   AGE
# mcp-server-pdb   1               N/A               1                     3d

The ALLOWED-DISRUPTIONS column reflects status.disruptionsAllowed. If this reads 0 when you expect a non-zero value, check whether replicas are healthy and whether the selector is correct.

PDB and Deployment YAML

The selector in the PDB must match the labels your Deployment stamps onto pods. The safest pattern is to define a dedicated label like app: mcp-server and reference it in both objects. Here is a complete example for a two-replica MCP server deployment:

# pdb.yaml
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: mcp-server-pdb
  namespace: prod
spec:
  minAvailable: 1          # at least 1 pod must stay up during voluntary disruption
  selector:
    matchLabels:
      app: mcp-server      # must match .spec.template.metadata.labels in Deployment

# deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mcp-server
  namespace: prod
spec:
  replicas: 2
  selector:
    matchLabels:
      app: mcp-server
  template:
    metadata:
      labels:
        app: mcp-server    # PDB matchLabels targets this
    spec:
      terminationGracePeriodSeconds: 60   # time to finish in-flight tool calls
      containers:
        - name: mcp-server
          image: your-registry/mcp-server:latest
          ports:
            - name: http
              containerPort: 3000
          lifecycle:
            preStop:
              exec:
                command: ["/bin/sh", "-c", "sleep 5"]  # see preStop section below
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0   # never take a pod down during rollout without a replacement

With minAvailable: 1 and 2 replicas, the drain can evict one pod and then must wait for it to finish and the replacement to become ready before evicting the second. This is the right default for most MCP server deployments.

If you use a percentage instead of an integer, Kubernetes rounds down. minAvailable: "50%" on 3 replicas = 1 pod must remain (floor(1.5) = 1), which allows 2 pods to be evicted concurrently. For small replica counts, explicit integers are safer and more predictable.

Graceful shutdown: terminationGracePeriodSeconds and preStop hooks

The PDB controls when a pod gets evicted. terminationGracePeriodSeconds and the preStop lifecycle hook control how cleanly the pod exits after eviction is permitted.

The shutdown sequence when kubelet evicts a pod:

1. kubelet executes the preStop hook (if defined) and waits for it to complete.
2. kubelet sends SIGTERM to PID 1 in the container.
3. The process has until terminationGracePeriodSeconds expires (counting from step 1, not step 2) to exit cleanly.
4. If the process is still running when the grace period expires, kubelet sends SIGKILL.

For MCP servers, "clean exit" means: stop accepting new connections, finish all in-flight JSON-RPC tool calls, flush any buffered logs, and close database connection pools. Thirty to sixty seconds is typically enough for tool calls that hit external APIs; set it higher only if your tools have genuinely long-running operations.

The preStop hook runs before SIGTERM. A common pattern is a short sleep to give the load balancer time to remove the pod from its target group before the server stops accepting connections — without this, the load balancer may still route new requests to a pod that has already received SIGTERM and is closing its listener:

# More complete preStop hook for an MCP server behind an ALB or NLB
lifecycle:
  preStop:
    exec:
      command:
        - /bin/sh
        - -c
        - |
          # Signal the app to stop accepting new connections
          kill -SIGUSR1 1
          # Wait for the load balancer to deregister this target (ALB deregistration
          # delay is 30s by default; match or exceed that here)
          sleep 30

If your MCP server process does not handle SIGUSR1, use the HTTP variant to call a drain endpoint instead:

lifecycle:
  preStop:
    httpGet:
      path: /admin/drain
      port: 3000

Make sure terminationGracePeriodSeconds is greater than the preStop sleep plus your worst-case tool call duration. A common mistake is setting a 30s grace period with a 30s preStop sleep, leaving zero seconds for actual in-flight requests to complete before SIGKILL fires.

Check the effective shutdown sequence with:

kubectl describe pod <pod-name> -n prod
# Look for:
#   Terminating      True    <timestamp>
# and the "Events" section showing preStop, SIGTERM, SIGKILL timings

How EKS uses PDBs during managed node group upgrades

EKS managed node group rollovers (triggered by a Kubernetes version upgrade, AMI update, or launch template change) drain each old node before replacing it. The upgrade process calls the Eviction API in exactly the same way as kubectl drain — your PDB applies here without any extra configuration.

If the PDB is violated — for example, all replicas are on the node being drained — the upgrade will block and eventually time out (the default is 15 minutes per node). You will see an event like:

Cannot evict pod as it would violate the pod's disruption budget.
Eviction of pod prod/mcp-server-7f4b9c-xk9np is not allowed,
0 of 1 disruptions are allowed by PDB mcp-server-pdb

The fix is always one of: spread replicas across nodes (topology spread constraints or anti-affinity rules), or temporarily lower minAvailable before draining manually. Never reach for kubectl drain --force on production nodes — it bypasses the PDB entirely and evicts pods with SIGKILL regardless of in-flight requests.

The cluster autoscaler also uses the Eviction API when it decides to scale down an underutilized node. If your MCP server's PDB would be violated by removing the last pod from a node, the autoscaler will skip that node and leave it running (it logs pod-disruption-budget-related as the reason). This is the correct behavior — but if it causes nodes to never scale down, check whether your replica count and PDB settings together always allow at least one eviction.

Quick reference: PDB behavior per disruption type:

PDB and Deployment rolling update interaction

PDBs and Deployment rollingUpdate settings operate at different layers. The PDB governs Eviction API calls; maxUnavailable/maxSurge governs how the Deployment controller replaces pods during a rollout. They do not directly interact — but they can compound in surprising ways.

Example: you have 2 replicas, minAvailable: 1 in the PDB, maxUnavailable: 1 in the rolling update, and a node drain fires simultaneously with a rollout. The rollout may have already taken one pod down (within its maxUnavailable budget), leaving only 1 pod available. Now the drain tries to evict that remaining pod — and the PDB blocks it because dropping it would leave 0 available. The drain stalls until the rollout completes and brings the replacement pod to Ready.

The safest combination for MCP servers during a rollout is maxUnavailable: 0 and maxSurge: 1 on the Deployment. This means rollouts temporarily overprovision by one pod rather than underprovision — and the PDB never has to block a concurrent drain because the available pod count never drops below minAvailable.

Failure modes reference