Deep Dive · AWS EKS

Running MCP Servers on EKS: The Production Playbook — Availability, Autoscaling, and Isolation

Published 2026-09-18 · 22 min read

Most MCP server teams land on EKS and reach for the obvious knobs: set a replica count, configure a readiness probe, call it done. What they get is a deployment that looks healthy in a calm cluster but sheds requests during node group upgrades, restarts pods when the database hiccups, fails to scale before users see errors, and runs in a flat network where a single compromised pod can reach everything else. Each of those failure modes has a different root cause and a different Kubernetes primitive that fixes it. This post synthesizes five of them — topology spread constraints, PodDisruptionBudgets, readiness and liveness probes, Horizontal Pod Autoscaler with custom metrics, and NetworkPolicies — and shows how they compose into a single coherent production posture.

The failure taxonomy: five classes, five primitives

Before wiring up any YAML, it helps to think about what can actually go wrong. MCP server failures on EKS fall into five classes, and the important thing is that they are mostly independent — each class requires a different primitive to address, and the primitives don't overlap much:

Failure class Example trigger Kubernetes primitive What it does
Correlated placement failure All 3 replicas on the same AZ node that goes down topologySpreadConstraints + podAntiAffinity Ensures replicas are distributed across failure domains before the failure happens
Voluntary simultaneous eviction kubectl drain, node group upgrade, cluster autoscaler scale-down PodDisruptionBudget Forces the Eviction API to drain pods one at a time; blocks drain if it would violate the budget
Traffic sent to unready/unhealthy pod Dependency goes offline, pod still in Service endpoints; restart loop from shared probe endpoint readinessProbe + livenessProbe + startupProbe readinessProbe gates traffic without restarting; livenessProbe restarts only on unrecoverable failure
Capacity doesn't match load Bursty tool-call traffic; CPU stays flat while queue depth grows HPA with custom metrics (or KEDA) Adds pods on the right signal; avoids the CPU-miss problem for I/O-bound MCP servers
Lateral movement after compromise Injected LLM tool output reaches a malicious host; pod sends unauthorized traffic to other namespaces NetworkPolicy with default-deny Restricts ingress and egress to explicitly allowed pods, namespaces, and ports

You need all five. Topology spread without a PDB means replicas are distributed but can still be evicted simultaneously during a node upgrade. A PDB without topology spread means the eviction serialization works, but if you have two replicas on two nodes in the same AZ, an AZ outage takes both. A readiness probe without topology spread means the pod drains correctly but if all replicas live on one node, a hardware failure takes everything. And so on — each primitive plugs a different gap. The rest of this guide walks through each layer and ends with a reference YAML combining them.

Layer 1: Topology spread — prevent correlated placement failures before they happen

The scheduler's default behavior is bin-packing: it places pods wherever resources are available. For a three-replica MCP server deployment, the default policy will happily put all three on the same node or in the same AZ if that's where resources are. One node failure or AZ outage and you're at zero available pods — not because of a bug in your code, but because no placement constraint prevented it.

topologySpreadConstraints expresses this as a scheduling rule: "keep the difference in pod count across topology domains below maxSkew." Two constraints — one for AZ-level spread and one for node-level spread within each AZ — cover the most important failure domains:

# topology spread configuration for a 3+ replica MCP server
spec:
  topologySpreadConstraints:
    # AZ-level: no AZ should have more than 1 extra pod vs the least-loaded AZ
    - maxSkew: 1
      topologyKey: topology.kubernetes.io/zone
      whenUnsatisfiable: DoNotSchedule
      labelSelector:
        matchLabels:
          app: mcp-server
    # Node-level: soft preference to spread across nodes within each AZ
    - maxSkew: 1
      topologyKey: kubernetes.io/hostname
      whenUnsatisfiable: ScheduleAnyway   # soft — don't block if nodes are unequal
      labelSelector:
        matchLabels:
          app: mcp-server

The DoNotSchedule on the AZ-level constraint means new pods will not schedule if they'd violate the spread. Use this for AZ spread — you want it to fail loudly if the cluster can't honor it, rather than silently packing pods into one AZ. Use ScheduleAnyway for node-level spread so that the deployment can still scale beyond the node count when needed.

For explicit node separation of replicas, add podAntiAffinity alongside the spread constraint — it keeps any two replicas off the same node entirely:

affinity:
  podAntiAffinity:
    preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 100
        podAffinityTerm:
          labelSelector:
            matchExpressions:
              - key: app
                operator: In
                values:
                  - mcp-server
          topologyKey: kubernetes.io/hostname

Use preferredDuringScheduling (soft) here rather than requiredDuringScheduling (hard). The hard variant creates a scheduling deadlock if you ever need to scale beyond the node count: the 4th pod has no node to schedule on because every existing node already hosts a replica. The soft variant expresses the preference without creating an un-schedulable state.

Topology spread does not protect against voluntary eviction — that's what the PDB is for. But it is the prerequisite: if all replicas are co-located, the PDB's guarantee of "at least 1 available during voluntary disruption" is meaningless because a hardware failure (which the PDB doesn't cover) would still wipe everything simultaneously.

Layer 2: PodDisruptionBudgets — serialize voluntary evictions

EKS routine operations — node group upgrades, AMI updates, cluster autoscaler scale-down, even a manual kubectl drain — all go through the Kubernetes Eviction API. Every eviction request checks the relevant PodDisruptionBudgets before proceeding. If an eviction would push the available pod count below spec.minAvailable, the API returns HTTP 429 and the draining operation retries later.

A PDB with minAvailable: 1 on a two-replica deployment means the drain can evict one pod, but must wait for a replacement to become Ready before evicting the second:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: mcp-server-pdb
  namespace: mcp
spec:
  minAvailable: 1
  selector:
    matchLabels:
      app: mcp-server   # must match .spec.template.metadata.labels exactly

The selector is the most common source of silent PDB misconfiguration. If spec.selector.matchLabels doesn't exactly match the pod labels on your Deployment, the PDB governs zero pods and the drain ignores it entirely. Check with:

kubectl get pdb mcp-server-pdb -n mcp -o wide
# ALLOWED-DISRUPTIONS column should show a non-zero value when replicas are healthy
# If it shows N/A, the selector is wrong

Beyond the PDB object itself, the other half of graceful shutdown is terminationGracePeriodSeconds and the preStop lifecycle hook on the pod spec. The PDB controls when eviction is permitted; these control how cleanly the pod exits once eviction is allowed.

The shutdown sequence: kubelet executes the preStop hook, then sends SIGTERM to the container, then waits until terminationGracePeriodSeconds expires (counting from preStop start, not SIGTERM), then sends SIGKILL. For MCP servers behind an ALB or NLB, the preStop hook should signal the load balancer to deregister the target and wait for the deregistration delay before the server stops accepting connections:

# pod spec — add alongside the container image/ports/resources
terminationGracePeriodSeconds: 90
containers:
  - name: mcp-server
    # ...
    lifecycle:
      preStop:
        exec:
          command:
            - /bin/sh
            - -c
            - |
              # Signal app to stop accepting new tool calls
              kill -SIGUSR1 1
              # Wait for ALB deregistration (default 30s deregistration delay)
              sleep 35

The math: terminationGracePeriodSeconds must exceed preStop duration (35 s in the example) plus the p99 latency of your slowest tool call. If your LLM-calling tools can take up to 45 seconds, set terminationGracePeriodSeconds: 90 (35 s deregistration + 45 s worst case + 10 s buffer). Setting the grace period equal to the preStop sleep leaves zero time for in-flight requests to complete before SIGKILL fires.

One interaction to understand: the PDB and Deployment rollingUpdate settings operate independently but can compound. If a node drain fires during a rolling update, and the rollout has already taken one pod down (within maxUnavailable), the PDB may block the drain from evicting the remaining pod because that would drop below minAvailable. The safest combination is maxUnavailable: 0 with maxSurge: 1 on the Deployment — rollouts overprovision rather than underprovision, so the PDB budget is never crowded out by a simultaneous rollout.

Layer 3: Probe architecture — gate traffic without causing restart loops

The single most common MCP server incident on EKS comes from one configuration mistake: using the same /health endpoint for both the readiness and liveness probes. When a database goes offline for 30 seconds, the shared endpoint returns 503, the liveness probe counts failures, and kubelet restarts the container — which is exactly the wrong response. The pod didn't need a restart; it needed to be removed from the load balancer until the database came back. The restart compounded the incident: in-flight tool calls were killed, the pod went through its cold start cycle, and the remaining replicas had to absorb the traffic spike.

The fix is two separate endpoints with two fundamentally different semantics:

Complete probe configuration for an MCP server:

# Complete probe block — add to spec.template.spec.containers[0]
startupProbe:
  httpGet:
    path: /healthz/ready
    port: http
  initialDelaySeconds: 5
  periodSeconds: 5
  timeoutSeconds: 3
  failureThreshold: 30        # 150s startup budget — covers slow Secrets Manager init

readinessProbe:
  httpGet:
    path: /healthz/ready
    port: http
  initialDelaySeconds: 0      # startupProbe already handles the delay
  periodSeconds: 10
  timeoutSeconds: 5
  failureThreshold: 3         # 30s before traffic shed

livenessProbe:
  httpGet:
    path: /healthz/live
    port: http
  initialDelaySeconds: 0
  periodSeconds: 15
  timeoutSeconds: 10          # longer timeout — must survive CPU pressure
  failureThreshold: 3         # 45s before restart

The startupProbe is critical for MCP servers that initialize slowly — those that fetch configuration from AWS Secrets Manager, establish connection pools to multiple backends, or load large in-memory indexes. Without it, you have two bad options: set initialDelaySeconds high enough to cover every slow start (which delays detection of real startup failures), or accept restart loops during normal slow starts. The startupProbe disables both readiness and liveness until it passes once, then steps out of the way.

The Node.js implementation of these two endpoints should express the semantics cleanly. The readiness handler exercises actual dependencies; the liveness handler checks only process-level state:

// health.js — minimal implementation for the two endpoints

let isReady = false;     // set true after full async init
let isLive = true;       // set false only on unrecoverable error
let dbPool = null;

export function markReady(pool) {
  dbPool = pool;
  isReady = true;
}

// GET /healthz/ready
// Checks actual dependency health — slow dependency = 503, not restart
export async function readinessHandler(req, res) {
  if (!isReady) {
    return res.status(503).json({ status: 'not_ready', reason: 'initializing' });
  }
  try {
    const client = await Promise.race([
      dbPool.connect(),
      new Promise((_, reject) =>
        setTimeout(() => reject(new Error('timeout')), 2000)
      )
    ]);
    await client.query('SELECT 1');
    client.release();
    res.status(200).json({ status: 'ready' });
  } catch (err) {
    res.status(503).json({ status: 'not_ready', reason: err.message });
  }
}

// GET /healthz/live
// Only fails on unrecoverable process state — never checks dependencies
export function livenessHandler(req, res) {
  if (!isLive) {
    return res.status(500).json({ status: 'dead' });
  }
  // Optional: check event loop lag as a deadlock proxy
  const start = Date.now();
  setImmediate(() => {
    const lag = Date.now() - start;
    if (lag > 5000) {
      res.status(500).json({ status: 'event_loop_blocked', lag_ms: lag });
    } else {
      res.status(200).json({ status: 'live', lag_ms: lag });
    }
  });
}

Always mount these routes before any authentication middleware. Kubelet does not send auth headers when probing — if auth middleware runs first, every probe returns 401 and the pod never becomes Ready.

The I/O-bound signal problem: why CPU misleads the HPA for MCP servers

Before configuring the HPA, it is worth understanding why the default CPU-based autoscaling is wrong for most MCP workloads. A typical MCP tool call spends its latency budget waiting: waiting for an LLM API response (1–30 seconds), waiting for a database query (5–500ms), waiting for an external HTTP call. During all of that waiting, the pod's CPU utilization is near zero. If the HPA is configured to scale at 65% CPU utilization, it sees healthy numbers while 50 tool calls are queued and your users are staring at timeouts.

The right signal for I/O-bound MCP servers is one of:

CPU still belongs in the HPA as a guard for genuinely compute-heavy tools (code execution, image processing, heavy JSON transformation). Use a multi-metric HPA and let Kubernetes scale when any metric crosses its threshold — the system scales on the most-stressed signal, not the most-common one.

There is one prerequisite that blocks all CPU-based HPA: resource.requests.cpu must be set on every container in the pod spec. Without it, metrics-server cannot compute utilization and the HPA reports <unknown>/65% indefinitely. The fix is always just adding the resources block — but because it's easy to miss on new deployments, it's worth calling out explicitly.

Layer 4: HPA with custom metrics — autoscaling on the right signal

The custom metric HPA requires three components: metrics-server (already installed in EKS), Prometheus (scrapes your MCP server's application metric), and prometheus-adapter (bridges Prometheus into the Kubernetes custom metrics API).

First, instrument the server. A gauge counting in-flight tool calls is the most reliable signal and the easiest to implement:

// Prometheus instrumentation for Node.js MCP server
import { Gauge, register } from 'prom-client';
import express from 'express';

const inFlightToolCalls = new Gauge({
  name: 'mcp_tool_call_queue_depth',
  help: 'Number of tool calls currently in-flight or queued'
});

// Wrap your tool call handler
export async function handleToolCall(toolName, params) {
  inFlightToolCalls.inc();
  try {
    return await dispatchTool(toolName, params);
  } finally {
    inFlightToolCalls.dec();
  }
}

// Expose /metrics on a separate port so NetworkPolicy rules can target it specifically
const metricsApp = express();
metricsApp.get('/metrics', async (req, res) => {
  res.set('Content-Type', register.contentType);
  res.end(await register.metrics());
});
metricsApp.listen(9090);

Once the metric is exposed, configure the prometheus-adapter to map it into the custom metrics API:

# prometheus-adapter values.yaml (Helm)
rules:
  custom:
    - seriesQuery: 'mcp_tool_call_queue_depth{namespace!="",pod!=""}'
      resources:
        overrides:
          namespace: { resource: namespace }
          pod:       { resource: pod }
      name:
        matches: "mcp_tool_call_queue_depth"
        as:      "mcp_tool_call_queue_depth"
      metricsQuery: 'avg(<<.Series>>{<<.LabelMatchers>>})'

Verify the metric is visible in the Kubernetes API before wiring the HPA:

kubectl get --raw \
  "/apis/custom.metrics.k8s.io/v1beta1/namespaces/mcp/pods/*/mcp_tool_call_queue_depth" \
  | jq .

If this returns empty results, check: (1) the prometheus-adapter is running and its rules config matches, (2) the ServiceMonitor label matches the Prometheus operator's serviceMonitorSelector, (3) the metric is actually being exported by the running pods (kubectl exec -- curl localhost:9090/metrics | grep mcp_tool).

Now the multi-metric HPA combining CPU (as a guard) and the custom tool-call depth (as the primary signal):

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: mcp-server-hpa
  namespace: mcp
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: mcp-server
  minReplicas: 2
  maxReplicas: 20
  metrics:
    # Primary: scale on tool-call queue depth — the right signal for I/O-bound work
    - type: Pods
      pods:
        metric:
          name: mcp_tool_call_queue_depth
        target:
          type: AverageValue
          averageValue: "5"   # scale out when avg queue depth exceeds 5 per pod
    # Guard: CPU — catches compute-heavy tool spikes
    - type: Resource
      resource:
        name: cpu
        target:
          type: AverageUtilization
          averageUtilization: 65
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 0    # scale up immediately
      policies:
        - type: Pods
          value: 4
          periodSeconds: 60
        - type: Percent
          value: 100
          periodSeconds: 60
      selectPolicy: Max                # use whichever adds more pods
    scaleDown:
      stabilizationWindowSeconds: 180  # wait 3 min of sustained low load
      policies:
        - type: Pods
          value: 2
          periodSeconds: 60
      selectPolicy: Min

The scale-down stabilizationWindowSeconds: 180 matters for MCP servers with slow cold starts. If your server takes 60 seconds to pass its readiness probe, a load spike that causes HPA to scale out and then immediately back down would create a cycle of provisioning and terminating pods. The stabilization window prevents scale-down during transient load drops, giving the autoscaler time to observe that load has genuinely dropped before removing capacity.

For workloads that need scale-to-zero, use KEDA instead of the built-in HPA. KEDA creates its own HPA internally — do not also create a separate HPA targeting the same Deployment, or you will get conflicting scale decisions. The KEDA ScaledObject replaces the HPA entirely.

Layer 5: NetworkPolicies — network isolation and blast radius containment

The single most dangerous fact about Kubernetes NetworkPolicies on EKS is this: you can write them, apply them, and watch kubectl get networkpolicy return them — while every pod in your cluster still freely communicates with every other pod. The default EKS VPC CNI does not enforce NetworkPolicies. The policy objects are stored in etcd; enforcement depends on the CNI plugin.

You need one of:

Always verify enforcement is active by applying a test deny policy, confirming traffic is actually blocked, then removing it. Don't rely on the policy existing in the API server as evidence of enforcement.

The pattern that works reliably: default-deny, then add allowlists. Apply a deny-all policy first (to a test namespace), prove your allowlists work, then roll it out to production. This order prevents the "apply default-deny to a live namespace and break everything" failure mode.

Three policies that every MCP server namespace needs:

# 1. Default deny — applied LAST, after allowlists are in place
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
  namespace: mcp
spec:
  podSelector: {}         # matches every pod in the namespace
  policyTypes:
    - Ingress
    - Egress
  # No rules = deny everything

---
# 2. DNS egress — MUST NOT be omitted or all hostname resolution breaks silently
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-dns-egress
  namespace: mcp
spec:
  podSelector: {}
  policyTypes:
    - Egress
  egress:
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: kube-system
      ports:
        - protocol: UDP
          port: 53
        - protocol: TCP
          port: 53         # TCP fallback for large DNS responses

---
# 3. MCP server allowlist: ingress from API gateway + monitoring, egress to data layer + HTTPS
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: mcp-server-allowlist
  namespace: mcp
spec:
  podSelector:
    matchLabels:
      app: mcp-server
  policyTypes:
    - Ingress
    - Egress
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              role: api-gateway
      ports:
        - protocol: TCP
          port: 8080
    - from:
        - namespaceSelector:
            matchLabels:
              role: monitoring
      ports:
        - protocol: TCP
          port: 9090        # Prometheus metrics scraping
  egress:
    # External LLM APIs and AWS services (use VPC endpoints to avoid CIDR management)
    - ports:
        - protocol: TCP
          port: 443
    # Internal data layer (database, cache)
    - to:
        - namespaceSelector:
            matchLabels:
              role: data-layer
      ports:
        - protocol: TCP
          port: 5432         # PostgreSQL
        - protocol: TCP
          port: 6379         # Redis

The DNS egress rule deserves its own emphasis. When you apply any egress NetworkPolicy to a pod, all egress traffic not explicitly listed is blocked — including DNS queries to kube-dns. Without the DNS rule, every hostname lookup silently fails. Your MCP server can't connect to its database, its LLM provider, or any Kubernetes service by name. The symptom looks like connection refused to everything simultaneously — a confusing failure that takes a long time to diagnose if you don't know to look for missing DNS egress.

One subtle semantics issue: when namespaceSelector and podSelector appear inside the same from list entry, they are ANDed — the pod must be in a matching namespace AND have matching pod labels. If you put them in separate from list items, they are ORed. This distinction creates security bugs when you intend "pods in namespace X with label Y" but accidentally write two separate entries that instead mean "pods in namespace X OR pods with label Y anywhere".

For AWS SDK calls (Secrets Manager, S3, STS, SSM), the cleanest egress pattern is VPC Interface Endpoints: traffic stays within the VPC and uses private IP addresses, so your existing internal CIDR egress rules cover it. Without VPC endpoints, your MCP server needs egress to AWS public IP ranges, which are large, change periodically, and are annoying to maintain as CIDR lists.

Composing all five layers: the full Deployment

Each of the five primitives addresses a different failure class. Here they are in a single Deployment spec:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: mcp-server
  namespace: mcp
spec:
  replicas: 3
  selector:
    matchLabels:
      app: mcp-server
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0    # overprovision rather than underprovision during rollout
  template:
    metadata:
      labels:
        app: mcp-server
    spec:
      # Layer 1: Topology spread — distribute across AZs and nodes
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: topology.kubernetes.io/zone
          whenUnsatisfiable: DoNotSchedule
          labelSelector:
            matchLabels:
              app: mcp-server
        - maxSkew: 1
          topologyKey: kubernetes.io/hostname
          whenUnsatisfiable: ScheduleAnyway
          labelSelector:
            matchLabels:
              app: mcp-server
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
            - weight: 100
              podAffinityTerm:
                labelSelector:
                  matchExpressions:
                    - key: app
                      operator: In
                      values:
                        - mcp-server
                topologyKey: kubernetes.io/hostname

      # Layer 2: Graceful shutdown — works with the PDB to drain cleanly
      terminationGracePeriodSeconds: 90

      containers:
        - name: mcp-server
          image: your-ecr-repo/mcp-server:latest
          ports:
            - name: http
              containerPort: 8080
            - name: metrics
              containerPort: 9090

          # Resources are required for HPA CPU metric
          resources:
            requests:
              cpu: "250m"
              memory: "512Mi"
            limits:
              cpu: "1000m"
              memory: "1Gi"

          # Layer 2: preStop — deregister from load balancer before stopping
          lifecycle:
            preStop:
              exec:
                command: ["/bin/sh", "-c", "kill -SIGUSR1 1; sleep 35"]

          # Layer 3: Probe architecture — three separate probes with distinct semantics
          startupProbe:
            httpGet:
              path: /healthz/ready
              port: http
            initialDelaySeconds: 5
            periodSeconds: 5
            timeoutSeconds: 3
            failureThreshold: 30

          readinessProbe:
            httpGet:
              path: /healthz/ready
              port: http
            periodSeconds: 10
            timeoutSeconds: 5
            failureThreshold: 3

          livenessProbe:
            httpGet:
              path: /healthz/live
              port: http
            periodSeconds: 15
            timeoutSeconds: 10
            failureThreshold: 3

Plus the supporting objects:

---
# Layer 2: PodDisruptionBudget — serialize voluntary evictions
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: mcp-server-pdb
  namespace: mcp
spec:
  minAvailable: 1
  selector:
    matchLabels:
      app: mcp-server

---
# Layer 4: HPA — custom metric primary, CPU guard secondary
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: mcp-server-hpa
  namespace: mcp
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: mcp-server
  minReplicas: 2
  maxReplicas: 20
  metrics:
    - type: Pods
      pods:
        metric:
          name: mcp_tool_call_queue_depth
        target:
          type: AverageValue
          averageValue: "5"
    - type: Resource
      resource:
        name: cpu
        target:
          type: AverageUtilization
          averageUtilization: 65
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 0
      policies:
        - type: Percent
          value: 100
          periodSeconds: 60
      selectPolicy: Max
    scaleDown:
      stabilizationWindowSeconds: 180
      policies:
        - type: Pods
          value: 2
          periodSeconds: 60
      selectPolicy: Min

---
# Layer 5: NetworkPolicy — default deny + DNS + allowlist
# (see the three NetworkPolicy objects in the previous section)

The interaction matrix: what happens when two failure modes arrive simultaneously

Real incidents rarely involve a single failure class. Here are the most important interaction cases:

Simultaneous events Without both primitives With both primitives
Node drain + rolling deployment If rollout already has one pod down, drain evicts the remaining pod and the PDB blocks — upgrade stalls until rollout completes maxUnavailable: 0 on Deployment means rollout never takes a pod down; drain always finds minAvailable satisfied; no stall
Database blip + HPA scale-down readinessProbe on shared endpoint triggers liveness restart; scale-down simultaneously terminates another pod; cascading restart loop Separate readiness probe fails silently (no restart); pod removed from endpoints; HPA scale-down stabilization window holds off scale-down during the blip
AZ outage + load spike Replicas co-located in failed AZ → total outage; HPA tries to scale but requests fail immediately Topology spread means 1/3 of replicas survive in other AZs; HPA scales them out (topology spread on new pods targets surviving AZs); degraded but available
Spot interruption + high load Spot node evicts all pods on it with NoExecute; SIGKILL to all in-flight tool calls simultaneously PDB limits simultaneous evictions; preStop hook drains gracefully within terminationGracePeriodSeconds; topology spread means not all pods on spot nodes
Cluster autoscaler scale-down + LLM API latency spike CA evicts pod while it has 30 in-flight 20s tool calls; all return connection reset PDB blocks CA eviction if available count would drop below minAvailable; tool calls on the other pods complete normally; CA retries after stabilization

Consolidated failure modes reference

A single table covering the most common misconfiguration in each layer:

Layer Symptom Root cause Fix
Topology spread topologySpreadConstraints causes Pending pods DoNotSchedule on hostname with low replica count, or one AZ missing nodes Use ScheduleAnyway on hostname; keep DoNotSchedule only on zone; ensure all AZs have nodes
Topology spread Can't scale Deployment beyond 3 replicas Hard pod anti-affinity on kubernetes.io/hostname with only 3 nodes Switch to preferredDuringScheduling anti-affinity; use topologySpreadConstraints instead
PDB PDB exists but ALLOWED-DISRUPTIONS is N/A; drain ignores it spec.selector.matchLabels doesn't match pod labels kubectl get pods --show-labels and compare; fix the label mismatch
PDB In-flight tool calls dropped despite PDB terminationGracePeriodSeconds too short; preStop sleep eats the entire grace period Set terminationGracePeriodSeconds > preStop duration + p99 tool call latency + 10s buffer
Probes Restart loop during database blip Same endpoint for readiness and liveness; dependency failure triggers liveness restart Separate /healthz/ready (checks dependencies) and /healthz/live (checks only process state)
Probes Pods restart during traffic spikes Liveness probe too aggressive (short timeoutSeconds); high CPU slows response past timeout Set timeoutSeconds: 10 and failureThreshold: 3–5 on liveness; keep liveness handler trivially cheap
HPA <unknown>/65% in kubectl get hpa; no scaling resource.requests.cpu not set on the container Add resources.requests.cpu to every container; redeploy
HPA HPA never scales despite high queue depth prometheus-adapter misconfigured; metric not visible at custom.metrics.k8s.io kubectl get --raw the custom metrics API endpoint; check adapter logs and ServiceMonitor labels
NetworkPolicy Policy applied but all traffic still flows Default EKS VPC CNI not enforcing (no network policy controller enabled) Enable Amazon VPC CNI network policy controller or install Calico/Cilium; verify with active test
NetworkPolicy All DNS resolution fails after applying policy Missing DNS egress rule — port 53 UDP/TCP to kube-system blocked Apply allow-dns-egress NetworkPolicy first; test with kubectl exec -- nslookup kubernetes.default
NetworkPolicy AWS SDK calls (S3, Secrets Manager) fail after egress deny No VPC endpoint; traffic leaves VPC to AWS public IPs; blocked by egress policy Create VPC Interface Endpoints for each AWS service; traffic stays within VPC on private IPs

The monitoring layer: AliveMCP and EKS

All five of these primitives operate within the cluster — they handle what Kubernetes can handle. But there's a class of failure they can't see: the probe endpoint itself returning 200 when the MCP protocol is broken, the server accepting connections but returning malformed JSON-RPC responses, or a dependency that passes a shallow health check but returns corrupt data.

AliveMCP approaches this from outside the cluster: it probes the actual MCP protocol endpoint over HTTPS, validates JSON-RPC response structure, and tracks schema drift across server versions. This is complementary to the Kubernetes health layer, not a replacement for it — the cluster-internal probes handle the Kubernetes lifecycle (traffic gating, restart decisions, autoscaling signals), while external probing validates that the MCP server is actually fulfilling protocol contracts.

If you are deploying an MCP server on EKS, you want both: internal probes wired correctly per this guide, and an external probe verifying that what the cluster considers "healthy" is also what the MCP protocol considers "healthy." Those two things are not always the same.

Further reading