Deep Dive · AWS EKS
Running MCP Servers on EKS: The Production Playbook — Availability, Autoscaling, and Isolation
Most MCP server teams land on EKS and reach for the obvious knobs: set a replica count, configure a readiness probe, call it done. What they get is a deployment that looks healthy in a calm cluster but sheds requests during node group upgrades, restarts pods when the database hiccups, fails to scale before users see errors, and runs in a flat network where a single compromised pod can reach everything else. Each of those failure modes has a different root cause and a different Kubernetes primitive that fixes it. This post synthesizes five of them — topology spread constraints, PodDisruptionBudgets, readiness and liveness probes, Horizontal Pod Autoscaler with custom metrics, and NetworkPolicies — and shows how they compose into a single coherent production posture.
The failure taxonomy: five classes, five primitives
Before wiring up any YAML, it helps to think about what can actually go wrong. MCP server failures on EKS fall into five classes, and the important thing is that they are mostly independent — each class requires a different primitive to address, and the primitives don't overlap much:
| Failure class | Example trigger | Kubernetes primitive | What it does |
|---|---|---|---|
| Correlated placement failure | All 3 replicas on the same AZ node that goes down | topologySpreadConstraints + podAntiAffinity | Ensures replicas are distributed across failure domains before the failure happens |
| Voluntary simultaneous eviction | kubectl drain, node group upgrade, cluster autoscaler scale-down | PodDisruptionBudget | Forces the Eviction API to drain pods one at a time; blocks drain if it would violate the budget |
| Traffic sent to unready/unhealthy pod | Dependency goes offline, pod still in Service endpoints; restart loop from shared probe endpoint | readinessProbe + livenessProbe + startupProbe | readinessProbe gates traffic without restarting; livenessProbe restarts only on unrecoverable failure |
| Capacity doesn't match load | Bursty tool-call traffic; CPU stays flat while queue depth grows | HPA with custom metrics (or KEDA) | Adds pods on the right signal; avoids the CPU-miss problem for I/O-bound MCP servers |
| Lateral movement after compromise | Injected LLM tool output reaches a malicious host; pod sends unauthorized traffic to other namespaces | NetworkPolicy with default-deny | Restricts ingress and egress to explicitly allowed pods, namespaces, and ports |
You need all five. Topology spread without a PDB means replicas are distributed but can still be evicted simultaneously during a node upgrade. A PDB without topology spread means the eviction serialization works, but if you have two replicas on two nodes in the same AZ, an AZ outage takes both. A readiness probe without topology spread means the pod drains correctly but if all replicas live on one node, a hardware failure takes everything. And so on — each primitive plugs a different gap. The rest of this guide walks through each layer and ends with a reference YAML combining them.
Layer 1: Topology spread — prevent correlated placement failures before they happen
The scheduler's default behavior is bin-packing: it places pods wherever resources are available. For a three-replica MCP server deployment, the default policy will happily put all three on the same node or in the same AZ if that's where resources are. One node failure or AZ outage and you're at zero available pods — not because of a bug in your code, but because no placement constraint prevented it.
topologySpreadConstraints expresses this as a scheduling rule: "keep the difference in pod count across topology domains below maxSkew." Two constraints — one for AZ-level spread and one for node-level spread within each AZ — cover the most important failure domains:
# topology spread configuration for a 3+ replica MCP server
spec:
topologySpreadConstraints:
# AZ-level: no AZ should have more than 1 extra pod vs the least-loaded AZ
- maxSkew: 1
topologyKey: topology.kubernetes.io/zone
whenUnsatisfiable: DoNotSchedule
labelSelector:
matchLabels:
app: mcp-server
# Node-level: soft preference to spread across nodes within each AZ
- maxSkew: 1
topologyKey: kubernetes.io/hostname
whenUnsatisfiable: ScheduleAnyway # soft — don't block if nodes are unequal
labelSelector:
matchLabels:
app: mcp-server
The DoNotSchedule on the AZ-level constraint means new pods will not schedule if they'd violate the spread. Use this for AZ spread — you want it to fail loudly if the cluster can't honor it, rather than silently packing pods into one AZ. Use ScheduleAnyway for node-level spread so that the deployment can still scale beyond the node count when needed.
For explicit node separation of replicas, add podAntiAffinity alongside the spread constraint — it keeps any two replicas off the same node entirely:
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
podAffinityTerm:
labelSelector:
matchExpressions:
- key: app
operator: In
values:
- mcp-server
topologyKey: kubernetes.io/hostname
Use preferredDuringScheduling (soft) here rather than requiredDuringScheduling (hard). The hard variant creates a scheduling deadlock if you ever need to scale beyond the node count: the 4th pod has no node to schedule on because every existing node already hosts a replica. The soft variant expresses the preference without creating an un-schedulable state.
Topology spread does not protect against voluntary eviction — that's what the PDB is for. But it is the prerequisite: if all replicas are co-located, the PDB's guarantee of "at least 1 available during voluntary disruption" is meaningless because a hardware failure (which the PDB doesn't cover) would still wipe everything simultaneously.
Layer 2: PodDisruptionBudgets — serialize voluntary evictions
EKS routine operations — node group upgrades, AMI updates, cluster autoscaler scale-down, even a manual kubectl drain — all go through the Kubernetes Eviction API. Every eviction request checks the relevant PodDisruptionBudgets before proceeding. If an eviction would push the available pod count below spec.minAvailable, the API returns HTTP 429 and the draining operation retries later.
A PDB with minAvailable: 1 on a two-replica deployment means the drain can evict one pod, but must wait for a replacement to become Ready before evicting the second:
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: mcp-server-pdb
namespace: mcp
spec:
minAvailable: 1
selector:
matchLabels:
app: mcp-server # must match .spec.template.metadata.labels exactly
The selector is the most common source of silent PDB misconfiguration. If spec.selector.matchLabels doesn't exactly match the pod labels on your Deployment, the PDB governs zero pods and the drain ignores it entirely. Check with:
kubectl get pdb mcp-server-pdb -n mcp -o wide
# ALLOWED-DISRUPTIONS column should show a non-zero value when replicas are healthy
# If it shows N/A, the selector is wrong
Beyond the PDB object itself, the other half of graceful shutdown is terminationGracePeriodSeconds and the preStop lifecycle hook on the pod spec. The PDB controls when eviction is permitted; these control how cleanly the pod exits once eviction is allowed.
The shutdown sequence: kubelet executes the preStop hook, then sends SIGTERM to the container, then waits until terminationGracePeriodSeconds expires (counting from preStop start, not SIGTERM), then sends SIGKILL. For MCP servers behind an ALB or NLB, the preStop hook should signal the load balancer to deregister the target and wait for the deregistration delay before the server stops accepting connections:
# pod spec — add alongside the container image/ports/resources
terminationGracePeriodSeconds: 90
containers:
- name: mcp-server
# ...
lifecycle:
preStop:
exec:
command:
- /bin/sh
- -c
- |
# Signal app to stop accepting new tool calls
kill -SIGUSR1 1
# Wait for ALB deregistration (default 30s deregistration delay)
sleep 35
The math: terminationGracePeriodSeconds must exceed preStop duration (35 s in the example) plus the p99 latency of your slowest tool call. If your LLM-calling tools can take up to 45 seconds, set terminationGracePeriodSeconds: 90 (35 s deregistration + 45 s worst case + 10 s buffer). Setting the grace period equal to the preStop sleep leaves zero time for in-flight requests to complete before SIGKILL fires.
One interaction to understand: the PDB and Deployment rollingUpdate settings operate independently but can compound. If a node drain fires during a rolling update, and the rollout has already taken one pod down (within maxUnavailable), the PDB may block the drain from evicting the remaining pod because that would drop below minAvailable. The safest combination is maxUnavailable: 0 with maxSurge: 1 on the Deployment — rollouts overprovision rather than underprovision, so the PDB budget is never crowded out by a simultaneous rollout.
Layer 3: Probe architecture — gate traffic without causing restart loops
The single most common MCP server incident on EKS comes from one configuration mistake: using the same /health endpoint for both the readiness and liveness probes. When a database goes offline for 30 seconds, the shared endpoint returns 503, the liveness probe counts failures, and kubelet restarts the container — which is exactly the wrong response. The pod didn't need a restart; it needed to be removed from the load balancer until the database came back. The restart compounded the incident: in-flight tool calls were killed, the pod went through its cold start cycle, and the remaining replicas had to absorb the traffic spike.
The fix is two separate endpoints with two fundamentally different semantics:
/healthz/ready— returns 503 when the server cannot safely handle new requests: database connection pool exhausted, required secret expired, dependency unreachable, or initialization incomplete. These are transient conditions. No restart. Traffic shed./healthz/live— returns 500 only when the process is in a state it cannot recover from: event loop deadlocked, memory exhausted beyond recovery, critical background thread panicked permanently. These warrant a restart. The bar is high.
Complete probe configuration for an MCP server:
# Complete probe block — add to spec.template.spec.containers[0]
startupProbe:
httpGet:
path: /healthz/ready
port: http
initialDelaySeconds: 5
periodSeconds: 5
timeoutSeconds: 3
failureThreshold: 30 # 150s startup budget — covers slow Secrets Manager init
readinessProbe:
httpGet:
path: /healthz/ready
port: http
initialDelaySeconds: 0 # startupProbe already handles the delay
periodSeconds: 10
timeoutSeconds: 5
failureThreshold: 3 # 30s before traffic shed
livenessProbe:
httpGet:
path: /healthz/live
port: http
initialDelaySeconds: 0
periodSeconds: 15
timeoutSeconds: 10 # longer timeout — must survive CPU pressure
failureThreshold: 3 # 45s before restart
The startupProbe is critical for MCP servers that initialize slowly — those that fetch configuration from AWS Secrets Manager, establish connection pools to multiple backends, or load large in-memory indexes. Without it, you have two bad options: set initialDelaySeconds high enough to cover every slow start (which delays detection of real startup failures), or accept restart loops during normal slow starts. The startupProbe disables both readiness and liveness until it passes once, then steps out of the way.
The Node.js implementation of these two endpoints should express the semantics cleanly. The readiness handler exercises actual dependencies; the liveness handler checks only process-level state:
// health.js — minimal implementation for the two endpoints
let isReady = false; // set true after full async init
let isLive = true; // set false only on unrecoverable error
let dbPool = null;
export function markReady(pool) {
dbPool = pool;
isReady = true;
}
// GET /healthz/ready
// Checks actual dependency health — slow dependency = 503, not restart
export async function readinessHandler(req, res) {
if (!isReady) {
return res.status(503).json({ status: 'not_ready', reason: 'initializing' });
}
try {
const client = await Promise.race([
dbPool.connect(),
new Promise((_, reject) =>
setTimeout(() => reject(new Error('timeout')), 2000)
)
]);
await client.query('SELECT 1');
client.release();
res.status(200).json({ status: 'ready' });
} catch (err) {
res.status(503).json({ status: 'not_ready', reason: err.message });
}
}
// GET /healthz/live
// Only fails on unrecoverable process state — never checks dependencies
export function livenessHandler(req, res) {
if (!isLive) {
return res.status(500).json({ status: 'dead' });
}
// Optional: check event loop lag as a deadlock proxy
const start = Date.now();
setImmediate(() => {
const lag = Date.now() - start;
if (lag > 5000) {
res.status(500).json({ status: 'event_loop_blocked', lag_ms: lag });
} else {
res.status(200).json({ status: 'live', lag_ms: lag });
}
});
}
Always mount these routes before any authentication middleware. Kubelet does not send auth headers when probing — if auth middleware runs first, every probe returns 401 and the pod never becomes Ready.
The I/O-bound signal problem: why CPU misleads the HPA for MCP servers
Before configuring the HPA, it is worth understanding why the default CPU-based autoscaling is wrong for most MCP workloads. A typical MCP tool call spends its latency budget waiting: waiting for an LLM API response (1–30 seconds), waiting for a database query (5–500ms), waiting for an external HTTP call. During all of that waiting, the pod's CPU utilization is near zero. If the HPA is configured to scale at 65% CPU utilization, it sees healthy numbers while 50 tool calls are queued and your users are staring at timeouts.
The right signal for I/O-bound MCP servers is one of:
- In-flight concurrent tool calls (a gauge your server increments at each tool call start and decrements at finish)
- Tool-call queue depth (if you buffer requests before routing to handlers)
- ALB request count per target (external metric from CloudWatch via KEDA)
CPU still belongs in the HPA as a guard for genuinely compute-heavy tools (code execution, image processing, heavy JSON transformation). Use a multi-metric HPA and let Kubernetes scale when any metric crosses its threshold — the system scales on the most-stressed signal, not the most-common one.
There is one prerequisite that blocks all CPU-based HPA: resource.requests.cpu must be set on every container in the pod spec. Without it, metrics-server cannot compute utilization and the HPA reports <unknown>/65% indefinitely. The fix is always just adding the resources block — but because it's easy to miss on new deployments, it's worth calling out explicitly.
Layer 4: HPA with custom metrics — autoscaling on the right signal
The custom metric HPA requires three components: metrics-server (already installed in EKS), Prometheus (scrapes your MCP server's application metric), and prometheus-adapter (bridges Prometheus into the Kubernetes custom metrics API).
First, instrument the server. A gauge counting in-flight tool calls is the most reliable signal and the easiest to implement:
// Prometheus instrumentation for Node.js MCP server
import { Gauge, register } from 'prom-client';
import express from 'express';
const inFlightToolCalls = new Gauge({
name: 'mcp_tool_call_queue_depth',
help: 'Number of tool calls currently in-flight or queued'
});
// Wrap your tool call handler
export async function handleToolCall(toolName, params) {
inFlightToolCalls.inc();
try {
return await dispatchTool(toolName, params);
} finally {
inFlightToolCalls.dec();
}
}
// Expose /metrics on a separate port so NetworkPolicy rules can target it specifically
const metricsApp = express();
metricsApp.get('/metrics', async (req, res) => {
res.set('Content-Type', register.contentType);
res.end(await register.metrics());
});
metricsApp.listen(9090);
Once the metric is exposed, configure the prometheus-adapter to map it into the custom metrics API:
# prometheus-adapter values.yaml (Helm)
rules:
custom:
- seriesQuery: 'mcp_tool_call_queue_depth{namespace!="",pod!=""}'
resources:
overrides:
namespace: { resource: namespace }
pod: { resource: pod }
name:
matches: "mcp_tool_call_queue_depth"
as: "mcp_tool_call_queue_depth"
metricsQuery: 'avg(<<.Series>>{<<.LabelMatchers>>})'
Verify the metric is visible in the Kubernetes API before wiring the HPA:
kubectl get --raw \
"/apis/custom.metrics.k8s.io/v1beta1/namespaces/mcp/pods/*/mcp_tool_call_queue_depth" \
| jq .
If this returns empty results, check: (1) the prometheus-adapter is running and its rules config matches, (2) the ServiceMonitor label matches the Prometheus operator's serviceMonitorSelector, (3) the metric is actually being exported by the running pods (kubectl exec -- curl localhost:9090/metrics | grep mcp_tool).
Now the multi-metric HPA combining CPU (as a guard) and the custom tool-call depth (as the primary signal):
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: mcp-server-hpa
namespace: mcp
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: mcp-server
minReplicas: 2
maxReplicas: 20
metrics:
# Primary: scale on tool-call queue depth — the right signal for I/O-bound work
- type: Pods
pods:
metric:
name: mcp_tool_call_queue_depth
target:
type: AverageValue
averageValue: "5" # scale out when avg queue depth exceeds 5 per pod
# Guard: CPU — catches compute-heavy tool spikes
- type: Resource
resource:
name: cpu
target:
type: AverageUtilization
averageUtilization: 65
behavior:
scaleUp:
stabilizationWindowSeconds: 0 # scale up immediately
policies:
- type: Pods
value: 4
periodSeconds: 60
- type: Percent
value: 100
periodSeconds: 60
selectPolicy: Max # use whichever adds more pods
scaleDown:
stabilizationWindowSeconds: 180 # wait 3 min of sustained low load
policies:
- type: Pods
value: 2
periodSeconds: 60
selectPolicy: Min
The scale-down stabilizationWindowSeconds: 180 matters for MCP servers with slow cold starts. If your server takes 60 seconds to pass its readiness probe, a load spike that causes HPA to scale out and then immediately back down would create a cycle of provisioning and terminating pods. The stabilization window prevents scale-down during transient load drops, giving the autoscaler time to observe that load has genuinely dropped before removing capacity.
For workloads that need scale-to-zero, use KEDA instead of the built-in HPA. KEDA creates its own HPA internally — do not also create a separate HPA targeting the same Deployment, or you will get conflicting scale decisions. The KEDA ScaledObject replaces the HPA entirely.
Layer 5: NetworkPolicies — network isolation and blast radius containment
The single most dangerous fact about Kubernetes NetworkPolicies on EKS is this: you can write them, apply them, and watch kubectl get networkpolicy return them — while every pod in your cluster still freely communicates with every other pod. The default EKS VPC CNI does not enforce NetworkPolicies. The policy objects are stored in etcd; enforcement depends on the CNI plugin.
You need one of:
- Amazon VPC CNI with network policy controller — enable
ENABLE_NETWORK_POLICY_CONTROLLER=trueon theaws-nodeDaemonSet (requires add-on version v1.14+). Native AWS support, uses eBPF, no additional components to manage. - Calico in policy-only mode — runs alongside VPC CNI, handles only policy enforcement, does not replace VPC IP management. Mature, well-documented, adds GlobalNetworkPolicy CRD.
- Cilium — replaces or chains with VPC CNI; eBPF-native with L7 HTTP and DNS filtering, best observability via Hubble. Higher operational complexity.
Always verify enforcement is active by applying a test deny policy, confirming traffic is actually blocked, then removing it. Don't rely on the policy existing in the API server as evidence of enforcement.
The pattern that works reliably: default-deny, then add allowlists. Apply a deny-all policy first (to a test namespace), prove your allowlists work, then roll it out to production. This order prevents the "apply default-deny to a live namespace and break everything" failure mode.
Three policies that every MCP server namespace needs:
# 1. Default deny — applied LAST, after allowlists are in place
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: default-deny-all
namespace: mcp
spec:
podSelector: {} # matches every pod in the namespace
policyTypes:
- Ingress
- Egress
# No rules = deny everything
---
# 2. DNS egress — MUST NOT be omitted or all hostname resolution breaks silently
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-dns-egress
namespace: mcp
spec:
podSelector: {}
policyTypes:
- Egress
egress:
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: kube-system
ports:
- protocol: UDP
port: 53
- protocol: TCP
port: 53 # TCP fallback for large DNS responses
---
# 3. MCP server allowlist: ingress from API gateway + monitoring, egress to data layer + HTTPS
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: mcp-server-allowlist
namespace: mcp
spec:
podSelector:
matchLabels:
app: mcp-server
policyTypes:
- Ingress
- Egress
ingress:
- from:
- namespaceSelector:
matchLabels:
role: api-gateway
ports:
- protocol: TCP
port: 8080
- from:
- namespaceSelector:
matchLabels:
role: monitoring
ports:
- protocol: TCP
port: 9090 # Prometheus metrics scraping
egress:
# External LLM APIs and AWS services (use VPC endpoints to avoid CIDR management)
- ports:
- protocol: TCP
port: 443
# Internal data layer (database, cache)
- to:
- namespaceSelector:
matchLabels:
role: data-layer
ports:
- protocol: TCP
port: 5432 # PostgreSQL
- protocol: TCP
port: 6379 # Redis
The DNS egress rule deserves its own emphasis. When you apply any egress NetworkPolicy to a pod, all egress traffic not explicitly listed is blocked — including DNS queries to kube-dns. Without the DNS rule, every hostname lookup silently fails. Your MCP server can't connect to its database, its LLM provider, or any Kubernetes service by name. The symptom looks like connection refused to everything simultaneously — a confusing failure that takes a long time to diagnose if you don't know to look for missing DNS egress.
One subtle semantics issue: when namespaceSelector and podSelector appear inside the same from list entry, they are ANDed — the pod must be in a matching namespace AND have matching pod labels. If you put them in separate from list items, they are ORed. This distinction creates security bugs when you intend "pods in namespace X with label Y" but accidentally write two separate entries that instead mean "pods in namespace X OR pods with label Y anywhere".
For AWS SDK calls (Secrets Manager, S3, STS, SSM), the cleanest egress pattern is VPC Interface Endpoints: traffic stays within the VPC and uses private IP addresses, so your existing internal CIDR egress rules cover it. Without VPC endpoints, your MCP server needs egress to AWS public IP ranges, which are large, change periodically, and are annoying to maintain as CIDR lists.
Composing all five layers: the full Deployment
Each of the five primitives addresses a different failure class. Here they are in a single Deployment spec:
apiVersion: apps/v1
kind: Deployment
metadata:
name: mcp-server
namespace: mcp
spec:
replicas: 3
selector:
matchLabels:
app: mcp-server
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
maxUnavailable: 0 # overprovision rather than underprovision during rollout
template:
metadata:
labels:
app: mcp-server
spec:
# Layer 1: Topology spread — distribute across AZs and nodes
topologySpreadConstraints:
- maxSkew: 1
topologyKey: topology.kubernetes.io/zone
whenUnsatisfiable: DoNotSchedule
labelSelector:
matchLabels:
app: mcp-server
- maxSkew: 1
topologyKey: kubernetes.io/hostname
whenUnsatisfiable: ScheduleAnyway
labelSelector:
matchLabels:
app: mcp-server
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
podAffinityTerm:
labelSelector:
matchExpressions:
- key: app
operator: In
values:
- mcp-server
topologyKey: kubernetes.io/hostname
# Layer 2: Graceful shutdown — works with the PDB to drain cleanly
terminationGracePeriodSeconds: 90
containers:
- name: mcp-server
image: your-ecr-repo/mcp-server:latest
ports:
- name: http
containerPort: 8080
- name: metrics
containerPort: 9090
# Resources are required for HPA CPU metric
resources:
requests:
cpu: "250m"
memory: "512Mi"
limits:
cpu: "1000m"
memory: "1Gi"
# Layer 2: preStop — deregister from load balancer before stopping
lifecycle:
preStop:
exec:
command: ["/bin/sh", "-c", "kill -SIGUSR1 1; sleep 35"]
# Layer 3: Probe architecture — three separate probes with distinct semantics
startupProbe:
httpGet:
path: /healthz/ready
port: http
initialDelaySeconds: 5
periodSeconds: 5
timeoutSeconds: 3
failureThreshold: 30
readinessProbe:
httpGet:
path: /healthz/ready
port: http
periodSeconds: 10
timeoutSeconds: 5
failureThreshold: 3
livenessProbe:
httpGet:
path: /healthz/live
port: http
periodSeconds: 15
timeoutSeconds: 10
failureThreshold: 3
Plus the supporting objects:
---
# Layer 2: PodDisruptionBudget — serialize voluntary evictions
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: mcp-server-pdb
namespace: mcp
spec:
minAvailable: 1
selector:
matchLabels:
app: mcp-server
---
# Layer 4: HPA — custom metric primary, CPU guard secondary
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: mcp-server-hpa
namespace: mcp
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: mcp-server
minReplicas: 2
maxReplicas: 20
metrics:
- type: Pods
pods:
metric:
name: mcp_tool_call_queue_depth
target:
type: AverageValue
averageValue: "5"
- type: Resource
resource:
name: cpu
target:
type: AverageUtilization
averageUtilization: 65
behavior:
scaleUp:
stabilizationWindowSeconds: 0
policies:
- type: Percent
value: 100
periodSeconds: 60
selectPolicy: Max
scaleDown:
stabilizationWindowSeconds: 180
policies:
- type: Pods
value: 2
periodSeconds: 60
selectPolicy: Min
---
# Layer 5: NetworkPolicy — default deny + DNS + allowlist
# (see the three NetworkPolicy objects in the previous section)
The interaction matrix: what happens when two failure modes arrive simultaneously
Real incidents rarely involve a single failure class. Here are the most important interaction cases:
| Simultaneous events | Without both primitives | With both primitives |
|---|---|---|
| Node drain + rolling deployment | If rollout already has one pod down, drain evicts the remaining pod and the PDB blocks — upgrade stalls until rollout completes | maxUnavailable: 0 on Deployment means rollout never takes a pod down; drain always finds minAvailable satisfied; no stall |
| Database blip + HPA scale-down | readinessProbe on shared endpoint triggers liveness restart; scale-down simultaneously terminates another pod; cascading restart loop | Separate readiness probe fails silently (no restart); pod removed from endpoints; HPA scale-down stabilization window holds off scale-down during the blip |
| AZ outage + load spike | Replicas co-located in failed AZ → total outage; HPA tries to scale but requests fail immediately | Topology spread means 1/3 of replicas survive in other AZs; HPA scales them out (topology spread on new pods targets surviving AZs); degraded but available |
| Spot interruption + high load | Spot node evicts all pods on it with NoExecute; SIGKILL to all in-flight tool calls simultaneously | PDB limits simultaneous evictions; preStop hook drains gracefully within terminationGracePeriodSeconds; topology spread means not all pods on spot nodes |
| Cluster autoscaler scale-down + LLM API latency spike | CA evicts pod while it has 30 in-flight 20s tool calls; all return connection reset | PDB blocks CA eviction if available count would drop below minAvailable; tool calls on the other pods complete normally; CA retries after stabilization |
Consolidated failure modes reference
A single table covering the most common misconfiguration in each layer:
| Layer | Symptom | Root cause | Fix |
|---|---|---|---|
| Topology spread | topologySpreadConstraints causes Pending pods | DoNotSchedule on hostname with low replica count, or one AZ missing nodes | Use ScheduleAnyway on hostname; keep DoNotSchedule only on zone; ensure all AZs have nodes |
| Topology spread | Can't scale Deployment beyond 3 replicas | Hard pod anti-affinity on kubernetes.io/hostname with only 3 nodes | Switch to preferredDuringScheduling anti-affinity; use topologySpreadConstraints instead |
| PDB | PDB exists but ALLOWED-DISRUPTIONS is N/A; drain ignores it | spec.selector.matchLabels doesn't match pod labels | kubectl get pods --show-labels and compare; fix the label mismatch |
| PDB | In-flight tool calls dropped despite PDB | terminationGracePeriodSeconds too short; preStop sleep eats the entire grace period | Set terminationGracePeriodSeconds > preStop duration + p99 tool call latency + 10s buffer |
| Probes | Restart loop during database blip | Same endpoint for readiness and liveness; dependency failure triggers liveness restart | Separate /healthz/ready (checks dependencies) and /healthz/live (checks only process state) |
| Probes | Pods restart during traffic spikes | Liveness probe too aggressive (short timeoutSeconds); high CPU slows response past timeout | Set timeoutSeconds: 10 and failureThreshold: 3–5 on liveness; keep liveness handler trivially cheap |
| HPA | <unknown>/65% in kubectl get hpa; no scaling | resource.requests.cpu not set on the container | Add resources.requests.cpu to every container; redeploy |
| HPA | HPA never scales despite high queue depth | prometheus-adapter misconfigured; metric not visible at custom.metrics.k8s.io | kubectl get --raw the custom metrics API endpoint; check adapter logs and ServiceMonitor labels |
| NetworkPolicy | Policy applied but all traffic still flows | Default EKS VPC CNI not enforcing (no network policy controller enabled) | Enable Amazon VPC CNI network policy controller or install Calico/Cilium; verify with active test |
| NetworkPolicy | All DNS resolution fails after applying policy | Missing DNS egress rule — port 53 UDP/TCP to kube-system blocked | Apply allow-dns-egress NetworkPolicy first; test with kubectl exec -- nslookup kubernetes.default |
| NetworkPolicy | AWS SDK calls (S3, Secrets Manager) fail after egress deny | No VPC endpoint; traffic leaves VPC to AWS public IPs; blocked by egress policy | Create VPC Interface Endpoints for each AWS service; traffic stays within VPC on private IPs |
The monitoring layer: AliveMCP and EKS
All five of these primitives operate within the cluster — they handle what Kubernetes can handle. But there's a class of failure they can't see: the probe endpoint itself returning 200 when the MCP protocol is broken, the server accepting connections but returning malformed JSON-RPC responses, or a dependency that passes a shallow health check but returns corrupt data.
AliveMCP approaches this from outside the cluster: it probes the actual MCP protocol endpoint over HTTPS, validates JSON-RPC response structure, and tracks schema drift across server versions. This is complementary to the Kubernetes health layer, not a replacement for it — the cluster-internal probes handle the Kubernetes lifecycle (traffic gating, restart decisions, autoscaling signals), while external probing validates that the MCP server is actually fulfilling protocol contracts.
If you are deploying an MCP server on EKS, you want both: internal probes wired correctly per this guide, and an external probe verifying that what the cluster considers "healthy" is also what the MCP protocol considers "healthy." Those two things are not always the same.
Further reading
- Node affinity and topology spread for MCP servers on EKS — full reference with taint/toleration patterns and dedicated node pool setup
- PodDisruptionBudgets for MCP servers on EKS — complete PDB and graceful shutdown reference including preStop hook patterns
- Readiness and liveness probes for MCP servers on EKS — probe parameter tuning table and full Node.js health endpoint implementation
- Horizontal Pod Autoscaler for MCP servers on EKS — KEDA integration, custom metrics API verification, and scale-down behavior configuration
- Network policies for MCP servers on EKS — multi-tenant namespace isolation and CNI enforcement verification
- MCP server health check reference — MCP protocol-level health checking beyond Kubernetes probes