Readiness and Liveness Probes for MCP Servers on EKS

The three probe types and what each one does

Kubernetes supports three probe types, each with a distinct action on failure:

• readinessProbe — gates traffic routing. When the probe fails, kubelet removes the pod from the Service's endpoint list. The container is not restarted. Existing connections continue until they close. The pod becomes ready again automatically once the probe passes again.
• livenessProbe — restart trigger. When the probe fails failureThreshold consecutive times, kubelet kills and restarts the container. Use only for unrecoverable states — not for transient dependency failures.
• startupProbe — delays liveness and readiness checking during slow container starts. Both liveness and readiness probes are disabled until the startup probe succeeds. If the startup probe itself fails failureThreshold times, the container is killed.

There are three probe mechanisms — the same mechanism can be used for any of the three probe types:

• httpGet — kubelet makes an HTTP GET to the specified path and port; success is any 2xx or 3xx response code.
• tcpSocket — kubelet attempts a TCP connection to the port; success means the connection was accepted.
• exec — kubelet runs a command inside the container; success is exit code 0.

For MCP servers, httpGet is almost always the right choice: it exercises the actual HTTP server stack, and you can return structured JSON with dependency statuses that are easy to debug.

Why readiness and liveness must use different endpoints

The failure semantics are completely different and must be encoded in different endpoints:

• Readiness endpoint (/healthz/ready) should return a non-2xx code whenever the server cannot safely handle new requests: the database connection pool is exhausted, a required secret has expired, a required downstream service is unreachable, or the server is still warming up its in-memory index. These are transient conditions — the server is not broken, it just should not receive new traffic right now.
• Liveness endpoint (/healthz/live) should return a non-2xx code only when the process is in a state it cannot recover from on its own: the event loop is deadlocked, memory is exhausted and the garbage collector is not keeping up, or a critical background thread has panicked and left the server unable to process any request regardless of how much time passes. These are the cases where a restart is the right answer.

If you use the same endpoint for both and a database goes offline for 30 seconds, the liveness probe fails, the container restarts, and now you have compounded the incident: the pod missed the traffic it would otherwise have shed gracefully, its in-flight requests were killed, and it is now in the back-off restart loop — all because of a 30-second dependency blip that did not require a restart to resolve.

A slow or unavailable dependency should fail readiness but not liveness. A deadlocked event loop should fail liveness. These two facts require two separate endpoints.

Complete Deployment YAML with all three probes

The following Deployment configures a startup probe for slow initialization, separate readiness and liveness probes, and named container ports for clean probe references:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: mcp-server
  namespace: prod
spec:
  replicas: 3
  selector:
    matchLabels:
      app: mcp-server
  template:
    metadata:
      labels:
        app: mcp-server
    spec:
      terminationGracePeriodSeconds: 60
      containers:
        - name: mcp-server
          image: your-registry/mcp-server:latest
          ports:
            - name: http        # name the port; probes reference it by name
              containerPort: 3000
              protocol: TCP

          # startupProbe: disables readiness+liveness until this passes.
          # failureThreshold * periodSeconds = max startup time budget.
          # Here: 30 * 5s = 150s = 2.5 minutes to finish initialization.
          startupProbe:
            httpGet:
              path: /healthz/ready
              port: http
            initialDelaySeconds: 5
            periodSeconds: 5
            timeoutSeconds: 3
            failureThreshold: 30
            successThreshold: 1

          # readinessProbe: remove from Service endpoints on failure.
          # Does NOT restart the container.
          # Checks DB, secrets, warm-up. Returns 503 if any dependency is unhealthy.
          readinessProbe:
            httpGet:
              path: /healthz/ready
              port: http
            initialDelaySeconds: 0   # startupProbe gates this; no additional delay needed
            periodSeconds: 10
            timeoutSeconds: 5
            failureThreshold: 3      # 30s of failures before removing from endpoints
            successThreshold: 1

          # livenessProbe: restart the container ONLY on unrecoverable failure.
          # Points to a different endpoint that does minimal checking.
          # Give it a generous timeout so high CPU doesn't trigger false restarts.
          livenessProbe:
            httpGet:
              path: /healthz/live
              port: http
            initialDelaySeconds: 0   # startupProbe gates this too
            periodSeconds: 15
            timeoutSeconds: 10
            failureThreshold: 3      # 45s of failures before restart
            successThreshold: 1

          resources:
            requests:
              cpu: "250m"
              memory: "256Mi"
            limits:
              cpu: "1000m"
              memory: "512Mi"

Key decisions in this configuration:

• Both liveness and readiness set initialDelaySeconds: 0 because the startupProbe already handles the delay. Setting initialDelaySeconds on both startup and readiness probes doubles the wait time unnecessarily.
• The liveness probe uses a longer timeoutSeconds (10s vs 5s) and longer periodSeconds (15s vs 10s) than readiness. Under CPU pressure, probe responses slow down; an aggressive liveness probe turns high load into a restart loop.
• Both probes reference the port by name (port: http) rather than by number. If the container port changes, you update one place.

Node.js health endpoint implementation

Here is a production-grade implementation of both endpoints in Node.js. The readiness check exercises real dependency connections; the liveness check only inspects process-level state:

// health.js — mount this as a router in your Express/Fastify/Hono MCP server

// Shared state that other parts of the server update
let isReady = false;          // set to true after full initialization
let isLive = true;            // set to false only on unrecoverable error
let dbPool = null;            // set by the DB init code
let secretsLoaded = false;    // set by the secrets init code

// Called during server startup after all async init is complete
export function markReady(pool) {
  dbPool = pool;
  secretsLoaded = true;
  isReady = true;
}

// Called if the event loop watchdog detects a deadlock or OOM
export function markDead(reason) {
  console.error('Server marked dead:', reason);
  isLive = false;
}

// GET /healthz/ready
// Returns 200 with JSON status when all dependencies are healthy.
// Returns 503 when any dependency is unhealthy (transient — no restart).
export async function readinessHandler(req, res) {
  const checks = {};
  let healthy = true;

  // 1. Basic readiness flag (covers slow startup)
  checks.initialized = isReady;
  if (!isReady) healthy = false;

  // 2. Database connectivity — lightweight query, not a full round-trip
  if (dbPool) {
    try {
      const client = await Promise.race([
        dbPool.connect(),
        new Promise((_, reject) =>
          setTimeout(() => reject(new Error('timeout')), 2000)
        )
      ]);
      await client.query('SELECT 1');
      client.release();
      checks.database = 'ok';
    } catch (err) {
      checks.database = err.message;
      healthy = false;
    }
  } else {
    checks.database = 'not_initialized';
    healthy = false;
  }

  // 3. Secrets availability
  checks.secrets = secretsLoaded ? 'ok' : 'not_loaded';
  if (!secretsLoaded) healthy = false;

  const status = healthy ? 200 : 503;
  res.status(status).json({ status: healthy ? 'ready' : 'not_ready', checks });
}

// GET /healthz/live
// Returns 200 unless the process is in an unrecoverable state.
// Keep this extremely cheap — it runs every 15 seconds under load.
// Do NOT check external dependencies here.
export function livenessHandler(req, res) {
  if (!isLive) {
    // Process is wedged — restart will help
    res.status(500).json({ status: 'dead' });
    return;
  }

  // Optional: check event loop lag as a proxy for deadlock
  const start = Date.now();
  setImmediate(() => {
    const lag = Date.now() - start;
    if (lag > 5000) {
      // Event loop is blocked — this is an unrecoverable state
      res.status(500).json({ status: 'event_loop_blocked', lag_ms: lag });
    } else {
      res.status(200).json({ status: 'live', lag_ms: lag });
    }
  });
}

Mount these handlers before any authentication middleware — kubelet does not send auth headers when it probes. If your MCP server requires auth on all routes, exempt /healthz/* explicitly.

The readiness handler uses a 2-second timeout on the DB connection attempt. This is intentionally shorter than the probe's timeoutSeconds: 5 — it ensures the endpoint itself returns within the probe's window even if the DB is completely unreachable, rather than hanging and timing out at the kubelet level (which counts as a failure anyway, but produces less useful logs).

Startup probe tuning for slow-starting MCP servers

Some MCP servers load large in-memory indexes, pre-warm embedding caches, or perform migrations on start. These can take 30–120 seconds. Without a startupProbe, you have two bad options: set initialDelaySeconds high enough to cover the worst case (delaying detection of real startup failures), or keep it low and accept restart loops during normal slow starts.

The startupProbe solves this cleanly. Configure failureThreshold * periodSeconds to equal your maximum acceptable startup time:

Once the startupProbe succeeds once, it is disabled and the liveness and readiness probes take over. The startup probe does not run again unless the container restarts.

For MCP servers that fetch configuration from AWS Secrets Manager or Parameter Store at startup, account for the AWS API call latency (typically 100–500ms but can spike to 2–3 seconds on cold Lambda-adjacent environments). Budget at least 5 seconds of startup time per Secrets Manager call, plus your connection pool creation time.

Probe parameter tuning reference

The product failureThreshold * periodSeconds is the most important thing to reason about. For readiness, it is how long a dependency outage must persist before traffic is shed. For liveness, it is how long the pod must appear unrecoverable before a restart is triggered. Err on the side of longer liveness windows — an unnecessary restart during a traffic spike is almost always worse than waiting an extra 30 seconds.

Failure modes reference