Guide · MCP API Gateway Integration

MCP Server Envoy — Admin API cluster health, config dump, forced drain, and /ready monitoring

Envoy is the data-plane proxy powering Istio, AWS App Mesh, and many custom service mesh implementations. Its admin API (port 9901) exposes granular cluster health, per-endpoint statistics, live xDS configuration state, and manual health override commands that are essential for graceful deployment. This guide covers connecting to the Envoy admin API from TypeScript, building list_clusters, get_config_dump, get_stats, and force_health_drain tools, understanding the /ready vs cluster-level health distinction, and integrating the right endpoint with AliveMCP uptime monitoring.

TL;DR

Envoy's admin API runs on port 9901 by default — never expose it publicly. Use GET /ready as the liveness probe (returns 200 when Envoy has loaded config and is serving traffic), GET /clusters?format=json for per-endpoint health (look at health_flags: /healthy means healthy, /failed_active_hc means the active health check failed), and GET /config_dump for full xDS state. POST /healthcheck/fail marks Envoy as unhealthy without stopping the process — use it for graceful preStop drain in Kubernetes. Monitor /ready with AliveMCP; alert separately on cluster degradation via /clusters.

Admin API setup and security

Envoy's admin API is configured in the bootstrap YAML under admin.address. It has no built-in authentication — access control must be enforced at the network layer (localhost-only binding or firewall rules). Exposing the admin API publicly allows anyone to drain your proxy, read all xDS state, and send arbitrary runtime modifications.

# envoy.yaml — bootstrap configuration
admin:
  address:
    socket_address:
      address: 127.0.0.1   # Bind to localhost ONLY — never 0.0.0.0 in production
      port_value: 9901

# For Kubernetes sidecars, access via port-forward:
# kubectl port-forward pod/my-pod 9901:9901
# or use the Downward API to get the pod IP and call from within the cluster
// TypeScript client for Envoy admin API
const ENVOY_ADMIN_URL = process.env.ENVOY_ADMIN_URL ?? 'http://localhost:9901';

async function envoyRequest(path: string, options: RequestInit = {}): Promise<string> {
  const res = await fetch(`${ENVOY_ADMIN_URL}${path}`, options);

  if (!res.ok) {
    throw new Error(`Envoy admin ${options.method ?? 'GET'} ${path} → ${res.status}`);
  }

  return res.text();
}

async function envoyJsonRequest(path: string): Promise<any> {
  const text = await envoyRequest(path);
  try {
    return JSON.parse(text);
  } catch {
    throw new Error(`Envoy admin ${path} returned non-JSON: ${text.slice(0, 200)}`);
  }
}

export { envoyRequest, envoyJsonRequest };
Admin endpoint Method What it does Safe to call frequently?
/ready GET Returns 200 when Envoy is ready to serve traffic Yes — lightweight
/clusters?format=json GET Per-cluster per-endpoint health and stats Yes — ~1s/call is fine
/listeners?format=json GET All listener configurations Yes
/config_dump GET Full xDS state (can be MB in size) No — expensive; call on demand
/stats/prometheus GET All Prometheus metrics Sparingly — large payload
/healthcheck/fail POST Force Envoy to report itself as unhealthy Only on deploy drain
/healthcheck/ok POST Clear forced failure, return to healthy After restart/recovery
/drain_listeners POST Drain all listeners (start graceful shutdown) Only on pod termination

Core Envoy MCP tool patterns

import { z } from 'zod';
import { McpError, ErrorCode } from '@modelcontextprotocol/sdk/types.js';
import { envoyRequest, envoyJsonRequest } from './envoy-client.js';

// ---- get_ready_status ----
server.tool(
  'get_ready_status',
  {},
  async () => {
    try {
      const text = await envoyRequest('/ready');
      return { content: [{ type: 'text', text: JSON.stringify({ ready: true, message: text.trim() }) }] };
    } catch {
      return { content: [{ type: 'text', text: JSON.stringify({ ready: false }) }] };
    }
  }
);

// ---- list_clusters ----
// Returns per-endpoint health flags and connection stats
server.tool(
  'list_clusters',
  {
    show_unhealthy_only: z.boolean().default(false),
    cluster_name_filter: z.string().optional()
  },
  async ({ show_unhealthy_only, cluster_name_filter }) => {
    const data = await envoyJsonRequest('/clusters?format=json');

    // cluster_statuses is the array of clusters in JSON format
    let clusters = (data.cluster_statuses ?? []) as any[];

    if (cluster_name_filter) {
      clusters = clusters.filter(c =>
        c.name?.toLowerCase().includes(cluster_name_filter.toLowerCase())
      );
    }

    const result = clusters.map((cluster: any) => {
      const hostStatuses = cluster.host_statuses ?? [];

      const hosts = hostStatuses.map((h: any) => ({
        address: h.address?.socket_address
          ? `${h.address.socket_address.address}:${h.address.socket_address.port_value}`
          : 'unknown',
        health_flags: h.health_status?.failed_active_health_check
          ? '/failed_active_hc'
          : h.health_status?.failed_outlier_check
          ? '/failed_outlier_check'
          : '/healthy',
        healthy: !h.health_status?.failed_active_health_check && !h.health_status?.failed_outlier_check,
        weight: h.weight,
        // Stats: successful, total, error counts
        rq_success: h.stats?.find((s: any) => s.name === 'rq_success')?.value ?? 0,
        rq_total: h.stats?.find((s: any) => s.name === 'rq_total')?.value ?? 0,
        rq_timeout: h.stats?.find((s: any) => s.name === 'rq_timeout')?.value ?? 0,
        cx_active: h.stats?.find((s: any) => s.name === 'cx_active')?.value ?? 0
      }));

      return {
        name: cluster.name,
        is_passthrough: cluster.circuit_breakers === undefined,
        total_hosts: hosts.length,
        healthy_hosts: hosts.filter(h => h.healthy).length,
        unhealthy_hosts: hosts.filter(h => !h.healthy).length,
        hosts: show_unhealthy_only ? hosts.filter(h => !h.healthy) : hosts
      };
    });

    const filtered = show_unhealthy_only
      ? result.filter(c => c.unhealthy_hosts > 0)
      : result;

    return { content: [{ type: 'text', text: JSON.stringify(filtered, null, 2) }] };
  }
);

// ---- get_config_dump ----
// Full xDS state — expensive, call on demand only
server.tool(
  'get_config_dump',
  {
    section: z.enum(['clusters', 'listeners', 'routes', 'endpoints', 'secrets', 'all']).default('clusters')
  },
  async ({ section }) => {
    const path = section === 'all'
      ? '/config_dump'
      : `/config_dump?resource=${section}`;

    const data = await envoyJsonRequest(path);

    // Config dump returns { configs: [ { @type: ..., ... }, ... ] }
    const configs = data.configs ?? [];

    const summary = configs.map((c: any) => ({
      type: c['@type']?.split('/').pop() ?? 'unknown',
      count: c.dynamic_active_clusters?.length
        ?? c.dynamic_listeners?.length
        ?? c.dynamic_route_configs?.length
        ?? c.dynamic_endpoint_configs?.length
        ?? '(see full dump)'
    }));

    return {
      content: [{
        type: 'text',
        text: JSON.stringify({ section, config_sections: summary, full_dump: section === 'all' ? data : undefined }, null, 2)
      }]
    };
  }
);

// ---- get_stats ----
// Returns specific named stats (avoids returning the full prometheus dump)
server.tool(
  'get_stats',
  {
    filter: z.string().describe('Stat name prefix to filter, e.g. "cluster.my-api" or "http.ingress"')
  },
  async ({ filter }) => {
    const text = await envoyRequest(`/stats?filter=${encodeURIComponent(filter)}&format=json`);
    const data = JSON.parse(text);

    return {
      content: [{
        type: 'text',
        text: JSON.stringify({
          filter,
          stats: data.stats ?? []
        }, null, 2)
      }]
    };
  }
);

// ---- force_health_drain ----
// Marks Envoy as unhealthy to drain upstream load balancers before pod termination
server.tool(
  'force_health_drain',
  {
    reason: z.string().describe('Why this instance is being drained (for logging)'),
    confirm: z.literal(true)
  },
  async ({ reason }) => {
    // POST /healthcheck/fail — causes Envoy to return 503 on its health check endpoint
    // Does NOT stop accepting traffic immediately — use /drain_listeners for that
    await envoyRequest('/healthcheck/fail', { method: 'POST' });

    return {
      content: [{
        type: 'text',
        text: JSON.stringify({
          drained: true,
          reason,
          note: 'Envoy will now return 503 on health checks. Upstream load balancers will stop routing new requests within their next health check interval. Use /drain_listeners to start full graceful shutdown.'
        })
      }]
    };
  }
);

// ---- restore_health ----
server.tool(
  'restore_health',
  { confirm: z.literal(true) },
  async () => {
    await envoyRequest('/healthcheck/ok', { method: 'POST' });
    return { content: [{ type: 'text', text: JSON.stringify({ restored: true }) }] };
  }
);

Understanding health_flags in cluster status

Envoy uses a health flags bitmask to represent the health state of each endpoint. The JSON format from /clusters?format=json exposes individual boolean fields per flag. Understanding which flag caused an endpoint to be marked unhealthy is essential for debugging.

health_status field Flag string Meaning Common cause
failed_active_health_check /failed_active_hc Active health check returned non-2xx or timed out Backend /health endpoint returning 503 or not responding
failed_outlier_check /failed_outlier_check Outlier detection ejected this host Too many consecutive 5xx responses from this endpoint
failed_active_degraded_check /degraded_failed_active_hc Active health check returned "degraded" response Health check returned 2xx but endpoint is degraded
pending_dynamic_removal /pending_dynamic_removal Endpoint was removed from EDS but still draining xDS control plane removed this endpoint
eds_health_status == UNHEALTHY /eds_status_unhealthy EDS (control plane) explicitly marked endpoint unhealthy Control plane health check at the xDS level

The most common flags in practice are failed_active_hc (backend health check failed) and failed_outlier_check (too many errors from real traffic). If you see failed_outlier_check, check cluster.<name>.outlier_detection.ejections_active in the stats to see how many hosts are currently ejected and ejections_total for the total ejection count since startup.

Health endpoint: /health/envoy

import express from 'express';
import { envoyRequest, envoyJsonRequest } from './envoy-client.js';

const app = express();

app.get('/health/envoy', async (_req, res) => {
  const start = Date.now();
  try {
    // Layer 1: Envoy process readiness
    const readyRes = await fetch(`${process.env.ENVOY_ADMIN_URL ?? 'http://localhost:9901'}/ready`);
    const isReady = readyRes.status === 200;
    const readyText = await readyRes.text();

    if (!isReady) {
      return res.status(503).json({
        status: 'not_ready',
        message: readyText.trim(),
        elapsed_ms: Date.now() - start
      });
    }

    // Layer 2: Cluster health summary
    const data = await envoyJsonRequest('/clusters?format=json');
    const clusters = (data.cluster_statuses ?? []) as any[];

    const clusterSummary = clusters.map(c => {
      const hosts = c.host_statuses ?? [];
      const unhealthy = hosts.filter((h: any) =>
        h.health_status?.failed_active_health_check || h.health_status?.failed_outlier_check
      );
      return {
        name: c.name,
        total: hosts.length,
        unhealthy: unhealthy.length,
        healthy: hosts.length - unhealthy.length
      };
    });

    const degradedClusters = clusterSummary.filter(c => c.unhealthy > 0);
    const latencyMs = Date.now() - start;

    res.status(degradedClusters.length > 0 ? 503 : 200).json({
      status: degradedClusters.length > 0 ? 'degraded' : 'ok',
      envoy_ready: isReady,
      total_clusters: clusters.length,
      degraded_clusters: degradedClusters.length,
      cluster_detail: degradedClusters,
      latency_ms: latencyMs
    });
  } catch (err: any) {
    res.status(503).json({
      status: 'error',
      error: err.message,
      elapsed_ms: Date.now() - start
    });
  }
});

app.listen(3001);
Failure mode /ready response /clusters response Detected by /health/envoy
Envoy not yet initialized 503 ENVOY_PRE_INITIALIZING Empty Yes — isReady false
Envoy waiting for xDS config 503 ENVOY_INITIALIZING Empty or partial Yes — isReady false
All backends in a cluster down 200 LIVE Hosts show failed_active_hc Yes — degradedClusters > 0
Outlier detection ejected all hosts 200 LIVE Hosts show failed_outlier_check Yes — degradedClusters > 0
POST /healthcheck/fail was called 503 ENVOY_PRE_INITIALIZING (or custom status) Clusters still healthy Yes — via /ready check

Graceful drain pattern for Kubernetes pods

The correct Kubernetes preStop sequence for an Envoy sidecar (or standalone Envoy pod) is: (1) mark Envoy unhealthy via POST /healthcheck/fail, (2) wait for upstream load balancers to drain, (3) call POST /drain_listeners to stop accepting new connections, (4) wait for active connections to finish. This prevents the 502 burst that happens when Kubernetes sends SIGTERM and immediately removes the pod from the Endpoints object.

// MCP tool for initiating graceful Envoy shutdown
server.tool(
  'initiate_graceful_shutdown',
  {
    wait_seconds: z.number().int().min(0).max(120).default(10)
      .describe('Seconds to wait after /healthcheck/fail before draining listeners'),
    confirm: z.literal(true)
  },
  async ({ wait_seconds }) => {
    // Step 1: Mark unhealthy — stops new requests from upstream LBs
    await envoyRequest('/healthcheck/fail', { method: 'POST' });

    // Step 2: Wait for LBs to detect unhealthy state
    await new Promise(resolve => setTimeout(resolve, wait_seconds * 1000));

    // Step 3: Drain listeners — starts graceful connection draining
    await envoyRequest('/drain_listeners', { method: 'POST' });

    return {
      content: [{
        type: 'text',
        text: JSON.stringify({
          shutdown_initiated: true,
          health_fail_called: true,
          drain_listeners_called: true,
          wait_seconds,
          note: `Waited ${wait_seconds}s between health fail and listener drain. Active connections will continue until they complete or timeout.`
        })
      }]
    };
  }
);

// Kubernetes preStop hook equivalent (shell):
// lifecycle:
//   preStop:
//     exec:
//       command: ["/bin/sh", "-c", "curl -s -X POST http://localhost:9901/healthcheck/fail; sleep 10; curl -s -X POST http://localhost:9901/drain_listeners"]

Frequently asked questions

What's the difference between /ready and /live in Envoy?

In Envoy 1.26+, GET /ready is the canonical readiness probe. It returns 200 with the text LIVE when Envoy has initialized and is ready to serve traffic. Prior to this, some operators used GET /server_info to check the state field — that still works but /ready is simpler and more standard. There is no separate /live endpoint in Envoy (unlike some other systems). For Kubernetes liveness/readiness probes, use /ready for both — a non-ready Envoy that's crashed is also not live, and Kubernetes will kill it on failed liveness checks.

How do I debug a cluster where outlier detection keeps ejecting all hosts?

When all hosts are ejected, Envoy uses panic mode — it will send traffic to all ejected hosts (the panic fallback). Check: (1) cluster.<name>.outlier_detection.ejections_active via /stats?filter=cluster.<name>.outlier_detection to see current ejection count; (2) ejections_consecutive_5xx to understand if consecutive 5xxs are the trigger; (3) ejections_enforced_total shows actual ejections vs detected (enforcement_percentage < 100% means not all detections result in ejection). If you see panic mode, check cluster.<name>.membership_healthy vs membership_total — when healthy/total drops below the panic threshold (50% by default), Envoy ignores health status and routes to all hosts to prevent complete outage.

How do I access the Envoy admin API inside a Kubernetes cluster?

Use kubectl port-forward for ad-hoc access: kubectl port-forward pod/<pod-name> 9901:9901. For MCP tools running inside the cluster, access the admin API via the pod's localhost using the Downward API to inject the pod IP: set MY_POD_IP via fieldRef: fieldPath: status.podIP, then connect to http://$(MY_POD_IP):9901. Do not use the pod's hostname or service name — the admin port is typically not included in any Kubernetes Service. For Istio sidecar admin access from within the same pod, use http://localhost:9901 directly since the sidecar runs in the same network namespace.

Can I use MCP tools to update Envoy's xDS config dynamically?

Not directly via the admin API — the admin API is read-only for xDS configuration. To update routes, clusters, or listeners dynamically, you need to push changes through the xDS control plane: Istio Pilot (via VirtualService/DestinationRule CRDs), a custom xDS server (using Go Control Plane or Java xDS), or static file config with a reload (send SIGUSR1 for hot reload of file config). Your MCP tool would interact with the control plane's API (e.g., Kubernetes API for Istio CRDs) rather than Envoy's admin API directly. The admin API's /config_dump is useful for verifying that xDS changes from the control plane have been applied to a specific Envoy instance.

Further reading

Know when Envoy's clusters go unhealthy before traffic degrades

AliveMCP polls your /health/envoy endpoint every 60 seconds and alerts you the moment any cluster shows failed health checks or outlier ejections — before your SLO breach window starts accumulating.

Start monitoring free