MCP Tools for API Gateways and Service Meshes: Process Health vs Backend Target Health, Authentication Model Diversity, and the Write-Path Health Trap Across Kong, Traefik, Envoy, APISIX, and Istio

Pattern 1: Process health vs backend target health

Every gateway in this arc maintains two separate health states: the health of the gateway process itself, and the health of the backend services it is proxying traffic to. These are always different endpoints, and conflating them is the most common mistake in gateway health probe design. A gateway that is fully alive and processing traffic can be forwarding every request to dead backends — its own process health endpoint will return 200 while every proxied request fails.

Kong: two-layer health with different scopes

Kong's GET /health on the Admin API port 8001 reports the health of the Kong node process itself — whether the Kong Go/Lua runtime is alive, whether the database connection is active (in DB mode), and whether the node is reachable. This endpoint says nothing about whether any upstream service behind Kong is alive.

Kong's upstream (backend target) health lives at GET /upstreams/:name/health, which returns a per-target health status for each IP:port in the upstream pool. The status values carry important semantic distinctions:

The critical operational note: HEALTHCHECKS_OFF means Kong is passively tracking failures based on real traffic — unhealthy targets are only ejected if passive health checks are enabled. An upstream with all targets showing HEALTHCHECKS_OFF can still be routing all traffic to dead backends if passive checks are also disabled. A correctly implemented Kong MCP health tool checks for DNS_ERROR, TIMEOUT, and UNHEALTHY across all targets in all upstreams, not just the node-level GET /health.

There is also an important active health check timing asymmetry: Kong's active health checks use separate healthchecks.active.healthy.interval and healthchecks.active.unhealthy.interval values. Setting the unhealthy interval lower than the healthy interval (e.g., unhealthy: 5s vs healthy: 30s) means Kong probes a degraded target more aggressively than a healthy one — recovering faster from transient outages. MCP tools that inspect upstream configuration should surface this interval asymmetry when it is not configured.

// Kong composite health check — node process AND upstream targets
async function checkKongHealth(upstreamName: string): Promise<{
  status: 'healthy' | 'degraded' | 'error';
  node: object;
  targets: Array<{ address: string; health: string }>;
}> {
  const [nodeHealth, upstreamHealth] = await Promise.all([
    kongRequest('/health'),
    kongRequest(`/upstreams/${upstreamName}/health`)
  ]);

  const targets = (upstreamHealth.data ?? []).map((t: any) => ({
    address: t.target.target,
    health: t.health
  }));

  const degradedTargets = targets.filter(
    (t: any) => t.health === 'UNHEALTHY' || t.health === 'DNS_ERROR' || t.health === 'TIMEOUT'
  );

  return {
    status: degradedTargets.length > 0 ? 'degraded' : 'healthy',
    node: nodeHealth,
    targets
  };
}

Traefik: /ping says nothing about backends

Traefik's GET /ping endpoint returns 200 as long as the Traefik process is running and able to receive requests. It explicitly does not check whether any backend service behind Traefik is alive — this is by design. /ping is a Kubernetes liveness probe target; it is there so the container orchestrator knows whether to restart the Traefik process, not whether traffic is flowing successfully.

Backend service health in Traefik lives in the serverStatus map within GET /api/http/services. Each service object contains a serverStatus field that maps the backend server URL to its current health state (UP or DOWN). But there is a prerequisite that is easy to miss: serverStatus is only populated when loadBalancer.healthCheck is configured in the dynamic configuration for that service. If health checking is not configured, serverStatus will be absent or empty — not because all servers are UP, but because Traefik is not checking them at all.

// Traefik composite health check — /ping AND serverStatus
async function checkTraefikHealth(): Promise<{
  status: 'healthy' | 'degraded' | 'error';
  processAlive: boolean;
  downServers: string[];
  uncheckedServices: string[];
}> {
  const [pingRes, services] = await Promise.all([
    fetch(`${TRAEFIK_URL}/ping`),
    traefikRequest('/api/http/services') as Promise<any[]>
  ]);

  const downServers: string[] = [];
  const uncheckedServices: string[] = [];

  for (const svc of services) {
    const statusMap: Record<string, string> = svc.serverStatus ?? {};
    const entries = Object.entries(statusMap);

    if (entries.length === 0) {
      // No serverStatus — health checking not configured
      uncheckedServices.push(svc.name);
      continue;
    }

    for (const [url, health] of entries) {
      if (health === 'DOWN') {
        downServers.push(`${svc.name}: ${url}`);
      }
    }
  }

  return {
    status: downServers.length > 0 ? 'degraded' : 'healthy',
    processAlive: pingRes.ok,
    downServers,
    uncheckedServices  // Services with no health check configured — caller should alert
  };
}

The uncheckedServices list is as operationally important as downServers: if a service has no health check configured in Traefik, Traefik will happily route traffic to dead backends indefinitely. An MCP tool that only alerts on DOWN entries misses the configuration gap entirely.

Envoy: /ready vs cluster health_flags

Envoy's GET /ready endpoint returns 200 LIVE once Envoy has loaded its initial xDS configuration and is ready to serve traffic. During startup, it returns 503 with ENVOY_PRE_INITIALIZING (bootstrap not yet loaded) or ENVOY_INITIALIZING (waiting for initial xDS response). After /ready returns 200, Envoy is a running process — but this says nothing about the health of the backend clusters it is routing to.

Cluster (backend) health in Envoy lives in GET /clusters?format=json, which returns per-host health information inside each cluster's host_statuses array. Each host entry has a health_flags field — a set of flags describing why a host is considered unhealthy:

// Envoy composite health check — /ready AND cluster health_flags
async function checkEnvoyHealth(): Promise<{
  status: 'healthy' | 'degraded' | 'error';
  ready: boolean;
  unhealthyHosts: Array<{ cluster: string; address: string; flags: string }>;
}> {
  const readyText = await envoyRequest('/ready');
  const clusters = await envoyJsonRequest('/clusters?format=json');

  const unhealthyHosts: Array<{ cluster: string; address: string; flags: string }> = [];

  for (const clusterStatus of clusters.cluster_statuses ?? []) {
    for (const host of clusterStatus.host_statuses ?? []) {
      const flags: string = host.health_status?.eds_health_status ?? '';
      const failedActive = host.health_status?.failed_active_health_check ?? false;
      const failedOutlier = host.health_status?.failed_outlier_check ?? false;

      if (failedActive || failedOutlier || flags === 'UNHEALTHY') {
        const addr = host.address?.socket_address;
        unhealthyHosts.push({
          cluster: clusterStatus.name,
          address: addr ? `${addr.address}:${addr.port_value}` : 'unknown',
          flags: [
            failedActive ? 'failed_active_hc' : null,
            failedOutlier ? 'failed_outlier_check' : null,
            flags === 'UNHEALTHY' ? 'eds_unhealthy' : null
          ].filter(Boolean).join(', ')
        });
      }
    }
  }

  return {
    status: unhealthyHosts.length > 0 ? 'degraded' : 'healthy',
    ready: readyText.includes('LIVE'),
    unhealthyHosts
  };
}

There is also a separate concept of Envoy's panic mode: when the fraction of healthy hosts in a cluster drops below 50%, Envoy ignores health status and load-balances across all hosts anyway (to avoid sending all traffic to a single remaining healthy instance). This means cluster health state in /clusters does not directly map to whether traffic is being sent to those unhealthy hosts — in panic mode, it is. The panic threshold is configurable per cluster; an MCP tool monitoring production Envoy deployments should surface both the healthy/total ratio and whether any cluster is likely in panic mode.

APISIX: data plane vs Control API health check

APISIX exposes two health surfaces on two separate ports. Port 9080 is the data plane — if it responds to HTTP requests, APISIX is serving traffic. Port 9090 is the Control API, which exposes GET /v1/healthcheck. This endpoint shows the active health check status for each upstream target — whether APISIX's periodic probes have marked upstream hosts as healthy or sick.

Crucially, active health check results (from probes APISIX initiates against upstreams) only appear in /v1/healthcheck. Passive health checks — where APISIX observes real request failures to decide a target is bad — do not appear in this API at all. So /v1/healthcheck shows only what APISIX is actively probing; it cannot tell you about passively detected degradation. An MCP tool probing APISIX health should check both the Control API active health check results and use its own judgment about routes that have no active health check configured.

Istio: istiod readiness vs proxy xDS sync state

Istio has the most layered health model of the five platforms because it separates the control plane (istiod) from the data plane (Envoy sidecars). GET /healthz/ready on istiod port 15014 tells you whether the Istio control plane process itself is ready — whether it has loaded CRDs, connected to the Kubernetes API server, and is ready to serve xDS. This endpoint says nothing about whether the sidecars across your cluster have received the current configuration.

Sidecar sync state lives in the Pilot debug API at GET /debug/syncz on port 15014. This endpoint returns a list of all connected Envoy proxies and their xDS state: how many resources were sent (cluster_sent, route_sent, etc.) and how many were acknowledged (cluster_acked, route_acked). A divergence between sent and acked counts means a sidecar has stale config:

// Istio composite health check — /healthz/ready AND /debug/syncz
async function checkIstioHealth(): Promise<{
  status: 'healthy' | 'degraded' | 'error';
  istiodReady: boolean;
  outOfSyncProxies: Array<{ proxyId: string; clusterLag: number; routeLag: number }>;
}> {
  const istiodBaseUrl = process.env.ISTIOD_URL ?? 'http://istiod.istio-system:15014';

  const [readyRes, syncData] = await Promise.all([
    fetch(`${istiodBaseUrl}/healthz/ready`),
    fetch(`${istiodBaseUrl}/debug/syncz`).then(r => r.json())
  ]);

  const outOfSyncProxies = (syncData ?? [])
    .filter((proxy: any) => {
      const clusterLag = (proxy.cluster_sent ?? 0) - (proxy.cluster_acked ?? 0);
      const routeLag = (proxy.route_sent ?? 0) - (proxy.route_acked ?? 0);
      return clusterLag > 0 || routeLag > 0;
    })
    .map((proxy: any) => ({
      proxyId: proxy.proxy,
      clusterLag: (proxy.cluster_sent ?? 0) - (proxy.cluster_acked ?? 0),
      routeLag: (proxy.route_sent ?? 0) - (proxy.route_acked ?? 0)
    }));

  return {
    status: outOfSyncProxies.length > 0 ? 'degraded' : 'healthy',
    istiodReady: readyRes.ok,
    outOfSyncProxies
  };
}

A composable multi-gateway health endpoint using Promise.allSettled() aggregates all five platform checks into a single URL that AliveMCP can poll at 60-second intervals, surfacing both process-level and backend-level failures in one response:

// Composite health endpoint — register this URL with AliveMCP
app.get('/health/gateways', async (req, res) => {
  const [kong, traefik, envoy, apisix, istio] = await Promise.allSettled([
    checkKongHealth(process.env.KONG_UPSTREAM_NAME ?? 'backend'),
    checkTraefikHealth(),
    checkEnvoyHealth(),
    checkApisixHealth(),
    checkIstioHealth()
  ]);

  const results = { kong, traefik, envoy, apisix, istio };
  const anyDegraded = Object.values(results).some(
    r => r.status === 'fulfilled' && r.value.status !== 'healthy'
  );
  const anyError = Object.values(results).some(r => r.status === 'rejected');

  res.status(anyError ? 503 : anyDegraded ? 207 : 200).json(results);
});

Pattern 2: Authentication model diversity

The five platforms use five structurally different approaches to securing access to their admin or control-plane APIs. This diversity is not cosmetic — each model implies a different credential lifecycle, different rotation procedure, and different failure mode when credentials expire or are revoked.

Kong: OSS vs Enterprise auth divergence

Kong OSS and Kong Enterprise (including Konnect) handle admin API authentication completely differently, and the difference is not always obvious from the client code — if you add a Kong-Admin-Token header to a request against a Kong OSS instance, the header is silently ignored. The OSS instance accepts the request without checking the header value, which means a client written for Enterprise will appear to work against an OSS instance even with an invalid token. This silent success masks the fact that there is no authentication enforced at all on that endpoint.

The correct approach in a shared codebase:

// Kong client — detects mode from environment, enforces intent
const KONG_ADMIN_TOKEN = process.env.KONG_ADMIN_TOKEN;
const KONG_EXPECT_AUTH = process.env.KONG_EXPECT_AUTH === 'true';

async function kongRequest(path: string, options: RequestInit = {}): Promise<any> {
  if (KONG_EXPECT_AUTH && !KONG_ADMIN_TOKEN) {
    throw new Error('KONG_EXPECT_AUTH is set but KONG_ADMIN_TOKEN is missing');
  }

  const headers: Record<string, string> = {
    'Content-Type': 'application/json',
    ...(options.headers as Record<string, string> ?? {})
  };

  if (KONG_ADMIN_TOKEN) {
    headers['Kong-Admin-Token'] = KONG_ADMIN_TOKEN;
  }

  const res = await fetch(`${KONG_ADMIN_URL}${path}`, { ...options, headers });

  if (!res.ok) {
    throw new Error(`Kong Admin ${options.method ?? 'GET'} ${path} → ${res.status}`);
  }

  return res.json();
}

The KONG_EXPECT_AUTH environment variable is the key guard: it forces the operator to declare intent at deployment time rather than silently falling back to unauthenticated access when the Enterprise token is missing.

Traefik: auth is always delegated

Traefik has no built-in auth on its own dashboard/API — this is a deliberate design choice. In development, --api.insecure=true exposes the API on port 8080 with no authentication. In production, you route the Traefik dashboard through an IngressRoute that applies a BasicAuth or ForwardAuth middleware. This means the auth layer is outside Traefik's own code, configured in Traefik's dynamic config, and enforced by a middleware before requests reach /api.

For an MCP server connecting to Traefik's API, this means the connection pattern depends on deployment context. In Kubernetes, the MCP server typically runs in-cluster and accesses the Traefik dashboard service directly (bypassing the IngressRoute that protects external access). In bare-metal deployments, the MCP server needs to be on the same network as the Traefik API port. There is no Traefik-specific auth credential to rotate — rotation is a middleware concern.

Envoy: localhost-only is not optional

Envoy's admin API has no authentication mechanism. The security model is entirely network-based: bind the admin API socket to 127.0.0.1:9901, never to 0.0.0.0. Exposing Envoy's admin API on a public interface allows anyone to call POST /healthcheck/fail (immediately removes the pod from load balancing), POST /drain_listeners (graceful shutdown of all traffic), and GET /config_dump (full xDS state including all cluster credentials and routing rules).

In Kubernetes sidecar deployments, an MCP tool running in the same pod can access the admin API directly at http://localhost:9901. For cross-pod inspection, use kubectl port-forward — never add a Kubernetes Service that exposes port 9901 cluster-wide, because that exposes the admin API to any pod in the cluster:

# Correct: port-forward from your MCP development environment
kubectl port-forward pod/my-envoy-pod 9901:9901

# Correct: in-pod access (MCP server co-deployed with Envoy sidecar)
ENVOY_ADMIN_URL=http://localhost:9901

# WRONG: never create a Service exposing the admin port cluster-wide
# (This gives any pod in the cluster access to drain/fail your proxy)

APISIX: X-API-KEY is not Bearer and not optional

APISIX uses X-API-KEY as its header name for Admin API authentication. This is not a standard Authorization: Bearer header — a client that mistakenly sends Authorization: Bearer <key> to APISIX's Admin API will receive a 401 even with the correct key value. The APISIX Admin API key has a role property — admin grants full read/write access, viewer grants read-only access. MCP tools that only need to inspect routing state should use a viewer key to limit the blast radius if the key is compromised.

The default APISIX Admin API key (edd1c9f034335f136f87ad84b625c8f1) is well-known and must be replaced before any network exposure. Unlike Kong Enterprise's token rotation API, APISIX key rotation requires editing config.yaml and restarting the APISIX process — there is no hot-reload for key changes in the static configuration section. Build this into your rotation runbook.

Istio: Kubernetes RBAC is the auth layer

Istio has no separate authentication layer — access to Istio configuration is access to Kubernetes CRDs (VirtualService, DestinationRule, PeerAuthentication, etc.) and the Pilot debug API. The @kubernetes/client-node library handles credential loading transparently: loadFromCluster() inside the cluster uses the pod's ServiceAccount token at /var/run/secrets/kubernetes.io/serviceaccount/token; loadFromDefault() outside the cluster uses ~/.kube/config.

RBAC for an Istio MCP server should be scoped to the minimum necessary operations on each resource type. A read-only MCP tool inspecting routing state only needs:

# Minimum RBAC for a read-only Istio MCP tool
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: istio-mcp-reader
rules:
  - apiGroups: ["networking.istio.io"]
    resources: ["virtualservices", "destinationrules", "gateways", "serviceentries"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["security.istio.io"]
    resources: ["peerauthentications", "authorizationpolicies"]
    verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: istio-mcp-reader-binding
subjects:
  - kind: ServiceAccount
    name: istio-mcp-server
    namespace: default
roleRef:
  kind: ClusterRole
  name: istio-mcp-reader
  apiGroup: rbac.authorization.k8s.io

An MCP tool that also writes Istio config (traffic splits, circuit breakers, mTLS policy) needs patch and update verbs added to the ClusterRole. Kubernetes 1.24+ automatically rotates ServiceAccount tokens (default TTL 1 hour), so there is no manual credential rotation needed — the kubelet refreshes the projected token file in-place, and @kubernetes/client-node's in-cluster credential provider reads the file on each request.

Pattern 3: The write-path health trap

The most dangerous pattern in this arc is what I call the write-path health trap: a gateway that appears healthy from any external health check can have a broken configuration write path. When the write path is broken, configuration changes fail silently — new routes do not take effect, plugin changes are not applied, security policy updates are not enforced. The gateway continues serving traffic from its last good state while appearing healthy, making it invisible to any monitoring that checks only whether the gateway is alive and routing.

APISIX: etcd is the write path

APISIX's write-path health trap is the most severe of the five platforms. APISIX stores all routing configuration in etcd. At startup, it loads the full configuration from etcd into memory. In production, it continuously watches etcd for changes. If etcd becomes unavailable, APISIX continues serving traffic from its in-memory cache — requests succeed, responses come back, health checks pass. But all Admin API write operations fail.

The failure mode is deceptive: POST /apisix/admin/routes to add a new route returns an error, but the data-plane port 9080 continues returning 200 for existing routes. A monitoring system watching only the data plane sees a healthy gateway. An operator who just deployed a new route via the Admin API cannot tell whether the route was written to etcd or whether etcd was down and the write failed — not without checking the Admin API response explicitly and separately checking etcd connectivity.

// APISIX health check — must include etcd, not just data-plane
async function checkApisixHealth(): Promise<{
  status: 'healthy' | 'degraded' | 'error';
  dataPlane: boolean;
  adminApiReachable: boolean;
  etcdConnected: boolean;
  activeHealthChecks: any[];
}> {
  const APISIX_DATA_URL = process.env.APISIX_DATA_URL ?? 'http://localhost:9080';
  const ETCD_URL = process.env.ETCD_URL ?? 'http://localhost:2379';

  const [dataPlane, adminCheck, etcdCheck, healthChecks] = await Promise.allSettled([
    fetch(`${APISIX_DATA_URL}/`),  // Data plane — any 2xx or 4xx means it's up
    fetch(`${APISIX_ADMIN_URL}/apisix/admin/routes?page_size=1`, {
      headers: { 'X-API-KEY': APISIX_API_KEY! }
    }),
    fetch(`${ETCD_URL}/health`),   // etcd health endpoint
    apisixControlRequest('/v1/healthcheck')
  ]);

  const dataPlaneOk = dataPlane.status === 'fulfilled' &&
    (dataPlane.value.status < 500 || dataPlane.value.status === 404);  // 404 is fine
  const adminOk = adminCheck.status === 'fulfilled' && adminCheck.value.ok;
  const etcdOk = etcdCheck.status === 'fulfilled' && etcdCheck.value.ok;

  const status = (!adminOk || !etcdOk) ? 'degraded' : 'healthy';

  return {
    status,
    dataPlane: dataPlaneOk,
    adminApiReachable: adminOk,
    etcdConnected: etcdOk,
    activeHealthChecks: healthChecks.status === 'fulfilled' ? healthChecks.value : []
  };
}

Register this composite health URL with AliveMCP — not the data-plane URL. The data-plane URL will return healthy even when etcd is down and all config writes are failing. The composite endpoint surfaces the etcd dependency separately, so AliveMCP can alert the moment etcd disconnects, before an operator tries to push a config change and wonders why it didn't take effect.

Traefik: the API is entirely read-only for dynamic config

Traefik's write-path trap is different: the API has no write path at all. GET /api/http/routers shows you the current routing state, but there is no POST /api/http/routers to add a new route. Traefik's dynamic configuration is write-only via external mechanisms — the file provider (watching a directory for *.yaml and *.toml files), Docker labels, or Kubernetes IngressRoute CRDs. The API only reflects what Traefik has read from those external sources.

This has two health implications. First, if the file provider directory is on a network mount that becomes unavailable, Traefik continues serving traffic from its last-loaded config, but new config file changes will not be picked up — with no error visible in /api. Second, if a Kubernetes IngressRoute CRD is misconfigured (invalid syntax, reference to a non-existent middleware), Traefik logs an error and skips that route — the route simply does not appear in GET /api/http/routers without any indication in the API response that a route was attempted but rejected.

An MCP tool can surface rejected routes by checking for routers with status: "disabled" or a non-empty error field in the GET /api/http/routers response:

// Surface misconfigured Traefik routers
server.tool('list_router_errors', {}, async () => {
  const routers: any[] = await traefikRequest('/api/http/routers');
  const errored = routers.filter(r => r.status === 'disabled' || r.error);

  if (errored.length === 0) {
    return { content: [{ type: 'text', text: 'No misconfigured routers found.' }] };
  }

  const lines = errored.map(r =>
    `${r.name} (${r.status ?? 'unknown'}): ${r.error ?? 'no error message'}`
  ).join('\n');

  return { content: [{ type: 'text', text: `Misconfigured routers:\n${lines}` }] };
});

Kong: DB mode vs DBless write-path semantics

Kong has two operating modes with completely different write-path semantics. In DB mode, Kong stores all configuration in PostgreSQL. The Admin API is read-write — POST /services and POST /routes write directly to PostgreSQL, and all Kong nodes in a cluster see the change via the database. If PostgreSQL becomes unavailable, existing routes continue to work (cached in memory) but Admin API writes fail with database error responses.

In DBless mode (declarative configuration), Kong loads a kong.yaml file at startup and holds it in memory. The Admin API becomes read-only for configuration — you cannot add routes via POST /services. The only way to change configuration in DBless mode is to update the kong.yaml file and either restart Kong or POST the full configuration to /config (which atomically replaces the entire in-memory config). There is no incremental change API.

An MCP tool should detect the operating mode from the GET / root endpoint response:

// Detect Kong operating mode
async function getKongMode(): Promise<'db' | 'dbless'> {
  const root = await kongRequest('/');
  return root.configuration?.database === 'off' ? 'dbless' : 'db';
}

// Guard write operations for DBless mode
server.tool('add_route', {
  service_name: z.string(),
  path: z.string()
}, async ({ service_name, path }) => {
  const mode = await getKongMode();
  if (mode === 'dbless') {
    return {
      content: [{
        type: 'text',
        text: 'Kong is running in DBless mode. Routes cannot be added via Admin API. Update kong.yaml and POST to /config instead.'
      }],
      isError: true
    };
  }
  // proceed with DB-mode write...
});

Envoy: admin API is entirely observe-and-drain, not configure

Envoy's admin API has the clearest write-path model: there is essentially no general configuration write path. All Envoy configuration comes from its bootstrap YAML (static config) or from an xDS control plane (dynamic config) — neither of which is accessible via the admin API. The only write operations the admin API supports are operational commands: POST /healthcheck/fail, POST /healthcheck/ok, POST /drain_listeners, and POST /runtime_modify (for runtime overrides).

This means the write-path question for Envoy is actually a question about the xDS control plane, not about the admin API. If Envoy's xDS stream to its control plane (Pilot, xDS server, etc.) breaks, Envoy continues serving traffic from its last received xDS state, and the admin API continues returning 200 from /ready. The only way to detect a broken xDS stream from the admin API is to check GET /config_dump and look at the timestamp of the last xDS update — if the dynamic resources have not been updated in an unexpected long time, the xDS stream may be stale.

In Istio-managed deployments, the /debug/syncz endpoint on istiod port 15014 is more reliable than /config_dump for detecting xDS staleness — it shows the per-sidecar sent vs acked counts and lets you identify exactly which proxies have stale configuration.

Istio: CRD writes are eventually consistent

Istio's write path goes through the Kubernetes API server, then through istiod, then to each sidecar via xDS. When you PATCH a VirtualService CRD, the write succeeds at the Kubernetes layer immediately — the API server returns 200. But the change is not yet in effect in the Envoy sidecars. Istiod must read the updated CRD from the Kubernetes watch stream, translate it to xDS configuration, and push it to all connected sidecars. Each sidecar must receive and acknowledge the push. This end-to-end propagation typically takes under a second in healthy clusters with few proxies, but can take tens of seconds in large clusters with many sidecars or under high load.

The practical implication for MCP tools: a set_traffic_split tool that patches a VirtualService and immediately reports success is technically correct about the CRD write, but potentially misleading about the actual traffic state. An honest implementation reports the CRD write status separately from the sidecar acknowledgment status:

// Istio traffic split with write-path transparency
server.tool('set_traffic_split', {
  namespace: z.string(),
  virtual_service_name: z.string(),
  v1_weight: z.number().int().min(0).max(100),
  v2_weight: z.number().int().min(0).max(100)
}, async ({ namespace, virtual_service_name, v1_weight, v2_weight }) => {
  if (v1_weight + v2_weight !== 100) {
    throw new McpError(ErrorCode.InvalidParams, 'Weights must sum to 100');
  }

  // Fetch current spec to avoid overwriting unrelated fields
  const current = await customApi.getNamespacedCustomObject(
    'networking.istio.io', 'v1beta1', namespace, 'virtualservices', virtual_service_name
  );
  const body = current.body as any;
  body.spec.http[0].route[0].weight = v1_weight;
  body.spec.http[0].route[1].weight = v2_weight;

  await customApi.patchNamespacedCustomObject(
    'networking.istio.io', 'v1beta1', namespace, 'virtualservices', virtual_service_name,
    body, undefined, undefined, undefined,
    { headers: { 'Content-Type': 'application/merge-patch+json' } }
  );

  return {
    content: [{
      type: 'text',
      text: [
        `VirtualService ${virtual_service_name} updated: ${v1_weight}% / ${v2_weight}%`,
        `CRD write: COMPLETE`,
        `Sidecar propagation: IN PROGRESS (typically <1s, check /debug/syncz on istiod for large clusters)`,
        `Verify effective traffic state with: get_traffic_split (allow 5s for propagation)`
      ].join('\n')
    }]
  };
});

Pagination and response shape traps

Beyond the three main patterns, each gateway exposes response shape traps that cause MCP tools to miss entries or produce incorrect results.

Kong pagination: Kong Admin API list endpoints return a next field in the response body when there are more items. The value of next is an absolute URL — http://localhost:8001/services?offset=abc123 — not a relative path. A tool that tries to use next as a path directly appended to the base URL will construct an incorrect double-URL. Strip the base URL before reusing:

async function kongListAll(path: string): Promise<any[]> {
  const results: any[] = [];
  let next: string | null = path;

  while (next) {
    // Strip base URL if next is absolute
    const requestPath = next.startsWith('http') ? new URL(next).pathname + new URL(next).search : next;
    const page = await kongRequest(requestPath);
    results.push(...(page.data ?? []));
    next = page.next ? (new URL(page.next).pathname + new URL(page.next).search) : null;
  }

  return results;
}

APISIX response shape: APISIX Admin API list responses wrap items in a { list: [{ key, value }] } structure. Each item in list has a key (the etcd key, e.g., /apisix/routes/00000000000000000001) and a value (the actual route or upstream object). Tools that expect a flat array miss all items:

// APISIX list — extract .value from each item
async function apisixListRoutes(): Promise<any[]> {
  const data = await apisixRequest('/apisix/admin/routes');
  return (data.list ?? []).map((item: any) => item.value);
}

APISIX PATCH semantics: APISIX's Admin API uses PATCH to replace entire nested objects, not merge-patch them. If a route has plugins.rate-limiting configured and you PATCH with a body that includes only plugins.key-auth, the PATCH replaces the entire plugins object — deleting rate-limiting silently. Always fetch-then-merge before patching any object with nested structures:

// APISIX safe plugin update — fetch, merge, then PATCH
async function addPluginToRoute(routeId: string, pluginName: string, pluginConfig: object) {
  const existing = await apisixRequest(`/apisix/admin/routes/${routeId}`);
  const plugins = { ...(existing.value?.plugins ?? {}), [pluginName]: pluginConfig };

  return apisixRequest(`/apisix/admin/routes/${routeId}`, {
    method: 'PATCH',
    body: JSON.stringify({ plugins })
  });
}

Envoy /config_dump cost: GET /config_dump returns Envoy's full xDS state, which can be tens or hundreds of megabytes in a large deployment. It should never be called on a polling interval. Use the resource= query parameter to filter to a specific xDS resource type: /config_dump?resource=ListenersConfigDump or /config_dump?resource=ClustersConfigDump. For targeted stats, use GET /stats?filter=http&format=json with a filter regex rather than retrieving the full stats payload.

Plugin priority and traffic split confirm guards

Two additional concerns apply across multiple gateways in this arc: plugin execution order and the confirm guard for traffic-modifying operations.

Plugin priority (Kong and APISIX)

Both Kong and APISIX execute plugins in priority order, not in the order they were added. In Kong, plugin priority is hardcoded per plugin type (e.g., jwt runs before rate-limiting which runs before request-transformer). In APISIX, plugin priority is a numeric property in the plugin schema — higher numbers run first:

An MCP tool that enables a plugin without surfacing its priority relative to existing plugins can create unexpected behavior: adding ip-restriction at priority 3000 blocks requests before jwt-auth at 2510 runs, which means blocklisted IPs are rejected before the request is authenticated. An operator who expected to block authenticated-but-blocklisted IPs specifically would get a different behavior. A well-designed enable_plugin tool should return the effective execution order after the change, not just confirm the plugin was added.

Confirm guard for traffic-modifying operations

Several operations across these five gateways are immediate, traffic-affecting, and hard to reverse cleanly. These require a confirm guard pattern — a preview/confirm tool pair that shows the effect before applying it:

Summary: the three patterns as a checklist

For every API gateway and service mesh MCP integration:

1. Process health vs backend target health: Never register only the gateway's process health endpoint with external monitoring. Every gateway in this arc has a separate mechanism for backend target health, and a healthy process can be routing 100% of traffic to dead backends. Kong: check /upstreams/:name/health for UNHEALTHY, DNS_ERROR, and TIMEOUT targets alongside /health. Traefik: check serverStatus in /api/http/services and alert on services with no health check configured (empty serverStatus) alongside /ping. Envoy: check health_flags in /clusters?format=json for failed_active_hc and failed_outlier_check alongside /ready. APISIX: check active health check results from /v1/healthcheck on port 9090 alongside the data-plane port. Istio: check /debug/syncz for sidecar xDS lag alongside /healthz/ready.

2. Authentication model diversity: Five platforms, five auth models, five different failure modes. Kong OSS: no auth enforced — client headers are silently ignored; detect this with KONG_EXPECT_AUTH intent declaration. Kong Enterprise: Kong-Admin-Token header — 403 on failure. Traefik: no built-in auth — auth is a middleware external to Traefik's own API layer; MCP tools should connect in-cluster to bypass the external middleware. Envoy: no auth — 127.0.0.1 binding is the security boundary; never create a Kubernetes Service exposing port 9901. APISIX: X-API-KEY header (not Authorization: Bearer) — use a viewer key for read-only tools; key rotation requires a restart. Istio: Kubernetes RBAC via ServiceAccount — scope the ClusterRole to the minimum necessary verbs on each CRD type; token rotation is automatic in Kubernetes 1.24+.

3. The write-path health trap: The most dangerous pattern is a healthy-looking gateway with a broken config write path. APISIX's write path depends on etcd — check GET /health on port 2379 directly, not just APISIX's data-plane port; a data-plane 200 during etcd downtime means all Admin API writes are failing silently. Traefik's API has no write path — the write path is the file provider directory or Kubernetes IngressRoute CRDs; surface rejected routes by checking for status: "disabled" or non-empty error fields in /api/http/routers. Kong DB mode: the write path is PostgreSQL — the Admin API returns database errors on write when PostgreSQL is unavailable; Kong DBless mode has no Admin API write path at all (detect via configuration.database === 'off' in GET /). Envoy: no admin API write path for routing config — config changes require the xDS control plane; detect xDS staleness via /debug/syncz on Pilot/istiod. Istio: CRD writes are eventually consistent — report CRD write status and sidecar propagation status separately; allow 1–10s for propagation in production clusters.

All five gateways need external monitoring with AliveMCP because their built-in health endpoints are designed for container orchestrator liveness probes, not for detecting the failure modes that matter most to operators: backends that are alive but sick, credentials that are valid but scope-reduced, and write paths that accept requests but fail to persist changes.

Further reading

• MCP Server Kong Gateway — Admin API tools, upstream health checks, and plugin management
• MCP Server Traefik — Router inspection, middleware management, service health, and /ping monitoring
• MCP Server Envoy — Admin API tools, cluster health flags, graceful drain, and /ready monitoring
• MCP Server Apache APISIX — Admin API tools, etcd health dependency, and plugin priority
• MCP Server Istio — CRD-based traffic management, mTLS policy, and Pilot debug API monitoring
• MCP Server NGINX — status module, upstream health, and reload-path monitoring
• MCP Tools for Service Meshes — patterns for traffic splitting, circuit breaking, and policy enforcement
• Building MCP Tools for DevOps Platforms — the three patterns for CloudWatch, Jenkins, CircleCI, Vault, and ArgoCD
• MCP Server Health Check — probe depth and alert tier design
• JSON-RPC Health Checks vs HTTP Probes — what each layer catches