Horizontal Pod Autoscaler for MCP servers on EKS

Why CPU alone misleads the HPA for MCP servers

A typical MCP tool call spends most of its time waiting for an upstream LLM API response, a database query, or an external HTTP service. The pod's CPU utilization during that wait is near zero. If you scale on CPU averageUtilization: 70, the HPA sees healthy-looking numbers while fifty tool calls are queued and your users are staring at a spinner.

The right mental model: CPU is a proxy for compute demand. For I/O-heavy MCP servers, the real demand signal is one of:

• In-flight concurrent requests (a gauge your server increments/decrements around each tool call)
• Tool-call queue depth (if you buffer requests)
• Request rate measured externally (ALB request count, SQS queue depth)

CPU still belongs in the HPA as a secondary guard for genuinely CPU-heavy tools (code execution, image processing). Use a multi-metric HPA and let Kubernetes scale when any metric crosses its threshold.

Basic CPU-based HPA

Before adding custom metrics, get the baseline working. The only non-obvious requirement: every container in your pod spec must have resources.requests.cpu set. Without it the metrics-server cannot compute utilization and the HPA reports <unknown>/70% forever.

# hpa-mcp-cpu.yaml
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: mcp-server-hpa
  namespace: mcp
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: mcp-server
  minReplicas: 2
  maxReplicas: 20
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: AverageUtilization
          averageUtilization: 65   # scale up when avg CPU > 65% of request
    - type: Resource
      resource:
        name: memory
        target:
          type: AverageUtilization
          averageUtilization: 80   # secondary guard only — see memory note below

And the matching Deployment fragment with explicit resource requests:

# deployment fragment — resources section is mandatory for HPA
spec:
  template:
    spec:
      containers:
        - name: mcp-server
          image: your-ecr-repo/mcp-server:latest
          resources:
            requests:
              cpu: "250m"      # HPA divides current usage by this
              memory: "512Mi"
            limits:
              cpu: "1000m"
              memory: "1Gi"

Memory-based HPA caveat: JVM-based MCP servers (and Node.js servers with large V8 heap pools) inflate memory constantly regardless of request load. The HPA sees high memory utilization and scales out, but adding replicas doesn't reduce memory on the existing pods. Use memory as an upper-bound alert, not a primary scale driver.

Custom metric HPA via Prometheus adapter

To scale on application-level metrics, you need three components: metrics-server (already present in EKS), kube-prometheus-stack (Prometheus + Grafana), and prometheus-adapter (bridges Prometheus metrics into the Kubernetes custom metrics API).

First, instrument your MCP server to expose a Prometheus metric. A gauge counting in-flight tool calls is the most reliable signal:

# Python example — instrument with prometheus_client
from prometheus_client import Gauge, start_http_server

mcp_tool_call_queue_depth = Gauge(
    'mcp_tool_call_queue_depth',
    'Number of tool calls currently queued or in-flight'
)

# Expose on :9090/metrics alongside your MCP endpoint
start_http_server(9090)

Next, create a ServiceMonitor so Prometheus scrapes it:

# servicemonitor-mcp.yaml
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: mcp-server-metrics
  namespace: mcp
  labels:
    release: kube-prometheus-stack   # must match Prometheus operator selector
spec:
  selector:
    matchLabels:
      app: mcp-server
  endpoints:
    - port: metrics          # must match a named port on the Service
      path: /metrics
      interval: 15s
  namespaceSelector:
    matchNames:
      - mcp

Configure the prometheus-adapter to expose that metric to the custom metrics API:

# prometheus-adapter values.yaml (Helm)
rules:
  custom:
    - seriesQuery: 'mcp_tool_call_queue_depth{namespace!="",pod!=""}'
      resources:
        overrides:
          namespace: { resource: namespace }
          pod:       { resource: pod }
      name:
        matches:     "mcp_tool_call_queue_depth"
        as:          "mcp_tool_call_queue_depth"
      metricsQuery: 'avg(<<.Series>>{<<.LabelMatchers>>})'

Verify the metric is visible before wiring the HPA:

kubectl get --raw \
  "/apis/custom.metrics.k8s.io/v1beta1/namespaces/mcp/pods/*/mcp_tool_call_queue_depth" \
  | jq .

Now the HPA referencing that custom metric:

# hpa-mcp-custom.yaml
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: mcp-server-hpa
  namespace: mcp
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: mcp-server
  minReplicas: 2
  maxReplicas: 20
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: AverageUtilization
          averageUtilization: 65
    - type: Pods
      pods:
        metric:
          name: mcp_tool_call_queue_depth
        target:
          type: AverageValue
          averageValue: "5"   # scale out when avg queue depth per pod exceeds 5

The type: Pods metric aggregates across all pods and divides by replica count. Scale when the average queue depth exceeds 5 per pod — tune this number for your expected tool-call latency and acceptable queue length.

Scale-down behavior and MCP server cold start

The default HPA scale-down stabilization window is 300 seconds (5 minutes). That means after load drops, the HPA waits 5 minutes before removing a pod. This is intentionally conservative. For MCP servers with fast cold starts, you can reduce it. For servers that take 60+ seconds to initialize, you may want to increase minReplicas instead of fighting the stabilization window.

The spec.behavior block gives fine-grained control over both directions:

# hpa-mcp-behavior.yaml
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: mcp-server-hpa
  namespace: mcp
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: mcp-server
  minReplicas: 2
  maxReplicas: 20
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: AverageUtilization
          averageUtilization: 65
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 0     # scale up immediately — don't wait
      policies:
        - type: Pods
          value: 4                       # add at most 4 pods per period
          periodSeconds: 60
        - type: Percent
          value: 100                     # or double the pod count per period
          periodSeconds: 60
      selectPolicy: Max                  # use whichever policy adds more pods
    scaleDown:
      stabilizationWindowSeconds: 180   # wait 3 min of sustained low load
      policies:
        - type: Pods
          value: 2                       # remove at most 2 pods per period
          periodSeconds: 60
        - type: Percent
          value: 20                      # or remove at most 20% per period
          periodSeconds: 60
      selectPolicy: Min                  # use whichever policy removes fewer pods

Cold start math: if your MCP server takes 60 seconds to pass its readiness probe after boot, a sudden 4x traffic spike will cause degraded service for at least 60 seconds even with stabilizationWindowSeconds: 0 on scale-up. The only remedies are: lower minReplicas floor, faster initialization (lazy-load tools, reduce startup I/O), or predictive scaling via KEDA's cron-based scaler.

KEDA for event-driven MCP workloads

KEDA (Kubernetes Event-Driven Autoscaler) replaces or augments the HPA for workloads that scale on external event sources. Key advantages over HPA for MCP servers:

• Can scale to zero replicas when idle (HPA minimum is 1)
• Scales on SQS queue depth, Kafka consumer lag, HTTP request rate, or any of 50+ built-in scalers
• First-class support for HTTP-triggered scaling via the http-add-on

# keda-scaledobject-mcp.yaml
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: mcp-server-scaledobject
  namespace: mcp
spec:
  scaleTargetRef:
    name: mcp-server
  minReplicaCount: 0          # scale to zero when no requests
  maxReplicaCount: 20
  cooldownPeriod: 120         # seconds to wait before scaling to zero
  pollingInterval: 10         # how often to check scalers (seconds)
  triggers:
    - type: prometheus
      metadata:
        serverAddress: http://kube-prometheus-stack-prometheus.monitoring:9090
        metricName: mcp_tool_call_queue_depth
        threshold: "5"
        query: avg(mcp_tool_call_queue_depth{namespace="mcp"})
    - type: aws-sqs-queue
      metadata:
        queueURL: https://sqs.us-east-1.amazonaws.com/123456789012/mcp-tool-jobs
        queueLength: "10"     # scale out when >10 messages visible
        awsRegion: us-east-1
      authenticationRef:
        name: keda-aws-credentials

KEDA and HPA can conflict if both target the same Deployment — pick one. When KEDA is installed, it creates an HPA internally and manages it. Do not create a separate HPA for the same target.

HPA + Cluster Autoscaler interaction: when HPA scales pods out and no node has sufficient capacity, new pods land in Pending. The Cluster Autoscaler (CA) detects Pending pods and provisions new EC2 nodes. The end-to-end delay is: HPA decision time (15–30 s) + CA reaction time (up to 60 s) + EC2 node launch (~90 s) + node join + pod schedule + readiness probe. Plan for 3–5 minutes of latency on a cold scale-out event and set minReplicas accordingly.

Failure modes reference