AWS ECS Fargate for MCP Servers: Three Operational Patterns for Scale-In Safety, Deployment Reliability, and Observability

Task definition foundations before the patterns

The Fargate task definition is the immutable blueprint for your MCP server container. Three task definition settings have non-obvious defaults that cause production incidents:

Valid CPU and memory combinations

Fargate does not allow arbitrary CPU/memory values — ECS validates the combination at registration time and rejects invalid pairs with InvalidParameterException. This happens at registration not at deploy time, which means your CI/CD pipeline fails immediately (before any infrastructure is affected), but it also means the error message is cryptic if you don't know the constraint:

initProcessEnabled: true — zombie reaping and SIGTERM forwarding

Without initProcessEnabled: true, your application process runs as PID 1 inside the container. Linux PID 1 is responsible for reaping zombie processes — processes that have exited but whose exit status has not been collected. Node.js and most language runtimes do not implement zombie reaping. MCP servers that spawn subprocesses — shell tools, Python scripts, file processors — accumulate zombie entries in /proc. Under sustained load this eventually exhausts process table slots, and new fork() calls fail with EAGAIN.

The second reason initProcessEnabled: true matters: SIGTERM forwarding. When ECS sends SIGTERM to a task during scale-in or deployment, it goes to PID 1. Without an init process, that's your application. But child processes spawned via child_process.spawn() run in their own process groups and do not receive the forwarded signal — they continue running until the container is force-killed after the 30-second grace period. Adding init (which ECS implements via tini) costs ~100KB of overhead and ensures both zombie reaping and full process group signal propagation.

"containerDefinitions": [{
  "initProcessEnabled": true,
  // ... rest of container definition
}]

nofile ulimit 65536

The default nofile ulimit in Fargate containers is 1024. An MCP server under moderate load can reach this quickly: each persistent WebSocket or SSE connection consumes one file descriptor, as does each database connection pool entry, each upstream HTTP keepalive connection, and Node.js's internal libuv handles. A server with 100 concurrent sessions, a 20-connection database pool, 5 upstream services, and 30 Node.js internals reaches 155 descriptors — safely under 1024. But under burst conditions or with a connection leak, the limit is reachable. Set both soft and hard limits to 65536 as a precaution:

"ulimits": [{
  "name": "nofile",
  "softLimit": 65536,
  "hardLimit": 65536
}]

Secrets array injection

Inject secrets via the secrets array, not environment. The environment array stores values as plain text in the task definition revision — visible to anyone with ECS read access and logged in CloudTrail. The secrets array fetches values from Secrets Manager or SSM Parameter Store at task startup and injects them as environment variables without storing them in the task definition:

"secrets": [
  {
    "name": "DATABASE_URL",
    "valueFrom": "arn:aws:secretsmanager:us-east-1:123456789012:secret:prod/mcp-server/database-AbCdEf:connection_string::"
  }
]

The execution role (not the task role) needs secretsmanager:GetSecretValue for secrets injection. The task role is for permissions your application code uses at runtime; the execution role is for what ECS needs at task startup (ECR pull, Secrets Manager fetch, log delivery). Confusing the two produces a ResourceInitializationError at task startup rather than an IAM error at runtime.

Pattern 1 — The scale-in protection contract

MCP tool calls are stateful within a session and have no automatic retry at the client when a task terminates mid-execution. This means standard ECS scale-in — which sends SIGTERM to a selected task and forcibly kills it after 30 seconds — drops any in-flight tool call on that task with no error visible to the caller. The MCP client receives a connection reset or a truncated response. There is no log entry on the server side because the task terminated.

The solution is per-task scale-in protection, enabled via the ECS UpdateTaskProtection API called from inside the container. When scale-in protection is enabled on a task, ECS will not select that task for termination during a scale-in event — it preferentially terminates unprotected (idle) tasks instead. The protection must be toggled in application code at the start and end of each tool call:

import { ECSClient, UpdateTaskProtectionCommand } from "@aws-sdk/client-ecs";

const ecsClient = new ECSClient({ region: process.env.AWS_REGION });
const CLUSTER = process.env.ECS_CLUSTER_NAME!;
const TASK_ARN = process.env.ECS_TASK_ARN!;

async function protectTaskDuringToolCall<T>(fn: () => Promise<T>): Promise<T> {
  await ecsClient.send(new UpdateTaskProtectionCommand({
    cluster: CLUSTER,
    tasks: [TASK_ARN],
    protectionEnabled: true,
    expiresInMinutes: 60, // safety ceiling — adjust to your max tool call duration
  }));
  try {
    return await fn();
  } finally {
    await ecsClient.send(new UpdateTaskProtectionCommand({
      cluster: CLUSTER,
      tasks: [TASK_ARN],
      protectionEnabled: false,
    }));
  }
}

The ECS_TASK_ARN environment variable is available via the ECS container metadata endpoint. If it is not injected directly by your task definition, retrieve it at startup:

const METADATA_URI = process.env.ECS_CONTAINER_METADATA_URI_V4;
const taskMetadata = await fetch(`${METADATA_URI}/task`).then(r => r.json());
const TASK_ARN = taskMetadata.TaskARN; // cache this — it is constant for the task lifetime

The task role (not the execution role) must grant ecs:UpdateTaskProtection:

{
  "Effect": "Allow",
  "Action": "ecs:UpdateTaskProtection",
  "Resource": "arn:aws:ecs:*:*:task/my-cluster/*"
}

ScaleInCooldown 300s — the LLM idle window rationale

Scale-in protection guards individual in-flight calls. But there is a second failure mode at the auto-scaling policy level. MCP workloads have a characteristic traffic pattern: an LLM agent dispatches a batch of tool calls simultaneously, processes the results for 30–60 seconds while computing its next step, then dispatches another batch. During the processing window, active request count per task drops to near zero. A target-tracking policy with the default 300-second scale-in cooldown handles this correctly, but shorter cooldowns — or step scaling policies with no cooldown — trigger scale-in during the idle window and remove capacity that the next burst immediately needs.

The recommended configuration for ALB-based target tracking on an MCP server service:

--target-tracking-scaling-policy-configuration '{
  "TargetValue": 50.0,
  "PredefinedMetricSpecification": {
    "PredefinedMetricType": "ALBRequestCountPerTarget",
    "ResourceLabel": "app/my-alb/abc123/targetgroup/mcp-server-tg/def456"
  },
  "ScaleOutCooldown": 60,
  "ScaleInCooldown": 300,
  "DisableScaleIn": false
}'

ScaleOutCooldown: 60 — new Fargate tasks take 20–40 seconds to provision and pass health checks; 60 seconds ensures new capacity is registered before the next scale-out evaluation. ScaleInCooldown: 300 — protects the idle window between tool call batches. DisableScaleIn: false — keep scale-in enabled and use per-task protection to guard active calls. Disabling scale-in globally prevents the service from ever shrinking and eliminates one of the main cost benefits of Fargate.

Step scaling for burst traffic

Target tracking is reactive — it converges toward the target over multiple evaluation periods. For MCP servers that receive sudden bursts (20–50 parallel tool calls dispatched simultaneously by an agent), add a step scaling policy on CPU utilization that fires faster:

aws cloudwatch put-metric-alarm \
  --alarm-name mcp-server-cpu-high \
  --metric-name CPUUtilization \
  --namespace AWS/ECS \
  --dimensions Name=ClusterName,Value=my-cluster Name=ServiceName,Value=mcp-server-svc \
  --period 60 \
  --threshold 70 \
  --evaluation-periods 1 \
  --alarm-actions arn:aws:autoscaling:...:policyName/mcp-server-step-scale-out

EvaluationPeriods: 1 fires the alarm after a single 60-second period of elevated CPU, rather than the default 3-period (3-minute) wait. By the time 3 minutes have elapsed, the burst is typically over and the capacity shortage has already caused tool call failures.

Scheduled scaling for predictable agent workflows

If your MCP server serves a scheduled agent workflow — daily batch jobs, nightly reports, morning data-ingestion runs — pre-warm capacity with a scheduled scaling action to avoid scale-out latency at job start:

aws application-autoscaling put-scheduled-action \
  --service-namespace ecs \
  --resource-id service/my-cluster/mcp-server-svc \
  --scalable-dimension ecs:service:DesiredCount \
  --scheduled-action-name mcp-server-morning-scaleout \
  --schedule "cron(0 8 * * ? *)" \
  --scalable-target-action MinCapacity=10,MaxCapacity=20

The cron uses UTC. Pre-warming 10 minutes before the job starts is sufficient — Fargate task provisioning takes 20–60 seconds, leaving tasks healthy well before the first tool call arrives.

Pattern 2 — The deployment safety contract

The ECS deployment circuit breaker is the first line of defense against broken deployments, but its detection scope is narrower than most operators assume. Understanding exactly what it catches — and what it misses — determines whether you need alarm-based rollback as a complement.

What the circuit breaker detects

The circuit breaker monitors tasks that ECS launches as part of a deployment. When a task starts and then fails the ECS health check (the ALB target group health check, or the container-level health check in the task definition), ECS counts that as a circuit breaker failure. The threshold is derived from the service's desired count:

Enable it with automatic rollback on the service:

--deployment-configuration '{
  "deploymentCircuitBreaker": {
    "enable": true,
    "rollback": true
  },
  "maximumPercent": 200,
  "minimumHealthyPercent": 100
}'
--health-check-grace-period-seconds 60

The critical gap: application errors bypass the circuit breaker

A deployment where the new task starts successfully, passes the /health check, registers with the ALB, and then returns 500 on every tool call will not trip the circuit breaker. From ECS's perspective, the task is healthy. The circuit breaker only sees ECS health check results — it has no visibility into the application's actual behavior once the task is running and registered.

This covers a wide range of real deployment failures: a bad database migration that broke the schema the new code expects, a removed environment variable that the old code used but the new code fails on without crashing, an SDK version mismatch that produces correct health check responses but broken tool responses. These all manifest as HTTP 500s from a task that ECS considers healthy.

Alarm-based rollback closes the gap

Wire a CloudWatch alarm into the deployment rollback configuration to catch application-level regressions:

# Create a CloudWatch alarm on ALB 5xx count
aws cloudwatch put-metric-alarm \
  --alarm-name mcp-server-5xx-high \
  --metric-name HTTPCode_Target_5XX_Count \
  --namespace AWS/ApplicationELB \
  --dimensions \
      Name=LoadBalancer,Value=app/my-alb/abc123 \
      Name=TargetGroup,Value=targetgroup/mcp-server-tg/def456 \
  --statistic Sum \
  --period 60 \
  --threshold 10 \
  --comparison-operator GreaterThanOrEqualToThreshold \
  --evaluation-periods 2 \
  --treat-missing-data notBreaching

# Wire it into the deployment configuration
aws ecs update-service \
  --cluster my-cluster \
  --service mcp-server-svc \
  --deployment-configuration '{
    "deploymentCircuitBreaker": { "enable": true, "rollback": true },
    "alarms": {
      "alarmNames": ["mcp-server-5xx-high"],
      "enable": true,
      "rollback": true
    },
    "maximumPercent": 200,
    "minimumHealthyPercent": 100
  }'

With alarm-based rollback configured, ECS monitors the alarm throughout the deployment window. If it enters ALARM state — indicating that the new deployment is producing errors even though all tasks pass health checks — ECS automatically rolls back to the previous task definition revision. The alarm evaluates every 60 seconds, so rollback triggers within 2 minutes of the first sustained 5xx spike.

Recommended alarms to wire into deployment rollback for MCP servers:

Zero-downtime rolling deployment: maximumPercent and minimumHealthyPercent

Two deployment configuration parameters control how many tasks ECS replaces simultaneously. The defaults — maximumPercent: 200, minimumHealthyPercent: 100 — are correct for zero-downtime deployments but are not always set explicitly, and some CDK versions default to maximumPercent: 100:

# How a deployment proceeds with maximumPercent:200, minimumHealthyPercent:100
# Desired count: 4 tasks

Step 1: Start 4 new tasks  →  8 tasks running (4 old + 4 new)
Step 2: New tasks pass health check and register with ALB
Step 3: Stop 4 old tasks   →  4 tasks running (4 new only)

# At no point are fewer than 4 healthy tasks serving traffic.
# With maximumPercent:100: ECS must stop 2 old tasks before starting new ones,
# halving capacity during the deployment window.

For a 4-task MCP server, a deployment with maximumPercent: 100 temporarily runs only 2 tasks — at exactly the moment when extra load from the deployment monitoring period would be expected. Confirm the values explicitly in your service definition rather than relying on framework defaults.

Health check grace period — p95 startup, not mean

The health check grace period tells ECS to ignore ALB health check failures for the first N seconds after a task starts. MCP servers frequently have non-trivial startup sequences:

# Typical MCP server startup sequence:
# 1. Node.js process start                   (0–2s)
# 2. Load and validate tool schemas           (2–10s)
# 3. Warm-up connections to downstream APIs   (10–30s)
# 4. Register with MCP registry (if enabled)  (30–60s)
# → ALB /health returns 503 during steps 1–4

# Without grace period: circuit breaker fires on startup,
# rollback triggered before the new code runs a single tool call.

Set the grace period to the 95th percentile startup time — not the average. You want to protect against slow starts (which happen when a downstream service is briefly slow to accept connections at task launch time), not just typical starts. A healthCheckGracePeriodSeconds: 60 value covers most production MCP servers; servers that register with external registries or warm large in-memory caches may need 90–120 seconds.

Pattern 3 — The observability stack

The Container Insights observability stack for MCP servers has three layers: infrastructure metrics (Container Insights), application metrics (EMF), and inter-service metrics (Service Connect proxy). Each answers a different question.

Container Insights: enabling and what it costs

Container Insights is disabled by default and must be enabled at the cluster level. After enabling, existing tasks must be replaced for the setting to take effect — it is not retroactively applied:

# Enable on new cluster
aws ecs create-cluster \
  --cluster-name my-cluster \
  --configuration '{"containerInsights": "enabled"}'

# Enable on existing cluster
aws ecs update-cluster-settings \
  --cluster my-cluster \
  --settings name=containerInsights,value=enabled

# Force existing tasks to pick up the setting
aws ecs update-service \
  --cluster my-cluster \
  --service mcp-server-svc \
  --force-new-deployment

Standard Container Insights costs approximately $0.50 per 1,000 metrics per month — typically $2–5/month per active service, which is worth the cost for the task-level CPU and memory breakdown. The enhanced tier costs roughly 2× and adds GPU metrics and deeper network I/O; only use enhanced if you have GPU-based MCP tools or need CloudWatch Application Signals integration.

Infrastructure metrics that matter for MCP servers

The MemoryUtilized > 85% alarm deserves emphasis. When Fargate OOM-kills a task, the task simply stops — no crash log, no exit message, no CloudWatch error. You see a task enter STOPPED state in the ECS console, and the ECS service event says "task stopped: essential container exited." There is no explicit out-of-memory indicator unless you have the memory utilization alarm to correlate with. Without the alarm, these OOM kills are invisible until a user reports dropped tool calls.

Custom application metrics via Embedded Metric Format

Container Insights gives you infrastructure-level signals. For application-level observability — which tools are slow, which tools are failing, what is the error rate per tool type — you need custom metrics. Embedded Metric Format (EMF) is the correct mechanism for Fargate: write structured JSON to stdout, and the CloudWatch Logs agent inside the Fargate host converts it to CloudWatch Metrics automatically. No PutMetricData API calls, no SDK dependency in your critical path, zero added latency:

function emitToolMetric(toolName: string, durationMs: number, success: boolean): void {
  process.stdout.write(JSON.stringify({
    _aws: {
      Timestamp: Date.now(),
      CloudWatchMetrics: [{
        Namespace: "MCPServer/Tools",
        Dimensions: [["ToolName"], ["ToolName", "ServiceName"]],
        Metrics: [
          { Name: "ToolCallDuration", Unit: "Milliseconds" },
          { Name: "ToolCallCount", Unit: "Count" },
          { Name: "ToolCallErrors", Unit: "Count" },
        ],
      }],
    },
    ToolName: toolName,
    ServiceName: process.env.ECS_SERVICE_NAME ?? "mcp-server",
    ToolCallDuration: durationMs,
    ToolCallCount: 1,
    ToolCallErrors: success ? 0 : 1,
  }) + "\n");
}

async function handleToolCall(name: string, args: unknown): Promise<unknown> {
  const start = Date.now();
  let success = false;
  try {
    const result = await executeToolLogic(name, args);
    success = true;
    return result;
  } finally {
    emitToolMetric(name, Date.now() - start, success);
  }
}

This produces metrics in a custom MCPServer/Tools namespace, dimensioned by ToolName and ServiceName. You can set an alarm on ToolCallErrors / ToolCallCount > 0.05 (5% per-tool error rate) and wire it into alarm-based deployment rollback — catching tool-level regressions that 5xx alarms at the ALB level might miss (some MCP server implementations return HTTP 200 with an error payload in the MCP protocol layer).

The TaskArn dimension cost trap

A natural extension of EMF metrics is to add TaskArn as a dimension to correlate application-level errors with a specific task's infrastructure metrics. This is useful for debugging but expensive in production: each unique TaskArn generates a separate CloudWatch metric stream at $0.30/stream/month. A service that deploys daily and runs 10 tasks generates roughly 10 unique TaskArns per day × 30 days = 300 unique streams/month = $90/month in metric stream costs alone, on top of metric storage costs.

Use TaskArn as a dimension only on debug dashboards that you view on demand during incidents, not on production alarms that continuously emit metrics for every unique ARN:

// Production: dimension by tool name only (bounded cardinality)
Dimensions: [["ToolName"], ["ToolName", "ServiceName"]],

// Debug dashboard only: add TaskArn dimension
Dimensions: [["ToolName"], ["ToolName", "TaskArn"]],
// → costly: $0.30/month × unique TaskArns × number of metrics

Log group retention

CloudWatch Logs default retention is infinite, at $0.50/GB/month for ingestion and $0.03/GB/month for storage. Set 30-day retention on all /ecs/ log groups as a baseline:

aws logs describe-log-groups \
  --log-group-name-prefix /ecs/ \
  --query 'logGroups[?!retentionInDays].logGroupName' \
  --output text | tr '\t' '\n' | while read lg; do
    aws logs put-retention-policy --log-group-name "$lg" --retention-in-days 30
  done

In the task definition, set the log group name explicitly with a stream prefix. Do not use awslogs-create-group: "true" in production — it creates log groups with infinite retention. Create them explicitly (with a retention policy) before deploying the service:

"logConfiguration": {
  "logDriver": "awslogs",
  "options": {
    "awslogs-group": "/ecs/mcp-server",
    "awslogs-region": "us-east-1",
    "awslogs-stream-prefix": "mcp-server"
  }
}

Service Connect: free proxy metrics for multi-service MCP architectures

For MCP server architectures that split responsibilities across multiple ECS services — a tool-dispatch service, an auth service, a registry-lookup service — ECS Service Connect provides both service discovery and automatic observability at no additional cost beyond standard CloudWatch metric ingestion.

The proxy sidecar that Service Connect injects into each task automatically emits per-service-to-service metrics to CloudWatch:

Service Connect requires an HTTP Cloud Map namespace (not a DNS namespace — a common configuration mistake that produces a cryptic InvalidParameterException when creating the service). Services that only make outbound calls use client mode; services that receive inbound calls from other services use client-server mode. The proxy sidecar listens on ephemeral ports (32768–65535) for inbound connections — if tasks are in different security groups, add an inbound rule allowing TCP on that range between the security groups, or Service Connect proxy connections will silently time out after 10 seconds.

The appProtocol: "http" field in the port mapping is required for detailed per-request proxy metrics to appear in CloudWatch. Without it, the proxy emits only coarse connection-level metrics.

Consolidated failure modes

Production checklist

• Task definition: valid CPU/memory combination from the Fargate table; initProcessEnabled: true; nofile ulimit 65536/65536; credentials in secrets array, not environment; execution role has secretsmanager:GetSecretValue
• Auto scaling: scalable target registered with MinCapacity: 2 (not 1); target tracking with ScaleInCooldown: 300; step scaling alarm with EvaluationPeriods: 1 for burst; scale-in protection called at the start of every tool call handler; task role has ecs:UpdateTaskProtection
• Deployment safety: circuit breaker enabled with rollback; alarm-based rollback wired to at least one CloudWatch alarm covering application errors; maximumPercent: 200, minimumHealthyPercent: 100 set explicitly; healthCheckGracePeriodSeconds set to p95 startup time
• Observability: Container Insights enabled + force-new-deployment run after enabling; MemoryUtilized > 85% alarm; EMF metrics emitting ToolCallCount, ToolCallDuration, ToolCallErrors per tool name; 30-day retention policy on all /ecs/ log groups; log groups created explicitly (not via awslogs-create-group: true)
• Service Connect (if multi-service): HTTP Cloud Map namespace (not DNS); appProtocol: "http" in port mapping; inbound TCP 32768–65535 security group rule between services; ConnectionErrors > 0 alarm per discovery name