AWS FIS Chaos Experiments on ECS Fargate MCP Servers

Fargate-compatible FIS action types

Not all FIS actions are available for Fargate. The compatibility matrix:

The aws:ecs:stop-task action is available without any agent — it calls the ECS API directly. All the other container-level actions (CPU, IO, process kill, network port blackhole) require enabling the ECS FIS agent as a sidecar container in your task definition.

Task stop experiment — the most important Fargate test

Task stop is the simplest and most valuable FIS experiment for Fargate-hosted MCP servers. It replicates the most common production failure mode: the ECS task exits (OOM kill, process crash, container registry pull failure, host node replacement) and the ECS service must start a replacement.

import * as fis from 'aws-cdk-lib/aws-fis';
import * as cloudwatch from 'aws-cdk-lib/aws-cloudwatch';

const taskStopTemplate = new fis.CfnExperimentTemplate(this, 'McpTaskStop', {
  description: 'Stop 50% of MCP server Fargate tasks — verify ECS replaces them within 30s',
  roleArn: fisRole.roleArn,

  targets: {
    McpTasks: {
      resourceType: 'aws:ecs:task',
      resourceTags: {
        Role: 'mcp-server',
        ChaosEnabled: 'true',
        Environment: 'staging',
      },
      filters: [
        {
          path: 'clusterArn',
          values: ['arn:aws:ecs:us-east-1:123456789012:cluster/mcp-cluster'],
        },
      ],
      selectionMode: 'PERCENT(50)',
    },
  },

  actions: {
    StopTask: {
      actionId: 'aws:ecs:stop-task',
      targets: { Tasks: 'McpTasks' },
    },
  },

  stopConditions: [
    {
      source: 'aws:cloudwatch:alarm',
      value: mcpRunningTaskCountAlarm.alarmArn,
    },
  ],

  tags: { Name: 'mcp-task-stop', Environment: 'staging' },
});

The filters array narrows targets beyond tag matching. Use it to scope the experiment to a specific ECS cluster so the template cannot accidentally target the wrong environment even if tag drift occurs. The filter path clusterArn is the ECS resource attribute — available filter paths are documented per resource type in the FIS user guide.

ECS service configuration for safe task stop experiments

An ECS service with desiredCount: 1 and minimumHealthyPercent: 0 will not automatically restart a stopped task until the health check grace period expires. This is the most common misconfiguration that makes chaos experiments cause unacceptable downtime. Configure your ECS service defensively:

const mcpService = new ecs.FargateService(this, 'McpService', {
  cluster,
  taskDefinition: mcpTaskDef,

  // Run at least 2 tasks so stopping one doesn't take the service to 0
  desiredCount: 2,

  // ECS will not stop an old task until a new one is healthy
  // With PERCENT(50) for 2 tasks: ECS maintains 1 running task minimum during deploys
  minHealthyPercent: 50,
  maxHealthyPercent: 200,

  // Health check grace period: how long ECS waits after task start before checking health
  // Keep this short (< your AliveMCP probe interval) to detect bad starts quickly
  healthCheckGracePeriod: cdk.Duration.seconds(15),

  // Ensure tasks propagate service tags so FIS can target them by tag
  propagateTags: ecs.PropagatedTagSource.SERVICE,

  enableExecuteCommand: true,  // ECS Exec for debugging during/after experiments
});

With desiredCount: 2 and selectionMode: PERCENT(50), the task stop experiment will stop one of the two running tasks. ECS will immediately start a replacement task. Your AliveMCP probe, hitting the load balancer in front of the ECS service, will route traffic to the surviving healthy task during the replacement period — total user-visible downtime should be zero if the ALB health check removes the stopped task from the target group before its connections are dropped.

Enabling the ECS FIS agent sidecar

To use container-level FIS actions (CPU stress, process kill, network port blackhole) on Fargate, you must add the ECS FIS agent as a sidecar container to your task definition. The agent is an AWS-managed container that FIS communicates with to inject faults inside the task boundary.

const mcpTaskDef = new ecs.FargateTaskDefinition(this, 'McpTaskDef', {
  memoryLimitMiB: 512,
  cpu: 256,
});

// Main MCP server container
const mcpContainer = mcpTaskDef.addContainer('McpServer', {
  image: ecs.ContainerImage.fromEcrRepository(repo, 'latest'),
  portMappings: [{ containerPort: 3000 }],
  logging: ecs.LogDrivers.awsLogs({ streamPrefix: 'mcp-server' }),
});

// ECS FIS agent sidecar — required for container-level chaos actions
mcpTaskDef.addContainer('FisAgent', {
  image: ecs.ContainerImage.fromRegistry(
    'public.ecr.aws/aws-fault-injection-simulator/aws-fis-agent:latest',
  ),
  essential: false,  // agent crash should not kill the MCP server container
  environment: {
    // Required: tells the FIS agent which task it's running in
    FIS_TASK_ARN: 'arn:aws:ecs:us-east-1:123456789012:task/cluster-name/TASK_ID',
  },
  // FIS agent does not expose any ports to the application
  logging: ecs.LogDrivers.awsLogs({ streamPrefix: 'fis-agent' }),
});

// Grant the task role permissions for the FIS agent to call back to FIS
mcpTaskDef.taskRole.addToPrincipalPolicy(new iam.PolicyStatement({
  actions: [
    'fis:SendExperimentReportS3BucketPolicy',
    'fis:GetExperimentTargetAccountConfiguration',
  ],
  resources: ['*'],
}));

Set essential: false on the FIS agent sidecar — if the agent container crashes (e.g., during a network disruption experiment that cuts FIS connectivity), you do not want ECS to stop the entire task. The essential: false flag means ECS will only stop the task if the main (McpServer) container exits.

Container process kill with aws:ecs:task-kill-process

Once the FIS agent sidecar is added to your task, you can use aws:ecs:task-kill-process to send a named signal to a specific process inside the container. This is more precise than stopping the entire task — you can kill just the MCP server process to test systemd/supervisord or Node.js process manager restart behavior inside the container:

actions: {
  KillMcpProcess: {
    actionId: 'aws:ecs:task-kill-process',
    parameters: {
      processName: 'node',       // the process name as seen in /proc (not the full command)
      signal: 'SIGKILL',         // SIGKILL, SIGTERM, SIGHUP, SIGUSR1, SIGUSR2
      installDependencies: 'true',  // FIS agent will install kill/pkill if not present
    },
    targets: { Tasks: 'McpTasks' },
  },
},

The signal parameter is important for testing your MCP server's graceful shutdown path. Use SIGTERM first to verify that your server handles termination signals properly (closes connections, flushes logs, returns in-flight tool calls with errors). Then use SIGKILL to test the unrecoverable crash path where no cleanup is possible. Most MCP server operators discover they handle SIGTERM but not SIGKILL — the difference matters for in-flight streaming tool calls that partially wrote results before the crash.

Network port blackhole — aws:ecs:task-network-blackhole-port

The network port blackhole action drops all traffic to or from a specific port inside the Fargate container. This is the Fargate equivalent of the EC2 network blackhole but scoped to a single port — useful for simulating "can't reach the database" without fully isolating the container:

actions: {
  BlockDatabasePort: {
    actionId: 'aws:ecs:task-network-blackhole-port',
    parameters: {
      port: '5432',         // drop all TCP to PostgreSQL
      protocol: 'tcp',
      trafficType: 'egress',  // 'egress' = outbound from container, 'ingress' = inbound
      duration: 'PT2M',
    },
    targets: { Tasks: 'McpTasks' },
  },
},

Blocking the database port tests your MCP server's dependency failure handling. What to verify:

• Does the MCP server return -32603 errors with "message": "Database temporarily unavailable", or does it time out silently and drop the WebSocket connection?
• Does your connection pool detect the broken connections immediately (via keepalive settings) or only after the TCP timeout (~4 minutes by default)?
• Does the MCP server's health check endpoint return 503 during database unavailability (causing ALB to remove it from rotation) or still return 200 (keeping it in rotation but returning errors to tool callers)?

The correct behavior depends on your SLO: if your MCP server can serve some tools without the database, the health check should return 200 and the failing tools should return structured errors. If all tools require the database, returning 503 from the health check is better — ALB will route traffic to healthy replicas while the affected replica waits for database reconnection.

Measuring recovery with AliveMCP during experiments

AliveMCP probes your MCP endpoint every 60 seconds and records the response time and status. When you run FIS experiments, the AliveMCP dashboard becomes your ground-truth measurement of user-visible recovery time. The sequence to observe:

1. T+0s: FIS starts experiment (e.g., stops one Fargate task)
2. T+5s: ECS detects task stopped, starts replacement task placement
3. T+15s: New task container starts, MCP server process initializes
4. T+25s: Health check grace period expires, ALB health check runs against new task
5. T+30s: Health check passes, new task added to ALB target group
6. T+35s: First AliveMCP probe against new task — should return success

If the AliveMCP probe fires during the window between T+0 and T+35, it will likely show a probe failure. If your AliveMCP alert is configured to fire after 2 consecutive failures (2 × 60s = 120s), you should not receive an alert for a healthy task stop experiment — recovery completes in ~35s, well within the 120s window. If you do receive an AliveMCP alert, your recovery time exceeds your probe tolerance and needs to be optimized.

Common failures