AWS FIS · 2026-11-07 · FIS arc

AWS FIS Chaos Engineering for MCP Servers: Fault Injection, ECS Fargate Experiments, and Stop Conditions

AliveMCP probes thousands of MCP endpoints every 60 seconds, and among the AWS-deployed servers we monitor, the failure patterns that cause the longest outages share a common trait: they are failures that the server operator never validated in advance because they never ran a controlled experiment to see how the server behaves under them. The MCP protocol is stateful — a client connects over SSE or WebSocket, negotiates a capability list, and then issues tool calls against the same persistent connection. When the server process is killed by an OOM event, the client does not get a clean error; it gets a dropped connection that surfaces as a timeout, a confusing "connection reset" exception, or — worst of all — no error at all from an agent framework that silently swallows the reconnect failure and proceeds as if the tool returned an empty result. A REST API can often be retried trivially; a stateful MCP session that crashes mid-way through a multi-step tool chain leaves the agent in an indeterminate state. The developer learns about these failure modes in one of two ways: a user files a bug report at the worst possible time, or the developer runs a chaos experiment and observes them in a controlled environment first. AWS Fault Injection Simulator (FIS) is the second path — a managed chaos engineering service that lets you inject controlled, time-bounded, automatically-reversible faults into your production or staging infrastructure without writing custom scripts. For MCP server operators, FIS answers four questions that you cannot answer by reading logs or reviewing code: Does the ECS Fargate service replace a stopped task before the first AliveMCP probe returns a failure? Does the Node.js process return meaningful JSON-RPC error codes under CPU saturation or does it drop connections silently? Does the connection pool detect a database network partition immediately or does it queue requests for the full 4-minute TCP timeout? Does your stop condition alarm fire fast enough to abort the experiment before it burns too much of your monthly error budget? This guide synthesizes four deep-dives — FIS concepts and IAM setup, writing experiment templates for the four key fault types, chaos on ECS Fargate, and stop conditions as your kill switch — into a complete FIS chaos engineering playbook for MCP server operators.

TL;DR

FIS concepts and IAM — an FIS experiment has four required components: a target (which AWS resources to inject faults into, selected by resource tag with optional selectionMode: PERCENT(50) for blast-radius limiting), an action (the fault to inject — aws:ec2:cpu-stress, aws:ecs:stop-task, aws:ssm:send-command, etc. — with ISO 8601 duration and fault-specific parameters), a stop condition (a CloudWatch alarm ARN that FIS monitors every 15 seconds during the experiment — if the alarm fires, FIS rolls back all injected faults immediately and marks the run stopped), and an IAM role (a dedicated FIS role with fis.amazonaws.com as the trust principal, and service-specific permissions scoped with tag conditions so FIS cannot accidentally target instances outside your MCP server fleet): The opt-in tag pattern — adding ChaosEnabled=true and Role=mcp-server to every resource you intentionally expose to FIS, then scoping the IAM role with StringEquals: { 'ec2:ResourceTag/ChaosEnabled': 'true' } — is both a safety net and a documentation mechanism. Any resource that has not been explicitly reviewed and tagged for chaos is invisible to FIS. When you scale up from one EC2 instance to a fleet, the new instances are excluded from experiments until a team member deliberately adds the tags, forcing a conscious decision about blast radius before the experiment runs. Enable CloudWatch Logs logging on every experiment template using logSchemaVersion: 2 (structured JSON, not plain text) — the logs record each action's start and stop time, the specific resources targeted, stop condition triggers, and error messages when FIS cannot inject a fault due to missing SSM agent or insufficient IAM permissions. These logs are the first place to look when an experiment result shows failed rather than completed or stopped.
Experiment templates — four fault families cover the most valuable chaos scenarios for MCP servers: CPU stress (aws:ec2:cpu-stress uses SSM to run stress-ng --cpu-load 80 for a configurable duration — requires SSM agent and AmazonSSMManagedInstanceCore on the instance profile; observe whether the Node.js event loop backs up and prevents health checks from responding, causing ECS to restart the task while your MCP server was under load); memory pressure (aws:ec2:memory-stress consumes a percentage of available memory via SSM — reveals OOM kill risk; if the kernel sends SIGKILL before Node.js triggers its own GC, in-flight tool calls get zero cleanup; fix by setting --max-old-space-size below the container's memory limit so Node.js returns a structured error before the kernel intervenes); network blackhole (aws:network:disrupt-connectivity{scope:'egress'} drops outbound traffic — tests whether the MCP server returns -32603 errors when it can't reach its database or whether it hangs until the client TCP timeout; reveals whether connectTimeout settings on the Secrets Manager SDK are configured); process kill via SSM (aws:ssm:send-command targeting a custom SSM document running pkill -SIGKILL -f "node.*mcp-server" — the most realistic crash scenario because SIGKILL gives the process no chance for cleanup; measures time-to-recovery as the key metric — if ECS restart takes 25 seconds and your MCP clients have a 20-second timeout, every crash produces a client-visible failure even after recovery). Chain actions sequentially with startAfter: ['ActionName'] to simulate compound failures — CPU stress followed by process kill tests crash behavior when the server is already degraded. Use emptyTargetResolutionMode: 'skip' in experimentOptions for CI pipelines where target environments may be torn down between pipeline runs: The recovery time metric from process kill experiments is the most actionable output from MCP server chaos testing. If the gap between SIGKILL and the first successful probe response exceeds your MCP client's retry timeout, you have a configuration change to make — smaller container images, fewer startup dependencies, pre-warmed connections — not a code change. FIS surfaces this measurement reliably and repeatably in staging before it surfaces as user-facing downtime in production.
ECS Fargate chaos — Fargate tasks have no SSM agent, so aws:ec2:cpu-stress, aws:ec2:memory-stress, and aws:network:disrupt-connectivity are unavailable; Fargate's FIS action set is: aws:ecs:stop-task (no agent required — calls the ECS API directly to stop selected tasks; the most important Fargate chaos action because it replicates the most common production failure mode — ECS task exits due to OOM kill, process crash, or host replacement — and verifies the ECS service replaces the task fast enough for your availability target), plus aws:ecs:task-cpu-stress, aws:ecs:task-io-stress, aws:ecs:task-kill-process, and aws:ecs:task-network-blackhole-port (these four require enabling the ECS FIS agent as a sidecar container with essential: false in the task definition — set essential: false so a FIS agent crash during a network disruption experiment does not kill the MCP server container). ECS service configuration for safe task stop experiments: desiredCount: 2, minimumHealthyPercent: 50, maxHealthyPercent: 200, healthCheckGracePeriod: 15s, and propagateTags: SERVICE (ECS tasks inherit service tags only when this is set — FIS targets tasks, not services, so tasks must have the ChaosEnabled=true tag propagated). Filter target tasks by clusterArn in the target selector to prevent accidentally targeting tasks in the wrong environment even if tag drift occurs: For a 2-task Fargate service with desiredCount: 2 and selectionMode: PERCENT(50), a task stop experiment stops exactly one task. ECS immediately starts a replacement. Your ALB routes traffic to the surviving healthy task during the replacement period. Total user-visible downtime should be zero if the ALB health check removes the stopped task from the target group before its connections are dropped — typically within 10 seconds of the task stopping. If AliveMCP shows a probe failure during this window, your ALB deregistration delay or health check interval needs tuning.
Stop conditions — always add at least one CloudWatch alarm ARN as a stop condition to every FIS experiment template; FIS polls the alarm state every 15 seconds and aborts the experiment (rolling back all injected faults) if any stop condition alarm transitions to ALARM; create three dedicated alarms for FIS stop conditions (not shared with PagerDuty): an availability alarm on AWS/ApplicationELB HealthyHostCount Minimum with a 1-minute period (not the 5-minute period you use for on-call alerting), evaluationPeriods: 1, and treatMissingData: BREACHING (critical — without BREACHING, a catastrophic failure where all targets die simultaneously stops emitting the metric and the alarm stays in OK); a latency alarm on AWS/ApplicationELB TargetResponseTime p99 with the threshold set at your SLO minus client timeout buffer (not at normal-operations best-case latency); and an ECS running task count alarm on ECS/ContainerInsights RunningTaskCount Minimum (requires Container Insights enabled — without it, the metric doesn't appear and the alarm stays in INSUFFICIENT_DATA); combine all three into one cloudwatch.CompositeAlarm using AlarmRule.anyOf() and use the composite alarm ARN as the single stop condition in all experiment templates; verify the composite alarm is in OK state before calling StartExperiment — FIS aborts immediately if any stop condition alarm is already in ALARM at experiment start: The sensitivity calibration distinction between FIS stop conditions and production alarms is what allows you to run chaos experiments safely in staging without false-positive PagerDuty alerts and without experiments running too long when faults exceed safe thresholds. Production alarms use 5-minute periods with 3-of-5 consecutive breach requirements to reduce alert fatigue — a brief spike during a deploy should not wake someone up. FIS stop condition alarms use 1-minute periods with 1 consecutive breach requirement — the goal is fast abort, not confidence building. FIS polls every 15 seconds, so a 1-minute alarm enables abort within 1–2 minutes of the fault exceeding the threshold. Never reuse a production alarm as a FIS stop condition.

Part 1: Why MCP servers need chaos engineering — and what FIS adds

Most MCP servers are deployed as stateful long-running processes — they accept persistent connections from AI agents, negotiate capability lists over the MCP handshake, and then process tool calls over the same connection for the duration of a session. This architecture is efficient and powerful, but it means failures are qualitatively different from REST API failures. When a REST endpoint goes down, the client typically gets an HTTP 5xx response, logs the error, and either retries or surfaces the failure cleanly to the user. When a stateful MCP server process dies mid-session, the client's outstanding tool calls may get a TCP RST (connection reset), a WebSocket close frame, or nothing at all — just a hanging request that eventually times out. Agent frameworks that wrap MCP tool calls often do not distinguish between "tool returned an error" and "tool call timed out due to server crash" — both become failures upstream, but the diagnostics and remediation paths are completely different.

The failure modes that chaos engineering surfaces before users do:

OOM kill: The MCP server process grows its heap during a tool call that returns large payloads (embedding vectors, database query results, file content). The kernel sends SIGKILL. No cleanup. In-flight tool calls receive no response. The ECS service starts a replacement task — but does the replacement complete within your MCP client's retry window?
CPU saturation: A tool handler calls an expensive external API or processes a large dataset. The Node.js event loop backs up. Health check requests queue behind tool calls. If the ALB health check times out before the queue drains, ECS marks the task unhealthy and stops it — now a performance degradation has become an availability incident.
Backend network partition: The MCP server can receive tool call requests from the client but cannot reach its database or secrets manager. Does it return structured JSON-RPC errors (-32603 with a human-readable message)? Or does it hang until the connection pool timeout fires — 4 minutes by default for most PostgreSQL drivers — causing all in-flight requests to stall for 4 minutes before returning any error?
Partial tool availability: Some tools work correctly while others fail because they depend on a backend service that is degraded. Does the server mark individual tools as unavailable in the capability response, or does it advertise all tools as available and silently return corrupted or empty results for the failing subset?

AWS FIS addresses all four scenarios with a managed, controlled, automatically-reversible fault injection system. The key operational advantage over custom scripts: FIS wires a CloudWatch alarm as a kill switch. If the fault causes more damage than intended, the alarm fires within 1–2 minutes and FIS rolls back all injected faults without manual intervention. You can't accidentally take yourself down permanently.

The three FIS primitives you build everything from:

Targets: Which AWS resources to inject faults into. Selected by resource ID, tag, or filter. For MCP servers, use tag selectors (Role=mcp-server + ChaosEnabled=true) combined with selectionMode: PERCENT(50) to limit blast radius to half the fleet rather than all instances.
Actions: The fault to inject. FIS provides built-in actions organized by service namespace. Actions run in parallel by default; use startAfter to chain them sequentially. Each action runs for a configured duration (ISO 8601, e.g., PT3M = 3 minutes) after which FIS rolls back the fault automatically.
Stop conditions: CloudWatch alarm ARNs. FIS polls these every 15 seconds. If any alarm transitions to ALARM, FIS immediately stops all injected faults and marks the experiment stopped. Always wire at least one — without a stop condition, you have no automatic kill switch.

IAM role scoping with tag conditions

Create a dedicated FIS IAM role — do not reuse Lambda execution roles or deployment roles. Trust principal is fis.amazonaws.com. The critical safety configuration is scoping EC2 and ECS permissions with tag conditions so FIS cannot accidentally target resources outside the MCP server fleet:

const fisRole = new iam.Role(this, 'FisRole', {
  assumedBy: new iam.ServicePrincipal('fis.amazonaws.com'),
});

// Scope EC2 permissions to only chaos-tagged instances
fisRole.addToPolicy(new iam.PolicyStatement({
  actions: ['ec2:StopInstances', 'ec2:RebootInstances', 'ec2:TerminateInstances'],
  resources: ['*'],
  conditions: {
    StringEquals: { 'ec2:ResourceTag/ChaosEnabled': 'true' },
  },
}));

// ECS task stop (no tag condition available — use target selector filter instead)
fisRole.addToPolicy(new iam.PolicyStatement({
  actions: ['ecs:StopTask', 'ecs:DescribeTasks', 'ecs:ListTasks'],
  resources: ['*'],
}));

// SSM for CPU/memory stress and custom process kill commands
fisRole.addToPolicy(new iam.PolicyStatement({
  actions: ['ssm:SendCommand', 'ssm:GetCommandInvocation', 'ssm:DescribeInstanceInformation'],
  resources: ['*'],
}));

// CloudWatch for stop condition polling and experiment logging
fisRole.addToPolicy(new iam.PolicyStatement({
  actions: ['cloudwatch:DescribeAlarms', 'logs:CreateLogDelivery', 'logs:PutLogEvents'],
  resources: ['*'],
}));

Tag every EC2 instance and ECS service you intentionally expose to chaos with ChaosEnabled=true and Role=mcp-server. For ECS, set propagateTags: ecs.PropagatedTagSource.SERVICE on the Fargate service — FIS targets ECS tasks (not services), and tasks only inherit service tags when propagation is enabled.

Part 2: Writing experiment templates for four MCP server fault scenarios

An FIS experiment template is a versioned, reusable document declaring targets, actions, stop conditions, and an IAM role. Store templates in CDK — they become version-controlled reliability tests that you re-run on every significant deploy. Here are the four fault templates most valuable for MCP server operators.

Template 1: CPU stress — verify event loop behavior under saturation

aws:ec2:cpu-stress uses the SSM agent to run stress-ng --cpu N --cpu-load 80 on the target instances. Requires the SSM agent to be installed and the instance profile to have AmazonSSMManagedInstanceCore.

new fis.CfnExperimentTemplate(this, 'McpCpuStress', {
  description: 'Apply 80% CPU load for 5 minutes — verify event loop behavior and health check response',
  roleArn: fisRole.roleArn,

  targets: {
    McpInstances: {
      resourceType: 'aws:ec2:instance',
      resourceTags: { Role: 'mcp-server', ChaosEnabled: 'true' },
      selectionMode: 'PERCENT(50)',
    },
  },

  actions: {
    CpuStress: {
      actionId: 'aws:ec2:cpu-stress',
      parameters: { duration: 'PT5M', percent: '80' },
      targets: { Instances: 'McpInstances' },
    },
  },

  stopConditions: [{ source: 'aws:cloudwatch:alarm', value: fisStopConditionAlarm.alarmArn }],
  logConfiguration: {
    cloudWatchLogsConfiguration: { logGroupArn: `arn:aws:logs:us-east-1:ACCOUNT:log-group:/fis/mcp-experiments:*` },
    logSchemaVersion: 2,
  },
});

What to observe during CPU stress: Does the Node.js event loop back up and prevent the ALB health check endpoint from responding? If the health check exceeds its timeout (default 5 seconds in ALB), ECS marks the task unhealthy and stops it — a performance degradation has become an availability incident. Does the MCP server return meaningful JSON-RPC error codes under saturation, or does it drop connections? A well-behaved server returns -32603 Internal error with a "server overloaded, retry after Ns" message rather than a bare TCP close.

Template 2: Memory pressure — find OOM risk before the kernel does

aws:ec2:memory-stress uses SSM and stress-ng --vm to consume a percentage of available memory. The critical question: does the kernel OOM-kill the Node.js process before it triggers its own GC?

actions: {
  MemoryPressure: {
    actionId: 'aws:ec2:memory-stress',
    parameters: {
      duration: 'PT3M',
      percent: '70',    // consume 70% of total instance memory
      workers: '4',
    },
    targets: { Instances: 'McpInstances' },
  },
},

If you observe OOM kills in this experiment, the fix is to set Node.js --max-old-space-size to a value below the container's memory limit. With --max-old-space-size=384 in a 512 MB container, Node.js triggers its own GC and returns structured errors when the heap limit approaches — the kernel never intervenes. Without this setting, the kernel OOM-kills the process with SIGKILL and in-flight tool calls get zero cleanup.

Template 3: Network blackhole — test backend dependency failure handling

aws:network:disrupt-connectivity{scope:'egress'} drops all outbound traffic from the MCP server instance, simulating inability to reach the database, Secrets Manager, or external APIs while clients can still submit tool calls. This is the most common silent failure mode — the MCP server accepts new connections but all tool calls that require backend access stall until the TCP timeout fires.

actions: {
  EgressBlackhole: {
    actionId: 'aws:network:disrupt-connectivity',
    parameters: { duration: 'PT2M', scope: 'egress' },
    targets: { Instances: 'McpInstances' },
  },
},

Key questions this experiment answers: Does the connection pool detect broken connections immediately (via TCP keepalive) or only after the full 4-minute TCP timeout? Does the Secrets Manager SDK respect its connectTimeout setting, or does it hang for the kernel default? Does the MCP server return a -32603 error with "message": "Database temporarily unavailable" or just drop the WebSocket connection?

Template 4: Process kill — measure time-to-recovery

No dedicated FIS action exists for killing a process — use aws:ssm:send-command targeting a custom SSM document. This is the most realistic crash scenario: SIGKILL, no cleanup, no graceful shutdown, no connection drain.

# Create the SSM document once per region
aws ssm create-document \
  --name 'KillMcpServerProcess' \
  --document-type 'Command' \
  --content '{
    "schemaVersion": "2.2",
    "mainSteps": [{
      "action": "aws:runShellScript",
      "name": "kill",
      "inputs": {
        "runCommand": ["pkill -SIGKILL -f \"node.*mcp-server\" || true"],
        "timeoutSeconds": "10"
      }
    }]
  }'

actions: {
  KillProcess: {
    actionId: 'aws:ssm:send-command',
    parameters: {
      documentArn: 'arn:aws:ssm:us-east-1:ACCOUNT:document/KillMcpServerProcess',
      documentVersion: '$DEFAULT',
      duration: 'PT1M',
    },
    targets: { Instances: 'McpInstances' },
  },
},

The key metric is time-to-recovery: seconds between the SIGKILL signal and the first successful AliveMCP probe response. If ECS task restart takes 25 seconds and your MCP clients have a 20-second timeout, every crash produces a client-visible failure even after full recovery. The fix is not in the code — it's in the deployment: smaller container images (fewer startup dependencies), pre-warmed connection pools (connect at startup, not on first tool call), and shorter health check grace periods.

Sequencing: compound failures with startAfter

Chain actions sequentially to simulate compound failures. First CPU stress to degrade the server, then process kill to test crash behavior when the server is already under load:

actions: {
  CpuStress: {
    actionId: 'aws:ec2:cpu-stress',
    parameters: { duration: 'PT2M', percent: '90' },
    targets: { Instances: 'McpInstances' },
  },
  KillProcess: {
    actionId: 'aws:ssm:send-command',
    parameters: { documentArn: '...KillMcpServerProcess', documentVersion: '$DEFAULT', duration: 'PT30S' },
    targets: { Instances: 'McpInstances' },
    startAfter: ['CpuStress'],  // waits for CpuStress to COMPLETE (not just start)
  },
},

startAfter takes a list of action name strings. FIS starts the dependent action only after all listed actions have reached completed state. Actions without startAfter run in parallel from experiment start. Plan total experiment duration as the sum of all sequential action durations.

Part 3: ECS Fargate chaos — task stop, FIS agent sidecar, and port-level network disruption

ECS Fargate is a popular hosting choice for MCP servers: no instance management, pay-per-task, built-in ECS service restart on failure. The managed nature means a different FIS action set. Fargate tasks have no SSM agent, so the EC2-flavored actions (aws:ec2:cpu-stress, aws:ec2:memory-stress, aws:network:disrupt-connectivity) are unavailable. Here is what Fargate supports:

Action	Agent required	Key use case
`aws:ecs:stop-task`	None	Simulate ECS task crash — most important Fargate test
`aws:ecs:task-cpu-stress`	FIS agent sidecar	CPU load inside the container namespace
`aws:ecs:task-kill-process`	FIS agent sidecar	Send SIGTERM or SIGKILL to a named process inside the container
`aws:ecs:task-network-blackhole-port`	FIS agent sidecar	Drop traffic to/from a specific port — test database unavailability
`aws:ec2:cpu-stress`	N/A	Not available on Fargate (uses SSM)
`aws:network:disrupt-connectivity`	N/A	Not available on Fargate (uses SSM tc rules)

Task stop: the most important Fargate experiment

Task stop replicates the most common production failure mode: the ECS task exits (OOM kill, process crash, host node replacement) and the ECS service must start a replacement. It calls the ECS API directly — no agent required.

new fis.CfnExperimentTemplate(this, 'McpTaskStop', {
  description: 'Stop 50% of MCP server Fargate tasks — verify ECS replaces within 30s',
  roleArn: fisRole.roleArn,

  targets: {
    McpTasks: {
      resourceType: 'aws:ecs:task',
      resourceTags: { Role: 'mcp-server', ChaosEnabled: 'true' },
      filters: [{
        path: 'clusterArn',
        values: ['arn:aws:ecs:us-east-1:ACCOUNT:cluster/mcp-cluster'],
      }],
      selectionMode: 'PERCENT(50)',
    },
  },

  actions: {
    StopTask: {
      actionId: 'aws:ecs:stop-task',
      targets: { Tasks: 'McpTasks' },
    },
  },

  stopConditions: [{ source: 'aws:cloudwatch:alarm', value: runningTaskAlarm.alarmArn }],
});

The filters array narrows targets beyond tag matching — scope to a specific ECS cluster ARN to prevent accidentally targeting tasks in the wrong environment even if tag drift occurs.

ECS service configuration for safe experiments

An ECS service misconfigured for chaos experiments causes accidents. The required settings:

new ecs.FargateService(this, 'McpService', {
  cluster,
  taskDefinition: mcpTaskDef,
  desiredCount: 2,            // minimum 2 so stopping one doesn't take service to 0
  minHealthyPercent: 50,      // ECS maintains at least 1 running task
  maxHealthyPercent: 200,
  healthCheckGracePeriod: cdk.Duration.seconds(15),
  propagateTags: ecs.PropagatedTagSource.SERVICE,  // tasks must have ChaosEnabled=true tag
  enableExecuteCommand: true,
});

With desiredCount: 2 and selectionMode: PERCENT(50), the task stop experiment stops exactly one task. ECS immediately starts a replacement. Your ALB routes traffic to the surviving healthy task during replacement — total user-visible downtime should be zero. If AliveMCP shows a probe failure during this window, your ALB deregistration delay or health check interval needs tuning.

FIS agent sidecar for container-level actions

To use CPU stress, process kill, or port-level network disruption on Fargate, add the ECS FIS agent as a sidecar container. Set essential: false — if the agent crashes during a network disruption experiment (losing FIS connectivity), the MCP server container must continue running.

mcpTaskDef.addContainer('FisAgent', {
  image: ecs.ContainerImage.fromRegistry(
    'public.ecr.aws/aws-fault-injection-simulator/aws-fis-agent:latest',
  ),
  essential: false,  // agent crash must not stop the MCP server container
  logging: ecs.LogDrivers.awsLogs({ streamPrefix: 'fis-agent' }),
});

// Task role needs permission for the FIS agent to call back to FIS service
mcpTaskDef.taskRole.addToPrincipalPolicy(new iam.PolicyStatement({
  actions: ['fis:GetExperimentTargetAccountConfiguration'],
  resources: ['*'],
}));

Port-level network disruption: testing database failure handling

The network port blackhole action is the Fargate equivalent of the EC2 egress blackhole but scoped to a single port — test "can't reach the database" without fully isolating the container:

actions: {
  BlockDatabase: {
    actionId: 'aws:ecs:task-network-blackhole-port',
    parameters: {
      port: '5432',
      protocol: 'tcp',
      trafficType: 'egress',
      duration: 'PT2M',
    },
    targets: { Tasks: 'McpTasks' },
  },
},

What to verify: Does the MCP server return -32603 errors with a readable message during database unavailability, or just drop connections? Does the connection pool detect broken connections immediately via TCP keepalive, or only after the 4-minute TCP timeout? Does the ALB health check return 503 (removing the task from rotation) or 200 (keeping it in rotation but returning errors to tool callers)? The correct behavior depends on whether some tools can serve requests without the database — if yes, return 200 and structured errors from affected tools; if no tool works without the database, return 503 so ALB routes to replicas that have reconnected.

Measuring recovery with AliveMCP

During FIS experiments, the AliveMCP dashboard is your ground-truth measurement of user-visible recovery time. For a task stop experiment on a 2-task Fargate service, the expected timeline is: T+0s FIS stops one task → T+5s ECS detects the stop and starts replacement → T+15s new task container initializes → T+25s health check grace period expires → T+30s ALB health check passes, task added to target group → T+35s first AliveMCP probe against new task returns success. If AliveMCP is configured to alert after 2 consecutive failures (2 × 60s = 120s), you should not receive an alert for a healthy task stop experiment — recovery completes in ~35 seconds, well within the 120-second alert window. If you receive an alert, recovery time exceeds your probe tolerance.

Part 4: Stop conditions — the CloudWatch alarm kill switch

A chaos experiment without a stop condition is uncontrolled fault injection. With stop conditions, it is a hypothesis-driven reliability test with an automatic kill switch. FIS monitors stop condition alarms every 15 seconds during an experiment — if any alarm transitions to ALARM, FIS immediately rolls back all injected faults and marks the experiment stopped.

What happens on stop condition trigger: FIS sets the experiment to stopping, sends rollback commands to each running action (removes CPU stress via SSM, restores network routes, etc.), waits up to 10 minutes for rollback confirmation, then sets the experiment to stopped. Not all actions are reversible — aws:ecs:stop-task cannot un-stop a task; ECS replaces it as normal service behavior, but FIS does not control the replacement timeline. This is why selectionMode: PERCENT(50) and desiredCount: 2+ matter: they limit blast radius so irreversible actions don't cause total outage even when FIS can't roll them back.

Three stop condition alarms for MCP servers

Availability alarm — fires when fewer than the minimum acceptable number of ALB healthy hosts are registered:

const availabilityStop = new cloudwatch.Alarm(this, 'FisStopAvailability', {
  alarmName: 'fis-stop-mcp-healthy-host-count',
  metric: new cloudwatch.Metric({
    namespace: 'AWS/ApplicationELB',
    metricName: 'HealthyHostCount',
    dimensionsMap: { LoadBalancer: alb.loadBalancerFullName, TargetGroup: tg.targetGroupFullName },
    statistic: 'Minimum',
    period: cdk.Duration.minutes(1),
  }),
  threshold: 1,
  comparisonOperator: cloudwatch.ComparisonOperator.LESS_THAN_THRESHOLD,
  evaluationPeriods: 1,
  treatMissingData: cloudwatch.TreatMissingData.BREACHING,  // critical — missing metric = outage
});

treatMissingData: BREACHING is non-negotiable. When all targets go unhealthy simultaneously, the ALB stops emitting HealthyHostCount entirely — without BREACHING, the alarm stays in OK state through a total outage. With BREACHING, missing data is treated as a breach and the alarm fires within the first evaluation period.

Latency alarm — catches experiments that degrade performance without killing the health check:

const latencyStop = new cloudwatch.Alarm(this, 'FisStopLatency', {
  alarmName: 'fis-stop-mcp-p99-latency',
  metric: new cloudwatch.Metric({
    namespace: 'AWS/ApplicationELB',
    metricName: 'TargetResponseTime',
    dimensionsMap: { LoadBalancer: alb.loadBalancerFullName },
    statistic: 'p99',
    period: cdk.Duration.minutes(1),
  }),
  threshold: 2,  // seconds — calibrate against your SLO minus client timeout buffer
  comparisonOperator: cloudwatch.ComparisonOperator.GREATER_THAN_THRESHOLD,
  evaluationPeriods: 1,
  treatMissingData: cloudwatch.TreatMissingData.NOT_BREACHING,
});

Set the threshold based on your MCP clients' actual timeout settings. If your client has a 5-second tool call timeout, 2-second p99 leaves a 3-second buffer. If you set the threshold too low (500ms), every CPU stress experiment immediately triggers the stop condition — calibrate against your SLO, not against normal-operations latency.

ECS running task count alarm — for Fargate, catches experiments where ECS is not replacing stopped tasks fast enough:

const taskCountStop = new cloudwatch.Alarm(this, 'FisStopTaskCount', {
  alarmName: 'fis-stop-mcp-running-task-count',
  metric: new cloudwatch.Metric({
    namespace: 'ECS/ContainerInsights',  // requires Container Insights enabled
    metricName: 'RunningTaskCount',
    dimensionsMap: { ClusterName: 'mcp-cluster', ServiceName: 'mcp-server' },
    statistic: 'Minimum',
    period: cdk.Duration.minutes(1),
  }),
  threshold: 1,
  comparisonOperator: cloudwatch.ComparisonOperator.LESS_THAN_THRESHOLD,
  evaluationPeriods: 1,
  treatMissingData: cloudwatch.TreatMissingData.BREACHING,
});

Container Insights must be enabled on the ECS cluster or this metric will not appear. Enable with containerInsights: true on the ECS cluster CDK construct. Without it, this alarm stays in INSUFFICIENT_DATA and will not stop experiments regardless of how many tasks are running.

Composite alarm: one stop condition ARN for all templates

Combine all three alarms into one composite alarm using OR logic — a single ARN to reference in every experiment template:

const fisStopCondition = new cloudwatch.CompositeAlarm(this, 'FisStopComposite', {
  compositeAlarmName: 'fis-stop-condition-mcp',
  alarmRule: cloudwatch.AlarmRule.anyOf(
    cloudwatch.AlarmRule.fromAlarm(availabilityStop, cloudwatch.AlarmState.ALARM),
    cloudwatch.AlarmRule.fromAlarm(latencyStop, cloudwatch.AlarmState.ALARM),
    cloudwatch.AlarmRule.fromAlarm(taskCountStop, cloudwatch.AlarmState.ALARM),
  ),
  actionsEnabled: false,  // suppress SNS — this alarm is only for FIS stop condition
});

Calibrating for FIS vs production alerting

Your production alarms (the ones that page you) use 5-minute periods with 3-of-5 consecutive breach requirements — this reduces alert fatigue by ignoring brief spikes during deploys. Your FIS stop condition alarms use 1-minute periods with 1 consecutive breach requirement. The goal is completely different: for PagerDuty alarms you want high confidence before waking someone up; for FIS stop conditions you want fast abort to minimize blast radius. Never share the same alarm ARN for both purposes — a 5-minute evaluation period lets an experiment run for up to 5 minutes causing damage before stopping. With dedicated 1-minute stop condition alarms, FIS aborts within 1–2 minutes of the fault exceeding threshold.

Pre-experiment health check

Always verify the stop condition alarm is in OK state before calling StartExperiment. FIS evaluates stop conditions at experiment initiation — if any stop condition is already in ALARM, FIS aborts immediately with "stop condition triggered at experiment start."

#!/bin/bash
set -euo pipefail

ALARM_NAME="fis-stop-condition-mcp"
ALARM_STATE=$(aws cloudwatch describe-alarms \
  --alarm-names "$ALARM_NAME" \
  --alarm-types "CompositeAlarm" \
  --query 'CompositeAlarms[0].StateValue' \
  --output text)

if [ "$ALARM_STATE" != "OK" ]; then
  echo "Stop condition alarm is $ALARM_STATE — resolve before running experiments"
  exit 1
fi

EXPERIMENT_ID=$(aws fis start-experiment \
  --experiment-template-id "$TEMPLATE_ID" \
  --query 'experiment.id' --output text)

echo "Experiment running: $EXPERIMENT_ID"

Putting it together: the MCP server chaos runbook

An FIS experiment without a runbook is a random fault injection. With a runbook, it is a hypothesis-driven reliability test. For every experiment against your MCP server, write down the following before running:

Hypothesis: State the expected behavior. "If the MCP server process is killed (SIGKILL), the ECS Fargate service will start a replacement task within 30 seconds and the AliveMCP probe will show at most one failed check."
Pre-conditions: Environment (staging only until verified safe), active AliveMCP probe on the endpoint, stop condition alarm in OK state, desiredCount ≥ 2.
Experiment: Which FIS template, which fault type, which target selection mode, what duration.
Monitoring: AliveMCP dashboard for probe failures, ECS service events for task replacement, CloudWatch alarm state.
Abort criteria: Stop condition alarm fires. Task not replaced after 60 seconds (manual abort). Any production alarm triggers.
Success criteria: Task replaced within 30 seconds. No more than 2 consecutive AliveMCP probe failures. Stop condition alarm does not fire. All in-flight tool calls receive JSON-RPC error responses (not bare connection drops).

Document the results. Failed experiments — where the hypothesis is wrong — are the most valuable. They reveal real production risk before users surface it. If ECS takes 45 seconds to replace a stopped task (not 30), you now have a concrete target: reduce the container startup time to under 20 seconds so recovery completes within the AliveMCP 2-consecutive-failure window. FIS gives you that measurement, repeatably, in staging.

The sequence for each FIS experiment set:

Deploy the experiment templates via CDK to staging
Run pre-flight alarm check (bash script above)
Start experiment, watch AliveMCP dashboard and CloudWatch alarm state simultaneously
Record recovery time metrics in the runbook
Compare to success criteria — if any criteria fails, file a reliability issue before promoting to production
After 3 successful runs in staging with the hypothesis confirmed, promote to a production experiment with PERCENT(10) selection mode and a tighter stop condition (lower threshold, shorter evaluation period)

Start with the task stop experiment — it requires no agent, is immediately reversible by ECS, and tests the highest-impact failure mode for Fargate-hosted MCP servers. Once you have confidence in task restart behavior, add the network port blackhole to test database failure handling. Add CPU stress last — it is the most disruptive experiment and the hardest to tune for environments where you don't yet have baseline latency measurements.

AliveMCP probes your MCP endpoint on a 60-second interval and surfaces recovery time measurements in the dashboard. Combined with FIS experiment logs (structured JSON in CloudWatch Logs, queryable via Logs Insights), you have full visibility into both the fault injection timeline and the user-visible impact — the two measurements you need to close the loop between chaos experiment results and reliability improvement decisions.