MCP Server Fault Injection Simulator — AWS FIS Chaos Engineering for Uptime

Why MCP servers need chaos engineering

MCP servers fail in ways that unit tests and integration tests do not catch. The MCP protocol is stateful — a client connects, negotiates capabilities, and then issues tool calls against the same persistent connection. When the server process crashes mid-conversation, most clients will not automatically reconnect and resume; they report the tool as unavailable, silently fail the agent workflow, or surface a confusing error to the end user. The developer only learns about the failure when a user complains or when an external monitoring service (like AliveMCP) detects the endpoint is down.

Common failure modes that chaos engineering surfaces before users do:

• Process OOM kill: The MCP server process is killed by the kernel under memory pressure. The ECS task or systemd unit may restart it — but does the restart complete within the MCP timeout? Do in-flight tool calls get retried, or silently dropped?
• CPU saturation: A single tool handler that calls an expensive external API can starve all other concurrent requests. Under CPU stress, does your MCP server return meaningful errors or just hang until the client times out?
• Network partition: The MCP server can reach the client but cannot reach its backend dependencies (database, secrets manager, external API). Does it return a structured JSON-RPC error or just drop the connection?
• Partial availability: Some tools return correctly while others fail. Does your server mark those tools as unavailable or silently return malformed results that confuse the agent?

AWS FIS lets you inject each of these conditions in a controlled, time-bounded, automatically-reversible way — with a kill switch tied to your CloudWatch alarms so you can't accidentally take yourself down permanently.

Core FIS concepts

An FIS experiment has four required components:

• Targets: The AWS resources FIS will inject faults into. Targets can be selected by resource ID, resource tag, or a resource filter. For MCP servers, you will typically target EC2 instances or ECS tasks by an Environment=staging or Role=mcp-server tag. You can limit blast radius with selectionMode: use PERCENT(50) to target only half your fleet rather than all instances at once.
• Actions: The fault to inject. FIS provides built-in actions organized by service namespace (e.g., aws:ec2:cpu-stress, aws:ecs:stop-task, aws:ssm:send-command). Actions have parameters (duration, percent CPU load, etc.) and can run in sequence or parallel. Each action has a start delay so you can stagger faults in multi-step experiments.
• Stop conditions: CloudWatch alarm ARNs. FIS polls these alarms every 15 seconds during the experiment. If any alarm transitions to ALARM state, FIS immediately stops all injected actions (reverting CPU stress, restoring network routes, etc.) and marks the experiment result as stopped. You should always wire at least one stop condition — without one, FIS still runs but you have no automatic kill switch.
• IAM role: The IAM role FIS assumes to inject faults. It needs permissions to call the target service APIs (e.g., ec2:StopInstances, ecs:StopTask, ssm:SendCommand) and to describe CloudWatch alarms. Create a dedicated FIS role — do not reuse Lambda execution roles or deployment roles.

IAM role for FIS

FIS requires an IAM role with a trust policy allowing fis.amazonaws.com to assume it. The permissions you need depend on which actions you run. A minimal role for EC2 and ECS experiments:

import * as iam from 'aws-cdk-lib/aws-iam';

const fisRole = new iam.Role(this, 'FisRole', {
  assumedBy: new iam.ServicePrincipal('fis.amazonaws.com'),
  description: 'FIS role for MCP server chaos experiments',
});

// Allow FIS to inject faults into EC2 instances
fisRole.addToPolicy(new iam.PolicyStatement({
  actions: [
    'ec2:DescribeInstances',
    'ec2:StopInstances',
    'ec2:RebootInstances',
    'ec2:TerminateInstances',
  ],
  resources: ['*'],
  conditions: {
    // Only target instances tagged for chaos testing
    StringEquals: { 'ec2:ResourceTag/ChaosEnabled': 'true' },
  },
}));

// Allow FIS to stop ECS tasks (Fargate and EC2 launch types)
fisRole.addToPolicy(new iam.PolicyStatement({
  actions: [
    'ecs:DescribeTasks',
    'ecs:ListTasks',
    'ecs:StopTask',
  ],
  resources: ['*'],
}));

// Allow FIS to send SSM commands (for CPU/memory stress via stress-ng)
fisRole.addToPolicy(new iam.PolicyStatement({
  actions: [
    'ssm:SendCommand',
    'ssm:GetCommandInvocation',
    'ssm:ListCommands',
    'ssm:ListCommandInvocations',
    'ssm:DescribeInstanceInformation',
  ],
  resources: ['*'],
}));

// Allow FIS to read CloudWatch alarms for stop conditions
fisRole.addToPolicy(new iam.PolicyStatement({
  actions: [
    'cloudwatch:DescribeAlarms',
  ],
  resources: ['*'],
}));

// Allow FIS to write experiment logs to CloudWatch Logs
fisRole.addToPolicy(new iam.PolicyStatement({
  actions: [
    'logs:CreateLogDelivery',
    'logs:PutLogEvents',
    'logs:DescribeLogGroups',
  ],
  resources: ['*'],
}));

Scope the ec2:StopInstances and ec2:TerminateInstances permissions with tag conditions — you do not want FIS to accidentally target instances that do not belong to your MCP server fleet. Add a ChaosEnabled=true tag to every instance you intentionally expose to FIS experiments, and use that tag in the resource policy condition.

Creating your first experiment template via CDK

Use aws-cdk-lib/aws-fis to define experiment templates as code. This ensures your chaos experiments are version-controlled, reviewable, and consistent across environments.

import * as fis from 'aws-cdk-lib/aws-fis';
import * as cloudwatch from 'aws-cdk-lib/aws-cloudwatch';
import * as cdk from 'aws-cdk-lib';

// Existing CloudWatch alarm for MCP server availability
const availabilityAlarm = cloudwatch.Alarm.fromAlarmArn(
  this,
  'McpAvailabilityAlarm',
  'arn:aws:cloudwatch:us-east-1:123456789012:alarm:McpServerAvailability',
);

// FIS experiment: 3-minute CPU stress on the MCP server EC2 instance
const cpuStressTemplate = new fis.CfnExperimentTemplate(this, 'McpCpuStress', {
  description: 'Inject 80% CPU load on MCP server for 3 minutes; verify restart and reconnect',
  roleArn: fisRole.roleArn,

  targets: {
    McpServerInstances: {
      resourceType: 'aws:ec2:instance',
      resourceTags: { Role: 'mcp-server', ChaosEnabled: 'true' },
      selectionMode: 'PERCENT(50)',  // target only half the fleet
    },
  },

  actions: {
    CpuStress: {
      actionId: 'aws:ec2:cpu-stress',
      description: 'Apply 80% CPU load for 3 minutes',
      parameters: {
        duration: 'PT3M',   // ISO 8601 duration
        percent: '80',
      },
      targets: { Instances: 'McpServerInstances' },
    },
  },

  stopConditions: [
    {
      source: 'aws:cloudwatch:alarm',
      value: availabilityAlarm.alarmArn,
    },
  ],

  tags: {
    Name: 'mcp-cpu-stress',
    Environment: 'staging',
  },

  logConfiguration: {
    cloudWatchLogsConfiguration: {
      logGroupArn: `arn:aws:logs:us-east-1:123456789012:log-group:/fis/mcp-experiments:*`,
    },
    logSchemaVersion: 2,
  },
});

The selectionMode: 'PERCENT(50)' tells FIS to randomly select 50% of matching instances as targets. For a single-instance staging environment this effectively means "one instance," but as you scale up to a fleet of MCP servers the same template safely limits blast radius without code changes.

Enabling FIS experiment logs

FIS can emit experiment event logs to Amazon CloudWatch Logs, Amazon S3, or both. Enable CloudWatch Logs logging for every experiment — the logs record each action's start/stop time, the resources targeted, any stop condition triggers, and error messages when FIS cannot inject a fault (e.g., SSM agent not running, insufficient IAM permissions).

# Create the log group first (FIS does not create it automatically)
aws logs create-log-group \
  --log-group-name /fis/mcp-experiments \
  --region us-east-1

# FIS log schema version 2 emits structured JSON per event
# Log entries look like:
# {
#   "version": "1.0",
#   "time": "2026-11-07T14:23:01Z",
#   "type": "action-state-change",
#   "data": {
#     "experimentId": "EXPT123",
#     "actionId": "CpuStress",
#     "state": "running",
#     "reason": "Action started successfully"
#   }
# }

# Query FIS experiment logs via CloudWatch Logs Insights
aws logs start-query \
  --log-group-name /fis/mcp-experiments \
  --start-time $(date -d '1 hour ago' +%s) \
  --end-time $(date +%s) \
  --query-string 'fields @timestamp, type, data.state, data.reason
    | filter type = "action-state-change"
    | sort @timestamp desc
    | limit 50'

Log schema version 2 is preferred over version 1 — it provides structured JSON events that are easier to query in Logs Insights. Version 1 (plain text) is kept for backwards compatibility but does not support Logs Insights filtering on data.* fields.

Running an experiment and reading results

Start an experiment from the template and monitor it via the FIS console or CLI. FIS returns an experiment ID immediately; the experiment transitions through states: initiating → running → completed (or stopped if a stop condition triggered).

# Start the experiment
EXPERIMENT_ID=$(aws fis start-experiment \
  --experiment-template-id ET123456789 \
  --query 'experiment.id' \
  --output text)

echo "Experiment running: $EXPERIMENT_ID"

# Poll status (experiment runs for ~3 minutes)
watch -n 15 "aws fis get-experiment \
  --id $EXPERIMENT_ID \
  --query 'experiment.state' \
  --output json"

# Full experiment result after completion
aws fis get-experiment --id $EXPERIMENT_ID

The state field in the result tells you the outcome:

• completed — FIS ran the full experiment duration with no stop condition trigger. This means the fault was injected and ran to completion. Check whether your MCP server recovered as expected (e.g., process restarted, health check passed).
• stopped — A stop condition alarm fired during the experiment. FIS aborted and rolled back all actions. This is the desired outcome if the fault was too severe — your alarm caught it before the experiment caused unacceptable downtime.
• failed — FIS could not start the experiment (IAM permissions missing, target not found, SSM agent not running). Check the FIS console error message and review the CloudWatch Logs for the experiment.

Tagging instances for FIS targeting

FIS targets resources by tag. Add two tags to every resource you want to expose to chaos experiments: a role tag (so FIS can distinguish your MCP server instances from your database instances) and a ChaosEnabled=true tag as an explicit opt-in. The opt-in tag pattern prevents accidents — if you forget to tag a new resource, FIS ignores it.

# Add chaos tags via CDK to your MCP server EC2 instance or ECS task
const mcpInstance = new ec2.Instance(this, 'McpServer', {
  // ... instance config ...
});

cdk.Tags.of(mcpInstance).add('Role', 'mcp-server');
cdk.Tags.of(mcpInstance).add('ChaosEnabled', 'true');
cdk.Tags.of(mcpInstance).add('Environment', 'staging');

// For ECS tasks, add tags to the ECS service:
const mcpService = new ecs.FargateService(this, 'McpService', {
  // ... service config ...
  taskDefinition: mcpTaskDef,
  propagateTags: ecs.PropagatedTagSource.SERVICE,  // propagate tags to tasks
});

cdk.Tags.of(mcpService).add('Role', 'mcp-server');
cdk.Tags.of(mcpService).add('ChaosEnabled', 'true');

The propagateTags: PropagatedTagSource.SERVICE setting on Fargate services ensures that the ECS tasks inherit the service's tags. FIS targets ECS tasks, not services, so the tags must be present on the task resource itself — not just the service or task definition.

Building a chaos experiment runbook

An FIS experiment without a runbook is a random fault injection. With a runbook, it is a hypothesis-driven reliability test. For each MCP server experiment, write down:

1. Hypothesis: "If the MCP server process is killed (SIGKILL), ECS Fargate will restart it within 30 seconds and the AliveMCP probe will show at most one failed check."
2. Pre-conditions: Staging environment. At least one active AliveMCP probe on the endpoint. CloudWatch availability alarm threshold set at < 99% over 1-minute period.
3. Experiment: FIS aws:ecs:stop-task targeting Role=mcp-server, ChaosEnabled=true. Duration: immediate (task stop is instantaneous).
4. Monitoring: Watch AliveMCP dashboard for probe failures. Watch ECS service events for task restart. Watch CloudWatch alarm state.
5. Abort criteria: Alarm fires (stop condition). Task has not restarted after 60 seconds (manual abort).
6. Success criteria: Task restarts within 30 seconds. No more than 2 consecutive AliveMCP probe failures. Alarm does not fire.

Document the results in your incident runbook. Failed experiments (where the hypothesis is proven wrong) are the most valuable — they reveal real production risk before your users surface it.

Common failures