AWS FIS Experiment Templates for MCP Server Infrastructure

Template anatomy

The minimal experiment template JSON (before wrapping in CDK) looks like this:

{
  "description": "CPU stress on MCP server — verify process restart under load",
  "roleArn": "arn:aws:iam::123456789012:role/FisExperimentRole",

  "targets": {
    "McpInstances": {
      "resourceType": "aws:ec2:instance",
      "resourceTags": {
        "Role": "mcp-server",
        "ChaosEnabled": "true"
      },
      "selectionMode": "PERCENT(50)"
    }
  },

  "actions": {
    "CpuStress": {
      "actionId": "aws:ec2:cpu-stress",
      "parameters": {
        "duration": "PT5M",
        "percent": "80"
      },
      "targets": {
        "Instances": "McpInstances"
      }
    }
  },

  "stopConditions": [
    {
      "source": "aws:cloudwatch:alarm",
      "value": "arn:aws:cloudwatch:us-east-1:123456789012:alarm:McpServerAvailability"
    }
  ],

  "tags": {
    "Name": "mcp-cpu-stress",
    "Team": "platform"
  }
}

The targets key is a map of logical names (you choose the name, e.g., McpInstances) to target selectors. Actions reference these logical names in their targets sub-object. This indirection means you can reuse the same target selector across multiple actions in one template.

CPU stress action — aws:ec2:cpu-stress

aws:ec2:cpu-stress uses the SSM agent to run stress-ng --cpu N --cpu-load PERCENT on the target EC2 instances. It requires the SSM agent to be installed and running, and the instance profile must have the AmazonSSMManagedInstanceCore managed policy attached.

actions: {
  CpuStress: {
    actionId: 'aws:ec2:cpu-stress',
    description: 'Saturate 80% of CPU cores for 5 minutes to test MCP server behavior under load',
    parameters: {
      duration: 'PT5M',   // ISO 8601: PT5M = 5 minutes, PT30S = 30 seconds
      percent: '80',      // CPU utilization percent (0–100); uses all available cores
    },
    targets: { Instances: 'McpInstances' },
  },
},

What to observe during CPU stress:

• Does the MCP server's response latency increase proportionally, or does it spike nonlinearly?
• Do tool calls that call external APIs time out before the client disconnects?
• Does the Node.js event loop back up and prevent health checks from responding, causing ECS to kill and restart the task?
• After the fault ends (CPU returns to normal), do queued requests drain or are they lost?

A well-behaved MCP server under CPU stress should return JSON-RPC error responses with meaningful codes (e.g., -32603 Internal error with a "server overloaded, retry after Ns" message) rather than dropping connections silently.

Memory pressure action — aws:ec2:memory-stress

aws:ec2:memory-stress also uses SSM and stress-ng --vm to consume a percentage of available memory. This simulates the MCP server process growing its heap under load — a common pattern when tool results contain large payloads (e.g., embedding vectors, large database query results).

actions: {
  MemoryPressure: {
    actionId: 'aws:ec2:memory-stress',
    parameters: {
      duration: 'PT3M',
      percent: '70',    // consume 70% of total instance memory
      workers: '4',     // number of stress-ng --vm workers
    },
    targets: { Instances: 'McpInstances' },
  },
},

Memory stress experiments on MCP servers reveal OOM kill risk. If the kernel sends SIGKILL to your server process (OOM kill), the ECS task or EC2 instance will show an abnormal termination — the process won't get a chance to respond to in-flight requests. After the fault ends, verify:

• Did ECS restart the task automatically?
• Did the restart complete before your AliveMCP probe timeout?
• Were any tool call results corrupted (partial writes to DynamoDB, incomplete S3 uploads)?

If you observe OOM kills in staging, the fix is usually to set Node.js --max-old-space-size to a value below the container's memory limit, so Node.js triggers its own GC and returns a structured error before the kernel intervenes.

Network blackhole — aws:network:disrupt-connectivity

aws:network:disrupt-connectivity drops all network traffic to or from the target instances using traffic control (tc) rules applied via SSM. This simulates a network partition between the MCP server and its backend dependencies.

actions: {
  NetworkBlackhole: {
    actionId: 'aws:network:disrupt-connectivity',
    parameters: {
      duration: 'PT2M',
      scope: 'all',   // 'all' = bidirectional; 'egress' = outbound only; 'ingress' = inbound only
    },
    targets: { Instances: 'McpInstances' },
  },
},

For MCP servers, a full network blackhole (scope: all) tests total isolation. More interesting for MCP operators is scope: egress — this simulates the MCP server being unable to reach its backend (the database, Secrets Manager, external APIs) while the client can still send tool call requests. Observed outcomes to test:

• Does the MCP server return -32603 errors with a meaningful description, or just hang until the client connection times out?
• Does the connection pooling library (e.g., node-postgres, mysql2) detect the broken connection and surface an error, or does it queue requests indefinitely?
• Does Secrets Manager SDK respect connectTimeout settings, or does it hang for the full 60-second TCP timeout?

Process kill via SSM — aws:ssm:send-command

FIS does not have a dedicated "kill process" action, but you can simulate it with aws:ssm:send-command targeting a custom SSM document that sends SIGKILL to the MCP server process. This is the most realistic "crash" scenario — no graceful shutdown, no connection drain, no SIGTERM handling.

# First, create an SSM document for the process kill
# (Do this once; reference it in multiple experiment templates)
aws ssm create-document \
  --name 'KillMcpServerProcess' \
  --document-type 'Command' \
  --content '{
    "schemaVersion": "2.2",
    "description": "Send SIGKILL to the MCP server Node.js process",
    "mainSteps": [{
      "action": "aws:runShellScript",
      "name": "killMcpServer",
      "inputs": {
        "runCommand": [
          "pkill -SIGKILL -f \"node.*mcp-server\" || true",
          "echo \"Exit code: $?\""
        ],
        "timeoutSeconds": "10"
      }
    }]
  }' \
  --region us-east-1

// Reference the custom SSM document in the FIS experiment template
actions: {
  KillProcess: {
    actionId: 'aws:ssm:send-command',
    parameters: {
      documentArn: 'arn:aws:ssm:us-east-1:123456789012:document/KillMcpServerProcess',
      documentVersion: '$DEFAULT',
      duration: 'PT1M',   // how long to wait for the command to complete
    },
    targets: { Instances: 'McpInstances' },
  },
},

After a SIGKILL experiment, the key metric is time to recovery: how many seconds between the kill signal and the first successful AliveMCP probe response? If your ECS task takes 25 seconds to restart and your MCP clients have a 20-second timeout, every crash will produce a client-visible failure even after recovery. The fix: reduce ECS task startup time (smaller container image, pre-cached dependencies, faster health check grace period) or increase client retry timeout.

Sequencing actions with startAfter

Use startAfter to chain actions sequentially — for example, first inject CPU stress to degrade performance, then kill the process to test behavior when the server is already under load when it crashes.

actions: {
  // Step 1: apply CPU stress first (runs immediately at experiment start)
  CpuStress: {
    actionId: 'aws:ec2:cpu-stress',
    parameters: { duration: 'PT2M', percent: '90' },
    targets: { Instances: 'McpInstances' },
  },

  // Step 2: kill process AFTER CpuStress has been running for 2 minutes
  KillProcess: {
    actionId: 'aws:ssm:send-command',
    parameters: {
      documentArn: 'arn:aws:ssm:...:document/KillMcpServerProcess',
      documentVersion: '$DEFAULT',
      duration: 'PT30S',
    },
    targets: { Instances: 'McpInstances' },
    startAfter: ['CpuStress'],  // wait for CpuStress to COMPLETE before starting
  },
},

startAfter takes a list of action names. FIS starts the dependent action only after all listed actions have reached completed state (not just started). Actions without startAfter run in parallel from experiment start. Plan your total experiment duration to be the sum of all sequential action durations.

Parameterizing templates with experiment options

FIS experiment templates support experimentOptions to override key behavior at runtime without changing the template definition. Use this to vary target selection mode (smaller blast radius for first runs) or duration:

// In the CDK template definition, declare experiment options
const template = new fis.CfnExperimentTemplate(this, 'McpCpuStress', {
  // ... targets, actions, stopConditions ...

  experimentOptions: {
    accountTargeting: 'single-account',
    emptyTargetResolutionMode: 'skip',   // skip the experiment if no targets found (vs 'fail')
  },
});

// When starting the experiment, you can override target count at runtime:
aws fis start-experiment \
  --experiment-template-id ET123 \
  --experiment-options '{"actionsMode": "run-all"}'

Set emptyTargetResolutionMode: 'skip' when running experiments in CI pipelines where the target environment may not have any MCP server instances running (e.g., a PR environment that was torn down). This prevents the experiment from failing the CI pipeline with an error when the target doesn't exist.

Common failures