AWS FIS Stop Conditions — CloudWatch Alarm Kill Switch for MCP Server Chaos

What happens when a stop condition triggers

When the stop condition alarm fires, FIS:

1. Sets the experiment state to stopping
2. Sends a rollback command to each running action (e.g., removes CPU stress via SSM, removes network blackhole tc rules, reverses the ECS task stop by allowing ECS service replacement)
3. Waits up to 10 minutes for all actions to confirm rollback completion
4. Sets the experiment state to stopped

Not all actions are reversible by FIS. aws:ecs:stop-task cannot "un-stop" a task — ECS will restart the task as part of its normal service maintenance, but FIS does not control that. aws:ec2:terminate-instances is similarly irreversible. For actions that are truly irreversible, the stop condition prevents FIS from starting additional actions (if the template has sequential steps) but cannot undo what has already happened.

This is why selectionMode: PERCENT(50) and desiredCount: 2+ matter so much — they limit blast radius so that even irreversible actions (like task stop) don't cause total outage.

Availability alarm stop condition

The most direct stop condition for an MCP server: a CloudWatch alarm on the ALB HealthyHostCount metric that fires when too few targets are healthy.

import * as cloudwatch from 'aws-cdk-lib/aws-cloudwatch';
import * as elbv2 from 'aws-cdk-lib/aws-elasticloadbalancingv2';

// Reference the ALB and target group
const alb = elbv2.ApplicationLoadBalancer.fromLookup(this, 'McpAlb', {
  loadBalancerArn: 'arn:aws:elasticloadbalancing:...',
});

// Alarm: fire when fewer than 1 healthy host is in the target group
// (for a 2-task deployment, this means both tasks are unhealthy)
const healthyHostAlarm = new cloudwatch.Alarm(this, 'McpHealthyHostStop', {
  alarmName: 'fis-stop-mcp-healthy-host-count',
  metric: new cloudwatch.Metric({
    namespace: 'AWS/ApplicationELB',
    metricName: 'HealthyHostCount',
    dimensionsMap: {
      LoadBalancer: alb.loadBalancerFullName,
      TargetGroup: mcpTargetGroup.targetGroupFullName,
    },
    statistic: 'Minimum',
    period: cdk.Duration.minutes(1),
  }),
  // Fire if minimum healthy hosts drops below 1 for any single 1-minute period
  threshold: 1,
  comparisonOperator: cloudwatch.ComparisonOperator.LESS_THAN_THRESHOLD,
  evaluationPeriods: 1,
  treatMissingData: cloudwatch.TreatMissingData.BREACHING,
});

The treatMissingData: BREACHING setting is critical for stop conditions. If the ALB stops emitting the HealthyHostCount metric (because all targets are unhealthy and the ALB has no data to report), you want the alarm to fire — not to remain in OK state due to missing data. Without this, a catastrophic failure where all targets die simultaneously would not trigger the stop condition.

Latency alarm stop condition

Add a latency-based stop condition to catch experiments that degrade performance without causing complete failure. A CPU stress experiment that pushes p99 latency to 10 seconds may not kill the ALB health check (which uses a 5-second timeout by default) but it will make your MCP server unusable in practice.

const latencyAlarm = new cloudwatch.Alarm(this, 'McpLatencyStop', {
  alarmName: 'fis-stop-mcp-p99-latency',
  metric: new cloudwatch.Metric({
    namespace: 'AWS/ApplicationELB',
    metricName: 'TargetResponseTime',
    dimensionsMap: {
      LoadBalancer: alb.loadBalancerFullName,
    },
    statistic: 'p99',
    period: cdk.Duration.minutes(1),
  }),
  // Stop the experiment if p99 latency exceeds 2 seconds for 1 consecutive period
  threshold: 2,  // seconds
  comparisonOperator: cloudwatch.ComparisonOperator.GREATER_THAN_THRESHOLD,
  evaluationPeriods: 1,
  treatMissingData: cloudwatch.TreatMissingData.NOT_BREACHING,
});

Set the latency threshold based on your MCP clients' actual timeout settings. If your MCP client has a 5-second tool call timeout, a 2-second p99 latency still leaves a 3-second buffer. If you set the threshold too low (e.g., 500ms), every experiment with any CPU stress will immediately trigger the stop condition — calibrate against your SLO, not against normal-operation latency.

ECS running task count alarm

For Fargate-hosted MCP servers, add a stop condition that fires when ECS running task count drops below your minimum acceptable count. This catches experiments where ECS is not replacing stopped tasks fast enough.

const runningTaskAlarm = new cloudwatch.Alarm(this, 'McpRunningTaskStop', {
  alarmName: 'fis-stop-mcp-running-task-count',
  metric: new cloudwatch.Metric({
    namespace: 'ECS/ContainerInsights',
    metricName: 'RunningTaskCount',
    dimensionsMap: {
      ClusterName: 'mcp-cluster',
      ServiceName: 'mcp-server',
    },
    statistic: 'Minimum',
    period: cdk.Duration.minutes(1),
  }),
  // Fire if running task count drops below 1 for any single period
  // (for desiredCount: 2, this means we lost both tasks — stop immediately)
  threshold: 1,
  comparisonOperator: cloudwatch.ComparisonOperator.LESS_THAN_THRESHOLD,
  evaluationPeriods: 1,
  treatMissingData: cloudwatch.TreatMissingData.BREACHING,
});

ECS/ContainerInsights metrics require Container Insights to be enabled on the ECS cluster. Enable it with containerInsights: true on the ECS cluster CDK construct or via aws ecs update-cluster-settings --cluster mcp-cluster --settings name=containerInsights,value=enabled. Without Container Insights, the RunningTaskCount metric will not appear and the alarm will stay in INSUFFICIENT_DATA state — which will not stop experiments.

Composite alarm — single stop condition for all metrics

FIS experiment templates accept a list of stop conditions but each must be a CloudWatch alarm ARN. Rather than listing three separate alarms in every template, create one composite alarm that combines all your stop signals. The composite alarm fires when any child alarm fires (using OR logic via ALARM(child1) OR ALARM(child2)).

const fisStopCondition = new cloudwatch.CompositeAlarm(this, 'FisStopComposite', {
  compositeAlarmName: 'fis-stop-condition-mcp',
  alarmRule: cloudwatch.AlarmRule.anyOf(
    cloudwatch.AlarmRule.fromAlarm(healthyHostAlarm, cloudwatch.AlarmState.ALARM),
    cloudwatch.AlarmRule.fromAlarm(latencyAlarm, cloudwatch.AlarmState.ALARM),
    cloudwatch.AlarmRule.fromAlarm(runningTaskAlarm, cloudwatch.AlarmState.ALARM),
  ),
  // Suppress actions on the composite alarm itself (we only use it for FIS stop condition)
  actionsEnabled: false,
});

// All experiment templates use the same composite alarm ARN
const experiments = [cpuStressTemplate, memoryStressTemplate, taskStopTemplate];
experiments.forEach((tpl) => {
  // Update each template to use the composite alarm as stop condition
  // (In CDK, set stopConditions in the template definition — shown here conceptually)
});

Composite alarms cannot reference other composite alarms as children — only metric alarms. If you want to nest logic (e.g., "stop if healthy host count is low AND it has been low for more than 2 periods"), use a separate metric alarm with the appropriate evaluationPeriods and datapointsToAlarm settings, then include that alarm in the composite.

Calibrating alarm sensitivity for chaos experiments

Your production alarms (the ones that page you at 3am) should have conservative evaluation periods — 5 minutes with 3 out of 5 consecutive breaches — to reduce false positive alerts. FIS stop condition alarms should be more sensitive: 1-minute period with 1 consecutive breach. The goal is different: for PagerDuty alarms you want high confidence before waking someone up; for FIS stop conditions you want fast abort to minimize experiment blast radius.

Use separate alarm definitions for FIS stop conditions and production alerting — do not share the same alarm ARN for both purposes. A shared alarm with a 5-minute evaluation period will let an experiment run for up to 5 minutes causing damage before stopping. With a 1-minute period dedicated stop condition alarm, FIS aborts within 1–2 minutes of the fault exceeding the threshold.

# Production alarm (5-min period, 3-of-5 breaches) — pages on-call
aws cloudwatch put-metric-alarm \
  --alarm-name mcp-healthy-host-prod \
  --metric-name HealthyHostCount \
  --namespace AWS/ApplicationELB \
  --statistic Minimum \
  --period 300 \
  --threshold 1 \
  --comparison-operator LessThanThreshold \
  --evaluation-periods 3 \
  --datapoints-to-alarm 3 \
  --treat-missing-data breaching \
  --alarm-actions arn:aws:sns:...:pagerduty-topic

# FIS stop condition alarm (1-min period, 1 breach) — fast abort
aws cloudwatch put-metric-alarm \
  --alarm-name fis-stop-mcp-healthy-host \
  --metric-name HealthyHostCount \
  --namespace AWS/ApplicationELB \
  --statistic Minimum \
  --period 60 \
  --threshold 1 \
  --comparison-operator LessThanThreshold \
  --evaluation-periods 1 \
  --datapoints-to-alarm 1 \
  --treat-missing-data breaching
  # No alarm-actions needed — FIS reads the state directly

Pre-experiment health check

Always verify the stop condition alarm is in OK state before starting an experiment. FIS evaluates stop conditions at experiment initiation — if any stop condition alarm is in ALARM state when you call StartExperiment, FIS aborts the experiment immediately with the reason "stop condition triggered at experiment start."

#!/bin/bash
# Pre-flight check before running FIS experiment
set -euo pipefail

ALARM_NAME="fis-stop-condition-mcp"
TEMPLATE_ID="ET123456789"

echo "Checking stop condition alarm state..."
ALARM_STATE=$(aws cloudwatch describe-alarms \
  --alarm-names "$ALARM_NAME" \
  --query 'MetricAlarms[0].StateValue || CompositeAlarms[0].StateValue' \
  --output text 2>/dev/null || echo "COMPOSITE")

# For composite alarms, use a different query
if [ "$ALARM_STATE" = "COMPOSITE" ] || [ "$ALARM_STATE" = "None" ]; then
  ALARM_STATE=$(aws cloudwatch describe-alarms \
    --alarm-names "$ALARM_NAME" \
    --alarm-types "CompositeAlarm" \
    --query 'CompositeAlarms[0].StateValue' \
    --output text)
fi

if [ "$ALARM_STATE" != "OK" ]; then
  echo "ERROR: Stop condition alarm is in $ALARM_STATE state — aborting experiment"
  echo "Resolve the alarm before running chaos experiments"
  exit 1
fi

echo "Alarm is OK — starting experiment"
EXPERIMENT_ID=$(aws fis start-experiment \
  --experiment-template-id "$TEMPLATE_ID" \
  --query 'experiment.id' \
  --output text)

echo "Experiment started: $EXPERIMENT_ID"
echo "Monitor at: https://console.aws.amazon.com/fis/home#/experiments/$EXPERIMENT_ID"

Incorporate this pre-flight check into your CI pipeline or runbook. Never run FIS experiments when you have active alarms or degraded service — chaos experiments on already-degraded infrastructure compound the blast radius and make it impossible to attribute failures to the experiment versus pre-existing issues.

Error budget stop condition

If you track error budgets for your MCP server (the standard SRE practice of allowing a fixed percentage of downtime per month), you can create a stop condition that halts experiments when the remaining error budget for the current month drops below a threshold. This prevents chaos experiments from burning through your monthly error budget in a single day.

# CloudWatch metric math: compute remaining error budget as a percentage
# Assumes you have a custom metric 'McpProbeFailureCount' and 'McpProbeTotalCount'
# published by your monitoring tool (e.g., from AliveMCP probe results)

aws cloudwatch put-metric-alarm \
  --alarm-name fis-stop-error-budget-mcp \
  --alarm-description "Stop FIS experiments when error budget burns too fast" \
  --metrics '[
    {
      "Id": "failures",
      "MetricStat": {
        "Metric": {
          "Namespace": "AliveMCP",
          "MetricName": "ProbeFailureCount",
          "Dimensions": [{"Name": "Endpoint", "Value": "alivemcp.com/status/mcp-server-slug"}]
        },
        "Period": 300,
        "Stat": "Sum"
      },
      "ReturnData": false
    },
    {
      "Id": "total",
      "MetricStat": {
        "Metric": {
          "Namespace": "AliveMCP",
          "MetricName": "ProbeTotalCount",
          "Dimensions": [{"Name": "Endpoint", "Value": "alivemcp.com/status/mcp-server-slug"}]
        },
        "Period": 300,
        "Stat": "Sum"
      },
      "ReturnData": false
    },
    {
      "Id": "error_rate",
      "Expression": "100 * failures / total",
      "Label": "Error Rate %",
      "ReturnData": true
    }
  ]' \
  --threshold 1.5 \
  --comparison-operator GreaterThanThreshold \
  --evaluation-periods 1 \
  --treat-missing-data notBreaching

An error rate alarm threshold of 1.5% means: "stop the experiment if the 5-minute error rate exceeds 1.5%." For a server with a 99.9% monthly uptime SLO (43 minutes of allowed downtime per month), burning 1.5% in 5 minutes is 450 seconds — roughly 10% of your entire monthly budget. Setting this as a chaos experiment limit ensures you don't blow the SLO doing reliability testing.

Common failures