Guide · AWS CloudWatch · Alerting

CloudWatch Composite Alarms for MCP Servers — AND/OR Health Gates and Alert Suppression

CloudWatch Composite Alarms evaluate boolean expressions over multiple child alarms — standard metric alarms, anomaly detection alarms, and other composite alarms — and fire a single action only when the combined condition is true. For MCP server operators, composite alarms solve two critical problems that simple per-metric alarms cannot: alert fatigue from independent signals firing separately, and false positives from noisy metrics alarming during expected maintenance windows. A composite alarm can express "notify my on-call only if BOTH latency is degraded AND error rate is elevated, AND we are NOT currently in a deployment window" — logic that is impossible to express in a single CloudWatch metric alarm. For the underlying individual alarms, see CloudWatch Alarms for MCP servers; for dynamic thresholds that remove the need for manual p99 targets, see CloudWatch Anomaly Detection for MCP servers.

TL;DR

Create standard metric alarms for each signal (latency, error rate, DLQ depth). Then create a composite alarm with --alarm-rule "ALARM(\"p99-latency\") OR ALARM(\"error-rate\") OR ALARM(\"dlq-depth\")" and attach your SNS topic to the composite, not to the individual alarms. For maintenance suppression, create a "maintenance-window" alarm that you transition manually to ALARM state before deployments, then wrap your composite rule with AND NOT ALARM("maintenance-window"). This way, your SNS pager goes silent during planned downtime and rings only during true incidents.

How composite alarms evaluate state

A composite alarm has an alarm rule — a boolean expression over child alarm states. CloudWatch evaluates the rule every time any child alarm changes state. If the rule evaluates to true, the composite alarm enters ALARM state. If the rule evaluates to false, it enters OK state. Unlike standard metric alarms, composite alarms do not have their own evaluation periods or data points — they re-evaluate immediately whenever a child state changes.

The rule language supports:

ALARM("alarm-name") — true if the named alarm is in ALARM state
OK("alarm-name") — true if the named alarm is in OK state
INSUFFICIENT_DATA("alarm-name") — true if the named alarm is in INSUFFICIENT_DATA state
AND, OR, NOT — boolean operators
Parentheses for grouping

Child alarms referenced in a composite rule can be standard metric alarms, anomaly detection alarms, or other composite alarms. You can nest composite alarms up to a depth of 4 levels. Composite alarms themselves cannot have datapoints_to_alarm or evaluation period settings — those live on the individual metric alarms.

Pattern 1: Single pager for correlated signals

MCP server failures typically manifest as a cluster of correlated signals: p99 latency rises, error rate rises, and the dead-letter queue starts filling — all at the same time. If you attach SNS notifications to each alarm independently, an on-call engineer receives 3 simultaneous pages for what is actually one incident. A composite OR alarm collapses these into a single page:

# Step 1: Create individual metric alarms (no SNS actions — composite will own notification)
aws cloudwatch put-metric-alarm \
  --alarm-name "mcp-p99-latency" \
  --namespace "McpServer" \
  --metric-name "McpToolDuration" \
  --extended-statistic p99 \
  --dimensions Name=FunctionName,Value=mcp-server-prod \
  --period 60 \
  --evaluation-periods 3 \
  --threshold 2000 \
  --comparison-operator GreaterThanOrEqualToThreshold \
  --treat-missing-data notBreaching

aws cloudwatch put-metric-alarm \
  --alarm-name "mcp-error-rate" \
  --namespace "McpServer" \
  --metric-name "McpToolErrors" \
  --statistic Sum \
  --dimensions Name=FunctionName,Value=mcp-server-prod \
  --period 60 \
  --evaluation-periods 2 \
  --threshold 10 \
  --comparison-operator GreaterThanOrEqualToThreshold \
  --treat-missing-data notBreaching

aws cloudwatch put-metric-alarm \
  --alarm-name "mcp-dlq-depth" \
  --namespace "AWS/SQS" \
  --metric-name "ApproximateNumberOfMessagesVisible" \
  --statistic Maximum \
  --dimensions Name=QueueName,Value=mcp-probe-dlq \
  --period 60 \
  --evaluation-periods 1 \
  --threshold 1 \
  --comparison-operator GreaterThanOrEqualToThreshold \
  --treat-missing-data notBreaching

# Step 2: Create composite alarm — page on any signal
aws cloudwatch put-composite-alarm \
  --alarm-name "mcp-health-gate" \
  --alarm-description "Any MCP health signal degraded" \
  --alarm-rule 'ALARM("mcp-p99-latency") OR ALARM("mcp-error-rate") OR ALARM("mcp-dlq-depth")' \
  --alarm-actions arn:aws:sns:us-east-1:123456789012:mcp-oncall-pager \
  --ok-actions arn:aws:sns:us-east-1:123456789012:mcp-oncall-pager

The SNS topic is attached only to the composite alarm. The individual metric alarms have no actions. This means one page per incident, not three — and the composite OK action fires once the health gate clears, sending a single "all clear" notification.

Pattern 2: Maintenance window suppression

During a deployment or planned maintenance, your MCP server will legitimately fail health checks and trigger latency/error alarms. Without suppression, your on-call receives pages for a known event, trains alert fatigue, and starts ignoring pages — a dangerous habit. Composite alarms provide a manual suppression mechanism:

# Create a "maintenance window" alarm that starts OK and is manually set to ALARM during deployments
# This alarm has no metric — it uses a static threshold that will never trigger automatically
# We set it to ALARM manually with set-alarm-state before deployments
aws cloudwatch put-metric-alarm \
  --alarm-name "mcp-maintenance-window" \
  --alarm-description "Set to ALARM manually before deployments to suppress on-call pages" \
  --namespace "AWS/CloudWatch" \
  --metric-name "NumberOfMetricFilters" \
  --statistic Average \
  --period 3600 \
  --evaluation-periods 1 \
  --threshold 999999 \
  --comparison-operator GreaterThanThreshold \
  --treat-missing-data notBreaching
  # This alarm will never enter ALARM state automatically (threshold too high)
  # Use set-alarm-state to flip it manually

# Composite alarm that suppresses during maintenance
aws cloudwatch put-composite-alarm \
  --alarm-name "mcp-health-gate-suppressed" \
  --alarm-rule 'ALARM("mcp-health-gate") AND NOT ALARM("mcp-maintenance-window")' \
  --alarm-actions arn:aws:sns:us-east-1:123456789012:mcp-oncall-pager \
  --ok-actions arn:aws:sns:us-east-1:123456789012:mcp-oncall-pager

# Before deployment: suppress pager
aws cloudwatch set-alarm-state \
  --alarm-name mcp-maintenance-window \
  --state-value ALARM \
  --state-reason "Planned deployment $(date -u +%Y-%m-%dT%H:%M:%SZ)"

# After deployment: restore pager
aws cloudwatch set-alarm-state \
  --alarm-name mcp-maintenance-window \
  --state-value OK \
  --state-reason "Deployment complete"

During the window where mcp-maintenance-window is in ALARM state, the composite rule evaluates to false even if all health signals are alarming — no pages. After the deployment completes and you restore mcp-maintenance-window to OK, the composite re-evaluates immediately and pages if any signal is still degraded.

Pattern 3: Severity tiering with AND logic

Some alerts warrant an immediate page, while others should only alert during business hours via a Slack notification. Composite alarms with AND logic let you express "this is a critical incident only if MULTIPLE signals are simultaneously degraded" — which is a better proxy for severity than any single metric:

# Critical: both latency AND errors degraded simultaneously (true incident, not flap)
aws cloudwatch put-composite-alarm \
  --alarm-name "mcp-critical-incident" \
  --alarm-description "Both latency and errors degraded — probable total outage" \
  --alarm-rule 'ALARM("mcp-p99-latency") AND ALARM("mcp-error-rate")' \
  --alarm-actions arn:aws:sns:us-east-1:123456789012:mcp-pagerduty-critical

# Warning: any single signal degraded (notify Slack, do not page)
aws cloudwatch put-composite-alarm \
  --alarm-name "mcp-warning-indicator" \
  --alarm-description "Single signal degraded — monitor but do not page" \
  --alarm-rule '(ALARM("mcp-p99-latency") OR ALARM("mcp-error-rate") OR ALARM("mcp-dlq-depth")) AND NOT ALARM("mcp-critical-incident")' \
  --alarm-actions arn:aws:sns:us-east-1:123456789012:mcp-slack-warnings

The mcp-warning-indicator rule fires when any single signal is alarming but the critical composite is not — i.e., exactly one signal is degraded. This two-tier approach pages on correlations (likely real incidents) and notifies without paging on isolated signals (likely transient flaps or single-signal issues).

Composite alarms in CDK

import * as cloudwatch from 'aws-cdk-lib/aws-cloudwatch';
import * as cw_actions from 'aws-cdk-lib/aws-cloudwatch-actions';
import * as sns from 'aws-cdk-lib/aws-sns';

const oncallTopic = sns.Topic.fromTopicArn(this, 'OncallTopic',
  'arn:aws:sns:us-east-1:123456789012:mcp-oncall-pager');

const p99Alarm = new cloudwatch.Alarm(this, 'P99LatencyAlarm', {
  alarmName: 'mcp-p99-latency',
  metric: new cloudwatch.Metric({
    namespace: 'McpServer',
    metricName: 'McpToolDuration',
    statistic: 'p99',
    period: cdk.Duration.minutes(1),
    dimensionsMap: { FunctionName: 'mcp-server-prod' },
  }),
  threshold: 2000,
  evaluationPeriods: 3,
  treatMissingData: cloudwatch.TreatMissingData.NOT_BREACHING,
});

const errorRateAlarm = new cloudwatch.Alarm(this, 'ErrorRateAlarm', {
  alarmName: 'mcp-error-rate',
  metric: new cloudwatch.Metric({
    namespace: 'McpServer',
    metricName: 'McpToolErrors',
    statistic: 'Sum',
    period: cdk.Duration.minutes(1),
    dimensionsMap: { FunctionName: 'mcp-server-prod' },
  }),
  threshold: 10,
  evaluationPeriods: 2,
  treatMissingData: cloudwatch.TreatMissingData.NOT_BREACHING,
});

const healthGate = new cloudwatch.CompositeAlarm(this, 'HealthGate', {
  alarmDescription: 'Any MCP health signal degraded',
  alarmRule: cloudwatch.AlarmRule.anyOf(p99Alarm, errorRateAlarm),
});

healthGate.addAlarmAction(new cw_actions.SnsAction(oncallTopic));
healthGate.addOkAction(new cw_actions.SnsAction(oncallTopic));

Composite alarm billing

Composite alarms cost $0.50 per alarm per month. Standard metric alarms cost $0.10 per alarm per month. For a typical MCP server setup with 5 metric alarms and 2 composite alarms, the total cost is (5 × $0.10) + (2 × $0.50) = $1.50/month — a negligible cost compared to the on-call engineer time saved by eliminating correlated alert storms.

Note: there is no additional cost for the alarm evaluations triggered by child state changes. You pay only the flat per-alarm monthly rate regardless of how frequently child alarms flip.

Failure modes

Symptom	Cause	Fix
Composite alarm stays in INSUFFICIENT_DATA on creation	One or more child alarms are in INSUFFICIENT_DATA because their metric has no data yet; composite alarm reflects the most restrictive child state	Wait for child alarms to receive metric data and transition to OK; or set child alarms' treat-missing-data to `notBreaching` so they start in OK state
Maintenance suppression not working — composite still pages during deployment	Composite rule evaluated before `set-alarm-state` command completed; CLI call returned before CloudWatch propagated the state change	Add 5–10 second delay after `set-alarm-state` before starting deployment; verify state with `aws cloudwatch describe-alarms --alarm-names mcp-maintenance-window`
Composite alarm fires but SNS notification not received	SNS topic has no subscriptions, or the subscription's email endpoint has not been confirmed	Verify topic subscriptions with `aws sns list-subscriptions-by-topic`; confirm email subscription in inbox
Rule validation error on composite alarm creation	Alarm names in the rule contain special characters that need proper quoting, or a referenced alarm name does not exist yet	Create all child alarms before the composite alarm; use double-quotes around alarm names in the rule string and escape properly for shell; verify rule syntax with the CloudWatch console rule editor first
Composite alarm does not fire even though all child alarms are in ALARM	Using `AND` when you meant `OR`: rule `ALARM(a) AND NOT ALARM(b)` requires b to be in OK, but b is also in ALARM from the same incident	Review the rule logic; for general health gates use OR; AND is for "both signals simultaneously" critical-severity detection