CloudWatch Advanced Patterns for MCP Server Observability: Metric Streams, Composite Alarms, Contributor Insights, and Anomaly Detection

Pattern 1: Metric Streams — replacing polling with push delivery

Every major observability SaaS (Datadog, New Relic, Grafana Cloud, Dynatrace) ships a "CloudWatch integration" that calls GetMetricData on a schedule — typically every 5 or 10 minutes. This polling architecture has three fundamental problems for MCP server monitoring: the 5–15 minute lag means a probe-failure spike that lasts 4 minutes can disappear before the next poll; the rate limits on GetMetricData cause throttling on accounts with hundreds of Lambda function dimensions; and the cost of $0.01 per 1,000 GetMetricData data points exceeds the Metric Streams cost of $0.003 per 1,000 updates by 3×.

Push vs. polling comparison

Output format decision: JSON vs. OpenTelemetry 0.7

Metric Streams supports exactly two output formats, and the choice is not interchangeable:

• json — newline-delimited JSON records in Firehose batches. Works with any S3-based consumer, custom Lambda processors, and the full set of CloudWatch namespaces including custom namespaces like McpServer. Use this for S3 data lakes queried with Athena, or for any consumer you control.
• opentelemetry0.7 — protobuf-encoded OTel ExportMetricsServiceRequest. Datadog, New Relic, and Grafana Cloud all have Firehose HTTP endpoint targets that ingest this format natively without a translation Lambda. Custom namespaces (like McpServer) are typically not supported by third-party OTel ingestion endpoints — they map only the standard AWS namespaces they know about.

The failure mode here is format mismatch: if you create a Metric Stream with json format pointed at a Datadog Firehose HTTP endpoint, Datadog's ingestion pipeline rejects the payload silently, metrics stop appearing in Datadog, and the stream's state still shows RUNNING. Output format cannot be changed on an existing stream — you must delete and recreate it.

Namespace include filters and percentile statistics

Without a namespace include filter, a Metric Stream pushes every metric in your AWS account — EC2, RDS, S3, CloudFront, and dozens of other services. Always apply include filters:

aws cloudwatch put-metric-stream \
  --name mcp-metrics-stream \
  --firehose-arn arn:aws:firehose:us-east-1:123456789012:deliverystream/mcp-metrics-to-s3 \
  --role-arn arn:aws:iam::123456789012:role/cloudwatch-metric-streams-role \
  --output-format json \
  --include-filters '[
    {"Namespace": "AWS/Lambda"},
    {"Namespace": "AWS/SQS"},
    {"Namespace": "McpServer"}
  ]' \
  --statistics-configurations '[
    {
      "IncludeMetrics": [
        {"Namespace": "AWS/Lambda", "MetricName": "Duration"},
        {"Namespace": "McpServer", "MetricName": "McpToolDuration"}
      ],
      "AdditionalStatistics": ["p50", "p95", "p99"]
    }
  ]'

The --statistics-configurations parameter adds percentile statistics to streamed records. Without it, the stream only includes min, max, sum, and count — you cannot compute p99 in the destination from those four values. Each additional statistic counts as an additional metric update at the same $0.003/1,000 rate.

Multi-account probe metric aggregation

If AliveMCP probe workers run across multiple AWS regions or accounts, Metric Streams enables central S3 aggregation without an APM SaaS subscription. Each probe worker account streams its McpServer namespace to a regional Firehose pointing at a central S3 bucket in the management account (via cross-account bucket policies). An Athena table over the central S3 prefix enables SQL queries across all regional probe data — for example, identifying which MCP endpoints are failing in eu-west-1 but healthy in us-east-1, which would indicate a regional DNS or routing issue rather than a server-side failure.

At 1,667 probe executions per second generating one McpToolDuration metric update each, Metric Streams cost for the McpServer namespace is approximately 1,667 × 60 × 60 × 24 × $0.003 / 1,000 = approximately $0.43/day — less than a single Datadog APM host per month for full metric streaming.

Pattern 2: Composite Alarms — health gates and maintenance suppression

A CloudWatch metric alarm evaluates one metric against one threshold. An MCP server failure manifests as a cluster: p99 latency rises, error count rises, and the dead-letter queue fills — all within the same minute. If each of these three alarms has its own SNS notification, the on-call engineer receives three pages simultaneously for one incident. Alert fatigue from correlated storms is one of the primary drivers of on-call engineers starting to ignore alerts — exactly the failure mode AliveMCP's customers pay to avoid at the server level.

Rule language and evaluation model

A composite alarm's rule is a boolean expression evaluated immediately whenever any child alarm changes state. The rule supports ALARM("name"), OK("name"), INSUFFICIENT_DATA("name"), AND, OR, NOT, and parentheses for grouping. Nesting depth is limited to 4 levels.

Critical difference from metric alarms: composite alarms have no evaluation period of their own. They do not poll on a schedule — they re-evaluate the moment a child alarm transitions. If three alarms fire within 5 seconds of each other, the composite alarm enters ALARM state once and triggers exactly one SNS notification.

Pattern 2a: Single pager for correlated signals (OR gate)

The canonical setup: create individual metric alarms with no SNS actions, then attach SNS only to the composite alarm:

# Individual alarms — no alarm-actions (composite owns the notification)
aws cloudwatch put-metric-alarm \
  --alarm-name "mcp-p99-latency" \
  --namespace McpServer --metric-name McpToolDuration \
  --extended-statistic p99 \
  --dimensions Name=FunctionName,Value=mcp-server-prod \
  --period 60 --evaluation-periods 3 --threshold 2000 \
  --comparison-operator GreaterThanOrEqualToThreshold \
  --treat-missing-data notBreaching

aws cloudwatch put-metric-alarm \
  --alarm-name "mcp-error-rate" \
  --namespace McpServer --metric-name McpToolErrors \
  --statistic Sum \
  --dimensions Name=FunctionName,Value=mcp-server-prod \
  --period 60 --evaluation-periods 2 --threshold 10 \
  --comparison-operator GreaterThanOrEqualToThreshold \
  --treat-missing-data notBreaching

# Composite OR gate — one SNS page per incident
aws cloudwatch put-composite-alarm \
  --alarm-name "mcp-health-gate" \
  --alarm-rule 'ALARM("mcp-p99-latency") OR ALARM("mcp-error-rate") OR ALARM("mcp-dlq-depth")' \
  --alarm-actions arn:aws:sns:us-east-1:123456789012:mcp-oncall-pager \
  --ok-actions arn:aws:sns:us-east-1:123456789012:mcp-oncall-pager

The --ok-actions on the composite sends a single "all clear" notification when the health gate recovers — one resolution notification matching the one incident notification.

Pattern 2b: Maintenance window suppression

Planned deployments legitimately break health checks. Without suppression, the on-call receives expected pages during known maintenance, trains the response "it's probably just a deploy," and eventually starts dismissing pages during real incidents. The solution is a dummy metric alarm that you manually flip:

# Dummy alarm — threshold 9999999 means it never fires automatically
# aws:cloudwatch:NumberOfMetricFilters is a real metric but will never reach this threshold
aws cloudwatch put-metric-alarm \
  --alarm-name "mcp-maintenance-window" \
  --namespace AWS/CloudWatch --metric-name NumberOfMetricFilters \
  --statistic Average --period 3600 --evaluation-periods 1 \
  --threshold 999999 --comparison-operator GreaterThanThreshold \
  --treat-missing-data notBreaching

# Composite with suppression
aws cloudwatch put-composite-alarm \
  --alarm-name "mcp-health-gate-suppressed" \
  --alarm-rule 'ALARM("mcp-health-gate") AND NOT ALARM("mcp-maintenance-window")' \
  --alarm-actions arn:aws:sns:us-east-1:123456789012:mcp-oncall-pager

# Before deployment
aws cloudwatch set-alarm-state --alarm-name mcp-maintenance-window \
  --state-value ALARM --state-reason "Deploy $(date -u +%Y-%m-%dT%H:%M:%SZ)"

# After deployment
aws cloudwatch set-alarm-state --alarm-name mcp-maintenance-window \
  --state-value OK --state-reason "Deploy complete"

Add a 5–10 second delay between set-alarm-state and starting the deployment. CloudWatch state propagation is near-instantaneous but not instantaneous — the composite re-evaluates as soon as the child state is received, so you want the maintenance window alarm to be in ALARM state before the health signals start degrading.

Pattern 2c: Severity tiering with AND logic

Some incidents warrant immediate paging; others need only a Slack message during business hours. AND logic distinguishes real outages (multiple signals simultaneously degraded) from transient flaps (single isolated signal):

# Critical: both latency AND errors degraded simultaneously — probable total outage
aws cloudwatch put-composite-alarm \
  --alarm-name "mcp-critical-incident" \
  --alarm-rule 'ALARM("mcp-p99-latency") AND ALARM("mcp-error-rate")' \
  --alarm-actions arn:aws:sns:us-east-1:123456789012:mcp-pagerduty-critical

# Warning: any single signal degraded but not critical — notify Slack, no page
aws cloudwatch put-composite-alarm \
  --alarm-name "mcp-warning-indicator" \
  --alarm-rule '(ALARM("mcp-p99-latency") OR ALARM("mcp-error-rate") OR ALARM("mcp-dlq-depth")) AND NOT ALARM("mcp-critical-incident")' \
  --alarm-actions arn:aws:sns:us-east-1:123456789012:mcp-slack-warnings

The mcp-warning-indicator fires when exactly one signal is degraded (any-OR minus the AND combination). This two-tier approach pages on multi-signal correlations (high-confidence real incidents) and notifies without paging on isolated signals (high-confidence transient flaps). Composite alarm cost: $0.50/alarm/month, versus $0.10/alarm/month for standard metric alarms — for a typical setup with 5 metric alarms and 3 composite alarms, the total is $2.00/month.

Pattern 3: Contributor Insights — continuous top-N error source identification

CloudWatch Logs Insights is excellent for ad-hoc investigation — "show me every probe result for api.mcp-broken.com between 3:40 and 3:55 AM." But answering "which MCP endpoints are failing the most right now?" requires you to remember to run a query, write the right aggregation syntax, wait for the query to complete, and repeat every time you want an update. For AliveMCP's operational dashboard, that operational question needs a continuously updating answer without manual intervention.

Contributor Insights vs. Logs Insights

Rule structure

A Contributor Insights rule is a JSON document defining which log group to watch, how to identify contributors, and how to aggregate their contributions:

{
  "Schema": { "Name": "CloudWatchLogRule", "Version": 1 },
  "LogGroupNames": ["/mcp/probe-results"],
  "LogFormat": "JSON",
  "Contribution": {
    "Keys": ["$.endpoint"],          // group-by — up to 4 keys
    "ValueOf": "$.latency_ms",       // for Sum aggregation (omit for Count)
    "Filters": [
      { "Match": "$.status", "In": ["error", "timeout", "auth_failure"] }
    ]
  },
  "AggregateOn": "Count"             // Count or Sum (Sum requires ValueOf)
}

The Keys array defines the contributor identity. With Keys: ["$.endpoint"], each unique endpoint URL is a separate contributor. With Keys: ["$.registry", "$.error_class"], each registry+error-class pair is a contributor — enabling the "which registries host the worst-quality servers?" operational question. Cardinality limit: 10,000 unique contributor values per rule per hour, which is not a constraint for MCP endpoint monitoring with the current ~2,200 registered servers.

Three practical rules for AliveMCP probe logs

Assuming probe workers write structured JSON lines like {"endpoint": "https://api.mcp-example.com/mcp", "status": "error", "latency_ms": 8432, "error_class": "upstream_timeout", "registry": "mcp.so"} to /mcp/probe-results:

# Rule 1: Top failing endpoints by error count (standard operational dashboard)
aws cloudwatch put-insight-rule \
  --rule-name "mcp-top-failing-endpoints" --rule-state ENABLED \
  --rule-definition '{
    "Schema": {"Name": "CloudWatchLogRule", "Version": 1},
    "LogGroupNames": ["/mcp/probe-results"], "LogFormat": "JSON",
    "Contribution": {
      "Keys": ["$.endpoint"],
      "Filters": [{"Match": "$.status", "In": ["error", "timeout", "auth_failure"]}]
    }, "AggregateOn": "Count"
  }'

# Rule 2: Top slow endpoints by accumulated latency (finds degraded-but-not-erroring servers)
aws cloudwatch put-insight-rule \
  --rule-name "mcp-top-slow-endpoints" --rule-state ENABLED \
  --rule-definition '{
    "Schema": {"Name": "CloudWatchLogRule", "Version": 1},
    "LogGroupNames": ["/mcp/probe-results"], "LogFormat": "JSON",
    "Contribution": {
      "Keys": ["$.endpoint"], "ValueOf": "$.latency_ms",
      "Filters": [{"Match": "$.status", "NotIn": ["error"]}]
    }, "AggregateOn": "Sum"
  }'

# Rule 3: Error class by registry (reveals which registries host worst servers)
aws cloudwatch put-insight-rule \
  --rule-name "mcp-error-class-by-registry" --rule-state ENABLED \
  --rule-definition '{
    "Schema": {"Name": "CloudWatchLogRule", "Version": 1},
    "LogGroupNames": ["/mcp/probe-results"], "LogFormat": "JSON",
    "Contribution": {
      "Keys": ["$.registry", "$.error_class"],
      "Filters": [{"Match": "$.status", "In": ["error"]}]
    }, "AggregateOn": "Count"
  }'

Querying results programmatically and alarming on MaxContributorValue

Rule results are available in the CloudWatch console and via the GetInsightRuleReport API for dashboard automation:

const { GetInsightRuleReportCommand } = require('@aws-sdk/client-cloudwatch');

const report = await cloudwatch.send(new GetInsightRuleReportCommand({
  RuleName: 'mcp-top-failing-endpoints',
  StartTime: new Date(Date.now() - 3600 * 1000),
  EndTime: new Date(),
  Period: 3600,
  MaxContributorCount: 10,
  Metrics: ['UniqueContributors', 'MaxContributorValue', 'Sum'],
  OrderBy: 'Sum',
}));

report.Contributors.forEach(c => {
  console.log(`${c.Keys[0]}: ${c.Approximation} errors in last hour`);
});

In sampled mode (default 25%), c.Approximation is a statistical estimate with ±15% accuracy at the 95th confidence level — sufficient for "which endpoints are the worst" operational ranking. For exact counts and for alarming, switch to full-scan mode by adding "InsightRuleSamplingPercentage": 100 to the rule definition. Full-scan mode publishes CloudWatch metrics under the CloudWatchLogs namespace, enabling a standard alarm on MaxContributorValue to fire when any single endpoint's error count exceeds a threshold — a signal that one bad actor endpoint is skewing overall probe health metrics.

Sampling vs. full-scan cost analysis

At AliveMCP scale (50M probe log events/day), sampled mode costs approximately $0.44/day per rule ($13/month); full-scan doubles this to $26/month. For a 3-rule setup, expect $39–$78/month depending on mode — modest for the operational visibility gained by replacing manual Logs Insights queries with continuously-running top-N charts.

Pattern 4: Anomaly Detection — ML baselines for off-peak degradation

Static threshold alarms have a fundamental problem for MCP server monitoring: the threshold that is meaningful during peak traffic is meaningless at night. A 1,000ms p99 threshold is reasonable during business hours when the baseline is 400ms — a 2.5× ratio. But at 3 AM when the baseline is 80ms, a server responding in 500ms is 6× its normal latency (clearly degraded) yet still under the 1,000ms threshold (alarm stays silent). An engineer who investigates a reported MCP failure at 3:15 AM will find a 500ms p99 that looks "fine" by the threshold, but which has been 6× the normal value for two hours.

How the model works

CloudWatch Anomaly Detection uses seasonal decomposition to model three components of metric behavior:

• Hourly seasonality — traffic patterns within a day (morning ramp, lunch peak, overnight quiet)
• Daily seasonality — weekday vs. weekend patterns (business hours vs. flat weekend traffic for B2B MCP servers)
• Trend — gradual drift in the baseline over weeks

The model requires at least 15 minutes of metric history to initialize and improves with up to 2 weeks of history. A single model covers one metric+statistic+dimension combination — a model on McpServer/McpToolDuration p99 FunctionName=mcp-server-prod is independent of a model on the same metric for FunctionName=mcp-server-staging. The standard deviations parameter controls band width: at stddev=2, approximately 95% of normal data points fall within the band (5% false positive rate); at stddev=3, ~99.7% fall within the band (fewer false positives but requires more pronounced anomalies to trigger).

Off-peak detection vs. static threshold comparison

Setup: anomaly detector and alarm configuration

# Step 1: Create anomaly detector model — starts training on last 2 weeks of history
aws cloudwatch put-anomaly-detector \
  --namespace McpServer \
  --metric-name McpToolDuration \
  --dimensions '[{"Name": "FunctionName", "Value": "mcp-server-prod"}]' \
  --stat p99 \
  --configuration '{
    "ExcludedTimeRanges": [
      {"StartTime": "2026-10-15T02:00:00Z", "EndTime": "2026-10-15T04:00:00Z"}
    ],
    "MetricTimezone": "America/New_York"
  }'
# ExcludedTimeRanges: prevents deployment restart spikes from being learned as normal
# MetricTimezone: aligns day-of-week seasonality with your business timezone

# Step 2: Wait 15+ minutes for model initialization
# (check CloudWatch console — grey band appears on metric graph when model is ready)

# Step 3: Create anomaly detection alarm
aws cloudwatch put-metric-alarm \
  --alarm-name "mcp-p99-anomaly" \
  --metrics '[
    {
      "Id": "m1",
      "MetricStat": {
        "Metric": {
          "Namespace": "McpServer", "MetricName": "McpToolDuration",
          "Dimensions": [{"Name": "FunctionName", "Value": "mcp-server-prod"}]
        },
        "Period": 60, "Stat": "p99"
      }
    },
    {
      "Id": "ad1",
      "Expression": "ANOMALY_DETECTION_BAND(m1, 2)",
      "Label": "McpToolDuration p99 expected band"
    }
  ]' \
  --comparison-operator GreaterThanUpperThreshold \
  --threshold-metric-id ad1 \
  --evaluation-periods 3 \
  --treat-missing-data notBreaching \
  --alarm-actions arn:aws:sns:us-east-1:123456789012:mcp-alerts

Detecting worker failures with LessThanLowerThreshold

Anomaly detection works in both directions. For probe throughput monitoring, a drop in probes-per-minute is the anomaly to detect — a worker that handles 1,000 probes/minute and drops to 50 is clearly broken, even if the remaining 50 probes show healthy latency. The latency metric looks fine; only throughput reveals the failure:

aws cloudwatch put-metric-alarm \
  --alarm-name "mcp-probe-throughput-drop" \
  --metrics '[
    {
      "Id": "m1",
      "MetricStat": {
        "Metric": {
          "Namespace": "McpServer", "MetricName": "ProbesExecuted",
          "Dimensions": [{"Name": "WorkerGroup", "Value": "probe-workers-prod"}]
        },
        "Period": 60, "Stat": "Sum"
      }
    },
    {"Id": "ad1", "Expression": "ANOMALY_DETECTION_BAND(m1, 2)"}
  ]' \
  --comparison-operator LessThanLowerThreshold \
  --threshold-metric-id ad1 \
  --evaluation-periods 5 \
  --treat-missing-data breaching \
  --alarm-actions arn:aws:sns:us-east-1:123456789012:mcp-alerts

Note --treat-missing-data breaching here — if the metric stops reporting entirely because the worker is completely dead, that absence should trigger the alarm. For latency alarms, notBreaching is correct because absent latency data means no traffic (not an incident), but for throughput alarms, absent data means the worker has stopped entirely.

CDK limitation and model drift mitigation

As of late 2026, the CloudWatch L2 CDK Alarm construct does not fully support anomaly detection threshold metrics. The comparisonOperator: ComparisonOperator.GREATER_THAN_UPPER_THRESHOLD option requires a thresholdMetricId pointing at the anomaly band expression, which the L2 construct cannot wire correctly. Use CfnAlarm (L1) with the metrics array matching the CLI structure above.

Model drift is the primary operational risk: a gradual degradation sustained for weeks gets incorporated into the model's baseline, causing the alarm to stop firing even though the metric is permanently elevated. Mitigate with two practices:

1. Backup static threshold alarm — always deploy a parallel GreaterThanOrEqualToThreshold alarm at the absolute maximum acceptable value (e.g., 3,000ms). Even if the anomaly detection model normalizes around 2,500ms, the static alarm still fires when further degradation reaches 3,000ms.
2. Exclude degradation periods — when you identify that a regression was introduced at a specific date, update ExcludedTimeRanges to exclude that period from future training, so the model reverts toward the pre-regression baseline rather than continuing to learn the degraded state as normal.

Anomaly detection pricing: $0.30 per model per month. A typical MCP server setup with 3 models (p99 latency, error rate, probe throughput) plus 3 anomaly detection alarms costs $0.90 + $0.30 = $1.20/month — the same order of magnitude as a single standard metric alarm, making it cost-neutral if it catches even one off-peak degradation per month that a static threshold would miss.

Production checklist