CloudWatch Anomaly Detection for MCP Servers — ML Baselines for Dynamic Latency Thresholds

How the model works

CloudWatch Anomaly Detection uses a combination of seasonal decomposition and statistical regression to model the expected value of a metric. It captures:

• Hourly seasonality — traffic patterns within a day (peak hours vs. night-time lows)
• Daily seasonality — weekday vs. weekend patterns
• Trend — gradual upward or downward drift in the baseline

The model requires at least 15 minutes of metric history to initialize, and improves with up to 2 weeks of history. Models are per metric+statistic+dimension combination — a model trained on p99 latency for FunctionName=mcp-server-prod is independent of a model for the same metric with FunctionName=mcp-server-staging.

The standard deviations parameter controls band width. At stddev=2, approximately 95% of historical data points fall within the band — meaning roughly 5% of normal data points will trigger the alarm even without a real incident. At stddev=3, ~99.7% fall within the band — fewer false positives, but degradations must be more pronounced to trigger. For MCP server p99 latency, start at stddev=2 and increase if you see too many false positives during normal traffic spikes.

Static threshold vs. anomaly detection for MCP latency

The most valuable scenario for MCP server monitoring is the off-peak degradation case: a server that responds in 80ms at night and starts responding in 500ms is clearly broken, but a static 1,000ms threshold misses it entirely. AliveMCP's probe workers detect this at the HTTP level (non-200 response or timeout), but CloudWatch Anomaly Detection provides a complementary layer that catches performance degradation before it becomes a hard failure.

Creating an anomaly detector and alarm

# Step 1: Create the anomaly detector model
# This starts model training on the last 2 weeks of metric history
aws cloudwatch put-anomaly-detector \
  --namespace McpServer \
  --metric-name McpToolDuration \
  --dimensions '[{"Name": "FunctionName", "Value": "mcp-server-prod"}]' \
  --stat p99 \
  --configuration '{
    "ExcludedTimeRanges": [
      {
        "StartTime": "2026-10-15T02:00:00Z",
        "EndTime": "2026-10-15T04:00:00Z"
      }
    ],
    "MetricTimezone": "America/New_York"
  }'
# ExcludedTimeRanges: exclude known maintenance/deployment windows from training data
# MetricTimezone: ensures day-of-week seasonality aligns with your business timezone

# Step 2: Wait for model to initialize (15 minutes minimum, better after 2 weeks)
# Check readiness: the band will appear in CloudWatch console with a grey shaded region

# Step 3: Create alarm using anomaly detection band as threshold
aws cloudwatch put-metric-alarm \
  --alarm-name "mcp-p99-anomaly" \
  --alarm-description "MCP p99 latency outside expected band" \
  --metrics '[
    {
      "Id": "m1",
      "MetricStat": {
        "Metric": {
          "Namespace": "McpServer",
          "MetricName": "McpToolDuration",
          "Dimensions": [{"Name": "FunctionName", "Value": "mcp-server-prod"}]
        },
        "Period": 60,
        "Stat": "p99"
      }
    },
    {
      "Id": "ad1",
      "Expression": "ANOMALY_DETECTION_BAND(m1, 2)",
      "Label": "McpToolDuration p99 (expected)"
    }
  ]' \
  --comparison-operator GreaterThanUpperThreshold \
  --threshold-metric-id ad1 \
  --evaluation-periods 3 \
  --treat-missing-data notBreaching \
  --alarm-actions arn:aws:sns:us-east-1:123456789012:mcp-alerts

The ANOMALY_DETECTION_BAND(m1, 2) math expression refers to the model trained on metric m1 with 2 standard deviations. The --threshold-metric-id ad1 tells CloudWatch that the "threshold" is the upper bound of the band (for GreaterThanUpperThreshold) or lower bound (for LessThanLowerThreshold).

CDK setup

import * as cloudwatch from 'aws-cdk-lib/aws-cloudwatch';
import * as cw_actions from 'aws-cdk-lib/aws-cloudwatch-actions';
import * as sns from 'aws-cdk-lib/aws-sns';

const alertTopic = sns.Topic.fromTopicArn(this, 'AlertTopic',
  'arn:aws:sns:us-east-1:123456789012:mcp-alerts');

const p99Metric = new cloudwatch.Metric({
  namespace: 'McpServer',
  metricName: 'McpToolDuration',
  statistic: 'p99',
  period: cdk.Duration.minutes(1),
  dimensionsMap: { FunctionName: 'mcp-server-prod' },
});

// ANOMALY_DETECTION_BAND math expression over the metric
const anomalyBand = new cloudwatch.MathExpression({
  expression: 'ANOMALY_DETECTION_BAND(m1, 2)',
  usingMetrics: { m1: p99Metric },
  label: 'Expected p99 band (±2 std dev)',
  period: cdk.Duration.minutes(1),
});

const anomalyAlarm = new cloudwatch.Alarm(this, 'P99AnomalyAlarm', {
  alarmDescription: 'MCP p99 latency outside expected seasonal band',
  metric: p99Metric,
  threshold: 0,                                 // ignored when comparisonOperator uses anomaly band
  evaluationPeriods: 3,
  comparisonOperator: cloudwatch.ComparisonOperator.GREATER_THAN_UPPER_THRESHOLD,
  treatMissingData: cloudwatch.TreatMissingData.NOT_BREACHING,
  // Specify the anomaly band as the threshold metric
  // CDK requires CfnAlarm L1 for full anomaly detection support
});
// For full anomaly detection alarm configuration, use CfnAlarm with Metrics array
// until the CloudWatch L2 construct fully supports anomaly detection thresholds

anomalyAlarm.addAlarmAction(new cw_actions.SnsAction(alertTopic));

As of late 2026, the CloudWatch L2 CDK construct has incomplete support for the anomaly detection band as a threshold metric. Use CfnAlarm (L1) with the Metrics array for production deployments, following the CLI command structure above.

Detecting lower-than-expected values

For some MCP server metrics, a lower-than-expected value is the anomaly — not a higher one. For example, if your probe worker normally handles 1,000 probes/minute and that drops to 50 probes/minute, something is wrong with the worker (stuck, throttled, or dead), even though the latency metric looks fine. Use LessThanLowerThreshold for this pattern:

# Alarm fires when probe throughput drops below expected seasonal minimum
aws cloudwatch put-metric-alarm \
  --alarm-name "mcp-probe-throughput-drop" \
  --alarm-description "Probe worker throughput below expected minimum — possible worker failure" \
  --metrics '[
    {
      "Id": "m1",
      "MetricStat": {
        "Metric": {
          "Namespace": "McpServer",
          "MetricName": "ProbesExecuted",
          "Dimensions": [{"Name": "WorkerGroup", "Value": "probe-workers-prod"}]
        },
        "Period": 60,
        "Stat": "Sum"
      }
    },
    {
      "Id": "ad1",
      "Expression": "ANOMALY_DETECTION_BAND(m1, 2)"
    }
  ]' \
  --comparison-operator LessThanLowerThreshold \
  --threshold-metric-id ad1 \
  --evaluation-periods 5 \
  --treat-missing-data breaching \
  --alarm-actions arn:aws:sns:us-east-1:123456789012:mcp-alerts

Note the --treat-missing-data breaching for the throughput drop alarm — if the metric stops reporting entirely (worker completely dead), treat that as ALARM, not as "not breaching." For latency alarms, notBreaching is correct because missing latency data usually means no traffic, not an incident.

Model management

Anomaly detectors continue training on new data indefinitely — the model adapts over time. This means a sustained degradation (p99 that stays elevated for weeks) will eventually be "learned" as normal, causing the alarm to stop firing. This is intentional behavior: if you have a genuine problem, you should fix it rather than let the model normalize around it. To prevent model drift from masking ongoing degradation:

1. Use anomaly detection alongside a static threshold alarm as a backup. If p99 stabilizes at 800ms (learned as normal), a static 1,000ms alarm still catches further escalation.
2. Periodically review the expected band width in the CloudWatch console — if the band has widened significantly over time, investigate whether the underlying metric has genuinely degraded.
3. Add the degraded period to ExcludedTimeRanges when you know a performance regression was introduced, so the model does not learn the bad state as normal.

# Update anomaly detector to exclude a known regression period from training
aws cloudwatch put-anomaly-detector \
  --namespace McpServer \
  --metric-name McpToolDuration \
  --dimensions '[{"Name": "FunctionName", "Value": "mcp-server-prod"}]' \
  --stat p99 \
  --configuration '{
    "ExcludedTimeRanges": [
      {
        "StartTime": "2026-10-25T00:00:00Z",
        "EndTime": "2026-10-27T12:00:00Z"
      }
    ],
    "MetricTimezone": "America/New_York"
  }'

Pricing

Anomaly detection models cost $0.30 per model per month. A model is per metric+statistic+dimension combination. A typical MCP server setup with 3 anomaly detectors (p99 latency, error rate, and probe throughput) costs $0.90/month for the models, plus the standard alarm cost ($0.10/alarm/month) for the alarms that reference them — approximately $1.20/month total. This is the same order of magnitude as a single standard metric alarm per week, making anomaly detection cost-neutral if it eliminates even one false-alarm page per month from the on-call rotation.

Failure modes