CloudWatch Observability for MCP Servers: Logs, Metrics, Alarms, and Container Insights

Pattern 1: The telemetry data plane — structured logs and EMF as two projections of the same event

When an MCP tool handler fires, it produces one meaningful unit of information: a tool invocation with a name, a duration, an outcome, and optionally an upstream service that was called. CloudWatch gives you two ways to project that single event into its observability systems, and the design insight is that you should emit both projections in the same console.log call — because both are synchronous stdout writes and neither adds handler latency.

The first projection is a structured JSON log line. Logs Insights parses it at query time, treating every JSON key as a queryable field. This projection is the right home for high-cardinality context: the aws_request_id that ties back to the Lambda REPORT line, the mcp_session_id from the transport header that identifies the specific agent that made the call, the error message string containing the full exception text, and the upstream_service field that names which AWS API returned an error. These fields have unbounded cardinality — thousands or millions of unique values — and belong in logs where you query them ad hoc during an incident, not in metrics where high cardinality creates millions of CloudWatch time-series and significant cost.

The second projection is an EMF (Embedded Metric Format) metric. The Lambda runtime flushes stdout to CloudWatch Logs synchronously; CloudWatch Logs detects the EMF envelope structure and extracts the numeric values as CloudWatch Metrics data points. This projection is the right home for bounded-cardinality aggregatable signals: McpToolDuration in milliseconds dimensioned by Tool (5–20 unique tool names), McpToolErrors as a count dimensioned by Tool, McpToolOutputSizeBytes dimensioned by Tool. These signals aggregate across thousands of invocations into percentile statistics you can alarm on — a p99(McpToolDuration) alarm on your dynamo_query tool is impossible with raw Lambda Duration metrics because the standard metric has no tool-name dimension.

The key constraint on EMF dimensions is cardinality. CloudWatch stores one time-series per unique dimension combination — use RequestId as a dimension and you create a new time-series for every single Lambda invocation, generating an enormous and unexpected bill. The correct split: tool, function_name, and environment are EMF dimensions (bounded); aws_request_id, mcp_session_id, error, and input_size_bytes are structured log fields (unbounded). The same tool field appears in both projections — in the structured log as a queryable filter, in the EMF envelope as a CloudWatch metric dimension. The EMF guide shows how EMF's Dimensions array in the _aws envelope references the same top-level JSON keys as your regular log fields, so you write one JSON object and CloudWatch Logs performs both the log indexing and the metric extraction from the same bytes.

// Single console.log call that produces BOTH a structured log line (queryable via Logs Insights)
// AND a set of CloudWatch custom metrics (via EMF envelope detection)
// The _aws key triggers EMF extraction; all other keys are indexed as Logs Insights fields

console.log(JSON.stringify({
  // EMF envelope — tells CloudWatch which keys are metrics
  _aws: {
    Timestamp: Date.now(),
    CloudWatchMetrics: [{
      Namespace: 'McpServer',
      Dimensions: [['Tool', 'FunctionName']],   // low-cardinality — ~10 unique combinations
      Metrics: [
        { Name: 'McpToolDuration', Unit: 'Milliseconds' },
        { Name: 'McpToolErrors', Unit: 'Count' },
        { Name: 'McpToolOutputSizeBytes', Unit: 'Bytes' }
      ]
    }]
  },

  // CloudWatch metric dimension values (low-cardinality, alarmable)
  Tool: toolName,
  FunctionName: context.functionName,

  // Structured log fields for Logs Insights (high-cardinality, incident debugging)
  level: 'INFO',
  duration_ms: durationMs,
  cold_start: isFirstInvocation,
  aws_request_id: context.awsRequestId,
  mcp_session_id: mcpSessionId,
  output_size_bytes: outputSizeBytes,
  upstream_service: 'DynamoDB',

  // These numeric values satisfy the EMF Metrics[] references above
  McpToolDuration: durationMs,
  McpToolErrors: 0,
  McpToolOutputSizeBytes: outputSizeBytes,
}));

If you use AWS Lambda Powertools, the @aws-lambda-powertools/metrics package builds the EMF envelope for you and the Logger builds the structured log. You call both in the same handler invocation with separate library calls — they write to stdout independently. The practical benefit is that a single deploy wires up both Logs Insights queryability and CloudWatch metric alarmability with no infrastructure changes. The EMF guide covers the Powertools @metrics.logMetrics decorator in detail, including the critical clearDimensions() call needed in warm containers that handle multiple sequential tool calls — without it, tool-name dimensions from the first invocation bleed into the second.

Pattern 2: The alarm stack — three tiers that catch different failure classes

With the telemetry data plane from Pattern 1 in place, you have two metric sources to alarm on: the standard AWS/Lambda namespace (or ECS/ContainerInsights for Fargate) and your custom McpServer namespace from EMF. The right alarm architecture uses both, organized into three tiers by what each tier can detect.

Tier 1: Standard infrastructure metrics. These exist immediately, require no code changes, and cover the most common catastrophic failures. For Lambda-hosted MCP servers, the four that matter are Errors (sum >5 per minute for 2 consecutive windows — the 2-window requirement filters one-shot deployment errors), Throttles (sum >1 per minute immediately, because a throttled MCP tool call is an instant failure from the caller's perspective), Duration p99 above 5 seconds for 3 consecutive minutes (note: the CLI requires --extended-statistic p99, not --statistic p99 — they are separate parameters), and ConcurrentExecutions above 80% of your reserved concurrency limit. For ECS/Fargate MCP servers, Container Insights provides RunningTaskCount dropping to zero (service-down alarm, use treat-missing-data breaching so the alarm fires even when the metric stops flowing) and memory utilization computed as (MemoryUtilized / MemoryReserved) * 100 above 80% for 2 consecutive 5-minute windows. The raw MemoryUtilized in MiB is not directly alarmable — you need the metric math expression because Fargate kills tasks at the reservation limit, not at some fixed absolute value. The Container Insights guide has the full alarm JSON with the metric math expression.

Tier 2: Custom per-tool metrics via EMF. Tier 1 tells you that the function as a whole is degraded. Tier 2 tells you which specific MCP tool is degraded. A McpToolErrors alarm with dimension Tool=dynamo_query fires before the aggregate AWS/Lambda Errors alarm if only one tool out of ten is failing — the aggregate metric absorbs the errors from the healthy tools and stays below threshold. The alarm configuration for a custom EMF metric is identical to a standard metric alarm; the only difference is the namespace (McpServer instead of AWS/Lambda) and the custom dimensions. The alarms guide shows the CLI command for per-tool alarms on custom EMF metrics.

Tier 3: Composite alarms for noise reduction. Tier 1 and Tier 2 alarms fire individually, which means a deployment that briefly elevates errors triggers an alert that wakes someone up for a non-incident. Composite alarms evaluate a Boolean expression over component alarm states — ALARM(errors) AND ALARM(latency) fires only when both conditions are simultaneously true, dramatically reducing alert fatigue. The expression language supports AND, OR, and NOT, enabling you to distinguish a bug (ALARM(errors) AND ALARM(latency) AND NOT ALARM(throttles)) from a capacity problem (ALARM(throttles) AND ALARM(concurrency)) in the alarm name itself, so the on-call rotation knows what to look at before opening a browser.

One composite alarm gotcha: the AND operator treats INSUFFICIENT_DATA component alarm state as neither OK nor ALARM, so the composite alarm never fires if a component alarm is in INSUFFICIENT_DATA. This happens on newly created alarms with no invocations yet, or after a function is deleted and its metric stream stops. The fix is to set treatMissingData: MISSING on each component alarm — when data stops flowing, the alarm transitions to ALARM state rather than INSUFFICIENT_DATA, which lets the composite alarm's AND logic evaluate correctly.

For variable-traffic MCP servers — more invocations during business hours, near-zero at night — static thresholds on the error rate produce false positives at low-traffic hours and miss real incidents at high-traffic hours. CloudWatch anomaly detection trains a 2-week baseline model and expresses thresholds as a band around the predicted value. The alarm fires when the observed metric exits the band, regardless of the absolute value. The CLI uses ANOMALY_DETECTION_BAND(m1, 2) as a threshold metric ID where the second argument is the band width in standard deviations (2 = roughly a 95% confidence interval).

Pattern 3: The investigation layer — Logs Insights after the alarm fires

When a composite alarm fires and the on-call engineer opens CloudWatch, the alarm state tells them which signal combination triggered. Logs Insights is what they use next to go from "what metric breached" to "what happened in the code." The investigation workflow is a sequence of queries, each narrowing the scope of the next.

The standard MCP server triage sequence: start with error rate by tool name over the last hour to identify the failing tool, then run p50/p90/p99 latency for that specific tool to determine if it's a latency degradation or a hard error, then filter to the specific error messages and upstream service fields to see what exception class is being thrown. Three queries, two to three minutes, and you have a working hypothesis for the root cause. This only works if the structured log fields from Pattern 1 are in place — the tool field is what makes the first query possible; the upstream_service field is what makes the third query possible. The Logs Insights guide has all eight query patterns with the exact syntax.

-- Step 1: Which tool is failing? (last hour)
fields @timestamp, tool, level, error, aws_request_id
| filter level = "ERROR"
| stats count(*) as error_count by tool
| sort error_count desc
| limit 20

-- Step 2: Latency breakdown for the failing tool (last 24h for trend context)
fields @timestamp, tool, duration_ms
| filter ispresent(duration_ms) and tool = "dynamo_query"
| stats
    percentile(duration_ms, 50) as p50_ms,
    percentile(duration_ms, 90) as p90_ms,
    percentile(duration_ms, 99) as p99_ms,
    count(*) as invocations
  by bin(5m)
| sort @timestamp asc

-- Step 3: What upstream service is causing it?
fields @timestamp, tool, error, error_code, aws_request_id
| filter tool = "dynamo_query" and level = "ERROR"
| stats count(*) as errors by error_code, upstream_service
| sort errors desc

For MCP servers built with one Lambda function per tool — a common CDK pattern that isolates cold start and concurrency settings per tool — the investigation query needs to span multiple log groups. Logs Insights supports up to 50 log groups in a single query via the console's log group selector or the CLI's --log-group-names array. The @logGroup field in query results tells you which function each log line came from, so a single query can rank all tool functions by p99 latency without switching between log groups manually.

For ECS/Fargate MCP servers, Container Insights adds a second log-based data source: the /aws/ecs/containerinsights/mcp-cluster/performance log group. CloudWatch publishes container-level metrics (CPU, memory, network per container name) to this log group as structured JSON — the standard ECS/ContainerInsights CloudWatch namespace only has service-level and cluster-level granularity, not per-container. If your MCP server task definition has multiple containers (a server container plus a log router sidecar), Logs Insights queries against the performance log group show you exactly which container is consuming the memory that is approaching the reservation limit. The Container Insights guide has the query syntax and the field names (ContainerName, CpuUtilized, MemoryUtilized) that appear in that performance log group.

Logs Insights queries can be saved as named favorites and pinned to CloudWatch Dashboards as widgets alongside standard metric graphs. The recommended MCP server ops dashboard layout: a "MCP tool error rate" table widget (Logs Insights, last hour), a "p99 latency by tool" bar chart (Logs Insights, last 24h), a "cold start frequency" time-series (Logs Insights, last 7d), and standard metric graphs for Lambda Errors, Throttles, and Duration p99. This gives a single-pane view that answers both "is anything on fire right now" and "what does the trend look like."

One common Logs Insights failure mode worth flagging explicitly: fields like tool show as - (undefined) in query results even though the field clearly exists in the log line. This happens when the log line is not valid JSON — an uncaught exception from the Node.js MCP SDK often produces a plain text stack trace that Logs Insights cannot parse. The diagnosis query is filter NOT ispresent(tool) | sort @timestamp desc, which surfaces all the log lines Logs Insights could not parse. The fix is a top-level try/catch in the Lambda handler that converts any thrown error into a structured JSON log line before re-throwing, ensuring that even uncaught exceptions produce queryable structured output.

The external blind spot: what all three layers miss

EMF metrics, CloudWatch Alarms, and Logs Insights are all instrumentation inside the AWS account. They share a common blind spot: they can only observe what happens inside the Lambda execution environment or the ECS task. A request that fails before reaching your function — DNS resolution failure, TLS handshake error, API Gateway routing misconfiguration, a Lambda cold start that times out before the invocation is recorded — produces no Lambda log line, no EMF metric, no CloudWatch alarm. From the internal observability stack's perspective, the failure never happened.

From the caller's perspective — an AI agent making JSON-RPC calls to your MCP server — that external failure is the only failure that matters. AliveMCP probes your MCP endpoint from outside your AWS account every 60 seconds using a real tools/list JSON-RPC handshake over the public internet. If DNS fails, TLS fails, API Gateway drops the request, or the Lambda cold start exceeds the timeout the caller would experience, AliveMCP catches it and alerts you — before your users do. The internal CloudWatch stack catches bugs in your tool handlers, capacity limits, and OOM kills. AliveMCP catches the reachability failures the internal stack is blind to. Both are needed for complete MCP server observability.

Consolidated failure modes