AWS API Gateway Canary Deployments for MCP Servers — Stage Variables, Traffic Splitting, and Safe Rollouts

How canary deployments work in API Gateway

To understand canary deployments, you first need to understand how API Gateway manages deployments. A deployment is an immutable snapshot of your API configuration — routes, integrations, method settings — at the moment create-deployment was called. A stage is a named pointer to one of those deployment snapshots, plus stage-specific settings like variables, throttling, and logging.

Normally, a stage has a single deploymentId field. When you call create-deployment and associate it with a stage, you update that pointer. All traffic goes to the configuration captured in the active deployment.

Canary deployments extend this model by adding a secondary deployment pointer to the stage. The stage now has:

• deploymentId: the baseline deployment (currently serving most traffic)
• canarySettings.deploymentId: the new canary deployment (serving the canary percentage)
• canarySettings.percentTraffic: a number from 0.01 to 100 specifying what fraction of requests go to the canary
• canarySettings.stageVariableOverrides: a map of stage variable name overrides that apply only to canary requests

API Gateway routes each incoming request to either the baseline or canary deployment using a random weighted decision before the request reaches Lambda. This decision happens at the API Gateway layer — the client has no input. Requests routed to the canary receive an additional response header: X-Amzn-ApiGateway-Experimental: true. This header is useful for monitoring, debugging, and — as we'll cover later — for AliveMCP health probe discrimination between the two code paths.

Canary deployments work on REST API stages. HTTP API (v2) does not support canary deployments at the API Gateway level — use Lambda weighted aliases directly for HTTP API canary traffic splitting.

Stage variables for Lambda alias routing

Hard-coding a Lambda ARN in the integration URI makes canary deployments less flexible — you would need to create a new API Gateway deployment every time you want to point the canary at a different Lambda version. Stage variables solve this by making the Lambda alias part of the stage configuration, not the API configuration.

The integration URI for a Lambda proxy integration with a stage variable looks like:

arn:aws:lambda:us-east-1:123456789012:function:mcp-handler:${stageVariables.lambdaAlias}

With this setup, the stage variable lambdaAlias controls which Lambda alias receives traffic. You configure the baseline stage to use lambdaAlias=stable and the canary settings override to use lambdaAlias=canary:

# Create Lambda aliases (assuming you have versions v1 and v2 published)
aws lambda create-alias \
  --function-name mcp-handler \
  --name stable \
  --function-version 1

aws lambda create-alias \
  --function-name mcp-handler \
  --name canary \
  --function-version 2

# Grant API Gateway permission to invoke via both aliases
aws lambda add-permission \
  --function-name mcp-handler:stable \
  --statement-id apigateway-stable \
  --action lambda:InvokeFunction \
  --principal apigateway.amazonaws.com \
  --source-arn "arn:aws:execute-api:us-east-1:123456789012:abc123def/prod/*/*"

aws lambda add-permission \
  --function-name mcp-handler:canary \
  --statement-id apigateway-canary \
  --action lambda:InvokeFunction \
  --principal apigateway.amazonaws.com \
  --source-arn "arn:aws:execute-api:us-east-1:123456789012:abc123def/prod/*/*"

With stage variable routing, deploying a new Lambda version to canary does not require a new API Gateway deployment — only an alias update and a canary settings update on the stage. This significantly reduces the frequency of API Gateway deployment operations, which helps avoid the deployment limit (1,000 deployments per API).

Creating a canary deployment via CLI

The full canary workflow has five phases: prepare, activate, monitor, promote, and finalize.

# --- PHASE 1: PREPARE ---

# Publish a new Lambda version from the current $LATEST
NEW_VERSION=$(aws lambda publish-version \
  --function-name mcp-handler \
  --description "v2 - new tool schema format" \
  --query Version --output text)

echo "New Lambda version: $NEW_VERSION"

# Update the canary alias to point to the new version
aws lambda update-alias \
  --function-name mcp-handler \
  --name canary \
  --function-version $NEW_VERSION

# --- PHASE 2: ACTIVATE CANARY ---

# Create a new deployment (this creates a snapshot of the current API configuration)
NEW_DEPLOYMENT_ID=$(aws apigateway create-deployment \
  --rest-api-id abc123def \
  --stage-name prod \
  --description "v2 canary deployment - new tool schema" \
  --canary-settings \
    percentTraffic=5,\
    stageVariableOverrides=\{lambdaAlias=canary\},\
    useStageCache=false \
  --query id --output text)

echo "Canary deployment ID: $NEW_DEPLOYMENT_ID"

# Verify canary is active
aws apigateway get-stage \
  --rest-api-id abc123def \
  --stage-name prod \
  --query '{baseline: deploymentId, canaryDeployment: canarySettings.deploymentId, canaryPercent: canarySettings.percentTraffic}'

# --- PHASE 3: MONITOR (10-30 minutes) ---

# Watch error rate on canary vs baseline using dimension filter
# Canary requests carry X-Amzn-ApiGateway-Experimental: true header
# Lambda function errors broken down by alias:

aws cloudwatch get-metric-statistics \
  --namespace AWS/Lambda \
  --metric-name Errors \
  --dimensions Name=FunctionName,Value=mcp-handler Name=Resource,Value=mcp-handler:canary \
  --start-time $(date -u -d '30 minutes ago' +%Y-%m-%dT%H:%M:%SZ) \
  --end-time $(date -u +%Y-%m-%dT%H:%M:%SZ) \
  --period 300 \
  --statistics Sum

# If error rate is elevated on canary: ROLL BACK immediately
# aws apigateway delete-stage-canary --rest-api-id abc123def --stage-name prod

# --- PHASE 4: PROMOTE ---

# Ramp canary to 100% before promoting (required step)
aws apigateway update-stage \
  --rest-api-id abc123def \
  --stage-name prod \
  --patch-operations \
    op=replace,path=/canarySettings/percentTraffic,value=100

# Wait 60 seconds to ensure in-flight requests have settled

# --- PHASE 5: FINALIZE ---

# Promote: update the stage's main deploymentId to the canary deploymentId
aws apigateway update-stage \
  --rest-api-id abc123def \
  --stage-name prod \
  --patch-operations \
    op=copy,path=/deploymentId,from=/canarySettings/deploymentId

# Remove canary settings — all traffic now on the promoted deployment
aws apigateway delete-stage-canary \
  --rest-api-id abc123def \
  --stage-name prod

# Update stable alias to point to the new version
aws lambda update-alias \
  --function-name mcp-handler \
  --name stable \
  --function-version $NEW_VERSION

echo "Canary promotion complete. All traffic on v$NEW_VERSION."

The critical ordering constraint is in Phase 4: you must set percentTraffic=100 before promoting. Promoting directly from a partial canary percentage leaves the baseline deployment pointing at old configuration and the stage in an inconsistent state. The AWS documentation describes this as a hard requirement, not a recommendation.

AliveMCP canary monitoring

Standard uptime monitoring treats all responses from an endpoint equally — a 200 is a 200 whether it came from the baseline or canary code path. AliveMCP health probes can be configured to distinguish canary vs baseline responses by inspecting the X-Amzn-ApiGateway-Experimental response header.

When API Gateway routes a request to the canary deployment, it adds X-Amzn-ApiGateway-Experimental: true to the response. Requests served by the baseline do not carry this header. By configuring AliveMCP to probe your MCP endpoint and check for this header's presence, you can create two separate health timelines within a single monitoring configuration:

• Baseline health timeline: Probes that did not receive the X-Amzn-ApiGateway-Experimental header — the stable code path's availability and latency over time.
• Canary health timeline: Probes that did receive the header — the new code path's availability and latency during the rollout window.

To configure AliveMCP for canary-aware monitoring, add a custom header assertion to your probe:

# AliveMCP probe configuration (conceptual — use AliveMCP dashboard to configure)
# Probe: POST https://api.yourmcp.com/prod/mcp
# Expected: 200 OK
# Header assertion: track "X-Amzn-ApiGateway-Experimental" presence

# When the probe receives X-Amzn-ApiGateway-Experimental: true,
# AliveMCP logs the response latency and status under the "canary" label.
# When the header is absent, the response is logged under "baseline".
# Error rate divergence between the two labels signals a canary regression.

# You can also set up a separate AliveMCP monitor pointing to the same endpoint
# but with a header check that fails (returns unhealthy) when
# X-Amzn-ApiGateway-Experimental is NOT present — this gives you a dedicated
# canary-specific health signal rather than a blended one.

The practical benefit during a canary rollout: your AliveMCP dashboard shows two latency curves — baseline P99 (stable, expected) and canary P99 (new version, being evaluated). If the canary P99 climbs while the baseline holds steady, you have clear signal to roll back before the canary reaches a higher traffic percentage. This is more reliable than CloudWatch Lambda duration metrics alone because AliveMCP probes the full round-trip latency including API Gateway overhead, not just Lambda execution time.

Failure modes