CloudFormation Drift Detection for MCP Server Infrastructure

Drift detection API

Drift detection is asynchronous. StartStackDriftDetection returns a detection ID; DescribeStackDriftDetectionStatus polls for completion:

# Start drift detection for a stack
DETECTION_ID=$(aws cloudformation detect-stack-drift \
  --stack-name mcp-server-prod \
  --query "StackDriftDetectionId" \
  --output text)

echo "Detection ID: $DETECTION_ID"

# Poll until DETECTION_COMPLETE
while true; do
  STATUS=$(aws cloudformation describe-stack-drift-detection-status \
    --stack-drift-detection-id "$DETECTION_ID" \
    --query "DetectionStatus" \
    --output text)
  echo "Status: $STATUS"
  if [ "$STATUS" = "DETECTION_COMPLETE" ]; then break; fi
  if [ "$STATUS" = "DETECTION_FAILED" ]; then
    echo "Detection failed" && exit 1
  fi
  sleep 10
done

# Check overall stack drift status
STACK_STATUS=$(aws cloudformation describe-stack-drift-detection-status \
  --stack-drift-detection-id "$DETECTION_ID" \
  --query "StackDriftStatus" \
  --output text)
echo "Stack drift status: $STACK_STATUS"
# Outputs: DRIFTED | IN_SYNC | NOT_CHECKED

Detection takes 1–5 minutes depending on stack size. During detection, the stack status shows DRIFT_DETECTION_IN_PROGRESS. Stack updates are blocked while detection is running — do not run detection during active deployments.

Listing drifted resources and property differences

After detection completes, describe-stack-resource-drifts returns the per-resource breakdown. The PropertyDifferences array shows exactly which properties diverge, with expected (template) and actual (live) values:

# List all drifted resources
aws cloudformation describe-stack-resource-drifts \
  --stack-name mcp-server-prod \
  --stack-resource-drift-status-filters MODIFIED DELETED \
  --query "StackResourceDrifts[*].{Resource:LogicalResourceId,Type:ResourceType,Status:StackResourceDriftStatus}" \
  --output table

# Get full property differences for a specific resource
aws cloudformation describe-stack-resource-drifts \
  --stack-name mcp-server-prod \
  --stack-resource-drift-status-filters MODIFIED \
  --query "StackResourceDrifts[?LogicalResourceId=='McpServerService'].PropertyDifferences"

# Example output — ECS service DesiredCount drifted from 2 to 4
[
  {
    "PropertyPath": "/DesiredCount",
    "ExpectedValue": "2",       // value in CloudFormation template
    "ActualValue": "4",         // value currently deployed in AWS
    "DifferenceType": "NOT_EQUAL"
  }
]

# Security group rule added manually — shows up as ADD difference
[
  {
    "PropertyPath": "/SecurityGroupIngress/2",
    "ExpectedValue": null,
    "ActualValue": "{\"CidrIp\":\"10.0.5.0/24\",\"FromPort\":8080,\"ToPort\":8080,\"IpProtocol\":\"tcp\"}",
    "DifferenceType": "ADD"
  }
]

DifferenceType values: NOT_EQUAL (property exists in both but values differ), ADD (property exists in actual but not in template), REMOVE (property exists in template but not in actual). ADD is the most dangerous for security — it means something was added outside of IaC.

Common drift sources in MCP server deployments

These are the most common manual changes that cause drift in MCP server infrastructure:

# 1. ECS desired count adjusted during incident response
#    Template says: DesiredCount: 2
#    Console changed to: DesiredCount: 6 for traffic spike
#    Next cfn deploy: silently reverts to 2 — traffic spike returns
#    Fix: use Application Auto Scaling instead of hardcoded DesiredCount

# 2. Lambda environment variable updated to rotate a secret
#    Team changed OPENAI_API_KEY in Lambda console (not in Secrets Manager)
#    Next cfn deploy: reverts to old value — secret is invalid
#    Fix: store secrets in Secrets Manager, not in Lambda environment variables

# 3. Security group rule added for emergency debugging access
#    Engineer added port 9229 inbound rule for Node.js debugger
#    Rule was never removed, not in template
#    Fix: use SSM Session Manager for debugging — no inbound port required

# 4. IAM role inline policy modified to grant temporary access
#    Extra s3:GetObject permission added directly to role
#    Not tracked in template — appears as ADD in drift output
#    Fix: always update template, submit PR, and deploy via pipeline

# 5. CloudFront cache behavior modified manually
#    Operator added a new Cache-Control header override in CloudFront console
#    Next deploy: override is removed
#    Fix: manage all CloudFront settings via CloudFormation/CDK

# Scheduled drift detection via EventBridge — run weekly
DetectDriftRule:
  Type: AWS::Events::Rule
  Properties:
    ScheduleExpression: "cron(0 9 ? * MON *)"  # Mondays at 9am UTC
    Targets:
      - Id: DetectDrift
        Arn: !GetAtt DetectDriftFunction.Arn

Which resource types support drift detection

Not all CloudFormation resource types support drift detection. The most relevant for MCP servers:

# Supported (drift detection works)
AWS::ECS::Service          # DesiredCount, TaskDefinition, NetworkConfiguration
AWS::ECS::TaskDefinition   # ContainerDefinitions (immutable — replacement only)
AWS::Lambda::Function      # Code, Environment, Timeout, MemorySize, Layers
AWS::Lambda::Alias         # FunctionVersion, RoutingConfig
AWS::EC2::SecurityGroup    # SecurityGroupIngress, SecurityGroupEgress
AWS::IAM::Role             # AssumeRolePolicyDocument, Policies, ManagedPolicyArns
AWS::IAM::Policy            # PolicyDocument
AWS::RDS::DBInstance        # DBInstanceClass, MultiAZ, AutoMinorVersionUpgrade
AWS::DynamoDB::Table       # ProvisionedThroughput, GlobalSecondaryIndexes, TTLSpecification
AWS::ElastiCache::ReplicationGroup  # AutomaticFailoverEnabled, NumCacheClusters
AWS::CloudFront::Distribution  # DistributionConfig (most properties)
AWS::ApiGateway::Stage     # MethodSettings, Variables
AWS::SSM::Parameter        # Value, Type (checks current value vs template)
AWS::SQS::Queue            # VisibilityTimeout, MessageRetentionPeriod

# NOT SUPPORTED — always shows NOT_CHECKED
Custom::*                  # Any custom resource type
AWS::CloudFormation::Stack # Nested stacks — must detect drift per stack
AWS::StepFunctions::StateMachine  # Definition changes not detectable
AWS::CodePipeline::Pipeline  # Limited support

Nested stacks must have drift detection run independently on each nested stack — the parent stack's drift detection does not cascade into nested stack resources. Run detection on every stack that contains stateful or security-sensitive resources.

Remediation — three approaches

Drift detection does not auto-remediate. Three options after finding drift:

# Option 1: Update the template to match reality (accept the drift)
# Use when the manual change was intentional and correct
# 1. Export actual resource state to CloudFormation format
aws cloudformation describe-stack-resource-drifts \
  --stack-name mcp-server-prod \
  --stack-resource-drift-status-filters MODIFIED \
  --query "StackResourceDrifts[?LogicalResourceId=='McpServerService'].PropertyDifferences"
# 2. Update template property to match actual value
# 3. Submit PR, review, merge, deploy
# 4. Drift is resolved because template now matches actual

# Option 2: Revert the resource to match the template
# Use when the manual change was a mistake or temporary
# Simply run a stack update — CloudFormation reverts resource to template state
aws cloudformation deploy \
  --stack-name mcp-server-prod \
  --template-file template.yaml \
  --parameter-overrides file://params-prod.json \
  --no-fail-on-empty-changeset

# Option 3: Import the resource under CloudFormation management
# Use when a resource was created outside CF and needs to be brought under IaC
aws cloudformation create-change-set \
  --stack-name mcp-server-prod \
  --change-set-name import-existing-resource \
  --change-set-type IMPORT \
  --resources-to-import '[
    {
      "ResourceType": "AWS::DynamoDB::Table",
      "LogicalResourceId": "McpSessionsTable",
      "ResourceIdentifier": { "TableName": "mcp-sessions-prod" }
    }
  ]' \
  --template-body file://template-with-import.yaml