AWS CloudFormation Advanced Patterns for MCP Servers: Custom Resource Contract, Change Set Safety Gates, and Operational Hygiene

Pattern 1 — Custom resource handler contract

CloudFormation custom resources bridge the gap between what CloudFormation supports natively and what your MCP server infrastructure actually needs: registering an endpoint in a discovery service that has no AWS resource type, seeding a database record at stack creation time, creating a DNS record in a provider that CloudFormation doesn't natively support. The Lambda handler receives a lifecycle event (Create, Update, or Delete), performs the operation, and signals back to CloudFormation via a pre-signed S3 URL. Three contract invariants govern correctness.

The one-hour silence trap — always wrap in try/finally

CloudFormation does not time out immediately if the Lambda handler fails to call cfnresponse.send(). It waits the full ServiceTimeout (default 3,600 seconds) before marking the resource failed. An unhandled exception that crashes the Lambda function before the cfnresponse PUT is sent causes a one-hour stall on every affected CloudFormation operation. The fix is a try/finally that unconditionally sends the response:

import cfnresponse from "cfn-response";

export const handler = async (
  event: CloudFormationCustomResourceEvent,
  context: AWSLambda.Context
): Promise<void> => {
  let physicalResourceId = event.PhysicalResourceId ?? "";
  const responseData: Record<string, string> = {};

  try {
    if (event.RequestType === "Create") {
      physicalResourceId = await handleCreate(event.ResourceProperties);
    } else if (event.RequestType === "Update") {
      physicalResourceId = await handleUpdate(
        event.PhysicalResourceId!,
        event.ResourceProperties,
        event.OldResourceProperties!
      );
    } else if (event.RequestType === "Delete") {
      await handleDelete(event.PhysicalResourceId!);
    }

    await cfnresponse.send(event, context, cfnresponse.SUCCESS, responseData, physicalResourceId);
  } catch (err) {
    console.error("Custom resource handler failed:", err);
    await cfnresponse.send(event, context, cfnresponse.FAILED, {
      Error: String(err),
    }, physicalResourceId);
  }
};

The physicalResourceId is initialized before the try block because the FAILED response must also include a physical resource ID. Without it, a failure on Create leaves CloudFormation unable to call Delete on cleanup. Set the Lambda function timeout to 300 seconds (5 minutes) for most operations — this gives the handler 240 seconds of execution time and 60 seconds of buffer to send cfnresponse before hitting the Lambda limit.

Physical resource ID determinism — the Update replacement trap

The physical resource ID uniquely identifies the resource instance across the stack's lifetime. The critical invariant: if your Update handler returns a different physical ID than Create returned, CloudFormation treats this as a replacement — it calls Delete on the old ID after the update succeeds. A random UUID generated per invocation causes replacement on every update, deleting the existing resource on every stack deployment:

// WRONG — random ID breaks the Update contract
async function handleCreate(props: Record<string, string>): Promise<string> {
  const id = randomUUID();   // new ID on every Create AND every Update retry
  await registerEndpoint(id, props.endpoint);
  return id;
}

// CORRECT — deterministic ID derived from stable inputs
async function handleCreate(props: Record<string, string>): Promise<string> {
  const id = `mcp-endpoint-${props.serviceName}-${props.environment}`;

  // Idempotency: check before creating (CloudFormation retries failed Creates)
  const existing = await getEndpointById(id);
  if (existing) {
    console.log(`Endpoint ${id} already exists — returning existing ID (retry)`);
    return id;
  }

  await registerEndpoint(id, props.endpoint);
  return id;
}

// Update — return the SAME physical ID to signal in-place update
async function handleUpdate(
  physicalId: string,
  newProps: Record<string, string>,
  _oldProps: Record<string, string>
): Promise<string> {
  await updateEndpoint(physicalId, newProps.endpoint);
  return physicalId;   // same ID = in-place update
}

// Delete — succeed even if the resource is already gone
async function handleDelete(physicalId: string): Promise<void> {
  try {
    await deregisterEndpoint(physicalId);
  } catch (err: any) {
    if (err.code === "EndpointNotFound") return;  // idempotent Delete
    throw err;
  }
}

The idempotency check in Create is equally important. CloudFormation retries failed Create calls, so a handler that creates the resource then fails before sending SUCCESS creates the resource twice on the retry. Check for existence before creating. For Delete, succeed silently if the resource is already gone — the stack delete must complete even if the resource was cleaned up out-of-band.

CDK cr.Provider for long-running operations

When the operation takes longer than Lambda's 15-minute limit — certificate issuance, large database migrations, waiting for an external API to complete provisioning — use the CDK cr.Provider framework from aws-cdk-lib/custom-resources. It wraps the Lambda boilerplate and adds an optional Step Functions Express Workflow for async stabilization:

import * as cr from "aws-cdk-lib/custom-resources";
import * as lambda from "aws-cdk-lib/aws-lambda";

// Primary handler — kicks off the operation and returns immediately
const handlerFn = new lambda.Function(this, "McpRegistrationHandler", {
  runtime: lambda.Runtime.NODEJS_22_X,
  handler: "index.handler",
  code: lambda.Code.fromAsset("./lambda/mcp-registration"),
  timeout: cdk.Duration.minutes(5),
});

// isComplete handler — polled until it returns IsComplete: true
const isCompleteFn = new lambda.Function(this, "McpRegistrationIsComplete", {
  runtime: lambda.Runtime.NODEJS_22_X,
  handler: "isComplete.handler",
  code: lambda.Code.fromAsset("./lambda/mcp-registration"),
  timeout: cdk.Duration.minutes(2),
});

const provider = new cr.Provider(this, "McpRegistrationProvider", {
  onEventHandler: handlerFn,
  isCompleteHandler: isCompleteFn,   // Step Functions polls this
  queryInterval: cdk.Duration.seconds(30),
  totalTimeout: cdk.Duration.minutes(30),
});

const registration = new cdk.CustomResource(this, "McpRegistration", {
  serviceToken: provider.serviceToken,
  properties: {
    ServiceName: "mcp-tool-server",
    Endpoint: `https://${api.restApiId}.execute-api.${this.region}.amazonaws.com/prod`,
    Environment: props.environment,
  },
  removalPolicy: cdk.RemovalPolicy.DESTROY,
});

// Reference outputs from the custom resource
const registrationId = registration.getAttString("RegistrationId");

cr.Provider handles cfnresponse signaling, the isComplete polling loop via Step Functions Express Workflow, and the IAM permissions between the provider Lambda and Step Functions. The responseData returned from the handler becomes available as !GetAtt LogicalId.AttributeName in CloudFormation templates — flatten nested objects to strings since only string values are supported.

Pattern 2 — Change set safety gates

A CloudFormation change set is a preview of what will change before any infrastructure is modified. For MCP server deployments, change sets answer the question that matters most before execution: will this update trigger a resource replacement that causes downtime or data loss? The change set lifecycle is: create → poll CREATE_COMPLETE → inspect for destructive changes → execute or delete.

Replacement detection in CI/CD

The Replacement field in each change entry is the key signal. Parse it from describe-change-set output and block the pipeline if any stateful resource is being replaced or removed:

# Create and inspect the change set
aws cloudformation create-change-set \
  --stack-name mcp-server-prod \
  --change-set-name "deploy-${{ github.sha }}" \
  --template-body file://template.yaml \
  --parameters file://params-prod.json \
  --capabilities CAPABILITY_IAM

# Wait for completion
aws cloudformation wait change-set-create-complete \
  --stack-name mcp-server-prod \
  --change-set-name "deploy-${{ github.sha }}"

# Check for destructive changes on stateful resource types
CHANGES=$(aws cloudformation describe-change-set \
  --stack-name mcp-server-prod \
  --change-set-name "deploy-${{ github.sha }}" \
  --query "Changes[?ResourceChange.Action=='Remove' || (ResourceChange.Replacement=='True' && (ResourceChange.ResourceType=='AWS::RDS::DBInstance' || ResourceChange.ResourceType=='AWS::DynamoDB::Table' || ResourceChange.ResourceType=='AWS::EFS::FileSystem'))]" \
  --output json)

if [[ "$CHANGES" != "[]" ]]; then
  echo "DESTRUCTIVE CHANGES DETECTED — manual approval required"
  echo "$CHANGES"
  exit 1  # Block pipeline until human review
fi

# Safe to execute
aws cloudformation execute-change-set \
  --stack-name mcp-server-prod \
  --change-set-name "deploy-${{ github.sha }}"

The important distinction: Replacement: True on an ECS TaskDefinition is expected and safe — CloudFormation creates a new revision and the ECS service rolls to it. Replacement: True on an RDS DBInstance or DynamoDB Table is data loss if DeletionPolicy is not set. Filter your replacement checks by resource type, not globally. Action: Remove on any resource always warrants review. The Evaluation: Dynamic change source indicates the replacement depends on a resource reference that changed — the actual replacement will be determined at execution time, not statically.

DeletionPolicy and UpdateReplacePolicy — set them before you need them

These two resource-level policies control what happens when CloudFormation deletes or replaces a resource. The critical gap: they must be set before the destructive event. Adding DeletionPolicy: Retain in the same change set that triggers a replacement does not protect the old resource — the deletion happens in the same stack update before the new policy takes effect:

# CloudFormation YAML — set on every stateful resource at stack creation
McpSessionsTable:
  Type: AWS::DynamoDB::Table
  DeletionPolicy: Retain          # table persists if stack is deleted
  UpdateReplacePolicy: Retain     # table persists if CF replaces it during an update
  Properties:
    TableName: mcp-sessions
    BillingMode: PAY_PER_REQUEST

McpPostgresDatabase:
  Type: AWS::RDS::DBInstance
  DeletionPolicy: Snapshot        # creates final snapshot before deleting
  UpdateReplacePolicy: Snapshot   # creates snapshot before replacement
  Properties:
    DBInstanceClass: db.t4g.medium
    Engine: postgres

McpEfsFileSystem:
  Type: AWS::EFS::FileSystem
  DeletionPolicy: Retain
  UpdateReplacePolicy: Retain
  Properties:
    Encrypted: true

The four values for each policy: Delete (default — resource is deleted), Retain (resource persists, no snapshot), Snapshot (creates final snapshot before deletion — RDS and EFS only), RetainExceptOnCreate (retain on replacement but delete on failed-creation rollback). For production MCP server databases, use plain Retain or Snapshot — not RetainExceptOnCreate.

Stack Policy — the second line of defense IAM cannot bypass

A stack policy is a JSON document attached to the stack that blocks specific update operations on named resources, independently of IAM. An IAM policy that allows cloudformation:UpdateStack does not override a stack policy that denies Update:Replace on a specific resource:

{
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "Update:*",
      "Principal": "*",
      "Resource": "*"
    },
    {
      "Effect": "Deny",
      "Action": ["Update:Replace", "Update:Delete"],
      "Principal": "*",
      "Resource": "LogicalResourceId/McpSessionsTable"
    },
    {
      "Effect": "Deny",
      "Action": ["Update:Replace", "Update:Delete"],
      "Principal": "*",
      "Resource": "LogicalResourceId/McpPostgresDatabase"
    }
  ]
}

# Apply to the production stack
aws cloudformation set-stack-policy \
  --stack-name mcp-server-prod \
  --stack-policy-body file://stack-policy.json

# Override for a planned maintenance window (audit this operation)
aws cloudformation execute-change-set \
  --stack-name mcp-server-prod \
  --change-set-name maintenance-2026-09-26 \
  --stack-policy-during-update-body '{"Statement":[{"Effect":"Allow","Action":"Update:*","Principal":"*","Resource":"*"}]}'

Stack policies cannot be removed once applied — only modified. The override path via --stack-policy-during-update-body requires elevated IAM permissions and should trigger an audit log. This is the correct behavior: the policy is a guardrail, not a setting to flip casually.

Pattern 3 — Operational hygiene: drift, policy enforcement, and fleet management

The third pattern covers what happens after stacks are deployed: detecting manual changes that break the IaC contract, enforcing security policies at synthesis time before templates are generated, and managing infrastructure across a fleet of AWS accounts. All three are recurring operational concerns for MCP server teams that grow beyond a single account.

Drift detection — finding the gaps between template and reality

CloudFormation drift detection compares the actual state of deployed resources against the expected state in the template. For MCP server infrastructure, manual changes are the primary drift source — an engineer adjusts ECS DesiredCount during an incident, adds a security group rule for emergency debugging, or updates a Lambda environment variable to rotate a secret without a stack update. These changes break the IaC contract silently: the next cfn deploy reverts them without warning.

# Start detection and poll to completion
DETECTION_ID=$(aws cloudformation detect-stack-drift \
  --stack-name mcp-server-prod \
  --query "StackDriftDetectionId" \
  --output text)

while true; do
  STATUS=$(aws cloudformation describe-stack-drift-detection-status \
    --stack-drift-detection-id "$DETECTION_ID" \
    --query "DetectionStatus" --output text)
  if [ "$STATUS" = "DETECTION_COMPLETE" ]; then break; fi
  if [ "$STATUS" = "DETECTION_FAILED" ]; then exit 1; fi
  sleep 10
done

# List drifted resources with property-level differences
aws cloudformation describe-stack-resource-drifts \
  --stack-name mcp-server-prod \
  --stack-resource-drift-status-filters MODIFIED DELETED \
  --query "StackResourceDrifts[*].{Resource:LogicalResourceId,Type:ResourceType,Status:StackResourceDriftStatus,Differences:PropertyDifferences}" \
  --output json

The five most common MCP server drift sources, and the structural fixes that prevent them:

The DifferenceType: ADD case is the most dangerous for security — it means something was added outside IaC. Schedule drift detection weekly via EventBridge (cron(0 9 ? * MON *)) to catch these. Custom resources (Custom::*) and Step Functions StateMachines always show NOT_CHECKED — drift detection cannot inspect resources CloudFormation doesn't model internally. Run detection independently on each nested stack; the parent stack's detection does not cascade.

CDK Aspects — policy enforcement at synthesis time

CDK Aspects implement the visitor pattern over the construct tree. Every construct in the tree receives a visit(node: IConstruct) callback during synthesis, before the CloudFormation template is generated. Annotations.of(node).addError() fails the synthesis with a non-zero exit code — the CloudFormation template is never written and the deployment cannot start. This is the correct place for cross-cutting security requirements that must apply to every resource of a given type:

import * as cdk from "aws-cdk-lib";
import * as s3 from "aws-cdk-lib/aws-s3";
import * as dynamodb from "aws-cdk-lib/aws-dynamodb";
import { IConstruct } from "constructs";

class McpSecurityAspect implements cdk.IAspect {
  visit(node: IConstruct): void {
    if (node instanceof s3.Bucket) {
      const cfnBucket = node.node.defaultChild as s3.CfnBucket;
      if (!cfnBucket.bucketEncryption) {
        cdk.Annotations.of(node).addError(
          `S3 bucket '${node.node.path}' must have encryption configured.`
        );
      }
    }

    if (node instanceof dynamodb.Table) {
      const cfnTable = node.node.defaultChild as dynamodb.CfnTable;
      const pitrEnabled =
        cfnTable.pointInTimeRecoverySpecification !== undefined &&
        (cfnTable.pointInTimeRecoverySpecification as any).pointInTimeRecoveryEnabled === true;
      if (!pitrEnabled) {
        cdk.Annotations.of(node).addError(
          `DynamoDB table '${node.node.path}' must have PITR enabled.`
        );
      }
    }

    // Escape hatch — set Lambda logging config CDK doesn't expose yet
    if (node instanceof lambda.Function) {
      const cfnFn = node.node.defaultChild as lambda.CfnFunction;
      if (!cfnFn.loggingConfig) {
        cfnFn.addPropertyOverride("LoggingConfig", {
          LogFormat: "JSON",
          SystemLogLevel: "WARN",
          ApplicationLogLevel: "INFO",
        });
      }
    }
  }
}

const app = new cdk.App();
const stack = new McpServerStack(app, "McpServerProd");
cdk.Aspects.of(stack).add(new McpSecurityAspect());

The escape hatch pattern — accessing the underlying CfnResource via node.defaultChild and calling addPropertyOverride() — lets Aspects mutate properties that the CDK L2 construct doesn't expose. Use this for CDK gaps, not as a workaround for your own construct design.

cdk-nag — 200+ pre-built compliance rules as a single Aspect

The cdk-nag library from AWS Solutions implements a full suite of security and compliance rules as CDK Aspects. Add AwsSolutionsChecks to the app-level scope and every resource in the application gets checked against ~150 rules covering encryption, IAM least privilege, logging, and secure defaults:

import { AwsSolutionsChecks, NagSuppressions } from "cdk-nag";

// Apply to the entire app — runs ~150 checks at synthesis time
cdk.Aspects.of(app).add(new AwsSolutionsChecks({ verbose: true }));

// Every suppression requires a reason string — this creates an auditable record
NagSuppressions.addResourceSuppressions(
  stack.mcpServerRole,
  [
    {
      id: "AwsSolutions-IAM4",
      reason: "AWSLambdaBasicExecutionRole is the minimal policy for Lambda CloudWatch logging",
    }
  ],
  true  // apply to child constructs
);

NagSuppressions.addStackSuppressions(stack, [
  {
    id: "AwsSolutions-S1",
    reason: "Server access logging disabled on artifacts bucket — no sensitive data, recursive log growth risk",
  }
]);

// Rules most relevant to MCP server infrastructure:
// AwsSolutions-IAM5  — wildcard permissions (s3:*, lambda:*)
// AwsSolutions-L1   — non-latest Lambda runtime (nodejs18.x vs nodejs22.x)
// AwsSolutions-SQS3 — SQS queue without DLQ
// AwsSolutions-EC23 — security group with 0.0.0.0/0 inbound
// AwsSolutions-DDB3 — DynamoDB table without PITR
// AwsSolutions-RDS2 — RDS without storage encryption
// AwsSolutions-CFR3 — CloudFront without access logging

The reason field on every suppression is enforced — a suppression without reason is rejected by the library. This creates a documented exception record for every intentional policy deviation, which is valuable for security reviews and audits. In CI/CD, cdk-nag failures are synthesis errors and block deployment before any CloudFormation call is made.

StackSets for fleet management — DELEGATED_ADMIN and operation preferences

CloudFormation StackSets deploy a single template to multiple AWS accounts and regions simultaneously. For MCP server infrastructure, the common use case is deploying a baseline IAM role, CloudWatch log group, and SSM parameter to every account in an AWS Organization so the monitoring infrastructure is pre-provisioned wherever MCP servers are deployed.

# Register a member account as DELEGATED_ADMIN (run from management account)
aws organizations register-delegated-administrator \
  --account-id 222222222222 \
  --service-principal stacksets.cloudformation.amazonaws.com

# Create the StackSet from the delegated admin account
aws cloudformation create-stack-set \
  --stack-set-name mcp-server-baseline \
  --template-body file://mcp-baseline.yaml \
  --permission-model SERVICE_MANAGED \
  --auto-deployment \
    Enabled=true,RetainStacksOnAccountRemoval=false \
  --capabilities CAPABILITY_IAM CAPABILITY_NAMED_IAM

# Deploy to all accounts in an OU — always add --call-as DELEGATED_ADMIN
aws cloudformation create-stack-instances \
  --stack-set-name mcp-server-baseline \
  --deployment-targets \
    OrganizationalUnitIds="ou-prod-abc12345" \
  --regions us-east-1 eu-west-1 ap-southeast-1 \
  --call-as DELEGATED_ADMIN \
  --operation-preferences \
    MaxConcurrentPercentage=25,FailureTolerancePercentage=10,RegionConcurrencyType=PARALLEL

The single most common StackSets mistake: omitting --call-as DELEGATED_ADMIN from the delegated admin account. Without it, the call is treated as a SELF_MANAGED operation, which fails because the legacy admin IAM roles are not present. Every StackSets API call — create, update, delete, list — from the delegated admin account requires this flag.

Operation preferences are the second source of production incidents with StackSets. MaxConcurrentPercentage: 25 means one quarter of the target accounts update at a time. FailureTolerancePercentage: 10 means if 10% of accounts fail, the entire operation halts — protecting the remaining 90% from a broken update. For non-critical baseline infrastructure, you can push to MaxConcurrentPercentage: 50. For production changes, conservative settings at 10/5 are the right default.

Stack instance statuses tell you the fleet's health: CURRENT means up to date with the template; OUTDATED means the StackSet template was updated but this instance hasn't received the update yet — OUTDATED instances do not auto-update, you must run update-stack-instances; INOPERABLE means the stack instance failed in an unrecoverable state requiring manual intervention. Run list-stack-instances as part of your weekly operational review to catch OUTDATED and INOPERABLE instances.

Consolidated failure modes

Production checklist

Custom resources

• Handler body wrapped in try/finally with cfnresponse.send() in finally block
• Physical resource ID derived from stable inputs (not randomUUID())
• Create handler checks for existence before creating (idempotency)
• Delete handler succeeds silently if resource already gone
• Lambda timeout set to 300s; operation completes within 240s
• Long-running operations use cr.Provider with isCompleteHandler + Step Functions polling
• responseData attributes are strings; complex objects flattened

Change set safety

• CI/CD pipeline parses change set output; blocks on Action: Remove and Replacement: True/Conditional for stateful resource types
• DeletionPolicy and UpdateReplacePolicy set on every stateful resource at stack creation
• RDS and EFS: DeletionPolicy: Snapshot; DynamoDB: DeletionPolicy: Retain
• Stack Policy applied to production stacks covering database resources
• --stack-policy-during-update-body restricted to separate elevated IAM role with CloudTrail alert

Operational hygiene

• Weekly drift detection scheduled via EventBridge
• ECS DesiredCount managed by Application Auto Scaling, not hardcoded in template
• Secrets in Secrets Manager or SSM Parameter Store, not Lambda environment variables
• Emergency access via SSM Session Manager, not inbound security group rules
• CDK Aspects applied at app level for encryption, PITR, and log configuration requirements
• cdk-nag AwsSolutionsChecks applied to app; all suppressions have reason strings
• StackSets using SERVICE_MANAGED with DELEGATED_ADMIN member account
• StackSets targeting OUs not individual account IDs
• MaxConcurrentPercentage: 25 and FailureTolerancePercentage: 10 for production StackSet updates
• Weekly list-stack-instances review to catch OUTDATED and INOPERABLE instances