SNS Subscription DLQ and Delivery Retry for MCP Servers — Per-Subscription Dead-Letter Queues, Exponential Backoff

SNS subscription DLQ vs SQS queue DLQ

For an SNS→SQS fan-out pipeline, you need both: the SNS subscription DLQ catches messages that SNS could not write to the SQS queue; the SQS DLQ catches messages that were delivered to the queue but the Lambda consumer failed to process. An SQS queue with a resource policy error would produce SNS subscription DLQ entries; a consumer bug would produce SQS DLQ entries.

SNS delivery retry policy phases

SNS retries failed deliveries in four sequential phases. The default policy totals 100,015 attempts over 23 days — far too many retries for most MCP monitoring workloads where stale events are useless. Configure a custom policy to fail fast and route to DLQ.

Recommended policy for MCP monitoring (fast-fail): 3 immediate retries, 2 at 5-second delay, 3 at exponential backoff up to 60 seconds, 0 post-backoff retries. Exhaustion (8 retries total) routes to DLQ within ~3 minutes. A probe result that cannot be delivered within 3 minutes is stale for alerting purposes.

Configuring a subscription DLQ

// Step 1: Create DLQ for SNS subscription failures
const snsDlq = new sqs.Queue(this, "SnsDeliveryDlq", {
  queueName: "sns-delivery-failures",
  retentionPeriod: Duration.days(14),
});

// Step 2: Add resource policy to allow SNS to write to the DLQ
// Note: the Source must be the SUBSCRIPTION ARN, not the topic ARN
// Use a wildcard for the source when the subscription ARN is not yet known (created after topic)
snsDlq.addToResourcePolicy(new iam.PolicyStatement({
  effect: iam.Effect.ALLOW,
  principals: [new iam.ServicePrincipal("sns.amazonaws.com")],
  actions: ["sqs:SendMessage"],
  resources: [snsDlq.queueArn],
  conditions: {
    ArnLike: {
      "aws:SourceArn": `arn:aws:sns:${Stack.of(this).region}:${Stack.of(this).account}:*`,
    },
  },
}));

// Step 3: Configure subscription with RedrivePolicy (via CfnSubscription for DLQ support)
// CDK L2 SqsSubscription does not directly expose RedrivePolicy — use escape hatch
const subscription = new sns_subscriptions.SqsSubscription(myQueue, {
  rawMessageDelivery: true,
});
const cfnSubscription = topic.addSubscription(subscription).node
  .defaultChild as sns.CfnSubscription;

cfnSubscription.redrivePolicy = {
  deadLetterTargetArn: snsDlq.queueArn,
};

CDK escape hatch note: As of CDK v2, the SqsSubscription construct exposes deadLetterQueue as a prop — use that directly instead of the escape hatch above if available in your CDK version:

// CDK v2.x — preferred approach
topic.addSubscription(
  new sns_subscriptions.SqsSubscription(myQueue, {
    rawMessageDelivery: true,
    deadLetterQueue: snsDlq,  // sets RedrivePolicy automatically
  })
);

Custom delivery retry policy for fast-fail

// Set custom delivery policy on the subscription via CLI (CDK L2 does not expose DeliveryPolicy)
aws sns set-subscription-attributes \
  --subscription-arn arn:aws:sns:us-east-1:123456789012:mcp-probe-results:uuid \
  --attribute-name DeliveryPolicy \
  --attribute-value '{
    "healthyRetryPolicy": {
      "numRetries": 8,
      "numNoDelayRetries": 3,
      "numMinDelayRetries": 2,
      "numMaxDelayRetries": 0,
      "minDelayTarget": 5,
      "maxDelayTarget": 60,
      "backoffFunction": "exponential"
    }
  }'

# Verify current policy
aws sns get-subscription-attributes \
  --subscription-arn arn:aws:sns:us-east-1:123456789012:mcp-probe-results:uuid \
  --query 'Attributes.DeliveryPolicy'

Delivery status logging to CloudWatch Logs

SNS can log per-delivery outcomes (success and failure) to CloudWatch Logs. This is the only way to see why a delivery failed — without logging, SNS delivery failures are invisible (the Publish API call succeeds; failures are silent). Enable logging per-protocol on the topic (not per-subscription).

// IAM role for SNS to write delivery logs
const snsLoggingRole = new iam.Role(this, "SnsLoggingRole", {
  assumedBy: new iam.ServicePrincipal("sns.amazonaws.com"),
  inlinePolicies: {
    Logging: new iam.PolicyDocument({
      statements: [new iam.PolicyStatement({
        actions: [
          "logs:CreateLogGroup",
          "logs:CreateLogStream",
          "logs:PutLogEvents",
          "logs:DescribeLogGroups",
          "logs:DescribeLogStreams",
        ],
        resources: ["*"],
      })],
    }),
  },
});

// Attach logging role to topic for SQS protocol deliveries
const cfnTopic = topic.node.defaultChild as sns.CfnTopic;
cfnTopic.sqsSuccessFeedbackRoleArn = snsLoggingRole.roleArn;
cfnTopic.sqsFailureFeedbackRoleArn = snsLoggingRole.roleArn;
cfnTopic.sqsSuccessFeedbackSampleRate = "5";  // log 5% of successes; 100% of failures always logged

Delivery logs appear in the CloudWatch log group /aws/sns/us-east-1/123456789012/DirectPublishToPhoneNumber (for SMS) or the platform-specific group. For SQS subscribers, logs are in /aws/sns/{region}/{account-id}/{topic-name}. Each log event contains status, destination, providerResponse, dwellTimeMs, and deliveryId.

Monitoring DLQ depth with CloudWatch

# Alert when SNS delivery DLQ is non-empty (any message = delivery infrastructure problem)
aws cloudwatch put-metric-alarm \
  --alarm-name "sns-delivery-dlq-non-empty" \
  --namespace AWS/SQS \
  --metric-name ApproximateNumberOfMessagesVisible \
  --dimensions Name=QueueName,Value=sns-delivery-failures \
  --threshold 1 \
  --comparison-operator GreaterThanOrEqualToThreshold \
  --evaluation-periods 1 \
  --period 60 \
  --statistic Sum \
  --alarm-actions arn:aws:sns:us-east-1:123456789012:ops-alerts \
  --treat-missing-data notBreaching

For AliveMCP monitoring pipelines, an SNS delivery DLQ alarm is a tier-1 alert: if probe results are not reaching the alerting consumer, AliveMCP itself has a blind spot and may not be generating alerts for endpoints that are down. Treat a non-empty SNS DLQ with the same urgency as a monitoring system outage.

DLQ message structure

When SNS writes to the subscription DLQ after exhausting retries, the SQS message contains the original SNS message body plus a special deadLetterReason field in the message attributes:

{
  "Body": "<original SNS message JSON or raw body>",
  "MessageAttributes": {
    "deadLetterReason": {
      "DataType": "String",
      "StringValue": "NO_ENDPOINT_REACHABLE"
    },
    "RetryAttempts": {
      "DataType": "Number",
      "StringValue": "8"
    },
    "UndeliverableTimestamp": {
      "DataType": "String",
      "StringValue": "2026-10-29T12:34:56.789Z"
    }
  }
}

Common deadLetterReason values and their root causes:

Common failure modes