SNS Fan-Out to SQS for MCP Servers — Parallel Processing, Raw Message Delivery, Cross-Account

Fan-out topology

A standard fan-out topology for an MCP monitoring pipeline:

                     ┌──────────────────────────┐
                     │   MCP Probe Worker         │
                     │   (Lambda / ECS task)      │
                     └──────────┬───────────────-─┘
                                │ SNS Publish
                                ▼
                     ┌──────────────────────────┐
                     │   SNS Topic               │
                     │   mcp-probe-results        │
                     └─────────┬────────────────-─┘
          ┌──────────┬─────────┼─────────┬──────────────┐
          │          │         │         │              │
          ▼          ▼         ▼         ▼              ▼
       SQS:        SQS:      SQS:      SQS:          SQS:
     alerting    analytics   audit   s3-archiver   dashboard
     (filter:    (all)      (all)    (all)         (filter:
     unhealthy)                                    latencyMs
                                                   >2000)

Each queue is a separate subscription with its own retry policy, DLQ, consumer, and optionally its own filter policy. The alerting queue might only receive unhealthy events via a filter policy; the analytics queue receives all events. A failure in the s3-archiver consumer (e.g., S3 bucket permissions error) does not affect the alerting queue — they are completely independent pipelines.

SQS queue resource policy requirement

Every SQS queue that receives SNS deliveries must have an access policy explicitly granting sqs:SendMessage to the SNS topic principal. Without this policy, SNS cannot write to the queue and silently discards the message (no error on the Publish call — the delivery simply fails and enters SNS retry/DLQ logic).

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": { "Service": "sns.amazonaws.com" },
    "Action": "sqs:SendMessage",
    "Resource": "arn:aws:sqs:us-east-1:123456789012:mcp-alerting-queue",
    "Condition": {
      "ArnEquals": {
        "aws:SourceArn": "arn:aws:sns:us-east-1:123456789012:mcp-probe-results"
      }
    }
  }]
}

Always scope with aws:SourceArn. A policy that grants sqs:SendMessage to sns.amazonaws.com without a source ARN condition allows any SNS topic in any account to write to your queue — a confused deputy vulnerability. The ArnEquals condition locks permission to your specific topic only.

Raw message delivery vs SNS envelope

By default, SNS wraps the original message in a JSON envelope before writing to the SQS queue. The SQS consumer must parse the envelope to reach the original body. With RawMessageDelivery: true, the original message body is written directly — the SQS consumer receives exactly what the publisher sent.

// Without raw delivery — consumer must unwrap SNS envelope
const sqsMessage = await receiveMessage(queueUrl);
const snsEnvelope = JSON.parse(sqsMessage.Body);
const probeResult = JSON.parse(snsEnvelope.Message);  // double-parse

// With raw delivery — consumer receives original JSON directly
const sqsMessage = await receiveMessage(queueUrl);
const probeResult = JSON.parse(sqsMessage.Body);  // single-parse

CDK: fan-out with multiple queue subscribers

import * as sns from "aws-cdk-lib/aws-sns";
import * as sqs from "aws-cdk-lib/aws-sqs";
import * as subscriptions from "aws-cdk-lib/aws-sns-subscriptions";
import { Duration } from "aws-cdk-lib";

const probeResultsTopic = new sns.Topic(this, "ProbeResultsTopic", {
  topicName: "mcp-probe-results",
});

// Queue 1: alerting — only unhealthy/degraded events
const alertingQueue = new sqs.Queue(this, "AlertingQueue", {
  visibilityTimeout: Duration.seconds(30),
  deadLetterQueue: { queue: alertingDlq, maxReceiveCount: 3 },
});
probeResultsTopic.addSubscription(
  new subscriptions.SqsSubscription(alertingQueue, {
    rawMessageDelivery: true,
    filterPolicy: {
      status: sns.SubscriptionFilter.stringFilter({
        allowlist: ["degraded", "unhealthy"],
      }),
    },
  })
);

// Queue 2: analytics — all events, raw delivery
const analyticsQueue = new sqs.Queue(this, "AnalyticsQueue", {
  visibilityTimeout: Duration.minutes(5),   // analytics may be slower to process
  retentionPeriod: Duration.days(4),
});
probeResultsTopic.addSubscription(
  new subscriptions.SqsSubscription(analyticsQueue, {
    rawMessageDelivery: true,
    // no filter policy — receives all events
  })
);

// Queue 3: s3-archiver — all events, with envelope (consumer uses MessageId for dedup)
const archiverQueue = new sqs.Queue(this, "ArchiverQueue", {
  visibilityTimeout: Duration.seconds(60),
});
probeResultsTopic.addSubscription(
  new subscriptions.SqsSubscription(archiverQueue, {
    rawMessageDelivery: false,  // keep envelope — archiver uses MessageId for idempotency
  })
);

Cross-account fan-out

SNS can deliver to SQS queues in a different AWS account. The queue account must grant sqs:SendMessage in its queue resource policy, and the SNS topic must be configured with a policy that allows cross-account publish if the topic is in a different account than the publisher.

// Cross-account SQS queue resource policy (in the target account)
{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": { "Service": "sns.amazonaws.com" },
    "Action": "sqs:SendMessage",
    "Resource": "arn:aws:sqs:us-east-1:999999999999:cross-account-analytics-queue",
    "Condition": {
      "ArnEquals": {
        "aws:SourceArn": "arn:aws:sns:us-east-1:123456789012:mcp-probe-results"
      }
    }
  }]
}

// Subscribe the cross-account queue via CLI (CDK does not support cross-account topic subscriptions natively)
aws sns subscribe \
  --topic-arn arn:aws:sns:us-east-1:123456789012:mcp-probe-results \
  --protocol sqs \
  --notification-endpoint arn:aws:sqs:us-east-1:999999999999:cross-account-analytics-queue \
  --profile source-account-profile

Encryption and cross-account: If the SQS queue uses an AWS-managed key (aws/sqs), SNS cannot deliver cross-account because SNS does not have permission to use the managed key from another account. Use a customer-managed KMS key (CMK) with an explicit key policy granting kms:GenerateDataKey and kms:Decrypt to the sns.amazonaws.com service principal for cross-account encrypted fan-out.

Cross-region fan-out constraints

Common failure modes