SNS Advanced Patterns for MCP Server Monitoring Pipelines: FIFO Topics, Filter Policies, Fan-Out, and Delivery Reliability

Pattern 1 — SNS Fan-Out: Single Publish, Parallel Independent Consumers

Why fan-out, and why it is not the same as multiple SQS queues

A naive solution to "I need multiple consumers for probe results" is to write to multiple SQS queues from the probe worker. This works, but it has two structural problems. First, the probe worker is now tightly coupled to the consumer topology — adding a fourth consumer requires modifying and redeploying the probe worker. Second, a slow or failing consumer cannot be isolated: if the probe worker writes to queues A, B, and C sequentially, a timeout on queue B delays the write to C.

The SNS fan-out pattern solves both problems. The probe worker publishes one message to one SNS topic. SNS delivers it to all subscribed queues in parallel, before the Publish call returns to the caller. Each queue gets an independent copy with its own visibility timeout, DLQ, consumer Lambda, and failure domain. Adding a fifth consumer requires creating a new SQS queue and subscription — no changes to the probe worker.

                     ┌──────────────────────────┐
                     │   MCP Probe Worker         │
                     │   (Lambda / ECS task)      │
                     └──────────┬───────────────-─┘
                                │ SNS Publish (one call)
                                ▼
                     ┌──────────────────────────┐
                     │   SNS Topic               │
                     │   mcp-probe-results        │
                     └─────────┬────────────────-─┘
          ┌──────────┬─────────┼─────────┬──────────────┐
          │          │         │         │              │
          ▼          ▼         ▼         ▼              ▼
       SQS:        SQS:      SQS:      SQS:          SQS:
     alerting    analytics   audit   s3-archiver   dashboard
     (filter:    (all)      (all)    (all)         (filter:
     unhealthy)                                    latencyMs
                                                   >2000)

The queue resource policy requirement — and the security trap

SNS will silently discard messages and enter its retry loop if it cannot write to an SQS queue. No error is returned to the publisher — from the probe worker's perspective, every Publish call succeeded. The most common cause is a missing or overly broad SQS queue resource policy.

Every subscribed queue must have an access policy statement granting sqs:SendMessage to the sns.amazonaws.com service principal, conditioned on the source topic ARN:

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": { "Service": "sns.amazonaws.com" },
    "Action": "sqs:SendMessage",
    "Resource": "arn:aws:sqs:us-east-1:123456789012:mcp-alerting-queue",
    "Condition": {
      "ArnEquals": {
        "aws:SourceArn": "arn:aws:sns:us-east-1:123456789012:mcp-probe-results"
      }
    }
  }]
}

Omitting the ArnEquals condition — granting sqs:SendMessage to all sns.amazonaws.com principals unconditionally — is a confused deputy vulnerability. Any SNS topic in any AWS account can write to your queue. CDK's SqsSubscription construct adds the correct conditioned policy automatically when you subscribe via topic.addSubscription(); manual subscription via the CLI or direct CloudFormation does not add it — you must write the resource policy explicitly.

Raw message delivery vs the SNS envelope

By default, SNS wraps the original message body in a JSON envelope before writing to the SQS queue. The SQS consumer receives an outer JSON object with fields Type, MessageId, TopicArn, Timestamp, SignatureVersion, and — critically — Message, which is the original body serialized as a JSON string. The consumer must parse twice: JSON.parse(sqsMsg.Body).Message gives the envelope, then JSON.parse(envelope.Message) gives the original payload.

Enable RawMessageDelivery: true per subscription when the consumer is purpose-built for this topic and does not need SNS metadata. The SQS message body is then the exact bytes the publisher sent — one JSON parse instead of two:

// CDK: three-queue fan-out with different raw delivery settings
const probeResultsTopic = new sns.Topic(this, "ProbeResultsTopic");

// Alerting queue: raw delivery + filter for non-healthy events
probeResultsTopic.addSubscription(
  new subscriptions.SqsSubscription(alertingQueue, {
    rawMessageDelivery: true,
    filterPolicy: {
      status: sns.SubscriptionFilter.stringFilter({
        allowlist: ["degraded", "unhealthy"],
      }),
    },
  })
);

// Analytics queue: raw delivery, all events
probeResultsTopic.addSubscription(
  new subscriptions.SqsSubscription(analyticsQueue, {
    rawMessageDelivery: true,
    // no filter — receives every probe result
  })
);

// Archiver queue: envelope delivery — archiver uses MessageId for idempotency
probeResultsTopic.addSubscription(
  new subscriptions.SqsSubscription(archiverQueue, {
    rawMessageDelivery: false,
  })
);

Cross-account fan-out

SNS can deliver to an SQS queue in a different AWS account. The destination queue's resource policy must explicitly grant sqs:SendMessage to the source account's SNS topic ARN. CDK does not support cross-account topic subscriptions natively — create the subscription via CLI from the account that owns the topic:

aws sns subscribe \
  --topic-arn arn:aws:sns:us-east-1:123456789012:mcp-probe-results \
  --protocol sqs \
  --notification-endpoint arn:aws:sqs:us-east-1:999999999999:cross-account-analytics-queue

If the destination queue uses an AWS-managed key (aws/sqs), cross-account SNS delivery fails with KMSAccessDeniedException. AWS-managed keys cannot have key policies modified to grant cross-account access. Switch to a customer-managed KMS key (CMK) and add a key policy statement granting kms:GenerateDataKey and kms:Decrypt to the sns.amazonaws.com principal from the source account.

Pattern 2 — Subscription Filter Policies: Attribute-Based Routing Without a Router

Why filter policies instead of consumer-side filtering

Without filter policies, every subscriber receives every message published to the topic. An alerting Lambda that only cares about status=unhealthy still receives — and is billed for — every healthy probe result. At 60-second probe intervals across 1,000 MCP endpoints, that is 1,440,000 Lambda invocations per day for a consumer that acts on roughly 0.1% of them.

SNS filter policies move the filtering decision to the SNS delivery layer. Messages that do not match a subscription's filter policy are never delivered to that subscriber — they are evaluated at the topic side before writing to the queue or invoking the Lambda. Non-matching messages are still billed as published at the topic level, but they do not generate SQS messages, Lambda invocations, or SQS queue storage costs downstream.

Filter scope: message attributes vs message body

The filter policy scope determines what SNS evaluates. Set via the FilterPolicyScope subscription attribute:

For AliveMCP, the probe worker controls the publish call, so it sets explicit MessageAttributes:

await snsClient.send(new PublishCommand({
  TopicArn: process.env.STATUS_TOPIC_ARN,
  Message: JSON.stringify({ endpointId, status, latencyMs, region }),
  MessageAttributes: {
    status:    { DataType: "String", StringValue: status },          // "healthy"|"degraded"|"unhealthy"
    region:    { DataType: "String", StringValue: region },          // "us-east-1", "eu-west-1", ...
    severity:  { DataType: "String", StringValue: getSeverity(status, latencyMs) },
    latencyMs: { DataType: "Number", StringValue: String(latencyMs) },
  },
}));

Note DataType: "Number" for latencyMs. Numeric filter operators only evaluate attributes published with DataType: "Number". Publishing latency as a String attribute and then writing a numeric filter policy results in the filter never matching — a silent failure that routes no events to the queue regardless of the latency value.

String and numeric operators

String operators apply to String-typed attributes or string-valued body fields:

Numeric operators apply to Number-typed attributes:

AND and OR semantics — the most common confusion

Every team hits this eventually. The rules are:

• OR: Multiple values within one key's array. "status": ["degraded","unhealthy"] matches if status is either value.
• AND: Multiple top-level keys in the filter policy object. Every key condition must be satisfied.

// Subscription 1: critical alerts — European endpoints, non-healthy status
// (status is "degraded" OR "unhealthy") AND (region starts with "eu-")
{
  "status": ["degraded", "unhealthy"],
  "region": [{ "prefix": "eu-" }]
}

// Subscription 2: SLO latency breach — any region, any status, latency > 5 seconds
{
  "latencyMs": [{ "numeric": [">", 5000] }]
}

// Subscription 3: ops pager — high latency AND critical/high severity
{
  "latencyMs": [{ "numeric": [">", 3000] }],
  "severity": ["high", "critical"]
}

The third filter: a message with latencyMs=4500 (matches >3000) but severity="low" (does not match ["high","critical"]) will not be delivered to that subscription. AND semantics require all conditions to match.

Limit: 5 KB, 5 attributes, 150 values per attribute

Each subscription has a 5 KB filter policy size limit, a maximum of 5 AND conditions (top-level keys), and a maximum of 150 values per key's array. These limits are per subscription — you can create multiple subscriptions to the same queue with different filter policies if you need more complex routing.

The 15-minute propagation delay after SetSubscriptionAttributes is the most operationally dangerous property. Testing a filter policy change immediately after applying it will often show the old behavior. When debugging filter policies in staging, wait the full 15 minutes before concluding the policy is wrong.

Pattern 3 — FIFO Topics: Per-Endpoint Ordered State Transitions

When ordering matters and when it does not

Standard SNS topics deliver at-least-once with best-effort ordering. For most probe result events — independent snapshots of endpoint health that are each self-contained — out-of-order delivery is harmless. If a healthy result for endpoint A arrives before a degraded result for endpoint A, but both are processed within seconds, the downstream state reflects the latest state after both are processed.

The ordering invariant breaks when consumers implement a state machine over a sequence of events. An MCP endpoint monitoring system that transitions through healthy → degraded → unhealthy → recovering → healthy produces incorrect alert states if the state machine receives events out of order. If unhealthy arrives before degraded, a consumer that suppresses alerts for repeated same-state events may suppress the degraded transition entirely, producing a gap in the alert history.

FIFO topics enforce strict delivery order within each MessageGroupId and deduplicate within a 5-minute window, at the cost of subscriber type restrictions and throughput limits.

FIFO vs Standard: the tradeoff table

MessageGroupId granularity: why per-endpoint is the right choice

Every message published to a FIFO topic requires a MessageGroupId. Messages in the same group are delivered in strict publish order. Choosing the granularity of your GroupId has a direct impact on throughput and blast radius:

The per-endpoint model is almost always correct for MCP monitoring. The 300 msg/s FIFO limit that seems like a bottleneck is per message group — with 10,000 monitored endpoints, you have 10,000 independent groups each with their own 300 msg/s budget.

await client.send(new PublishCommand({
  TopicArn: "arn:aws:sns:us-east-1:123456789012:mcp-status.fifo",
  MessageGroupId: `endpoint-${endpointId}`,         // per-endpoint ordering scope
  MessageDeduplicationId: `${endpointId}-${probeRunId}`,  // explicit dedup ID
  Message: JSON.stringify({ endpointId, status, probeRunId, probeAt: new Date().toISOString() }),
  MessageAttributes: {
    status: { DataType: "String", StringValue: status },
  },
}));

Deduplication: explicit ID vs content-based

Two approaches, with a non-obvious trap in the content-based path:

Monitoring FIFO delivery: the silent stall pattern

The blocking property of FIFO queues means a single unprocessable message can silently delay all subsequent events in the same message group. If an endpoint's probe-result consumer fails to process message N and moves it to the DLQ, all subsequent probe results for that endpoint queue behind message N+1. The main queue depth appears non-zero for that group while the DLQ has one message.

AliveMCP detects this indirectly: if the probe result timestamp in the status page diverges from the real probe time by more than the expected processing lag, the consumer is likely blocked on a DLQ item. The direct detection is a CloudWatch alarm on FIFO DLQ depth:

# DLQ depth alarm — any item in FIFO DLQ = a message group is blocked
aws cloudwatch put-metric-alarm \
  --alarm-name "mcp-status-fifo-dlq-blocked" \
  --namespace AWS/SQS \
  --metric-name ApproximateNumberOfMessagesVisible \
  --dimensions Name=QueueName,Value=mcp-status-dlq.fifo \
  --threshold 1 \
  --comparison-operator GreaterThanOrEqualToThreshold \
  --evaluation-periods 1 \
  --period 60 \
  --statistic Sum \
  --alarm-actions arn:aws:sns:us-east-1:123456789012:ops-alerts

Pattern 4 — Per-Subscription DLQ and Fast-Fail Delivery Policy: Making Silent Failures Visible

The fundamental visibility problem with SNS

The SNS Publish API always returns HTTP 200 if the message was accepted by the topic — regardless of what happens to downstream delivery. A probe worker that publishes a result successfully has no way to know from the Publish response whether the alerting queue received the message, whether the Lambda subscriber was invoked, or whether the delivery failed and SNS entered its internal retry loop.

This means three classes of production failures are invisible by default:

• SQS queue resource policy missing or wrong — SNS silently retries, eventually routes to subscription DLQ
• Lambda subscriber under concurrency throttle — SNS retries, may exhaust and route to DLQ if throttle persists
• FIFO queue DLQ blocking a group — main queue accumulates, no error, no alert unless you instrument it

Two mechanisms close this gap: delivery status logging (makes per-delivery outcomes visible in CloudWatch Logs) and per-subscription DLQ with custom retry policy (provides a last-resort recovery path and an alarming surface).

SNS subscription DLQ vs SQS queue DLQ — a critical distinction

Both types are SQS queues. Their difference is what failure they capture and where they are configured:

For an SNS→SQS fan-out pipeline, you need both types. A missing queue resource policy produces SNS subscription DLQ entries. A consumer bug produces SQS DLQ entries. Looking only at the SQS DLQ leaves delivery infrastructure failures invisible.

The 23-day default retry problem

SNS's default delivery retry policy retries a failed SQS delivery 100,015 times over 23 days before routing to the subscription DLQ. The breakdown: 3 immediate retries (default numRetries=3), followed by 100,000 retries at the max delay (numMaxDelayRetries=100000, default maxDelayTarget=20s). The result: a message with a misconfigured queue resource policy sits in SNS retry hell for 23 days while your alerting pipeline silently discards every probe result for that endpoint.

For MCP monitoring, a probe result that cannot be delivered within a few minutes is stale and useless for alerting. Configure a fast-fail policy:

aws sns set-subscription-attributes \
  --subscription-arn arn:aws:sns:us-east-1:123456789012:mcp-probe-results:uuid \
  --attribute-name DeliveryPolicy \
  --attribute-value '{
    "healthyRetryPolicy": {
      "numRetries": 8,
      "numNoDelayRetries": 3,
      "numMinDelayRetries": 2,
      "numMaxDelayRetries": 0,
      "minDelayTarget": 5,
      "maxDelayTarget": 60,
      "backoffFunction": "exponential"
    }
  }'

This policy exhausts in approximately 3 minutes: 3 immediate retries, 2 at 5 seconds, 3 at exponential backoff up to 60 seconds, then route to DLQ. Zero post-backoff retries — numMaxDelayRetries: 0 is the key setting that eliminates the 23-day tail.

CDK: wiring the subscription DLQ

import * as sns from "aws-cdk-lib/aws-sns";
import * as sqs from "aws-cdk-lib/aws-sqs";
import * as subscriptions from "aws-cdk-lib/aws-sns-subscriptions";
import * as iam from "aws-cdk-lib/aws-iam";
import { Duration, Stack } from "aws-cdk-lib";

// Create DLQ for SNS delivery failures (distinct from the main queue's consumer DLQ)
const snsDeliveryDlq = new sqs.Queue(this, "SnsDeliveryDlq", {
  queueName: "sns-delivery-failures",
  retentionPeriod: Duration.days(14),
});

// Resource policy: SNS must write to the DLQ on delivery failure
// Use ArnLike (not ArnEquals) because the subscription ARN is not known at stack creation time
snsDeliveryDlq.addToResourcePolicy(new iam.PolicyStatement({
  effect: iam.Effect.ALLOW,
  principals: [new iam.ServicePrincipal("sns.amazonaws.com")],
  actions: ["sqs:SendMessage"],
  resources: [snsDeliveryDlq.queueArn],
  conditions: {
    ArnLike: {
      "aws:SourceArn": `arn:aws:sns:${Stack.of(this).region}:${Stack.of(this).account}:*`,
    },
  },
}));

// CDK v2.x: deadLetterQueue prop on SqsSubscription (preferred — no escape hatch needed)
topic.addSubscription(
  new subscriptions.SqsSubscription(alertingQueue, {
    rawMessageDelivery: true,
    deadLetterQueue: snsDeliveryDlq,  // sets RedrivePolicy on the subscription automatically
  })
);

Delivery status logging: the only way to see why a delivery failed

Without delivery logging, SNS delivery failures produce no CloudWatch metrics, no logs, and no errors. Enable per-protocol delivery logging on the topic by attaching an IAM role that allows SNS to write to CloudWatch Logs:

// IAM role for delivery logs
const snsLoggingRole = new iam.Role(this, "SnsLoggingRole", {
  assumedBy: new iam.ServicePrincipal("sns.amazonaws.com"),
  inlinePolicies: {
    Logging: new iam.PolicyDocument({
      statements: [new iam.PolicyStatement({
        actions: ["logs:CreateLogGroup","logs:CreateLogStream","logs:PutLogEvents",
                  "logs:DescribeLogGroups","logs:DescribeLogStreams"],
        resources: ["*"],
      })],
    }),
  },
});

// Attach to the SNS topic (CDK escape hatch — L2 does not expose these props)
const cfnTopic = topic.node.defaultChild as sns.CfnTopic;
cfnTopic.sqsSuccessFeedbackRoleArn = snsLoggingRole.roleArn;
cfnTopic.sqsFailureFeedbackRoleArn = snsLoggingRole.roleArn;
cfnTopic.sqsSuccessFeedbackSampleRate = "5";  // log 5% of successes; 100% of failures always logged

Logs appear in /aws/sns/{region}/{account-id}/{topic-name}. Each log event contains status (SUCCESS or FAILURE), destination (queue ARN), providerResponse (the SQS error message when status is FAILURE), and dwellTimeMs (time SNS spent on this delivery attempt).

Reading DLQ messages: the deadLetterReason taxonomy

When SNS routes a message to the subscription DLQ after exhausting retries, the SQS message has a deadLetterReason message attribute that identifies the root cause:

Cross-Pattern Failure Modes

Sequencing Guidance: Build in Layers

The four patterns compose naturally into a layered architecture. Build in this order:

1. Fan-out first. Establish the topology: one SNS topic, three or four SQS queues, correct resource policies. This is the structural layer everything else builds on. Do not add filtering or FIFO semantics until the basic fan-out is verified working end-to-end with a test message.
2. Filter policies second. Once the topology is confirmed, add subscription filter policies to reduce downstream processing cost and noise. Test each filter by publishing test messages with known attributes and verifying only the expected queues receive them — wait the full 15-minute propagation window before concluding a policy is wrong.
3. FIFO topics third (only if ordering matters). Migrate the main status topic to FIFO if the downstream state machine requires ordered delivery. This is an irreversible change to the topic type and affects all subscriptions simultaneously. Most MCP monitoring workloads do not need FIFO — per-endpoint per-minute probe results are rarely order-sensitive if the timestamp is in the payload.
4. DLQ + delivery logging last — but not optional. A monitoring pipeline without delivery visibility is itself an unmonitored system. Add per-subscription DLQs, the fast-fail delivery policy, and delivery status logging before declaring the pipeline production-ready. The DLQ alarm on non-empty depth is the signal that the monitoring infrastructure has a blind spot.

For AliveMCP's probe architecture — Lambda probe workers publishing results every 60 seconds per endpoint — the practical deployment is: standard SNS topic (not FIFO, since probe timestamps in the payload make ordering redundant), fan-out to alerting/analytics/archiving queues with subscription filter policies, and per-subscription DLQ with 8-retry fast-fail policy on all three subscriptions. The alerting queue subscription uses status: ["degraded","unhealthy"] and severity: ["high","critical"] AND filter; the analytics queue has no filter; the archiving queue has no filter but enables envelope delivery for MessageId-based deduplication in the S3 writer.