SQS FIFO Message Deduplication for MCP Servers — MessageDeduplicationId vs ContentBasedDeduplication

How the deduplication window works

When a message arrives at a FIFO queue, SQS records the deduplication token and the arrival time. Any subsequent message with the same token received within 5 minutes of the first is silently accepted (SQS returns success) but not enqueued — the consumer never sees it. After 5 minutes the token expires, and a message with the same token is treated as a new, distinct message.

Two important consequences for MCP probe pipelines:

• Retry safety: if your probe worker crashes after sending to SQS but before recording the send locally, a retry within 5 minutes is safe — the duplicate is dropped. After 5 minutes, a retry enqueues a second copy. Size your retry timeout accordingly.
• Idempotent reprocessing: replaying a probe result from a dead-letter queue that was originally sent within the last 5 minutes will be deduplicated. Wait for the window to expire before replaying DLQ messages that should re-enter the pipeline.

ContentBasedDeduplication: when it works and when it silently breaks

When ContentBasedDeduplication is enabled on a queue, SQS computes a SHA-256 hash of the raw message body and uses that as the deduplication token. You do not supply a MessageDeduplicationId — SQS derives it automatically.

This is safe only when your message bodies are fully deterministic for logically identical events:

The failure mode is silent: SQS accepts the message, returns HTTP 200, but never enqueues a second copy when you expect it to. Your consumer processes only the first copy of a legitimately distinct probe result, and you may never notice — the DLQ stays empty, no errors are thrown, messages simply vanish.

MessageDeduplicationId: format best practices

An explicit MessageDeduplicationId is a string up to 128 characters, alphanumeric and !"#$%&'()*+,-./:;<=>?@[\]^_`{|}~. It must be unique per distinct logical event within the 5-minute window.

Recommended formats for MCP monitoring pipelines:

// Probe result: one result per endpoint per run — {endpointId}-{probeRunId}
const deduplicationId = `${endpointId}-${probeRunId}`;
// Example: "server-abc123-run-9f4e2b"

// Alert dispatch: one alert per endpoint per incident — {endpointId}-{incidentId}
const deduplicationId = `alert-${endpointId}-${incidentId}`;
// Example: "alert-server-abc123-inc-20261029T120000Z"

// Verdict-minute aggregation: one aggregated verdict per endpoint per minute
const minute = new Date().toISOString().slice(0, 16).replace(':', '');  // "2026-10-29T1200"
const deduplicationId = `verdict-${endpointId}-${minute}`;
// Example: "verdict-server-abc123-2026-10-29T1200"

Key rules:

• Do not use raw UUIDs as the sole component — they are unique per invocation, so two retries of the same logical event get different IDs and both enqueue.
• Derive the ID from the logical identity of the event (endpoint + run + type), not from the message content or send timestamp.
• Keep the ID stable across retries of the same event — if your worker retries on timeout, the ID must be pre-computed before the first send and stored locally, not re-derived from a new UUID.
• For consumer-side retry from a DLQ, preserve the original MessageDeduplicationId from the DLQ message attributes when re-enqueuing to the source queue after the 5-minute window.

Enabling deduplication in CDK

import { Queue, QueueEncryption, DeduplicationScope, FifoThroughputLimit } from 'aws-cdk-lib/aws-sqs';

// Option A — content-based (only for deterministic payloads)
const probeCommandQueue = new Queue(this, 'ProbeCommandQueue', {
  queueName: 'mcp-probe-commands.fifo',
  fifo: true,
  contentBasedDeduplication: true,         // SQS derives token from SHA-256(body)
  visibilityTimeout: Duration.seconds(30),
  encryption: QueueEncryption.KMS_MANAGED,
});

// Option B — explicit MessageDeduplicationId (recommended for probe results)
const probeResultQueue = new Queue(this, 'ProbeResultQueue', {
  queueName: 'mcp-probe-results.fifo',
  fifo: true,
  contentBasedDeduplication: false,        // caller must supply MessageDeduplicationId
  visibilityTimeout: Duration.seconds(30),
  encryption: QueueEncryption.KMS_MANAGED,
});

// Sending with explicit ID (AWS SDK v3)
import { SQSClient, SendMessageCommand } from '@aws-sdk/client-sqs';
const sqs = new SQSClient({});

await sqs.send(new SendMessageCommand({
  QueueUrl: probeResultQueueUrl,
  MessageGroupId: endpointId,              // per-endpoint ordering
  MessageDeduplicationId: `${endpointId}-${probeRunId}`,
  MessageBody: JSON.stringify(probeResult),
}));

Deduplication scope in high-throughput FIFO mode

Standard FIFO queues deduplicate across the entire queue — the token is global. High-throughput FIFO queues (enabled by setting DeduplicationScope=messageGroup and FifoThroughputLimit=perMessageGroupId) scope deduplication to the message group: the same MessageDeduplicationId can be safely reused across different MessageGroupId values.

const highThroughputQueue = new Queue(this, 'HighThroughputProbeQueue', {
  queueName: 'mcp-probe-ht.fifo',
  fifo: true,
  contentBasedDeduplication: false,
});

// Enable high-throughput FIFO via escape hatch (CDK L1)
const cfnQueue = highThroughputQueue.node.defaultChild as CfnQueue;
cfnQueue.deduplicationScope = 'messageGroup';
cfnQueue.fifoThroughputLimit = 'perMessageGroupId';

For AliveMCP's per-endpoint probe pipeline, high-throughput FIFO makes sense only if you have many endpoints each generating probe results at high frequency. With message-group scoped deduplication, a MessageDeduplicationId of run-{probeRunId} (short form, without endpoint prefix) is sufficient — two different endpoints can use the same short form without collision, because their MessageGroupId values differ.

Common failure modes