MCP Server SQS Dead-Letter Queue — maxReceiveCount, redrive, poison pill detection

What the DLQ catches and why the default (no DLQ) is dangerous

SQS standard queues have an "approximate receive count" for each message. When a consumer receives a message, processes it, and does not delete it (because the consumer crashed, threw an exception, or exceeded the visibility timeout), SQS increments the receive count and re-delivers the message after the visibility timeout expires. This continues indefinitely until the message's MessageRetentionPeriod expires.

Without a DLQ this creates the following observable failure pattern in production: a single malformed tool call message causes the worker to throw on every delivery attempt. The exception lands in logs but the message is re-delivered every few minutes. If the worker has multiple threads or Lambda concurrency > 1, multiple workers may crash simultaneously on the same message. The queue depth does not decrease because the message is never deleted. Other good messages queue behind it in the same visibility window. Engineers notice elevated error rates and queue depth but must grep logs to correlate the failure to a specific message ID.

With a DLQ (and maxReceiveCount: 3), the same message is delivered 3 times, then moved to the DLQ. The DLQ depth alarm fires within 1 minute. An engineer inspects the DLQ message (full payload preserved) and identifies the root cause — usually within 5 minutes of the alarm.

import {
  SQSClient,
  GetQueueAttributesCommand,
  SetQueueAttributesCommand,
} from '@aws-sdk/client-sqs';

const sqs = new SQSClient({ region: process.env.AWS_REGION ?? 'us-east-1' });

// Verify DLQ is configured on source queue
async function verifyDLQConfigured(sourceQueueUrl: string): Promise {
  const attrs = await sqs.send(new GetQueueAttributesCommand({
    QueueUrl: sourceQueueUrl,
    AttributeNames: ['RedrivePolicy', 'VisibilityTimeout'],
  }));

  const redrivePolicy = attrs.Attributes?.RedrivePolicy;
  if (!redrivePolicy) {
    throw new Error(`Source queue ${sourceQueueUrl} has no DLQ configured — add RedrivePolicy before production traffic`);
  }

  const policy = JSON.parse(redrivePolicy);
  console.log({
    dlqArn: policy.deadLetterTargetArn,
    maxReceiveCount: policy.maxReceiveCount,
  });
}

maxReceiveCount sizing

maxReceiveCount is the number of times SQS delivers a message before moving it to the DLQ. The right value depends on whether failures are expected to be transient (network blip, downstream throttle that clears in seconds) or deterministic (bad message that always fails).

For MCP tool workers that call external APIs, transient failures are normal — a downstream service throttles for 5 seconds, the worker throws, the message is re-delivered after the visibility timeout, and the second attempt succeeds. maxReceiveCount: 1 would move the message to the DLQ on the first transient failure, generating false alerts. maxReceiveCount: 5 gives 5 delivery attempts, which covers most transient failure windows without masking genuine bugs for too long.

For tool calls that should never fail (deterministic business logic, no external dependency), a lower threshold like maxReceiveCount: 2 catches bugs faster. The tradeoff: an accidental Lambda timeout on the first attempt (cold start + slow tool) sends the message to DLQ after just two attempts at maxReceiveCount: 2.

A practical default for MCP tool workers: maxReceiveCount: 3 with a visibility timeout set to 3× p99 tool duration. This gives 3 genuine processing attempts with enough time for each attempt to complete before being considered failed.

// Setting up source queue with DLQ via AWS CLI
// Step 1: Create the DLQ first
// aws sqs create-queue --queue-name tool-calls-dlq \
//   --attributes MessageRetentionPeriod=1209600  # 14 days

// Step 2: Get DLQ ARN
// DLQ_ARN=$(aws sqs get-queue-attributes \
//   --queue-url $DLQ_URL \
//   --attribute-names QueueArn \
//   --query Attributes.QueueArn --output text)

// Step 3: Set redrive policy on source queue
// aws sqs set-queue-attributes --queue-url $SOURCE_URL \
//   --attributes "{\"RedrivePolicy\":\"{\\\"deadLetterTargetArn\\\":\\\"$DLQ_ARN\\\",\\\"maxReceiveCount\\\":3}\"}"

// CDK equivalent:
import { Queue } from 'aws-cdk-lib/aws-sqs';
import { Duration } from 'aws-cdk-lib';

const dlq = new Queue(stack, 'ToolDLQ', {
  queueName: 'tool-calls-dlq',
  retentionPeriod: Duration.days(14),   // longer than source queue retention
});

const toolQueue = new Queue(stack, 'ToolQueue', {
  queueName: 'tool-calls',
  visibilityTimeout: Duration.seconds(360),
  retentionPeriod: Duration.days(4),    // default; DLQ holds messages 14 days for inspection
  deadLetterQueue: {
    queue: dlq,
    maxReceiveCount: 3,
  },
});

ApproximateReceiveCount: near-DLQ diagnostics

The ApproximateReceiveCount message system attribute tells a consumer how many times SQS has delivered this message. It is only available if you request it explicitly in ReceiveMessage — passing AttributeNames: ['ApproximateReceiveCount'] or AttributeNames: ['All'].

Reading this value in every consumer gives you visibility into messages that are consistently failing before they exhaust the threshold and land in the DLQ. A message that has been received twice with maxReceiveCount: 3 is one failed attempt away from the DLQ — that is the best time to emit detailed diagnostics, not after the fact from the DLQ.

import { ReceiveMessageCommand, DeleteMessageCommand } from '@aws-sdk/client-sqs';

async function pollWithReceiveCountDiagnostics(): Promise {
  while (true) {
    const { Messages } = await sqs.send(new ReceiveMessageCommand({
      QueueUrl: process.env.TOOL_QUEUE_URL!,
      MaxNumberOfMessages: 10,
      WaitTimeSeconds: 20,
      MessageAttributeNames: ['All'],
      AttributeNames: ['All'], // includes ApproximateReceiveCount, SentTimestamp, etc.
    }));

    if (!Messages?.length) continue;

    for (const msg of Messages) {
      const receiveCount = parseInt(msg.Attributes?.ApproximateReceiveCount ?? '1', 10);
      const sentTimestamp = parseInt(msg.Attributes?.SentTimestamp ?? '0', 10);
      const ageMs = Date.now() - sentTimestamp;

      if (receiveCount >= 2) {
        // This message has already failed at least once
        console.warn({
          event: 'message_near_dlq_threshold',
          messageId: msg.MessageId,
          receiveCount,
          ageMs,
          body: msg.Body, // log full payload for inspection
        });
      }

      try {
        const payload = JSON.parse(msg.Body!);
        await executeTool(payload);
        await sqs.send(new DeleteMessageCommand({
          QueueUrl: process.env.TOOL_QUEUE_URL!,
          ReceiptHandle: msg.ReceiptHandle!,
        }));
      } catch (err) {
        // Do NOT re-throw; let VT expire and SQS re-deliver
        // After maxReceiveCount attempts, SQS moves to DLQ automatically
        console.error({
          event: 'tool_call_processing_failed',
          messageId: msg.MessageId,
          receiveCount,
          error: String(err),
        });
      }
    }
  }
}

Replaying DLQ messages with StartMessageMoveTask

After identifying and fixing the root cause of a DLQ batch (bad deserialization, a missing field in the worker, a downstream API bug that's now patched), you need to move the quarantined messages back to the source queue for reprocessing. The correct API is StartMessageMoveTask.

Do not manually re-send DLQ messages using SendMessage: that creates new messages with new message IDs and loses the original message attributes, timestamps, and deduplication IDs. StartMessageMoveTask moves the original message objects, preserving all attributes.

import { StartMessageMoveTaskCommand, ListMessageMoveTasksCommand, CancelMessageMoveTaskCommand } from '@aws-sdk/client-sqs';

async function replayDLQ(
  dlqArn: string,
  destinationArn: string, // the source queue ARN, or omit to move back to source
): Promise {
  const { TaskHandle } = await sqs.send(new StartMessageMoveTaskCommand({
    SourceArn: dlqArn,
    DestinationArn: destinationArn,
    MaxNumberOfMessagesPerSecond: 10, // throttle replay to avoid flooding workers
  }));

  console.log({ event: 'dlq_replay_started', taskHandle: TaskHandle });
  return TaskHandle!;
}

// Monitor replay progress
async function monitorReplay(sourceArn: string): Promise {
  const { Results } = await sqs.send(new ListMessageMoveTasksCommand({
    SourceArn: sourceArn,
    MaxResults: 10,
  }));

  for (const task of Results ?? []) {
    console.log({
      taskHandle: task.TaskHandle,
      status: task.Status,          // RUNNING, COMPLETED, CANCELLING, CANCELLED, FAILED
      approximateNumberOfMessagesMoved: task.ApproximateNumberOfMessagesMoved,
      approximateNumberOfMessagesToMove: task.ApproximateNumberOfMessagesToMove,
      failureReason: task.FailureReason,
    });
  }
}

// Cancel a replay if you realize the fix was incomplete
async function cancelReplay(taskHandle: string): Promise {
  await sqs.send(new CancelMessageMoveTaskCommand({ TaskHandle: taskHandle }));
}

Important constraints on StartMessageMoveTask: only one move task can be running per source queue at a time. The source queue must have a redrive allow policy that permits the move (by default DLQs allow moves to any destination). After a successful move task, the messages in the DLQ are deleted — there is no undo once the task completes.

FIFO queues require FIFO DLQs

An SQS FIFO source queue (FifoQueue=true) requires a DLQ that is also a FIFO queue. Attempting to configure a standard queue as a DLQ for a FIFO source returns a InvalidParameterValue error: "The dead-letter queue of a FIFO queue must also be a FIFO queue." This is a creation-time validation for CDK/CloudFormation; it surfaces at aws sqs set-queue-attributes time for manual configuration.

The FIFO DLQ must not share its name with any other queue — create it as a dedicated DLQ with the .fifo suffix. Messages moved to a FIFO DLQ preserve their MessageGroupId and deduplication ID.

// CORRECT: FIFO source with FIFO DLQ
const fifoDlq = new Queue(stack, 'ToolFifoDLQ', {
  queueName: 'tool-calls-dlq.fifo',
  fifo: true,
  retentionPeriod: Duration.days(14),
});

const fifoQueue = new Queue(stack, 'ToolFifoQueue', {
  queueName: 'tool-calls.fifo',
  fifo: true,
  contentBasedDeduplication: false,
  visibilityTimeout: Duration.seconds(360),
  deadLetterQueue: {
    queue: fifoDlq,   // must also be FIFO
    maxReceiveCount: 3,
  },
});

// WRONG: standard DLQ for FIFO source — CreateQueue throws at deploy time
const standardDlq = new Queue(stack, 'WrongDLQ', {
  queueName: 'tool-calls-dlq', // NOT .fifo
  retentionPeriod: Duration.days(14),
});
// fifoQueue.addDeadLetterQueue({ queue: standardDlq, maxReceiveCount: 3 })
// → InvalidParameterValue: The dead-letter queue of a FIFO queue must also be a FIFO queue

DLQ CloudWatch alarm and SNS fan-out

A DLQ without an alarm is nearly useless — you won't know messages landed there until you check manually. The correct alarm metric is NumberOfMessagesSent on the DLQ (not ApproximateNumberOfMessagesVisible, which is a lagging indicator and accumulates across multiple incidents). Set the threshold to 1 with a 1-minute period: alert the moment the first message lands.

import { Alarm, ComparisonOperator, TreatMissingData } from 'aws-cdk-lib/aws-cloudwatch';
import { SnsAction } from 'aws-cdk-lib/aws-cloudwatch-actions';
import { Topic } from 'aws-cdk-lib/aws-sns';

const alertTopic = new Topic(stack, 'ToolAlerts');

// Add your on-call endpoint (email, PagerDuty, Slack webhook) as a subscriber
// alertTopic.addSubscription(new EmailSubscription('oncall@example.com'));

new Alarm(stack, 'DLQAlarm', {
  metric: dlq.metricNumberOfMessagesSent({
    period: Duration.minutes(1),
    statistic: 'Sum',
  }),
  threshold: 1,
  comparisonOperator: ComparisonOperator.GREATER_THAN_OR_EQUAL_TO_THRESHOLD,
  evaluationPeriods: 1,
  treatMissingData: TreatMissingData.NOT_BREACHING,
  alarmDescription: 'Tool call DLQ received messages — investigate immediately',
}).addAlarmAction(new SnsAction(alertTopic));

Failure modes reference