AWS SQS and EventBridge for MCP Servers: Queuing, FIFO, DLQ, Pub/Sub, and Lambda ESM — Five Production Patterns

The case for async tool call processing

MCP tool calls in a synchronous architecture run inside the JSON-RPC request-response cycle. The MCP client sends tools/call and blocks. For tools that complete in under a second — cache lookups, arithmetic, small database reads — synchronous execution is fine and introduces no unnecessary complexity.

Three failure modes expose the limits of synchronous MCP tool handlers at production scale:

Protocol timeout: MCP clients impose a timeout on tool calls, typically 30–120 seconds depending on client and configuration. A Bedrock Agents tool call that waits for a multi-step workflow, a web_scrape tool that fetches and parses a slow page, or a code execution tool that compiles and runs a project can exceed this. The client receives a timeout error. The tool worker continues executing. The result is silently discarded.

Unordered concurrent writes: An agent that fans out three parallel tool calls to a shared resource (a document, a database row, a state machine) with no serialization guarantee can apply writes in any order. Two calls racing to update the same document may produce a merged result that is a random interleaving of both writes. Standard SQS queues cannot solve this — only FIFO queues with a per-session MessageGroupId enforce sequential delivery within a session context.

Synchronous fan-out brittleness: A tool that must notify an audit log, a webhook dispatcher, and a session summarizer when it completes must call all three synchronously. If any one of the three is slow, the tool blocks. If any one is down, the tool fails. EventBridge pub/sub removes this coupling — the tool emits one event and returns; subscribers react independently.

The five patterns below address each failure mode with the minimum AWS infrastructure that solves it cleanly.

Pattern 1: SQS standard queue — tool call buffering and retry absorption

The SQS standard queue pattern decouples the MCP server's request-acceptance latency from its tool execution latency. The MCP server enqueues the tool call and returns a pending token immediately; a pool of workers drains the queue at their own pace; results are written to a shared store (DynamoDB, Redis) keyed by the tool call ID that the MCP server returns to the client.

The operational parameter that causes the most production incidents is visibility timeout. The SQS default is 30 seconds. For any tool that can run longer than 30 seconds — which includes almost all tools that call external APIs — the default causes duplicate execution: the first worker's visibility timeout expires, SQS re-delivers to a second worker, and both execute simultaneously. Set visibility timeout to at least 3× your p99 tool execution time and extend it within the worker loop for long-running tools using ChangeMessageVisibilityCommand on a fixed interval shorter than half the visibility timeout.

Long polling is the second misconfiguration source. Short polling (WaitTimeSeconds omitted or 0) returns immediately whether or not messages are available — burning 144,000 SQS API calls per day per worker on empty responses. Set WaitTimeSeconds: 20 on every ReceiveMessage call. Combined with MaxNumberOfMessages: 10, a single worker thread processes up to 10 messages per call and idles efficiently between bursts.

The combined failure mode table for standard SQS tool queues:

Pattern 2: SQS FIFO — ordered tool execution within a session

The FIFO queue pattern enforces strict ordering within a message group. For MCP tool sequences that are stateful — create_document followed by append_section, or a database transaction spanning multiple INSERT calls — FIFO with MessageGroupId = sessionId guarantees that tool calls within a session are delivered to workers in the order they were sent.

MessageGroupId granularity is the key design decision. Using a single constant across all messages collapses throughput to one worker at a time. Using sessionId limits concurrency to one worker per active session — which is correct for stateful sequences — while allowing all sessions to proceed in parallel. Using userId provides global ordering per user at the cost of forcing all of a user's tool calls to serialize even when they're independent.

Deduplication strategy is the second design decision. ContentBasedDeduplication (SHA-256 of message body) silently fails to deduplicate when bodies are non-deterministic — a sentAt timestamp, a request UUID, or any field that changes between retries. Always use explicit MessageDeduplicationId set to the stable toolCallId that the MCP server generates when it receives the tools/call RPC. This ID is stable across retries and exactly identifies the logical invocation.

FIFO queues support up to 300 messages/second per queue by default (3,000 with batching). Enable high-throughput FIFO mode by setting DeduplicationScope=messageGroup and FifoThroughputLimit=perMessageGroupId — this scales to 300,000 messages/second when the message group count is high. High-throughput FIFO requires ContentBasedDeduplication=false, which reinforces the recommendation to use explicit MessageDeduplicationId.

Pattern 3: Dead-letter queue — poison pill quarantine and replay

The DLQ pattern converts an infinite retry loop into an inspectable, replayable queue. Without a DLQ, a message that always fails processing (malformed JSON, a missing required field, a downstream API that always 500s on this specific input) recirculates through the source queue until its retention period expires — masking root causes and generating log noise that buries actionable failures.

Three operational details determine DLQ correctness:

maxReceiveCount: Set to 3–5 for tool workers that call external APIs where transient failures are expected (network blip, momentary throttle). Set to 2 for pure business logic with no external dependencies. Too low (1) causes transient failures to DLQ immediately — false positive alerts. Too high (10+) delays detection of genuine bugs — a bad message loops for 10 attempts before being quarantined.

DLQ retention longer than source retention: Set the DLQ's MessageRetentionPeriod to 14 days (maximum) and the source queue's retention to 4 days (default). If the DLQ retention is shorter than the source, messages can expire from the DLQ before an engineer has a chance to inspect them — the original payload is lost.

FIFO DLQ for FIFO source: A FIFO source queue requires a FIFO DLQ. Attempting to configure a standard queue as the DLQ for a FIFO source returns InvalidParameterValue: The dead-letter queue of a FIFO queue must also be a FIFO queue. This is a creation-time error — it surfaces at CDK deploy or CloudFormation update time, but only if you try to set it. Always pre-create the FIFO DLQ with the .fifo suffix before configuring the source queue's redrive policy.

After fixing the root cause of a DLQ batch, use StartMessageMoveTask to replay the quarantined messages back to the source queue. Never manually re-send DLQ messages using SendMessage — that creates new message objects with new IDs, losing the original message attributes, deduplication IDs, and FIFO group membership.

Pattern 4: EventBridge pub/sub — decoupled tool side effects

The EventBridge pattern decouples a tool handler from its downstream side effects. Instead of the tool calling the audit logger, the webhook dispatcher, and the session summarizer synchronously, it emits a ToolCallCompleted event to a custom EventBridge bus and returns. Each subscriber has its own EventBridge rule that matches the event pattern and delivers the event to a Lambda, SQS queue, or API Gateway endpoint. Adding a new subscriber requires no changes to the tool handler. Removing a broken subscriber doesn't affect tool execution.

Custom bus always: Never publish application events to the default event bus. The default bus receives events from dozens of AWS services — mixing application events with AWS service events makes pattern matching noisy and exposes tool call payloads to any rule running on the default bus. Custom buses are free; use one per application domain.

Event pattern priority: EventBridge evaluates event patterns by matching source first (fastest), then detail-type, then detail field matches (slowest, requires deserializing the full payload). Always filter on source and detail-type first. Only add detail field filters for further narrowing within a detail-type.

Target DLQ is not the same as SQS queue DLQ: When EventBridge delivers an event to an SQS queue and the delivery fails (the SQS queue is throttling, or the event size exceeds the queue's max message size), EventBridge retries with exponential backoff for up to 24 hours by default. Events that exhaust all retries are dropped unless you configure a dead-letter queue on the EventBridge target — this is separate from the SQS queue's own redrive policy. Always attach a target DLQ to every EventBridge rule target and set maxEventAge to 1–6 hours to fail fast rather than retrying for 24 hours.

256 KB event size limit: EventBridge events are capped at 256 KB. Tool results containing file contents, LLM responses, or code execution output can exceed this. Store large results in S3 and include only the S3 URI in the event detail. Downstream subscribers retrieve the full result from S3. This also reduces EventBridge costs — pricing is per event, not per byte, but staying well under the size limit avoids validation failures.

Pattern 5: Lambda SQS event source mapping — managed queue drain

The Lambda SQS ESM pattern eliminates polling infrastructure: Lambda manages the queue drain automatically, scales to match queue depth, and handles visibility timeout extension during function execution. For MCP tool workers, the ESM is the lowest-operational-overhead consumption pattern — no poller to maintain, automatic concurrency scaling, and direct integration with SQS DLQ via maxReceiveCount.

ReportBatchItemFailures is mandatory: Without it, a single failed message in a batch of 10 causes all 10 to retry. With it, the handler returns a batchItemFailures array naming only the failed message IDs — Lambda deletes the successful messages and re-queues only the failures. For MCP tool workers where batches contain independent tool calls from different sessions, this prevents a single bad tool call from causing unrelated tool calls to re-execute.

Visibility timeout vs Lambda function timeout: Lambda extends the visibility timeout of messages it's processing automatically, every minute, during function execution. But if the queue's VisibilityTimeout is shorter than the Lambda function's timeout, Lambda's extension calls come too late after the VT has already expired. The SQS queue delivers the message to another Lambda invocation simultaneously. Set the queue's visibility timeout to at least 6× the Lambda function's timeout.

ScalingConfig.MaximumConcurrency: Without it, a queue burst scales the ESM to consume all of the function's unreserved concurrency — potentially thousands of simultaneous invocations. For MCP tool workers calling LLM APIs or database connections, this saturates downstream rate limits and connection pools. Set MaximumConcurrency to match the downstream API's sustainable request rate divided by average requests per tool call.

FIFO queues and ESM ordering: A Lambda ESM consuming a FIFO queue processes one message group at a time, preserving in-group order. A failure in one batch blocks the entire group until the failing message is successfully processed or moved to the DLQ. Return a batchItemFailures entry for the failed message and also for all subsequent messages in the group — then stop processing. This preserves FIFO ordering semantics: you don't want to execute step 5 of a tool sequence if step 3 failed.

Combined failure mode reference

All five patterns interact in a complete async MCP tool infrastructure. The combined failure mode table covers the cross-pattern failure modes that aren't obvious from reading each pattern in isolation:

Choosing the right pattern for your MCP tool

The five patterns compose — a real MCP async infrastructure typically uses all of them together. The decision flow:

Start with standard SQS + Lambda ESM for any tool that can exceed the MCP client timeout or has bursty demand. This handles 90% of async requirements: visibility timeout sized to tool duration, long polling via ESM, partial failure reporting for batch resilience, and SQS DLQ for poison pill quarantine.

Add FIFO only when tool calls within a session have a hard ordering dependency — when call N must complete before call N+1 begins. If tool calls are independent (parallel web searches, independent document reads), standard queues have higher throughput and simpler configuration.

Add EventBridge when tool side effects need to fan out to multiple independent subscribers. Don't add it for single-subscriber fan-out — direct invocation or an SQS queue is simpler. EventBridge's value is the decoupling: subscribers can be added or removed without touching the tool handler.

Scale Lambda ESM MaximumConcurrency to match your downstream API's rate limit, not your queue depth. A queue can hold millions of messages; your downstream LLM API might sustain 100 requests/second. Cap the ESM at 100 concurrent invocations (assuming each invocation makes one downstream call) to avoid amplifying throttle pressure.