AWS Step Functions for MCP Servers: Three Depth Patterns for Express Workflow Semantics, Callback Orchestration, and Distributed Map Scaling

CDK foundations before the patterns

The aws-stepfunctions and aws-stepfunctions-tasks CDK modules handle ASL synthesis, IAM execution role creation, and CloudWatch Logs delivery configuration automatically — eliminating the most error-prone parts of Step Functions setup when done correctly. Two CDK defaults cause the most production incidents.

payloadResponseOnly: true — the single most impactful LambdaInvoke setting

Without payloadResponseOnly: true, every LambdaInvoke wraps the Lambda return value in a three-field SDK envelope:

{
  "Payload": { "statusCode": 200, "body": "..." },
  "SdkHttpMetadata": { "AllHttpHeaders": { ... }, "HttpStatusCode": 200 },
  "SdkResponseMetadata": { "RequestId": "..." }
}

Every subsequent state that reads $.toolResult is actually reading the envelope, not the Lambda output — the actual body is at $.toolResult.Payload.body. This produces silent data mismatches that only surface when a downstream state tries to access a field that is one level too shallow. With payloadResponseOnly: true, the state receives the Lambda return value directly with no wrapper.

Express state machines require explicit logging — CDK does not add it by default

Standard state machines surface execution history via GetExecutionHistory for 90 days. Express state machines have no execution history API — all execution data exists only in CloudWatch Logs if you configured logging. CDK creates Express state machines without logging unless you pass a logs configuration explicitly:

import * as sfn from "aws-cdk-lib/aws-stepfunctions";
import * as logs from "aws-cdk-lib/aws-logs";

const toolChainLogs = new logs.LogGroup(this, "ToolChainLogs", {
  logGroupName: "/stepfunctions/mcp-tool-chain",
  retention: logs.RetentionDays.ONE_MONTH,
  removalPolicy: cdk.RemovalPolicy.DESTROY,
});

const toolChain = new sfn.StateMachine(this, "ToolChain", {
  definition,
  stateMachineType: sfn.StateMachineType.EXPRESS,
  timeout: cdk.Duration.minutes(5),
  tracingEnabled: true,
  logs: {
    destination: toolChainLogs,
    level: sfn.LogLevel.ALL,         // ALL captures input/output at each state transition
    includeExecutionData: true,      // without this, only state names are logged — no data
  },
});

The execution role needs nine logs:* permissions in addition to invoking Lambda. The non-obvious ones are logs:PutResourcePolicy and logs:DescribeResourcePolicies — Step Functions creates a CloudWatch Logs resource policy to establish the delivery channel, not just writes log events. Missing these two permissions causes the state machine to fail at start time with a permissions error even though it can technically write log events.

Pattern 1 — Express workflow semantics

Express and Standard workflows are not equivalent with different names. They make fundamentally different guarantees around execution semantics, observability, pricing, and maximum duration. Getting the choice wrong means either a state machine that times out on long agent jobs (Express misused as Standard) or one that silently doubles work on retries (Standard misused where at-least-once is acceptable).

Express Sync blocking contract

Express Sync (StartSyncExecution) is the natural fit for MCP tool chains that must complete within 5 minutes and return a result synchronously to the LLM. The API call blocks until the state machine finishes or hits the 5-minute wall:

import { SFNClient, StartSyncExecutionCommand } from "@aws-sdk/client-sfn";

const sfn = new SFNClient({ region: process.env.AWS_REGION });

async function runToolChain(input: Record<string, unknown>): Promise<unknown> {
  const result = await sfn.send(new StartSyncExecutionCommand({
    stateMachineArn: process.env.TOOL_CHAIN_ARN,
    input: JSON.stringify(input),
  }));

  if (result.status === "FAILED") {
    throw new Error(`State machine failed: ${result.error} — ${result.cause}`);
  }
  if (result.status === "TIMED_OUT") {
    throw new Error("Tool chain exceeded 5-minute execution limit");
  }
  return JSON.parse(result.output ?? "{}");
}

The calling Lambda's Timeout must exceed the state machine's expected p99 execution time — not just the average. If Lambda times out before Step Functions returns, the state machine continues running but the caller receives a Lambda timeout error. The LLM sees a tool failure, the state machine continues executing, and the work is eventually discarded with no return path. Set the calling Lambda timeout to the state machine's p99 plus 30 seconds margin.

CloudWatch Logs requirement — the non-obvious IAM

Express state machines charge at $1 per million state transitions. A 6-state tool chain run 200,000 times per month generates 1.2M transitions at $1.20. Standard workflows charge $0.025 per 1,000 execution starts — the same 200,000 runs cost $5. Express is cheaper at high volume with many state transitions per execution. Standard is cheaper for low-volume, long-duration agent jobs where per-start cost is amortized over hours of execution time.

The pricing crossover is approximately 500,000 starts/month for a 6-state pipeline — above that, Express wins on cost. Below that, Standard's exactly-once semantics and 90-day queryable execution history may be worth the premium.

Express Async — the ExecutionDoesNotExist gotcha

Express Async (StartExecution on an Express state machine) returns an execution ARN immediately. The most common mistake: passing that ARN to GetExecutionHistory, which returns ExecutionDoesNotExist — Express executions are simply not visible to the Step Functions history API. Query execution state via CloudWatch Logs Insights or have the state machine write results to DynamoDB as its final state:

// CloudWatch Logs Insights query for Express Async result
// fields @timestamp, @message
// | filter executionArn = "arn:aws:states:..." and type = "ExecutionSucceeded"
// | limit 1

// Or: have the final state write output to DynamoDB directly
// Then poll DynamoDB for the result via a separate MCP "check-status" tool

Monitoring Express health

Without queryable execution history, Express workflow health comes entirely from CloudWatch metrics and Logs. Alarm on all four of these — dimension by StateMachineArn or a noisy Standard pipeline can mask failures in a silent Express one:

Pattern 2 — Callback pattern depth

The wait-for-task-token callback pattern pauses a Step Functions execution mid-workflow and resumes it only when an external system calls SendTaskSuccess — the standard solution for MCP tool calls that trigger asynchronous processing (file conversion, model inference, webhook delivery) where the state machine needs to wait for a result without polling.

Token injection — the .waitForTaskToken suffix

Adding .waitForTaskToken to the resource ARN is what activates the callback pattern. Without the suffix, the Lambda integration returns immediately after Lambda responds. The task token lives in the execution context at $$.Task.Token and must be injected via the Parameters field using the .$ suffix on the key name:

{
  "InvokeAsyncTool": {
    "Type": "Task",
    "Resource": "arn:aws:states:::lambda:invoke.waitForTaskToken",
    "Parameters": {
      "FunctionName": "arn:aws:lambda:us-east-1:ACCOUNT:function:AsyncToolHandler",
      "Payload": {
        "taskToken.$": "$$.Task.Token",
        "toolName.$": "$.toolName",
        "toolInput.$": "$.toolInput",
        "sessionId.$": "$.sessionId"
      }
    },
    "HeartbeatSeconds": 300,
    "TimeoutSeconds": 600,
    "Catch": [
      {
        "ErrorEquals": ["States.HeartbeatTimeout", "States.Timeout"],
        "ResultPath": "$.error",
        "Next": "HandleTimeout"
      }
    ],
    "End": true
  }
}

In CDK, sfn.JsonPath.taskToken is the constant for $$.Task.Token. CDK validates at synth time that JsonPath.taskToken is used only within a WAIT_FOR_TASK_TOKEN integration — a useful guard against the common mistake of injecting the token in a REQUEST_RESPONSE task where it has no effect.

Token durability before work — the correct operation order

The single most important operational rule for the callback pattern: store the task token in DynamoDB before starting the asynchronous work, not after. A worker crash between completing work and writing the token to DynamoDB is unrecoverable — the execution is permanently stuck waiting with no way to resume:

export async function handler(event: {
  taskToken: string;
  toolName: string;
  toolInput: Record<string, unknown>;
  sessionId: string;
}) {
  // Step 1: store token durably BEFORE starting work
  await dynamo.send(new PutItemCommand({
    TableName: process.env.PENDING_TOKENS_TABLE,
    Item: {
      sessionId: { S: event.sessionId },
      taskToken: { S: event.taskToken },
      createdAt: { S: new Date().toISOString() },
      ttl: { N: String(Math.floor(Date.now() / 1000) + 3600) },
    },
    ConditionExpression: "attribute_not_exists(sessionId)",  // idempotency guard
  }));

  // Step 2: dispatch async work — result comes back via webhook
  await dispatchWork(event.toolName, event.toolInput, event.sessionId);

  // Lambda returns here; Step Functions execution remains paused
  // (do NOT call SendTaskSuccess here — that defeats the async pattern)
}

The ConditionExpression: "attribute_not_exists(sessionId)" idempotency guard prevents duplicate token writes if the Lambda is retried by Step Functions (which uses at-least-once invocation semantics for Lambda integrations).

SQS optimized integration — eliminating the Lambda intermediary

The SQS optimized integration avoids the Lambda intermediary entirely for queue-based worker patterns. Step Functions sends a message containing the task token directly to SQS, and a pool of workers picks it up — no Lambda layer needed to store the token or route work:

{
  "SendToWorkerQueue": {
    "Type": "Task",
    "Resource": "arn:aws:states:::sqs:sendMessage.waitForTaskToken",
    "Parameters": {
      "QueueUrl": "https://sqs.us-east-1.amazonaws.com/ACCOUNT/mcp-tool-workers",
      "MessageBody": {
        "taskToken.$": "$$.Task.Token",
        "toolName.$": "$.toolName",
        "toolInput.$": "$.toolInput",
        "sessionId.$": "$.sessionId"
      },
      "MessageGroupId.$": "$.sessionId"
    },
    "HeartbeatSeconds": 120,
    "TimeoutSeconds": 300,
    "End": true
  }
}

The state machine execution role needs only sqs:SendMessage on the queue. SQS guarantees delivery of the token to exactly one worker, so separate DynamoDB token storage is optional — the message is the durable storage. The worker Lambda consuming from SQS calls SendTaskSuccess when done.

HeartbeatSeconds — detecting crashed workers

Without HeartbeatSeconds, a worker crash leaves the execution waiting until TimeoutSeconds elapses — potentially hours. With HeartbeatSeconds: 30, the worker must call SendTaskHeartbeat every 30 seconds or the task fails with States.HeartbeatTimeout. Set the heartbeat interval to 60–70% of HeartbeatSeconds to buffer for network latency — a heartbeat interval equal to HeartbeatSeconds may miss the deadline on a slow call and trigger a false timeout:

async function processWithHeartbeat(taskToken: string, work: () => Promise<unknown>) {
  let done = false;
  const heartbeatInterval = setInterval(async () => {
    if (done) return;
    try {
      await sfn.send(new SendTaskHeartbeatCommand({ taskToken }));
    } catch (err: any) {
      if (err.name === "TaskTimedOut") clearInterval(heartbeatInterval);
    }
  }, 20_000);  // every 20s, 67% of HeartbeatSeconds: 30

  try {
    const result = await work();
    done = true;
    clearInterval(heartbeatInterval);
    return result;
  } catch (err) {
    done = true;
    clearInterval(heartbeatInterval);
    throw err;
  }
}

TaskTimedOut and InvalidToken — non-fatal handling

When the webhook receiver calls SendTaskSuccess, two errors are non-fatal and must be handled gracefully rather than thrown:

• TaskTimedOut — the execution already timed out or the heartbeat window elapsed before the callback arrived. The execution is already done; clean up the DynamoDB token record and return.
• InvalidToken — the token was already used (called twice) or the execution was aborted. Treat as non-fatal; delete the stale record and continue.

try {
  await sfn.send(new SendTaskSuccessCommand({
    taskToken: record.taskToken,
    output: JSON.stringify({ result }),
  }));
  await deleteToken(sessionId);
} catch (err: any) {
  if (err.name === "TaskTimedOut" || err.name === "InvalidToken") {
    await deleteToken(sessionId);
    return;
  }
  throw err;
}

Pattern 3 — Distributed Map scaling

The Distributed Map state runs the same sub-workflow against every item in a large input set with configurable concurrency — the right primitive for MCP tool chains that call the same tool against N inputs (scan N URLs, process N documents, query N endpoints) without writing fan-out coordination code.

Inline mode vs Distributed mode — the 40-item threshold

Inline mode processes items from the execution state input array and collects all results into the parent execution history. Use it for up to 40 concurrent items where you need immediate access to all results without querying S3. Distributed mode creates separate child state machine executions for thousands of items, with S3 as both input source and result store to avoid the 256 KB execution history size limit.

The critical difference is observability: Inline results are visible in the Step Functions console on the parent execution; Distributed results are in S3 files listed in a manifest. The cost difference is also significant — Distributed mode creates real child executions billed at Standard rates per execution-start, while Inline mode has no per-child overhead.

ItemBatcher — 25× cold start reduction

Without ItemBatcher, each item in the input array creates one child execution or Map iteration. For 10,000 items this means 10,000 Lambda cold starts if containers aren't warm. ItemBatcher groups items into batches before each child execution, passing the batch in event.Items:

"ItemBatcher": {
  "MaxItemsPerBatch": 25,            // 25 items per child execution
  "MaxInputBytesPerBatch": 204800,   // 200 KB ceiling — whichever limit hits first
  "BatchInput": {
    "environment": "production",     // static fields merged into every batch's event
    "toolVersion": "2"               // available as event.environment, event.toolVersion
  }
}

// Lambda receives:
export async function handler(event: {
  Items: Array<{ url: string; timeout: number }>;
  environment: string;
  toolVersion: string;
}): Promise<{ results: Array<{ url: string; status: number }> }> {
  const results = await Promise.all(
    event.Items.map(item =>
      probe(item.url, item.timeout)
        .catch(err => ({ url: item.url, status: 0, error: err.message }))
    )
  );
  return { results };
}

With MaxItemsPerBatch: 25 and 10,000 total items, Distributed Map creates 400 child executions instead of 10,000 — a 25× reduction in Lambda cold starts. 400 child executions also stays well within the 300 child-execution-starts-per-second throttle (400 ÷ 300 = 1.3 seconds to start all children at full throttle, plus compute time).

S3ItemReader — processing datasets larger than 256 KB

The 256 KB execution input limit prevents passing large item arrays directly to the Map state. S3ItemReader streams items from S3 objects without materializing the full file in memory — multi-GB input files are supported:

"ItemReader": {
  "Resource": "arn:aws:states:::s3:getObject",
  "ReaderConfig": {
    "InputType": "NDJSON"   // or "JSON" for JSON array
  },
  "Parameters": {
    "Bucket": "my-mcp-batch-inputs",
    "Key": "runs/2026-09-26/inputs.ndjson"
  }
}

Use NDJSON (InputType: "NDJSON") for large files — each line is one Map item, the format can be produced incrementally (append-only), and it parses line-by-line without loading the entire structure. JSON array format (InputType: "JSON") requires the entire file to be valid JSON — a partially written file is unparseable. The execution role needs s3:GetObject on the input bucket.

ToleratedFailurePercentage — partial success for batch tool calls

By default a single item failure causes the entire Map state to fail and cancel all in-flight iterations. For MCP batch calls where partial success is acceptable (scan 10,000 endpoints; if 5% timeout, that's fine — process the 9,500 that responded):

"ToleratedFailurePercentage": 15,   // fail Map only if >15% of items fail
"ToleratedFailureCount": 100,       // fail Map if >100 items fail (absolute count)
// Both can be set — Map fails when EITHER threshold is crossed

When the Map completes within tolerance, the ResultWriter writes a manifest.json to the output S3 prefix listing which child executions succeeded and which failed — along with the S3 keys for result files of each batch. The parent state can read the manifest and re-queue the failed subset as a second pass:

# manifest.json structure
{
  "DestinationBucket": "my-results-bucket",
  "DestinationPrefix": "results/",
  "ResultFiles": {
    "SUCCEEDED": [
      { "Key": "results/SUCCESS_0.json", "ItemCount": 25 },
      { "Key": "results/SUCCESS_1.json", "ItemCount": 25 }
    ],
    "FAILED": [
      { "Key": "results/FAILED_0.json", "ItemCount": 3 }
    ]
  }
}

CDK composition — all three patterns in one stack

The CDK grant methods handle the execution role IAM for each pattern automatically. One pattern that trips teams: you need separate grants for starting executions (called by the MCP server Lambda) and responding to executions (called by callback workers). CDK exposes these separately:

declare const mcpServerFn: lambda.Function;
declare const workerFn: lambda.Function;

// Express Sync — MCP server starts and waits for result
toolChain.grantStartSyncExecution(mcpServerFn);

// Standard / Express Async — MCP server starts and gets ARN
toolChain.grantStartExecution(mcpServerFn);

// Callback pattern — worker resolves execution via SendTaskSuccess/Failure
toolChain.grantTaskResponse(workerFn);
// Adds: states:SendTaskSuccess, states:SendTaskFailure, states:SendTaskHeartbeat

For Distributed Map with S3ItemReader and ResultWriter, the state machine execution role needs S3 permissions CDK does not synthesize automatically. Scope them to the specific buckets:

// S3ItemReader requires GetObject on input bucket
inputBucket.grantRead(toolChain.role);

// ResultWriter requires PutObject on output bucket
outputBucket.grantWrite(toolChain.role);

// DynamoDB GetItem optimized integration — CDK synthesizes automatically
const getSession = new tasks.DynamoGetItem(this, "GetSession", {
  table: sessionsTable,
  key: { sessionId: tasks.DynamoAttributeValue.fromString(
    sfn.JsonPath.stringAt("$.sessionId")
  )},
  resultPath: "$.session",
});
// CDK adds dynamodb:GetItem on sessionsTable.tableArn only — narrower than writing by hand

DefinitionSubstitutions keeps environment-specific ARNs out of committed ASL JSON files and resolves them at CloudFormation deploy time via Fn::Sub — the state machine definition stays readable across dev and production stacks without duplication.

Consolidated failure modes

Production checklist

For any Step Functions state machine powering MCP tool orchestration:

Express state machine

• logs: { level: LogLevel.ALL, includeExecutionData: true } — configured before first deploy, cannot be added retroactively for existing executions
• Execution role has all nine logs:* permissions including PutResourcePolicy and DescribeResourcePolicies
• Calling Lambda timeout = state machine p99 execution time + 30s margin
• Alarm on ExecutionsFailed, ExecutionThrottled, ExecutionTime p95 > 240,000ms dimensioned by StateMachineArn
• Express Async results written to DynamoDB by final state — not queried via GetExecutionHistory

Callback pattern

• Resource ARN ends with .waitForTaskToken — missing suffix means Lambda returns immediately with no pause
• Task token stored in DynamoDB before worker dispatches work, not after
• DynamoDB ConditionExpression: "attribute_not_exists(sessionId)" for idempotency on Lambda retry
• HeartbeatSeconds set; worker heartbeat loop fires at 60–70% of that value
• SendTaskSuccess caller handles TaskTimedOut and InvalidToken as non-fatal

Distributed Map

• ItemBatcher.MaxItemsPerBatch: 25 for inputs > 40 items — reduces cold starts and child execution count
• S3ItemReader for inputs > 256 KB — use NDJSON format for incremental production
• ToleratedFailurePercentage set for batch tool calls where partial success is acceptable
• State machine execution role has s3:GetObject on input bucket and s3:PutObject on output bucket
• ResultWriter configured; manifest.json read by downstream state for failed-subset retry logic

CDK

• payloadResponseOnly: true on all LambdaInvoke tasks — default wraps response in SDK envelope
• grantStartSyncExecution vs grantStartExecution vs grantTaskResponse — use the correct grant for each caller
• tracingEnabled: true with xray:PutTraceSegments and xray:PutTelemetryRecords on execution role
• DefinitionSubstitutions for multi-environment ARN injection — no hardcoded ARNs in committed ASL files