AWS Kinesis Data Streams for MCP Servers

PutRecord vs PutRecords — when each matters

Kinesis exposes two write APIs. PutRecord publishes a single record synchronously and returns the shard ID and sequence number — straightforward but wasteful for MCP tools that emit events in tight loops. PutRecords accepts up to 500 records or 5 MB in one API call, significantly reducing per-record API overhead and improving throughput. The critical difference: PutRecords is partially atomic. The HTTP response is always 200 — you must check response.FailedRecordCount and then re-examine response.Records[i].ErrorCode for each record to identify which ones failed with ProvisionedThroughputExceededException or InternalFailure.

import {
  KinesisClient,
  PutRecordsCommand,
  type PutRecordsRequestEntry,
} from "@aws-sdk/client-kinesis";
import { randomUUID } from "node:crypto";

const kinesis = new KinesisClient({ region: process.env.AWS_REGION ?? "us-east-1" });

async function publishToolCallEvents(
  streamName: string,
  events: Array<{ toolName: string; input: unknown; sessionId: string }>
): Promise<void> {
  const records: PutRecordsRequestEntry[] = events.map((event) => ({
    Data: Buffer.from(JSON.stringify(event)),
    // High-cardinality partition key: sessionId distributes load across shards
    // Do NOT use toolName — 5 tool names across 10 shards = hot-spotting
    PartitionKey: event.sessionId,
  }));

  let remaining = records;
  let attempts = 0;

  while (remaining.length > 0 && attempts < 3) {
    attempts++;
    const response = await kinesis.send(
      new PutRecordsCommand({ StreamName: streamName, Records: remaining })
    );

    if ((response.FailedRecordCount ?? 0) === 0) break;

    // Collect only the records that failed — preserve original order mapping
    const failed: PutRecordsRequestEntry[] = [];
    for (let i = 0; i < (response.Records?.length ?? 0); i++) {
      if (response.Records![i].ErrorCode) {
        failed.push(remaining[i]);
      }
    }

    remaining = failed;
    if (remaining.length > 0) {
      await new Promise((r) => setTimeout(r, 100 * Math.pow(2, attempts)));
    }
  }

  if (remaining.length > 0) {
    console.error(`${remaining.length} records failed after ${attempts} attempts`);
  }
}

Shard throughput limits and partition key selection

Each Kinesis shard supports 1 MB/s or 1,000 records/s for writes and 2 MB/s for reads. These limits apply per shard, not per stream — and the assignment of records to shards is determined by the MD5 hash of the partition key. A partition key that hashes to the same shard range overloads that shard while others remain idle, a condition called hot-spotting.

For MCP server tools, the best partition keys are high-cardinality identifiers already present on the event: session ID, request ID (UUID), or tenant ID in multi-tenant deployments. Avoid using MCP tool names (low cardinality — your tool registry typically has 5–30 tools), timestamp rounded to the second (bursts all land on the same key), or a constant string (all records to one shard).

To detect hot-spotting, enable per-shard CloudWatch metrics (GetShardLevelMetrics in enhanced monitoring) and watch IncomingRecords per shard. A healthy stream shows roughly even distribution. A hot shard typically shows one shard consuming >80% of total IncomingRecords.

# Enable enhanced per-shard monitoring for throughput and error visibility
aws kinesis enable-enhanced-monitoring \
  --stream-name mcp-tool-events \
  --shard-level-metrics \
    IncomingBytes \
    IncomingRecords \
    WriteProvisionedThroughputExceeded \
    ReadProvisionedThroughputExceeded \
    IteratorAgeMilliseconds

# Check shard count and iterator age (rising iterator age = consumer falling behind)
aws cloudwatch get-metric-statistics \
  --namespace AWS/Kinesis \
  --metric-name GetRecords.IteratorAgeMilliseconds \
  --dimensions Name=StreamName,Value=mcp-tool-events \
  --start-time 2026-09-20T00:00:00Z \
  --end-time 2026-09-20T01:00:00Z \
  --period 300 \
  --statistics Maximum \
  --query 'Datapoints[*].{Time:Timestamp,Age:Maximum}' \
  --output table

GetShardIterator types — which one to use

When building a standard (polling) consumer, you first call GetShardIterator to get a cursor into the shard, then call GetRecords in a loop. The iterator type controls where in the shard the cursor starts.

The 5-minute iterator expiry trap: GetShardIterator returns a cursor that expires in exactly 5 minutes if unused. If your MCP tool calls GetShardIterator and then does slow work before calling GetRecords — connecting to a database, making external API calls, waiting for a mutex — the iterator expires and the next GetRecords call returns ExpiredIteratorException. The fix: never cache iterators across tool invocations. Call GetShardIterator immediately before the GetRecords loop, and refresh it using the NextShardIterator returned by each GetRecords call rather than requesting a new one.

import {
  KinesisClient,
  GetShardIteratorCommand,
  GetRecordsCommand,
  ShardIteratorType,
} from "@aws-sdk/client-kinesis";

const kinesis = new KinesisClient({ region: "us-east-1" });

async function readFromShard(
  streamName: string,
  shardId: string,
  maxRecords = 100
): Promise<unknown[]> {
  // Call GetShardIterator immediately before the read loop — never cache this
  const iteratorResponse = await kinesis.send(
    new GetShardIteratorCommand({
      StreamName: streamName,
      ShardId: shardId,
      ShardIteratorType: ShardIteratorType.LATEST,
    })
  );

  let iterator = iteratorResponse.ShardIterator!;
  const collected: unknown[] = [];

  while (collected.length < maxRecords) {
    const records = await kinesis.send(
      new GetRecordsCommand({ ShardIterator: iterator, Limit: 100 })
    );

    for (const record of records.Records ?? []) {
      collected.push(JSON.parse(Buffer.from(record.Data!).toString("utf-8")));
    }

    // Always use NextShardIterator from the response — never call GetShardIterator again
    if (!records.NextShardIterator) break; // shard closed (resharding)
    iterator = records.NextShardIterator;

    // Standard polling delay: GetRecords has a 5 TPS limit per shard per consumer
    if ((records.Records?.length ?? 0) === 0) {
      await new Promise((r) => setTimeout(r, 1000));
    }
  }

  return collected;
}

Standard polling vs enhanced fan-out consumers

Standard GetRecords polling is throttled at 5 transactions per second per shard and the 2 MB/s read throughput is shared across all standard consumers reading the same shard. If three Lambda functions all poll the same shard, they collectively get 2 MB/s — roughly 667 KB/s each — and compete for the 5 TPS limit.

Enhanced fan-out consumers register with RegisterStreamConsumer and receive records via SubscribeToShard, a long-polling HTTP/2 connection. Each enhanced fan-out consumer gets a dedicated 2 MB/s per shard pipe, completely independent of other consumers. You can have up to 20 enhanced fan-out consumers per stream (compared to effectively 3–5 practical concurrent standard consumers before throughput degrades). The trade-off: enhanced fan-out costs $0.015 per consumer-shard-hour on top of the shard-hour cost, and the SubscribeToShard call times out after 5 minutes — your consumer must re-subscribe in a loop.

# Register an enhanced fan-out consumer (one-time setup per consumer application)
aws kinesis register-stream-consumer \
  --stream-arn arn:aws:kinesis:us-east-1:123456789012:stream/mcp-tool-events \
  --consumer-name mcp-audit-archiver

# Verify registration (status transitions: CREATING → ACTIVE within ~10s)
aws kinesis describe-stream-consumer \
  --stream-arn arn:aws:kinesis:us-east-1:123456789012:stream/mcp-tool-events \
  --consumer-name mcp-audit-archiver \
  --query 'ConsumerDescription.{Status:ConsumerStatus,ARN:ConsumerARN}'

Shard-count planning for MCP tool call volumes

Kinesis Data Streams shards are elastic within limits: you can scale up with UpdateShardCount (doubles shards, causes a brief pause), or enable on-demand mode to let AWS handle scaling automatically. For MCP server tools, the write-side throughput calculation is straightforward:

# Sizing formula:
# peak_records_per_sec × avg_record_bytes / 1_000_000 = MB/s needed
# shards_needed = ceil(MB/s_needed / 0.85)  ← 0.85 = 85% utilization target (15% headroom for retry bursts)

# Example: 500 MCP tool calls/sec × 2 KB average event = 1 MB/s → ceil(1/0.85) = 2 shards

# Check current shard count and stream summary
aws kinesis describe-stream-summary \
  --stream-name mcp-tool-events \
  --query 'StreamDescriptionSummary.{Shards:OpenShardCount,RetentionHours:RetentionPeriodHours,StreamStatus:StreamStatus}'

# Switch to on-demand mode (auto-scaling up to 200 shards, billed per GB)
aws kinesis update-stream-mode \
  --stream-arn arn:aws:kinesis:us-east-1:123456789012:stream/mcp-tool-events \
  --stream-mode-details StreamMode=ON_DEMAND

# Scale up provisioned stream manually (API halves or doubles shard count only)
aws kinesis update-shard-count \
  --stream-name mcp-tool-events \
  --target-shard-count 4 \
  --scaling-type UNIFORM_SCALING

On-demand mode is appropriate for MCP servers with spiky or unpredictable tool call volumes — it automatically scales between 4 and 200 shards and you pay per GB ingested rather than per shard-hour. Provisioned mode is cheaper for steady-state workloads above ~10 shards. The break-even is roughly at $0.015/shard-hour × 730 hours/month ≈ $11/shard/month vs on-demand's $0.08/GB — if your stream ingests more than 137 GB/month per logical shard, provisioned mode costs less.

Kinesis Data Streams retention and data availability

Default retention is 24 hours. Extended retention costs $0.023/shard-hour of data retained and can be extended to 7 days via IncreaseStreamRetentionPeriod, or up to 365 days with long-term retention (additional cost tier). For MCP server audit event streams, 7-day retention is the practical minimum — it gives you a replay window for incident investigation and allows consumers to catch up after extended downtime without losing data.

# Extend retention from 24h to 7 days
aws kinesis increase-stream-retention-period \
  --stream-name mcp-tool-events \
  --retention-period-in-hours 168

# Long-term retention (up to 8760h / 365 days — higher cost tier)
aws kinesis increase-stream-retention-period \
  --stream-name mcp-tool-events \
  --retention-period-in-hours 720  # 30 days

Common failure modes reference