Kinesis Data Streams Resharding for MCP Servers — UpdateShardCount, On-Demand Mode, Hot Shard Remediation

Shard capacity limits and cost model

For AliveMCP, the write side is the binding constraint: 5,000 MCP endpoints × 1 probe/60s = ~83 probe results/second at steady state, roughly 83 × 2 KB = 166 KiB/s — well within 1 shard. During a scan wave (all endpoints probed within a 10-second burst), that same volume arrives as ~500 records/second — still within 1 shard's 1,000 records/second limit. The 1 MiB/s write limit becomes a concern only if average probe result size exceeds ~2 KB at 500 req/s. At 10,000 endpoints, a 2-shard stream provides comfortable headroom.

On-demand mode: auto-scaling without operator action

On-demand mode eliminates shard capacity planning. Kinesis scales the stream up automatically when ingest exceeds the current capacity (up to a maximum of 200 shards), and scales down (with a 4-hour cooldown) when ingest decreases. The trade-off is cost: on-demand pricing is approximately 3× higher per GB than provisioned mode at steady-state load.

import { KinesisClient, UpdateStreamModeCommand } from '@aws-sdk/client-kinesis';

const client = new KinesisClient({ region: 'us-east-1' });

// Switch existing provisioned stream to on-demand
await client.send(new UpdateStreamModeCommand({
  StreamARN: 'arn:aws:kinesis:us-east-1:123456789012:stream/alivemcp-probe-results',
  StreamModeDetails: {
    StreamMode: 'ON_DEMAND',  // 'PROVISIONED' to switch back
  },
}));

On-demand mode constraints:

• Maximum scale-up: 200 shards per stream (hard limit, not adjustable)
• Scale-up granularity: doubles shard count on sustained over-capacity (not per-record smooth scaling)
• Scale-down cooldown: 4 hours after last scale-up before a scale-down occurs
• No control over which shards split — Kinesis chooses the split point based on observed hot ranges
• Cannot switch to on-demand more than twice in a 24-hour period

When to use on-demand vs provisioned

UpdateShardCount: bulk provisioned resize

UpdateShardCount is the recommended way to resize a provisioned stream when you need to add or remove multiple shards at once. It targets a new shard count and executes the splits or merges needed to reach it:

import { UpdateShardCountCommand } from '@aws-sdk/client-kinesis';

// Scale from 4 shards to 8 shards
await client.send(new UpdateShardCountCommand({
  StreamName: 'alivemcp-probe-results',
  TargetShardCount: 8,
  ScalingType: 'UNIFORM_SCALING',  // Only supported value; scales evenly across hash key range
}));

Constraints on UpdateShardCount:

• Can scale up to 2× the current shard count per call (so 4→8 is valid; 4→20 requires multiple calls)
• Can scale down to 50% of the current shard count per call (8→4 valid; 8→1 requires multiple calls)
• Maximum 10 UpdateShardCount calls per 24 hours per stream
• Takes 1–5 minutes to complete; stream remains fully available during the operation

After UpdateShardCount completes, the parent shards are closed (no new writes) and child shards accept new writes. Parent shards continue to serve reads until all existing records age out of retention. KCL consumers automatically detect the new child shards via lease table updates. Lambda ESM automatically begins processing child shards after detecting SHARD_END on parent shards — no ESM configuration change required.

Hot shard detection and targeted SplitShard

A hot shard occurs when a disproportionate fraction of records hash to the same shard's hash key range. For AliveMCP, this could happen if probe results are keyed by endpoint owner ID and one owner has 1,000× more endpoints than others — all their traffic lands on the same shard.

Detecting hot shards with enhanced monitoring

Enable shard-level metrics to identify which specific shards are approaching capacity:

import { EnableEnhancedMonitoringCommand } from '@aws-sdk/client-kinesis';

await client.send(new EnableEnhancedMonitoringCommand({
  StreamName: 'alivemcp-probe-results',
  ShardLevelMetrics: [
    'IncomingBytes',       // Write bytes per shard
    'IncomingRecords',     // Write records per shard
    'OutgoingBytes',       // Read bytes per shard
    'WriteProvisionedThroughputExceeded',  // Throttled writes — the hot shard signal
    'ReadProvisionedThroughputExceeded',   // Throttled reads
    'IteratorAgeMilliseconds',             // Consumer lag per shard
  ],
}));

Alarm on WriteProvisionedThroughputExceeded > 0 per shard. Any non-zero value means records are being dropped on that shard — Kinesis rejects the PutRecord call with ProvisionedThroughputExceededException. Producers with automatic retry will retry with exponential backoff, but this adds latency to probe result delivery.

SplitShard: surgical hot shard remediation

When a specific shard is hot, SplitShard splits it at a chosen hash key midpoint, giving you control over where the split occurs (unlike UpdateShardCount which always splits uniformly):

import { SplitShardCommand, DescribeStreamSummaryCommand,
         ListShardsCommand } from '@aws-sdk/client-kinesis';

// Find the hot shard's starting and ending hash keys
const shards = await client.send(new ListShardsCommand({
  StreamName: 'alivemcp-probe-results',
}));
const hotShard = shards.Shards!.find(s => s.ShardId === 'shardId-000000000001')!;

// Split at the midpoint of the hot shard's hash key range
const startHash = BigInt(hotShard.HashKeyRange!.StartingHashKey);
const endHash = BigInt(hotShard.HashKeyRange!.EndingHashKey);
const midHash = ((startHash + endHash) / 2n).toString();

await client.send(new SplitShardCommand({
  StreamName: 'alivemcp-probe-results',
  ShardToSplit: 'shardId-000000000001',
  NewStartingHashKey: midHash,
}));

After the split, two child shards are created with hash ranges [startHash, midHash-1] and [midHash, endHash]. If the hot traffic is concentrated at one end of the hash range (e.g., most endpoint IDs hash to the upper half), adjust NewStartingHashKey to put the split asymmetrically — 75% of the range in the child that will receive less traffic, 25% in the child that will receive more.

Retention period: 24 hours to 365 days

Default retention is 24 hours. Extended retention (up to 365 days) is available at additional cost per GB-hour. Choose retention based on your replay and backfill needs:

For AliveMCP, 7-day retention is the practical default. It covers the scenario where a new consumer (e.g., an analytics backfiller) comes online over a weekend and needs to catch up on the prior 7 days of probe results. Retention beyond 7 days makes sense only if probe results are not being archived to S3 or DynamoDB — if they are, Kinesis retention is redundant storage cost.

KDS vs SQS vs EventBridge vs MSK: when to use what

For AliveMCP's probe result pipeline: KDS is the right choice when you need multiple independent consumers (alerter, archiver, analytics, status-page) reading the same results simultaneously with full-speed delivery. Use SQS if you have a single consumer and message durability (DLQ) matters more than ordering or replay. Use EventBridge if you are routing probe events to different AWS services based on content filters (e.g., failed probes → SNS, schema-drift probes → DynamoDB). Use MSK if you already operate Kafka infrastructure and need schema evolution guarantees.

Common failure modes