KDS Advanced Patterns for MCP Server Monitoring Pipelines: Enhanced Fan-Out, KCL 2.x, Lambda ESM, and Capacity Management

Pattern 1: Enhanced Fan-Out — dedicated bandwidth for multiple independent consumers

An MCP probe pipeline with a single consumer is simple: one application calls GetRecords on each shard, processes the results, and moves on. Multiple consumers on the same stream introduce a throughput-sharing problem that is invisible until it degrades. Standard GetRecords splits the shard's 2 MiB/s read bandwidth equally across all consumers polling that shard — add a second consumer and each gets 1 MiB/s; add a fourth and each gets 500 KiB/s; a slow archiver filling its 500 KiB/s budget blocks the alerter from reading new records until the archiver releases the connection.

Enhanced Fan-Out (EFO) eliminates this by allocating a separate 2 MiB/s HTTP/2 push connection for each registered consumer. The alerter and archiver each get a dedicated 2 MiB/s stream from Kinesis, independent of each other. Backpressure on the archiver does not affect the alerter's delivery latency. EFO also reduces propagation latency from ~200ms (GetRecords polling interval) to ~70ms (HTTP/2 server push).

RegisterStreamConsumer lifecycle: CREATING → ACTIVE

EFO requires pre-registering each consumer application by name. Registration is idempotent by consumer name — calling it twice with the same name returns the existing consumer ARN rather than creating a duplicate. The status transition takes 10–30 seconds: a consumer in CREATING state cannot subscribe, and calling SubscribeToShard too early returns ResourceInUseException. Poll DescribeStreamConsumer until ConsumerStatus is ACTIVE before opening the subscription.

Consumer naming discipline matters more than it looks. Using deployment metadata in consumer names (commit SHA, timestamp) creates orphaned DELETING registrations on each deploy that count against the 20-consumer limit. Stale DELETING consumers do not expire automatically — they must be explicitly deregistered with DeregisterStreamConsumer, and then waited for before re-registering the same name. The right pattern is a stable, human-readable name scoped to function role: alivemcp-alerter-consumer, alivemcp-archiver-consumer. Add a pre-deploy deregister step in CI/CD that calls ListStreamConsumers and deregisters any existing registration with the same name before re-registering.

SubscribeToShard: 5-minute subscription expiry and continuation

Each SubscribeToShard call opens an HTTP/2 stream that Kinesis closes after ~5 minutes. This is expected behavior, not an error — the consumer must re-subscribe using the continuation sequence number from the last processed record. Resuming from TRIM_HORIZON instead of the checkpointed position on restart causes duplicate processing. Persist the last sequence number per shard; use AFTER_SEQUENCE_NUMBER as the starting position.

IAM dual-resource requirement

The most common EFO misconfiguration is granting only the stream ARN in the IAM policy. RegisterStreamConsumer acts on the stream ARN, but SubscribeToShard and DeregisterStreamConsumer act on the consumer ARN (a child resource of the stream: stream/name/consumer/*:*). Granting only the stream ARN causes SubscribeToShard to return AccessDeniedException — an error that looks like a permissions problem but is actually a missing resource scope in the policy statement.

CloudWatch metrics for EFO consumers

EFO consumers do not contribute to ReadProvisionedThroughputExceeded — they bypass the shared bandwidth pool entirely. To monitor EFO lag, use SubscribeToShard.MillisBehindLatest (not GetRecords.IteratorAgeMilliseconds, which only applies to standard polling consumers). Alarm on MillisBehindLatest exceeding 60,000ms — this means the consumer is one full probe cycle behind the write rate and may start missing real-time alerting windows.

Pattern 2: KCL 2.x — distributed consumer coordination with DynamoDB lease management

When probe consumers run as an ECS fleet — multiple task instances that need to share shard assignment, recover from instance failures, and redistribute shards on scale-out — KCL 2.x is the right layer. It stores shard assignment state in a DynamoDB table and coordinates workers via optimistic locking: each shard has one lease row, and a worker "owns" a shard by holding the lease (writing its worker ID and incrementing the leaseCounter before the renewal deadline). If a worker task is killed, another worker steals the expired lease and resumes from the last checkpoint.

DynamoDB lease table structure

KCL creates the lease table automatically, named after your application (e.g., alivemcp-probe-consumer). Each row stores: leaseKey (shard ID, the partition key), leaseOwner (current worker ID), leaseCounter (monotonic version for optimistic locking), checkpoint (last committed sequence number), checkpointSubSequenceNumber (for KPL aggregated records), ownerSwitchesSinceCheckpoint (debugging starvation), and parentShardIds (populated after merge — lists the parent shard IDs that must all reach SHARD_END before this child shard starts processing).

Use on-demand DynamoDB capacity for the lease table. Steady-state is low-frequency writes (~1 WCU per shard per 10-second renewal interval), but a cold-start of 8 workers simultaneously produces 16 conditional lease-claim writes in under 1 second. Provisioned capacity with a low WCU setting throttles this burst, causing KCL startup delays and log noise. On-demand handles the spike without intervention.

ShardRecordProcessor: the shardEnded invariant

KCL 2.x requires implementing four lifecycle methods. Three are straightforward: initialize (receives the starting sequence number), processRecords (processes the batch and optionally checkpoints), and leaseLost (called when another worker steals the lease — do not checkpoint here, the new owner resumes from the last checkpoint). The fourth — shardEnded — is where most production incidents originate.

When a shard is split or merged, the parent shard runs out of records and KCL calls shardEnded. The critical invariant: KCL will not start processing child shards until all parent shards have called checkpoint() in shardEnded with the SHARD_END sentinel. If shardEnded returns without calling checkpoint, the child shard rows in the lease table remain in waiting state indefinitely. New probe results written to those child shards accumulate unread; MillisBehindLatest may not alarm on the parent (which appears healthy, just exhausted) while the child lag grows silently. The fix is mechanical: always call checkpointer.checkpoint() before returning from shardEnded.

Split vs merge topology after resharding

After a shard split, one parent becomes two children. Each child's parentShardIds contains only the single parent — KCL starts processing the child as soon as the parent reaches SHARD_END. After a merge, two parents become one child. The child's parentShardIds contains both parent shard IDs — KCL waits for both parents to be checkpointed at SHARD_END before starting the child. This ordering guarantee ensures no records from either parent stream are skipped when the merged shard begins. Missing a SHARD_END checkpoint on either parent in a merge scenario leaves the child waiting for both — a deadlock that requires manual intervention to resolve.

Checkpointing frequency: per-batch for idempotent probe results

Checkpointing writes to DynamoDB. Per-record checkpointing (one DynamoDB write per probe result) is 10–1,000× more expensive than per-batch checkpointing and is only justified for non-idempotent, high-value processing where reprocessing a single record has meaningful side effects. For AliveMCP probe results, upsert-by-probeRunId makes reprocessing idempotent — the result of reprocessing a batch is identical to the original. Per-batch checkpointing (once per processRecords call) is the correct choice: it keeps DynamoDB write costs proportional to shard count rather than record throughput.

MultiLangDaemon for Node.js

KCL 2.x is a Java library. Node.js consumers use the MultiLangDaemon (MLD) — a Java subprocess that manages all Kinesis API calls and lease management, communicating with the Node.js processor via a stdin/stdout JSON protocol wrapped by the amazon-kinesis-client npm package. This requires JRE 11+ in the container image (typically adding ~200 MB via apt-get install -y default-jre-headless). If image size is a hard constraint, use Lambda ESM instead — it does not require MLD and handles the equivalent coordination internally.

Pattern 3: Lambda ESM — serverless consumer with parallelization and built-in error isolation

Lambda's Kinesis Event Source Mapping is the right consumer architecture when the probe pipeline does not need a persistent fleet: no ECS tasks to size, no lease table to provision, no worker coordination to debug. Lambda polls each shard on your behalf and invokes your function with a batch of records. The tradeoff is that the default configuration — one serial invocation per shard — caps throughput at roughly the function's invocation rate, which lags during scan-wave bursts when all endpoint probes arrive within a 10-second window.

Parallelization factor: concurrent invocations per shard

parallelizationFactor (1–10) controls how many concurrent Lambda invocations can process different record windows from the same shard simultaneously. A 4-shard stream with parallelizationFactor=3 runs up to 12 concurrent invocations. Within-shard ordering is preserved within each parallel window (records from the same shard go to one invocation at a time), but records in different parallel windows may process out of order relative to each other. For AliveMCP probe results, this is correct: probe result upserts are commutative (last write by probeRunAt timestamp wins), so parallel processing does not create consistency problems.

Start at parallelizationFactor=2–3 for moderate scan wave bursts. Monitor GetRecords.IteratorAgeMilliseconds: if iterator age grows continuously despite no Lambda throttles or errors, increase the factor. At parallelizationFactor=10 (40 concurrent invocations on a 4-shard stream), verify that Lambda reserved concurrency is set high enough — each invocation counts against the function's concurrency limit.

bisectBatchOnFunctionError: binary search for poison-pill records

Without bisectBatchOnFunctionError, a single malformed probe result in a batch of 100 retries the entire batch until retryAttempts exhausts, blocking all 100 records — including the 99 healthy ones — for minutes. With it enabled, ESM responds to a function error by splitting the failing batch in half and retrying each half independently. The half containing the bad record fails again; ESM splits it again. This continues until the batch size is 1 — the single failing record routes to the onFailure SQS destination while the healthy records process normally.

For a batch of 8 records where record 5 is malformed: attempt 1 fails on [1–8]; attempts 2a [1–4] succeed; attempt 2b [5–8] fails and bisects; attempt 3a [5–6] fails and bisects; attempt 3b [7–8] succeeds; attempt 4a [5] fails and routes to DLQ; attempt 4b [6] succeeds. Total: 7 of 8 records processed, 1 quarantined in DLQ. The shard is unblocked after O(log n) retry rounds rather than exhausting the full retry budget on every attempt.

Always configure an onFailure SQS destination alongside bisect. Without a DLQ, records that exhaust retries are silently dropped. The DLQ captures them for investigation and potential replay once the producer bug is fixed.

Batch window: latency vs cost for different consumers

maximumBatchingWindowInSeconds (0–300) controls how long ESM waits to fill a batch before invoking Lambda. The right value depends on the consumer's latency tolerance: the real-time alerter driving the status page needs 0 (invoke immediately on any available record); the probe result archiver writing to S3 benefits from 30 seconds (one PutObject with 500 records costs the same as 500 individual PutObjects at 1/500th the request count). The analytics aggregator writing to DynamoDB benefits from 60 seconds (consolidate records into fewer, larger batch writes). Use per-consumer ESM configurations — not one shared ESM with a batch window that suits only one use case.

Tumbling window aggregations: per-endpoint counts without external state

ESM tumbling windows accumulate aggregation state across multiple Lambda invocations within a time-bounded window. The event.state object persists between invocations; event.isFinalInvokeForWindow signals when the window closes and the flush should happen. This is the right pattern for computing per-endpoint probe counts per minute (for rate-of-change alerts), uptime percentages within a rolling window, or alert suppression counts. Without tumbling windows, these aggregations require DynamoDB atomic counters or ElastiCache sorted sets — tumbling windows provide equivalent semantics internally at no extra infrastructure cost.

Set the Lambda function timeout to exceed the tumbling window duration. If the function times out mid-window, the accumulated state is lost and the window restarts. A 5-minute tumbling window requires a minimum function timeout of 5 minutes plus processing headroom.

Pattern 4: KDS capacity management — on-demand mode, UpdateShardCount, and hot shard remediation

Each shard handles 1 MiB/s ingest (1,000 records/second via PutRecord) and 2 MiB/s read for standard consumers. For AliveMCP at steady state — 5,000 MCP endpoints × 1 probe per 60s = ~83 probe results/second × ~2 KB each = ~166 KiB/s — a single shard is sufficient. During a scan wave (all endpoints probed within 10 seconds), the same volume arrives as ~500 records/second — still within the 1,000 records/second limit. The capacity question becomes relevant at 10,000+ endpoints, during pipeline backlogs, or if a poor partition key choice concentrates all traffic onto one shard.

On-demand mode vs provisioned: the right default for each scenario

On-demand mode is the right starting point for a new probe pipeline with unpredictable growth: it scales automatically to 200 shards without operator action, and requires no shard count estimation. The trade-off is cost (~3× per GB vs provisioned at steady-state load) and loss of split-point control (Kinesis chooses where to split based on observed hot hash ranges). Provisioned mode is correct once traffic is stable and predictable, or when hot shard remediation requires a specific hash key split point.

Switch from on-demand to provisioned after traffic stabilizes using UpdateStreamModeCommand with StreamMode: 'PROVISIONED'. Do not switch more than twice in a 24-hour period — AWS enforces this limit. After the switch, enable shard-level enhanced monitoring immediately so hot shards surface before they drop records.

UpdateShardCount constraints

UpdateShardCount resizes a provisioned stream by executing the splits or merges needed to reach the target shard count. Constraints: can scale up to 2× the current count per call (4→8 valid; 4→20 requires two calls: 4→8, then 8→20 after the first completes); can scale down to 50% per call (8→4 valid; 8→1 requires two calls: 8→4, then 4→2); maximum 10 calls per stream per 24 hours. The operation takes 1–5 minutes; the stream remains fully available throughout. KCL consumers detect child shards via lease table updates; Lambda ESM detects them after parent shards reach SHARD_END — no consumer reconfiguration is required.

Hot shard detection and targeted SplitShard

A hot shard occurs when a disproportionate fraction of records hash to the same shard's hash key range. The signal is WriteProvisionedThroughputExceeded > 0 on a specific shard (not stream-wide). Enable shard-level metrics with EnableEnhancedMonitoringCommand and alarm on any shard exceeding 0 — this means records are being dropped on that shard, not just throttled.

When a specific shard is hot, SplitShard is preferable to UpdateShardCount: it splits only the hot shard at a chosen hash key midpoint rather than doubling the entire stream. Compute the midpoint as a BigInt: ((BigInt(startingHashKey) + BigInt(endingHashKey)) / 2n).toString(). If the hot traffic is concentrated at one end of the hash range, adjust the split point asymmetrically — 75% of the range to the lighter-traffic child, 25% to the heavier-traffic child. This surgical approach avoids the 2× cost increase of doubling all shards when only one is hot.

The root cause of hot shards is almost always a non-random partition key. Using a constant string, a monotonic timestamp, or a low-cardinality owner ID as the partition key means all records hash to the same shard. The fix is a high-cardinality random partition key — endpoint UUID, or a random bucket prefix combined with the endpoint ID — so records distribute evenly across all shards.

Retention period: 7 days as the practical default

The default 24-hour retention is too short for most MCP monitoring pipelines: a consumer catch-up after a deployment outage, a weekend DLQ replay, or a new analytics consumer onboarding all require more than 24 hours of replay window. Seven days covers these scenarios. Extended retention beyond 7 days (up to 365 days) makes sense only when probe results are not being archived to S3 or DynamoDB — if they are, Kinesis retention is redundant storage that adds cost without adding recovery capability. Long-term retention (365 days, priced at a discounted per-GB-hour tier) is appropriate for regulated industries requiring immutable probe result history.

When to use KDS vs SQS vs EventBridge vs MSK

KDS is the right choice for the AliveMCP probe result pipeline because it satisfies all four requirements simultaneously: multiple independent consumers (alerter, archiver, analytics, status-page), ordered results per endpoint (partition-key ordering within shards), replay capability (rewind to any sequence number within retention), and high throughput (scales via shard count). SQS is correct for a single consumer where durability (DLQ) matters more than ordering or replay — each SQS message is consumed once and deleted. EventBridge is correct for content-based routing to multiple AWS services — route failed probes to SNS, schema-drift probes to DynamoDB — but provides no message retention or replay. MSK (Kafka) is correct when schema evolution guarantees (Glue Schema Registry) or existing Kafka infrastructure matter more than operational simplicity.

Cross-pattern failure modes