Kinesis Lambda Event Source Mapping for MCP Servers — Parallelization Factor, BisectBatchOnError, and Tumbling Windows

Parallelization factor: concurrent invocations per shard

Without parallelization, one Lambda invocation processes one batch from one shard at a time — the next invocation for that shard starts only after the current one finishes. With parallelizationFactor: N, ESM maintains up to N in-flight invocations per shard simultaneously, processing N non-overlapping record windows in parallel:

Parallelization factor does not change shard ordering guarantees within a single shard: records from the same shard are still processed in partition-key order within each parallel window. Records in different parallel windows may be processed out of order relative to each other. For probe results, this is typically acceptable — probe result upserts are commutative (last write wins by probeRunAt timestamp). If your consumer requires strict FIFO ordering across all records in a shard, keep parallelizationFactor: 1.

import * as lambda from 'aws-cdk-lib/aws-lambda';
import * as kinesis from 'aws-cdk-lib/aws-kinesis';
import * as eventsources from 'aws-cdk-lib/aws-lambda-event-sources';
import * as sqs from 'aws-cdk-lib/aws-sqs';

const probeResultStream = kinesis.Stream.fromStreamArn(this, 'ProbeStream',
  'arn:aws:kinesis:us-east-1:123456789012:stream/alivemcp-probe-results'
);
const dlq = new sqs.Queue(this, 'ProbeResultDlq', {
  queueName: 'alivemcp-probe-result-dlq',
  retentionPeriod: cdk.Duration.days(14),
});

probeConsumer.addEventSource(new eventsources.KinesisEventSource(probeResultStream, {
  startingPosition: lambda.StartingPosition.LATEST,
  batchSize: 100,
  parallelizationFactor: 3,
  bisectBatchOnError: true,
  maxBatchingWindow: cdk.Duration.seconds(10),
  retryAttempts: 3,
  onFailure: new eventsources.SqsDlq(dlq),
}));

bisectBatchOnFunctionError: isolating poison-pill records

Without bisectBatchOnFunctionError, a single malformed probe result record in a batch of 100 causes the entire batch to retry until it exhausts retryAttempts and the records age out or hit maximumRecordAgeInSeconds. The healthy 99 records in the batch are blocked behind the one bad record for minutes.

With bisectBatchOnFunctionError: true, ESM responds to a function error by splitting the failing batch in half and retrying each half independently. The half containing the bad record fails again; ESM splits it again. This binary search continues until the batch size is 1 — the single failing record is then routed to the onFailure destination (SQS DLQ) and processing resumes for the healthy records.

# Bisect behavior on a batch of 8 records where record 5 is malformed:
Attempt 1: batch [1,2,3,4,5,6,7,8] → function throws → bisect
Attempt 2a: batch [1,2,3,4] → success ✓
Attempt 2b: batch [5,6,7,8] → function throws → bisect
Attempt 3a: batch [5,6] → function throws → bisect
Attempt 3b: batch [7,8] → success ✓
Attempt 4a: batch [5] → function throws → onFailure DLQ → poison pill quarantined
Attempt 4b: batch [6] → success ✓

Total successful processing: records 1,2,3,4,6,7,8 ✓
Quarantined: record 5 → DLQ for investigation

Enable bisect for any consumer where the batch may contain independently-processable records (probe results from different endpoints). Do not rely on bisect as the primary error handling strategy — fix the root cause. Bisect is a safety net that prevents one bad record from blocking the entire shard indefinitely.

Batch window and batch size: cost vs latency

maximumBatchingWindowInSeconds (0–300 seconds) controls how long ESM waits before invoking Lambda even if fewer than batchSize records are available. At 0 (default), ESM invokes Lambda as soon as any records are available — maximizing freshness but potentially invoking Lambda with very small batches during low-traffic periods.

The probe result archiver that writes to S3 benefits most from a large batch window: each Lambda invocation opens an S3 PutObject, and consolidating 500 records into one S3 object costs the same as 500 separate objects at 1/500th of the request cost. For the real-time alerter that drives the status page UI, any batch window adds noticeable latency — keep it at 0.

Tumbling window aggregations

ESM supports tumbling windows for computing aggregations over a time-bounded window of records without needing an external state store. Specify tumblingWindow as a duration (15 seconds to 15 minutes); ESM provides a state object in the event that persists between invocations within the same window:

export const handler = async (event: KinesisStreamEvent): Promise => {
  // event.state: accumulated from prior invocations in this window
  // event.isFinalInvokeForWindow: true on last call before window closes
  // event.isWindowTerminatedEarly: true if window ended early (batch too large)

  const state = (event.state as { counters?: Record }) ?? { counters: {} };
  const counters = state.counters ?? {};

  for (const record of event.Records) {
    const probe = JSON.parse(Buffer.from(record.kinesis.data, 'base64').toString());
    counters[probe.endpointId] = (counters[probe.endpointId] ?? 0) + 1;
  }

  if (event.isFinalInvokeForWindow) {
    // Window closed — flush aggregated counters to DynamoDB
    await writeWindowCounters(counters, event.window);
    return { state: {}, batchItemFailures: [] };
  }

  // Return updated state for next invocation in this window
  return { state: { counters }, batchItemFailures: [] };
};

Tumbling windows are ideal for computing per-endpoint probe counts per minute (for rate-of-change alerts) or uptime percentages within a rolling window. Without tumbling windows, you would need DynamoDB atomic counters or ElastiCache to accumulate partial results between Lambda invocations — tumbling windows provide this state internally at no extra infrastructure cost.

Enhanced Fan-Out mode for Lambda ESM

By default, Lambda ESM uses standard GetRecords polling (shared 2 MiB/s per shard). Enable EFO mode on the ESM by setting startingPosition: LATEST and adding the consumerArn parameter pointing to a registered EFO consumer:

# CDK: enable EFO on Lambda ESM
probeConsumer.addEventSource(new eventsources.KinesisEventSource(probeResultStream, {
  startingPosition: lambda.StartingPosition.LATEST,
  // EFO: dedicated 2 MiB/s regardless of other consumers on the stream
  // Register consumer first via RegisterStreamConsumer, then pass ARN here
  // Note: CDK does not yet have native EFO support — use CfnEventSourceMapping directly
}));

// CfnEventSourceMapping for EFO
new lambda.CfnEventSourceMapping(this, 'ProbeResultEsmEfo', {
  functionName: probeConsumer.functionName,
  eventSourceArn: probeResultStream.streamArn,
  startingPosition: 'LATEST',
  parallelizationFactor: 3,
  bisectBatchOnFunctionError: true,
  selfManagedEventSource: undefined,
  amazonManagedKafkaEventSourceConfig: undefined,
  // EFO via consumer ARN in FilterCriteria is not available yet — use standard ESM
  // and RegisterStreamConsumer separately for EFO control
});

As of 2026, Lambda ESM supports EFO via the DestinationConfig and consumer ARN fields in CreateEventSourceMapping. Check the current SDK documentation for the exact parameter name — the CDK L2 construct may not expose it yet; use the L1 CfnEventSourceMapping when needed.

CloudWatch monitoring for Kinesis ESM

The most important metric for ESM consumers is GetRecords.IteratorAgeMilliseconds — the age of the oldest unread record in the shard. A growing iterator age means Lambda is processing records slower than Kinesis is receiving them. First check for Lambda throttles (concurrency limit); then check for slow processing (increase parallelizationFactor or optimize the function); then check for Kinesis shard throughput limits (consider adding shards).

Common failure modes