Kinesis Firehose Error Handling and Monitoring for MCP Servers — processing-failed Prefix, DataFreshness Alarm, Retry Duration

The Firehose error routing model

Firehose routes failed records to the error prefix but never surfaces the failure to the producer. Understanding which failure type routes where is essential for setting up correct monitoring:

A subtle distinction: processing-failed is written when Firehose cannot process the record (transformation or format conversion error). delivery-failed is written when Firehose successfully processed the record but could not deliver it to the destination (e.g., S3 bucket deleted after stream creation). In practice, processing-failed is far more common in MCP probe pipelines because it catches every Lambda transformation bug.

ErrorOutputPrefix configuration

If ErrorOutputPrefix is not set, Firehose writes error records to the same S3 prefix as successful deliveries. This corrupts the Athena table (non-Parquet error payloads mixed into Parquet data files) and makes error records unfindable. Always set it explicitly:

// Delivery stream configuration
ExtendedS3DestinationConfiguration: {
  BucketARN: 'arn:aws:s3:::alivemcp-probe-archive',
  RoleARN: 'arn:aws:iam::123456789012:role/firehose-delivery-role',

  // Success prefix — Hive-compatible for Athena partition projection
  Prefix: 'probe-results/year=!{timestamp:yyyy}/month=!{timestamp:MM}/day=!{timestamp:dd}/hour=!{timestamp:HH}/',

  // Error prefix — separate bucket path per error type
  // !{firehose:error-output-type} resolves to: processing-failed, delivery-failed, or backup-failed
  ErrorOutputPrefix: 'probe-errors/type=!{firehose:error-output-type}/year=!{timestamp:yyyy}/month=!{timestamp:MM}/day=!{timestamp:dd}/hour=!{timestamp:HH}/',

  // ... buffer hints, format conversion, etc.
}

Error records written to the error prefix are the original source records (before transformation), base64-decoded, in the original format your producer sent. For AliveMCP, this means the raw JSON probe result is preserved in the error prefix — you can inspect it, fix the Lambda, and replay it by reading from the error prefix and writing back to the Firehose stream or the source Kinesis stream.

CloudWatch monitoring: the critical metrics

Firehose emits metrics to CloudWatch under the AWS/Firehose namespace. The minimal monitoring setup for a production AliveMCP pipeline:

The DeliveryToS3.DataFreshness alarm is the most important. Set the threshold to your buffer interval plus 2 minutes — for a 60-second buffer interval, alarm at >180 seconds; for a 300-second buffer interval, alarm at >420 seconds. If this metric exceeds the threshold, Firehose is not delivering records and the error prefix is accumulating data that must be replayed.

// CloudFormation / CDK snippet: DataFreshness alarm
{
  "Type": "AWS::CloudWatch::Alarm",
  "Properties": {
    "AlarmName": "alivemcp-firehose-delivery-stalled",
    "AlarmDescription": "Firehose delivery stalled — oldest record not delivered within buffer window",
    "Namespace": "AWS/Firehose",
    "MetricName": "DeliveryToS3.DataFreshness",
    "Dimensions": [{ "Name": "DeliveryStreamName", "Value": "alivemcp-probe-results" }],
    "Statistic": "Maximum",
    "Period": 60,
    "EvaluationPeriods": 3,
    "Threshold": 180,        // buffer_interval (60s) + 120s grace period
    "ComparisonOperator": "GreaterThanThreshold",
    "TreatMissingData": "notBreaching",  // No data = no records arriving = not a delivery failure
    "AlarmActions": ["arn:aws:sns:us-east-1:123456789012:alivemcp-ops-alerts"]
  }
}

Backup S3 bucket: safety net for failed records

Firehose supports a backup S3 destination that captures records before transformation and delivery. This provides a safety net if the transformation Lambda has a bug, the format conversion schema is wrong, or the destination bucket has a permission error — you always have the raw source records in the backup bucket to replay from.

ExtendedS3DestinationConfiguration: {
  // Primary delivery configuration...

  S3BackupMode: 'FailedDataOnly',  // 'Disabled' | 'FailedDataOnly' | 'Enabled' (all records)
  S3BackupConfiguration: {
    BucketARN: 'arn:aws:s3:::alivemcp-probe-backup',
    RoleARN: 'arn:aws:iam::123456789012:role/firehose-delivery-role',
    Prefix: 'raw-backup/!{timestamp:yyyy/MM/dd/HH}/',
    ErrorOutputPrefix: 'raw-backup-errors/!{firehose:error-output-type}/!{timestamp:yyyy/MM/dd/HH}/',
    BufferingHints: {
      SizeInMBs: 128,
      IntervalInSeconds: 900,  // Backup can use longer intervals — it's a safety net, not a query source
    },
    CompressionFormat: 'GZIP',
  },
}

Backup modes:

• Disabled: No backup. If transformation fails and the error prefix is misconfigured, records are lost permanently.
• FailedDataOnly: Captures only records that failed transformation or delivery. Provides a recovery path without doubling storage costs. Recommended for production.
• Enabled (All): Captures every record regardless of success or failure. Doubles your storage bill. Use only during initial pipeline validation or when doing a live schema migration where you need a guaranteed copy of all records.

Retry duration configuration

Firehose retries failed S3 deliveries for a configurable duration before routing to the error prefix. This applies to destination delivery failures (S3 unavailable, permission error) not transformation failures (those are retried by Lambda's own retry mechanism, not Firehose's retry duration).

ExtendedS3DestinationConfiguration: {
  // S3 delivery has no separate RetryOptions — Firehose retries S3 deliveries
  // indefinitely until the RetryDuration expires (for S3 destination, the default
  // is effectively unlimited — S3 is highly available and rarely causes permanent failures)

  // For HTTP endpoint destinations, configure retry duration explicitly:
  // HttpEndpointDestinationConfiguration: {
  //   RetryOptions: {
  //     DurationInSeconds: 300,  // Retry HTTP endpoint failures for 5 minutes before error prefix
  //   },
  // }
}

For S3 destinations, Firehose retries delivery with exponential backoff for up to 24 hours before permanently routing to the error prefix. In practice, S3 delivery failures are almost always transient permission issues or bucket policy changes that are corrected within minutes — the 24-hour retry window means no data is lost unless the issue persists for more than a day.

For HTTP endpoint destinations (webhooks, custom analytics receivers), set RetryOptions.DurationInSeconds based on your endpoint's expected recovery time. A value of 300 seconds (5 minutes) is appropriate for ephemeral endpoint failures; use 3,600 seconds (1 hour) if the endpoint runs on infrastructure that might be down for routine maintenance windows.

Replaying failed records from the error prefix

When records accumulate in the processing-failed/ prefix due to a Lambda bug, the fix requires three steps: fix the Lambda, then replay the failed records through the corrected pipeline:

import { FirehoseClient, PutRecordBatchCommand } from '@aws-sdk/client-firehose';
import { S3Client, GetObjectCommand, ListObjectsV2Command } from '@aws-sdk/client-s3';

const s3 = new S3Client({ region: 'us-east-1' });
const firehose = new FirehoseClient({ region: 'us-east-1' });

async function replayFailedRecords(
  bucket: string,
  prefix: string,
  deliveryStreamName: string
): Promise {
  // List all error objects in the specified time range
  const list = await s3.send(new ListObjectsV2Command({
    Bucket: bucket,
    Prefix: prefix,  // e.g., 'probe-errors/type=processing-failed/year=2026/month=10/day=30/'
  }));

  for (const obj of list.Contents ?? []) {
    // Download and decompress the error file
    const get = await s3.send(new GetObjectCommand({ Bucket: bucket, Key: obj.Key! }));
    const body = await get.Body!.transformToString('utf-8');

    // Each line is one original record (NDJSON format in error prefix)
    const records = body.trim().split('\n').map(line => ({
      Data: Buffer.from(line, 'utf-8'),
    }));

    // Replay in batches of 500 (PutRecordBatch limit)
    for (let i = 0; i < records.length; i += 500) {
      const batch = records.slice(i, i + 500);
      await firehose.send(new PutRecordBatchCommand({
        DeliveryStreamName: deliveryStreamName,
        Records: batch,
      }));
      // Firehose throttle: 1,000 records/s per delivery stream
      await new Promise(r => setTimeout(r, 500));
    }
  }
}

Replay considerations for AliveMCP probe data:

• Replayed records will have a different approximateArrivalTimestamp (the replay time, not the original probe time). Ensure downstream Athena queries use probedAt from the record payload as the event time, not the Firehose arrival timestamp.
• If the error prefix contains duplicate records (from multiple retry attempts before routing to error), deduplicate on probeId or the combination of serverSlug + probedAt before replaying.
• Replay at a rate below 1,000 records/second per delivery stream to avoid Firehose throttling.

Complete failure mode reference