Kinesis Firehose Dynamic Partitioning for MCP Servers — JQ Expressions, Partition Keys, S3 Prefix Routing

What dynamic partitioning solves

The default Firehose S3 delivery uses a single prefix with optional datetime components: probe-results/!{timestamp:yyyy/MM/dd/HH}/. Every record — regardless of which MCP endpoint it describes or which tenant owns it — lands in the same hourly files. An Athena query asking "show me all probe failures for tenant XYZ in the last 7 days" must scan every record from every tenant for every hour over 7 days. At 5 million probe records per day across 1,000 tenants, that is a full 35 million record scan for what is conceptually a ~35,000 record query.

Dynamic partitioning routes each record to its own S3 key at ingest time. The same query scans only the files under probe-results/tenant=XYZ/, reducing data scanned by 1,000×. This is especially valuable for AliveMCP where probe results naturally segment by serverSlug (one per MCP endpoint) and status (healthy/degraded/down), and where tenants on paid plans expect query isolation.

Enabling dynamic partitioning

Dynamic partitioning is configured at delivery stream creation time. It cannot be enabled on an existing stream — you must create a new stream or recreate the existing one. The following CloudFormation / IaC excerpt shows the relevant parameters:

// AWS SDK v3 — CreateDeliveryStream with dynamic partitioning
import { FirehoseClient, CreateDeliveryStreamCommand } from '@aws-sdk/client-firehose';

const client = new FirehoseClient({ region: 'us-east-1' });

await client.send(new CreateDeliveryStreamCommand({
  DeliveryStreamName: 'alivemcp-probe-results',
  DeliveryStreamType: 'KinesisStreamAsSource',
  KinesisStreamSourceConfiguration: {
    KinesisStreamARN: 'arn:aws:kinesis:us-east-1:123456789012:stream/alivemcp-results',
    RoleARN: 'arn:aws:iam::123456789012:role/firehose-kinesis-role',
  },
  ExtendedS3DestinationConfiguration: {
    BucketARN: 'arn:aws:s3:::alivemcp-probe-archive',
    RoleARN: 'arn:aws:iam::123456789012:role/firehose-s3-role',

    // Prefix uses dynamic partition keys + timestamp
    Prefix: 'probe-results/tenant=!{partitionKeyFromQuery:tenantId}/tool=!{partitionKeyFromQuery:toolName}/!{timestamp:yyyy/MM/dd/HH}/',
    ErrorOutputPrefix: 'probe-results-errors/!{firehose:error-output-type}/!{timestamp:yyyy/MM/dd/HH}/',

    BufferingHints: {
      SizeInMBs: 64,
      IntervalInSeconds: 60,
    },

    // Enable dynamic partitioning
    DynamicPartitioningConfiguration: {
      Enabled: true,
      RetryOptions: {
        DurationInSeconds: 300,  // Retry failed partition key extractions for 5 min
      },
    },

    // Inline parsing: extract partition keys from JSON records
    ProcessingConfiguration: {
      Enabled: true,
      Processors: [
        {
          Type: 'MetadataExtraction',
          Parameters: [
            {
              ParameterName: 'MetadataExtractionQuery',
              // JQ-style: extract tenantId and toolName from the JSON record
              ParameterValue: '{tenantId: .tenantId, toolName: .toolName}',
            },
            {
              ParameterName: 'JsonParsingEngine',
              ParameterValue: 'JQ-1.6',
            },
          ],
        },
        // Multi-record deaggregation: required for dynamic partitioning
        {
          Type: 'AppendDelimiterToRecord',
          Parameters: [
            { ParameterName: 'Delimiter', ParameterValue: '\\n' },
          ],
        },
      ],
    },
  },
}));

Key configuration decisions in the above:

• Prefix syntax: !{partitionKeyFromQuery:tenantId} references the tenantId key extracted by the JQ expression. The key name must match exactly — case-sensitive — between the JQ output object and the prefix reference.
• ErrorOutputPrefix: !{firehose:error-output-type} is a built-in variable that resolves to processing-failed, delivery-failed, or backup-failed depending on the failure type. Always specify this to keep error records separate from successful delivery paths.
• AppendDelimiterToRecord processor: Required alongside MetadataExtraction when writing JSON records to S3. Firehose concatenates records without delimiters by default — the newline processor ensures each record is on its own line, making the S3 files directly parseable by Athena.

JQ expression syntax and limits

Firehose inline parsing uses JQ 1.6 syntax to extract partition keys from JSON records. The expression must return an object where each key becomes a partition key name:

Hard constraints on dynamic partition keys:

• Maximum 50 partition keys per delivery stream
• Partition key values are truncated to 2,048 bytes
• If the JQ expression fails to extract a key (null result or invalid JSON), the record is routed to the error prefix — it is NOT delivered with a fallback default unless you explicitly provide one with // "fallback"
• JQ expressions cannot make network calls or access external state
• The extracted partition key value is included in the S3 object key as-is — special characters are NOT URL-encoded. Avoid values containing /, ?, #, or spaces in partition key fields

Buffer semantics per partition

This is the most misunderstood aspect of dynamic partitioning. The buffer hints (SizeInMBs and IntervalInSeconds) apply per active partition, not globally. A Firehose stream with 1,000 active tenant partitions receiving 1 record per tenant per minute will flush each partition's buffer every 60 seconds — even though the global record rate may be high enough to flush a single-partition stream every second.

Implications for AliveMCP:

• High-traffic tenants: A tenant with 500 monitored endpoints generating 500 probe results per minute will fill a 64 MB buffer quickly. The buffer flushes on whichever limit is hit first — likely time (60 seconds) at typical probe result sizes (~2 KB each: 500 × 2 KB = 1 MB per minute, well under 64 MB).
• Low-traffic tenants: A free-tier user with 1 monitored endpoint generates 1 probe result per minute. Their partition's buffer holds 1 MB/60s = 1 MB/hour. The buffer will flush on the time limit (60 seconds), creating one small S3 file per minute. This is wasteful — consider routing free-tier tenants to a shared "free" partition rather than per-tenant partitioning.
• Cost impact: Each S3 PUT costs $0.005 per 1,000 requests. A low-traffic partition flushing 1,440 times per day = 1,440 PUT requests = $0.0072/day per partition. With 10,000 free-tier tenants, that's $72/day in PUT costs alone — 22× more than if free-tier traffic went to a single shared partition.

Recommendation: partition by coarse-grained keys first (e.g., tier = paid/free, then date), and add fine-grained per-tenant partitioning only for paid tiers with sufficient traffic to justify the S3 file fragmentation cost.

Multi-record deaggregation

When dynamic partitioning is enabled, Firehose must deaggregate multi-record batches before extracting partition keys — because different records in the same batch may belong to different partitions. The AppendDelimiterToRecord processor handles this by ensuring each record is newline-terminated before the MetadataExtraction processor runs.

If you are using KPL (Kinesis Producer Library) aggregation to batch records at the producer, you must add the Decompression and KPL deaggregation processors before MetadataExtraction:

// If producer uses KPL aggregation, add KPL deaggregation first
Processors: [
  {
    Type: 'RecordDeAggregation',
    Parameters: [
      { ParameterName: 'SubRecordType', ParameterValue: 'DELIMITED' },
    ],
  },
  {
    Type: 'MetadataExtraction',
    Parameters: [
      { ParameterName: 'MetadataExtractionQuery', ParameterValue: '{tenantId: .tenantId, toolName: .toolName}' },
      { ParameterName: 'JsonParsingEngine', ParameterValue: 'JQ-1.6' },
    ],
  },
  {
    Type: 'AppendDelimiterToRecord',
    Parameters: [{ ParameterName: 'Delimiter', ParameterValue: '\\n' }],
  },
]

Processor ordering matters: deaggregation must precede metadata extraction, which must precede the delimiter appender. Reversing the order causes the metadata extractor to see the aggregated record envelope rather than the individual JSON records, causing all keys to fail extraction and route to the error prefix.

S3 prefix expression reference

Firehose prefix expressions support three variable types:

The !{timestamp:} format uses Java DateTimeFormatter patterns. Common patterns:

• yyyy/MM/dd/HH — hourly Hive-style prefix (recommended for Athena)
• yyyy-MM-dd — daily prefix, simpler but less Athena-efficient
• yyyy/MM/dd — daily Hive-style (works with Athena partition projection)

Use Hive-compatible prefixes (key=value/ format) when registering the S3 location as an Athena table — it enables Athena partition projection, eliminating the need to run MSCK REPAIR TABLE after each day's data arrives.

Common failure modes