Kinesis Firehose S3 Delivery Configuration for MCP Servers — Buffer Hints, Parquet Format Conversion, Compression

Buffer hint tuning: size vs interval tradeoffs

Firehose buffers records in memory and flushes to S3 when either the size limit or the interval limit is reached — whichever comes first. Lower limits mean more frequent, smaller S3 files; higher limits mean fewer, larger files.

Buffer presets for AliveMCP use cases:

S3 PUT pricing is $0.005 per 1,000 requests. The cost difference between 60-second and 900-second buffer intervals is minimal in absolute dollars (approximately $0.001/day per partition difference for a single delivery stream). The more important consideration is S3 small-file fragmentation: thousands of 5–10 KB files per prefix cause Athena to spend most of its time on metadata operations rather than actual data scanning, increasing query time and cost even without increasing data volume.

S3 prefix patterns and Hive-compatible partitioning

The S3 prefix determines how Firehose organizes delivered objects. Athena and Glue perform best with Hive-compatible prefixes that use key=value format — these enable partition projection, eliminating manual partition maintenance.

// Hive-compatible prefix (recommended for Athena)
Prefix: 'probe-results/year=!{timestamp:yyyy}/month=!{timestamp:MM}/day=!{timestamp:dd}/hour=!{timestamp:HH}/',
ErrorOutputPrefix: 'probe-results-errors/!{firehose:error-output-type}/year=!{timestamp:yyyy}/month=!{timestamp:MM}/day=!{timestamp:dd}/hour=!{timestamp:HH}/',

// Standard datetime prefix (simpler; requires MSCK REPAIR TABLE for new partitions)
Prefix: 'probe-results/!{timestamp:yyyy/MM/dd/HH}/',
ErrorOutputPrefix: 'probe-results-errors/!{firehose:error-output-type}/!{timestamp:yyyy/MM/dd/HH}/',

With Hive-compatible prefixes, configure Athena partition projection on the Glue table to auto-detect new time partitions without any operator action:

-- Athena table with partition projection (no MSCK REPAIR TABLE needed)
CREATE EXTERNAL TABLE probe_results (
  server_slug   STRING,
  tenant_id     STRING,
  status        STRING,
  response_time_ms BIGINT,
  error_code    STRING,
  probed_at     STRING
)
PARTITIONED BY (year STRING, month STRING, day STRING, hour STRING)
STORED AS PARQUET
LOCATION 's3://alivemcp-probe-archive/probe-results/'
TBLPROPERTIES (
  'projection.enabled'       = 'true',
  'projection.year.type'     = 'integer',
  'projection.year.range'    = '2026,2030',
  'projection.year.digits'   = '4',
  'projection.month.type'    = 'integer',
  'projection.month.range'   = '01,12',
  'projection.month.digits'  = '2',
  'projection.day.type'      = 'integer',
  'projection.day.range'     = '01,31',
  'projection.day.digits'    = '2',
  'projection.hour.type'     = 'integer',
  'projection.hour.range'    = '00,23',
  'projection.hour.digits'   = '2',
  'storage.location.template'= 's3://alivemcp-probe-archive/probe-results/year=${year}/month=${month}/day=${day}/hour=${hour}/'
);

Format conversion: JSON to Parquet via Glue Data Catalog

Format conversion transforms incoming JSON records to Parquet (columnar) or ORC format before writing to S3. Athena scans 5–10× less data on a Parquet table with the same records as a JSON table, because columnar format allows Athena to read only the columns referenced in the query rather than parsing entire JSON records.

Requirements for format conversion:

• A Glue Data Catalog database and table with the schema matching your records
• An IAM role granting Firehose glue:GetTable and glue:GetTableVersions permissions on the Glue table
• Source records must be JSON (not binary, not CSV)
• Format conversion is incompatible with dynamic partitioning inline parsing — if you use both, the Lambda preprocessor must handle both schema extraction and format normalization

// Enable format conversion in delivery stream configuration
ExtendedS3DestinationConfiguration: {
  // ... prefix, buffer hints, etc. ...

  DataFormatConversionConfiguration: {
    Enabled: true,
    InputFormatConfiguration: {
      Deserializer: {
        OpenXJsonSerDe: {},  // Parse input as JSON (handles nested objects, arrays)
        // Alternative: HiveJsonSerDe for strict Hive JSON format
      },
    },
    OutputFormatConfiguration: {
      Serializer: {
        ParquetSerDe: {
          Compression: 'SNAPPY',        // SNAPPY (fast, Parquet-native) or GZIP (better ratio)
          BlockSizeBytes: 268435456,     // 256 MB Parquet row group size (default)
          PageSizeBytes: 65536,          // 64 KB page size (default)
          EnableDictionaryCompression: true,  // Dictionary encoding for low-cardinality columns
          WriterVersion: 'V2',           // Parquet writer version (V2 improves null handling)
        },
        // Alternative: OrcSerDe for ORC format
      },
    },
    SchemaConfiguration: {
      RoleARN: 'arn:aws:iam::123456789012:role/firehose-glue-role',
      DatabaseName: 'alivemcp',
      TableName: 'probe_results',
      Region: 'us-east-1',
      VersionId: 'LATEST',  // Always use latest Glue table schema version
    },
  },
}

Format conversion adds approximately 3–5 seconds of processing overhead per buffer flush. This is negligible for batch archiving workloads but means the effective minimum delivery latency is ~65 seconds even with a 60-second buffer interval.

Parquet vs ORC: which to choose

Compression options for JSON delivery (without format conversion)

When not using format conversion (delivering plain JSON or NDJSON), Firehose applies compression to the entire S3 object after buffering:

For AliveMCP probe results as plain JSON, GZIP is the recommended compression. A probe result record averages ~500 bytes uncompressed. With GZIP at 8:1 ratio, a 64 MB buffer holding 128,000 records compresses to ~8 MB — a significant reduction in S3 storage costs over months of data accumulation.

Important: do not use ZIP (ZIP) if the table will be queried by Athena — Athena cannot read ZIP-compressed S3 files directly and will fail with an unreadable format error. Use GZIP instead.

IAM permissions for Firehose S3 delivery

The Firehose service role needs the following S3 permissions on the destination bucket:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "FirehoseS3Delivery",
      "Effect": "Allow",
      "Action": [
        "s3:AbortMultipartUpload",
        "s3:GetBucketLocation",
        "s3:GetObject",
        "s3:ListBucket",
        "s3:ListBucketMultipartUploads",
        "s3:PutObject"
      ],
      "Resource": [
        "arn:aws:s3:::alivemcp-probe-archive",
        "arn:aws:s3:::alivemcp-probe-archive/*"
      ]
    },
    {
      "Sid": "GlueSchemaAccess",
      "Effect": "Allow",
      "Action": [
        "glue:GetTable",
        "glue:GetTableVersion",
        "glue:GetTableVersions"
      ],
      "Resource": [
        "arn:aws:glue:us-east-1:123456789012:catalog",
        "arn:aws:glue:us-east-1:123456789012:database/alivemcp",
        "arn:aws:glue:us-east-1:123456789012:table/alivemcp/probe_results"
      ]
    }
  ]
}

The GetObject and ListBucket permissions are required for Firehose to verify the S3 bucket is accessible during stream creation, not for the actual delivery. Omitting them causes the delivery stream creation to fail with a permission validation error even though the object PUT operation would succeed.

Common failure modes