AWS Timestream WriteRecords for MCP Server Monitoring: Batch Writes, Multi-Measure Records, and Error Handling

Single-measure vs. multi-measure records

Timestream supports two record formats:

• Single-measure records: One record per metric per timestamp. A probe with five metrics (response_ms, status_code, is_up, schema_checksum, error_class) produces five separate records. Simple to write but expensive: 5× the write units, 5× the storage, and queries that correlate metrics require time-join operations.
• Multi-measure records: All metrics for a single event in one record. One probe cycle = one record with five MeasureValues. AWS recommends this format for monitoring workloads — it uses fewer write units, stores correlated data together for efficient range queries, and returns a single row per probe cycle rather than five.

For MCP monitoring, always use multi-measure records. The schema is fixed at the first write, so getting this right before you have production data matters.

// WRONG: single-measure — five records per probe cycle, expensive and awkward to query
const singleMeasureRecords = [
  { MeasureName: 'response_ms', MeasureValue: '142', MeasureValueType: 'BIGINT', Time: ts },
  { MeasureName: 'status_code', MeasureValue: '200', MeasureValueType: 'BIGINT', Time: ts },
  { MeasureName: 'is_up', MeasureValue: 'true', MeasureValueType: 'VARCHAR', Time: ts },
  // ...two more records
];

// RIGHT: multi-measure — one record per probe cycle
const multiMeasureRecord = {
  MeasureName: 'probe',           // logical group name
  MeasureValueType: 'MULTI',
  MeasureValues: [
    { Name: 'response_ms', Value: '142', Type: 'BIGINT' },
    { Name: 'status_code', Value: '200', Type: 'BIGINT' },
    { Name: 'is_up', Value: 'true', Type: 'VARCHAR' },
    { Name: 'schema_checksum', Value: 'a3f9b...', Type: 'VARCHAR' },
    { Name: 'error_class', Value: '', Type: 'VARCHAR' },
  ],
  Time: ts,
  TimeUnit: 'MILLISECONDS',
};

CommonAttributes: reduce payload for same-dimension batches

When a batch of records shares the same dimensions — for example, all 100 records come from the same probe worker in the same region running the same probe version — you can move those shared dimensions into CommonAttributes to avoid repeating them on every record. The Timestream service merges CommonAttributes with each record's Dimensions before storing.

import {
  TimestreamWriteClient,
  WriteRecordsCommand,
  type _Record,
} from '@aws-sdk/client-timestream-write';

const client = new TimestreamWriteClient({ region: 'us-east-1' });

// Probe worker that runs one region at a time
async function flushBatch(
  batchResults: ProbeResult[],
  probeRegion: string,
  probeVersion: string,
): Promise {
  const records: _Record[] = batchResults.map(r => ({
    // Only dimensions that differ per endpoint go here
    Dimensions: [
      { Name: 'server_slug', Value: r.serverSlug },
      { Name: 'protocol', Value: r.protocol },
    ],
    MeasureName: 'probe',
    MeasureValueType: 'MULTI',
    MeasureValues: [
      { Name: 'response_ms', Value: String(r.responseMs), Type: 'BIGINT' },
      { Name: 'status_code', Value: String(r.statusCode), Type: 'BIGINT' },
      { Name: 'is_up', Value: String(r.isUp), Type: 'VARCHAR' },
      { Name: 'schema_checksum', Value: r.schemaChecksum, Type: 'VARCHAR' },
      { Name: 'error_class', Value: r.errorClass, Type: 'VARCHAR' },
    ],
    Time: String(r.checkTime.getTime()),
    TimeUnit: 'MILLISECONDS',
  }));

  await client.send(new WriteRecordsCommand({
    DatabaseName: 'mcp-probe-metrics',
    TableName: 'probe-results',
    Records: records,
    // Shared dimensions merged into every record — saves ~30% payload size for
    // batches where probe_version and region are constant
    CommonAttributes: {
      Dimensions: [
        { Name: 'region', Value: probeRegion },
        { Name: 'probe_version', Value: probeVersion },
      ],
    },
  }));
}

Note: if a record's own Dimensions array contains a dimension name that also appears in CommonAttributes.Dimensions, the record-level value wins. This is useful for multi-region probe workers that want to override the common region for specific endpoints.

Handling RejectedRecordsException without data loss

RejectedRecordsException is a partial-failure response: the records that Timestream accepted were written successfully; only the records in the RejectedRecords[] array were dropped. This exception is not a complete failure — do not retry the entire batch on this error.

async function writeBatchWithRejectionHandling(
  results: ProbeResult[],
): Promise {
  const records = results.map(toTimestreamRecord);

  try {
    await client.send(new WriteRecordsCommand({
      DatabaseName: 'mcp-probe-metrics',
      TableName: 'probe-results',
      Records: records,
    }));
  } catch (err: any) {
    if (err.name === 'RejectedRecordsException') {
      // RejectedRecords is an array of { RecordIndex: number, Reason: string, ExistingVersion?: number }
      const rejected: Array<{ RecordIndex: number; Reason: string }> =
        err.RejectedRecords ?? [];

      for (const r of rejected) {
        const source = results[r.RecordIndex];
        // Common reasons:
        // "The record timestamp is outside the time range of the table's retention settings"
        //   → record is too old for memory store and magnetic store writes are disabled
        // "The record has multiple measure value types for the same measure name"
        //   → schema conflict; you changed a measure value type
        // "Version must be greater than the existing version for upsert"
        //   → duplicate record with a lower version number
        console.error(
          `[timestream-reject] slug=${source.serverSlug} ` +
          `time=${source.checkTime.toISOString()} reason=${r.Reason}`
        );
        // Route to dead-letter queue or S3 for manual inspection
        await deadLetterQueue.send(JSON.stringify({ source, reason: r.Reason }));
      }
      // Non-rejected records were written — do NOT retry the entire batch
      const acceptedCount = records.length - rejected.length;
      console.info(`[timestream] ${acceptedCount}/${records.length} records written`);
    } else {
      throw err;
    }
  }
}

The most common rejection reason for MCP probe workers is the late-arrival case: a probe worker was offline for more than memoryStoreRetentionPeriodInHours hours and is now replaying buffered records. If enableMagneticStoreWrites is false, these are rejected. Enable magnetic store writes and check the S3 error bucket to catch rejections that even the error handler misses (e.g. network errors that prevent the exception from being received).

Late-arrival records: configuring magnetic store writes

A fault-tolerant probe worker buffers records locally when Timestream is unreachable and replays them when connectivity recovers. If the outage lasted longer than the memory store retention window (e.g. 24 hours), those replayed records would normally be rejected. Magnetic store writes accept records with older timestamps directly into the magnetic store, bypassing the memory store entirely.

// CDK: enable late-arrival writes with S3 error bucket
import * as timestream from 'aws-cdk-lib/aws-timestream';
import * as s3 from 'aws-cdk-lib/aws-s3';

const table = new timestream.CfnTable(this, 'ProbeTable', {
  databaseName: 'mcp-probe-metrics',
  tableName: 'probe-results',
  retentionProperties: {
    memoryStoreRetentionPeriodInHours: 24,
    magneticStoreRetentionPeriodInDays: 93,
  },
  magneticStoreWriteProperties: {
    enableMagneticStoreWrites: true,
    magneticStoreRejectedDataLocation: {
      s3Configuration: {
        bucketName: errorBucket.bucketName,
        objectKeyPrefix: 'timestream-rejected/',
        encryptionOption: 'SSE_S3',
      },
    },
  },
});

The S3 error bucket captures records that were rejected even from the magnetic store (e.g. schema conflicts, records older than the magnetic store retention period). Each rejected record is written as a JSON object under the prefix you specify. Inspect this bucket periodically to catch silent data quality issues in your probe pipeline.

Late arrivals written directly to the magnetic store are not visible in queries that target only the memory store. Timestream's SQL automatically spans both tiers when you query with a WHERE time BETWEEN ago(90d) AND now() clause — no special handling needed in the query layer.

Write throughput and batching strategy

At 1,000 monitored endpoints probed every 60 seconds, the probe worker produces 1,000 records per minute. This fits comfortably in 10 WriteRecordsCommand calls of 100 records each, sent every 60 seconds. The account-level default write quota is 1,000 records/s per table; 10 calls × 100 records ÷ 60 seconds = 16.7 records/s — well within limits.

At 50,000 endpoints (a large-scale deployment), the math changes: 50,000 records per 60-second window = 833 records/s sustained, with bursts during the first second of each probe cycle. Use a queue-and-drain pattern:

class TimestreamBatchWriter {
  private queue: _Record[] = [];
  private flushTimer: ReturnType<typeof setInterval> | null = null;

  constructor(
    private client: TimestreamWriteClient,
    private dbName: string,
    private tableName: string,
    private maxBatchSize = 100,
    private flushIntervalMs = 1000,
  ) {
    this.flushTimer = setInterval(() => this.flush(), this.flushIntervalMs);
  }

  enqueue(record: _Record): void {
    this.queue.push(record);
    if (this.queue.length >= this.maxBatchSize) {
      // Don't wait for the timer — flush immediately when full
      setImmediate(() => this.flush());
    }
  }

  private async flush(): Promise {
    if (this.queue.length === 0) return;
    const batch = this.queue.splice(0, this.maxBatchSize);
    try {
      await this.client.send(new WriteRecordsCommand({
        DatabaseName: this.dbName,
        TableName: this.tableName,
        Records: batch,
      }));
    } catch (err) {
      // Re-enqueue on transient errors (ThrottlingException, ServiceUnavailableException)
      this.queue.unshift(...batch);
      throw err;
    }
  }

  destroy(): void {
    if (this.flushTimer) clearInterval(this.flushTimer);
  }
}

IAM policy for the probe worker

The probe worker only needs to write records and look up the regional endpoint. The DescribeEndpoints permission is required — the SDK calls it automatically to discover the regional Timestream endpoint on startup and caches the response for 1 hour.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "TimestreamProbeWrite",
      "Effect": "Allow",
      "Action": [
        "timestream:WriteRecords"
      ],
      "Resource": "arn:aws:timestream:us-east-1:123456789012:database/mcp-probe-metrics/table/probe-results"
    },
    {
      "Sid": "TimestreamEndpointDiscovery",
      "Effect": "Allow",
      "Action": [
        "timestream:DescribeEndpoints"
      ],
      "Resource": "*"
    }
  ]
}

Keep the write policy scoped to the specific table ARN, not * — probe workers should not be able to write to or read from other tables in the same database. If you run multiple probe workers for different customers (e.g. one per team on the Team plan), give each its own IAM role scoped to its own table.

Failure modes