MSK Advanced Patterns for MCP Server Monitoring Pipelines: Serverless vs Provisioned, Durability Model, Schema Evolution, and Producer Tuning

Pattern 1 — MSK Serverless: Capacity Model, IAM Auth, and Feature Trade-offs

Why MSK Serverless changes the capacity planning conversation

On a provisioned MSK cluster, throughput is a product of broker count, instance type, and partition count: you plan in advance, size conservatively to handle peak probe waves, and pay for that capacity whether you use it or not. MSK Serverless replaces that model with a single per-cluster soft quota:

For AliveMCP probe pipelines, MSK Serverless is a good fit when probe ingest volume is bursty — many probes completing simultaneously during a scan wave — and when consumer throughput is predictable and bounded by the 120-partition limit. At 2,000 probe completions per second producing roughly 1.5 KB each, peak ingress is approximately 3 MiBps, well within the 200 MiBps quota. The binding constraint for scale-out is the 120-partition soft limit: each consumer thread needs its own partition for ordered processing, so a probe fleet with more than ~120 concurrent consumer threads requires provisioned MSK.

IAM-only authentication: what changes at the client level

MSK Serverless enforces SASL/IAM for every client connection. There is no path to unauthenticated, SCRAM-SHA-512, or mTLS access — the cluster rejects connections that do not present a valid IAM credential. Every producer and consumer must:

1. Have an IAM principal (Lambda execution role, ECS task role, EC2 instance profile, or explicit access key)
2. Use the SASL_SSL protocol with sasl.mechanism=AWS_MSK_IAM
3. Include the aws-msk-iam-sasl-signer-js (Node.js), aws-msk-iam-auth (Java), or equivalent library that signs tokens using SigV4
4. Hold an IAM policy granting at minimum: kafka-cluster:Connect, kafka-cluster:DescribeTopic, and kafka-cluster:WriteData (for producers) or kafka-cluster:ReadData + kafka-cluster:DescribeGroup + kafka-cluster:AlterGroup (for consumers)

MSK IAM tokens expire after 15 minutes; the client library handles token refresh automatically. If the IAM role is missing kafka-cluster:AlterGroup, the consumer can read messages but cannot commit offsets — commits fail silently in some client versions, causing the consumer to re-read from the last committed offset after every restart. Always verify the full permission set in staging before moving to production.

When to choose provisioned MSK instead

MSK Serverless is the wrong choice for probe pipelines that need any of the following. If you need even one of these, start with provisioned MSK:

• Per-topic retention controls: MSK Serverless applies a single retention policy at the cluster level. If probe results must be retained for 7 days while probe commands can be deleted after 1 hour, provisioned MSK is required.
• Log compaction: A probe pipeline that maintains a compacted "last known status" topic (one record per endpoint, always the latest probe result) requires cleanup.policy=compact, which is not available on Serverless.
• Cross-region DR via MSK Replication: MSK Replication is only available on provisioned clusters. If your uptime SLA requires a standby consumer in a different AWS region with a committed failover time, plan for provisioned from the start.
• MSK Connect sink connectors: If the probe result pipeline needs to sink to S3, OpenSearch, or a JDBC target via a managed connector, provisioned MSK is required.
• More than 120 concurrent consumer partitions: The 120-partition soft limit is a real ceiling, not a guideline. A probe fleet with more than 120 parallel consumers reading ordered partitions will hit this limit.

// CDK: MSK Serverless cluster with IAM auth
import { aws_msk as msk, aws_iam as iam } from 'aws-cdk-lib';

const serverlessCluster = new msk.CfnServerlessCluster(this, 'ProbeCluster', {
  clusterName: 'alivemcp-probe-serverless',
  clientAuthentication: {
    sasl: { iam: { enabled: true } }
  },
  vpcConfigs: [{
    subnetIds: vpc.privateSubnets.map(s => s.subnetId),
    securityGroups: [probeClusterSg.securityGroupId],
  }],
});

// Grant producer access to probe worker role
const producerPolicy = new iam.PolicyStatement({
  actions: [
    'kafka-cluster:Connect',
    'kafka-cluster:DescribeTopic',
    'kafka-cluster:WriteData',
    'kafka-cluster:CreateTopic',
  ],
  resources: [
    serverlessCluster.attrArn,
    `${serverlessCluster.attrArn}/topic/probe-results`,
    `${serverlessCluster.attrArn}/topic/probe-commands`,
  ],
});
probeWorkerRole.addToPolicy(producerPolicy);

Pattern 2 — Kafka Durability Model: RF, min.ISR, acks, and Idempotence

The three-way durability system

Kafka durability is not controlled by a single setting. It is the product of three independent configurations that must be set consistently to achieve the intended guarantee. Getting one wrong without the others renders the guarantee weaker than intended:

1. Replication factor (RF) — how many broker copies of each partition exist. RF=1 means there is only one copy; if that broker fails before the message is consumed, the data is gone. RF=3 means three brokers each hold a copy; the cluster can survive one broker failure without losing data.
2. min.insync.replicas (min.ISR) — the minimum number of in-sync replicas (replicas caught up within the lag threshold) that must acknowledge a write with acks=all before the producer considers it durable. With RF=3 and min.ISR=2: the write is acknowledged when two of three brokers confirm it; one broker can fail after acknowledgement without data loss.
3. Producer acks — which acknowledgement the broker sends before the producer's send() call returns. acks=0: no ack (fire and forget); acks=1: leader only; acks=all: all ISR members (subject to min.ISR).

The standard AliveMCP configuration for the probe-results topic is RF=3/min.ISR=2/acks=all. This means: a probe result is considered written when two of the three MSK brokers have confirmed it. If one broker fails after acknowledgement, the two remaining copies are sufficient. If a second broker fails before the first recovers, the topic becomes unavailable for acks=all producers — they receive NotEnoughReplicasException and must retry. This is the right trade-off: briefly unavailable is better than silently losing monitoring data.

Idempotent producer: preventing duplicates on retry

With acks=all and retries enabled, there is a subtle failure mode: the producer sends a probe result, all in-sync replicas write it, the leader sends the acknowledgement, but the network drops the ack before it reaches the producer. The producer times out and retries. The broker receives the retry and writes it again — now there are two copies of the same probe result. Downstream consumers process it twice: duplicate alert, double-counted latency sample, two rows in the verdict store with slightly different processing timestamps.

Setting enable.idempotence=true solves this. The broker assigns each producer a session-scoped producer ID (PID) and sequences every message per-partition. When a retry arrives with the same PID and sequence number as an already-written message, the broker deduplicates it silently — the retry succeeds (returns success to the producer) but only one copy is stored. Key constraints:

• Idempotence is scoped to the producer session. If the producer process restarts, it gets a new PID — messages sent before the restart are not deduplicated against messages sent after.
• enable.idempotence=true automatically enforces: acks=all (upgraded if not set), retries=Integer.MAX_VALUE (upgraded if below), and max.in.flight.requests.per.connection=5 (capped if above). Do not set these to conflicting values.
• Idempotence is per-partition. A message written to partition 3 with PID=X and sequence=100 is only deduplicated against other writes to partition 3 from the same PID.

Bounding retries with delivery.timeout.ms

With retries=Integer.MAX_VALUE, a producer that hits a NotEnoughReplicasException will retry indefinitely. For a 60-second probe cycle, retrying indefinitely means a probe result from one cycle can still be in-flight when the next cycle begins — creating a time-ordering problem in the verdict store. Setting delivery.timeout.ms=30000 bounds the total retry window to 30 seconds: after 30 seconds, the send fails with a TimeoutException and the probe result is dropped. The next probe cycle will re-verify the endpoint status — a 30-second delivery window aligns probe result delivery with the probe cycle rather than leaking across cycle boundaries.

// CDK: MSK topic configuration for durable probe result ingestion
import { aws_msk as msk } from 'aws-cdk-lib';

const probeCluster = new msk.Cluster(this, 'ProbeCluster', {
  clusterName: 'alivemcp-probe',
  kafkaVersion: msk.KafkaVersion.V3_5_1,
  numberOfBrokerNodes: 3,
  instanceType: ec2.InstanceType.of(ec2.InstanceClass.M5, ec2.InstanceSize.LARGE),
  encryptionInTransit: {
    clientBroker: msk.ClientBrokerEncryption.TLS,
  },
});

// Producer config for probe-results topic (Node.js kafkajs)
const producer = kafka.producer({
  acks: -1,                          // acks=all
  idempotent: true,                  // enable.idempotence=true
  maxInFlightRequests: 5,            // enforced by idempotence
  transactionTimeout: 30000,
});

// delivery.timeout.ms equivalent: message-level timeout
const result = await producer.send({
  topic: 'probe-results',
  timeout: 30000,                    // delivery.timeout.ms=30000
  messages: [{ key: endpointId, value: JSON.stringify(probeResult) }],
});

MSK Replication for cross-region DR (provisioned only)

MSK Replication (generally available since 2024) copies topics from a source provisioned MSK cluster to a destination cluster in a different AWS region. Unlike MirrorMaker2, MSK Replication is fully managed: no connector to run, no consumer group to monitor. On failover, restart the standby consumer group from the last committed offset on the destination cluster — synchroniseConsumerGroupOffsets translates source offsets to destination offsets during replication, eliminating the need for manual offset reset. Monitor ReplicationByteLag in CloudWatch: a steady non-zero lag means the destination is catching up; a continuously increasing lag means replication is falling behind and the failover RTO will include a replay period. Alert at >50 MB lag for probe pipelines where freshness matters.

Pattern 3 — Glue Schema Registry: Compatibility Modes, Probe Payload Versioning, and Cross-Account Access

How Glue Schema Registry integrates with MSK messages

When a Glue-aware Kafka producer sends a message, the serializer prepends a 5-byte header to the message value: 1 magic byte (0x03 — the Glue magic byte, not the Confluent 0x00) followed by a 4-byte schema-version UUID index. The consumer deserializer reads the header, calls glue:GetSchemaVersion to retrieve the schema definition, and uses it to decode the payload. Schemas are cached per producer/consumer session — the Glue API is not called per message. This architecture means a producer code change that makes an incompatible schema change fails at schema registration time (when the producer starts and tries to register its schema version), not at runtime when a consumer fails to deserialize.

The Glue Schema Registry uses a different API surface than the Confluent Schema Registry. There is no schema.registry.url configuration property. Instead, producers and consumers are configured with AWS credentials and a Glue registry name; the Glue SDK resolves schema versions via the regional Glue API endpoint. This is important for library selection: Confluent's KafkaAvroSerializer does not work with Glue's registry — you need the AWS Glue Schema Registry SerDe libraries.

Compatibility modes: what each allows and rejects

For AliveMCP probe result schemas, BACKWARD is the correct mode. The probe worker fleet is updated before the archiver and alert Lambda fleet — producers upgrade first, consumers catch up. BACKWARD allows adding new optional fields (for example, tlsCertExpiresAt with a null default when certificate expiry monitoring is added) while ensuring that old consumer versions — still running during a rolling deployment — can still deserialize the new message format because the new field has a default.

Schema evolution examples for probe payloads

The evolution rules are concrete once mapped to probe payload fields:

The Glue serializer validates the new schema version against the compatibility mode at registration time. A BACKWARD violation causes the RegisterSchemaVersion API call to fail with a compatibility error — the producer fails to start, not at message send time. This makes schema violations a CI/CD-time failure rather than a production incident: run the producer startup in a staging environment with schema registration enabled, and any BACKWARD violation fails the deployment pipeline before reaching production consumers.

Cross-account consumer access

For analytics pipelines in a separate AWS account that need to read and deserialize probe events, the Glue Schema Registry access model does not require VPC peering or PrivateLink. The Glue API is a regional public endpoint. The pattern is a resource-based policy on the registry:

// Resource-based policy on Glue registry granting cross-account read access
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::ANALYTICS-ACCOUNT-ID:root"
      },
      "Action": [
        "glue:GetSchemaVersion",
        "glue:GetSchemaByDefinition",
        "glue:QuerySchemaVersionMetadata",
        "glue:GetRegistry"
      ],
      "Resource": "arn:aws:glue:us-east-1:PROBE-ACCOUNT-ID:registry/alivemcp-probe-schemas"
    }
  ]
}

With this policy, the analytics account consumer calls glue:GetSchemaVersion using its own IAM credentials — no need for the producer account to assume roles or pre-share schema definitions. Schema lookups are cached in the consumer process for the session lifetime, so the cross-account API call is not on the hot path for message deserialization.

Pattern 4 — Producer Configuration: acks, Compression, Batching, and Idempotence

Two topic types, two producer profiles

A probe pipeline typically has two distinct Kafka topics with different requirements:

• probe-results: High volume (thousands per second), must not be lost (verdict store depends on it), each result is unique, producers may retry on transient failure — requires acks=all, compression, large batch size, idempotence.
• probe-commands: Lower volume, latency-sensitive (probe workers need the command quickly to start the next probe), idempotent by design (if a command is lost, the next scheduled probe re-issues it), consumers are fast — requires acks=1, no compression, default batch size, zero linger.

Running the same producer configuration for both topics optimizes for neither. Configure separate Kafka.producer() instances with different settings, or use per-topic overrides where supported by your client library.

LZ4 compression: the right choice for probe JSON

Probe result messages are structured JSON with predictable field names (endpointId, status, responseTimeMs, probedAt, etc.) — highly compressible because the field names repeat identically across millions of messages. Kafka compresses at batch level — all messages in a batch are compressed together, which achieves better compression than per-message compression because repeated field names across messages become a single dictionary entry.

LZ4 is the preferred codec for the probe-results topic. It achieves 3–5× compression at very low CPU overhead, keeping Lambda and ECS probe worker CPU free for the actual HTTP probe requests rather than message compression. The broker stores messages in their compressed form; consumers pay the decompression cost on fetch, but LZ4 decompression is also fast. Set compression.type=producer on the broker (the default) to respect the producer's codec choice rather than recompressing.

batch.size and linger.ms: the batching math

The producer accumulates messages into a batch before flushing to the broker. The batch is flushed when either the batch reaches batch.size bytes or linger.ms elapses, whichever comes first.

Default settings: batch.size=16384 (16 KB), linger.ms=0. With linger.ms=0, the producer sends every message the instant it arrives — the batch fills only if the producer is faster than the network layer picks it up. For a probe worker producing one message per probe result, this means one Kafka request per probe result — 2,000 results/second = 2,000 Kafka requests/second, each incurring a SASL/TLS round-trip overhead.

Tuned settings: batch.size=65536 (64 KB), linger.ms=5. At 2,000 probe results/second with average probe result size of 1.5 KB (uncompressed) and LZ4 compression achieving 4× compression, each message compresses to roughly 375 bytes. A 64 KB batch holds roughly 170 compressed messages. At 2,000 messages/second, the batch fills in about 85ms — the batch.size trigger fires before linger.ms in most cases. The result is roughly 24 Kafka requests per second — an 83× reduction in broker connection overhead. Even without LZ4, each 1.5 KB uncompressed message means the 64 KB batch fills in ~32ms at 2,000 messages/second, yielding ~62 requests/second instead of 2,000.

Idempotence: enforcement and constraints

Setting enable.idempotence=true on the producer activates the broker-side deduplication mechanism described in Pattern 2. At the producer configuration level, idempotence automatically enforces three constraints — setting conflicting values causes the Kafka client to throw a configuration error at startup:

• acks must be all (or -1). If set to 1 or 0, the client throws ConfigException: Must set acks to all in order to use the idempotent producer.
• retries must be greater than 0. The client sets it to Integer.MAX_VALUE if not specified.
• max.in.flight.requests.per.connection must be ≤ 5. This is the Kafka protocol constraint for per-partition sequence tracking. Values above 5 are rejected with a configuration error.

For the Node.js kafkajs library, idempotence is enabled via idempotent: true in the producer config, which maps to enable.idempotence=true. The library will enforce maxInFlightRequests: 5 automatically. Do not set maxInFlightRequests above 5 in the same config or the producer will fail to initialize.

Combined CDK configuration

// Full producer config for probe-results and probe-commands topics
import { Kafka } from 'kafkajs';
import { MSKAuthProvider } from 'aws-msk-iam-sasl-signer-js';

const kafka = new Kafka({
  clientId: 'alivemcp-probe-worker',
  brokers: [process.env.MSK_BOOTSTRAP_SERVERS!],
  ssl: true,
  sasl: {
    mechanism: 'oauthbearer',
    oauthBearerProvider: async () => {
      const signer = new MSKAuthProvider({ region: 'us-east-1' });
      const token = await signer.tokenProvider();
      return { value: token.token };
    },
  },
});

// High-durability producer for probe results
const resultsProducer = kafka.producer({
  idempotent: true,            // enable.idempotence=true; forces acks=all, retries=MAX
  maxInFlightRequests: 5,      // required for idempotence
  compression: CompressionTypes.LZ4,
  // batch.size=65536 and linger.ms=5 via producer-level config:
  // Note: kafkajs sets these per send() call; for global defaults use
  // the KafkaJS 2.x createPartitioner + lingerMs option
});

// Low-latency producer for probe commands
const commandsProducer = kafka.producer({
  acks: 1,                     // leader only — commands are idempotent
  maxInFlightRequests: 10,     // higher throughput for command dispatch
  // no compression, default batch.size, linger.ms=0
});

// Send probe result with delivery timeout
await resultsProducer.send({
  topic: 'probe-results',
  timeout: 30000,              // delivery.timeout.ms bound
  messages: [{ key: endpointId, value: JSON.stringify(probeResult) }],
});

// Send probe command (fire-and-forget style, idempotent receiver)
await commandsProducer.send({
  topic: 'probe-commands',
  messages: [{ key: endpointId, value: JSON.stringify(probeCommand) }],
});

Cross-Pattern Failure Modes

Related guides

• MSK Serverless for MCP Servers — Capacity Model, IAM-Only Auth, and Probe Pipeline Trade-offs
• MSK Replication for MCP Servers — Cross-Region DR, Replication Factor, and Producer Acks
• MSK Schema Registry for MCP Servers — Glue Schema Registry, Compatibility Modes, and Probe Payload Versioning
• MSK Producer Configuration for MCP Servers — acks, Compression, batch.size, linger.ms, and Idempotence
• MSK for MCP Servers — Kafka Cluster Setup, Topic Design, and Consumer Groups
• SQS Advanced Patterns for MCP Server Monitoring Pipelines
• SNS Advanced Patterns for MCP Server Monitoring Pipelines
• Kinesis for MCP Server Monitoring Pipelines