MSK Producer Configuration for MCP Servers — acks, Compression, batch.size, linger.ms, and Idempotence

acks: durability vs latency

The acks config controls how many broker acknowledgements the producer waits for before returning success from send(). It interacts directly with min.insync.replicas on the cluster (see MSK Replication guide for the full durability matrix). For probe pipelines:

• acks=all (recommended for probe-results topic): write acknowledged only after all in-sync replicas confirm. With min.ISR=2 and RF=3, one broker can fail without data loss. Added latency is typically 2–5ms on same-region brokers.
• acks=1 (acceptable for probe-commands topic): leader acknowledgement only. Probe commands are idempotent by design — if a command is lost, the next probe cycle re-issues it. The latency saving (1–2ms) is worth the theoretical data loss window.
• acks=0: never use for durable data. Appropriate only for telemetry where high send rate matters more than completeness.

Compression: codec comparison for probe JSON

Probe result messages are structured JSON with predictable field names (endpointId, status, responseTimeMs, probedAt, etc.) — highly compressible. The producer compresses at batch level: all messages in a batch are compressed together, which is more efficient than per-message compression.

LZ4 is the preferred codec for AliveMCP's probe result producer: it gives 3–5× compression at very low CPU overhead, keeping Lambda and ECS probe worker CPU free for the actual probe HTTP requests. ZSTD is worth evaluating if broker storage cost is the binding constraint and producer CPU is not.

The compression codec is a producer-side config; the broker stores messages in compressed form and decompresses only when needed (consumer fetch). Setting a different codec on the broker-side (compression.type=producer, the default) tells the broker to respect whatever the producer chose — do not set a specific codec on the broker unless you want to override producers.

batch.size and linger.ms: throughput tuning

The producer accumulates messages into a batch before sending to the broker. Two configs control when the batch is flushed:

• batch.size (default 16384 bytes): maximum batch size. When the batch reaches this size, it is sent immediately regardless of linger.ms.
• linger.ms (default 0): time to wait before sending an incomplete batch. With the default of 0, the producer sends every message as soon as it arrives — never accumulating a batch unless the producer is so fast that the batch fills before the network layer picks it up.

At 2,000 probe completions/second with 2 KB average message size, the default settings generate 2,000 Kafka requests/second to the broker. With batch.size=65536 and linger.ms=10, the same volume generates roughly 60 requests/second (each containing ~32 messages). Fewer, larger requests reduce broker CPU for request handling and SASL/TLS connection overhead.

// Node.js kafkajs producer config for high-throughput probe results
const producer = kafka.producer({
  // Idempotence requires acks=-1, retries>0, max.in.flight.per.connection<=5
  idempotent: true,             // sets acks=-1 and retries=Integer.MAX_VALUE automatically
  transactionTimeout: 30000,
  // kafkajs-specific batching equivalent
  // (kafkajs does not expose batch.size/linger.ms directly;
  //  use sendBatch() for explicit control, or a bridge via node-rdkafka)
});

// For fine-grained producer config, use node-rdkafka (librdkafka bindings):
// librdkafka exposes Kafka's full producer config namespace
const rdkafka = require('node-rdkafka');

const highThroughputProducer = new rdkafka.Producer({
  'metadata.broker.list': bootstrapBrokers,
  'security.protocol': 'SASL_SSL',
  'sasl.mechanisms': 'OAUTHBEARER',
  // credential refresh callback for MSK IAM (see MSK IAM auth guide)
  'acks': '-1',              // acks=all
  'compression.codec': 'lz4',
  'batch.size': 65536,
  'linger.ms': 10,
  'enable.idempotence': 'true',
  'max.in.flight.requests.per.connection': '5',
  'retries': '2147483647',   // INT_MAX
  'message.send.max.retries': '2147483647',
});

Idempotent producer: exactly-once send semantics

With retries enabled (retries>0), a network timeout between the producer and broker can cause a message to be written to the broker and acknowledged but the ACK to be lost — the producer retries and the broker receives a duplicate. The idempotent producer prevents this: each producer instance is assigned a producer ID (PID) and the broker tracks a sequence number per PID per partition. Duplicate sends (same PID, same sequence number) are silently deduplicated at the broker.

Enable idempotence with enable.idempotence=true (librdkafka) or idempotent: true (kafkajs). This automatically enforces:

• acks=all (cannot be changed when idempotence is on)
• retries=Integer.MAX_VALUE
• max.in.flight.requests.per.connection=5 (the maximum that preserves ordering guarantees)

Idempotence is per-session (producer instance). If the producer process restarts, the PID changes and the deduplication window resets — a message in-flight at crash time may be re-sent after restart and the broker will not deduplicate it. For cross-session exactly-once semantics, you need the transactional producer (transactional.id set). For probe result ingestion, idempotent-but-not-transactional is almost always the right choice: probe results are idempotent at the consumer side (upsert by probe run ID), so the rare duplicate on producer restart is harmless.

SASL_SSL security protocol configuration

MSK on provisioned clusters supports multiple authentication modes; IAM (SASL_SSL with AWS_MSK_IAM mechanism) is the recommended mode. The full producer security config for MSK IAM on librdkafka:

const mskProducerConfig = {
  'metadata.broker.list': bootstrapBrokers,    // port 9098 for SASL_SSL
  'security.protocol': 'SASL_SSL',
  'sasl.mechanisms': 'OAUTHBEARER',
  // MSK IAM uses OAUTHBEARER with a token provider that calls STS
  'oauthbearer_token_refresh_cb': async (kafkaHandle, oauthConfig) => {
    const provider = new MSKAuthProvider({ region: 'us-east-1' });
    const token = await provider.generateAuthToken();
    kafkaHandle.setOauthBearerToken(token, /* expiry */ Date.now() + 900000);
  },
  'ssl.ca.location': '/etc/ssl/certs/ca-certificates.crt',
  // TLS is mandatory for MSK; do NOT set ssl.endpoint.identification.algorithm=none
};

Port 9098 is the MSK port for SASL_SSL (IAM auth). Port 9092 is plaintext (no auth — disabled by default on provisioned clusters), 9094 is TLS with client certificate auth (mTLS). Always use port 9098 for IAM-authenticated clients.

CloudWatch producer metrics

Common failure modes