AWS MSK for MCP Servers

MSK provisioned vs MSK Serverless

MSK offers two deployment models with different trade-offs for MCP probe workloads:

For AliveMCP monitoring 5,000 endpoints at 60-second intervals, the steady write rate is ~83 probe results per second. MSK Serverless handles this comfortably. At 50,000 endpoints (833 writes/second), provisioned MSK with kafka.m5.large brokers (3 brokers in 3 AZs) is cheaper. The crossover is typically around 10,000+ consistently monitored endpoints.

MSK Serverless partition limits

MSK Serverless limits each topic to 120 partitions and the cluster to 120 Kafka units (RUs). One RU roughly equals 1 MB/s write + 2 MB/s read. For a probe-events topic receiving 1 MB/s of probe payloads, you consume one RU and can use up to 120 partitions. The partition count determines consumer parallelism — 120 partitions means up to 120 Lambda probe-result processors running in parallel.

Kafka topic design for MCP probe events

Topic design determines throughput, ordering, and consumer parallelism. For MCP probe pipelines, the key decisions are partition count, message key, and retention policy.

Topic: mcp-probe-results

The central topic receives one message per completed probe. The message key is the MCP server slug. Kafka routes messages with the same key to the same partition, so all probes for one server arrive in order at one consumer — enabling stateful transition detection (was down, now up) without shared state.

// Topic structure
// Key:   Buffer.from(serverSlug)                 → consistent partition assignment per server
// Value: JSON probe result payload
// Retention: 7 days (probe SLA report window)

// CDK topic via MSK Construct Library (L2 available in CDK v2 for Serverless)
import { aws_msk as msk } from 'aws-cdk-lib';

const serverlessCluster = new msk.CfnServerlessCluster(this, 'McpProbeMsk', {
  clusterName: 'mcp-probe-cluster',
  vpcConfigs: [{
    subnetIds: vpc.selectSubnets({ subnetType: ec2.SubnetType.PRIVATE_WITH_EGRESS }).subnetIds,
    securityGroups: [mskSg.securityGroupId],
  }],
  clientAuthentication: {
    sasl: { iam: { enabled: true } },  // IAM-only for Serverless
  },
});

// Topics created via Kafka AdminClient (MSK API doesn't create topics directly)
// Run once at deploy time from a custom resource Lambda
const adminConfig = {
  clientId: 'topic-setup',
  brokers: [clusterBootstrapBroker],
  ssl: true,
  sasl: {
    mechanism: 'oauthbearer',
    oauthBearerProvider: async () => ({ value: await getIamToken() }),
  },
};

const admin = kafka.admin(adminConfig);
await admin.connect();
await admin.createTopics({
  topics: [
    {
      topic: 'mcp-probe-results',
      numPartitions: 60,            // 60 partitions = 60 max parallel consumers
      replicationFactor: 3,         // 3 = survive loss of 1 AZ
      configEntries: [
        { name: 'retention.ms',     value: String(7 * 24 * 60 * 60 * 1000) },  // 7 days
        { name: 'compression.type', value: 'lz4' },   // probe payloads compress well
        { name: 'min.insync.replicas', value: '2' },  // require 2 of 3 AZs for write ack
      ],
    },
    {
      topic: 'mcp-status-changes',  // filtered topic: only up↔down transitions
      numPartitions: 10,
      replicationFactor: 3,
      configEntries: [{ name: 'retention.ms', value: String(30 * 24 * 60 * 60 * 1000) }],
    },
  ],
});

Message key strategy and partition assignment

Using the server slug as the message key guarantees that a given consumer always sees all probes for a given server in sequence. This is the prerequisite for stateful transition detection: the consumer maintains in-memory state of each server's last known status, and a new probe result with a different status triggers an alert. Without key-based routing, two consecutive probes for the same server might land on different consumers, each missing the other's state.

For a 60-partition topic with 2,000 servers: each partition handles ~33 servers on average. The consumer's in-memory state map has ~33 entries per partition, making state management trivial. Increase partitions when consumer processing falls behind (partition lag growing); the partition count sets the hard ceiling on parallelism.

CDK cluster setup — MSK Provisioned

For workloads that outgrow MSK Serverless, provisioned MSK with kafka.m5.large brokers (3 brokers across 3 AZs) handles up to ~300 MB/s aggregate throughput:

import { aws_msk as msk } from 'aws-cdk-lib';

const mskCluster = new msk.Cluster(this, 'McpProbeMskCluster', {
  clusterName: 'mcp-probe-cluster',
  kafkaVersion: msk.KafkaVersion.V3_6_0,
  numberOfBrokerNodes: 3,                  // 1 per AZ
  instanceType: ec2.InstanceType.of(
    ec2.InstanceClass.M5,
    ec2.InstanceSize.LARGE
  ),
  ebs: { volumeSize: 1000 },              // 1 TB per broker = 3 TB total
  vpc,
  vpcSubnets: { subnetType: ec2.SubnetType.PRIVATE_WITH_EGRESS },
  securityGroups: [mskSg],

  // Authentication: prefer IAM (no secret rotation needed)
  clientAuthentication: msk.ClientAuthentication.sasl({
    iam: true,
    scram: true,   // also enable SCRAM for non-AWS clients (dev tools)
  }),

  encryptionInTransit: {
    clientBroker: msk.ClientBrokerEncryption.TLS,  // TLS required
    enableInCluster: true,
  },

  // Monitoring: enable enhanced monitoring for partition-level metrics
  monitoring: {
    clusterMonitoringLevel: msk.ClusterMonitoringLevel.PER_TOPIC_PER_PARTITION,
    enablePrometheusJmxExporter: true,
    enablePrometheusNodeExporter: true,
  },

  logging: {
    s3: { enabled: true, bucket: logsBucket, prefix: 'msk/broker-logs/' },
    cloudwatch: { enabled: true, logGroup: mskLogGroup },
  },
});

The PER_TOPIC_PER_PARTITION monitoring level enables CloudWatch metrics for individual partition lag — critical for detecting when one probe worker is falling behind while others are healthy. Without per-partition metrics, a single lagging partition hides in aggregate consumer-group lag.

Producing probe results to MSK

Probe workers publish results after each probe run. KafkaJS with IAM authentication using the aws-msk-iam-sasl-signer-js package is the standard approach for Node.js probe workers:

import { Kafka } from 'kafkajs';
import { generateAuthToken } from 'aws-msk-iam-sasl-signer-js';

const kafka = new Kafka({
  clientId: `probe-worker-${process.env.WORKER_ID}`,
  brokers: process.env.MSK_BOOTSTRAP_BROKERS!.split(','),
  ssl: true,
  sasl: {
    mechanism: 'oauthbearer',
    oauthBearerProvider: async () => {
      const { token, expiryTime } = await generateAuthToken({ region: 'us-east-1' });
      return { value: token, extensions: { expiryMs: String(expiryTime) } };
    },
  },
});

const producer = kafka.producer({
  idempotent: true,             // exactly-once delivery; requires acks: -1 (all)
  transactionTimeout: 30_000,
});

await producer.connect();

// After each probe run, publish the result
await producer.send({
  topic: 'mcp-probe-results',
  messages: [{
    key:   serverSlug,          // consistent partition assignment per server
    value: JSON.stringify({
      serverSlug,
      status:         verdict,       // 'up' | 'degraded' | 'down'
      responseTimeMs: responseTimeMs,
      checkedAt:      new Date().toISOString(),
      probeWorkerId:  process.env.WORKER_ID,
    }),
    headers: { 'content-type': 'application/json' },
  }],
  acks: -1,                     // wait for all in-sync replicas (min.insync.replicas = 2)
});

The idempotent: true producer setting prevents duplicate probe results if the producer retries a failed send. Without idempotency, a network timeout on the broker ACK causes the producer to resend, and if the original write succeeded, the duplicate inflates probe counts. With idempotency, Kafka deduplicates via producer sequence numbers on the broker side.

How AliveMCP monitors MSK clusters for MCP teams

An MSK cluster serving as the probe event backbone introduces a new failure mode: the cluster is up but broker leader election is in progress, causing producer hangs and consumer lag spikes. AliveMCP monitors MSK clusters at three levels:

• Produce-path canary: every 60 seconds, AliveMCP publishes a canary message to a designated alivemcp-canary topic. A timeout or error on the canary produce flags the cluster as degraded — even if the MSK service status shows green and no CloudWatch alarms have fired.
• Consumer lag: AliveMCP reads the kafka.consumer_group.offset_lag CloudWatch metric per consumer group per partition. When any partition's lag exceeds 60 seconds of probe throughput (configurable), AliveMCP alerts — the probe pipeline is falling behind and alert delivery will be delayed.
• Under-replicated partitions: the KafkaDataLogs/UnderReplicatedPartitions metric going non-zero indicates a broker is lagging replication — a precursor to potential data loss if a broker fails before it catches up. AliveMCP pages immediately on any under-replicated partition in a production cluster.

MSK monitoring is available on the Team tier. AliveMCP generates a unified health view: MSK broker health, consumer group lag, and MCP endpoint probe results in one dashboard, so the MCP team can distinguish "probe pipeline degraded" from "MCP endpoint degraded."

Related guides

• MSK IAM Authentication for MCP Probe Workers
• MSK Consumer Groups for Parallel Probe Processing
• MSK Connect for Probe Archive and State Sync
• Kafka MCP Server Tools — KafkaJS Producer and Consumer