MSK Serverless for MCP Servers — Capacity Model, IAM-Only Auth, and Probe Pipeline Trade-offs

Capacity model: how throughput scales without partition management

On a provisioned MSK cluster, throughput is bounded by (broker count) × (per-broker throughput ceiling). You plan ahead: N brokers × M topics × P partitions. MSK Serverless replaces that model with a single per-cluster soft quota:

For AliveMCP, the probe pipeline produces one Kafka message per probe completion — typically 1–4 KB of JSON containing endpoint ID, HTTP status, response time, and tool-schema hash. At 2,000 probe completions per second that is roughly 8 MiBps ingress, well within the 200 MiBps quota. The 120-partition soft limit is the binding constraint for scale-out: if the probe worker fleet grows beyond ~120 concurrent consumer threads and each needs its own partition for ordered processing, you will need provisioned MSK.

IAM-only authentication: required configuration

MSK Serverless enforces SASL/IAM for all clients. There is no option to enable unauthenticated, SCRAM, or mTLS access. Every producer and consumer needs:

1. A boto3 or AWS SDK credential chain (instance profile, ECS task role, Lambda execution role, or explicit access key)
2. The aws-msk-iam-auth library (Java) or a compatible SASL/IAM plugin for the client language
3. An IAM policy granting kafka-cluster:Connect, kafka-cluster:DescribeGroup, kafka-cluster:AlterGroup, kafka:DescribeTopic, kafka-cluster:ReadData, and/or kafka-cluster:WriteData on the cluster and topic ARNs

# Node.js producer with MSK IAM auth (using kafkajs + msk-iam-auth-library)
import { Kafka } from 'kafkajs';
import { MSKAuthProvider } from 'aws-msk-iam-sasl-signer-js';

const kafka = new Kafka({
  clientId: 'alivemcp-probe-producer',
  brokers: ['b-1.cluster.msk.us-east-1.amazonaws.com:9098'],
  ssl: true,
  sasl: {
    mechanism: 'oauthbearer',
    oauthBearerProvider: async () => {
      const provider = new MSKAuthProvider({ region: 'us-east-1' });
      const token = await provider.generateAuthToken();
      return { value: token };
    },
  },
});

The IAM token has a 15-minute expiry. The MSKAuthProvider re-fetches a token before each connection and on token expiry — your probe worker does not need a refresh loop. On Lambda, the execution role is the IAM principal; the SDK credential chain resolves it automatically.

Minimum IAM policy for a probe result producer

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "kafka-cluster:Connect",
        "kafka-cluster:DescribeTopic",
        "kafka-cluster:WriteData"
      ],
      "Resource": [
        "arn:aws:kafka:us-east-1:123456789012:cluster/alivemcp-probe-results/*",
        "arn:aws:kafka:us-east-1:123456789012:topic/alivemcp-probe-results/*/probe-results"
      ]
    }
  ]
}

Tiered storage: default on Serverless

MSK Serverless uses tiered storage by default. Data is written to local broker SSD first, then offloaded to S3-backed tiered storage. There is no fixed local storage limit — the cluster will not fill a disk. Retention is controlled at the cluster level via the log.retention.ms default (7 days). You cannot override retention.ms per-topic on MSK Serverless — all topics share the cluster-level retention.

For AliveMCP probe result topics, 7-day retention covers the DLQ replay window (probe results are archived to S3 via a separate archiver consumer within 1 hour; the Kafka retention is an emergency replay buffer). If you need per-topic retention tuning — for example, a 1-day topic for ephemeral probe commands and a 30-day topic for historical raw results — you need provisioned MSK with topic-level configs.

What MSK Serverless does NOT support

CDK provisioning for MSK Serverless

MSK Serverless uses the L1 construct CfnServerlessCluster — there is no L2 wrapper yet. The cluster creation includes an inline VPC config and security group, and a client authentication block that enforces IAM.

import * as cdk from 'aws-cdk-lib';
import { aws_msk as msk, aws_ec2 as ec2 } from 'aws-cdk-lib';

const vpc = ec2.Vpc.fromLookup(this, 'Vpc', { isDefault: false });

const mskSg = new ec2.SecurityGroup(this, 'MskSg', {
  vpc,
  description: 'MSK Serverless probe results',
  allowAllOutbound: true,
});

// Allow probe workers (in their own SG) to reach MSK on port 9098
mskSg.addIngressRule(probeWorkerSg, ec2.Port.tcp(9098), 'Kafka SASL/TLS');

const serverlessCluster = new msk.CfnServerlessCluster(this, 'ProbeCluster', {
  clusterName: 'alivemcp-probe-results',
  vpcConfigs: [{
    subnetIds: vpc.privateSubnets.map(s => s.subnetId),
    securityGroups: [mskSg.securityGroupId],
  }],
  clientAuthentication: {
    sasl: { iam: { enabled: true } },
  },
});

new cdk.CfnOutput(this, 'ClusterArn', { value: serverlessCluster.attrArn });

CloudWatch metrics for MSK Serverless

MSK Serverless emits a subset of the metrics available on provisioned clusters. Key metrics for probe pipeline monitoring:

Note: UnderReplicatedPartitions is not meaningful for MSK Serverless (replication is managed by AWS). Do not build alarms on it for serverless clusters — it will always read 0.

Common failure modes