MSK IAM Authentication for MCP Servers

SASL/IAM mechanism: how it works

MSK's IAM authentication uses the Kafka SASL/OAUTHBEARER extension with a custom token format. The client calls the MSK IAM signing library to generate a short-lived signed token from the EC2/Lambda instance's IAM credentials (via IMDS or the Lambda execution role). The token is sent to the broker during SASL handshake. The broker verifies the signature against IAM and checks the principal's IAM policy for the requested operation (Produce, Consume, CreateTopic, etc.).

Tokens expire after approximately 15 minutes. The KafkaJS client must call the oauthBearerProvider function to refresh the token before expiry. The aws-msk-iam-sasl-signer-js package handles signing and returns the expiry time so KafkaJS schedules the refresh automatically.

import { Kafka, logLevel } from 'kafkajs';
import { generateAuthToken } from 'aws-msk-iam-sasl-signer-js';

// The oauthBearerProvider is called on connection and before every token expiry.
// Returning { value: token, extensions: { expiryMs } } tells KafkaJS when to refresh.
async function mskIamProvider() {
  const region = process.env.AWS_REGION ?? 'us-east-1';
  const { token, expiryTime } = await generateAuthToken({ region });
  return {
    value: token,
    extensions: {
      expiryMs: String(expiryTime),   // Unix ms timestamp — KafkaJS refreshes before this
    },
  };
}

export const kafka = new Kafka({
  clientId: `mcp-probe-worker-${process.env.WORKER_ID}`,
  brokers:   process.env.MSK_BOOTSTRAP_BROKERS!.split(','),
  ssl: true,                              // TLS always required with IAM auth
  sasl: {
    mechanism: 'oauthbearer',
    oauthBearerProvider: mskIamProvider,  // called on connect + before token expiry
  },
  logLevel:  logLevel.WARN,
  retry: {
    retries: 5,
    initialRetryTime: 300,
    factor: 2,                            // exponential backoff on transient auth failures
  },
});

A common mistake: omitting ssl: true when using SASL/IAM. The IAM auth handshake fails silently or with a generic connection reset error if TLS is not enabled. MSK requires TLS on port 9098 (SASL/IAM) and 9096 (SASL/SCRAM) — port 9092 is plaintext and does not support authentication on production clusters.

MSK bootstrap broker ports

Retrieve the correct bootstrap endpoint for each port type from CDK via the MSK Cluster construct:

// CDK: retrieve IAM bootstrap broker endpoint
const iamBrokers = cluster.bootstrapBrokersSaslIam;     // port 9098
const scramBrokers = cluster.bootstrapBrokersSaslScram;  // port 9096
const tlsBrokers = cluster.bootstrapBrokersTls;          // port 9094

new cdk.CfnOutput(this, 'IamBrokers', { value: iamBrokers });

IAM policies for topic-level access control

MSK IAM authorization operates at the topic and consumer-group level. The IAM policy uses the MSK Kafka resource ARN format: arn:aws:kafka:{region}:{account}:topic/{cluster-name}/{cluster-uuid}/{topic-name}. MSK resolves the kafka-cluster:* actions against these ARNs during the authentication handshake.

Least-privilege producer role (probe worker)

// Probe worker: can produce to mcp-probe-results only
probeWorkerRole.addToPrincipalPolicy(new iam.PolicyStatement({
  effect: iam.Effect.ALLOW,
  actions: [
    'kafka-cluster:Connect',                  // connect to cluster
    'kafka-cluster:WriteData',                // produce to topic
    'kafka-cluster:DescribeTopic',            // metadata fetch (required for leader discovery)
  ],
  resources: [
    cluster.clusterArn,
    `arn:aws:kafka:${region}:${account}:topic/${clusterName}/${clusterId}/mcp-probe-results`,
  ],
}));

Least-privilege consumer role (alert processor)

// Alert processor: can consume from mcp-probe-results and mcp-status-changes
alertProcessorRole.addToPrincipalPolicy(new iam.PolicyStatement({
  effect: iam.Effect.ALLOW,
  actions: [
    'kafka-cluster:Connect',
    'kafka-cluster:ReadData',                 // consume from topic
    'kafka-cluster:DescribeTopic',
    'kafka-cluster:DescribeGroup',            // consumer group metadata
    'kafka-cluster:AlterGroup',               // commit offsets
  ],
  resources: [
    cluster.clusterArn,
    `arn:aws:kafka:${region}:${account}:topic/${clusterName}/${clusterId}/mcp-probe-results`,
    `arn:aws:kafka:${region}:${account}:topic/${clusterName}/${clusterId}/mcp-status-changes`,
    `arn:aws:kafka:${region}:${account}:group/${clusterName}/${clusterId}/alert-processor-group`,
  ],
}));

The kafka-cluster:AlterGroup action is required for offset commits. Without it, consumers can read messages but cannot commit their position — every restart re-reads from the beginning of the topic. This is the most common IAM misconfiguration for MSK consumers: the role has ReadData but not AlterGroup, causing duplicate alert delivery after every consumer restart.

SASL/SCRAM for non-AWS clients

When MCP team members need to inspect the probe topic with desktop tools (kcat, Kafka UI, Conduktor), IAM auth is impractical — these tools don't have an IAM role. SASL/SCRAM with credentials stored in AWS Secrets Manager provides a user/password flow that these tools support natively.

// CDK: create SCRAM user in Secrets Manager and associate with MSK
const scramSecret = new secretsmanager.Secret(this, 'McpMskScramSecret', {
  secretName: 'AmazonMSK_mcp-probe-cluster_devuser',  // MSK requires this prefix
  generateSecretString: {
    secretStringTemplate: JSON.stringify({ username: 'devuser' }),
    generateStringKey: 'password',
    excludePunctuation: true,
    passwordLength: 32,
  },
});

// Associate the secret with the MSK cluster
const cfnCluster = cluster.node.defaultChild as msk.CfnCluster;
// Use MSK BatchAssociateScramSecret API (via custom resource) to link the secret
// MSK Provisioned only — Serverless does not support SCRAM, only IAM

SCRAM credentials rotate via Secrets Manager rotation (configure a 30-day rotation window). After rotation, MSK automatically picks up the new password — no cluster restart required. The old password remains valid for 5 minutes after rotation to allow in-flight connections to drain.

Authentication failure modes

Token refresh lifecycle in long-running MCP probe workers

Probe workers are long-running processes that produce messages every 60 seconds indefinitely. The IAM token expires every ~15 minutes. KafkaJS calls the oauthBearerProvider before each token expiry — but only when a produce or consume operation is in flight or the connection is active. If the probe worker is idle for more than 15 minutes (e.g., all monitored endpoints are paused), the token expires and the next produce attempt triggers re-authentication.

Re-authentication generates a brief connection reset that the KafkaJS retry logic handles automatically. However, ensure your probe worker does not suppress KafkaJS WARN logs during this window — the re-auth cycle emits WARN level logs that confirm the token refresh succeeded. Silent failures during re-auth are harder to diagnose.

// Probe worker keepalive: send a heartbeat canary message every 10 minutes
// to keep the connection active and prevent token expiry during quiet periods
setInterval(async () => {
  await producer.send({
    topic: 'mcp-probe-results',
    messages: [{
      key:   '__heartbeat__',
      value: JSON.stringify({ type: 'heartbeat', workerId: process.env.WORKER_ID, ts: Date.now() }),
    }],
  });
}, 10 * 60 * 1000);

How AliveMCP monitors MSK IAM auth health

AliveMCP tracks authentication health as a first-class signal for MSK-backed MCP probe pipelines. An IAM policy misconfiguration — such as a missing AlterGroup action after a role rotation, or a CloudFormation rollback that reverted the IAM policy — silently stops offset commits without causing the consumer to crash. Probes continue running, but alerts stop firing because the consumer re-reads old messages on every restart.

AliveMCP detects this via consumer lag correlation: if consumer group offset lag is growing despite the consumer reporting active reads, the most likely cause is a missing AlterGroup permission. AliveMCP surfaces this as a distinct alert type ("consumer offset commit failure") rather than the generic "consumer lag" alert, so the MCP team knows exactly which IAM action to check.

Related guides

• AWS MSK Overview for MCP Servers
• MSK Consumer Groups for Parallel Probe Processing
• MSK Connect for Probe Archive and State Sync