MSK Replication for MCP Servers — Cross-Region DR, Replication Factor, and Producer Acks

Replication factor and in-sync replicas

The replication factor (RF) is the number of broker copies for each partition. For a 3-broker cluster, RF=3 means all three brokers hold a copy of every partition. The in-sync replica set (ISR) is the subset of replicas that are caught up to within the replica lag threshold — by default, any replica that is more than 30 seconds behind the leader is removed from the ISR.

min.insync.replicas (ISR) controls the minimum number of ISR members required before a producer write with acks=all is acknowledged. With RF=3 and min.insync.replicas=2:

• The cluster can tolerate one broker failure (ISR drops from 3 to 2, still meets the minimum of 2)
• If a second broker fails, the ISR drops to 1, which is below min.insync.replicas=2 — the topic becomes unavailable for acks=all producers (they receive NotEnoughReplicasException)
• Consumers can still read from the leader even when ISR < min.insync.replicas

Producer acks: three settings compared

The acks producer config controls which acknowledgement the broker sends back before the producer considers a send complete:

The interaction with retries: with acks=all, a producer that gets a NotEnoughReplicasException (ISR < min.insync.replicas) will retry if retries > 0. Set retries=Integer.MAX_VALUE or a high value with delivery.timeout.ms to bound the total retry window. For probe result ingestion, a 30-second delivery timeout aligns with the 60-second probe cycle — if the write hasn't succeeded in 30 seconds, the result is dropped and the next probe will re-verify the endpoint status.

// CDK: MSK cluster with RF=3 and min.insync.replicas=2
import { aws_msk as msk } from 'aws-cdk-lib';

const clusterConfig = new msk.ClusterConfigurationProperty({
  // MSK configuration resource (applied to cluster)
  // Set default topic-level min.insync.replicas
  serverProperties: [
    'min.insync.replicas=2',
    'default.replication.factor=3',
    'auto.create.topics.enable=false',  // prevent accidental topic creation
  ].join('\n'),
});

const cluster = new msk.CfnCluster(this, 'ProbeCluster', {
  clusterName: 'alivemcp-probe-us-east-1',
  kafkaVersion: '3.6.0',
  numberOfBrokerNodes: 3,
  brokerNodeGroupInfo: { /* ... */ },
  configurationInfo: {
    arn: clusterConfig.ref,
    revision: 1,
  },
  encryptionInfo: {
    encryptionInTransit: { clientBroker: 'TLS', inCluster: true },
  },
});

MSK Replication: cross-region active-passive setup

MSK Replication copies topics from a source cluster to a destination cluster in a different region. The destination cluster is read-only during normal operations — probe result consumers are directed to the destination cluster only during failover. To configure MSK Replication:

1. Create a destination MSK cluster in the DR region (eu-west-1 for us-east-1 primary). It does not need the same broker count — a smaller cluster is fine for standby.
2. Create an MSK Replicator resource in the source region specifying source cluster ARN, destination cluster ARN, and topic patterns.
3. Grant the MSK Replication service IAM permissions to read from the source and write to the destination cluster.

// CloudFormation / CDK L1 for MSK Replication
// (No CDK L2 construct as of late 2026 — use CfnReplicator)
import { aws_msk as msk } from 'aws-cdk-lib';

const replicator = new msk.CfnReplicator(this, 'ProbeResultReplicator', {
  replicatorName: 'probe-results-us-to-eu',
  description: 'Replicate probe result topics to EU DR cluster',
  serviceExecutionRoleArn: replicationRole.roleArn,
  kafkaClusters: [
    {
      amazonMskCluster: { mskClusterArn: sourceClusterArn },
      vpcConfig: { subnetIds: sourceSubnetIds, securityGroupIds: [sourceSgId] },
    },
    {
      amazonMskCluster: { mskClusterArn: destClusterArn },
      vpcConfig: { subnetIds: destSubnetIds, securityGroupIds: [destSgId] },
    },
  ],
  replicationInfoList: [{
    sourceKafkaClusterArn: sourceClusterArn,
    targetKafkaClusterArn: destClusterArn,
    targetCompressionType: 'SNAPPY',
    topicReplication: {
      topicsToReplicate: ['probe-results', 'alert-dispatch', 'audit-events'],
      detectAndCopyNewTopics: true,
      copyTopicConfigurations: true,
    },
    consumerGroupReplication: {
      consumerGroupsToReplicate: ['probe-archiver', 'alert-worker'],
      detectAndCopyNewConsumerGroups: true,
      synchroniseConsumerGroupOffsets: true,
    },
  }],
});

With synchroniseConsumerGroupOffsets: true, MSK Replication translates source-cluster offsets to destination-cluster offsets and writes them to the __consumer_offsets topic on the destination. On failover, a consumer pointed at the destination cluster resumes from the translated offset — you do not need to manually reset offsets.

Monitoring replication lag

MSK Replication emits the ReplicationByteLag CloudWatch metric (bytes) per topic partition. A lag value that grows over time indicates the destination is falling behind — the DR cluster would not be a safe failover target until it catches up.

// CDK alarm: alert if replication lag exceeds 50 MB on any partition
import { aws_cloudwatch as cloudwatch, aws_cloudwatch_actions as cw_actions } from 'aws-cdk-lib';

new cloudwatch.Alarm(this, 'ReplicationLagAlarm', {
  alarmName: 'msk-probe-replication-lag',
  metric: new cloudwatch.Metric({
    namespace: 'AWS/MSK/Replicator',
    metricName: 'ReplicationByteLag',
    dimensionsMap: {
      ReplicatorName: 'probe-results-us-to-eu',
    },
    statistic: 'Maximum',
    period: cdk.Duration.minutes(1),
  }),
  threshold: 50 * 1024 * 1024,  // 50 MB
  evaluationPeriods: 3,          // sustained for 3 minutes
  comparisonOperator: cloudwatch.ComparisonOperator.GREATER_THAN_THRESHOLD,
  alarmDescription: 'MSK DR replication falling behind — failover would replay this lag on resume',
});

Common failure modes