DynamoDB Global Tables Replication Monitoring for MCP Servers — ReplicationLatency, Conflict Resolution

Global Tables v2 architecture and consistency model

Global Tables v2 (the current version, replacing v1's independent stream-replication model) uses DynamoDB's internal replication infrastructure. Each region holds a full copy of the table. Writes to any region replica propagate to all other replicas asynchronously through the DynamoDB replication layer — not through an application-managed stream pipeline.

The most critical operational constraint: conditional writes are not globally atomic. If your probe collector in us-east-1 runs UpdateItem with ConditionExpression: "attribute_not_exists(PK)" and your probe collector in eu-west-1 runs the same expression for the same item within the replication window, both can succeed. Both writes will be applied to their local replica and replicated; one will survive as the final state. For MCP server registration (creating the canonical server record), this means two concurrent registrations may both succeed locally — use application-level deduplication (check for existing records after a write, then delete the duplicate) rather than relying on conditional write atomicity across regions.

ReplicationLatency metric and alarm setup

DynamoDB publishes ReplicationLatency to CloudWatch in the AWS/DynamoDB namespace. The metric reports the time (in milliseconds) for a write in the source region to be replicated to each destination region. There is one metric stream per source-destination region pair.

// CDK: CloudWatch alarm for Global Tables replication latency
import * as cloudwatch from 'aws-cdk-lib/aws-cloudwatch';

// Alarm: replication from us-east-1 to eu-west-1 exceeding 30 seconds
const replicationLatencyAlarm = new cloudwatch.Alarm(this, 'GlobalTablesReplicationLag', {
  metric: new cloudwatch.Metric({
    namespace: 'AWS/DynamoDB',
    metricName: 'ReplicationLatency',
    dimensionsMap: {
      TableName: probeTable.tableName,
      ReceivingRegion: 'eu-west-1'  // One alarm per destination region
    },
    statistic: 'p99',
    period: cdk.Duration.minutes(1),
    region: 'us-east-1'  // Metric is published in the source region
  }),
  threshold: 30000,  // 30 seconds in milliseconds
  evaluationPeriods: 3,
  datapointsToAlarm: 2,
  comparisonOperator: cloudwatch.ComparisonOperator.GREATER_THAN_THRESHOLD,
  treatMissingData: cloudwatch.TreatMissingData.NOT_BREACHING,
  alarmDescription: 'Global Tables replication from us-east-1 to eu-west-1 exceeds 30s p99'
});

// Also alarm on maximum latency for tail-latency outliers
const replicationMaxAlarm = new cloudwatch.Alarm(this, 'GlobalTablesReplicationMaxLag', {
  metric: new cloudwatch.Metric({
    namespace: 'AWS/DynamoDB',
    metricName: 'ReplicationLatency',
    dimensionsMap: { TableName: probeTable.tableName, ReceivingRegion: 'eu-west-1' },
    statistic: 'Maximum',
    period: cdk.Duration.minutes(5),
    region: 'us-east-1'
  }),
  threshold: 60000,  // 60 seconds maximum — indicates replication stall
  evaluationPeriods: 1,
  comparisonOperator: cloudwatch.ComparisonOperator.GREATER_THAN_THRESHOLD
});

Create one alarm per source-destination region pair. For a 3-region setup (us-east-1, eu-west-1, ap-southeast-1), you need 6 alarms (3 sources × 2 destinations each). The metric is published in the source region's CloudWatch, so use cross-region CloudWatch metric access or a global dashboard aggregating metrics from all regions.

Conflict resolution: last-writer-wins and aws:rep:updatetime

When two replicas receive conflicting writes to the same item, Global Tables resolves the conflict by keeping the write with the higher aws:rep:updatetime value — a microsecond-precision timestamp that DynamoDB assigns to every write at the time it's applied to the local replica. The losing write is silently discarded; no error is returned to either writer.

// aws:rep:updatetime is NOT a user-visible attribute in most SDKs
// It's visible in DynamoDB Streams NEW_AND_OLD_IMAGES records:
{
  "eventName": "MODIFY",
  "dynamodb": {
    "NewImage": {
      "PK": { "S": "SERVER#api.example.com" },
      "status": { "S": "HEALTHY" },
      "aws:rep:updatetime": { "N": "1735600000.123456" },  // Microsecond Unix timestamp
      "aws:rep:deleting": { "BOOL": false }
    }
  }
}

// You CAN read aws:rep:updatetime in DynamoDB Streams records
// to detect which write won a conflict — compare aws:rep:updatetime
// in OldImage vs NewImage to see if replication overwrote a local write

// Example: Lambda on DynamoDB Stream detecting a replication overwrite
export const handler = async (event) => {
  for (const record of event.Records) {
    if (record.eventName !== 'MODIFY') continue;
    const oldTime = parseFloat(record.dynamodb.OldImage?.['aws:rep:updatetime']?.N ?? '0');
    const newTime = parseFloat(record.dynamodb.NewImage?.['aws:rep:updatetime']?.N ?? '0');
    // If new timestamp is from the far future relative to old, this may be a replication overwrite
    // from a different region (not a local write)
    if (newTime - oldTime > 5.0) {  // >5 second jump suggests cross-region replication win
      console.log('Possible replication overwrite detected for', record.dynamodb.Keys.PK.S);
    }
  }
};

For MCP probe state, write conflicts arise when the same MCP server is monitored by probes running in two regions simultaneously. If probe A in us-east-1 writes status=HEALTHY at the same microsecond that probe B in eu-west-1 writes status=DEGRADED for the same server, one status silently wins. The winning status may not reflect the true endpoint health as seen from either region.

Write routing strategy to minimize conflicts

The most effective way to prevent conflicts in a multi-region probe deployment is to assign each MCP server slug to a single "home region" that owns writes for that server's status record. Other regions can read from their local replica for alerting but route status writes to the home region.

// Home region routing: deterministic assignment of server slug to write region
function getHomeRegion(serverSlug: string): string {
  const regions = ['us-east-1', 'eu-west-1', 'ap-southeast-1'];
  // FNV-1a hash of the slug, modulo region count
  let hash = 2166136261;  // FNV offset basis
  for (let i = 0; i < serverSlug.length; i++) {
    hash ^= serverSlug.charCodeAt(i);
    hash = (hash * 16777619) >>> 0;  // FNV prime, keep 32-bit
  }
  return regions[hash % regions.length];
}

// In each region's probe runner:
async function writeProbeResult(slug: string, status: string, latencyMs: number) {
  const homeRegion = getHomeRegion(slug);
  const currentRegion = process.env.AWS_REGION!;

  if (homeRegion === currentRegion) {
    // Local write — fast, strongly consistent
    await dynamodb.updateItem({
      TableName: TABLE_NAME,
      Key: { PK: { S: `SERVER#${slug}` }, SK: { S: 'STATUS' } },
      UpdateExpression: 'SET #s = :s, latencyMs = :l, lastCheckedAt = :t',
      ExpressionAttributeNames: { '#s': 'status' },
      ExpressionAttributeValues: {
        ':s': { S: status },
        ':l': { N: String(latencyMs) },
        ':t': { N: String(Date.now()) }
      }
    }).promise();
  } else {
    // Route write to home region via cross-region DynamoDB client
    const homeClient = getDynamoClientForRegion(homeRegion);
    await homeClient.updateItem({ /* same params */ }).promise();
    // Note: cross-region write adds ~50-150ms latency vs local write
  }
}

The tradeoff: cross-region writes add 50–150 ms latency compared to writing locally. For a 60-second probe interval, this is acceptable. The benefit is zero write conflicts — each server's status is written by exactly one region at a time.

Reading from the correct replica for different use cases

Strongly consistent reads (ConsistentRead: true) only guarantee consistency within a single region — they do not provide cross-region consistency. A strongly consistent read in eu-west-1 returns the latest committed write in eu-west-1, which may lag behind the latest write in us-east-1 by the current replication latency. For true cross-region consistency, route reads to the home region (the region that owns writes for that item).

Complete failure mode reference