MSK Schema Registry for MCP Servers — Glue Schema Registry, Compatibility Modes, and Probe Payload Versioning

How Glue Schema Registry integrates with MSK messages

When a Glue-aware producer sends a Kafka message, the serializer prepends a 5-byte header to the message value: 1 magic byte (0x03 for Glue, not the Confluent 0x00), followed by a 4-byte schema-version UUID index. The consumer deserializer reads that header, calls glue:GetSchemaByDefinition or glue:GetSchemaVersion to retrieve the schema, and uses it to decode the payload. Schemas are cached locally for the session lifetime — the Glue API is not called per message.

The header format difference is important if you are mixing Glue-aware and Confluent-aware clients: a Confluent deserializer will fail to parse a Glue-serialized message because the magic byte does not match, and vice versa. Keep one schema registry system per MSK cluster to avoid this.

Supported schema formats

AliveMCP uses Avro for the probe-results topic (high volume, same consumer codebase) and JSON Schema for the audit-events topic (low volume, consumed by diverse Lambda functions without a shared library).

Compatibility modes: what each allows

Compatibility mode is set per schema group (a logical container for schema versions). It controls which schema evolution operations are allowed when registering a new version:

Probe result schema evolution example

AliveMCP probe result schema v1 has fields: endpointId (string, required), status (enum: UP/DOWN/DEGRADED), responseTimeMs (int), probedAt (long, unix ms). For v2, we want to add tlsCertExpiresAt (long, optional) for TLS expiry tracking.

Adding an optional field with a default (null in Avro) is a BACKWARD-compatible change. The schema registration will succeed if the group is in BACKWARD or FULL mode. Old consumers that have not been updated will read v2 messages and ignore the new field — their Avro deserializer maps unknown fields to the default specified in their locally-cached v1 schema. New consumers use the v2 schema and read the TLS field when present.

// Avro schema v2 — adding optional tlsCertExpiresAt
{
  "type": "record",
  "name": "ProbeResult",
  "namespace": "com.alivemcp",
  "fields": [
    { "name": "endpointId",       "type": "string" },
    { "name": "status",           "type": { "type": "enum", "name": "Status", "symbols": ["UP", "DOWN", "DEGRADED"] } },
    { "name": "responseTimeMs",   "type": "int" },
    { "name": "probedAt",         "type": "long" },
    { "name": "tlsCertExpiresAt", "type": ["null", "long"], "default": null }
  ]
}

What is NOT backward-compatible: renaming responseTimeMs to latencyMs, changing status from enum to string, or removing the probedAt field. The Glue registry rejects these registrations at schema-push time, before any producer deploys — the failure is at CI/CD, not at runtime.

Producer and consumer configuration (Node.js)

// Producer with Glue Schema Registry serialization (kafkajs + @aws-sdk/client-glue)
import { GlueSchemaRegistrySerializer } from 'aws-glue-schema-registry';

const serializer = new GlueSchemaRegistrySerializer({
  region: 'us-east-1',
  registryName: 'alivemcp-probe-schemas',
  schemaName: 'ProbeResult',
  compatibility: 'BACKWARD',
  dataFormat: 'AVRO',
  avroSchema: probeResultSchemaV2,  // the Avro schema JSON object
});

// Before sending to Kafka:
const encodedPayload = await serializer.encode(probeResultObject);
await producer.send({
  topic: 'probe-results',
  messages: [{ value: encodedPayload }],
});

// Consumer with Glue deserializer
import { GlueSchemaRegistryDeserializer } from 'aws-glue-schema-registry';

const deserializer = new GlueSchemaRegistryDeserializer({ region: 'us-east-1' });

const consumer = kafka.consumer({ groupId: 'probe-archiver-v2' });
await consumer.run({
  eachMessage: async ({ message }) => {
    const decoded = await deserializer.decode(message.value);
    // decoded is a plain JS object matching the schema
    await archiveProbeResult(decoded);
  },
});

Cross-account consumer access

The Glue Schema Registry is regional and account-scoped. If you have a consumer Lambda in a separate AWS account (for example, an analytics pipeline in an analytics account reading probe events from the MSK cluster in your application account), you need a resource-based policy on the Glue registry:

// CDK: attach resource policy to the Glue registry
import { aws_glue as glue, aws_iam as iam } from 'aws-cdk-lib';

const registry = new glue.CfnRegistry(this, 'ProbeSchemas', {
  name: 'alivemcp-probe-schemas',
  description: 'Schema versions for AliveMCP probe result topics',
});

// Allow analytics account to read schema versions
const registryPolicy = new iam.PolicyDocument({
  statements: [new iam.PolicyStatement({
    principals: [new iam.AccountPrincipal('987654321098')],  // analytics account
    actions: ['glue:GetSchemaVersion', 'glue:GetSchemaByDefinition', 'glue:QuerySchemaVersionMetadata'],
    resources: [registry.attrArn],
  })],
});

new glue.CfnResourcePolicy(this, 'RegistryResourcePolicy', {
  resourceArn: registry.attrArn,
  policyDocument: registryPolicy.toJSON(),
});

The consumer in the analytics account must use IAM credentials from that account; it then calls Glue directly to fetch schemas. The MSK cluster itself still needs its own cross-account access policy (or VPC peering / PrivateLink) if the consumer Kafka client connects to MSK directly.

Common failure modes