Binary Serialization in MCP Tool Handlers: Format Selection, the BigInt Problem, and Schema Evolution

Pattern 1 — Choosing the right format for your MCP server's data source

The right binary serialization format is determined by where your data comes from, not by any intrinsic property of the data itself. MCP servers are bridges — they translate between a data source's native encoding and the text-based JSON that agent runtimes expect. Choosing the wrong format adds conversion complexity with no benefit; the right format uses the source's own encoding library to handle the conversion in a single line.

JSON (the default — no binary serialization needed)

Most MCP servers should return JSON from their tool handlers without any binary format in the path. PostgreSQL via node-postgres, REST APIs, SQLite, most NoSQL databases — all return data in forms that JSON.stringify handles correctly after minimal transformation (converting Date to ISO string, removing circular references). Choose a binary format only when you have a concrete reason: a gRPC backend, a Kafka topic, a binary cache, an analytics backend, or a Parquet data lake.

Protocol Buffers — gRPC backends

When an MCP server proxies a gRPC backend, Protocol Buffers is the native encoding and there is no alternative. The decision is which library to use: @bufbuild/protobuf with generated TypeScript classes (preferred) or protobufjs with runtime .proto loading (use only when codegen is not possible).

The generated approach via @bufbuild/protobuf eliminates three manual steps: Timestamp conversion (ts.toDate().toISOString() is built into toJson()), enum-to-string mapping (enumAsInteger: false in toJson()), and zero-value field inclusion (optional field syntax makes the encoder emit the field even when its value is 0, false, or "", which proto3's default behavior would silently drop). The @connectrpc/connect transport handles HTTP/2 framing, deadlines, and error-to-exception mapping, so tool handlers read as straightforward async function calls against a typed client:

import { createClient } from '@connectrpc/connect';
import { createGrpcTransport } from '@connectrpc/connect-node';
import { OrderService } from './gen/orders/v1/orders_connect.js';

const transport = createGrpcTransport({ baseUrl: process.env.ORDERS_GRPC_URL!, httpVersion: '2' });
const client = createClient(OrderService, transport);

server.tool('get_order', { order_id: z.string() }, async ({ order_id }) => {
  const response = await client.getOrder(new GetOrderRequest({ orderId: order_id }));
  // toJson() converts Timestamp → ISO string, enum → name, bytes → base64
  // optional fields are included even at zero value
  return { content: [{ type: 'text', text: JSON.stringify(response.order?.toJson({ enumAsInteger: false })) }] };
});

The protobufjs runtime approach requires two extra flags in toObject() that are easy to miss: defaults: true (include zero-value fields, matching proto3 optional semantics) and longs: String (convert Long objects to decimal strings, avoiding the BigInt problem). Without defaults: true, a field whose value is 0 or false will be absent from the tool response, and the agent will treat it as a missing key rather than a meaningful default. Verify all expected proto types resolve at startup via root.lookupType(typeName) so the server fails fast rather than producing confusing decode errors at request time. See the Protocol Buffers guide for the full startup verification probe.

Apache Avro — Kafka pipelines with Schema Registry

MCP servers that consume Kafka topics encoded with Confluent-compatible Avro serializers must use @kafkajs/confluent-schema-registry rather than calling an Avro parser directly on the raw message bytes. Every Confluent-wire-format Avro message begins with a 5-byte header: magic byte 0x00 followed by a 4-byte big-endian schema ID. Passing the raw message value to an Avro parser produces a decode error because the parser interprets the 5-byte header as schema content. registry.decode(message.value) handles the prefix transparently — it reads the schema ID, fetches the schema from the registry on first call and caches it for subsequent calls, then decodes the remaining bytes with the cached schema:

import { SchemaRegistry } from '@kafkajs/confluent-schema-registry';

// Module-scope registry client — NOT per-request (caches schemas by ID)
const registry = new SchemaRegistry({
  host: process.env.SCHEMA_REGISTRY_URL!,
  auth: { username: process.env.SCHEMA_REGISTRY_API_KEY!, password: process.env.SCHEMA_REGISTRY_API_SECRET! },
});

server.tool('get_recent_events', { topic: z.string(), max_events: z.number().int().min(1).max(100).default(20) },
  async ({ topic, max_events }) => {
    // registry.decode() strips the 5-byte prefix, fetches schema, decodes binary
    const decoded = await registry.decode(rawKafkaMessage.value);
    return { content: [{ type: 'text', text: JSON.stringify(decoded) }] };
  }
);

For MCP servers that also produce Avro messages, registry.encode(schemaId, payload) performs the inverse — it adds the 5-byte prefix automatically. Fetch the latest schema ID via registry.getLatestSchemaId(subject) and cache it at module scope to avoid repeated registry HTTP calls per message. See the Avro guide for nullable field union syntax and compatibility mode testing.

MessagePack — numeric-heavy caches and binary inter-service buffers

MessagePack makes sense when an MCP server acts as a cache layer for expensive computation results that contain many integers and floats — telemetry rows, feature vectors, aggregated analytics reports. For this use case it cuts payload size by 30–60% compared to JSON for integer-heavy payloads and 10–20% for string-heavy payloads. The @msgpack/msgpack library is the correct choice for cross-language compatibility (with Python msgpack and Go's vmihailenco/msgpack) — msgpackr with useRecords: true uses an ext type 0 record extension that no other implementation understands.

The key discipline when using MessagePack in MCP tool handlers is that the decoded binary never goes directly into a tool response — it always passes through a serialization step that converts bigint to string and Uint8Array to base64 before JSON.stringify:

import { decode, encode, ExtensionCodec } from '@msgpack/msgpack';

// Custom Date ext type (ext type 1 — int64 milliseconds)
const extensionCodec = new ExtensionCodec();
extensionCodec.register({
  type: 1,
  encode: (v) => {
    if (!(v instanceof Date)) return null;
    const buf = new ArrayBuffer(8);
    new DataView(buf).setBigInt64(0, BigInt(v.getTime()));
    return new Uint8Array(buf);
  },
  decode: (data) => new Date(Number(new DataView(data.buffer, data.byteOffset).getBigInt64(0))),
});

// Safe tool response serialiser — handles bigint and binary fields
function toJsonSafe(value: unknown): string {
  return JSON.stringify(value, (_, v) => {
    if (typeof v === 'bigint') return v.toString();
    if (v instanceof Uint8Array || Buffer.isBuffer(v)) return Buffer.from(v).toString('base64');
    return v;
  });
}

server.tool('get_cached_report', { report_id: z.string() }, async ({ report_id }) => {
  const raw = await redis.getBuffer(`report:${report_id}`);
  if (!raw) return { content: [{ type: 'text', text: JSON.stringify({ error: 'not_found' }) }] };
  const decoded = decode(raw, { extensionCodec });
  return { content: [{ type: 'text', text: toJsonSafe(decoded) }] };
});

Note that the toJsonSafe function above is the same pattern that works for Arrow and Parquet results — write it once and use it across all binary formats in the MCP server. See the MessagePack guide for size comparison utilities and NATS JetStream integration patterns.

Apache Arrow — analytics backends and DuckDB queries

Apache Arrow appears in MCP servers that front analytics backends: DuckDB running in-process, Python FastAPI services that return application/vnd.apache.arrow.stream responses, or Arrow Flight endpoints for columnar data access. The key property of Arrow is that it is already columnar — DuckDB's internal storage format is Arrow-compatible, so connection.arrowIPCStream() returns results without any column-to-row pivot, which is 3–5× faster than row-by-row conn.all() iteration for result sets in the thousands of rows.

The Arrow-specific complexity is that buffers live off the JavaScript heap. A 500 MB Arrow table does not show up in process.memoryUsage().heapUsed. For MCP servers that run large queries in a loop, this means memory grows invisibly until the process is OOM-killed. Scope Arrow table reads inside bounded functions so references are dropped after conversion, and call table.delete() in environments that expose Arrow's GC handle. The conversion step from Arrow row proxies to plain objects is mandatory before returning from a tool handler — Arrow row proxies are not JSON.stringify-safe:

import { tableFromIPC } from 'apache-arrow';
import Database from 'duckdb-async';

const db = await Database.create(':memory:');

server.tool('query_analytics', { sql: z.string().max(4000), max_rows: z.number().int().min(1).max(10000).default(1000) },
  async ({ sql, max_rows }) => {
    if (!/^(select|with)/i.test(sql.trim())) throw new Error('Only SELECT queries allowed');

    const conn = await db.connect();
    try {
      const stream = await conn.arrowIPCStream(`SELECT * FROM (${sql}) LIMIT ${max_rows}`);
      const batches: Uint8Array[] = [];
      for await (const chunk of stream) batches.push(chunk);

      const combined = new Uint8Array(batches.reduce((n, b) => n + b.byteLength, 0));
      let offset = 0;
      for (const b of batches) { combined.set(b, offset); offset += b.byteLength; }

      const table = tableFromIPC(combined);
      // Convert to plain objects — Arrow row proxies are not JSON-serialisable
      const rows = tableToJsonRows(table);  // see Pattern 2 for the full converter
      return { content: [{ type: 'text', text: JSON.stringify({ row_count: rows.length, rows }) }] };
    } finally {
      await conn.close();
    }
  }
);

For Python FastAPI backends returning Arrow IPC streams, use RecordBatchReader.from(resp.body) to consume the stream incrementally without buffering the full response. Always verify the content-type header is application/vnd.apache.arrow.stream before attempting to parse as Arrow — an HTML error page passed to the Arrow parser produces an inscrutable decoding exception. See the Arrow guide for the full schema-driven type conversion map and IPC startup verification probe.

Apache Parquet — S3/GCS data lakes via DuckDB HTTPFS

Parquet appears in MCP servers that read data lake files: event logs, audit trails, analytics snapshots stored on S3 or GCS. DuckDB's HTTPFS extension turns Parquet into a first-class query target — it performs byte-range HTTP requests so DuckDB can fetch only the column chunks and row groups it needs, combining column projection (SELECT col1, col2 instead of SELECT *) with predicate pushdown (WHERE date >= '2026-01-01' filters at the row-group statistics level before fetching data).

The critical discipline with Parquet in MCP tools is always projecting columns explicitly. A Parquet table with 200 columns but an agent query that needs 3 will transfer all 200 columns' data from S3 on a SELECT *. This is not just slow — it can exhaust the MCP server's memory budget for large files. Always build the SELECT col1, col2, col3 list from the tool's columns parameter and sanitize it to identifier-safe characters before interpolating into the query. The hive_partitioning=true option is equally important for partitioned datasets: without it, a path like s3://bucket/events/date=2026-07-01/*.parquet will not expose date as a queryable column, and the tool response will include rows from all partitions regardless of any WHERE date = filter:

const safeCols = columns
  .map(c => c.replace(/[^a-zA-Z0-9_]/g, ''))
  .filter(Boolean)
  .map(c => `"${c}"`)
  .join(', ');

const rows = await conn.all(`
  SELECT ${safeCols}
  FROM read_parquet('${s3Path.replace(/'/g, "''")}',
    hive_partitioning = true,
    union_by_name     = true
  )
  WHERE date >= '${dateFrom}' AND date <= '${dateTo}'
  LIMIT ${limit}
`, (_, val) => typeof val === 'bigint' ? val.toString() : val);

Use parquet_schema() to discover column names and types before building tool queries, and parquet_metadata() to estimate whether a full scan is feasible given the file size. See the Parquet guide for the HTTPFS health probe and schema drift handling with union_by_name=true.

Pattern 2 — The cross-cutting BigInt problem

Every binary serialization format in this article produces JavaScript values that JSON.stringify cannot handle. The root cause is that all five formats have native 64-bit integer types — proto3's int64/uint64/fixed64, Avro's long, MessagePack's uint64 for values above 2³², Arrow's Int64/Uint64/Timestamp[ns], and Parquet's INT64 — and none of these fit in JavaScript's 53-bit safe integer range. The JavaScript ecosystem's answer is bigint, but JSON.stringify predates bigint and throws TypeError: Do not know how to serialize a BigInt when it encounters one. The failure is silent in development when all test values happen to be small integers, then breaks in production when the first real 64-bit value arrives.

Where each format produces BigInt

The unified replacer — write it once

Rather than per-format BigInt handling, write a single toJsonSafe function and use it in every tool handler that touches binary formats. The replacer also handles Uint8Array and Buffer values that MessagePack's bin type and Arrow's binary columns produce:

// One replacer function — handles bigint and binary from ALL five formats
function toJsonSafe(value: unknown): string {
  return JSON.stringify(value, (_key, val) => {
    if (typeof val === 'bigint') {
      return val.toString(); // decimal string — agents parse this correctly
    }
    if (val instanceof Uint8Array || Buffer.isBuffer(val)) {
      return Buffer.from(val).toString('base64');
    }
    return val;
  });
}

// Usage in every binary-format tool handler:
return { content: [{ type: 'text', text: toJsonSafe(decoded) }] };

Format-specific BigInt conversion for timestamps

When the bigint value is a timestamp, converting it to a decimal string gives agents an opaque number. Better: convert it to an ISO-8601 string at the point you know what the value represents. Each format encodes timestamps differently:

// Protocol Buffers — Timestamp → ISO string
// @bufbuild/protobuf:
const iso = ts.toDate().toISOString();
// protobufjs (Long seconds + nanos):
const iso = new Date(Number(ts.seconds.toString()) * 1000 + Math.floor(ts.nanos / 1_000_000)).toISOString();

// Apache Arrow — Timestamp[ns] arrives as bigint nanoseconds
const iso = new Date(Number(bigintNs) / 1_000_000).toISOString();

// Apache Arrow — Timestamp[us] arrives as bigint microseconds
const iso = new Date(Number(bigintUs) / 1_000).toISOString();

// Apache Arrow — Timestamp[ms] arrives as number (safe integer), no bigint
const iso = new Date(tsMs).toISOString();

// Apache Arrow — Timestamp[s] arrives as number seconds
const iso = new Date(tsS * 1000).toISOString();

// Avro — timestamp-millis logical type arrives as number (safe)
const iso = new Date(tsMs).toISOString();

// MessagePack — custom Date ext type (ext type 1, int64 ms)
// (decoded by extensionCodec.register, arrives as a JS Date already)
const iso = date.toISOString();

The Arrow schema inspection pattern described in the Arrow guide generalizes this — build a conversion map from the schema's field types and apply the correct conversion rule per field rather than assuming all bigint values are the same kind of timestamp.

Why Number(bigint) is often wrong

A common fix attempt is Number(bigintValue). This works for timestamps (Arrow Timestamp[ns] divided by 1,000,000 still fits in a 53-bit integer for any modern timestamp) but fails silently for IDs and counters above Number.MAX_SAFE_INTEGER (9,007,199,254,740,991) — the conversion rounds to the nearest representable float, so a user ID of 9007199254740993n becomes 9007199254740992 and two distinct IDs can map to the same number. Use .toString() for IDs, counters, and any value that must be preserved exactly; use Number(bigint) / scale only for timestamps where you control the scale and know the result stays in safe integer range.

Pattern 3 — Schema evolution safety across the five formats

MCP servers that depend on a binary format's schema face a different class of failure than connection errors or type conversion bugs: the tool handler decodes data correctly today, but after a schema change in the upstream service it decodes different data silently, produces garbled results, or throws an exception from inside the decoder. Each format has a different model for what a "schema" is and what evolution rules it enforces.

Protocol Buffers — field numbers are forever

Proto3 encodes every field by its number, not its name. A message on the wire looks like a sequence of (field_number, value) pairs. If you delete field 5 (string coupon_code = 5) and later add a different field with number 5 (int64 discount_cents = 5), any consumer running the old schema will silently decode the new int64 bytes as a string, producing corrupt data — no error, no warning, just wrong values. The reserved keyword prevents this at the proto compiler level:

message Order {
  string order_id = 1;
  string user_id  = 2;
  // Fields 3 and 4 removed in v2 — permanently blocked from reuse
  reserved 3, 4;
  reserved "coupon_code", "is_priority"; // names blocked too

  repeated LineItem items     = 5;
  optional int32 retry_count  = 8; // 'optional' emits field even at zero value
}

// Safe evolution operations:
// ✅ Add new fields with new numbers
// ✅ Rename a field (wire uses numbers, not names)
// ✅ Add new enum values
// ❌ Remove a field without reserving its number AND name
// ❌ Change a field's type (silent misparse)
// ❌ Change a field's number

Unknown fields — fields the current schema doesn't know about — are preserved in a binary buffer by both protobufjs and @bufbuild/protobuf by default. This means an MCP server with an older proto descriptor can safely proxy messages that contain newer fields without data loss.

Avro — Schema Registry compatibility modes as a gate

Avro's Schema Registry enforces evolution rules at registration time, not at read time. Every POST /subjects/{subject}/versions call runs a compatibility check against the current schema before accepting the new version. The default mode is BACKWARD: new consumers can read messages written by old producers. This allows adding optional fields (union with null first, default null) and deleting defaulted fields, but rejects adding required fields or changing types.

Before registering a new schema version, test it with a dry-run compatibility call — this avoids an unexpected 409 Conflict that blocks the deployment:

// Dry-run compatibility check — run in CI before deploying schema changes
async function testSchemaCompatibility(subject: string, candidate: object): Promise<boolean> {
  const resp = await fetch(
    `${process.env.SCHEMA_REGISTRY_URL}/compatibility/subjects/${subject}/versions/latest`,
    {
      method: 'POST',
      headers: {
        'Content-Type': 'application/vnd.schemaregistry.v1+json',
        Authorization: `Basic ${btoa(`${process.env.SCHEMA_REGISTRY_API_KEY}:${process.env.SCHEMA_REGISTRY_API_SECRET}`)}`,
      },
      body: JSON.stringify({ schema: JSON.stringify(candidate) }),
    }
  );
  return ((await resp.json()) as { is_compatible: boolean }).is_compatible;
}

// Nullable field pattern — null FIRST in union so default: null is valid
const schema = {
  type: 'record', name: 'UserEvent', namespace: 'com.example',
  fields: [
    { name: 'event_id', type: 'string' },          // required
    { name: 'display_name', type: ['null', 'string'], default: null }, // nullable
  ],
};

The nullable field rule is the most commonly violated Avro constraint in MCP servers: { "type": "string" } is required, ["string", "null"] with a non-null default is effectively required for backward compatibility, and only ["null", "string"] with "default": null is truly optional. The order matters because Avro's default must match the first branch of the union.

MessagePack — ext type IDs as stable contracts

MessagePack has no schema — the format is self-describing (each value's type tag is in the byte stream) for the core types (nil, bool, int, float, str, bin, array, map), and ext types are just an integer ID between 0 and 127 with an opaque byte payload. This makes MessagePack extremely flexible but shifts all schema contracts to convention: if two services agree that ext type 1 is a millisecond-precision timestamp, that agreement must be maintained forever. Changing ext type 1 to carry nanosecond precision breaks all consumers of the old format with no error — they decode the 8 bytes correctly as int64 but interpret the value as milliseconds instead of nanoseconds.

The correct evolution strategy is additive: introduce ext type 2 for nanosecond timestamps while continuing to write ext type 1 for millisecond timestamps until all consumers migrate. Never reuse an ext type ID for a different semantic. Document the ext type ID table in a shared config file or package that both publishers and consumers import:

// Shared ext type contract — treat as a stable interface
export const EXT_TYPES = {
  DATE_MS:    1, // int64 milliseconds since epoch (stable since v1)
  DATE_NS:    2, // int64 nanoseconds since epoch (added v2, use for new data)
  DECIMAL:    3, // string representation of arbitrary-precision decimal
  GEO_POINT:  4, // [float64 lat, float64 lon]
} as const;

// DO NOT reuse IDs, even for similar purposes — use a new ID

For cross-language interop: stick to the 8 core MessagePack types (nil, bool, int, float, str, bin, array, map) for any value that crosses language boundaries. Custom ext types require matching decoder registration on every consumer. Python's msgpack library uses ext type 5 for its Timestamp type — do not use ext type 5 for anything else in a cross-language system.

Apache Arrow — IPC format version and stream vs. file format

Arrow's schema evolution model is per-batch: each IPC record batch carries its own schema in its header, so schema can change across batches in a stream and the reader handles it gracefully. The evolution failure mode in Arrow is not a schema change — it is using the wrong IPC format variant. The Arrow IPC spec has two wire formats:

• Stream format — prefixed with a continuation marker (0xFFFFFFFF); used for streaming scenarios where the total batch count is unknown; tableToIPC(table, 'stream')
• File format — prefixed with the ASCII magic bytes ARROW1 followed by a footer; designed for seek-able file access; tableToIPC(table, 'file')

Passing a stream-format buffer to a parser expecting file format silently reads zero batches — it neither throws nor returns an error. This is the Arrow equivalent of a schema mismatch. Always use stream format for data passed between MCP server components, and verify with the round-trip startup probe:

import { tableFromIPC, tableToIPC, makeTable, vectorFromArray } from 'apache-arrow';

async function verifyArrowIpc(): Promise<void> {
  const source = makeTable({ id: vectorFromArray([1, 2, 3]), name: vectorFromArray(['a', 'b', 'c']) });
  const roundTripped = tableFromIPC(tableToIPC(source, 'stream')); // always use 'stream'
  if (roundTripped.numRows !== 3) throw new Error(`Arrow IPC round-trip failed: expected 3 rows, got ${roundTripped.numRows}`);
  console.log('Arrow IPC verification passed');
}

Apache Parquet — union_by_name for multi-file schema drift

Parquet files in data lake paths are commonly written by multiple jobs over time, each potentially adding or renaming columns. Without union_by_name=true, DuckDB throws a schema mismatch error on the first file whose schema differs from the first file it read. With union_by_name=true, DuckDB reads all file footers, builds a union schema, and fills missing columns with NULL for files that don't have them. The tradeoff is startup latency — for a glob that matches hundreds of files, reading all footers adds several seconds before the first row is returned.

The recommended pattern for MCP tools over large partitioned datasets is to always pass union_by_name=true and to validate requested column names against the discovered schema before executing the data query:

// Step 1: discover union schema (reads footers only, no row data)
const schema = await conn.all(
  `SELECT DISTINCT name FROM parquet_schema('${s3Glob}') WHERE name != 'schema' ORDER BY name`
);
const knownCols = new Set(schema.map((r: { name: string }) => r.name));

// Step 2: validate requested columns
const invalid = requestedColumns.filter(c => !knownCols.has(c));
if (invalid.length > 0) {
  return { content: [{ type: 'text', text: JSON.stringify({ error: `Unknown columns: ${invalid.join(', ')}`, available: [...knownCols] }) }] };
}

// Step 3: execute data query with union schema and column projection
const rows = await conn.all(
  `SELECT ${safeCols} FROM read_parquet('${s3Glob}', union_by_name=true, hive_partitioning=true) LIMIT ${limit}`
);

Composite health probe comparison

Each binary format has a different health probe surface — some require a live server (Schema Registry, gRPC backend), some test a local library round-trip, and some verify both a library and a storage backend. Register all probes as MCP resources so AliveMCP can poll them every 60 seconds and alert you before agents start receiving decode errors from a broken upstream.

The Avro probe should cache its result for 30 seconds to avoid flooding the Schema Registry with health checks from all concurrent tool calls in a busy MCP session. The proto descriptor probe runs once at startup and panics if any required type is missing — at runtime, assume descriptors are valid (they cannot change without a redeploy). The Parquet probe is the only one that separates two independent failure modes (HTTPFS extension and S3 connectivity) into distinct fields in the probe result — this makes it easier to diagnose which layer failed when AliveMCP reports the endpoint as unhealthy.

Format selection guide — use-case matrix

When an MCP server needs to integrate with a new data source or backend, this matrix maps the most common use cases to the right serialization format and the key decision criteria that rules out the alternatives:

Keeping your MCP server visible when it depends on binary backends

An MCP server that proxies a gRPC backend, consumes a Kafka topic, or reads Parquet from S3 has more failure modes than a server that only talks to a database: the gRPC backend can be unhealthy, the Schema Registry can be unreachable, the S3 credentials can expire, or the HTTPFS extension can fail to load after a DuckDB upgrade. Each of these failures presents to agents as a tool handler error rather than an infrastructure outage — agents retry, report confusing errors, and authors don't find out until a user complains.

The health probe pattern addresses this: register each binary backend as an MCP resource at the appropriate URI (proto://health, avro://health, etc.) and configure AliveMCP to monitor those endpoints. When the Schema Registry goes down at 2 AM, AliveMCP pings the avro://health resource, sees the registry failure in the JSON response, and sends an alert — before any agent traffic reaches the broken tool handler. The 90-day uptime history on the AliveMCP dashboard shows which binary backend is the reliability bottleneck, which is the data you need to make the case for a more resilient deployment.