AWS Glue Data Catalog for MCP Servers

Catalog hierarchy: databases, tables, partitions

The Glue Data Catalog is organized as a three-level hierarchy. Each AWS account has one catalog per region.

• Database — logical grouping of tables; maps to a namespace in SQL. One database per product domain is a clean pattern (e.g., mcp_logs, mcp_analytics).
• Table — schema definition + location. For S3-backed tables the location is an S3 prefix; for JDBC tables it's a connection + table name. Tables are not the data themselves — they're the schema contract and pointer.
• Partition — a subset of a table's data stored at a specific S3 sub-prefix. A partition is identified by its partition key values (e.g., dt=2026-09-24, server_id=xyz). Partitions are registered separately from the table; adding a new S3 partition without registering it in the catalog makes it invisible to Athena and Glue ETL.

import boto3

glue = boto3.client("glue")

# Create a database for MCP log tables
glue.create_database(
    DatabaseInput={
        "Name": "mcp_logs",
        "Description": "Raw and transformed MCP server event logs",
        "LocationUri": "s3://mcp-data/logs/",  # Optional default location hint
    }
)

# Create a partitioned table for raw tool call events
glue.create_table(
    DatabaseName="mcp_logs",
    TableInput={
        "Name": "raw_tool_calls",
        "StorageDescriptor": {
            "Columns": [
                {"Name": "server_id", "Type": "string"},
                {"Name": "tool_name", "Type": "string"},
                {"Name": "duration_ms", "Type": "bigint"},
                {"Name": "error", "Type": "boolean"},
                {"Name": "timestamp_ms", "Type": "bigint"},
                {"Name": "payload", "Type": "string"},
            ],
            "Location": "s3://mcp-data/logs/raw_tool_calls/",
            "InputFormat": "org.apache.hadoop.mapred.TextInputFormat",
            "OutputFormat": "org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat",
            "SerdeInfo": {
                "SerializationLibrary": "org.openx.data.jsonserde.JsonSerDe",
                "Parameters": {"serialization.format": "1"},
            },
        },
        "PartitionKeys": [
            {"Name": "dt", "Type": "string"},   # Hive-style: s3://.../dt=2026-09-24/
        ],
        "TableType": "EXTERNAL_TABLE",
        "Parameters": {
            "classification": "json",
            "compressionType": "none",
            "typeOfData": "file",
        },
    }
)

Partition projection — eliminate crawlers for predictable partitions

Partition projection is a Glue/Athena feature that computes partition locations on-the-fly from a formula rather than reading them from the catalog. For MCP log tables that use date-based partitioning (dt=YYYY-MM-DD), partition projection means:

• No crawler needed to register new daily partitions.
• New S3 prefixes are immediately queryable — zero propagation delay.
• No partition metadata accumulates in the catalog (tables with years of daily partitions can have 1,000+ partition rows — projection eliminates all of them).

# Create a table with partition projection for daily MCP logs
# date-projected partition: dt ranges from 2026-01-01 to NOW
glue.create_table(
    DatabaseName="mcp_logs",
    TableInput={
        "Name": "raw_tool_calls_projected",
        "StorageDescriptor": {
            "Columns": [
                {"Name": "server_id", "Type": "string"},
                {"Name": "tool_name", "Type": "string"},
                {"Name": "duration_ms", "Type": "bigint"},
                {"Name": "error", "Type": "boolean"},
                {"Name": "timestamp_ms", "Type": "bigint"},
            ],
            "Location": "s3://mcp-data/logs/raw_tool_calls/",
            "InputFormat": "org.apache.hadoop.mapred.TextInputFormat",
            "OutputFormat": "org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat",
            "SerdeInfo": {
                "SerializationLibrary": "org.apache.hive.hcatalog.data.JsonSerDe",
            },
        },
        "PartitionKeys": [{"Name": "dt", "Type": "string"}],
        "TableType": "EXTERNAL_TABLE",
        "Parameters": {
            # Partition projection — Athena and Glue ETL read these properties
            "projection.enabled": "true",
            "projection.dt.type": "date",
            "projection.dt.format": "yyyy-MM-dd",
            "projection.dt.range": "2026-01-01,NOW",
            "projection.dt.interval": "1",
            "projection.dt.interval.unit": "DAYS",
            # Tell Athena where to find each partition
            "storage.location.template": "s3://mcp-data/logs/raw_tool_calls/dt=${dt}/",
            "classification": "json",
        },
    }
)

Once projection is configured, queries like WHERE dt = '2026-09-24' in Athena resolve the S3 path directly without a catalog partition lookup. For multi-column partitions (e.g., dt + region), define both projection properties — Athena combines them using the storage.location.template.

Schema versioning and change history

Every time a Glue Crawler detects a schema change (new column, changed type, dropped column), it creates a new schema version for the table. The current schema version is queryable; all previous versions are retained.

# List all schema versions for a table
versions = glue.get_table_versions(
    DatabaseName="mcp_logs",
    TableName="raw_tool_calls",
)["TableVersions"]

for v in versions:
    cols = v["Table"]["StorageDescriptor"]["Columns"]
    print(f"Version {v['VersionId']}: {len(cols)} columns, updated {v['Table']['UpdateTime']}")

# Get a specific historical version
v3 = glue.get_table_version(
    DatabaseName="mcp_logs",
    TableName="raw_tool_calls",
    VersionId="3",
)

Schema version count as a health signal: a table with dozens of schema versions means the source data schema is not stable — your MCP server is emitting tool call events with variable structure across versions. In a healthy system, schema versions should be low (under 5) and only increment when you intentionally add a new field. If versions are accumulating rapidly, fix the schema at the source and consider locking the Glue table schema with UpdateTable and disabling the crawler's schema update permission.

Catalog encryption and cross-account access

By default the Glue Data Catalog is not encrypted. You can enable server-side encryption via the Glue console (Settings → Data Catalog encryption) using a KMS CMK. This encrypts table metadata at rest — it does not encrypt the S3 data itself.

# Enable catalog encryption (account-level setting)
glue.put_data_catalog_encryption_settings(
    DataCatalogEncryptionSettings={
        "EncryptionAtRest": {
            "CatalogEncryptionMode": "SSE-KMS",
            "SseAwsKmsKeyId": "arn:aws:kms:us-east-1:123456789012:key/abc123",
        },
        "ConnectionPasswordEncryption": {
            "ReturnConnectionPasswordEncrypted": True,
            "AwsKmsKeyId": "arn:aws:kms:us-east-1:123456789012:key/abc123",
        },
    }
)

Cross-account catalog access: to share a Glue catalog table with another AWS account (e.g., sharing MCP log tables between a logging account and an analytics account), use AWS Lake Formation to grant cross-account permissions. Direct IAM cross-account access to the Glue catalog is not supported — Lake Formation is the only supported mechanism for catalog-level cross-account sharing.

Hive metastore compatibility

The Glue Data Catalog is a drop-in replacement for Apache Hive Metastore. EMR clusters, Spark on EKS, and Presto/Trino deployments can all use the Glue catalog as their metastore by setting the appropriate Spark/Hive configuration properties — no Hive Metastore service to manage.

# EMR cluster configuration to use Glue catalog
{
    "Classification": "spark-hive-site",
    "Properties": {
        "hive.metastore.client.factory.class":
            "com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory"
    }
}

# Spark on EKS — set in SparkConf
spark_conf = {
    "spark.hadoop.hive.metastore.client.factory.class":
        "com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory",
}

With the Glue metastore connector, any SHOW TABLES, DESCRIBE TABLE, or CREATE TABLE issued from your Spark session goes through the Glue Catalog API. This means the same table registered by a Glue ETL job is immediately visible to Athena, EMR Spark, and Presto — one schema definition, multiple query engines.

GetTable API latency and caching patterns

The GetTable API typically responds in 20–100 ms. For Glue ETL jobs and Athena queries, table metadata is cached for the duration of the job/query — the overhead is one-time per job run. For applications that call GetTable on every Lambda invocation or in a hot loop, the latency accumulates.

import boto3
import functools
import time

glue = boto3.client("glue")

# Simple in-process cache with TTL — avoid repeated GetTable calls in Lambda
_table_cache: dict = {}

def get_table_cached(database: str, table: str, ttl_seconds: int = 300):
    key = f"{database}.{table}"
    entry = _table_cache.get(key)
    if entry and (time.monotonic() - entry["ts"]) < ttl_seconds:
        return entry["data"]
    result = glue.get_table(DatabaseName=database, Name=table)["Table"]
    _table_cache[key] = {"data": result, "ts": time.monotonic()}
    return result

For Lambda functions that need table column metadata at runtime (e.g., to validate tool call payloads against the catalog schema), cache the schema in the Lambda initialization path (outside the handler) to avoid per-invocation catalog calls. The 5-minute default TTL is safe for most schemas; reduce to 60 seconds if your pipeline frequently adds columns.