Guide · AWS Glue

AWS Glue Crawlers for MCP Servers

AWS Glue Crawlers scan data stores (S3, JDBC, DynamoDB, MongoDB) and automatically populate the Glue Data Catalog with table definitions, column types, and partition metadata — eliminating manual schema registration. For MCP servers this is the entry point for analytics pipelines: drop JSON events into S3, run a crawler, and they're immediately queryable in Athena. Three things teams consistently get wrong: the 10-minute crawl cold-start delay — even with the shortest schedule ("every 5 minutes"), Glue adds a startup overhead that means new data is typically not catalogued for 10–15 minutes after landing; if sub-minute catalog freshness matters, use partition projection instead. Not configuring schema change behavior — the default is UPDATE_IN_DATABASE, which silently alters the Glue table schema when new columns appear in your source data; if downstream Athena queries are column-typed, silent schema evolution breaks them; for production pipelines, set the schema change policy to LOG or FAIL and control schema changes explicitly. Confusing classifier order — crawlers try classifiers in a defined priority chain; if a custom classifier runs before the built-in JSON classifier and returns the wrong type, all subsequent classifiers are skipped.

TL;DR

Use S3 event-triggered crawlers for near-real-time catalog updates (note: minimum crawl granularity is still ~5 minutes). Set schema change policy to LOG on production tables to prevent silent schema evolution. Use partition projection instead of crawlers for predictable date-partitioned S3 paths. Custom classifiers go first in priority order — be precise with grok patterns to avoid misclassification.

Classifier priority chain

When a crawler scans an S3 object, it runs classifiers in order until one returns a confidence of 1.0 (certain match). The classifier chain is:

Custom classifiers (user-defined, run first in the order you specify)
Built-in classifiers in this priority order: CSV → JSON → XML → Parquet → ORC → Avro → Ion → Grok → AWS Glue ServiceLogs

The first classifier to return a match wins and all remaining classifiers are skipped. This means a poorly written custom classifier that matches JSON as "custom_type" will prevent the built-in JSON classifier from ever running on those files.

import boto3

glue = boto3.client("glue")

# Create a custom grok classifier for MCP server access logs
# (non-JSON structured text logs from a load balancer in front of MCP endpoints)
glue.create_classifier(
    GrokClassifier={
        "Classification": "mcp-access-log",
        "Name": "mcp-alb-access-log",
        # Grok pattern: timestamp method path status duration server_id
        "GrokPattern": "%{TIMESTAMP_ISO8601:timestamp} %{WORD:method} %{URIPATHPARAM:path} %{NUMBER:status_code:int} %{NUMBER:duration_ms:float} %{WORD:server_id}",
        # CustomPatterns: extend with additional named patterns
        "CustomPatterns": "",
    }
)

# Create a crawler that uses this custom classifier first,
# then falls back to built-ins for JSON files in the same S3 prefix
glue.create_crawler(
    Name="mcp-logs-crawler",
    Role="arn:aws:iam::123456789012:role/GlueCrawlerRole",
    DatabaseName="mcp_logs",
    Targets={
        "S3Targets": [
            {"Path": "s3://mcp-data/logs/", "Exclusions": ["**.tmp", "**/_temporary/**"]},
        ]
    },
    Classifiers=["mcp-alb-access-log"],  # Custom classifiers run FIRST
    SchemaChangePolicy={
        "UpdateBehavior": "LOG",           # Don't silently update schema; log the change
        "DeleteBehavior": "LOG",           # Don't delete tables when S3 path is gone
    },
    RecrawlPolicy={"RecrawlBehavior": "CRAWL_NEW_FOLDERS_ONLY"},  # Skip already-crawled partitions
)

Schema change detection modes

The SchemaChangePolicy.UpdateBehavior controls what happens when a crawler detects that the schema of an existing table has changed (new columns, changed types, dropped columns).

UpdateBehavior	What happens	When to use
`UPDATE_IN_DATABASE`	Glue silently updates the table schema to match the new data. Downstream queries may break if they relied on old column types.	Development/exploratory workflows where schema is actively evolving
`LOG`	Glue logs the schema change as a warning in CloudWatch Logs but does NOT update the catalog table. The table stays on the old schema.	Production tables where schema must be explicitly managed; detect drift without breaking anything
`FAIL` (via crawler run failure)	Not a direct setting — achieved by setting `UpdateBehavior: LOG` and adding a CloudWatch alarm on the crawler log group for schema change warnings, then triggering a Lambda to mark the pipeline as degraded	Pipelines with strict schema contracts where any change must halt the pipeline

The DeleteBehavior setting controls what happens when S3 files that were previously catalogued are no longer present:

LOG — logs a warning; table remains in catalog.
DELETE_FROM_DATABASE — removes the table from the catalog. Dangerous if S3 lifecycle policies archive old data to Glacier — the crawler will see the files as "gone" and delete the table definition.
DEPRECATE_IN_DATABASE — marks the table as deprecated without removing it.

S3 event-based crawl triggers

Glue supports triggering a crawler run from an S3 event notification rather than only on a fixed schedule. This reduces latency between data landing in S3 and the partition being queryable in Athena.

# Create an S3 event trigger for the crawler
# Step 1: configure S3 to send EventBridge notifications for the bucket
# (S3 → Properties → Event notifications → Send to EventBridge: On)

# Step 2: Create a Glue trigger of type EVENT
glue.create_trigger(
    Name="mcp-logs-s3-event-trigger",
    Type="EVENT",
    WorkflowName="mcp-log-pipeline",  # Optional: tie to a Glue workflow
    Actions=[{"CrawlerName": "mcp-logs-crawler"}],
    EventBatchingCondition={
        "BatchSize": 1,     # Trigger after this many S3 events (1 = immediate)
        "BatchWindow": 900, # OR after 900 seconds (15 min), whichever comes first
    },
)

Important: even with S3 event triggering, Glue crawlers have an inherent startup time of 5–10 minutes before they begin scanning. The event trigger fires the crawler start; it does not bypass the crawler startup process. For partition-projection tables, the new partition is queryable immediately with zero delay — event-triggered crawlers are a middle ground for tables that can't use projection.

Partition inference from S3 path structure

Glue Crawlers can infer partition columns from the S3 path structure using Hive-style partitioning (key=value path segments). For MCP log buckets organized as s3://bucket/logs/year=2026/month=09/day=24/, the crawler automatically creates partition columns year, month, and day in the catalog table.

For non-Hive paths (e.g., s3://bucket/logs/2026/09/24/ with no key= prefix), the crawler creates a single partition per top-level prefix — it doesn't infer multi-level partitions. You have two options:

Rename S3 prefixes to Hive-style (year=2026/month=09/day=24/) — the crawler infers partition columns automatically.
Use partition projection — define the partition structure explicitly in the table properties; no path renaming needed and no crawler required.

# RecrawlBehavior options — control how the crawler handles already-catalogued S3 paths
glue.update_crawler(
    Name="mcp-logs-crawler",
    RecrawlPolicy={
        # CRAWL_EVERYTHING: re-scan all S3 objects on every run (expensive for large buckets)
        # CRAWL_NEW_FOLDERS_ONLY: only scan S3 prefixes not yet in the catalog (fast, incremental)
        # CRAWL_EVENT_MODE: only process S3 objects that triggered the event (event-trigger only)
        "RecrawlBehavior": "CRAWL_NEW_FOLDERS_ONLY",
    }
)

Use CRAWL_NEW_FOLDERS_ONLY for production crawlers on large S3 buckets to avoid re-scanning historical data on every run. Switch to CRAWL_EVERYTHING only when you need to detect schema changes in existing partitions (not just new ones).

Crawler IAM role requirements

# Minimum IAM policy for a Glue Crawler on S3 + Glue Catalog
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::mcp-data",
                "arn:aws:s3:::mcp-data/*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "glue:CreateTable",
                "glue:UpdateTable",
                "glue:DeleteTable",
                "glue:CreatePartition",
                "glue:BatchCreatePartition",
                "glue:UpdatePartition",
                "glue:DeletePartition",
                "glue:GetTables",
                "glue:GetTable",
                "glue:GetDatabase",
                "glue:CreateDatabase"
            ],
            "Resource": [
                "arn:aws:glue:*:123456789012:catalog",
                "arn:aws:glue:*:123456789012:database/mcp_logs",
                "arn:aws:glue:*:123456789012:table/mcp_logs/*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogGroup",
                "logs:CreateLogStream",
                "logs:PutLogEvents"
            ],
            "Resource": "arn:aws:logs:*:123456789012:log-group:/aws-glue/crawlers:*"
        }
    ]
}

Crawler run timing and cold-start delay

Scenario	Typical latency from data landing to catalog update	Notes
Scheduled crawl (5-min interval)	5–15 min	Schedule fires, then Glue allocates crawler infrastructure (5–10 min startup)
S3 event trigger	5–12 min	Event fires immediately but crawler still has 5–10 min startup overhead
On-demand via `start_crawler` API	5–10 min	Startup overhead is the dominant factor; only marginally faster than scheduled
Partition projection (no crawler)	0 min	New S3 partition is immediately queryable; no crawl needed
Manual `add_partition` API call	<1 s	Register the partition directly — fast but requires caller to know the schema

For MCP log pipelines where Kinesis Firehose delivers hourly S3 partitions and the SLA for Athena query freshness is 30 minutes, a scheduled crawler at 15-minute intervals is sufficient. For pipelines where freshness SLA is under 5 minutes, skip the crawler and use glue.batch_create_partition() called from the Firehose delivery Lambda after each S3 write.