AWS Glue for MCP Servers: Three ETL Pipeline Patterns for Worker Sizing, Incremental Processing, and Data Quality Enforcement

The core mental model: Glue is a Spark job manager, not a streaming pipeline

Before the three patterns, it helps to be precise about what Glue is and where it sits relative to other AWS data primitives:

Glue ETL is not a replacement for Kinesis or Lambda event-driven patterns. It's the right tool when you have batch data (accumulated since the last job run), need Spark's parallel processing across workers, and want managed infrastructure without running EMR clusters. The minimum viable Glue ETL pipeline for MCP server logs has four steps: read from S3 (or JDBC) into a DynamicFrame, apply transforms (flatten nested tool_call fields, resolve choice types), run a Data Quality ruleset, and write the clean rows to an output S3 path — with bookmarks tracking which objects were processed so the next run only reads new data.

Pattern 1: The ETL job configuration triad — worker type, bookmarks, and DynamicFrame strategy

Three job-level configuration decisions determine most of the cost and correctness behavior of a Glue ETL job: which worker type to allocate, whether bookmarks are enabled, and when to convert a DynamicFrame to a DataFrame. Teams frequently get all three wrong in the same job.

Worker type selection — G.025X is usually right for MCP log pipelines

Glue charges by DPU-hour (Data Processing Unit-hour). Choosing the wrong worker type is the largest single cost lever in Glue ETL.

A typical MCP server generates tool call logs in the range of hundreds of MB to a few GB per day. A daily Glue job that flattens and deduplicates that day's logs, resolves choice types, and writes Parquet belongs on G.025X with 5–10 workers — not G.1X. The G.025X worker formerly called "Standard 2 DPU" runs the same GlueVersion 4.0 Spark runtime; the only difference is the fraction-DPU billing. Profile with Spark UI (enabled via --enable-spark-ui) before committing to G.1X.

import boto3

glue = boto3.client("glue")

glue.create_job(
    Name="mcp-log-transform",
    Role="arn:aws:iam::123456789012:role/GlueJobRole",
    Command={
        "Name": "glueetl",            # "glueetl" = Spark; "pythonshell" = Python shell
        "ScriptLocation": "s3://my-scripts/mcp-log-transform.py",
        "PythonVersion": "3",
    },
    GlueVersion="4.0",                # Always pin to latest — Glue 4.0 = Spark 3.3, Python 3.10
    WorkerType="G.025X",              # Start here; step up only if Spark UI shows OOM or high GC
    NumberOfWorkers=10,
    DefaultArguments={
        "--job-bookmark-option": "job-bookmark-enable",
        "--enable-metrics": "true",
        "--enable-spark-ui": "true",
        "--spark-event-logs-path": "s3://my-logs/spark-ui/",
        "--enable-continuous-cloudwatch-log": "true",
    },
    MaxRetries=0,     # Don't auto-retry — bookmarks make duplicate processing risky
    Timeout=120,      # Minutes; default 2880 (48 h) wastes DPUs on hung jobs
)

Two flags worth calling out: MaxRetries=0 is intentional. With bookmarks enabled, an auto-retry after a partial run creates a race condition — if the job wrote some output before failing, the retry re-reads the same input (bookmark only advances on success via job.commit()) and potentially writes duplicates. Set MaxRetries to 0, add an alarm on job failure, and let the on-call operator decide whether to retry manually after inspecting what was partially written. The Timeout=120 cap prevents a hung Spark job from consuming DPUs for 48 hours — the default if you leave it unset.

Job bookmarks — incremental processing and the job.commit() gate

Without bookmarks, every job run scans all input data from the beginning. With bookmarks enabled, Glue tracks which S3 objects (by ETag and modification timestamp) and JDBC rows (by bookmark key column value) have already been processed, and skips them on subsequent runs.

How S3 bookmarks work. Glue records the ETag and modification time of every S3 object read in a successful run. The next run only reads objects with a newer modification time than the high-water mark. This requires that your S3 source uses date-partitioned prefixes (e.g., s3://bucket/logs/year=2026/month=09/day=24/) so new data lands in new prefixes rather than overwriting existing objects. If your pipeline appends to an existing S3 object, Glue cannot distinguish new bytes from old and will re-read the entire object on the next run.

How JDBC bookmarks work. Glue uses a monotonically increasing bookmark key column (typically created_at, id, or updated_at) to track the maximum value seen in the last successful run. The next run applies a WHERE bookmark_col > last_seen_max predicate before pulling rows. You specify the key via the --job-bookmark-keys job argument and the sort order via --job-bookmark-keys-sorting-order.

The job.commit() gate — the most important call in any Glue script. Glue does not advance the bookmark automatically when the job run succeeds at the infrastructure level. It advances the bookmark only when your script explicitly calls job.commit(). If your script writes output but never calls job.commit(), the next run starts from the same high-water mark and re-reads all the same input, producing duplicate rows in the target.

import sys
from awsglue.transforms import *
from awsglue.utils import getResolvedOptions
from pyspark.context import SparkContext
from awsglue.context import GlueContext
from awsglue.job import Job

args = getResolvedOptions(sys.argv, ["JOB_NAME"])
sc = SparkContext()
glueContext = GlueContext(sc)
spark = glueContext.spark_session
job = Job(glueContext)
job.init(args["JOB_NAME"], args)        # Initialize bookmark tracking

# Read from S3 — bookmark filters out already-processed objects automatically
dyf = glueContext.create_dynamic_frame.from_options(
    format_options={"jsonPath": "$", "multiline": False},
    connection_type="s3",
    format="json",
    connection_options={
        "paths": ["s3://mcp-data/logs/raw_tool_calls/"],
        "recurse": True,
    },
    transformation_ctx="dyf_read",     # transformation_ctx is required for bookmarks to work
)

# Apply transforms — keep as DynamicFrame for schema-flexible operations
dyf_resolved = dyf.resolveChoice(specs=[("duration_ms", "cast:long"), ("error", "cast:boolean")])
dyf_dropped = dyf_resolved.drop_nulls()

# Convert to DataFrame ONLY if you need a Spark operation not available on DynamicFrame
# df = dyf_resolved.toDF()
# ... Spark-only operations here ...
# dyf_resolved = DynamicFrame.fromDF(df, glueContext, "back_to_dyf")

# Write to output S3 path
glueContext.write_dynamic_frame.from_options(
    frame=dyf_dropped,
    connection_type="s3",
    format="parquet",
    connection_options={
        "path": "s3://mcp-data/transformed/tool_calls/",
        "partitionKeys": ["year", "month", "day"],
    },
    transformation_ctx="dyf_write",
)

# CRITICAL: advance the bookmark — without this, the next run reprocesses everything
job.commit()

The transformation_ctx parameter on both the read and write frames is also required for bookmark tracking — Glue uses it as a key to store bookmark state per logical operation. Omitting transformation_ctx disables bookmarking for that operation even when --job-bookmark-option job-bookmark-enable is set at the job level.

DynamicFrame-first strategy — when to convert and when not to

Glue's DynamicFrame is a Spark Dataset wrapper that adds Glue-specific schema flexibility: it can hold fields with multiple types in the same column (called "choice types") without failing the schema inference, which is common in semi-structured MCP event payloads where duration_ms might be an integer in some records and a string in others. DataFrames require a fixed schema; DynamicFrames don't.

The correct strategy is: start as a DynamicFrame, stay as a DynamicFrame through all schema-flexible operations (resolveChoice, drop_nulls, unnest_columns, apply_mapping), then convert to DataFrame only when you need a Spark-specific operation that DynamicFrame doesn't support (complex joins, window functions, Spark ML), and then convert back to DynamicFrame before writing to the Glue Catalog. Converting early loses the choice-type safety net and forces you to handle schema mismatches manually in Spark.

# DynamicFrame-first strategy for MCP tool call log ETL

# Step 1: Resolve choice types before doing anything else
dyf_resolved = dyf.resolveChoice(specs=[
    ("duration_ms", "cast:long"),
    ("payload", "make_struct"),     # nested fields with varying schemas → struct
    ("error_code", "cast:int"),
])

# Step 2: Apply Glue-native transforms (stays as DynamicFrame)
dyf_clean = dyf_resolved.drop_nulls()  # drop rows where any field is null
dyf_flat = dyf_clean.unnest_columns()  # flatten nested structs one level

# Step 3: Convert to DataFrame ONLY for Spark-specific operation
# (e.g., windowed deduplication by session + timestamp)
df = dyf_flat.toDF()
from pyspark.sql import Window
from pyspark.sql.functions import row_number
w = Window.partitionBy("session_id").orderBy(df["timestamp_ms"].desc())
df_deduped = df.withColumn("rn", row_number().over(w)).filter("rn = 1").drop("rn")

# Step 4: Convert back to DynamicFrame for Catalog write
from awsglue.dynamicframe import DynamicFrame
dyf_final = DynamicFrame.fromDF(df_deduped, glueContext, "dyf_final")

# Step 5: Write via DynamicFrame — gets Catalog schema update automatically
glueContext.write_dynamic_frame.from_catalog(
    frame=dyf_final,
    database="mcp_analytics",
    table_name="deduped_tool_calls",
    transformation_ctx="dyf_catalog_write",
)

One additional Glue ETL optimization worth adding to every MCP log pipeline: pushdown predicates. If your S3 source is Hive-partitioned (e.g., year=2026/month=09/day=24/), you can tell Glue to only scan specific partition directories without reading all objects first:

# Pushdown predicate — reduces S3 scan to only today's partition
dyf = glueContext.create_dynamic_frame.from_catalog(
    database="mcp_logs",
    table_name="raw_tool_calls",
    push_down_predicate="(year == '2026' and month == '09' and day == '24')",
    transformation_ctx="dyf_pushdown",
)
# Note: pushdown predicate syntax is a Python expression string — NOT SQL.
# Only partition columns (from the catalog partition keys) are available.
# Regular columns cannot be used in pushdown predicates.

The pushdown predicate filter is evaluated against the catalog partition keys before any data is read from S3. For a job that runs daily on only the previous day's partition, this reduces scan cost to 1/365th of scanning the entire prefix.

Pattern 2: The catalog-as-contracts layer — partition projection, schema version health, and Hive compatibility

The Glue Data Catalog is a managed metadata repository — a Hive-compatible metastore — that stores the schema definitions and partition maps for your S3-backed and JDBC-backed data stores. For MCP server pipelines, the catalog is the schema contract layer that lets Glue ETL jobs, Athena queries, and EMR Spark jobs all read from the same table definition without each maintaining its own schema copy. Three catalog capabilities matter in practice: partition projection (the correct way to handle date-partitioned S3 tables), schema version count as a pipeline health signal, and Hive metastore compatibility.

Partition projection — eliminate the crawler for date-partitioned tables

The naive approach to keeping Athena and Glue ETL aware of new daily partitions is to run a Glue crawler on a schedule. Crawlers have two problems for date-partitioned MCP log tables: they have a 5–10 minute startup overhead even when event-triggered, and they create a new schema version in the catalog every time they detect a schema change — which can accumulate thousands of versions for volatile source schemas.

Partition projection is the correct solution for any table with a predictable partitioning scheme. With projection enabled, Athena and Glue ETL compute partition paths from the projection configuration rather than looking up partition metadata in the catalog. New S3 prefixes matching the projection formula are immediately queryable — no crawler run required, no partition registration call required, and no 5-minute lag.

import boto3

glue = boto3.client("glue")

# Create a table with partition projection for daily MCP log partitions
# Partition scheme: s3://mcp-data/logs/dt=YYYY-MM-DD/
glue.create_table(
    DatabaseName="mcp_logs",
    TableInput={
        "Name": "raw_tool_calls",
        "StorageDescriptor": {
            "Columns": [
                {"Name": "server_id", "Type": "string"},
                {"Name": "tool_name", "Type": "string"},
                {"Name": "duration_ms", "Type": "bigint"},
                {"Name": "error", "Type": "boolean"},
                {"Name": "timestamp_ms", "Type": "bigint"},
            ],
            "Location": "s3://mcp-data/logs/",
            "InputFormat": "org.apache.hadoop.mapred.TextInputFormat",
            "OutputFormat": "org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat",
            "SerdeInfo": {
                "SerializationLibrary": "org.openx.data.jsonserde.JsonSerDe",
                "Parameters": {"serialization.format": "1"},
            },
        },
        "PartitionKeys": [{"Name": "dt", "Type": "date"}],
        "Parameters": {
            # Enable partition projection — all partition lookup bypasses catalog
            "projection.enabled": "true",
            # Define the dt partition as an injection-format date range
            "projection.dt.type": "date",
            "projection.dt.format": "yyyy-MM-dd",
            "projection.dt.range": "2026-01-01,NOW",
            "projection.dt.interval": "1",
            "projection.dt.interval.unit": "DAYS",
            # S3 path template — ${dt} is replaced with the partition value
            "storage.location.template": "s3://mcp-data/logs/dt=${dt}/",
        },
        "TableType": "EXTERNAL_TABLE",
    }
)

Partition projection completely eliminates the need to run ALTER TABLE ADD PARTITION or trigger a crawler when new daily data arrives. When Athena runs SELECT * FROM raw_tool_calls WHERE dt = '2026-09-24', it computes the S3 path from the template and reads directly — no catalog partition metadata required. The projection.dt.range value NOW means the projection automatically includes today's date as the upper bound, so new partitions are immediately accessible after the job writes data to the dt=YYYY-MM-DD prefix.

The one limitation: partition projection doesn't work for unpredictable partitioning schemes — if your partition keys are tenant IDs, server IDs, or other values you can't enumerate in advance, you still need crawlers or explicit partition registration. For date-partitioned time-series tables (which cover most MCP log pipelines), projection is always the right choice.

Schema version count as a pipeline health signal

Every time a crawler detects a schema change and updates a table, Glue creates a new schema version. Old versions are kept indefinitely. The GetTableVersions API returns all schema versions with their timestamps — a table that has accumulated hundreds or thousands of versions in a short period indicates that the source schema is unstable.

import boto3

glue = boto3.client("glue")

# Check schema version count for a table
paginator = glue.get_paginator("get_table_versions")
versions = []
for page in paginator.paginate(DatabaseName="mcp_logs", TableName="raw_tool_calls"):
    versions.extend(page["TableVersions"])

print(f"Schema version count: {len(versions)}")
print(f"Oldest version: {versions[-1]['Table']['UpdateTime']}")
print(f"Latest version: {versions[0]['Table']['UpdateTime']}")

# High version count (>50 in a month) = schema drift in source data
# Inspect the diffs to understand what's changing
if len(versions) > 50:
    v1 = versions[0]["Table"]["StorageDescriptor"]["Columns"]
    v2 = versions[1]["Table"]["StorageDescriptor"]["Columns"]
    v1_cols = {c["Name"]: c["Type"] for c in v1}
    v2_cols = {c["Name"]: c["Type"] for c in v2}
    added = set(v1_cols) - set(v2_cols)
    removed = set(v2_cols) - set(v1_cols)
    type_changes = {k for k in v1_cols if k in v2_cols and v1_cols[k] != v2_cols[k]}
    print(f"Latest change: +{added} -{removed} type_changes={type_changes}")

For MCP server log tables, a healthy schema version count is 0–10 (schema defined once at table creation, crawlers run infrequently or not at all with partition projection). A high version count (50+) means the source JSON structure is changing frequently — tool definitions are being added or removed, field types are shifting, or the serialization layer is inconsistent. Track this as a CloudWatch metric and alarm when version count growth rate exceeds 1 per day.

Hive metastore compatibility — one table definition for Glue ETL, Athena, and EMR

The Glue Data Catalog implements the Apache Hive metastore API, which means any Hive-compatible engine can use the same table definitions. For MCP server teams this is valuable: the same mcp_logs.raw_tool_calls table can be queried by Athena (ad-hoc SQL), Glue ETL (batch transforms), EMR Spark (large-scale analytics), and Presto/Trino — all pointing to the same S3 data with the same schema.

To use the Glue Data Catalog as the metastore for an EMR cluster:

# EMR cluster configuration to use Glue Catalog as Hive metastore
{
  "Classification": "hive-site",
  "Properties": {
    "hive.metastore.client.factory.class":
      "com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory"
  }
}

# For Spark on EMR (or Glue ETL scripts), set SparkConf:
# spark.hadoop.hive.metastore.client.factory.class =
#   com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory

# Then in Spark:
# spark.sql("SELECT * FROM mcp_logs.raw_tool_calls WHERE dt = '2026-09-24'")
# — same table, same data, same schema as Athena queries

GetTable latency and caching. The Glue Catalog API adds 20–100 ms per GetTable call. Athena caches table metadata within a query (one call per table per query). Glue ETL scripts that call glueContext.create_dynamic_frame.from_catalog() inside a loop will pay 20–100 ms per iteration. The fix: retrieve the table definition once at script startup and reuse it, or use a module-scope TTL cache with a 5-minute expiry so that repeated calls within the job run hit the cache rather than the catalog API.

Pattern 3: Data pipeline reliability — the self-referential SG trap, JDBC_ENFORCE_SSL, and QUARANTINE mode

Three reliability and security issues account for most Glue production incidents: Glue jobs that hang silently at startup because of a missing security group rule, JDBC connections that transmit credentials in plaintext because the default behavior is insecure, and Data Quality pipelines that halt on bad rows when they should quarantine them. Each is a configuration detail that's easy to miss and whose failure mode is non-obvious.

The self-referential security group rule — the silent hang at startup

When a Glue ETL job uses a Connection (required for any JDBC data source inside your VPC), Glue places an Elastic Network Interface (ENI) in the subnet specified in the connection's PhysicalConnectionRequirements. The Spark driver and executor nodes communicate with each other over this ENI. For that intra-cluster communication to work, the security group assigned to the ENI must allow all inbound TCP from itself — a rule where both the source and the destination are the same security group ID.

If this rule is missing, the Glue job appears to start successfully but hangs in the RUNNING state with no logged error. The Spark driver launches, the executors try to connect back to the driver, the inbound TCP connection is rejected by the security group, and the job sits at RUNNING until it hits the Timeout parameter (which defaults to 2,880 minutes — 48 hours — if you don't set it). The only signal is the Spark UI showing zero tasks ever executed.

import boto3

ec2 = boto3.client("ec2")

GLUE_SG_ID = "sg-0abc123def456789"

# Add self-referential inbound rule — all TCP from the same security group
ec2.authorize_security_group_ingress(
    GroupId=GLUE_SG_ID,
    IpPermissions=[
        {
            "IpProtocol": "tcp",
            "FromPort": 0,
            "ToPort": 65535,
            "UserIdGroupPairs": [
                {
                    "GroupId": GLUE_SG_ID,    # Source = same SG (self-referential)
                    "Description": "Glue intra-cluster Spark communication",
                }
            ],
        }
    ],
)

# Also add the target data store's security group inbound rule
# (the RDS security group must allow inbound on port 5432 from the Glue SG)
ec2.authorize_security_group_ingress(
    GroupId="sg-rds-postgres-sg",   # RDS instance's security group
    IpPermissions=[
        {
            "IpProtocol": "tcp",
            "FromPort": 5432,
            "ToPort": 5432,
            "UserIdGroupPairs": [
                {
                    "GroupId": GLUE_SG_ID,
                    "Description": "Allow Glue ETL to connect to RDS",
                }
            ],
        }
    ],
)

The two-SG setup (Glue SG with self-referential rule + RDS SG allowing inbound from Glue SG) is the complete networking requirement for a Glue JDBC job running inside a VPC. Both rules must be in place before the Glue Connection test will pass for JDBC types, and before any job using that connection will start successfully.

JDBC_ENFORCE_SSL — the default-off security trap

Glue JDBC connections do not enforce SSL encryption by default. A Glue job connecting to an RDS PostgreSQL instance inside your VPC will, by default, transmit the database username and password over a plaintext JDBC connection. The assumption teams make — that VPC-internal traffic is already private — is correct for the network layer but does not protect against packet capture inside the VPC by a compromised instance, or against the connection being proxied through an unexpected path.

The fix is a single connection property: always set JDBC_ENFORCE_SSL: "true" when creating or updating a Glue Connection. Combined with storing credentials in AWS Secrets Manager (via SECRET_ID rather than hardcoded USERNAME/PASSWORD properties), this is the minimum security baseline for any Glue JDBC connection.

import boto3

glue = boto3.client("glue")

# Secure JDBC connection to RDS PostgreSQL
# Credentials in Secrets Manager, SSL enforced
glue.create_connection(
    ConnectionInput={
        "Name": "mcp-registry-rds-secure",
        "ConnectionType": "JDBC",
        "ConnectionProperties": {
            "JDBC_CONNECTION_URL": "jdbc:postgresql://mcp-registry.cluster-abc.us-east-1.rds.amazonaws.com:5432/mcp_registry",
            # Use Secrets Manager — not hardcoded USERNAME/PASSWORD
            # Glue retrieves credentials at runtime; rotation is automatic
            "SECRET_ID": "arn:aws:secretsmanager:us-east-1:123456789012:secret:glue-rds-creds",
            # CRITICAL: enforce SSL — default is plaintext
            "JDBC_ENFORCE_SSL": "true",
            "JDBC_CONNECTION_TIMEOUT": "10",
        },
        "PhysicalConnectionRequirements": {
            "SubnetId": "subnet-0abc123def456789",
            "SecurityGroupIdList": ["sg-0abc123def456789"],  # Must have self-referential rule
            "AvailabilityZone": "us-east-1a",
        },
    }
)

# The Secrets Manager secret must be JSON with "username" and "password" keys:
# { "username": "glue_etl_user", "password": "..." }
# When RDS rotates the password, Glue picks up the new credentials on the next run
# without any connection update required — this is the primary value of SECRET_ID

One additional check: the Glue console's "Test connection" button only works for JDBC connection types. For MongoDB, Kafka, and NETWORK connections, the test always returns success regardless of whether the endpoint is actually reachable. For those types, test connectivity by running a minimal Glue job that reads one row and exits — the only reliable end-to-end connectivity test for non-JDBC connections.

QUARANTINE mode — the production default for Data Quality

Glue Data Quality lets you define DQDL (Data Quality Definition Language) rules that run inline in your ETL job and evaluate each rule against the DynamicFrame. Three evaluation modes exist, and the wrong default choice is the most common mistake:

EVALUATE is the wrong choice for production because it never affects data flow — your pipeline writes all rows (including the bad ones) to the target regardless of what the quality rules say. FAIL is too blunt for most MCP log pipelines — a handful of malformed rows should not halt the job and prevent all the valid records from being processed. QUARANTINE is the production default: it separates the bad rows into a quarantine S3 path for inspection without blocking the good rows from reaching the analytics table.

from awsglue.transforms import EvaluateDataQuality

# Define DQDL ruleset for MCP tool call events
dqdl_ruleset = """
    Rules = [
        Completeness "server_id" >= 0.99,
        Completeness "tool_name" >= 0.99,
        IsComplete "timestamp_ms",
        IsPositive "duration_ms",
        Uniqueness "event_id" >= 0.999,
        RowCount >= 1000,
        ColumnValues "duration_ms" <= 60000
    ]
"""

# QUARANTINE mode — split into passing and failing DynamicFrames
result = EvaluateDataQuality.apply(
    frame=dyf_clean,
    ruleset=dqdl_ruleset,
    publishing_options={
        "dataQualityEvaluationContext": "mcp_tool_calls_quality",
        "enableDataQualityCloudWatchMetrics": True,
        "enableDataQualityResultsPublishing": True,
    },
    additional_options={
        "performanceTuning.caching": "CACHE_NOTHING",
    },
    output="rowLevelOutcomes",  # Required for QUARANTINE mode row routing
)

# Get the split frames from the result
dyf_passing = result.select_fields(["originalData"]).filter(
    lambda x: x.getField("DataQualityEvaluationResult") == "Passed"
)
dyf_failing = result.select_fields(["originalData"]).filter(
    lambda x: x.getField("DataQualityEvaluationResult") == "Failed"
)

# Write passing rows to the analytics target
glueContext.write_dynamic_frame.from_options(
    frame=dyf_passing,
    connection_type="s3",
    format="parquet",
    connection_options={"path": "s3://mcp-data/analytics/tool_calls/"},
    transformation_ctx="write_passing",
)

# Write failing rows to quarantine for inspection
glueContext.write_dynamic_frame.from_options(
    frame=dyf_failing,
    connection_type="s3",
    format="json",
    connection_options={"path": "s3://mcp-data/quarantine/tool_calls/"},
    transformation_ctx="write_quarantine",
)

job.commit()

Anomaly detection warm-up requirement. Glue Data Quality includes anomaly detection rules that automatically threshold based on historical values (e.g., flag a RowCount that is more than 3 standard deviations below the running average). These rules need a minimum of 10 prior successful evaluations to build a stable baseline. If you add an anomaly detection rule to a brand-new pipeline, the first 10 runs produce no signal and trivially pass — build the baseline by running the pipeline in EVALUATE mode first, then switch to QUARANTINE after 10 runs.

# Anomaly detection rule — only add after 10+ prior runs have built the baseline
Rules = [
    # These rules work from first run:
    Completeness "server_id" >= 0.99,
    IsPositive "duration_ms",

    # These require 10+ prior evaluations before they produce signal:
    AnomalyDetection "RowCount" BETWEEN 0.7 AND 1.3,      # flag if row count drops >30%
    AnomalyDetection "Completeness" "server_id" BETWEEN 0.95 AND 1.0,
    AnomalyDetection "Mean" "duration_ms" BETWEEN 0.5 AND 2.0,  # flag if avg latency doubles
]

The AnomalyDetection threshold type (BETWEEN 0.7 AND 1.3) specifies the acceptable ratio of the current metric to the historical baseline — 0.7 means no more than 30% below the baseline, 1.3 means no more than 30% above. The 3σ default is configurable via the threshold bounds. Use tighter bounds (0.85–1.15) for metrics that should be stable (RowCount on a predictable daily pipeline) and looser bounds (0.5–2.0) for metrics with high natural variance (duration_ms for diverse tool types).

Consolidated failure modes reference

The five most common Glue failures in MCP server data pipelines, along with their root causes and fixes:

Putting it together: the production MCP log ETL pipeline

A complete MCP server log pipeline that applies all three patterns looks like this in terms of resource configuration and script structure:

Job configuration (via boto3 or CDK):

• WorkerType: G.025X, NumberOfWorkers: 10 — start here, profile before stepping up
• --job-bookmark-option: job-bookmark-enable — incremental reads from first run
• MaxRetries: 0, Timeout: 120 — prevent silent hangs and double-processing on failure
• --enable-spark-ui, --enable-continuous-cloudwatch-log — observability from day one

Catalog configuration:

• Partition projection on the source table — no crawlers for date-partitioned paths
• One database per product domain (mcp_logs, mcp_analytics) — shared metastore across Glue ETL, Athena, and EMR
• Schema version count monitored via CloudWatch metric — alert at >1 new version per day

Connection configuration:

• JDBC_ENFORCE_SSL: true on every JDBC connection — no plaintext credentials in transit
• SECRET_ID pointing to Secrets Manager — credentials rotatable without connection update
• Self-referential security group rule (all TCP from same SG) — required before any VPC-connected job will start

Script structure:

1. Read from catalog with pushdown predicate (last partition only)
2. ResolveChoice and apply Glue-native transforms (stay as DynamicFrame)
3. Apply Data Quality ruleset in QUARANTINE mode — passing rows continue, failing rows to quarantine S3 path
4. Convert to DataFrame only if needed for a complex join or window function
5. Convert back to DynamicFrame and write to catalog output table
6. job.commit() — advance bookmark, final line of script

Each of the five Glue guide pages goes deeper on individual components: ETL jobs covers the full worker type comparison and bookmark configuration options; the Data Catalog covers multi-column partition projection and GetTable latency caching; Crawlers covers the schema change policy options (UPDATE_IN_DATABASE vs LOG vs FAIL-via-alarm) and when crawlers are still the right tool; Connections covers all connection types and the connection test limitations; and Data Quality covers the full DQDL rule reference and anomaly detection configuration.

The patterns compound: a job with G.025X workers, bookmarks enabled, and QUARANTINE-mode data quality running on a partition-projected table over a Secrets-Manager-backed SSL connection is not a complex architecture — it's five configuration choices made correctly. The individual mistakes (wrong worker type, missing job.commit(), crawler dependency, plaintext JDBC, EVALUATE mode data quality) each cause a different failure mode, but they share a common root: they're all defaults that look correct in a development environment and break silently in production.

Monitoring AliveMCP with Glue

The AliveMCP platform monitors MCP servers for availability, response time, and schema drift. Glue ETL is the backend that processes raw probe logs, computes uptime percentages per server, and populates the analytics tables that back the public status dashboard. The three patterns described here — G.025X worker sizing for small daily batches, partition projection for zero-latency new-day queries, and QUARANTINE mode to keep the analytics table clean without halting the pipeline on malformed probe responses — are the same patterns we run in production.

If you're running MCP servers yourself and want uptime monitoring, alerting, and the kind of schema drift detection that catches breaking tool definition changes before your agents do, AliveMCP has a free tier for up to three servers.