AWS Glue ETL Jobs for MCP Servers

Worker types and DPU model

Glue charges by DPU-hour. Choosing the wrong worker type is the single largest cost lever in Glue ETL.

For a typical MCP server log pipeline — reading JSON from S3, flattening nested tool_call fields, writing Parquet — G.1X with 5–10 workers is the right starting point. Profile with Spark UI (available via Glue job run details) and scale down if executors are idle.

import boto3

glue = boto3.client("glue")

# Create a Glue ETL job for MCP tool-call log transformation
glue.create_job(
    Name="mcp-log-transform",
    Role="arn:aws:iam::123456789012:role/GlueJobRole",
    Command={
        "Name": "glueetl",          # "glueetl" = Spark, "pythonshell" = Python shell
        "ScriptLocation": "s3://my-scripts/mcp-log-transform.py",
        "PythonVersion": "3",
    },
    GlueVersion="4.0",              # Always use latest — Glue 4.0 = Spark 3.3, Python 3.10
    WorkerType="G.1X",
    NumberOfWorkers=10,
    DefaultArguments={
        "--job-bookmark-option": "job-bookmark-enable",
        "--enable-metrics": "true",
        "--enable-spark-ui": "true",
        "--spark-event-logs-path": "s3://my-logs/spark-ui/",
        "--enable-continuous-cloudwatch-log": "true",
    },
    MaxRetries=0,                   # Don't auto-retry — bookmarks make duplicate processing risky
    Timeout=120,                    # Minutes; default 2880 (48h) wastes DPUs on hung jobs
)

Job bookmarks — incremental processing

Without bookmarks, every job run scans all input data from the start. With bookmarks enabled, Glue tracks which S3 objects and JDBC rows have already been processed and skips them on subsequent runs.

How S3 bookmarks work: Glue records the S3 object ETags and modification timestamps it processed. On the next run it only reads objects newer than the last successful run's high-water mark. This requires that your S3 source uses date-partitioned prefixes (e.g., s3://bucket/logs/year=2026/month=09/day=24/) so new partitions are added rather than existing objects modified — modified objects re-trigger processing.

How JDBC bookmarks work: Glue uses a bookmark key column (a monotonically increasing column like created_at or id) to track the maximum value seen. Next run it applies a WHERE bookmark_col > last_seen_max filter. You must specify the bookmark key via --job-bookmark-keys and the sort order via --job-bookmark-keys-sorting-order.

# PySpark Glue script — MCP log transform with bookmark-aware S3 read
import sys
from awsglue.transforms import *
from awsglue.utils import getResolvedOptions
from pyspark.context import SparkContext
from awsglue.context import GlueContext
from awsglue.job import Job

args = getResolvedOptions(sys.argv, ["JOB_NAME"])
sc = SparkContext()
glueContext = GlueContext(sc)
spark = glueContext.spark_session
job = Job(glueContext)
job.init(args["JOB_NAME"], args)

# Read from S3 — bookmark tracks which objects are new since last run
source_df = glueContext.create_dynamic_frame.from_options(
    connection_type="s3",
    connection_options={
        "paths": ["s3://mcp-logs/raw/"],
        "recurse": True,
        # groupFiles: combine small files into larger partitions to reduce task overhead
        "groupFiles": "inPartition",
        "groupSize": "134217728",   # 128 MB per group
    },
    format="json",
    format_options={"withHeader": False, "multiline": True},
    transformation_ctx="source_df",  # Must match the bookmark context key
)

# ... transform ...

job.commit()  # Advances the bookmark — MUST be called or next run re-processes everything

Critical: job.commit() at the end of the script advances the bookmark. If your job fails after writing output but before commit(), the next run will re-process the same data — make sure your target write is idempotent (e.g., UPSERT by primary key rather than INSERT).

DynamicFrame vs DataFrame

Glue introduces DynamicFrame as a wrapper around Spark's DataFrame with added capabilities for schema-flexible data. For MCP event logs — which often have inconsistent field presence (some tool calls have error, others don't; some have nested metadata, others have a flat structure) — DynamicFrame provides key advantages.

# Flatten MCP tool call events — keep as DynamicFrame through schema operations
from awsglue.transforms import ApplyMapping, ResolveChoice, DropNullFields

# Resolve choice types: if "duration_ms" is sometimes int, sometimes string
resolved = ResolveChoice.apply(
    frame=source_df,
    choice="cast:long",  # All choice fields cast to long
    transformation_ctx="resolved",
)

# Flatten nested metadata — Glue's built-in transform, no Spark needed
flattened = resolved.unnest_columns(
    colName="metadata",
    transformation_ctx="flattened",
)

# Drop rows where required fields are null
cleaned = DropNullFields.apply(frame=flattened, transformation_ctx="cleaned")

# Only convert to DataFrame for a Spark operation (here: window function for session ID)
from pyspark.sql.functions import col, row_number
from pyspark.sql.window import Window

df = cleaned.toDF()
w = Window.partitionBy("session_id").orderBy("timestamp_ms")
df_with_seq = df.withColumn("call_seq", row_number().over(w))

# Convert back to DynamicFrame to write via Glue catalog
from awsglue.dynamicframe import DynamicFrame
output_dyf = DynamicFrame.fromDF(df_with_seq, glueContext, "output_dyf")

Pushdown predicates — avoid full partition scans

When reading from a partitioned S3 source (Hive-style partitions: s3://bucket/dt=2026-09-24/), Glue can push partition filters down to the S3 lister so it only reads matching partitions — without scanning all objects first. This is called a pushdown predicate.

# Read only today's MCP log partition — avoids listing and reading all historical partitions
from datetime import date

today = date.today().isoformat()  # "2026-09-24"

source_df = glueContext.create_dynamic_frame.from_catalog(
    database="mcp_logs",
    table_name="raw_tool_calls",
    push_down_predicate=f"(dt == '{today}')",
    transformation_ctx="source_df",
)

Pushdown predicate syntax uses Python expression syntax, not SQL. Columns must be partition columns (registered in the Glue Data Catalog partition metadata). Non-partition columns are not pushed down — Glue reads the files and then filters in Spark.

For date-range queries common in MCP log pipelines (last 7 days, this month), use in with a generated list or a range expression:

from datetime import date, timedelta

# Last 7 days
last_7 = [
    (date.today() - timedelta(days=i)).isoformat()
    for i in range(7)
]
predicate = "dt in (" + ",".join(f"'{d}'" for d in last_7) + ")"
# Results in: dt in ('2026-09-24','2026-09-23',...,'2026-09-18')

source_df = glueContext.create_dynamic_frame.from_catalog(
    database="mcp_logs",
    table_name="raw_tool_calls",
    push_down_predicate=predicate,
    transformation_ctx="source_df",
)

Glue Spark UI and job metrics

Enable the Spark UI and continuous CloudWatch logging to diagnose performance problems. The default Glue job run view only shows start/end time and status — the Spark UI shows executor utilization, stage DAG, task skew, and spill-to-disk metrics that are essential for tuning.

DefaultArguments={
    "--enable-spark-ui": "true",
    "--spark-event-logs-path": "s3://my-logs/spark-ui/mcp-log-transform/",
    "--enable-metrics": "true",
    "--enable-continuous-cloudwatch-log": "true",
    "--enable-continuous-cloudwatch-log-filter": "true",  # Suppress INFO spam, keep WARN/ERROR
}

Access the Spark UI from the Glue console: job run → Spark UI tab. Common patterns in MCP log pipelines:

• Few tasks completing, most pending: small file problem — use groupFiles: "inPartition" to combine files.
• One task takes 10× longer than others: partition skew — one tenant generates most events. Add a secondary sort key (e.g., tenant_id + "_" + hour) to re-partition before writing.
• Spill to disk on executors: insufficient memory for the sort/join — step up from G.1X to G.2X.

Common job failures and fixes