AWS Glue Connections for MCP Servers

Connection types and when to use each

Creating a JDBC connection with SSL enforcement

import boto3

glue = boto3.client("glue")

# Create a JDBC connection to an RDS PostgreSQL instance
# Credentials are stored in Secrets Manager — Glue retrieves them at runtime
glue.create_connection(
    ConnectionInput={
        "Name": "mcp-registry-rds-postgres",
        "ConnectionType": "JDBC",
        "ConnectionProperties": {
            "JDBC_CONNECTION_URL": "jdbc:postgresql://mcp-registry.cluster-xyz.us-east-1.rds.amazonaws.com:5432/mcp_registry",
            # Use Secrets Manager — NOT hardcoded username/password
            "SECRET_ID": "arn:aws:secretsmanager:us-east-1:123456789012:secret:mcp-rds-creds",
            # CRITICAL: enforce SSL — default is plaintext
            "JDBC_ENFORCE_SSL": "true",
            # For RDS with a custom CA, specify the root CA cert path on the Glue worker
            # (or use RDS default cert — valid for most use cases)
            "JDBC_CONNECTION_TIMEOUT": "10",  # Connection timeout in seconds
        },
        "PhysicalConnectionRequirements": {
            "SubnetId": "subnet-0abc123def456789",       # Subnet where Glue places its ENI
            "SecurityGroupIdList": ["sg-0abc123def456789"],  # Must have self-referential rule
            "AvailabilityZone": "us-east-1a",
        },
    }
)

Secret format: the Secrets Manager secret referenced by SECRET_ID must be a JSON object with username and password keys:

{
  "username": "glue_etl_user",
  "password": "secure-password-here"
}

When credentials are stored in Secrets Manager and the RDS password is rotated, the Glue connection automatically picks up the new credentials on the next job run — no connection update required. This is the primary reason to use SECRET_ID over hardcoded credentials in the connection properties.

VPC networking — the self-referential security group rule

When a Glue ETL job runs in your VPC (required for JDBC connections to resources inside the VPC), Glue places an ENI in the subnet you specify. The Spark driver and executor nodes communicate over this ENI. The security group attached to the ENI must allow this intra-cluster communication.

import boto3

ec2 = boto3.client("ec2")

# Step 1: Create or identify the security group for Glue workers
# Step 2: Add the self-referential inbound rule — ALL TCP from the same security group
ec2.authorize_security_group_ingress(
    GroupId="sg-0abc123def456789",  # The SG assigned to the Glue connection
    IpPermissions=[
        {
            "IpProtocol": "tcp",
            "FromPort": 0,
            "ToPort": 65535,
            "UserIdGroupPairs": [
                {
                    "GroupId": "sg-0abc123def456789",  # Source = same SG (self-referential)
                    "Description": "Glue intra-cluster communication",
                }
            ],
        }
    ],
)

# Step 3: Allow outbound to the RDS instance's security group on port 5432
ec2.authorize_security_group_egress(
    GroupId="sg-0abc123def456789",
    IpPermissions=[
        {
            "IpProtocol": "tcp",
            "FromPort": 5432,
            "ToPort": 5432,
            "UserIdGroupPairs": [
                {
                    "GroupId": "sg-rds-0xyz789",  # RDS security group
                    "Description": "Glue to RDS PostgreSQL",
                }
            ],
        }
    ],
)

The RDS security group must also allow inbound TCP on port 5432 from the Glue security group. Without the self-referential rule in the Glue SG, the Glue job will appear to start (status: RUNNING) but hang indefinitely until it times out — no error is logged because the failure is at the Spark executor communication layer, not at the JDBC layer.

Using connections in ETL scripts and crawlers

# Use a Glue connection in an ETL script to read from RDS
from awsglue.context import GlueContext

glueContext = GlueContext(sc)

# Read from JDBC via the named connection
# Glue resolves credentials from the connection's SECRET_ID automatically
rds_data = glueContext.create_dynamic_frame.from_options(
    connection_type="postgresql",
    connection_options={
        "useConnectionProperties": "true",
        "connectionName": "mcp-registry-rds-postgres",
        "dbtable": "public.server_registry",
        # For large tables: push down a predicate to read only recent rows
        # (Glue JDBC push-down uses a WHERE clause, not partition-level filtering)
        "hashfield": "id",        # Split large reads across multiple Spark partitions
        "hashpartitions": "10",
    },
    transformation_ctx="rds_data",
)

# Configure a crawler to use a Glue connection for JDBC table discovery
glue.create_crawler(
    Name="mcp-rds-schema-crawler",
    Role="arn:aws:iam::123456789012:role/GlueCrawlerRole",
    DatabaseName="mcp_registry",
    Targets={
        "JdbcTargets": [
            {
                "ConnectionName": "mcp-registry-rds-postgres",
                "Path": "mcp_registry/%",    # Crawl all tables in mcp_registry schema
                "Exclusions": ["pg_*", "information_schema.*"],
            }
        ]
    },
)

Connection test limitations

For Kafka, MongoDB, and NETWORK connections, validate connectivity by running a minimal Glue job that reads one row from the source. Wrap the read in a try/except and log the error to CloudWatch to surface connection failures without running the full ETL workload.

Common connection failures