Guide · AWS RDS · Database Migrations

RDS Blue/Green Deployments for MCP Servers — Zero-Downtime Schema Changes, Switchover, Rollback

RDS Blue/Green Deployments create a managed staging copy (green) of your production database (blue) that continuously replicates changes from blue via binlog or logical replication — MCP server teams use it to test schema migrations, engine upgrades, or parameter group changes against a production data replica before switching live traffic to the new environment. Three operational realities matter: switchover is not instant — applications experience approximately 60–90 seconds of write interruption while RDS stops writes on blue, waits for green's replication lag to reach zero, then swaps the DB instance identifiers and endpoints; the application's connection pool sees the old endpoint DNS resolve to the new instance, but connections that were in-flight at the moment of the quorum stop are dropped; the green environment receives every write made to blue in real time, meaning any destructive change made on green (like truncating a table) does not appear on blue — green is a safe laboratory — but schema changes on green must be backward-compatible with the application version running against blue if you want the option to roll back after switchover; and Blue/Green Deployments support RDS MySQL, RDS PostgreSQL, RDS MariaDB, Aurora MySQL, and Aurora PostgreSQL as of 2024, but the feature requires the source cluster to have binlog enabled (MySQL/MariaDB) or logical replication enabled (PostgreSQL) before the deployment is created.

TL;DR

Create a blue/green deployment from the RDS console or CLI. AWS provisions a green copy of your cluster and sets up replication from blue to green. Run your schema migration on green and validate. Click "Switch over" (or call switchover-blue-green-deployment): RDS pauses writes on blue (~60–90 seconds), drains lag, swaps endpoints, then resumes writes on the new writer (formerly green). Rollback is simply deleting the green environment; blue was never modified.

How Blue/Green works

Under the hood, RDS provisions a complete replica of your cluster (same instance class, same parameter group, same VPC and subnet group) and creates a managed replication channel from blue (production) to green (staging):

MySQL/MariaDB: binlog replication. The source cluster must have binlog_format = ROW and binlog_retention_hours ≥ 24.
PostgreSQL/Aurora PostgreSQL: logical replication using pglogical. The source cluster's rds.logical_replication parameter must be set to 1 before creating the deployment.

Blue and green run simultaneously during the staging period. Green receives all writes made to blue in near-real-time (typically 1–10 ms lag for same-region). You can connect to green directly using its own endpoint to test your application against the migrated schema.

Phase	Blue state	Green state	Duration
Provisioning	Normal production writes	Snapshot restore + initial replication catch-up	Minutes to hours (depends on data size)
Staging (testing)	Normal production writes	Continuously replicating from blue; you apply schema changes here	As long as needed
Switchover	Write quorum stop (~60–90s downtime)	Lag drains; endpoints swap; green becomes new production	60–90 seconds
Post-switchover	Old blue available as read-only fallback; replication from new blue (old green) to old blue begins	Now the production writer	—

Prerequisites and setup

# For RDS PostgreSQL: enable logical replication (requires parameter group change + reboot)
aws rds modify-db-cluster \
  --db-cluster-identifier mcp-probe-db \
  --db-cluster-parameter-group-name mcp-probe-pg16-params \
  --apply-immediately

# mcp-probe-pg16-params must have rds.logical_replication = 1
# Verify:
aws rds describe-db-cluster-parameters \
  --db-cluster-parameter-group-name mcp-probe-pg16-params \
  --query "Parameters[?ParameterName=='rds.logical_replication']"

# For RDS MySQL: ensure binlog is enabled
aws rds describe-db-clusters \
  --db-cluster-identifier mcp-probe-db \
  --query "DBClusters[0].{BacktrackWindow:BacktrackWindow,EnabledCloudwatchLogsExports:EnabledCloudwatchLogsExports}"
# binlog is enabled by default on Aurora MySQL 3.x; check with:
# SELECT @@binlog_format, @@log_bin FROM dual;  (should return ROW, 1)

# Create the blue/green deployment
aws rds create-blue-green-deployment \
  --blue-green-deployment-name mcp-probe-bgd-v3-migration \
  --source arn:aws:rds:us-east-1:123456789012:cluster:mcp-probe-db \
  --target-engine-version "16.3" \           # optional: upgrade engine version in green
  --target-db-cluster-parameter-group-name mcp-probe-pg163-params \
  --tags Key=Purpose,Value=SchemaV3Migration

# Monitor provisioning status
aws rds describe-blue-green-deployments \
  --blue-green-deployment-identifier bgd-1234567890abcdef0 \
  --query "BlueGreenDeployments[0].{Status:Status,GreenClusterArn:GreenClusterArn}"

Running schema migrations on green

Connect to the green environment's writer endpoint (visible in the blue/green deployment details) and run your migrations. The green endpoint is separate from the blue endpoint — it does not serve any production traffic.

# Get the green endpoint
GREEN_ENDPOINT=$(aws rds describe-blue-green-deployments \
  --blue-green-deployment-identifier bgd-1234567890abcdef0 \
  --query "BlueGreenDeployments[0].SwitchoverDetails[?SourceMember==null].TargetMember" \
  --output text)

# Connect and run migration
psql "host=$GREEN_ENDPOINT dbname=mcp_probe user=admin sslmode=require" \
  -f migrations/v3_add_schema_version_column.sql

# Verify the migration
psql "host=$GREEN_ENDPOINT dbname=mcp_probe user=admin sslmode=require" \
  -c "SELECT column_name, data_type FROM information_schema.columns WHERE table_name='probe_results' ORDER BY ordinal_position;"

# Test your application against green by temporarily pointing a canary deployment
# at the green endpoint (keep the main application on blue)

Schema compatibility rule: if you want the option to roll back after switchover, the schema changes on green must be backward-compatible with the application code running against blue. An ALTER TABLE ADD COLUMN DEFAULT NULL is backward-compatible; an ALTER TABLE DROP COLUMN is not — if you roll back by deleting green (which restores the old blue as writer), the application that shipped schema-aware code for the dropped column will break.

Switchover execution

# Pre-switchover checklist:
# 1. Replication lag < 100ms consistently for 10+ minutes (check CloudWatch)
# 2. Application code deployed that handles the new schema on both blue and green
# 3. Maintenance window or low-traffic period selected
# 4. RDS Proxy (if used) will automatically reroute connections post-switchover

# Check current replication lag
aws cloudwatch get-metric-statistics \
  --namespace "AWS/RDS" \
  --metric-name "AuroraBinlogReplicaLag" \
  --dimensions Name=DBClusterIdentifier,Value=mcp-probe-db \
  --period 60 --statistics Maximum \
  --start-time $(date -u -d '10 minutes ago' +%Y-%m-%dT%H:%M:%SZ) \
  --end-time $(date -u +%Y-%m-%dT%H:%M:%SZ)

# Execute switchover (this blocks until complete or times out)
aws rds switchover-blue-green-deployment \
  --blue-green-deployment-identifier bgd-1234567890abcdef0 \
  --switchover-timeout 300   # maximum seconds to wait for lag drain; fail if exceeded

During switchover, RDS performs these steps in order:

Places a write quorum stop on the blue cluster (new writes are held, not rejected — connections see ~1–3 second hanging writes).
Waits for green's replication lag to reach 0.
Renames the DB instance and cluster identifiers — blue's identifiers move to the green instances, and vice versa. The writer endpoint DNS record now resolves to the former green instance.
Lifts the write quorum stop. Writes resume on the new writer (formerly green).

Total application-visible downtime: typically 60–90 seconds for Aurora, up to 3 minutes for large RDS MySQL instances due to binlog catch-up.

Using RDS Proxy with Blue/Green

If you have RDS Proxy in front of the cluster, the proxy transparently handles the endpoint swap. Applications connected to the proxy endpoint experience a brief connection interruption during switchover (their in-flight connections are dropped when the quorum stop fires), but they reconnect to the proxy endpoint — which now routes to the new writer — without any DNS or connection string changes. This is the recommended pattern for Lambda workloads:

# RDS Proxy automatically detects the endpoint swap and reroutes new connections
# to the new writer after switchover. No proxy configuration change needed.

# Verify post-switchover: proxy target should now point to the former green cluster
aws rds describe-db-proxies \
  --db-proxy-name mcp-probe-proxy \
  --query "DBProxies[0].Endpoint"

aws rds describe-db-proxy-targets \
  --db-proxy-name mcp-probe-proxy \
  --query "Targets[*].{Endpoint:Endpoint,Type:Type,State:State}"

AliveMCP probe-database migration pattern

AliveMCP uses Blue/Green Deployments for every schema migration to the probe results database. The migration workflow: create the deployment (green provisions overnight), run the Rails/Node migration script against the green endpoint the following morning, run the full integration test suite against a staging AliveMCP instance pointed at green, verify probe write throughput matches baseline (167 writes/second). Two hours before the scheduled maintenance window, check that lag is consistently below 50 ms. During the maintenance window (Saturday 02:00 UTC, lowest global MCP endpoint traffic), execute switchover-blue-green-deployment. The 90-second write interruption causes a gap in the uptime timeline for monitored endpoints — the probe workers reconnect automatically via RDS Proxy and backfill the gap with a "monitoring interrupted" status rather than reporting false outages. After switchover, the old blue environment stays available for 48 hours as a read-only fallback before deletion.

Common failure modes

Symptom	Cause	Fix
Green provisioning stuck in "PROVISIONING" for hours	Initial snapshot copy is large; a 500 GB database takes 2–4 hours to provision green. Or the source cluster has a heavy write load making initial replication catch-up slow.	Wait; monitor with `describe-blue-green-deployments`. Avoid creating deployments during bulk data imports on blue.
Switchover timeout: "replication lag did not drain to 0 within timeout"	Long-running transactions or high write throughput on blue prevents lag from reaching zero within the switchover timeout window	Increase `--switchover-timeout`; reduce write throughput before switchover (pause probe workers temporarily); terminate long-running transactions
"ERROR: logical replication is not enabled" when creating deployment	PostgreSQL source cluster does not have `rds.logical_replication = 1` in its parameter group	Update parameter group, apply immediately, and reboot the cluster (this requires a brief outage — schedule it before creating the deployment)
Application throws "column does not exist" after switchover	Schema on green dropped or renamed a column; old application code still expects the original schema on what is now the writer	Deploy application code that handles both old and new schema (expand-contract pattern) before executing the switchover; do not delete columns until all application code is updated
RDS Proxy continues routing to old writer after switchover	Proxy target is not updating; rare edge case if the proxy was created with a pinned instance ARN rather than the cluster ARN	Delete and recreate the proxy target pointing to the cluster ARN, not a specific instance ARN; instance ARNs change during switchover
Blue environment still appears in console after switchover and cannot be deleted	Post-switchover, the blue/green deployment resource must be deleted before the old instances can be terminated	Delete the blue/green deployment resource first (`delete-blue-green-deployment`), which releases the old blue instances; then optionally terminate them