MCP Server RDS Multi-AZ — failover impact on connection pools, DNS TTL, reconnect logic

The Multi-AZ failover timeline

RDS Multi-AZ maintains a hot standby in a different Availability Zone. When a failover event occurs (instance failure, AZ failure, or manual failover for maintenance), the following happens:

1. 0s: Primary becomes unavailable. Existing connections begin failing with TCP errors. New connection attempts fail.
2. ~20-60s: RDS detects the primary is down. The detection mechanism varies: hardware failures are detected in seconds; OS-level failures may take 30-60 seconds.
3. ~60-120s: Standby is promoted to primary. The RDS endpoint DNS CNAME record is updated to point to the new primary's IP. TTL is 5 seconds.
4. ~65-125s: After DNS propagation, new connections using the RDS endpoint reach the new primary. Existing connections in the pool still point to the old (dead) IP.
5. After failover: Old primary may come back as the new standby, but the CNAME now points to the promoted instance.

Why connection pools don't automatically recover

A pg.Pool (or any connection pool) maintains a set of open TCP sockets. When Multi-AZ failover occurs, those sockets are connected to the primary's IP address. When the primary becomes unavailable, the OS-level TCP connection receives a RST or times out — the socket transitions to an error state.

The pool typically surfaces this error on the next query attempt as one of:

• Error: Connection terminated unexpectedly
• Error: read ECONNRESET
• Error: connect ETIMEDOUT
• Error: terminating connection due to administrator command (PostgreSQL shutdown)

The pool may automatically remove the errored connection and create a new one — but here is the DNS caching problem: when it opens the new connection, it resolves the hostname again. If the OS DNS cache still holds the old IP (which it will for up to its TTL), the new connection also fails. Node.js's DNS resolution uses the OS resolver, which caches based on the TTL returned by the DNS server. RDS uses a 5-second TTL, but the OS may cache longer depending on the system's /etc/resolv.conf and ndots settings.

// This is the problem: pg.Pool reconnects using the cached IP
const pool = new Pool({
  host: "mcp-prod.cluster-xxxx.us-east-1.rds.amazonaws.com",
  // ... other options
});

// After failover, pool.connect() may resolve to the OLD IP
// if the OS DNS cache hasn't expired yet
const client = await pool.connect(); // Can fail even after CNAME flip

Pattern 1: Use RDS Proxy (eliminates the problem)

The recommended pattern for Lambda-based MCP servers is to use RDS Proxy. The proxy's endpoint is a stable hostname that never changes. The proxy itself handles reconnection to the new RDS primary after failover — the proxy reconnects in a matter of seconds, not minutes. Lambda connection pools connected to the proxy endpoint experience a much shorter outage (the proxy reconnection time, typically 5-30 seconds, versus the full 60-120 second RDS failover + DNS propagation window).

// Connect to the proxy endpoint, not the RDS cluster endpoint
const pool = new Pool({
  host: process.env.RDS_PROXY_ENDPOINT,
  // proxy-name.proxy-xxxx.us-east-1.rds.amazonaws.com
  // This endpoint never changes, even across failovers
  port: 5432,
  database: process.env.DB_NAME,
  // ...
});

Pattern 2: Pool drain-and-recreate on failover errors (direct connection)

If you connect directly to RDS without a proxy, implement a failover error handler that drains the pool and forces DNS re-resolution by creating a new pool instance.

import { Pool, DatabaseError } from "pg";
import dns from "dns/promises";

const DB_HOST = "mcp-prod.cluster-xxxx.us-east-1.rds.amazonaws.com";

function createPool(): Pool {
  return new Pool({
    host: DB_HOST,
    port: 5432,
    database: process.env.DB_NAME!,
    user: process.env.DB_USER!,
    password: process.env.DB_PASSWORD!,
    ssl: { rejectUnauthorized: true },
    max: 1,
    connectionTimeoutMillis: 5_000,
    idleTimeoutMillis: 10_000,
  });
}

let pool = createPool();
let recreating = false;

function isFailoverError(err: Error): boolean {
  return (
    err.message.includes("Connection terminated unexpectedly") ||
    err.message.includes("ECONNRESET") ||
    err.message.includes("ETIMEDOUT") ||
    err.message.includes("terminating connection due to administrator command") ||
    // PostgreSQL error code 57P01: admin_shutdown
    (err instanceof DatabaseError && err.code === "57P01")
  );
}

async function drainAndRecreatePool(): Promise {
  if (recreating) return; // Prevent concurrent recreation
  recreating = true;

  try {
    // Wait for DNS TTL to expire before recreating (RDS TTL is 5s)
    // Force DNS re-resolution: lookup the new IP before creating the pool
    await new Promise((resolve) => setTimeout(resolve, 6_000));
    const resolved = await dns.lookup(DB_HOST);
    console.log(`DNS re-resolved to: ${resolved.address}`);

    const oldPool = pool;
    pool = createPool(); // New pool will use the freshly resolved IP

    // End old pool (fire and forget — old connections are already dead)
    oldPool.end().catch(() => {});
  } finally {
    recreating = false;
  }
}

export async function query(sql: string, values?: any[]): Promise {
  const client = await pool.connect();
  try {
    return await client.query(sql, values);
  } catch (err) {
    if (err instanceof Error && isFailoverError(err)) {
      // Trigger async pool recreation; return error to MCP client immediately
      drainAndRecreatePool().catch(console.error);
      throw new Error(
        "Database unavailable during failover. Retry in 30-60 seconds."
      );
    }
    throw err;
  } finally {
    client.release();
  }
}

Key decision: fail fast vs. wait. During a Multi-AZ failover (60-120s), you have two choices: queue incoming MCP tool calls and wait for the DB to recover, or immediately return an error and let the MCP client retry. For interactive MCP sessions, returning an error quickly is better UX than hanging for 60-120 seconds. The client's LLM can surface the error and suggest retrying — hanging connections appear broken to the client.

Subscribing to RDS failover events

For proactive notification, subscribe to the RDS event SNS topic and forward failover events to your MCP server's operational channel (PagerDuty, Slack, CloudWatch dashboard). Failover events allow you to take proactive action (e.g., temporarily rate-limit MCP tool calls that require DB access).

# Subscribe to RDS DB instance events (covers failover events)
aws rds create-event-subscription \
  --subscription-name mcp-db-failover-alerts \
  --sns-topic-arn arn:aws:sns:us-east-1:123456789012:mcp-ops-alerts \
  --source-type db-instance \
  --source-ids mcp-prod \
  --event-categories '["failover","failure","availability"]' \
  --enabled

# RDS will send SNS notifications for:
# - RDS-EVENT-0025: Multi-AZ failover initiated
# - RDS-EVENT-0049: Multi-AZ failover completed
# - RDS-EVENT-0013: Insufficient capacity in AZ

Common failure modes