MCP Server Secrets Manager Rotation — Lambda rotation function, staging labels, multi-user pattern

The four rotation Lambda steps

Every Secrets Manager rotation Lambda must handle all four steps via a Step field in the event payload. Secrets Manager calls the Lambda four times per rotation cycle — once for each step. If any step throws an exception, the entire rotation cycle fails and the secret remains in its prior state.

// rotation-lambda/index.mjs — skeleton for all four steps
import {
  SecretsManagerClient,
  GetSecretValueCommand,
  PutSecretValueCommand,
  UpdateSecretVersionStageCommand,
  DescribeSecretCommand,
} from "@aws-sdk/client-secrets-manager";

const sm = new SecretsManagerClient({});

export const handler = async (event) => {
  const { SecretId, ClientRequestToken, Step } = event;

  // Validate that this version is still pending (idempotency guard)
  const meta = await sm.send(new DescribeSecretCommand({ SecretId }));
  const versionStages = meta.VersionIdsToStages?.[ClientRequestToken] ?? [];
  if (versionStages.includes("AWSCURRENT")) {
    // Already finished — idempotent exit
    return;
  }
  if (!versionStages.includes("AWSPENDING")) {
    throw new Error(`Version ${ClientRequestToken} is not AWSPENDING`);
  }

  switch (Step) {
    case "createSecret":  return createSecret(SecretId, ClientRequestToken);
    case "setSecret":     return setSecret(SecretId, ClientRequestToken);
    case "testSecret":    return testSecret(SecretId, ClientRequestToken);
    case "finishSecret":  return finishSecret(SecretId, ClientRequestToken);
    default:
      throw new Error(`Unknown step: ${Step}`);
  }
};

async function createSecret(secretId, token) {
  // 1. Check if AWSPENDING version already exists (retry safety)
  try {
    await sm.send(new GetSecretValueCommand({
      SecretId: secretId,
      VersionStage: "AWSPENDING",
    }));
    return; // already created in a prior invocation
  } catch (e) {
    if (e.name !== "ResourceNotFoundException") throw e;
  }

  // 2. Get current secret to use as template for new credential
  const current = await sm.send(new GetSecretValueCommand({
    SecretId: secretId,
    VersionStage: "AWSCURRENT",
  }));
  const currentSecret = JSON.parse(current.SecretString);

  // 3. Generate new credential (service-specific)
  const newPassword = generateSecurePassword(32);

  // 4. Store as AWSPENDING — Secrets Manager assigns the version ID
  await sm.send(new PutSecretValueCommand({
    SecretId: secretId,
    ClientRequestToken: token,
    SecretString: JSON.stringify({
      ...currentSecret,
      password: newPassword,
    }),
    VersionStages: ["AWSPENDING"],
  }));
}

async function setSecret(secretId, token) {
  // Retrieve AWSPENDING credential and set it at the backend service
  const pending = await sm.send(new GetSecretValueCommand({
    SecretId: secretId,
    VersionStage: "AWSPENDING",
  }));
  const { username, password, host, port, dbname } = JSON.parse(pending.SecretString);

  // Connect to the database using the CURRENT credential to update the password
  const current = await sm.send(new GetSecretValueCommand({
    SecretId: secretId,
    VersionStage: "AWSCURRENT",
  }));
  const currentCreds = JSON.parse(current.SecretString);

  // Use an admin/master credential to change the user password
  // (for single-user rotation — for multi-user rotation, create a new user)
  await changePasswordInDatabase(currentCreds, username, password);
}

async function testSecret(secretId, token) {
  // Retrieve AWSPENDING and verify it can connect to the backend
  const pending = await sm.send(new GetSecretValueCommand({
    SecretId: secretId,
    VersionStage: "AWSPENDING",
  }));
  const creds = JSON.parse(pending.SecretString);

  // Attempt a real connection — throw on failure so rotation aborts
  await verifyDatabaseConnection(creds);
}

async function finishSecret(secretId, token) {
  // Find current AWSCURRENT version ID (to demote to AWSPREVIOUS)
  const meta = await sm.send(new DescribeSecretCommand({ SecretId: secretId }));
  const currentVersionId = Object.entries(meta.VersionIdsToStages ?? {})
    .find(([_, stages]) => stages.includes("AWSCURRENT"))?.[0];

  if (currentVersionId === token) {
    return; // already finished in a prior invocation
  }

  // Promote AWSPENDING to AWSCURRENT; Secrets Manager automatically demotes
  // old AWSCURRENT to AWSPREVIOUS and removes old AWSPREVIOUS
  await sm.send(new UpdateSecretVersionStageCommand({
    SecretId: secretId,
    VersionStage: "AWSCURRENT",
    MoveToVersionId: token,
    RemoveFromVersionId: currentVersionId,
  }));
}

Staging label lifecycle

AWSPREVIOUS is critical for MCP servers because MCP server instances may cache credentials in memory for performance. If your server caches a credential for 5 minutes and rotation completes, the cached credential is AWSPREVIOUS for 5 minutes — it still works at the backend because the backend was set to accept the new password in setSecret but the old one wasn't revoked yet. The multi-user pattern makes this window explicit and configurable.

Multi-user rotation pattern for zero-downtime cycling

In single-user rotation, there is a window between setSecret (when the backend password changes) and the MCP server reading the new AWSCURRENT credential where connections using the old credential fail. For MCP servers under load, this causes transient tool call failures. The multi-user pattern eliminates this window by alternating between two backend users: mcp_user_a and mcp_user_b. When rotating, the inactive user's password is changed and that user becomes AWSCURRENT. The previously active user becomes AWSPREVIOUS and remains valid at the database indefinitely (until the next rotation cycle promotes the other user).

// Multi-user rotation: the secret stores which user is active
// { "username": "mcp_user_a", "password": "...", "host": "...", "port": 5432, "dbname": "..." }

async function createSecret(secretId, token) {
  // Determine which user alternates to next
  const current = await sm.send(new GetSecretValueCommand({
    SecretId: secretId, VersionStage: "AWSCURRENT"
  }));
  const currentCreds = JSON.parse(current.SecretString);

  // Alternate: if current user is mcp_user_a, next is mcp_user_b
  const nextUsername = currentCreds.username === "mcp_user_a"
    ? "mcp_user_b"
    : "mcp_user_a";

  const newPassword = generateSecurePassword(32);

  await sm.send(new PutSecretValueCommand({
    SecretId: secretId,
    ClientRequestToken: token,
    SecretString: JSON.stringify({
      ...currentCreds,
      username: nextUsername,
      password: newPassword,
    }),
    VersionStages: ["AWSPENDING"],
  }));
}

async function setSecret(secretId, token) {
  // Change the password of the PENDING user at the database
  // (does NOT affect the current active user — zero downtime)
  const pending = await sm.send(new GetSecretValueCommand({
    SecretId: secretId, VersionStage: "AWSPENDING"
  }));
  const { username, password } = JSON.parse(pending.SecretString);

  // Use admin credentials (separate master secret) to ALTER USER
  const master = await sm.send(new GetSecretValueCommand({
    SecretId: process.env.MASTER_SECRET_ARN
  }));
  const masterCreds = JSON.parse(master.SecretString);
  await alterUserPassword(masterCreds, username, password);
}

// testSecret and finishSecret are identical to single-user rotation

RotationRules: scheduling and timing

Secrets Manager supports two rotation schedule modes: AutomaticallyAfterDays (legacy integer — rotate every N days) and ScheduleExpression (cron or rate expression — preferred for predictable windows). Use ScheduleExpression with a maintenance window that avoids peak traffic for MCP server database credentials.

# Terraform: enable rotation with a weekly Sunday 02:00 UTC schedule
resource "aws_secretsmanager_secret_rotation" "mcp_db" {
  secret_id           = aws_secretsmanager_secret.mcp_db.id
  rotation_lambda_arn = aws_lambda_function.secret_rotator.arn

  rotation_rules {
    # Rotate every Sunday between 02:00 and 02:30 UTC
    schedule_expression        = "cron(0 2 ? * 1 *)"
    # Duration window in which rotation must complete (ISO 8601 duration)
    duration                   = "PT30M"
    # IMPORTANT: set false to avoid an immediate rotation on terraform apply
    rotate_immediately_on_update = false
  }
}

# Lambda must have secretsmanager.amazonaws.com as allowed invoker
resource "aws_lambda_permission" "sm_invoke" {
  statement_id   = "SecretsManagerInvoke"
  action         = "lambda:InvokeFunction"
  function_name  = aws_lambda_function.secret_rotator.function_name
  principal      = "secretsmanager.amazonaws.com"
  source_account = data.aws_caller_identity.current.account_id
}

rotate_immediately_on_update: Defaults to true in the AWS API. If you update RotationRules (e.g., change the schedule) while RotateImmediatelyOnUpdate is true, Secrets Manager triggers an unplanned rotation immediately — which may land during peak traffic. Always set it to false in Terraform to control when rotations actually fire.

MCP server credential refresh pattern

MCP servers should not call GetSecretValue on every tool invocation — that adds latency and hits Secrets Manager API rate limits. The recommended pattern is an in-memory cache with TTL just under the rotation window, combined with retry-on-auth-failure to pick up a just-rotated credential without restarting the server.

// MCP server: cached credential with retry on auth failure
const SECRET_ARN = process.env.DB_SECRET_ARN;
const CACHE_TTL_MS = 4 * 60 * 1000; // 4 minutes — rotate every 5m minimum

let credentialCache = { value: null, expiresAt: 0 };

async function getDbCredential() {
  const now = Date.now();
  if (credentialCache.value && now < credentialCache.expiresAt) {
    return credentialCache.value;
  }
  const resp = await sm.send(new GetSecretValueCommand({ SecretId: SECRET_ARN }));
  const creds = JSON.parse(resp.SecretString);
  credentialCache = { value: creds, expiresAt: now + CACHE_TTL_MS };
  return creds;
}

// In tool handler: retry once on auth failure to handle mid-rotation state
async function executeDbQuery(sql, params) {
  let creds = await getDbCredential();
  try {
    return await queryWithCredentials(creds, sql, params);
  } catch (err) {
    if (isAuthenticationError(err)) {
      // Force cache refresh — rotation may have just completed
      credentialCache = { value: null, expiresAt: 0 };
      creds = await getDbCredential();
      return queryWithCredentials(creds, sql, params);
    }
    throw err;
  }
}

Failure modes reference