AWS Secrets Manager for MCP Servers: Three Patterns for Credential Rotation, Lambda Caching, and Cross-Account Access

The core mental model: AWSCURRENT, AWSPENDING, AWSPREVIOUS version labels

Before the three patterns, the AWSCURRENT/AWSPENDING/AWSPREVIOUS version model is the one concept that every other rotation, caching, and cross-account pattern depends on. Secrets Manager does not store a single secret value — it stores versioned values with labels that indicate their role in the rotation lifecycle:

The rotation state machine moves a version through labels: a new version starts as AWSPENDING during rotation, is promoted to AWSCURRENT in finishSecret, and the old AWSCURRENT is simultaneously relabeled as AWSPREVIOUS. Each label can only be held by one version at a time — assigning AWSCURRENT to the new version atomically removes it from the old version. This atomicity is the mechanism that makes rotation consistent: there is never a moment where zero versions hold AWSCURRENT.

The version label model also explains why cross-account secrets require the full ARN: GetSecretValue with a name (not ARN) resolves in the calling account's namespace. In a cross-account scenario, the secret name does not exist in the consuming account, so name resolution fails with ResourceNotFoundException even though the secret is accessible by ARN via the cross-account permission chain.

Pattern 1 — The credential lifecycle contract: TTL, cache isolation, and force-refresh

The first structural pattern governs how MCP server Lambda functions consume secrets from Secrets Manager: how they cache them, how long they hold them, and how they respond to rotation events. The decisions here determine whether rotation is transparent to users (zero auth failures, zero service interruption) or whether rotation causes a wave of authentication errors that take minutes to self-heal.

Why calling GetSecretValue on every invocation is wrong

The naive implementation — calling GetSecretValue inside the handler function on every invocation — has two problems. Cost: Secrets Manager charges $0.05 per 10,000 requests. A function handling 50,000 invocations per hour fetches one secret on each call — $2.50/hour in API costs per secret per function, before data transfer. Latency: each GetSecretValue call adds 5–50 ms of network round-trip to every invocation (within-region to the Secrets Manager API endpoint).

The correct approach is a module-level cache: the cache object is created once when the Lambda execution context initializes (the "init phase"), and subsequent invocations within the same warm context reuse the cached value until the TTL expires:

from aws_secretsmanager_caching import SecretCache, SecretCacheConfig
import boto3
import json

# Module-level — persists across warm invocations
_cache = SecretCache(
    config=SecretCacheConfig(
        max_cache_size=1000,
        secret_refresh_interval=300,   # 5 minutes — NOT the default 3600s
        secret_version_stage_refresh_interval=300,
    ),
    client=boto3.client("secretsmanager", region_name="us-east-1"),
)


def handler(event: dict, context) -> dict:
    secret_str = _cache.get_secret_string(
        "arn:aws:secretsmanager:us-east-1:123456789012:secret:prod/mcp/database-AbCdEf"
    )
    creds = json.loads(secret_str)
    # ... use creds["password"] etc ...

The default secret_refresh_interval in the AWS caching library is 3600 seconds — one hour. This is the most dangerous default in the library. If your rotation policy is monthly (30 days), using a 1-hour TTL means a rotation that completes at 02:00 UTC can leave Lambda functions using the old credential until 03:00 UTC, causing an hour of authentication failures if any downstream system invalidates the old credential immediately on rotation. Set secret_refresh_interval=300 (5 minutes) as a safe default for any secret that rotates.

The per-execution-context cache isolation trap

The most misunderstood aspect of Lambda secrets caching is that the module-level cache is per-execution-context, not per-function. Lambda may maintain many execution contexts simultaneously for a concurrently-invoked function — each context has its own Python interpreter state, its own module-level variables, and its own independent cache. At 100 concurrent invocations, there are up to 100 separate caches.

This has a specific failure mode after SingleUser rotation:

1. Rotation completes at T=0. Secrets Manager atomically promotes AWSPENDING → AWSCURRENT. The old AWSCURRENT becomes AWSPREVIOUS. The downstream database immediately invalidates the old password (SingleUser rotation changes the single database user's password).
2. Containers whose TTL expired before T=0 have already fetched the new credential. They work fine.
3. Containers whose TTL has not yet expired still hold the cached old AWSCURRENT value. When they next attempt to authenticate against the database, they get rejected — the old password is no longer valid.
4. Each affected container recovers individually when its TTL expires and it fetches the new credential. At a 300-second TTL, the last affected container recovers by T=300s.

This is not a TTL configuration bug — it is inherent to SingleUser rotation. The correct solution is MultiUser rotation: two database users alternate, and the old user remains valid until the next rotation cycle. With MultiUser rotation, the per-context cache propagation window is benign — both the old and new credentials authenticate successfully during the window.

TTL selection table

EventBridge force-refresh — the publish-subscribe pattern for zero-downtime rotation

For environments that use SingleUser rotation (or need zero-downtime guarantee regardless of rotation strategy), an EventBridge rule on the RotationSucceeded event provides an active invalidation signal:

# EventBridge rule — fires when Secrets Manager rotation succeeds
{
  "source": ["aws.secretsmanager"],
  "detail-type": ["Secret Rotation Event"],
  "detail": { "eventName": ["RotationSucceeded"] }
}

# Force-refresh Lambda: writes a rotation counter to SSM Parameter Store
def cache_invalidation_handler(event: dict, context) -> None:
    secret_arn = event["detail"]["secretArn"]
    ssm = boto3.client("ssm")
    ssm.put_parameter(
        Name=f"/mcp-server/rotation-counter/{secret_arn.split(':')[-1]}",
        Value=str(int(time.time())),
        Type="String",
        Overwrite=True,
    )

The MCP server Lambda reads the SSM parameter on each invocation (cheap — SSM GetParameter adds ~1 ms) and compares it to the counter value it saw when it last fetched the secret. If the counter has changed, it force-refreshes regardless of the TTL. This is the publish-subscribe pattern for credential rotation: the rotation event increments the counter (publish), and each execution context consumes it lazily on the next invocation (subscribe). No direct Lambda-to-Lambda invocation is needed — the SSM parameter acts as a durable signal that survives container recycling.

Pattern 2 — The rotation function contract: four stages with mandatory idempotency

The second structural pattern governs the rotation Lambda function itself — the code that Secrets Manager invokes four times in sequence to execute a rotation cycle. The pattern here is not about which rotation strategy to use (SingleUser vs MultiUser), but about the correctness invariants that apply to all rotation functions regardless of strategy. Each of the four stages has a critical detail that, if missed, causes the rotation to appear successful while leaving the system in a broken or inconsistent state.

The dispatch skeleton and why MaxRetries must be 0

Every rotation Lambda follows the same dispatch pattern: receive an event with a Step field, route to the corresponding stage function:

import boto3
import json

secrets_client = boto3.client("secretsmanager")

def handler(event: dict, context) -> None:
    arn = event["SecretId"]
    token = event["ClientRequestToken"]  # version token for the AWSPENDING version
    step = event["Step"]

    if step == "createSecret":   create_secret(secrets_client, arn, token)
    elif step == "setSecret":    set_secret(secrets_client, arn, token)
    elif step == "testSecret":   test_secret(secrets_client, arn, token)
    elif step == "finishSecret": finish_secret(secrets_client, arn, token)
    else: raise ValueError(f"Unknown step: {step}")

The ClientRequestToken is the version token Secrets Manager assigned to the AWSPENDING version. Every API call in the rotation function that operates on the new version should use this token. This enables idempotency: if the Lambda is invoked a second time for the same step (due to a Lambda retry after a timeout or error), the token links all operations to the same version — preventing the creation of duplicate AWSPENDING versions or duplicate database users.

This is why the rotation Lambda's event source configuration must set MaxRetries=0. Secrets Manager already handles its own retry logic — it calls the Lambda again with the same ClientRequestToken if the invocation fails. A Lambda-level retry (from the event source mapping) calls createSecret a second time with the same token. If createSecret is not idempotent, this creates a second AWSPENDING version with the same token but different content, corrupting the rotation state machine. The safest configuration is Lambda-level retries disabled, Secrets Manager retries only.

createSecret — idempotency check before generation

The createSecret stage generates a new credential value and stores it as AWSPENDING. The critical detail is the idempotency check at the start: if AWSPENDING already exists for this token (from a previous invocation that completed partially), skip re-creation:

import secrets, string

def create_secret(client, arn: str, token: str) -> None:
    # Idempotency check — skip if AWSPENDING already exists with this token
    try:
        client.get_secret_value(SecretId=arn, VersionId=token, VersionStage="AWSPENDING")
        return  # Already created in a previous (retried) invocation
    except client.exceptions.ResourceNotFoundException:
        pass  # Expected — proceed to create

    # Fetch current secret as template
    current = json.loads(
        client.get_secret_value(SecretId=arn, VersionStage="AWSCURRENT")["SecretString"]
    )
    alphabet = string.ascii_letters + string.digits + "!@#$%^&*"
    new_password = "".join(secrets.choice(alphabet) for _ in range(32))

    client.put_secret_value(
        SecretId=arn,
        ClientRequestToken=token,     # links this value to the AWSPENDING version
        SecretString=json.dumps({**current, "password": new_password}),
        VersionStages=["AWSPENDING"],
    )

Without the idempotency check, a retried createSecret invocation calls put_secret_value with the same token but generates a different password. Secrets Manager raises ResourceExistsException because a version with that token already exists — the rotation fails, and the secret is left with a stale AWSPENDING entry that Secrets Manager cannot clean up automatically.

setSecret — connect with AWSCURRENT, update with AWSPENDING

The setSecret stage provisions the new credential in the target system. For a database, this means connecting with the current (AWSCURRENT) admin credentials and changing the password to the new (AWSPENDING) value:

def set_secret(client, arn: str, token: str) -> None:
    pending = json.loads(
        client.get_secret_value(SecretId=arn, VersionId=token, VersionStage="AWSPENDING")["SecretString"]
    )
    current = json.loads(
        client.get_secret_value(SecretId=arn, VersionStage="AWSCURRENT")["SecretString"]
    )
    # Connect with CURRENT admin credentials, update password to PENDING value
    conn = connect_db(current["host"], current["port"], current["username"], current["password"])
    try:
        conn.execute("ALTER USER %s WITH PASSWORD %s", (pending["username"], pending["password"]))
        conn.commit()
    finally:
        conn.close()

This stage must be idempotent: if the password was already updated in a previous invocation of setSecret, the function should succeed quietly rather than raising a duplicate-update error. For databases, ALTER USER ... WITH PASSWORD is naturally idempotent — running it twice with the same password succeeds both times. For systems that create a new API key or token, implement an explicit check-before-create pattern.

testSecret — MUST use AWSPENDING, never AWSCURRENT

The testSecret stage validates that the new credential actually works. This is the single most commonly misimplemented rotation stage:

def test_secret(client, arn: str, token: str) -> None:
    # MUST use AWSPENDING — this is what the function is validating
    pending = json.loads(
        client.get_secret_value(SecretId=arn, VersionId=token, VersionStage="AWSPENDING")["SecretString"]
    )
    # Connect with the PENDING credential to verify it works
    conn = connect_db(pending["host"], pending["port"], pending["username"], pending["password"])
    try:
        conn.execute("SELECT 1")
    finally:
        conn.close()
    # If connect_db raises, Secrets Manager marks this rotation cycle as failed

The error: fetching AWSCURRENT instead of AWSPENDING in the get_secret_value call. This validates the old credential — which always succeeds because it's the credential that's been working for months. Secrets Manager receives a successful return from testSecret and proceeds to finishSecret, promoting what might be an invalid AWSPENDING credential to AWSCURRENT. The rotation appears successful. The MCP server then picks up the new (invalid) AWSCURRENT credential on the next cache expiry and begins failing authentication against a credential that was never actually validated. This failure manifests minutes or hours after the rotation completed, with no apparent cause in the rotation logs.

finishSecret — atomic promotion with version ID lookup

The finishSecret stage promotes AWSPENDING → AWSCURRENT. The correct implementation first retrieves the current AWSCURRENT version ID, then calls update_secret_version_stage with both the version to promote and the version to demote:

def finish_secret(client, arn: str, token: str) -> None:
    # Find the version currently holding AWSCURRENT (so we can demote it)
    metadata = client.describe_secret(SecretId=arn)
    current_version_id = None
    for version_id, stages in metadata["VersionIdsToStages"].items():
        if "AWSCURRENT" in stages and version_id != token:
            current_version_id = version_id
            break

    if current_version_id is None:
        return  # Already promoted — idempotent

    client.update_secret_version_stage(
        SecretId=arn,
        VersionStage="AWSCURRENT",
        MoveToVersionId=token,           # promote AWSPENDING
        RemoveFromVersionId=current_version_id,  # demote old AWSCURRENT
    )
    # Secrets Manager automatically assigns AWSPREVIOUS to the demoted version

The describe_secret call at the start is required because update_secret_version_stage requires the explicit version ID to remove the AWSCURRENT label from — it does not auto-discover which version currently holds AWSCURRENT. The idempotency check (if current_version_id is None: return) handles the case where finishSecret was retried after a successful promotion: if the token version already holds AWSCURRENT, there is no old version to demote, and the function returns cleanly.

EventBridge monitoring for RotationFailed

Rotation failures are silent by default — Secrets Manager does not send notifications unless you configure EventBridge rules. A rotation failure means the old credential is still in use: this is a security gap that needs immediate attention. Wire an alert on RotationFailed:

{
  "source": ["aws.secretsmanager"],
  "detail-type": ["Secret Rotation Event"],
  "detail": { "eventName": ["RotationFailed"] }
}

Route this to SNS → Slack (or PagerDuty). A rotation failure is not a minor warning — it means the credential is not rotating, and depending on your compliance requirements, an unrotated credential may trigger an audit finding within days. The alert should include the secretArn from the event detail so the on-call engineer can immediately identify which credential stopped rotating and investigate the rotation Lambda's CloudWatch logs.

Pattern 3 — Cross-account and RDS: the three-policy requirement and network paths

The third structural pattern covers the two most complex Secrets Manager configurations: cross-account access in multi-account AWS organizations, and RDS credential rotation using AWS-managed rotation Lambdas. Both patterns have specific configuration requirements that differ meaningfully from single-account, non-database usage — and both have failure modes that manifest as AccessDeniedException with no indication of which specific policy or network path is missing.

Why cross-account requires a CMK — the managed key limitation

Secrets Manager encrypts every secret with a KMS key. The default key is aws/secretsmanager — an AWS-managed key whose key policy is entirely controlled by AWS and cannot be modified. When a cross-account principal calls GetSecretValue on a secret encrypted with the managed key, KMS evaluates the key policy, finds no cross-account grant, and returns AccessDeniedException. This happens regardless of what the secret's resource policy says or what the consuming account's IAM policies allow — KMS rejects the decryption attempt before the secret value is returned.

A customer-managed KMS key (CMK) has an explicit key policy that you control. Creating one and using it to encrypt the secret is the prerequisite for all cross-account access scenarios:

# Key policy statement in the OWNER account (111111111111)
# — grants the consuming role kms:Decrypt on this CMK
{
  "Sid": "AllowCrossAccountDecrypt",
  "Effect": "Allow",
  "Principal": {
    "AWS": "arn:aws:iam::222222222222:role/mcp-lambda-execution-role"
  },
  "Action": ["kms:Decrypt", "kms:DescribeKey"],
  "Resource": "*"
}

The three simultaneous policies — all three required

Cross-account GetSecretValue requires three independent permission grants, each in a different policy and potentially in a different account. Missing any one produces AccessDeniedException:

Unlike S3 — where a bucket resource policy alone is sufficient for cross-account access — Secrets Manager requires both the resource policy (owner account) and the IAM policy (consuming account) to permit the action. The dual-policy requirement is intentional: it prevents a scenario where an owner account can silently grant access to any consuming-account identity without the consuming account's knowledge or consent.

# 1. Secret resource policy (owner account — aws CLI)
aws secretsmanager put-resource-policy \
  --secret-id "arn:aws:secretsmanager:us-east-1:111111111111:secret:shared/mcp-creds-AbCdEf" \
  --resource-policy '{
    "Version": "2012-10-17",
    "Statement": [{
      "Effect": "Allow",
      "Principal": { "AWS": "arn:aws:iam::222222222222:role/mcp-lambda-execution-role" },
      "Action": ["secretsmanager:GetSecretValue", "secretsmanager:DescribeSecret"],
      "Resource": "*"
    }]
  }'

# 3. IAM policy on the Lambda execution role (consuming account)
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["secretsmanager:GetSecretValue", "secretsmanager:DescribeSecret"],
      "Resource": "arn:aws:secretsmanager:us-east-1:111111111111:secret:shared/mcp-creds-AbCdEf"
    },
    {
      "Effect": "Allow",
      "Action": ["kms:Decrypt", "kms:DescribeKey"],
      "Resource": "arn:aws:kms:us-east-1:111111111111:key/cmk-key-id-here"
    }
  ]
}

Hub-and-spoke assume-role for multi-account fleets

For organizations with more than two accounts — a hub security account holding shared secrets accessed by 10+ application accounts — adding each application account's role to the secret's resource policy becomes operationally unmaintainable. The scalable pattern is an intermediate role in the owner account:

import boto3, time

sts = boto3.client("sts", region_name="us-east-1")

_cross_account_client = None
_client_expiry = 0.0

def get_secrets_manager_client():
    global _cross_account_client, _client_expiry
    # Refresh the assumed-role session 60s before expiry
    if _cross_account_client is None or time.monotonic() > _client_expiry - 60:
        assumed = sts.assume_role(
            RoleArn="arn:aws:iam::111111111111:role/secrets-reader-for-mcp",
            RoleSessionName="mcp-server-secrets-session",
            DurationSeconds=900,  # 15-minute session
        )
        creds = assumed["Credentials"]
        _cross_account_client = boto3.client(
            "secretsmanager",
            region_name="us-east-1",
            aws_access_key_id=creds["AccessKeyId"],
            aws_secret_access_key=creds["SecretAccessKey"],
            aws_session_token=creds["SessionToken"],
        )
        _client_expiry = time.monotonic() + 900
    return _cross_account_client

The intermediate role secrets-reader-for-mcp in the owner account consolidates all secret access grants into a single role. New application accounts get access by updating that role's trust policy to allow their Lambda execution roles to assume it. The secret's resource policy only needs to grant the intermediate role — a one-time configuration that never needs updating as the organization grows. The intermediate role holds the CMK Decrypt permission and the GetSecretValue permission in the owner account, so consuming accounts only need the sts:AssumeRole permission to call the role.

RDS MultiUser rotation via CDK HostedRotation

For RDS databases, AWS provides managed rotation Lambda functions for each engine. These implement all four rotation stages correctly for each database protocol — you do not need to write the rotation logic. The CDK HostedRotation construct deploys the managed rotation Lambda and wires it to the secret in a single call:

from aws_cdk import aws_secretsmanager as sm, aws_rds as rds, aws_ec2 as ec2, Duration

# App user secret — alternates between mcp_app_a and mcp_app_b
app_secret = sm.Secret(self, "MCPAppSecret",
    generate_secret_string=sm.SecretStringGenerator(
        secret_string_template='{"username": "mcp_app_a"}',
        generate_string_key="password",
        exclude_characters="/@\"\\",
    ),
)

# Master secret — admin credentials the rotation Lambda uses to manage alternating users
master_secret = sm.Secret(self, "MCPMasterSecret",
    secret_string_value=SecretValue.unsafe_plain_text(
        '{"username": "mcp_admin", "password": "change-me-on-first-deploy"}'
    ),
)

# MultiUser rotation — CDK deploys SAR rotation Lambda automatically
app_secret.add_rotation_schedule("AppSecretRotation",
    hosted_rotation=sm.HostedRotation.postgre_sql_multi_user(
        master_secret=master_secret,
        vpc=vpc,
        vpc_subnets=ec2.SubnetSelection(subnet_type=ec2.SubnetType.PRIVATE_WITH_EGRESS),
        security_groups=[rotation_lambda_sg],
    ),
    automatically_after=Duration.days(30),
)

The master_secret must contain admin credentials for the database — an account that has permission to CREATE USER, ALTER USER, and GRANT in PostgreSQL (or equivalent in other engines). The rotation Lambda uses the master secret to create and manage the alternating users (mcp_app_a and mcp_app_b). If the master secret itself is stale or its credential has been invalidated, all subsequent MultiUser rotations will fail at the setSecret stage.

MultiUser alternating-user lifecycle

MultiUser rotation keeps two database users active and alternates between them. After each rotation cycle, the previously-inactive user becomes AWSCURRENT and the previously-active user becomes AWSPREVIOUS (still valid):

# Before rotation
AWSCURRENT  → { "username": "mcp_app_a", "password": "pw-abc123" }  # active
AWSPREVIOUS → { "username": "mcp_app_b", "password": "pw-xyz789" }  # inactive (still valid)

# Rotation cycle:
# createSecret: generate new password for mcp_app_b (the AWSPREVIOUS user)
AWSPENDING  → { "username": "mcp_app_b", "password": "pw-new456" }
# setSecret: update mcp_app_b's DB password using master secret admin credentials
# testSecret: connect with AWSPENDING (mcp_app_b / pw-new456) → SELECT 1 passes
# finishSecret: promote AWSPENDING → AWSCURRENT

# After rotation
AWSCURRENT  → { "username": "mcp_app_b", "password": "pw-new456" }  # newly active
AWSPREVIOUS → { "username": "mcp_app_a", "password": "pw-abc123" }  # still valid

During the entire rotation window, mcp_app_a remains a valid database user with its unchanged password. Lambda functions with cached mcp_app_a credentials continue authenticating successfully until their cache TTL expires — at which point they fetch AWSCURRENT and begin using mcp_app_b. The key property: at no point during the rotation is the old credential invalidated while live consumers hold it.

VPC network requirements for the rotation Lambda

The rotation Lambda must reach two endpoints: the RDS instance (to change the password) and the Secrets Manager API (to read and write secret versions). For private-VPC RDS deployments, this requires explicit network path configuration:

The security group wiring is equally important and equally easy to miss. The rotation Lambda's security group must have an outbound rule to the RDS security group on the database port (5432 for PostgreSQL, 3306 for MySQL). The RDS security group must have an inbound rule from the rotation Lambda's security group on the same port. Without this explicit inbound rule, the rotation Lambda's connection attempt times out at setSecret — the connection attempt is dropped silently by the security group rather than rejected with an error, making the failure look like a network timeout rather than a permission problem.

Rotation testing procedure

Verify that rotation works before depending on it in production. The four-step procedure catches the most common failures before they affect live traffic:

# 1. Trigger an immediate rotation (without waiting for the schedule)
aws secretsmanager rotate-secret \
  --secret-id "arn:aws:secretsmanager:us-east-1:123456789012:secret:prod/mcp/database-AbCdEf"

# 2. Tail the rotation Lambda's CloudWatch logs for errors
aws logs tail /aws/lambda/SecretsManager-RDSPostgreSQL-MultiUser-Suffix \
  --follow --filter-pattern "ERROR"

# 3. Verify the version stage transition
aws secretsmanager describe-secret \
  --secret-id "arn:..." --query "VersionIdsToStages"
# Expected: {"new-version-id": ["AWSCURRENT"], "old-version-id": ["AWSPREVIOUS"]}

# 4. Verify the new credential actually authenticates and can query data
python3 -c "
import boto3, json, psycopg2
client = boto3.client('secretsmanager', region_name='us-east-1')
creds = json.loads(client.get_secret_value(SecretId='arn:...', VersionStage='AWSCURRENT')['SecretString'])
conn = psycopg2.connect(host=creds['host'], user=creds['username'], password=creds['password'])
cur = conn.cursor()
cur.execute('SELECT count(*) FROM mcp_tool_calls LIMIT 1')
print('Rotation verified — new credential can query application tables')
conn.close()
"

Step 4 is critical: the most common post-rotation failure is a rotation that Secrets Manager marks as successful but that leaves the new database user without the necessary schema permissions. The managed rotation Lambda creates the new user and sets its password, but it does not grant application-level permissions (SELECT, INSERT, UPDATE on specific tables). These must be pre-granted via ALTER DEFAULT PRIVILEGES on the database owner, or applied by a custom rotation Lambda that extends the managed one. Verifying that the new credential can actually execute application queries (not just connect) catches this failure mode before it reaches production.

Consolidated failure modes

Quick-start checklist for production MCP servers

The following checklist covers the configuration required to deploy Secrets Manager credential management for a production MCP server Lambda function with RDS and optional cross-account access:

• Secret encryption: Create a CMK in the owner account. Encrypt all secrets with the CMK (not the default aws/secretsmanager key). Required for cross-account; recommended for all production secrets because CMK usage is auditable in CloudTrail.
• Rotation strategy: Choose MultiUser for any database used by a production MCP server. Create both alternating users (mcp_app_a, mcp_app_b) with identical schema grants before enabling rotation. Create the master secret with admin credentials.
• Rotation Lambda network: Deploy the rotation Lambda in the same VPC as the RDS instance. Add a VPC interface endpoint for Secrets Manager (com.amazonaws.{region}.secretsmanager). Wire security group inbound rules on the database port from the rotation Lambda SG to the RDS SG.
• Application Lambda caching: Use the AWS Secrets Manager caching library at module level with secret_refresh_interval=300 (not the default 3600). Cache the SecretCache object at module level — not inside the handler.
• Cross-account access: Configure all three policies — secret resource policy, CMK key policy (in owner account), IAM role policy (in consuming account). Always use the full ARN in GetSecretValue calls, never the secret name.
• Force-refresh: Create an EventBridge rule on RotationSucceeded that writes an SSM Parameter Store counter. Have application Lambdas poll the counter and force-flush their cache when it changes.
• Monitoring: Create an EventBridge rule on RotationFailed routing to SNS → alert channel. Treat rotation failures as P1 — the credential is not rotating, and it will eventually become a compliance issue or a security incident.
• Test rotation: Run rotate-secret --rotate-immediately in staging. Verify VersionIdsToStages transitions from AWSPENDING to AWSCURRENT. Verify that the new credential can execute application queries (not just authenticate). Only then enable rotation on the production secret.

AliveMCP probes MCP server endpoints every 60 seconds from outside your AWS account. Rotation bugs — a cached stale credential, a new user missing table grants, a cross-account permission change — manifest as authentication failures or tool-call errors that AliveMCP catches within one minute. The external probe gives you the signal that CloudWatch metrics on Lambda errors cannot: that an end-to-end tool call actually succeeded, including the credential lookup, the database authentication, and the query execution. If you run MCP tools in production with Secrets Manager-managed credentials, AliveMCP monitoring gives you the fast feedback loop for catching the failure modes documented in this guide.