Secrets Caching in Lambda for MCP Servers

The AWS Secrets Manager caching library

AWS provides an official caching library for Python (aws-secretsmanager-caching) and Java. The Python library wraps GetSecretValue with an in-memory LRU cache keyed on (secret_id, version_stage):

# Install: pip install aws-secretsmanager-caching
from aws_secretsmanager_caching import SecretCache, SecretCacheConfig
import boto3
import json

# Create at module level — cache persists across warm invocations
_cache = SecretCache(
    config=SecretCacheConfig(
        max_cache_size=1000,          # max number of secrets cached per execution context
        exception_retry_delay_base=1, # base seconds before retrying after a GetSecretValue exception
        exception_retry_growth_factor=2,
        exception_retry_delay_max=3600,
        default_version_stage_name="AWSCURRENT",
        secret_refresh_interval=300,  # TTL in seconds (default is 3600 — too long for most rotation policies)
        secret_version_stage_refresh_interval=300,
    ),
    client=boto3.client("secretsmanager", region_name="us-east-1"),
)


def handler(event: dict, context) -> dict:
    # Returns the SecretString value (str) — parse as JSON for structured secrets
    secret_str = _cache.get_secret_string(
        "arn:aws:secretsmanager:us-east-1:123456789012:secret:prod/mcp-server/database-AbCdEf"
    )
    creds = json.loads(secret_str)
    # ... use creds["password"] etc ...

The first call within a new execution context fetches from Secrets Manager. Subsequent calls within secret_refresh_interval seconds return the cached value. After the interval expires, the next call triggers a background refresh — it fetches the new value and updates the cache, but still returns the cached value for that invocation (the refresh is asynchronous to avoid adding latency to the call that triggered it).

Manual module-level cache (no library dependency)

If you prefer not to add a dependency, a simple module-level dict with a timestamp achieves the same effect with full control over the refresh logic:

import boto3
import json
import time

_client = boto3.client("secretsmanager", region_name="us-east-1")
_cache: dict[str, dict] = {}
_TTL_SECONDS = 300


def get_secret(secret_arn: str) -> dict:
    entry = _cache.get(secret_arn)
    if entry and (time.monotonic() - entry["ts"]) < _TTL_SECONDS:
        return entry["value"]

    resp = _client.get_secret_value(
        SecretId=secret_arn,
        VersionStage="AWSCURRENT",
    )
    value = json.loads(resp["SecretString"])
    _cache[secret_arn] = {"value": value, "ts": time.monotonic()}
    return value


def force_refresh_secret(secret_arn: str) -> dict:
    """Call this after a rotation event to immediately use the new credential."""
    _cache.pop(secret_arn, None)  # invalidate
    return get_secret(secret_arn)  # re-fetch

time.monotonic() is the right clock for cache TTL — it is not affected by system time changes (NTP adjustments, DST) and does not go backward. time.time() can jump backward on AWS Lambda if the VPC's time source is adjusted, which would reset your TTL logic incorrectly.

TTL selection — balancing staleness and API cost

The right TTL depends on your rotation schedule and your tolerance for stale credentials:

The default TTL in the AWS caching library (secret_refresh_interval) is 3600 seconds — one hour. This is safe for environments that never rotate secrets, but dangerous if you rotate monthly: a rotation that completes at 02:00 UTC can leave Lambda functions using the old credential until 03:00 UTC, causing an hour of auth failures if any downstream system invalidates the old credential immediately after rotation.

Force-refresh after rotation via EventBridge

For zero-downtime rotation, pair the cache with an EventBridge rule that invokes a force-refresh Lambda immediately after rotation completes:

# EventBridge rule — fires when Secrets Manager rotation succeeds
{
  "source": ["aws.secretsmanager"],
  "detail-type": ["Secret Rotation Event"],
  "detail": {
    "eventName": ["RotationSucceeded"]
  }
}

# Force-refresh Lambda handler
def cache_invalidation_handler(event: dict, context) -> None:
    secret_arn = event["detail"]["secretArn"]
    # Option 1: invoke the MCP server Lambdas directly to flush their caches
    # (requires a cache-flush endpoint or an SNS topic they're subscribed to)

    # Option 2: use SSM Parameter Store to signal a "rotation version counter"
    # MCP server reads the counter and compares to its cached version
    ssm = boto3.client("ssm")
    ssm.put_parameter(
        Name=f"/mcp-server/rotation-counter/{secret_arn.split(':')[-1]}",
        Value=str(int(time.time())),
        Type="String",
        Overwrite=True,
    )

The SSM parameter approach works because Lambda execution contexts poll the parameter on each invocation (cheap — SSM GetParameter is ~1 ms) and compare its value to the last time they fetched the secret. If the counter changed, they force-refresh regardless of the TTL. This is a publish-subscribe pattern: the rotation event increments the counter, all execution contexts consume it lazily.

Caching binary secrets and multiple versions

The AWS caching library also supports SecretBinary and explicit version stages:

# Cache a binary secret (TLS certificate, PKCS#12 keystore)
cert_bytes = _cache.get_secret_binary(
    "arn:aws:secretsmanager:us-east-1:123456789012:secret:prod/mcp-server/tls-cert-XyZwWv"
)

# Explicitly request AWSPREVIOUS for rollback (e.g., in a rotation Lambda's error handler)
old_secret_str = _cache.get_secret_string(
    "arn:aws:secretsmanager:us-east-1:123456789012:secret:prod/mcp-server/database-AbCdEf",
    version_stage="AWSPREVIOUS",
)

The cache keys on both the ARN and the version stage — so AWSCURRENT and AWSPREVIOUS versions of the same secret are cached independently. This is important for the rollback scenario: fetching AWSPREVIOUS does not evict the AWSCURRENT cache entry.

Per-execution-context isolation — the concurrency trap

The most misunderstood aspect of Lambda secrets caching is that the module-level cache is per-execution-context, not per-function. At 100 concurrent invocations, Lambda may have 100 separate execution contexts — each with its own independent cache. This has two implications:

1. After rotation, the new credential propagates to each execution context only when that context's cache TTL expires. At 300-second TTL, a rotation that completes at T=0 is fully propagated across all contexts by T=300s — but individual contexts may see the new credential anywhere from T=0 (cold start after rotation) to T=300s (warm context at the end of its TTL).
2. During the propagation window, different Lambda instances are using different credentials. For MultiUser rotation (where both the old and new credentials are simultaneously valid), this is fine. For SingleUser rotation (where the old credential is invalidated immediately), this window causes auth failures on warm contexts.

This is why MultiUser rotation is the correct strategy for high-concurrency MCP servers — both credentials are valid simultaneously, so the per-context cache propagation window is benign.

Common failures