AWS ECR · 2026-10-23 · ECR advanced arc
Amazon ECR Advanced Operations for MCP Server Images: Lifecycle Policies, Supply Chain Security, Pull-Through Cache, and Multi-Region Replication
Most MCP server teams treat ECR as a passive storage layer — you push an image, ECS pulls it, done. That works until the first production incident caused by a lifecycle policy that expires the image an active ECS task definition is pointing to, a zero-day CVE in a production npm dependency that scan-on-push never caught because the CVE was published after the push, a Docker Hub rate limit failure in CI that silently builds with a cached stale base image, or a multi-region deployment that triggers ECS task definition updates before the image has finished replicating to the destination region. Four facts stop these incidents: (1) tagPatternList supports * wildcards but tagPrefixList does not — "tagPrefixList": ["v*"] matches the literal tag v*, not all tags starting with v, and the rule silently does nothing useful; (2) ECR Enhanced Scanning uses Amazon Inspector to scan application-layer dependencies like npm package-lock.json that BasicScanning entirely misses, and continuous scanning fires Inspector2:FindingCreated EventBridge events for stored images when new CVEs are published — not just on push; (3) the IAM principal pulling through the pull-through cache must have ecr:CreateRepository on Resource: "*" because ECR creates the destination repository on first cache miss — AmazonECSTaskExecutionRolePolicy does not include this permission and the pull silently falls back to direct upstream; (4) put-replication-configuration replaces the entire existing replication config — adding a new destination region by PUTting only the new rule drops all existing destination rules. This guide synthesises three structural patterns for MCP teams that have moved beyond the basics. Pattern 1 covers image lifecycle management — lifecycle policy JSON structure, tagPatternList wildcards, rule evaluation order with the first-match-wins trap, imageCountMoreThan vs sinceImagePushed countType semantics, lifecycle policy preview dry-run before touching production, CDK custom resource workaround for tagPatternList support, and the AliveMCP policy that keeps probe worker storage under 2 GB — synthesised from ECR Lifecycle Policies — JSON Format, Rule Evaluation, Preview API. Pattern 2 covers supply chain security — BasicScanning vs Enhanced Scanning feature comparison, enabling continuous scanning at the registry level, CI pipeline polling and CRITICAL+EPSS threshold blocking, proactive EventBridge alerts for new CVEs in stored production images, SBOM export in CycloneDX and SPDX formats, and Inspector suppression filters for accepted risks — synthesised from ECR Enhanced Scanning — Amazon Inspector, SBOM, EventBridge Findings. Pattern 3 covers multi-registry and multi-region distribution: the pull-through cache setup (supported upstream registries, IAM gap, Docker Hub official image library/ namespace, rate-limit bypass via Secrets Manager credentials, cache storage and lifecycle policies) synthesised from ECR Pull-Through Cache — Docker Hub, GHCR, Upstream Registry Caching; and cross-region and cross-account replication (non-retroactive semantics, full-replace configuration model, cross-account registry policy, replication status API, wait-for-replication CI pattern, cost model) synthesised from ECR Replication — Cross-Region and Cross-Account Image Distribution.
TL;DR
- Lifecycle policies — JSON structure, tagPatternList, evaluation order, preview: A lifecycle policy is a single JSON document with a
rulesarray. Each rule has a uniquerulePriority(lower number = higher priority, evaluated first), aselectionobject, andaction.type: "expire". UsetagPatternList: ["v*"]for wildcard tag matching;tagPrefixListdoes not support wildcards and["v*"]matches only the literal tagv*. countTypeimageCountMoreThankeeps the N most recently pushed images in the matching set;sinceImagePushed(requirescountUnit: "days") expires images older than N days. Place specific-tag rules at low priority numbers and thetagStatus: "any"hard-cap rule at the highest numeric priority — a catch-all at priority 1 matches everything first, preventing more-specific keep rules from running. Always test withstart-lifecycle-policy-preview+get-lifecycle-policy-previewbefore applying to production — theappliedRulePriorityfield shows which rule matched each image. CDK'slifecycleRulesprop generatestagPrefixListnottagPatternList— use a CDK custom resource to apply the raw JSON directly. - Enhanced Scanning — continuous scan, CI blocking, EventBridge, SBOM: Enable at registry level via
put-registry-scanning-configuration --scan-type ENHANCEDwithCONTINUOUS_SCANfrequency for production repositories. BasicScanning (Clair) checks OS packages only; Enhanced Scanning (Inspector) also scans npm, pip, Maven, and Go dependencies at $0.11/image/month. In CI, polldescribe-image-scan-findingsuntilimageScanStatus.status == "ACTIVE"(up to 5 minutes for large images) then fail on CRITICAL severity — or more precisely on CRITICAL with EPSS > 0.1 to avoid blocking on CVSS-critical CVEs that have near-zero exploit probability. WireInspector2:FindingCreatedEventBridge events to SNS or Lambda to get proactive alerts when a zero-day CVE hits a dependency in already-deployed images. Export SBOM for compliance attestation viacreate-sbom-exportin CycloneDX_1_4 or SPDX_2_3 format — the output is a pre-signed S3 URL valid 24 hours. Suppression filters hide accepted findings from severity counts and EventBridge events without disabling scanning. - Pull-through cache — IAM gap, library/ namespace, Docker Hub auth, lifecycle policies: Create a cache rule with
create-pull-through-cache-rule --ecr-repository-prefix docker-hub --upstream-registry-url registry-1.docker.io. The IAM principal pulling images must haveecr:CreateRepositoryandecr:BatchImportUpstreamImageonResource: "*"— ECR creates the destination repository on the first cache miss andAmazonECSTaskExecutionRolePolicydoes not include these actions. Official Docker Hub images require thelibrary/namespace prefix:node:22-alpinebecomes<account>.dkr.ecr.<region>.amazonaws.com/docker-hub/library/node:22-alpine. For Docker Hub rate-limit bypass, create a Secrets Manager secret namedecr-pullthroughcache/docker-hub(the prefix is required) containing Docker Hub credentials and reference its ARN in the cache rule. Cached images have no built-in TTL — apply lifecycle policies to the auto-created cache repositories immediately, otherwise storage accumulates like any ECR repository. - Replication — full-replace model, non-retroactive, cross-account registry policy, wait-for-replication CI: Call
put-replication-configurationwith the complete destination array — the call replaces the entire existing configuration, so adding a new region requires PUTting all existing destinations plus the new one. Replication only applies to images pushed after the configuration is set — pre-existing images must be re-pushed or copied manually. For cross-account replication, also callput-registry-policyin the destination account granting the source accountecr:CreateRepositoryandecr:ReplicateImage— without this policy, replication fails silently and the failure is logged in the destination account's CloudTrail, not the source. Check replication status withdescribe-image-replication-statusand use a wait-for-replication script in CI before updating ECS task definitions in secondary regions. Budget $0.02/GB per replication transfer plus destination ECR storage at standard rates.
Pattern 1 — Image Lifecycle Management
Why lifecycle policies are non-optional
An ECR repository without a lifecycle policy accumulates images indefinitely. Every push to an existing tag (e.g., :latest) leaves the previous manifest as an untagged image because ECR stores images by digest, not by tag — the tag moves to the new digest, but the old digest stays in the repository billed at $0.10/GB-month. A CI system building MCP server images on every commit to a feature branch can accumulate hundreds of SHA-tagged images within weeks. At 200 MB per image and 500 images, that is 100 GB = $10/month of silent storage cost before anyone notices. More critically, without lifecycle management, a hard-cap rule at an unexpected moment can expire an image that a running ECS task definition is still pointing to, causing new task launches to fail with a CannotPullContainerError.
Lifecycle policy JSON structure
A lifecycle policy is a JSON document with a single rules array. Each rule specifies which images to expire (via selection) and when (via the countType). There is no "archive" action — only "expire".
// Production lifecycle policy for a Node.js MCP server repository
{
"rules": [
{
"rulePriority": 1,
"description": "Never expire protected environment tags",
"selection": {
"tagStatus": "tagged",
"tagPatternList": ["latest", "stable", "prod", "staging"],
"countType": "imageCountMoreThan",
"countNumber": 99999 // effectively never expire these tags
},
"action": { "type": "expire" }
},
{
"rulePriority": 2,
"description": "Keep last 30 versioned releases",
"selection": {
"tagStatus": "tagged",
"tagPatternList": ["v*"], // wildcard: v1.0.0, v2.3.1-rc1, etc.
"countType": "imageCountMoreThan",
"countNumber": 30
},
"action": { "type": "expire" }
},
{
"rulePriority": 3,
"description": "Keep last 10 CI SHA images",
"selection": {
"tagStatus": "tagged",
"tagPatternList": ["sha-*", "ci-*", "pr-*"],
"countType": "imageCountMoreThan",
"countNumber": 10
},
"action": { "type": "expire" }
},
{
"rulePriority": 4,
"description": "Delete untagged images after 1 day",
"selection": {
"tagStatus": "untagged",
"countType": "sinceImagePushed",
"countUnit": "days", // only "days" is valid for countUnit
"countNumber": 1
},
"action": { "type": "expire" }
},
{
"rulePriority": 5,
"description": "Hard cap: expire anything beyond 100 total images",
"selection": {
"tagStatus": "any", // catch-all must be highest numeric priority
"countType": "imageCountMoreThan",
"countNumber": 100
},
"action": { "type": "expire" }
}
]
}
tagPatternList vs tagPrefixList — the wildcard trap
ECR supports two tag selection fields. Only one supports wildcards.
| Field | Wildcard support | Example | What it matches |
|---|---|---|---|
tagPatternList | Yes — * only | ["v*"] | All tags starting with v: v1.0.0, v2.3.1-rc1 |
tagPatternList | Yes | ["*-prod"] | All tags ending with -prod: v1.0.0-prod, release-prod |
tagPrefixList | No — literal prefix | ["v"] | Tags whose text starts with the literal character v |
tagPrefixList | No | ["v*"] | Only the literal tag v* — not a wildcard, matches nothing useful |
tagPrefixList predates tagPatternList and exists for backwards compatibility. Use tagPatternList for all new rules. A rule cannot have both fields — use one or the other. CDK's lifecycleRules prop generates tagPrefixList internally and does not support tagPatternList. Use a CDK custom resource to apply the raw policy JSON:
// CDK: apply raw lifecycle policy with tagPatternList via custom resource
import * as ecr from "aws-cdk-lib/aws-ecr";
import * as cr from "aws-cdk-lib/custom-resources";
const repo = new ecr.Repository(this, "McpServerRepo", {
repositoryName: "mcp-server",
imageTagMutability: ecr.TagMutability.IMMUTABLE,
});
new cr.AwsCustomResource(this, "LifecyclePolicy", {
onCreate: {
service: "ECR",
action: "putLifecyclePolicy",
parameters: {
repositoryName: repo.repositoryName,
lifecyclePolicyText: JSON.stringify({
rules: [
{
rulePriority: 1,
description: "Keep last 30 versioned releases",
selection: {
tagStatus: "tagged",
tagPatternList: ["v*"], // tagPatternList — not available via CDK L2
countType: "imageCountMoreThan",
countNumber: 30,
},
action: { type: "expire" },
},
{
rulePriority: 2,
description: "Delete untagged after 1 day",
selection: {
tagStatus: "untagged",
countType: "sinceImagePushed",
countUnit: "days",
countNumber: 1,
},
action: { type: "expire" },
},
],
}),
},
physicalResourceId: cr.PhysicalResourceId.of("McpServerLifecyclePolicy"),
},
policy: cr.AwsCustomResourcePolicy.fromSdkCalls({ resources: [repo.repositoryArn] }),
});
Rule evaluation order and the first-match trap
ECR evaluates rules in ascending priority order (priority 1 first). When a rule matches an image, no subsequent rules are evaluated for that image. This means rule ordering determines which images survive. The most dangerous pattern is placing a tagStatus: "any" hard-cap rule at priority 1 — it matches every image first, potentially expiring the newest images before the keep rules have a chance to protect them.
// WRONG: hard cap at priority 1 may expire the images the keep rule would protect
{
"rules": [
{ "rulePriority": 1, "selection": { "tagStatus": "any", "countType": "imageCountMoreThan", "countNumber": 50 }, "action": { "type": "expire" } },
{ "rulePriority": 2, "selection": { "tagStatus": "tagged", "tagPatternList": ["v*"], "countType": "imageCountMoreThan", "countNumber": 30 }, "action": { "type": "expire" } }
]
}
// CORRECT: specific rules first, catch-all last
{
"rules": [
{ "rulePriority": 1, "selection": { "tagStatus": "tagged", "tagPatternList": ["v*"], "countType": "imageCountMoreThan", "countNumber": 30 }, "action": { "type": "expire" } },
{ "rulePriority": 2, "selection": { "tagStatus": "untagged", "countType": "sinceImagePushed", "countUnit": "days", "countNumber": 1 }, "action": { "type": "expire" } },
{ "rulePriority": 3, "selection": { "tagStatus": "any", "countType": "imageCountMoreThan", "countNumber": 100 }, "action": { "type": "expire" } }
]
}
Lifecycle policy preview — dry-run before applying
The preview API evaluates a lifecycle policy against the current repository state and reports which images would be expired — without actually expiring them. Run it before applying any policy change to a production repository. The appliedRulePriority in the output tells you exactly which rule matched each image, which is essential for debugging unexpected expirations.
## Step 1: Start a preview (evaluates the policy against the current repo)
aws ecr start-lifecycle-policy-preview \
--repository-name mcp-server \
--lifecycle-policy-text file://lifecycle-policy.json
## Step 2: Poll until COMPLETE (usually < 30 seconds)
aws ecr get-lifecycle-policy-preview \
--repository-name mcp-server \
--query 'status'
# "COMPLETE"
## Step 3: Inspect which images would be expired and which rule matched
aws ecr get-lifecycle-policy-preview \
--repository-name mcp-server \
--query 'previewResults[?action.type==`EXPIRE`].[imageDigest,imageTags,appliedRulePriority]' \
--output table
## Step 4: If the preview is clean, apply the policy
aws ecr put-lifecycle-policy \
--repository-name mcp-server \
--lifecycle-policy-text file://lifecycle-policy.json
Important limitations: the preview captures repository state at the moment start-lifecycle-policy-preview is called. Images pushed during the preview run are not included. Lifecycle evaluation runs once per day at a time chosen by AWS — a newly applied policy does not take effect until the next daily cycle.
AliveMCP runs the lifecycle policy preview as part of the CDK deployment pipeline for the alivemcp/probe-worker repository. If the preview shows the current prod tag would be expired, the pipeline fails and alerts on-call before the policy is applied. This catches misconfigured rule ordering before it can cause an ECS task launch failure in production.
Pattern 2 — Supply Chain Security with Inspector Enhanced Scanning
Why BasicScanning misses the most important vulnerabilities for Node.js MCP servers
BasicScanning uses Clair to scan OS-level packages (rpm, deb, apk). For a typical MCP server built on node:22-alpine, BasicScanning checks the Alpine OS packages and stops. It does not look inside node_modules. A typical Node.js MCP server has 50–200 transitive npm dependencies, each of which can have CVEs. BasicScanning misses all of them. Enhanced Scanning uses Amazon Inspector to analyze the full dependency tree including npm package-lock.json, Python requirements.txt, Java POMs, Go modules, Ruby gems, and .NET packages — the same attack surface that supply chain attackers target.
| Feature | BasicScanning (Clair) | Enhanced Scanning (Inspector) |
|---|---|---|
| OS packages (rpm, deb, apk) | Yes | Yes |
| npm, pip, Maven, Gem, Go, .NET | No | Yes |
| Scan frequency | On-push only | On-push + continuous re-scan |
| SBOM export | No | Yes (CycloneDX, SPDX2) |
| EPSS score | No | Yes |
| EventBridge finding events | ScanCompleted only | FindingCreated, FindingUpdated, FindingClosed |
| Security Hub integration | No | Yes |
| Cost | Free | $0.11/image/month |
Cost planning note: at $0.11/image/month, a repository with 200 stored images costs $22/month on scanning alone. Enable Enhanced Scanning after applying lifecycle policies — cap image counts first, then turn on the continuous scanner.
Enabling Enhanced Scanning
Scanning is configured at the registry level (per account per region) with a filter pattern determining which repositories get Enhanced Scanning versus BasicScanning.
## Enable CONTINUOUS_SCAN for production repositories, SCAN_ON_PUSH for others
aws ecr put-registry-scanning-configuration \
--scan-type ENHANCED \
--rules '[
{
"repositoryFilters": [
{ "filter": "mcp-server*", "filterType": "WILDCARD" }
],
"scanFrequency": "CONTINUOUS_SCAN"
},
{
"repositoryFilters": [
{ "filter": "*", "filterType": "WILDCARD" }
],
"scanFrequency": "SCAN_ON_PUSH"
}
]'
## Verify
aws ecr get-registry-scanning-configuration
CONTINUOUS_SCAN re-scans stored images as new CVEs are published — typically within 24 hours of NVD or VulnDB publication. This is the key capability that scan-on-push misses: a zero-day published three weeks after your last push is still detected. SCAN_ON_PUSH for non-production repositories saves cost on CI build repositories where continuous coverage isn't needed.
CI pipeline: poll for completion, block on CRITICAL+EPSS
Enhanced Scanning is asynchronous — findings take 2–5 minutes to become available after push for a large image. CI must poll for imageScanStatus.status == "ACTIVE" before acting on findings.
#!/usr/bin/env bash
# check-ecr-scan.sh — fail CI on CRITICAL vulnerabilities with EPSS > threshold
REPO=$1; TAG=$2; MAX_WAIT=300; ELAPSED=0
echo "Waiting for ECR Enhanced Scan to complete for $REPO:$TAG..."
while true; do
STATUS=$(aws ecr describe-image-scan-findings \
--repository-name "$REPO" --image-id imageTag="$TAG" \
--query 'imageScanStatus.status' --output text 2>/dev/null)
[ "$STATUS" = "ACTIVE" ] && { echo "Scan complete."; break; }
[ "$STATUS" = "FAILED" ] && { echo "Scan failed — check Inspector configuration"; exit 1; }
[ "$ELAPSED" -ge "$MAX_WAIT" ] && { echo "Scan timed out — proceeding"; exit 0; }
sleep 15; ELAPSED=$((ELAPSED + 15))
done
CRITICAL=$(aws ecr describe-image-scan-findings \
--repository-name "$REPO" --image-id imageTag="$TAG" \
--query 'imageScanFindings.findingSeverityCounts.CRITICAL' --output text)
CRITICAL=${CRITICAL:-0}
if [ "$CRITICAL" != "None" ] && [ "$CRITICAL" -gt 0 ]; then
echo "BLOCKING DEPLOY: $CRITICAL CRITICAL vulnerabilities found"
aws ecr describe-image-scan-findings \
--repository-name "$REPO" --image-id imageTag="$TAG" \
--query 'imageScanFindings.enhancedFindings[?severity==`CRITICAL`].[title,packageVulnerabilityDetails.vulnerabilityId,epss.score]' \
--output table
exit 1
fi
echo "No CRITICAL findings — deploy approved"
EPSS-gated blocking reduces false-positive deploy blocks. The EPSS (Exploit Prediction Scoring System) score (0.0–1.0) estimates the probability a CVE will be exploited in the wild within 30 days. A CRITICAL CVSS score does not imply high exploit probability — many CRITICAL CVEs are theoretical with EPSS near zero. Block on CRITICAL with EPSS > 0.1 (10% exploit probability) to focus blocks on actionable vulnerabilities:
// Node.js: EPSS-gated blocking using AWS SDK v3
import { ECRClient, DescribeImageScanFindingsCommand } from "@aws-sdk/client-ecr";
const ecr = new ECRClient({ region: process.env.AWS_REGION });
async function checkScanFindings(repo: string, tag: string): Promise<void> {
const result = await ecr.send(new DescribeImageScanFindingsCommand({
repositoryName: repo,
imageId: { imageTag: tag },
}));
const findings = result.imageScanFindings?.enhancedFindings ?? [];
const blockers = findings.filter(f => {
if (f.severity !== "CRITICAL") return false;
const epss = f.epss?.score ?? 1.0; // block if no EPSS data (conservative)
return epss > 0.1;
});
if (blockers.length > 0) {
console.error(`BLOCKING: ${blockers.length} CRITICAL+EPSS findings:`);
blockers.forEach(f =>
console.error(` ${f.packageVulnerabilityDetails?.vulnerabilityId} — EPSS: ${f.epss?.score?.toFixed(3)} — pkg: ${f.packageVulnerabilityDetails?.vulnerablePackages?.[0]?.name}`)
);
process.exit(1);
}
}
EventBridge: proactive alerts on new CVEs in stored images
The most important operational pattern with Enhanced Scanning is wiring Inspector2:FindingCreated to an alert channel. When a new CVE is published that matches a dependency in your production image, Inspector fires a FindingCreated event — even if the image was pushed months ago. This is the zero-day detection capability that scan-on-push cannot provide.
// CDK: wire CRITICAL Inspector findings on ECR images to SNS for Slack/PagerDuty
import * as events from "aws-cdk-lib/aws-events";
import * as targets from "aws-cdk-lib/aws-events-targets";
import * as sns from "aws-cdk-lib/aws-sns";
const alertTopic = new sns.Topic(this, "CriticalCveTopic");
new events.Rule(this, "CriticalEcrFinding", {
eventPattern: {
source: ["aws.inspector2"],
detailType: ["Inspector2 Finding"],
detail: {
severity: ["CRITICAL"],
status: ["ACTIVE"],
resources: {
type: ["AWS_ECR_CONTAINER_IMAGE"],
},
},
},
targets: [new targets.SnsTopic(alertTopic)],
});
The event payload includes the CVE ID, EPSS score, affected package name, fixed version (if available), and which ECR repository and tag is affected. A Lambda behind the SNS topic can check whether the image is currently deployed (cross-reference with ECS task definitions) and escalate to PagerDuty only for CVEs in actively running images.
SBOM export for compliance
Enhanced Scanning generates a Software Bill of Materials (SBOM) for each scanned image. Export it in CycloneDX or SPDX2 format for FedRAMP, SOC 2, or internal supply-chain attestation requirements. Inspector derives the SBOM from image layers after push — no build-time instrumentation required.
## Export SBOM in CycloneDX format
curl -o sbom-v1.2.3.json "$(aws ecr create-sbom-export \
--repository-name mcp-server \
--image-id imageTag=v1.2.3 \
--output-format CYCLONEDX_1_4 \
--query 'downloadUrl' --output text)"
## Available formats:
## CYCLONEDX_1_4 — CycloneDX JSON (broadest tool support)
## SPDX_2_3 — SPDX JSON format
The download URL is a pre-signed S3 URL valid for 24 hours. Store the SBOM alongside the image tag in an S3 bucket as part of the release artifact bundle — timestamped, versioned, and retrievable for compliance audits.
AliveMCP's probe worker runs in every customer's AWS region and makes outbound HTTP connections to thousands of MCP endpoints. It has a larger attack surface than a typical internal service. Enhanced Scanning is enabled with CONTINUOUS_SCAN, CI blocks on CRITICAL with EPSS > 0.05 (5% threshold — lower than the default because of the broad attack surface), and new CRITICAL findings on stored production images trigger PagerDuty alerts via the EventBridge rule within 5 minutes. SBOMs are exported and stored in S3 on every release for supply-chain attestation.
Pattern 3 — Multi-Registry and Multi-Region Distribution
Part A: Pull-through cache
Why pull-through cache matters for MCP server CI
Docker Hub rate-limits anonymous pulls to 100 per 6 hours per IP. AWS CodeBuild instances and GitHub Actions runners share public IPs across many customers — a single AWS availability zone can exhaust the rate limit across all teams using it simultaneously. This produces intermittent CI failures where docker pull node:22-alpine returns toomanyrequests: Rate exceeded — not a build failure, not a code problem, just infrastructure noise that causes CI reruns. ECR pull-through cache routes all base image pulls through your private ECR endpoint, where the cached image is served directly from ECR within the same VPC, eliminating the Docker Hub rate limit and reducing build times by 15–25 seconds (ECR intra-VPC pull versus Docker Hub over the public internet).
Supported upstream registries
| Registry | Upstream URL | Auth required |
|---|---|---|
| Docker Hub | registry-1.docker.io | Optional (for private images or rate-limit bypass) |
| Amazon ECR Public | public.ecr.aws | No |
| Quay.io | quay.io | Optional |
| GitHub Container Registry | ghcr.io | Optional |
| Kubernetes registry | registry.k8s.io | No |
| Azure Container Registry | *.azurecr.io | Yes |
| GitLab Container Registry | registry.gitlab.com | Optional |
## Create pull-through cache rules
aws ecr create-pull-through-cache-rule \
--ecr-repository-prefix docker-hub \
--upstream-registry-url registry-1.docker.io \
--region us-east-1
aws ecr create-pull-through-cache-rule \
--ecr-repository-prefix ghcr \
--upstream-registry-url ghcr.io
## List rules
aws ecr describe-pull-through-cache-rules
The IAM gap — CreateRepository and BatchImportUpstreamImage
On the first pull of an upstream image, ECR automatically creates the destination repository (e.g., docker-hub/library/node). The IAM principal pulling must have permission to create that repository. AmazonECSTaskExecutionRolePolicy does not include ecr:CreateRepository or ecr:BatchImportUpstreamImage. Without these permissions, the pull silently falls back to direct upstream — no explicit error, just the Docker Hub rate limit problem reappearing. Check CloudTrail for AccessDenied on these actions if pull-through cache appears not to be working.
// IAM policy additions for ECS task execution roles using pull-through cache
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "ECRPullThroughCache",
"Effect": "Allow",
"Action": [
"ecr:CreateRepository", // required: ECR creates repo on first cache miss
"ecr:BatchImportUpstreamImage", // required: fetches image from upstream
"ecr:GetDownloadUrlForLayer",
"ecr:BatchGetImage",
"ecr:BatchCheckLayerAvailability"
],
"Resource": "*" // must be "*" — target ARN not known at pull time
}
]
}
Pull URI format — the library/ namespace trap
The ECR pull-through cache URI appends the upstream path after the prefix. Official Docker Hub images (maintained by Docker, e.g., node, nginx, postgres) live under the library/ namespace on Docker Hub. Omitting library/ causes a cache miss that falls through to Docker Hub — often silently, making it look like the cache rule isn't working.
# Dockerfile update for ECR pull-through cache
ARG ECR_REGISTRY
# Official Docker Hub image — requires library/ namespace
FROM ${ECR_REGISTRY}/docker-hub/library/node:22-alpine
# Non-library Docker Hub image (e.g., bitnami):
# FROM ${ECR_REGISTRY}/docker-hub/bitnami/postgresql:15
# GHCR:
# FROM ${ECR_REGISTRY}/ghcr/owner/repo:tag
# Build:
# docker build \
# --build-arg ECR_REGISTRY=123456789012.dkr.ecr.us-east-1.amazonaws.com \
# -t mcp-server:latest .
Docker Hub authentication via Secrets Manager
To bypass Docker Hub rate limits, create a Secrets Manager secret with Docker Hub credentials and reference it in the cache rule. The secret name must start with ecr-pullthroughcache/ — ECR refuses to read secrets that don't have this prefix.
## Step 1: Create the Secrets Manager secret
## Name must start with "ecr-pullthroughcache/" — required by ECR
aws secretsmanager create-secret \
--name ecr-pullthroughcache/docker-hub \
--secret-string '{"username":"your-dockerhub-username","accessToken":"dckr_pat_..."}'
# Use an Access Token (from hub.docker.com/settings/security), not your password
## Step 2: Create the cache rule with credential ARN
aws ecr create-pull-through-cache-rule \
--ecr-repository-prefix docker-hub \
--upstream-registry-url registry-1.docker.io \
--credential-arn arn:aws:secretsmanager:us-east-1:123456789012:secret:ecr-pullthroughcache/docker-hub-AbCdEf
ECR needs a resource policy on the Secrets Manager secret to read it on your behalf:
// Secrets Manager resource policy: allow ECR service to read the credential secret
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "AllowECRPullThroughCache",
"Effect": "Allow",
"Principal": { "Service": "ecr.amazonaws.com" },
"Action": "secretsmanager:GetSecretValue",
"Resource": "arn:aws:secretsmanager:us-east-1:123456789012:secret:ecr-pullthroughcache/docker-hub-*",
"Condition": {
"StringEquals": { "aws:sourceAccount": "123456789012" }
}
}
]
}
Cache storage — no built-in TTL, lifecycle policies required
Cached images have no built-in expiry. The auto-created cache repositories accumulate images at the same rate as any ECR repository. Apply lifecycle policies to all cache repositories immediately after setup — before you forget they exist and discover $30/month of unexpected ECR storage six months later.
## Apply lifecycle policy to all docker-hub/* cache repositories in bulk
for repo in $(aws ecr describe-repositories \
--query 'repositories[?starts_with(repositoryName, `docker-hub/`)].repositoryName' \
--output text); do
aws ecr put-lifecycle-policy \
--repository-name "$repo" \
--lifecycle-policy-text '{
"rules": [{
"rulePriority": 1,
"description": "Keep only 5 most recently pulled tags",
"selection": { "tagStatus": "any", "countType": "imageCountMoreThan", "countNumber": 5 },
"action": { "type": "expire" }
}]
}'
done
Part B: Cross-region and cross-account replication
Why replication beats multi-region push in CI
The naive multi-region approach — run docker push to each regional ECR endpoint in CI — has two problems. First, it adds N × image-size of data transfer time to every build for N regions. A 500 MB image with 3 regions adds 1.5 GB of upload time to every CI run. Second, if CI fails mid-push (network error, credentials expire), some regions get the new image and some don't, leaving the deployment in an inconsistent state. ECR replication pushes once to the source region and lets ECR handle distribution — asynchronously, reliably, with a dedicated transfer path.
Replication configuration: full-replace semantics
put-replication-configuration is a complete replacement, not an append. The entire existing configuration is discarded and replaced by the new one. Teams get burned when adding a new destination region by PUTting only the new rule — all existing destinations are silently dropped.
## CORRECT: PUT the complete destination list every time you add a region
aws ecr put-replication-configuration \
--replication-configuration '{
"rules": [
{
"destinations": [
{ "region": "eu-west-1", "registryId": "123456789012" },
{ "region": "ap-southeast-1", "registryId": "123456789012" }
// When adding ap-northeast-1, include ALL three destinations here
],
"repositoryFilters": [
{ "filter": "mcp-server", "filterType": "PREFIX_MATCH" }
]
}
]
}'
## Verify current configuration
aws ecr describe-registry
repositoryFilters with PREFIX_MATCH restricts replication to repositories whose names start with the specified string. In most MCP deployments, filtering to mcp-server is preferable to replicating everything — dev, tooling, and migration images don't need to exist in every production region.
Cross-account replication — source config + destination registry policy
Cross-account replication copies images from a centralized build account to regional production accounts. It requires configuration in both accounts — the source account sets the replication destination, and the destination account grants the source account permission via a registry policy. Without the destination registry policy, replication fails silently. The failure is logged in the destination account's CloudTrail, not the source — a fact that makes debugging cross-account replication failures non-obvious.
## Step 1: SOURCE account — configure replication to destination account
aws ecr put-replication-configuration \
--replication-configuration '{
"rules": [
{
"destinations": [
{ "region": "eu-west-1", "registryId": "999888777666" }
],
"repositoryFilters": [
{ "filter": "mcp-server", "filterType": "PREFIX_MATCH" }
]
}
]
}'
## Step 2: DESTINATION account (999888777666) — create registry policy
## granting source account permission to replicate images
aws ecr put-registry-policy \
--policy-text '{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "AllowSourceAccountReplication",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::123456789012:root"
},
"Action": [
"ecr:CreateRepository", // required: ECR creates destination repo on first replication
"ecr:ReplicateImage"
],
"Resource": "arn:aws:ecr:eu-west-1:999888777666:repository/*"
}
]
}'
Replication is not retroactive
Only images pushed after the replication configuration is applied are replicated. Images already in the source repository at configuration time do not get backfilled. Teams who set up replication after months of pushes find their destination registries empty — the configuration is correct, but there is nothing to replicate until the next push. Re-push or programmatically copy existing images:
## Programmatic copy: get manifest from source, put to destination
## (useful for backfilling after replication is enabled)
MANIFEST=$(aws ecr batch-get-image \
--repository-name mcp-server \
--image-ids imageTag=v1.2.3 \
--query 'images[0].imageManifest' --output text)
aws ecr put-image \
--repository-name mcp-server \
--image-tag v1.2.3 \
--image-manifest "$MANIFEST" \
--region eu-west-1
Replication status and CI wait pattern
Cross-region replication within the same account completes in 1–2 minutes. Cross-account replication takes 2–5 minutes. ECS task definition updates in secondary regions should wait for replication to complete — otherwise ECS may try to pull an image that hasn't arrived yet, causing task launch failures.
#!/usr/bin/env bash
# wait-for-replication.sh — wait until image is in all destination regions
REPO=$1; TAG=$2
REGIONS=("eu-west-1" "ap-southeast-1")
MAX_WAIT=300; ELAPSED=0
echo "Waiting for $REPO:$TAG to replicate to ${REGIONS[*]}..."
while [ "$ELAPSED" -lt "$MAX_WAIT" ]; do
ALL_COMPLETE=true
for REGION in "${REGIONS[@]}"; do
STATUS=$(aws ecr describe-image-replication-status \
--repository-name "$REPO" --image-id imageTag="$TAG" \
--query "replicationStatuses[?region=='$REGION'].status" \
--output text)
if [ "$STATUS" != "COMPLETE" ]; then
ALL_COMPLETE=false
echo " $REGION: ${STATUS:-PENDING}"
fi
done
if [ "$ALL_COMPLETE" = true ]; then
echo "Replication complete in all regions."
exit 0
fi
sleep 15; ELAPSED=$((ELAPSED + 15))
done
echo "Replication timed out after ${MAX_WAIT}s"
exit 1
Alternatively, subscribe to ECR Image Replication EventBridge events to get async notification when status changes, eliminating polling entirely.
Cost model for replication
| Cost component | Rate | Notes |
|---|---|---|
| Replication transfer | $0.02/GB | Charged in source region per destination |
| Destination ECR storage | $0.10/GB-month | Apply lifecycle policies in each destination region |
| ECS pull from same-region ECR | Free | No data transfer charge for ECS pulling from ECR in the same region |
Example: a 500 MB MCP server image replicated to 3 destination regions costs $0.02 × 0.5 GB × 3 = $0.03 per release. Two releases per week for a month = $0.24 transfer. Destination storage for 10 kept images × 500 MB × 3 regions = 15 GB × $0.10 = $1.50/month. Total: ~$1.74/month — negligible compared to the ECS compute costs the deployment drives.
AliveMCP's probe workers run in us-east-1 (primary, CI build and push), eu-west-1 (European MCP endpoints), and ap-southeast-1 (Asia-Pacific MCP endpoints). ECR cross-region replication propagates each new probe worker image from us-east-1 to the two secondary regions within 2 minutes of push. The CI deployment pipeline calls wait-for-replication.sh before updating ECS task definitions in eu-west-1 and ap-southeast-1. The replication configuration uses a PREFIX_MATCH filter on mcp-server — dev, tooling, and migration images are not replicated to the two secondary regions.
Failure modes reference
| Symptom | Cause | Fix |
|---|---|---|
| ECS task fails to start after lifecycle policy runs — image pull fails | Lifecycle policy expired an image referenced by an active ECS task definition | Add a rule protecting the tags referenced by task definitions (prod, latest, specific SHA); verify with preview before applying |
tagPrefixList: ["v*"] does not match versioned tags | tagPrefixList does not support wildcards; v* is treated as the literal prefix v* | Replace with tagPatternList: ["v*"] |
| Hard-cap rule at priority 1 expires newest images before keep rules run | Rules evaluate in ascending priority order; catch-all at priority 1 matches everything first | Move the hard-cap rule to the highest numeric priority (last rule) |
CDK lifecycleRules generates prefix rules instead of pattern rules | CDK L2 uses tagPrefixList internally, no support for tagPatternList | Apply raw lifecycle policy JSON via a CDK custom resource |
imageScanStatus.status stays IN_PROGRESS indefinitely | Inspector2 may not be enabled in this region, or image was pushed before Enhanced Scanning was configured | Check aws inspector2 list-delegated-admin-accounts; re-push image to trigger a new scan |
| No enhanced findings returned for known-vulnerable Node.js image | BasicScanning (not Enhanced) is active for this repository; enhancedFindings is only populated for Enhanced Scanning | Verify with get-registry-scanning-configuration; check repository name matches the filter pattern |
| EventBridge does not fire for re-scanned images | Repository uses SCAN_ON_PUSH frequency, not CONTINUOUS_SCAN | Update scan configuration to CONTINUOUS_SCAN for repositories needing ongoing alerts |
| Pull falls back to Docker Hub instead of using ECR cache | IAM principal lacks ecr:CreateRepository or ecr:BatchImportUpstreamImage | Add these actions to the execution role's inline policy; check CloudTrail for AccessDenied |
docker pull docker-hub/node:22-alpine fails: "no such image" | Official Docker Hub images require the library/ namespace prefix in the ECR path | Use docker-hub/library/node:22-alpine |
| Secrets Manager credential not found by ECR for pull-through cache | Secret name must start with ecr-pullthroughcache/ for ECR to access it | Rename the secret with the required prefix; update the ARN in the cache rule |
| ECR storage grows in pull-through cache repositories without bound | No lifecycle policy on auto-created cache repositories; cached images never expire | Apply lifecycle policies to all docker-hub/* repositories; use bulk script |
Replication status stays IN_PROGRESS; never COMPLETE | Missing registry policy in destination account; or ecr:ReplicateImage not granted to source account | Add registry policy to destination account with ecr:CreateRepository and ecr:ReplicateImage |
| Existing images not replicated after enabling replication | Replication is not retroactive — only new pushes after configuration is applied are replicated | Re-push or programmatically copy existing images to destination regions |
| Adding a new destination region removed existing replication rules | put-replication-configuration replaces the entire config, not appends | Always PUT the complete configuration with all destinations; maintain the full config in IaC |
| Cross-account replication fails silently; no error in source CloudTrail | Failure is logged in the destination account's CloudTrail, not the source | Check CloudTrail in the destination account for ECR:ReplicateImage AccessDenied events |
| Storage grows in destination regions despite lifecycle policy in source | Lifecycle policies are per-repository per-region; a policy in us-east-1 does not apply to replicated repos in other regions | Apply the same lifecycle policy to replicated repositories in each destination region |