AWS EKS Security Architecture for MCP Server Infrastructure: IRSA, Pod Security Admission, Cluster Endpoint Access, and Add-On Lifecycle

Pattern 1 — Per-Pod AWS Identity Isolation: IRSA Trust Policy, Token Injection, Cross-Account Chaining, and Pod Identity

Why the EC2 node role is the wrong model for MCP probe workers

Before IRSA, all pods on an EKS node shared the EC2 instance profile. In a standard web application cluster, this is inconvenient but manageable — if all pods need similar AWS access, one node role works. For AliveMCP probe workers, it is a security failure: every probe makes an outbound HTTP connection to a third-party MCP server and parses its response. A malformed payload exploiting a vulnerability in the HTTP parsing library could give an attacker code execution. If they escape the container, they inherit the node role's permissions — potentially S3 read/write, MSK access, or SNS publish rights across all workloads on the node.

IRSA solves this by issuing each pod a projected OIDC service account token that identifies exactly one Kubernetes service account. STS validates the token's cryptographic signature against the OIDC provider's public keys, enforces the trust policy conditions, and issues time-limited credentials scoped to that service account's role. A compromised probe worker can access only the Kafka topic its role permits — nothing else in the account.

The trust policy condition that matters most

The OIDC subject claim (:sub) uniquely identifies the Kubernetes service account. It must match exactly, using StringEquals — not StringLike. A wildcard in the subject value (system:serviceaccount:alivemcp:*) allows any service account in the namespace to assume the role. This is the most common IRSA misconfiguration and it defeats per-workload isolation entirely.

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": {
      "Federated": "arn:aws:iam::111122223333:oidc-provider/oidc.eks.us-east-1.amazonaws.com/id/EXAMPLED539D4633E53DE1B716D3041E"
    },
    "Action": "sts:AssumeRoleWithWebIdentity",
    "Condition": {
      "StringEquals": {
        "oidc.eks.us-east-1.amazonaws.com/id/EXAMPLED539D4633E53DE1B716D3041E:sub":
          "system:serviceaccount:alivemcp:probe-worker-sa",
        "oidc.eks.us-east-1.amazonaws.com/id/EXAMPLED539D4633E53DE1B716D3041E:aud":
          "sts.amazonaws.com"
      }
    }
  }]
}

Two annotations on the condition key that trip people up: the OIDC provider hostname appears without the https:// scheme as the condition key prefix; and the audience condition (:aud) must match the client ID you specified when registering the OIDC provider — sts.amazonaws.com. Without the audience condition, tokens with any audience value are accepted.

How credentials reach the pod

Once the service account is annotated with eks.amazonaws.com/role-arn, the EKS pod identity webhook mutates every new pod backed by that service account — injecting AWS_ROLE_ARN and AWS_WEB_IDENTITY_TOKEN_FILE environment variables and mounting the OIDC token at /var/run/secrets/eks.amazonaws.com/serviceaccount/token. The annotation is only read at pod creation time; if you annotate after pods exist, those pods keep the old behavior until restarted.

The token file is a projected service account token issued by the Kubernetes API server and signed with the cluster's OIDC private key. It is not the same as the default Kubernetes service account token at /var/run/secrets/kubernetes.io/serviceaccount/token — that token identifies the pod to the Kubernetes API, not to AWS STS. The OIDC token has an aud of sts.amazonaws.com and a bounded expiry (default 24h, minimum 3600s).

Token rotation is transparent: the kubelet rewrites the projected token at approximately 80% of its lifetime; the AWS SDK detects the file change and re-calls AssumeRoleWithWebIdentity before the STS credentials expire (1h default). No pod restarts, no secret rotations, no ConfigMap updates required.

Cross-account IRSA for MSK in a separate data account

AliveMCP probe workers run in the EKS account but write results to an MSK cluster in a separate data account. IRSA supports this via role chaining: the EKS-account probe-worker role has sts:AssumeRole permission targeting the data-account MSK-write role; the data-account role trusts the EKS-account role (standard IAM role trust, not OIDC). The OIDC trust relationship exists only in the EKS account. Account B does not need to know about your cluster's OIDC provider — it only trusts one named role in account A.

The blast radius consequence: a compromised probe worker can write to MSK probe results in the data account — and nothing else in account B. The data account's MSK-write role has no S3, DynamoDB, SNS, or other permissions. CloudTrail in both accounts records API calls attributed to the specific IRSA role ARN, not a generic node role, making incident investigation precise.

EKS Pod Identity: simpler for same-account use cases

EKS Pod Identity (GA November 2023) removes the OIDC provider setup and the trust policy :sub condition for same-account workloads. Install the eks-pod-identity-agent add-on, create an association between a service account and a role using the EKS API, and the trust policy uses pods.eks.amazonaws.com as the principal with aws:SourceAccount and aws:SourceArn conditions that are much harder to misconfigure than the OIDC subject claim syntax.

# Pod Identity trust policy — no OIDC provider ARN, no :sub condition
{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": { "Service": "pods.eks.amazonaws.com" },
    "Action": ["sts:AssumeRole", "sts:TagSession"],
    "Condition": {
      "StringEquals": { "aws:SourceAccount": "111122223333" },
      "ArnLike": {
        "aws:SourceArn": "arn:aws:eks:us-east-1:111122223333:cluster/my-cluster"
      }
    }
  }]
}

The tradeoff: Pod Identity does not support cross-account role assumptions as of 2026. For same-account workloads — writing to S3 in the same account, reading from Secrets Manager, publishing to SNS — Pod Identity is the lower-friction choice. For AliveMCP's cross-account MSK pattern, IRSA is required.

IRSA failure modes reference

Pattern 2 — Pod Security Hardening: PSA Levels, Restricted securityContext, emptyDir Workarounds, and Gatekeeper Complement

PSA replaces PodSecurityPolicy

PodSecurityPolicy was deprecated in Kubernetes 1.21 and removed in 1.25. Pod Security Admission is its built-in replacement — enforced by a webhook already present in every Kubernetes API server, requiring no add-ons, no CRDs. It implements three fixed policy levels applied per-namespace via labels. The cost of simplicity is coarser granularity: PSP allowed per-field control; PSA provides three levels and nothing in between. For policies beyond the three levels, the standard pattern is PSA as the security floor plus OPA Gatekeeper or Kyverno for custom rules.

The three policy levels and where to apply them

The migration sequence: add warn and audit mode labels first. Violations surface in kubectl warnings and audit logs without blocking any deployments. Fix all workloads until no warnings appear, then add enforce. Never add any PSA enforce label to kube-system — even baseline blocks features that CoreDNS requires, causing cluster-wide DNS failure.

The complete securityContext for restricted compliance

spec:
  template:
    spec:
      securityContext:
        runAsNonRoot: true
        runAsUser: 1000
        runAsGroup: 1000
        fsGroup: 1000
        seccompProfile:
          type: RuntimeDefault
      containers:
        - name: probe-worker
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities:
              drop: ["ALL"]
          volumeMounts:
            - name: tmp
              mountPath: /tmp
            - name: npm-cache
              mountPath: /home/app/.npm
      volumes:
        - name: tmp
          emptyDir: {}
        - name: npm-cache
          emptyDir: {}

Two distinctions matter: runAsNonRoot: true is a check, not a setter — it rejects pods where the container image's USER is root (UID 0), but it does not change the UID. You must set runAsUser explicitly, or ensure the Dockerfile sets a non-root USER directive. Setting both is belt-and-suspenders. The seccompProfile can be set at pod level (applies to all containers and init containers) or container level — pod level is less error-prone. RuntimeDefault uses containerd's built-in seccomp profile, which blocks ~40 uncommon syscalls while permitting all syscalls normal server applications need. For MCP probe workers, this profile directly blocks syscalls that exfiltration tools and reverse shells commonly require: ptrace, open_by_handle_at, socket operations to unexpected destinations.

The readOnlyRootFilesystem trap

readOnlyRootFilesystem: true applies the overlay mount with the ro flag at the kernel level. Any write to a path not backed by a mounted volume fails with EROFS: read-only file system. The paths that trip production deployments most often:

• /tmp — Node.js HTTP response parsing, multipart uploads, format conversion libraries
• /var/run — PID files, Unix domain sockets, lock files
• ~/.npm, ~/.cache — package manager caches that write on first use, even in production containers
• /app/logs or /app/data — if the app writes SQLite or log files to its own working directory

The fix is always the same: mount an emptyDir volume at each writable path. For npm's runtime cache, also redirect it with ENV NPM_CONFIG_CACHE=/tmp/.npm-cache in the Dockerfile so it writes to the emptyDir-backed /tmp rather than its default location in the home directory.

Common migration breakages

Alpine base image running as root. node:20-alpine and python:3.12-alpine default to UID 0. Add a non-root user in the Dockerfile:

RUN addgroup -g 1000 app && adduser -D -u 1000 -G app app
USER app

hostPath volumes. baseline and restricted both block hostPath. Replace with ConfigMap or Secret volumes. Legitimate hostPath users (node monitoring agents, log collectors) belong in a privileged-mode namespace.

Init containers running as root. PSA applies the policy to init containers too. Common pattern: an init container runs chmod or chown as root before the main container starts. Fix: set permissions at image build time, use fsGroup for volume ownership, or rewrite the init container to run as a non-root user with sufficient permissions for the setup task.

OPA Gatekeeper for custom rules beyond PSA

PSA enforces three fixed levels. Gatekeeper adds a policy engine (Rego) for arbitrary custom rules. The pattern: PSA sets the namespace security floor; Gatekeeper enforces organisation-specific rules above that floor. Two ConstraintTemplate rules AliveMCP uses:

# Require CPU and memory limits on all containers in the alivemcp namespace
violation[{"msg": msg}] {
  container := input.review.object.spec.containers[_]
  not container.resources.limits.cpu
  msg := sprintf("Container '%v' missing cpu limit", [container.name])
}

# Require images from approved ECR registry only
violation[{"msg": msg}] {
  container := input.review.object.spec.containers[_]
  not startswith(container.image, "111122223333.dkr.ecr.us-east-1.amazonaws.com/")
  msg := sprintf("Container '%v' uses unapproved registry: %v",
    [container.name, container.image])
}

Gatekeeper's critical operational concern: its webhook is fail-closed by default. If both Gatekeeper controller pods are unavailable, all pod creation attempts fail cluster-wide. Run at least two replicas with a PodDisruptionBudget of minAvailable: 1. Monitor Gatekeeper pod health separately from the workloads it governs.

Kyverno is the YAML-native alternative for teams that prefer not to write Rego. Its +(field) strategic merge patch syntax adds a field only if absent — idempotent mutation that will not override an explicit Localhost seccomp profile with RuntimeDefault. Kyverno's generate rules can automatically create a default NetworkPolicy for every new namespace, preventing namespaces from being accidentally left open to all ingress.

Pattern 3 — Cluster Infrastructure Hardening: Endpoint Access, Audit Logging, VPC CNI IP Arithmetic, CoreDNS Tuning, and Add-On Lifecycle

EKS endpoint modes and the publicAccessCidrs lockout trap

The EKS control plane lives outside your VPC. The public endpoint is not a raw exposed port — it is protected by AWS infrastructure — but a stolen kubeconfig or compromised IAM credentials can reach it from any IP on the internet. publicAccessCidrs restricts which source IPs can initiate connections to the public endpoint, regardless of credential validity.

The lock-out trap: set publicAccessCidrs to a CIDR that does not include your current IP before enabling the private endpoint — you lose all API server access immediately. Recovery: the AWS console uses the AWS Management API (not the Kubernetes API) so it still works; or run aws eks update-cluster-config --resources-vpc-config endpointPublicAccess=true,publicAccessCidrs="0.0.0.0/0" from a machine with IAM access to the AWS API.

SSM Session Manager port-forwarding: bastion-free kubectl for private clusters

For private-only clusters, SSM port-forwarding is operationally the cheapest kubectl access path. A small EC2 instance in the VPC — no public IP, no inbound security group rules, no SSH keys — runs the SSM agent outbound. You forward a local port through it to the API server's private hostname:

# Find the private API server hostname
aws eks describe-cluster --name my-cluster \
  --query 'cluster.endpoint' --output text

# Start SSM port-forwarding session
aws ssm start-session \
  --target i-0abc1234def567890 \
  --document-name AWS-StartPortForwardingSessionToRemoteHost \
  --parameters '{
    "host": ["XXXXXXXXXXXXXXXX.gr7.us-east-1.eks.amazonaws.com"],
    "portNumber": ["443"],
    "localPortNumber": ["6443"]
  }'

# In a separate terminal: kubectl via the tunnel
kubectl --server=https://localhost:6443 \
  --insecure-skip-tls-verify \
  get nodes

The --insecure-skip-tls-verify flag is safe for this tunnel: the API server cert is issued for its original hostname, not localhost, and the SSM tunnel itself is already TLS-encrypted. The attack surface is the SSM IAM authorization plane, not an open SSH port. AliveMCP CI/CD pipelines (GitHub Actions) authenticate via IRSA from a self-hosted runner in the same VPC and reach the private endpoint directly — no SSM overhead for automated pipelines, SSM only for interactive developer sessions.

CloudWatch control plane logging: the five types and audit log forensics

All five log types are disabled by default. Enable them explicitly:

aws eks update-cluster-config --name my-cluster \
  --logging '{"clusterLogging":[{
    "types":["api","audit","authenticator","controllerManager","scheduler"],
    "enabled":true
  }]}'

The audit log type is the security-critical one: every API request with who called what, on which resource, with which verb, and what the response code was. Without it, after a security incident — a compromised service account, an unauthorized kubectl delete, an unexpected pod exec session — you have no forensic trail. The api log type is far more voluminous (includes every list-watch long-poll from every node's kubelet) and should only be enabled when actively debugging API server latency. Enable audit unconditionally; gate api on need.

Save these Logs Insights queries in the CloudWatch console for the on-call runbook:

-- Unauthorized API calls (403 responses) — alert signal
fields @timestamp, user.username, verb, objectRef.resource, sourceIPs
| filter responseStatus.code = 403
| sort @timestamp desc
| limit 200

-- Pod exec sessions — anomaly for probe-worker clusters
fields @timestamp, user.username, objectRef.name, objectRef.namespace
| filter requestURI like "/exec"
| sort @timestamp desc

-- All pod deletions in last 24 hours — on-call runbook
fields @timestamp, user.username, objectRef.name, objectRef.namespace
| filter verb = "delete" and objectRef.resource = "pods"
| sort @timestamp desc

Create a metric filter on 403 responses and an alarm on 5+ unauthorized calls in any 5-minute window. For AliveMCP, a second metric filter on requestURI contains "/exec" pages the on-call engineer immediately — probe workers are stateless, any exec session against a probe-worker pod is anomalous.

VPC CNI IP arithmetic and prefix delegation

The Amazon VPC CNI assigns each pod a real VPC IP address from a secondary ENI — different from overlay-network CNIs that NAT through the node IP. The benefit is direct routability within the VPC. The constraint is the pod ceiling per node: (maxENIs × maxIPsPerENI) - 2. For m5.xlarge (4 ENIs × 15 IPs - 2 = 58 max pods), lightweight probe workers hit the IP ceiling before the compute ceiling since each worker consumes minimal CPU and memory.

Enable prefix delegation to multiply pod capacity ~16×:

kubectl set env daemonset aws-node -n kube-system \
  ENABLE_PREFIX_DELEGATION=true \
  WARM_PREFIX_TARGET=1

Prefix delegation allocates /28 CIDR blocks (16 IPs each) to ENI slots instead of individual IPs. An m5.xlarge goes from 58 max pods to 240+ — the same instance type at 4× pod density. Ensure your VPC subnet has enough address space to accommodate the prefix allocations before enabling.

CoreDNS ndots:5 amplification — the DNS failure that looks like MCP server outages

Kubernetes injects five search domain suffixes into every pod's /etc/resolv.conf by default. When a probe worker looks up an external MCP endpoint hostname like api.example-mcp.com, the resolver tries all five suffixes first:

1. api.example-mcp.com.alivemcp.svc.cluster.local → NXDOMAIN
2. api.example-mcp.com.svc.cluster.local → NXDOMAIN
3. api.example-mcp.com.cluster.local → NXDOMAIN
4. api.example-mcp.com.us-east-1.compute.internal → NXDOMAIN
5. api.example-mcp.com → correct answer

At 2,000 external MCP endpoint probes per minute, that is 8,000 unnecessary NXDOMAIN queries per minute — 4× the actual DNS load. When CoreDNS throttles under this load, probe HTTP connections fail because DNS resolution timed out. Those failures appear identical in your monitoring data to real MCP server outages.

# Override ndots on probe-worker pods only
spec:
  template:
    spec:
      dnsConfig:
        options:
          - name: ndots
            value: "2"    # external FQDNs with 3+ dots resolve directly

With ndots:2, a hostname with 3 or more dots (api.example-mcp.com has 3) resolves immediately as a bare FQDN without searching the suffix list. Each external probe goes from 5 DNS queries to 1-2, keeping CoreDNS load proportional to probe volume rather than multiplied by 5.

EBS CSI driver, gp3 StorageClass, and the silent PVC Pending trap

From EKS 1.23, the in-tree EBS provisioner is deprecated. Any PVC using a StorageClass with provisioner: ebs.csi.aws.com requires the aws-ebs-csi-driver add-on to be installed. Without it, the PVC stays Pending indefinitely. The PVC object itself shows no error — only kubectl describe pod on a pod trying to mount it reveals no volume plugin matched. Install the CSI driver add-on before creating PVCs.

Always create a gp3 StorageClass and annotate it as the default. The default StorageClass in many clusters is gp2, which provides 3 IOPS per GB — a 10 GB volume gets 30 IOPS baseline, enough to make SQLite write latency visible under batch commit load. gp3 provides 3,000 IOPS and 125 MB/s throughput baseline at the same cost regardless of volume size:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: gp3
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
provisioner: ebs.csi.aws.com
volumeBindingMode: WaitForFirstConsumer   # creates volume in pod's AZ
reclaimPolicy: Retain
allowVolumeExpansion: true
parameters:
  type: gp3
  encrypted: "true"

volumeBindingMode: WaitForFirstConsumer defers volume creation until a pod is scheduled, ensuring the EBS volume is created in the same AZ as the pod. Without this, a pod rescheduled to a different AZ cannot attach its volume — a silent failure at scale when node failure triggers rescheduling across AZs.

Add-on conflict policy: PRESERVE protects CoreDNS customisations

When EKS upgrades a managed add-on, it reconciles the add-on's Kubernetes resources back to the canonical version. If you have manually edited those resources — including the CoreDNS Corefile for ndots overrides — the resolveConflicts policy controls what happens. OVERWRITE silently reverts all manual edits. PRESERVE keeps your edits but may block the upgrade if there is an irreconcilable conflict. Always use PRESERVE for add-ons where you have intentional customisations.

# Re-install CoreDNS with PRESERVE so Corefile edits survive upgrades
aws eks update-addon \
  --cluster-name my-cluster \
  --addon-name coredns \
  --resolve-conflicts PRESERVE

Cluster infrastructure failure modes reference

End-to-end EKS security architecture for AliveMCP probe infrastructure

AliveMCP's EKS cluster for probe workers integrates all three patterns into a coherent security posture. The cluster uses private endpoint mode with SSM Session Manager port-forwarding as the sole developer kubectl access path — no public IP on the SSM relay instance, no SSH keys, no bastion maintenance overhead. CI/CD pipelines (GitHub Actions self-hosted runner in the same VPC) reach the private endpoint directly via IRSA credentials.

Three service accounts in the alivemcp namespace, each with a dedicated IRSA role scoped to exactly the actions its worker type needs: probe-worker-sa (Kafka WriteData + ReadData + DescribeGroup on the mcp-probe-results topic only), probe-archive-sa (s3:PutObject on the probe-archive prefix only), alert-worker-sa (sns:Publish on the alert topic ARN only). Cross-account MSK access goes through the two-hop chain: IRSA in the EKS account → data-account MSK-write role. A compromised probe worker cannot read archives, publish alerts, call any other AWS service, or assume any other role. CloudTrail in both accounts attributes every API call to the specific IRSA role ARN.

The alivemcp namespace carries enforce: restricted with a version pin to the running Kubernetes minor version. Probe-worker Dockerfiles start from node:20-alpine, create a non-root user (UID/GID 1000), and switch with USER app. Two emptyDir volumes are mounted: one at /tmp for temporary probe response buffering during HTTP response parsing, one at /home/app/.npm for npm's runtime cache writes. The seccompProfile: RuntimeDefault policy blocks syscalls that exfiltration tools and reverse shells commonly require before the kernel executes them — the defence-in-depth layer that operates below user space. OPA Gatekeeper runs alongside PSA with two controller replicas and a PodDisruptionBudget of minAvailable: 1, enforcing ECR-only image registry and CPU/memory limit requirements across all containers in the namespace.

All five EKS log types are enabled with 30-day audit log retention. Two CloudWatch metric filters are active: one on responseStatus.code = 403 (unauthorized kubectl activity) and one on requestURI contains "/exec" (pod exec sessions are anomalous for stateless probe workers). Each probe-worker pod carries ndots: "2" — at 2,000 external MCP endpoint probes per 60-second window, this prevents 8,000 unnecessary NXDOMAIN queries per minute that would otherwise saturate CoreDNS and produce DNS timeout failures indistinguishable from real MCP server outages. Probe results write to a 20 GB gp3 PersistentVolume (WaitForFirstConsumer binding mode, ReadWriteOnce, 3,000 IOPS baseline) — SQLite's batch commit at 60-second intervals generates a burst of perhaps 50–100 IOPS, well within the gp3 baseline. The CoreDNS add-on is managed with PRESERVE conflict policy so Corefile customisations survive add-on version upgrades.

Related guides

• IRSA for EKS Probe Workers: Pod-Level AWS Credential Isolation
• Pod Security Admission for MCP Servers: Enforcing Security Contexts on EKS
• EKS Cluster Endpoint Access Control for MCP Monitoring Infrastructure
• EKS Managed Add-Ons: VPC CNI, EBS CSI, CoreDNS, and Pod Identity for MCP Servers
• Network Policies for MCP Servers on EKS
• Horizontal Pod Autoscaler for MCP Servers on EKS
• AWS OpenSearch Service Advanced Patterns for MCP Monitoring
• AWS MSK for MCP Server Probe Pipelines: Event Architecture, IAM Auth, and Operational Reliability