IRSA for EKS Probe Workers: Pod-Level AWS Credential Isolation

Why the EC2 instance profile is the wrong model for pods

Before IRSA, EKS workers used the EC2 node's instance profile for AWS API calls. Every pod on that node — your MCP probe workers, your log shippers, your debug containers — all shared the same IAM principal. If an adversary escaped one pod, they inherited the full set of permissions granted to every other workload that happened to run on that node. Blast radius: maximum.

The specific failure mode matters: an attacker who can exec into any pod on the node can call the EC2 instance metadata endpoint (http://169.254.169.254/latest/meta-data/iam/security-credentials/ROLE_NAME) and retrieve temporary credentials for the node role. Those credentials are valid for up to one hour. There is no per-pod restriction, no namespace boundary, and no audit trail linking API calls to a specific workload — all calls appear as the node role in CloudTrail.

IRSA solves this by issuing each pod a short-lived, cryptographically signed OIDC token that identifies the exact Kubernetes service account. STS validates the token signature against the OIDC provider's public keys and enforces the trust policy conditions before issuing credentials. A compromised pod can only access the specific AWS resources its own service account role permits.

Setting up the OIDC provider in IAM

EKS automatically creates an OIDC issuer URL for each cluster. You retrieve it and then register it as an Identity Provider in IAM — this registration is the step most teams forget to do when setting up a new cluster.

# Step 1: get the OIDC issuer URL for your cluster
aws eks describe-cluster \
  --name my-eks-cluster \
  --query "cluster.identity.oidc.issuer" \
  --output text
# Returns something like:
# https://oidc.eks.us-east-1.amazonaws.com/id/EXAMPLED539D4633E53DE1B716D3041E

# Step 2: extract the hostname and get its TLS thumbprint
# The thumbprint is the SHA-1 fingerprint of the root CA cert
# AWS provides a known thumbprint for the EKS OIDC endpoint:
# 9e99a48a9960b14926bb7f3b02e22da2b0ab7280

# Step 3: create the OIDC identity provider in IAM
aws iam create-open-id-connect-provider \
  --url https://oidc.eks.us-east-1.amazonaws.com/id/EXAMPLED539D4633E53DE1B716D3041E \
  --client-id-list sts.amazonaws.com \
  --thumbprint-list 9e99a48a9960b14926bb7f3b02e22da2b0ab7280

# The client-id (audience) must be sts.amazonaws.com — this is what the
# SDK sends when calling AssumeRoleWithWebIdentity

If you use eksctl, the command eksctl utils associate-iam-oidc-provider --cluster my-eks-cluster --approve performs all three steps. The OIDC provider ARN follows the pattern arn:aws:iam::ACCOUNT_ID:oidc-provider/oidc.eks.REGION.amazonaws.com/id/CLUSTER_ID — you will need this when writing trust policies.

One OIDC provider is needed per cluster, not per service account or namespace. Creating it is idempotent if the URL is already registered. Check existing providers with aws iam list-open-id-connect-providers.

IAM trust policy: the exact condition that matters

The trust policy is where precision is critical. The OIDC subject claim (:sub) uniquely identifies the Kubernetes service account that the token was issued for. It must match exactly — both the namespace and the service account name are part of the string.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::111122223333:oidc-provider/oidc.eks.us-east-1.amazonaws.com/id/EXAMPLED539D4633E53DE1B716D3041E"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "oidc.eks.us-east-1.amazonaws.com/id/EXAMPLED539D4633E53DE1B716D3041E:sub":
            "system:serviceaccount:alivemcp:probe-worker-sa",
          "oidc.eks.us-east-1.amazonaws.com/id/EXAMPLED539D4633E53DE1B716D3041E:aud":
            "sts.amazonaws.com"
        }
      }
    }
  ]
}

The key fields to understand:

• :sub (subject) — must equal system:serviceaccount:NAMESPACE:SERVICE_ACCOUNT_NAME exactly. The namespace and service account name are case-sensitive. A value of system:serviceaccount:alivemcp:* using StringLike allows any service account in the namespace to assume this role — that defeats per-workload isolation entirely.
• :aud (audience) — must equal sts.amazonaws.com. This was set when creating the OIDC provider. Without this condition, tokens with any audience would be accepted.
• The OIDC provider hostname appears twice — once in the Principal.Federated ARN and once as the condition key prefix. The condition key does not have the https:// scheme.

# Create the IAM role with the trust policy above
aws iam create-role \
  --role-name probe-worker-role \
  --assume-role-policy-document file://trust-policy.json

# Attach a permissions policy (minimal: only what probe-worker-sa needs)
aws iam put-role-policy \
  --role-name probe-worker-role \
  --policy-name probe-worker-policy \
  --policy-document '{
    "Version": "2012-10-17",
    "Statement": [
      {
        "Effect": "Allow",
        "Action": [
          "kafka-cluster:Connect",
          "kafka-cluster:DescribeGroup",
          "kafka-cluster:AlterGroup",
          "kafka-cluster:ReadData",
          "kafka-cluster:WriteData",
          "kafka-cluster:DescribeTopic"
        ],
        "Resource": [
          "arn:aws:kafka:us-east-1:111122223333:cluster/mcp-results/*",
          "arn:aws:kafka:us-east-1:111122223333:topic/mcp-results/*/mcp-probe-results",
          "arn:aws:kafka:us-east-1:111122223333:group/mcp-results/*/probe-consumer-*"
        ]
      }
    ]
  }'

Service account annotation and how credentials reach the pod

With the IAM role and trust policy in place, annotate the Kubernetes service account with the role ARN. The EKS Pod Identity Webhook watches for this annotation and mutates new pods before they are scheduled.

# service-account.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: probe-worker-sa
  namespace: alivemcp
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::111122223333:role/probe-worker-role
    # Optional: override the token expiry (default 86400s = 24h, min 3600s)
    eks.amazonaws.com/token-expiration: "86400"

When a pod backed by this service account is created, the webhook mutates the pod spec to inject two environment variables and mount the projected token:

# What the webhook injects automatically — you never write this yourself
# (shown here for debugging reference)
env:
  - name: AWS_ROLE_ARN
    value: arn:aws:iam::111122223333:role/probe-worker-role
  - name: AWS_WEB_IDENTITY_TOKEN_FILE
    value: /var/run/secrets/eks.amazonaws.com/serviceaccount/token
volumeMounts:
  - name: aws-iam-token
    mountPath: /var/run/secrets/eks.amazonaws.com/serviceaccount
    readOnly: true
volumes:
  - name: aws-iam-token
    projected:
      sources:
        - serviceAccountToken:
            audience: sts.amazonaws.com
            expirationSeconds: 86400
            path: token

The token at /var/run/secrets/eks.amazonaws.com/serviceaccount/token is a projected service account token issued by the Kubernetes API server, signed with the cluster's OIDC private key. It is not the same as the default Kubernetes service account token at /var/run/secrets/kubernetes.io/serviceaccount/token — that default token is a long-lived credential for calling the Kubernetes API and cannot be used with AWS STS. The IRSA token has an aud claim of sts.amazonaws.com and a bounded expiry.

The AWS SDK credential provider chain checks for AWS_ROLE_ARN and AWS_WEB_IDENTITY_TOKEN_FILE environment variables and automatically calls AssumeRoleWithWebIdentity using the token file. All major SDK versions support this: Python Boto3, Node.js SDK v3, Go SDK v2, Java SDK v2. The one exception: if you pin an extremely old SDK version (e.g., AWS SDK for JavaScript v2 prior to 2.726.0), the web identity provider is not in the credential chain and the pod silently falls back to node instance profile credentials or no credentials at all.

Token expiry and automatic rotation

The projected token defaults to expirationSeconds: 86400 (24 hours). Unlike legacy Kubernetes service account tokens, these tokens are short-lived by design and the kubelet continuously refreshes them before expiry — roughly when 80% of the token lifetime has elapsed. The AWS SDK also refreshes the STS credentials it derives from the token well before they expire (STS credentials last 1 hour by default for AssumeRoleWithWebIdentity).

This means credential rotation happens transparently in the background — you do not need to restart pods, rotate secrets, or update ConfigMaps. The refresh chain is: kubelet writes a new projected token → SDK detects the file has changed → SDK calls STS again with the new token → new AWS credentials are cached in memory. The minimum token lifetime is 3600 seconds; setting it lower is not allowed and the annotation is clamped to 3600 if you try.

To verify token rotation is working in a running pod:

# Check the token expiry from inside the pod
kubectl exec -n alivemcp deploy/probe-worker -- \
  cat /var/run/secrets/eks.amazonaws.com/serviceaccount/token | \
  cut -d'.' -f2 | base64 -d 2>/dev/null | python3 -m json.tool | grep exp

# Check which role the SDK is actually assuming
kubectl exec -n alivemcp deploy/probe-worker -- \
  aws sts get-caller-identity

Cross-account IRSA for accessing resources in another AWS account

A common pattern for AliveMCP: probe workers run in account A (the EKS cluster account) but need to write results to an MSK cluster in account B (a dedicated data account). IRSA supports this via role chaining — the source account role trusts the OIDC provider, assumes a role in the target account, and the target account role trusts the source account role.

# Account B (data account) — role that permits MSK access
# Trust policy: allow account A's probe-worker-role to assume this role
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::111122223333:role/probe-worker-role"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

# Account A (EKS account) — probe-worker-role permissions policy
# Must include sts:AssumeRole on the target account role
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "sts:AssumeRole",
      "Resource": "arn:aws:iam::444455556666:role/msk-write-role"
    }
  ]
}

The probe worker code then performs an explicit AssumeRole call to get credentials for the target account role. The SDK handles the STS call transparently when you construct the client with the source credentials and target role ARN. For most workloads, one hop (OIDC → source role → target role) is sufficient; deeper chains are allowed by STS but add latency and complexity.

Important: the OIDC trust relationship only exists in account A. Account B's role uses a standard IAM role trust, not an OIDC condition. This is intentional — account B does not need to know about your EKS cluster's OIDC provider. The security guarantee is: account B trusts exactly one named role in account A, and account A's OIDC trust policy ensures that role can only be assumed by the specific Kubernetes service account.

EKS Pod Identity: the newer alternative

EKS Pod Identity became generally available in November 2023 as a simpler alternative to IRSA for same-account use cases. Instead of managing an OIDC provider in IAM and writing trust policy conditions by hand, you install the EKS Pod Identity Agent add-on on your cluster and use an association API to bind a service account to an IAM role.

# Install the EKS Pod Identity Agent add-on
aws eks create-addon \
  --cluster-name my-eks-cluster \
  --addon-name eks-pod-identity-agent

# Create the association — no trust policy editing required
aws eks create-pod-identity-association \
  --cluster-name my-eks-cluster \
  --namespace alivemcp \
  --service-account probe-worker-sa \
  --role-arn arn:aws:iam::111122223333:role/probe-worker-role

The IAM role's trust policy for Pod Identity uses a different principal — pods.eks.amazonaws.com rather than the OIDC provider — and there are no :sub conditions to get right:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "pods.eks.amazonaws.com"
      },
      "Action": [
        "sts:AssumeRole",
        "sts:TagSession"
      ]
    }
  ]
}

The tradeoff table:

For AliveMCP's cross-account MSK access pattern, IRSA is the only option today. For same-account workloads like writing to an S3 bucket in the same account, Pod Identity is simpler to operate and reduces the surface area for trust policy misconfiguration.

Failure modes reference

How AliveMCP uses IRSA

AliveMCP runs three distinct worker types in the alivemcp namespace, each with its own service account and dedicated IAM role. The probe-worker-sa service account maps to a role with Kafka WriteData, ReadData, and DescribeGroup permissions scoped to the mcp-probe-results topic on the mcp-results MSK cluster — nothing else. The probe-archive-sa maps to a role with only s3:PutObject on the probe-archive bucket prefix. The alert-worker-sa maps to a role with only sns:Publish on the alert topic ARN.

The consequence of this partitioning: if a vulnerability in an MCP server under test caused a probe worker to execute arbitrary code, the attacker would have write access to MSK probe results but could not read S3 archives, publish SNS alerts, call any other AWS service, or pivot to other pods' roles. AliveMCP also enables CloudTrail data events on the MSK and S3 resources, so every API call is tied to a specific IRSA role ARN — not a generic node role — making incident investigation precise. For the cross-account MSK cluster in the data account, probe workers use IRSA to assume the source role in the EKS account, then explicitly chain to the MSK-write role in the data account, so the data account's blast radius is confined to one named caller.

Related guides

• Pod Security Admission for MCP Servers: Enforcing Security Contexts on EKS
• EKS Cluster Endpoint Access Control for MCP Infrastructure
• Managing EKS Add-Ons for MCP Server Clusters
• Network Policies for MCP Servers on EKS
• Horizontal Pod Autoscaler for MCP Servers on EKS