EKS managed add-ons for MCP server infrastructure: VPC CNI, EBS CSI, CoreDNS, and Pod Identity

Managed add-on lifecycle: OVERWRITE vs PRESERVE conflict policy

When EKS upgrades a managed add-on, it may need to reconcile the add-on's Kubernetes resources (ConfigMaps, DaemonSet specs, RBAC rules) back to the canonical state for that version. If you have manually edited those resources, a conflict occurs. The resolveConflicts field on the add-on controls what happens:

# Install an add-on with PRESERVE conflict policy
aws eks create-addon \
  --cluster-name my-cluster \
  --addon-name coredns \
  --resolve-conflicts PRESERVE

# Update an existing add-on
aws eks update-addon \
  --cluster-name my-cluster \
  --addon-name coredns \
  --addon-version v1.11.3-eksbuild.1 \
  --resolve-conflicts PRESERVE

Add-on version compatibility is separate from the Kubernetes version. Before upgrading your cluster from, say, 1.29 to 1.30, check whether each add-on version you are running is compatible with the target Kubernetes version. Some add-ons must be upgraded before the cluster upgrade; others require upgrading after. The compatibility matrix lives at aws eks describe-addon-versions:

# List available versions for vpc-cni compatible with K8s 1.30
aws eks describe-addon-versions \
  --addon-name vpc-cni \
  --kubernetes-version 1.30 \
  --query 'addons[].addonVersions[].addonVersion'

VPC CNI: IP-per-pod model and instance type limits

The Amazon VPC CNI (the aws-node DaemonSet) assigns each pod a real VPC IP address from a secondary ENI attached to the node. This is fundamentally different from overlay-network CNIs like Flannel or Calico, where pods share the node's IP through NAT. The benefit: pods are directly routable within the VPC, simplifying security group rules and AWS SDK calls. The constraint: the number of pods a node can run is bounded by how many secondary IPs the node's instance type supports.

The formula is: (maxENIs × maxIPsPerENI) - 2, where 2 IPs are reserved for the primary ENI's primary IP and the node's networking. Common instance types:

Note that m5.xlarge and m5.2xlarge have identical pod limits despite m5.2xlarge having double the vCPU and RAM. If your MCP probe workers are lightweight (low CPU, low memory) and you want density, you hit the IP ceiling before the compute ceiling. Two approaches address this:

# Option 1: Tune WARM_ENI_TARGET and MINIMUM_IP_TARGET on aws-node DaemonSet
# Reduce pre-attached warm ENIs for large nodes with sparse pod placement
kubectl set env daemonset aws-node \
  -n kube-system \
  WARM_ENI_TARGET=0 \
  MINIMUM_IP_TARGET=10

# Option 2: Enable prefix delegation — allocate /28 CIDR blocks per ENI
# instead of individual IPs; multiplies effective pod capacity by 16
kubectl set env daemonset aws-node \
  -n kube-system \
  ENABLE_PREFIX_DELEGATION=true \
  WARM_PREFIX_TARGET=1

Prefix delegation mode assigns /28 CIDR prefixes (16 IPs each) to ENI slots instead of individual IP addresses. An m5.xlarge with prefix delegation can host up to (4 ENIs - 1) × 16 addresses × 1 prefix/slot = 240+ pods instead of 58 — a 4x improvement without changing instance type. Enable it when your workload is pod-dense relative to the instance's ENI/IP limits, and ensure your VPC subnet has enough IP space to accommodate the prefix allocations.

CoreDNS: ndots, search domains, and DNS query amplification

CoreDNS is the in-cluster DNS server — a Deployment (typically 2 replicas) that resolves Kubernetes service names to ClusterIP addresses and forwards external hostname lookups to the VPC resolver. It is critically important: if CoreDNS is unavailable, any pod that resolves a service name at startup will fail to initialize, and all outbound connections by hostname will fail.

The less obvious problem for MCP monitoring workloads is DNS query amplification caused by the ndots:5 default. Kubernetes injects five search domain suffixes into every pod's /etc/resolv.conf. When a pod looks up an external hostname like api.example-mcp.com, the resolver tries each suffix before falling back to the bare domain:

1. api.example-mcp.com.mcp.svc.cluster.local → NXDOMAIN
2. api.example-mcp.com.svc.cluster.local → NXDOMAIN
3. api.example-mcp.com.cluster.local → NXDOMAIN
4. api.example-mcp.com.us-east-1.compute.internal → NXDOMAIN
5. api.example-mcp.com → the correct answer

For a single probe, 4 wasted NXDOMAIN lookups are harmless. At 2,000 probes per minute hitting external MCP endpoint URLs, that is 8,000 unnecessary DNS queries per minute — 4x the load on CoreDNS and the VPC resolver. At scale, this causes DNS throttling, SERVFAIL responses, and probe HTTP connections that fail not because the MCP server is down, but because DNS resolution timed out. Those failures look identical in your monitoring data to real outages.

# Fix: override ndots in the probe-worker pod spec
# ndots:2 means: only try search suffixes if the hostname has fewer than 2 dots
# api.example-mcp.com has 3 dots — resolves directly without search suffix iteration
apiVersion: apps/v1
kind: Deployment
metadata:
  name: probe-worker
  namespace: mcp
spec:
  template:
    spec:
      dnsConfig:
        options:
          - name: ndots
            value: "2"      # down from the default of 5
      containers:
        - name: probe-worker
          image: your-ecr-repo/probe-worker:latest

CoreDNS scaling: the default 2 replicas handles approximately 100–150 pods adequately. Scale to 4 replicas for clusters above 200 pods, or enable the CoreDNS Autoscaler add-on which adjusts replica count based on node and core count:

# Scale CoreDNS manually
kubectl scale deployment coredns -n kube-system --replicas=4

# Or install the cluster-proportional-autoscaler for CoreDNS
# This is a separate add-on; check eks describe-addon-versions for availability
aws eks create-addon \
  --cluster-name my-cluster \
  --addon-name coredns-autoscaler

EBS CSI driver: gp3 StorageClass and volume lifecycle

Before EKS 1.23, the Kubernetes built-in kubernetes.io/aws-ebs in-tree provisioner handled EBS volume creation. From EKS 1.23 onward, that provisioner is deprecated and the aws-ebs-csi-driver add-on is required. Without it installed, any PersistentVolumeClaim using a StorageClass with provisioner: ebs.csi.aws.com will stay in Pending indefinitely. The PVC object itself shows no error. Only kubectl describe pod on a pod trying to mount that PVC reveals no volume plugin matched or waiting for a volume to be created.

# Install EBS CSI driver add-on
aws eks create-addon \
  --cluster-name my-cluster \
  --addon-name aws-ebs-csi-driver \
  --service-account-role-arn arn:aws:iam::123456789012:role/EBSCSIDriverRole \
  --resolve-conflicts PRESERVE

The CSI driver's service account needs IAM permissions to create, attach, describe, and delete EBS volumes. The required actions are: ec2:CreateVolume, ec2:AttachVolume, ec2:DetachVolume, ec2:DeleteVolume, ec2:DescribeVolumes, ec2:DescribeVolumeStatus, ec2:CreateTags, ec2:DescribeInstances, and ec2:DescribeAvailabilityZones. Use the AWS-managed policy arn:aws:iam::aws:policy/service-role/AmazonEBSCSIDriverPolicy to avoid maintaining this list manually.

Always use gp3 StorageClass. The default EKS StorageClass in many clusters is gp2, which provides 3 IOPS per GB of provisioned storage — meaning a 10 GB volume gets only 30 IOPS baseline, enough to make SQLite write latency visible under load. gp3 provides a flat 3,000 IOPS and 125 MB/s throughput baseline at the same cost regardless of volume size:

# gp3 StorageClass — use this instead of the default gp2
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: gp3
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
provisioner: ebs.csi.aws.com
volumeBindingMode: WaitForFirstConsumer   # waits for pod scheduling to pick AZ
reclaimPolicy: Retain                     # don't delete EBS volume on PVC delete
allowVolumeExpansion: true                # allow online resize without pod restart
parameters:
  type: gp3
  encrypted: "true"
  # Optional: raise above 3000 IOPS baseline (costs extra)
  # iops: "6000"
  # throughput: "250"

# PersistentVolumeClaim using gp3
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: mcp-probe-db
  namespace: mcp
spec:
  accessModes:
    - ReadWriteOnce     # EBS: single node at a time
  storageClassName: gp3
  resources:
    requests:
      storage: 20Gi

Important constraint: EBS volumes use ReadWriteOnce access mode — they can only be mounted by one node at a time, and the volume is bound to a specific Availability Zone. If the pod is rescheduled to a node in a different AZ, the volume cannot attach. The volumeBindingMode: WaitForFirstConsumer setting defers volume creation until a pod is scheduled, ensuring the volume is created in the same AZ as the node. For workloads that genuinely need multi-replica shared storage, use the EFS CSI driver instead.

Online volume resizing requires allowVolumeExpansion: true in the StorageClass. To resize, update the PVC's spec.resources.requests.storage field. The EBS CSI driver handles the resize without requiring a pod restart — the filesystem is expanded while the volume is mounted. Shrinking EBS volumes is not supported; only expansion is allowed.

EKS Pod Identity: the simpler alternative to IRSA

IRSA (IAM Roles for Service Accounts) has been the standard way to give EKS pods AWS credentials since 2019. It works by annotating a Kubernetes ServiceAccount with an IAM role ARN, with the trust policy using OIDC subject claim conditions to scope the role to a specific namespace and service account name. It works well but has two friction points: you must set up an OIDC identity provider in IAM for each cluster, and the trust policy condition syntax is verbose and error-prone.

EKS Pod Identity (the eks-pod-identity-agent add-on) is a newer approach that moves OIDC token management entirely inside EKS. There is no OIDC identity provider to configure in IAM. Instead, you create an association between a service account and a role using the EKS API, and EKS injects credentials into pods via the instance metadata endpoint proxy (169.254.170.23) on the node.

# Step 1: Install the Pod Identity agent add-on
aws eks create-addon \
  --cluster-name my-cluster \
  --addon-name eks-pod-identity-agent \
  --resolve-conflicts PRESERVE

# Step 2: Create the IAM role with the correct trust policy
# Note: principal is pods.eks.amazonaws.com, not an OIDC provider ARN
cat > trust-policy.json <<'EOF'
{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": {
      "Service": "pods.eks.amazonaws.com"
    },
    "Action": ["sts:AssumeRole", "sts:TagSession"],
    "Condition": {
      "StringEquals": {
        "aws:SourceAccount": "123456789012"
      },
      "ArnLike": {
        "aws:SourceArn": "arn:aws:eks:us-east-1:123456789012:cluster/my-cluster"
      }
    }
  }]
}
EOF

aws iam create-role \
  --role-name ProbeWorkerRole \
  --assume-role-policy-document file://trust-policy.json

# Step 3: Create the Pod Identity association
aws eks create-pod-identity-association \
  --cluster-name my-cluster \
  --namespace mcp \
  --service-account probe-worker \
  --role-arn arn:aws:iam::123456789012:role/ProbeWorkerRole

The trust policy conditions are much simpler than IRSA's OIDC sub claim pattern: aws:SourceAccount restricts to your AWS account, and aws:SourceArn restricts to a specific cluster ARN. You do not need to remember the OIDC provider URL or the exact sub claim format (system:serviceaccount:NAMESPACE:SA_NAME). The agent architecture is a DaemonSet (eks-pod-identity-agent) on every node that intercepts credential requests from pods and exchanges the EKS-issued projected service account token for temporary IAM credentials.

Current limitation: EKS Pod Identity works for same-account role assumptions only (as of late 2026). If your workload needs to assume a role in a different AWS account (cross-account access), you must continue using IRSA for that specific use case. For all same-account credential needs — S3 access, DynamoDB, Secrets Manager, CloudWatch — Pod Identity is the recommended path for new clusters.

How AliveMCP uses EKS add-ons

AliveMCP probe workers write check results to a SQLite database file (mcp-probe.db) on a 20 GB gp3 PersistentVolume. The gp3 baseline of 3,000 IOPS is more than adequate for the 60-second batch write interval — SQLite batch commits at that frequency generate a burst of maybe 50–100 IOPS, well within the baseline. The EBS volume is ReadWriteOnce, which is fine because the probe worker is a single-replica Deployment with a single writer; SQLite's single-writer model matches this constraint naturally. If a future sharding design requires multiple probe-worker replicas writing separate shard databases, each shard gets its own gp3 PVC — still ReadWriteOnce, still one writer per volume. An EFS CSI migration would only be needed if multiple replicas needed to share a single database file, which is explicitly avoided in the current architecture.

Each probe-worker pod has dnsConfig.options: - name: ndots value: "2". AliveMCP monitors roughly 2,000 public MCP endpoints, each with a distinct external FQDN. With the default ndots:5, probing all 2,000 endpoints in a 60-second window would generate approximately 8,000 NXDOMAIN queries per minute against CoreDNS before the 2,000 correct lookups — a 5x DNS amplification factor. With ndots:2, each external FQDN (which has 3 or more dots) resolves in at most 2 queries: one potential search-suffix attempt, one correct bare-domain lookup. This keeps CoreDNS load proportional to probe volume rather than multiplied by the search domain list length. The OVERWRITE conflict policy is explicitly avoided on the CoreDNS add-on; PRESERVE is used so that the Corefile customizations (including the ndots override in the ConfigMap) survive add-on version upgrades.

Failure modes reference

Related guides

• IRSA: IAM Roles for Service Accounts on EKS for MCP servers
• EKS Pod Security Admission for MCP server workloads
• EKS cluster endpoint access control for MCP monitoring infrastructure
• Horizontal Pod Autoscaler for MCP servers on EKS
• Network policies for MCP servers on EKS