Node affinity and pod placement for MCP servers on EKS

Node affinity: controlling which nodes MCP pods land on

Node affinity is the structured replacement for nodeSelector. It lives at spec.affinity.nodeAffinity in your pod or Deployment spec and supports two scheduling modes:

• requiredDuringSchedulingIgnoredDuringExecution — hard rule. The pod will not schedule if no node matches. "IgnoredDuringExecution" means existing pods are not evicted if a node's labels change after scheduling.
• preferredDuringSchedulingIgnoredDuringExecution — soft preference with a weight (1–100). The scheduler prefers matching nodes but will fall back to non-matching nodes if necessary.

Common MCP server use case: require a specific instance family (memory-optimized r7g nodes for large tool context windows) while preferring a particular AZ for latency reasons:

# deployment-mcp-affinity.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mcp-server
  namespace: mcp
spec:
  replicas: 3
  selector:
    matchLabels:
      app: mcp-server
  template:
    metadata:
      labels:
        app: mcp-server
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                  - key: node.kubernetes.io/instance-type
                    operator: In
                    values:
                      - r7g.large
                      - r7g.xlarge
                      - r7g.2xlarge
          preferredDuringSchedulingIgnoredDuringExecution:
            - weight: 70
              preference:
                matchExpressions:
                  - key: topology.kubernetes.io/zone
                    operator: In
                    values:
                      - us-east-1a
            - weight: 30
              preference:
                matchExpressions:
                  - key: eks.amazonaws.com/nodegroup
                    operator: In
                    values:
                      - mcp-primary-ng
      containers:
        - name: mcp-server
          image: your-ecr-repo/mcp-server:latest
          resources:
            requests:
              cpu: "500m"
              memory: "1Gi"

Node labels EKS sets automatically: topology.kubernetes.io/zone (the AZ), node.kubernetes.io/instance-type (EC2 instance type), and eks.amazonaws.com/nodegroup (managed node group name). You don't need to add these — they're present on every EKS node.

Pod anti-affinity: keeping replicas on different nodes

Pod anti-affinity keeps your replicas spread by repelling pods that share the same label selector. The topologyKey defines what "same" means — kubernetes.io/hostname means "same physical node", topology.kubernetes.io/zone means "same AZ".

# pod-anti-affinity fragment — add to spec.template.spec.affinity
affinity:
  podAntiAffinity:
    # Hard rule: no two mcp-server pods on the same node
    requiredDuringSchedulingIgnoredDuringExecution:
      - labelSelector:
          matchExpressions:
            - key: app
              operator: In
              values:
                - mcp-server
        topologyKey: kubernetes.io/hostname
    # Soft preference: prefer different AZs
    preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 100
        podAffinityTerm:
          labelSelector:
            matchExpressions:
              - key: app
                operator: In
                values:
                  - mcp-server
          topologyKey: topology.kubernetes.io/zone

Critical limitation of hard pod anti-affinity: if you have 3 nodes and set requiredDuringSchedulingIgnoredDuringExecution on hostname, you can never scale beyond 3 replicas. The 4th pod will be Pending indefinitely because every node already has one mcp-server pod. Use preferredDuringSchedulingIgnoredDuringExecution if you need to scale beyond node count, or pair with topology spread constraints instead.

Topology spread constraints: flexible cross-AZ distribution

Topology spread constraints are a newer, more flexible replacement for pod anti-affinity for distribution use cases. Instead of "no two pods here", they express "the difference between the most-loaded and least-loaded topology domain should not exceed N".

# topology-spread-mcp.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mcp-server
  namespace: mcp
spec:
  replicas: 6
  selector:
    matchLabels:
      app: mcp-server
  template:
    metadata:
      labels:
        app: mcp-server
    spec:
      topologySpreadConstraints:
        # Cross-AZ spread: no AZ should have more than 1 extra pod vs others
        - maxSkew: 1
          topologyKey: topology.kubernetes.io/zone
          whenUnsatisfiable: DoNotSchedule
          labelSelector:
            matchLabels:
              app: mcp-server
        # Cross-node spread within each AZ
        - maxSkew: 1
          topologyKey: kubernetes.io/hostname
          whenUnsatisfiable: ScheduleAnyway  # soft — don't block if nodes are unequal
          labelSelector:
            matchLabels:
              app: mcp-server
      containers:
        - name: mcp-server
          image: your-ecr-repo/mcp-server:latest

The maxSkew: 1 rule with 3 AZs means pod counts across AZs can differ by at most 1. For 6 pods across 3 AZs you get 2-2-2. For 7 pods you get 3-2-2. This works cleanly.

The unsatisfiable case: 2 replicas, 3 AZs, maxSkew: 1, whenUnsatisfiable: DoNotSchedule. The only valid distributions are 1-1-0, but the constraint requires the max difference to be ≤ 1 — that means max is 1 and min is 0, difference is 1, which is actually satisfiable. But for maxSkew: 0 with 2 pods and 3 AZs it is impossible — keep this in mind when setting very tight skew values at low replica counts.

Dedicated node pools with taints and tolerations

If your MCP servers have special hardware requirements (GPUs, high memory, specific instance families) or you want to prevent noisy-neighbor interference from other workloads, create a dedicated node group and taint it so only MCP pods land there.

Add the taint to the EKS managed node group (eksctl example):

# eksctl cluster config fragment
managedNodeGroups:
  - name: mcp-dedicated
    instanceType: r7g.xlarge
    minSize: 2
    maxSize: 10
    taints:
      - key: dedicated
        value: mcp
        effect: NoSchedule   # pods without matching toleration won't schedule here
    labels:
      workload: mcp-server
    tags:
      k8s.io/cluster-autoscaler/enabled: "true"
      k8s.io/cluster-autoscaler/my-cluster: "owned"

Add the matching toleration to the MCP pod spec:

# pod spec fragment — tolerations allow scheduling on the tainted node group
spec:
  tolerations:
    - key: dedicated
      operator: Equal
      value: mcp
      effect: NoSchedule
  # Also add nodeAffinity to *require* landing on these nodes
  # (toleration alone only permits — it doesn't enforce)
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchExpressions:
              - key: workload
                operator: In
                values:
                  - mcp-server

Important: a toleration is permission, not direction. Adding only a toleration lets the pod schedule on the tainted nodes but doesn't force it there — the scheduler may still place it on untainted nodes. Add a requiredDuringScheduling nodeAffinity for the node group label to enforce exclusive placement.

Spot instance pattern: use NoSchedule taint on spot node groups and add tolerations only to fault-tolerant MCP workloads. Keep critical MCP servers on on-demand nodes via preferredDuringScheduling nodeAffinity for the on-demand node group label with a high weight.

Failure modes reference