EKS cluster endpoint access control for MCP monitoring infrastructure

EKS endpoint types: what each mode actually does

The EKS control plane — the Kubernetes API server — is managed entirely by AWS and lives outside your VPC. How your nodes and your operators reach it depends on which endpoint modes you enable. The distinction matters because AWS nodes must be able to reach the API server continuously (for pod scheduling, certificate renewal, and drain operations), whereas human operators only need access for kubectl sessions.

A common misunderstanding: "public endpoint" does not mean the API server is exposed on a raw public IP with port 6443 visible in a port scan. AWS places it behind their own networking infrastructure. The risk is that anyone with a stolen kubeconfig (or exploited IAM credentials) can reach the API server from any IP — not that it is trivially port-scanned. That is why publicAccessCidrs matters: it restricts which source IPs can initiate connections to the public endpoint, regardless of credential validity.

# Check current endpoint configuration
aws eks describe-cluster --name my-cluster \
  --query 'cluster.resourcesVpcConfig.{publicAccess:endpointPublicAccess,privateAccess:endpointPrivateAccess,cidrs:publicAccessCidrs}'

# Enable private endpoint, restrict public endpoint to VPN + CI/CD CIDRs
aws eks update-cluster-config --name my-cluster \
  --resources-vpc-config \
    endpointPublicAccess=true,\
    endpointPrivateAccess=true,\
    publicAccessCidrs="203.0.113.0/24,198.51.100.10/32"
# 203.0.113.0/24 = office/VPN CIDR
# 198.51.100.10/32 = CI/CD runner fixed IP

The lock-out trap: if you set publicAccessCidrs to a CIDR that does not include your current IP, and you have not yet enabled the private endpoint, you will immediately lose access to the API server. If this happens, you can recover by updating the cluster from the AWS console (which bypasses the kubeconfig path) or by temporarily setting publicAccessCidrs back to 0.0.0.0/0 via the AWS CLI from a machine with access to the AWS API (not the Kubernetes API).

kubectl access patterns for private endpoint clusters

Once you enable private-only endpoints, your local kubectl cannot reach the API server directly. You need an access path that traverses your VPC. There are four common approaches, each with different operational costs.

Option 1: Bastion host

An EC2 instance in the same VPC with an SSH jump entry in your ~/.ssh/config. Operationally simple but expensive to maintain: you must patch the bastion OS, rotate SSH keys, and keep the bastion running 24/7. The bastion is also an attack surface — a compromised bastion gives an attacker VPC-interior access to the API server.

Option 2: VPC peering

Peer your developer VPC to the cluster VPC. Developers can run kubectl from a machine in the developer VPC without a hop. Requires route table entries on both sides that include the cluster VPC CIDR, and security groups that permit traffic on port 443 from the developer VPC CIDR range. Useful when your development environment already runs in a separate AWS account with Transit Gateway or VPC peering set up.

Option 3: AWS Client VPN

Users connect to AWS Client VPN before running kubectl. Once connected, their machine is logically inside the VPC and can reach the private endpoint directly. Scales well for larger teams. Requires configuring a Client VPN endpoint (cost: ~$0.10/hour plus ~$0.05/hour per active connection) and distributing client configuration files.

Option 4: SSM Session Manager port-forwarding (no bastion, no SSH keys)

This is the operationally cheapest option for teams that already use AWS SSM. You place a small EC2 instance inside the VPC — it needs no public IP, no inbound security group rules, and no SSH keys. The SSM agent phones home to AWS outbound. You then use the SSM StartPortForwardingSessionToRemoteHost document to forward a local port on your developer machine through that EC2 instance to the API server's private hostname.

# Step 1: Find the private API server hostname
aws eks describe-cluster --name my-cluster \
  --query 'cluster.endpoint' --output text
# Returns: https://XXXXXXXXXXXXXXXX.gr7.us-east-1.eks.amazonaws.com

# Step 2: Start SSM port-forwarding session
# Forwards localhost:6443 → EC2 instance → API server private hostname:443
aws ssm start-session \
  --target i-0abc1234def567890 \
  --document-name AWS-StartPortForwardingSessionToRemoteHost \
  --parameters '{
    "host": ["XXXXXXXXXXXXXXXX.gr7.us-east-1.eks.amazonaws.com"],
    "portNumber": ["443"],
    "localPortNumber": ["6443"]
  }'

# Step 3: In a separate terminal, use kubectl against localhost:6443
# The API server TLS cert is issued for its original hostname, not localhost,
# so you must either add the cert's SAN to /etc/hosts or skip TLS verification
# for the tunnel session (the tunnel itself is already encrypted by SSM).
kubectl --server=https://localhost:6443 \
  --insecure-skip-tls-verify \
  get nodes

The EC2 instance used for the tunnel needs an IAM instance profile with ssm:StartSession (granted via the AmazonSSMManagedInstanceCore managed policy). The instance requires outbound HTTPS to SSM endpoints (or a VPC endpoint for SSM). You need the SSM Session Manager plugin installed locally alongside the AWS CLI. No SSH, no inbound ports, no key rotation — the attack surface is the SSM service authorization plane, which is controlled by IAM.

EKS control plane logging: the five log types

EKS ships five categories of control plane logs to a CloudWatch Logs group named /aws/eks/CLUSTER_NAME/cluster. All five are disabled by default — you must enable them explicitly. Each covers a different part of the control plane's operation.

# Enable all five log types via AWS CLI
aws eks update-cluster-config --name my-cluster \
  --logging '{"clusterLogging":[{"types":["api","audit","authenticator","controllerManager","scheduler"],"enabled":true}]}'

# Enable via eksctl (preferred for cluster lifecycle management)
# In cluster.yaml:
cloudWatch:
  clusterLogging:
    enableTypes:
      - "api"
      - "audit"
      - "authenticator"
      - "controllerManager"
      - "scheduler"

Minimum viable logging: enable at least api and audit. The api logs give you latency and error rate data for the control plane itself. The audit logs give you the security view. Running without audit logs means that after a security incident — a compromised service account, an unauthorized kubectl delete, an unexpected exec session into a pod — you have no forensic trail. The cost of audit logs at moderate cluster size is negligible compared to the investigative cost of having none.

Audit log structure and CloudWatch Logs Insights queries

EKS audit log entries are JSON objects. The fields you will use most frequently in investigations are: verb (the HTTP verb mapped to a Kubernetes operation: create, get, list, update, patch, delete, watch), objectRef.resource (the Kubernetes resource type: pods, secrets, deployments), objectRef.namespace, user.username (for IAM users, this is the full ARN; for service accounts, it is system:serviceaccount:NAMESPACE:SA_NAME), user.groups, sourceIPs, and responseStatus.code.

Save these queries in the CloudWatch Logs Insights console as saved queries for your on-call runbook:

-- All kubectl apply operations by a specific IAM user
fields @timestamp, verb, objectRef.resource, objectRef.namespace, responseStatus.code
| filter verb = "create" or verb = "update" or verb = "patch"
| filter user.username = "arn:aws:iam::123456789012:user/developer-name"
| sort @timestamp desc
| limit 100

-- Unauthorized API calls (403 responses) — alert signal
fields @timestamp, user.username, verb, objectRef.resource, objectRef.namespace, sourceIPs
| filter responseStatus.code = 403
| sort @timestamp desc
| limit 200

-- All pod deletions in the last 24 hours — on-call runbook query
fields @timestamp, user.username, objectRef.name, objectRef.namespace
| filter verb = "delete" and objectRef.resource = "pods"
| sort @timestamp desc

-- API server errors (5xx) — control plane health indicator
fields @timestamp, verb, objectRef.resource, requestURI, responseStatus.code, responseStatus.message
| filter responseStatus.code >= 500
| sort @timestamp desc

-- Pod exec sessions — anomaly detection for probe worker clusters
fields @timestamp, user.username, objectRef.name, objectRef.namespace, sourceIPs
| filter requestURI like "/exec"
| sort @timestamp desc

Audit log retention and alerting configuration

CloudWatch Logs groups default to Never expire retention, but this is not the same as secure storage — anyone with IAM write access to the log group can delete log events. Set explicit retention to enforce a compliance window and reduce storage costs on verbose log types.

# Set 30-day retention on the audit log group
aws logs put-retention-policy \
  --log-group-name "/aws/eks/my-cluster/cluster" \
  --retention-in-days 30

# For tighter compliance, archive to S3 with object lock
# Use a CloudWatch Logs subscription filter → Kinesis Firehose → S3
aws logs put-subscription-filter \
  --log-group-name "/aws/eks/my-cluster/cluster" \
  --filter-name "audit-to-s3" \
  --filter-pattern "" \
  --destination-arn arn:aws:firehose:us-east-1:123456789012:deliverystream/eks-audit-archive

Create a metric filter on unauthorized API calls so CloudWatch Alarms can page your on-call team:

# Create metric filter: count 403 responses in audit logs
aws logs put-metric-filter \
  --log-group-name "/aws/eks/my-cluster/cluster" \
  --filter-name "UnauthorizedAPICallCount" \
  --filter-pattern '{ $.responseStatus.code = 403 }' \
  --metric-transformations \
    metricName=UnauthorizedAPICallCount,\
    metricNamespace=EKS/Security,\
    metricValue=1,\
    defaultValue=0

# Create alarm: alert if 5+ unauthorized calls in any 5-minute window
aws cloudwatch put-metric-alarm \
  --alarm-name "EKS-Unauthorized-API-Calls" \
  --metric-name UnauthorizedAPICallCount \
  --namespace EKS/Security \
  --statistic Sum \
  --period 300 \
  --threshold 5 \
  --comparison-operator GreaterThanOrEqualToThreshold \
  --evaluation-periods 1 \
  --alarm-actions arn:aws:sns:us-east-1:123456789012:on-call-alerts

One nuance: the api log type and the audit log type both land in the same CloudWatch log group but are differentiated by a logType field in the JSON. The api logs are far more voluminous — they include every list-watch long-poll from every node's kubelet — and add significant cost if you enable them. The audit logs are the security-relevant view and are much smaller. Enable audit unconditionally; enable api only when you are actively debugging API server latency.

How AliveMCP uses EKS endpoint access control

AliveMCP runs its probe-worker EKS cluster in private endpoint mode with the public endpoint disabled. Developers reach the API server exclusively via SSM Session Manager port-forwarding through a small EC2 instance in the cluster VPC — no public IP, no SSH keys, no bastion to patch. The EC2 instance uses an IAM instance profile scoped to SSM only. CI/CD pipelines (GitHub Actions) authenticate using IRSA and interact with the cluster via the private endpoint from a self-hosted runner in the same VPC. This eliminates the entire class of "rotate this SSH key or certificate" maintenance work that a bastion-based approach requires.

All five log types are enabled and shipped to CloudWatch. Audit log retention is set to 30 days — enough for a forensics window on any incident without accumulating unbounded storage costs. Two metric filters are active: one on responseStatus.code = 403 (unauthorized kubectl activity alert) and one on requestURI contains "/exec" (pod exec session alert). AliveMCP probe workers are stateless and never use kubectl exec; any exec session opening against a probe-worker pod is anomalous and pages the on-call engineer immediately. A saved Logs Insights query — "probe-worker pod deletions last 24h" — is part of the on-call runbook for investigating unexpected probe gaps.

Failure modes reference

Related guides

• IRSA: IAM Roles for Service Accounts on EKS for MCP servers
• EKS Pod Security Admission for MCP server workloads
• EKS Managed Add-Ons: VPC CNI, EBS CSI, CoreDNS, and Pod Identity for MCP servers
• Horizontal Pod Autoscaler for MCP servers on EKS
• Network policies for MCP servers on EKS