AWS SSM Operations for MCP Servers: Run Command, Session Manager, Maintenance Windows, and Patch Manager

Pattern 1 — The Shared Infrastructure Model

One policy, four tools, one CDK stack

Every SSM tool — Run Command, Session Manager, Maintenance Windows, Patch Manager — requires the same two things on the instance side: the SSM Agent running and the AmazonSSMManagedInstanceCore managed policy on the instance role. Both are free. The managed policy grants permission for the agent to register with the SSM service, receive commands, and write output to CloudWatch Logs and S3. It does not grant permission to call SSM APIs from application code — that is a separate role and a separate concern.

In CDK, the minimal MCP server instance setup that enables all four tools:

import * as ec2 from 'aws-cdk-lib/aws-ec2';
import * as iam from 'aws-cdk-lib/aws-iam';
import * as cdk from 'aws-cdk-lib';

const instanceRole = new iam.Role(this, 'McpInstanceRole', {
  assumedBy: new iam.ServicePrincipal('ec2.amazonaws.com'),
  managedPolicies: [
    iam.ManagedPolicy.fromAwsManagedPolicyName('AmazonSSMManagedInstanceCore'),
  ],
});

const instance = new ec2.Instance(this, 'McpServer', {
  vpc,
  instanceType: ec2.InstanceType.of(ec2.InstanceClass.T3, ec2.InstanceSize.SMALL),
  machineImage: ec2.MachineImage.latestAmazonLinux2023(), // SSM Agent pre-installed
  role: instanceRole,
  // No keyPair — SSM Session Manager replaces SSH
  vpcSubnets: { subnetType: ec2.SubnetType.PRIVATE_WITH_EGRESS },
  securityGroup: mcpSecurityGroup, // no inbound rules at all
});

// Tags are the universal targeting mechanism across all four SSM tools
cdk.Tags.of(instance).add('Service', 'mcp-server');
cdk.Tags.of(instance).add('Env', 'production');
cdk.Tags.of(instance).add('Patch Group', 'mcp-production'); // Patch Manager targeting

The Service=mcp-server and Env=production tags flow through to Run Command's --targets parameter, to Session Manager's IAM condition key scoping, to Maintenance Window target registration, and — via the Patch Group tag — to Patch Manager's baseline assignment. Define them once at instance launch and they work across all four tools without further configuration.

VPC endpoints for private subnets

MCP servers in private subnets without a NAT gateway cannot reach the SSM service endpoints over the public internet. Three VPC Interface Endpoints are required — and all three must be present; missing any one causes the SSM Agent to fail silently with no informative error to CloudWatch.

const ssmEndpoints = [
  'com.amazonaws.us-east-1.ssm',
  'com.amazonaws.us-east-1.ssmmessages',
  'com.amazonaws.us-east-1.ec2messages',
];

for (const service of ssmEndpoints) {
  new ec2.InterfaceVpcEndpoint(this, `Endpoint-${service.split('.').pop()}`, {
    vpc,
    service: new ec2.InterfaceVpcEndpointService(service),
    subnets: { subnetType: ec2.SubnetType.PRIVATE_ISOLATED },
    securityGroups: [endpointSecurityGroup],
    privateDnsEnabled: true, // required — SSM Agent uses standard regional DNS names
  });
}

privateDnsEnabled: true is mandatory. The SSM Agent resolves ssm.us-east-1.amazonaws.com using the standard regional DNS hostname. Without private DNS enabled on the endpoint, that name resolves to a public IP the instance cannot reach. The endpoint exists but the agent cannot find it. The symptom is the instance appearing as "Connection Lost" in Fleet Manager with no actionable error in instance logs.

The ssmmessages endpoint is the WebSocket relay used by Session Manager; ssm is the main API; ec2messages carries Run Command message delivery. All three are required regardless of which SSM tools you use — the agent needs all three endpoints to stay registered.

Pattern 2 — Access Control and the Audit Stack

Scoping Session Manager access by tag

By default, any IAM principal with ssm:StartSession on resource * can session into any SSM-managed instance in the account. For MCP server environments that hold downstream API keys, that default is wrong. Production MCP server instances should only be accessible to on-call engineers or a designated incident-response role. The ssm:resourceTag IAM condition key lets you enforce this at the policy level using the same tags you attached at instance launch.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowSessionToMcpProductionOnly",
      "Effect": "Allow",
      "Action": [
        "ssm:StartSession",
        "ssm:TerminateSession",
        "ssm:ResumeSession",
        "ssm:DescribeSessions",
        "ssm:GetConnectionStatus"
      ],
      "Resource": [
        "arn:aws:ec2:*:*:instance/*",
        "arn:aws:ssm:*:*:document/AWS-StartPortForwardingSession"
      ],
      "Condition": {
        "StringEquals": {
          "ssm:resourceTag/Service": "mcp-server",
          "ssm:resourceTag/Env": "production"
        }
      }
    },
    {
      "Sid": "DenySessionToNonMcpInstances",
      "Effect": "Deny",
      "Action": "ssm:StartSession",
      "Resource": "arn:aws:ec2:*:*:instance/*",
      "Condition": {
        "StringNotEquals": {
          "ssm:resourceTag/Service": "mcp-server"
        }
      }
    }
  ]
}

The explicit Deny in the second statement is necessary. Without it, a principal could have a separate broad Allow policy elsewhere in their attached roles and the condition on the first statement would not protect non-MCP instances. The Deny overrides any Allow — an engineer with this policy cannot session into a database instance, a build server, or a bastion host even if a separate policy would allow it.

This policy works regardless of instance count or instance ID. Tag every new MCP server instance at CDK deploy time and it automatically falls under the policy. Decommission an instance and remove its tags; the policy no longer covers it. No policy updates needed as the fleet scales.

Session logging: two controls, two audiences

For MCP servers handling sensitive downstream credentials, session logging serves two distinct purposes that require two separate controls. CloudWatch session streaming captures all session activity for your security team's audit and post-incident investigation. Setting HISTFILE=/dev/null in the shell profile prevents commands from writing to the instance's bash history, protecting against secrets leaking into files that other users on the same instance can read.

import * as ssm from 'aws-cdk-lib/aws-ssm';
import * as logs from 'aws-cdk-lib/aws-logs';

const sessionLogGroup = new logs.LogGroup(this, 'SsmSessionLogs', {
  logGroupName: '/mcp/ssm-sessions',
  retention: logs.RetentionDays.NINETY_DAYS,
});

// Document name must be exactly SSM-SessionManagerRunShell for automatic application
const sessionPreferences = new ssm.CfnDocument(this, 'SessionPreferences', {
  name: 'SSM-SessionManagerRunShell',
  documentType: 'Session',
  content: {
    schemaVersion: '1.0',
    sessionType: 'InteractiveCommands',
    inputs: {
      cloudWatchLogGroupName: sessionLogGroup.logGroupName,
      cloudWatchEncryptionEnabled: true,
      cloudWatchStreamingEnabled: true,
      idleSessionTimeout: '20',
      maxSessionDuration: '60',
      shellProfile: {
        // Suppress instance-level history; CloudWatch captures everything at session level
        linux: 'export HISTFILE=/dev/null; echo "Session on $(hostname) at $(date)"',
      },
    },
  },
});

The document name SSM-SessionManagerRunShell is not arbitrary — this exact name is the default that SSM applies automatically to all sessions in the account. Name it anything else and you must explicitly specify it on each start-session call, which means engineers working from the CLI during an incident may bypass logging entirely. Use the exact name and the preferences apply to every session automatically.

The instance role also needs logs:CreateLogStream and logs:PutLogEvents on the session log group ARN. The AmazonSSMManagedInstanceCore managed policy does not include CloudWatch Logs write permissions — you must add them explicitly or sessions will connect successfully but log silently.

Run Command: caller audit trail and operator policy

Every Run Command invocation is recorded in CloudWatch Logs (when CloudWatchOutputConfig is configured) with the executing IAM identity, the exact document name and version, all parameters, and the full stdout/stderr output from each targeted instance. This is the audit trail that makes post-incident analysis tractable: "what command ran on which instance, in which order, by which IAM role, with what output."

The caller IAM policy for Run Command operators should scope document access to your own documents rather than granting access to all built-in documents:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "ssm:SendCommand",
        "ssm:GetCommandInvocation",
        "ssm:ListCommandInvocations",
        "ssm:ListCommands"
      ],
      "Resource": [
        "arn:aws:ec2:*:*:instance/*",
        "arn:aws:ssm:*:*:document/AWS-RunShellScript",
        "arn:aws:ssm:*:*:document/AWS-RunPatchBaseline",
        "arn:aws:ssm:*:*:document/mcp-*"
      ]
    }
  ]
}

Scoping to arn:aws:ssm:*:*:document/mcp-* for custom documents means that an operator with this policy can run your approved runbooks but cannot run arbitrary AWS-RunShellScript commands outside of the explicitly listed built-in documents. For key rotation and config reload operations, this scoping matters: the runbook document encodes the approved procedure, and operators execute the runbook — not ad-hoc shell commands.

Pattern 3 — Fleet Operations: Scheduled Maintenance and Incident Response

Maintenance Windows: the scheduling model

A Maintenance Window is not an orchestration engine — it is a scheduling construct with four components: a cron or rate schedule, a duration in hours (how long the window stays open), a cutoff (how many hours before the window end to stop dispatching new tasks), and a set of tasks registered against tagged targets. The window knows when to open and when to stop dispatching; it does not know whether the tasks inside it succeeded. That validation is external — from Run Command's CloudWatch output and from AliveMCP's probe history.

The CDK stack for a weekly restart window:

import * as ssm from 'aws-cdk-lib/aws-ssm';
import * as iam from 'aws-cdk-lib/aws-iam';

// Weekly: Sunday 03:00 UTC, 2-hour window, stop dispatching 1 hour before close
const maintenanceWindow = new ssm.CfnMaintenanceWindow(this, 'McpMaintenanceWindow', {
  name: 'mcp-server-weekly-maintenance',
  schedule: 'cron(0 3 ? * SUN *)',
  duration: 2,
  cutoff: 1,
  allowUnassociatedTargets: false,
  enabled: true,
  scheduleTimezone: 'UTC',
});

const windowTargets = new ssm.CfnMaintenanceWindowTarget(this, 'McpWindowTargets', {
  windowId: maintenanceWindow.ref,
  resourceType: 'INSTANCE',
  targets: [
    { key: 'tag:Service', values: ['mcp-server'] },
    { key: 'tag:Env', values: ['production'] },
  ],
  name: 'mcp-production-fleet',
});

const maintenanceRole = new iam.Role(this, 'MaintenanceWindowRole', {
  assumedBy: new iam.ServicePrincipal('ssm.amazonaws.com'),
  managedPolicies: [
    iam.ManagedPolicy.fromAwsManagedPolicyName('service-role/AmazonSSMMaintenanceWindowRole'),
  ],
});

const restartTask = new ssm.CfnMaintenanceWindowTask(this, 'McpRestartTask', {
  windowId: maintenanceWindow.ref,
  taskArn: 'AWS-RunShellScript',
  taskType: 'RUN_COMMAND',
  targets: [{ key: 'WindowTargetIds', values: [windowTargets.ref] }],
  priority: 1,
  maxConcurrency: '25%',
  maxErrors: '20%',
  serviceRoleArn: maintenanceRole.roleArn,
  taskInvocationParameters: {
    maintenanceWindowRunCommandParameters: {
      parameters: {
        commands: [
          'systemctl restart mcp-server',
          'sleep 10',
          'systemctl is-active mcp-server && echo "RESTART_OK" || echo "RESTART_FAILED"',
        ],
      },
      cloudWatchOutputConfig: {
        cloudWatchLogGroupName: '/mcp/maintenance',
        cloudWatchOutputEnabled: true,
      },
      timeoutSeconds: 300,
    },
  },
});

Understanding cutoff behavior

Cutoff is the most commonly misunderstood Maintenance Window setting. It controls when SSM stops dispatching new task invocations — not when running invocations are killed. With a 2-hour window and 1-hour cutoff, tasks dispatched in the first hour continue until completion or their own TimeoutSeconds limit. Tasks not yet dispatched when the cutoff fires are cancelled.

For a rolling fleet restart, estimate the required window duration before creating the window:

const instanceCount = 100;
const concurrencyPercent = 0.25;           // 25%
const batchSize = Math.ceil(instanceCount * concurrencyPercent); // 25 instances/batch
const minutesPerBatch = 3;                 // restart + 10s sleep + health check
const batches = Math.ceil(instanceCount / batchSize); // 4 batches
const totalMinutes = batches * minutesPerBatch; // 12 minutes total

// For a patch install with reboot (20 min/instance, 10% concurrency):
const patchInstancesPerBatch = Math.ceil(100 * 0.10); // 10 instances/batch
const patchBatches = 100 / patchInstancesPerBatch;    // 10 batches
const patchTotalMinutes = patchBatches * 20;           // 200 minutes
const patchDurationHours = Math.ceil(patchTotalMinutes / 60) + 1; // 5 hours
// Set duration=5, cutoff=1 for patch install windows

Setting the wrong duration is the most common reason a fleet emerges from a Maintenance Window with a subset of instances that never received the task — the cutoff fired before all batches were dispatched. For restarts (2–3 minutes each), a 2-hour window is more than sufficient for fleets under 1,000 instances. For OS patch installs that may trigger kernel updates and reboots (20–30 minutes each at 10% concurrency), size the window at approximately ceil((instances / concurrency_count) * minutes_per_instance / 60) + 1 hours.

Patch Manager: from compliance scan to install

Patch Manager operates in two modes. Scan mode reads the instance's installed packages against the approved patch baseline and writes compliance data to Systems Manager — it changes nothing on the instance. Install mode applies missing patches and reboots if required. The two modes are separate invocations and the compliance data is not automatically refreshed after an Install — you need a follow-up Scan to update the compliance view in the console.

Run a fleet-wide compliance scan at any time without disruption:

aws ssm send-command \
  --document-name "AWS-RunPatchBaseline" \
  --targets '[{"Key":"tag:Service","Values":["mcp-server"]}]' \
  --parameters '{"Operation":["Scan"]}' \
  --max-concurrency "100%" \
  --cloud-watch-output-config '{"CloudWatchLogGroupName":"/mcp/patch-scans","CloudWatchOutputEnabled":true}' \
  --output text \
  --query "Command.CommandId"

100% concurrency is safe for Scan because it changes nothing. For Install, always use a Maintenance Window with 10% concurrency and never run Operation=Install directly with send-command on a production fleet — patching triggers reboots, and uncontrolled concurrent reboots take the entire fleet offline simultaneously.

For deployments where the default AWS-managed patch baselines are insufficient — you're managing Node.js with nvm and need to block the system package manager from upgrading it, or you're pinning a specific openssl version for a compliance requirement — create a custom CfnPatchBaseline with rejectedPatchesAction: 'BLOCK':

const mcpPatchBaseline = new ssm.CfnPatchBaseline(this, 'McpPatchBaseline', {
  name: 'mcp-server-patch-baseline',
  operatingSystem: 'AMAZON_LINUX_2023',
  approvalRules: {
    patchRules: [{
      patchFilterGroup: {
        patchFilters: [
          { key: 'CLASSIFICATION', values: ['Security', 'Bugfix'] },
          { key: 'SEVERITY', values: ['Critical', 'Important'] },
        ],
      },
      approveAfterDays: 7,
    }],
  },
  // Block packages that would upgrade Node.js from the nvm-managed version
  rejectedPatches: ['nodejs20', 'nodejs18'],
  rejectedPatchesAction: 'BLOCK', // takes precedence over approval rules
  patchGroups: ['mcp-production'], // matches the Patch Group tag value
});

BLOCK takes precedence over approval rules — if a rejected package appears in the approval criteria, it is still blocked. This is the correct behavior for pinned runtimes: you want security patches to install automatically, but not if they would silently upgrade a pinned runtime dependency. Use ALLOW_AS_DEPENDENCY if you want to allow a blocked package to install when pulled in as a dependency of an approved package (for transitive dependency scenarios).

The incident response workflow with AliveMCP

AliveMCP monitors MCP servers from outside the AWS account using a real JSON-RPC tools/list handshake every 60 seconds. When a probe starts failing, the incident workflow that SSM enables is faster than any SSH-based alternative:

1. AliveMCP alert fires → on-call engineer is paged.
2. Engineer opens AliveMCP dashboard → identifies which instance ID is failing and since when.
3. aws ssm start-session --target i-{id} — no bastion host, no VPN, no SSH key lookup. Session opens in under 10 seconds.
4. On the instance: systemctl status mcp-server, journalctl -u mcp-server -n 100 --no-pager.
5. If the issue is a hung process or memory overflow: systemctl restart mcp-server.
6. AliveMCP probe recovers (next 60-second interval confirms JSON-RPC response) → alert resolves.
7. Post-mortem: pull the Run Command history or session log from CloudWatch to see whether this instance had the restart applied correctly in the last Maintenance Window.

For debugging a probe failure where the LB health check still shows the instance as healthy — a common AliveMCP detection pattern where the load balancer uses a shallow TCP check but AliveMCP's JSON-RPC probe is deeper — use port forwarding to reproduce exactly what AliveMCP sees from the instance's perspective:

# Forward localhost:13000 → instance:3000 (MCP server JSON-RPC port)
aws ssm start-session \
  --target i-0abc123def456789a \
  --document-name AWS-StartPortForwardingSession \
  --parameters '{"portNumber":["3000"],"localPortNumber":["13000"]}'

# In a separate terminal: send the same probe AliveMCP sends
curl -s -X POST http://localhost:13000/ \
  -H 'Content-Type: application/json' \
  -d '{"jsonrpc":"2.0","id":1,"method":"tools/list","params":{}}' | jq .

If the port-forwarded probe returns a valid tools/list response but AliveMCP shows the instance as down, the failure is upstream of the instance (load balancer misconfiguration, security group change, Route 53 health check). If the port-forwarded probe also fails, the MCP server process itself is unhealthy — the LB is presenting a stale health check result from before the failure.

AliveMCP during scheduled maintenance

AliveMCP continues probing during Maintenance Windows by design. The probe validates that the maintenance operation actually recovered the server — not just that the maintenance script completed. During a rolling restart with 25% concurrency, the expected AliveMCP pattern is: batch of 25 instances go offline (probe failures), recover within 2–3 minutes (probe success), next batch goes offline, and so on. The aggregate probe success rate for the fleet dips during each batch but never reaches zero because 75% of instances are always running while the current batch restarts.

Two deviations from this expected pattern that warrant immediate investigation:

• Aggregate probe rate drops to zero — all instances offline simultaneously. This means MaxConcurrency was set to 100% or the Maintenance Window dispatched all batches before AliveMCP could confirm individual instance recovery. Check the Run Command execution history for the window's task ID in CloudWatch.
• Individual instance does not recover within 5 minutes of its restart — the restart command succeeded (otherwise MaxErrors would have halted the window) but the MCP server process failed to start after the systemd restart. Use Session Manager to access the instance and run journalctl -u mcp-server -n 50 to see the startup failure reason.

After the Maintenance Window closes, verify that every instance in the AliveMCP dashboard has returned to green before considering maintenance complete. A single instance still failing after the window is a residual — not an expected post-maintenance state.

Failure modes consolidated

Production checklist

Shared infrastructure

• AmazonSSMManagedInstanceCore attached to every EC2 instance role
• Service=mcp-server, Env=production, and Patch Group=mcp-production tags on every instance at launch
• Three SSM VPC Interface Endpoints deployed with privateDnsEnabled: true for private subnets
• Session Manager plugin installed on every operator's workstation

Access control and audit

• Session Manager access policy uses ssm:resourceTag/Service condition with explicit Deny on non-MCP instances
• Session preferences document named exactly SSM-SessionManagerRunShell with CloudWatch streaming enabled
• Instance role includes logs:CreateLogStream and logs:PutLogEvents for session log group
• Run Command caller policy scoped to specific documents, not *

Scheduled maintenance

• Maintenance Window duration estimated using the formula: ceil((instance_count / concurrency_count) × minutes_per_task / 60) + 1
• Cutoff set to 1 hour (for 2-4 hour windows) to ensure in-progress tasks complete
• Restart task uses 25% concurrency; patch install uses 10% concurrency
• Post-patch follow-up Scan scheduled to refresh compliance data after Install window
• AliveMCP alert thresholds account for expected per-instance downtime during rolling restarts
• Post-window check: verify all instances return to green in AliveMCP dashboard before closing the incident