NLB Health Checks for MCP Server Endpoints

TCP vs HTTP health checks: why TCP isn't enough for MCP

NLB supports three health check protocols: TCP, HTTP, and HTTPS. A TCP health check opens a connection to the target port and immediately closes it — if the connection succeeds, the target is "healthy." An HTTP check opens a connection, sends a GET request to a path, and evaluates the response code.

For MCP servers, TCP health checks produce false positives in two common failure scenarios:

• Node.js event loop blocked: A CPU-intensive MCP tool handler blocks the event loop. The TCP port is still open (the OS handles TCP handshake), but new HTTP requests queue behind the blocked handler. NLB's TCP check passes; clients time out. AliveMCP's external probe catches this as a slow response (>5s) and marks the endpoint degraded.
• Database connection pool exhausted: The MCP server's SQLite or PostgreSQL connection pool is full. New tool calls fail with 503; the port is still listening. TCP check passes; HTTP check on /health fails if the health endpoint exercises the pool (e.g., SELECT 1).

// CDK: HTTP health check for MCP server target group
const targetGroup = new elbv2.NetworkTargetGroup(scope, 'McpTg', {
  vpc,
  port: 8080,
  protocol: elbv2.Protocol.TCP,
  targetType: elbv2.TargetType.IP,
  healthCheck: {
    // HTTP check validates the server processes requests, not just accepts TCP connections
    protocol: elbv2.Protocol.HTTP,
    path: '/health',
    port: '8080',
    // Minimum NLB interval is 10 seconds (ALB can go to 5s)
    interval: Duration.seconds(10),
    // Mark unhealthy after 2 consecutive failures (20 seconds)
    unhealthyThresholdCount: 2,
    // Mark healthy after 2 consecutive successes (20 seconds recovery)
    healthyThresholdCount: 2,
    // Accept 200-299 as healthy; 503 from an overloaded pool fails the check
    // NLB only supports a single status code, not ranges — specify 200
    // (ALB supports ranges like 200-499; NLB does not)
  },
  // 120s = ECS container stop timeout; in-flight SSE streams drain before SIGKILL
  deregistrationDelay: Duration.seconds(120),
});

One sharp NLB edge: NLB HTTP health checks only accept a single HTTP response code, not a range. ALB lets you specify 200-499 to treat client errors as healthy; NLB requires specifying 200. If your /health endpoint returns 204 (No Content), you must configure NLB with 200 and change the endpoint to return 200 with a body — or keep it 204 and NLB marks it unhealthy.

Health check timing: how long until NLB stops sending traffic

The time from failure to traffic removal is:

time_to_unhealthy = interval × unhealthy_threshold
                  = 10s × 2 = 20 seconds

And time from fix to traffic restoration:

time_to_healthy = interval × healthy_threshold
                = 10s × 2 = 20 seconds

During the 20-second window between first failure and traffic removal, NLB continues sending new connections to the failing target. Clients see failures. The right way to minimize user impact is:

1. ECS container stop signal → MCP server's SIGTERM handler begins graceful shutdown, stops accepting new SSE subscriptions, returns 503 from /health
2. NLB health check sees two consecutive 503 responses (20 seconds)
3. NLB marks target draining — no new connections, existing SSE streams continue
4. Deregistration delay timer starts (120 seconds)
5. After 120 seconds, existing connections are RST and the target is deregistered
6. ECS sends SIGKILL if the container is still running (120s matches the stopTimeout)

AliveMCP monitors the NLB UnHealthyHostCount metric. When a target enters the draining state, AliveMCP will see probe failures during the 20-second health check window, then see probe results resume as the replaced task becomes healthy (another 20 seconds). The total outage window visible externally: ~40 seconds per rolling deployment with a healthy replacement task.

Slow start: warming up cold MCP containers

MCP servers often have significant startup latency: they load a tool registry from a database, initialize HTTP clients, pre-compile JSON schema validators, or establish connection pools. A freshly started ECS task that passes the first two health checks (20 seconds after starting) may still be handling requests slowly while warm-up completes in the background.

NLB target groups support a slow start duration (30–900 seconds) that gradually ramps traffic to newly healthy targets:

// CDK: slow start via L1 CfnTargetGroup override
import * as elbv2 from 'aws-cdk-lib/aws-elasticloadbalancingv2';
import { CfnTargetGroup } from 'aws-cdk-lib/aws-elasticloadbalancingv2';

const targetGroup = new elbv2.NetworkTargetGroup(scope, 'McpTg', {
  // ... health check config as above
});

// L1 override to set slow start duration (CDK L2 doesn't expose this yet for NLB)
const cfnTg = targetGroup.node.defaultChild as CfnTargetGroup;
cfnTg.addPropertyOverride('SlowStartDurationSeconds', 60);
// During slow start, NLB uses round-robin with linearly increasing weight for the new target
// At 0s: new target gets 0% of traffic; at 60s: full traffic weight restored

With 60-second slow start and a 20-second healthy threshold, the new target receives its first health check pass at T+20s, enters slow start, ramps to full traffic weight by T+80s. The total cold-start impact window is 80 seconds vs 20 seconds without slow start — but P99 latency during the ramp is much lower because the cold target only handles a fraction of traffic while it's still warming up.

HTTPS health checks: validating TLS certificate chain

If your MCP server uses end-to-end TLS (NLB in TCP pass-through mode, TLS terminates at the container), NLB can perform HTTPS health checks that also validate the server's TLS certificate:

healthCheck: {
  protocol: elbv2.Protocol.HTTPS,
  path: '/health',
  port: '8080',
  interval: Duration.seconds(10),
  unhealthyThresholdCount: 2,
  healthyThresholdCount: 2,
  // NLB validates TLS cert during health check — expired or self-signed cert fails the check
  // unless you explicitly allow self-signed via target group attribute
}

NLB's HTTPS health check validates the certificate chain by default. For MCP servers with self-signed internal certificates, this causes health check failures. Two options:

• Use HTTP health check on a non-TLS port: Expose port 8080 for plain HTTP health checks and port 8443 for TLS MCP traffic. NLB checks port 8080 with HTTP, serves clients on port 8443 with TLS. Simplest solution.
• Use ACM certificates on the backend: ACM private CA issues internal TLS certs trusted by NLB. Health checks pass because the cert chain is valid. Adds cert management complexity.

AliveMCP's external probes independently verify TLS certificate validity for monitored MCP endpoints. If a monitored endpoint's TLS certificate expires, AliveMCP marks it degraded and alerts the owner — separate from whether NLB's internal health check passes.

Monitoring health check failures with CloudWatch

import * as cloudwatch from 'aws-cdk-lib/aws-cloudwatch';
import * as cloudwatchActions from 'aws-cdk-lib/aws-cloudwatch-actions';

// Alarm: any targets failing health checks (leading indicator of outage)
const unhealthyAlarm = new cloudwatch.Alarm(scope, 'UnhealthyTargets', {
  alarmName: 'mcp-nlb-unhealthy-targets',
  alarmDescription: 'One or more MCP server tasks failing NLB health checks',
  metric: new cloudwatch.Metric({
    namespace: 'AWS/NetworkELB',
    metricName: 'UnHealthyHostCount',
    dimensionsMap: {
      LoadBalancer: nlb.loadBalancerFullName,
      TargetGroup: targetGroup.targetGroupFullName,
    },
    period: Duration.minutes(1),
    statistic: 'Average',
  }),
  threshold: 0,
  comparisonOperator: cloudwatch.ComparisonOperator.GREATER_THAN_THRESHOLD,
  // Evaluate for 3 consecutive minutes before alarming — avoids false positives during deployments
  evaluationPeriods: 3,
  treatMissingData: cloudwatch.TreatMissingData.NOT_BREACHING,
});

// Critical alarm: all targets unhealthy (full outage)
const zeroHealthyAlarm = new cloudwatch.Alarm(scope, 'NoHealthyTargets', {
  alarmName: 'mcp-nlb-no-healthy-targets',
  alarmDescription: 'All MCP server tasks unhealthy — total probe coverage outage',
  metric: new cloudwatch.Metric({
    namespace: 'AWS/NetworkELB',
    metricName: 'HealthyHostCount',
    dimensionsMap: {
      LoadBalancer: nlb.loadBalancerFullName,
      TargetGroup: targetGroup.targetGroupFullName,
    },
    period: Duration.minutes(1),
    statistic: 'Minimum',
  }),
  threshold: 1,
  comparisonOperator: cloudwatch.ComparisonOperator.LESS_THAN_THRESHOLD,
  evaluationPeriods: 1,     // alert immediately on full outage
  treatMissingData: cloudwatch.TreatMissingData.BREACHING,
});

// Both alarms notify the same SNS topic; on-call receives PagerDuty alert
[unhealthyAlarm, zeroHealthyAlarm].forEach(alarm =>
  alarm.addAlarmAction(new cloudwatchActions.SnsAction(alertTopic)),
);

The UnHealthyHostCount > 0 alarm with a 3-minute evaluation window strikes the right balance: it catches real health check failures without firing on every deployment (where one target briefly fails during rolling update). The HealthyHostCount < 1 alarm fires immediately — zero healthy targets means all probe traffic is being dropped, which is an immediate outage.

Failure modes