WebSocket and SSE Connections over NLB for MCP Servers

NLB TCP idle timeout: the 350-second rule

NLB tracks every TCP connection as a "flow" entry in its connection table. A flow entry expires after 350 seconds of inactivity — no bytes in either direction. When the entry expires, NLB sends a TCP RST to both the client and the target.

MCP SSE connections are one-directional during quiet periods: the server sends events, the client only reads them. If an MCP server has no events to send (e.g., all monitored endpoints are healthy and the client is subscribed to the live status feed), the connection is silent from the client side. The server may also be silent if it's waiting for a new probe cycle. This creates idle periods that can exceed 350 seconds and trigger NLB's timeout RST.

For AliveMCP's live status feed (SSE stream that pushes events when endpoint status changes), quiet periods during "all healthy" windows can last several minutes. Setting ALB idle timeout to 400+ seconds avoids the issue but requires explicit configuration and has an upper limit. NLB's 350s is better than ALB's 60s default but still requires keep-alives for long-quiet SSE streams.

Keep-alive strategies: preventing NLB timeout on quiet SSE streams

Two independent keep-alive mechanisms work together to prevent NLB from RST-ing quiet SSE connections:

1. SSE comment keep-alives (application layer)

The SSE specification allows comment lines that the browser ignores: a line starting with : followed by any text. Sending these every 30 seconds creates traffic in the NLB flow and resets the 350-second idle timer.

// MCP SSE server: send keep-alive comments every 30 seconds
function createMcpSseHandler(req, res) {
  res.writeHead(200, {
    'Content-Type': 'text/event-stream',
    'Cache-Control': 'no-cache',
    'Connection': 'keep-alive',
    'X-Accel-Buffering': 'no',   // prevents Nginx/proxy buffering of SSE chunks
  });

  // Send initial connection event
  res.write('data: {"type":"connected","serverTime":"' + new Date().toISOString() + '"}\n\n');

  // Keep-alive: comment line every 30s (well within NLB's 350s timeout)
  const keepAliveTimer = setInterval(() => {
    if (res.writableEnded) {
      clearInterval(keepAliveTimer);
      return;
    }
    res.write(': keepalive\n\n');
  }, 30_000);

  req.on('close', () => {
    clearInterval(keepAliveTimer);
  });

  // Send real MCP events when they arrive
  const unsubscribe = subscribeToProbeResults((event) => {
    if (res.writableEnded) return;
    res.write('data: ' + JSON.stringify(event) + '\n\n');
  });

  req.on('close', unsubscribe);
}

2. TCP socket keep-alive (transport layer)

TCP keep-alive probes run at the OS/transport layer independently of application data. When enabled, the OS sends TCP ACK probes after a period of inactivity. If the other end responds, the connection is considered alive and the NLB flow entry is reset.

// Node.js: enable TCP keep-alive on accepted sockets
const server = require('node:http').createServer(app);
server.on('connection', (socket) => {
  // Enable TCP keep-alive after 60 seconds of inactivity
  socket.setKeepAlive(true, 60_000);
  // socket.keepAlive = true at the OS level; probes sent every ~75s by default on Linux
});

// Or globally via server options (Node.js v22+):
const server = require('node:http').createServer({
  keepAliveTimeout: 120_000,   // server-side HTTP keep-alive timeout (matches ALB/NLB)
  headersTimeout: 60_000,
}, app);

With both mechanisms active, the effective keep-alive interval is 30 seconds (SSE comment) + TCP keep-alive at 60 seconds as backup. The NLB flow entry is refreshed every 30 seconds; the 350-second timeout is never reached under normal operation. Only a client that closes its browser tab without sending TCP FIN (e.g., machine suspended mid-session) would create a truly silent connection, which TCP keep-alive will detect and clean up.

Sticky sessions for stateful MCP tool sessions

Most MCP tools are stateless: each tool call is independent. Some tools maintain session state — a "browser automation" MCP might open a browser session on one container and expect subsequent calls to reach the same container. NLB supports sticky sessions via source IP hash: all connections from the same client IP are routed to the same target.

// CDK: enable source IP hash sticky sessions on target group
const cfnTg = targetGroup.node.defaultChild as elbv2.CfnTargetGroup;
cfnTg.addPropertyOverride('TargetGroupAttributes', [
  { Key: 'stickiness.enabled', Value: 'true' },
  {
    // source_ip: hash based on client IP and port (default)
    // source_ip_dest_ip: hash includes destination IP — for multi-NLB setups
    Key: 'stickiness.type',
    Value: 'source_ip',
  },
]);

Caveats for MCP with source IP sticky sessions:

• Enterprise clients behind NAT: All clients from the same corporate network appear as one IP. Source IP stickiness routes them all to the same container, defeating horizontal scaling. For enterprise MCPs, implement session stickiness at the application layer (session token in Authorization header → consistent hash to a container instance) rather than at the NLB level.
• Container replacement disrupts sessions: If a sticky target becomes unhealthy (Fargate Spot interruption, ECS rolling update), NLB reassigns those clients to a healthy target. The stateful session context on the old container is gone. MCP servers using sticky sessions should persist session state to Redis or DynamoDB so a new target can resume the session.
• Cross-zone stickiness: With crossZoneEnabled: false (NLB default), a client always reaches an NLB node in its nearest AZ. If that AZ has fewer healthy targets, load imbalance can occur. Enable cross-zone or ensure even target distribution across AZs.

WebSocket over NLB for MCP bidirectional communication

The MCP specification's SSE transport is unidirectional: the server pushes to the client. For MCP patterns that require bidirectional streaming (tool calls that produce intermediate results while accepting streaming input), WebSocket transport is emerging. NLB handles WebSocket at the TCP level with no special configuration.

// MCP WebSocket server with NLB compatibility
import { WebSocketServer } from 'ws';
import * as http from 'node:http';

const httpServer = http.createServer((req, res) => {
  if (req.url === '/health') { res.writeHead(200); res.end('OK'); return; }
  res.writeHead(404); res.end();
});

const wss = new WebSocketServer({ server: httpServer });

wss.on('connection', (ws, req) => {
  const clientIp = req.headers['x-forwarded-for'] || req.socket.remoteAddress;

  // Send WebSocket ping every 30s to keep NLB flow alive
  const pingInterval = setInterval(() => {
    if (ws.readyState === ws.OPEN) ws.ping();
  }, 30_000);

  ws.on('pong', () => {
    // NLB flow refreshed; client connection confirmed alive
  });

  ws.on('message', (data) => {
    // Handle MCP tool call JSON messages
    const call = JSON.parse(data.toString());
    handleMcpCall(call).then(result => {
      ws.send(JSON.stringify(result));
    });
  });

  ws.on('close', () => clearInterval(pingInterval));
  ws.on('error', () => clearInterval(pingInterval));
});

WebSocket ping/pong frames carry bytes through the NLB flow, preventing the 350-second idle timeout. Unlike SSE (HTTP GET), WebSocket upgrades from HTTP to a binary protocol — the NLB sees this as a TCP connection and doesn't interfere with the upgrade handshake. NLB supports WebSocket over TCP and TLS listeners without any special target group configuration.

Monitoring active flow count as an MCP health signal

NLB's ActiveFlowCount metric counts open TCP connections at any instant. For an MCP server, this is a proxy for active subscriber count (SSE connections) plus in-flight tool calls (WebSocket connections). Monitoring this metric reveals two failure modes:

import * as cloudwatch from 'aws-cdk-lib/aws-cloudwatch';

// Alert on connection leak: steady growth in active flows
const connectionLeakAlarm = new cloudwatch.Alarm(scope, 'ConnectionLeak', {
  alarmName: 'mcp-nlb-connection-leak',
  metric: new cloudwatch.Metric({
    namespace: 'AWS/NetworkELB',
    metricName: 'ActiveFlowCount',
    dimensionsMap: { LoadBalancer: nlb.loadBalancerFullName },
    period: Duration.minutes(5),
    statistic: 'Average',
  }),
  threshold: 5000,   // tune to 2× your expected max concurrent connections
  comparisonOperator: cloudwatch.ComparisonOperator.GREATER_THAN_THRESHOLD,
  evaluationPeriods: 6,   // sustained for 30 minutes before alarming
  treatMissingData: cloudwatch.TreatMissingData.NOT_BREACHING,
});

// Alert on TCP reset storm (NLB closing idle connections faster than expected)
const rstStormAlarm = new cloudwatch.Alarm(scope, 'TcpRstStorm', {
  alarmName: 'mcp-nlb-tcp-reset-storm',
  metric: new cloudwatch.Metric({
    namespace: 'AWS/NetworkELB',
    metricName: 'TCP_ELB_Reset_Count',
    dimensionsMap: { LoadBalancer: nlb.loadBalancerFullName },
    period: Duration.minutes(1),
    statistic: 'Sum',
  }),
  threshold: 100,   // more than 100 RSTs/minute suggests idle timeout fires
  comparisonOperator: cloudwatch.ComparisonOperator.GREATER_THAN_THRESHOLD,
  evaluationPeriods: 3,
  treatMissingData: cloudwatch.TreatMissingData.NOT_BREACHING,
});

A TCP_ELB_Reset_Count spike correlates with missing keep-alives: NLB kills idle SSE connections, clients see connection closed errors, and they reconnect — generating a NewFlowCount spike immediately after. The pattern (TCP RST spike → NewFlowCount spike with 5-second lag) is the fingerprint of "SSE keep-alive not implemented." AliveMCP's external probe catches this as intermittent disconnects on the monitored endpoint's uptime chart: brief gaps in the 60-second probe cadence where the probe had to reconnect before verifying the server was alive.

Graceful shutdown: draining SSE connections on ECS task stop

When ECS stops a Fargate task (during deployment or Spot interruption), it sends SIGTERM to the container. The MCP server should drain in-flight SSE streams gracefully rather than letting NLB RST them during deregistration:

// MCP SSE server graceful shutdown on SIGTERM
const activeSseClients = new Set();

process.on('SIGTERM', async () => {
  console.log('SIGTERM received — beginning graceful shutdown');

  // Stop NLB health check first: return 503 so NLB stops sending new connections
  isHealthy = false;

  // Send close event to all active SSE clients
  for (const res of activeSseClients) {
    if (!res.writableEnded) {
      // Final SSE event: client should reconnect to a healthy server
      res.write('event: server_shutdown\ndata: {"reconnect":true,"retryAfterMs":5000}\n\n');
      res.end();
    }
  }
  activeSseClients.clear();

  // Close the HTTP server — no new connections accepted
  httpServer.close(() => {
    console.log('HTTP server closed — exiting');
    process.exit(0);
  });

  // Force exit if drain takes too long (ECS will SIGKILL at stopTimeout anyway)
  setTimeout(() => process.exit(0), 100_000);  // 100s < 115s ECS stopTimeout
});

The server_shutdown SSE event is an application-level signal — MCP clients that handle it can display a "reconnecting" state rather than showing an error. Clients that don't handle it will see a connection close and retry automatically. The 5000ms retryAfterMs hint suggests clients wait 5 seconds before reconnecting — enough time for the new ECS task to become healthy and begin accepting connections through NLB.