Handling Fargate Spot Interruptions for MCP Server Workers: SIGTERM and Checkpoint Patterns

The 2-minute interruption timeline

When AWS decides to reclaim a Fargate Spot task's capacity, the following sequence occurs:

The ECS container stopTimeout defaults to 30 seconds — far too short for a probe worker processing a batch of 20 endpoints at 8 seconds each. Set stopTimeout to 115 in the container definition to give the handler 115 seconds of the 120-second window.

SIGTERM handling: the signal forwarding problem

In Docker containers, SIGTERM is sent to PID 1. If your Node.js process is not PID 1 — for example, if you start it with a shell script like CMD ["sh", "-c", "node dist/index.js"] — the shell receives SIGTERM but does not forward it to Node.js by default. Node exits without running its SIGTERM handler.

Two patterns avoid this:

# Option 1: Start Node as PID 1 by using exec form (no shell wrapper)
CMD ["node", "dist/index.js"]

# Option 2: Use exec to replace the shell process with Node when using a wrapper script
#!/bin/sh
exec node dist/index.js  # 'exec' replaces the shell process — Node becomes PID 1

# Wrong: this starts Node as a child of sh; SIGTERM goes to sh, not Node
#!/bin/sh
node dist/index.js  # do NOT do this

Verify your container's PID 1: when the container is running, docker exec <id> ps aux | head -3 should show node as PID 1, not sh.

Full SIGTERM handler for MCP probe workers

// probe-worker/src/shutdown.ts

import { SQSClient, ChangeMessageVisibilityCommand } from '@aws-sdk/client-sqs';
import type { Message } from '@aws-sdk/client-sqs';

interface ActiveBatch {
  sqsMessage: Message;        // the SQS message we dequeued for this batch
  endpointSlugs: string[];    // endpoints being probed in this batch
  startedAt: number;          // Date.now() when batch started
}

let isShuttingDown = false;
const activeBatches = new Map<string, ActiveBatch>();  // keyed by SQS message ID

export function registerShutdownHandler(sqsClient: SQSClient, queueUrl: string): void {
  process.on('SIGTERM', async () => {
    console.log('[shutdown] SIGTERM received at', new Date().toISOString());
    isShuttingDown = true;

    const DRAIN_DEADLINE_MS = 90_000;  // 90s: leave 25s buffer before ECS stopTimeout
    const started = Date.now();

    // Phase 1: wait for in-flight probes to complete
    while (activeBatches.size > 0 && Date.now() - started < DRAIN_DEADLINE_MS) {
      const slugs = [...activeBatches.values()].flatMap(b => b.endpointSlugs);
      console.log(`[shutdown] Draining ${activeBatches.size} batches (${slugs.length} probes)...`);
      await new Promise(res => setTimeout(res, 1000));
    }

    // Phase 2: re-queue any batches that didn't complete in time
    if (activeBatches.size > 0) {
      console.log(`[shutdown] ${activeBatches.size} incomplete batches — re-queuing to SQS`);
      const requeuePromises = [...activeBatches.values()].map(async (batch) => {
        if (!batch.sqsMessage.ReceiptHandle) return;
        try {
          // Setting visibility to 0 makes the message immediately visible again
          // so another worker can pick it up without waiting for the visibility timeout.
          await sqsClient.send(new ChangeMessageVisibilityCommand({
            QueueUrl: queueUrl,
            ReceiptHandle: batch.sqsMessage.ReceiptHandle,
            VisibilityTimeout: 0,
          }));
          console.log(`[shutdown] Re-queued batch ${batch.sqsMessage.MessageId} (${batch.endpointSlugs.length} endpoints)`);
        } catch (err) {
          // ReceiptHandle may have expired if visibility timeout < elapsed time.
          // The SQS visibility timeout will expire naturally and re-queue the message.
          console.warn(`[shutdown] Could not re-queue ${batch.sqsMessage.MessageId}:`, err);
        }
      });
      await Promise.allSettled(requeuePromises);
    }

    const elapsed = Date.now() - started;
    console.log(`[shutdown] Clean exit after ${elapsed}ms`);
    process.exit(0);
  });
}

// Called by probe loop to register an active batch
export function trackBatch(batch: ActiveBatch): void {
  activeBatches.set(batch.sqsMessage.MessageId!, batch);
}

// Called by probe loop when a batch completes
export function completeBatch(messageId: string): void {
  activeBatches.delete(messageId);
}

export function isShutdownRequested(): boolean {
  return isShuttingDown;
}

Integrating the SIGTERM handler into the probe loop

// probe-worker/src/index.ts

import { SQSClient, ReceiveMessageCommand, DeleteMessageCommand } from '@aws-sdk/client-sqs';
import {
  registerShutdownHandler,
  trackBatch,
  completeBatch,
  isShutdownRequested,
} from './shutdown.js';
import { runProbe } from './probe.js';
import { writeProbeResult } from './db.js';

const sqs = new SQSClient({ region: process.env.AWS_REGION ?? 'us-east-1' });
const QUEUE_URL = process.env.SQS_PROBE_QUEUE_URL!;

// Register SIGTERM handler BEFORE entering the probe loop.
registerShutdownHandler(sqs, QUEUE_URL);

async function probeLoop(): Promise<void> {
  console.log('[worker] Probe loop started');

  while (!isShutdownRequested()) {
    const result = await sqs.send(new ReceiveMessageCommand({
      QueueUrl: QUEUE_URL,
      MaxNumberOfMessages: 1,
      WaitTimeSeconds: 5,          // long-poll: up to 5s if queue is empty
      VisibilityTimeout: 90,       // 90s: matches SIGTERM drain deadline
      AttributeNames: ['MessageId'],
    }));

    if (!result.Messages?.length) continue;
    const message = result.Messages[0];
    const batch = JSON.parse(message.Body!) as { endpointSlugs: string[] };

    // Track the batch before starting probes so SIGTERM handler can re-queue if needed
    trackBatch({
      sqsMessage: message,
      endpointSlugs: batch.endpointSlugs,
      startedAt: Date.now(),
    });

    try {
      const results = await Promise.all(
        batch.endpointSlugs.map(slug => runProbe(slug))
      );
      await Promise.all(results.map(r => writeProbeResult(r)));

      // Delete the SQS message only after all results are written.
      // If the task is interrupted between write and delete, the message
      // becomes visible again (after visibility timeout) — a duplicate probe,
      // not a missed one. Prefer duplicates over gaps.
      await sqs.send(new DeleteMessageCommand({
        QueueUrl: QUEUE_URL,
        ReceiptHandle: message.ReceiptHandle!,
      }));

      completeBatch(message.MessageId!);
    } catch (err) {
      // Don't delete the message on error — SQS re-queues it after VisibilityTimeout.
      completeBatch(message.MessageId!);
      console.error('[worker] Probe batch failed:', err);
    }
  }

  console.log('[worker] Probe loop exited (shutdown requested)');
}

probeLoop().catch(err => {
  console.error('[worker] Fatal error:', err);
  process.exit(1);
});

SQS visibility timeout: the safety net

The SQS message visibility timeout is the time a message stays hidden from other consumers after it's received. If the consumer crashes (SIGKILL, OOM, power loss) without deleting the message, SQS automatically makes the message visible again after the timeout expires — so another worker picks it up.

For MCP probe workers, set the visibility timeout to match your SIGTERM drain deadline (90 seconds):

• Too short (e.g. 30s): A second worker picks up the message while the first is still processing it → duplicate probe results for the same batch in the same window.
• Too long (e.g. 300s): If a worker crashes, the batch is missing from probe results for 5 minutes. Status page shows stale data.
• 90s sweet spot: Matches the SIGTERM drain time. If SIGTERM handler drains cleanly, it re-queues immediately (visibility=0). If the task is force-killed by SIGKILL without running the handler, SQS re-queues after 90 seconds — at most one missed probe cycle.

import * as sqs from 'aws-cdk-lib/aws-sqs';
import { Duration } from 'aws-cdk-lib';

const probeQueue = new sqs.Queue(scope, 'ProbeQueue', {
  queueName: 'mcp-probe-batches',
  visibilityTimeout: Duration.seconds(90),
  // Dead-letter queue after 5 failed attempts (persistent probe errors)
  deadLetterQueue: {
    queue: new sqs.Queue(scope, 'ProbeDLQ', {
      queueName: 'mcp-probe-batches-dlq',
      retentionPeriod: Duration.days(7),
    }),
    maxReceiveCount: 5,
  },
  retentionPeriod: Duration.hours(4),  // probe batches older than 4h are stale; discard
});

ECS task metadata: proactive interruption detection

The ECS task metadata endpoint v4 exposes a /task/status path that you can poll to detect an impending Spot interruption before SIGTERM arrives. This gives you a few extra seconds to pause accepting new work before the signal:

// Optional: poll ECS metadata to detect Spot interruption proactively
const ECS_METADATA_URL = process.env.ECS_CONTAINER_METADATA_URI_V4;

async function pollForSpotInterruption(): Promise<void> {
  if (!ECS_METADATA_URL) return;  // not running in ECS (local dev)

  while (!isShutdownRequested()) {
    try {
      const res = await fetch(`${ECS_METADATA_URL}/task`);
      const metadata = await res.json();

      // When Spot capacity is being reclaimed, the task status transitions to
      // DEACTIVATING before SIGTERM. Detecting this gives ~5-10s extra warning.
      if (metadata.DesiredStatus === 'STOPPED' || metadata.KnownStatus === 'DEACTIVATING') {
        console.log('[metadata] Spot interruption detected via task metadata — preparing shutdown');
        process.emit('SIGTERM');  // trigger our SIGTERM handler early
        return;
      }
    } catch {
      // Metadata endpoint unavailable — continue, rely on SIGTERM
    }
    await new Promise(res => setTimeout(res, 5000));  // poll every 5 seconds
  }
}

// Start metadata polling in background alongside probe loop
pollForSpotInterruption();

The metadata polling is not a substitute for the SIGTERM handler — SIGTERM is the authoritative interruption signal. The metadata poll is a belt-and-suspenders approach that can give 5–10 extra seconds of warning in some cases, but it is not guaranteed: the status transition window is implementation-dependent and may shrink in future AWS platform versions.

Testing graceful shutdown locally

Test your SIGTERM handler locally before deploying to Fargate Spot:

# Terminal 1: run your probe worker locally (not in Docker yet)
node dist/index.js

# Terminal 2: send SIGTERM to simulate Fargate interruption
kill -SIGTERM $(pgrep -f "node dist/index.js")

# Expected output in Terminal 1:
# [shutdown] SIGTERM received at 2026-10-16T04:00:00.000Z
# [shutdown] Draining 1 batches (20 probes)...
# [shutdown] Draining 1 batches (20 probes)...
# [shutdown] Clean exit after 12340ms

# Test SIGKILL handling (no graceful shutdown — should be safe due to SQS visibility timeout)
kill -SIGKILL $(pgrep -f "node dist/index.js")
# Probe loop dies immediately; SQS re-queues incomplete batch after 90s visibility timeout

For Docker-level testing:

# Run container and simulate ECS SIGTERM
docker run -d --name probe-worker probe-worker:latest
docker stop --time 115 probe-worker  # sends SIGTERM, then SIGKILL after 115s

# Watch the logs for the SIGTERM handler output
docker logs -f probe-worker

The --time 115 flag mirrors the ECS stopTimeout: 115 setting. If docker stop with a 115-second timeout successfully stops the container (exit code 0), your handler is working correctly within the deadline.

Failure modes