AWS Fargate Spot for MCP Server Monitoring: Capacity Providers, SIGTERM Handlers, and the Mixed On-Demand Strategy

Pattern 1 — Cost Model and Capacity Provider Infrastructure

Why Fargate Spot matters for MCP monitoring at scale

Most MCP server monitoring systems start as a single loop: probe every endpoint every 60 seconds, write results, repeat. At 50 endpoints this fits in a single Lambda. At 500 endpoints you need a worker fleet. At 5,000 endpoints the fleet is running around the clock and ECS compute is the dominant line item in your infrastructure bill.

The raw numbers: a 0.5 vCPU / 1 GB Fargate task running 24/7 in us-east-1 costs approximately $18/month On-Demand. Ten probe workers cost $180/month. Twenty workers — the count AliveMCP needs at 5,000 endpoints — cost $360/month. Fargate Spot at a representative 66% discount brings that to $122/month for 20 workers. The coordinator stays On-Demand at ~$9/month. The registry crawler runs Spot at ~$6/month. Total at 5,000 endpoints: ~$33/month vs ~$99/month all-On-Demand.

The discount compounds further with ARM64 Graviton: Graviton Fargate tasks have a lower base price than x86 (approximately 20% cheaper) before applying the Spot discount. The two savings are orthogonal — Graviton On-Demand is already cheaper than x86 On-Demand, and Graviton Spot is cheaper still. For a 0.25 vCPU / 0.5 GB probe worker:

These numbers are estimates using us-east-1 pricing; actual Spot discounts vary by region and availability pool. The important pattern: ARM64 + Spot together deliver 70–75% savings vs the x86 On-Demand baseline that most engineers start with when they first deploy to Fargate.

Why MCP probe workers are a strong Spot fit

Fargate Spot is not the right choice for every workload. A database that holds in-memory state, a WebSocket server maintaining client connections, or any process that cannot safely be interrupted in the middle of its work is a poor Spot fit. MCP probe workers have properties that make them almost ideal:

The one property that requires care: workers maintain an open batch of in-progress probes at any given moment. If SIGTERM arrives mid-batch and no handler is registered, those probes are lost and their SQS messages stay invisible until the visibility timeout expires. A SIGTERM handler that re-queues the in-flight batch before exiting eliminates this gap entirely.

The ECS capacity provider base + weight formula

Capacity providers are the ECS control plane for mixing On-Demand and Spot in a single service. The strategy has two parameters per provider: base (a fixed task count always provisioned from this provider, regardless of desired count) and weight (relative proportion controlling how tasks beyond the base are distributed).

Only one provider in a strategy can have a non-zero base. The formula:

base_tasks = base (for the provider with base set)
remaining = desiredCount - base_tasks
for each provider:
  provider_share = remaining × weight[provider] / sum(all weights)

Example: desiredCount=10, FARGATE base=1 weight=1, FARGATE_SPOT base=0 weight=3
  base_tasks = 1  (FARGATE — always On-Demand)
  remaining = 10 - 1 = 9
  FARGATE remaining share = 9 × 1/(1+3) = 2.25 → 2 tasks
  FARGATE_SPOT remaining share = 9 × 3/(1+3) = 6.75 → 7 tasks
  Total: 3 On-Demand (1 base + 2 weight) + 7 Spot = 10 tasks

The base count is guaranteed: at minimum 1 On-Demand task always runs even if Spot capacity is completely unavailable in all configured AZs. The weight-based Spot tasks may not provision during a Spot drought, but the base On-Demand task continues running. This is why setting base: 0 with FARGATE_SPOT weight-only is risky for probe workers: during a Spot drought, the service may drop to 0 running tasks.

One pitfall at low desiredCount values: at desiredCount=4 with base=1, only 3 remaining tasks split by weight → roughly 1 more On-Demand + 2 Spot = 50% Spot at best, not 75%. The weight ratio only approaches the theoretical 75% as desiredCount grows large enough that the base is a small fraction of the total. For small worker fleets (2–4 tasks), use base: 0 with FARGATE weight to guarantee you always get some On-Demand fallback at the cost of losing the base guarantee.

CDK cluster and service setup

Before a service can use FARGATE_SPOT, the ECS cluster must have both capacity providers enabled. This is a one-time cluster configuration:

import * as ec2 from 'aws-cdk-lib/aws-ec2';
import * as ecs from 'aws-cdk-lib/aws-ecs';
import { Stack, StackProps } from 'aws-cdk-lib';
import { Construct } from 'constructs';

export class McpMonitoringStack extends Stack {
  public readonly cluster: ecs.Cluster;

  constructor(scope: Construct, id: string, props?: StackProps) {
    super(scope, id, props);

    const vpc = ec2.Vpc.fromLookup(this, 'Vpc', { isDefault: false });

    this.cluster = new ecs.Cluster(this, 'McpProbeCluster', {
      vpc,
      clusterName: 'mcp-probe-cluster',
      containerInsights: true,
    });

    // Without this call, services cannot use FARGATE_SPOT —
    // the API rejects CapacityProviderStrategy references to FARGATE_SPOT.
    this.cluster.enableFargateCapacityProviders();
  }
}

If you already have an ECS cluster created without enableFargateCapacityProviders(), run aws ecs put-cluster-capacity-providers --cluster <name> --capacity-providers FARGATE FARGATE_SPOT --default-capacity-provider-strategy '[]' to update the existing cluster without redeploying it.

The probe worker task definition is identical whether it runs On-Demand or Spot — Fargate Spot is selected at the service level, not the task definition level. Use ARM64 Graviton to compound savings:

const taskDef = new ecs.FargateTaskDefinition(scope, 'ProbeWorkerTaskDef', {
  cpu: 256,            // 0.25 vCPU — probe workers are I/O-bound, not CPU-bound
  memoryLimitMiB: 512,
  executionRole,
  taskRole,
  runtimePlatform: {
    cpuArchitecture: ecs.CpuArchitecture.ARM64,  // Graviton: ~20% off base Fargate price
    operatingSystemFamily: ecs.OperatingSystemFamily.LINUX,
  },
});

taskDef.addContainer('ProbeWorker', {
  image: ecs.ContainerImage.fromEcrRepository(repo, 'latest'),
  environment: {
    PROBE_CONCURRENCY: '20',
    PROBE_TIMEOUT_MS: '8000',
    SQS_PROBE_QUEUE_URL: process.env.SQS_PROBE_QUEUE_URL ?? '',
  },
  logging: ecs.LogDrivers.awsLogs({ streamPrefix: 'probe-worker' }),
  stopTimeout: Duration.seconds(115),  // must set this — default 30s is too short for SIGTERM drain
});

The stopTimeout: 115 on the container definition is critical — it gives the SIGTERM handler 115 seconds of the 120-second Spot interruption window to drain in-flight probes. The default of 30 seconds is far too short for a worker that may be mid-batch processing 20 endpoints at up to 8 seconds each.

The ECS service for probe workers:

const workerService = new ecs.FargateService(scope, 'ProbeWorkerService', {
  cluster,
  taskDefinition: taskDef,
  desiredCount: 8,
  capacityProviderStrategies: [
    { capacityProvider: 'FARGATE', base: 1, weight: 1 },      // 1 On-Demand always
    { capacityProvider: 'FARGATE_SPOT', base: 0, weight: 3 }, // 75% of remainder on Spot
  ],
  vpcSubnets: {
    availabilityZones: ['us-east-1a', 'us-east-1b', 'us-east-1c'],
    onePerAz: true,
  },
  assignPublicIp: false,
  minHealthyPercent: 50,
  maxHealthyPercent: 200,
  enableExecuteCommand: true,
});

Spreading tasks across 3+ AZs reduces simultaneous interruption risk — Spot capacity is managed per AZ, so a capacity shortage in us-east-1a does not affect tasks running in us-east-1b or 1c. ECS distributes tasks across subnets in multiple AZs automatically when you provide subnets spanning multiple AZs.

Pattern 2 — SIGTERM Handler and SQS Checkpoint

The 2-minute interruption timeline

Understanding the exact sequence of events during a Fargate Spot interruption is essential for sizing the SIGTERM handler correctly:

The default ECS container stopTimeout is 30 seconds — AWS's documentation notes this clearly, but many engineers miss it because most containers don't need more than 30 seconds to shut down. For MCP probe workers, 30 seconds is insufficient: a batch of 20 concurrent probes at up to 8 seconds each completes in 8 seconds (they run in parallel), but if the probe loop starts a new batch in the last 30 seconds before SIGTERM, the handler has no time to drain. Set stopTimeout to 115 in the container definition.

The PID 1 signal forwarding problem

SIGTERM is delivered to PID 1 in the container. If your Node.js process is not PID 1 — for example, because you start it with a shell script — the shell receives SIGTERM and exits without forwarding it to Node. Node exits too (when the parent exits), but without running its SIGTERM handler. Your checkpoint code never runs.

The fix is simple but easy to miss:

# CORRECT: exec form — Node is PID 1
CMD ["node", "dist/index.js"]

# CORRECT: shell script with exec — exec replaces the shell process, Node becomes PID 1
#!/bin/sh
exec node dist/index.js

# WRONG: shell script without exec — Node is a child of sh; SIGTERM goes to sh, not Node
#!/bin/sh
node dist/index.js  # do NOT do this

Verify which process is PID 1 while the container is running: docker exec <container-id> ps aux | head -3. The first line after the header should show node as PID 1, not sh or bash. If you see a shell as PID 1, fix the Dockerfile before deploying to Fargate Spot.

Full SIGTERM handler with SQS batch checkpoint

The complete SIGTERM handler tracks in-flight batches in a Map keyed by SQS message ID, drains them within a 90-second window, and re-queues incomplete batches to SQS by setting their visibility timeout to 0 (making them immediately visible to other workers):

// probe-worker/src/shutdown.ts

import { SQSClient, ChangeMessageVisibilityCommand } from '@aws-sdk/client-sqs';
import type { Message } from '@aws-sdk/client-sqs';

interface ActiveBatch {
  sqsMessage: Message;
  endpointSlugs: string[];
  startedAt: number;
}

let isShuttingDown = false;
const activeBatches = new Map<string, ActiveBatch>();

export function registerShutdownHandler(sqsClient: SQSClient, queueUrl: string): void {
  process.on('SIGTERM', async () => {
    console.log('[shutdown] SIGTERM received at', new Date().toISOString());
    isShuttingDown = true;

    const DRAIN_DEADLINE_MS = 90_000;  // 90s: 25s buffer before ECS stopTimeout
    const started = Date.now();

    // Phase 1: wait for in-flight probes to finish
    while (activeBatches.size > 0 && Date.now() - started < DRAIN_DEADLINE_MS) {
      const count = [...activeBatches.values()].reduce((n, b) => n + b.endpointSlugs.length, 0);
      console.log(`[shutdown] Draining ${activeBatches.size} batches (${count} probes)...`);
      await new Promise(res => setTimeout(res, 1000));
    }

    // Phase 2: re-queue batches that didn't complete in time
    if (activeBatches.size > 0) {
      console.log(`[shutdown] ${activeBatches.size} incomplete batches — re-queuing to SQS`);
      await Promise.allSettled(
        [...activeBatches.values()].map(async (batch) => {
          if (!batch.sqsMessage.ReceiptHandle) return;
          try {
            // VisibilityTimeout: 0 makes the message immediately visible again —
            // another worker picks it up without waiting for the 90s timeout to expire.
            await sqsClient.send(new ChangeMessageVisibilityCommand({
              QueueUrl: queueUrl,
              ReceiptHandle: batch.sqsMessage.ReceiptHandle,
              VisibilityTimeout: 0,
            }));
          } catch (err) {
            // ReceiptHandle may have expired if visibility timeout < elapsed time.
            // The message re-queues naturally when the timeout expires — log and continue.
            console.warn(`[shutdown] Could not re-queue ${batch.sqsMessage.MessageId}:`, err);
          }
        }),
      );
    }

    console.log(`[shutdown] Clean exit after ${Date.now() - started}ms`);
    process.exit(0);
  });
}

export function trackBatch(batch: ActiveBatch): void {
  activeBatches.set(batch.sqsMessage.MessageId!, batch);
}

export function completeBatch(messageId: string): void {
  activeBatches.delete(messageId);
}

export function isShutdownRequested(): boolean {
  return isShuttingDown;
}

Integrating the handler into the probe loop

The probe loop must call trackBatch before starting any probes and completeBatch in a finally block — not in the success path only. The finally guarantee is what prevents the activeBatches Map from growing unbounded on error paths:

// probe-worker/src/index.ts

import { SQSClient, ReceiveMessageCommand, DeleteMessageCommand } from '@aws-sdk/client-sqs';
import { registerShutdownHandler, trackBatch, completeBatch, isShutdownRequested } from './shutdown.js';
import { runProbe } from './probe.js';
import { writeProbeResult } from './db.js';

const sqs = new SQSClient({ region: process.env.AWS_REGION ?? 'us-east-1' });
const QUEUE_URL = process.env.SQS_PROBE_QUEUE_URL!;

// Register BEFORE entering the probe loop — SIGTERM may arrive at any time
registerShutdownHandler(sqs, QUEUE_URL);

async function probeLoop(): Promise<void> {
  while (!isShutdownRequested()) {
    const result = await sqs.send(new ReceiveMessageCommand({
      QueueUrl: QUEUE_URL,
      MaxNumberOfMessages: 1,
      WaitTimeSeconds: 5,
      VisibilityTimeout: 90,   // matches SIGTERM drain deadline
      AttributeNames: ['MessageId'],
    }));

    if (!result.Messages?.length) continue;
    const message = result.Messages[0];
    const batch = JSON.parse(message.Body!) as { endpointSlugs: string[] };

    // Track before probes so SIGTERM handler can re-queue if interrupted mid-batch
    trackBatch({ sqsMessage: message, endpointSlugs: batch.endpointSlugs, startedAt: Date.now() });

    try {
      const results = await Promise.all(batch.endpointSlugs.map(slug => runProbe(slug)));
      await Promise.all(results.map(r => writeProbeResult(r)));

      // Delete the SQS message only after all results are written.
      // If interrupted between write and delete, the message re-queues after the visibility
      // timeout — a duplicate probe, not a missed one. Always prefer duplicates over gaps.
      await sqs.send(new DeleteMessageCommand({
        QueueUrl: QUEUE_URL,
        ReceiptHandle: message.ReceiptHandle!,
      }));
    } finally {
      // Always call completeBatch in finally, never only on success
      completeBatch(message.MessageId!);
    }
  }
}

probeLoop().catch(err => { console.error('[worker] Fatal:', err); process.exit(1); });

The ordering within the try block matters: runProbe → writeProbeResult → DeleteMessage → completeBatch. If writeProbeResult fails, the SQS message is not deleted and not removed from activeBatches — but completeBatch in the finally block cleans up the Map entry. The SQS message becomes visible again after the visibility timeout; another worker retries the batch. If DeleteMessage fails after successful writes, the batch gets processed twice — accept this; dashboard aggregation deduplicates by time window.

SQS visibility timeout: the 90-second sweet spot

The SQS visibility timeout is the safety net that handles crashes the SIGTERM handler cannot: SIGKILL (the hard deadline at T+120s), OOM kills, and container engine failures. When a task is killed without deleting the SQS message, SQS makes the message visible again automatically after the timeout expires. Size this timeout relative to the SIGTERM drain time:

• Too short (30s): A second worker picks up the message while the first worker is still processing it → duplicate probe results for the same batch in the same time window (harmless but noisy).
• Too long (300s): If a worker crashes, affected endpoints are missing from probe results for 5 minutes. The status page shows stale data; users see endpoints "frozen" at their last known state.
• 90s sweet spot: Matches the SIGTERM drain time. Clean shutdown: handler re-queues immediately with VisibilityTimeout: 0. SIGKILL without handler: SQS re-queues after 90 seconds — at most one missed 60-second probe cycle per interrupted task.

const probeQueue = new sqs.Queue(scope, 'ProbeQueue', {
  queueName: 'mcp-probe-batches',
  visibilityTimeout: Duration.seconds(90),
  deadLetterQueue: {
    queue: new sqs.Queue(scope, 'ProbeDLQ', { retentionPeriod: Duration.days(7) }),
    maxReceiveCount: 5,
  },
  retentionPeriod: Duration.hours(4),  // batches older than 4h are stale — discard
});

ECS task metadata: proactive interruption detection

The ECS task metadata endpoint v4 exposes task status information that can detect an impending Spot interruption a few seconds before SIGTERM arrives. When AWS decides to reclaim the Fargate Spot capacity, the task's DesiredStatus transitions to STOPPED before the SIGTERM signal is sent:

const ECS_METADATA_URL = process.env.ECS_CONTAINER_METADATA_URI_V4;

async function pollForSpotInterruption(): Promise<void> {
  if (!ECS_METADATA_URL) return;  // not in ECS (local dev)

  while (!isShutdownRequested()) {
    try {
      const res = await fetch(`${ECS_METADATA_URL}/task`);
      const metadata = await res.json();

      if (metadata.DesiredStatus === 'STOPPED' || metadata.KnownStatus === 'DEACTIVATING') {
        console.log('[metadata] Spot interruption detected proactively');
        process.emit('SIGTERM');  // trigger shutdown handler early
        return;
      }
    } catch { /* metadata unavailable — rely on SIGTERM */ }
    await new Promise(res => setTimeout(res, 5000));
  }
}

pollForSpotInterruption();  // run in background alongside probe loop

The metadata poll is belt-and-suspenders, not a replacement for the SIGTERM handler. The status transition window is implementation-dependent — it may give 5–10 extra seconds in some interruption scenarios, or may not precede SIGTERM at all in others. Always register the SIGTERM handler; optionally add the metadata poll for slightly earlier warning.

Testing graceful shutdown locally

Test the SIGTERM handler before deploying to Fargate Spot — production interruptions are difficult to observe and reproduce:

# Process-level SIGTERM test
node dist/index.js &
kill -SIGTERM $(pgrep -f "node dist/index.js")
# Expected: [shutdown] SIGTERM received → [shutdown] Draining... → [shutdown] Clean exit

# Docker-level test matching ECS stopTimeout behavior
docker run -d --name probe-worker probe-worker:latest
docker stop --time 115 probe-worker
docker logs probe-worker  # should show SIGTERM handler completing within 115 seconds

# SIGKILL test: verify SQS re-queues correctly
docker run -d --name probe-worker probe-worker:latest
docker kill probe-worker  # SIGKILL — no handler
# verify: SQS message becomes visible again after 90 seconds

The docker stop --time 115 flag mirrors the ECS stopTimeout: 115 setting. If Docker stops the container cleanly (exit 0) within 115 seconds, your SIGTERM handler is working within the deadline. If it times out and falls through to SIGKILL, your drain logic is taking too long — check for batches that are not calling completeBatch in a finally block.

Pattern 3 — Coordinator + Worker Split Architecture, Alarms, and Drought Fallback

Why the split matters

A naive Fargate Spot deployment runs every component on Spot — coordinator and workers alike. The problem: the coordinator is the only process that knows which endpoints need probing and when. If it is interrupted, probe batches stop being sent to SQS for 30–90 seconds (the time for ECS to provision a replacement task and for the coordinator to reload its state from the database). During that window, workers have nothing to probe. At a 60-second probe interval, a 30–90 second coordinator outage means one to two full missed cycles across every monitored endpoint.

Running the coordinator On-Demand eliminates this for approximately $9/month — the cost of one 0.5 vCPU / 1 GB On-Demand task. Every other component can run Spot. This is the core architectural trade-off: pay $9/month for coordinator stability; save 60–70% on the worker fleet that does the actual HTTP probing.

Coordinator CDK: single-task, On-Demand, minHealthyPercent=0

The coordinator service has one task and must never have two tasks running simultaneously — two coordinators each sending SQS batches doubles the probe distribution and creates duplicate probe records for the same endpoints in the same time window:

export function createCoordinatorService(
  scope: Construct,
  cluster: ecs.Cluster,
  coordinatorTaskDef: ecs.FargateTaskDefinition,
  probeQueue: sqs.IQueue,
): ecs.FargateService {
  const service = new ecs.FargateService(scope, 'ProbeCoordinatorService', {
    cluster,
    taskDefinition: coordinatorTaskDef,
    desiredCount: 1,
    capacityProviderStrategies: [
      { capacityProvider: 'FARGATE', base: 1, weight: 1 },  // On-Demand only
    ],
    // minHealthyPercent: 0 tells ECS to stop the old task before starting a new one.
    // With minHealthyPercent: 100 (the default for desiredCount=1), ECS would start
    // a second coordinator alongside the first — briefly two coordinators running.
    minHealthyPercent: 0,
    maxHealthyPercent: 100,
    circuitBreaker: { rollback: true },
  });

  probeQueue.grantSendMessages(service.taskDefinition.taskRole);
  return service;
}

The minHealthyPercent: 0 + maxHealthyPercent: 100 combination is the standard pattern for singleton ECS services: during a rolling update, ECS stops the old task first (bringing the service to 0 running tasks momentarily) then starts the new one. The brief 0-task window causes at most one missed probe distribution cycle — an acceptable trade-off for preventing dual-coordinator data corruption.

If even a brief coordinator outage during deployment is unacceptable, implement a distributed lease using DynamoDB conditional writes: the coordinator acquires a lease with a 70-second TTL at startup and renews it every 30 seconds. A second coordinator that starts cannot proceed if a valid lease exists. Both can start, but only one distributes batches.

Worker service CDK with Auto Scaling

export function createWorkerService(
  scope: Construct,
  cluster: ecs.Cluster,
  workerTaskDef: ecs.FargateTaskDefinition,
  probeQueue: sqs.IQueue,
): ecs.FargateService {
  const service = new ecs.FargateService(scope, 'ProbeWorkerService', {
    cluster,
    taskDefinition: workerTaskDef,
    desiredCount: 4,
    capacityProviderStrategies: [
      { capacityProvider: 'FARGATE', base: 1, weight: 1 },
      { capacityProvider: 'FARGATE_SPOT', base: 0, weight: 3 },
    ],
    minHealthyPercent: 50,
    maxHealthyPercent: 200,
    circuitBreaker: { rollback: true },
    vpcSubnets: { availabilityZones: [`${Aws.REGION}a`, `${Aws.REGION}b`, `${Aws.REGION}c`] },
  });

  probeQueue.grantConsumeMessages(service.taskDefinition.taskRole);

  // Scale on SQS visible messages per running task
  const scaling = service.autoScaleTaskCount({ minCapacity: 2, maxCapacity: 50 });
  scaling.scaleOnMetric('SqsDepthPerTask', {
    metric: new cloudwatch.MathExpression({
      expression: 'queueDepth / MAX([taskCount, 1])',
      usingMetrics: {
        queueDepth: probeQueue.metricApproximateNumberOfMessagesVisible({
          period: Duration.minutes(1), statistic: 'Average',
        }),
        taskCount: new cloudwatch.Metric({
          namespace: 'ECS/ContainerInsights',
          metricName: 'RunningTaskCount',
          dimensionsMap: { ServiceName: service.serviceName, ClusterName: cluster.clusterName },
          period: Duration.minutes(1), statistic: 'Average',
        }),
      },
      period: Duration.minutes(1),
    }),
    scalingSteps: [
      { upper: 5, change: -2 },
      { lower: 5, upper: 20, change: 0 },
      { lower: 20, upper: 50, change: +2 },
      { lower: 50, change: +5 },
    ],
    adjustmentType: applicationautoscaling.AdjustmentType.CHANGE_IN_CAPACITY,
    cooldown: Duration.minutes(2),  // matches Fargate task launch latency (30–60s)
  });

  return service;
}

The MAX([taskCount, 1]) in the MathExpression prevents a division-by-zero alarm during Spot drought when RunningTaskCount drops to 0. The metric collapses to queueDepth / 1 = queueDepth — which is correctly very large — triggering scale-out when the service has no running tasks.

The 2-minute cooldown is intentional: Fargate tasks take 30–60 seconds to provision (image pull + container init). Allowing Auto Scaling decisions within 2 minutes of the previous one risks scaling decisions based on stale queue depth metrics before new tasks have had time to start consuming from the queue. A 2-minute cooldown prevents thrashing where Auto Scaling adds 5 tasks, observes the queue hasn't emptied yet (because the tasks haven't started), and adds 5 more.

CloudWatch alarms for Spot droughts and interruptions

Two alarms cover the failure modes that matter most: mass task loss (simultaneous Spot interruption across all AZs, or persistent Spot unavailability) and probe backlog buildup (workers can't keep up with the coordinator's dispatch rate):

// Alarm 1: fewer than 2 workers running for 6 consecutive minutes
const taskCountAlarm = new cloudwatch.Alarm(scope, 'LowWorkerTaskCount', {
  alarmName: 'mcp-probe-low-worker-count',
  metric: new cloudwatch.Metric({
    namespace: 'ECS/ContainerInsights',
    metricName: 'RunningTaskCount',
    dimensionsMap: { ServiceName: service.serviceName, ClusterName: cluster.clusterName },
    period: Duration.minutes(2),
    statistic: 'Minimum',
  }),
  threshold: 2,
  comparisonOperator: cloudwatch.ComparisonOperator.LESS_THAN_THRESHOLD,
  evaluationPeriods: 3,  // 3 × 2-min periods = 6 minutes
  treatMissingData: cloudwatch.TreatMissingData.BREACHING,
});

// Alarm 2: SQS backlog growing despite workers running
const queueBacklogAlarm = new cloudwatch.Alarm(scope, 'ProbeQueueBacklog', {
  alarmName: 'mcp-probe-queue-backlog',
  metric: probeQueue.metricApproximateNumberOfMessagesVisible({
    period: Duration.minutes(2), statistic: 'Average',
  }),
  threshold: 100,
  comparisonOperator: cloudwatch.ComparisonOperator.GREATER_THAN_THRESHOLD,
  evaluationPeriods: 3,
  treatMissingData: cloudwatch.TreatMissingData.NOT_BREACHING,
});

The 6-minute evaluation window for the task count alarm is important: during a rolling deployment, ECS temporarily reduces running task count (due to minHealthyPercent: 50) by up to 2 tasks for 2–3 minutes. A 6-minute alarm threshold rides through a normal deployment without firing. A Spot mass-interruption event, by contrast, causes a sustained task count drop lasting longer than 6 minutes as ECS struggles to reprovision in constrained availability pools.

Spot drought fallback

A Spot drought is when AWS has no available Fargate Spot capacity in your region's AZs and cannot provision new Spot tasks. This is distinct from Spot interruption (AWS reclaiming existing tasks): drought only blocks new task provisioning, not existing running tasks. Existing Spot tasks in a drought continue running until they are interrupted for other reasons.

Drought response options:

1. Automatic (no action needed): ECS uses the FARGATE weight allocation in the capacity strategy to provision On-Demand tasks when Spot is unavailable. Spot workers that exit during a drought are replaced with On-Demand workers using the FARGATE weight. This costs more (no Spot discount) but keeps the probe fleet running automatically.
2. Manual base increase (for sustained drought >1 hour): If Auto Scaling needs to scale out during a drought and Spot is unavailable, new tasks may not provision. Temporarily raise the On-Demand base to force explicit On-Demand provisioning:

   aws ecs update-service \
     --cluster mcp-probe-cluster \
     --service ProbeWorkerService \
     --capacity-provider-strategy \
       "capacityProvider=FARGATE,base=5,weight=1" \
       "capacityProvider=FARGATE_SPOT,base=0,weight=3" \
     --force-new-deployment

   Revert by redeploying via CDK once the Spot drought passes.

Fargate Spot droughts are relatively rare and typically resolve within an hour. The FARGATE weight-based fallback handles the vast majority of cases automatically. The manual override exists for the uncommon scenario where a sustained multi-hour drought coincides with an endpoint count spike requiring rapid scale-out.

Cost scaling model

The complete coordinator + worker split architecture cost at different endpoint scales (worker count assumes 0.25 vCPU / 0.5 GB ARM64 Graviton, 20 concurrent probes per worker; coordinator is 0.5 vCPU / 1 GB On-Demand; 66% Spot discount):

The savings percentage stabilizes at 67% as endpoint count grows and the fixed coordinator On-Demand cost ($9/month) becomes a smaller fraction of the total. At 500 endpoints, the coordinator represents a larger share of total cost, so the blended savings percentage is lower (59%). Beyond 1,000 endpoints, the savings curve flattens near 67% — the long-run discount rate of the worker fleet at 75% Spot composition.

AliveMCP runs this architecture at 1,000 monitored endpoints: 1 probe coordinator ($9/mo On-Demand), 8 probe workers ($12/mo blended), 1 registry crawler ($3/mo Spot) = $24/mo total. The equivalent all-On-Demand fleet costs $82/mo. The $58/month savings funds the Timestream and SQS costs at this scale with room to spare.

Consolidated failure modes