Mixed On-Demand and Fargate Spot Strategy for MCP Server Monitoring

Architecture: coordinator vs worker split

A naive MCP monitoring design runs a single ECS service with a cron-triggered Lambda or a looping process that probes every endpoint every 60 seconds. This works at small scale but becomes a bottleneck at >100 endpoints (sequential probing can't hit a 60-second interval) and is a poor fit for Spot (interrupting the single process loses the entire probe cycle).

The split architecture uses two roles:

This separation lets you scale the worker fleet from 2 to 50 tasks based on SQS queue depth (more endpoints = more SQS messages per 60s = more workers needed) while keeping the coordinator stable and cheap.

CDK: coordinator service (On-Demand)

import * as ecs from 'aws-cdk-lib/aws-ecs';
import * as sqs from 'aws-cdk-lib/aws-sqs';
import { Duration } from 'aws-cdk-lib';
import { Construct } from 'constructs';

export function createCoordinatorService(
  scope: Construct,
  cluster: ecs.Cluster,
  coordinatorTaskDef: ecs.FargateTaskDefinition,
  probeQueue: sqs.IQueue,
): ecs.FargateService {
  const service = new ecs.FargateService(scope, 'ProbeCoordinatorService', {
    cluster,
    taskDefinition: coordinatorTaskDef,
    desiredCount: 1,
    // Explicit On-Demand: no FARGATE_SPOT in the strategy.
    // This is equivalent to launchType: FARGATE but makes the cost decision visible.
    capacityProviderStrategies: [
      { capacityProvider: 'FARGATE', base: 1, weight: 1 },
    ],
    // Coordinator must not have two instances running simultaneously —
    // two coordinators would double-distribute SQS batches → duplicate probe records.
    minHealthyPercent: 0,    // allow ECS to stop the old task before starting new (rolling update)
    maxHealthyPercent: 100,
    // Stop old coordinator before starting new during deployments to avoid dual-coordinator
    deploymentController: { type: ecs.DeploymentControllerType.ECS },
    circuitBreaker: { rollback: true },
  });

  // Coordinator needs SQS SendMessage and DB read access
  probeQueue.grantSendMessages(service.taskDefinition.taskRole);

  return service;
}

The coordinator uses minHealthyPercent: 0 so ECS terminates the old task before starting the new one during a deployment. With minHealthyPercent: 100, ECS would start a second coordinator alongside the first — a brief dual-coordinator window where both are sending SQS batches and probe results are duplicated. For a single-task service, always use minHealthyPercent: 0.

CDK: worker service (Fargate Spot)

import * as ecs from 'aws-cdk-lib/aws-ecs';
import * as sqs from 'aws-cdk-lib/aws-sqs';
import * as applicationautoscaling from 'aws-cdk-lib/aws-applicationautoscaling';
import { Duration, Aws } from 'aws-cdk-lib';

export function createWorkerService(
  scope: Construct,
  cluster: ecs.Cluster,
  workerTaskDef: ecs.FargateTaskDefinition,
  probeQueue: sqs.IQueue,
): ecs.FargateService {
  const service = new ecs.FargateService(scope, 'ProbeWorkerService', {
    cluster,
    taskDefinition: workerTaskDef,
    desiredCount: 4,
    capacityProviderStrategies: [
      // 1 On-Demand task always running: survives total Spot drought
      { capacityProvider: 'FARGATE', base: 1, weight: 1 },
      // Remaining tasks 75% on Spot
      { capacityProvider: 'FARGATE_SPOT', base: 0, weight: 3 },
    ],
    minHealthyPercent: 50,
    maxHealthyPercent: 200,
    circuitBreaker: { rollback: true },
    // Spread across AZs for Spot capacity diversity
    vpcSubnets: { availabilityZones: [`${Aws.REGION}a`, `${Aws.REGION}b`, `${Aws.REGION}c`] },
  });

  // Workers need SQS ReceiveMessage, DeleteMessage, ChangeMessageVisibility
  probeQueue.grantConsumeMessages(service.taskDefinition.taskRole);

  // Auto Scaling: scale workers based on SQS visible messages per task
  const scaling = service.autoScaleTaskCount({ minCapacity: 2, maxCapacity: 50 });
  scaling.scaleOnMetric('SqsDepthPerTask', {
    metric: probeQueue.metricApproximateNumberOfMessagesVisible({
      period: Duration.minutes(1),
      statistic: 'Average',
    }),
    scalingSteps: [
      { upper: 0, change: -2 },
      { lower: 1, upper: 10, change: 0 },
      { lower: 10, upper: 40, change: +3 },
      { lower: 40, change: +8 },
    ],
    adjustmentType: applicationautoscaling.AdjustmentType.CHANGE_IN_CAPACITY,
    cooldown: Duration.minutes(2),
  });

  return service;
}

CloudWatch alarms: detecting Spot droughts

A "Spot drought" occurs when AWS has no available Fargate Spot capacity in your region/AZ and ECS cannot provision new Spot tasks. Existing Spot tasks continue running (they aren't interrupted by capacity shortage — only by AWS reclaiming capacity for On-Demand), but the service can't scale out. If probe load grows and workers can't scale, the SQS queue depth climbs and probes fall behind their 60-second interval.

import * as cloudwatch from 'aws-cdk-lib/aws-cloudwatch';
import * as cloudwatchActions from 'aws-cdk-lib/aws-cloudwatch-actions';
import * as sns from 'aws-cdk-lib/aws-sns';

export function createSpotAlarms(
  scope: Construct,
  workerService: ecs.FargateService,
  probeQueue: sqs.IQueue,
  alertTopic: sns.ITopic,
): void {
  // Alarm 1: running task count drops below minimum (mass Spot interruption or drought)
  const taskCountAlarm = new cloudwatch.Alarm(scope, 'LowWorkerTaskCount', {
    alarmName: 'mcp-probe-low-worker-count',
    alarmDescription: 'Probe worker count below 2 — possible Spot interruption or drought',
    metric: new cloudwatch.Metric({
      namespace: 'ECS/ContainerInsights',
      metricName: 'RunningTaskCount',
      dimensionsMap: {
        ServiceName: workerService.serviceName,
        ClusterName: workerService.cluster.clusterName,
      },
      period: Duration.minutes(2),
      statistic: 'Minimum',
    }),
    threshold: 2,
    comparisonOperator: cloudwatch.ComparisonOperator.LESS_THAN_THRESHOLD,
    evaluationPeriods: 3,        // alarm after 3 consecutive 2-minute periods (6 minutes total)
    treatMissingData: cloudwatch.TreatMissingData.BREACHING,
  });
  taskCountAlarm.addAlarmAction(new cloudwatchActions.SnsAction(alertTopic));

  // Alarm 2: SQS queue depth growing despite running workers (workers not keeping up)
  const queueBacklogAlarm = new cloudwatch.Alarm(scope, 'ProbeQueueBacklog', {
    alarmName: 'mcp-probe-queue-backlog',
    alarmDescription: 'Probe SQS queue depth exceeds 100 — workers not keeping up with probe demand',
    metric: probeQueue.metricApproximateNumberOfMessagesVisible({
      period: Duration.minutes(2),
      statistic: 'Average',
    }),
    threshold: 100,
    comparisonOperator: cloudwatch.ComparisonOperator.GREATER_THAN_THRESHOLD,
    evaluationPeriods: 3,
    treatMissingData: cloudwatch.TreatMissingData.NOT_BREACHING,
  });
  queueBacklogAlarm.addAlarmAction(new cloudwatchActions.SnsAction(alertTopic));
}

Spot drought fallback: raising the On-Demand base

When a Spot drought alarm fires, you have two response options:

1. Automatic fallback (CDK, via Auto Scaling): ECS will fall back to On-Demand automatically using the FARGATE weight in the capacity strategy — it just won't get the Spot discount. No action needed; just accept temporarily higher costs.
2. Manual base increase (AWS CLI): If Auto Scaling isn't scaling out fast enough during a drought, temporarily increase the FARGATE base to force more On-Demand tasks:

   aws ecs update-service \
     --cluster mcp-probe-cluster \
     --service ProbeWorkerService \
     --capacity-provider-strategy \
       "capacityProvider=FARGATE,base=5,weight=1" \
       "capacityProvider=FARGATE_SPOT,base=0,weight=3" \
     --force-new-deployment

   Revert by re-deploying via CDK once the Spot drought passes.

In practice, Spot droughts for Fargate are rare (typically <1 hour) and affect only specific AZ/instance families. Tasks already running on Spot continue until AWS needs capacity — the drought only affects provisioning new Spot tasks. If your worker service has enough running tasks to handle current probe load, a Spot drought is invisible to your monitoring output.

Cost model: mixed vs all-On-Demand at different scales

Worker count assumes 0.25 vCPU / 0.5 GB (ARM64 Graviton) with 20 concurrent probes per worker; coordinator is 0.5 vCPU / 1 GB On-Demand. Cost estimates use Fargate On-Demand pricing ~$0.04048/vCPU-hr + $0.004445/GB-hr in us-east-1 and a 66% Spot discount. Actual costs vary by region and current Spot pricing.

Failure modes