OpenSearch Anomaly Detection for MCP Endpoint Health

Detector types: single-metric vs multi-metric vs HC

OpenSearch Anomaly Detection offers three detector configurations:

For AliveMCP monitoring tens of thousands of MCP endpoints, the HC detector with category_field: "server_slug" is the right choice. Each server gets its own RCF model instance — its normal p95 latency baseline is learned independently, so a server that always runs at 800ms doesn't trigger anomalies while a server that normally runs at 50ms but suddenly hits 800ms does.

Creating a High Cardinality detector via API

// Create HC anomaly detector for per-MCP-server latency anomalies
const detectorResponse = await client.transport.request({
  method: 'POST',
  path: '/_plugins/_anomaly_detection/detectors',
  body: {
    name: 'mcp-endpoint-latency-hc',
    description: 'Per-server-slug p95 latency anomaly detection for MCP endpoints',
    time_field: 'checked_at',
    indices: ['probe-results-*'],          // All probe-results rollover indices
    feature_attributes: [
      {
        feature_name: 'p95_latency_ms',
        feature_enabled: true,
        aggregation_query: {
          p95_latency_ms: {
            percentiles: {
              field: 'latency_ms',
              percents: [95],
            },
          },
        },
      },
    ],
    filter_query: {
      bool: {
        must: [
          { term: { probe_type: 'http_check' } },
          // Exclude known-maintenance servers from anomaly scoring
          { term: { maintenance_mode: false } },
        ],
      },
    },
    detector_interval: {
      period: { interval: 1, unit: 'Minutes' },   // 1-minute detection cadence
    },
    window_delay: {
      period: { interval: 30, unit: 'Seconds' },  // Wait 30s for late-arriving probe data
    },
    shingle_size: 8,          // RCF context window: 8 consecutive intervals = 8 minutes of history
    category_field: ['server_slug'],  // HC: one sub-detector per unique server_slug value
  },
});

const detectorId = detectorResponse.body._id;
console.log('Created detector:', detectorId);

The window_delay parameter compensates for probe data arriving late (network jitter, retry delays). Setting it to 30 seconds means the detector evaluates the window [now-1m-30s, now-30s] rather than [now-1m, now], ensuring all probes for the interval have been indexed before the RCF model processes them. Without this, a probe that took 25 seconds to arrive would be missed by the detector, causing sparse training data and noisy anomaly scores.

Random Cut Forest algorithm: shingle size and warm-up

The RCF algorithm builds a forest of random binary trees over a sliding window of shingle_size data points. Each data point in the shingle includes all feature values at that timestamp, creating a multidimensional representation of recent history. A point is anomalous if its inclusion in the tree increases the complexity of the forest more than expected — the anomaly score is the average complexity increase across all trees.

Warm-up: after detector creation or after a data gap exceeding shingle_size × interval (8 minutes with defaults), the RCF forest contains too few trees to produce reliable scores. During warm-up, the detector sets "anomaly_grade": 0 for all results regardless of actual latency — do not trigger alerts on results with anomaly_grade: 0 combined with confidence: 0.

Preview mode: validate feature extraction before starting detection

Preview mode runs the detector against historical data without writing results to the anomaly results index. Use it to validate that the feature extraction (the percentiles aggregation) returns non-null values and that the training data density is sufficient.

// Preview the detector against historical data
const preview = await client.transport.request({
  method: 'POST',
  path: `/_plugins/_anomaly_detection/detectors/${detectorId}/_preview`,
  body: {
    period_start: Date.now() - 7 * 24 * 60 * 60 * 1000,  // 7 days ago
    period_end: Date.now(),
  },
});

// Check that feature values are populated
const sampleResults = preview.body.anomaly_result.slice(0, 5);
for (const result of sampleResults) {
  if (result.feature_data[0].data === null) {
    console.warn('Feature returned null — check field name and index pattern');
  }
}

// Count anomalies in the preview window
const anomalyCount = preview.body.anomaly_result.filter(
  r => r.anomaly_grade > 0
).length;
const totalIntervals = preview.body.anomaly_result.length;
const anomalyRate = anomalyCount / totalIntervals;

// A healthy detector should show 0.5–5% anomaly rate on normal data
// >10% = feature is too noisy (use a longer interval or different aggregation)
// 0% = feature has no variance (check data quality)
console.log(`Anomaly rate: ${(anomalyRate * 100).toFixed(1)}% over ${totalIntervals} intervals`);

Starting the detector and reading results

// Start real-time detection
await client.transport.request({
  method: 'POST',
  path: `/_plugins/_anomaly_detection/detectors/${detectorId}/_start`,
});

// Query anomaly results for a specific server
const anomalyResults = await client.search({
  index: '.opendistro-anomaly-results-history-*',  // Results are in this system index
  body: {
    query: {
      bool: {
        must: [
          { term: { detector_id: detectorId } },
          { term: { 'entity.0.value': 'my-mcp-server-slug' } },  // HC entity filter
          {
            range: {
              data_start_time: {
                gte: 'now-1h',
              },
            },
          },
          {
            range: {
              anomaly_grade: { gt: 0.5 },  // Only results above 0.5 threshold
            },
          },
        ],
      },
    },
    sort: [{ data_start_time: 'desc' }],
    _source: ['data_start_time', 'anomaly_grade', 'confidence', 'feature_data', 'entity'],
  },
});

for (const hit of anomalyResults.body.hits.hits) {
  const { data_start_time, anomaly_grade, confidence, feature_data, entity } = hit._source;
  console.log({
    server: entity[0].value,
    time: new Date(data_start_time).toISOString(),
    grade: anomaly_grade.toFixed(3),       // 0.0–1.0; higher = more anomalous
    confidence: confidence.toFixed(3),     // 0.0–1.0; higher = model is more certain
    p95_latency: feature_data[0].data,
  });
}

Alerting integration: anomaly detector → monitor → destination

OpenSearch Alerting can consume anomaly detector results and fire notifications via webhook, SNS, Slack, or custom destinations. The monitor type "anomaly_detector" polls the anomaly results index automatically — no manual query schedule needed.

// Create an alerting monitor that fires when anomaly_grade > 0.7
const monitor = await client.transport.request({
  method: 'POST',
  path: '/_plugins/_alerting/monitors',
  body: {
    type: 'monitor',
    name: 'mcp-endpoint-latency-anomaly-alert',
    monitor_type: 'anomaly_detector_monitor',
    enabled: true,
    schedule: {
      period: { interval: 1, unit: 'MINUTES' },
    },
    inputs: [
      {
        anomaly_detector: {
          detector_id: detectorId,
        },
      },
    ],
    triggers: [
      {
        anomaly_detector_trigger: {
          name: 'high-anomaly-grade',
          severity: '2',
          min_anomaly_grade: 0.7,   // Fire when grade > 0.7
          min_confidence: 0.5,      // Only fire when model is confident
        },
      },
    ],
    actions: [
      {
        name: 'notify-alivemcp-webhook',
        destination_id: 'WEBHOOK_DESTINATION_ID',  // Pre-created destination
        message_template: {
          source: `{
            "text": "MCP Endpoint Anomaly: {{ctx.results[0].hits.hits[0]._source.entity[0].value}} — grade={{ctx.results[0].hits.hits[0]._source.anomaly_grade}} confidence={{ctx.results[0].hits.hits[0]._source.confidence}}"
          }`,
        },
        action_execution_policy: {
          action_execution_scope: 'PER_ALERT',  // One notification per unique anomalous entity
        },
      },
    ],
  },
});

HC detector entity limit: scaling beyond 1,000 servers

HC detectors process at most 1,000 unique entity values (server slugs) per detector. When more than 1,000 unique values appear in the training data, the detector uses a priority queue to select the 1,000 most frequently appearing entities — low-frequency servers (newly added, infrequently probed) may be silently excluded.

To monitor more than 1,000 MCP servers with per-entity detection, partition them across multiple HC detectors. A practical split for AliveMCP: one HC detector per registry source (MCP.so, Glama, Smithery, PulseMCP), each covering ~500 servers, with a global single-metric detector on top for fleet-wide anomalies like probe worker failures.

// Check which entities are being actively monitored by an HC detector
const entityStats = await client.transport.request({
  method: 'GET',
  path: `/_plugins/_anomaly_detection/detectors/${detectorId}/profile/models`,
});

// Response includes per-entity model state:
// "entities": [ { "modelId": "...", "modelSize": 12345, "nodeId": "..." } ]
// Each entity's model shows its last update time and RCF forest state
const activeEntities = entityStats.body.entities?.length ?? 0;
console.log(`Active entity models: ${activeEntities} / 1000 max`);

How AliveMCP uses anomaly detection as an early-warning system

AliveMCP's binary probe result (HTTP 200 or not) tells you a server has failed — but the failure was already visible 3–15 minutes earlier as a p95 latency increase before the first timeout. The HC anomaly detector on the probe-results-* indices runs at 1-minute intervals and detects per-server p95 latency anomalies before any timeout occurs.

When the detector flags a server with anomaly_grade > 0.7 for two consecutive intervals, AliveMCP promotes it from "live" to "degraded" status on the status page and sends an early-warning webhook to the server's Author-tier subscriber. The Author receives a notification 5–15 minutes before the server would have tripped the binary down threshold — enough time to check their hosting dashboard and restart a stuck process before users notice.

The confidence field prevents spurious alerts: during the first 8 minutes after a server is first seen (warm-up period), confidence = 0 and alerts are suppressed. Only after the RCF forest has been trained on at least 8 consecutive interval samples does the monitor consider the anomaly grade actionable.

Related guides

• OpenSearch Overview for MCP Servers — bulk indexing, scroll, SigV4 auth
• OpenSearch k-NN Vector Search for Semantic MCP Endpoint Discovery
• OpenSearch Index State Management for Probe Time-Series Data
• Amazon OpenSearch Serverless — collection types, access policies, OCU model