Guide · Alerting Infrastructure

MCP Tools for Prometheus Alertmanager — alerts, silences, and inhibition rules

Prometheus Alertmanager receives firing alerts from Prometheus and Grafana Mimir, deduplicates them, groups them by label sets, and routes them to receivers (PagerDuty, Slack, email, webhook). When building MCP tools that integrate with Alertmanager — listing active alerts, silencing noisy alerts during maintenance, checking route configuration, or auditing inhibition rules — three distinctions are critical: alert state vs alert status (an alert can be active and simultaneously suppressed due to a silence or inhibition rule — suppressed alerts are not notified), silences vs inhibitions (silences suppress individual matching alerts by label matchers; inhibitions suppress a class of alerts whenever a higher-severity "source" alert is firing), and Alertmanager cluster health vs notification health (a healthy cluster can still fail to send notifications if a receiver's webhook returns 5xx or a PagerDuty integration key is revoked).

TL;DR

Query active alerts at GET /api/v2/alerts?active=true&silenced=false&inhibited=false (omit filters to see all states including silenced). Create a silence via POST /api/v2/silences with a matchers array and an endsAt timestamp. Delete a silence (expire it immediately) via DELETE /api/v2/silences/{silenceId}. List all silences at GET /api/v2/silences — include filter=state=active to exclude expired ones. Check Alertmanager's status at GET /api/v2/status — this returns the cluster configuration and peer list but not notification success/failure (Alertmanager does not expose notification delivery metrics via the v2 API; use the /metrics Prometheus endpoint instead).

Alertmanager API v2 architecture for MCP tool authors

Alertmanager exposes the v2 REST API at /api/v2/ on port 9093 (default). The API is OpenAPI-documented — a Swagger UI is available at the Alertmanager root UI. Alertmanager can run as a cluster of multiple instances for high availability; all instances in the cluster share state via a gossip protocol. In a clustered setup, you can send API requests to any instance and the state (silences, active alerts) is eventually consistent across peers within seconds.

Alertmanager itself has no built-in authentication. In production deployments, it is placed behind a reverse proxy (nginx, Traefik, or the Grafana Cloud auth proxy) that handles authentication. When building MCP tools that connect to a production Alertmanager, always use the authenticated reverse-proxy endpoint — connecting directly to port 9093 exposes the silence creation API to all callers.

async function createAlertmanagerClient(config) {
  const headers = { 'Content-Type': 'application/json' };

  if (config.token) {
    // Grafana Cloud managed Alertmanager: Bearer token
    headers['Authorization'] = `Bearer ${config.token}`;
  } else if (config.basicAuth) {
    const encoded = Buffer.from(`${config.basicAuth.user}:${config.basicAuth.pass}`).toString('base64');
    headers['Authorization'] = `Basic ${encoded}`;
  }

  const baseUrl = config.baseUrl.replace(/\/$/, ''); // strip trailing slash

  async function request(method, path, body) {
    const response = await fetch(`${baseUrl}${path}`, {
      method,
      headers,
      body: body ? JSON.stringify(body) : undefined,
    });

    if (!response.ok) {
      const text = await response.text();
      throw new Error(`Alertmanager ${method} ${path} → ${response.status}: ${text}`);
    }

    // DELETE /silences/{id} returns 200 with empty body
    if (response.headers.get('content-length') === '0' || response.status === 200 && method === 'DELETE') {
      return null;
    }

    return response.json();
  }

  return { request };
}

const am = await createAlertmanagerClient({
  baseUrl: 'https://alertmanager.internal',
  basicAuth: { user: 'admin', pass: 'secret' },
});

Listing and filtering active alerts

The GET /api/v2/alerts endpoint returns all alerts currently known to Alertmanager. By default this includes alerts in all states: active (firing), suppressed (silenced or inhibited), and unprocessed (received but not yet routed). The query parameters active, silenced, inhibited, and unprocessed are boolean filters — set them to true or false to include or exclude each state.

Each alert object has a labels map (the alert's label set from Prometheus), an annotations map (human-readable context like runbook URLs), a status object with a state field (active, suppressed, unprocessed) and a silencedBy array (silence IDs suppressing this alert), an inhibitedBy array (alert fingerprints of the inhibiting source alerts), and a receivers array (the receiver names this alert routes to). The startsAt timestamp indicates when Prometheus first sent this alert as firing — useful for computing alert duration.

async function listAlerts(am, options = {}) {
  const {
    activeOnly = true,   // Only truly firing (not silenced)
    filterLabels = {},   // e.g. { severity: 'critical', team: 'backend' }
  } = options;

  const params = new URLSearchParams();
  if (activeOnly) {
    params.set('active', 'true');
    params.set('silenced', 'false');
    params.set('inhibited', 'false');
  }

  // Label filter matchers: 'labelname=value' (comma-separated)
  const filterMatchers = Object.entries(filterLabels)
    .map(([k, v]) => `${k}="${v}"`)
    .join(',');
  if (filterMatchers) params.set('filter', filterMatchers);

  const alerts = await am.request('GET', `/api/v2/alerts?${params.toString()}`);

  return alerts.map((alert) => ({
    fingerprint: alert.fingerprint,
    labels: alert.labels,
    annotations: alert.annotations,
    state: alert.status.state,           // 'active' | 'suppressed' | 'unprocessed'
    silencedBy: alert.status.silencedBy,
    inhibitedBy: alert.status.inhibitedBy,
    receivers: alert.receivers.map((r) => r.name),
    startsAt: alert.startsAt,
    updatedAt: alert.updatedAt,
    // Derived: how long has this alert been firing?
    durationMs: Date.now() - new Date(alert.startsAt).getTime(),
    severity: alert.labels.severity,
    alertname: alert.labels.alertname,
  }));
}

// Summarize alerts by severity for an ops dashboard
async function getAlertSummary(am) {
  const allAlerts = await listAlerts(am, { activeOnly: false });

  const byState = {};
  const bySeverity = {};

  for (const alert of allAlerts) {
    byState[alert.state] = (byState[alert.state] ?? 0) + 1;
    bySeverity[alert.severity ?? 'none'] = (bySeverity[alert.severity ?? 'none'] ?? 0) + 1;
  }

  const criticalFiring = allAlerts.filter(
    (a) => a.state === 'active' && a.labels.severity === 'critical'
  );

  return {
    total: allAlerts.length,
    byState,
    bySeverity,
    criticalFiringCount: criticalFiring.length,
    criticalAlerts: criticalFiring.slice(0, 5), // Top 5 for display
  };
}

Creating and managing silences

A silence suppresses alerts whose labels match all of the silence's matchers. Silences are the primary mechanism for "muting known noise" during deployments, maintenance windows, or while an incident is being investigated. A silence does not stop Prometheus from evaluating the alert rule or Alertmanager from receiving the alert — it only stops the notification from being routed to receivers. The alert remains visible in the Alertmanager UI and API with state: suppressed.

Silence matchers use three operators: = (equality), != (inequality), =~ (regex), !~ (negative regex). Always include at least one equality matcher to avoid accidentally silencing all alerts. The createdBy and comment fields are required by the API and should include the MCP tool name, requesting user, and reason — these appear in the Alertmanager UI and are essential for incident retrospectives to understand why notifications were suppressed.

async function createSilence(am, params) {
  const {
    matchers,        // Array of { name, value, isRegex, isEqual }
    durationMinutes, // How long the silence should last
    createdBy,       // Identity of the requestor (MCP tool + user)
    comment,         // Required: reason for the silence
  } = params;

  const startsAt = new Date().toISOString();
  const endsAt = new Date(Date.now() + durationMinutes * 60 * 1000).toISOString();

  const silenceBody = {
    matchers: matchers.map((m) => ({
      name: m.name,
      value: m.value,
      isRegex: m.isRegex ?? false,
      isEqual: m.isEqual ?? true,
    })),
    startsAt,
    endsAt,
    createdBy,
    comment,
  };

  const result = await am.request('POST', '/api/v2/silences', silenceBody);
  return { silenceId: result.silenceID, startsAt, endsAt };
}

// Expire a silence immediately (Alertmanager calls this "delete")
async function expireSilence(am, silenceId) {
  await am.request('DELETE', `/api/v2/silences/${silenceId}`);
  return { silenceId, expired: true, expiredAt: new Date().toISOString() };
}

// List all active silences (not expired)
async function listActiveSilences(am) {
  const silences = await am.request('GET', '/api/v2/silences');

  return silences
    .filter((s) => s.status.state === 'active')
    .map((s) => ({
      id: s.id,
      state: s.status.state, // 'active' | 'expired' | 'pending'
      matchers: s.matchers,
      startsAt: s.startsAt,
      endsAt: s.endsAt,
      createdBy: s.createdBy,
      comment: s.comment,
      // How much time remains?
      remainingMs: new Date(s.endsAt).getTime() - Date.now(),
    }));
}

// Example: silence a specific deployment's alerts for 30 minutes
const silence = await createSilence(am, {
  matchers: [
    { name: 'alertname', value: 'HighErrorRate', isRegex: false, isEqual: true },
    { name: 'deployment', value: 'api-v2.3.1', isRegex: false, isEqual: true },
  ],
  durationMinutes: 30,
  createdBy: 'mcp-ops-tool/deploy-script',
  comment: 'Silencing during api-v2.3.1 rollout — expected elevated error rate during canary phase',
});

Inspecting routing configuration and status

The GET /api/v2/status endpoint returns Alertmanager's current configuration and cluster state. The config.original field contains the raw YAML configuration as a string — useful for MCP tools that audit routing rules or validate that a specific receiver is configured. The cluster field reports the HA peer list and their connection status, enabling detection of cluster split-brain conditions where nodes cannot communicate with each other.

A common operational issue: Alertmanager is "healthy" (HTTP 200, cluster healthy) but alerts are not being delivered. The v2 API does not expose notification delivery success/failure — this information is available only via Prometheus metrics scraped from Alertmanager's /metrics endpoint. The key metrics are alertmanager_notifications_total{integration="slack",status="success"} and alertmanager_notifications_failed_total{integration="slack"}. Building a combined probe that checks both the API status and the notification failure rate gives a complete picture of Alertmanager health.

async function getAlertmanagerStatus(am) {
  const status = await am.request('GET', '/api/v2/status');

  const peers = (status.cluster?.peers ?? []).map((p) => ({
    name: p.name,
    address: p.address,
    // 'alive' | 'failed' — 'failed' means gossip connection is down
    state: p.state,
  }));

  const failedPeers = peers.filter((p) => p.state !== 'alive');

  return {
    uptime: status.uptime,
    versionInfo: status.versionInfo,
    clusterStatus: status.cluster?.status, // 'ready' | 'settling' | 'disabled'
    clusterPeers: peers,
    clusterHealthy: failedPeers.length === 0,
    failedPeers,
    // Parse routing config summary from YAML (requires a YAML parser)
    configReceivers: extractReceiversFromConfig(status.config?.original),
    configRoutes: extractRoutesFromConfig(status.config?.original),
  };
}

// Fetch notification success/failure from Prometheus metrics endpoint
async function getNotificationMetrics(metricsEndpoint) {
  const response = await fetch(metricsEndpoint + '/metrics');
  const text = await response.text();

  // Parse Prometheus text format for alertmanager notification metrics
  const totalByIntegration = {};
  const failedByIntegration = {};

  for (const line of text.split('\n')) {
    if (line.startsWith('#') || !line.trim()) continue;

    // alertmanager_notifications_total{integration="pagerduty",status="success"} 42
    const totalMatch = line.match(
      /alertmanager_notifications_total\{integration="([^"]+)",status="([^"]+)"\}\s+([\d.]+)/
    );
    if (totalMatch) {
      const [, integration, status, count] = totalMatch;
      if (!totalByIntegration[integration]) totalByIntegration[integration] = {};
      totalByIntegration[integration][status] = parseFloat(count);
    }

    // alertmanager_notifications_failed_total{integration="slack"} 5
    const failedMatch = line.match(
      /alertmanager_notifications_failed_total\{integration="([^"]+)"\}\s+([\d.]+)/
    );
    if (failedMatch) {
      failedByIntegration[failedMatch[1]] = parseFloat(failedMatch[2]);
    }
  }

  return {
    notifications: totalByIntegration,
    failed: failedByIntegration,
    hasFailures: Object.values(failedByIntegration).some((v) => v > 0),
  };
}

function extractReceiversFromConfig(yamlStr) {
  if (!yamlStr) return [];
  // Simple extraction — replace with a YAML parser for production use
  const matches = [...yamlStr.matchAll(/- name: ['"]?([^'"\n]+)/g)];
  return matches.map((m) => m[1].trim());
}

function extractRoutesFromConfig(yamlStr) {
  if (!yamlStr) return null;
  // Return the raw YAML routes section for display or further parsing
  const routeMatch = yamlStr.match(/^route:(.+?)(?=^[a-z]|\z)/ms);
  return routeMatch?.[0] ?? null;
}

AliveMCP integration for Alertmanager health monitoring

Alertmanager's own health is a distinct concern from the alerts it manages. The Alertmanager process can crash, partition from its cluster peers, or fail to deliver notifications while appearing superficially healthy. Register AliveMCP probes for each Alertmanager layer: a direct HTTP probe at /-/healthy (returns OK on success) for process liveness, an API probe at /api/v2/status for cluster membership, and a Prometheus-scraping probe for notification failure rate.

An important deployment note: in a clustered Alertmanager setup, only one instance sends notifications for each grouped alert (the elected "leader" for that group). If you probe only one instance's notification metrics, you may see zero notifications on a non-leader instance even when delivery is working correctly. Aggregate notification metrics across all instances before alerting on failure rate.

// Minimal health probe for AliveMCP — register this at /health/alertmanager
async function alertmanagerHealthProbe(am, metricsUrl) {
  const [status, metrics] = await Promise.allSettled([
    am.request('GET', '/api/v2/status'),
    getNotificationMetrics(metricsUrl),
  ]);

  const clusterOk =
    status.status === 'fulfilled' &&
    status.value.cluster?.status === 'ready';

  const notificationsOk =
    metrics.status === 'fulfilled' &&
    !metrics.value.hasFailures;

  return {
    // HTTP 200 when clusterOk; HTTP 503 otherwise (AliveMCP sees non-200 as DOWN)
    healthy: clusterOk,
    clusterStatus: status.status === 'fulfilled'
      ? status.value.cluster?.status
      : 'unreachable',
    notificationFailures: metrics.status === 'fulfilled'
      ? metrics.value.failed
      : 'metrics_unavailable',
    warning: !notificationsOk ? 'notification_delivery_failures_detected' : null,
  };
}