MCP Server Alert Routing Architecture — multi-channel, deduplication, and escalation

The complete alert routing pipeline

An MCP server alert begins at detection and ends at resolution. The pipeline has six stages:

1. Detect. AliveMCP probes your server every minute via a full MCP initialize handshake. If the server fails 2 consecutive checks (2 minutes of confirmed downtime), it sends alert.triggered to your configured webhook URL.
2. Classify. Your bridge receives the webhook and classifies the alert severity based on the server's tier (internal vs third-party), the failure type (protocol failure vs connection refused vs timeout), and any business rules you configure.
3. Deduplicate. Check whether an active alert already exists for this server slug. If yes, update the existing incident rather than creating a new one.
4. Route. Send the alert to the appropriate channel(s) based on severity. Critical alerts go to PagerDuty + Slack. Warning alerts go to Slack only. Info alerts go to an email digest queue.
5. Escalate. If the alert is not acknowledged within the configured timeout, upgrade to the next escalation level (push → phone call → secondary contact).
6. Resolve. When AliveMCP sends alert.resolved, close the incident in every channel where it was opened: resolve the PagerDuty incident, update the Slack message to green, delete from the email queue.

Severity taxonomy for MCP server alerts

Not all MCP server failures have the same urgency. The severity taxonomy maps failure characteristics to response expectations.

The classification logic in your bridge should be deterministic and fast: if any condition matches a higher severity, use the higher severity. Do not attempt ML-based severity prediction for MCP server alerts — the rule table above covers 95% of real incidents, and a misclassified severe incident is worse than an overclassified minor one.

function classifySeverity(event) {
  const { failure_type, server_tier, downtime_seconds, schema_drift } = event;

  // P1: connection refused means the process is dead
  if (failure_type === 'connection_refused') return 'P1';

  // P1: internal servers are always critical
  if (server_tier === 'internal') return 'P1';

  // P2: author-claimed servers or sustained high latency
  if (server_tier === 'author') return 'P2';
  if (event.p95_latency_ms > event.p95_baseline_ms * 10) return 'P2';

  // P3: third-party or schema drift
  if (schema_drift) return 'P3';
  if (server_tier === 'thirdparty') return 'P3';

  // P4: everything else (short blips, recovered quickly)
  if (downtime_seconds < 180) return 'P4';

  return 'P3';  // default: medium
}

Deduplication strategy

AliveMCP sends alert.triggered every minute while a server is down. Without deduplication, a 30-minute outage generates 30 separate alerts. Deduplication requires two things: a stable key per server, and state to track whether an alert is already open.

The deduplication key pattern:

• Key: alivemcp-{serverSlug} — use the server slug, not the server name (names can change, slugs don't).
• State store: A key-value store mapping dedup_key → {channelAlertIds}, where channelAlertIds contains the identifier for the active alert in each routing channel (Slack message ID, PagerDuty incident key, OpsGenie alert alias).
• TTL: Set a 24-hour TTL on stored entries. If a server is still down after 24 hours, create a new alert rather than updating a 24-hour-old one — the original context has scrolled out of view.

// Deduplication state store (Redis-backed)
class AlertStateStore {
  async getActiveAlert(serverSlug) {
    const raw = await redis.get(`alert:${serverSlug}`);
    return raw ? JSON.parse(raw) : null;
  }

  async setActiveAlert(serverSlug, alertIds) {
    await redis.set(
      `alert:${serverSlug}`,
      JSON.stringify(alertIds),
      'EX', 86400   // 24-hour TTL
    );
  }

  async clearActiveAlert(serverSlug) {
    await redis.del(`alert:${serverSlug}`);
  }
}

// In your webhook handler:
const existing = await store.getActiveAlert(event.server_slug);
if (existing) {
  // Update all channels with new timestamp / failure count
  await updateAllChannels(existing, event);
} else {
  // Create new alerts in all channels, collect their IDs
  const alertIds = await createInAllChannels(event, severity);
  await store.setActiveAlert(event.server_slug, alertIds);
}

Escalation ladder design

An escalation ladder upgrades the alert to a louder channel if no one responds within a time threshold. For MCP servers, the standard ladder is:

The escalation ladder lives in PagerDuty or OpsGenie's escalation policy configuration, not in your bridge code. Your bridge only needs to trigger the initial incident; the on-call tool handles the escalation timing. What your bridge does control is the severity mapping: a P1 alert starts the ladder at T+0 with phone call enabled; a P3 alert starts with push only and no escalation ladder (it is handled during business hours).

Multi-channel fan-out

Critical MCP server failures should reach multiple channels simultaneously, not sequentially. A bridge that posts to Slack first and then PagerDuty means the first responder might see the Slack message and start investigating without acknowledging in PagerDuty — which causes the escalation ladder to continue even though someone is already handling the incident.

Fan out to all channels in parallel using Promise.allSettled, then collect results. Promise.allSettled rather than Promise.all ensures a failure in one channel (e.g., Slack is having an outage) does not prevent the alert from reaching PagerDuty.

async function routeAlert(event, severity) {
  const routes = [];

  if (severity === 'P1' || severity === 'P2') {
    routes.push(routeToPagerDuty(event, severity));
    routes.push(routeToSlack(event, severity, { ping: true }));
  }

  if (severity === 'P3') {
    routes.push(routeToSlack(event, severity, { ping: false }));
    routes.push(createJiraTicket(event));
  }

  if (severity === 'P4') {
    routes.push(addToEmailDigest(event));
  }

  // Always include Discord for community visibility
  routes.push(routeToDiscord(event, severity));

  const results = await Promise.allSettled(routes);
  const failures = results.filter(r => r.status === 'rejected');
  if (failures.length > 0) {
    console.error('Alert routing partial failure:', failures.map(f => f.reason));
    // Don't throw — partial delivery is better than no delivery
  }

  return results;
}

Maintenance window suppression

Planned MCP server deployments cause brief downtime. Without suppression, a 30-second rolling restart during a deployment generates a flurry of alerts that train your team to ignore notifications.

Implement maintenance windows as time-bounded suppression rules. AliveMCP's Team tier supports API-configurable maintenance windows. Alternatively, manage suppression in your bridge:

// maintenance-windows.json: [{serverSlug, startISO, endISO, reason}]
const windows = JSON.parse(fs.readFileSync('./maintenance-windows.json'));

function isInMaintenanceWindow(serverSlug) {
  const now = Date.now();
  return windows.some(w =>
    w.serverSlug === serverSlug &&
    new Date(w.startISO).getTime() <= now &&
    new Date(w.endISO).getTime() >= now
  );
}

// In your webhook handler:
if (isInMaintenanceWindow(event.server_slug)) {
  console.log(`Suppressing alert for ${event.server_slug}: maintenance window active`);
  return res.status(200).json({ suppressed: true });
}

Integrate maintenance window management into your deploy pipeline: before running helm upgrade or kubectl rollout, add the server slug to the maintenance window file with a 10-minute window. After the deploy completes, remove the window. This way alert suppression is a code-committed artifact that goes through your normal deployment review process.

Further reading

• PagerDuty for MCP Servers — Events API integration and escalation policies
• OpsGenie for MCP Servers — team-based routing and on-call schedules
• Discord Alerts for MCP Servers — webhook routing and embed formatting
• MCP Server Slack Alerts — channel routing and Slack Block Kit formatting
• MCP Server Incident Runbook — response playbook for common failure modes
• MCP Server Flapping — detecting and suppressing oscillating alerts