MCP Tools for Observability Platforms: Dual-Credential Auth, Timestamp Units, and Health Probe Depth Across Datadog, New Relic, Grafana, Prometheus, and Jaeger

Why observability platforms are harder to integrate than they appear

Observability platforms are the tools developers use to understand whether their systems are working. The irony is that integrating them as MCP tools surfaces all the same reliability patterns they were built to monitor — and those patterns are more varied across platforms than you would expect from a category with a unified purpose.

Every platform in this arc has a working REST or GraphQL API. The TypeScript clients are mostly straightforward. The failure modes are almost all invisible until production: a Datadog MCP server where the Application Key was rotated three weeks ago and reads started silently returning 403; a New Relic integration that always returns 200 and never logs an error even after key expiry; a Jaeger trace search that returns empty results for every query because the timestamps are off by a factor of 1,000.

None of these failures look like failures. They look like empty query results. A general-purpose HTTP monitor — checking that port 80 responds — won't catch any of them. Understanding why requires walking through all three patterns across all five platforms.

Pattern 1 — Dual-credential auth models: which key validates what

The most surprising structural pattern across the observability arc is that the platforms which charge the most (Datadog, New Relic) have the most complex credential models — and the most dangerous silent failure modes when credentials rotate.

Datadog: the read/write credential split

Datadog's credential model separates write access from read access at the API level, not just at the role level. The DD-API-KEY header identifies your Datadog organization and authorizes ingest — submitting metrics, posting events, and registering service checks. It cannot read anything. The DD-APPLICATION-KEY header is tied to a specific Datadog user account and grants read access — querying timeseries metrics, listing monitors, searching logs, reading dashboards.

Both headers must be present on every read-after-write operation. A write to /api/v1/series succeeds with just the API key; reading those metrics back via v1.MetricsApi.queryMetrics() requires the Application Key. If only the API key is present on a read request, Datadog returns 403.

import { client, v1 } from '@datadog/datadog-api-client';

const configuration = client.createConfiguration({
  authMethods: {
    apiKeyAuth: process.env.DD_API_KEY!,
    appKeyAuth: process.env.DD_APP_KEY!,
  },
  // EU customers: serverVariables: { site: 'datadoghq.eu' }
});

const metricsApi = new v1.MetricsApi(configuration);
const monitorsApi = new v1.MonitorsApi(configuration);

// Raw axios for endpoints not yet in the typed SDK:
const ddHttp = axios.create({
  baseURL: 'https://api.datadoghq.com',
  headers: {
    'DD-API-KEY': process.env.DD_API_KEY!,
    'DD-APPLICATION-KEY': process.env.DD_APP_KEY!,
    'Content-Type': 'application/json',
  }
});

The dangerous failure mode: Application Keys inherit the role permissions of the Datadog user who created them. If that user's role is changed, or if the key owner leaves the organization and their account is deactivated, the Application Key stops working — but no error is logged at key creation time and no rotation notification is sent. The API key continues to accept metric submissions (writes succeed) while all reads silently return 403. An MCP server that only writes (posting deployment events) will appear fully healthy while every metric query tool fails.

Use a dedicated Datadog service account user for the Application Key. This decouples the key lifecycle from any individual team member's employment status. When you monitor with AliveMCP, the two-step health check catches the split — a failed Application Key validation triggers an alert even when the API key remains valid.

New Relic: different keys for different API families

New Relic's credential model is more fragmented: different API families use structurally different credential types, and the header names are not consistent.

NerdGraph (the primary GraphQL API for NRQL queries, alert policy management, and entity search) uses a "User Key" with the header name Api-Key — not Authorization: Bearer as you might expect from a modern API. The Insights Events Collector API (for recording custom events) uses a separate "Insert Key" with the header X-Insert-Key. These are different credentials, created in different places in the New Relic UI, with completely separate expiry and rotation lifecycles.

// NerdGraph (NRQL queries, alert policies, entity search):
const nerdgraph = axios.create({
  baseURL: 'https://api.newrelic.com',
  headers: {
    'Api-Key': process.env.NEW_RELIC_USER_KEY!,   // NOT "Authorization: Bearer"
    'Content-Type': 'application/json',
  },
});

// Insights Events API (recording custom events):
const insightsCollector = axios.create({
  baseURL: `https://insights-collector.newrelic.com`,
  headers: {
    'X-Insert-Key': process.env.NEW_RELIC_INSERT_KEY!,  // Separate from User Key
    'Content-Type': 'application/json',
  },
});

The critical gotcha: NerdGraph always returns HTTP 200, even for authentication failures. An expired User Key does not produce an HTTP 401 or 403. Instead, the response body contains an errors array with the error message. If your MCP tool only checks HTTP status codes and doesn't inspect the response body, it will treat every auth failure as a successful empty result. Your tool handler must always check res.data.errors?.length before accessing the data.

async function nrQuery(query: string, variables: Record<string, unknown> = {}) {
  const res = await nerdgraph.post('/graphql', { query, variables });
  // Do NOT check HTTP status — it is always 200.
  // Auth failures, syntax errors, and missing data all return HTTP 200.
  if (res.data.errors?.length) {
    throw new Error(`NerdGraph error: ${res.data.errors[0].message}`);
  }
  return res.data.data;
}

HTTP 403 from NerdGraph is not an authentication error — it means the request format was wrong (malformed JSON body, missing Content-Type). This is the opposite of every conventional API. A naïve auth-failure handler that retries on 403 will loop infinitely on a format error while the actual auth failure (which returns 200) goes undetected.

Grafana: the token deprecation migration

Grafana's credential model changed significantly in version 10. Legacy API keys — the long-lived JWT tokens from previous Grafana versions — are deprecated in Grafana 10 and scheduled for removal in a future major release. Service Account tokens are the replacement: they use the same Authorization: Bearer header, carry a role assignment (Viewer / Editor / Admin), and have explicit token names that make rotation auditable.

const grafana = axios.create({
  baseURL: process.env.GRAFANA_URL!,
  headers: {
    Authorization: `Bearer ${process.env.GRAFANA_SERVICE_TOKEN!}`,
    'Content-Type': 'application/json',
    Accept: 'application/json',
  },
});

The practical difference matters if you have a legacy Grafana MCP integration: a legacy API key token starts with eyJ (JWT format), while a Service Account token starts with glsa_. If you attempt to use a legacy API key against a Grafana 10+ instance that has disabled legacy key support, you get a 401. Check your token prefix before debugging auth issues — the credential type determines which Grafana administration screen you need to visit to rotate it.

Prometheus and Jaeger: no built-in auth

Prometheus and Jaeger have no built-in authentication. Both are designed for cluster-internal deployment and rely on the surrounding infrastructure (a reverse proxy, a service mesh, or network-level access controls) to restrict access. This simplifies your MCP server's credential management — there is no credential to rotate — but it shifts the auth burden to the deployment layer. If your MCP server needs to reach a Prometheus or Jaeger instance that is protected by basic auth at the reverse proxy, pass those credentials via axios's auth option, not as API keys.

const prom = axios.create({
  baseURL: process.env.PROMETHEUS_URL ?? 'http://localhost:9090',
  ...(process.env.PROMETHEUS_USERNAME && {
    auth: {
      username: process.env.PROMETHEUS_USERNAME,
      password: process.env.PROMETHEUS_PASSWORD!,
    }
  }),
  timeout: 30_000,
});

const jaeger = axios.create({
  baseURL: process.env.JAEGER_URL ?? 'http://localhost:16686',
  ...(process.env.JAEGER_USERNAME && {
    auth: {
      username: process.env.JAEGER_USERNAME,
      password: process.env.JAEGER_PASSWORD!,
    }
  }),
  timeout: 30_000,
});

The credential table across all five platforms:

The architectural takeaway: Datadog and New Relic both have split credential models where different operations require different credentials. This means rotating one credential does not validate the other. Your health check must independently validate every credential in the model — not just the one that controls your most-used operation.

Pattern 2 — The timestamp unit diversity trap

Every observability platform stores and queries time-series data, and every platform uses a different timestamp unit. JavaScript's Date.now() returns milliseconds. Exactly one of the five platforms (Grafana annotations) uses milliseconds natively. The others use seconds or microseconds. Passing the wrong unit produces empty results or silently queries the wrong time range — no error message, just missing data.

The unit table

Datadog: the within-platform unit mismatch

Datadog's timestamp inconsistency is internal to a single API call. The from and to query parameters for v1.MetricsApi.queryMetrics() are Unix epoch seconds. But each data point in the returned pointlist array uses millisecond timestamps. This means you must use different conversion logic for the same metric value depending on whether you're reading the query parameters or the result timestamps.

const result = await metricsApi.queryMetrics({
  from: Math.floor(Date.now() / 1000) - 3600,  // 1 hour ago, in SECONDS
  to: Math.floor(Date.now() / 1000),            // now, in SECONDS
  query: 'avg:system.cpu.user{host:web-01}',
});

// result.series[n].pointlist is [[timestamp_ms, value], ...]
// The timestamp here is MILLISECONDS, not seconds
const points = (result.series ?? []).flatMap(s =>
  (s.pointlist ?? []).map(([ts, val]) => ({
    // ts is milliseconds — divide by 1000 for seconds, or use directly in new Date(ts)
    timestamp_iso: new Date(ts).toISOString(),
    value: val,
  }))
);

The rate limit adds another dimension: Datadog's metrics query endpoint is limited to 300 requests per hour — far lower than most other Datadog endpoints. The monitors API and events API are far more permissive. Design your MCP server's tool set with this asymmetry in mind: an agent that queries metrics in a loop will exhaust the quota in five minutes.

Prometheus: values are always strings

Prometheus uses Unix seconds for all timestamps. The conversion is consistent: always Math.floor(Date.now() / 1000) for the time (instant query) or start/end (range query) parameters. The silent failure when you pass milliseconds: Prometheus queries the year 50,000 and returns an empty vector. No error — just empty results that look like "no matching metrics found."

There is a second, equally silent bug that trips up first implementations: Prometheus always returns metric values as strings, not numbers. The value[1] field in both instant and range query results is always a JSON string like "0.23456", even for numeric metrics. Always apply parseFloat() before returning the value to the LLM — comparing a string "0.2" to a threshold number 0.5 will produce wrong results that look arithmetically correct.

// Instant query result shape:
// result = [{ metric: {__name__: "...", ...labels}, value: [unix_seconds, "numeric_string"] }]

const results = envelope.data.result.map((r: any) => ({
  labels: r.metric,
  timestamp: r.value[0],             // seconds — correct
  // Always parseFloat — Prometheus returns values as strings
  value: parseFloat(r.value[1]),     // "0.23456" → 0.23456
  // parseFloat handles "NaN", "+Inf", "-Inf" correctly
}));

For range queries, add a data-point guard. Prometheus will return up to the number of points implied by (end - start) / step. A 7-day range with a 1-second step would request 604,800 data points. Prometheus will serve them, but your MCP tool response will be enormous and exhaust the LLM's context window. Cap the effective number of data points:

server.tool('query_range', {
  query: z.string(),
  start: z.number().int().describe('Start time as Unix epoch SECONDS'),
  end: z.number().int().describe('End time as Unix epoch SECONDS'),
  step: z.string().describe('Step as a Prometheus duration string: "5m", "1h", "30s"')
}, async ({ query, start, end, step }) => {
  // Parse step to seconds to check point count
  const stepSeconds = parsePromDuration(step);
  const expectedPoints = Math.ceil((end - start) / stepSeconds);
  if (expectedPoints > 11_000) {
    return {
      isError: true,
      content: [{ type: 'text', text: `Query would return ~${expectedPoints} data points. Reduce time range or increase step.` }]
    };
  }
  // ... proceed with query
});

Jaeger: microseconds — the most common integration bug

Jaeger uses Unix microseconds for all timestamp parameters in trace search — one millionth of a second. JavaScript's Date.now() returns milliseconds. The conversion is Date.now() * 1000 (multiply, not divide). This is the single most common first-implementation bug in Jaeger MCP tools.

When you pass milliseconds to Jaeger's start and end parameters, you are specifying a time range in the year 1970. Jaeger's storage backend has no data for that range. The response is { "data": [], "total": 0 } — empty, no error, no indication that the time range was invalid. The tool appears to work, returns an empty trace list, and the agent concludes there are no matching traces.

// WRONG — passes milliseconds, Jaeger returns 0 results:
const start = Date.now() - 60 * 60 * 1000;  // 1 hour ago in ms
const end   = Date.now();                     // now in ms

// CORRECT — multiply by 1000 to convert ms → microseconds:
const nowUs   = Date.now() * 1000;
const startUs = Date.now() * 1000 - 60 * 60 * 1_000_000;

// Build a dedicated helper module — never compute raw timestamps inline:
// jaeger-time.ts
export const nowUs        = ()          => Date.now() * 1000;
export const hoursAgoUs   = (h: number) => nowUs() - h * 3_600_000_000;
export const minutesAgoUs = (m: number) => nowUs() - m *    60_000_000;

// Usage:
import { hoursAgoUs, nowUs } from './jaeger-time.js';

server.tool('search_traces', {
  service: z.string(),
  lookback_hours: z.number().min(0.1).max(24).default(1)
}, async ({ service, lookback_hours }) => {
  const { data } = await jaeger.get('/api/traces', {
    params: {
      service,
      start: hoursAgoUs(lookback_hours),  // microseconds
      end:   nowUs(),                      // microseconds
      limit: 20,
    }
  });
  // ...
});

The microsecond conversion applies to every timestamp parameter in the Jaeger Query API: start, end, and the minDuration/maxDuration filters (which use Prometheus-style duration strings like "500ms" and "5s" rather than microseconds). Only the trace ID is not a timestamp. Build the conversion helper module as a first step in any Jaeger integration, and import it everywhere rather than computing timestamps inline in tool handlers.

New Relic: NRQL makes timestamp units mostly invisible

New Relic's NRQL query language has the most ergonomic timestamp handling across the five platforms — and it is not a coincidence. Because NRQL queries express time ranges as relative clauses (SINCE 1 hour ago, SINCE 30 minutes ago UNTIL 15 minutes ago), the client never constructs raw timestamp values for most queries. The New Relic API server handles the translation. The only place where raw timestamps appear is when using absolute NRQL clauses (SINCE 1720000000) or when calling the Insights Events Collector directly to record custom events — and there the unit is Unix seconds.

// NRQL with relative time — no raw timestamp handling needed:
const NRQL_QUERY = `
  query NrqlQuery($accountId: Int!, $nrql: Nrql!) {
    actor {
      account(id: $accountId) {
        nrql(query: $nrql) {
          results
        }
      }
    }
  }
`;

// Safe: NRQL handles time resolution
await nrQuery(NRQL_QUERY, {
  accountId: ACCOUNT_ID,
  nrql: 'SELECT average(duration) FROM Transaction SINCE 1 hour ago'
});

// Insights Events API — rare case where you need Unix seconds:
await insightsCollector.post(
  `/v1/accounts/${ACCOUNT_ID}/events`,
  [{
    eventType: 'McpToolCall',
    timestamp: Math.floor(Date.now() / 1000),  // SECONDS
    tool: 'query_nrql',
    duration_ms: 42,
  }]
);

Pattern 3 — Health probe depth: why process-alive checks miss every important failure

The third pattern is the one that connects most directly to what AliveMCP monitors: the difference between a health endpoint that tells you the server process is alive and a health endpoint that tells you whether your MCP tools will actually return useful data. Every platform in this arc has an invisible failure mode — one where the server process is running and health endpoints return 200 while every MCP tool call fails or returns empty results.

Datadog: the two-step credential validation

Datadog provides a GET /api/v1/validate endpoint specifically for health checking API keys. It is not sufficient on its own. The validate endpoint only confirms that the DD-API-KEY is valid — it does not check the Application Key at all. A Datadog integration where the API Key is valid but the Application Key has been rotated or revoked will pass the validate check and then fail on every query_metrics, list_monitors, and other read operation.

app.get('/health/datadog', async (req, res) => {
  const checks: Record<string, unknown> = {};

  // Step 1: validate the API key — but this does NOT check the Application key
  try {
    const validateRes = await ddHttp.get('/api/v1/validate');
    checks.api_key = validateRes.data.valid === true ? 'valid' : 'invalid';
  } catch (err: any) {
    checks.api_key = `invalid (HTTP ${err.response?.status})`;
    return res.status(503).json({ status: 'unhealthy', checks });
  }

  // Step 2: validate the Application key by making a lightweight read
  // GET /api/v1/monitor?page_size=1 requires DD-APPLICATION-KEY
  try {
    await ddHttp.get('/api/v1/monitor', { params: { page_size: 1 } });
    checks.app_key = 'valid';
  } catch (err: any) {
    checks.app_key = err.response?.status === 403
      ? 'invalid or insufficient permissions'
      : `error (HTTP ${err.response?.status})`;
    return res.status(503).json({ status: 'unhealthy', checks });
  }

  return res.json({ status: 'healthy', checks });
});

A single probe can't validate both keys simultaneously because the validate endpoint intentionally ignores the Application Key. The two-step pattern is not optional — it is the minimum necessary check to confirm that read MCP tools will function.

New Relic: reading the response body, not the HTTP status

New Relic's NerdGraph API always returns HTTP 200 for every request, including authentication failures. This means every conventional HTTP health monitoring tool — including most generic uptime monitors — will always show New Relic as "healthy" even when the User Key is expired. To detect an authentication failure, you must parse the response body and check the errors array.

app.get('/health/newrelic', async (req, res) => {
  try {
    // Minimal query that validates the User Key and requires no specific permissions
    const HEALTH_QUERY = `{ actor { user { name email } } }`;
    const response = await nerdgraph.post('/graphql', { query: HEALTH_QUERY });

    // HTTP status is always 200 — check the body for errors
    if (response.data.errors?.length) {
      return res.status(503).json({
        status: 'unhealthy',
        error: response.data.errors[0].message,
        note: 'NerdGraph returns HTTP 200 even for auth failures — see errors[]'
      });
    }

    const user = response.data.data?.actor?.user;
    return res.json({
      status: 'healthy',
      user_email: user?.email,
      user_name: user?.name,
    });
  } catch (err: any) {
    // Actual HTTP errors from NerdGraph mean a format problem, not auth
    return res.status(503).json({
      status: 'unhealthy',
      error: err.message,
    });
  }
});

The health query { actor { user { name email } } } is deliberately minimal. It validates the User Key and returns the authenticated user's name and email — enough confirmation that the credential is valid and the account is active. It does not require any specific Datadog or account permissions beyond basic authentication.

Grafana: the database field versus datasource connectivity

Grafana's GET /api/health endpoint returns {"database": "ok", "version": "9.5.x"} with HTTP 200 when Grafana is operational. The database field refers to Grafana's own internal database (SQLite or PostgreSQL, depending on deployment) — not to any of the datasources (Prometheus, Loki, Elasticsearch) that your MCP tools query through Grafana. A Grafana instance with a healthy internal database but all upstream datasources unreachable will pass the /api/health check and fail on every query.

app.get('/health/grafana', async (req, res) => {
  try {
    const { data } = await grafana.get('/api/health');

    // data.database: "ok" means Grafana's own DB is healthy
    // This does NOT indicate whether your datasources (Prometheus, Loki, etc.) are reachable
    if (data.database !== 'ok') {
      return res.status(503).json({ status: 'unhealthy', database: data.database });
    }

    // For MCP tools that query specific datasources, extend the health check:
    // GET /api/datasources/:id/health (Grafana 9+) validates a specific datasource
    // This is optional but recommended if your tools are datasource-specific

    return res.json({
      status: 'healthy',
      grafana_version: data.version,
      database: data.database,
    });
  } catch (err: any) {
    // HTTP 503 from /api/health means Grafana is starting up or DB is degraded
    return res.status(503).json({ status: 'unhealthy', error: err.message });
  }
});

For MCP tools that proxy queries to specific datasources via POST /api/ds/query, add a per-datasource health check using GET /api/datasources/:id/health (available since Grafana 9). This endpoint actually tests connectivity to the upstream datasource, not just Grafana's own health. Get the datasource ID from GET /api/datasources at startup and cache it.

Prometheus: ready versus healthy

Prometheus exposes two health endpoints with critically different semantics. GET /-/healthy returns HTTP 200 when the Prometheus process is alive — even if it just started and has not yet loaded any TSDB data, and even if it cannot serve any queries. GET /-/ready returns HTTP 200 only when Prometheus has fully loaded its data and is ready to serve queries. An agent that queries /-/healthy will see a green health check during a Prometheus restart window, while all actual queries return errors because the TSDB is not yet ready.

app.get('/health/prometheus', async (req, res) => {
  try {
    // Use /-/ready, not /-/healthy
    // /-/healthy returns 200 even when TSDB is not yet loaded
    // /-/ready returns 200 only when Prometheus can serve queries
    await prom.get('/-/ready');

    // Optionally validate with a real query to confirm PromQL works:
    const check = await prom.get('/api/v1/query', {
      params: { query: 'up', time: String(Math.floor(Date.now() / 1000)) }
    });

    return res.json({
      status: 'healthy',
      ready: true,
      targets_up: check.data.data?.result?.length ?? 0,
    });
  } catch (err: any) {
    // /-/ready returns 503 when not ready (startup, WAL replay in progress)
    return res.status(503).json({
      status: 'unhealthy',
      error: err.message,
      note: 'Use /-/ready, not /-/healthy — /-/healthy is true before TSDB loads'
    });
  }
});

Jaeger: the storage backend proof

Jaeger's process health check — whether the Query API process is accepting connections on port 16686 — tells you nothing about whether the storage backend (Elasticsearch, Cassandra, or BadgerDB) is reachable. If the storage backend is down, Jaeger's process stays running but every trace search returns empty results. The correct Jaeger health probe is not a connection check to port 16686 — it is an actual API call that exercises the storage backend.

app.get('/health/jaeger', async (req, res) => {
  try {
    // GET /api/services does two things:
    // 1. Confirms the Jaeger Query API is accepting requests
    // 2. Queries the storage backend to retrieve service names
    // If the storage backend is down, this call returns an error — not 200
    const { data } = await jaeger.get('/api/services');

    // data.data is an array of service names
    // An empty array means no traces have been indexed — not necessarily unhealthy
    // An error or exception means the storage backend is unreachable
    return res.json({
      status: 'healthy',
      service_count: data.data?.length ?? 0,
      services: data.data ?? [],
    });
  } catch (err: any) {
    return res.status(503).json({
      status: 'unhealthy',
      error: err.message,
      note: '/api/services is the correct Jaeger health probe — it validates storage backend connectivity'
    });
  }
});

The reason /api/services is the right probe: it is the same query that returns meaningful data for the agent, so it exercises exactly the storage path that your MCP tools use. A generic HTTP check on the Jaeger UI port would pass even if Elasticsearch is down and every search_traces call returns empty.

The health probe comparison table

Where the integrations diverge: query models and cross-platform architecture

Beyond the three patterns, the five platforms diverge significantly in their query models. Understanding where they differ helps you design a multi-platform observability MCP server that does not accidentally abstract too much.

Metric query languages are incompatible

Datadog uses its own metrics query DSL (avg:system.cpu.user{host:web-01}). New Relic uses NRQL (SELECT average(duration) FROM Transaction SINCE 1 hour ago). Prometheus uses PromQL (rate(http_requests_total[5m])). These are three structurally different query languages. An MCP tool that accepts a generic "query" parameter and routes it to the platform the agent is asking about must surface the query language as an explicit part of the tool schema — agents trained on one platform's syntax will produce invalid queries on another.

The correct design: separate tools per platform, with input schema descriptions that explicitly name the query language. query_datadog_metrics({ query: 'avg:...' }), query_nrql({ nrql_query: 'SELECT ...' }), query_prometheus({ promql: '...' }). Do not create a query_metrics({ platform, query }) abstraction — it looks useful but produces confusing tool descriptions for the LLM.

Trace search only exists in Jaeger

Only Jaeger in this arc is a distributed tracing backend. Datadog and New Relic also have tracing products (APM), but their tracing APIs are separate from the monitoring APIs covered here. Prometheus and Grafana do not store or search distributed traces natively. If an agent needs to debug a slow request using a trace ID from a log line, the Jaeger MCP tool is the only one in this arc that can resolve a trace ID to its span tree. The trace ID format is 16 or 32 hexadecimal characters — the W3C traceparent header uses 32-character trace IDs.

server.tool('get_trace', {
  trace_id: z.string().regex(/^[0-9a-f]{16,32}$/, {
    message: 'Trace ID must be 16 or 32 lowercase hex characters'
  }).describe('Jaeger trace ID — from a traceparent header, log line, or search_traces result.')
}, async ({ trace_id }) => {
  const { data } = await jaeger.get(`/api/traces/${trace_id}`);
  // Returns a trace with all its spans, processes, and warnings
  // ...
});

Grafana is a proxy layer, not a primary data store

Grafana does not store metrics or traces — it proxies queries to backend datasources (Prometheus, Loki, Elasticsearch, InfluxDB, etc.) and renders them in dashboards. POST /api/ds/query proxies a PromQL query to a Prometheus datasource without requiring your MCP server to have direct access to Prometheus. This is powerful for teams that have locked down direct Prometheus access but granted Grafana service account tokens to their MCP servers. The datasource ID is required for the proxy call and must be discovered at startup via GET /api/datasources.

// Discover datasource IDs at startup:
const { data: datasources } = await grafana.get('/api/datasources');
const promDatasource = datasources.find((ds: any) => ds.type === 'prometheus');

// Proxy a PromQL query through Grafana without direct Prometheus access:
const queryResult = await grafana.post('/api/ds/query', {
  queries: [{
    datasourceId: promDatasource.id,
    expr: 'rate(http_requests_total[5m])',
    instant: true,
    refId: 'A',
  }],
  from: 'now-1h',
  to: 'now',
});

Rate limit comparison

The composable observability MCP server

If you are building a single MCP server that integrates multiple observability platforms, the three patterns above suggest a consistent structure across all five platform integrations.

Initialize all platform clients at startup, not inside tool handlers. Validate all credentials during initialization and fail fast if any credential check fails — a health check at startup is cheaper than discovering a broken credential during the first agent interaction.

// At server startup — validate all platforms before registering tools
async function initializeClients() {
  // Datadog: two-step validation
  await ddHttp.get('/api/v1/validate');
  await ddHttp.get('/api/v1/monitor', { params: { page_size: 1 } });

  // New Relic: body inspection
  const nrHealth = await nerdgraph.post('/graphql', {
    query: '{ actor { user { name } } }'
  });
  if (nrHealth.data.errors?.length) throw new Error('New Relic auth failed');

  // Grafana: health endpoint
  const grafanaHealth = await grafana.get('/api/health');
  if (grafanaHealth.data.database !== 'ok') throw new Error('Grafana DB unhealthy');

  // Prometheus: ready check
  await prom.get('/-/ready');

  // Jaeger: storage backend check
  await jaeger.get('/api/services');
}

// Expose a single /health endpoint that checks all five:
app.get('/health', async (req, res) => {
  const checks: Record<string, 'healthy' | 'unhealthy'> = {};
  const promises = [
    checkDatadog().then(() => { checks.datadog = 'healthy'; }).catch(() => { checks.datadog = 'unhealthy'; }),
    checkNewRelic().then(() => { checks.newrelic = 'healthy'; }).catch(() => { checks.newrelic = 'unhealthy'; }),
    checkGrafana().then(() => { checks.grafana = 'healthy'; }).catch(() => { checks.grafana = 'unhealthy'; }),
    checkPrometheus().then(() => { checks.prometheus = 'healthy'; }).catch(() => { checks.prometheus = 'unhealthy'; }),
    checkJaeger().then(() => { checks.jaeger = 'healthy'; }).catch(() => { checks.jaeger = 'unhealthy'; }),
  ];
  await Promise.allSettled(promises);
  const allHealthy = Object.values(checks).every(v => v === 'healthy');
  return res.status(allHealthy ? 200 : 503).json({ status: allHealthy ? 'healthy' : 'degraded', checks });
});

Register this /health endpoint with AliveMCP. Because the endpoint aggregates all five platform checks — including the body-inspection check for New Relic and the two-step validation for Datadog — AliveMCP's 60-second polling will catch any platform credential failure within one minute, before your agents start returning empty observability data with no explanation.

Summary: the three patterns as a checklist

For every observability platform MCP integration:

1. Dual-credential auth: Identify all credentials the platform requires. Validate each one independently in your health check. Do not assume that a successful write validates your read credentials. Datadog and New Relic both have split credential models where a successful API key validation says nothing about Application Key or User Key validity.

2. Timestamp units: Check the platform's timestamp unit for both input parameters and output values before writing any query tool. Build conversion helpers and import them rather than computing raw timestamps inline. Datadog mixes seconds and milliseconds within a single API call; Prometheus uses seconds consistently; Jaeger uses microseconds. Never use Date.now() directly as a timestamp parameter without verifying the expected unit.

3. Health probe depth: Use the probe endpoint that exercises the actual failure modes your MCP tools depend on — not the process health endpoint. For Datadog: two-step credential validation. For New Relic: body inspection, not HTTP status. For Grafana: datasource health in addition to internal DB health. For Prometheus: /-/ready, not /-/healthy. For Jaeger: /api/services, not a TCP port check.

All five platforms need external monitoring with AliveMCP because their internal health endpoints are designed to report on the platform process, not on the observability integration your MCP server depends on. An expired Datadog Application Key passes Datadog's own validate endpoint. An expired New Relic User Key returns HTTP 200. A Grafana instance with unreachable datasources reports a healthy database. None of these failures appear in the platform's own health response — they appear as empty tool results after your agent has already called the tool and reported no metrics.

Further reading

• MCP Server Datadog — dual-key auth, query_metrics, list_monitors, and /health/datadog
• MCP Server New Relic — NerdGraph NRQL queries, alert policies, and /health/newrelic
• MCP Server Grafana — dashboards by UID, annotations, datasource proxy, and /health/grafana
• MCP Server Prometheus — instant and range PromQL queries, target listing, and /-/ready health
• MCP Server Jaeger — microsecond timestamps, trace search, get by ID, and /health/jaeger
• Building MCP Tools for DevOps Platforms — singleton clients, credential lifecycle, and health transparency
• MCP Server Health Check — probe depth and alert tier design
• MCP Server Error Handling — structured errors and retry patterns
• MCP Monitoring Tool — evaluating options for MCP-aware uptime monitoring
• JSON-RPC Health Checks vs HTTP Probes — what each layer catches