MCP Tools for Observability Data Stores: Query Language Diversity, Health Check Deception, and Data Gap Semantics Across Elasticsearch, Loki, Alertmanager, InfluxDB, and Dynatrace

Pattern 1: Query languages are mutually incompatible

Every observability data store in this arc uses a distinct query language built around a different fundamental data model. The mental model that works for Elasticsearch's document-centric relevance search does not transfer to Loki's log-stream filter model. The pipe-forward functional transforms of Flux do not resemble the time-series selector chains of Dynatrace's metricSelector. And Alertmanager's label-matcher syntax applies only to alert label sets — it has no concept of data retrieval at all.

For MCP tools that span multiple observability systems — a common pattern in agent-driven incident investigation workflows — each platform requires entirely independent query construction logic, error handling, and result parsing. There is no useful abstraction that maps concepts across these five languages.

Elasticsearch: Query DSL scoring vs filtering

Elasticsearch's Query DSL is built around the bool compound query type with four clause types: must (match required, contributes to relevance score), filter (match required, binary-cached, no score contribution), should (match preferred, boosts score if present), and must_not (match excluded, cached like filter). The critical performance distinction for MCP tools building monitoring or investigation queries is between must and filter.

A clause in must runs the relevance scoring calculation for every matching document on every query — a floating-point computation that cannot be cached. A clause in filter is cached as a bitset of document IDs after the first execution and reused on subsequent queries at near-zero cost. Every date range, status field check, and categorical filter in a production MCP tool belongs in the filter array, not in must. Using must for these common filter patterns makes each query re-score every document on each request, burning CPU and increasing latency at exactly the moment an incident is under investigation and query throughput matters most.

// Wrong: date range and status in must (uncached, scored on every query)
{
  "query": {
    "bool": {
      "must": [
        { "range": { "@timestamp": { "gte": "now-1h" } } },
        { "term": { "level": "error" } }
      ]
    }
  }
}

// Correct: filters in filter (binary-cached after first run)
{
  "query": {
    "bool": {
      "filter": [
        { "range": { "@timestamp": { "gte": "now-1h" } } },
        { "term": { "level": "error" } }
      ],
      "must": [
        { "match": { "message": "connection refused" } }
      ]
    }
  }
}

Deep pagination in Elasticsearch has a separate incompatibility hazard. The from/size approach hits a hard limit at 10,000 results by default (configurable but memory-expensive). The scroll API, used for deep pagination in older versions, was deprecated in 7.10 in favor of search_after combined with Point-in-Time (PIT). A PIT is a lightweight snapshot of index state obtained via POST /index/_pit?keep_alive=5m; pass the PIT ID in each subsequent search_after request. PITs must be explicitly deleted via DELETE /_pit after pagination completes — abandoned PITs accumulate file descriptors and memory. An MCP tool that uses scroll in 2026 is both using a deprecated API and managing a more expensive resource class.

Loki: LogQL stream selectors are mandatory

Loki's LogQL requires every query to begin with a stream selector — a label-value matcher set in curly braces that restricts which log streams are examined. A query string with no stream selector returns a parse error from the API rather than an empty result. This surprises engineers familiar with Elasticsearch full-text search (which scans all indices by default) or SQL (which has implicit FROM clauses).

The GET /loki/api/v1/query_range endpoint returns two incompatible response structures depending on the query type. A log filter query (stream selector plus optional pipeline filter) returns resultType: "streams" — an array of {stream, values} objects where each value is a [nanosecond_timestamp_string, log_line] pair. A metric query using rate(), count_over_time(), or other range aggregations returns resultType: "matrix" — {metric, values} objects with the same shape as Prometheus's range query response. These require different parsing code paths, and the response type is determined entirely by the query string content, not by an API parameter the caller controls.

Timestamps in the Loki push API are Unix nanoseconds expressed as strings. Submitting millisecond timestamps (e.g., "1720000000000") to the push endpoint produces HTTP 204 success responses but silently writes all entries to 1970-01-01, because the millisecond value interpreted as nanoseconds is 0.00000172 seconds past Unix epoch. Push entries must also be in ascending timestamp order per stream — out-of-order entries return HTTP 422. Both constraints produce no error when violated at the wrong precision, making the data corruption silent until you notice that queries return no data for expected recent windows.

InfluxDB: Flux as a functional pipeline

InfluxDB v2 uses Flux, a purpose-built query language that bears no syntactic resemblance to SQL, PromQL, LogQL, or Elasticsearch's Query DSL. Every Flux query is a pipeline of pipe-forward operators starting with a mandatory source and range restriction: from(bucket: "name") |> range(start: -1h). Missing either produces a compile error. The complete query is submitted as a string to POST /api/v2/query with Content-Type application/vnd.flux, and the response is annotated CSV — not JSON — requiring a CSV parser in the MCP tool.

InfluxDB's data model separates tags (indexed, immutable after write, used in stream selectors) from fields (not indexed, mutable, used in aggregation). A filter on a tag is efficient; a filter on a field requires a full measurement scan with no error or warning from the API. An MCP tool that passes a field name where a tag selector is expected produces correct results with much higher latency — and silence from the API about the inefficiency.

Data gap detection uses aggregateWindow(every: 5m, fn: mean, createEmpty: true). The createEmpty: true parameter fills null values (not zero, not missing rows) for windows where no data points arrived, enabling a query to surface gaps in metric collection. The integer field suffix requirement — appending i to integer values in line protocol writes (field1=1i) — is another incompatibility trap: omitting it writes the value as a float, changing the field type, causing subsequent integer writes to fail with a type conflict error.

Dynatrace: metricSelector chain notation

Dynatrace's Metrics API v2 uses a metricSelector chain in the format key:aggregation:transformation:filter — unique to Dynatrace and resembling no other observability query language. Before querying per-entity metrics, an MCP tool must resolve the entity's stable ID (e.g., SERVICE-1234abcd) via the Entities API using an entitySelector expression with operators like type("SERVICE"), tag("env:prod"), or entityId("SERVICE-1234abcd"). Entity IDs persist across renames and configuration changes, making them stable keys for monitoring logic.

The Metrics API's resolution parameter controls data point density. resolution=Inf collapses the entire time range into a single aggregated value per entity — useful for current-state health checks. resolution=5m returns a time series with one data point per 5-minute window. These produce different response shapes, and an MCP tool querying for current state must choose resolution=Inf explicitly rather than receiving an unexpected time series and selecting only the last point.

Alertmanager: matcher operators for alert label sets

Alertmanager's REST API v2 uses label matcher operators for silence creation: = (exact match), != (negative exact), =~ (regex match), !~ (negative regex). These apply to the matchers array in POST /api/v2/silences and match against active alert label sets — suppressing matching alerts from routing. Silences require endsAt, createdBy, and comment fields for audit trail purposes; comment is required even in the open-source version.

The critical incompatibility: Alertmanager's matcher syntax is for alert label filtering only. It has no concept of data retrieval, time-series queries, or aggregation. Inhibition rules — which suppress alert classes based on a source alert firing — are defined in the static Alertmanager configuration file, not via the REST API. An MCP tool cannot create or modify inhibition rules at runtime; it can only manage silences (timed suppressions) and retrieve current alert state.

Pattern 2: The health check deception problem

Every observability data store in this arc presents at least one way to appear healthy at the HTTP layer while being operationally degraded in ways that produce silent data loss, missed notifications, or incorrect monitoring results. This "health check deception" pattern is especially dangerous for observability tools: a degraded Elasticsearch means search results are incomplete; a degraded Loki means log queries return stale data; a degraded Alertmanager means critical alerts are being silently swallowed; a degraded InfluxDB means metric data is corrupted or missing.

Elasticsearch: green cluster with an unsearchable index

Elasticsearch's GET /_cluster/health returns one of three colors aggregated across all indices: green (all shards assigned), yellow (all primaries assigned, some replicas unassigned), red (at least one primary shard unassigned). The aggregate color reflects the worst-performing index — one red index makes the entire cluster red. Knowing the cluster color without knowing which index is the problem leaves you unable to act.

Call GET /_cluster/health?level=indices to get per-index health breakdown in a single request. Then call GET /_cluster/allocation/explain (no body) for a root cause analysis of the first unassigned shard — disk low watermark reached, node not matching allocation filters, corrupt shard data, or insufficient replica count. Two health signals that never change the cluster color but actively degrade service: JVM heap above 75% causes GC pressure visible in query latency spikes (monitor via GET /_nodes/stats/jvm — the heap_used_percent field); disk usage at 85% per node stops new shard allocation on that node silently, without changing cluster health color, causing recovery and reindexing operations to stall.

Loki: query endpoint returns 200 while /ready returns 404

Loki's health architecture separates process liveness from ingester ring membership. The /ready endpoint checks specifically whether the Loki instance's ingester component has joined the gossip ring and is ready to accept log writes. A Loki instance that started but hasn't yet joined the ring — or was evicted due to a heartbeat timeout — returns 404 or 503 from /ready.

The deception: Loki simultaneously returns 404 from /ready and serves query results from /loki/api/v1/query_range. The query endpoint reads from the backend store (S3, GCS, local filesystem) and can serve historical log data from cache without an active ingester ring membership. An MCP tool that probes only the query endpoint gets HTTP 200 and concludes Loki is healthy — while new log writes are either buffering with uncertain durability or failing to reach a ring member for ingestion.

The correct probe for a production Loki instance is /ready, not the query endpoint. An AliveMCP probe on /ready detects ingester ring membership loss before write failures accumulate in application logs. An AliveMCP probe on the query endpoint detects only total process failure — missing the far more common degraded-but-not-dead state where reads work but writes are failing.

Alertmanager: notification failures are invisible to REST

Alertmanager provides two REST API health signals: /-/healthy confirms the process is alive; /api/v2/status returns cluster peer information and active configuration YAML. Neither reveals anything about notification delivery outcomes. Whether PagerDuty received an alert, whether the Slack webhook returned 200, whether the email SMTP relay accepted the message — all of these outcomes are tracked exclusively in Prometheus metrics, exposed via the /metrics endpoint.

The counter alertmanager_notifications_failed_total{integration="pagerduty"} accumulates permanently. An Alertmanager that has been silently dropping all PagerDuty notifications for hours due to an expired API key is indistinguishable from a correctly functioning Alertmanager via /-/healthy or /api/v2/status. The only detection path from the Alertmanager side is to scrape /metrics and alert on rate(alertmanager_notifications_failed_total[5m]) > 0. MCP tools monitoring Alertmanager health via REST API alone miss this failure class entirely.

Alertmanager has no built-in authentication. An unauthenticated port 9093 allows anyone to create silences suppressing all alerts or delete existing silences. Production deployments place Alertmanager behind a reverse proxy that handles authentication. MCP tools should connect to the authenticated proxy endpoint, not directly to port 9093, regardless of the network topology assurances of the deployment environment.

InfluxDB: /health passes while ingestion writes to 1970

InfluxDB v2's GET /health returns {"status":"pass"} as a liveness check. It does not verify that ingestion is landing in the expected buckets, that timestamp precision is correctly configured, or that retention policies are running. The most treacherous failure mode is timestamp precision mismatch in line protocol writes.

The write endpoint at POST /api/v2/write accepts a precision query parameter: ns (nanoseconds), us (microseconds), ms (milliseconds), or s (seconds). If a client sends millisecond timestamps (e.g., 1720000000000) with precision=ns, InfluxDB interprets the value as nanoseconds — placing the data point at 0.00000172 seconds past Unix epoch, January 1, 1970. The write response is HTTP 204 (success). No error. The data is "written" and invisible to any query over a real time range. /health says pass throughout the entire period of data corruption.

The only reliable health check for InfluxDB ingestion is an active canary probe: write a sentinel measurement with a known timestamp and field value, then immediately query it back. If the query returns the sentinel data, ingestion is functioning correctly. If it returns nothing, a precision mismatch, bucket misconfiguration, or ingestion pipeline failure is the likely cause. Register this active canary probe with AliveMCP, not the passive /health endpoint, for production ingestion validation.

Dynatrace: 403 MISSING_PERMISSION is not an auth failure

Dynatrace's Environment API v2 uses scoped API tokens with granular permissions granted per token at creation time. When an API call requires a permission the token doesn't have, the response is HTTP 403 with a body containing "MISSING_PERMISSION" and the specific permission name that was absent (e.g., "metrics.read").

This differs semantically from a standard 403: the token is valid and authenticated, but its scope list doesn't include the required permission. The fix is to add the permission to the token configuration or create a new token with the correct scope — not to retry, rotate credentials, or treat it as a transient auth failure. An MCP tool that catches 403 as a generic auth error and retries indefinitely rather than surfacing the configuration problem will never recover without human intervention.

Active problems on Dynatrace entities are separate from entity availability. A service entity can return entity metadata and current metrics with HTTP 200 while simultaneously having open problems with severityLevel: "AVAILABILITY". The Problems API — GET /api/v2/problems?problemStatus=OPEN — is the correct endpoint for active incident detection. Davis AI, Dynatrace's anomaly detection engine, auto-correlates anomalies into problems with rootCauseEntity, affectedEntities, impactLevel, and severityLevel fields, and auto-resolves them when metrics return to baseline. An MCP tool that monitors Dynatrace for active incidents must poll the Problems API, not the entity or metrics endpoints.

Pattern 3: Null vs zero vs absent in time series

Every time-series observability platform must represent the concept "no data arrived in this time window." The choice of representation — null, zero, a missing data point, or a special sentinel — determines how monitoring logic built on top behaves when data collection gaps occur. Five platforms, five representations, five different semantics. Treating null as zero (or zero as null, or absent as null) in monitoring logic produces silent false positives or false negatives in alert rules, SLO calculations, and capacity planning queries.

Elasticsearch: zero from healthy vs zero from unsearchable

When an Elasticsearch aggregation returns a date_histogram bucket with doc_count: 0, two explanations exist: the healthy case (no documents matching the query filters existed in that time window) and the degraded case (shards for that index are unassigned and the documents that exist cannot be searched — the count is zero not because nothing happened, but because the data is inaccessible).

Both produce identical aggregation output. Distinguishing them requires a separate GET /_cluster/health?level=indices call that includes the queried index. If the index status is red, zero counts in that time window are data access failures, not quiet periods. For production MCP tools that use Elasticsearch aggregations to derive error rates, request counts, or latency percentiles, the index health check must precede result interpretation.

The date_histogram aggregation with extended_bounds and min_doc_count: 0 fills empty buckets with zero — convenient for time-series plotting, but it makes "empty because quiet" and "empty because degraded" produce the same output for any downstream logic that doesn't also check shard health.

Loki: null means no log lines matched in window

Loki's metric query functions — rate(), count_over_time(), bytes_rate() — return time-series data over the requested interval. When createEmpty: true is included in the query, time buckets where no log lines matched the stream selector are filled with null values rather than being omitted from the response entirely. Null here means "no matching log lines arrived in this time window" — which has two operationally distinct causes.

The first is a genuinely quiet window: the application logged nothing matching the stream selector in that period. A healthy service with no errors will have rate({job="api", level="error"}[5m]) returning null during quiet periods — this is correct and expected. The second is a data collection gap: the log shipper (Promtail, Alloy, or a custom agent) went down, and no log lines were pushed to Loki in that window even while application activity continued. The null looks identical in both cases.

Distinguishing these requires out-of-band knowledge about whether application traffic was occurring, or a secondary probe checking the log shipper's health. Loki cannot distinguish them — it knows only that no matching log lines arrived, not whether that was because none were generated or none were shipped. The /ready endpoint is a partial signal: if Loki was not ready during that window, ingestion may have been impaired — but a healthy /ready during a null window does not prove the shipper was functioning correctly.

InfluxDB: null means scrape target went silent

InfluxDB's Flux aggregateWindow() with createEmpty: true generates null values for windows where no data points were written to the bucket for the queried measurement and tag set. The null specifically means "the source that generates this measurement did not produce any data points in this window" — a gap in the source's reporting, not a genuine zero measurement.

For Prometheus-compatible metrics forwarded to InfluxDB, a null in a 5-minute aggregation window means the Prometheus scrape target went silent — either the target stopped responding or the scrape itself failed. A zero means the target actively reported and reported zero: zero active connections, zero error rate. These are categorically different operational states. An alerting rule built on top that fires when a value exceeds a threshold must explicitly handle null: in Flux, a null comparison (null > 100.0) returns null, not false — which means an alert condition that doesn't guard against null may fail to fire when expected. The fill(usePrevious: true) Flux function can forward-fill nulls with the last known value — appropriate for slowly-changing gauges but incorrect for fast-changing rates where a null represents actual data absence.

Dynatrace: null means OneAgent connectivity lost

Dynatrace metric API responses contain null values when the OneAgent monitoring a host or service has lost network connectivity to the Dynatrace cluster. The OneAgent runs inside the monitored environment and pushes measurements continuously. When it loses connectivity, metric collection stops and the cluster records null values for that entity in the time series — not zero, because zero would be a valid measurement (zero CPU usage, zero error rate). A null is an explicit representation of a connectivity gap.

Davis AI, Dynatrace's anomaly detection engine, understands null semantics and suppresses anomaly detection during entity communication loss — avoiding false alerts when metrics disappear due to infrastructure events. Custom monitoring logic in an MCP tool must handle null explicitly rather than treating it as zero. A metric that normally reports 50ms latency showing null values during an outage is not the same as reporting zero latency — an MCP tool that substitutes zero for null will fire false "latency is zero" health alerts during an actual outage.

The entity-level signal for connectivity loss is available separately via the Entities API. An entity that has lost OneAgent communication receives a monitoring state update detectable via GET /api/v2/entities/{entityId}. Combining metric null detection with entity monitoring state provides a fuller picture: null metric with degraded monitoring state = OneAgent connectivity loss; null metric with healthy monitoring state = genuine period of zero activity on a correctly monitored entity.

Alertmanager: counters are never null — zero means healthy

Alertmanager is event-driven rather than time-series. Its Prometheus metrics endpoint exposes counters (alertmanager_notifications_total, alertmanager_notifications_failed_total, keyed by integration name) that are cumulative and never null once the endpoint has been scraped. A counter at zero is genuinely zero — Alertmanager has successfully delivered all notifications since startup (or there have been no notifications). A counter absent from the scrape output means the Prometheus scrape of Alertmanager's /metrics endpoint has failed — a monitoring infrastructure failure separate from Alertmanager's own operation.

The correct check for recent notification delivery health is rate(alertmanager_notifications_failed_total[5m]) > 0 — a positive rate of failure in the recent window indicates active delivery problems. A zero rate means no recent failures. An absent metric means the scrape failed and delivery health is unknown. These three states map cleanly to healthy, degraded, and unknown — which is the correct trichotomy for any monitoring check.

Cross-tool comparison: five observability data stores

Composite health probe for a multi-platform observability stack

A production observability stack commonly combines several of these platforms: Elasticsearch for search and log analytics, Loki for log aggregation, Alertmanager for alert routing, InfluxDB for Prometheus-compatible metrics, and Dynatrace for APM. An MCP tool that monitors the monitoring stack itself requires platform-specific probe strategies for each tier.

// Composite health probe for an observability data store stack
async function checkObservabilityStack(clients) {
  const results = await Promise.allSettled([

    // Elasticsearch: per-index health, not just cluster color
    clients.elasticsearch
      .request('GET', '/_cluster/health', { level: 'indices' })
      .then(r => {
        const redIndexes = Object.entries(r.indices ?? {})
          .filter(([, idx]) => idx.status === 'red')
          .map(([name]) => name);
        return {
          platform: 'elasticsearch',
          healthy: redIndexes.length === 0 && r.status !== 'red',
          detail: redIndexes.length > 0
            ? `red indexes: ${redIndexes.join(', ')}`
            : r.status,
        };
      }),

    // Loki: probe /ready (not the query endpoint)
    clients.loki
      .request('GET', '/ready')
      .then(r => ({
        platform: 'loki',
        healthy: r.statusCode === 200,
        detail: r.statusCode === 200
          ? 'ingester in ring'
          : `ingester not in ring (${r.statusCode})`,
      })),

    // Alertmanager: liveness + delivery failure rate
    Promise.all([
      clients.alertmanager.request('GET', '/-/healthy'),
      clients.alertmanager
        .request('GET', '/metrics', { format: 'text' })
        .then(parseNotificationFailureRate),
    ]).then(([liveness, failureRate]) => ({
      platform: 'alertmanager',
      healthy: liveness.statusCode === 200 && failureRate === 0,
      detail: failureRate > 0
        ? `delivery failures: ${failureRate}/min`
        : 'no delivery failures',
    })),

    // InfluxDB: active canary write+read
    (async () => {
      const ts = Date.now() * 1_000_000; // ms → ns
      const measurement = `_alivemcp_canary`;
      await clients.influxdb.write(
        `${measurement} value=1i ${ts}`,
        { precision: 'ns' }
      );
      // Small delay to allow ingestion flush
      await new Promise(r => setTimeout(r, 500));
      const result = await clients.influxdb.query(
        `from(bucket:"monitoring")|>range(start:-10s)|>filter(fn:(r)=>r._measurement=="${measurement}")`
      );
      const ok = result.some(row => row._value === 1);
      return {
        platform: 'influxdb',
        healthy: ok,
        detail: ok
          ? 'canary round-trip succeeded'
          : 'canary write not readable — check precision/bucket config',
      };
    })(),

    // Dynatrace: check for open problems + token scope validation
    clients.dynatrace
      .request('GET', '/api/v2/problems', { problemStatus: 'OPEN', pageSize: 5 })
      .then(r => ({
        platform: 'dynatrace',
        healthy: r.totalCount === 0,
        detail: r.totalCount === 0
          ? 'no open problems'
          : `${r.totalCount} open problem(s) — first: ${r.problems?.[0]?.displayId}`,
      }))
      .catch(e => ({
        platform: 'dynatrace',
        healthy: false,
        detail: e.code === 403
          ? `token missing scope: ${e.missingPermission}`
          : String(e),
      })),
  ]);

  return results.map(r =>
    r.status === 'fulfilled'
      ? r.value
      : { platform: 'unknown', healthy: false, detail: String(r.reason) }
  );
}

Three principles from these failure modes drive the composite probe design. First, probe the signal that reveals degraded-but-not-dead states: /ready for Loki rather than the query endpoint; level=indices for Elasticsearch; /metrics delivery rate for Alertmanager in addition to /-/healthy. Second, active canary probes beat passive liveness checks for storage systems: InfluxDB's canary write+read catches precision mismatch, bucket misconfiguration, and ingestion pipeline failures that /health misses entirely. Third, distinguish null from zero in all monitoring logic built on top: all four time-series data stores in this arc (Elasticsearch, Loki, InfluxDB, Dynatrace) use null with semantics distinct from zero, and confusing them produces silent false positives or negatives in derived alerting rules and SLO calculations.

Register each platform-specific probe target with AliveMCP at the appropriate poll interval — 60 seconds for storage liveness and delivery health, with alerting on sustained degradation. AliveMCP probes the URLs you specify and alerts the moment a platform enters the health check deception state — catching Loki ring membership loss, Elasticsearch shard stalls, Alertmanager delivery failures, and InfluxDB ingestion corruption before they produce blind spots in the monitoring coverage that was supposed to catch production incidents.

Further reading

• MCP Tools for Elasticsearch — search queries, cluster health, and index management
• MCP Tools for Grafana Loki — LogQL queries, log push, and label streaming
• MCP Tools for Prometheus Alertmanager — silences, inhibitions, and notification health
• MCP Tools for InfluxDB v2 — Flux queries, line protocol writes, and bucket management
• MCP Tools for Dynatrace — entity model, metrics API, and problems integration
• MCP Tools for Workflow Orchestration: State Machines, Engine Health vs Execution Progress, and Completion Polling
• MCP Tools for Message Queues: Consumer Lag Signals, Dead-Letter Verification, and At-Least-Once Delivery
• MCP Server Uptime Monitoring — probe design and health endpoint conventions
• MCP Server Health Check — patterns for composite health endpoints