Guide · OpenSearch · Index Lifecycle

OpenSearch Index State Management for MCP Probe Data

OpenSearch Index State Management (ISM) automates the lifecycle of time-series probe data — rolling over when an index gets too large, moving old shards to warm nodes, merging segments to reclaim storage, and deleting data after the retention window closes. Three ISM pitfalls catch teams building MCP monitoring pipelines: rollover only triggers on the write alias, not the index — writing directly to the index name bypasses rollover entirely; the warm allocation action sets index.routing.allocation.require.box_type: warm, but if your cluster has no warm-tier nodes, the index enters a stuck INITIALIZING shard state; and forced merge in the ISM policy runs as a blocking operation — scheduling it during peak query hours increases p99 latency by 3–5× while the merge completes.

TL;DR

Write to a rollover alias, not the index directly. Define transitions with at least one condition (min_doc_count, min_size, or min_index_age). Move to warm only if you have warm-tier nodes (check box_type attribute). Schedule forced merge in ISM to run outside query-peak windows. Use ISM template to auto-attach policy to all indices matching a pattern — attach policy once, it applies to new rollover indices automatically.

ISM policy lifecycle: states and transitions

An ISM policy is a state machine applied to an index. Each state has actions (work to do while in that state) and transitions (conditions that move the index to the next state). The ISM worker checks each index's state every 5 minutes by default.

// ISM policy for MCP probe data — hot → warm → cold → delete
const policy = {
  policy: {
    description: 'AliveMCP probe results: 7d hot, 30d warm, 60d cold, delete at 90d',
    default_state: 'hot',
    states: [
      {
        name: 'hot',
        actions: [
          {
            rollover: {
              min_doc_count: 50_000_000,   // 50M probe results per index
              min_size: '50gb',            // OR 50 GB of storage
              min_index_age: '7d',         // OR 7 days old — whichever comes first
            },
          },
        ],
        transitions: [
          {
            state_name: 'warm',
            conditions: {
              min_rollover_age: '1d',       // Move to warm 1 day after rollover
            },
          },
        ],
      },
      {
        name: 'warm',
        actions: [
          {
            // Reduce replicas on warm indices to cut storage cost
            replica_count: { number_of_replicas: 0 },
          },
          {
            // Merge all segments into 1 — reduces storage 20-40%, improves search speed
            // Runs once on transition into this state
            force_merge: { max_num_segments: 1 },
          },
          {
            // Move shards to warm-tier nodes (requires box_type=warm node attribute)
            allocation: {
              require: { box_type: 'warm' },
            },
          },
        ],
        transitions: [
          {
            state_name: 'cold',
            conditions: {
              min_index_age: '30d',
            },
          },
        ],
      },
      {
        name: 'cold',
        actions: [
          {
            // Make index read-only to prevent accidental writes
            read_only: {},
          },
          {
            // Further reduce to cold-tier nodes
            allocation: {
              require: { box_type: 'cold' },
            },
          },
        ],
        transitions: [
          {
            state_name: 'delete',
            conditions: {
              min_index_age: '90d',
            },
          },
        ],
      },
      {
        name: 'delete',
        actions: [
          { delete: {} },
        ],
        transitions: [],  // Terminal state
      },
    ],
    ism_template: [
      {
        index_patterns: ['probe-results-*'],
        priority: 100,  // Higher priority wins if multiple templates match
      },
    ],
  },
};

await client.transport.request({
  method: 'PUT',
  path: '/_plugins/_ism/policies/probe-results-lifecycle',
  body: policy,
});

Rollover alias pattern

ISM rollover only triggers when writes go through the alias — not the backing index. The alias must have "is_write_index": true on exactly one index. After rollover, ISM automatically updates the alias to point to the new index and sets the old index's is_write_index to false.

// Step 1: Create the initial backing index with the rollover alias
await client.indices.create({
  index: 'probe-results-000001',  // Starting index — number suffix required for rollover
  body: {
    aliases: {
      'probe-results-write': {
        is_write_index: true,    // All writes go to this index via the alias
      },
      'probe-results-read': {},  // Read queries go to all indices matching this alias
    },
    settings: {
      'index.plugins.index_state_management.policy_id': 'probe-results-lifecycle',
    },
  },
});

// Step 2: Write probe results via the ALIAS, not the index name
// Correct — triggers rollover
await client.index({
  index: 'probe-results-write',  // Write alias
  body: { server_slug: 'my-mcp-server', status: 'live', latency_ms: 42, checked_at: new Date() },
});

// Wrong — bypasses rollover, ISM never triggers
await client.index({
  index: 'probe-results-000001',  // Direct index write — rollover never fires
  body: probeResult,
});

After rollover, the backing index sequence is probe-results-000001, probe-results-000002, etc. The probe-results-read alias automatically spans all backing indices — queries that need all historical data use this alias; queries against today's data can filter on checked_at >= now-1d.

ISM template vs explicit policy attachment

ISM templates auto-attach a policy to any new index whose name matches an index pattern. Without a template, new rollover indices (e.g., probe-results-000002 through probe-results-000099) would be created without a policy attached and accumulate unbounded.

Approach	How it works	When to use
ISM template (in policy)	Policy includes `ism_template` array with index pattern; applied to all matching new indices	Standard — set once, works for all future rollover indices
Explicit attach via API	`POST /_plugins/_ism/add/{index}` with `policy_id`	For existing indices, one-off backfill, or overriding template
Index template settings	Set `index.plugins.index_state_management.policy_id` in the index template's settings section	When the policy ID must be embedded in the index template, not the ISM policy itself

Use the ism_template approach for rolling probe-data indices — it requires zero maintenance as new backing indices are created by rollover. Verify it's working by checking that new indices show "index.plugins.index_state_management.policy_id" in their settings after creation.

Forced merge scheduling and I/O impact

Forced merge merges all Lucene segments in a shard into a single segment. For a 50 GB warm index with 200 segments, this operation takes 10–40 minutes and saturates disk I/O throughout. Running forced merge while the index is still receiving queries causes significant latency spikes because Lucene segment files are being rewritten while the query path reads them.

Best practice: schedule the warm state to run its force_merge action only after the index has transitioned away from the write alias (guaranteed by the min_rollover_age: "1d" transition condition) and during off-peak hours. ISM actions within a state run sequentially — the replica_count reduction and allocation actions will execute before force_merge if listed first. Because the ISM worker only runs every 5 minutes, the window for forced merge to run at an unfortunate time is small but non-zero; for production environments with strict p99 SLAs, consider splitting the warm state into two: warm-reduce-replicas (immediate) and warm-merge (after a 2h buffer).

// Check ISM execution history for a specific index
const history = await client.transport.request({
  method: 'GET',
  path: `/_plugins/_ism/explain/probe-results-000001`,
});

// Key fields in the response:
// index.index: index name
// index.index_uuid: UUID for cross-check
// index.policy_id: attached policy
// index.state.name: current ISM state (hot, warm, cold, delete)
// index.state.action.name: current action running
// index.failed_steps: list of failed ISM steps (check reason)
// index.info.message: last action message
// index.info.cause: error cause if any

console.log(JSON.stringify(history.body.probe-results-000001, null, 2));

Hot-warm-cold cluster node attributes

The allocation ISM action uses index.routing.allocation.require.box_type to move shards to nodes tagged with matching node attributes. Without these attributes set on at least one cluster node, the shard reallocation step will remain in INITIALIZING indefinitely — the index stays in the warm ISM state but shards never actually move, and the force_merge action behind it never runs.

# Set node attributes in opensearch.yml for each node tier
# Hot data nodes (SSD-backed, high IOPS)
node.attr.box_type: hot

# Warm data nodes (HDD-backed, lower IOPS, larger capacity)
node.attr.box_type: warm

# Cold data nodes (lowest cost, object storage or HDD)
node.attr.box_type: cold

# Verify node attributes are set
# GET /_cat/nodeattrs?v&h=node,attr,value

In Amazon OpenSearch Service, UltraWarm nodes serve as the warm tier — they store index data on S3 backed by an S3-based cache layer (not on instance storage). UltraWarm nodes appear as ultrawarm data nodes in the cluster. UltraWarm shards are read-only by design, which aligns with the ISM warm state pattern. UltraWarm costs roughly 90% less per GB than hot nodes but adds 10–50ms query latency versus SSD-backed hot nodes.

How AliveMCP uses ISM for probe retention

AliveMCP writes 60-second probe results to the probe-results-write alias at approximately 83 documents per second for 300,000 monitored endpoints. At that rate, a single index reaches 50 GB in about 7 days — matching the ISM rollover condition. The 90-day delete policy means AliveMCP retains roughly 13 rolling indices at any time, totalling ~650 GB of hot data.

Subscribers on the Team and Enterprise plans get 90-day response-time history on their status pages — data served entirely from the OpenSearch warm tier (indices 8–13, age 7–90 days) at a fraction of the hot-tier cost. The forced merge on warm transition reduces query time for the 90-day histogram endpoint from ~800ms (200 segments) to ~120ms (1 merged segment), making the response-time chart render under 200ms for p95 of requests.