MCP Tools for Workflow Orchestration: State Machines, Engine Health vs Execution Progress, and Completion Polling Across Temporal, Airflow, Step Functions, Prefect, and Dagster

Pattern 1: State machine diversity hides stuck semi-terminal states

Every workflow orchestrator has at least one state that looks active — polling returns a non-terminal status — but is actually frozen, waiting for something external that may never arrive. The instinct when building an MCP tool is to poll until you see a "finished" state and then report the result. This breaks predictably when the execution is stuck in a non-terminal limbo state that your terminal set doesn't include.

The five orchestrators each have different stuck states with different mechanisms.

Temporal: CONTINUED_AS_NEW and the stale run ID trap

Temporal's six-value execution status enum includes CONTINUED_AS_NEW — a status that no other major orchestrator shares and that surprises MCP tool authors who assume a non-RUNNING status means the workflow is done. When a long-running workflow calls workflow.continueAsNew() to reset its event history before hitting Temporal's 50,000-event hard limit, the current execution enters CONTINUED_AS_NEW state and a new execution starts with the same Workflow ID but a new Run ID. From the perspective of code holding a WorkflowHandle pinned to the original Run ID, the execution is "finished" with status CONTINUED_AS_NEW. But the workflow has not finished — it is actively running under a new Run ID.

The correct pattern is to always get a workflow handle using only the Workflow ID (not the Run ID) when you want the "current" execution: client.workflow.getHandle(workflowId) without specifying a Run ID. This resolves to the latest execution automatically. If you pinned the Run ID at handle creation time (because you started the workflow and saved the Run ID from the start response), you must be prepared to re-fetch the latest run when you encounter CONTINUED_AS_NEW status. Treating CONTINUED_AS_NEW as a terminal state in your poll loop will cause your MCP tool to incorrectly report a completion on an actively-running workflow.

Prefect: CANCELLING requires a live worker

In Prefect, cancellation is not instantaneous. When you request cancellation via PATCH /api/flow_runs/{id} or via the API, Prefect sets the flow run to CANCELLING state and sends a signal to the worker process handling the run. The worker must receive this signal, interrupt the flow function, and call back to the Prefect server to set the final CANCELLED state. If the worker that was running the flow has since crashed, restarted, or been replaced by a different worker that doesn't know about this cancellation request, the flow run sits in CANCELLING forever — it is not terminal, but it will never complete the transition on its own.

MCP tools that cancel Prefect flow runs must account for this. After requesting cancellation, poll for up to a reasonable grace period (5 minutes is typical) and if the state is still CANCELLING, force-cancel via POST /api/flow_runs/{id}/set_state with the body {"type": "CANCELLED", "force": true}. The force: true flag bypasses the worker handshake and sets the state directly. Include a stuck-CANCELLING detector in any monitoring MCP tool: if a flow run has been in CANCELLING state for more than your grace period threshold, it requires intervention.

// Detect flow runs stuck in CANCELLING beyond threshold
async function findStuckCancellingRuns(client, thresholdMinutes = 10) {
  // POST /api/flow_runs/filter with state type filter
  const runs = await client.request('POST', '/flow_runs/filter', {
    flow_runs: {
      state: { type: { any_: ['CANCELLING'] } },
    },
  });

  const thresholdMs = thresholdMinutes * 60_000;
  return runs.filter((run) => {
    const stateTimestamp = run.state?.timestamp;
    return stateTimestamp
      && Date.now() - new Date(stateTimestamp).getTime() > thresholdMs;
  });
}

// Force-cancel a run stuck in CANCELLING
async function forceCancelRun(client, flowRunId) {
  return client.request('POST', `/flow_runs/${flowRunId}/set_state`, {
    state: { type: 'CANCELLED', name: 'Cancelled', message: 'Force-cancelled by MCP tool after CANCELLING timeout' },
    force: true,
  });
}

Step Functions: waitForTaskToken pauses until the callback arrives

AWS Step Functions' waitForTaskToken integration pattern pauses a state machine execution until an external system calls SendTaskSuccess or SendTaskFailure with the task token. From the perspective of DescribeExecution, the execution is RUNNING — which it legitimately is, waiting for the callback. But if the external system that was supposed to call back has died, the task token has expired (via the state's HeartbeatSeconds setting), or the token was lost (e.g., delivered to a dead SQS consumer that never processed it), the execution will remain RUNNING indefinitely until the TimeoutSeconds for the entire state machine execution expires.

There is no Step Functions API call that tells you "this execution is waiting on a waitForTaskToken" without reading the execution history. The symptom is a RUNNING execution that has been RUNNING for longer than the expected completion time. To detect these, combine ListExecutions filtered to RUNNING with an age threshold, then call GetExecutionHistory (STANDARD only — EXPRESS executions don't store history in Step Functions) with reverseOrder: true and look for a TaskStateEntered event without a subsequent TaskStateExited, which indicates the task is still waiting.

Airflow: up_for_reschedule and sensor timeout

Apache Airflow's up_for_reschedule task state is exclusive to sensors. A sensor that hasn't found its target condition yet returns up_for_reschedule between poke intervals — it's not running continuously, it's sleeping and will wake up for the next poke. This looks like a semi-stuck state from the outside: the TaskInstance alternates between running and up_for_reschedule on each poke cycle. If the sensor's poke condition never becomes true — because the source system is down, the expected file never arrives, or the upstream data never lands — the TaskInstance will cycle through up_for_reschedule indefinitely as long as the sensor's timeout parameter hasn't expired.

An MCP tool checking for stuck Airflow runs should treat TaskInstances in up_for_reschedule for longer than their configured timeout plus a safety margin as candidates for investigation. The try_number field on the TaskInstance advances on each poke, so a sensor that has been poking for 500+ times without finding its condition is almost certainly stuck on a broken upstream dependency rather than just slow.

Dagster: CANCELING requires the code server

Dagster's CANCELING state (note: single-L, unlike Prefect's double-L CANCELLING) behaves similarly to Prefect's CANCELLING: it requires the code server to coordinate the actual cancellation. If the code server is offline, the run stays in CANCELING indefinitely. The instance.daemonHealth GraphQL query can tell you if the relevant daemons are online, but daemon health doesn't directly tell you whether the code server that was running a specific job is responsive. Monitor for runs in CANCELING state beyond your threshold and escalate via the Dagster UI's force-terminate option or by directly manipulating run state via the GraphQL mutation terminateRun with the terminatePolicy: MARK_AS_CANCELED_IMMEDIATELY argument.

Pattern 2: Engine health does not equal execution progress

The second pattern is more subtle and more dangerous: every orchestrator separates the health of the API-serving layer (which tells you the system is alive) from the health of the execution engine (which tells you workflows are actually running). An MCP tool that checks only the API endpoint will report a healthy orchestrator when the system is silently failing to run any new workflows.

This architectural split exists in all five orchestrators but manifests differently in each.

Temporal: gRPC reachable ≠ workers polling

The Temporal Server handles workflow state persistence, event history, and the Visibility index — but it does not execute workflow or activity code. That execution happens in worker processes that poll task queues. A Temporal Server that passes a gRPC health check (getSystemInfo() returns successfully) but has zero workers polling its task queues will accept every API call, store every new workflow execution, and return RUNNING status on them — while making zero progress on any of them. Workflows are submitted and sit frozen indefinitely.

The correct health signal for execution progress is task queue poller count via describeTaskQueue(). Zero pollers for a task queue means zero execution capacity on that queue. Alert on this condition independently of server reachability. A Temporal deployment can have multiple task queues (one per workflow or activity type), so you need to check pollers for each queue that has active workloads. Register both signals with AliveMCP: one endpoint for server reachability and one per task queue for poller presence.

async function checkTemporalHealth(client, namespace, taskQueues) {
  const result = { serverReachable: false, queues: {} };

  try {
    await client.workflowService.getSystemInfo({});
    result.serverReachable = true;
  } catch {
    return result; // Server down — stop here
  }

  for (const queueName of taskQueues) {
    const tq = await client.workflowService.describeTaskQueue({
      namespace,
      taskQueue: { name: queueName },
      taskQueueType: 1, // WORKFLOW type
    });
    result.queues[queueName] = {
      pollerCount: tq.pollers?.length ?? 0,
      healthy: (tq.pollers?.length ?? 0) > 0,
    };
  }

  return result;
}

Airflow: webserver health ≠ scheduler health

Apache Airflow runs the webserver (REST API + UI) and the scheduler (task slot assignment, DagRun creation, dependency resolution) as separate processes. The GET /api/v1/health endpoint (no authentication required) returns both metadatabase.status and scheduler.status plus the scheduler.latest_scheduler_heartbeat timestamp. This heartbeat is the key signal: it is the last time the scheduler wrote a "I'm alive" record to the metadata database. If this timestamp is more than 30–60 seconds old, the scheduler process has crashed or become unresponsive — the webserver may be serving API calls normally while zero new task slots are being created and no scheduled DagRuns are being triggered.

Build your Airflow health MCP tool around heartbeat freshness, not HTTP status. A stale heartbeat with a 200 OK from the webserver is a more serious incident than a 503 from the webserver (which is obvious) because the stale heartbeat is invisible to anything that only checks the HTTP layer. AliveMCP's scheduler-heartbeat probe endpoint should alert if Date.now() - new Date(heartbeat).getTime() > 60_000.

Dagster: webserver health ≠ daemon health

Dagster has the most granular daemon health model of the five orchestrators. The instance.daemonHealth.allDaemonStatuses GraphQL field returns the health of each daemon type independently:

• SCHEDULER daemon: creates runs for scheduled jobs. If dead, no scheduled runs trigger.
• SENSOR daemon: polls sensors and creates runs when sensor conditions are met. If dead, no sensor-driven runs trigger.
• BACKFILL daemon: manages partition backfills. If dead, in-progress backfills freeze.
• AUTO_MATERIALIZE daemon: drives the auto-materialization policy engine. If dead, no automatic asset materializations trigger.

Each daemon reports a healthy boolean and a lastHeartbeatTime. The webserver can be serving GraphQL at HTTP 200 while all four daemons are offline. An MCP tool that only checks the GraphQL endpoint health will miss this entirely. Include daemon heartbeat freshness in your Dagster health probe and register each daemon as a separate named endpoint in AliveMCP — a dead sensor daemon is a different alerting context from a dead scheduler daemon.

const DAEMON_HEALTH_QUERY = `
  query DaemonHealth {
    instance {
      daemonHealth {
        allDaemonStatuses {
          daemonType
          healthy
          lastHeartbeatTime
          lastHeartbeatErrors { message }
        }
      }
    }
  }
`;

async function checkDagsterHealth(client) {
  const data = await client.query(DAEMON_HEALTH_QUERY);
  const daemons = data.instance.daemonHealth.allDaemonStatuses;

  return daemons.map((d) => ({
    type: d.daemonType,
    healthy: d.healthy,
    lastHeartbeat: d.lastHeartbeatTime
      ? new Date(d.lastHeartbeatTime * 1000)
      : null,
    heartbeatAgeMs: d.lastHeartbeatTime
      ? Date.now() - d.lastHeartbeatTime * 1000
      : Infinity,
    errors: d.lastHeartbeatErrors?.map((e) => e.message) ?? [],
  }));
}

Prefect: work pool paused + worker heartbeat = two independent signals

Prefect's execution capacity depends on two independent conditions: the work pool must not be paused (is_paused: false), and at least one worker must be actively polling that work pool. A paused work pool will not send work to any workers even if workers are online. An active work pool with no live workers will queue runs in SCHEDULED state indefinitely with no start.

Check both signals: GET /api/work_pools/{name} for the is_paused flag, and then check worker heartbeat age. Workers that haven't sent a heartbeat in more than 120 seconds are considered offline by Prefect. You can check this via GET /api/work_pools/{name}/workers (or via filtering) and examining last_heartbeat_time on each worker. Zero workers with a recent heartbeat on any work pool that has queued runs is the actionable signal.

Step Functions: no scheduler to check — but EXPRESS logging matters

Step Functions is a managed AWS service — there is no "scheduler process" to monitor. But there is a critical configuration health check that your MCP tool should include: for EXPRESS state machines, execution events are only available in CloudWatch Logs if loggingConfiguration was set on the state machine with level: ALL or level: ERROR. If EXPRESS logging is not configured and an EXPRESS execution fails, there is no record of what went wrong — GetExecutionHistory throws ExecutionDoesNotExist, and CloudWatch Logs has nothing either.

Include a configuration probe for any Step Functions MCP tool: call DescribeStateMachine and check type === 'EXPRESS' plus loggingConfiguration.destinations.length > 0. Report a configuration warning if an EXPRESS state machine has no logging configured.

Pattern 3: Completion polling requires orchestrator-specific terminal sets

All five orchestrators require polling for completion — none provide native webhook callbacks for "my workflow finished" events in their primary API. But each has a different definition of "terminal", and treating any "not currently active" state as terminal will cause your MCP tool to exit its poll loop prematurely on states that are not, in fact, done.

The correct terminal state sets are:

• Temporal: COMPLETED, FAILED, CANCELED, TERMINATED. Not CONTINUED_AS_NEW (active workflow, new run ID) and not TIMED_OUT in older API versions (check SDK version).
• Airflow DagRun: success and failed. Not queued (waiting for scheduler pickup) and not dataset_triggered in some versions.
• Step Functions: SUCCEEDED, FAILED, TIMED_OUT, ABORTED. All four are terminal; none require additional checks.
• Prefect (state type): COMPLETED, FAILED, CRASHED, CANCELLED. Not CANCELLING (stuck semi-terminal) and not PAUSED (waiting for resume signal).
• Dagster: SUCCESS, FAILURE, CANCELED. Not CANCELING (in-progress cancellation).

The Crashed vs Failed distinction in Prefect also matters for error reporting. CRASHED means the worker process died — an infrastructure event (OOM kill, node failure, signal) — not a code defect. FAILED means the flow function raised an exception. These have different remediation paths: CRASHED suggests investigating the worker's infrastructure (memory pressure, node health), while FAILED suggests investigating the flow code and its inputs. An MCP tool should surface this distinction rather than treating both as "failed".

In Dagster, the equivalent distinction is between a step's STEP_FAILURE event (the step code raised an exception — check error.className and error.message) and a PIPELINE_FAILURE event (the run-level coordination failed, often a code server issue). Filter for these event types separately via the logsForRun GraphQL query with ofTypes: ['STEP_FAILURE', 'PIPELINE_FAILURE'] rather than scanning all log events for error text.

// Universal completion poll pattern — fills in terminal sets per orchestrator
const TERMINAL = {
  temporal: new Set(['COMPLETED', 'FAILED', 'CANCELED', 'TERMINATED']),
  airflow:  new Set(['success', 'failed']),
  sfn:      new Set(['SUCCEEDED', 'FAILED', 'TIMED_OUT', 'ABORTED']),
  prefect:  new Set(['COMPLETED', 'FAILED', 'CRASHED', 'CANCELLED']), // state *types*
  dagster:  new Set(['SUCCESS', 'FAILURE', 'CANCELED']),
};

// Airflow example — illustrates the pattern
async function pollAirflowDagRun(client, dagId, dagRunId, pollMs = 10_000, timeoutMs = 3_600_000) {
  const terminal = TERMINAL.airflow;
  const deadline = Date.now() + timeoutMs;

  while (Date.now() < deadline) {
    const run = await client.request('GET', `/dags/${dagId}/dagRuns/${dagRunId}`);
    if (terminal.has(run.state)) {
      return {
        state: run.state,
        success: run.state === 'success',
        endDate: run.end_date,
        durationSec: run.end_date
          ? (new Date(run.end_date) - new Date(run.start_date)) / 1000
          : null,
      };
    }
    await new Promise((r) => setTimeout(r, pollMs));
  }
  throw new Error(`DagRun ${dagRunId} timed out after ${timeoutMs}ms`);
}

One structural pattern unifies the poll target across all five orchestrators: each has a "describe this specific execution" call that returns current state without side effects. For Temporal it's handle.describe(), for Airflow it's GET /api/v1/dags/{dag_id}/dagRuns/{dag_run_id}, for Step Functions it's DescribeExecution, for Prefect it's GET /api/flow_runs/{id}, and for Dagster it's the runOrError GraphQL query. All five are safe to call in a tight polling loop (every 5–30 seconds depending on expected run duration). None of them modify state — they are read-only status snapshots.

Cross-tool comparison: five orchestrators at a glance

Registering orchestrator health with AliveMCP

The three patterns above converge on a single operational conclusion: health check design for workflow orchestrators requires multiple independent probes, not a single HTTP endpoint ping. An MCP tool that registers all three health layers with AliveMCP gives you alerting coverage for the full failure mode surface.

For each orchestrator, implement three probe tiers:

1. API reachability probe (30–60 second interval): can you reach the API at all? For Temporal, getSystemInfo(). For Airflow, GET /api/v1/health. For Step Functions, ListStateMachines with a 1-item limit. For Prefect, GET /api/health. For Dagster, a lightweight GraphQL query like { version }.
2. Engine health probe (60-second interval): is the execution engine running? For Temporal, non-zero pollers on each active task queue. For Airflow, scheduler heartbeat age <60 seconds. For Prefect, work pool not paused and at least one live worker. For Dagster, SCHEDULER and SENSOR daemon heartbeat freshness. Step Functions has no engine to probe — skip this tier.
3. Execution progress probe (5-minute interval): are workflows completing within expected SLA? Count stuck RUNNING executions older than threshold. Count failed DagRuns in the last hour vs historical baseline. Count Prefect CRASHED runs (infrastructure-failure signal distinct from FAILED). Track Dagster asset freshness staleness for assets with materialization SLA policies.

The second tier is the one most MCP tools skip because it requires orchestrator-specific knowledge. It is also the tier that catches the highest-impact incidents: a dead Airflow scheduler or a Temporal namespace with no workers will run for hours in this degraded state before any workflow SLA violations become visible at tier 3. Tier 2 probes catch these silently-degraded states at 60-second granularity.

This same monitoring architecture applies to other external integrations your MCP server depends on. See our guides on MCP server error handling, message queue health patterns, and DevOps tooling integration patterns for the equivalent analysis applied to message brokers and CI/CD pipelines.

Related guides

• MCP tools for Temporal — workflow execution queries, signals, and namespace health
• MCP tools for Apache Airflow — DAG run states, task instance hierarchy, and REST API v1
• MCP tools for AWS Step Functions — STANDARD vs EXPRESS, execution history, and waitForTaskToken
• MCP tools for Prefect — flow run states, work pool health, and deployment triggers
• MCP tools for Dagster — asset materializations, run status, and GraphQL API
• MCP server health check patterns — composite endpoints and failure classification
• MCP server error handling — retries, timeouts, and structured error responses
• MCP Tools for Message Queues — Visibility Timeout, Consumer Lag, and Dead Letter Queue Patterns
• MCP Tools for Security Scanning — Severity Models, Null-Result Traps, and Subprocess Integration
• MCP Tools for DevOps Tooling — CI/CD Pipelines, Container Registries, and Infrastructure
• MCP server uptime monitoring patterns