AliveMCP

Guide · GitOps & Kubernetes

MCP Tools for Flux CD — GitRepository reconciliation, Kustomization health, and HelmRelease status

Flux CD exposes its entire state through Kubernetes Custom Resources — there is no separate Flux API server. When building MCP tools that query Flux state, three distinctions determine correctness: source controller health vs kustomize controller health vs helm controller health (each controller manages a separate CRD group and can independently fail), controller health vs resource reconciliation health (a healthy Flux controller pod does not mean your GitRepository or Kustomization is reconciling correctly), and generation vs observedGeneration (if metadata.generation exceeds status.observedGeneration, the controller has not yet processed your last spec change — distinguishing "stuck" from "pending").

TL;DR

Flux CD resources are read via the Kubernetes API — GET /apis/source.toolkit.fluxcd.io/v1/namespaces/{ns}/gitrepositories/{name} for source, /apis/kustomize.toolkit.fluxcd.io/v1/kustomizations for apply state, /apis/helm.toolkit.fluxcd.io/v2/helmreleases for Helm. The status.conditions array is the canonical health signal — look for a condition with type: Ready and check its status field (True = healthy, False = failed, Unknown = reconciling). The status.lastAppliedRevision on a Kustomization and status.lastAttemptedRevision on a GitRepository tell you which commit was applied vs which commit the controller last tried. A Kustomization can have Ready: True while showing a stale revision if spec.suspend: true — always check the suspend field before diagnosing a "stuck" reconciliation.

Flux CD architecture for MCP tool authors

Flux CD is a set of Kubernetes controllers installed into a cluster. The primary controllers are: source-controller (fetches Git repos, OCI images, Helm charts — produces an artifact tarball stored in-cluster), kustomize-controller (watches Kustomization CRs, fetches the artifact from source-controller, runs kustomize build, applies manifests), and helm-controller (watches HelmRelease CRs, fetches Helm charts via source-controller, manages Helm releases).

MCP tools authenticate to the Kubernetes API using a ServiceAccount token or kubeconfig. Use an RBAC ClusterRole that grants get, list, and watch on the Flux CRD groups: source.toolkit.fluxcd.io, kustomize.toolkit.fluxcd.io, and helm.toolkit.fluxcd.io. Never use a cluster-admin ServiceAccount — if your MCP tool's token is leaked or used from a compromised agent, the blast radius must be bounded to read-only Flux state.

async function createFluxClient(k8sBaseUrl, token) {
  const headers = {
    'Authorization': `Bearer ${token}`,
    'Accept': 'application/json',
  };

  async function get(path) {
    const res = await fetch(`${k8sBaseUrl}${path}`, { headers });
    if (!res.ok) {
      const body = await res.text();
      throw new Error(`K8s API ${path} → ${res.status}: ${body}`);
    }
    return res.json();
  }

  // List all GitRepository sources in a namespace
  async function listGitRepositories(namespace = 'flux-system') {
    const path = `/apis/source.toolkit.fluxcd.io/v1/namespaces/${namespace}/gitrepositories`;
    const result = await get(path);
    return result.items.map(parseFluxResource);
  }

  // List all Kustomizations in a namespace
  async function listKustomizations(namespace = 'flux-system') {
    const path = `/apis/kustomize.toolkit.fluxcd.io/v1/namespaces/${namespace}/kustomizations`;
    const result = await get(path);
    return result.items.map(parseFluxResource);
  }

  // List all HelmReleases in a namespace
  async function listHelmReleases(namespace = 'flux-system') {
    const path = `/apis/helm.toolkit.fluxcd.io/v2/namespaces/${namespace}/helmreleases`;
    const result = await get(path);
    return result.items.map(parseFluxResource);
  }

  return { listGitRepositories, listKustomizations, listHelmReleases };
}

Reading Flux resource status: conditions, generation, and revision

Every Flux resource follows the same status convention. The status.conditions array contains typed conditions. The most important is type: Ready. A Ready: True condition with a recent lastTransitionTime means the controller successfully reconciled. Ready: False means reconciliation failed — read the message field for the error. Ready: Unknown means reconciliation is in progress (controller picked up the resource but hasn't finished).

The generation vs observedGeneration gap is the key signal for stuck reconciliation. Every time a resource's spec changes, Kubernetes increments metadata.generation`. The Flux controller sets status.observedGeneration to match when it finishes processing that spec version. If metadata.generation > status.observedGeneration for more than a few seconds, the controller has not yet picked up the change — check if the controller pod is running and healthy.

function parseFluxResource(resource) {
  const conditions = resource.status?.conditions ?? [];
  const readyCondition = conditions.find(c => c.type === 'Ready');
  const reconcilingCondition = conditions.find(c => c.type === 'Reconciling');

  const generation = resource.metadata.generation ?? 0;
  const observedGeneration = resource.status?.observedGeneration ?? 0;
  const generationLag = generation - observedGeneration;

  const suspended = resource.spec?.suspend === true;

  return {
    name: resource.metadata.name,
    namespace: resource.metadata.namespace,
    kind: resource.kind,

    // Primary health signal
    ready: readyCondition?.status === 'True',
    readyStatus: readyCondition?.status ?? 'Unknown',   // 'True' | 'False' | 'Unknown'
    readyReason: readyCondition?.reason ?? null,
    readyMessage: readyCondition?.message ?? null,
    lastTransitionTime: readyCondition?.lastTransitionTime ?? null,

    // Reconciliation progress
    isReconciling: reconcilingCondition?.status === 'True',
    generationLag,         // > 0 = controller hasn't processed latest spec change
    stuckReconciliation: generationLag > 0 && !reconcilingCondition,

    suspended,             // spec.suspend: true = reconciliation paused by operator

    // Revision tracking (GitRepository + Kustomization)
    lastAppliedRevision: resource.status?.lastAppliedRevision ?? null,
    lastAttemptedRevision: resource.status?.lastAttemptedRevision ?? null,
    // Applied != Attempted = last apply failed, showing previous good state
    revisionDrift: (
      resource.status?.lastAppliedRevision !== resource.status?.lastAttemptedRevision &&
      resource.status?.lastAttemptedRevision !== null
    ),
  };
}

// Composite health check for all Flux resources in a namespace
async function fluxNamespaceHealth(flux, namespace) {
  const [gitRepos, kustomizations, helmReleases] = await Promise.all([
    flux.listGitRepositories(namespace),
    flux.listKustomizations(namespace),
    flux.listHelmReleases(namespace),
  ]);

  const all = [...gitRepos, ...kustomizations, ...helmReleases];
  const failing = all.filter(r => !r.suspended && r.readyStatus === 'False');
  const stuck = all.filter(r => !r.suspended && r.stuckReconciliation);
  const suspended = all.filter(r => r.suspended);

  return {
    totalResources: all.length,
    failingCount: failing.length,
    stuckCount: stuck.length,
    suspendedCount: suspended.length,
    healthy: failing.length === 0 && stuck.length === 0,
    failing: failing.map(r => ({ kind: r.kind, name: r.name, reason: r.readyReason, message: r.readyMessage })),
    stuck: stuck.map(r => ({ kind: r.kind, name: r.name, generationLag: r.generationLag })),
  };
}

GitRepository: source fetch errors and artifact freshness

A GitRepository with Ready: False blocks every Kustomization and HelmRelease that references it via spec.sourceRef. The most common GitRepository failures are authentication errors (SSH key or token expired), network errors (Git host unreachable), and revision errors (branch deleted, tag not found). The readyCondition.message contains the underlying error string — surface this directly in your MCP tool response rather than a generic "not ready".

The status.artifact field shows the last successfully fetched artifact: its revision (git commit SHA), digest (sha256 of the tarball), and lastUpdateTime. If lastUpdateTime is older than spec.interval by more than 2x, the source-controller may be failing silently on retries — check the source-controller pod logs and the status.conditions array for a FetchFailed type condition.

async function getGitRepositoryDetail(k8sBaseUrl, token, namespace, name) {
  const res = await fetch(
    `${k8sBaseUrl}/apis/source.toolkit.fluxcd.io/v1/namespaces/${namespace}/gitrepositories/${name}`,
    { headers: { 'Authorization': `Bearer ${token}`, 'Accept': 'application/json' } }
  );
  const repo = await res.json();

  const artifact = repo.status?.artifact;
  const intervalMs = parseFluxInterval(repo.spec?.interval ?? '1m');
  const lastUpdate = artifact?.lastUpdateTime ? new Date(artifact.lastUpdateTime) : null;
  const staleSince = lastUpdate ? Date.now() - lastUpdate.getTime() : null;

  return {
    ...parseFluxResource(repo),
    url: repo.spec?.url,
    branch: repo.spec?.ref?.branch ?? null,
    tag: repo.spec?.ref?.tag ?? null,
    intervalMs,
    artifact: artifact ? {
      revision: artifact.revision,   // e.g. "main@sha1:abc1234..."
      digest: artifact.digest,
      lastUpdateTime: artifact.lastUpdateTime,
      stale: staleSince !== null && staleSince > intervalMs * 2,
      staleDurationMs: staleSince,
    } : null,
  };
}

// Parse Flux interval strings ("1m", "5m", "1h") to milliseconds
function parseFluxInterval(interval) {
  const match = interval.match(/^(\d+)(s|m|h)$/);
  if (!match) return 60_000;
  const [, num, unit] = match;
  const multipliers = { s: 1000, m: 60_000, h: 3_600_000 };
  return parseInt(num, 10) * multipliers[unit];
}

Kustomization: apply status, drift detection, and cross-namespace references

A Kustomization object ties a source (GitRepository) to a path within that source and applies the manifests. The status.lastAppliedRevision is the last Git revision that was successfully applied to the cluster. The status.lastAttemptedRevision is the revision the controller tried most recently. When these differ, the most recent apply failed — the cluster is running the previous revision and Flux is retrying. This is the Flux equivalent of a deployment rollout failure: the old version is "live" but the new version is stuck.

Cross-namespace source references are common in multi-tenant Flux setups: a Kustomization in namespace team-a references a GitRepository in namespace flux-system. In Flux v2, this requires the GitRepository to have a spec.accessFrom or the Kustomization to specify spec.sourceRef.namespace. When an MCP tool queries Kustomizations, always resolve the spec.sourceRef to retrieve the actual source status — a Kustomization may show Ready: False because its GitRepository source (in a different namespace) failed, not because of an apply error.

async function getKustomizationWithSource(fluxClient, k8sClient, namespace, name) {
  const [ks, allGitRepos] = await Promise.all([
    k8sClient.getResource(
      `/apis/kustomize.toolkit.fluxcd.io/v1/namespaces/${namespace}/kustomizations/${name}`
    ),
    // Source may be in a different namespace (cross-namespace ref)
    k8sClient.listAllNamespaces(
      `/apis/source.toolkit.fluxcd.io/v1/gitrepositories`
    ),
  ]);

  const ksResource = parseFluxResource(ks);
  const sourceRef = ks.spec?.sourceRef;
  const sourceNamespace = sourceRef?.namespace ?? namespace;
  const source = allGitRepos.items?.find(
    r => r.metadata.name === sourceRef?.name && r.metadata.namespace === sourceNamespace
  );

  return {
    kustomization: {
      ...ksResource,
      path: ks.spec?.path ?? './',
      prune: ks.spec?.prune ?? false,      // prune: true = delete removed resources
      force: ks.spec?.force ?? false,      // force: true = recreate resources on immutable field change
      targetNamespace: ks.spec?.targetNamespace ?? null,
      revisionDrift: ks.status?.lastAppliedRevision !== ks.status?.lastAttemptedRevision,
      lastAppliedRevision: ks.status?.lastAppliedRevision ?? null,
      lastAttemptedRevision: ks.status?.lastAttemptedRevision ?? null,
    },
    source: source ? parseFluxResource(source) : null,
    sourceReady: source ? (source.status?.conditions?.find(c => c.type === 'Ready')?.status === 'True') : null,
    // Root cause: if source is not ready, that's why Kustomization is failing
    rootCause: source && source.status?.conditions?.find(c => c.type === 'Ready')?.status !== 'True'
      ? 'source_not_ready'
      : ksResource.readyStatus === 'False' ? 'apply_failed' : null,
  };
}

HelmRelease: upgrade failure, rollback detection, and remediation

Flux HelmRelease resources manage the lifecycle of Helm releases — install, upgrade, rollback, and uninstall. The status.history array (Flux v2 HelmRelease v2beta2+) records every Helm action. The status.conditions array includes a type: Released condition (did the last Helm action succeed?) and a type: Ready condition (is the overall HelmRelease healthy?). These can disagree: a HelmRelease can be Ready: True while Released: False if Flux performed a successful rollback after a failed upgrade.

The spec.upgrade.remediation field controls what Flux does on upgrade failure: retries: 3 means retry up to 3 times before marking Ready: False; remediateLastFailure: true plus strategy: rollback means auto-rollback to the previous Helm release on failure. Track status.upgradeFailures and status.installFailures counters — if these increase across reconciliation cycles, the HelmRelease is stuck in a failure loop and automated remediation is not resolving it.

function parseHelmReleaseStatus(hr) {
  const conditions = hr.status?.conditions ?? [];
  const releasedCondition = conditions.find(c => c.type === 'Released');
  const testSuccessCondition = conditions.find(c => c.type === 'TestSuccess');

  // Flux v2 helm release history
  const history = hr.status?.history ?? [];
  const lastRelease = history[0];  // Most recent action first

  return {
    ...parseFluxResource(hr),

    // Helm-specific status
    releaseName: hr.status?.helmChart?.split('/').pop() ?? hr.spec?.releaseName ?? hr.metadata.name,
    releaseNamespace: hr.spec?.targetNamespace ?? hr.metadata.namespace,
    chartVersion: hr.status?.lastAttemptedValuesChecksum ? hr.status?.lastAttemptedRevision : null,

    // Release action result
    lastReleaseSucceeded: releasedCondition?.status === 'True',
    lastReleaseReason: releasedCondition?.reason ?? null,
    lastReleaseMessage: releasedCondition?.message ?? null,

    // Test results (helm test, if spec.test.enable: true)
    testsEnabled: hr.spec?.test?.enable ?? false,
    testsSucceeded: testSuccessCondition?.status === 'True',

    // Failure counters — increasing = stuck in retry loop
    upgradeFailures: hr.status?.upgradeFailures ?? 0,
    installFailures: hr.status?.installFailures ?? 0,

    // Remediation config
    remediationStrategy: hr.spec?.upgrade?.remediation?.strategy ?? 'none',
    remediationRetries: hr.spec?.upgrade?.remediation?.retries ?? 0,

    // Last Helm action detail
    lastAction: lastRelease ? {
      status: lastRelease.status,  // 'deployed' | 'failed' | 'pending-upgrade' | etc.
      chartVersion: lastRelease.chartVersion,
      appVersion: lastRelease.appVersion,
      deployedAt: lastRelease.firstDeployed,
    } : null,
  };
}

AliveMCP integration for Flux CD health monitoring

Flux CD health requires monitoring at two independent layers: the Flux controller pods (source-controller, kustomize-controller, helm-controller deployments in the flux-system namespace) and the individual resource conditions. A healthy controller deployment does not mean your GitRepository and Kustomizations are reconciling — the controller may be running but silently failing on specific resources due to authentication errors, misconfigured paths, or downstream Kubernetes API errors.

Register a custom health probe with AliveMCP that checks both layers and reports the aggregate. Use a 60-second probe interval — Flux's default spec.interval is 1 minute for most resources, so sub-60s polling adds no signal. Set a 15-second timeout on the Kubernetes API calls — the K8s API can be slow under load but rarely takes longer than 10 seconds for a simple resource GET.

async function fluxHealthProbe(k8sBaseUrl, token, namespace = 'flux-system') {
  const headers = {
    'Authorization': `Bearer ${token}`,
    'Accept': 'application/json',
  };

  // Layer 1: Are Flux controller pods healthy?
  const deployments = await Promise.allSettled([
    fetch(`${k8sBaseUrl}/apis/apps/v1/namespaces/${namespace}/deployments/source-controller`, { headers }),
    fetch(`${k8sBaseUrl}/apis/apps/v1/namespaces/${namespace}/deployments/kustomize-controller`, { headers }),
    fetch(`${k8sBaseUrl}/apis/apps/v1/namespaces/${namespace}/deployments/helm-controller`, { headers }),
  ]);

  const controllerHealth = await Promise.all(
    deployments.map(async (result, i) => {
      const names = ['source-controller', 'kustomize-controller', 'helm-controller'];
      if (result.status === 'rejected') return { name: names[i], ready: false, error: result.reason.message };
      const d = await result.value.json();
      const ready = d.status?.readyReplicas > 0;
      return {
        name: names[i],
        ready,
        readyReplicas: d.status?.readyReplicas ?? 0,
        desiredReplicas: d.status?.replicas ?? 0,
      };
    })
  );

  // Layer 2: Are all Flux resources reconciling correctly?
  const [gitRepos, kustomizations, helmReleases] = await Promise.all([
    fetch(`${k8sBaseUrl}/apis/source.toolkit.fluxcd.io/v1/namespaces/${namespace}/gitrepositories`, { headers }).then(r => r.json()),
    fetch(`${k8sBaseUrl}/apis/kustomize.toolkit.fluxcd.io/v1/namespaces/${namespace}/kustomizations`, { headers }).then(r => r.json()),
    fetch(`${k8sBaseUrl}/apis/helm.toolkit.fluxcd.io/v2/namespaces/${namespace}/helmreleases`, { headers }).then(r => r.json()),
  ]);

  const allResources = [
    ...(gitRepos.items ?? []),
    ...(kustomizations.items ?? []),
    ...(helmReleases.items ?? []),
  ].filter(r => !r.spec?.suspend);  // Exclude intentionally suspended resources

  const failing = allResources.filter(r => {
    const ready = r.status?.conditions?.find(c => c.type === 'Ready');
    return ready?.status === 'False';
  });

  const controllersHealthy = controllerHealth.every(c => c.ready);
  const resourcesHealthy = failing.length === 0;

  return {
    healthy: controllersHealthy && resourcesHealthy,
    controllers: controllerHealth,
    resources: {
      total: allResources.length,
      failing: failing.length,
      failingDetails: failing.map(r => ({
        kind: r.kind,
        name: r.metadata.name,
        message: r.status?.conditions?.find(c => c.type === 'Ready')?.message,
      })),
    },
  };
}

Register the Flux probe with AliveMCP at a 60-second interval with a 15-second timeout. Configure separate AliveMCP alerts for controller pod failures (immediate page — no Flux reconciliation happening) vs resource condition failures (warn — cluster may be running stale manifests). The distinction matters: a controller crash blocks all reconciliation, while a single failing Kustomization may be an application-level misconfiguration that doesn't affect other resources.

Related guides