MCP Tools for Kubernetes Operators: Controller Health vs Resource Reconciliation, status.conditions as the Error DSL, and the Silent Acceptance Problem Across Flux CD, KEDA, cert-manager, ESO, and Crossplane

Pattern 1: Controller health does not equal resource reconciliation health

The Kubernetes operator pattern separates the control plane (controller pods that watch CRDs and reconcile resources) from the data plane (the actual resources being managed). This separation is what makes operators scalable — a single controller pod can manage thousands of resources. But it also creates a fundamental monitoring split that every MCP tool must handle explicitly: the health of the controller pod and the health of any specific resource are independently observable, independently failable, and must be checked independently.

The surface area of this split is larger than it appears across the five operators in this arc.

Flux CD: three controllers, each with independent failure modes

Flux CD installs three controller deployments: source-controller (fetches Git repos and Helm charts), kustomize-controller (applies manifests), and helm-controller (manages Helm releases). Each controller is independently healthy or unhealthy, and each manages a distinct CRD group. A healthy source-controller does not mean your Kustomizations are applying — if the kustomize-controller crashes, GitRepositories will continue fetching new commits while every Kustomization stops updating. A healthy kustomize-controller does not mean your GitRepository sources are fetching — if source-controller loses its Git credentials, Kustomizations will continue showing their last-applied revision as healthy while the underlying source has silently stopped fetching new commits.

The correct health probe for Flux requires checking all three controller deployments plus every non-suspended resource's Ready condition. The generation vs observedGeneration gap adds a third check: if metadata.generation > status.observedGeneration on any resource, the controller has received a spec change it hasn't processed — either the controller is down, overwhelmed, or the resource has entered a permanent error state where the controller stopped incrementing observedGeneration.

KEDA: two controller layers, with a non-obvious Active condition

KEDA installs two deployments: keda-operator (watches ScaledObjects, manages the HPA) and keda-operator-metrics-apiserver (implements the Custom Metrics API that HPAs query). Both must be healthy for autoscaling to work — a healthy keda-operator with a crashed metrics API server means the HPA cannot retrieve metric values and will scale based on stale or fallback data without explicit error.

KEDA introduces a condition type that is genuinely not an error: Active: False. When a ScaledObject's trigger metric is at zero (empty RabbitMQ queue, no Kafka consumer lag, idle SQS queue), KEDA sets Active: False and scales the workload to spec.minReplicaCount (which is often 0). This is correct behavior. An MCP tool that alerts on every Active: False will produce false alarms overnight for every queue-based autoscaling workload in the cluster. Only Ready: False indicates a configuration failure — the ScaledObject cannot establish the scaling relationship at all.

cert-manager: the Ready condition vs the actual expiry timestamp

cert-manager separates Issuer health from Certificate health from certificate validity. An Issuer with Ready: True means cert-manager has validated the connection to the CA (Let's Encrypt, HashiCorp Vault, an internal PKI). This does not mean any specific Certificate is healthy. A Certificate with Ready: True means the last issuance succeeded — this does not mean the certificate is still valid. The authoritative expiry timestamp is status.notAfter on the Certificate object, not the Ready condition. A Certificate can have Ready: True and status.notAfter in the past simultaneously, in the window between expiry and cert-manager's next renewal attempt.

cert-manager renews certificates when they enter the renewal window — by default, 2/3 of the certificate's total lifetime. This means a 90-day Let's Encrypt certificate starts renewing at day 60. An MCP tool should alert when status.notAfter is less than 30 days away regardless of the Ready condition — and separately alert when Issuing: True persists for more than 10 minutes, which indicates a stuck renewal.

ESO: the Valid condition on SecretStore is not the Ready condition

External Secrets Operator uses a different condition type on SecretStore resources: type: Valid, not type: Ready. Valid: True means ESO successfully authenticated to the external provider (AWS Secrets Manager, HashiCorp Vault, GCP Secret Manager, etc.). Valid: False means authentication failed — expired IAM role, revoked Vault token, incorrect service account annotation. This condition is on the SecretStore, not on individual ExternalSecrets. When a SecretStore goes Valid: False, every ExternalSecret referencing that store will fail to sync — they will show Ready: False with reason: "SecretSyncError" and their data will become stale at their next refresh interval.

ExternalSecrets have an additional freshness check that the condition alone doesn't cover: status.refreshTime. If status.refreshTime is older than spec.refreshInterval by more than 2x, the controller may be failing silently on retries without updating the condition. One exception: spec.refreshInterval: "0" is intentional — it means "never refresh." Exclude these from staleness alerts.

Crossplane: three health layers with compounding failures

Crossplane has the deepest layering of any operator in this arc. A Provider resource has two conditions: Installed (CRDs installed into the cluster) and Healthy (provider pod running). The most dangerous Crossplane state is Installed: True, Healthy: False — the CRDs exist and the Kubernetes API accepts ManagedResource creation, but no controller pod is running to process them. Resources accumulate silently in an unreconciled state.

Below the Provider level, ManagedResources have two separate conditions: Synced (the provider sent the desired state to the cloud API without error) and Ready (the cloud resource is operational). Synced: True, Ready: False is normal for slow-provisioning resources like RDS instances — the API call succeeded but the database hasn't finished provisioning. Synced: False means the cloud API returned an error — read status.conditions[Synced].message for the raw cloud API error string. CompositeResources add a fourth layer: they aggregate readiness across all composed ManagedResources via a status.resources array, so a CompositeResource showing Ready: False requires tracing through the dependency chain to find the leaf ManagedResource that actually failed.

Pattern 2: status.conditions is the structured error DSL

Every Kubernetes operator in this arc exposes errors through status.conditions — a typed array of condition objects, each with four diagnostic fields: type (what aspect of the resource is being reported), status (True / False / Unknown), reason (a PascalCase machine-readable error category), and message (a human-readable string describing the specific error). The status field is the boolean health signal most monitoring tools read. The reason and message fields are the error DSL that identifies root cause — and they're routinely ignored.

This matters for MCP tools because the reason field enables programmatic failure classification without string parsing. A Flux Ready: False condition with reason: "ArtifactFailed" means the source-controller fetched the artifact but it's invalid; reason: "BuildFailed" means kustomize build failed — different failure class, different remediation. A KEDA Ready: False with reason: "ScalerNotFound" means the trigger plugin isn't installed (cluster-level fix required); reason: "GeneralTriggerConfigError" means the TriggerAuthentication has invalid credentials (secret rotation required). Routing alerts based on reason values dramatically improves signal quality over routing on status === 'False' alone.

Flux: condition types across three CRD groups

Flux uses consistent condition types across GitRepository, Kustomization, and HelmRelease: Ready (overall reconciliation health), Reconciling (reconciliation in progress), and Stalled (reconciliation will never succeed without operator intervention). The lastTransitionTime field on each condition anchors duration calculations — a Reconciling: True condition with lastTransitionTime older than 10 minutes indicates a stuck reconciliation rather than a normal in-progress one. HelmRelease adds Released and TestSuccess condition types, and the status.upgradeFailures integer counter tracks cumulative upgrade failures — an incrementing counter across polling cycles indicates a stuck retry loop that auto-remediation is not resolving.

cert-manager: multi-resource condition chains

cert-manager's condition chain spans four resource types for a single certificate renewal. The Certificate has Ready and Issuing conditions. When renewal begins, cert-manager creates a CertificateRequest with Approved, Ready, and optionally Denied or InvalidRequest conditions. For ACME issuers, the CertificateRequest creates an Order, which creates one or more Challenge resources with their own status. An MCP tool diagnosing a stuck renewal must walk this chain: Certificate (Issuing: True) → CertificateRequest (Approved: True, Ready: False) → Order (pending) → Challenge (pending with reason). Common Challenge failure reasons: PresentChallengeFailed for HTTP-01 (Ingress controller class mismatch or network policy blocking .well-known/acme-challenge/), WrongDNSResolver for DNS-01 (propagation delay), or provider errors in the message field (expired API credentials for the DNS provider).

ESO: the condition type mismatch trap

ESO uses type: Valid on SecretStore resources instead of type: Ready. This is intentional — ESO differentiates between "the store configuration is semantically valid and the provider is reachable" (Valid) from "the last secret sync operation succeeded" (Ready, on ExternalSecrets). An MCP tool that queries SecretStore and looks for a Ready condition will always find no matching condition — the condition exists but is named Valid. This is a common implementation bug in operator monitoring tools that assume all Kubernetes operators use type: Ready as their primary health condition. Always read the operator's CRD documentation to discover the actual condition type names.

Crossplane: Synced vs Ready as orthogonal signals

Crossplane's two-condition model on ManagedResources makes Synced and Ready orthogonal health dimensions rather than sequential ones. A resource can be Synced: True, Ready: True (nominal), Synced: True, Ready: False (cloud API accepted the request; resource provisioning in progress — normal for slow resources), Synced: False, Ready: False (cloud API returned an error — read Synced.message for the error string), or Synced: False, Ready: True (the resource was previously healthy but a spec change produced a cloud API error on the latest reconciliation attempt — the cloud resource still exists in its previous state). Treating Ready as the sole health signal misses the Synced: False case where the provider received an error but the cloud resource was not changed.

// Universal Kubernetes operator condition reader
// Works across Flux, KEDA, cert-manager, ESO, Crossplane
function readConditions(resource) {
  const conditions = resource.status?.conditions ?? [];

  // Index by type for O(1) lookup
  const byType = Object.fromEntries(
    conditions.map(c => [c.type, c])
  );

  function get(type) {
    const c = byType[type];
    if (!c) return { present: false };
    return {
      present: true,
      status: c.status,          // 'True' | 'False' | 'Unknown'
      reason: c.reason ?? null,  // PascalCase machine-readable error category
      message: c.message ?? null,
      lastTransitionTime: c.lastTransitionTime ?? null,
      ageSeconds: c.lastTransitionTime
        ? (Date.now() - new Date(c.lastTransitionTime).getTime()) / 1000
        : null,
    };
  }

  return {
    // Common across most operators
    ready: get('Ready'),

    // Flux-specific
    reconciling: get('Reconciling'),
    stalled: get('Stalled'),
    released: get('Released'),       // HelmRelease only

    // KEDA-specific
    active: get('Active'),           // False = scaled to zero (NOT an error)
    fallback: get('Fallback'),       // True = metric source unavailable

    // cert-manager-specific
    issuing: get('Issuing'),         // True = renewal in progress
    approved: get('Approved'),       // On CertificateRequest
    denied: get('Denied'),           // On CertificateRequest

    // ESO-specific (SecretStore uses Valid, not Ready)
    valid: get('Valid'),             // SecretStore provider auth health

    // Crossplane-specific
    synced: get('Synced'),           // Desired state sent to cloud API
    installed: get('Installed'),     // Provider: CRDs installed
    healthy: get('Healthy'),         // Provider: controller pod running
  };
}

Pattern 3: The silent acceptance problem

The Kubernetes API server validates resources at admission time against the CRD's OpenAPI schema. It checks required fields are present, field types match, and enum values are in the allowed set. It does not validate semantic correctness — whether the credentials will authenticate, whether the referenced secret key exists, whether the Git repository URL is reachable, or whether the cloud provider will accept the resource configuration. Every operator in this arc creates resources that appear fully provisioned from the API's perspective — they have a name, a UID, a creationTimestamp, appear in list responses — but will never reconcile to their desired state because the semantic validation only happens during reconciliation, asynchronously, after creation.

This is not a bug in any of these operators — it's a deliberate consequence of the Kubernetes controller model, where the API server is a dumb store and controllers are responsible for validating and acting on resources. But it produces a monitoring gap: there is no admission webhook that rejects "this ExternalSecret references a Vault path that doesn't exist." The resource is created, the error surfaces in status.conditions only after the first reconciliation attempt, and if no monitoring watches that condition, the failure is silent.

Concrete examples across the five operators

In Flux CD: a GitRepository created with an SSH key secret that has a typo in the key field name passes API admission, appears in kubectl get gitrepositories with no indication of a problem, and only shows Ready: False, reason: "GitOperationFailed" after the source-controller's first fetch attempt. Every Kustomization referencing this GitRepository silently stops receiving updates.

In KEDA: a ScaledObject created with a spec.triggers[].authenticationRef pointing to a TriggerAuthentication whose spec.secretTargetRef[].key does not exist in the referenced Secret passes API admission without error. The ScaledObject appears created. keda-operator discovers the missing key during the first metric poll attempt and sets Ready: False — but the workload was already running at spec.minReplicaCount since before the ScaledObject was created, so no pod restart occurs to signal the failure.

In cert-manager: a Certificate created with an spec.issuerRef pointing to a ClusterIssuer that uses DNS-01 validation via a DNS provider whose API credentials are stored in a Secret that doesn't yet exist in the cert-manager namespace passes schema validation. cert-manager creates the resource, begins reconciliation, creates a CertificateRequest, attempts the Challenge, and only then discovers the Secret is missing. The Certificate stays in Issuing: True indefinitely until the Secret is created or the Certificate is deleted.

In ESO: an ExternalSecret created with spec.data[].remoteRef.key: "prod/db-password" when the actual Vault path is secret/prod/db-password (missing the secret/ mount prefix) passes admission. ESO attempts the sync, receives a 404 from Vault, sets Ready: False, reason: "SecretSyncError" on the ExternalSecret, and the application's Pod reads a stale Kubernetes Secret (if one existed before) or fails to start (if this is a new Secret).

In Crossplane: a ManagedResource created with spec.managementPolicies: ["Observe"] will observe an existing cloud resource but never manage its lifecycle — it won't create it if absent, won't update it if the spec changes, and won't delete it when the Kubernetes resource is deleted. This is intentional for observe-only mode, but it can be configured accidentally. The resource shows Ready: True if the observed cloud resource exists — the absence of lifecycle management is invisible in conditions unless you explicitly check spec.managementPolicies.

What MCP tools must do differently

The silent acceptance problem makes polling status.conditions after resource creation non-optional. An MCP tool that creates a Flux GitRepository and immediately reports success based on the HTTP 201 from the Kubernetes API is reporting a false positive. Correct behavior requires polling the resource's conditions until either (a) the relevant condition transitions to the expected state within a reasonable timeout, or (b) the condition transitions to False with a diagnosable reason, or (c) the timeout expires and the tool reports "provisioning in progress" rather than "success."

For creation workflows, the polling pattern should be: create resource → poll every 5 seconds → check the primary condition (Ready, Valid, Synced) → on False, read reason and message and surface them immediately → on True, report success → on timeout (30–120 seconds depending on the operator), report "still reconciling, check conditions manually."

Cross-operator comparison

The table below captures the five operators' health model at each layer, enabling systematic health check design for MCP tools that span multiple operators in the same cluster.

Composite health probe for Kubernetes operators

An MCP tool monitoring a cluster that runs multiple operators from this arc should implement a layered health probe that checks each layer independently and aggregates the results without collapsing them into a single binary. The following pattern uses Promise.allSettled() so that a failure in one operator's API path does not prevent the others from reporting.

async function kubernetesOperatorsHealthProbe(k8sBaseUrl, token) {
  const headers = { 'Authorization': `Bearer ${token}`, 'Accept': 'application/json' };
  const get = path => fetch(`${k8sBaseUrl}${path}`, { headers }).then(r => {
    if (!r.ok) throw new Error(`K8s ${path} → ${r.status}`);
    return r.json();
  });

  const [
    fluxControllers,
    kedaControllers,
    certManagerController,
    esoController,
    crossplaneProviders,
  ] = await Promise.allSettled([
    // Flux: all three controllers
    Promise.all([
      get('/apis/apps/v1/namespaces/flux-system/deployments/source-controller'),
      get('/apis/apps/v1/namespaces/flux-system/deployments/kustomize-controller'),
      get('/apis/apps/v1/namespaces/flux-system/deployments/helm-controller'),
    ]),
    // KEDA: both deployments
    Promise.all([
      get('/apis/apps/v1/namespaces/keda/deployments/keda-operator'),
      get('/apis/apps/v1/namespaces/keda/deployments/keda-operator-metrics-apiserver'),
    ]),
    // cert-manager
    get('/apis/apps/v1/namespaces/cert-manager/deployments/cert-manager'),
    // ESO
    get('/apis/apps/v1/namespaces/external-secrets/deployments/external-secrets'),
    // Crossplane providers
    get('/apis/pkg.crossplane.io/v1/providers'),
  ]);

  function deploymentReady(d) {
    return (d.status?.readyReplicas ?? 0) > 0;
  }

  function getCondition(resource, type) {
    return resource.status?.conditions?.find(c => c.type === type) ?? null;
  }

  // Layer 1: Controller health
  const controllerHealth = {
    flux: fluxControllers.status === 'fulfilled'
      ? { healthy: fluxControllers.value.every(deploymentReady), controllers: fluxControllers.value.map(d => ({ name: d.metadata.name, ready: deploymentReady(d) })) }
      : { healthy: false, error: fluxControllers.reason?.message },
    keda: kedaControllers.status === 'fulfilled'
      ? { healthy: kedaControllers.value.every(deploymentReady), controllers: kedaControllers.value.map(d => ({ name: d.metadata.name, ready: deploymentReady(d) })) }
      : { healthy: false, error: kedaControllers.reason?.message },
    certManager: certManagerController.status === 'fulfilled'
      ? { healthy: deploymentReady(certManagerController.value) }
      : { healthy: false, error: certManagerController.reason?.message },
    eso: esoController.status === 'fulfilled'
      ? { healthy: deploymentReady(esoController.value) }
      : { healthy: false, error: esoController.reason?.message },
    crossplane: crossplaneProviders.status === 'fulfilled'
      ? {
          providers: (crossplaneProviders.value.items ?? []).map(p => ({
            name: p.metadata.name,
            installed: getCondition(p, 'Installed')?.status === 'True',
            healthy: getCondition(p, 'Healthy')?.status === 'True',
            // Most dangerous: installed but not healthy
            silentlyBroken: getCondition(p, 'Installed')?.status === 'True' &&
                            getCondition(p, 'Healthy')?.status !== 'True',
          })),
        }
      : { providers: [], error: crossplaneProviders.reason?.message },
  };

  return controllerHealth;
}

// Layer 2: Per-resource condition health (run after layer 1)
async function resourceConditionHealth(k8sBaseUrl, token, namespace) {
  const headers = { 'Authorization': `Bearer ${token}`, 'Accept': 'application/json' };
  const get = path => fetch(`${k8sBaseUrl}${path}`, { headers }).then(r => r.json());

  const [gitRepos, kustomizations, scaledObjects, certificates, externalSecrets] = await Promise.all([
    get(`/apis/source.toolkit.fluxcd.io/v1/namespaces/${namespace}/gitrepositories`),
    get(`/apis/kustomize.toolkit.fluxcd.io/v1/namespaces/${namespace}/kustomizations`),
    get(`/apis/keda.sh/v1alpha1/namespaces/${namespace}/scaledobjects`),
    get(`/apis/cert-manager.io/v1/namespaces/${namespace}/certificates`),
    get(`/apis/external-secrets.io/v1beta1/namespaces/${namespace}/externalsecrets`),
  ]);

  function failing(items, conditionType = 'Ready') {
    return (items ?? []).filter(r => {
      if (r.spec?.suspend) return false;
      const c = r.status?.conditions?.find(x => x.type === conditionType);
      return c?.status === 'False';
    }).map(r => {
      const c = r.status?.conditions?.find(x => x.type === conditionType);
      return { name: r.metadata.name, reason: c?.reason, message: c?.message };
    });
  }

  return {
    failingGitRepositories: failing(gitRepos.items),
    failingKustomizations: failing(kustomizations.items),
    failingScaledObjects: failing(scaledObjects.items),   // Active:False is normal; only Ready:False
    expiredCertificates: (certificates.items ?? []).filter(c => {
      const notAfter = c.status?.notAfter;
      return notAfter && new Date(notAfter) < new Date();
    }).map(c => ({ name: c.metadata.name, expiredAt: c.status.notAfter })),
    stuckRenewals: (certificates.items ?? []).filter(c => {
      const issuing = c.status?.conditions?.find(x => x.type === 'Issuing');
      if (issuing?.status !== 'True') return false;
      const age = Date.now() - new Date(issuing.lastTransitionTime ?? 0).getTime();
      return age > 10 * 60 * 1000; // > 10 minutes
    }).map(c => ({ name: c.metadata.name })),
    failingExternalSecrets: failing(externalSecrets.items),
  };
}

Register both probe targets with AliveMCP: the controller-layer probe at 60-second intervals (controller pod crashes need immediate paging) and the resource-layer probe at 60-second intervals with a higher alert threshold for transient failures (a resource that fails for 5 consecutive checks is more likely a configuration problem than a transient retry). The generation vs observedGeneration gap check belongs in the resource-layer probe for Flux and Crossplane resources.

Further reading

• MCP Tools for Flux CD — GitRepository reconciliation, Kustomization health, and HelmRelease status
• MCP Tools for KEDA — ScaledObject status, TriggerAuthentication health, and metrics server monitoring
• MCP Tools for cert-manager — Certificate lifecycle, Issuer health, and ACME challenge debugging
• MCP Tools for External Secrets Operator — SecretStore health, ExternalSecret sync status, and refresh monitoring
• MCP Tools for Crossplane — Provider health, ManagedResource status, and CompositeResource reconciliation
• MCP Tools for Observability Data Stores: Query Language Diversity, Health Check Deception, and Data Gap Semantics
• MCP Tools for Workflow Orchestration: State Machines, Engine Health vs Execution Progress, and Completion Polling
• MCP Tools for Kubernetes API — Deployment rollout status, Pod conditions, and Event-based failure diagnosis
• MCP Server Health Check Patterns — composite endpoints and failure classification