AI Model Serving for MCP Servers: Provider Routing, Loading Gates, and Protocol Escape Hatches

Pattern 1 — Serving Contract Discovery: The Two-Level Identity Problem

Every serving system has a two-level identity contract: which name you call the model by, and what request shape it expects at that name. These two levels can be misconfigured independently, and the errors they produce rarely point to the actual cause. The safest pattern is to fetch the serving contract once at MCP server startup, cache it, and derive all subsequent call parameters from the cached contract rather than hardcoding values that can drift.

LiteLLM — provider prefix routing and startup model validation

LiteLLM's routing layer uses the {provider}/{model_name} prefix as the routing key. The same proxy instance simultaneously holds credentials for dozens of providers — a model name without a prefix causes LiteLLM to route to OpenAI by default, which produces a 401 if no OpenAI key is configured or (worse) a valid OpenAI response if an OpenAI key is configured but you intended to route to Anthropic. This class of bug produces correct-looking responses from the wrong model.

import OpenAI from 'openai';

const litellm = new OpenAI({
  baseURL: process.env.LITELLM_BASE_URL ?? 'http://localhost:4000',
  apiKey:  process.env.LITELLM_API_KEY  ?? 'sk-1234',
});

// Startup validation — verify the configured model is actually routable
async function validateLiteLLMModel(model: string): Promise<void> {
  const models = await litellm.models.list();
  const known  = new Set(models.data.map(m => m.id));

  // model aliases may appear as exact IDs or as provider/model pairs
  if (!known.has(model)) {
    throw new Error(
      `Model '${model}' not found in LiteLLM router. ` +
      `Available models: ${[...known].slice(0, 5).join(', ')}...`
    );
  }
}

// At MCP server startup — gate tool registration behind this check
const MODEL = process.env.LLM_MODEL ?? 'anthropic/claude-haiku-4-5-20251001';
await validateLiteLLMModel(MODEL);  // throws if prefix missing or model not configured

LiteLLM model aliases (short names like fast-model mapped to a real provider/model in config.yaml) solve the prefix problem for operational teams who want to change provider/model without touching MCP server code. If you are deploying LiteLLM in a shared infrastructure context, prefer aliases for all model references and keep the actual provider/model in LiteLLM's config rather than in each MCP server's environment variables.

vLLM — startup model name discovery

vLLM allows the operator to override the served model name with --served-model-name. This is commonly used to shorten the HuggingFace path (meta-llama/Llama-3.1-8B-Instruct → llama3) or to present a generic name (default) when the model may be swapped between deployments. The MCP server has no way to know which name was chosen at startup unless it queries GET /v1/models.

The second level of the contract: vLLM's chat completions endpoint (/v1/chat/completions) silently falls back to naive message concatenation if the model's tokenizer has no chat template. Base models (not instruction-tuned -Instruct or -Chat variants) often lack a chat template, and the fallback produces output that literally starts with "Human: " or "<s>[INST] " rather than a model-generated response. The fix is to use /v1/completions with an explicit prompt string for base models.

const VLLM_BASE = process.env.VLLM_BASE_URL ?? 'http://localhost:8000/v1';

const vllm = new OpenAI({ baseURL: VLLM_BASE, apiKey: 'EMPTY' });

// Discover served model name and detect chat template presence at startup
let servedModelId: string;
let hasChatTemplate: boolean;

async function initVllm(): Promise<void> {
  await waitForVllm();  // poll GET /health until 200

  const models = await vllm.models.list();
  if (!models.data.length) throw new Error('vLLM returned no models');

  servedModelId = models.data[0].id;  // use first — or filter by expected name

  // Detect chat template by checking if the tokenizer model has one
  // Instruction-tuned models have it; base models do not
  // Heuristic: -Instruct, -Chat, -it, -chat suffix in model name
  hasChatTemplate = /(-instruct|-chat|-it|-chat)$/i.test(servedModelId) ||
                    models.data[0].id.includes('llama') && models.data[0].id.includes('Instruct');
}

// At MCP server startup
await initVllm();

// Route to chat vs completion endpoint based on template availability
async function vllmInfer(prompt: string, maxTokens = 512): Promise<string> {
  if (hasChatTemplate) {
    const c = await vllm.chat.completions.create({
      model:      servedModelId,
      messages:   [{ role: 'user', content: prompt }],
      max_tokens: maxTokens,
      extra_body: { skip_special_tokens: true } as Record<string, unknown>,
    });
    return c.choices[0]?.message?.content ?? '';
  } else {
    // Base model — use raw completions endpoint
    const c = await vllm.completions.create({
      model:      servedModelId,
      prompt,
      max_tokens: maxTokens,
      extra_body: { skip_special_tokens: true } as Record<string, unknown>,
    });
    return c.choices[0]?.text ?? '';
  }
}

Triton — fetch config.pbtxt tensor names once at startup

Triton's KServe v2 protocol requires every request to name inputs and outputs by the exact strings declared in the model's config.pbtxt. The Python backend author may have named the primary text input TEXT, text_input, INPUT_IDS, or anything else — there is no convention. A mismatch between what you send and what config.pbtxt declares produces {"error": "input 'TEXT' is not found in the model configuration"}, which reads like a model identity error rather than a tensor naming error. Fetching the config once at startup and caching the tensor names is the safest approach.

const TRITON = process.env.TRITON_HTTP_URL ?? 'http://localhost:8000';

interface ModelConfig {
  inputName:    string;
  outputName:   string;
  inputDtype:   string;
  outputDtype:  string;
}

const configCache = new Map<string, ModelConfig>();

async function fetchTritonConfig(model: string): Promise<ModelConfig> {
  if (configCache.has(model)) return configCache.get(model)!;

  const res = await fetch(`${TRITON}/v2/models/${model}/config`);
  if (!res.ok) throw new Error(`Triton config for '${model}': HTTP ${res.status}`);

  const cfg = await res.json() as {
    input:  Array<{ name: string; data_type: string }>;
    output: Array<{ name: string; data_type: string }>;
  };

  const config: ModelConfig = {
    inputName:   cfg.input[0]?.name  ?? 'text_input',
    outputName:  cfg.output[0]?.name ?? 'text_output',
    inputDtype:  cfg.input[0]?.data_type  ?? 'BYTES',
    outputDtype: cfg.output[0]?.data_type ?? 'BYTES',
  };

  configCache.set(model, config);
  return config;
}

// MCP tool — text inference using config-derived tensor names
server.tool(
  'triton_infer',
  { model: z.string().min(1), text: z.string().min(1).max(8000) },
  async ({ model, text }) => {
    const { inputName, outputName, outputDtype } = await fetchTritonConfig(model);

    const res = await fetch(`${TRITON}/v2/models/${model}/infer`, {
      method:  'POST',
      headers: { 'Content-Type': 'application/json' },
      body:    JSON.stringify({
        inputs:  [{ name: inputName, shape: [1], datatype: 'BYTES', data: [text] }],
        outputs: [{ name: outputName }],
      }),
      signal: AbortSignal.timeout(30_000),
    });

    if (!res.ok) throw new Error(`Triton infer: HTTP ${res.status}`);

    const body = await res.json() as {
      outputs: Array<{ name: string; datatype: string; data: unknown[] }>;
    };

    const tensor   = body.outputs.find(o => o.name === outputName);
    const rawValue = tensor?.data[0];

    // BYTES output is base64-encoded in HTTP responses — always decode
    const text_out = outputDtype === 'BYTES' && typeof rawValue === 'string'
      ? Buffer.from(rawValue, 'base64').toString('utf-8')
      : String(rawValue ?? '');

    return { content: [{ type: 'text', text: text_out }] };
  }
);

BentoML — inspect /docs before coding the call

BentoML maps Python service class methods to HTTP endpoints: a method named generate becomes POST /generate. The request body is JSON whose keys match the Python function's parameter names exactly — if the Python function is def generate(self, prompt: str, max_new_tokens: int = 256), the JSON body must use prompt and max_new_tokens, not text or max_tokens. Unlike OpenAI-style APIs, there is no universal schema. The /docs endpoint on the live service returns a Swagger UI showing the exact input and output models for each method. Check this before writing MCP tool code, especially when integrating with a BentoML service you did not write yourself.

Hugging Face — task type determines SDK method and response shape

The HF Inference API exposes multiple task types: textGeneration, chatCompletion (Messages API), text2TextGeneration (T5, BART encoder-decoder), featureExtraction (embeddings), textClassification, and others. The task type is declared on the model's Hub card, not in the API endpoint itself. Calling a text-generation model via featureExtraction returns a 2D array of token embeddings rather than an error. Calling a T5 model via textGeneration produces output starting with the full encoder-decoder attention result rather than the summarization or translation you expected. Always verify the task type on the model card before selecting the SDK method.

Pattern 2 — Readiness Hierarchy: Process Alive ≠ Model Loaded ≠ Backends Reachable

Every serving system has at least three distinct health signal levels: the serving process started and its HTTP port is accepting connections; the model weights are fully loaded into memory (GPU VRAM for accelerated serving) and inference is possible; and any upstream dependencies (provider APIs, model backends behind a proxy) are currently reachable. None of the five systems surfaces all three levels through a single endpoint, and the incorrect use of a process-level probe for a Kubernetes readiness check is the most common way MCP tool handlers produce errors that look like model failures but are actually startup timing bugs.

vLLM — one endpoint, two states (loading vs ready)

vLLM uses a single GET /health endpoint that returns HTTP 503 while the model is loading and HTTP 200 when inference is ready. For large models (70B+ parameters), loading takes 5–15 minutes depending on GPU bandwidth and NVMe read speed. An MCP server that registers tools and begins accepting agent connections before GET /health returns 200 will surface ECONNREFUSED errors on the first tool call — agents interpret these as broken tools rather than transient startup conditions.

async function waitForVllm(maxWaitMs = 300_000): Promise<void> {
  const base  = VLLM_BASE.replace('/v1', '');
  const start = Date.now();
  let   n     = 0;

  while (Date.now() - start < maxWaitMs) {
    try {
      const res = await fetch(`${base}/health`, { signal: AbortSignal.timeout(5_000) });
      if (res.status === 200) return;
      // 503 = still loading — expected, keep waiting
    } catch { /* ECONNREFUSED before process starts */ }

    const delay = Math.min(5_000 * (n + 1), 30_000);  // 5s, 10s, 15s, max 30s
    await new Promise(r => setTimeout(r, delay));
    n++;
  }

  throw new Error(`vLLM not ready after ${maxWaitMs / 1000}s`);
}

BentoML — /healthz vs /readyz: choose the right level

BentoML's two health endpoints serve different purposes and the choice matters for both Kubernetes probes and MCP server startup gating. GET /healthz returns 200 as soon as the BentoML runner process starts — this is appropriate for a Kubernetes liveness probe (is the process alive?), but returning 200 during model loading means the process is alive but cannot yet serve requests. GET /readyz returns 200 only after the model weights are fully loaded into memory. Use /readyz for Kubernetes readiness probes and for the MCP server startup gate. Never use /healthz as a readiness gate.

// BentoML startup gate — polls /readyz until model is loaded
async function waitForBento(maxWaitMs = 180_000): Promise<void> {
  const start   = Date.now();
  const headers = process.env.BENTO_API_KEY
    ? { Authorization: `Bearer ${process.env.BENTO_API_KEY}` }
    : {} as Record<string, string>;

  while (Date.now() - start < maxWaitMs) {
    try {
      const res = await fetch(`${process.env.BENTO_URL}/readyz`, {
        headers,
        signal: AbortSignal.timeout(5_000),
      });
      if (res.status === 200) return;
      // /healthz would return 200 here even during loading — /readyz does not
    } catch { /* process not yet started */ }
    await new Promise(r => setTimeout(r, 5_000));
  }
  throw new Error('BentoML service not ready');
}

LiteLLM — three endpoints for three different health concerns

LiteLLM exposes three health endpoints with distinct semantics. GET /health/readiness checks only that the LiteLLM proxy process is running — it returns 200 within milliseconds and is appropriate for Kubernetes readiness probes. GET /health/liveliness is similar, also checking proxy process liveness. GET /health makes a live probe to each configured model backend and returns a JSON body with per-model status including unhealthy endpoints — this is the endpoint to call for business-logic health monitoring, but it can take 10–15 seconds if backends are slow or unhealthy, making it unsuitable as a Kubernetes readiness probe target.

// MCP resource — composite LiteLLM health (full backend probe)
server.resource('litellm_health', 'litellm://health', async () => {
  const base = process.env.LITELLM_BASE_URL ?? 'http://localhost:4000';

  const [readiness, backends] = await Promise.allSettled([
    // Fast process-level check
    fetch(`${base}/health/readiness`, { signal: AbortSignal.timeout(3_000) }),
    // Slow per-model backend check — may timeout if backends are unreachable
    fetch(`${base}/health`, {
      headers: { Authorization: `Bearer ${process.env.LITELLM_API_KEY ?? ''}` },
      signal: AbortSignal.timeout(20_000),
    }),
  ]);

  const processAlive = readiness.status === 'fulfilled' && readiness.value.ok;
  let backendsBody: Record<string, unknown> | null = null;

  if (backends.status === 'fulfilled' && backends.value.ok) {
    backendsBody = await backends.value.json().catch(() => null);
  }

  return {
    contents: [{
      uri: 'litellm://health',
      text: JSON.stringify({
        process_alive:     processAlive,
        healthy_backends:  (backendsBody as any)?.healthy_count   ?? null,
        unhealthy_backends:(backendsBody as any)?.unhealthy_count ?? null,
        unhealthy_models:  (backendsBody as any)?.unhealthy_endpoints?.map((e: any) => e.model) ?? [],
        checked_at:        new Date().toISOString(),
      }),
    }],
  };
});

Triton — server ready vs per-model ready

Triton's GET /v2/health/ready returns 200 when the server is ready to accept inference requests, but individual model load failures do not cause this endpoint to return non-200. A model that failed to load due to a missing weight file or backend misconfiguration will simply return an error on subsequent infer calls — the server-level health probe is green while the model-level state is broken. Use GET /v2/models/{model}/ready for per-model readiness at startup.

// Triton per-model readiness check at MCP server startup
async function validateTritonModel(model: string): Promise<void> {
  // Server-level
  const serverRes = await fetch(`${TRITON}/v2/health/ready`, { signal: AbortSignal.timeout(5_000) });
  if (!serverRes.ok) throw new Error(`Triton server not ready: HTTP ${serverRes.status}`);

  // Per-model — 200 = loaded, 400 = failed to load
  const modelRes = await fetch(`${TRITON}/v2/models/${model}/ready`, { signal: AbortSignal.timeout(5_000) });
  if (!modelRes.ok) {
    throw new Error(`Triton model '${model}' not ready: HTTP ${modelRes.status} — check model load logs`);
  }
}

Hugging Face Serverless — cold start is the readiness signal

The Serverless Inference API has no dedicated health endpoint. Model load state is communicated inline with inference requests: when a serverless model has not been called recently and is not loaded on a backend, the API returns HTTP 503 with {"error": "Model is currently loading.", "estimated_time": 20}. The estimated_time field is the serving system telling you how many seconds to wait before retrying. Retrying immediately causes another 503 — the retry window is not immediate. The correct handler sleeps for estimated_time + 2 seconds (the buffer accounts for estimation error) then retries up to three times.

async function hfWithColdStartRetry<T>(fn: () => Promise<T>, maxAttempts = 3): Promise<T> {
  let lastErr: unknown;

  for (let attempt = 0; attempt < maxAttempts; attempt++) {
    try {
      return await fn();
    } catch (err: any) {
      lastErr = err;
      const msg  = err?.message ?? '';

      // HF cold start — sleep for estimated_time + 2s before retrying
      const match = msg.match(/"estimated_time"\s*:\s*(\d+(?:\.\d+)?)/);
      if (match) {
        const waitMs = Math.min((parseFloat(match[1]) + 2) * 1000, 120_000);
        await new Promise(r => setTimeout(r, waitMs));
        continue;
      }

      // Transient 503 without estimated_time — short backoff
      if (err?.httpStatus === 503 || msg.includes('503')) {
        await new Promise(r => setTimeout(r, (attempt + 1) * 5_000));
        continue;
      }

      throw err;  // non-retriable
    }
  }

  throw lastErr;
}

Pattern 3 — Protocol Escape Hatches and Silent Type Coercions

Four of the five systems present themselves as "OpenAI-compatible" to varying degrees. In practice, each adds non-standard parameters, response headers, error shapes, or output encoding conventions that the OpenAI SDK's TypeScript types do not represent. Production bugs cluster in three places: parameters you send that get silently dropped because they are outside the typed API, error responses you misclassify because the error shape differs from OpenAI's, and output values you misinterpret because of an encoding transformation you did not expect.

vLLM — extra_body for non-standard sampling parameters

vLLM's most powerful features — constrained JSON decoding, best-of-N sampling, token-level stopping, and attention sinks — are not in the OpenAI spec. When using the OpenAI Node.js SDK, these parameters must be passed in extra_body, a TypeScript-opaque dict that the SDK forwards verbatim to the underlying HTTP request without type checking. Passing these parameters at the top level of the request object causes the SDK to throw a TypeScript compile error or silently ignore the fields at runtime.

// vLLM guided JSON decoding — must use extra_body, not top-level params
const completion = await vllm.chat.completions.create({
  model:      servedModelId,
  messages:   [{ role: 'user', content: `Extract as JSON: ${text}` }],
  max_tokens: 1024,
  temperature: 0,
  // vLLM-specific params — not in OpenAI spec, silently dropped without extra_body
  extra_body: {
    guided_json:             schema,       // JSON Schema for constrained decoding
    guided_decoding_backend: 'outlines',   // 'outlines' | 'lm-format-enforcer'
    skip_special_tokens:     true,         // strip <|im_end|> etc. from output
    stop_token_ids:          [32000],      // stop on specific token IDs
  } as Record<string, unknown>,
});

LiteLLM — two 429 shapes and spend headers outside the type system

LiteLLM returns HTTP 429 for two distinct conditions that require different responses: provider-side rate limiting (error.type === 'RateLimitError', retry with backoff after retry-after header) and virtual key budget exhaustion (error.type === 'BudgetExceeded', do not retry, surface to user). Treating budget exhaustion as a rate limit causes infinite retry loops that never resolve. The spend-tracking headers LiteLLM adds to every response (x-litellm-spend, x-litellm-model-response-cost) are not part of the OpenAI SDK types — they are only accessible through the .withResponse() method which returns the raw Response alongside the typed data.

// LiteLLM error handling — differentiate budget exhaustion from rate limit
async function litellmSafe(params: OpenAI.Chat.ChatCompletionCreateParamsNonStreaming): Promise<string> {
  try {
    const { data: completion, response } = await litellm.chat.completions
      .create(params)
      .withResponse();

    // Read spend headers — not in SDK types, only available via withResponse()
    const spendUsd = parseFloat(response.headers.get('x-litellm-spend') ?? '0');
    if (spendUsd > 0.10) {
      console.warn(`High per-call spend: $${spendUsd.toFixed(4)}`);
    }

    return completion.choices[0]?.message?.content ?? '';
  } catch (err: any) {
    if (err?.status === 429) {
      const errType = err?.error?.type ?? err?.error?.error?.type ?? '';

      if (errType === 'BudgetExceeded') {
        // Hard budget cap — do NOT retry, surface to user
        throw new Error(`LiteLLM budget exhausted. ${err?.error?.message ?? ''}`);
      }

      // Provider rate limit — retry with backoff
      const retryAfter = parseInt(err?.headers?.['retry-after'] ?? '60', 10);
      throw new Error(`Rate limited. Retry after ${retryAfter}s. Model: ${params.model}`);
    }

    if (err?.status === 401 || err?.status === 403) {
      throw new Error(`Auth failed for '${params.model}' — check provider prefix in config.yaml`);
    }

    throw err;
  }
}

Triton — BYTES outputs are base64-encoded; FP32 outputs are not

Triton's HTTP REST interface applies different serialization rules to different tensor datatypes. BYTES tensors (the standard datatype for string I/O in Python backends) are base64-encoded in HTTP responses — this is an undocumented transport-level encoding that applies even though you sent the same BYTES tensor as a plain string in the request. FP32 tensors (floating-point embeddings) are serialized as JSON number arrays without any base64 encoding. The practical implication: a model that takes text input (BYTES) and returns text output (BYTES) requires decoding on every response, while a model that takes text input (BYTES) and returns float embeddings (FP32) requires no decoding. Always check outputTensor.datatype before deciding whether to base64-decode.

// Triton output decoding — check datatype before decoding
function decodeTritonOutput(tensor: { datatype: string; data: unknown[] }): (string | number)[] {
  return tensor.data.map(raw => {
    if (tensor.datatype === 'BYTES' && typeof raw === 'string') {
      // BYTES = base64-encoded in HTTP responses — must decode
      return Buffer.from(raw, 'base64').toString('utf-8');
    }
    // FP32, INT32, INT64, BOOL — JSON-native, no encoding
    return raw as number;
  });
}

BentoML — SSE streaming and multipart file upload

BentoML service methods that return AsyncGenerator[str, None] (Python) stream their output as Server-Sent Events. The request must set Accept: text/event-stream — without it, BentoML buffers the full generator output and returns it as a JSON string, losing all streaming benefits. Each SSE event line has the format data: {chunk}\n and the stream ends with a data: [DONE]\n\n sentinel. Multimodal services that accept images or audio files use multipart/form-data with form field names matching the Python parameter names annotated with bentoml.Image or bentoml.File — never send binary content as base64 in a JSON body for file-typed parameters.

// BentoML SSE streaming — requires Accept header and manual SSE parsing
async function bentStream(method: string, input: Record<string, unknown>): Promise<string> {
  const headers: Record<string, string> = {
    'Content-Type': 'application/json',
    'Accept':       'text/event-stream',  // REQUIRED — omitting this causes buffered JSON response
  };
  if (process.env.BENTO_API_KEY) headers['Authorization'] = `Bearer ${process.env.BENTO_API_KEY}`;

  const res = await fetch(`${process.env.BENTO_URL}/${method}`, {
    method: 'POST', headers, body: JSON.stringify(input),
    signal: AbortSignal.timeout(120_000),
  });

  if (!res.ok || !res.body) throw new Error(`BentoML ${method}: HTTP ${res.status}`);

  const reader  = res.body.getReader();
  const decoder = new TextDecoder();
  let   buf     = '';
  let   full    = '';

  while (true) {
    const { done, value } = await reader.read();
    if (done) break;
    buf += decoder.decode(value, { stream: true });

    const events = buf.split('\n\n');
    buf = events.pop() ?? '';

    for (const evt of events) {
      for (const line of evt.split('\n')) {
        if (!line.startsWith('data: ')) continue;
        const data = line.slice(6);
        if (data === '[DONE]') return full;  // stream complete
        full += data;
      }
    }
  }

  return full;
}

Hugging Face — return_full_text and feature extraction output normalization

textGeneration returns the full prompt concatenated with the generated text by default (return_full_text: true). For MCP tools that return only the model's completion to an agent, this means the tool output starts with the agent's own prompt — a confusing and wasteful duplication. Set return_full_text: false to receive only the newly generated tokens. For featureExtraction, sentence-transformer models return a flat number[] (one embedding for the full input, pooling applied by the model), while raw transformer models return number[][] (one row per token). Treating 2D output as 1D in a RAG pipeline means your embedding vector is the first token's representation rather than the sentence embedding, which silently degrades retrieval quality.

// HF feature extraction with output normalization
async function hfEmbed(text: string, model: string): Promise<number[]> {
  const raw = await hfWithColdStartRetry(() =>
    hf.featureExtraction({ model, inputs: text })
  );

  // Normalize shape — sentence-transformers: number[], raw transformers: number[][]
  if (Array.isArray((raw as number[][])[0])) {
    const matrix = raw as number[][];
    const dims   = matrix[0].length;
    // Mean pool token embeddings to get sentence embedding
    return Array.from({ length: dims }, (_, d) =>
      matrix.reduce((sum, tok) => sum + tok[d], 0) / matrix.length
    );
  }

  return raw as number[];
}

Health Probe Comparison Table

Use this table to configure Kubernetes probes, MCP server startup gates, and AliveMCP monitoring rules. "Process level" is appropriate for liveness probes only — never use it as a readiness gate for an MCP server that calls models.

Failure Modes Reference

Twelve failure modes across the five systems, organised by the pattern they belong to. Each entry includes root cause, symptom, and fix.

Technology Selection Guide

Ten common MCP server AI inference use cases with the serving system most appropriate for each:

Connecting the Three Patterns

The three patterns are load-ordered: serving contract discovery must happen first (before startup is complete), readiness gating must happen second (before tools are registered), and protocol escape hatch knowledge must be applied at call time. Skipping any step causes failures at the next step: without contract discovery you produce misnamed tensor or wrong model name errors; without readiness gating you produce connection errors that agents misinterpret as broken tools; without protocol knowledge you produce silent wrong-output bugs or error misclassifications that cause wrong retry behavior.

The unifying observation is that none of the five systems is fully described by its "OpenAI-compatible" label. LiteLLM adds routing, budgeting, and spend telemetry that lives outside the OpenAI API surface. vLLM extends completions with constrained decoding and sampling controls that require an escape hatch. BentoML is not OpenAI-compatible at all — it exposes whatever Python methods you define. Triton uses a formally specified tensor protocol that predates and is orthogonal to the OpenAI API. The HF Inference API is task-typed rather than model-typed. Writing MCP tool code that treats any of these as a drop-in OpenAI replacement without reading the system's actual contract is the most reliable way to ship tools that produce subtle, deployment-specific bugs.

From a monitoring perspective — which is what AliveMCP is built for — each system requires a different health probe strategy. A single TCP connectivity check tells you nothing about whether a vLLM instance has finished loading its 70B model or whether a LiteLLM backend's Anthropic API key is still valid. The health probe table in this post is designed to be translated directly into AliveMCP monitoring rules: process-level checks run every 60 seconds, model-level checks run every 5 minutes, and backend reachability checks (which make real API calls) run every 10 minutes with alerts on consecutive failures.