LLM Observability for MCP Servers: Trace Correlation, Cost Attribution, and Evaluation Feedback Loops

Pattern 1 — Trace Correlation Architecture: Threading Session Context Through Tool Handlers

The fundamental challenge of LLM observability in MCP servers is that a tool handler is stateless by design — it receives a call, runs, and returns — but observability requires state threading: the agent's session ID, parent span context, and run tree position must all be injected into the handler at call time and propagated to every inner LLM call. None of the five systems do this automatically. They all require you to extract the context from the incoming tool call arguments, build the appropriate handle (trace, run, span), and explicitly pass it to every inner operation. The failure mode when you skip this is deceptively subtle: all the LLM calls still get logged, but they appear as disconnected root-level events rather than a tree, making it impossible to reconstruct which agent session triggered which sequence of tool invocations.

Langfuse — trace/generation hierarchy and the flush-before-return contract

Langfuse's data model is a four-level hierarchy: trace → span → generation → score. For MCP servers, the mapping is: one trace per agent session (created once, ID shared across all tool calls in the session), one span per tool invocation (created when the handler starts, ended when it returns), one generation per LLM call inside the handler (captures prompt, completion, model, and token counts), and scores attached to any level after the fact.

The most common mistake is calling langfuse.trace() inside the tool handler rather than passing in an existing trace ID. Each call to langfuse.trace() creates a new root-level trace. If the agent calls five tools in one session and each handler creates its own trace, you get five disconnected timelines with no way to correlate them.

import Langfuse from 'langfuse';
import OpenAI from 'openai';

// Instantiate once at module scope — shares HTTP pool and batch queue
const langfuse = new Langfuse({
  secretKey: process.env.LANGFUSE_SECRET_KEY!,
  publicKey:  process.env.LANGFUSE_PUBLIC_KEY!,
  baseUrl:    process.env.LANGFUSE_HOST ?? 'https://cloud.langfuse.com',
});
const openai = new OpenAI();

// Fetch and cache the prompt version — fallback prevents API outages from
// propagating as tool errors (getPrompt() has 50-200ms latency on cache miss)
const promptCache: { obj: Awaited<ReturnType<typeof langfuse.getPrompt>>, at: number } | null = null;

async function getAnalysisPrompt() {
  if (promptCache && Date.now() - promptCache.at < 300_000) return promptCache.obj;
  try {
    const p = await langfuse.getPrompt('mcp-analysis-prompt', undefined, { cacheTtlSeconds: 300 });
    return p;
  } catch {
    return null; // use inline fallback below
  }
}

server.tool('analyzeEndpoint', { /* zod schema */ }, async (args) => {
  // Agent passes its session ID so all tool calls in this session share one trace
  const sessionId = args.sessionId ?? crypto.randomUUID();

  // Reuse the trace — do NOT call langfuse.trace() here; that creates a new root
  const trace = langfuse.trace({ id: sessionId, name: 'agent-session' });
  const span  = trace.span({ name: 'analyzeEndpoint', input: args });

  const promptObj = await getAnalysisPrompt();
  const systemPrompt = promptObj
    ? promptObj.compile({ endpoint: args.endpoint })
    : `Analyze the MCP endpoint health for: ${args.endpoint}`; // inline fallback

  const gen = trace.generation({
    name:   'analysis-llm-call',
    model:  'gpt-4o-mini',
    input:  [{ role: 'system', content: systemPrompt }, { role: 'user', content: args.query }],
    prompt: promptObj ?? undefined,   // links generation to prompt version in Langfuse UI
  });

  const resp = await openai.chat.completions.create({
    model:    'gpt-4o-mini',
    messages: [{ role: 'system', content: systemPrompt }, { role: 'user', content: args.query }],
  });

  const output = resp.choices[0].message.content ?? '';

  gen.end({
    output,
    usage: {
      input:  resp.usage?.prompt_tokens,
      output: resp.usage?.completion_tokens,
    },
  });
  span.end({ output });

  // CRITICAL: must await before handler returns
  // Langfuse batches events asynchronously; without flushAsync() the process
  // returns before the HTTP upload completes and all events are silently dropped
  await langfuse.flushAsync();

  return { content: [{ type: 'text', text: output }] };
});

The promptObj ?? undefined pattern on the prompt: field is intentional: Langfuse uses this reference to link the generation to a specific prompt version in its UI, enabling the "which prompt version produced the best outcomes" analysis. When the prompt fetch fails (Langfuse API outage, network timeout), the fallback inline string keeps the tool functional while the prompt: field is simply absent from the generation record.

For multi-step tools, create a span hierarchy: the handler span wraps all operations, and each LLM call gets its own generation child. A RAG tool, for example, would have: trace → handler-span → retrieval-span → llm-generation → scoring-generation. The flushAsync() must be called after all of these are ended, not after each one individually.

LangSmith — traceable() wrapper form and UUID v4 run IDs

LangSmith's tracing API centers on the traceable() higher-order function from langsmith/traceable. For standalone applications with class-based handlers, the @traceable decorator works. For MCP servers, it does not: server.tool('name', schema, handler) receives a plain function reference, not a class method, and the decorator form requires a method definition position to attach. The fix is the function wrapper form:

import { traceable }    from 'langsmith/traceable';
import { wrapOpenAI }   from 'langsmith/wrappers';
import { RunTree }      from 'langsmith';
import OpenAI           from 'openai';

// wrapOpenAI instruments the client — LANGCHAIN_TRACING_V2=true only
// auto-instruments LangChain objects, not raw OpenAI SDK instances
const rawClient = new OpenAI();
const openai    = wrapOpenAI(rawClient);

async function _analyzeHandler(args: { endpoint: string; query: string; parentRunId?: string }) {
  const completion = await openai.chat.completions.create({
    model:    'gpt-4o-mini',
    messages: [
      { role: 'system', content: `Analyze MCP endpoint: ${args.endpoint}` },
      { role: 'user',   content: args.query },
    ],
  });
  return completion.choices[0].message.content ?? '';
}

// Wrap with traceable, threading the agent's run ID as the parent
const analyzeTraced = traceable(_analyzeHandler, {
  name:      'analyzeEndpoint',
  run_type:  'tool',
  // id and parent_run_id are provided per-call via the second argument
});

server.tool('analyzeEndpoint', { /* zod schema */ }, async (args) => {
  // parentRunId from args must be a valid UUID v4 — non-UUID values are silently
  // rejected by the LangSmith API (HTTP 422 swallowed by the client) and the
  // run tree is broken with no local error signal
  const parentRunId = args.parentRunId; // caller must supply a UUID v4

  const result = await analyzeTraced(
    { endpoint: args.endpoint, query: args.query, parentRunId },
    { id: crypto.randomUUID(), parent_run_id: parentRunId }
  );

  return { content: [{ type: 'text', text: result }] };
});

The wrapOpenAI call is mandatory if you want inner LLM spans to appear as children of the traceable wrapper. Without it, the raw OpenAI client makes calls that LangSmith cannot intercept, and the trace shows the tool invocation but none of the inner LLM calls — you see cost and latency for the tool but not which prompts were used or what the completion tokens were.

For multi-step tools that need explicit parent-child relationships between run tree nodes, use RunTree directly:

const parent = new RunTree({
  name:          'multi-step-analysis',
  run_type:      'chain',
  id:            crypto.randomUUID(),  // must be UUID v4
  parent_run_id: args.parentRunId,     // must be UUID v4 or undefined
  project_name:  process.env.LANGCHAIN_PROJECT,
});
await parent.postRun();

// ... do step 1 ...

await parent.patchRun({ outputs: { step1: result1 }, end_time: Date.now() });

Project isolation via LANGCHAIN_PROJECT prevents test traces from polluting production baselines. Set a different project name in unit test environments to keep synthetic data out of the dataset that feeds your evaluation pipelines and regression checks.

Arize Phoenix — OpenTelemetry span kinds and instrumentation ordering

Arize Phoenix uses the OpenInference convention on top of OpenTelemetry. The critical constraint is that instrumentOpenAI() patches the global OpenAI constructor — it must be called before any new OpenAI() instantiation, and calling it after the client has been created results in an uninstrumented client with no error or warning. The correct module initialization order is: register instrumentations → create clients → register tools.

import { registerInstrumentations }  from '@opentelemetry/instrumentation';
import { OpenAIInstrumentation }     from '@arize-ai/openinference-instrumentation-openai';
import { NodeTracerProvider }        from '@opentelemetry/sdk-trace-node';
import { OTLPTraceExporter }         from '@opentelemetry/exporter-trace-otlp-http';
import { BatchSpanProcessor }        from '@opentelemetry/sdk-trace-base';
import { SemanticConventions }       from '@arize-ai/openinference-semantic-conventions';
import OpenAI                        from 'openai';

// Step 1: register instrumentations BEFORE any client creation
const provider = new NodeTracerProvider();
const exporter = new OTLPTraceExporter({
  url: process.env.PHOENIX_ENDPOINT ?? 'http://localhost:4318/v1/traces',
  headers: process.env.PHOENIX_API_KEY
    ? { 'api_key': process.env.PHOENIX_API_KEY }  // required for Phoenix Cloud
    : {},
});
// SimpleSpanProcessor in dev (synchronous, immediate export for debugging)
// BatchSpanProcessor in production (async, avoids blocking tool handlers)
provider.addSpanProcessor(new BatchSpanProcessor(exporter));
provider.register();

registerInstrumentations({ instrumentations: [new OpenAIInstrumentation()] });

// Step 2: create client AFTER registration — now it's instrumented
const openai = new OpenAI();
const tracer  = provider.getTracer('mcp-server');

const SEMATTRS = SemanticConventions;

server.tool('analyzeEndpoint', { /* zod schema */ }, async (args) => {
  // Wrap the handler in a TOOL span — this becomes the parent for inner LLM spans
  return tracer.startActiveSpan('analyzeEndpoint', {
    attributes: {
      [SEMATTRS.OPENINFERENCE_SPAN_KIND]: 'TOOL',
      [SEMATTRS.INPUT_VALUE]: JSON.stringify(args),
    },
  }, async (span) => {
    try {
      // OpenAI call automatically creates a child LLM span via instrumentation
      const resp = await openai.chat.completions.create({
        model:    'gpt-4o-mini',
        messages: [{ role: 'user', content: args.query }],
      });

      const output = resp.choices[0].message.content ?? '';
      span.setAttribute(SEMATTRS.OUTPUT_VALUE, output);

      // Return the spanId so the caller can submit annotations later
      const spanId = span.spanContext().spanId;
      span.end();

      return { content: [{ type: 'text', text: output }], spanId };
    } catch (err) {
      span.recordException(err as Error);
      span.end();
      throw err;
    }
  });
});

The span.spanContext().spanId in the response deserves attention: by returning the span ID to the calling agent, you create a hook for post-hoc feedback. The agent can submit a span annotation to Phoenix's REST API after it has evaluated the tool's response, attaching a quality score or correction that accumulates in the annotation dataset without requiring any synchronous feedback loop inside the handler itself.

For RAG tools, wrap the retrieval step in a RETRIEVER span so Phoenix's context precision and recall evaluators run against it automatically. Phoenix's eval harness distinguishes LLM spans (run generation quality evals), RETRIEVER spans (run context relevance evals), and TOOL spans (aggregate only — no auto-eval).

Pattern 2 — Cost Attribution and Budget Enforcement: Per-Tool Spend Accounting

MCP servers that make LLM calls have a structural budget problem: every tool invocation can trigger multiple LLM calls, the token counts compound across a multi-step agent session, and without per-tool attribution the monthly invoice is a single aggregate number that provides no signal about which tools are responsible for overruns. The two approaches — proxy-based attribution (Helicone) and hand-rolled cost tracking — are not mutually exclusive. Helicone provides attribution and dashboards with no code instrumentation; hand-rolled tracking provides budget enforcement, anomaly detection, and provider-level normalization that Helicone cannot do when you mix Anthropic and non-OpenAI-compatible models.

Helicone — proxy attribution and cache safety

Helicone operates as a transparent HTTP proxy between your MCP server and any OpenAI-compatible provider. The integration requires two changes: the baseURL change to route through Helicone, and the authentication change where the Helicone API key goes in a custom header rather than replacing the provider key:

import OpenAI from 'openai';

// The Helicone API key goes in Helicone-Auth, NOT in apiKey
// Swapping these makes Helicone log correctly but forward an invalid key
// to OpenAI, producing a silent 401 from the provider with no Helicone error
const openai = new OpenAI({
  baseURL: 'https://oai.helicone.ai/v1',
  apiKey:  process.env.OPENAI_API_KEY,     // provider key stays here
  defaultHeaders: {
    'Helicone-Auth':                `Bearer ${process.env.HELICONE_API_KEY}`,
    'Helicone-Property-Service':    'alivemcp',
    'Helicone-Property-Environment': process.env.NODE_ENV ?? 'production',
  },
});

// Per-tool attribution — set ToolName in defaultHeaders at construction time
// or pass it per-request to break down spend by tool in the Helicone dashboard
function makeToolClient(toolName: string): OpenAI {
  return new OpenAI({
    baseURL: 'https://oai.helicone.ai/v1',
    apiKey:  process.env.OPENAI_API_KEY,
    defaultHeaders: {
      'Helicone-Auth':             `Bearer ${process.env.HELICONE_API_KEY}`,
      'Helicone-Property-ToolName': toolName,
      // Disable cache for action tools — stale completions for side-effect tools
      // (those that send emails, update records, or call APIs) cause silent bugs
      'Helicone-Cache-Enabled':    'false',
    },
  });
}

// Cache hit detection — a cache hit returns HTTP 200 with a stale completion
// and no exception; the only signal is the response header
server.tool('analyzeEndpoint', { /* zod schema */ }, async (args) => {
  const client = makeToolClient('analyzeEndpoint');

  const { data, response } = await client.chat.completions
    .create({
      model:    'gpt-4o-mini',
      messages: [{ role: 'user', content: args.query }],
      // For query tools (idempotent, no side effects) — enable cache:
      // 'Helicone-Cache-Enabled': 'true', 'Helicone-Cache-Ttl': '3600'
    })
    .withResponse();

  const cacheHit   = response.headers.get('helicone-cache-hit') === 'true';
  const heliconeId = response.headers.get('helicone-id');
  const spend      = response.headers.get('helicone-response-body-size'); // approx cost signal

  if (cacheHit) {
    // Tool logic may need to handle cache hits differently — e.g., add a
    // cache: true flag to the response so the agent knows this wasn't a fresh call
  }

  return {
    content: [{ type: 'text', text: data.choices[0].message.content ?? '' }],
    metadata: { heliconeId, cacheHit },
  };
});

The per-tool client construction pattern (calling new OpenAI() inside a factory function rather than at module scope) costs one object allocation per handler invocation but buys clean per-tool attribution. The alternative — setting the Helicone-Property-ToolName header as a per-request second argument — works but is error-prone: the OpenAI Node.js SDK's extra headers API is a second positional argument on each method call, easy to forget when adding new LLM calls inside existing handlers.

Helicone's rate limit header forwarding is a useful secondary signal: x-ratelimit-remaining-requests and x-ratelimit-remaining-tokens are forwarded from the provider and accessible via .withResponse(). Monitoring these in the MCP server allows proactive backpressure — slowing tool execution before hitting a 429 — rather than reactive retry-loop handling after the rate limit is hit.

LLM cost tracking — provider normalization, streaming usage, and budget gates

When you cannot route all LLM traffic through a single proxy (mixed Anthropic and OpenAI clients, local vLLM models, or compliance requirements that prohibit third-party proxies), a per-tool cost ledger in SQLite provides attribution with no external dependency:

import Database from 'better-sqlite3';

const db = new Database('./data.db');
db.exec(`
  CREATE TABLE IF NOT EXISTS llm_cost_log (
    id          INTEGER PRIMARY KEY AUTOINCREMENT,
    tool_name   TEXT    NOT NULL,
    model_id    TEXT    NOT NULL,   -- from response, not request
    provider    TEXT    NOT NULL,   -- 'openai' | 'anthropic' | 'local'
    input_tok   INTEGER NOT NULL,
    output_tok  INTEGER NOT NULL,
    cache_read_tok  INTEGER NOT NULL DEFAULT 0,
    cache_write_tok INTEGER NOT NULL DEFAULT 0,
    cost_usd    REAL    NOT NULL,
    created_at  DATETIME DEFAULT (datetime('now'))
  )
`);

// Pricing table — use response model IDs, not request model IDs
// Proxies and fallback routing may change the model
const PRICING: Record<string, { input: number; output: number; cacheRead?: number }> = {
  'gpt-4o':                  { input: 2.50,  output: 10.00, cacheRead: 1.25  },
  'gpt-4o-mini':             { input: 0.15,  output: 0.60,  cacheRead: 0.075 },
  'claude-haiku-4-5':        { input: 0.80,  output: 4.00,  cacheRead: 0.08  },
  'claude-sonnet-4-6':       { input: 3.00,  output: 15.00, cacheRead: 0.30  },
};

// Normalize provider-specific usage shapes to a single internal shape
function normalizeUsage(provider: 'openai' | 'anthropic', raw: Record<string, number>) {
  if (provider === 'openai') {
    return {
      inputTokens:      raw['prompt_tokens']     ?? 0,
      outputTokens:     raw['completion_tokens'] ?? 0,
      cacheReadTokens:  raw['prompt_tokens_details']?.['cached_tokens'] ?? 0,
      cacheWriteTokens: 0,
    };
  }
  return { // anthropic
    inputTokens:      raw['input_tokens']               ?? 0,
    outputTokens:     raw['output_tokens']              ?? 0,
    cacheReadTokens:  raw['cache_read_input_tokens']    ?? 0,
    cacheWriteTokens: raw['cache_creation_input_tokens'] ?? 0,
  };
}

function calcCost(modelId: string, usage: ReturnType<typeof normalizeUsage>): number {
  // Match versioned model IDs — 'gpt-4o-mini-2024-07-18' → 'gpt-4o-mini'
  const key = Object.keys(PRICING).find(k => modelId.startsWith(k));
  if (!key) return 0; // unknown model — log but don't throw
  const p = PRICING[key];
  return (
    (usage.inputTokens      * p.input      / 1_000_000) +
    (usage.outputTokens     * p.output     / 1_000_000) +
    (usage.cacheReadTokens  * (p.cacheRead ?? 0) / 1_000_000)
  );
}

const logCost = db.prepare(`
  INSERT INTO llm_cost_log
    (tool_name, model_id, provider, input_tok, output_tok, cache_read_tok, cache_write_tok, cost_usd)
  VALUES
    (@toolName, @modelId, @provider, @inputTok, @outputTok, @cacheReadTok, @cacheWriteTok, @costUsd)
`);

const checkHourlyBudget = db.prepare(
  `SELECT COALESCE(SUM(cost_usd), 0) AS total
   FROM llm_cost_log
   WHERE created_at > datetime('now', '-1 hour')`
);

const HOURLY_BUDGET_USD = Number(process.env.HOURLY_LLM_BUDGET ?? '5.00');

server.tool('analyzeEndpoint', { /* zod schema */ }, async (args) => {
  // Pre-call budget gate — reject before making the call if already over limit
  const { total } = checkHourlyBudget.get() as { total: number };
  if (total >= HOURLY_BUDGET_USD) {
    throw new Error(`Hourly LLM budget of $${HOURLY_BUDGET_USD} exceeded (current: $${total.toFixed(4)})`);
  }

  // IMPORTANT: stream_options.include_usage MUST be set for streaming calls
  // Without it, the final chunk has usage: null and all token counts are zero
  const stream = openai.beta.chat.completions.stream({
    model:          'gpt-4o-mini',
    messages:       [{ role: 'user', content: args.query }],
    stream_options: { include_usage: true },
  });

  let outputText = '';
  for await (const chunk of stream) {
    outputText += chunk.choices[0]?.delta?.content ?? '';
  }
  const finalCompletion = await stream.finalChatCompletion();

  // Read model from response — not from the request — proxy may have rerouted
  const responseModelId = finalCompletion.model;
  const usage = normalizeUsage('openai', finalCompletion.usage as Record<string, number>);
  const costUsd = calcCost(responseModelId, usage);

  logCost.run({
    toolName:      'analyzeEndpoint',
    modelId:       responseModelId,
    provider:      'openai',
    inputTok:      usage.inputTokens,
    outputTok:     usage.outputTokens,
    cacheReadTok:  usage.cacheReadTokens,
    cacheWriteTok: usage.cacheWriteTokens,
    costUsd,
  });

  // Anomaly detection — 10× above 7-day per-tool average signals runaway loop,
  // prompt injection, or accidental model upgrade
  const baseline = (db.prepare(
    `SELECT COALESCE(AVG(cost_usd), 0) AS avg
     FROM llm_cost_log
     WHERE tool_name = 'analyzeEndpoint'
       AND created_at > datetime('now', '-7 days')`
  ).get() as { avg: number }).avg;

  if (baseline > 0 && costUsd > baseline * 10) {
    console.warn(`[cost-anomaly] analyzeEndpoint: $${costUsd.toFixed(6)} is 10× above 7-day avg $${baseline.toFixed(6)}`);
    // Optionally surface to alerting system here
  }

  return { content: [{ type: 'text', text: outputText }] };
});

The stream_options: { include_usage: true } requirement is the most common silent bug in cost tracking implementations. The OpenAI API does not include usage in streaming responses by default — the final chunk's usage field is null unless you opt in. This means every streaming tool invocation reports zero input and output tokens, and the cost ledger accumulates rows with cost_usd = 0.00 that look correct until you notice the dashboard shows no spend despite millions of tokens processed.

The response model vs request model distinction matters whenever LiteLLM, BedRock, or any other proxy layer is in the call path. A request for anthropic/claude-haiku-4-5 may be served by claude-haiku-4-5-20251001 — the exact versioned model ID — and the pricing table must handle the prefix-match case to avoid logging zero cost for successful calls where the model ID didn't exactly match.

Pattern 3 — Evaluation Feedback Loops: Building Quality Signals from Production Traces

The third pattern transforms observability from a debugging tool into a continuous improvement mechanism. Production traces accumulate evidence about which prompts, which models, and which tool designs produce high-quality outputs — but that evidence is inert unless there is a structured way to attach quality signals to specific runs and then use those signals to select prompt versions, build regression datasets, and automate quality gates. All three tracing tools (Langfuse, LangSmith, Arize Phoenix) provide this feedback loop, but they implement it differently and have different constraints on when and how feedback can be submitted.

Langfuse — score() and the async LLM judge pattern

Langfuse's scoring API accepts scores on any level of the hierarchy — trace, span, or generation — via langfuse.score(). The key design decision is timing: synchronous scoring (running an evaluation inside the handler before returning) adds latency to every tool invocation; asynchronous scoring (running evaluation in a background function after the handler returns) has no latency cost but requires maintaining the traceId across the async boundary.

// Async LLM judge — runs fire-and-forget after the handler returns
// Uses a separate Langfuse client to avoid blocking on the main batch queue
async function scoreLlmJudge(traceId: string, output: string, groundTruth?: string) {
  try {
    const judgeResp = await openai.chat.completions.create({
      model:    'gpt-4o-mini',
      messages: [{
        role:    'user',
        content: groundTruth
          ? `Score this output 0-1 vs ground truth.\nOutput: ${output}\nGround truth: ${groundTruth}`
          : `Score this output 0-1 for helpfulness and accuracy.\nOutput: ${output}`,
      }],
    });

    const rawScore = parseFloat(judgeResp.choices[0].message.content ?? '0');
    const score    = Math.min(1, Math.max(0, isNaN(rawScore) ? 0 : rawScore));

    await langfuse.score({
      traceId,
      name:   'llm-judge',
      value:  score,
      source: 'API',
      comment: `judge model: ${judgeResp.model}`,
    });
  } catch (err) {
    // Scoring failure must never propagate to the handler's caller
    console.error('[langfuse-judge] failed:', err);
  }
}

server.tool('analyzeEndpoint', { /* zod schema */ }, async (args) => {
  const sessionId = args.sessionId ?? crypto.randomUUID();
  const trace     = langfuse.trace({ id: sessionId });
  const gen       = trace.generation({ name: 'analysis', model: 'gpt-4o-mini', input: args });

  const resp   = await openai.chat.completions.create({ model: 'gpt-4o-mini', messages: [{ role: 'user', content: args.query }] });
  const output = resp.choices[0].message.content ?? '';

  gen.end({ output, usage: { input: resp.usage?.prompt_tokens, output: resp.usage?.completion_tokens } });
  await langfuse.flushAsync();

  // Fire-and-forget — does not block the handler response
  // void prevents unhandled promise rejection if the outer scope catches
  void scoreLlmJudge(sessionId, output, args.groundTruth);

  return { content: [{ type: 'text', text: output }] };
});

Prompt version selection via Langfuse scores works because Langfuse can aggregate average scores per prompt version over any time window. The workflow: ship a new prompt version to production, let it accumulate ~50 generations, compare average llm-judge scores between versions, promote the higher-scoring version to default. The prompt: reference on each generation is what enables this — without it, Langfuse cannot attribute a score to the prompt version that produced the output.

LangSmith — feedback API and golden dataset construction

LangSmith's feedback model distinguishes between automated scores (submitted programmatically during or after runs) and human annotations (submitted via the UI or API by reviewers). Both are submitted through the same client.createFeedback() API:

import { Client } from 'langsmith';

const lsClient = new Client();

// Submit feedback on a completed run — runId is the UUID v4 from the traceable call
async function submitLangSmithFeedback(
  runId: string,
  score: number,
  correction?: string
) {
  await lsClient.createFeedback(runId, 'correctness', {
    score,
    comment:    correction,
    source_info: { judge: 'gpt-4o-mini', timestamp: new Date().toISOString() },
  });
}

// Build golden dataset by filtering high-score runs
// Runs with score === 1 become examples for regression testing
async function buildGoldenDataset(projectName: string, minScore: number = 0.9) {
  const runs = lsClient.listRuns({
    projectName,
    filter: `and(eq(run_type, "tool"), gte(feedback_stats.correctness.avg, ${minScore}))`,
  });

  const examples = [];
  for await (const run of runs) {
    examples.push({
      inputs:   run.inputs,
      outputs:  run.outputs,
      metadata: { runId: run.id, score: run.feedback_stats?.['correctness']?.avg },
    });
  }

  // Create or update a dataset with these examples
  // Datasets feed into LangSmith's test runner for automated regression
  const dataset = await lsClient.createDataset('mcp-golden-' + projectName, {
    description: `Golden examples from ${projectName} with score >= ${minScore}`,
  });

  await lsClient.createExamples({
    inputs:    examples.map(e => e.inputs),
    outputs:   examples.map(e => e.outputs),
    datasetId: dataset.id,
  });
}

The golden dataset construction pattern closes the quality loop: production runs that score highly become the expected outputs for future regression tests. When you update a prompt or change a model, running the test suite against the golden dataset immediately surfaces regressions before they reach production. The dataset grows organically from production traffic rather than requiring expensive manual annotation campaigns.

Project isolation is the enforcement mechanism for keeping synthetic data out of production quality metrics. Unit tests and integration tests should set LANGCHAIN_PROJECT=mcp-test-<branch> so their traces land in a separate project. The production dashboard's score averages and cost aggregates are then accurate representations of real user traffic, not inflated or deflated by test executions.

Arize Phoenix — span annotations and the eval harness contract

Arize Phoenix runs evaluations automatically on LLM spans, but the mechanism is the Phoenix evaluation SDK operating on collected spans — not inline code inside the MCP server. The MCP server's job is to produce well-formed spans with the correct attributes and export them to the Phoenix backend; the eval harness reads those spans asynchronously and writes annotation results back.

After-the-fact feedback from agents or users can be submitted via the Phoenix REST API using the span ID returned from the handler:

// Submit span annotation after the agent has evaluated the tool's response
// The spanId was returned by the analyzeEndpoint tool handler (see Pattern 1)
async function submitPhoenixAnnotation(
  spanId: string,
  score: number,
  label: string,
  explanation: string
) {
  const response = await fetch(
    `${process.env.PHOENIX_ENDPOINT ?? 'http://localhost:6006'}/v1/span_annotations`,
    {
      method:  'POST',
      headers: {
        'Content-Type': 'application/json',
        ...(process.env.PHOENIX_API_KEY
          ? { 'api_key': process.env.PHOENIX_API_KEY }
          : {}),
      },
      body: JSON.stringify({
        data: [{
          span_id:     spanId,
          name:        'agent-evaluation',
          annotator_kind: 'LLM',
          result: {
            label,
            score,
            explanation,
          },
        }],
      }),
    }
  );

  if (!response.ok) {
    const body = await response.text();
    throw new Error(`Phoenix annotation failed ${response.status}: ${body}`);
  }
}

The Phoenix eval harness distinction between span kinds (LLM, RETRIEVER, TOOL) is worth examining in detail. When you define a custom evaluator in Phoenix, it receives a dataframe of spans filtered by kind. Evaluators that measure generation quality (coherence, groundedness, answer relevance) run on LLM spans. Evaluators that measure retrieval quality (context relevance, context precision, context recall) run on RETRIEVER spans. TOOL spans are aggregates only — they show up in the trace view and in latency/cost rollups, but no built-in evaluator runs against them automatically.

This means a RAG MCP tool must produce at least two levels of instrumented spans to get full eval coverage: a RETRIEVER span for the document fetch step (so context quality evals run) and an LLM span for the answer generation step (so answer quality evals run), all nested under a TOOL span for the handler boundary. Collapsing all three into one span — even if it is an LLM span — means the retrieval-specific evals never see the retrieved documents, and the context precision scores are never computed.

Observability Tool Comparison

Five systems across the three patterns. Rows are technologies; columns are the key integration dimensions for MCP server authors.

Failure Modes Reference

Twelve failure modes across the five systems. Each entry has symptom, root cause, and fix.

Technology Selection Guide

Ten observability and cost-tracking use cases with the recommended approach and the key tradeoff.

Connecting the Three Patterns

The three patterns are temporally ordered: trace correlation must be configured at server startup (instrumentations registered, Langfuse/LangSmith clients initialized), cost attribution is applied per-invocation (budget gate checked before the LLM call, token counts logged after), and evaluation feedback loops are collected asynchronously after responses are delivered. Skipping any one pattern degrades the value of the others: without trace correlation, cost and quality data are attributed to individual calls but cannot be aggregated by session or user; without cost attribution, quality improvements may be economically unsustainable; without evaluation feedback, the observability data accumulates as a debug artifact rather than becoming a flywheel for quality improvement.

The unifying observation across all five systems is that MCP servers require explicit context threading that single-process applications get implicitly through request middleware. In a standard web framework, a session cookie or request ID propagates through middleware automatically; in an MCP server, the handler is a stateless function that receives args, runs, and returns, and any context that needs to span multiple tool invocations (the agent session ID, the parent trace ID, the parent run ID, the OTel span context) must be passed explicitly in the tool's input arguments. The most reliable pattern is to define a _meta field in every tool's input schema that carries observability context — sessionId, traceId, parentRunId, spanContext — and let the orchestrating agent populate it from its own context at tool invocation time.

From a monitoring perspective — which is what AliveMCP is built around — LLM observability tools and MCP uptime monitoring complement each other at different layers. AliveMCP monitors whether your MCP server is reachable and responding correctly at the protocol level; Langfuse, LangSmith, and Phoenix monitor whether the LLM calls inside your server are producing high-quality outputs; and per-tool cost tracking monitors whether the economics of those LLM calls are sustainable. A tool that is reliably reachable (green on AliveMCP) but making LLM calls that are degrading in quality (declining Langfuse scores) or climbing in cost (cost ledger anomalies) is a different failure mode than a tool that is simply down — and catching it requires the full observability stack, not just a liveness probe.