MCP Server LangSmith — run trees, traceable wrappers, feedback API, project scoping

Traceable wrappers for MCP tool handlers

LangSmith's traceable function wraps any async function and records its input, output, errors, and child spans as a LangSmith run. For MCP tools, the pattern is to define the handler body as a separate async function and pass it to traceable(), then register the wrapped version with server.tool().

import { traceable, wrapOpenAI } from 'langsmith/wrappers';
import { Client as LangSmithClient } from 'langsmith';
import OpenAI from 'openai';
import { z } from 'zod';
import { randomUUID } from 'crypto';

// LANGCHAIN_TRACING_V2=true + LANGCHAIN_API_KEY + LANGCHAIN_PROJECT in env

const openai     = new OpenAI();
const tracedOAI  = wrapOpenAI(openai);  // wraps all calls with LangSmith spans
const lsClient   = new LangSmithClient();

// Define handler separately — traceable() needs a named function reference
async function handleSearchTool(args: {
  query: string;
  max_results: number;
  run_id?: string;  // optional UUID from agent for run tree correlation
}): Promise<{ content: Array<{ type: 'text'; text: string }> }> {
  // Use tracedOAI — LLM calls inside here become child spans in LangSmith
  const completion = await tracedOAI.chat.completions.create({
    model:    'gpt-4o-mini',
    messages: [{
      role:    'user',
      content: `Generate ${args.max_results} search result snippets for: "${args.query}"`,
    }],
    max_tokens: 512,
  });
  const results = completion.choices[0]?.message?.content ?? '';

  return { content: [{ type: 'text', text: results }] };
}

// Wrap with traceable BEFORE registering with MCP
// run_type: 'tool' tells LangSmith this is a tool invocation (not chain/llm/retriever)
const traceableSearchHandler = traceable(handleSearchTool, {
  name:     'mcp.tool.search',
  run_type: 'tool',
  // id can be set per-call by passing a closure, or fixed at registration time
  // For per-call run IDs, see the factory pattern below
});

server.tool(
  'search',
  {
    query:       z.string().min(1).max(500),
    max_results: z.number().int().min(1).max(10).default(5),
    run_id:      z.string().uuid().optional(),
  },
  (args) => traceableSearchHandler(args)
);

The run_type: 'tool' metadata ensures LangSmith renders this run with a wrench icon in the trace tree and includes it in tool-specific analytics. Using run_type: 'chain' (the default when omitted) works but misrepresents the run type in dashboards and eval filters.

Run trees and parent run ID threading

When an agent makes multiple sequential tool calls in one session, you want those calls to appear as siblings under one parent run in LangSmith, not as disconnected roots. Thread the session ID as a UUID-formatted parent run ID. The agent should pass its session ID as a tool argument; your MCP server converts it to a parent run context using LangSmith's context API.

import { traceable } from 'langsmith/wrappers';
import { RunTree } from 'langsmith';
import { randomUUID } from 'crypto';

// Factory that creates a per-call traceable with the correct parent run ID
function makeTracedToolHandler<TArgs, TResult>(
  name: string,
  handler: (args: TArgs, parentRunId: string) => Promise<TResult>
) {
  return async (args: TArgs & { session_run_id?: string }): Promise<TResult> => {
    // session_run_id must be a UUID v4 — non-UUID values are silently rejected
    const parentRunId = args.session_run_id ?? randomUUID();

    // Create a run tree node manually when you need fine-grained control
    const runTree = new RunTree({
      run_type:      'tool',
      name:          name,
      inputs:        { ...args },
      parent_run_id: parentRunId,
      id:            randomUUID(),  // must be UUID v4
      project_name:  process.env.LANGCHAIN_PROJECT ?? 'default',
    });

    await runTree.postRun();

    let result: TResult;
    try {
      result = await handler(args, runTree.id);
      await runTree.end({ outputs: { result } });
    } catch (err: any) {
      await runTree.end({ error: err.message });
      await runTree.patchRun();
      throw err;
    }

    await runTree.patchRun();
    return result;
  };
}

// Usage — each tool call is a child of the agent's session run
const analyzeTool = makeTracedToolHandler(
  'mcp.tool.analyze',
  async ({ text, session_run_id: _sid }: { text: string; session_run_id?: string }, _parentId) => {
    const completion = await tracedOAI.chat.completions.create({
      model:    'gpt-4o-mini',
      messages: [{ role: 'user', content: `Analyze the sentiment of: ${text}` }],
    });
    return { sentiment: completion.choices[0]?.message?.content ?? '' };
  }
);

server.tool(
  'analyze_sentiment',
  {
    text:           z.string().min(1).max(5000),
    session_run_id: z.string().uuid().optional(),
  },
  (args) => analyzeTool(args).then(r => ({
    content: [{ type: 'text', text: JSON.stringify(r) }],
  }))
);

If the agent doesn't pass a session_run_id, generate a fresh UUID so each invocation is still a valid trace root — invalid IDs cause silent 422 errors from the LangSmith API with no client-side exception. Never use human-readable strings like 'tool-call-1' as run IDs.

Feedback submission and dataset building

LangSmith feedback lets you annotate runs with quality scores after the fact. For MCP tools, collect feedback via a dedicated feedback tool that the agent (or user) calls when it knows whether a previous tool output was correct. Accumulate good and bad examples in a LangSmith dataset for regression testing and prompt improvement.

import { Client as LangSmithClient } from 'langsmith';

const lsClient = new LangSmithClient();

// Tool the agent calls to rate a previous tool's output
server.tool(
  'submit_tool_feedback',
  {
    run_id:    z.string().uuid(),
    score:     z.number().min(0).max(1),
    label:     z.enum(['correct', 'incorrect', 'partial']).optional(),
    comment:   z.string().max(1000).optional(),
  },
  async ({ run_id, score, label, comment }) => {
    await lsClient.createFeedback(run_id, 'tool_correctness', {
      score,
      value:   label,
      comment: comment ?? '',
    });

    // If the output was correct, add it to a golden dataset for regression testing
    if (score === 1) {
      // Fetch the run to get its inputs/outputs
      const run = await lsClient.readRun(run_id);
      const datasetName = `mcp-tool-golden-${process.env.LANGCHAIN_PROJECT ?? 'default'}`;

      // Ensure dataset exists
      let dataset;
      try {
        dataset = await lsClient.readDataset({ datasetName });
      } catch {
        dataset = await lsClient.createDataset(datasetName, {
          description: 'Golden examples from MCP tool runs scored 1.0',
        });
      }

      await lsClient.createExample(
        run.inputs ?? {},
        run.outputs ?? {},
        { datasetId: dataset.id }
      );
    }

    return {
      content: [{ type: 'text', text: JSON.stringify({ feedback_recorded: true, run_id }) }],
    };
  }
);

Project scoping and environment isolation

LangSmith projects (set via LANGCHAIN_PROJECT) separate traces by environment or feature. Use distinct project names for development, staging, and production so eval baselines don't get contaminated by test runs. You can also set the project per-run by passing project_name to traceable() options or RunTree, which overrides the env var for that specific run.

// Project isolation pattern for multi-environment MCP deployments
import { traceable } from 'langsmith/wrappers';

const ENV = process.env.NODE_ENV ?? 'development';
const LS_PROJECT = `alivemcp-${ENV}`;  // e.g. 'alivemcp-production', 'alivemcp-staging'

// Override project at the traceable level — useful when NODE_ENV isn't a reliable signal
function makeEnvScopedTool<TArgs, TResult>(
  toolName: string,
  fn: (args: TArgs) => Promise<TResult>
) {
  return traceable(fn, {
    name:         `mcp.tool.${toolName}`,
    run_type:     'tool',
    project_name: LS_PROJECT,
    tags:         [ENV, 'mcp-server', toolName],
  });
}

// Tool tags let you filter traces by tool name in LangSmith without a separate project
const tracedExtract = makeEnvScopedTool('extract_entities', async ({ text }: { text: string }) => {
  const completion = await tracedOAI.chat.completions.create({
    model:    'gpt-4o-mini',
    messages: [{ role: 'user', content: `Extract named entities as JSON array from: ${text}` }],
    response_format: { type: 'json_object' },
  });
  return JSON.parse(completion.choices[0]?.message?.content ?? '{}');
});

server.tool(
  'extract_entities',
  { text: z.string().min(1).max(10_000) },
  (args) => tracedExtract(args).then(r => ({
    content: [{ type: 'text', text: JSON.stringify(r) }],
  }))
);

// MCP resource — expose LangSmith project URL for debugging
server.resource('langsmith_project', 'langsmith://project', async () => ({
  contents: [{
    uri:  'langsmith://project',
    text: JSON.stringify({
      project:  LS_PROJECT,
      url:      `https://smith.langchain.com/o/${process.env.LANGCHAIN_ORG_ID}/projects/p/${LS_PROJECT}`,
      tracing:  process.env.LANGCHAIN_TRACING_V2 === 'true',
    }),
  }],
}));

Keep LANGCHAIN_TRACING_V2=false in unit test environments to avoid polluting production LangSmith projects with synthetic data. The env var check is synchronous and free — setting it to false adds no overhead to test suite runs, and your tools behave identically except LangSmith upload is skipped.