MCP Server Arize Phoenix — OpenInference spans, OTLP export, tool/LLM span kinds, evals

OTLP exporter setup and provider configuration

Arize Phoenix uses the standard OpenTelemetry OTLP exporter. Local Phoenix (self-hosted via pip install arize-phoenix) listens on http://localhost:4317 (gRPC) and http://localhost:4318 (HTTP/protobuf) with no authentication. Phoenix Cloud uses https://app.phoenix.arize.com with api_key in the OTLP header. Mixing the two configurations causes silent export failures with no local error because the OTLP exporter is fire-and-forget by default.

import { NodeTracerProvider } from '@opentelemetry/sdk-trace-node';
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http';
import { SimpleSpanProcessor } from '@opentelemetry/sdk-trace-base';
import { registerInstrumentations } from '@opentelemetry/instrumentation';
import { OpenAIInstrumentation } from '@arize-ai/openinference-instrumentation-openai';
import { trace } from '@opentelemetry/api';

// Detect Phoenix Cloud vs local from endpoint format
const phoenixEndpoint = process.env.OTEL_EXPORTER_OTLP_ENDPOINT ?? 'http://localhost:4318';
const phoenixApiKey   = process.env.PHOENIX_API_KEY;

const isCloud = phoenixEndpoint.includes('app.phoenix.arize.com') ||
                phoenixEndpoint.includes('phoenix.arize.com');

if (isCloud && !phoenixApiKey) {
  console.warn('[phoenix] PHOENIX_API_KEY not set — spans will be dropped by cloud endpoint silently');
}

const exporter = new OTLPTraceExporter({
  url: `${phoenixEndpoint}/v1/traces`,
  headers: isCloud && phoenixApiKey
    ? { 'api_key': phoenixApiKey }  // Phoenix Cloud authentication header
    : {},                            // Local Phoenix: no auth
  timeoutMillis: 5_000,
});

const provider = new NodeTracerProvider();
provider.addSpanProcessor(new SimpleSpanProcessor(exporter));
provider.register();

// Auto-instrumentation: patches the global OpenAI constructor
// Must be called BEFORE any `new OpenAI()` constructor call
registerInstrumentations({
  instrumentations: [
    new OpenAIInstrumentation({
      // enrich_token_usage: true adds prompt/completion token counts to spans
      enrich_token_usage: true,
    }),
  ],
});

// Get a named tracer — use your MCP server's package name
const tracer = trace.getTracer('alivemcp-server', '1.0.0');

// Now safe to create the OpenAI client — instrumentation is active
import OpenAI from 'openai';
const openai = new OpenAI();

Use SimpleSpanProcessor in development (synchronous, immediate export — easy to debug) and swap to BatchSpanProcessor in production (buffers spans and exports in batches — lower per-request overhead). With SimpleSpanProcessor, a Phoenix export failure blocks the request for up to timeoutMillis; with BatchSpanProcessor, export happens out-of-band and never delays tool responses.

Span kinds and semantic conventions for MCP tools

OpenInference defines openinference.span.kind with values TOOL, LLM, CHAIN, RETRIEVER, RERANKER, EMBEDDING, and AGENT. For MCP servers, use TOOL for the MCP tool handler container span and LLM for any inner inference call (the auto-instrumentation handles the LLM spans for you when using OpenAIInstrumentation). Using CHAIN instead of TOOL is the most common mislabelling — it works visually but breaks Phoenix's tool-specific analytics filters.

import { SpanKind } from '@opentelemetry/api';
import { SemanticConventions } from '@arize-ai/openinference-semantic-conventions';
import { z } from 'zod';

// SemanticConventions exports string constants — use these, not string literals
// SemanticConventions.OPENINFERENCE_SPAN_KIND === 'openinference.span.kind'
// SemanticConventions.INPUT_VALUE              === 'input.value'
// SemanticConventions.OUTPUT_VALUE             === 'output.value'
// SemanticConventions.LLM_MODEL_NAME           === 'llm.model_name'

server.tool(
  'analyze_code',
  {
    code:         z.string().min(1).max(20_000),
    language:     z.string().min(1).max(50).default('typescript'),
    session_id:   z.string().optional(),
  },
  async ({ code, language, session_id }) => {
    // Tool container span — kind TOOL so Phoenix routes it correctly
    return tracer.startActiveSpan(
      'mcp.tool.analyze_code',
      {
        kind: SpanKind.INTERNAL,
        attributes: {
          [SemanticConventions.OPENINFERENCE_SPAN_KIND]: 'TOOL',
          [SemanticConventions.INPUT_VALUE]:              JSON.stringify({ language, codeLength: code.length }),
          // Metadata — surfaced in Phoenix's span detail panel
          'tool.name':     'analyze_code',
          'tool.language': language,
          ...(session_id ? { 'session.id': session_id } : {}),
        },
      },
      async (span) => {
        try {
          // openai call is auto-instrumented with kind=LLM and nested under this span
          const completion = await openai.chat.completions.create({
            model:    'gpt-4o-mini',
            messages: [{
              role:    'user',
              content: `Analyze this ${language} code for bugs and improvements:\n\`\`\`${language}\n${code}\n\`\`\``,
            }],
            max_tokens: 1024,
          });

          const analysis = completion.choices[0]?.message?.content ?? '';

          // Set output on the TOOL span — Phoenix links input→output for eval display
          span.setAttribute(SemanticConventions.OUTPUT_VALUE, analysis);
          span.end();

          return { content: [{ type: 'text', text: analysis }] };
        } catch (err: any) {
          span.recordException(err);
          span.setStatus({ code: 2, message: err.message });  // SpanStatusCode.ERROR = 2
          span.end();
          throw err;
        }
      }
    );
  }
);

Retrieval spans for RAG tools

If your MCP tool retrieves documents before sending them to an LLM, create a RETRIEVER span around the retrieval step. Phoenix uses retrieval spans to calculate retrieval-specific metrics like context precision and context recall when you run evals.

import { SemanticConventions } from '@arize-ai/openinference-semantic-conventions';

// MCP tool: retrieve relevant documents, then answer with context
server.tool(
  'rag_query',
  {
    question:   z.string().min(1).max(500),
    session_id: z.string().optional(),
  },
  async ({ question, session_id }) => {
    return tracer.startActiveSpan(
      'mcp.tool.rag_query',
      {
        attributes: {
          [SemanticConventions.OPENINFERENCE_SPAN_KIND]: 'TOOL',
          [SemanticConventions.INPUT_VALUE]:              question,
          ...(session_id ? { 'session.id': session_id } : {}),
        },
      },
      async (toolSpan) => {
        // Step 1: Retrieve documents — RETRIEVER span so Phoenix tracks recall
        const documents = await tracer.startActiveSpan(
          'retrieve_documents',
          {
            attributes: {
              [SemanticConventions.OPENINFERENCE_SPAN_KIND]: 'RETRIEVER',
              [SemanticConventions.INPUT_VALUE]:              question,
            },
          },
          async (retrievalSpan) => {
            // Replace with actual vector search
            const docs = [
              { id: 'doc1', content: 'MCP servers implement the Model Context Protocol...' },
              { id: 'doc2', content: 'Uptime monitoring for MCP endpoints requires...' },
            ];

            // OpenInference retrieval document attributes
            docs.forEach((doc, i) => {
              retrievalSpan.setAttribute(
                `${SemanticConventions.RETRIEVAL_DOCUMENTS}.${i}.document.id`,
                doc.id
              );
              retrievalSpan.setAttribute(
                `${SemanticConventions.RETRIEVAL_DOCUMENTS}.${i}.document.content`,
                doc.content
              );
            });

            retrievalSpan.end();
            return docs;
          }
        );

        // Step 2: LLM generation — auto-instrumented as LLM span
        const context = documents.map(d => d.content).join('\n\n');
        const completion = await openai.chat.completions.create({
          model:    'gpt-4o-mini',
          messages: [
            { role: 'system', content: `Answer using this context:\n${context}` },
            { role: 'user',   content: question },
          ],
          max_tokens: 512,
        });

        const answer = completion.choices[0]?.message?.content ?? '';
        toolSpan.setAttribute(SemanticConventions.OUTPUT_VALUE, answer);
        toolSpan.end();

        return { content: [{ type: 'text', text: answer }] };
      }
    );
  }
);

Online evaluations and annotation

Phoenix's online eval feature runs evaluation functions against incoming spans automatically. Configure evals in the Phoenix UI and they execute in the Phoenix backend. You can also run evals programmatically from your MCP server using Phoenix's Python client, or submit annotations directly via the Phoenix REST API for spans already exported.

// Submit a quality annotation for a completed span via Phoenix REST API
// Phoenix local: http://localhost:6006/v1/span_annotations
// Phoenix Cloud: https://app.phoenix.arize.com/v1/span_annotations
async function annotateSpan(spanId: string, score: number, label: string): Promise<void> {
  const phoenixBase = process.env.PHOENIX_BASE_URL ?? 'http://localhost:6006';
  const apiKey      = process.env.PHOENIX_API_KEY ?? '';

  const body = {
    data: [{
      span_id:    spanId,
      name:       'tool_quality',
      annotator_kind: 'HUMAN',
      result: {
        label,
        score,
        explanation: `Annotated by MCP tool outcome reporting`,
      },
    }],
  };

  const headers: Record<string, string> = {
    'Content-Type': 'application/json',
  };
  if (apiKey) headers['api_key'] = apiKey;

  const res = await fetch(`${phoenixBase}/v1/span_annotations`, {
    method:  'POST',
    headers,
    body:    JSON.stringify(body),
    signal:  AbortSignal.timeout(5_000),
  });

  if (!res.ok) {
    // Non-blocking — annotation failure should not fail the tool
    console.error(`[phoenix] span annotation failed: ${res.status}`);
  }
}

// MCP tool: submit outcome annotation for a previous tool span
server.tool(
  'annotate_tool_outcome',
  {
    span_id: z.string().min(1),
    outcome: z.enum(['correct', 'incorrect', 'partial']),
  },
  async ({ span_id, outcome }) => {
    const score = outcome === 'correct' ? 1.0 : outcome === 'partial' ? 0.5 : 0.0;

    // Fire-and-forget — don't block the tool response on annotation upload
    annotateSpan(span_id, score, outcome).catch(() => {});

    return {
      content: [{ type: 'text', text: JSON.stringify({ annotated: true, span_id, score }) }],
    };
  }
);

Span IDs are accessible inside tracer.startActiveSpan() callbacks via span.spanContext().spanId. Pass the span ID in your tool's response metadata so the calling agent can submit annotations after reporting task outcome, creating a feedback loop that accumulates in Phoenix's annotation dataset.