Embedding Tools in MCP Servers — generate and store vectors via MCP

Why MCP as the embedding layer

Direct embedding API calls from agent code create several problems. Every agent component that needs embeddings duplicates the API client, key management, retry logic, and error handling. Rate limits are managed per-client rather than globally. When you switch from text-embedding-ada-002 to text-embedding-3-small, you update every agent component separately. Caching — the biggest cost reduction lever for embedding — requires a shared cache, but each embedded client has its own in-process cache that doesn't help other processes.

An MCP embedding server centralizes all of this. Agents call generate_embedding({ text: "...", model: "default" }) and receive a vector. The server decides which model to call, manages API keys, enforces rate limits across all callers, and maintains a shared embedding cache. Switching models means updating one server and re-indexing the corpus — agent code doesn't change.

Embedding model selection

A critical constraint: the embedding model used at indexing time must be used at query time. The vector space is model-specific — embeddings from text-embedding-3-small and embeddings from bge-small-en-v1.5 are not interchangeable in the same index. If you switch models, you must re-embed the entire corpus and rebuild the index. Plan model choice before committing to production scale.

OpenAI's text-embedding-3 models support dimension reduction via the dimensions parameter. Reducing text-embedding-3-small from 1536 to 256 dimensions reduces storage and query latency with measurable but often acceptable quality loss. Test against your specific domain before committing to reduced dimensions in production.

MCP tool schema and implementation

// Tool schema
const generateEmbeddingTool = {
  name: "generate_embedding",
  description: "Generate a vector embedding for text using the configured embedding model. Returns a numeric array suitable for similarity search.",
  inputSchema: {
    type: "object",
    properties: {
      text: {
        oneOf: [
          { type: "string", description: "Single text to embed" },
          { type: "array", items: { type: "string" }, maxItems: 100, description: "Batch of texts (max 100)" }
        ],
        description: "Text or array of texts to embed"
      },
      model: {
        type: "string",
        enum: ["default", "small", "large"],
        default: "default",
        description: "Embedding model variant. 'default' uses text-embedding-3-small."
      }
    },
    required: ["text"]
  }
};

// Implementation with SHA-256 caching
import OpenAI from 'openai';
import { createHash } from 'crypto';
import Database from 'better-sqlite3';

const openai = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });
const db = new Database('./embedding_cache.db');

db.exec(`
  CREATE TABLE IF NOT EXISTS embedding_cache (
    hash       TEXT PRIMARY KEY,
    model      TEXT NOT NULL,
    dimensions INTEGER NOT NULL,
    vector     BLOB NOT NULL,   -- Float32Array as binary
    created_at INTEGER DEFAULT (unixepoch())
  )
`);

const MODEL_MAP = {
  default: 'text-embedding-3-small',
  small: 'text-embedding-3-small',
  large: 'text-embedding-3-large',
};

function cacheKey(text, model) {
  return createHash('sha256').update(`${model}:${text}`).digest('hex');
}

function vectorToBlob(vector) {
  const buf = Buffer.allocUnsafe(vector.length * 4);
  vector.forEach((v, i) => buf.writeFloatLE(v, i * 4));
  return buf;
}

function blobToVector(blob) {
  const arr = [];
  for (let i = 0; i < blob.length; i += 4) {
    arr.push(blob.readFloatLE(i));
  }
  return arr;
}

async function generateEmbedding(text, modelAlias = 'default') {
  const model = MODEL_MAP[modelAlias] || MODEL_MAP.default;
  const isBatch = Array.isArray(text);
  const inputs = isBatch ? text : [text];

  // Check cache for each input
  const results = new Array(inputs.length);
  const uncachedIndexes = [];

  for (let i = 0; i < inputs.length; i++) {
    const key = cacheKey(inputs[i], model);
    const cached = db.prepare('SELECT vector FROM embedding_cache WHERE hash = ?').get(key);
    if (cached) {
      results[i] = blobToVector(cached.vector);
    } else {
      uncachedIndexes.push(i);
    }
  }

  if (uncachedIndexes.length > 0) {
    const uncachedTexts = uncachedIndexes.map(i => inputs[i]);

    // OpenAI batch API: up to 2048 inputs per request
    const response = await openai.embeddings.create({
      model,
      input: uncachedTexts,
      encoding_format: 'float',
    });

    const insertStmt = db.prepare(
      'INSERT OR REPLACE INTO embedding_cache (hash, model, dimensions, vector) VALUES (?, ?, ?, ?)'
    );

    uncachedIndexes.forEach((originalIndex, batchIndex) => {
      const vector = response.data[batchIndex].embedding;
      results[originalIndex] = vector;

      const key = cacheKey(inputs[originalIndex], model);
      insertStmt.run(key, model, vector.length, vectorToBlob(vector));
    });
  }

  return {
    content: [{
      type: "text",
      text: JSON.stringify({
        embeddings: isBatch ? results : results[0],
        model,
        dimensions: results[0].length,
        cache_hits: inputs.length - uncachedIndexes.length,
        api_calls: uncachedIndexes.length > 0 ? 1 : 0,
      })
    }]
  };
}

Batch embedding for efficiency

Calling the embedding API once per text is the most expensive pattern. OpenAI's API allows up to 2048 texts per request (subject to total token limits). A single API request with 100 short texts costs essentially the same as a single request with 1 text in terms of latency — the per-request overhead dominates, not the per-token cost.

For document indexing workloads (where an agent calls index_document with a large corpus), batch the embedding calls:

async function embedCorpus(chunks, batchSize = 200) {
  const embeddings = [];
  const tokenCounts = [];

  for (let i = 0; i < chunks.length; i += batchSize) {
    const batch = chunks.slice(i, i + batchSize);
    const texts = batch.map(c => c.text);

    const response = await openai.embeddings.create({
      model: 'text-embedding-3-small',
      input: texts,
    });

    response.data.forEach(item => {
      embeddings.push(item.embedding);
    });

    tokenCounts.push(response.usage.total_tokens);

    // Rate limit: 3,000 requests/min on tier 1; 1M tokens/min
    // At batch size 200 × avg 100 tokens = 20K tokens per batch
    // Safe to send 50 batches/min = ~10M tokens/min (above tier 1 limit, but tier 2+)
    if (i + batchSize < chunks.length) {
      await new Promise(r => setTimeout(r, 100));  // 10 batches/sec = safe for tier 2
    }
  }

  const totalTokens = tokenCounts.reduce((sum, t) => sum + t, 0);
  const costUsd = (totalTokens / 1_000_000) * 0.02;  // $0.02 per 1M tokens

  console.log(JSON.stringify({
    event: 'corpus_embedded',
    chunks: chunks.length,
    total_tokens: totalTokens,
    cost_usd: costUsd.toFixed(4),
  }));

  return embeddings;
}

Local embedding with sentence-transformers

For privacy-sensitive corpora or cost-zero deployments, run embedding inference locally. The @xenova/transformers package runs ONNX-format models in Node.js without a Python dependency or GPU requirement — though GPU acceleration significantly reduces inference time.

import { pipeline } from '@xenova/transformers';

let embedder = null;

async function getEmbedder() {
  if (!embedder) {
    // Downloads model on first call (~90MB for bge-small-en-v1.5)
    embedder = await pipeline('feature-extraction', 'Xenova/bge-small-en-v1.5');
  }
  return embedder;
}

async function localEmbed(texts) {
  const model = await getEmbedder();
  const isArray = Array.isArray(texts);
  const inputs = isArray ? texts : [texts];

  const outputs = await model(inputs, { pooling: 'mean', normalize: true });

  // outputs.tolist() returns number[][] — one vector per input
  const vectors = outputs.tolist();

  return isArray ? vectors : vectors[0];
}

The normalize: true flag applies L2 normalization, producing unit vectors. With normalized vectors, cosine similarity equals dot product — you can use the faster dot product operator in pgvector (<#>) instead of the cosine operator (<=>).

Local inference adds 5–30ms per batch call (CPU) compared to 50–100ms for the OpenAI API. However, there's no per-call cost and no rate limit. For high-throughput indexing pipelines, local inference is faster and cheaper. For sporadic queries where the model initialization cost (model download + load, ~1–3 seconds) would dominate, the OpenAI API is faster unless you keep the model loaded in memory.

Embedding cache TTL and invalidation

Embedding cache entries are keyed by SHA-256(model + text) — they never expire from correctness concerns, because the same text with the same model always produces the same vector. The only reason to expire cache entries is disk space management.

For a corpus of 100,000 chunks averaging 200 tokens (approximately 150 characters), the SQLite cache database is approximately 100,000 × (256 bytes SHA256 hash + 1536 × 4 bytes float32 vector + metadata) ≈ 700MB. On modern servers, this is negligible. Increase corpus size by 10× and you still fit in 7GB — a single SQLite file.

-- Prune cache entries older than 90 days for text no longer in the corpus
-- Run this periodically via cron, not on every tool call
DELETE FROM embedding_cache
WHERE created_at < unixepoch('now', '-90 days')
  AND hash NOT IN (
    SELECT embedding_hash FROM document_chunks  -- keep if still referenced
  );

If you switch embedding models, the old cache entries are still valid but will never be accessed (the new model produces different hashes for the same text, since the key includes the model name). Either delete the old model's entries manually or let them age out via TTL pruning.

Health probes: separating process liveness from API reachability

An MCP embedding server has two independent failure modes that require separate health probes:

1. Process dead: the Node.js/Python process crashed. AliveMCP's protocol probe detects this within 60 seconds via connection refused.
2. Embedding API down: the process is running but calls to OpenAI/Cohere are failing. AliveMCP's protocol probe cannot detect this — the MCP server responds to initialize and tools/list normally. Only a test tool call reveals the failure.

// /live: process liveness
app.get('/live', (req, res) => {
  res.json({ status: 'ok' });
});

// /ready: embedding API reachability
app.get('/ready', async (req, res) => {
  try {
    const start = Date.now();

    // Test call with a minimal 1-token input
    await openai.embeddings.create({
      model: 'text-embedding-3-small',
      input: 'ok',
      encoding_format: 'float',
    });

    const latencyMs = Date.now() - start;

    if (latencyMs > 2000) {
      return res.status(503).json({
        status: 'degraded',
        reason: 'embedding_api_slow',
        latency_ms: latencyMs,
      });
    }

    res.json({ status: 'ok', latency_ms: latencyMs });
  } catch (err) {
    res.status(503).json({
      status: 'unhealthy',
      reason: 'embedding_api_unreachable',
      error: err.message,
    });
  }
});

Configure AliveMCP's custom health check URL to point at /ready. This makes embedding API outages visible as MCP server failures in your AliveMCP dashboard. The failure_reason field in AliveMCP's alert payload will read external_api_failure (triggered by your 503 response), distinguishing it from connection_refused (process dead) and timeout (process overloaded). This distinction matters for the runbook: process dead → restart the server; embedding API down → check OpenAI status page, activate local model fallback.

Fallback to local model on API outage

async function generateEmbeddingWithFallback(text, model = 'text-embedding-3-small') {
  try {
    const response = await openai.embeddings.create({ model, input: text });
    return { vector: response.data[0].embedding, source: 'openai' };
  } catch (err) {
    // Circuit open or API down: fall back to local model
    console.warn('OpenAI embedding API failed, using local model:', err.message);
    const vector = await localEmbed(text);
    return { vector, source: 'local' };
  }
}

The fallback introduces a subtle correctness issue: the corpus was indexed with OpenAI vectors, but queries on the fallback path use local model vectors. The two vector spaces are not compatible — cosine similarity between them is meaningless. The fallback produces wrong results, not an error. Document this trade-off: failing fast with an error response is often preferable to silently returning incorrect results.

Frequently asked questions

Further reading

• RAG with MCP Servers — retrieval-augmented generation tool patterns
• Vector Search MCP Tools — pgvector, Chroma, and Pinecone integration
• Semantic Caching for MCP Servers — reduce latency for similar queries
• MCP Server Cost Monitoring — track API spend per tool call
• MCP Server Circuit Breaker — prevent cascading failures from external APIs
• MCP Server Health Checks — liveness, readiness, and custom probes