Vector Search MCP Tools — pgvector, Chroma, and Pinecone integration

Vector store deployment models

pgvector — HNSW vs IVFFlat, connection pooling, and the MCP concurrency problem

pgvector exposes two index types. IVFFlat (Inverted File Flat) partitions vectors into lists and searches the nearest lists. Fast to build, uses less memory, but requires setting ivfflat.probes at query time — higher probes improves recall at the cost of latency. HNSW (Hierarchical Navigable Small World) builds a multi-layer graph for navigation, offers better recall at the same latency, and does not require probes tuning. For MCP tools where query latency matters more than index build time, prefer HNSW.

-- Enable pgvector extension
CREATE EXTENSION IF NOT EXISTS vector;

-- Document chunks table with HNSW index
CREATE TABLE document_chunks (
  id          BIGSERIAL PRIMARY KEY,
  source_id   TEXT NOT NULL,
  chunk_index INTEGER NOT NULL,
  content     TEXT NOT NULL,
  embedding   vector(1536),       -- match your embedding model's dimensions
  created_at  TIMESTAMPTZ DEFAULT NOW()
);

-- HNSW index on cosine distance
-- m: max connections per node (higher = better recall, more memory)
-- ef_construction: search width during build (higher = better recall, slower build)
CREATE INDEX ON document_chunks USING hnsw (embedding vector_cosine_ops)
  WITH (m = 16, ef_construction = 64);

-- Set ef for query-time recall vs latency tradeoff (64 is a good default)
SET hnsw.ef_search = 64;

The MCP concurrency problem: an MCP server handling 10 simultaneous agent sessions issues 10 concurrent embedding lookups to PostgreSQL. Each embedding query acquires a connection from the pool. If the pool is sized at 5 connections (pg's default), 5 queries wait, causing P99 latency spikes that look like a slow server but are actually a pool exhaustion event.

// Node.js: size the pool to match expected MCP concurrency
import { Pool } from 'pg';

const pool = new Pool({
  connectionString: process.env.DATABASE_URL,
  max: 20,             // max connections; tune to: (CPU cores × 2) + effective_spindle_count
  idleTimeoutMillis: 30000,
  connectionTimeoutMillis: 2000,  // fail fast rather than queue indefinitely
});

// Health check endpoint: verify pool has idle connections available
app.get('/ready', async (req, res) => {
  if (pool.idleCount === 0 && pool.waitingCount > 0) {
    return res.status(503).json({
      status: 'degraded',
      reason: 'connection_pool_exhausted',
      idle: pool.idleCount,
      waiting: pool.waitingCount,
    });
  }
  res.json({ status: 'ok', idle: pool.idleCount });
});

async function vectorSearch(queryEmbedding, topK = 5, sourceFilter = null) {
  const client = await pool.connect();
  try {
    const params = [JSON.stringify(queryEmbedding), topK];
    const whereClause = sourceFilter ? 'AND source_id = $3' : '';
    if (sourceFilter) params.push(sourceFilter);

    const result = await client.query(`
      SELECT id, source_id, chunk_index, content,
             1 - (embedding <=> $1::vector) AS similarity
      FROM document_chunks
      WHERE 1=1 ${whereClause}
      ORDER BY embedding <=> $1::vector
      LIMIT $2
    `, params);

    return result.rows;
  } finally {
    client.release();
  }
}

The pool.idleCount === 0 && pool.waitingCount > 0 check in the readiness endpoint is the canary for pool exhaustion. AliveMCP polling your /ready endpoint catches this condition before it becomes a full outage. Configure your AliveMCP Author plan webhook to send an alert when /ready returns 503 — this gives you time to scale the connection pool before latency degrades enough to break the agent's context window budget.

Chroma — PersistentClient vs EphemeralClient, query API, and restart behavior

Chroma's two client modes have critically different behavior for MCP servers. EphemeralClient stores the vector index in memory — when the MCP server process restarts, all indexed documents are lost. PersistentClient persists the index to disk in a LanceDB-backed store. Always use PersistentClient in production. The directory you provide must be on a persistent volume — not a tmpfs or container ephemeral layer.

import chromadb
from chromadb.config import Settings

# PersistentClient: survives process restarts
client = chromadb.PersistentClient(
    path="/data/chroma",           # must be on a persistent volume
    settings=Settings(
        anonymized_telemetry=False,
        allow_reset=False,         # prevent accidental collection deletion
    )
)

collection = client.get_or_create_collection(
    name="mcp_documents",
    metadata={"hnsw:space": "cosine"}  # or "l2" or "ip"
)

async def chroma_search(query_text: str, top_k: int = 5, source_filter: str = None):
    where_filter = {"source": source_filter} if source_filter else None

    results = collection.query(
        query_texts=[query_text],         # Chroma embeds internally if embedding_function is set
        n_results=top_k,
        include=["documents", "distances", "metadatas"],
        where=where_filter,
    )

    chunks = []
    for doc, dist, meta in zip(
        results["documents"][0],
        results["distances"][0],
        results["metadatas"][0],
    ):
        # Chroma cosine distance: 0 = identical, 2 = opposite
        # Convert to similarity: 1 - (distance / 2)
        similarity = 1.0 - (dist / 2.0)
        chunks.append({
            "text": doc,
            "source": meta.get("source", "unknown"),
            "similarity": round(similarity, 4),
        })

    return chunks

Chroma uses HNSW internally via the hnswlib library. The hnsw:space metadata field at collection creation time sets the distance function — it cannot be changed after creation without rebuilding the collection. Use cosine for text embeddings from transformer models; use l2 for embeddings from models that were trained with L2 normalization.

For the Chroma HTTP server mode (running Chroma as a separate service instead of embedded), use chromadb.HttpClient(host="localhost", port=8000). This adds an internal network hop but allows the Chroma server to be scaled independently and monitored separately. AliveMCP can probe the Chroma HTTP server's /api/v1/heartbeat endpoint directly if you add it as a secondary monitor.

Pinecone — serverless vs pod-based, upsert batching, and latency characterization

Pinecone offers two architectures. Serverless indexes auto-scale and bill per query — no provisioning, but latency is variable (50–300ms depending on index warmth). Pod-based indexes use dedicated compute, offer predictable latency (~30–80ms), and charge per pod per hour. For MCP tools with a 500ms total latency budget, serverless Pinecone with a large index can consume the entire budget just on the vector search step.

import { Pinecone } from '@pinecone-database/pinecone';

const pc = new Pinecone({ apiKey: process.env.PINECONE_API_KEY });
const index = pc.index(process.env.PINECONE_INDEX_NAME);

async function pineconeSearch(queryEmbedding, topK = 5, namespace = 'default') {
  const start = Date.now();

  const response = await index.namespace(namespace).query({
    vector: queryEmbedding,
    topK,
    includeMetadata: true,
    includeValues: false,
  });

  const latencyMs = Date.now() - start;

  // Log for P95 tracking — emit to your metrics system
  console.log(JSON.stringify({
    event: 'pinecone_query',
    latency_ms: latencyMs,
    matches: response.matches.length,
    namespace,
  }));

  if (latencyMs > 400) {
    console.warn(`Pinecone query slow: ${latencyMs}ms — may exceed tool call latency budget`);
  }

  return response.matches.map(m => ({
    text: m.metadata.text,
    source: m.metadata.source,
    score: m.score,
  }));
}

// Upsert in batches of 100 (Pinecone hard limit is 2MB per request)
async function upsertBatch(vectors, namespace = 'default') {
  const BATCH_SIZE = 100;
  for (let i = 0; i < vectors.length; i += BATCH_SIZE) {
    const batch = vectors.slice(i, i + BATCH_SIZE);
    await index.namespace(namespace).upsert(batch);
    // Small delay to avoid rate limiting
    if (i + BATCH_SIZE < vectors.length) {
      await new Promise(resolve => setTimeout(resolve, 100));
    }
  }
}

Pinecone namespaces partition vectors within a single index. Use namespaces for multi-tenancy in an MCP server: each client gets their own namespace, queries are isolated, and you pay for one index instead of N. The namespace is determined at query time from the agent's authentication context — include it in the tool's internal routing, not in the tool's public input schema.

Latency budget across the full MCP tool call stack

An MCP agent calling search_documents has a total patience budget — typically 2,000–5,000ms before it times out the tool call. That budget is consumed by every step in the retrieval chain.

The P95 tool call latency target is 800ms for documentation retrieval (users tolerate it as a background operation) and 300ms for real-time assistant responses (latency visible to users). With Pinecone, hitting the 300ms target requires skipping cross-encoder reranking or using a very small candidate set. With pgvector HNSW and a warm connection pool, 300ms is achievable including cross-encoding of 10 candidates.

AliveMCP tracks your server's P95 response time at the protocol level. A spike in P95 that correlates with increased traffic usually indicates connection pool exhaustion in pgvector or Pinecone API rate limiting — both produce slow responses before they produce errors.

Index warmth and cold-start latency

HNSW indexes (in pgvector and Chroma) must be loaded into memory before they serve queries at full speed. On a fresh process start, the first query triggers a cold read from disk that can take 1–10 seconds for large indexes. Subsequent queries run from the OS page cache — orders of magnitude faster.

Warm the index on startup before marking the MCP server ready:

async function warmVectorIndex() {
  console.log('Warming vector index...');
  const start = Date.now();

  // Send a no-op query to force index load
  const warmupEmbedding = new Array(1536).fill(0);  // zero vector forces full scan of nearest neighbors
  await vectorSearch(warmupEmbedding, 1);

  console.log(`Vector index warm in ${Date.now() - start}ms`);
}

async function startServer() {
  await warmVectorIndex();   // wait for warmup before accepting traffic
  server.listen(3000);
}

The /ready endpoint should return 503 until warmup completes. If you deploy with Kubernetes, AliveMCP's external probe detects failed readiness probes and alerts — combine AliveMCP with Kubernetes readiness to catch the case where the pod is running but the index never finishes loading (out-of-memory during index load produces a zombie process that passes liveness but fails readiness).

Frequently asked questions

Further reading

• RAG with MCP Servers — retrieval-augmented generation tool patterns
• Embedding Tools in MCP Servers — generate and store vectors via MCP
• MCP Server Connection Pooling — sizing pools for concurrent tool calls
• MCP Server Performance — latency profiling and optimization
• MCP Server Health Checks — liveness, readiness, and custom probes
• MCP Server Observability — metrics, tracing, and structured logging