MCP Servers as the Retrieval Layer: RAG, Vector Search, Embeddings, Context Management, and Semantic Caching

The RAG layer: why MCP is the right retrieval boundary

Before MCP, adding retrieval to an LLM application meant coupling retrieval logic into agent code. Every agent that needed document search had to manage vector store credentials, embedding API keys, chunking parameters, and context assembly. When you switched from text-embedding-ada-002 to text-embedding-3-small, you updated every agent component separately. When the retrieval logic needed tuning — chunk size, rerank strategy, context format — you changed it in N places.

MCP externalizes retrieval behind a tool boundary. The agent calls search_documents({ query: "how do I configure rate limiting?", top_k: 5 }) and receives ranked chunks with metadata. A single MCP RAG server can serve a support bot, a documentation assistant, and a code review helper — all sharing one indexed corpus. Re-indexing on corpus change happens in one place; all agents see the update immediately.

A minimal RAG MCP server exposes three tools. search_documents is the inference-time tool: embed the query, run hybrid retrieval, rerank, return ranked chunks. index_document is the ingestion tool: chunk the document, embed each chunk, upsert to the vector store. list_sources enumerates indexed sources with chunk counts and last-indexed timestamps — letting agents surface staleness to users before issuing retrieval calls that might return outdated results.

Chunking strategy is the highest-leverage tuning decision for retrieval quality. Chunks too large dilute the relevance signal; chunks too small lose cross-sentence context. For technical documentation, the sentence-boundary strategy — 4–6 sentences per chunk with one sentence of overlap, enforced with a max-token limit from js-tiktoken — produces more coherent chunks than fixed-character splitting, which cuts mid-sentence and breaks the passage that would have ranked highest.

For retrieval, combine BM25 keyword search with vector similarity via Reciprocal Rank Fusion. Pure vector search misses exact technical term matches: "MCP initialize timeout" and "initialization timeout handling" are close in embedding space but not identical — BM25 handles the exact-match case that vector search deprioritizes. The RRF merge formula score = 1/(rank + 60) (the k=60 constant reduces sensitivity to high-ranked outliers) produces a merged ranking that outperforms either signal alone. Over-fetch top-20 candidates from hybrid retrieval, then rerank with a cross-encoder (ms-marco-MiniLM-L-6-v2 in 200–600ms on CPU) to return top-5 with joint query-document scoring.

Vector stores: what each backend gets wrong in an MCP context

Every vector store has a different failure surface. The choice is not just about latency and scale — it is about which class of silent failure you are accepting and how you monitor for it.

The pgvector concurrency problem deserves more detail because it is the most common production failure mode for MCP servers. A default PostgreSQL connection limit of 100, a pool size of 10, and 10 concurrent MCP sessions sounds fine — until those 10 sessions each make 3 tool calls simultaneously, producing 30 concurrent connection requests against a pool of 10. The pool blocks, connections queue, and the connectionTimeoutMillis fires before any results return. The search_documents tool returns an empty array. No error. The LLM never knows retrieval failed.

Configure the /ready endpoint to check pool.idleCount > 0 before accepting traffic. When idle connections hit zero, AliveMCP's next probe returns 503. You see the saturation signal before users see empty results. Size the pool at roughly twice your expected peak concurrent sessions to absorb burst overlap.

HNSW indexes have a cold-start problem that is worth knowing before it bites you in production. The HNSW graph is memory-mapped at startup; the first queries traverse an unwarmed graph with high cache-miss rates and return wrong nearest neighbors — not slow results, wrong results. Warm the index on startup with a zero-vector query before accepting traffic: SELECT embedding <=> '[0,0,...,0]' FROM document_chunks ORDER BY 1 LIMIT 1. Gate your /ready endpoint on warmup completion. AliveMCP will not route traffic (in the sense that it will not report healthy) until your server reports ready.

Embedding servers: centralizing the layer that fails silently across all agents

When agents call embedding APIs directly, you get N-fold rate limit exposure, per-agent in-process caches that don't share hits, fragmented cost tracking, and per-agent update surface when you switch models. An MCP embedding server centralizes all of this: agents call generate_embedding({ text: "...", model: "default" }) and receive a vector; the server manages credentials, batches calls, maintains a shared SHA-256 cache, and handles model switching in one place.

The SHA-256 cache is the highest-leverage optimization in the embedding layer. Cache key is SHA256(model:text), stored in SQLite with the embedding as a FLOAT32 binary blob. Cache hits are free — no API cost, ~1ms latency — and the hit rate is high for RAG workloads where the same documentation chunks are embedded repeatedly across indexing jobs. Batch uncached texts in a single API call (up to 2048 inputs per OpenAI batch request) to minimize per-request overhead.

The critical monitoring distinction for an embedding server is the separation between process liveness and embedding API reachability. If the process dies, AliveMCP detects connection_refused immediately. If the embedding API (OpenAI, Cohere, a local model) becomes unavailable but the process is alive, the protocol probe still succeeds — initialize works, tools/list works — but every generate_embedding call silently fails or falls back to a local model.

The fallback correctness issue is subtle and important: if your production corpus was indexed with OpenAI text-embedding-3-small vectors and your fallback under API outage generates vectors with all-MiniLM-L6-v2, the two models live in incompatible vector spaces. The cosine similarities between a local-model query embedding and OpenAI corpus embeddings are meaningless — retrieval returns wrong results rather than no results. The LLM confabulates confidently.

The correct monitoring architecture uses two probes. /live checks the process: can the Node.js event loop respond? Always fast; never fails unless the process is dead. /ready calls the embedding API with a 1-token test input and returns 503 if the call fails or takes more than 2 seconds. Point AliveMCP's custom health check URL at /ready. When the embedding API goes down, AliveMCP fires an alert with failure_reason: external_api_failure — distinguishable from connection_refused (process death) in the same monitoring dashboard, routing to different playbooks.

Context window management: the failure mode measured in tokens, not errors

Tool responses consume the LLM's context window as tool messages. A search_documents call returning 10 chunks at 300 tokens each consumes 3,000 tokens — before the model writes a word of response. A 20-turn agent session with repeated retrieval calls can reach 80,000 tokens, approaching the practical limit of even a 128K-context model. When the context overflows, the LLM silently drops early history. There is no error. The response quality degrades in ways that are hard to distinguish from hallucination.

Token counting must be exact. Character-based estimates are wrong by up to 40% for code and structured data: a 1,000-character JSON blob might be 180 tokens (compact key names) or 350 tokens (verbose keys, nested structure, Unicode characters). Use js-tiktoken in Node.js or tiktoken in Python. Counting 10,000 characters takes under 1ms — the overhead is negligible against the latency of a vector store query.

// Token-budget-aware result assembly
function assembleContext(chunks, tokenBudget = 2000) {
  const parts = [];
  let usedTokens = 0;

  for (const chunk of chunks) {
    const formatted = `[Source: ${chunk.source}]\n${chunk.text}\n`;
    const tokens = countTokens(formatted); // js-tiktoken, not character estimate

    if (usedTokens + tokens > tokenBudget) break; // Stop here, not at some character estimate

    parts.push(formatted);
    usedTokens += tokens;
  }

  return {
    context: parts.join('\n---\n'),
    chunks_included: parts.length,
    tokens_used: usedTokens,
    truncated: parts.length < chunks.length, // Let the agent know it got a partial result
  };
}

The truncated: true field is the key design choice. It tells the agent it received a partial result set and can request a smaller top_k or a more specific query. Without this signal, the agent assumes the retrieval returned all relevant results — which is wrong if the highest-ranked chunks didn't fit in the budget.

Multi-turn deduplication prevents the same chunks from consuming context on every turn. Track which chunk IDs have already been returned in the current session (an in-memory Map<sessionId, Set<chunkId>> for dev; Redis with a 1-hour TTL for production). On each subsequent turn, prioritize new chunks (80% of the token budget) and only re-include previously-seen chunks if they score significantly higher than new candidates. Deduplicated retrieval produces tighter, more informative context across long agent sessions.

The server restart problem: in-memory session state is lost on every restart. A user mid-conversation gets context continuity broken — the next tool call returns chunks the agent has already seen (because the deduplication set is gone), or the agent loses its understanding of what documents it has already read. AliveMCP detects server restarts by noticing the initialize handshake fails and then recovers within 60 seconds of the process coming back up. With a webhook on the restart recovery event, you can notify users that their active sessions have been reset — which is dramatically better than silent context loss.

Semantic caching: the latency savings that create their own monitoring problem

Exact-match caching has a near-100% miss rate for LLM-generated queries. "What is the rate limit policy?" and "how many requests per minute can I make?" are different strings that retrieve the same documentation chunks. Semantic caching catches these paraphrase variants by embedding the incoming query and checking cosine similarity against previous queries in the cache. A cache hit returns the full formatted tool response in ~1ms instead of running 100ms of vector search, API calls, and reranking.

The implementation pattern is a middleware wrapper around the tool handler:

async function withSemanticCache(toolFn, options = {}) {
  const { similarityThreshold = 0.92, ttlSeconds = 3600 } = options;

  return async function cachedTool(args) {
    const { query } = args;
    if (!query) return toolFn(args); // non-query tools bypass the cache

    const queryEmbedding = await embedText(query);
    const cacheHit = await findSimilarCacheEntry(queryEmbedding, similarityThreshold);

    if (cacheHit) return cacheHit.response; // ~1ms — no retrieval cost

    const response = await toolFn(args); // Full retrieval path
    await storeCacheEntry(query, queryEmbedding, response, ttlSeconds);
    return response;
  };
}

The similarity threshold of 0.92 is a starting point, not a law. At 0.99, only near-identical strings hit the cache. At 0.85, semantically related but factually different queries return wrong cached results. Tune the threshold by logging all hits in the 0.90–0.95 similarity band for one week, then sampling whether the cached response was actually correct for the incoming query. A threshold that feels right in testing often proves wrong at production query diversity.

TTL should match data volatility, not a uniform policy:

Document update invalidation requires tagging cache entries by source. When a document is re-indexed, delete all cache entries associated with that source before the new index is available. Without invalidation, the cache serves stale results from the old document even after re-indexing completes — the exact failure mode (wrong answer, HTTP 200, confident LLM) that makes retrieval degradation hard to detect.

Cold-start latency is the semantic cache's monitoring edge case. After a server restart or Redis flush, every query misses the cache. The MCP server serves every request via the full retrieval path — 100ms+ per tool call instead of 1ms cache hits. P95 latency spikes dramatically for 15–30 minutes as the cache warms, then decays back to baseline as hit rate climbs. AliveMCP will observe this latency spike and may fire a P95 alert — which is technically correct but operationally a false positive if the threshold is set for steady-state performance.

The monitoring design for this: alert on sustained P95 elevation (>800ms for more than 20 minutes) rather than any P95 spike. A cold-start spike decays within 30 minutes as the cache warms. A permanent regression from a slow new retrieval path does not decay. The 20-minute window distinguishes the two. This is the only place in the retrieval stack where you want the alert to fire slowly.

The unified monitoring architecture: protocol probes and canary queries

The standard AliveMCP probe (MCP initialize handshake + tools/list) catches five classes of visible failure: connection refused (process dead), protocol error (process alive but MCP initialize fails), timeout (initialize succeeds but takes too long), schema drift (tools/list response changed), and elevated error rate (tool calls returning errors above threshold). For a retrieval MCP server, all five are necessary. None are sufficient.

The retrieval-layer monitoring gap is that the most common failure mode — degraded retrieval quality — produces HTTP 200 responses with empty or stale result sets. The protocol probe sees a healthy server. The LLM sees wrong answers. Closing this gap requires an application-level canary query in your /health endpoint:

app.get('/health', async (req, res) => {
  try {
    // Protocol layer: vector store reachability
    const storeOk = await vectorStore.ping();
    if (!storeOk) {
      return res.status(503).json({ status: 'unhealthy', reason: 'vector_store_unreachable' });
    }

    // Embedding layer: API reachability (if applicable)
    if (useExternalEmbeddingApi) {
      const embedOk = await embeddingApi.ping({ timeout: 2000 });
      if (!embedOk) {
        return res.status(503).json({ status: 'degraded', reason: 'external_api_failure' });
      }
    }

    // Semantic layer: canary query must return non-empty results
    const canaryResult = await searchDocuments({
      query: 'MCP server health check',  // A query you know has indexed results
      top_k: 1,
    });
    const parsed = JSON.parse(canaryResult.content[0].text);
    if (parsed.total_results === 0) {
      return res.status(503).json({ status: 'degraded', reason: 'index_empty_or_stale' });
    }

    // Token budget: count tokens on the canary result
    const tokens = countTokens(parsed.results[0].text);
    if (tokens > 500) { // Unexpectedly large chunk — chunking pipeline issue
      return res.status(503).json({ status: 'degraded', reason: 'chunk_size_anomaly', tokens });
    }

    res.json({ status: 'ok', results_returned: parsed.total_results });
  } catch (err) {
    res.status(503).json({ status: 'unhealthy', reason: err.message });
  }
});

Configure AliveMCP's custom health check URL to point at this endpoint rather than the default MCP protocol probe. This combines three layers of verification into one HTTP call: vector store connectivity, embedding API reachability, and semantic retrieval quality. When any layer degrades, AliveMCP detects it within 60 seconds and routes to the correct alert channel with an appropriate failure_reason.

For the semantic cache, add one more field to the /health response: cache_hit_rate (from a rolling 5-minute window) and cache_warm (true when hit rate > 20%). AliveMCP can use the cache_warm: false signal to suppress P95 latency alerts during the expected cold-start warmup period — you get the cold-start protection without the false-positive storm.

Architecture by team size and use case

Not all five components are required simultaneously. The right architecture depends on corpus size, team size, and the cost of wrong answers.

The monitoring architecture scales differently from the retrieval architecture. A solo dev with Chroma still needs the canary query /health endpoint — the failure mode (empty results, LLM confabulation) is the same whether you have 100 users or 100,000. The only thing that changes is how many people experience wrong answers before you notice. AliveMCP's 60-second external probe runs the same check for a single-developer deployment as for an enterprise fleet. The cost of the monitoring is the same; the blast radius of missing it is not.

Frequently asked questions

How do I know whether my RAG server's retrieval quality is actually degrading in production?

Three signals in combination. First, the canary query in /health: a known-good query that should always return results; if total_results === 0, retrieval has degraded. Second, AliveMCP's P95 latency tracking: if the vector store connection pool is saturating, latency rises before empty results appear — you get early warning before users experience failures. Third, the list_sources tool's last-indexed timestamps: if a source was last indexed 72 hours ago and the underlying documents are updated daily, the retrieval might succeed but return stale facts. Add a staleness check to your /health endpoint for sources that should be re-indexed on a schedule.

Why is semantic caching dangerous with a fallback embedding model?

The semantic cache assumes that cached query vectors and incoming query vectors live in the same embedding space — so cosine similarity between them is meaningful. If your primary model (OpenAI) goes down and your server falls back to a local model (all-MiniLM-L6-v2), the incoming query embedding is in a completely different space from the cache entries indexed under the primary model. The cosine similarities are numerically valid but semantically meaningless. A query about database connection pooling might hit a cached entry about file permissions — both look like infrastructure queries in the local model's space — and the wrong response is returned. The right design is to fail fast on embedding API outage (return 503 from /ready) rather than silently falling back to an incompatible model.

What happens to active agent sessions when the MCP server restarts?

Any in-memory session state is lost: multi-turn deduplication sets, token usage tallies, active session context. The next tool call from the LLM starts fresh — the server doesn't know which chunks the agent has already seen. If you're using Redis-backed session state (the production pattern for multi-turn deduplication), you get continuity across restarts because the deduplication sets survive in Redis. AliveMCP detects the restart by observing the brief period when the initialize handshake fails (the process is not yet listening) followed by recovery. A webhook on the recovery event lets you send a session continuity notification to active users: "Your session context was preserved in our retrieval layer, but the server restarted — some context may need to be re-established."

How do I distinguish a pgvector connection pool saturation from an index problem?

Pool saturation shows up in latency first: P95 rises as connections queue, then calls start timing out and returning empty results. An index problem (corrupted HNSW graph, wrong hnsw:space at collection creation) shows up in result quality immediately with no latency increase — queries return wrong or zero neighbors at normal speed. In your /health endpoint, track both: a pool check (pool.idleCount > 0) returns 503 with reason: "pool_saturated"; the canary query returns 503 with reason: "index_empty_or_stale". AliveMCP reports both as alert events but with different failure_reason fields that route to different runbook playbooks: pool saturation leads you to connection pool sizing; an index issue leads you to re-indexing the corpus.

Is there a minimum viable monitoring setup for a RAG MCP server I can set up in one hour?

Yes. Add one /health endpoint that: (1) pings your vector store, (2) runs a canary search_documents call with a query you know should return at least one result, (3) returns 503 if total_results === 0 or if either step throws. Point AliveMCP's custom health check URL at this endpoint. Add the runbook URL to your AliveMCP alert configuration so that when the 503 fires, the webhook payload already contains the link to the investigation steps. This takes under an hour, costs nothing in AliveMCP's free tier, and covers the failure class that is most likely to go undetected in a RAG deployment: the vector store returning empty results while the MCP protocol layer reports healthy.