MCP server backoff guidance

Backoff-relevant error types in MCP

MCP uses isError: true for all tool-level failures — there is no built-in distinction between a permanent error and a transient one. Your server must encode that distinction in the error payload so callers can decide whether to retry.

Encoding retry hints in error payloads

The MCP protocol does not define a standard retry-after field — you must define your own convention in the content[].text payload. Use a consistent JSON structure so callers can parse it reliably.

// src/rate-limit/error-response.ts
export interface RateLimitError {
  error: 'rate_limited' | 'server_overloaded' | 'transient_error';
  message: string;
  retry_after_ms: number;     // how long the caller should wait before retrying
  retry_after_iso: string;    // ISO 8601 timestamp when the limit resets (human-readable)
  retryable: true;
}

export function makeRateLimitError(retryAfterMs: number): RateLimitError {
  return {
    error: 'rate_limited',
    message: `Rate limit exceeded. Retry after ${Math.ceil(retryAfterMs / 1000)} seconds.`,
    retry_after_ms: retryAfterMs,
    retry_after_iso: new Date(Date.now() + retryAfterMs).toISOString(),
    retryable: true,
  };
}

// In your CallToolRequestSchema handler:
if (!rateLimiter.allow(toolName)) {
  // Calculate how long until the bucket has at least 1 token
  const refillTimeMs = rateLimiter.msUntilToken(toolName); // implement in your limiter
  return {
    content: [{ type: 'text', text: JSON.stringify(makeRateLimitError(refillTimeMs)) }],
    isError: true,
  };
}

Add msUntilToken() to your token bucket to calculate the exact wait time:

// Add to TokenBucket class
msUntilToken(): number {
  this.refill();
  if (this.tokens >= 1) return 0;
  const deficit = 1 - this.tokens;
  return Math.ceil((deficit / this.refillRate) * 1000); // ms
}

Exponential backoff with jitter — client-side implementation

Callers should implement exponential backoff with full jitter. "Full jitter" means the wait time is a random value between 0 and the exponential cap, rather than a deterministic exponential. This prevents thundering herd: if 100 clients all hit the same rate limit at the same time and all wait exactly 2 seconds, they will all hammer the server simultaneously again 2 seconds later. With full jitter, they spread out across the 0–2 second window.

// Backoff utility for MCP tool callers
interface BackoffOptions {
  baseMs?: number;      // initial wait (default 200ms)
  maxMs?: number;       // cap on total wait (default 30s)
  maxAttempts?: number; // give up after this many retries (default 5)
  jitterType?: 'full' | 'equal' | 'decorrelated';
}

function computeBackoff(attempt: number, options: BackoffOptions = {}): number {
  const { baseMs = 200, maxMs = 30_000, jitterType = 'full' } = options;
  const exponential = baseMs * Math.pow(2, attempt);
  const capped = Math.min(maxMs, exponential);

  switch (jitterType) {
    case 'full':
      // Random between 0 and the capped exponential — best for thundering herd prevention
      return Math.random() * capped;
    case 'equal':
      // Half deterministic, half random — retries are not too short and not synchronized
      return capped / 2 + Math.random() * (capped / 2);
    case 'decorrelated':
      // Each attempt's wait is based on the previous attempt's wait
      // Not shown here — see AWS blog for implementation
      return Math.random() * capped;
  }
}

async function callToolWithRetry(
  client: MCPClient,
  toolName: string,
  args: Record<string, unknown>,
  options: BackoffOptions = {}
): Promise<MCPToolResult> {
  const { maxAttempts = 5 } = options;

  for (let attempt = 0; attempt < maxAttempts; attempt++) {
    const result = await client.callTool({ name: toolName, arguments: args });

    if (!result.isError) return result;

    // Parse the error payload
    let errorPayload: Record<string, unknown> = {};
    try {
      const text = (result.content as Array<{ type: string; text?: string }>)
        .find(c => c.type === 'text')?.text ?? '{}';
      errorPayload = JSON.parse(text);
    } catch { /* non-JSON error */ }

    // Don't retry non-retryable errors
    if (errorPayload.retryable !== true) {
      throw new Error(`Tool error (non-retryable): ${JSON.stringify(errorPayload)}`);
    }

    if (attempt === maxAttempts - 1) {
      throw new Error(`Tool failed after ${maxAttempts} attempts: ${JSON.stringify(errorPayload)}`);
    }

    // Use server-provided retry_after_ms if available, else compute our own backoff
    const serverHint = typeof errorPayload.retry_after_ms === 'number'
      ? errorPayload.retry_after_ms
      : null;
    const backoffMs = serverHint !== null
      ? serverHint + Math.random() * 200  // add 0–200ms jitter to server hint
      : computeBackoff(attempt, options);

    console.log(`Tool '${toolName}' rate limited. Retrying in ${Math.round(backoffMs)}ms (attempt ${attempt + 1}/${maxAttempts})`);
    await new Promise(resolve => setTimeout(resolve, backoffMs));
  }

  throw new Error('unreachable');
}

Jitter comparison

Not all jitter strategies perform equally under load. The table below compares the three common approaches for MCP clients that might have multiple agents calling the same server simultaneously.

For most MCP use cases, full jitter is the right default. The only time to prefer equal jitter is when retrying too quickly would cause a thundering herd at zero delay (e.g., all clients finishing at the same time and all retrying immediately). In those cases, the guaranteed minimum wait from equal jitter helps.

Communicating retry guidance in tool descriptions

LLM clients that receive an isError: true response may decide to retry on their own without consulting your retry-after hint. Guide the model explicitly in the tool description:

server.setRequestHandler(ListToolsRequestSchema, async () => ({
  tools: [
    {
      name: 'search_documents',
      description: [
        'Search documents by keyword. Returns matching document excerpts.',
        'If this tool returns an error with "error":"rate_limited", wait for the',
        'number of milliseconds specified in "retry_after_ms" before calling again.',
        'Do not retry more than 3 times.',
      ].join(' '),
      inputSchema: { /* ... */ },
    },
  ],
}));

This is belt-and-suspenders guidance: your retry wrapper in the calling code handles most retries automatically, but the model description prevents the model itself from hammering the tool in a loop when the wrapper isn't in place.

Related questions

Further reading

• MCP server rate limiting — token bucket and sliding window implementation
• MCP server per-tool rate limiting — per-tool token buckets
• MCP server client throttling — per-identity fair queuing
• MCP server circuit breaker — stopping cascading failures
• MCP server timeout — deadline propagation