MCP Server Bedrock Converse API — model IDs, InferenceConfiguration, streaming, throttling

Why Converse API instead of InvokeModel

The Bedrock API surface has two entry points for text generation: InvokeModel and Converse. InvokeModel sends a raw byte body to the model and receives a raw byte body back — the schema of that body is model-specific. Anthropic Claude uses one JSON structure, Amazon Titan uses another, Meta Llama uses a third. When an MCP server needs to support multiple foundation models — or needs to swap models based on cost, latency, or capability — InvokeModel requires a separate serialization/deserialization path for every model family.

The Converse API normalizes the request and response format across all supported models. You send a messages array with role and content, an optional system array, and an optional toolConfig. The API handles model-specific translation internally. The response always contains an output.message with a structured content array of typed blocks: TextBlock, ToolUseBlock, ImageBlock, DocumentBlock.

For MCP tool handlers this matters concretely: you can expose a single bedrock_generate tool that accepts a modelId parameter, and the same handler code runs against Claude 3.5 Sonnet, Llama 3.1 70B, and Command R+ without modification. The only model-specific code is InferenceConfiguration validation, covered in the next section.

import {
  BedrockRuntimeClient,
  ConverseCommand,
  type ConverseCommandInput,
  type Message,
} from '@aws-sdk/client-bedrock-runtime';

const client = new BedrockRuntimeClient({ region: process.env.AWS_REGION ?? 'us-east-1' });

// Minimum IAM permissions:
// bedrock:InvokeModel on the specific model ARN, OR
// bedrock:Converse on arn:aws:bedrock:REGION::foundation-model/*
// For cross-region inference profiles: bedrock:InvokeModel on the profile ARN

async function bedrockConverse(
  modelId: string,
  userMessage: string,
  systemPrompt?: string,
): Promise {
  const messages: Message[] = [
    { role: 'user', content: [{ text: userMessage }] },
  ];

  const input: ConverseCommandInput = {
    modelId,
    messages,
    system: systemPrompt ? [{ text: systemPrompt }] : undefined,
    inferenceConfig: {
      maxTokens: 2048,
      temperature: 0.3,
    },
  };

  const response = await client.send(new ConverseCommand(input));

  const outputMessage = response.output?.message;
  if (!outputMessage) throw new Error('No output message from Bedrock Converse');

  const textBlocks = outputMessage.content
    ?.filter(block => 'text' in block)
    .map(block => (block as { text: string }).text);

  return textBlocks?.join('') ?? '';
}

The stopReason field on the response is the primary signal for what the model did: end_turn means the model finished naturally, tool_use means it emitted one or more ToolUseBlock entries and is waiting for results, max_tokens means the output was truncated at maxTokens (increase it or handle the partial response), and stop_sequence means a custom stop string was matched. Always check stopReason — silently ignoring max_tokens is the most common source of truncated agent responses.

Model IDs: on-demand vs cross-region inference profiles

Bedrock model IDs come in two formats that behave differently for throughput and routing. Choosing the wrong one causes either hard 429s under load or wasted latency budget from unnecessary cross-region hops.

On-demand model IDs are single-region identifiers: anthropic.claude-3-5-sonnet-20241022-v2:0. They route to the model in exactly the region your BedrockRuntimeClient is configured for. Throughput is bounded by that region's on-demand quota. If the regional quota is exhausted you get ThrottlingException with no automatic failover. Use on-demand IDs only when you have provisioned throughput in that region or your request rate is well below the default quota.

Cross-region inference profiles use a two-letter prefix that identifies the inference profile geography: us.anthropic.claude-3-5-sonnet-20241022-v2:0 (US), eu.anthropic.claude-3-5-haiku-20241022-v1:0 (EU), ap.anthropic.claude-3-5-sonnet-20241022-v2:0 (Asia Pacific). These route across all enabled regions in the geography, giving you access to the combined quota. They add 5–50 ms of routing overhead on a cache-cold request. For MCP servers where a single slow tool call stalls the agent loop, the combined quota headroom is worth the small latency premium.

// On-demand IDs — regional, bounded by that region's quota
const MODELS = {
  claudeSonnet35:        'anthropic.claude-3-5-sonnet-20241022-v2:0',
  claudeHaiku35:         'anthropic.claude-3-5-haiku-20241022-v1:0',
  claudeHaiku3:          'anthropic.claude-3-haiku-20240307-v1:0',
  titanPremier:          'amazon.titan-text-premier-v1:0',
  llama31_70b:           'meta.llama3-1-70b-instruct-v1:0',
  mistralLarge2:         'mistral.mistral-large-2402-v1:0',
} as const;

// Cross-region inference profiles — US geography, pooled quota across us-east-1/us-west-2
const US_INFERENCE_PROFILES = {
  claudeSonnet35:        'us.anthropic.claude-3-5-sonnet-20241022-v2:0',
  claudeHaiku35:         'us.anthropic.claude-3-5-haiku-20241022-v1:0',
  claudeSonnet4:         'us.anthropic.claude-sonnet-4-5-20251001-v1:0',
  llama31_70b:           'us.meta.llama3-1-70b-instruct-v1:0',
} as const;

// To check which models support Converse API with tool use in the current region:
import { BedrockClient, ListFoundationModelsCommand } from '@aws-sdk/client-bedrock';
async function listConverseToolModels(region: string) {
  const bedrock = new BedrockClient({ region });
  const { modelSummaries } = await bedrock.send(new ListFoundationModelsCommand({}));
  return modelSummaries
    ?.filter(m => m.responseStreamingSupported && m.inputModalities?.includes('TEXT'))
    .map(m => m.modelId);
}

Cross-region inference profiles require the bedrock:InvokeModel permission on the inference profile ARN, not the foundation model ARN. The ARN format is arn:aws:bedrock:us-east-1::inference-profile/us.anthropic.claude-3-5-sonnet-20241022-v2:0. If you use wildcard ARNs (arn:aws:bedrock:*::foundation-model/*) in your IAM policy, also add arn:aws:bedrock:*::inference-profile/* or inference profile calls will get 403s.

InferenceConfiguration: per-model valid ranges and gotchas

InferenceConfiguration passes sampling parameters to the model. The Converse API forwards them as-is to the model backend, which enforces its own validation. The validation error surfaces as a ValidationException with a message like "Value at 'inferenceConfig.topP' failed to satisfy constraint" — but only at request time, not when you build the config object. This means misconfigured parameters silently pass type checking but fail at runtime.

The key fields and per-model behavior:

The Titan topP-when-temperature-zero gotcha: Titan Premier's documentation states that when temperature is 0, sampling is effectively greedy and topP has no effect — but the model does not error; it silently ignores topP. If you copy a Claude config that uses temperature: 0, topP: 0.9 to a Titan call, Titan will respond without error but topP does nothing. Claude and Llama apply both parameters simultaneously; they interact: low temperature narrows the token distribution before topP applies, so temperature: 0.2, topP: 0.9 behaves very differently from temperature: 0.8, topP: 0.9.

// Type-safe InferenceConfiguration per model family
type ModelFamily = 'claude' | 'titan' | 'llama' | 'mistral';

function safeInferenceConfig(
  family: ModelFamily,
  opts: { maxTokens?: number; temperature?: number; topP?: number },
) {
  const base = { maxTokens: opts.maxTokens ?? 1024 };

  const temperature = Math.min(1, Math.max(0, opts.temperature ?? 0.5));
  const topP        = Math.min(1, Math.max(0, opts.topP ?? 1.0));

  switch (family) {
    case 'claude':
      return { ...base, temperature, topP };
    case 'titan':
      // Titan ignores topP when temperature is 0; omit it to avoid confusion
      return temperature === 0 ? { ...base, temperature } : { ...base, temperature, topP };
    case 'llama':
    case 'mistral':
      return { ...base, temperature, topP };
  }
}

// Usage in MCP tool handler
const inferenceConfig = safeInferenceConfig('claude', {
  maxTokens: 4096,
  temperature: 0.3,
  topP: 0.95,
});

Streaming with converseStream — parsing ConverseStreamOutput

The ConverseStreamCommand returns a response stream that emits a sequence of typed events. The events arrive as a ConverseStreamOutput async iterable — each element is a union type with a discriminated member indicating which event type it is. Failing to handle all union variants correctly is the most common source of dropped chunks or silently missed stop reasons.

The event sequence for a non-tool response is: messageStart (role), one or more contentBlockStart / contentBlockDelta / contentBlockStop triples for each content block, then messageStop with the stopReason. For a tool-use response, the content blocks include a toolUse block whose input accumulates across multiple contentBlockDelta events as JSON fragments — you must concatenate the fragments and parse the complete JSON when contentBlockStop arrives.

import {
  ConverseStreamCommand,
  type ConverseStreamOutput,
  type ContentBlockDeltaEvent,
  type ContentBlockStartEvent,
  type ContentBlockStopEvent,
  type MessageStopEvent,
} from '@aws-sdk/client-bedrock-runtime';

interface StreamResult {
  text: string;
  stopReason: string;
  toolUses: Array<{ toolUseId: string; name: string; input: unknown }>;
}

async function converseStream(
  modelId: string,
  messages: Array<{ role: 'user' | 'assistant'; content: Array<{ text: string }> }>,
  onChunk?: (text: string) => void,
): Promise {
  const response = await client.send(new ConverseStreamCommand({
    modelId,
    messages,
    inferenceConfig: { maxTokens: 4096, temperature: 0.3 },
  }));

  if (!response.stream) throw new Error('No stream in ConverseStream response');

  let fullText = '';
  let stopReason = '';
  const toolUses: StreamResult['toolUses'] = [];

  // Per-block accumulation state
  const blockAccumulators = new Map();

  for await (const event of response.stream as AsyncIterable) {
    if ('contentBlockStart' in event) {
      const e = event.contentBlockStart as ContentBlockStartEvent;
      const idx = e.contentBlockIndex ?? 0;
      const start = e.start;

      if (start && 'toolUse' in start && start.toolUse) {
        blockAccumulators.set(idx, {
          type: 'toolUse',
          toolUseId: start.toolUse.toolUseId,
          name: start.toolUse.name,
          inputJson: '',
        });
      } else {
        blockAccumulators.set(idx, { type: 'text', text: '' });
      }
    } else if ('contentBlockDelta' in event) {
      const e = event.contentBlockDelta as ContentBlockDeltaEvent;
      const idx = e.contentBlockIndex ?? 0;
      const acc = blockAccumulators.get(idx);
      const delta = e.delta;
      if (!acc || !delta) continue;

      if (acc.type === 'text' && 'text' in delta) {
        acc.text = (acc.text ?? '') + (delta as { text: string }).text;
        onChunk?.((delta as { text: string }).text);
      } else if (acc.type === 'toolUse' && 'toolUse' in delta) {
        // delta.toolUse.input is a JSON fragment string
        const fragment = (delta as { toolUse: { input: string } }).toolUse.input;
        acc.inputJson = (acc.inputJson ?? '') + (fragment ?? '');
      }
    } else if ('contentBlockStop' in event) {
      const e = event.contentBlockStop as ContentBlockStopEvent;
      const idx = e.contentBlockIndex ?? 0;
      const acc = blockAccumulators.get(idx);
      if (!acc) continue;

      if (acc.type === 'text') {
        fullText += acc.text ?? '';
      } else if (acc.type === 'toolUse') {
        toolUses.push({
          toolUseId: acc.toolUseId!,
          name: acc.name!,
          input: JSON.parse(acc.inputJson || '{}'),
        });
      }
      blockAccumulators.delete(idx);
    } else if ('messageStop' in event) {
      const e = event.messageStop as MessageStopEvent;
      stopReason = e.stopReason ?? '';
    }
    // Also handle 'metadata' events (token usage, latency) if needed — safe to ignore
  }

  return { text: fullText, stopReason, toolUses };
}

One non-obvious behavior: when the model produces both text and a tool call in the same turn (rare but valid for some models), both block types appear in the same stream. The block index (contentBlockIndex) is the stable identifier — do not assume text always has index 0 or that tool use always comes last. The accumulator map keyed on block index handles this correctly.

Tool use with ConverseToolConfig and ToolInputSchema

The Converse API supports tool use through the toolConfig field. Each tool is described by a ToolSpec that contains a name, description, and inputSchema — a JSON Schema wrapped in a json key. When the model decides to call a tool, it returns a ToolUseBlock in the response content. Your MCP tool handler runs the tool, then injects a ToolResultBlock back into the messages array and calls Converse again. The loop continues until stopReason is end_turn.

import {
  ConverseCommand,
  type ToolConfiguration,
  type Message,
} from '@aws-sdk/client-bedrock-runtime';

// Define tools using ConverseToolConfig
const toolConfig: ToolConfiguration = {
  tools: [
    {
      toolSpec: {
        name: 'get_weather',
        description: 'Get current weather conditions for a city.',
        inputSchema: {
          json: {
            type: 'object',
            properties: {
              city:    { type: 'string', description: 'City name' },
              country: { type: 'string', description: 'ISO 3166-1 alpha-2 country code' },
            },
            required: ['city'],
          },
        },
      },
    },
    {
      toolSpec: {
        name: 'search_knowledge_base',
        description: 'Search the internal knowledge base for relevant documents.',
        inputSchema: {
          json: {
            type: 'object',
            properties: {
              query:      { type: 'string', description: 'Search query' },
              max_results: { type: 'number', description: 'Maximum number of results to return', default: 5 },
            },
            required: ['query'],
          },
        },
      },
    },
  ],
  // toolChoice omitted — defaults to 'auto' (model decides when to use tools)
  // toolChoice: { auto: {} }     — explicit auto
  // toolChoice: { any: {} }      — force at least one tool call
  // toolChoice: { tool: { name: 'get_weather' } } — force a specific tool
};

// Agentic loop with tool use
async function converseWithTools(
  modelId: string,
  userQuery: string,
  toolDispatcher: (name: string, input: unknown) => Promise,
): Promise {
  const messages: Message[] = [
    { role: 'user', content: [{ text: userQuery }] },
  ];

  for (let turn = 0; turn < 10; turn++) {
    const response = await client.send(new ConverseCommand({
      modelId,
      messages,
      toolConfig,
      inferenceConfig: { maxTokens: 4096 },
    }));

    const outputMessage = response.output?.message!;
    messages.push(outputMessage);

    if (response.stopReason === 'end_turn') {
      const textBlocks = outputMessage.content
        ?.filter(b => 'text' in b)
        .map(b => (b as { text: string }).text);
      return textBlocks?.join('') ?? '';
    }

    if (response.stopReason === 'tool_use') {
      const toolResults: Message['content'] = [];

      for (const block of outputMessage.content ?? []) {
        if (!('toolUse' in block)) continue;
        const toolUse = (block as { toolUse: { toolUseId: string; name: string; input: unknown } }).toolUse;

        let resultContent: string;
        let status: 'success' | 'error' = 'success';
        try {
          resultContent = await toolDispatcher(toolUse.name, toolUse.input);
        } catch (err) {
          resultContent = `Tool error: ${(err as Error).message}`;
          status = 'error';
        }

        toolResults.push({
          toolResult: {
            toolUseId: toolUse.toolUseId,
            content: [{ text: resultContent }],
            status,
          },
        } as never);
      }

      messages.push({ role: 'user', content: toolResults });
      continue;
    }

    // Handle max_tokens, stop_sequence, content_filtered
    throw new Error(`Unexpected stopReason: ${response.stopReason}`);
  }

  throw new Error('Max turns exceeded in tool use loop');
}

Throttling: ThrottlingException, backoff, and provisioned throughput

Bedrock on-demand throughput is quota-limited per model per region. When you exceed the tokens-per-minute (TPM) or requests-per-minute (RPM) quota, Bedrock returns a ThrottlingException with HTTP 429. The exception is retryable — the AWS SDK v3's default retry handler retries it three times with exponential backoff, but the default backoff is often too aggressive for sustained load.

Two categories of throttling you will encounter in MCP server tool handlers running agentic loops:

Burst throttling: A single large agent run triggers many tool calls in rapid succession. Each tool call generates a Converse API request. Five simultaneous MCP client sessions each running a 5-step agent loop can send 25 requests/minute to the same model. At Claude 3.5 Sonnet's default quota of 50 RPM in us-east-1 this is borderline; at 10 simultaneous sessions it becomes a steady source of 429s.

Sustained throttling: Your MCP server is called by downstream systems that themselves run at high request rates. The Bedrock quota is shared across all calls from the same AWS account to the same model — including calls from other services, Lambda functions, and Bedrock Agents in the same account.

import { ThrottlingException } from '@aws-sdk/client-bedrock-runtime';

async function converseWithBackoff(
  input: ConverseCommandInput,
  maxAttempts = 8,
): Promise>> {
  let attempt = 0;
  let delay = 1000; // start at 1 second

  while (true) {
    try {
      return await client.send(new ConverseCommand(input));
    } catch (err) {
      if (!(err instanceof ThrottlingException)) throw err;
      if (attempt >= maxAttempts) throw err;

      // Exponential backoff with full jitter (best for reducing collision)
      const jitter = Math.random() * delay;
      const waitMs = Math.min(delay + jitter, 60_000);

      console.warn(
        `ThrottlingException on attempt ${attempt + 1}/${maxAttempts}. ` +
        `Retrying in ${Math.round(waitMs)}ms`,
      );

      await new Promise(r => setTimeout(r, waitMs));
      delay = Math.min(delay * 2, 30_000); // cap base at 30s
      attempt++;
    }
  }
}

// For provisioned throughput: use a model unit ARN instead of a model ID
// Provisioned throughput ARN format:
// arn:aws:bedrock:us-east-1:123456789012:provisioned-model/abc123xyz
// With provisioned throughput: no ThrottlingException below the provisioned RPM/TPM;
// requests above the provisioned capacity are rejected (not queued).
const PROVISIONED_MODEL_ARN = process.env.BEDROCK_PROVISIONED_MODEL_ARN;
const MODEL_ID = PROVISIONED_MODEL_ARN || 'us.anthropic.claude-3-5-sonnet-20241022-v2:0';

Model latency differences matter when designing the retry budget. Claude 3.5 Haiku is 2–4x faster than Claude 3.5 Sonnet on short prompts; Sonnet is 2–3x faster than Opus on most workloads. In a 30-second MCP tool call timeout budget, a Haiku call (median 800 ms) leaves room for 3 retries; a Sonnet call (median 3–6 s for a 1k-token response) leaves room for 1 retry. Size your timeout and retry budget to the specific model, not a generic value.

Common failure modes