Bedrock Multimodal in MCP Servers — image analysis, document processing, Claude Vision

Supported models for multimodal inputs

The safe default for production multimodal MCP tools is Claude 3.5 Sonnet or Claude Sonnet 4.x — both support images, all document formats, and have the strongest multimodal reasoning capabilities. Use Claude 3.5 Haiku for high-volume, cost-sensitive image tasks where response latency matters more than reasoning depth.

Sending images via inline bytes

The most straightforward approach is reading the image into memory and sending it as base64-encoded bytes in the request. This works for images up to 5MB and does not require any S3 setup.

import {
  BedrockRuntimeClient,
  ConverseCommand,
  type Message,
  type ContentBlock,
} from '@aws-sdk/client-bedrock-runtime';
import { readFileSync } from 'fs';
import { extname } from 'path';

const client = new BedrockRuntimeClient({ region: 'us-east-1' });

type ImageFormat = 'jpeg' | 'png' | 'gif' | 'webp';

function getImageFormat(filePath: string): ImageFormat {
  const ext = extname(filePath).slice(1).toLowerCase();
  const formats: Record = {
    jpg: 'jpeg', jpeg: 'jpeg', png: 'png', gif: 'gif', webp: 'webp',
  };
  const format = formats[ext];
  if (!format) throw new Error(`Unsupported image format: ${ext}`);
  return format;
}

async function analyzeImage(
  imagePath: string,
  prompt: string,
  systemPrompt?: string,
): Promise {
  const imageBytes = readFileSync(imagePath);
  const format = getImageFormat(imagePath);

  // Validate size: 5MB limit
  if (imageBytes.length > 5 * 1024 * 1024) {
    throw new Error(`Image ${imagePath} exceeds 5MB limit (${imageBytes.length} bytes)`);
  }

  const imageBlock: ContentBlock = {
    image: {
      format,
      source: {
        bytes: imageBytes,  // Uint8Array — SDK serializes to base64 automatically
      },
    },
  } as ContentBlock;

  const messages: Message[] = [
    {
      role: 'user',
      content: [
        imageBlock,
        { text: prompt },
      ],
    },
  ];

  const response = await client.send(new ConverseCommand({
    modelId: 'us.anthropic.claude-3-5-sonnet-20241022-v2:0',
    messages,
    system: systemPrompt ? [{ text: systemPrompt }] : undefined,
    inferenceConfig: { maxTokens: 2048 },
  }));

  const textBlocks = response.output?.message?.content
    ?.filter(b => 'text' in b)
    .map(b => (b as { text: string }).text);

  return textBlocks?.join('') ?? '';
}

// Example MCP tool: analyze a monitoring dashboard screenshot
// Tool input: { imagePath: string, question: string }
async function analyzeDashboardScreenshot(
  imagePath: string,
  question: string,
): Promise {
  return analyzeImage(
    imagePath,
    question,
    'You are an expert at analyzing monitoring dashboards and uptime graphs. '
    + 'Identify anomalies, downtime periods, and trends from visual charts.',
  );
}

The order of blocks in the content array matters: the model processes blocks in order. Place the image before the text prompt for tasks where the image is the subject of the question. For multi-image tasks (compare these two screenshots), place images in sequence before the analysis prompt.

Sending images from S3

For MCP servers that store images in S3 — common for audit screenshots, report attachments, or user-uploaded content — Bedrock can fetch images directly via the s3Location source. This avoids loading the image into the MCP server's memory and is faster for large images.

// S3 source: Bedrock fetches the image directly during inference
// Requires: the IAM role executing the Bedrock call must have s3:GetObject
// on the bucket. Bedrock does NOT use a separate service role for S3 reads
// when using the Converse API — it uses the caller's credentials.

async function analyzeImageFromS3(
  s3Uri: string,        // e.g. s3://my-bucket/screenshots/dashboard-2026-11-02.png
  format: ImageFormat,
  prompt: string,
): Promise {
  // Parse S3 URI
  const [, , bucket, ...keyParts] = s3Uri.split('/');
  const key = keyParts.join('/');

  const imageBlock: ContentBlock = {
    image: {
      format,
      source: {
        s3Location: {
          uri: s3Uri,
          bucketOwner: process.env.AWS_ACCOUNT_ID,  // required for cross-account buckets
        },
      },
    },
  } as ContentBlock;

  const response = await client.send(new ConverseCommand({
    modelId: 'us.anthropic.claude-3-5-sonnet-20241022-v2:0',
    messages: [{
      role: 'user',
      content: [imageBlock, { text: prompt }],
    }],
    inferenceConfig: { maxTokens: 1024 },
  }));

  return response.output?.message?.content
    ?.filter(b => 'text' in b)
    .map(b => (b as { text: string }).text)
    .join('') ?? '';
}

// IAM policy addition required for S3 source images:
// {
//   "Effect": "Allow",
//   "Action": ["s3:GetObject"],
//   "Resource": "arn:aws:s3:::your-bucket/*"
// }

S3 source images must be in an S3 bucket in the same AWS partition as the Bedrock endpoint (aws, aws-cn, aws-us-gov). Cross-region S3 access works but adds latency. The bucket must allow reads from the calling IAM principal — bucket policies and IAM policies both apply. If using S3 source images from Lambda or ECS, ensure the execution role has s3:GetObject on the image bucket; if using S3 source from a local development environment, your local AWS credentials need the same permission.

Document processing: PDF, DOCX, XLSX, CSV

The DocumentBlock type handles non-image documents. Bedrock extracts text and structure from these formats and presents them to the model as semantic content — not as images. This means the model can reason about tables in Excel, paragraph structure in Word documents, and page layout in PDFs without needing to "see" the rendered pixels.

type DocumentFormat = 'pdf' | 'docx' | 'xlsx' | 'xls' | 'csv' | 'txt' | 'md' | 'html';

async function analyzeDocument(
  documentPath: string,
  format: DocumentFormat,
  question: string,
): Promise {
  const docBytes = readFileSync(documentPath);

  // Document size limits: 4.5MB for PDF/DOCX, 4.5MB for XLSX, 50 pages max for PDF
  if (docBytes.length > 4.5 * 1024 * 1024) {
    throw new Error(`Document exceeds 4.5MB size limit`);
  }

  const documentBlock: ContentBlock = {
    document: {
      format,
      name: documentPath.split('/').pop()!,   // filename for model context
      source: {
        bytes: docBytes,
      },
    },
  } as ContentBlock;

  const response = await client.send(new ConverseCommand({
    modelId: 'us.anthropic.claude-3-5-sonnet-20241022-v2:0',
    messages: [{
      role: 'user',
      content: [documentBlock, { text: question }],
    }],
    inferenceConfig: { maxTokens: 4096 },
  }));

  return response.output?.message?.content
    ?.filter(b => 'text' in b)
    .map(b => (b as { text: string }).text)
    .join('') ?? '';
}

// Example: extract structured data from a CSV of MCP endpoint health reports
async function parseHealthReport(csvPath: string): Promise {
  return analyzeDocument(
    csvPath,
    'csv',
    'Parse this health report CSV and return a JSON array of objects with '
    + 'fields: endpoint, uptime_percent, avg_response_ms, status. '
    + 'Return only the JSON array, no surrounding text.',
  );
}

// Example: extract configuration from a PDF architecture diagram doc
async function extractArchitectureFromPdf(pdfPath: string): Promise {
  return analyzeDocument(
    pdfPath,
    'pdf',
    'List all MCP server endpoints, their protocols, and dependencies '
    + 'mentioned in this architecture document.',
  );
}

For PDFs specifically, Bedrock processes up to 50 pages. Multi-page PDFs beyond 50 pages are truncated — the model will only see the first 50 pages. For longer documents, split the PDF before sending, or extract the relevant sections. CSV and plain text formats have no page limit, only the overall size limit.

Image token pricing and cost estimation

Images are charged as additional input tokens. The token count for an image depends on its pixel dimensions. For Claude models, Bedrock uses the same token approximation as the Anthropic API: approximately (width × height) / 750 image tokens for images that are not resized before processing. Claude automatically downsamples images larger than 1568 pixels on the longest edge.

// Token cost estimation for images (Claude approximation)
function estimateImageTokens(widthPx: number, heightPx: number): number {
  // Claude automatically resizes: max 1568px on longest edge
  const maxDim = 1568;
  let w = widthPx;
  let h = heightPx;
  if (Math.max(w, h) > maxDim) {
    const scale = maxDim / Math.max(w, h);
    w = Math.round(w * scale);
    h = Math.round(h * scale);
  }
  return Math.ceil((w * h) / 750);
}

// Examples:
// 1024×1024 → 1398 tokens
// 1920×1080 (1568 resize applies) → 1568×882 → 1847 tokens
// 640×480 → 410 tokens
// 100×100 (tiny icon) → 14 tokens

// At Claude 3.5 Sonnet pricing (~$3/M input tokens):
// 1024×1024 image: 1398 tokens × $0.000003 = $0.0042/image
// 20 images per request: ~$0.084 in image tokens + text tokens
console.log('1024×1024 image tokens:', estimateImageTokens(1024, 1024));
console.log('1920×1080 image tokens:', estimateImageTokens(1920, 1080));

Amazon Nova models use megapixel-based pricing rather than the token-count approximation: Nova Lite charges $0.000060/image token (0.006 cents/image) and Nova Pro charges $0.001/megapixel. For high-volume image analysis where cost is a constraint, Nova Lite is significantly cheaper than Claude Sonnet for tasks that don't require deep reasoning — screenshot classification, label extraction, OCR-adjacent tasks.

Common errors and solutions

Multi-turn conversation with images

Images can appear in any user message turn in a multi-turn conversation. The model retains visual context from previous turns — you can ask follow-up questions about an image sent in turn 1 without resending the image in turn 2.

// Multi-turn conversation about a monitoring dashboard
async function analyzeWithFollowUp(
  imagePath: string,
): Promise<{ initialAnalysis: string; followUpAnswer: string }> {
  const imageBytes = readFileSync(imagePath);
  const messages: Message[] = [
    {
      role: 'user',
      content: [
        { image: { format: 'png', source: { bytes: imageBytes } } } as ContentBlock,
        { text: 'Describe what this monitoring dashboard shows.' },
      ],
    },
  ];

  // First turn
  const response1 = await client.send(new ConverseCommand({
    modelId: 'us.anthropic.claude-3-5-sonnet-20241022-v2:0',
    messages,
    inferenceConfig: { maxTokens: 1024 },
  }));

  const analysis = response1.output?.message?.content
    ?.filter(b => 'text' in b)
    .map(b => (b as { text: string }).text)
    .join('') ?? '';

  // Add assistant response to conversation history
  messages.push(response1.output!.message!);

  // Follow-up question — no need to resend the image
  messages.push({
    role: 'user',
    content: [{ text: 'What time period shows the most downtime in this chart?' }],
  });

  const response2 = await client.send(new ConverseCommand({
    modelId: 'us.anthropic.claude-3-5-sonnet-20241022-v2:0',
    messages,
    inferenceConfig: { maxTokens: 512 },
  }));

  const followUpAnswer = response2.output?.message?.content
    ?.filter(b => 'text' in b)
    .map(b => (b as { text: string }).text)
    .join('') ?? '';

  return { initialAnalysis: analysis, followUpAnswer };
}

Important: when images are included in earlier turns of the conversation history, those images remain in the context window and continue consuming image tokens on each subsequent inference call. For a 10-turn conversation with a 1024×1024 image in turn 1, every subsequent turn charges approximately 1400 extra image tokens on top of the growing text context. For long conversations about images, extract the relevant text description in the first turn and discard the image from subsequent turns if memory or cost is a constraint.