MCP Server Claude Vision — image inputs, base64, URL sources, PDF multimodal

Base64 image input from MCP tool arguments

Base64 encoding is the most reliable image delivery method for MCP servers — it works regardless of network configuration, does not require public URLs, and lets you process images from the filesystem, memory, or blob storage. The four supported MIME types are image/jpeg, image/png, image/gif, and image/webp.

import Anthropic from '@anthropic-ai/sdk';
import { z } from 'zod';
import sharp from 'sharp';

const anthropic = new Anthropic();

// Resize an image to a safe resolution before sending to Claude.
// 1,568 px is the maximum dimension that does not incur extra token overhead.
// Without resizing, a 4K screenshot (3840×2160) costs ~6,000 tokens for the image alone.
async function downscaleIfNeeded(input: Buffer): Promise<{ buffer: Buffer; mediaType: string }> {
  const img   = sharp(input);
  const meta  = await img.metadata();
  const MAX_DIM = 1_568;

  const maxDim = Math.max(meta.width ?? 0, meta.height ?? 0);
  const resized = maxDim > MAX_DIM
    ? await img.resize({ width: MAX_DIM, height: MAX_DIM, fit: 'inside' }).jpeg({ quality: 85 }).toBuffer()
    : await img.jpeg({ quality: 90 }).toBuffer();

  return { buffer: resized, mediaType: 'image/jpeg' };
}

server.tool(
  'analyze_screenshot',
  {
    image_base64: z.string().describe('Base64-encoded PNG or JPEG screenshot'),
    question:     z.string().min(1).max(1_000).default('Describe what you see in this image.'),
  },
  async ({ image_base64, question }) => {
    // Decode, downscale, re-encode to JPEG for smaller base64 payload
    const raw = Buffer.from(image_base64, 'base64');
    const { buffer: processed, mediaType } = await downscaleIfNeeded(raw);
    const data = processed.toString('base64');

    const response = await anthropic.messages.create({
      model:      'claude-sonnet-4-6',
      max_tokens: 1024,
      messages: [
        {
          role: 'user',
          content: [
            {
              type:   'image',
              source: { type: 'base64', media_type: mediaType as any, data },
            },
            { type: 'text', text: question },
          ],
        },
      ],
    });

    const text = response.content.find(b => b.type === 'text')?.text ?? '';
    return { content: [{ type: 'text', text }] };
  }
);

The base64 string must not include the data URI prefix (data:image/png;base64,). If the calling agent passes a data URI, strip the prefix before sending: data.replace(/^data:[^;]+;base64,/, ''). Sending the prefix as part of the data field causes a base64 decode error on the Anthropic side.

URL-sourced images and public reachability

URL-sourced images let you avoid base64 encoding entirely — Claude fetches the image server-side during the API call. The URL must be HTTPS and publicly reachable without authentication at the exact moment the API call is made. Pre-signed URLs are safe only if the expiry is long enough to survive typical API call latency (add at least 60 seconds to the expected expiry).

server.tool(
  'analyze_public_image',
  {
    image_url: z.string().url(),
    prompt:    z.string().default('What is shown in this image?'),
  },
  async ({ image_url, prompt }) => {
    // Guard against localhost and private IPs — Claude cannot reach these
    const url = new URL(image_url);
    const PRIVATE_HOSTS = ['localhost', '127.0.0.1', '0.0.0.0', '::1'];
    const isPrivate = PRIVATE_HOSTS.includes(url.hostname) ||
      /^10\.|^192\.168\.|^172\.(1[6-9]|2[0-9]|3[01])\./.test(url.hostname);

    if (isPrivate) {
      return {
        content: [{ type: 'text', text: 'Error: image URL must be publicly accessible. Private/localhost URLs are not supported.' }],
        isError: true,
      };
    }

    const response = await anthropic.messages.create({
      model:      'claude-sonnet-4-6',
      max_tokens: 512,
      messages: [
        {
          role: 'user',
          content: [
            {
              type:   'image',
              source: { type: 'url', url: image_url },
            },
            { type: 'text', text: prompt },
          ],
        },
      ],
    });

    const text = response.content.find(b => b.type === 'text')?.text ?? '';
    return { content: [{ type: 'text', text }] };
  }
);

When Claude fetches a URL-sourced image, it follows redirects (up to a platform-defined limit) and respects standard HTTP caching headers. However, it does not send session cookies, Authorization headers, or other authentication material — purely public URLs only. For authenticated content (internal dashboards, signed CDN URLs), use base64 instead: fetch the image bytes server-side in your MCP tool, then send as base64.

PDF inputs as document blocks

PDFs use the document block type rather than the image type, even though they contain visual content. Claude extracts text, understands layout, tables, and embedded images when processing PDFs. The base64 approach is mandatory for PDFs — there is no URL-source option for documents in the same way as images.

server.tool(
  'analyze_pdf',
  {
    pdf_base64: z.string().describe('Base64-encoded PDF content'),
    question:   z.string().min(1).max(2_000),
  },
  async ({ pdf_base64, question }) => {
    const response = await anthropic.messages.create({
      model:      'claude-sonnet-4-6',
      max_tokens: 2_048,
      messages: [
        {
          role: 'user',
          content: [
            {
              type:   'document',           // NOT 'image' — PDFs are documents
              source: {
                type:       'base64',
                media_type: 'application/pdf',
                data:       pdf_base64,    // no data: URI prefix
              },
            },
            { type: 'text', text: question },
          ],
        },
      ],
    });

    const text = response.content.find(b => b.type === 'text')?.text ?? '';
    return { content: [{ type: 'text', text }] };
  }
);

Multi-image requests and token cost estimation

Multiple images can be included in a single request by adding multiple image blocks to the user content array. The token cost is per-image: each image block costs approximately 1,334–1,601 input tokens depending on dimensions, regardless of the question length. Before sending 5 high-resolution images in one call, estimate the token cost so you can guard against runaway spending.

// Approximate token cost of an image (rough estimate — actual cost varies by model)
// Claude 3 / Sonnet: base 85 tokens + 170 tokens per 512×512 tile
function estimateImageTokens(widthPx: number, heightPx: number): number {
  const tilesW = Math.ceil(widthPx / 512);
  const tilesH = Math.ceil(heightPx / 512);
  return 85 + (170 * tilesW * tilesH);
}

server.tool(
  'compare_screenshots',
  {
    images: z.array(z.object({
      base64:    z.string(),
      label:     z.string().max(100),
      width_px:  z.number().int().min(1),
      height_px: z.number().int().min(1),
    })).min(2).max(5),
    comparison_prompt: z.string().default('Compare these screenshots and highlight the key differences.'),
  },
  async ({ images, comparison_prompt }) => {
    // Estimate total token cost before calling API
    const estimatedImageTokens = images.reduce(
      (sum, img) => sum + estimateImageTokens(img.width_px, img.height_px),
      0,
    );
    const TOKEN_BUDGET = 20_000;
    if (estimatedImageTokens > TOKEN_BUDGET) {
      return {
        content: [{
          type: 'text',
          text: `Estimated ${estimatedImageTokens} image tokens exceeds budget of ${TOKEN_BUDGET}. Reduce image resolution or number of images.`,
        }],
        isError: true,
      };
    }

    const contentBlocks: any[] = images.flatMap(({ base64, label }) => [
      { type: 'text', text: `[${label}]` },
      { type: 'image', source: { type: 'base64', media_type: 'image/jpeg', data: base64 } },
    ]);
    contentBlocks.push({ type: 'text', text: comparison_prompt });

    const response = await anthropic.messages.create({
      model:      'claude-sonnet-4-6',
      max_tokens: 1_024,
      messages:   [{ role: 'user', content: contentBlocks }],
    });

    const text = response.content.find(b => b.type === 'text')?.text ?? '';
    return { content: [{ type: 'text', text }] };
  }
);