AWS Rekognition DetectLabels in MCP Servers — Object Detection, Confidence, Bounding Boxes

Basic tool implementation

import { RekognitionClient, DetectLabelsCommand } from '@aws-sdk/client-rekognition';
import { McpServer } from '@modelcontextprotocol/sdk/server/mcp.js';
import { z } from 'zod';

const rekognition = new RekognitionClient({ region: 'us-east-1' });
const server = new McpServer({ name: 'vision-tools', version: '1.0.0' });

server.tool('detect_image_labels', {
  // Accept either an S3 URI (s3://bucket/key) or base64-encoded image bytes
  image_source: z.union([
    z.object({ s3_uri: z.string().regex(/^s3:\/\//) }),
    z.object({ base64: z.string(), media_type: z.enum(['image/jpeg', 'image/png', 'image/webp', 'image/gif']) }),
  ]),
  min_confidence: z.number().min(0).max(100).default(70),
  max_labels: z.number().int().min(1).max(1000).default(50),
}, async ({ image_source, min_confidence, max_labels }) => {
  const image =
    'base64' in image_source
      ? { Bytes: Buffer.from(image_source.base64, 'base64') }
      : (() => {
          const url = new URL(image_source.s3_uri);
          return { S3Object: { Bucket: url.hostname, Name: url.pathname.slice(1) } };
        })();

  const { Labels = [] } = await rekognition.send(new DetectLabelsCommand({
    Image: image,
    MinConfidence: min_confidence,
    MaxLabels: max_labels,
    Features: ['GENERAL_LABELS'],
  }));

  const structured = Labels.map(label => ({
    name: label.Name,
    confidence: Math.round((label.Confidence ?? 0) * 10) / 10,
    parents: label.Parents?.map(p => p.Name) ?? [],
    categories: label.Categories?.map(c => c.Name) ?? [],
    instance_count: label.Instances?.length ?? 0,
  }));

  return {
    content: [{
      type: 'text',
      text: JSON.stringify({ labels: structured, label_count: structured.length }, null, 2),
    }],
  };
});

The image_source union keeps the tool flexible: agents can pass S3 URIs for images already in cloud storage, or base64 for freshly-uploaded bytes. The S3 path avoids the 5 MB base64 payload ceiling — S3 objects up to 15 MB are accepted. Both JPEG and PNG are common; Rekognition also accepts GIF (first frame only) and WEBP.

Label structure — Names, Parents, Instances, Categories

A single DetectLabels response can return labels at multiple levels of the taxonomy simultaneously. For an image of a golden retriever playing fetch on a beach, you might see:

// Example response structure
{
  "Labels": [
    {
      "Name": "Golden Retriever",
      "Confidence": 98.4,
      "Parents": [
        { "Name": "Dog" },
        { "Name": "Canine" },
        { "Name": "Pet" },
        { "Name": "Animal" }
      ],
      "Instances": [
        {
          "BoundingBox": { "Left": 0.12, "Top": 0.35, "Width": 0.44, "Height": 0.52 },
          "Confidence": 98.4
        }
      ],
      "Categories": [{ "Name": "Animals and Pets" }]
    },
    {
      "Name": "Beach",
      "Confidence": 94.1,
      "Parents": [{ "Name": "Outdoors" }, { "Name": "Nature" }],
      "Instances": [],  // scene labels have no instances
      "Categories": [{ "Name": "Nature and Outdoors" }]
    },
    {
      "Name": "Sports",
      "Confidence": 72.3,
      "Parents": [],
      "Instances": [],
      "Categories": [{ "Name": "Activities and Events" }]
    }
  ]
}

Key observations:

• Parents[] — hierarchical ancestors, most specific first. Use the parent chain to match at a category level (e.g., "any Animal") without hardcoding every breed name.
• Instances[] — only populated for localizable objects (dogs, cars, people). Scene labels (Beach, Outdoors) have empty Instances arrays.
• Categories[] — coarse groupings (Animals and Pets, Nature and Outdoors). Useful for a quick content-category classification without inspecting every label name.

When building an MCP tool that checks whether an image contains a person, match against label.Name === 'Person' or check label.Parents.some(p => p.Name === 'Person') on more specific labels (e.g., "Man", "Woman", "Child") — both patterns fire.

Bounding box fractional-to-pixel conversion

Rekognition expresses all bounding boxes as fractions of the image dimensions, not pixel offsets. This makes the coordinates resolution-independent but requires a conversion step before overlaying boxes in a UI or passing positions to downstream image processing:

interface BoundingBox {
  Left: number;   // x-offset of left edge as fraction of image width  (0.0–1.0)
  Top: number;    // y-offset of top edge as fraction of image height  (0.0–1.0)
  Width: number;  // box width as fraction of image width
  Height: number; // box height as fraction of image height
}

function toPixelBox(bb: BoundingBox, imageWidth: number, imageHeight: number) {
  return {
    x: Math.round(bb.Left * imageWidth),
    y: Math.round(bb.Top * imageHeight),
    w: Math.round(bb.Width * imageWidth),
    h: Math.round(bb.Height * imageHeight),
  };
}

// Example: 1920×1080 image, BoundingBox { Left: 0.12, Top: 0.35, Width: 0.44, Height: 0.52 }
const box = toPixelBox({ Left: 0.12, Top: 0.35, Width: 0.44, Height: 0.52 }, 1920, 1080);
// { x: 230, y: 378, w: 845, h: 562 }

// Enhanced tool that includes pixel boxes when image dimensions are known
server.tool('detect_labels_with_positions', {
  image_source: z.union([
    z.object({ s3_uri: z.string() }),
    z.object({ base64: z.string() }),
  ]),
  image_width: z.number().int().optional(),
  image_height: z.number().int().optional(),
  min_confidence: z.number().default(70),
}, async ({ image_source, image_width, image_height, min_confidence }) => {
  const image = resolveImage(image_source);
  const { Labels = [] } = await rekognition.send(new DetectLabelsCommand({
    Image: image,
    MinConfidence: min_confidence,
    MaxLabels: 100,
    Features: ['GENERAL_LABELS'],
  }));

  return {
    content: [{
      type: 'text',
      text: JSON.stringify(
        Labels.map(label => ({
          name: label.Name,
          confidence: label.Confidence,
          instances: label.Instances?.map(inst => ({
            bounding_box: inst.BoundingBox,
            pixel_box: image_width && image_height
              ? toPixelBox(inst.BoundingBox!, image_width, image_height)
              : null,
          })) ?? [],
        })),
        null, 2
      ),
    }],
  };
});

When the agent doesn't know the image dimensions ahead of time, return the raw fractional values and let the calling application convert. The fractional format is lossless — you lose no precision regardless of display resolution.

Tuning MinConfidence and MaxLabels

The API default is MinConfidence: 55 and MaxLabels: 10. For production MCP tools, explicitly setting both avoids depending on API defaults that may change. The MinConfidence cut-off applies server-side, so you're not paying to transfer low-confidence labels you'd filter client-side — set it as high as the use case allows.

ImageProperties feature set

Adding 'IMAGE_PROPERTIES' to the Features array returns dominant colour information alongside labels — useful for image search, thumbnail generation metadata, or design tools:

const { Labels = [], ImageProperties } = await rekognition.send(new DetectLabelsCommand({
  Image: image,
  Features: ['GENERAL_LABELS', 'IMAGE_PROPERTIES'],
  MaxLabels: 50,
}));

// ImageProperties.DominantColors[]: each has Red, Green, Blue (0–255), HexCode, CSSColor, SimplifiedColor
const palette = ImageProperties?.DominantColors?.slice(0, 5).map(c => ({
  hex: c.HexCode,       // e.g. '#4A7FC1'
  css: c.CSSColor,      // e.g. 'Blue'
  simple: c.SimplifiedColor,  // e.g. 'Blue'
  rgb: [c.Red, c.Green, c.Blue],
  pixel_percent: c.PixelPercent,  // fraction of image pixels this colour dominates
})) ?? [];

// ImageProperties.Foreground and Background are also available
const foregroundBrightness = ImageProperties?.Foreground?.Luminance;

ImageProperties does not cost extra API calls — it's part of the same DetectLabels request. SimplifiedColor collapses the full RGB space into a small vocabulary (Red, Orange, Yellow, Green, Blue, Purple, Pink, Black, White, Grey) — useful for filtering images by colour in a search UI.

Async video label detection

For video files, the synchronous DetectLabels API is unavailable. Use the async pair instead:

import {
  StartLabelDetectionCommand,
  GetLabelDetectionCommand,
} from '@aws-sdk/client-rekognition';

// Step 1: start the job (video must be in S3)
const { JobId } = await rekognition.send(new StartLabelDetectionCommand({
  Video: {
    S3Object: { Bucket: 'my-videos-bucket', Name: 'clip.mp4' },
  },
  MinConfidence: 70,
  // Optional: get notified via SNS when complete
  NotificationChannel: {
    SNSTopicArn: 'arn:aws:sns:us-east-1:123456789012:rekognition-jobs',
    RoleArn: 'arn:aws:iam::123456789012:role/RekognitionSNSRole',
  },
  JobTag: 'mcp-label-detection',
}));

// Step 2: poll until complete (or handle SNS notification in a webhook)
async function getLabelDetectionResults(jobId: string) {
  let nextToken: string | undefined;
  const allLabels: any[] = [];

  do {
    const result = await rekognition.send(new GetLabelDetectionCommand({
      JobId: jobId,
      SortBy: 'TIMESTAMP',
      MaxResults: 1000,
      NextToken: nextToken,
    }));

    if (result.JobStatus === 'FAILED') {
      throw new Error(`Label detection failed: ${result.StatusMessage}`);
    }
    if (result.JobStatus !== 'SUCCEEDED') {
      return null; // still in progress
    }

    allLabels.push(...(result.Labels ?? []));
    nextToken = result.NextToken;
  } while (nextToken);

  return allLabels;
}

// Two-tool MCP pattern for async video
server.tool('start_video_label_detection', {
  s3_uri: z.string(),
}, async ({ s3_uri }) => {
  const url = new URL(s3_uri);
  const { JobId } = await rekognition.send(new StartLabelDetectionCommand({
    Video: { S3Object: { Bucket: url.hostname, Name: url.pathname.slice(1) } },
    MinConfidence: 70,
  }));
  return { content: [{ type: 'text', text: JSON.stringify({ job_id: JobId }) }] };
});

server.tool('get_video_label_results', {
  job_id: z.string(),
}, async ({ job_id }) => {
  const labels = await getLabelDetectionResults(job_id);
  if (!labels) {
    return { content: [{ type: 'text', text: JSON.stringify({ status: 'in_progress' }) }] };
  }
  // Labels include Timestamp (ms) so you can find when objects appear/disappear
  const summary = labels.map(l => ({
    timestamp_ms: l.Timestamp,
    label: l.Label?.Name,
    confidence: l.Label?.Confidence,
  }));
  return { content: [{ type: 'text', text: JSON.stringify({ labels: summary, count: summary.length }) }] };
});

The Timestamp field (milliseconds from video start) lets you build temporal queries: "at what timestamps does a person appear?", "when does a car first enter the frame?" Pagination is required for long videos — GetLabelDetection returns at most 1000 entries per call; loop using NextToken until it's undefined.

IAM permissions required

{
  "Effect": "Allow",
  "Action": [
    "rekognition:DetectLabels",
    "rekognition:StartLabelDetection",
    "rekognition:GetLabelDetection"
  ],
  "Resource": "*"
}

// If using S3 as image source, also need:
{
  "Effect": "Allow",
  "Action": ["s3:GetObject"],
  "Resource": "arn:aws:s3:::my-images-bucket/*"
}

// For async SNS notification:
{
  "Effect": "Allow",
  "Action": ["iam:PassRole"],
  "Resource": "arn:aws:iam::123456789012:role/RekognitionSNSRole",
  "Condition": {
    "StringEquals": { "iam:PassedToService": "rekognition.amazonaws.com" }
  }
}

The SNS role trust policy must include rekognition.amazonaws.com as a principal with sns:Publish permission on the topic. If using iam:PassRole, scope the resource ARN to the specific role rather than *.

Image input constraints

For MCP server tools that accept arbitrary user-uploaded images, enforce a size check before the API call: reject anything over 4.5 MB when using bytes input to leave headroom for base64 encoding overhead. If the image exceeds the limit, upload it to S3 first and pass the S3 reference — or return an error instructing the agent to provide a smaller image.