AWS Rekognition Content Moderation in MCP Servers — DetectModerationLabels, A2I Human Review, MinConfidence

Basic tool implementation

import { RekognitionClient, DetectModerationLabelsCommand } from '@aws-sdk/client-rekognition';
import { McpServer } from '@modelcontextprotocol/sdk/server/mcp.js';
import { z } from 'zod';

const rekognition = new RekognitionClient({ region: 'us-east-1' });
const server = new McpServer({ name: 'moderation-tools', version: '1.0.0' });

// Categories that trigger an immediate hard block regardless of subcategory
const HARD_BLOCK_CATEGORIES = new Set([
  'Explicit Nudity',
  'Violence',
  'Visually Disturbing',
  'Hate Symbols',
]);

server.tool('check_image_safety', {
  s3_bucket: z.string(),
  s3_key: z.string(),
  // MinConfidence: labels below this are not returned (server-side filter)
  // Lower = more sensitive (flags more borderline content)
  min_confidence: z.number().min(0).max(100).default(60),
}, async ({ s3_bucket, s3_key, min_confidence }) => {
  const { ModerationLabels = [], ModerationModelVersion } =
    await rekognition.send(new DetectModerationLabelsCommand({
      Image: { S3Object: { Bucket: s3_bucket, Name: s3_key } },
      MinConfidence: min_confidence,
    }));

  if (ModerationLabels.length === 0) {
    return {
      content: [{
        type: 'text',
        text: JSON.stringify({ safe: true, flagged_labels: [], model_version: ModerationModelVersion }),
      }],
    };
  }

  // Split into top-level categories (ParentName === '') and subcategories
  const categories = ModerationLabels.filter(l => !l.ParentName || l.ParentName === '');
  const subcategories = ModerationLabels.filter(l => l.ParentName && l.ParentName !== '');

  const hardBlocked = categories.filter(l => HARD_BLOCK_CATEGORIES.has(l.Name ?? ''));

  const result = {
    safe: hardBlocked.length === 0,
    blocked_reason: hardBlocked.length > 0
      ? hardBlocked.map(l => `${l.Name} (${l.Confidence?.toFixed(1)}%)`).join(', ')
      : null,
    flagged_categories: categories.map(l => ({ name: l.Name, confidence: l.Confidence })),
    flagged_subcategories: subcategories.map(l => ({
      name: l.Name,
      parent: l.ParentName,
      confidence: l.Confidence,
    })),
    model_version: ModerationModelVersion,
  };

  return { content: [{ type: 'text', text: JSON.stringify(result, null, 2) }] };
});

ModerationModelVersion is returned (not set by the caller) — it identifies the model snapshot that evaluated the image. Log this alongside the detection results for audit trails; Rekognition updates the model periodically, and version pinning lets you correlate any change in false-positive rate with a model version bump.

Label taxonomy

The two-tier taxonomy has top-level categories and subcategory children. Categories where ParentName === '' are the coarse classification layer; subcategories provide specificity for allow-lists:

Subcategory allow-lists are powerful for context-aware moderation. A fitness platform blocks "Explicit Nudity" but allows "Barechested Male" (a child of "Suggestive"). A news site blocks "Graphic Violence or Gore" but allows "Weapons" (sports photography with firearms). Implement the allow-list at the subcategory level, not the category level — blocking the whole "Suggestive" category would eliminate many legitimate fitness and medical images.

// Context-aware allow-list example
const ALLOW_LISTED_SUBCATEGORIES = new Set([
  'Barechested Male',   // fitness platform
  'Female Swimwear or Underwear',  // fashion/swimwear retailer
  'Weapons',           // news/sporting photography
  'Alcoholic Beverages', // bar/restaurant platform
]);

function isImageSafe(labels: any[], hardBlockCategories: Set<string>): boolean {
  for (const label of labels) {
    const isCategory = !label.ParentName || label.ParentName === '';
    const isSubcategory = !isCategory;

    if (isCategory && hardBlockCategories.has(label.Name)) {
      // Check if ALL subcategories of this category are allow-listed
      const thisParentSubcats = labels.filter(
        l => l.ParentName === label.Name
      );
      const allAllowListed = thisParentSubcats.every(
        s => ALLOW_LISTED_SUBCATEGORIES.has(s.Name)
      );
      if (!allAllowListed) return false;
    }
  }
  return true;
}

MinConfidence threshold design

A common production pattern uses two thresholds: a hard block threshold (e.g., 80) where the image is rejected automatically, and a review threshold (e.g., 50) where it's routed to human reviewers. Set MinConfidence to the lower review threshold so the API returns labels in the 50–79 band. Your application code then branches: confidence ≥ 80 → block; 50–79 → human review queue; no labels above 50 → safe.

A2I human review integration

// A2I (Amazon Augmented AI) routes borderline images to human reviewers
// Requires: A2I flow definition ARN with Rekognition task type configured in AWS Console

const HARD_BLOCK_THRESHOLD = 80;
const REVIEW_THRESHOLD = 50;    // passed as MinConfidence to get borderline labels

const { ModerationLabels = [] } = await rekognition.send(
  new DetectModerationLabelsCommand({
    Image: { S3Object: { Bucket: s3_bucket, Name: s3_key } },
    MinConfidence: REVIEW_THRESHOLD,
    HumanLoopConfig: {
      HumanLoopName: `review-${Date.now()}-${s3_key.replace(/\//g, '-')}`,  // unique per request
      FlowDefinitionArn: 'arn:aws:sagemaker:us-east-1:123456789012:flow-definition/image-moderation-review',
      DataAttributes: {
        ContentClassifiers: ['FreeOfPersonallyIdentifiableInformation'],
      },
    },
  })
);

// HumanLoopActivationOutput (returned only when A2I loop was started)
// Check result.HumanLoopActivationOutput?.HumanLoopArn to know if review was triggered

// Application logic:
const hardBlockLabels = ModerationLabels.filter(
  l => (l.Confidence ?? 0) >= HARD_BLOCK_THRESHOLD
);
const reviewLabels = ModerationLabels.filter(
  l => {
    const c = l.Confidence ?? 0;
    return c >= REVIEW_THRESHOLD && c < HARD_BLOCK_THRESHOLD;
  }
);

if (hardBlockLabels.length > 0) {
  // Auto-block — image stored in quarantine, not published
  return { action: 'block', reason: hardBlockLabels.map(l => l.Name).join(', ') };
} else if (reviewLabels.length > 0) {
  // A2I loop was triggered above — image held pending human decision
  return { action: 'pending_review', labels: reviewLabels.map(l => l.Name) };
} else {
  return { action: 'allow' };
}

HumanLoopName must be unique per request (per AWS account per region). Using a timestamp plus a sanitised S3 key provides uniqueness without a UUID library. The A2I flow definition is configured in the AWS Console or via the SageMaker API — it specifies the workforce (Mechanical Turk, private team, or vendor workforce), the task UI template, and the work team.

DataAttributes.ContentClassifiers tells A2I reviewers how to handle the data. FreeOfPersonallyIdentifiableInformation means reviewers can safely view the image. If the image might contain PII (faces in a photo), use FreeOfAdultContent (and separately check for PII before routing). These classifiers affect workforce eligibility — private teams can review any content; Mechanical Turk workers are only shown content marked with appropriate classifiers.

Pre-upload gate pattern

// Full pre-upload gate: moderate BEFORE storing permanently
server.tool('upload_image_with_safety_check', {
  // Caller provides a presigned S3 upload URL they've already received;
  // the image is uploaded to a quarantine prefix first, checked, then moved
  quarantine_bucket: z.string(),
  quarantine_key: z.string(),   // e.g., 'quarantine/user-123/photo.jpg'
  final_bucket: z.string(),
  final_key: z.string(),        // e.g., 'uploads/user-123/photo.jpg'
}, async ({ quarantine_bucket, quarantine_key, final_bucket, final_key }) => {
  // 1. Moderate the image while still in quarantine
  const { ModerationLabels = [] } = await rekognition.send(
    new DetectModerationLabelsCommand({
      Image: { S3Object: { Bucket: quarantine_bucket, Name: quarantine_key } },
      MinConfidence: 60,
    })
  );

  const violations = ModerationLabels.filter(
    l => HARD_BLOCK_CATEGORIES.has(l.ParentName || l.Name || '') ||
         HARD_BLOCK_CATEGORIES.has(l.Name || '')
  );

  if (violations.length > 0) {
    // Delete from quarantine — never make it to permanent storage
    await s3.send(new DeleteObjectCommand({ Bucket: quarantine_bucket, Key: quarantine_key }));
    return {
      content: [{ type: 'text', text: JSON.stringify({
        success: false,
        reason: 'Image violates content policy',
        // Don't expose specific label names to end-users (prevents targeted evasion)
      }) }],
    };
  }

  // 2. Safe — move from quarantine to permanent storage
  await s3.send(new CopyObjectCommand({
    CopySource: `${quarantine_bucket}/${quarantine_key}`,
    Bucket: final_bucket,
    Key: final_key,
  }));
  await s3.send(new DeleteObjectCommand({ Bucket: quarantine_bucket, Key: quarantine_key }));

  return {
    content: [{ type: 'text', text: JSON.stringify({ success: true, stored_at: `s3://${final_bucket}/${final_key}` }) }],
  };
});

The quarantine-then-move pattern ensures harmful images are never published even if the moderation step experiences a transient error — the image stays in quarantine (which is not publicly accessible) until explicitly moved. Use an S3 lifecycle rule to auto-delete quarantine objects older than 24 hours to prevent quarantine bucket accumulation.

Async video content moderation

import {
  StartContentModerationCommand,
  GetContentModerationCommand,
} from '@aws-sdk/client-rekognition';

server.tool('start_video_moderation', {
  s3_bucket: z.string(),
  s3_key: z.string(),
  min_confidence: z.number().default(60),
}, async ({ s3_bucket, s3_key, min_confidence }) => {
  const { JobId } = await rekognition.send(new StartContentModerationCommand({
    Video: { S3Object: { Bucket: s3_bucket, Name: s3_key } },
    MinConfidence: min_confidence,
  }));
  return { content: [{ type: 'text', text: JSON.stringify({ job_id: JobId }) }] };
});

server.tool('get_video_moderation_results', {
  job_id: z.string(),
}, async ({ job_id }) => {
  const allDetections: any[] = [];
  let nextToken: string | undefined;

  do {
    const result = await rekognition.send(new GetContentModerationCommand({
      JobId: job_id,
      SortBy: 'TIMESTAMP',
      MaxResults: 1000,
      NextToken: nextToken,
    }));

    if (result.JobStatus !== 'SUCCEEDED') {
      return { content: [{ type: 'text', text: JSON.stringify({ status: result.JobStatus }) }] };
    }

    allDetections.push(...(result.ModerationLabels ?? []));
    nextToken = result.NextToken;
  } while (nextToken);

  // Video: each detection has Timestamp (ms). Group by violation type.
  const violations = allDetections.filter(
    d => HARD_BLOCK_CATEGORIES.has(d.ModerationLabel?.ParentName || d.ModerationLabel?.Name || '')
  );

  const summary = {
    total_detections: allDetections.length,
    violation_count: violations.length,
    safe: violations.length === 0,
    first_violation_at_ms: violations[0]?.Timestamp ?? null,
    violation_timeline: violations.slice(0, 20).map(v => ({
      timestamp_ms: v.Timestamp,
      category: v.ModerationLabel?.ParentName || v.ModerationLabel?.Name,
      confidence: v.ModerationLabel?.Confidence,
    })),
  };

  return { content: [{ type: 'text', text: JSON.stringify(summary) }] };
});

first_violation_at_ms lets you jump directly to the offending moment in the video for human review rather than scanning the entire clip. For video platforms, use this timestamp to auto-generate a clip of the flagged segment as the human review artifact, rather than submitting the entire video to A2I.

IAM permissions

{
  "Effect": "Allow",
  "Action": [
    "rekognition:DetectModerationLabels",
    "rekognition:StartContentModeration",
    "rekognition:GetContentModeration"
  ],
  "Resource": "*"
}

// For A2I human loops:
{
  "Effect": "Allow",
  "Action": ["sagemaker:StartHumanLoop"],
  "Resource": "arn:aws:sagemaker:us-east-1:123456789012:flow-definition/image-moderation-review"
}

// For S3 quarantine pattern:
{
  "Effect": "Allow",
  "Action": ["s3:GetObject", "s3:DeleteObject"],
  "Resource": "arn:aws:s3:::quarantine-bucket/*"
}
{
  "Effect": "Allow",
  "Action": ["s3:PutObject", "s3:CopyObject"],
  "Resource": "arn:aws:s3:::uploads-bucket/uploads/*"
}