AWS Comprehend · 2026-11-03 · Comprehend Advanced arc
AWS Comprehend Advanced NLP for MCP Servers: PII Detection, Custom Entity Recognition, Custom Classification, and Events Extraction
AliveMCP monitors MCP endpoints every 60 seconds — and the endpoints that produce the most ambiguous failure signals are NLP processing tools, because language model APIs fail softly: Comprehend's ContainsPiiEntities returns an empty Labels array when the input exceeds 5,000 bytes rather than throwing, a custom entity recognizer deployed against unrelated text returns an empty Entities array without indicating that the model was invoked on out-of-distribution input, a multi-class classifier at 0.52 confidence routes to the wrong handler with no visible error, and an Events detection job with a misspelled event type silently runs without detecting any events. Four Comprehend capabilities close the gap between a prototype NLP tool and a production-grade text intelligence MCP server. PII detection addresses the most urgent safety concern: MCP servers that process user-supplied text must guard against forwarding personally identifiable information — and credentials — to downstream systems and logs before consent gates are in place; ContainsPiiEntities provides a fast binary pre-screen (returning Labels[] with entity type and score, no spans, responding in under 200ms) while DetectPiiEntities returns exact character offsets for redaction, and the async StartPiiEntitiesDetectionJob processes S3 document batches in either ONLY_REDACTION mode (producing a copy of the document with PII masked character-for-character) or ONLY_OFFSETS mode (returning coordinates without modifying the document); critically, Comprehend detects AWS_ACCESS_KEY and AWS_SECRET_KEY as PII entity types — users occasionally paste AWS credentials into tool arguments, and a PII gate before logging catches this before it reaches CloudWatch logs. Custom entity recognition addresses the limits of built-in NER: Comprehend's standard DetectEntities identifies 12 general named entity types — PERSON, LOCATION, ORGANIZATION, DATE, and others — but cannot identify "alivemcp-status-server" as an MCP_SERVER_SLUG or "CVE-2026-12345" as a VULNERABILITY_ID; custom recognizers train on your annotated documents and learn domain-specific entity patterns, with two annotation formats (CSV with tab-separated offset spans for manually-curated small datasets, and augmented manifests from SageMaker Ground Truth for large annotation workflows), real-time endpoint deployment at 100 characters per second per inference unit, and async batch via StartEntitiesDetectionJobCommand without endpoint overhead. Custom classification provides the routing layer: instead of branching on keywords or calling a general-purpose LLM for routing decisions, a trained Comprehend classifier applies a purpose-built model — either MULTI_CLASS (softmax across labels; output scores sum to 1.0; pick Classes[0]) for mutually exclusive intent routing, or MULTI_LABEL (independent probability per label; scores do not sum to 1.0; apply per-label threshold) for topic tagging — and exposes it via the same real-time endpoint or async batch path as entity recognizers. Events detection takes NLP output from entities to structured knowledge: instead of returning "AWS acquired Anthropic" as two ORGANIZATION entities and a verb, Events detection returns a structured ACQUISITION event with SUBJECT=AWS, OBJECT=Anthropic, DATE_TIME, and ARTIFACT roles linked by EntityId references into the accompanying Entities array — output that an AI orchestrator can reason over directly without re-parsing. This guide synthesizes all four Comprehend patterns into a complete NLP MCP server stack. Pattern 1 covers PII detection and the credential-leak gate. Pattern 2 covers custom entity recognizer training, deployment, and inference. Pattern 3 covers document classification in both modes with real-time and batch paths. Pattern 4 covers Events detection with role semantics and the two-tool async MCP pattern.
TL;DR
- PII detection — use
ContainsPiiEntitiesas a fast pre-screen gate before forwarding text to external APIs or logging systems; process spans right-to-left when redacting to avoid offset drift; gate onAWS_ACCESS_KEYandAWS_SECRET_KEYas hard blocks before any log write: Pre-screen:ContainsPiiEntitiesCommand({ Text, LanguageCode: 'en' })→Labels[]with entity type and score (no offsets). Hard-block types: SSN, CREDIT_DEBIT_NUMBER, BANK_ACCOUNT_NUMBER, AWS_ACCESS_KEY, AWS_SECRET_KEY, PASSWORD, PASSPORT_NUMBER. Soft-block or redact: NAME, EMAIL, PHONE, ADDRESS, IP_ADDRESS. Span extraction:DetectPiiEntitiesCommand→Entities[]withBeginOffset,EndOffset,Score. Redact right-to-left: sort byBeginOffsetdescending, replace each span — replacing from the highest offset first keeps all lower-offset span coordinates valid. Async batch:StartPiiEntitiesDetectionJobCommandwithMode: 'ONLY_REDACTION'andRedactionConfig.MaskMode: 'REPLACE_WITH_PII_ENTITY_TYPE'produces[NAME]/[EMAIL]-masked documents in S3. Both sync APIs have a 5,000 UTF-8 byte hard limit — split on paragraph boundaries for longer text, flag any chunk as PII-containing if any sub-chunk triggers a positive. - Custom entity recognition — design entity type names carefully (uppercase + underscores only, no digits or hyphens), provide 200+ annotated examples per type minimum, deploy to an endpoint for real-time MCP tool inference, use async batch for large document sets without endpoint overhead; F1 ≥ 0.80 is the production readiness gate: Training:
CreateEntityRecognizerCommandwithEntityTypes: [{Type: 'MCP_SERVER_SLUG'}, ...],DataFormat: 'COMPREHEND_CSV'(tab-separated File/Line/BeginOffset/EndOffset/Type), orDataFormat: 'AUGMENTED_MANIFEST'for Ground Truth output. PollDescribeEntityRecognizeruntilStatus === 'TRAINED'(30–90 min for <100k docs). Read F1 fromRecognizerMetadata.EvaluationMetrics.F1Score; per-type breakdown inRecognizerMetadata.EntityTypes[]. Endpoint:CreateEndpointCommand({ ModelArn: recognizerArn, DesiredInferenceUnits: 1 }); 1 IU = 100 chars/sec;~$0.0005/hour. Real-time inference:DetectEntitiesCommand({ Text, EndpointArn }). Filter:entities.filter(e => (e.Score ?? 0) >= 0.80). Async batch:StartEntitiesDetectionJobCommand({ EntityRecognizerArn, InputDataConfig, OutputDataConfig })— no endpoint required, cheaper for large jobs. - Custom classification — choose MULTI_CLASS (softmax scores summing to 1.0, pick Classes[0]) for mutually exclusive routing, MULTI_LABEL (independent 0–1 per label, apply threshold) for topic tagging; keep endpoint running only for latency-critical paths — async batch costs the same per character with no endpoint surcharge; in-place model version upgrade via UpdateEndpointCommand has no traffic interruption: Training CSV:
CLASS,TEXTfor MULTI_CLASS (no quotes needed unless text contains commas);"L1|L2",TEXTfor MULTI_LABEL (pipe-separated labels, quoted). Min examples: 10 per class (MULTI_CLASS), 5 per label (MULTI_LABEL). Train:CreateDocumentClassifierCommand({ Mode: 'MULTI_CLASS', VersionName: 'v1-2026-11-03' }). Same endpoint lifecycle as entity recognizers. Real-time:ClassifyDocumentCommand({ Text, EndpointArn })→Classes[](MULTI_CLASS, sorted by score) orLabels[](MULTI_LABEL). Route onClasses[0].NameifClasses[0].Score >= 0.70; below threshold route to fallback handler. Async:StartDocumentClassificationJobCommand({ DocumentClassifierArn })— no endpoint required. Output JSONL:{"Line": 0, "Classes": [{...}]}. Upgrade:UpdateEndpointCommand({ EndpointArn, DesiredModelArn: newVersionArn })— in-place, no cold-start. - Events detection is async-only — submit via StartEventsDetectionJobCommand with TargetEventTypes, retrieve output JSON Lines from S3; each event has Triggers[] (what phrase triggered it) and Arguments[] (entity participants with roles); EntityId references back into the top-level Entities[] array by Id; apply independent thresholds to trigger GroupScore and argument role Score: Submit:
StartEventsDetectionJobCommand({ TargetEventTypes: ['ACQUISITION', 'EMPLOYMENT', ...], InputDataConfig: {S3Uri, InputFormat: 'ONE_DOC_PER_LINE'}, OutputDataConfig: {S3Uri} }). PollDescribeEventsDetectionJobCommanduntilJobStatus === 'COMPLETED'. Output per line:{ Entities: [{Id, Text, Type, BeginOffset, EndOffset}], Events: [{Type, Triggers: [{Text, GroupScore}], Arguments: [{EntityId, Role, Score}]}] }. BuildentityMap = new Map(output.Entities.map(e => [e.Id, e])). Filter: triggerGroupScore >= 0.70to accept event; roleScore >= 0.75to assign participant. ACQUISITION vs CORPORATE_ACQUISITION: handle both and deduplicate by trigger text offset. Two-tool MCP pattern:extract_events_start(writes docs to S3, calls Start, returns jobId) +extract_events_results(polls status, reads output, returns structured events).
Pattern 1: PII Detection — the credential-leak gate every MCP server needs
MCP servers that accept user-supplied text — document uploads, natural-language queries, support ticket text, chat messages — face a compliance and security risk that manifests before any business logic runs: the user may have included personally identifiable information or credentials in their input. Forwarding this input directly to external APIs, writing it to CloudWatch Logs, or passing it to a downstream AI model before consent checks can expose PII to systems that were never authorized to process it. Comprehend PII detection provides two complementary APIs for different points in this pipeline.
ContainsPiiEntities: the fast pre-screen gate
ContainsPiiEntitiesCommand answers a binary question — does this document contain PII? — without returning character offsets. It typically responds in under 200ms and is the right tool for pre-screening user input before forwarding to external services:
import {
ComprehendClient,
ContainsPiiEntitiesCommand,
DetectPiiEntitiesCommand,
LanguageCode,
type PiiEntity,
} from '@aws-sdk/client-comprehend';
const client = new ComprehendClient({ region: 'us-east-1' });
// Types that hard-block processing — never forward these downstream
const HARD_BLOCK_TYPES = new Set([
'SSN', 'CREDIT_DEBIT_NUMBER', 'BANK_ACCOUNT_NUMBER',
'AWS_ACCESS_KEY', 'AWS_SECRET_KEY', 'PASSWORD', 'PASSPORT_NUMBER',
'DRIVER_ID',
]);
interface PiiScreenResult {
containsPii: boolean;
hardBlock: boolean;
detectedTypes: Array<{ type: string; score: number }>;
}
async function screenForPii(text: string, threshold = 0.80): Promise<PiiScreenResult> {
// Split long text — 5,000 UTF-8 byte hard limit per API call
const chunks = splitIntoChunks(text, 4500);
const allDetected: Array<{ type: string; score: number }> = [];
for (const chunk of chunks) {
const { Labels } = await client.send(new ContainsPiiEntitiesCommand({
Text: chunk,
LanguageCode: LanguageCode.EN,
}));
for (const label of Labels ?? []) {
if ((label.Score ?? 0) >= threshold) {
allDetected.push({ type: label.Name ?? '', score: label.Score ?? 0 });
}
}
}
return {
containsPii: allDetected.length > 0,
hardBlock: allDetected.some(d => HARD_BLOCK_TYPES.has(d.type)),
detectedTypes: allDetected,
};
}
function splitIntoChunks(text: string, maxBytes: number): string[] {
const encoder = new TextEncoder();
if (encoder.encode(text).length <= maxBytes) return [text];
const chunks: string[] = [];
const paragraphs = text.split(/\n\n+/);
let current = '';
for (const para of paragraphs) {
const candidate = current ? `${current}\n\n${para}` : para;
if (encoder.encode(candidate).length > maxBytes) {
if (current) chunks.push(current);
current = para;
} else {
current = candidate;
}
}
if (current) chunks.push(current);
return chunks;
}
The key distinction: ContainsPiiEntities returns Labels[] with entity type names and confidence scores only — it does NOT return character offsets. It is fast because it performs classification rather than extraction. Use it at the tool handler entry point before any other processing.
DetectPiiEntities: span-level extraction and right-to-left redaction
When you need to know exactly where the PII appears — to redact it, log only the non-PII portions, or present an audit trail — call DetectPiiEntitiesCommand. The critical implementation detail: always process spans right-to-left when building redacted text:
interface RedactionResult {
redactedText: string;
detectedEntities: Array<{ type: string; score: number; original: string }>;
}
async function detectAndRedact(text: string, threshold = 0.80): Promise<RedactionResult> {
const { Entities } = await client.send(new DetectPiiEntitiesCommand({
Text: text,
LanguageCode: LanguageCode.EN,
}));
// Filter by confidence and validate offset fields exist
const confident = (Entities ?? [])
.filter((e): e is Required<PiiEntity> =>
(e.Score ?? 0) >= threshold &&
e.BeginOffset !== undefined &&
e.EndOffset !== undefined
)
.sort((a, b) => a.BeginOffset - b.BeginOffset);
let redacted = text;
const detected: RedactionResult['detectedEntities'] = [];
// Process in reverse offset order to prevent offset drift
// Replacing a span at offset 200-210 changes string length —
// all offsets below 200 remain valid, but forward replacements would
// reference wrong positions after the first substitution
for (const entity of [...confident].reverse()) {
const original = text.slice(entity.BeginOffset, entity.EndOffset);
const mask = `[${entity.Type}]`;
redacted =
redacted.slice(0, entity.BeginOffset) +
mask +
redacted.slice(entity.EndOffset);
detected.unshift({ type: entity.Type, score: entity.Score, original });
}
return { redactedText: redacted, detectedEntities: detected };
}
The right-to-left requirement is non-obvious and easy to miss: if you process spans left-to-right and the mask string [NAME] is shorter than the original span "John Bartholomew Smith", the byte positions of all subsequent spans shift. Sorting descending by BeginOffset and replacing from the end of the string forward eliminates this entirely — lower-offset spans are unaffected by replacements at higher offsets.
The AWS credential detection use case
The most operationally significant PII types for MCP server operators are AWS_ACCESS_KEY and AWS_SECRET_KEY. Users paste credentials into tool arguments more often than you'd expect — copying a command from a tutorial, testing with temporary credentials, or using an MCP tool to process a configuration file that contains credentials. Comprehend detects AKIA...-prefixed access key IDs and 40-character secret keys reliably:
import { McpServer } from '@modelcontextprotocol/sdk/server/mcp.js';
import { z } from 'zod';
const server = new McpServer({ name: 'document-processor', version: '1.0.0' });
server.tool('process_document', {
document_text: z.string().max(4000),
}, async ({ document_text }) => {
const screen = await screenForPii(document_text);
if (screen.hardBlock) {
// Hard block — credentials or financial PII detected
// Do NOT log document_text here — that would defeat the gate
const types = screen.detectedTypes
.filter(t => HARD_BLOCK_TYPES.has(t.type))
.map(t => t.type);
return {
content: [{
type: 'text',
text: JSON.stringify({
error: 'Document contains sensitive PII that cannot be processed',
detected_types: types,
action: types.some(t => t.includes('KEY') || t === 'PASSWORD')
? 'Credentials detected — rotate immediately if these are real keys'
: 'Remove PII before resubmitting',
}),
}],
};
}
if (screen.containsPii) {
// Soft PII detected — redact before processing
const { redactedText } = await detectAndRedact(document_text);
return processRedacted(redactedText);
}
return processRedacted(document_text);
});
Async PII redaction for batch document pipelines
For batch workloads — nightly processing of accumulated document uploads, compliance-required redaction of an existing document archive — the async StartPiiEntitiesDetectionJobCommand processes S3 document sets without the 5,000-byte per-call limit:
import {
StartPiiEntitiesDetectionJobCommand,
DescribePiiEntitiesDetectionJobCommand,
} from '@aws-sdk/client-comprehend';
const { JobId } = await client.send(new StartPiiEntitiesDetectionJobCommand({
JobName: `pii-redact-${Date.now()}`,
DataAccessRoleArn: 'arn:aws:iam::123456789012:role/ComprehendDataAccessRole',
LanguageCode: LanguageCode.EN,
Mode: 'ONLY_REDACTION',
RedactionConfig: {
// REPLACE_WITH_PII_ENTITY_TYPE produces "[NAME]", "[EMAIL]" markers
// REDACT produces asterisk-filled spans: "***" — less human-readable
MaskMode: 'REPLACE_WITH_PII_ENTITY_TYPE',
// Optional: only redact specific types
PiiEntityTypes: ['NAME', 'EMAIL', 'SSN', 'AWS_ACCESS_KEY', 'AWS_SECRET_KEY'],
},
InputDataConfig: {
S3Uri: 's3://my-bucket/raw-documents/',
InputFormat: 'ONE_DOC_PER_LINE',
},
OutputDataConfig: {
S3Uri: 's3://my-bucket/redacted-documents/',
},
}));
The two mask modes produce different outputs: REPLACE_WITH_PII_ENTITY_TYPE replaces each span with its entity type label ([NAME], [AWS_ACCESS_KEY]) — preserving document structure and making the redaction auditable; REDACT replaces each span with asterisks (***) matching the original span length — appropriate when the entity type classification itself must not be disclosed. For compliance audit trails, REPLACE_WITH_PII_ENTITY_TYPE is almost always the right choice because it lets reviewers verify that redaction was complete without revealing the original values.
Full list of Comprehend PII entity types, pricing, and the MCP pre-screening gate pattern are covered in detail in AWS Comprehend PII Detection in MCP Servers.
Pattern 2: Custom Entity Recognition — training domain-specific NER models for MCP tool extraction
Comprehend's built-in entity recognizer is a general-purpose NER model. It correctly identifies standard named entities — people, places, organizations, dates, quantities — but it has never seen your domain vocabulary. An MCP server that needs to extract MCP server slugs from freeform issue text, parse CVE identifiers from security advisories, or identify proprietary product codes from support tickets cannot use DetectEntities for those extraction tasks. Custom entity recognizers fill this gap.
Designing your entity type list
Before writing annotation data, design the entity type list carefully. Comprehend supports up to 25 custom entity types per recognizer. Entity type names must match the regex [A-Z_]+ — uppercase letters and underscores only, no digits, hyphens, or spaces.
| Design principle | Good example | Bad example |
|---|---|---|
| Types map to distinct downstream handling | MCP_SERVER_SLUG, REGISTRY_NAME, TOOL_ERROR_CODE | PRODUCT (catches everything) |
| Types are mutually exclusive in practice | CVE_ID, ADVISORY_ID | IDENTIFIER (too broad) |
| Each type has consistent surface patterns | SERVER_VERSION (e.g., "1.2.3") | STATUS_INFO (highly variable) |
| You can provide 200+ examples per type | Any type you actually label | A type you can only annotate 20 times |
Too few types and precision suffers — a single PRODUCT type for all product-related text learns vague patterns. Too many types and you need far more annotation data per type to train a good model. A workable rule: if you would write separate code to handle two entity types differently downstream, they warrant separate types in the recognizer.
CSV annotation format
The CSV format requires two S3 objects: a documents file (one document per line, plain UTF-8 text) and a tab-separated annotations file:
# documents.txt — one document per line in S3
The mcp-server-filesystem endpoint has been down for 3 hours per the AliveMCP dashboard.
User reports mcp-server-brave-search returned TOOL_ERROR on all calls since deployment.
AliveMCP shows mcp-server-sequential-thinking at 99.9% uptime over the last 30 days.
CVE-2026-12345 affects all mcp-server-memory versions below 2.1.0.
# annotations.csv — tab-separated (use actual tabs, not spaces)
File Line Begin Offset End Offset Type
s3://my-bucket/train/documents.txt 0 4 26 MCP_SERVER_SLUG
s3://my-bucket/train/documents.txt 1 18 41 MCP_SERVER_SLUG
s3://my-bucket/train/documents.txt 1 50 60 TOOL_ERROR_CODE
s3://my-bucket/train/documents.txt 2 9 38 MCP_SERVER_SLUG
s3://my-bucket/train/documents.txt 3 0 14 CVE_ID
s3://my-bucket/train/documents.txt 3 42 63 MCP_SERVER_SLUG
Offset precision is the most common source of poor model quality. Off-by-one errors — annotating "mcp-server-filesystem " (trailing space) instead of "mcp-server-filesystem" — are systematic errors that consistently degrade F1 scores. When building annotation pipelines, always verify annotation spans with a snippet extraction test: line.slice(beginOffset, endOffset) should exactly equal the entity text.
Creating and training the recognizer
import {
ComprehendClient,
CreateEntityRecognizerCommand,
DescribeEntityRecognizerCommand,
LanguageCode,
} from '@aws-sdk/client-comprehend';
const client = new ComprehendClient({ region: 'us-east-1' });
const { EntityRecognizerArn } = await client.send(
new CreateEntityRecognizerCommand({
RecognizerName: 'mcp-server-entities-v1',
LanguageCode: LanguageCode.EN,
DataAccessRoleArn: 'arn:aws:iam::123456789012:role/ComprehendDataAccessRole',
InputDataConfig: {
DataFormat: 'COMPREHEND_CSV',
EntityTypes: [
{ Type: 'MCP_SERVER_SLUG' },
{ Type: 'REGISTRY_NAME' },
{ Type: 'TOOL_ERROR_CODE' },
{ Type: 'CVE_ID' },
],
Documents: { S3Uri: 's3://my-bucket/train/documents.txt' },
Annotations: { S3Uri: 's3://my-bucket/train/annotations.csv' },
},
})
);
// Training is async — poll until TRAINED (30–90 minutes)
async function waitForTraining(arn: string) {
while (true) {
const { EntityRecognizerProperties: props } = await client.send(
new DescribeEntityRecognizerCommand({ EntityRecognizerArn: arn })
);
if (props?.Status === 'TRAINED') {
const metrics = props.RecognizerMetadata?.EvaluationMetrics;
// Production gate: F1 >= 0.80
// If below threshold, check per-type metrics in RecognizerMetadata.EntityTypes[]
// Low-F1 types need more annotation examples or better annotation consistency
console.log('F1:', metrics?.F1Score, 'Precision:', metrics?.Precision, 'Recall:', metrics?.Recall);
return arn;
}
if (props?.Status === 'IN_ERROR') {
throw new Error(`Training failed: ${props.Message ?? 'unknown'}`);
}
await new Promise(r => setTimeout(r, 60_000));
}
}
The DataAccessRoleArn requires a trust policy allowing comprehend.amazonaws.com to assume it — always include an aws:SourceAccount condition on the trust policy to prevent confused deputy attacks where Comprehend is tricked into assuming your role for another customer's job. The role needs s3:GetObject and s3:ListBucket on the training bucket.
Endpoint deployment and real-time inference in MCP tool handlers
import {
CreateEndpointCommand,
DescribeEndpointCommand,
DetectEntitiesCommand,
} from '@aws-sdk/client-comprehend';
import { McpServer } from '@modelcontextprotocol/sdk/server/mcp.js';
import { z } from 'zod';
// Deploy endpoint — 1 inference unit = 100 chars/sec sustained throughput
const { EndpointArn } = await client.send(new CreateEndpointCommand({
EndpointName: 'mcp-entities-prod',
ModelArn: entityRecognizerArn,
DesiredInferenceUnits: 1,
}));
// Wait for IN_SERVICE (5–15 minutes)
await waitForEndpoint(EndpointArn);
const server = new McpServer({ name: 'entity-extractor', version: '1.0.0' });
const ENTITY_CONFIDENCE_THRESHOLD = 0.80;
server.tool('extract_entities', {
text: z.string().max(4000),
entity_types: z.array(z.string()).optional(),
}, async ({ text, entity_types }) => {
const { Entities } = await client.send(new DetectEntitiesCommand({
Text: text,
EndpointArn: process.env.COMPREHEND_ENTITY_ENDPOINT_ARN!,
}));
const extracted = (Entities ?? [])
.filter(e =>
(e.Score ?? 0) >= ENTITY_CONFIDENCE_THRESHOLD &&
(!entity_types || entity_types.includes(e.Type ?? ''))
)
.map(e => ({
text: e.Text,
type: e.Type,
score: e.Score,
begin: e.BeginOffset,
end: e.EndOffset,
}));
// Deduplicate overlapping spans — keep highest-confidence when spans overlap
const deduped = deduplicateOverlappingSpans(extracted);
return {
content: [{ type: 'text', text: JSON.stringify({ entities: deduped }) }],
};
});
Async batch recognition without an endpoint
For batch document processing tasks where real-time latency is not required, use StartEntitiesDetectionJobCommand with the recognizer ARN directly — no endpoint deployment needed, and endpoint hourly charges don't apply:
import { StartEntitiesDetectionJobCommand } from '@aws-sdk/client-comprehend';
const { JobId } = await client.send(new StartEntitiesDetectionJobCommand({
JobName: `entity-batch-${Date.now()}`,
EntityRecognizerArn: entityRecognizerArn,
DataAccessRoleArn: 'arn:aws:iam::123456789012:role/ComprehendDataAccessRole',
LanguageCode: LanguageCode.EN,
InputDataConfig: {
S3Uri: 's3://my-bucket/documents/',
InputFormat: 'ONE_DOC_PER_LINE',
},
OutputDataConfig: { S3Uri: 's3://my-bucket/entity-output/' },
}));
// Output: one JSON line per document
// {"Entities": [{"BeginOffset": 4, "EndOffset": 26, "Score": 0.93, "Text": "mcp-server-filesystem", "Type": "MCP_SERVER_SLUG"}]}
Cost comparison: endpoint inference is billed per character processed ($0.0005/100 chars) plus endpoint hours ($0.0005/hour per IU). A 24/7 endpoint at 1 IU costs ~$0.36/day regardless of volume. Async batch is billed per character only — for irregular or low-volume workloads, async batch is dramatically cheaper. Train patterns, confidence tuning, augmented manifest format, and per-type F1 diagnosis are covered in depth in AWS Comprehend Custom Entity Recognition in MCP Servers.
Pattern 3: Custom Classification — intent routing and document tagging with trained models
Routing MCP tool calls — sending server-unreachable issues to one handler, auth failures to another, schema drift to a third — is easy to prototype with keyword matching and fragile in production. Keyword matchers miss paraphrases, misfire on unrelated mentions, and require manual maintenance as vocabulary evolves. Custom Comprehend classifiers provide a trained model that generalizes across paraphrase variation and produces calibrated confidence scores that you can use to detect out-of-distribution inputs.
MULTI_CLASS vs MULTI_LABEL: the architectural choice
| Dimension | MULTI_CLASS | MULTI_LABEL |
|---|---|---|
| Labels per document | Exactly one winner | Zero or more |
| Output scores | Sum to 1.0 (softmax) | Independent 0–1 per label |
| Routing decision | Pick Classes[0] | Threshold each label independently |
| Training minimum | 10 examples/class | 5 examples/label |
| Training CSV format | CLASS,TEXT | "L1|L2",TEXT |
| Best for | Intent routing, ticket triage, language detection | Topic tagging, content labeling, multi-topic documents |
| Low-confidence signal | Classes[0].Score < 0.70 → fallback handler | No label scores above threshold → untagged |
For AliveMCP's primary use case — routing MCP server issue reports to the right handler — MULTI_CLASS is correct: each issue should route to exactly one primary category (SERVER_DOWN, AUTH_FAILURE, SCHEMA_DRIFT, CERT_EXPIRED, RATE_LIMITED, TIMEOUT). MULTI_LABEL would be appropriate for tagging blog posts or documentation with multiple applicable topics.
Training data: both CSV formats
# MULTI_CLASS format: CLASS,TEXT (no header row)
# Text with commas must be double-quoted
SERVER_DOWN,"The mcp-server-filesystem endpoint is not responding. AliveMCP shows 45 minutes down."
SERVER_DOWN,Getting connection refused on all calls to mcp-server-brave-search since 14:30 UTC.
AUTH_FAILURE,mcp-server-github returns 401 on every call. Credentials look correct from my side.
AUTH_FAILURE,"Authentication failing for mcp-server-gitlab. Token rotated yesterday but error persists."
SCHEMA_DRIFT,"mcp-server-postgres tool schema changed, orchestrator can't find query_table anymore."
CERT_EXPIRED,"SSL certificate error on mcp-server-everything. Browser also shows cert warning."
RATE_LIMITED,"Getting 429 Too Many Requests from mcp-server-openai since the traffic spike."
TIMEOUT,"mcp-server-memory calls hanging for 30+ seconds then timing out. Started 2 hours ago."
# MULTI_LABEL format: "L1|L2",TEXT (labels pipe-separated, quoted when multiple)
monitoring,"How do I set up uptime monitoring for my MCP server?"
"monitoring|alerting","Configure webhook alerts when an MCP server goes down."
"monitoring|metrics","What metrics does AliveMCP track for each MCP endpoint?"
"alerting|slack","How do I connect AliveMCP Slack alerts to my workspace?"
Training and deploying the classifier
import {
ComprehendClient,
CreateDocumentClassifierCommand,
ClassifyDocumentCommand,
DocumentClassifierMode,
LanguageCode,
} from '@aws-sdk/client-comprehend';
const { DocumentClassifierArn } = await client.send(
new CreateDocumentClassifierCommand({
DocumentClassifierName: 'mcp-issue-classifier',
LanguageCode: LanguageCode.EN,
Mode: DocumentClassifierMode.MULTI_CLASS,
VersionName: 'v1-2026-11-03', // becomes part of the ARN
DataAccessRoleArn: 'arn:aws:iam::123456789012:role/ComprehendDataAccessRole',
InputDataConfig: {
DataFormat: 'COMPREHEND_CSV',
S3Uri: 's3://my-bucket/training/classifier-train.csv',
},
OutputDataConfig: {
// Confusion matrix and per-class F1 written here after training
S3Uri: 's3://my-bucket/classifier-output/',
},
})
);
// Same endpoint lifecycle as entity recognizers
// CreateEndpointCommand → wait IN_SERVICE → ClassifyDocumentCommand
Real-time classification in MCP tool handlers
import { McpServer } from '@modelcontextprotocol/sdk/server/mcp.js';
import { z } from 'zod';
const server = new McpServer({ name: 'issue-router', version: '1.0.0' });
const CLASSIFICATION_THRESHOLD = 0.70;
const FALLBACK_CLASS = 'UNKNOWN';
server.tool('classify_issue', {
issue_text: z.string().max(4000),
}, async ({ issue_text }) => {
const { Classes, Labels } = await client.send(new ClassifyDocumentCommand({
Text: issue_text,
EndpointArn: process.env.COMPREHEND_CLASSIFIER_ARN!,
}));
// MULTI_CLASS: Classes[] sorted by Score descending
// Classes[0] is always the top prediction — do not sort manually
const topClass = Classes?.[0];
if (!topClass || (topClass.Score ?? 0) < CLASSIFICATION_THRESHOLD) {
return {
content: [{
type: 'text',
text: JSON.stringify({
classification: FALLBACK_CLASS,
confidence: topClass?.Score ?? 0,
all_classes: (Classes ?? []).slice(0, 3).map(c => ({
name: c.Name,
score: c.Score,
})),
message: 'Issue does not match known categories with sufficient confidence',
}),
}],
};
}
return {
content: [{
type: 'text',
text: JSON.stringify({
classification: topClass.Name,
confidence: topClass.Score,
top_classes: (Classes ?? []).slice(0, 3).map(c => ({
name: c.Name,
score: c.Score,
})),
}),
}],
};
});
The low-confidence fallback route is critical for production reliability. A classifier forced to produce a routing decision at 0.52 confidence routes to the wrong handler roughly half the time — the same as random assignment for a binary problem. Return UNKNOWN and escalate to a human or LLM fallback handler when Classes[0].Score < 0.70.
Async batch classification and cost comparison
import { StartDocumentClassificationJobCommand } from '@aws-sdk/client-comprehend';
// No endpoint required — same per-character cost, no endpoint surcharge
const { JobId } = await client.send(new StartDocumentClassificationJobCommand({
JobName: `classify-batch-${Date.now()}`,
DocumentClassifierArn: documentClassifierArn,
DataAccessRoleArn: 'arn:aws:iam::123456789012:role/ComprehendDataAccessRole',
InputDataConfig: {
S3Uri: 's3://my-bucket/batch-input/',
InputFormat: 'ONE_DOC_PER_LINE',
},
OutputDataConfig: { S3Uri: 's3://my-bucket/batch-output/' },
}));
// Output JSONL: {"Line": 0, "Classes": [{"Name": "SERVER_DOWN", "Score": 0.94}, ...]}
| Path | Per-character cost | Endpoint cost | Best for |
|---|---|---|---|
| Real-time endpoint | $0.0005/100 chars | ~$0.0005/hour per IU | Interactive MCP tools, <200ms latency requirement |
| Async batch job | $0.0005/100 chars | None | Nightly bulk triage, large document backlogs |
In-place model version upgrades
UpdateEndpointCommand accepts a new DesiredModelArn pointing at a retrained classifier version. The update takes 5–10 minutes and has no traffic interruption or cold-start window — the endpoint continues serving traffic from the old model until the transition completes. This is the correct zero-downtime upgrade path:
import { UpdateEndpointCommand } from '@aws-sdk/client-comprehend';
// Train new version (same DocumentClassifierName, new VersionName)
// Then upgrade the live endpoint to the new version in-place:
await client.send(new UpdateEndpointCommand({
EndpointArn: productionEndpointArn,
DesiredModelArn: newClassifierVersionArn,
// DesiredInferenceUnits: 2, // optionally resize at the same time
}));
Multi-version classifiers (same name, different VersionName values) support A/B testing: deploy the new version to a canary endpoint at a fraction of traffic, compare routing accuracy and Classes[0].Score distributions, then upgrade the production endpoint and delete the canary. AWS Comprehend Custom Classification in MCP Servers covers the full classifier lifecycle including confusion matrix interpretation and class imbalance handling.
Pattern 4: Events Detection — structured knowledge extraction from financial and operational text
Comprehend Events detection bridges the gap between named entity recognition and structured knowledge. Where DetectEntities returns "AWS" and "Anthropic" as organizations and "April 1, 2026" as a date, Events detection returns a structured ACQUISITION event: SUBJECT=AWS (the acquirer), OBJECT=Anthropic (the acquired), DATE_TIME=April 1, 2026. The difference in downstream usefulness is substantial — an AI orchestrator can reason over the structured event representation directly without re-parsing natural language.
Supported event types and their role semantics
Events detection is built-in — no custom model training required. It supports 11 event types across financial, employment, and corporate domains, and is available for English only:
| Event Type | Primary roles | Typical trigger words |
|---|---|---|
| ACQUISITION | SUBJECT (acquirer), OBJECT (acquired), DATE_TIME, ARTIFACT (deal value) | acquired, bought, purchased |
| CORPORATE_ACQUISITION | SUBJECT (acquirer), OBJECT (target), DATE_TIME, ARTIFACT | acquired by, taken over, merger |
| EMPLOYMENT | SUBJECT (person), OBJECT (org), ARTIFACT (title), DATE_TIME | joined, hired, appointed, named |
| BOARD_CHANGE | SUBJECT (person), OBJECT (company), ARTIFACT (role) | elected, resigned, stepped down, appointed |
| IPO | SUBJECT (company), DATE_TIME, ARTIFACT (exchange, price) | priced, went public, listed, IPO |
| BANKRUPTCY | SUBJECT (filing entity), DATE_TIME, ARTIFACT (chapter type) | filed for, Chapter 11, Chapter 7 |
| BUSINESS_EXPANSION | SUBJECT (company), ARTIFACT (location/product), DATE_TIME | expanded, launched, opened, extended |
| BUSINESS_REDUCTION | SUBJECT (company), ARTIFACT (scope), DATE_TIME | laid off, cut, reduced, downsized |
| DISSOLVED | SUBJECT (entity), DATE_TIME | dissolved, shut down, closed, wound up |
| RIGHTS_ISSUE | SUBJECT (company), ARTIFACT (ratio, price), DATE_TIME | rights issue, rights offering |
| SECONDARY_OFFERING | SUBJECT (company), ARTIFACT (shares, price), DATE_TIME | secondary offering, follow-on, dilutive |
ACQUISITION and CORPORATE_ACQUISITION both detect acquisition events but from different textual perspectives — ACQUISITION typically fires when the acquiring entity is the grammatical subject ("AWS acquired Anthropic"), while CORPORATE_ACQUISITION fires when the target or the transaction itself is foregrounded ("Anthropic was acquired by AWS"). In practice, both may fire for the same underlying event depending on how the text is written. Your extraction logic should handle both types and deduplicate events with overlapping trigger text spans.
Submitting an events detection job
Events detection is async-only — there is no synchronous DetectEvents API. Always structure MCP tools around the two-step submit/retrieve pattern:
import {
ComprehendClient,
StartEventsDetectionJobCommand,
DescribeEventsDetectionJobCommand,
LanguageCode,
} from '@aws-sdk/client-comprehend';
const client = new ComprehendClient({ region: 'us-east-1' });
const { JobId } = await client.send(new StartEventsDetectionJobCommand({
JobName: `events-detect-${Date.now()}`,
DataAccessRoleArn: 'arn:aws:iam::123456789012:role/ComprehendDataAccessRole',
LanguageCode: LanguageCode.EN,
// Specify only relevant types — reduces output noise
// Cost is per-character regardless of event type count
TargetEventTypes: [
'ACQUISITION',
'CORPORATE_ACQUISITION',
'EMPLOYMENT',
'BOARD_CHANGE',
'BANKRUPTCY',
'BUSINESS_EXPANSION',
'BUSINESS_REDUCTION',
],
InputDataConfig: {
S3Uri: 's3://my-bucket/news-articles/',
InputFormat: 'ONE_DOC_PER_LINE',
},
OutputDataConfig: {
S3Uri: 's3://my-bucket/events-output/',
},
}));
Output JSON structure and EntityId cross-references
The output JSON Lines file has one object per input document. Each document object contains two top-level arrays:
// Example output for: "AWS acquired Anthropic for $4B in a deal closed on April 1, 2026"
{
"Entities": [
{ "Id": 0, "Text": "AWS", "Type": "ORGANIZATION", "BeginOffset": 0, "EndOffset": 3 },
{ "Id": 1, "Text": "Anthropic","Type": "ORGANIZATION", "BeginOffset": 13, "EndOffset": 22 },
{ "Id": 2, "Text": "$4B", "Type": "QUANTITY", "BeginOffset": 27, "EndOffset": 30 },
{ "Id": 3, "Text": "April 1, 2026", "Type": "DATE", "BeginOffset": 52, "EndOffset": 65 }
],
"Events": [
{
"Type": "ACQUISITION",
"Triggers": [
{ "Id": 4, "Text": "acquired", "Type": "ACQUISITION", "BeginOffset": 4, "EndOffset": 12, "GroupScore": 0.97 }
],
"Arguments": [
{ "EntityId": 0, "Role": "SUBJECT", "Score": 0.98 },
{ "EntityId": 1, "Role": "OBJECT", "Score": 0.96 },
{ "EntityId": 2, "Role": "ARTIFACT", "Score": 0.87 },
{ "EntityId": 3, "Role": "DATE_TIME", "Score": 0.95 }
]
}
]
}
Key structural points for parsing:
- Entities[] uses integer Ids that Arguments[].EntityId references — build a
Map<number, Entity>for O(1) lookup - Triggers[].GroupScore is the confidence that the trigger phrase indicates a real event (as opposed to figurative language or historical reference)
- Arguments[].Score is the confidence that this specific entity fills this role in the event — independent from GroupScore
- Multiple triggers per event are possible when an event is mentioned multiple times in one document; each trigger has its own GroupScore
- Events without Arguments are valid — the event occurred but no entities were identified as participants
TypeScript parsing with independent confidence thresholds
interface ComprehendEntity {
Id: number;
Text: string;
Type: string;
BeginOffset: number;
EndOffset: number;
}
interface ComprehendEvent {
Type: string;
Triggers: Array<{ Id: number; Text: string; GroupScore: number }>;
Arguments: Array<{ EntityId: number; Role: string; Score: number }>;
}
interface DocumentEventsOutput {
Entities: ComprehendEntity[];
Events: ComprehendEvent[];
}
interface StructuredEvent {
type: string;
trigger: string;
triggerConfidence: number;
subject?: string;
object?: string;
date?: string;
artifact?: string;
investor?: string;
recipient?: string;
raw: ComprehendEvent;
}
// Two independent thresholds:
// - TRIGGER_THRESHOLD: confidence that the event actually happened
// - ROLE_THRESHOLD: confidence that a specific entity fills a role
const TRIGGER_THRESHOLD = 0.70;
const ROLE_THRESHOLD = 0.75;
function parseDocumentEvents(output: DocumentEventsOutput): StructuredEvent[] {
// Build integer-keyed entity map for fast role resolution
const entityMap = new Map<number, ComprehendEntity>(
output.Entities.map(e => [e.Id, e])
);
const results: StructuredEvent[] = [];
for (const event of output.Events) {
// Use the highest-confidence trigger as the event anchor
const primaryTrigger = [...event.Triggers].sort(
(a, b) => b.GroupScore - a.GroupScore
)[0];
if (!primaryTrigger || primaryTrigger.GroupScore < TRIGGER_THRESHOLD) continue;
const structured: StructuredEvent = {
type: event.Type,
trigger: primaryTrigger.Text,
triggerConfidence: primaryTrigger.GroupScore,
raw: event,
};
for (const arg of event.Arguments) {
if (arg.Score < ROLE_THRESHOLD) continue; // Low-confidence role — skip rather than misassign
const entity = entityMap.get(arg.EntityId);
if (!entity) continue;
switch (arg.Role) {
case 'SUBJECT': structured.subject = entity.Text; break;
case 'OBJECT': structured.object = entity.Text; break;
case 'DATE_TIME': structured.date = entity.Text; break;
case 'ARTIFACT': structured.artifact = entity.Text; break;
case 'INVESTOR': structured.investor = entity.Text; break;
case 'RECIPIENT': structured.recipient = entity.Text; break;
}
}
results.push(structured);
}
return results;
}
// Deduplicate ACQUISITION + CORPORATE_ACQUISITION with overlapping triggers
function deduplicateByTriggerOffset(events: StructuredEvent[]): StructuredEvent[] {
const seen = new Set<string>();
return events.filter(e => {
const trigger = e.raw.Triggers[0];
if (!trigger) return true;
const key = `${trigger.BeginOffset}-${trigger.EndOffset}`;
if (seen.has(key)) return false;
seen.add(key);
return true;
});
}
Two-tool MCP pattern for async events extraction
import { S3Client, GetObjectCommand, PutObjectCommand } from '@aws-sdk/client-s3';
import { McpServer } from '@modelcontextprotocol/sdk/server/mcp.js';
import { z } from 'zod';
import { Readable } from 'node:stream';
import * as readline from 'node:readline';
const server = new McpServer({ name: 'financial-events', version: '1.0.0' });
const s3 = new S3Client({ region: 'us-east-1' });
// Tool 1: submit documents for events extraction — returns jobId immediately
server.tool('extract_events_start', {
document_texts: z.array(z.string()).min(1).max(1000),
event_types: z.array(z.enum([
'ACQUISITION', 'CORPORATE_ACQUISITION', 'EMPLOYMENT', 'IPO',
'BOARD_CHANGE', 'BANKRUPTCY', 'BUSINESS_EXPANSION', 'BUSINESS_REDUCTION',
'DISSOLVED', 'RIGHTS_ISSUE', 'SECONDARY_OFFERING',
])).default(['ACQUISITION', 'CORPORATE_ACQUISITION', 'EMPLOYMENT', 'BOARD_CHANGE']),
}, async ({ document_texts, event_types }) => {
const inputKey = `events-input/${Date.now()}.txt`;
// Write documents to S3 as newline-separated file
await s3.send(new PutObjectCommand({
Bucket: process.env.COMPREHEND_BUCKET!,
Key: inputKey,
Body: document_texts.join('\n'),
ContentType: 'text/plain',
}));
const { JobId } = await client.send(new StartEventsDetectionJobCommand({
JobName: `events-${Date.now()}`,
DataAccessRoleArn: process.env.COMPREHEND_ROLE_ARN!,
LanguageCode: LanguageCode.EN,
TargetEventTypes: event_types,
InputDataConfig: {
S3Uri: `s3://${process.env.COMPREHEND_BUCKET!}/${inputKey}`,
InputFormat: 'ONE_DOC_PER_LINE',
},
OutputDataConfig: {
S3Uri: `s3://${process.env.COMPREHEND_BUCKET!}/events-output/`,
},
}));
return {
content: [{
type: 'text',
text: JSON.stringify({
jobId: JobId,
status: 'submitted',
document_count: document_texts.length,
message: 'Call extract_events_results with this jobId to retrieve results (typically ready in 2–10 minutes)',
}),
}],
};
});
// Tool 2: check status and retrieve structured events when ready
server.tool('extract_events_results', {
job_id: z.string(),
}, async ({ job_id }) => {
const { EventsDetectionJobProperties: props } = await client.send(
new DescribeEventsDetectionJobCommand({ JobId: job_id })
);
const status = props?.JobStatus;
if (status !== 'COMPLETED') {
return {
content: [{
type: 'text',
text: JSON.stringify({
jobId: job_id,
status: status ?? 'UNKNOWN',
message: status === 'FAILED'
? `Job failed: ${props?.Message ?? 'no message'}`
: 'Job not yet complete — retry in 30–60 seconds',
}),
}],
};
}
// Read and parse output JSONL from S3
const outputUri = props.OutputDataConfig?.S3Uri!;
const [bucket, ...keyParts] = outputUri.replace('s3://', '').split('/');
const key = `${keyParts.join('/')}/output`;
const s3Obj = await s3.send(new GetObjectCommand({ Bucket: bucket, Key: key }));
const stream = s3Obj.Body as Readable;
const allEvents: StructuredEvent[] = [];
const rl = readline.createInterface({ input: stream });
for await (const line of rl) {
if (!line.trim()) continue;
const doc = JSON.parse(line) as DocumentEventsOutput;
const docEvents = deduplicateByTriggerOffset(parseDocumentEvents(doc));
allEvents.push(...docEvents);
}
return {
content: [{
type: 'text',
text: JSON.stringify({
jobId: job_id,
status: 'COMPLETED',
event_count: allEvents.length,
events: allEvents.map(e => ({
type: e.type,
trigger: e.trigger,
confidence: e.triggerConfidence,
subject: e.subject,
object: e.object,
date: e.date,
artifact: e.artifact,
})),
}),
}],
};
});
The two-tool pattern is necessary because Events detection is async-only and jobs take 2–15 minutes depending on document volume. Blocking inside a single tool handler for 10 minutes is impractical — it holds the MCP server's tool call thread, may exceed orchestrator timeouts, and prevents the AI agent from doing other work while waiting. The submit/retrieve split lets the orchestrator poll at its own pace and parallelize other tasks. The complete Events API reference including all 11 event type role vocabularies, output deduplication, and async job lifecycle management is in AWS Comprehend Events Detection in MCP Servers.
Production checklist: Comprehend Advanced NLP stack
| Area | Check | Why it matters |
|---|---|---|
| PII gate | ContainsPiiEntities called before any log write or external API call for user-supplied text | AWS credential leaks to CloudWatch are a common accidental exposure vector |
| PII byte limit | Input split into ≤4,500 UTF-8 byte chunks before calling sync PII APIs | 5,000 byte hard limit — silently returns empty Labels if exceeded |
| Redaction direction | PII spans processed right-to-left (sorted descending by BeginOffset) | Left-to-right replacement corrupts all downstream offsets after the first substitution |
| Comprehend IAM role | Trust policy has aws:SourceAccount condition for comprehend.amazonaws.com | Prevents confused deputy attacks on the data access role |
| Entity type names | Entity types match [A-Z_]+ regex only | Digits or hyphens in type names cause CreateEntityRecognizer to fail |
| Annotation volume | ≥200 annotated examples per entity type for production use | Models trained with fewer examples often achieve F1 below 0.70 threshold |
| F1 gate | EvaluationMetrics.F1Score >= 0.80 before deploying endpoint to production | Below 0.70 F1 is near-random for many domains — check per-type breakdown first |
| Offset precision | Annotation spans verified: line.slice(begin, end) === entityText | Systematic off-by-one errors are the most common cause of low F1 scores |
| Classification threshold | Route to fallback handler when Classes[0].Score < 0.70 | Low-confidence routing is worse than no routing — creates silent misclassification |
| Events deduplication | ACQUISITION + CORPORATE_ACQUISITION deduplicated by trigger BeginOffset/EndOffset | Both types may fire for the same sentence, producing duplicate structured events |
| Events thresholds | Trigger GroupScore >= 0.70 and role Score >= 0.75 applied independently | High trigger confidence with low role score = event happened but participant unclear — don't assign wrong entity |
| Async pattern | Events tools use submit/retrieve split — no blocking for job completion inside tool handler | Events jobs take 2–15 min — single blocking tool would exceed orchestrator timeouts |