AWS Polly Text-to-Speech for MCP Servers: Synthesis, Engine Selection, SSML, and Async Tasks

Pattern 1: Basic synthesis — SynthesizeSpeechCommand, voice selection, output formats, and chunking

When an AI agent needs to convert a short notification, a conversational response, or a UI label into audio, basic synthesis with SynthesizeSpeechCommand is the right starting point. The call is synchronous — Polly processes the text in-memory and returns the audio stream directly in the response. The fundamental design constraint is the 3,000 UTF-8 character limit for plain-text input (6,000 for SSML). For most notification and conversational use cases this limit is irrelevant; for document synthesis it requires chunking.

Basic tool implementation

import {
  PollyClient,
  SynthesizeSpeechCommand,
  OutputFormat,
  Engine,
  VoiceId,
} from '@aws-sdk/client-polly';
import { S3Client, PutObjectCommand } from '@aws-sdk/client-s3';
import { getSignedUrl } from '@aws-sdk/s3-request-presigner';
import { GetObjectCommand } from '@aws-sdk/client-s3';
import { McpServer } from '@modelcontextprotocol/sdk/server/mcp.js';
import { z } from 'zod';

const polly = new PollyClient({ region: 'us-east-1' });
const s3 = new S3Client({ region: 'us-east-1' });
const server = new McpServer({ name: 'polly-tools', version: '1.0.0' });

server.tool('synthesize_speech', {
  text: z.string().max(3000).describe('Text to synthesize (max 3000 chars)'),
  voice_id: z.enum(['Joanna', 'Matthew', 'Ivy', 'Kendra', 'Salli', 'Joey', 'Ruth', 'Stephen'])
    .default('Joanna'),
  output_bucket: z.string().describe('S3 bucket to store the audio file'),
}, async ({ text, voice_id, output_bucket }) => {
  const key = `audio/${Date.now()}.mp3`;

  const { AudioStream, ContentType } = await polly.send(new SynthesizeSpeechCommand({
    Text: text,
    VoiceId: voice_id as VoiceId,
    OutputFormat: OutputFormat.MP3,
    Engine: Engine.NEURAL,
    SampleRate: '22050',
  }));

  if (!AudioStream) throw new Error('No audio stream returned from Polly');

  const chunks: Uint8Array[] = [];
  for await (const chunk of AudioStream as AsyncIterable<Uint8Array>) {
    chunks.push(chunk);
  }
  const audioBuffer = Buffer.concat(chunks);

  await s3.send(new PutObjectCommand({
    Bucket: output_bucket,
    Key: key,
    Body: audioBuffer,
    ContentType: ContentType ?? 'audio/mpeg',
  }));

  const url = await getSignedUrl(s3, new GetObjectCommand({
    Bucket: output_bucket,
    Key: key,
  }), { expiresIn: 3600 });

  return {
    content: [{
      type: 'text',
      text: JSON.stringify({
        audio_url: url,
        expires_in_seconds: 3600,
        bytes: audioBuffer.byteLength,
        voice: voice_id,
        engine: 'neural',
      }, null, 2),
    }],
  };
});

Output format and sample rate compatibility

The combination of OutputFormat and SampleRate determines the size, quality, and downstream compatibility of the audio file. Specifying an invalid combination causes an InvalidSampleRateException — Polly validates this server-side before synthesis begins:

Neural voices ignore SampleRate for internal synthesis — the specified rate affects only the output encoding. Requesting a lower sample rate from a neural voice still incurs the full neural synthesis cost and does not reduce processing time; it only reduces file size.

Chunking text longer than 3,000 characters

The critical mistake teams make with chunking is splitting at the character limit — which cuts sentences mid-word, creating an audible click or abrupt pause at the join point. The correct approach is to split on sentence boundaries and keep each chunk under the limit:

function splitIntoChunks(text: string, maxChars = 2900): string[] {
  const sentences = text.match(/[^.!?]+[.!?]+/g) ?? [text];
  const chunks: string[] = [];
  let current = '';

  for (const sentence of sentences) {
    if ((current + sentence).length > maxChars) {
      if (current) chunks.push(current.trim());
      current = sentence;
    } else {
      current += sentence;
    }
  }
  if (current.trim()) chunks.push(current.trim());
  return chunks;
}

async function synthesizeLong(text: string, voiceId: VoiceId): Promise<Buffer> {
  const chunks = splitIntoChunks(text);
  const buffers: Buffer[] = [];

  for (const chunk of chunks) {
    const { AudioStream } = await polly.send(new SynthesizeSpeechCommand({
      Text: chunk,
      VoiceId: voiceId,
      OutputFormat: OutputFormat.MP3,
      Engine: Engine.NEURAL,
    }));
    if (!AudioStream) throw new Error('No audio stream');
    const parts: Uint8Array[] = [];
    for await (const part of AudioStream as AsyncIterable<Uint8Array>) parts.push(part);
    buffers.push(Buffer.concat(parts));
  }

  // MP3 frames from consecutive calls concatenate cleanly — decoders handle multi-segment files
  return Buffer.concat(buffers);
}

MP3 and OGG Vorbis buffers from consecutive calls can be concatenated directly — most audio players handle multi-segment files correctly. For PCM, raw sample concatenation also works without headers since PCM is headerless. For long-form documents (articles, book chapters, full-document narration), use StartSpeechSynthesisTaskCommand instead — it accepts up to 100,000 characters and delivers output directly to S3 without manual chunking.

Returning audio to an AI agent

MCP tools return structured text, not binary streams. Two patterns work in practice for returning audio to the calling agent:

Pattern 1 — S3 pre-signed URL (preferred for all clips): Upload to S3 and return a pre-signed GetObject URL with a 1-hour expiry. The audio never touches the MCP transport layer. The agent passes the URL to a client or embeds it in a response.

Pattern 2 — base64 inline (acceptable for short clips under ~100 KB): Encode the buffer as base64 and return it in the tool result. This avoids the extra S3 upload but inflates response size by ~33% and can exceed MCP message limits for longer audio. Use only for sub-5-second clips such as notification sounds or short prompts.

Pattern 2: Engine selection — standard vs neural vs long-form vs generative

The choice of Engine is the most consequential Polly decision an MCP team makes. It determines audio quality, latency, cost, and which SSML features are available. Getting it wrong for the content type produces either robotic output that users find jarring, or unnecessarily high costs for a quality level the use case doesn't require. The four engine tiers map cleanly to four content categories.

The long-form engine costs the same per character as neural — the choice between them is purely about quality for the content type. For a single-sentence notification, neural is indistinguishable from long-form. For a five-paragraph article narration, long-form with Ruth or Stephen produces markedly better pacing and prosody because it was trained on narration-style content rather than conversational audio. The generative engine should be reserved for products where human-likeness is a primary differentiator — a voice assistant product where users will notice robotic output, or a content narration service where audio quality directly affects retention. At 20–30× the standard cost, it cannot be used in high-volume paths without careful cost analysis.

Discovering the voice catalog with GetVoicesCommand

import {
  PollyClient,
  GetVoicesCommand,
  Engine,
  LanguageCode,
} from '@aws-sdk/client-polly';

const polly = new PollyClient({ region: 'us-east-1' });

// Cache at startup — the voice catalog changes infrequently
let voiceCache: Map<string, string[]> | null = null;

async function getVoicesForEngine(
  engine: Engine,
  languageCode?: LanguageCode,
) {
  const { Voices } = await polly.send(new GetVoicesCommand({
    Engine: engine,
    LanguageCode: languageCode,
    IncludeAdditionalLanguageCodes: true,
  }));

  return (Voices ?? []).map(v => ({
    id: v.Id,
    name: v.Name,
    gender: v.Gender,
    language: v.LanguageCode,
    supportedEngines: v.SupportedEngines,
    additionalLanguages: v.AdditionalLanguageCodes,
  }));
}

// List long-form voices (English-only as of late 2026)
// Returns: Ruth (en-US, female), Stephen (en-US, male)
const longFormVoices = await getVoicesForEngine(Engine.LONG_FORM);

// List neural voices for US English
// Returns: Joanna, Matthew, Ivy, Kendra, Kimberly, Salli, Joey, Justin, Kevin
const neuralUsVoices = await getVoicesForEngine(Engine.NEURAL, LanguageCode.EN_US);

The SupportedEngines field on each voice object is the authoritative source of truth for which engines that voice supports — not all voices listed for one engine tier are available for another. Calling SynthesizeSpeechCommand with an engine the voice doesn't support raises InvalidParameterValueException: Engine not supported by voice. The Kevin voice (en-US, male child) is neural-only — there is no standard variant. Amy (en-GB, female) is one of the few voices available on the generative engine.

Engine selection by content type

type ContentType = 'notification' | 'conversation' | 'article' | 'presentation';

function pickEngine(contentType: ContentType): { engine: string; voiceId: string } {
  switch (contentType) {
    case 'notification':
      // Short, single-sentence alerts — neural is natural enough; standard cuts cost
      return { engine: 'neural', voiceId: 'Joanna' };

    case 'conversation':
      // Multi-turn dialogue, chatbot responses — consistent voice per persona
      return { engine: 'neural', voiceId: 'Matthew' };

    case 'article':
      // Blog posts, documents, 3+ paragraphs — long-form narrates better
      return { engine: 'long-form', voiceId: 'Ruth' };

    case 'presentation':
      // Slide narration or explainer video — long-form male voice
      return { engine: 'long-form', voiceId: 'Stephen' };
  }
}

Multi-language voice strategy

Some Polly voices support a secondary language via the LanguageCode override parameter on SynthesizeSpeechCommand — a voice trained primarily on US English can synthesize French text with a French accent. This is not the same as a native French voice: pronunciation is accented. For best results with non-English content, use a voice whose primary LanguageCode matches the input text. The AdditionalLanguageCodes field lists secondary languages the voice can synthesize with an accent — useful for mixed-language documents where a fully native voice isn't available. Neural voice availability varies by language — many non-English languages only have standard voices or limited neural coverage. Run GetVoicesCommand at startup and store the result to validate your language-to-voice map against what's actually available in your deployed region.

Pattern 3: SSML markup — prosody, phoneme, say-as, pauses, and XML escaping

SSML (Speech Synthesis Markup Language) is an XML vocabulary that gives MCP server tools precise control over how Polly speaks a given text — adding pauses between sections, slowing down for important numbers, correcting mispronounced product names, and formatting dates or phone numbers as a human would speak them. Using SSML instead of plain text unlocks two advantages: the character limit doubles to 6,000 characters, and you get fine-grained control over timing and pronunciation that plain text cannot express. The critical production discipline is XML character escaping — an unescaped ampersand in a product name or URL inserted into an SSML template causes InvalidSsmlException and silent synthesis failure.

SSML tool with escaping helper

import {
  PollyClient,
  SynthesizeSpeechCommand,
  OutputFormat,
  Engine,
  TextType,
  VoiceId,
} from '@aws-sdk/client-polly';
import { McpServer } from '@modelcontextprotocol/sdk/server/mcp.js';
import { z } from 'zod';

const polly = new PollyClient({ region: 'us-east-1' });
const server = new McpServer({ name: 'polly-ssml-tools', version: '1.0.0' });

// Always escape user-supplied text before inserting into SSML templates
function escapeXml(str: string): string {
  return str
    .replace(/&/g, '&')
    .replace(/</g, '&lt;')
    .replace(/>/g, '&gt;')
    .replace(/"/g, '&quot;')
    .replace(/'/g, '&apos;');
}

server.tool('speak_notification', {
  title: z.string().describe('Alert title — will be spoken at slow rate'),
  body: z.string().describe('Alert body text'),
  code: z.string().optional().describe('Numeric code to read digit-by-digit (e.g. error code, verification code)'),
  voice_id: z.string().default('Joanna'),
}, async ({ title, body, code, voice_id }) => {
  const ssml = `<speak>
  <prosody rate="slow" volume="loud">${escapeXml(title)}</prosody>
  <break time="800ms"/>
  ${escapeXml(body)}
  ${code ? `<break time="500ms"/>Code: <prosody rate="x-slow"><say-as interpret-as="digits">${escapeXml(code)}</say-as></prosody>` : ''}
</speak>`;

  const { AudioStream } = await polly.send(new SynthesizeSpeechCommand({
    Text: ssml,
    TextType: TextType.SSML,
    VoiceId: voice_id as VoiceId,
    OutputFormat: OutputFormat.MP3,
    Engine: Engine.NEURAL,
  }));

  if (!AudioStream) throw new Error('No audio stream');
  const chunks: Uint8Array[] = [];
  for await (const chunk of AudioStream as AsyncIterable<Uint8Array>) chunks.push(chunk);
  const buf = Buffer.concat(chunks);

  return {
    content: [{ type: 'text', text: JSON.stringify({
      audio_base64: buf.toString('base64'),
      content_type: 'audio/mpeg',
      bytes: buf.byteLength,
    }) }],
  };
});

Pauses with <break>

The <break> element inserts silence. It accepts either a time attribute (in seconds or milliseconds, up to 10 seconds) or a strength attribute for preset relative durations:

Prosody: rate, pitch, and volume

Prosody tags can be nested — inner tags take precedence over outer tags for the same attribute. Keep pitch adjustments under ±10% for natural-sounding output — extreme values produce an unnatural chipmunk or low-robot effect. The standard engine supports prosody more reliably than neural; the neural engine may ignore subtle rate and pitch adjustments and apply its own prosodic model instead.

<speak>
  Your verification code is
  <prosody rate="x-slow" volume="loud">
    <say-as interpret-as="digits">482910</say-as>
  </prosody>.
  <break time="800ms"/>
  <prosody rate="fast" volume="soft">
    Terms and conditions apply. See website for details.
  </prosody>
</speak>

Pronunciation with <phoneme>

Domain-specific terms, product names, and proper nouns are frequently mispronounced. Use <phoneme> with International Phonetic Alphabet (IPA) to override the default pronunciation. IPA transcriptions are not validated by the API — a wrong IPA string silently produces garbled audio; test in the Polly console before shipping to production:

<speak>
  Welcome to <phoneme alphabet="ipa" ph="ˈeɪlɪvˌɛm.siˈpiː">AliveMCP</phoneme>,
  the uptime monitoring dashboard for
  <phoneme alphabet="ipa" ph="ˌɛm.siˈpiː">MCP</phoneme> servers.
</speak>

Formatting with <say-as>

The <say-as interpret-as> element tells Polly how to interpret and speak a text value. Without it, Polly uses heuristics that are frequently wrong for structured data — it may read 404 as "four hundred and four" instead of "four zero four", or read API as a word instead of spelling out the letters:

Acronym expansion with <sub> and inline language switching with <lang>

Use <sub alias="sequel">SQL</sub> to replace a term with an alternative pronunciation without changing the displayed text — unlike <phoneme> which requires IPA, <sub alias> takes a plain-text string that Polly synthesizes normally. This is easier to use for most substitutions: write what you want spoken, not a phonetic encoding. <sub> is not supported by the generative engine.

Use <lang xml:lang="fr-FR">bonjour</lang> to switch pronunciation rules for enclosed text. This requires that the selected VoiceId supports the target language as an additional language — query GetVoicesCommand and inspect AdditionalLanguageCodes for the voice you're using.

Common SSML pitfalls

Pattern 4: Async tasks and speech marks — StartSpeechSynthesisTaskCommand, polling, and subtitle generation

When synthesizing long-form documents, articles, or any content where the chunking-and-concatenate approach adds too much complexity, StartSpeechSynthesisTaskCommand is the right tool. It accepts up to 100,000 characters per task, runs synthesis asynchronously in the background, and writes the output audio directly to an S3 key — no AudioStream to collect, no manual chunking, no buffer concatenation. The same call can produce a parallel speech marks file containing word-level timestamps that enable synchronized subtitles, audio players with word highlighting, and avatar lip-sync animation.

Async task implementation with polling

import {
  PollyClient,
  StartSpeechSynthesisTaskCommand,
  GetSpeechSynthesisTaskCommand,
  OutputFormat,
  Engine,
  VoiceId,
  SpeechMarkType,
  TaskStatus,
} from '@aws-sdk/client-polly';
import { McpServer } from '@modelcontextprotocol/sdk/server/mcp.js';
import { z } from 'zod';

const polly = new PollyClient({ region: 'us-east-1' });
const server = new McpServer({ name: 'polly-async-tools', version: '1.0.0' });

// Tool 1: submit the task and return immediately
server.tool('synthesize_long_text_start', {
  text: z.string().max(100000).describe('Text to synthesize (up to 100,000 characters)'),
  output_bucket: z.string(),
  output_key_prefix: z.string().default('polly-output/'),
  voice_id: z.enum(['Ruth', 'Stephen', 'Joanna', 'Matthew']).default('Ruth'),
  include_speech_marks: z.boolean().default(false),
}, async ({ text, output_bucket, output_key_prefix, voice_id, include_speech_marks }) => {
  const { SynthesisTask } = await polly.send(new StartSpeechSynthesisTaskCommand({
    Text: text,
    VoiceId: voice_id as VoiceId,
    OutputFormat: OutputFormat.MP3,
    Engine: voice_id === 'Ruth' || voice_id === 'Stephen' ? Engine.LONG_FORM : Engine.NEURAL,
    OutputS3BucketName: output_bucket,
    OutputS3KeyPrefix: output_key_prefix,
    SpeechMarkTypes: include_speech_marks
      ? [SpeechMarkType.WORD, SpeechMarkType.SENTENCE]
      : undefined,
  }));

  if (!SynthesisTask?.TaskId) throw new Error('No TaskId returned');

  return {
    content: [{
      type: 'text',
      text: JSON.stringify({
        task_id: SynthesisTask.TaskId,
        status: SynthesisTask.TaskStatus,
        // Store task_id — task metadata expires after 7 days
      }, null, 2),
    }],
  };
});

// Tool 2: poll for completion and return result when ready
server.tool('synthesize_long_text_result', {
  task_id: z.string().describe('TaskId returned by synthesize_long_text_start'),
}, async ({ task_id }) => {
  const { SynthesisTask } = await polly.send(
    new GetSpeechSynthesisTaskCommand({ TaskId: task_id })
  );

  if (!SynthesisTask) throw new Error('Task not found');

  if (SynthesisTask.TaskStatus === TaskStatus.FAILED) {
    return {
      content: [{
        type: 'text',
        text: JSON.stringify({ error: 'FAILED', reason: SynthesisTask.TaskStatusReason }),
      }],
      isError: true,
    };
  }

  if (SynthesisTask.TaskStatus !== TaskStatus.COMPLETED) {
    return {
      content: [{
        type: 'text',
        text: JSON.stringify({ status: SynthesisTask.TaskStatus, task_id }),
      }],
    };
  }

  return {
    content: [{
      type: 'text',
      text: JSON.stringify({
        task_id,
        audio_uri: SynthesisTask.OutputUri,
        speech_marks_uri: SynthesisTask.OutputUri?.replace('.mp3', '.mp3.marks'),
        creation_time: SynthesisTask.CreationTime,
        completed_time: SynthesisTask.CompletionTime,
        voice: SynthesisTask.VoiceId,
        engine: SynthesisTask.Engine,
      }, null, 2),
    }],
  };
});

Task lifecycle and S3 output location

The OutputUri is an S3 URI in the format s3://bucket/prefix<taskId>.mp3. Polly generates the key automatically — you cannot specify an exact output key, only a prefix. If you requested SpeechMarkTypes, the speech marks file is at the same key with .marks appended: s3://bucket/prefix<taskId>.mp3.marks. Store OutputUri in your database at task completion rather than relying on being able to re-query the task metadata later — Polly cleans up task records after 7 days.

Speech marks — word-level timing for subtitles and lip-sync

Speech marks are a JSON Lines file (one JSON object per line) where each line is a timing event. The word type gives you one object per word with its start time in milliseconds from audio start; the sentence type gives you sentence boundaries for subtitle cue groups:

{"time":0,"type":"sentence","start":0,"end":89,"value":"Hello, welcome to AliveMCP."}
{"time":0,"type":"word","start":0,"end":5,"value":"Hello"}
{"time":420,"type":"word","start":7,"end":14,"value":"welcome"}
{"time":820,"type":"word","start":15,"end":17,"value":"to"}
{"time":1010,"type":"word","start":18,"end":26,"value":"AliveMCP"}
{"time":1620,"type":"sentence","start":91,"end":140,"value":"Your server is healthy."}
{"time":1620,"type":"word","start":91,"end":95,"value":"Your"}
{"time":1850,"type":"word","start":96,"end":102,"value":"server"}
{"time":2200,"type":"word","start":103,"end":105,"value":"is"}
{"time":2340,"type":"word","start":106,"end":113,"value":"healthy"}

Parsing speech marks for WebVTT subtitle generation

import { S3Client, GetObjectCommand } from '@aws-sdk/client-s3';

const s3 = new S3Client({ region: 'us-east-1' });

interface SpeechMark {
  time: number;
  type: 'word' | 'sentence' | 'viseme' | 'ssml';
  start: number;
  end: number;
  value: string;
}

async function parseSpeechMarks(bucket: string, key: string): Promise<SpeechMark[]> {
  const { Body } = await s3.send(new GetObjectCommand({ Bucket: bucket, Key: key }));
  const chunks: Uint8Array[] = [];
  for await (const chunk of Body as AsyncIterable<Uint8Array>) chunks.push(chunk);
  const text = Buffer.concat(chunks).toString('utf-8');

  // JSON Lines — one JSON object per line
  return text
    .split('\n')
    .filter(line => line.trim())
    .map(line => JSON.parse(line) as SpeechMark);
}

function formatVttTime(ms: number): string {
  const totalSec = Math.floor(ms / 1000);
  const h = Math.floor(totalSec / 3600);
  const m = Math.floor((totalSec % 3600) / 60);
  const s = totalSec % 60;
  const msRemainder = ms % 1000;
  return `${String(h).padStart(2,'0')}:${String(m).padStart(2,'0')}:${String(s).padStart(2,'0')}.${String(msRemainder).padStart(3,'0')}`;
}

function buildWebVTT(marks: SpeechMark[]): string {
  const sentences = marks.filter(m => m.type === 'sentence');

  const cues = sentences.map((sentence, i) => {
    const nextSentence = sentences[i + 1];
    const endTime = nextSentence ? nextSentence.time : sentence.time + 5000;
    return `${formatVttTime(sentence.time)} --> ${formatVttTime(endTime)}\n${sentence.value}`;
  });

  return `WEBVTT\n\n${cues.join('\n\n')}`;
}

SNS notification instead of polling

For high-volume workloads where you're running many concurrent synthesis tasks, polling GetSpeechSynthesisTaskCommand per task adds overhead. Supply SnsTopicArn to StartSpeechSynthesisTaskCommand and Polly publishes the full SynthesisTask object to your SNS topic on completion or failure. The SNS topic must grant Polly permission to publish — add a resource-based policy with Service: polly.amazonaws.com as the principal and a Condition: StringEquals: "aws:SourceAccount": "<your-account-id>" to prevent confused-deputy attacks. Polling remains the simpler implementation for low-volume MCP tools.

IAM permissions for async tasks

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "polly:SynthesizeSpeech",
        "polly:GetVoices",
        "polly:StartSpeechSynthesisTask",
        "polly:GetSpeechSynthesisTask",
        "polly:ListSpeechSynthesisTasks"
      ],
      "Resource": "*"
    },
    {
      "Effect": "Allow",
      "Action": ["s3:PutObject", "s3:GetObject"],
      "Resource": "arn:aws:s3:::your-output-bucket/polly-output/*"
    }
  ]
}

All Polly actions require "Resource": "*" — Polly does not support resource-level restrictions. Polly uses your execution role's credentials to write to S3 — it does not assume a separate role. If the output bucket uses SSE-KMS, grant kms:GenerateDataKey and kms:Decrypt for the KMS key to the same role. The speech marks .marks file is written to the same bucket and key prefix as the audio, so the same S3 permissions cover both files.