AWS Polly Text-to-Speech in MCP Servers — SynthesizeSpeechCommand, Voice Selection, Output Formats

Basic tool implementation

import {
  PollyClient,
  SynthesizeSpeechCommand,
  OutputFormat,
  Engine,
  VoiceId,
} from '@aws-sdk/client-polly';
import { S3Client, PutObjectCommand } from '@aws-sdk/client-s3';
import { McpServer } from '@modelcontextprotocol/sdk/server/mcp.js';
import { z } from 'zod';

const polly = new PollyClient({ region: 'us-east-1' });
const s3 = new S3Client({ region: 'us-east-1' });
const server = new McpServer({ name: 'polly-tools', version: '1.0.0' });

server.tool('synthesize_speech', {
  text: z.string().max(3000).describe('Text to synthesize (max 3000 chars; use synthesize_speech_long for longer text)'),
  voice_id: z.enum(['Joanna', 'Matthew', 'Ivy', 'Kendra', 'Kimberly', 'Salli', 'Joey', 'Justin', 'Kevin']).default('Joanna'),
  output_bucket: z.string().describe('S3 bucket to store the audio file'),
  output_key: z.string().optional().describe('S3 key; defaults to audio/{timestamp}.mp3'),
}, async ({ text, voice_id, output_bucket, output_key }) => {
  const key = output_key ?? `audio/${Date.now()}.mp3`;

  const { AudioStream, ContentType } = await polly.send(new SynthesizeSpeechCommand({
    Text: text,
    VoiceId: voice_id as VoiceId,
    OutputFormat: OutputFormat.MP3,
    Engine: Engine.NEURAL,
    SampleRate: '22050',   // 22050 Hz is the default and recommended for mp3
  }));

  if (!AudioStream) throw new Error('No audio stream returned from Polly');

  // Collect stream into a buffer
  const chunks: Uint8Array[] = [];
  for await (const chunk of AudioStream as AsyncIterable<Uint8Array>) {
    chunks.push(chunk);
  }
  const audioBuffer = Buffer.concat(chunks);

  await s3.send(new PutObjectCommand({
    Bucket: output_bucket,
    Key: key,
    Body: audioBuffer,
    ContentType: ContentType ?? 'audio/mpeg',
  }));

  return {
    content: [{
      type: 'text',
      text: JSON.stringify({
        s3_uri: `s3://${output_bucket}/${key}`,
        bytes: audioBuffer.byteLength,
        voice: voice_id,
        engine: 'neural',
        format: 'mp3',
      }, null, 2),
    }],
  };
});

Output formats and sample rates

The SampleRate parameter applies only to mp3, ogg_vorbis, and pcm formats. Specifying an invalid combination (e.g. pcm at 22050 Hz) causes a InvalidSampleRateException. For telephony integrations via Amazon Connect or Chime, use pcm at 8000 Hz. For high-quality web audio, use mp3 at 22050 or 24000 Hz.

Neural voices ignore the SampleRate for internal synthesis — the specified rate is used only for the output encoding. Requesting a lower sample rate from a neural voice still incurs the neural synthesis cost; it doesn't save time, only file size.

Chunking text longer than 3,000 characters

Each SynthesizeSpeechCommand call accepts at most 3,000 UTF-8 characters for plain text (6,000 for SSML). Longer content requires splitting into chunks, synthesizing each chunk separately, and concatenating the audio buffers. For seamless concatenation, split on sentence boundaries rather than at the character limit:

function splitIntoChunks(text: string, maxChars = 2900): string[] {
  const sentences = text.match(/[^.!?]+[.!?]+/g) ?? [text];
  const chunks: string[] = [];
  let current = '';

  for (const sentence of sentences) {
    if ((current + sentence).length > maxChars) {
      if (current) chunks.push(current.trim());
      current = sentence;
    } else {
      current += sentence;
    }
  }
  if (current.trim()) chunks.push(current.trim());
  return chunks;
}

async function synthesizeLong(text: string, voiceId: VoiceId): Promise<Buffer> {
  const chunks = splitIntoChunks(text);
  const buffers: Buffer[] = [];

  for (const chunk of chunks) {
    const { AudioStream } = await polly.send(new SynthesizeSpeechCommand({
      Text: chunk,
      VoiceId: voiceId,
      OutputFormat: OutputFormat.MP3,
      Engine: Engine.NEURAL,
    }));
    if (!AudioStream) throw new Error('No audio stream');
    const parts: Uint8Array[] = [];
    for await (const part of AudioStream as AsyncIterable<Uint8Array>) parts.push(part);
    buffers.push(Buffer.concat(parts));
  }

  // MP3 frames can be concatenated directly — decoders handle multi-segment files
  return Buffer.concat(buffers);
}

MP3 and OGG Vorbis audio buffers from consecutive SynthesizeSpeechCommand calls can be concatenated directly — most audio players and decoders handle multi-segment files correctly. For PCM, raw sample concatenation also works without headers since PCM is headerless. For long-form documents, consider using StartSpeechSynthesisTaskCommand instead, which accepts up to 100,000 characters and delivers output directly to S3 without manual chunking.

Voice selection

Call GetVoicesCommand at startup to fetch the complete voice catalog for your region. Filter by Engine and LanguageCode to build a voice picker for your application. Not all voices are available in all regions — Polly is a global service but regional deployments can differ on neural and generative voice availability. See the neural engine guide for a full comparison of engine tiers.

Returning audio to an AI agent

MCP tools return structured text, not binary streams. For audio output, two patterns work in practice:

Pattern 1 — S3 pre-signed URL: Upload the audio to S3 and return a pre-signed GetObject URL. The agent can pass this URL to a client or embed it in a response. This is the preferred pattern — the audio never touches the MCP transport layer.

import { getSignedUrl } from '@aws-sdk/s3-request-presigner';
import { GetObjectCommand } from '@aws-sdk/client-s3';

const url = await getSignedUrl(s3, new GetObjectCommand({
  Bucket: output_bucket,
  Key: key,
}), { expiresIn: 3600 }); // 1-hour expiry

return {
  content: [{
    type: 'text',
    text: JSON.stringify({ audio_url: url, expires_in_seconds: 3600 }),
  }],
};

Pattern 2 — base64 inline: For short clips (under ~100 KB), encode the audio buffer as base64 and return it in the tool result. The agent or client decodes it. This avoids the extra S3 upload but inflates the response size by ~33% and can exceed MCP message limits for longer audio.

return {
  content: [{
    type: 'text',
    text: JSON.stringify({
      audio_base64: audioBuffer.toString('base64'),
      content_type: 'audio/mpeg',
      bytes: audioBuffer.byteLength,
    }),
  }],
};

IAM permissions

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "polly:SynthesizeSpeech",
        "polly:GetVoices"
      ],
      "Resource": "*"
    },
    {
      "Effect": "Allow",
      "Action": ["s3:PutObject"],
      "Resource": "arn:aws:s3:::your-audio-bucket/audio/*"
    }
  ]
}

polly:SynthesizeSpeech and polly:GetVoices do not support resource-level restrictions — the "Resource": "*" is required. Polly does not store your text; it processes it in-memory and returns the stream directly. Add polly:StartSpeechSynthesisTask, polly:GetSpeechSynthesisTask, and polly:ListSpeechSynthesisTasks for the async task API.