AWS Polly Async Speech Synthesis Tasks and Speech Marks in MCP Servers — StartSpeechSynthesisTask, Word Timing

Async task implementation

import {
  PollyClient,
  StartSpeechSynthesisTaskCommand,
  GetSpeechSynthesisTaskCommand,
  OutputFormat,
  Engine,
  VoiceId,
  SpeechMarkType,
  TaskStatus,
} from '@aws-sdk/client-polly';
import { S3Client, GetObjectCommand } from '@aws-sdk/client-s3';
import { McpServer } from '@modelcontextprotocol/sdk/server/mcp.js';
import { z } from 'zod';

const polly = new PollyClient({ region: 'us-east-1' });
const s3 = new S3Client({ region: 'us-east-1' });
const server = new McpServer({ name: 'polly-async-tools', version: '1.0.0' });

server.tool('synthesize_long_text', {
  text: z.string().max(100000).describe('Text to synthesize (up to 100,000 characters)'),
  output_bucket: z.string(),
  output_key_prefix: z.string().default('polly-output/'),
  include_speech_marks: z.boolean().default(false).describe('Also produce word/sentence timing marks'),
}, async ({ text, output_bucket, output_key_prefix, include_speech_marks }) => {
  const { SynthesisTask } = await polly.send(new StartSpeechSynthesisTaskCommand({
    Text: text,
    VoiceId: VoiceId.RUTH,
    OutputFormat: OutputFormat.MP3,
    Engine: Engine.LONG_FORM,
    OutputS3BucketName: output_bucket,
    OutputS3KeyPrefix: output_key_prefix,
    SpeechMarkTypes: include_speech_marks
      ? [SpeechMarkType.WORD, SpeechMarkType.SENTENCE]
      : undefined,
  }));

  if (!SynthesisTask?.TaskId) throw new Error('No TaskId returned');
  const taskId = SynthesisTask.TaskId;

  // Poll until complete (long-form documents take 30 seconds to several minutes)
  let task = SynthesisTask;
  for (let attempt = 0; attempt < 30; attempt++) {
    await new Promise(r => setTimeout(r, 15_000)); // 15-second poll interval

    const { SynthesisTask: updated } = await polly.send(
      new GetSpeechSynthesisTaskCommand({ TaskId: taskId })
    );
    task = updated!;

    if (task.TaskStatus === TaskStatus.COMPLETED) break;
    if (task.TaskStatus === TaskStatus.FAILED) {
      return {
        content: [{
          type: 'text',
          text: JSON.stringify({ error: 'FAILED', reason: task.TaskStatusReason }),
        }],
        isError: true,
      };
    }
  }

  if (task.TaskStatus !== TaskStatus.COMPLETED) {
    return {
      content: [{ type: 'text', text: JSON.stringify({ error: 'timeout', task_id: taskId }) }],
      isError: true,
    };
  }

  return {
    content: [{
      type: 'text',
      text: JSON.stringify({
        task_id: taskId,
        audio_uri: task.OutputUri,
        speech_marks_uri: include_speech_marks
          ? task.OutputUri?.replace('.mp3', '.mp3.marks')
          : undefined,
        creation_time: task.CreationTime,
        completed_time: task.CompletionTime,
        voice: task.VoiceId,
        engine: task.Engine,
      }, null, 2),
    }],
  };
});

Task lifecycle and S3 output location

The OutputUri in the completed task is an S3 URI in the format s3://bucket/prefix<taskId>.mp3. Polly generates the key automatically — you cannot specify an exact output key, only a prefix. The key contains the TaskId so it's always unique. If you requested SpeechMarkTypes, the speech marks file is at the same key with .marks appended: s3://bucket/prefix<taskId>.mp3.marks.

Polly stores completed task metadata for 7 days — after that, GetSpeechSynthesisTaskCommand returns a SynthesisTaskNotFoundException. The audio and marks files remain in S3 as long as your bucket lifecycle allows. Store the OutputUri in your database at task completion rather than relying on being able to re-query the task metadata later.

Speech marks — word-level timing output

Speech marks are a parallel output alongside the audio: a JSON Lines file (one JSON object per line) where each line represents a timing event. When SpeechMarkTypes includes 'word', Polly writes one object per word with its start time in milliseconds:

{"time":0,"type":"sentence","start":0,"end":89,"value":"Hello, welcome to AliveMCP."}
{"time":0,"type":"word","start":0,"end":5,"value":"Hello"}
{"time":420,"type":"word","start":7,"end":14,"value":"welcome"}
{"time":820,"type":"word","start":15,"end":17,"value":"to"}
{"time":1010,"type":"word","start":18,"end":26,"value":"AliveMCP"}
{"time":1620,"type":"sentence","start":91,"end":140,"value":"Your server is healthy."}
{"time":1620,"type":"word","start":91,"end":95,"value":"Your"}
{"time":1850,"type":"word","start":96,"end":102,"value":"server"}
{"time":2200,"type":"word","start":103,"end":105,"value":"is"}
{"time":2340,"type":"word","start":106,"end":113,"value":"healthy"}

Parsing speech marks for subtitle generation

import { S3Client, GetObjectCommand } from '@aws-sdk/client-s3';

interface SpeechMark {
  time: number;
  type: 'word' | 'sentence' | 'viseme' | 'ssml';
  start: number;
  end: number;
  value: string;
}

async function parseSpeechMarks(bucket: string, key: string): Promise<SpeechMark[]> {
  const { Body } = await s3.send(new GetObjectCommand({ Bucket: bucket, Key: key }));
  const chunks: Uint8Array[] = [];
  for await (const chunk of Body as AsyncIterable<Uint8Array>) chunks.push(chunk);
  const text = Buffer.concat(chunks).toString('utf-8');

  // JSON Lines — one JSON object per line
  return text
    .split('\n')
    .filter(line => line.trim())
    .map(line => JSON.parse(line) as SpeechMark);
}

// Build WebVTT subtitle cues from sentence and word marks
function buildWebVTT(marks: SpeechMark[]): string {
  const sentences = marks.filter(m => m.type === 'sentence');
  const words = marks.filter(m => m.type === 'word');

  const cues = sentences.map((sentence, i) => {
    const nextSentence = sentences[i + 1];
    const endTime = nextSentence ? nextSentence.time : sentence.time + 5000;

    // Find words in this sentence's character range
    const sentenceWords = words.filter(
      w => w.start >= sentence.start && w.end <= sentence.end
    );

    // Highlight the first word of this sentence (simplistic approach)
    const displayText = sentence.value;

    return `${formatVttTime(sentence.time)} --> ${formatVttTime(endTime)}\n${displayText}`;
  });

  return `WEBVTT\n\n${cues.join('\n\n')}`;
}

function formatVttTime(ms: number): string {
  const totalSec = Math.floor(ms / 1000);
  const h = Math.floor(totalSec / 3600);
  const m = Math.floor((totalSec % 3600) / 60);
  const s = totalSec % 60;
  const msRemainder = ms % 1000;
  return `${String(h).padStart(2,'0')}:${String(m).padStart(2,'0')}:${String(s).padStart(2,'0')}.${String(msRemainder).padStart(3,'0')}`;
}

Speech mark types

Speech mark types can be combined — requesting ['word', 'sentence', 'viseme'] emits all three event types interleaved in the JSON Lines output, sorted by time. The ssml type requires SSML input containing <mark name="..."> elements and is useful when you need to trigger external actions at specific audio positions: for example, advancing a presentation slide or showing a diagram when a specific paragraph is spoken.

When requesting speech marks without audio (for analysis only), set OutputFormat: 'json'. This produces only the JSON Lines file with no audio synthesis, at a fraction of the cost. This is useful for pre-computing timestamps from known scripts or validating SSML mark placement before running full synthesis.

Using SNS for async notification instead of polling

// Supply an SNS topic ARN to receive a notification when the task completes
const { SynthesisTask } = await polly.send(new StartSpeechSynthesisTaskCommand({
  Text: longFormText,
  VoiceId: VoiceId.STEPHEN,
  OutputFormat: OutputFormat.MP3,
  Engine: Engine.LONG_FORM,
  OutputS3BucketName: outputBucket,
  OutputS3KeyPrefix: 'documents/',
  SnsTopicArn: 'arn:aws:sns:us-east-1:123456789012:polly-task-complete',
}));

Polly publishes a JSON notification to the SNS topic when the task transitions to completed or failed. The message body contains the full SynthesisTask object including TaskId, TaskStatus, OutputUri, and TaskStatusReason. The SNS topic must grant Polly permission to publish — add a resource-based policy with Service: polly.amazonaws.com as the principal and Condition: StringEquals: "aws:SourceAccount": "<your-account-id>" to prevent confused-deputy attacks. Polling is fine for low-volume MCP tools; SNS becomes important when you're running many concurrent synthesis tasks and don't want to hold connections open in a polling loop.

IAM permissions for async tasks

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "polly:StartSpeechSynthesisTask",
        "polly:GetSpeechSynthesisTask",
        "polly:ListSpeechSynthesisTasks"
      ],
      "Resource": "*"
    },
    {
      "Effect": "Allow",
      "Action": ["s3:PutObject"],
      "Resource": "arn:aws:s3:::your-output-bucket/polly-output/*"
    },
    {
      "Effect": "Allow",
      "Action": ["s3:GetObject"],
      "Resource": "arn:aws:s3:::your-output-bucket/polly-output/*"
    }
  ]
}

The async task API requires the same Polly IAM actions as the synchronous API, plus S3 write permissions for the output bucket. Polly uses your execution role's credentials to write to S3 — it does not assume a separate role. If the output bucket uses SSE-KMS, grant kms:GenerateDataKey and kms:Decrypt for the KMS key to the same role. The speech marks .marks file is written to the same bucket and key prefix as the audio, so the same S3 permissions cover both files.