MCP Server Lex v2 Audio Streaming — RecognizeUtteranceCommand, audio formats, SigV4a

When to use audio vs text-only

Before implementing RecognizeUtteranceCommand, evaluate whether text-only integration is a better fit. Text mode has lower latency for most pipelines, is easier to test, and separates STT concerns from NLU concerns.

For agentic MCP server tools that orchestrate conversation flows in TypeScript, RecognizeTextCommand plus a separate Amazon Transcribe step gives you the most control. The Transcribe step lets you use custom vocabularies tuned to your domain, control speaker diarization, and inspect the raw transcript before it reaches Lex. Reserve RecognizeUtteranceCommand for cases where the audio source arrives ready and you want a single-hop call to Lex.

Audio format requirements

Lex v2 accepts four audio input formats. Each has different quality, compression, and streaming characteristics. PCM-16 at 16 kHz is the safest default — no codec dependencies, lossless, and always accepted.

MP3 is not supported for input. This surprises developers coming from Amazon Transcribe, which does accept MP3. If your audio source produces MP3, convert to PCM-16 before sending to Lex (use ffmpeg -i input.mp3 -ar 16000 -ac 1 -f s16le output.raw). Stereo audio (channels=2) is also not supported — convert to mono by mixing channels before submitting.

RecognizeUtteranceCommand: sending audio

RecognizeUtteranceCommand accepts audio as a Buffer, Uint8Array, or Readable stream via the inputStream field. The SDK handles SigV4a signing transparently when streaming — do not sign the request body manually.

import {
  LexRuntimeV2Client,
  RecognizeUtteranceCommand,
  type RecognizeUtteranceCommandInput,
} from '@aws-sdk/client-lex-runtime-v2';
import { readFileSync } from 'fs';

const lex = new LexRuntimeV2Client({ region: process.env.AWS_REGION ?? 'us-east-1' });

export async function recognizeAudio(
  audioBuffer: Buffer,
  sessionId: string,
): Promise<{ transcription: string; intentName: string; botReply: string }> {
  const input: RecognizeUtteranceCommandInput = {
    botId: process.env.LEX_BOT_ID!,
    botAliasId: process.env.LEX_ALIAS_ID ?? 'TSTALIASID',
    localeId: 'en_US',
    sessionId,
    requestContentType: 'audio/l16; rate=16000; channels=1', // PCM-16 mono at 16 kHz
    responseContentType: 'text/plain; charset=utf-8',         // request text back, not audio
    inputStream: audioBuffer,
  };

  const response = await lex.send(new RecognizeUtteranceCommand(input));

  // Lex returns several headers and an inputTranscript (what it heard)
  const transcription = response.inputTranscript ?? '';

  // interpretations and messages are base64-encoded JSON strings in the response
  // The SDK does NOT auto-decode them — you must decode and parse manually
  const interpretationsJson = response.interpretations
    ? Buffer.from(response.interpretations as unknown as string, 'base64').toString('utf8')
    : '[]';
  const messagesJson = response.messages
    ? Buffer.from(response.messages as unknown as string, 'base64').toString('utf8')
    : '[]';

  const interpretations: Array<{ intent?: { name: string } }> = JSON.parse(interpretationsJson);
  const messages: Array<{ content?: string }> = JSON.parse(messagesJson);

  const intentName = interpretations[0]?.intent?.name ?? 'Unknown';
  const botReply = messages[0]?.content ?? '';

  return { transcription, intentName, botReply };
}

The interpretations and messages fields in the RecognizeUtteranceCommand response are base64-encoded JSON strings — not parsed objects. This is different from RecognizeTextCommand, which returns parsed objects. Always decode with Buffer.from(value, 'base64').toString('utf8') and then JSON.parse. Forgetting to decode is the most common mistake when migrating from text to audio mode.

Requesting audio output from Lex

If your MCP server needs to return audio (e.g. for a voice interface), set responseContentType to an audio MIME type. Lex will synthesize the bot's response text using Amazon Polly and return the audio bytes in the response body.

import { createWriteStream } from 'fs';

const response = await lex.send(new RecognizeUtteranceCommand({
  botId: process.env.LEX_BOT_ID!,
  botAliasId: process.env.LEX_ALIAS_ID ?? 'TSTALIASID',
  localeId: 'en_US',
  sessionId,
  requestContentType: 'audio/l16; rate=16000; channels=1',
  responseContentType: 'audio/mpeg',  // Lex returns MP3 audio via Polly
  inputStream: audioBuffer,
}));

// response.audioStream is a Readable stream of MP3 bytes
if (response.audioStream) {
  const chunks: Buffer[] = [];
  for await (const chunk of response.audioStream) {
    chunks.push(Buffer.isBuffer(chunk) ? chunk : Buffer.from(chunk));
  }
  const audioResponse = Buffer.concat(chunks);
  // Stream audioResponse back to the voice client
}

// Supported responseContentType values for audio output:
// 'audio/mpeg'          — MP3, compressed, most compatible with playback clients
// 'audio/ogg; codecs=opus'  — Opus in OGG container, smaller than MP3
// 'audio/pcm'           — PCM 16-bit 16 kHz, raw uncompressed bytes

The Polly voice and engine used for audio output are configured in the bot's locale settings in the Lex console — you cannot override them per API call in RecognizeUtteranceCommand. If you need fine-grained voice control, use the text-only response from Lex and then call Polly separately with the full SynthesizeSpeechCommand configuration.

Bidirectional streaming with StartConversationCommand

StartConversationCommand enables true bidirectional streaming over HTTP/2: you send audio frames as a stream while simultaneously receiving transcription updates and audio responses. This is designed for real-time voice kiosk and telephony applications. It is significantly more complex than RecognizeUtteranceCommand and requires HTTP/2 support in your Node.js environment (Node 18+ recommended).

import {
  LexRuntimeV2Client,
  StartConversationCommand,
  type ConfigurationEvent,
  type AudioInputEvent,
  type DisconnectEvent,
} from '@aws-sdk/client-lex-runtime-v2';
import { Readable, PassThrough } from 'stream';

const lex = new LexRuntimeV2Client({ region: process.env.AWS_REGION ?? 'us-east-1' });

async function startVoiceConversation(sessionId: string, audioSource: Readable) {
  // Build the input event stream — Lex expects a sequence of event types
  async function* buildInputStream() {
    // 1. ConfigurationEvent must be first — defines bot, locale, and audio format
    const config: ConfigurationEvent = {
      botAliasId: process.env.LEX_ALIAS_ID ?? 'TSTALIASID',
      botId: process.env.LEX_BOT_ID!,
      localeId: 'en_US',
      sessionId,
      responseContentType: 'audio/mpeg',
      welcomeMessages: [],
    };
    yield { ConfigurationEvent: config };

    // 2. AudioInputEvent — stream audio frames
    for await (const chunk of audioSource) {
      const audioChunk: AudioInputEvent = {
        audioChunk: chunk as Uint8Array,
        contentType: 'audio/l16; rate=16000; channels=1',
        eventId: crypto.randomUUID(),
      };
      yield { AudioInputEvent: audioChunk };
    }

    // 3. DisconnectEvent — signal end of audio
    const disconnect: DisconnectEvent = { eventId: crypto.randomUUID() };
    yield { DisconnectEvent: disconnect };
  }

  const response = await lex.send(new StartConversationCommand({
    botId: process.env.LEX_BOT_ID!,
    botAliasId: process.env.LEX_ALIAS_ID ?? 'TSTALIASID',
    localeId: 'en_US',
    sessionId,
    requestEventStream: buildInputStream() as AsyncIterable<unknown>,
  }));

  // response.responseEventStream is an AsyncIterable of output events
  if (response.responseEventStream) {
    for await (const event of response.responseEventStream) {
      if (event.AudioResponseEvent) {
        // Stream audio bytes back to the client
        const audioChunk = event.AudioResponseEvent.audioChunk;
        // write audioChunk to output stream
      }
      if (event.IntentResultEvent) {
        const intent = event.IntentResultEvent.sessionState?.intent;
        console.log('Intent:', intent?.name, 'State:', intent?.state);
      }
      if (event.TranscriptEvent) {
        console.log('Transcript:', event.TranscriptEvent.transcript);
      }
    }
  }
}

For MCP server tools that orchestrate agent workflows, StartConversationCommand is rarely the right choice. The event stream architecture requires careful handling of incomplete audio frames, out-of-order events, and reconnection logic. Use it only when you are building a real-time voice interface where end-to-end audio latency is a hard requirement.

SigV4a signing: what you need to know

Audio streaming commands (RecognizeUtteranceCommand and StartConversationCommand) require SigV4a (the asymmetric, multi-region variant of AWS request signing) instead of the standard SigV4 used for most AWS API calls. The AWS SDK for JavaScript v3 handles this automatically — you do not need to configure anything extra when using the SDK client.

// No special configuration needed in the SDK client for SigV4a
const lex = new LexRuntimeV2Client({
  region: process.env.AWS_REGION ?? 'us-east-1',
  // credentials and signing are handled automatically by the SDK
});

// The SDK detects that RecognizeUtteranceCommand requires streaming and applies SigV4a signing.
// If you receive a SignatureDoesNotMatch error on audio commands:
// 1. Ensure you are using @aws-sdk/client-lex-runtime-v2 v3.400.0+
//    (older versions had SigV4a issues on certain Node.js versions)
// 2. Check that AWS_REGION is set — SigV4a requires a valid region
// 3. Ensure credentials have lex:RecognizeUtterance permission — not just lex:RecognizeText
// 4. Do NOT use a custom httpHandler or manual signing middleware — it breaks streaming

Developers building custom HTTP clients or proxying Lex audio calls through a backend encounter SigV4a the most. If your proxy server is re-signing the request with standard SigV4, Lex will reject it. The request must be signed by the original calling principal, not an intermediary. For MCP server tools that proxy voice from a client app, the preferred pattern is to generate pre-signed credentials on the server and have the client call Lex directly — or to use a WebSocket proxy that signs at the edge.