Measuring STT Latency in Production Voice Pipelines

A naive STT measurement can report 45 milliseconds while the caller waits 280 milliseconds to hear the bot respond, because the first number often excludes audio buffering, end-of-speech detection, network transfer, and downstream work. For a voice scheduling bot handling roughly 200 concurrent calls, using 8kHz telephone audio over WebSocket streaming, those omitted intervals determine whether the conversation feels immediate or delayed.

That distinction is the purpose of production latency instrumentation. Model latency is useful for comparing recognizers under controlled conditions. It is not the same measurement as the time between a caller finishing a phrase and the application receiving a transcript it can act on. To diagnose the latter, measure each boundary in the audio and event path.

Define the latency event before collecting data

“STT latency” can refer to several different intervals. A dashboard that combines them into one percentile may be precise while still answering the wrong question.

• Capture delay: the time between an audio sample entering the telephony or client stack and that sample being placed into an outbound frame.
• Transport delay: the time required for a WebSocket frame to reach the STT service, including queueing and packet retransmission effects where applicable.
• Recognition delay: the interval from receipt of an audio segment to a partial or final transcript event.
• Endpointing delay: the time spent waiting for voice activity detection, silence thresholds, or end-of-speech detection before a final result is emitted.
• Application delay: the time from transcript receipt to the next action, such as a tool call, response generation request, or TTS request.

For an interactive bot, a useful primary metric is caller_speech_end_to_actionable_transcript. That metric requires an estimate of speech end, which may come from VAD or a timestamp attached to the final audio frame. It should be paired with first_partial_transcript, because partial results can let an application begin work before the final transcript arrives.

Do not use the timestamp generated by the STT server as the only clock. Record a monotonic client-side timestamp when each audio chunk is captured, a send timestamp for each WebSocket frame, and receipt timestamps for partial and final events. Wall-clock timestamps can be retained for correlation, but elapsed time should use a monotonic clock to avoid errors caused by clock adjustment.

Instrument the audio path, not just the API call

A common implementation starts a timer immediately before an API request and stops it when a response arrives. That works for a single uploaded file. It is incomplete for streaming recognition, where audio is continuously captured, chunked, buffered, transmitted, decoded, and assembled.

Attach a sequence number and capture timestamp to every audio chunk. For PCM16 audio, record the sample rate, channel count, and number of samples. A 20 millisecond chunk at 8kHz mono contains 160 samples. The same duration at 16kHz contains 320 samples. If the producer accidentally sends 100 millisecond chunks, the recognizer may have excellent processing speed while the caller still waits for the next large frame to be assembled.

At minimum, log these fields for each session:

• Session ID and turn ID
• Audio format, sample rate, channel count, and codec
• Chunk sequence number and capture time
• Chunk duration and WebSocket send time
• Server acknowledgment or receipt time, if available
• Partial transcript time, final transcript time, and transcript stability state
• VAD speech-start and speech-end events
• Queue time before downstream processing

Keep the raw audio out of ordinary application logs unless it is required for a controlled debugging workflow. A compact event record is usually enough to reconstruct timing. If audio samples must be retained for analysis, apply access controls and a defined retention period, because phone conversations can contain sensitive information.

Separate streaming recognition from endpointing

Streaming STT often produces a partial transcript while the speaker is still talking. The final transcript may arrive later because the system is waiting for more context or for a stable endpoint. Treating the final event as the first useful event hides an important tradeoff.

Consider a caller saying, “Schedule an appointment for Thursday at three.” The bot may receive a partial transcript after “Schedule an appointment,” then revise it as more audio arrives. If the application launches an irreversible action on the first partial result, recognition speed creates an accuracy and control problem. If it waits for the final result before doing any work, it may add unnecessary delay.

Instrument both timestamps and make the application policy explicit:

1. Use partial results for speculative work, such as preparing candidate intents or warming a response path.
2. Use stable partial results for reversible work, such as updating a draft slot selection.
3. Use the final transcript, or a confirmed stable result, for actions that change records or send commitments.

Endpointing deserves its own histogram. A recognizer can return partial text quickly while a 700 millisecond silence threshold delays the final result. That delay may be intentional, especially for callers who pause between words, but it should not be mislabeled as model processing time.

VAD settings also interact with telephone audio. Narrowband 8kHz audio can contain codec artifacts and background energy that make speech-start and speech-end decisions less stable than they are with clean 16kHz microphone input. Record the VAD configuration and the audio source with every latency sample. Otherwise, a change in endpointing behavior can look like a change in STT performance.

Use a concrete production-style trace

In a synthetic test of a voice scheduling bot, the media gateway receives 8kHz PCM16 audio and sends 20 millisecond WebSocket frames to an STT endpoint. The caller finishes speaking at capture time 12,400 milliseconds. The last speech-containing frame is sent at 12,430 milliseconds because of a local buffer. The STT service receives it at 12,492 milliseconds. A partial transcript arrived at 12,465 milliseconds, but the application marked it unstable. VAD reports end of speech at 12,610 milliseconds, and the final transcript arrives at 12,655 milliseconds.

Several latency values can now be calculated from the same turn:

• Capture-to-send delay for the last frame: 30 milliseconds
• Transport and service receipt delay: 62 milliseconds
• Receipt of the first partial result after speech end: 65 milliseconds
• Final transcript after speech end: 255 milliseconds
• Endpointing interval from last speech-containing frame to VAD end: 180 milliseconds

A single “STT latency” field would conceal the fact that endpointing contributes more delay than recognition. If the bot starts preparing a response from the stable partial result, the perceived delay may be closer to 65 milliseconds. If it waits for the final transcript, the relevant figure is 255 milliseconds before application work can begin.

This is also where concurrency matters. At roughly 200 simultaneous calls, inspect event-loop delay, socket write queues, CPU saturation, garbage collection pauses, and per-session buffering. A service can show a low median while a small number of calls accumulate large queue delays. Report p50, p90, p95, and p99 by traffic condition, not only a single average.

Build latency distributions that explain variance

Latency data becomes useful when each percentile can be connected to a cause. Group measurements by audio source, language, chunk duration, codec, concurrency level, and turn length. Compare short acknowledgments with longer utterances. Separate first partial, stable partial, and final transcript events.

Track real-time factor, or RTFX, alongside wall-clock latency when it is available. RTFX describes processing time relative to audio duration. It can help distinguish a recognizer that processes a long utterance efficiently from a pipeline that delays delivery because it buffers too much audio. Neither metric replaces the caller-facing interval.

Use traces to identify queueing rather than averaging it away. A useful trace might show:

capture_start
audio_chunk_17_captured
audio_chunk_17_sent
audio_chunk_17_received
partial_transcript_received
vad_end
final_transcript_received
application_action_started

Attach a trace ID to the STT session and downstream agent events. Then compare STT timing with TTFT and TTS timing. Smallest AI’s internal benchmark tests report approximately 45ms STT latency and approximately 38ms TTS latency across more than 1,000 audio samples in English and Spanish. Actual results vary by audio length and language. Those figures describe component capabilities, not a guarantee for a complete production turn. A production trace still needs to account for capture, network, endpointing, agent processing, and audio playback.

What this measurement does not solve

Detailed timing will not resolve recognition errors, ambiguous caller intent, poor microphone placement, or a VAD threshold that is inappropriate for a particular conversation. It also cannot make a slow downstream tool call disappear. A bot may receive a transcript quickly and still respond slowly because a calendar lookup, policy check, or response model takes longer.

Nor are we claiming that the lowest STT number automatically produces the best caller experience. We are saying that component latency must be connected to the exact event the application is waiting for. In some flows, a trustworthy partial transcript is more valuable than a slightly faster final transcript. In others, premature partial text creates costly corrections.

Latency instrumentation also has operational costs. High-cardinality traces can increase storage and logging volume, while retaining audio for replay introduces privacy obligations. Sample detailed traces, retain aggregate histograms longer, and make the sampling policy visible when comparing releases.

For teams building voice agents, the practical starting point is a turn-level trace with monotonic timestamps, chunk metadata, VAD events, partial and final transcript events, and downstream action timing. Once those intervals are separate, a reported 45 milliseconds can be understood as one part of a caller’s wait rather than mistaken for the entire experience.