A team connects a telephony stream to an STT WebSocket and hears a 200 ms pause before every partial transcript. The STT engine is not necessarily slow. The audio may be arriving as 8 kHz G.711, being decoded, resampled, buffered, and only then sent for recognition. Each stage adds delay while the pipeline continues without returning an obvious error.
In early-access integrations, we have found that latency problems usually come from the boundaries between systems. The endpoint may report approximately 45 ms streaming STT latency, with full round-trip voice interaction under 100 ms in the right setup, but that budget can disappear before recognition begins. These are the five integration errors we check first.
1. Sending the wrong sample rate
The most common failure is sending audio at 8 kHz when the application is configured around 16 kHz. This happens frequently when a telephony source produces narrowband audio and the integration passes its format through without making the conversion explicit.
The pipeline may not fail. Instead, an intermediate service detects the mismatch and resamples the stream, or the recognizer receives audio whose timing and frequency characteristics do not match the declared format. A resampling step can add tens of milliseconds. If audio is first accumulated into a 20 ms frame and then resampled in batches, the effective delay can be closer to one or more frame intervals, depending on the implementation.
Use 16 kHz mono PCM as the default format. Smallest AI supports 8 kHz narrowband audio, but the sender must declare and preserve that choice throughout the path. Do not label 8 kHz samples as 16 kHz to avoid a configuration change. That changes playback speed and recognition timing rather than improving compatibility.
Diagnose this at the socket boundary. Log the sample rate, channel count, sample width, and number of samples in every test stream. A PCM16 frame containing 320 samples represents 20 ms at 16 kHz. The same frame represents 40 ms at 8 kHz. If your logs and packet timestamps disagree, the format declaration is probably wrong.
2. Buffering audio before opening the stream
Another common pattern is to collect a complete utterance, or a large audio buffer, before opening the STT connection. It is convenient for batch transcription, especially when the first version of an agent is built around an HTTP POST request. It is a poor fit for interactive speech.
Suppose the client waits for 500 ms of audio before sending the first WebSocket frame. Recognition cannot begin until that half second has elapsed. Add a 100 ms network exchange, decoding time, and end-of-speech handling, and the user hears a response noticeably after they finish speaking. The recognizer may still process the audio quickly, but the application has already spent most of its latency budget.
For live agents, open /v1/stt/stream before speech begins and send frames continuously. A practical starting point is 10 to 20 ms of PCM16 audio per WebSocket frame. Smaller frames can reduce waiting time but increase message overhead. Larger frames reduce overhead but make every partial transcript wait longer.
Keep the batch endpoint, /v1/stt/recognize, for recordings and asynchronous workflows. The boundary is simple: if a human is waiting for the next turn, stream audio as it arrives. If nobody is waiting, buffering may be a reasonable tradeoff.
To diagnose this error, compare the capture timestamp of the first audio sample with the send timestamp of the first WebSocket frame. A large gap there is client buffering, not STT latency.
3. Treating partial transcripts as final transcripts
Streaming recognition normally revises its interpretation as more audio arrives. A partial transcript can change when the next phoneme, word, or phrase provides context. Teams sometimes render every partial result as permanent text, then send the agent a new message for every revision.
This creates two problems. The user interface can show duplicated or visibly changing text, and the agent can begin responding before the speaker has finished. If a partial result arrives 150 ms after the beginning of a phrase but is revised 200 ms later, acting on it immediately can produce a response based on incomplete speech. The cost is not only a few hundred milliseconds of correction. It can trigger an unnecessary turn, barge into the speaker, or require a second recognition pass.
Maintain separate provisional and committed transcript state. Replace the provisional text when a newer partial arrives. Commit text only when the service marks it final, or when your own policy determines that the earlier words are stable enough for the next stage.
A useful design is to pass partials to the user interface immediately while sending only committed text to business logic. If an agent must react before finalization, use a transcript confidence threshold and a short stability window, such as waiting for the same leading phrase across two updates. The right interval depends on the model and conversation style, so measure it rather than hard-coding a universal value.
Inspect transcript event timestamps and revision identifiers during testing. If one spoken phrase produces several downstream actions, the integration is treating recognition updates as commands.
4. Using an oversized or irregular WebSocket frame
WebSocket streaming does not guarantee low latency by itself. The sender still decides how much audio goes into each frame. A client that sends 200 or 500 ms chunks may technically stream, but it forces the recognizer to wait for a large block before receiving new speech.
At 16 kHz mono PCM16, 100 ms contains 3,200 bytes of audio. A 500 ms chunk contains 16,000 bytes. In the latter case, the first word in the chunk can wait almost half a second before transmission, even if encoding and network transfer take only a few milliseconds. Irregular frame sizes create a related problem. A scheduler pause can cause several frames to arrive together, making the server see bursts instead of a continuous stream.
Send consistently sized frames, usually 10 to 20 ms to start, and timestamp them at capture time. Keep the capture loop independent from the transcript handling loop. A slow user interface or logging operation must not stop audio capture.
Test the entire path, not just the WebSocket client. A browser audio callback, media gateway, jitter buffer, or proxy may regroup frames. Record the arrival interval at the STT endpoint and calculate the difference between capture time and server receipt time. If the interval is regularly 100 ms or more, reduce the upstream chunk size or inspect the component that is coalescing messages.
This error is easy to confuse with network delay. A network trace may show fast delivery of each large frame, while the user still waits because the frame was created too late.
5. Letting codec conversion and VAD own the turn boundary
Telephony integrations often combine G.711 decoding, a jitter buffer, voice activity detection, and end-of-speech detection before sending text to an agent. Each component has a purpose, but their defaults can quietly add delay. A 60 ms jitter buffer followed by 300 ms of silence required by VAD already places roughly 360 ms between the end of speech and the next action, before STT and network time are included.
The mistake is not using these components. The mistake is allowing each one to add an unmeasured wait. A jitter buffer sized for unstable network conditions may be excessive on a local media path. A conservative VAD may wait for a long silence interval because it was tuned to avoid cutting off speakers. A codec decoder may hold a frame while waiting for a complete packet.
Measure each boundary separately: packet arrival, decoded PCM availability, audio sent to STT, last recognized speech, VAD stop, and agent response start. Then choose settings against the interaction you are building. Keep the jitter buffer large enough for the actual network, but no larger. Tune the VAD silence interval with real pauses, not only clean test recordings. Send decoded PCM to the streaming endpoint as soon as complete frames are available.
This is the situational error in the group. For asynchronous transcription, an extra 300 ms at the end of an utterance may not matter. For a voice agent, it can make the system feel inattentive. The correct setting depends on whether the priority is turn speed, resistance to clipped final words, or tolerance for packet jitter.
Make the latency budget visible
Most STT integration problems become straightforward once every stage has a timestamp. Track capture, codec conversion, buffering, WebSocket send, server receipt, partial transcript, final transcript, VAD stop, and response start as separate events.
Use the streaming endpoint for interactive audio, keep the format explicit, and inspect frame timing before changing models or infrastructure. In our testing, a clean 16 kHz PCM path with small, regular frames leaves the recognition service enough of the latency budget to matter. A pipeline that quietly waits 200 ms before sending its first frame cannot recover that time later.
The practical test is simple: play a recording with a spoken timestamp, such as “one” followed by a known pause, and compare capture time with the first partial transcript time at the client. That measurement usually identifies the waiting component faster than another round of endpoint tuning.