A DTMF IVR can respond to a keypress in under 50ms, while a conversational voice agent must detect speech, interpret intent, generate a response, and synthesize audio before the caller hears anything. Replacing an IVR therefore changes the latency problem from a mostly deterministic lookup into a distributed streaming pipeline with several variable stages.
That distinction matters most in the first few seconds of a call. A caller who hears a tone immediately after pressing "2" accepts the system as responsive. A caller who finishes saying, "I need to move my appointment to next Tuesday," and then hears silence for half a second starts to wonder whether the call dropped.
This article describes the pipeline decisions behind a conversational IVR replacement: where latency accumulates, which buffers should be short, how the telephony boundary affects response time, and how to keep perceived round-trip latency near the 300ms range.
Why DTMF latency is predictable
A traditional IVR generally performs a small number of bounded operations:
- Receive a DTMF event from the telephony platform.
- Map the digit to a menu transition.
- Start a pre-recorded audio prompt.
The system does not need to infer meaning from an acoustic signal or wait for a complete linguistic interpretation. The event has a known duration, the state transition is usually an in-memory operation, and the prompt is already encoded. Even when the call travels through a SIP provider and media gateway, the application-side work is relatively stable.
Speech introduces uncertainty at every boundary. The caller may pause mid-sentence, restart, speak over the prompt, or produce audio with background noise. The system needs a voice activity detector, streaming speech recognition, turn detection, an intent or response model, and speech synthesis. Each component can stream partial results, but each also has a decision point where it may wait for more evidence.
Replacing the IVR does not mean the latency problem disappears. It means latency has to be measured as a pipeline, rather than as the time required to execute one menu transition.
The three-stage conversational pipeline
The core application path is speech-to-text, language-model processing, and text-to-speech. The useful measurement is not just the total duration. It is the time to the first usable partial result, the time to a stable decision, and the time to the first audible response.
| Stage | What adds latency | Design target |
|---|---|---|
| STT | Audio buffering, voice activity detection, decoding, endpointing | Stream small frames and expose partial transcripts |
| LLM | Queueing, prompt size, retrieval, tool calls, token generation | Start generation as soon as the turn is sufficiently understood |
| TTS | Text buffering, synthesis, codec encoding, audio playback startup | Emit the first playable audio chunk without waiting for the full answer |
STT: endpointing is often the hidden wait
Streaming STT can produce hypotheses while the caller is still speaking, but an application should not treat every partial transcript as final. The recognizer may revise words as more context arrives. Waiting for a long silence improves final accuracy but creates a noticeable pause.
For an appointment bot, a practical approach is to use partial results for state and intent hints, then commit when one of several conditions is met: a stable final transcript arrives, the caller pauses beyond a configured threshold, or the recognizer detects a phrase that is complete enough for the current operation. Endpointing should be tuned by task. "Tuesday at three" can often be acted on sooner than an open-ended support explanation.
At Smallest AI, streaming STT has a median latency of approximately 45ms. That figure is useful for the recognizer itself, but it is not the caller's end-to-end wait. Audio framing, transport, endpointing, and downstream processing still belong in the measurement.
LLM: the least predictable stage
Language-model latency is harder to bound than STT or TTS. It depends on queue depth, input token count, output length, model load, tool calls, and whether the request requires external data. A prompt that merely confirms a date is materially different from one that checks availability, applies business rules, and books an appointment through a scheduling API.
The first token is not necessarily the first useful token. A model may generate a short acknowledgement before it has completed a tool call, or it may produce text that cannot be spoken until a sentence boundary is reached. The application should measure time to first token, time to first speakable phrase, and time to final response separately.
Keep the conversational context compact. Do not send an entire call transcript on every turn when a structured state object can represent the relevant facts. Separate system instructions, caller-provided content, and tool results clearly. For common operations, route predictable intents through constrained handlers instead of asking the model to reason through every step.
Streaming model output also needs a policy. Sending every token directly to TTS creates unnatural fragments and can increase synthesis overhead. Waiting for the complete response increases dead air. A useful compromise is to buffer until a short clause or sentence boundary, then begin synthesis while the model continues generating.
TTS: first audio matters more than final audio
For voice interaction, time to first audio chunk is usually more important than time to finish synthesizing the full response. A short acknowledgement can play while the system completes a lookup, provided the acknowledgement does not promise an outcome that has not been confirmed.
Smallest AI TTS has a first-audio-chunk latency of approximately 38ms. The remaining delay depends on text buffering, transport, audio queueing, and the telephony codec. The playback layer should begin sending a valid audio frame as soon as it has enough material, while preserving a small jitter buffer to prevent gaps between chunks.
Buffering and audio frame design
Buffering is a latency and quality decision, not simply a networking detail. Large buffers reduce per-message overhead and can make decoding easier, but they delay every downstream stage. Small buffers improve responsiveness but increase message frequency and make timing errors more visible.
For 16kHz mono PCM16, 20ms contains 320 samples, or 640 bytes. Frames in the 10ms to 30ms range are common for interactive streaming. A 20ms frame is a useful starting point because it gives the recognizer frequent updates without creating an excessive number of WebSocket messages. Eight-kilohertz narrowband audio can be supported when the telephony path requires it, although the reduced bandwidth limits some recognition quality.
- Capture buffer: Keep it short enough that speech reaches STT quickly. Avoid collecting several hundred milliseconds before transmission.
- STT aggregation: Let the recognizer combine frames internally, but expose partial hypotheses to the application.
- LLM text buffer: Accumulate enough text for a natural phrase, usually a clause or sentence, rather than a fixed number of tokens.
- TTS playback buffer: Hold enough audio to absorb normal network jitter, but do not wait for the complete response.
Batch APIs can be simpler for non-interactive jobs, but they are a poor fit for a conversational turn. Uploading a complete utterance, waiting for a complete transcript, generating a complete answer, and only then synthesizing audio adds the duration of each stage serially. Streaming allows partial work to overlap. The tradeoff is more complicated cancellation, backpressure, and error handling.
Backpressure must be explicit. If TTS produces audio faster than the telephony leg can send it, the system should pause or discard queued speculative audio rather than allowing memory and conversational state to drift. If the caller interrupts, cancel the current model and synthesis work, clear the playback queue, and resume recognition without waiting for stale tasks to finish.
Persistent connections and the telephony boundary
A persistent WebSocket is usually the right application transport for a low-latency voice agent. It avoids a new TCP and TLS handshake on each turn, maintains session state, and supports bidirectional streaming. The connection can carry audio frames, partial transcripts, control events, barge-in signals, and synthesized audio in both directions.
Per-turn HTTP connections introduce several avoidable costs. Even with connection reuse, request scheduling and serialization can create small pauses between stages. New connections add handshake and routing overhead. More importantly, an HTTP request model encourages batch boundaries, which work against continuous audio processing.
The connection should have explicit lifecycle and timing events. At minimum, record:
- Audio capture timestamp at the media gateway.
- Audio arrival and forwarding timestamps at the application.
- First partial and final STT timestamps.
- LLM request, first-token, tool-call, and first-speakable-text timestamps.
- TTS request, first-audio, and playback timestamps.
- Interruption, cancellation, and turn completion timestamps.
These measurements distinguish model latency from network latency. Without them, teams often optimize the wrong component. A slow first response may be caused by a 250ms endpointing threshold, not by the language model.
The PSTN or VoIP bridge is another common blind spot. A SIP provider, media server, or gateway may introduce packetization delay, jitter buffering, transcoding, and geographic network distance. G.711 audio is frequently used for telephony, while the application may prefer 16kHz PCM16. Moving between those formats requires resampling and possibly codec encode or decode operations.
Individual resampling operations are often inexpensive, especially for short frames, but the surrounding buffering can be significant. A converter that waits for a larger block, or a media gateway that maintains a conservative jitter buffer, can add more delay than the arithmetic itself. Avoid repeated format changes. Decode and resample once near the media boundary, keep one internal representation, and encode only when sending audio back to the call.
Packet loss concealment and jitter handling also affect perception. A small, adaptive jitter buffer is preferable to a large fixed buffer for an interactive agent. The correct value depends on the carrier and network, so it should be measured with real call paths rather than assumed from local tests.
Keeping the round-trip near 300ms
There is no universal threshold at which every caller rejects a voice system, but conversational turns begin to feel uncomfortable when silence becomes long enough to suggest a failure. A design target near 300ms from the end of the caller's turn to the first audible response is useful, even when some turns will exceed it.
That target should be treated as a latency budget, not a promise that every stage receives an equal allocation. For example:
| Segment | Illustrative budget |
|---|---|
| Media transport and endpointing | 40ms to 80ms |
| STT stabilization | 45ms to 70ms |
| LLM first useful output | 80ms to 130ms |
| TTS first audio and playback startup | 38ms to 70ms |
The values are illustrative and may overlap. In a streaming design, STT, LLM preparation, and TTS startup do not always run strictly one after another. The important constraint is the critical path to the first playable response.
Use short acknowledgements when a required operation will exceed the budget. "I'm checking that now" can prevent dead air, but it should be generated or selected carefully. Repeated filler becomes irritating, and a filler response should not mask an unhealthy dependency. For tool calls, the agent can acknowledge the request, then provide the result when the external operation completes.
Barge-in handling is part of the same budget. The system should stop playback quickly when the caller starts speaking, rather than waiting for the current audio buffer to drain. This requires a separate input path that remains active while TTS is playing, plus cancellation propagation through the model and synthesis layers.
When DTMF should remain available
The goal is not to replicate DTMF speed in a conversational system. Some workflows benefit from keeping both interfaces.
Complex menu trees with highly constrained choices may be easier to audit with DTMF. Reliability-sensitive flows, such as authentication steps or high-volume outage announcements, may need a deterministic path when model services, external APIs, or speech recognition are degraded. Some caller segments also prefer pressing keys, particularly when they are in a noisy environment, have limited speech confidence, or are using a handset with poor audio quality.
A practical design can offer DTMF fallback at the call level and within a turn. The caller might say "press zero" or use a configured escape key to reach an operator or legacy menu. The media and session layer should preserve the same call state so that a fallback does not require starting over.
For example, imagine replacing a synthetic healthcare provider's appointment menu. A caller says, "Move my appointment to next Tuesday afternoon." The agent streams the audio to STT, identifies the date and time window, calls an availability service, and begins speaking a confirmation as soon as the result is known. If the caller is in a noisy emergency department or the availability service times out, the system can offer DTMF choices or route to a conventional scheduling queue.
That architecture treats latency as a measurable property of the complete media path. Keep WebSocket sessions persistent, use small audio frames, overlap streaming stages, bound language-model work, start TTS on the first useful phrase, and instrument every boundary through the PSTN or VoIP bridge. Conversational behavior can tolerate occasional longer turns when the caller receives an immediate, truthful indication of progress. It cannot tolerate unexplained silence caused by buffers and handoffs that no one is measuring.
If you're replacing an IVR and want to walk through the latency architecture, email [email protected].