A 400 millisecond gap between a caller finishing a question and a voice agent beginning its reply is enough to make the interaction feel like an IVR, even when every component is technically working. That gap was one of the observations that motivated us to build Smallest AI. A person asks a question, hears nothing, and starts wondering whether the system heard them, whether the call dropped, or whether they need to repeat themselves.
The ear notices before the caller complains
Human conversation depends on very short turn-taking intervals. In ordinary dialogue, speakers continuously use tiny timing cues to decide whether someone has finished talking, is thinking, or has stopped responding. A voice agent does not need to reply instantaneously, but it does need to provide evidence that the turn is moving forward.
For practical voice engineering, the thresholds are roughly:
- Below 150 milliseconds: the response usually feels immediate, assuming the audio itself is natural.
- 150 to 250 milliseconds: noticeable in a careful comparison, but generally acceptable in a phone conversation.
- 250 to 400 milliseconds: the caller can perceive a pause and may begin to interpret it as processing.
- Above 400 milliseconds: the interaction starts to acquire the rhythm of a menu system or a delayed handoff.
- Above 700 milliseconds: many callers will speak again, ask if the agent is there, or abandon the turn.
These are engineering targets, not a universal law of perception. The acceptable interval changes with the task, the caller's expectations, network conditions, and whether the agent has already established a conversational rhythm. A 500 millisecond pause before a complex insurance explanation may be tolerated. The same pause after a caller says "yes" to an appointment time feels broken.
The issue is not that 400 milliseconds is technically noticeable in every conversation. The issue is that the pause arrives at the exact point where the caller expects confirmation that the system is listening. Silence removes that confirmation.
Why silence gets interpreted as failure
People do not experience a phone bot as a collection of APIs. They experience a sequence of social signals. A short acknowledgment, a partial phrase, or even a carefully designed filler tells the caller that the system has received the turn and is working on it. Silence supplies no such signal.
This creates a compounding problem. After roughly half a second of dead air, some callers start a second utterance. That second utterance can collide with the original response, trigger barge-in handling, or be captured as a new turn. The agent then has to distinguish among a repeated question, an interruption, and a continuation. Latency has become an accuracy problem.
In a text interface, a spinner can occupy this waiting period. On a call, the equivalent is audio. If the agent cannot produce the final answer yet, it may need to acknowledge the caller, stream an initial phrase, or provide a short progress cue. That cue should be semantically appropriate. "Let me check that" works for a scheduling lookup. It sounds strange when repeated before every simple answer.
A synthetic scheduling example
Consider a healthcare scheduling bot handling 20,000 calls per month, with an average of 50 concurrent calls during weekday peaks. Its main tasks are checking appointment availability, collecting insurance details, and confirming a time with the patient.
Suppose the bot takes 420 milliseconds after the caller finishes speaking to begin its response. On a simple request such as "Do you have anything next Tuesday morning?", the caller hears a full silence gap before the system says it is checking. Some callers repeat the question. Others talk over the reply. A smaller group hangs up and calls again, increasing demand on the contact center without producing a completed appointment.
Now consider the same workflow with a first audible response around 180 to 220 milliseconds. The agent can say "I'll check Tuesday morning" while the availability lookup continues. The total time to produce the final options may be unchanged, but the caller no longer has to infer whether the system is alive. That distinction matters because conversion and task completion are affected by the first response boundary, not only by the time required for the complete answer.
The scenario is synthetic, and the exact impact depends on the caller population and task. Still, it illustrates why teams should measure abandonment and interruption against response latency, rather than treating latency as a purely technical metric.
Build the latency budget from the turn boundary
"Model latency" is not the same as the silence a caller experiences. The relevant measurement starts when the caller's turn ends and ends when the first response audio reaches the phone. That path can include:
- End-of-turn detection and audio transport.
- Speech-to-text processing and transcript delivery.
- Agent orchestration, tool selection, and any external lookup.
- Text-to-speech generation up to the first playable audio chunk.
- Network transport, telephony buffering, and audio playback.
A useful first-pass budget for a simple response might allocate 20 to 40 milliseconds for transport and turn detection, about 45 milliseconds for streaming speech recognition, 20 to 80 milliseconds for orchestration, about 38 milliseconds to generate the first speech chunk, and 30 to 80 milliseconds for the remaining network and telephony path. Those numbers do not add up to a guaranteed experience. They show how quickly a nominal 200 millisecond target can disappear when every layer adds a little delay.
At Smallest AI, our real-time infrastructure is designed around this first-audio boundary. Streaming STT has a median latency of approximately 45ms, and TTS first-audio-chunk latency is approximately 38ms. Based on internal benchmark tests across 1,000+ audio samples in English and Spanish.
For a phone agent, a reasonable target is first audible response below 250 milliseconds for common turns, with a preferred range closer to 150 to 200 milliseconds when the interaction is short and transactional. More complex requests can take longer to resolve, but they should not remain acoustically empty while the system works.
Latency is not the only source of a bad call
Optimizing every component for the lowest possible number can create other problems. A system that interrupts the caller because endpoint detection is too aggressive may respond quickly but feel rude. A system that emits a filler phrase after every turn may meet a timing target while sounding repetitive. Very small TTS chunks can also produce clipped words, unnatural prosody, or extra buffering if the telephony path cannot play them smoothly.
Accuracy matters just as much. A fast response based on a bad transcript is worse than a slightly slower response that correctly understands a medication name, street address, or appointment date. Tool calls add another constraint: the agent should not invent availability merely to avoid silence. In those cases, an honest progress cue is preferable to a fabricated answer.
Teams should therefore track at least four measurements together: first-audio latency, final-answer latency, interruption rate, and task completion. Segment them by intent and turn type. A single average can hide the fact that greetings are fast while scheduling lookups regularly exceed 800 milliseconds.
Design for evidence of progress
The practical design question is not "How do we make every answer instantaneous?" It is "What should the caller hear while the system is completing the turn?" For simple confirmations, the answer should usually be the final response. For tool-backed actions, a brief acknowledgment can preserve the conversational frame while the lookup runs. For longer operations, a progress update may be appropriate, but it should be generated intentionally and not used as a blanket patch for slow infrastructure.
Measure from the caller's perspective. Log the timestamp of the final inbound speech frame, the timestamp when the first response audio is playable, and the timestamp when the complete answer finishes. Then review recordings around the 250, 400, and 700 millisecond boundaries. These recordings often reveal issues that a dashboard does not, including clipped acknowledgments, awkward filler timing, and double responses after a caller repeats themselves.
A voice agent can tolerate a complex answer taking several seconds if the conversation continues to provide evidence of progress. It struggles with a few hundred milliseconds of unexplained silence at the turn boundary. That is the founding insight behind focusing on streaming infrastructure rather than measuring only the time until a full response is ready. In a phone conversation, the first audible signal is part of the answer.
If you're building a phone bot and want to test latency against these thresholds, email [email protected].