A team building appointment scheduling bots for a healthcare platform described a familiar failure. A caller would finish saying, “I need to move my appointment to next Tuesday,” and the system would correctly detect the end of speech. Then nothing happened for nearly half a second. The answer was already being prepared, but the caller had no evidence of that. They said “hello?” or repeated the request. Sometimes they hung up before the bot spoke.
The team initially focused on the voice itself. They tested different speakers, adjusted prosody, and compared several TTS voices. Those changes helped, but they did not address the moment that made the caller uncomfortable. The problem was the empty space between the caller stopping and the bot starting.
During six months of conversations with a small set of early-access teams, this pattern came up repeatedly. Teams wanted phone bots that sounded human, but “human” did not primarily mean a perfect voice. It meant timely turn-taking, clear interruptibility, and enough evidence that the system was listening. A natural voice arriving too late still sounds like a machine.
The first lesson was about perceived delay
In a phone conversation, callers do not experience latency as a number in a dashboard. They experience it as uncertainty. Did the bot hear me? Is it thinking? Did the call drop? Should I speak again?
Sudarshan Kamath’s founding observation was that a 400ms gap between a caller’s question and the bot’s response can create an IVR feel. That gap is not necessarily long from an engineering perspective. It is long enough, however, for a caller to notice that the exchange is not behaving like a live conversation.
This distinction changed how we evaluated the voice stack. We stopped treating speech-to-text, reasoning, and text-to-speech as isolated components with independent benchmarks. The caller experiences one turn. The relevant question is how quickly the system can move from incoming audio to the first useful audio response.
That includes several intervals:
- When the caller’s speech reaches the recognizer.
- How quickly a partial transcript becomes available.
- How the system determines that the caller has finished speaking.
- When response generation begins.
- How quickly TTS produces its first audio chunk.
- When that audio reaches the telephone connection.
A team can improve one interval and still leave the caller waiting because another interval dominates the turn. For this reason, TTFA, or time to first audio, became one of the most useful measurements in our discussions with early-access teams.
Measure the gap before the voice starts
Many teams already measure transcription accuracy and the total time required to produce a response. Both matter. Neither tells the whole story of a live call.
For phone interactions, we recommend recording timestamps for at least four events: the end of incoming speech, the final or usable transcript, the start of response generation, and the arrival of the first outbound audio chunk. Those timestamps help separate end-of-speech detection from TTS delay. Without that separation, a team may spend days changing voices when the larger issue is a conservative silence timeout.
Our current infrastructure is designed around short streaming intervals. It provides approximately 45ms STT latency and approximately 38ms TTS first-audio-chunk latency, with a sub-100ms full round-trip target in the product path. The exact experience still depends on the application, network, model behavior, and telephony provider. The point is not that one number makes a phone bot conversational. The point is that the stack must make the early part of each turn small enough to preserve the rhythm of a call.
Streaming also changes what the application can do while a caller is still speaking. A partial transcript can inform routing, intent preparation, or response planning before the final transcript arrives. That does not mean the bot should interrupt every sentence fragment. It means the system does not need to wait for every downstream operation to begin before preparing the next step.
One early-access team running approximately 80 concurrent inbound calls found that aggregate latency reports hid the most disruptive cases. Most turns felt acceptable, but calls with longer caller utterances and brief pauses triggered a second silence window. The resulting delay was much more noticeable than the median. Looking at the timeline of individual turns made the issue visible: end-of-speech detection, not synthesis, was often the source of dead air.
End-of-speech detection is a product decision
There is no single correct silence timeout for every phone bot. A support line for short yes or no answers can respond quickly. An appointment scheduler may need to tolerate dates, names, addresses, and corrections. A system that cuts off callers too aggressively feels rude. A system that waits too long feels absent.
This is where latency and human factors meet. People pause inside a thought. They inhale. They search for a date. They say “next week, actually, Thursday.” If the bot interprets every pause as the end of speech, it will create unnatural interruptions. If it waits for a long period of silence after every turn, callers will fill the gap themselves.
We found it useful to treat end-of-speech detection as a policy that can vary by interaction. The policy can consider the likely utterance type, whether the caller has already provided a complete slot value, and whether the system is waiting for a confirmation. It can also distinguish a short pause from a stronger signal that the caller has yielded the turn.
Partial transcripts help here, but they do not remove the need for judgment. A partial transcript can be incomplete or wrong. Acting on it too early can cause the bot to answer a question the caller was still forming. The practical goal is not to respond at the first possible millisecond. It is to avoid unnecessary waiting after the caller has clearly finished.
For teams testing this behavior, we suggest reviewing recordings with a simple annotation: mark the moment the caller appears finished, the moment the system decides the turn is complete, and the moment the first audio begins. The distance between the first two points reflects detection policy. The distance between the second and third reflects downstream and TTS delay. These are different problems and need different fixes.
Barge-in is part of sounding human
A phone bot can respond quickly and still feel artificial if callers cannot interrupt it. In real conversations, people correct themselves, change direction, and answer before a question is finished. A system that continues speaking over those corrections forces the caller to wait or repeat themselves.
Barge-in requires more than detecting incoming audio while TTS is playing. The application must decide when that audio represents a meaningful interruption rather than background noise, line noise, or a cough. It must stop outbound audio quickly, preserve the new caller utterance, and prevent the previous response from continuing in the background.
The timing matters. If a caller says, “No, I meant Friday,” and the bot speaks for another second before stopping, the interaction feels controlled by the machine. If the bot stops too readily on every small sound, the call becomes fragmented. Interruptibility therefore belongs in the same test plan as latency and recognition quality.
We also learned that interruption handling should be tested with ordinary telephone audio, not only clean microphone recordings. Our product context includes 8kHz narrowband telephone audio support, where speech has less acoustic information than wideband audio. Initial language support is English and Spanish, and teams need to evaluate pauses, overlaps, and corrections in the languages they intend to use. A behavior that works in a quiet development environment may behave differently on an actual phone connection.
What teams expected to matter, but often did not
Several teams expected TTS voice naturalness to be the top complaint. They spent time comparing prosody, pronunciation, and different speaking styles. Those details matter, especially in longer responses. But early-access feedback often pointed somewhere else first: the gap before TTS started.
The surprise was not that voice quality was irrelevant. It was that callers tolerated a merely good voice when the response began at the right time, while they distrusted a polished voice that arrived after noticeable dead air. Pacing and response timing shaped the perceived quality of the entire exchange.
This does not mean teams should ignore pronunciation or prosody. A voice that emphasizes the wrong word, reads a date unnaturally, or speaks at an exhausting pace creates its own problems. It means optimization should follow the caller’s sequence of perception:
- Did the system hear me?
- Did it understand enough to take a turn?
- Did it begin responding without an unexplained pause?
- Can I interrupt it if I need to correct something?
- Does the voice remain understandable and appropriate?
Teams also sometimes expected model response quality to be the first bottleneck. In practice, a concise response that starts promptly can feel more capable than a more comprehensive response preceded by silence. This is not an argument for shallow answers. It is an argument for separating response planning from audio delivery, and for avoiding unnecessary work before the first useful words can be spoken.
A practical sequence for early testing
Our recommendation to teams is to start with a narrow call flow and inspect every turn. Choose one task, such as confirming an appointment or collecting a delivery address. Record the audio and event timestamps. Then test ordinary behavior, interruptions, corrections, incomplete phrases, and short periods of background noise.
Track at least these questions:
- How long is the silence after the caller finishes?
- How often does the bot answer before the caller is done?
- How quickly does barge-in stop outbound audio?
- Does the system use partial transcripts without committing too early?
- Does the first TTS chunk contain a useful acknowledgement or only a delayed preamble?
- Does performance change with 8kHz telephone audio or concurrent calls?
Then fix the largest visible gap before tuning the voice. A shorter end-of-speech window may help more than a new speaker. Streaming a response may help more than generating a longer one. Faster interruption handling may help more than adding another conversational feature.
Smallest AI is a bootstrapped, three-person company building real-time voice infrastructure for these conditions. We are working with teams in early access rather than presenting this as a finished, self-serve product. The feedback is useful precisely because it exposes the difference between a component benchmark and the caller’s experience of a complete turn.
If you are building a phone agent and want to evaluate this approach, email [email protected] to apply for early access. We are especially interested in call flows where timing, barge-in, and narrowband audio make a measurable difference.
After enough call reviews, “sounding human” becomes a technical description rather than a marketing phrase. It means the system detects the end of speech without rushing, begins useful audio without dead air, stops when the caller interrupts, and keeps the timing of each turn within the range people expect from one another.