STT at 45ms + LLM at 150ms + TTS at 38ms = about 233ms before considering network and buffering, but the practical target is different: keep the full response loop below the point where a caller perceives a pause. A voice agent is not one model. It is a sequence of pipeline legs, and each leg adds time before the caller hears a response.
The pipeline is straightforward:
[Caller audio] -> [STT] -> [LLM (user's stack)] -> [TTS] -> [Caller hears response]That sequence makes latency budgeting useful. It also exposes a common mistake. Teams often optimize one component in isolation, then discover that the interaction still feels slow because the middle leg, network path, buffering strategy, or turn-taking logic consumes the remaining budget.
The latency budget is a chain, not a component score
For a voice agent, total response time can be viewed as the sum of three primary processing intervals:
- STT latency: the time required to produce a usable partial or final transcript.
- LLM latency: the time until the model begins producing a response, usually measured as TTFT, or time-to-first-token.
- TTS latency: the time until the first playable audio chunk is available, often called TTFA, or time-to-first-audio.
A typical hosted voice stack might allocate roughly 200ms to STT, 150ms to the LLM, and 180ms to TTS first audio. That is approximately 530ms before accounting for transport, queueing, audio buffering, and application logic. The figure is illustrative, not a universal measurement, but it shows how quickly the budget grows when each leg is merely acceptable.
Smallest AI focuses on the two audio legs that sit around the model. Our internal benchmark tests across 1,000+ audio samples in English and Spanish show approximately 45ms STT latency and approximately 38ms TTS first-audio-chunk latency. Actual results vary by audio length and language.
Those figures do not mean the entire application responds in 83ms. The LLM remains in the middle, and the application still has to manage network transport, prompt construction, tool calls, playback, interruption, and turn detection. They mean the Smallest AI legs can fit within a sub-100ms processing target, leaving more of the budget available for the user's model and application.
A practical allocation, and where it breaks down
Consider a voice agent with a 400ms total latency target. A team might plan the budget this way:
| Pipeline leg | Planned allocation | What it covers |
|---|---|---|
| STT | 50ms | Producing a usable transcript from incoming audio |
| LLM | 250ms | Time to first useful token or response fragment |
| TTS | 100ms | Time to the first playable audio chunk |
On paper, the numbers add up. In a live call, the result can be different. The caller's audio may need to travel over a WebSocket before transcription begins. The application may wait for a final transcript instead of acting on a partial transcript. The prompt may include conversation history or retrieved context. The LLM may take longer on a larger model or a tool call. The TTS service may receive the whole answer instead of a stream of response text.
In that situation, the nominal 250ms LLM allocation may become 350ms, and a small network delay may push the total beyond 500ms. Alternatively, the LLM may produce a token quickly, but the application may wait for a complete sentence before starting synthesis. The measured TTFT looks good while TTFA remains poor.
This is why latency tests should measure the events a caller experiences, not only isolated API timings. Useful timestamps include the end of the caller's speech, arrival of the partial transcript, arrival of the first model token, dispatch of the first text segment to TTS, arrival of the first audio chunk, and start of playback. Those timestamps reveal whether the problem is inference, coordination, buffering, or transport.
STT: optimize for usable partial transcripts
STT is the first decision point in the loop. A system that waits for a perfect final transcript can add delay even when its recognition quality is high. For conversational agents, a partial transcript can be more valuable than a slightly more accurate final result if the application can begin preparing the next step safely.
The important question is not simply, "How fast is transcription?" It is, "When does the application receive enough reliable text to do useful work?" That may mean routing the request, selecting a prompt, or preparing a response path while the final words are still being resolved.
Teams should measure STT under the conditions their agents actually face. Audio length and language affect results, as does silence detection, packetization, and the point at which the system decides that a turn is complete. A short command and a long explanation will not produce the same timing profile.
STT also interacts with turn-taking. If endpoint detection waits too long after the caller stops speaking, a fast recognizer cannot fully compensate. If it cuts off too quickly, the system may interrupt the caller or send incomplete text to the LLM. The best allocation therefore includes both recognition latency and the delay introduced by deciding that the turn has ended.
LLM latency is often the real bottleneck
The middle leg is frequently the limiting factor. Typical LLM TTFT can range from 50ms to 300ms depending on model size and infrastructure. A model may also spend additional time on long context, structured output, retrieval, moderation, or a tool call. In a voice interaction, these delays are exposed directly because the caller is waiting for audio, not just a text response on a screen.
Teams control some of this leg. They can select a faster model, reduce unnecessary context, stream tokens, cache stable instructions, simplify tool paths, and send the first response fragment to TTS as soon as it is safe to speak. They can also design prompts that produce concise spoken answers rather than long written responses.
Other parts remain outside the application team's direct control. Model queueing, provider infrastructure, inference load, and the behavior of a selected model can change over time. A faster STT and TTS layer cannot remove those variables. If the LLM does not produce usable text, the surrounding audio services have nothing to synthesize.
There is also a quality trade-off. Choosing the lowest-latency model may reduce reasoning quality, increase repair turns, or create longer conversations overall. The right question is not whether the first token arrives as quickly as possible. It is whether the model produces a safe, useful first spoken fragment within the interaction's budget.
TTS: first audio matters more than final completion
TTS latency should be evaluated by TTFA, not only by the time required to synthesize the full answer. In a streaming synthesis design, the caller can begin hearing a response while later text is still being generated. That changes the experience substantially, provided the first audio chunk arrives quickly and the system can maintain playback without awkward pauses.
Smallest AI's TTS first-audio-chunk latency is approximately 38ms in our internal benchmark framing. The benefit is most apparent when the application streams model output into synthesis instead of waiting for the complete response. It is also important to control chunk boundaries. Sending fragments that are too small can create overhead or unnatural prosody. Sending fragments that are too large delays the first sound.
TTS has a second responsibility beyond speed: it must support interruption. Callers will often speak while an agent is responding. The application needs to stop playback, preserve the new caller turn, and begin the next cycle without allowing queued audio to continue. Fast first audio is valuable, but fast cancellation and clean turn-taking are part of the same architectural problem.
Measure ownership boundaries, not just vendor endpoints
Optimizing STT alone is not enough. The whole chain determines whether a call feels natural. A team can have excellent STT and TTS measurements while still delivering a slow interaction because its own orchestration layer waits for final text, serializes avoidable operations, or buffers too much audio.
The ownership boundary matters here. With Smallest AI, the audio legs can be treated as focused infrastructure: STT receives caller audio, TTS turns model output into speech, and the user's stack retains control of the LLM and application behavior. That division lets teams tune the middle leg without replacing their model strategy or business logic.
For early integrations, I recommend recording a trace for every turn with at least these timestamps:
- Caller speech ended or endpoint detected.
- First usable STT transcript received.
- LLM request sent.
- First model token received.
- First text segment sent to TTS.
- First audio chunk received.
- Playback started.
That trace produces a budget the team can act on. It shows whether the next improvement belongs in recognition, model selection, prompt design, WebSocket handling, streaming synthesis, or playback. It also prevents a misleading optimization, such as celebrating a lower STT number when the caller still waits for the application to finish the entire LLM response.
The architectural choice is ultimately about control. If the LLM remains in the user's stack, the team owns model selection, TTFT, context, and tool behavior. If Smallest AI owns the STT and TTS legs, those interfaces should make streaming, timing, and interruption visible enough to tune. The most useful voice stack is not the one with the lowest isolated number. It is the one whose ownership boundaries let each team reduce the part of the round-trip it can actually control.