The End-to-End Latency Budget for a Voice Agent

A voice agent with approximately 45ms of speech-to-text latency and 38ms to produce the first text-to-speech audio chunk can still take 160ms to 400ms to respond. The reason is simple: the total response time is the sum of several sequential components, and most teams measure only one of them.

When we broke down the pipeline in our testing, the useful question was not “How fast is the model?” It was “Where does every millisecond go between the caller finishing a turn and hearing the next audio chunk?” That shift produces a latency budget you can inspect before deployment, rather than a single vendor benchmark that says little about the complete interaction.

Think in legs, not product metrics

A streaming voice agent usually has five important legs in the response path:

1. Network transfer of the caller's audio to the speech endpoint
2. Speech-to-text processing, or STT
3. LLM processing, including time to first token and time to the usable response
4. Text-to-speech processing, or TTS
5. Network transfer of synthesized audio back to the caller

These legs are not always perfectly sequential. Audio can stream in small chunks, transcription can begin before the caller has finished speaking, and TTS can begin before the full LLM response is available. However, the first audible response still depends on a chain of events. Delays in one leg can prevent the next leg from starting, or reduce the amount of parallelism available.

For a first-pass budget, use this model:

First audible response =
network to STT
+ STT processing
+ LLM time to usable text
+ TTS time to first audio chunk
+ network to caller

This is different from measuring STT latency in isolation. It is also different from measuring the time required to generate the complete answer. For a conversational agent, TTFA, or time to first audio, often determines whether the system feels responsive. Total completion time matters as well, especially for longer answers, but it belongs in a separate budget.

An illustrative latency allocation

The following figures show how the legs can fit together in a typical domestic US path:

Pipeline leg	Illustrative latency	What it measures
Network to STT	10 to 30ms	Caller audio reaching the speech endpoint
STT	Approximately 45ms	Smallest AI speech recognition processing
LLM	Typically 50 to 300ms	Time to produce usable text from the team's infrastructure
TTS	Approximately 38ms	Smallest AI time to the first audio chunk
Network to caller	10 to 30ms	Synthesized audio reaching the caller

Smallest AI's STT and TTS figures are based on internal benchmark tests across 1,000+ audio samples in English and Spanish. Actual results vary by audio length and language.

The Smallest AI processing legs add up to approximately 83ms before network transfer. Adding both network legs gives a typical combined range of approximately 100ms to 160ms, excluding the LLM. With an efficient LLM, a realistic full end-to-end range is approximately 160ms to 400ms, depending on the model and infrastructure.

For comparison, an illustrative typical voice stack might allocate approximately 200ms to STT, 150ms to the LLM, and 180ms to TTS, or about 530ms before considering network transfer. This comparison is directional, not a claim about a specific vendor. The point is that the processing legs can consume most of a conversational turn before network conditions are included.

Work one concrete budget before deployment

Consider a voice scheduling bot with a target of 350ms from the end of a caller's turn to the first audible response. The team assigns:

• 45ms for STT
• 200ms for the LLM
• 38ms for TTS to first audio chunk
• 30ms for the network path to the STT endpoint
• 30ms for the network path back to the caller

The planned total is 343ms. That leaves only 7ms for jitter, buffering, audio framing, queueing, and measurement error. The design is already effectively over budget for a production P95 target, even though the nominal arithmetic fits under 350ms.

A more practical version might reserve an explicit 40ms tail allowance:

• 45ms for STT
• 170ms for the LLM
• 38ms for TTS
• 30ms for each network leg
• 40ms for jitter and pipeline overhead

That budget totals 353ms. It misses the nominal target by 3ms, but it tells the team something useful: the LLM or network assumptions must change, or the target must be revised. A budget that exposes a conflict before deployment is more valuable than a benchmark that hides it.

The LLM leg is usually the variable leg

Teams can spend considerable effort reducing STT and TTS latency while leaving the LLM path undefined. That creates a misleading result. A 45ms STT service does not produce a 45ms response if the agent waits 300ms for the model, then performs tool calls, validates the answer, and sends the result to TTS.

The LLM leg belongs to the team building the agent. We can control the STT and TTS processing legs in our service, but the model choice, prompt, retrieval path, tool execution, output validation, and hosting configuration are outside those legs. Network performance is shared between the application, infrastructure providers, and caller location.

There are several ways to compress the LLM portion:

• Measure TTFT separately from total generation time.
• Use a response format that lets TTS start from the first safe phrase instead of waiting for the full answer.
• Keep prompts and retrieved context within a size that the selected model can process quickly.
• Move nonessential work, logging, and analytics out of the critical response path.
• Set explicit timeouts for tools and define a short spoken fallback when a dependency is slow.
• Test the model and infrastructure under concurrent load, not only with one local request.

There are limits to this optimization. A shorter prompt may reduce context quality. A smaller model may respond faster but make more mistakes. Streaming partial text can reduce TTFA while increasing the risk that an early phrase must be revised. Parallel tool calls may help when dependencies are independent, but they cannot remove a dependency that must be resolved before the agent can speak accurately.

Measure the tail, not just the average

Average latency is useful for finding broad regressions, but it is not enough for voice interactions. A system that responds in 180ms on average and 600ms at P95 may feel inconsistent. Track at least P50, P95, and P99 for each leg and for the complete path.

Instrument timestamps for:

• Last caller audio frame received
• STT partial and final transcript availability
• LLM request start, TTFT, and usable text availability
• TTS request start and first audio chunk availability
• First audio frame sent to the caller
• First audio frame observed by the caller, where that measurement is available

Also record audio chunk size, buffering duration, network RTT, queue time, retries, and whether the agent was speaking or listening when the event occurred. Streaming systems can conceal serialization problems. For example, an application may receive a fast first TTS chunk but hold it in a buffer until a larger audio segment is complete. The service metric looks good while the caller hears silence.

Jitter deserves its own line in the budget. Network RTT changes by geography and route. Model queues change with concurrency. Audio devices and telephony paths can add buffering that is invisible from the application server. A nominal latency figure without a tail allowance is not a deployment plan.

Build the budget around the first audible response

Before deployment, write down the target event and the measurement boundaries. Decide whether the target is last caller audio to first synthesized audio, last caller audio to first audio heard, or complete answer time. These are different measurements.

Then allocate every leg, identify which team owns it, and reserve space for jitter. Run the same test across representative languages, audio lengths, network locations, concurrency levels, and response types. If the budget fails, fix the largest variable leg first. Improving a 45ms STT result by a few milliseconds will not compensate for an LLM that regularly adds hundreds of milliseconds or a pipeline that serializes avoidable work.

For teams evaluating Smallest AI during early access, the practical starting point is to share the target latency, expected traffic pattern, languages, and current pipeline measurements at [email protected]. A complete budget gives the engineering discussion something more useful than a single headline number: it shows which milliseconds are available, which are constrained, and which still belong to the agent architecture.