We set a 40 millisecond target for time-to-first-audio, then found that the model was only part of the problem: buffer flushes, WebSocket framing, and scheduler decisions could each add more delay than a small inference optimization removed.
At Smallest AI, our text-to-speech path is designed for voice agents rather than long-form narration. The useful question is not only how quickly a complete utterance can be synthesized. It is how quickly the agent can begin speaking after it has enough text to produce a natural first phrase.
Our current internal benchmark is approximately 38 milliseconds for TTS first-audio-chunk latency. The benchmark is based on internal tests across more than 1,000 audio samples in English and Spanish. Actual results vary by audio length and language. The number is also specifically about the first audio chunk, not the time required to synthesize and transmit an entire response.
Start with the latency budget, not the model
Voice latency is a chain. A response may pass through turn detection, language-model generation, text normalization, TTS inference, audio buffering, encoding, and network delivery before a caller hears anything. Optimizing only the synthesis model leaves the other queues untouched.
For TTS, we treated time-to-first-audio as a separate budget from total utterance duration. A useful decomposition looks like this:
- Text arrival and normalization
- Inference scheduling delay
- Autoregressive decoding for the initial acoustic frames
- Audio conversion and buffer assembly
- WebSocket frame creation and transmission
- Playback-side buffering before the first sample is rendered
Each step can be small in isolation and still produce an audible pause when added together. More importantly, the slowest requests are often governed by queue behavior rather than the median request. That is why we looked at tail latency, including P95 and P99 behavior, instead of relying on a single average.
The target was therefore not “make every synthesis operation complete in under 40 milliseconds.” The constraint we actually care about is getting the first usable PCM audio to the agent connection quickly enough for turn-taking to feel immediate. The rest of the utterance can continue streaming while playback begins.
Streaming changes what the first result means
A conventional request-response TTS endpoint often encourages a simple sequence: send all text, wait for a complete result, then return audio. That can be appropriate for file generation, but it creates the wrong dependency for an interactive agent. The caller does not need the final phoneme before hearing the first word.
Our TTS path uses streaming output. As soon as the decoder has enough material for a valid audio segment, the service can pass it into the output stream. The client receives audio incrementally over a WebSocket connection, with HTTP REST available as a fallback. Python and Node.js SDKs provide the same basic streaming model without requiring each application to implement its own frame handling.
This approach introduces an important design choice: how much audio should be placed in the first chunk?
A very small chunk can reduce the time until the first byte, but it increases framing overhead and can make playback sensitive to jitter. A large chunk gives the receiver more runway, but it delays the first audible output and may force the system to wait for audio that is not needed yet. The best setting is not the smallest possible frame. It is the smallest frame that starts playback without creating a new buffering problem.
We use 16 kHz mono PCM as the recommended input format for the voice pipeline, while 8 kHz narrowband audio remains supported for telephone-grade paths. The sample rate changes the amount of data carried by each time interval, so the framing and playback assumptions need to match the connection. A chunk policy that works for wideband audio may create different queue behavior when the same agent handles narrowband calls.
Chunk size is a scheduling decision
Chunk size is often discussed as a network optimization, but it also controls when inference work becomes visible to the client. If the service waits for a fixed number of samples before flushing, that threshold becomes part of TTFA. If the threshold is too high, the network sees fewer, larger frames and the user waits longer. If it is too low, the system spends more time managing frames and the receiver has less protection against jitter.
Consider a synthetic appointment-reminder bot handling roughly 50 concurrent calls over WebSocket connections. Some calls use 8 kHz telephone-grade audio, while the agent response is generated in short phrases such as “Your appointment is confirmed.” We might compare an initial frame policy around 512 bytes with one around 2,048 bytes. The smaller frame is more likely to reach the client earlier, but it provides less playback runway. The larger frame is more efficient to transmit, but it makes the first-audio decision wait for more decoded material.
The right evaluation is not simply which policy produces the lower median. We would inspect:
- TTFA at P50, P95, and P99
- Time between consecutive audio frames
- Playback underruns after the first frame
- WebSocket write and flush time under concurrent load
- Differences between English and Spanish samples
- Behavior for short responses compared with multi-sentence responses
This is also why we avoid treating a single chunk-size setting as universally correct. The first chunk has a different job from later chunks. The first chunk establishes responsiveness. Later chunks maintain continuity and should be large enough to avoid unnecessary scheduling overhead without accumulating avoidable delay.
Buffer management: flush early, but do not starve playback
Buffers exist at several layers, and each one can hide latency. The inference buffer holds generated acoustic information. The service buffer holds audio waiting for conversion or transmission. The WebSocket implementation may hold a frame before it is written. Finally, the client or audio device may queue samples before playback.
When those buffers are tuned independently, the system can look fast in service logs while still sounding slow to a caller. For example, the synthesis process may produce the first samples quickly, but a downstream flush threshold can wait for additional bytes. Conversely, flushing every tiny segment can expose the caller to jitter because there is not enough queued audio to bridge a short network delay.
We therefore separate first-chunk handling from steady-state streaming. The initial output is allowed to flush as soon as it satisfies the decoder and audio-format requirements. Once playback starts, subsequent chunks follow a pacing policy intended to keep the receiver supplied without building an unnecessary queue.
That policy also needs backpressure. If the consumer is slower than the producer, blindly generating more audio increases memory use and can make cancellation expensive. If the producer reacts too aggressively to a momentary slow write, it can create gaps. The service needs to observe the state of the stream and schedule the next inference work accordingly.
Cancellation is part of this path. Voice agents frequently interrupt themselves when the caller begins speaking. A long precomputed audio buffer may continue to arrive after the response is no longer relevant. Streaming lets us stop closer to the current decoding position, reducing wasted work and shortening the time before the next turn can be processed.
Inference scheduling controls the tail
Autoregressive decoding creates a scheduling problem that is easy to miss when testing one request at a time. A single request may have acceptable TTFA, while concurrent requests wait behind an inefficient batch or a busy worker. For an interactive system, the scheduler needs to account for the value of the first chunk, not only total throughput.
We prioritize getting an initial segment from each active stream before allowing one long utterance to consume disproportionate compute time. That does not mean every request receives identical priority. It means the scheduler recognizes that a request with no audio yet has a different latency state from one that is already streaming steadily.
There is a tradeoff between batching and responsiveness. Larger batches can improve hardware utilization, but waiting to form a batch can add queue delay. Smaller batches may reduce waiting for individual requests while increasing per-request overhead. We tuned around the interactive case, where an earlier first chunk is generally more valuable than squeezing maximum throughput from a complete long-form synthesis job.
Speculative decoding can be useful in some generative systems, but it does not eliminate the rest of the path. A faster token or frame prediction step still has to pass through audio conversion, buffering, scheduling, and the network. We treat those techniques as possible components of an optimization strategy, not as a substitute for measuring end-to-end TTFA.
What this approach does not solve
Low TTS first-audio latency cannot compensate for every delay in a voice agent. If the language model takes too long to produce the first phrase, TTS has nothing to synthesize. If turn detection waits too long to decide that the caller has finished, the perceived response remains slow. Network distance, packet loss, client buffering, and audio device behavior also sit outside the model's inference time.
There is a quality tradeoff as well. Generating and sending very short fragments can make prosody less stable if the system commits before enough linguistic context is available. Voice selection and SSML pacing can improve agent-style delivery, but pacing instructions may require additional context before the most natural output can be chosen. The goal is not to minimize every buffer regardless of speech quality. It is to avoid waiting for data that the listener does not need before playback can begin.
Our benchmark also has a defined scope. The approximately 38 millisecond figure is an internal first-audio-chunk result across English and Spanish samples, not a guarantee for every language, network, text length, or concurrency level. Actual results vary by audio length and language. Teams evaluating a production path should measure from their own text arrival event through playback, including their client buffers and network conditions.
Smallest AI is currently in early access, and we are still refining these boundaries with a small, three-person engineering team. Developers interested in testing the streaming path can email [email protected] to join early access.
The main lesson from the work was that sub-40-millisecond TTS was not one breakthrough inside the synthesizer. It was the result of treating the first audio chunk as a scheduling deadline, then making every buffer, flush, and frame justify the time it added.