Batch recognition looks cheaper because it turns speech into a completed request. For interactive voice agents, that accounting misses the cost of waiting. A caller does not experience an audio file, an API response, and a completed transcript as separate stages. They experience a conversation, and every delayed response changes its rhythm.
Streaming transcription is therefore the practical path for voice products that must listen and respond while a person is still speaking. The trade-off is not simply API price versus latency. It is a choice between processing a live stream continuously or accepting extra buffering, turn detection, and request overhead before the system can act.
Where the latency actually accumulates
A voice agent's response time starts before speech recognition returns text. The audio must be captured, encoded, divided into chunks, sent over the network, processed, and passed to the reasoning or application layer. The agent then generates an answer, converts it to audio, and sends that audio back to the caller.
With batch recognition, another wait appears at the front of this chain. The system usually buffers a phrase, waits for end-of-speech detection, uploads the completed segment, and only then receives a transcript. The batch API may process the audio quickly once it receives it, but the user has already spent time waiting for the segment to become available.
Streaming changes the timing model. A WebSocket carries audio chunks as they arrive, and the recognizer returns partial transcripts before the utterance is complete. The application can use those partial results to prepare intent classification, retrieve context, or begin other work while the speaker finishes.
| Stage | Batch recognition | Streaming transcription |
|---|---|---|
| Audio delivery | Buffered until a segment is ready | Sent continuously in chunks |
| Transcript availability | After request processing begins | Partial text arrives during speech |
| Turn handling | Often coupled to a completed segment | Can use partial text and VAD together |
| Best fit | Async transcription and analysis | Interactive agents and live controls |
Why first-word latency matters more than average latency
Average recognition time can hide the part of the interaction users notice most: first-word latency. If an agent waits for a complete sentence before it starts processing, a fast final transcript may still produce a slow conversation.
Consider a scheduling agent handling a call to move an appointment. The caller says, “Could you change my visit to next Thursday afternoon?” In a batch design, the application may wait for end-of-speech detection, package the utterance, send it to a batch API, and wait for the final result. If that path adds an illustrative 300 to 400 milliseconds after the caller finishes speaking, the response begins late enough to disrupt the conversational rhythm. The problem becomes more noticeable when the system also waits on a separate intent service or calendar lookup.
With streaming transcription, the recognizer can expose text such as “Could you change my visit” before the caller reaches “next Thursday afternoon?” The application should not commit to an action from an unstable partial transcript, but it can begin lightweight preparation. Once the final words arrive, less work remains on the critical path.
This distinction is important: partial transcripts are signals for early work, not necessarily final input. A streaming implementation needs to reconcile revisions, identify stable words, and wait for sufficient confidence before taking an irreversible action.
Streaming has costs, but they are visible and controllable
Streaming is not free of engineering work. A client must maintain a WebSocket, send audio at a consistent cadence, handle reconnects, and distinguish interim text from final text. The service must also account for backpressure, dropped chunks, connection state, and cancellation when a caller hangs up.
Buffering is another trade-off. Very small audio chunks can increase protocol overhead and scheduling pressure. Larger chunks reduce request frequency but add delay. Audio format matters too. Smallest AI recommends 16kHz mono PCM for streaming transcription and supports 8kHz narrowband audio, which is common in telephony environments. Keeping capture, resampling, and transport behavior explicit makes latency easier to reason about.
Voice activity detection introduces a second timing decision. An overly aggressive VAD configuration can cut off speech or create premature turns. A conservative configuration avoids interruption but adds silence to the interaction. The right setting depends on whether the agent is handling short commands, open-ended explanations, or noisy calls.
These costs are operational rather than mysterious. Engineers can measure audio chunk duration, network transit, first partial transcript, final transcript, end-of-speech detection, and agent response start separately. That makes it possible to find whether the delay comes from buffering, the recognizer, application logic, or the response stack.
How the Smallest AI interfaces map to the trade-off
Smallest AI exposes streaming transcription through /v1/stt/stream, a WebSocket endpoint designed for audio that arrives over time. Its stated streaming STT latency is approximately 45ms, and the stated full round-trip target is under 100ms. Those figures describe the speech infrastructure path, not the total time required for an agent to reason, access a tool, and produce speech.
For completed audio segments, /v1/stt/recognize provides an HTTP POST batch recognition path. HTTP is simpler for an uploaded file or a finished recording, and it can be a useful fallback when the caller does not need an immediate response.
WebSocket: /v1/stt/stream
HTTP POST: /v1/stt/recognize
Recommended audio: 16kHz mono PCM
Supported narrowband audio: 8kHzThe transport choice should follow the product's timing requirement. A live agent generally needs WebSocket streaming as its primary protocol, with HTTP REST available for batch or fallback workflows. A system that starts with batch recognition and later tries to retrofit partial transcripts often has to redesign turn management, state handling, and downstream service calls.
Where batch recognition is the right choice
Batch recognition remains appropriate when the user is not waiting for a conversational response. Recorded interviews, meeting archives, call-center review, searchable audio, caption post-processing, and offline analytics can all tolerate a completed file or segment. In these cases, a simpler HTTP request may reduce client complexity and make retries easier.
Batch processing also fits workflows where sentence-level latency is acceptable. An analytics pipeline can wait for a full recording, run recognition asynchronously, and process the resulting transcript in larger units. There is no need to keep a live socket open, track partial transcript revisions, or make a turn decision in real time.
The boundary is not that batch recognition is technically inadequate. It is that batch's waiting model becomes part of the user experience when an application must respond during a live exchange. For interactive voice, the relevant metric is not only real-time factor, or RTFX, after an utterance is available. It is the time from incoming speech to useful action.
Designing around the actual boundary
A practical architecture can use both modes. Send live audio to /v1/stt/stream, retain stable final text, and use VAD plus application rules to decide when a turn is complete. Route recordings and asynchronous jobs to /v1/stt/recognize. This keeps the low-latency path focused on interaction while preserving a simpler path for completed audio.
During early access, Smallest AI is focused on real-time voice infrastructure and speech agent systems. The company is bootstrapped and has not publicly launched. Teams evaluating the streaming path can reach out at [email protected].
The cost boundary sits where waiting becomes visible: not at the moment a batch request finishes, but between a caller's first words and the agent's first useful response. For non-interactive audio, batch recognition can be the sensible option. For a live voice agent, that waiting interval is part of the product, which makes streaming the necessary foundation.