Infrastructure Requirements for Production Voice AI

Production Starts With a Latency Budget

At 100 concurrent calls, a voice agent can remain available while becoming perceptibly slow, because queueing and buffer growth often appear before the service reports an outright failure.

Real-time voice infrastructure is a chain of deadlines. Audio must arrive from the caller, pass through transport and buffering, reach speech recognition, move through application logic, and then produce synthesized audio quickly enough to avoid an unnatural pause. Each stage has its own queue, and the system only feels responsive when the queues remain short under the expected load.

A useful latency budget separates at least four measurements:

• Transport latency: the time required to move audio between the client, media gateway, and inference service.
• Inference latency: the time required to produce a transcription or the first usable audio chunk.
• Application latency: the time spent in turn detection, business logic, retrieval, tool calls, and response generation.
• Playback latency: the time between receiving synthesized audio and actually playing it to the caller.

These measurements should be tracked separately. A median end-to-end response time can look acceptable while a smaller group of calls experiences long pauses caused by a saturated inference worker or a full audio buffer. Smallest AI's current service targets include approximately 45 milliseconds median latency for streaming speech recognition and approximately 38 milliseconds to the first audio chunk for text to speech. Those figures describe inference behavior under defined conditions, not the complete call experience. Network distance, application scheduling, and queueing still determine what the caller hears.

CPU and GPU Inference Are Different Capacity Decisions

CPU inference is often sufficient for smaller deployments, especially when the workload uses a compact model, moderate audio quality, and predictable concurrency. It can also simplify operations. A CPU worker is easier to place close to an application region, often has lower idle cost, and avoids the scheduling and memory characteristics of a shared accelerator.

CPU capacity becomes less attractive when the system must process many audio streams concurrently while maintaining low tail latency. Speech workloads are continuous rather than request-shaped: each call produces a stream of audio, and every active stream consumes compute even when the caller is speaking slowly or pausing. If workers cannot keep up with the incoming audio rate, recognition frames wait in an inference queue. The resulting delay is cumulative, not merely the processing time of one frame.

GPUs are useful when model size, concurrency, or the required processing rate exceeds practical CPU capacity. Batching can improve accelerator utilization, although streaming workloads limit how much batching is possible. A batch window that is too long adds delay before inference starts. A batch window that is too short may leave the GPU underused. The correct setting depends on audio frame size, model behavior, stream count, and the latency objective.

GPU deployments also introduce their own constraints:

• GPU memory must accommodate model weights, runtime allocations, and the working state of concurrent streams.
• Host-to-device transfers and synchronization can add overhead if the processing pipeline is not designed around streaming execution.
• Sharing an accelerator between workloads can create interference that is visible primarily in p95 and p99 latency.
• Scaling by adding GPU workers may be limited by regional availability, startup time, and the cost of keeping capacity warm.

This is not an argument that GPU inference is mandatory for every deployment. A contact center with a few simultaneous calls may achieve better economics and adequate latency with CPU workers. Conversely, a high-volume deployment can waste money on large GPU instances if the application layer, network path, or turn-taking logic is the actual source of delay. The decision should come from load tests that measure both throughput and tail latency at the intended audio rate.

Concurrency Creates Queues Before It Creates Outages

Concurrency is not simply the number of open WebSocket connections. A call may have an active inbound audio stream, a speech recognition task, application work, an outbound synthesis stream, and several buffers holding audio that has not yet been consumed. The system must be sized for those simultaneous activities.

Consider a synthetic contact center bot handling 80 concurrent calls during a peak hour. Suppose most callers alternate between speaking and listening, but several callers speak for extended periods at the same time. The average CPU utilization might remain below a nominal threshold while the active speaking group creates a recognition backlog. If each audio frame normally clears in time, the backlog stays invisible. Once processing falls behind, every later frame waits behind earlier work, and the caller hears delayed transcriptions or delayed responses.

The same pattern occurs on the synthesis side. A response that produces its first audio chunk quickly can still create a playback backlog if later chunks arrive faster than the client or media gateway can consume them. Without bounded buffers, a slow consumer causes memory usage to grow. With buffers that are too small, the system drops audio or repeatedly stalls playback.

Capacity planning should therefore measure:

• Maximum concurrent calls, not only average concurrent calls.
• Concurrent speaking streams and concurrent synthesis streams.
• Inference queue depth and queue wait time.
• Audio buffer age, size, and discard rate.
• WebSocket connection count per process, host, region, and upstream dependency.
• CPU, GPU, memory, and network utilization during burst periods.

Connection limits also appear at layers that are easy to overlook. A reverse proxy may have a maximum connection count. A process may run out of file descriptors. An upstream service may cap streams per account or per worker. NAT tables, load balancer idle timeouts, and container memory limits can all affect calls before the inference fleet reaches saturation.

For every queue, define a policy. Some work should wait briefly, some should be rejected, and some should degrade. A new call might receive a capacity response rather than entering an unbounded queue. A noncritical transcript enrichment task might be deferred. A voice response cannot usually tolerate indefinite waiting, so it needs a deadline and a fallback behavior.

Network Placement Determines the Floor for Responsiveness

Inference placement should be considered alongside model selection. A fast model in a distant region can produce a slower call experience than a slightly slower model located near the media ingress point. Every network hop adds propagation time, serialization, congestion risk, and another operational boundary.

A typical path may include the caller's carrier or browser, a media provider, an edge or load balancer, the application service, a speech recognition service, a language model or tool service, and a speech synthesis service. If these components are spread across multiple regions, the first audio chunk may be delayed by several round trips even when each individual service is healthy.

Regional deployment does not require every component to be duplicated everywhere on the first day. It does require an explicit choice about where latency matters most. For interactive calls, media ingress and streaming inference should generally be close to one another. Application services that perform slower work can sometimes be placed separately, provided the call path has timeouts and does not block audio handling.

WebSocket streaming adds an important operational detail: a connection is stateful. Once a call is attached to a region or worker, moving it is more difficult than routing an ordinary HTTP request. Load balancing should account for connection duration, uneven call distribution, and the effect of reconnects after a regional or process failure.

Measure network behavior from the actual deployment environment. Developer laptops and synthetic probes often produce clean results that do not represent carrier networks or customer browser locations. Useful measurements include connection establishment time, round-trip time, packet loss, jitter, reconnect frequency, and the time between an audio frame leaving the client and becoming available to inference.

Memory, Buffers, and Connection Lifecycles

Each active call carries state. That can include authentication context, turn-detection state, partial transcripts, audio buffers, synthesis playback state, tool-call metadata, and timestamps used for observability. The exact memory footprint varies by implementation, but the scaling relationship is direct: per-call state multiplied by concurrent calls becomes a material capacity requirement.

Audio buffers deserve special attention. A buffer should have a clear purpose and a bounded maximum size. Inbound audio may need a small jitter buffer to smooth network variation. Recognition may need a short rolling window for context. Outbound audio may need enough capacity to absorb normal scheduling variation without allowing an increasingly old response to play.

When memory pressure begins, latency can rise before the process is killed. Garbage collection can become more frequent, kernel memory reclaim can increase, and container scheduling can become less predictable. A process that appears healthy by connection count may already be producing degraded tail latency. Track resident memory, allocation rate, buffer age, and the number of calls approaching their limits.

Connection lifecycle management should be treated as part of the call protocol, not as an afterthought. A production WebSocket implementation needs:

• Authentication and authorization at connection establishment.
• Heartbeats that distinguish a quiet call from a dead connection.
• Idle and maximum connection timeouts appropriate to the use case.
• Reconnect handling that avoids duplicating audio or application actions.
• Cancellation propagation when a caller hangs up or starts a new turn.
• Graceful shutdown that stops accepting new calls and gives active calls a defined handoff or termination path.

Connection pooling requires careful terminology. Long-lived WebSockets are usually not pooled like short HTTP requests. The useful forms of pooling are often upstream session reuse, bounded worker allocation, and controlled reuse of established transport resources. Creating a new inference session for every small audio fragment adds setup overhead and can create connection storms during a burst.

Reconnection should also be idempotent. If a client reconnects after an ambiguous network failure, the server may not know whether the previous audio or tool request completed. Sequence numbers, call identifiers, and explicit acknowledgement boundaries can prevent duplicated actions. For a voice agent, repeating a tool call may be more serious than losing a partial transcript.

Operations Must Watch the Tail

Monitoring only median latency hides the calls that users describe as broken. Track p50, p95, p99, and maximum values for each stage, segmented by region, model, worker type, and call state. In particular, monitor time to first transcription result, time to stable transcription, time to first synthesized audio, and time between successive playable audio chunks.

Useful alerts are tied to user-visible behavior and queue conditions rather than a single infrastructure percentage. Examples include:

• p99 first-audio latency exceeding its service objective for several consecutive minutes.
• Inference queue wait time increasing while incoming call volume remains steady.
• Audio buffer age crossing a threshold on a material percentage of calls.
• Reconnect rates increasing by region or client version.
• WebSocket establishment failures or upstream connection refusals.
• Memory usage rising with call duration rather than returning to baseline after calls end.

Distributed traces should connect a call identifier across transport, inference, application logic, and playback. Logs need enough timing context to explain a pause without recording unnecessary audio content. Metrics should distinguish an inference slowdown from a client that stopped reading. These distinctions determine whether the response is capacity, placement, code, or client behavior.

When Simpler Infrastructure Is the Right Choice

Early deployments do not need a multi-region fleet, dedicated GPU pools, and independent queues for every component. Over-engineering can introduce more failure modes than it removes. A single region with CPU inference, bounded concurrency, a managed WebSocket gateway, and clear overload behavior may be the right architecture for validating call flows.

The key is to preserve the measurements and boundaries that allow a later scale-up. Keep inference behind an explicit service interface. Record queue wait separately from compute time. Make buffers bounded. Set connection and request deadlines. Exercise reconnects and graceful shutdown in testing. These choices cost little and prevent a prototype from becoming impossible to diagnose.

Scale when evidence requires it. If CPU utilization is low but p99 inference latency is high, adding CPUs may not help. If the model is compute-bound and queue wait grows with speaking concurrency, a GPU or more inference workers may be appropriate. If inference is fast but end-to-end latency is high, move services closer together or remove an unnecessary network hop. Production capacity is a systems property, not a label attached to one machine type.

The practical objective is predictable turn-taking under the concurrency profile the product actually serves. That means sizing compute, transport, memory, and operational response together, then testing the system at burst load rather than only at its average. A voice agent is ready for production when its slow paths are observable, its queues are bounded, and its degradation behavior is deliberate.

If you're scoping production infrastructure for a voice agent and want to discuss the concurrency and latency tradeoffs, [email protected] is the right starting point.