WebSocket vs gRPC for Real-Time Audio Streaming

At 16 kHz, mono PCM16 produces 32,000 bytes of audio per second, so a voice agent sending 20 ms chunks must deliver roughly 640 bytes every 20 ms while receiving transcription, control events, and audio output on the same conversational path. At that rate, choosing WebSocket or gRPC can appear like an implementation detail. It is not. Protocol behavior under packet loss, queue pressure, flow control, and concurrent streams often affects tail latency more than the median.

For real-time voice systems, the relevant question is not which protocol has the smallest header. It is which transport preserves predictable chunk delivery when the connection is busy, a packet is retransmitted, or several logical operations compete for bandwidth. WebSocket and gRPC can both support bidirectional audio, but they make different tradeoffs around framing, multiplexing, observability, and backpressure.

Two different streaming models

WebSocket begins as an HTTP upgrade and then becomes a persistent, full-duplex byte stream. Both endpoints can send messages independently without polling. An audio client can send binary PCM frames while the server sends partial transcripts, endpointing events, and synthesized audio over the same connection.

The protocol itself does not define an application schema for those messages. A service must decide whether each message is raw audio, JSON control data, a binary envelope, or a typed format such as Protocol Buffers. That flexibility is useful when a browser, telephony gateway, and native client all need to connect to the same endpoint. It also places more responsibility on the application layer. Message types, sequence numbers, timestamps, authentication refresh, error codes, and close semantics must be designed explicitly.

gRPC streaming uses HTTP/2 and usually Protocol Buffers. A client-streaming or bidirectional-streaming RPC creates a logical stream within an HTTP/2 connection. The client and server can send messages independently after the RPC starts. Each protobuf message is carried in a gRPC record with a one-byte compression flag and a four-byte message length, followed by the serialized payload.

That structure gives gRPC a formal contract through service definitions, generated client code, deadlines, status codes, interceptors, and built-in metadata handling. A bidirectional audio RPC can expose typed messages such as AudioChunk, TranscriptUpdate, and PlaybackMarker without inventing a separate envelope protocol.

The difference is therefore not that one protocol is bidirectional and the other is not. Both are. WebSocket offers a persistent full-duplex connection with application-defined messages. gRPC offers typed streams inside an HTTP/2 session, with transport and RPC semantics layered together.

Framing, overhead, and multiplexing

For small audio chunks, framing overhead is measurable but rarely the primary source of latency. A WebSocket binary frame has a small header, with client-to-server frames also carrying a four-byte masking key and a masking operation. The exact header size depends on payload length. For a 640-byte, 20 ms PCM16 chunk, this overhead is small relative to the payload.

gRPC adds its five-byte per-message record prefix, protobuf serialization, HTTP/2 framing, and connection-level metadata. Protobuf can be compact for structured events, though raw audio still occupies nearly the same number of bytes unless it is compressed or encoded differently. HTTP/2 also introduces stream identifiers, frame headers, flow-control windows, and HPACK-compressed headers. Most of those costs are amortized over a long-lived connection.

The important distinction is multiplexing. A WebSocket connection is one ordered message sequence. If an application wants four audio channels on one connection, it must define channel identifiers and scheduling rules inside the message format. The receiver still sees one ordered stream of WebSocket messages.

HTTP/2 can carry multiple gRPC streams over one TCP connection. A single connection could contain separate RPCs for audio uplink, synthesized audio downlink, control events, and diagnostics. HTTP/2 DATA frames from different streams can be interleaved, and the protocol maintains separate stream-level flow-control windows. This reduces the need for application-level multiplexing and makes per-RPC deadlines and status more natural.

Head-of-line blocking is more than one problem

Head-of-line, or HOL, blocking is often presented as an argument for HTTP/2 over a single WebSocket. That comparison needs precision. There are at least three queues to consider: the application message queue, the HTTP/2 stream scheduler, and the underlying transport.

With WebSocket, messages are ordered within one TCP byte stream. If a large message is queued ahead of an audio chunk, the audio chunk cannot pass it at the WebSocket layer. A poorly designed protocol that sends a large metadata object, file fragment, or buffered audio segment can therefore delay a time-sensitive 20 ms frame. The same issue appears when one application writer serializes unrelated work behind a slow operation.

HTTP/2 can interleave DATA frames from separate streams, so a large response on one gRPC stream does not have to occupy the entire application sequence before another stream sends a frame. Per-stream flow control also allows a receiver to constrain one stream without necessarily stopping every other stream.

However, HTTP/2 normally runs over TCP. TCP presents an ordered byte stream to HTTP/2, so a lost packet can delay delivery of later bytes even when those bytes belong to different HTTP/2 streams. HTTP/2 reduces application-level HOL between streams, but it does not remove transport-level HOL caused by TCP retransmission. A single congested or lossy connection can still affect all of its gRPC streams.

WebSocket and gRPC also differ in how visibly they expose backpressure. A WebSocket implementation may offer a buffered amount, write queue, or asynchronous drain signal, but the behavior varies by runtime. The application must decide when to stop accepting microphone frames, drop old data, or terminate a session. gRPC libraries generally connect writes to HTTP/2 flow control, though buffering can still occur in the client library, language runtime, proxy, or server.

For speech recognition, dropping arbitrary audio is usually unsafe because it creates gaps in the acoustic sequence. A better policy is to bound queues, measure send-to-server delay, and fail or degrade deliberately when the connection cannot keep up. For synthesized audio, a client may have more options, such as discarding superseded low-priority events while preserving playback order.

Tail latency in a voice agent

Median latency can look healthy while the 95th or 99th percentile produces audible stalls. A voice agent is sensitive to these tails because the user notices a delayed partial transcript, a late endpointing decision, or a missing first audio chunk even when most chunks arrive on time.

Consider a synthetic deployment in which a voice agent handles back-to-back calls through one long-lived connection from a regional media service. Each call sends 20 ms PCM16 frames and receives partial STT updates. The service also sends control messages for barge-in and routes TTS audio back to the media layer. In a WebSocket design, the application multiplexes calls with a call identifier in each binary envelope. If one call's TTS output is written in large buffers, or if the server's writer uses a single FIFO queue, another call's microphone frames can wait behind it. The median may remain acceptable, but the tail grows during overlap and bursty output.

In a gRPC design, each call can use a separate bidirectional RPC on the same HTTP/2 connection. Stream scheduling can prevent one RPC from occupying the entire application-level sequence, and per-RPC deadlines make failure handling clearer. But the streams still share the TCP congestion window and the connection-level flow-control window. A connection-level stall, an undersized window, or a proxy that buffers DATA frames can affect every call at once.

At Smallest AI, our streaming targets are a useful reference point for this distinction: median STT latency is approximately 45 ms, and TTS first-audio-chunk latency is approximately 38 ms. Those figures describe service behavior under measured conditions, not an end-to-end guarantee. A transport that adds occasional 100 ms queueing spikes can dominate the user experience even when model and service medians remain low.

Tail behavior should be measured with timestamps at each boundary: microphone capture, client enqueue, socket write, server receipt, decoder ingress, first transcript event, TTS request, first generated audio chunk, and client playback. Track p50, p95, p99, maximum queue depth, retransmission indicators where available, and the percentage of chunks that exceed their playback deadline. Test cold starts separately from warm persistent connections.

When HTTP/2 multiplexing helps, and when it hurts

HTTP/2 multiplexing helps when a product has many independent, moderate-volume streams and needs typed RPC semantics. For example, a backend may maintain separate streams for audio, tool events, agent state, and observability while sharing one authenticated connection. Stream-level cancellation is useful when a caller interrupts TTS or hangs up. Generated interfaces also reduce ambiguity between service teams.

It is particularly attractive for service-to-service deployments where browser compatibility is not a requirement and the organization already operates HTTP/2-aware proxies, load balancers, and gRPC telemetry. A fleet of native clients can benefit from consistent deadlines, status propagation, and schema evolution through protobuf.

Multiplexing can hurt when too many high-throughput streams share one connection. Connection-level flow control becomes a shared resource, and a single network path concentrates loss and congestion. Some implementations also batch writes, coalesce small messages, or use executor queues that add scheduling delay. Proxy support may introduce its own buffering, idle timeouts, maximum message sizes, and stream limits.

A common mistake is to interpret multiple gRPC streams as independent network paths. They are independent logical streams, not independent congestion domains. If tail isolation matters, several connections, connection pools, or explicit traffic classes may be more effective than placing every call on one HTTP/2 connection. This increases socket and handshake overhead, so it should be validated with production-like load rather than adopted automatically.

Practical protocol selection

WebSocket is usually the simpler default for browser-facing voice products, interactive demos, and APIs that need to work through a broad range of edge environments. It has direct browser support, straightforward binary message delivery, and fewer assumptions about generated client libraries. It also maps naturally to one conversational session per connection, which can make ordering and debugging easier.

Choose WebSocket with an explicit message contract. Include a protocol version, message type, session or channel identifier, monotonically increasing sequence number, capture timestamp, and sample metadata where needed. Keep audio chunks bounded, avoid mixing large payloads with real-time frames, and make queue limits visible in metrics. A binary envelope is often preferable to repeatedly parsing JSON around audio data.

gRPC is a strong choice for internal media services, native clients, and systems with many independently managed streams. It genuinely wins when typed contracts, generated code, cancellation, deadlines, and HTTP/2 stream multiplexing outweigh browser reach and deployment simplicity. This is not an argument that gRPC is the wrong protocol. It is an argument for treating HTTP/2 flow control, connection sharing, and proxy behavior as part of the latency design.

For either protocol, test under packet loss, bandwidth contention, concurrent calls, partial disconnects, and delayed consumers. Compare one stream per connection with pooled connections. Test 20 ms and 40 ms audio frames, because larger chunks reduce per-message overhead but increase the minimum unit of delay and the amount of audio lost when a frame misses its deadline. Confirm that intermediaries preserve streaming rather than buffering until a threshold is reached.

Our primary protocol is WebSocket streaming, with HTTP REST as a fallback for workflows that do not require continuous bidirectional audio. We recommend 16 kHz mono PCM16 for general speech input, with 8 kHz narrowband supported for telephony-oriented paths. That recommendation does not eliminate the value of gRPC. It reflects the need to support interactive clients where connection reachability, immediate binary delivery, and simple session semantics are often decisive.

The most useful protocol decision comes from measuring the complete queueing path, not from comparing wire headers in isolation. In a voice agent, a five-byte framing difference is usually less important than whether an audio chunk waits behind unrelated work, whether flow control is bounded, and whether a retransmitted packet stalls every logical stream sharing the connection. Those are the mechanisms that turn a low median into an unacceptable tail.

We're evaluating WebSocket and gRPC integration paths with early-access teams. Email [email protected] to discuss what fits your stack.