The interruption starts before the caller finishes a word
A caller who interrupts a voice agent mid-sentence and waits 800 milliseconds for it to stop talking experiences the delay as indecision, even if every individual component is behaving within its expected range.
Barge-in detection is the mechanism that prevents that delay. It identifies that the caller has started speaking while the agent's TTS audio is still playing, then changes the system from output mode to listening mode. In a real-time voice system, this is not a single classifier decision. It is a coordinated cancellation path across audio capture, streaming speech recognition, playback, language-model generation, and TTS.
Consider a synthetic outbound appointment reminder. The agent says, "Your appointment is scheduled for Thursday at two p.m. Please have your insurance card ready, and if you need to reschedule..." The caller interrupts with, "Thursday is wrong. I need Friday." A natural system should stop the original sentence, preserve the caller's correction, and begin responding to it. A system without carefully designed interruption handling may finish the sentence, continue generating an answer to the wrong conversational state, or play fragments of both responses over each other.
From my perspective working on real-time audio systems, the important question is not simply whether a system supports barge-in. It is how quickly and safely it moves through the interruption path.
The interruption signal chain
The core sequence looks like this:
- Incoming microphone audio reaches the streaming STT service.
- The recognizer detects speech activity and emits partial transcript updates.
- The orchestration layer decides whether the activity is a genuine interruption.
- Current TTS playback is cancelled and audio buffers are flushed.
- In-flight language-model generation is discarded or marked obsolete.
- The partial or final caller transcript becomes the input to a new turn.
- The STT to LLM to TTS pipeline begins again.
Each step has state associated with it. The system needs to know which transcript belongs to which turn, which TTS stream is currently audible, and whether a generated token is still relevant. Without turn identifiers or cancellation tokens, an old response can arrive after the caller has already changed the subject. That creates one of the most confusing failure modes in voice UX: the agent appears to hear the interruption, but answers the previous question anyway.
In a streaming architecture, cancellation needs to be explicit. Stopping the speaker output is not enough if the LLM continues generating, and cancelling the LLM is not enough if already generated TTS audio remains queued for playback. Barge-in is complete only when all stale work is prevented from reaching the caller.
Three latency contributions
The time from the caller beginning to speak to the agent becoming quiet and ready for the next response can be divided into three practical contributions:
- Detection latency: the time required to distinguish new caller speech from silence, background sound, or acoustic echo. With streaming STT, partial transcripts can provide evidence before the caller finishes a sentence.
- Abort latency: the time to cancel the TTS stream, stop the playback path, and flush audio that has not yet been played.
- Recovery latency: the time to stabilize the new transcript, discard stale generation, and begin the next STT to LLM to TTS turn.
These numbers do not add up in a perfectly serial way. Some cancellation and recognition work can happen concurrently. Still, the decomposition is useful because it shows where an apparently slow interruption originates. A team may optimize model generation while leaving a large audio buffer in the telephony layer. Or it may detect speech quickly but wait for a final transcript before cancelling playback, losing the main benefit of streaming recognition.
Our streaming STT has a median latency of approximately 45 milliseconds, and our TTS first-audio-chunk latency is approximately 38 milliseconds. Those figures describe important portions of the pipeline, not the complete interruption experience. Network transport, audio framing, device buffering, orchestration, and telephony behavior still contribute to the time a caller perceives.
Partial transcripts and the decision to interrupt
Streaming STT changes the timing of barge-in because the system receives partial transcripts rather than only a final result. The first partial update may contain only a short fragment such as "Thursday," "no," or even an unstable phonetic interpretation. Waiting for the final transcript improves certainty but makes the agent continue speaking over the caller. Acting on every partial update makes the system vulnerable to false interruptions.
The usual solution is to separate speech onset from semantic commitment. The onset signal can be enough to stop or attenuate TTS, while the transcript itself remains provisional. The agent does not need to know exactly what the caller said before yielding the floor. It needs enough evidence that the caller intends to speak.
That evidence can combine several signals:
- Voice activity over a minimum duration rather than a single noisy frame.
- A partial transcript with non-empty, changing content.
- Confidence or stability information from the recognizer.
- Whether the detected audio overlaps strongly with the agent's own playback.
- Conversation context, such as whether the caller has already started responding to a question.
A useful state machine might move from speaking to possible interruption, then to confirmed interruption. During the possible state, the system can reduce the amount of new TTS audio it schedules. Once the event is confirmed, it cancels playback and records the partial transcript. If the signal disappears immediately and the recognizer produces no meaningful content, the system can return to speaking instead of throwing away the response.
There is no universal threshold. Aggressive barge-in sensitivity is not always the right choice. A quiet support call in a controlled environment may tolerate a low threshold, while a speakerphone in a busy office may require more protection against noise and echo. The right policy depends on whether false continuation or false interruption is more damaging for the application.
What TTS abort actually means
"Stop TTS" can refer to several different operations, and they do not have identical user-visible effects.
At the application layer, the active TTS request should be cancelled so that no additional audio chunks are generated. At the playback layer, chunks already queued locally must be removed. At the transport layer, packets already sent may be impossible to retract. In a phone call, some audio may already be inside the carrier or endpoint buffers and will continue playing for a short interval.
This is why streaming TTS matters. If an entire response is rendered before playback begins, the system has fewer opportunities to stop cleanly and may have a large amount of stale audio waiting in a queue. With streaming output, the response is delivered incrementally, and the implementation can keep the unsent buffer small. The tradeoff is that cancellation must be handled at chunk boundaries and across every downstream buffer.
Playback should also carry a generation or turn identifier. Suppose the agent begins response 17 and sends six audio chunks. The caller interrupts, and response 18 begins. If chunk seven from response 17 arrives late, the playback consumer should reject it because its identifier is obsolete. A simple cancellation flag is often insufficient when asynchronous callbacks, retries, and network reordering are involved.
There is also a product decision around audible completion. Some systems cut audio immediately, even in the middle of a word. Others apply a very short fade or allow the current frame to finish to avoid a click. The fade must not become a hidden delay. An interruption handler that takes 100 milliseconds to sound polite may be less natural than one that stops abruptly but promptly.
Acoustic echo and false barge-in
In a phone conversation, the caller's microphone may capture the agent's TTS playback. This acoustic echo is especially difficult when the caller uses a speakerphone, when the phone's echo cancellation is imperfect, or when network audio introduces timing variation. Without protection, the recognizer can interpret the agent's own words as caller speech and trigger a false interruption.
Echo handling usually requires more than checking whether the recognizer returned text. The system knows the audio it sent toward the caller, so it can compare the incoming signal against a reference of the recent playback. Echo cancellation can reduce correlated playback energy, while a barge-in policy can discount transcript fragments that align closely with the agent's current words.
The comparison is not exact. The received echo can be delayed, filtered by the handset, mixed with room noise, or combined with genuine caller speech. A caller saying "yes" while the agent says "Thursday" produces a mixed signal, not a clean choice between two sources. For that reason, echo suppression should reduce false positives without making the system deaf to a real interruption.
One practical approach is to use multiple modes rather than a binary detector. During clean playback, require stronger evidence before confirming a barge-in. When the incoming audio contains speech that differs from the known TTS content, lower the confirmation threshold. When the system has already detected a stable partial transcript that is inconsistent with the agent's output, cancel quickly and let the next turn resolve the ambiguity.
Testing should include handset calls, speakerphone calls, noisy rooms, crosstalk, packet loss, and callers who begin speaking at different points in a sentence. A detector that performs well on isolated microphone recordings can behave very differently once its own streamed audio is present in the input.
When teams ship without interruption handling
Interruption handling is often underdeveloped because a basic voice demo can appear successful without it. The developer starts a response, plays the audio, waits for completion, then accepts the next turn. That sequence works as long as the caller remains silent until the agent finishes.
In production, callers do not follow that protocol. They correct names, answer questions before the prompt ends, say "wait," ask for repetition, or begin speaking because they assume the agent has finished. Without barge-in, the agent talks over them. The caller then repeats the correction, often louder, while the system continues processing the original response. This increases turn length and can cause the recognizer to receive a mixture of agent and caller speech.
The other failure is partial cancellation. The system may stop the speaker but leave the old LLM request running. It may then append the caller's words to the old context and generate a response that combines incompatible turns. Or it may start a new TTS stream while old audio remains buffered, producing an audible splice between two answers.
These are not only model-quality problems. They are lifecycle problems. Every response needs a clear owner, a cancellation path, and a rule for what happens to partial input when the owner is revoked. Observability should record speech onset, confirmation time, TTS cancellation time, last audible chunk, LLM cancellation time, and first audio from the replacement response. Without those timestamps, "the agent interrupted badly" is difficult to diagnose.
Choosing the interruption policy
The central design decision is how much evidence to require before yielding the floor. A highly sensitive policy makes the agent feel responsive, but it may stop for coughs, keyboard sounds, short acknowledgments, or its own echo. A conservative policy protects long responses from noise, but callers may have to wait for the agent to finish before they can correct it.
The policy can also vary by response type. A short confirmation may be allowed to finish, while a long explanation should yield quickly. A yes-or-no question may use a lower threshold because the caller is expected to answer during the turn. An outbound compliance script may need stricter controls if accidental interruptions create operational errors.
We generally find it useful to measure both sides of the tradeoff:
- Missed interruption: the caller starts speaking, but the agent continues audibly for too long.
- False interruption: the agent stops even though the caller did not intend to take the turn.
- Stale audio duration: how much old TTS remains audible after interruption onset.
- Recovery time: how long after confirmed interruption until replacement TTS begins.
- Transcript integrity: whether the caller's partial speech survives cancellation and reaches the next turn.
Streaming STT and streaming TTS make these measurements actionable. Partial transcripts can expose speech onset early, while incremental audio output limits the amount of stale response that must be discarded. The remaining work is orchestration: deciding when a partial signal is sufficient, cancelling every downstream operation, and preserving the new turn without importing artifacts from the old one.
The most reliable interruption path treats audio, transcript, generation, and playback as one coordinated state machine. Detection is only the first event. The system must also prove that obsolete audio cannot re-enter the output path, that echo is not mistaken for intent, and that the caller's unfinished words remain available when the next response is generated. Those invariants matter more than any isolated latency number because they determine whether a fast component produces a fast conversation.
Interruption handling is one of the first things teams want to understand during early access. Email [email protected] to start a conversation about your use case.