Speech interruption is a state machine

Cancellation travels through several different systems

A spoken response can exist in several places at once: pending language-model tokens, text already sent to synthesis, audio being produced, packets in transit, decoded samples in an application queue and samples handed to the output device. Cancelling the generation request reaches only one of those places. The synthesizer may finish an accepted segment, a network callback may deliver buffered data, and the player may already have scheduled audio. I would begin the interruption design by drawing these ownership boundaries and identifying which component can prevent each remaining piece of work from becoming audible.

The immediate user requirement is usually to stop the current response quickly enough to make room for the interruption. Reclaiming upstream computation is important, but it need not sit on the critical path for muting obsolete playback. Invalidate the current response at the playback gateway, clear or stop what that gateway owns, and cancel the upstream work independently. This ordering does not magically retract audio already heard or guarantee instantaneous hardware response. It does establish that a slow cancellation acknowledgement cannot authorize the application to keep admitting chunks from a response that the conversation has abandoned.

Give every response an identity that can expire

I would assign each assistant response an epoch, carried with every text segment, synthesis job and audio chunk belonging to it. The playback gateway admits only chunks whose epoch matches the currently active response. Interruption increments or replaces that identity before requesting upstream cancellation. A late callback then fails admission even if the cancelled worker continues briefly. Starting a new response must use a fresh identity as well, so old data cannot become valid merely because the player has returned to a speaking state. In a multi-process design, a session identifier plus a response identifier is safer than a process-local counter alone.

The small model below illustrates the admission rule using a single-threaded event order and placeholder chunks. It is not an audio driver and does not claim to synchronize concurrent callbacks by itself. A real implementation must serialize the state transition or use a synchronization mechanism that makes validation and queue insertion atomic relative to interruption. Otherwise a callback can pass its epoch check, pause, and append stale audio after the interrupt has cleared the queue. The invariant belongs at the final shared queue boundary, not only at an earlier network handler that assumes nothing can change afterward.

class PlaybackGate:
    def __init__(self):
        self.epoch = 0
        self.speaking = False
        self.queued = []

    def begin_reply(self):
        self.epoch += 1
        self.speaking = True
        self.queued.clear()
        return self.epoch

    def enqueue(self, epoch, chunk):
        if epoch != self.epoch or not self.speaking:
            return False
        self.queued.append(chunk)
        return True

    def interrupt(self):
        self.epoch += 1
        self.speaking = False
        self.queued.clear()

gate = PlaybackGate()
old = gate.begin_reply()
assert gate.enqueue(old, 'first audio')
gate.interrupt()
assert not gate.queued
assert not gate.enqueue(old, 'late audio')
current = gate.begin_reply()
assert not gate.enqueue(old, 'very late audio')
assert gate.enqueue(current, 'new response')
assert gate.queued == ['new response']

Packet order does not identify a conversation turn

RTP defines sequence numbers for packet ordering and loss detection, and timestamps tied to the media sampling clock. Those fields solve transport and playback problems; they do not tell an application whether a response remains conversationally relevant. A perfectly ordered packet can belong to an interrupted utterance. Conversely, a valid new response may arrive after a gap without becoming semantically stale. Keep media ordering and response identity as separate coordinates. The receiver needs both to decide whether a chunk is timely enough for playback and whether it belongs to the response currently allowed to speak.

An epoch should also be interpreted within its session lifetime. If a client reconnects and resets a small counter, an old message from the previous connection must not collide with a new response that happens to reuse the same number. A fresh session token prevents that confusion. For a distributed system, specify who creates the token and how components propagate it through retries and synthesis segmentation. These details look mundane until a delayed retry recreates an interrupted sentence. The protocol should make stale ownership observable, allowing logs to explain why a chunk was dropped without depending on its audio content.

References: [1] RFC 3550 — RTP: A Transport Protocol for Real-Time Applications

Measure the stop path at the speaker boundary

The Web Audio specification provides a scheduled stop operation for an AudioScheduledSourceNode, expressed against the audio context's time coordinate. That is a useful playback primitive, but it is not the same operation as cancelling a remote synthesis request. The application must track which sources have started, which queued buffers remain under its control and how new data is admitted. A stop call acts within the audio system's semantics; the full product still has detection, event delivery and buffering delays to account for. Measure the user-visible result rather than assuming the API call timestamp equals silence.

For an illustrative sequential budget, sixty milliseconds to detect an interruption, twenty to deliver the control event, ten to flush application work and thirty of unretractable device buffering produce a 120-millisecond path. Those numbers are invented to make the accounting concrete. Real stages can overlap, and measurements should identify their actual critical path. A three-hundred-millisecond server cancellation would not belong on this immediate stop path if local admission has already been revoked. A short fade may reduce an audible click while adding a little time; choose that tradeoff explicitly and include it in the measured response.

References: [2] W3C Recommendation — Web Audio API, AudioScheduledSourceNode.stop

Conversation history must distinguish planned from heard

The assistant may have generated an entire paragraph even though the user heard only its first sentence. Appending the whole paragraph to conversation history can make the next response assume knowledge the user never received. I would track distinct states for generated text, synthesized text, scheduled audio and the best available estimate of played content. Word-level alignment can help map progress back to text, but playback acknowledgement is not a perfect measurement of human perception. A conservative heard-prefix estimate, with explicit uncertainty at its end, is preferable to confidently treating every generated token as communicated.

Interruption can also occur after the assistant has performed a tool action. Stopping speech does not roll back that action, and deleting the entire response from history could hide a committed change from the next turn. Represent action outcomes separately from narration. The next response should know that the operation completed even if its explanation was cut off, and can clarify the outcome when relevant. This separation prevents the speech state machine from accidentally becoming a transaction protocol. Audio ownership determines what may play; the tool's own commit and cancellation rules determine what happened outside the conversation.

Interruption detection is a policy with false positives

Voice activity alone does not prove that the user intends to interrupt. Background speech, echo from the assistant's own output and a brief acknowledgement can all activate a detector. A system that cancels on every sound may feel as unresponsive as one that refuses to stop. I would separate the immediate acoustic signal from the policy that interprets it, and measure both missed interruptions and unnecessary stops. Depending on the application, a short provisional pause can buy time for classification, but it must have a clear recovery rule so the assistant does not oscillate indefinitely between speaking and listening.

There is no requirement that every environment use fully automatic interruption. A push-to-talk control or an explicit stop affordance can be the better contract in a noisy setting. If automatic detection is enabled, test with the actual microphone, speaker and echo-control arrangement the application expects. When a provisional interruption is dismissed, resuming should be a deliberate transition with a fresh playback decision. Do not simply reopen the gate for every stale chunk. The application must choose whether to continue from a known point, regenerate a shorter answer or wait for the user to clarify their intent.

Test hostile event orders before polishing the voice

The most revealing tests schedule inconvenient interleavings: interruption just before enqueue, interruption after scheduling, a cancellation timeout, a late chunk after the next response begins and a reconnect that delivers an old message. Assert that stale epochs cannot reach the active queue and that the new response remains playable. Then measure audible stop latency in an integration test, because a correct queue model cannot prove the behaviour of the device buffer. Keep these two kinds of evidence separate. The state machine establishes an ownership invariant; the playback measurement establishes whether the complete implementation meets the conversational budget.

My design priority is a small set of understandable transitions with explicit ownership. Speaking can become interrupted; interrupted work stays obsolete; a new response receives a new identity; committed actions retain their own history. That structure makes failures easier to reason about than a collection of cancellation flags spread across callbacks. Natural conversation depends on timing, but good timing starts with correct state. A polished voice cannot compensate for a pipeline that occasionally resurrects an abandoned sentence, and a clear interruption protocol is what prevents that resurrection from being a normal consequence of asynchronous delivery.