Voice AI should prepare before you stop talking

A silence threshold is only one observation

A voice interface needs an estimate of conversational readiness, not just a timer that expires after the microphone becomes quiet. A pause can separate turns, divide a sentence, or mean that the speaker is searching for a word. The same duration can mean different things depending on what has already been said. I would therefore treat acoustic silence as an observation that updates a turn hypothesis. It should not, by itself, authorize a response that assumes the request is complete.

The Moshi paper provides a useful point of comparison: it models user and system speech as parallel streams rather than requiring a conventional sequence of explicit turns. That is a research result about a particular speech model and architecture, not evidence for my proposed implementation. My argument here is narrower. Even a modular system with separate recognition, language generation and synthesis can stop making every stage wait for a final transcript. It can overlap reversible preparation while preserving a separate decision about when speaking becomes appropriate.

References: [1] Moshi: a speech-text foundation model for real-time dialogue

Give partial interpretations a version

Imagine hearing: find the cheapest train to London tomorrow, followed by actually, make that Friday. A provisional planner can discover destinations, normalize the date and prepare a search. It must not regard its first date interpretation as permanent. I would attach a monotonically increasing revision to each interpretation, and make every prepared result declare the revision and dependencies it used. A later transcript does not necessarily invalidate everything. The destination may remain useful while the date-dependent query becomes obsolete.

This distinction also prevents a common category error around recognition metadata. Google’s streaming speech documentation distinguishes stability, which concerns whether an interim result will change, from confidence in recognition correctness. A stable phrase can still be misunderstood, and an accurate partial phrase can still be unfinished. Neither signal alone measures permission to speak or act. The application needs its own state describing what is known, what remains provisional, and which unresolved fields matter for the proposed response. That state should survive beyond a temporary text string.

References: [2] Google Cloud Speech-to-Text: streaming recognition responses

Separate preparation from commitment

I would divide the response path into four states: interpreting, preparing, ready and committed. Preparing may include retrieval, a tentative plan and synthesis into a private audio buffer. Ready means the candidate has passed current checks and could be released if the conversation permits it. Committed means some externally visible effect has occurred. A spoken syllable cannot be retracted from the listener, even if the remaining audio can be stopped. This makes playback a meaningful commit boundary rather than an incidental output operation.

The following sketch illustrates the guard, not a complete turn detector. A candidate must still match the current interpretation, satisfy its dependencies and survive the turn policy immediately before release. A cancellation signal alone is insufficient because cancellation races with completion. A worker may finish after it was told to stop. Checking the revision at the output boundary closes that particular race, provided the check and release are serialized by the same owner of playback state. It does not solve recognition errors, authorization or transport buffering.

type Candidate = { revision: number; audio: Uint8Array };
function commit(candidate: Candidate) {
  if (candidate.revision !== currentRevision) return;
  if (!dependenciesSatisfied() || !turnPolicyAllowsSpeech()) return;
  playback.enqueue(candidate.audio);
}

Price the work that gets thrown away

Speculation moves work earlier; it does not make that work free. Consider an illustrative preparation stage costing 120 milliseconds and one unit of compute. Suppose it finishes before the turn ends, survives revisions 70 percent of the time, and therefore saves 120 milliseconds on those successful occasions. Across many turns, the simple expected latency saving is 84 milliseconds per turn. The unused preparation costs 0.3 compute units per turn. These invented values describe the accounting method, not a measured voice system.

That calculation omits congestion, which can reverse the result. If speculative jobs delay committed responses on the same accelerator, a locally useful guess can make the service slower overall. I would reserve a small speculative budget and prefer work with high reuse: fetching a stable document may survive a wording change, whereas synthesizing a long answer rarely does. Admission should depend on current load and remaining conversational uncertainty. A candidate that is cheap to produce but expensive to cancel may also be a poor speculation target.

Cancellation is an end-to-end operation

Stopping the language model is only part of stopping speech. Text may already be queued for synthesis, audio may be waiting in an application buffer, and encoded packets may have reached the device. Each stage needs the response identifier and a way to reject obsolete chunks. I would track the last generated, transmitted and played offsets separately. Otherwise the system can record an entire answer as spoken when the listener heard only its first sentence, causing the next response to assume knowledge that never arrived.

A useful interruption test is deliberately adversarial: begin playback, interrupt during a sentence, inject a late chunk from the cancelled response, and verify that it never resumes. Repeat the test around connection loss and reconnection. The acceptance criterion should concern the audible result and the subsequent conversation state, not merely whether a cancellation function returned successfully. A bounded tail of already buffered sound may be unavoidable on a particular client, but it should be measured and described as a product property rather than hidden inside a server latency chart.

Do not let speculation acquire authority

Reading a document and sending a message have different consequences. A speculative plan may prepare the message body, but it should not send it because a partial utterance sounded likely to authorize sending. I would give preparation workers read-only capabilities where possible. Any operation that changes external state should require a separately checked action proposal, complete arguments and whatever authorization the surrounding application requires. The conversational confidence score is not an access-control decision, and a fluent continuation is not evidence that consent occurred.

This also matters for privacy. A premature external lookup can disclose part of an unfinished request, even if no answer is spoken. Reversibility should therefore be evaluated from the user’s perspective, not solely from the database’s perspective. Local token generation may be discardable; an external query may already have crossed a boundary. The system should classify allowed speculative work before launching it. In a sensitive workflow the appropriate policy may permit local preparation while delaying every remote request until the speaker’s intent is sufficiently clear.

Measure conversational errors alongside speed

A useful evaluation set needs pauses within turns, corrections, overlapping speech, short acknowledgements and explicit interruptions. I would measure time to useful audible response, false starts, interruption recovery and wasted preparation together. A system that wins on first audio by emitting a generic acknowledgement on every pause has optimized a convenient proxy. The relevant question is whether the timing helped the conversation. Human judgments may be necessary, but the rubric should distinguish premature commitment from harmless preparation and from genuinely useful overlap.

Comparisons also need the same audio inputs and a defined playback environment. Network conditions, device buffering and synthesis chunk sizes can dominate an otherwise fast model. Report distributions rather than only a median, and separate turns requiring tools from turns answerable immediately. An interruption-heavy evaluation can expose cancellation defects that a clean question-and-answer benchmark never exercises. I would retain event traces with revisions and offsets so that a failure can be reconstructed without retaining more conversation content than the evaluation actually requires.

Sometimes waiting is the correct optimization

The strongest counterargument is that speculation introduces complexity exactly where a voice system already struggles: asynchronous state, uncertain transcripts and hard-to-reproduce timing. For a low-volume assistant with inexpensive, fast responses, the extra machinery may save little while creating many more races. I agree with that objection. A simple sequential design is an excellent baseline, and the burden of proof belongs to the speculative implementation. It should earn its complexity through measured improvements on the conversations the product actually serves.

I would begin with one bounded optimization: prepare a read-only response candidate from a stable prefix, retain strict revision checks, and discard it whenever uncertainty increases. Only then would I consider speculative synthesis or more aggressive overlap. The target is not a voice that always speaks sooner. It is a system that uses otherwise idle time without confusing possibility with commitment. Preparation can be eager while speech remains patient, and the ability to change course should be designed as carefully as the ability to answer.