·14 min read
A Pause Is Not the End of an Answer
Voice agents end your turn when you go quiet. In an interview, quiet is where the answer gets built. Let the person end the turn, then engineer the silence that follows.
AI / LLM / Voice / WebSockets / Streaming / Architecture
Most voice agents decide you have finished speaking when you stop making sound. Voice activity detection notices the silence, a timer runs out, usually in well under a second, and the system takes its turn. For booking a table, that is what you want. For answering "how would you shard this table?", it is a disaster, because the silence is where the answer is being built.
I built a mock interviewer that runs live voice rounds, and one of the decisions in its design document, written before any code, was that silence never ends a turn. The candidate taps Done. Nearly everything else in the voice pipeline follows from that choice: where the latency clock starts, why questions are spoken a sentence at a time, why some sentences are checked before anyone hears them, and what has to survive a dropped connection.
Silence is not a turn boundary
Endpointing is the part of a speech pipeline that declares an utterance finished. The usual version is a silence threshold: once the speaker has been quiet for long enough, the turn is over. It is tuned for conversation, where people reply to each other quickly and a gap really does mean someone is waiting for you.
In an interview, a gap usually means the opposite. "Let me think about the write path first" is followed by four seconds of nothing, and then the actual answer. With a conversational threshold, the interviewer jumps in during those four seconds, the candidate loses the thread, and whatever gets graded afterwards is partly a measurement of the system's impatience.
Raising the threshold does not fix it. Any fixed value is wrong for somebody: short enough to feel responsive is short enough to cut off a slow thinker, and long enough for the slow thinker leaves everyone else sitting in silence after they have plainly finished. Acoustically, the pause in the middle of an answer and the pause at the end of one are the same pause. Turn detectors that read the words as well as the silence do better in ordinary conversation, where "and then" is obviously unfinished. They help less here, because "I'd hash the user id." is a complete sentence that might be a complete answer or the first line of a five-minute one.
The only party who knows whether an answer is finished is the person giving it, so they say so. It costs one tap per answer. In return, nobody is cut off, and the end of a turn becomes an event rather than a guess. The Done message carries the sequence number of the last audio chunk the app sent, so the server waits for exactly that chunk (a second at most), asks the transcription service to finalise, and takes the result of that finalise as the answer. The provider's endpointing plays no part. If Done arrives with nothing heard, the interviewer says it did not catch that and the turn stays open, so even the failure case leaves the decision with the candidate.
The same respect for pauses carries into the analytics. Words per minute comes from the word timings the transcription returns, with every pause of two seconds or more taken out of the speaking time, so thinking does not count against your pace. Long pauses are reported separately, as what they are. Filler words get context rules: "um" always counts, "like" only as a discourse marker, so "I like Python" is not a filler, and "kind of" only after a hedge, so "a kind of cache" is not one either. These numbers feed the grader for communication and confidence, and nothing else.
The gap after Done is a budget
An explicit end of turn gives the pipeline a clean starting line. The number the candidate experiences is the time from tapping Done to the first byte of the interviewer's voice, and because that interval has a known start, it can be split into stages with a target each:
| Stage | Target p50 | Target p95 |
|---|---|---|
| Last chunk arrives, transcript finalised | 300 ms | 600 ms |
| Quick assessment of the answer (small model) | 600 ms | 1,100 ms |
| Next question, to its first complete sentence (strong model, streamed) | 500 ms | 900 ms |
| Speech synthesis, time to first byte | 150 ms | 350 ms |
| Server overhead: event bus, framing, ordering | under 200 ms | under 400 ms |
| Total | about 1.8 s | about 3.3 s |
Each row names where it is measured, the total is logged on every turn as a single field, and a smoke script measures the same interval from the client's side of the socket.
To be precise about what that table is: these are targets. The only numbers measured so far ran against fake providers, so they measure the pipeline's own overhead and nothing else: a median of 182 ms and a worst case of 864 ms across four turns from the client, and 100 to 150 ms on the server's log. That is useful (the plumbing is not where the time will go) and it is not a latency claim. Real figures need a run against the real providers, with the same script.
A per-stage budget is most useful on the day the total is over. One number tells you something is slow. Five rows tell you which, and the response can be decided in advance, while nobody is under pressure. Mine, in the order I would pull them: play a short pre-synthesised acknowledgement ("Okay.", "Thanks.") the moment Done lands, which costs cached audio and no model call; merge the quick assessment and the question into one call, trading a round trip for a larger prompt; lower the effort, or the model, for follow-up questions only; and run the service closer to the providers' regions.
Meanwhile the server tells the app that the interviewer is thinking, and at which stage: evaluating the answer, then composing the question. Silence on the interviewer's side is labelled, so the candidate never mistakes it for a dropped call.
Take the slow work off the critical path
Two things want a model after every answer: deciding what to ask next, and grading the answer properly. Only the first stands between Done and the interviewer's voice, so the two are split.
The quick assessment is a small, fast model returning a short structured verdict: whether the candidate answered the question, a rough quality, the gaps, whether a follow-up is warranted and on what, and whether the answer looks like an attempt to steer the interviewer. It does not choose the next question. Code does, from time remaining, follow-up caps and topic coverage, which is the same split as ranking: the model supplies the judgements and the decision is code you own. If the call fails, the verdict defaults to neutral (answered, adequate, no follow-up), the fallback a scoring step should declare, and the interview carries on instead of stalling on a grading error.
The full rubric scoring uses the strong model, writes long output (a better version of the answer is part of the feedback), and runs as a background job. The cost is that the scorecard arrives a few seconds after the session ends rather than instantly. Nobody is waiting on the scorecard in the middle of an interview. Everyone is waiting on the next question.
The general rule: sort every model call by who is waiting for it. A call with a person on the other side of a silence gets the small model, a short output and a fallback. Everything else goes to a queue.
Speak the first sentence while the model writes the second
The next question is generated as a stream. Waiting for the whole question before synthesising any of it would add the entire generation time to the gap, so the pipeline speaks as it goes: deltas accumulate in a buffer, and each time a full sentence is available it goes to speech synthesis straight away.
Sentences rather than tokens, because a sentence is the smallest piece a synthesiser can say naturally. It needs to know whether it is reading a statement or a question before it starts, and a fragment does not tell it. The boundary rule is a terminator followed by whitespace, and the whitespace matters in a stream: "3." might be followed by "5" in the next delta, and waiting for the space means a decimal or a version number is never cut in half.
SENTENCE_END = re.compile(r"(.+?[.?!])(?=\s)", re.DOTALL)
def on_delta(turn, delta):
if not turn.early_ok:
return # early speech already stopped for this turn
turn.buffer += delta
while (match := SENTENCE_END.match(turn.buffer)):
turn.buffer = turn.buffer[match.end():].lstrip()
sentence = clean_spoken_text(match.group(1))
if leaks_private_context(sentence, plan_objectives):
turn.early_ok = False # stop; only the checked final text is spoken
return
turn.spoken.append(sentence)
enqueue(turn.id, sentence) # synthesis starts now, in parallel
Every sentence starts synthesising the moment it is known, and a single publisher sends the audio out strictly in order. Synthesis is parallel, so by the time the first sentence finishes playing, the second is usually ready. Publishing is serial, so a slow first sentence can never let the second one overtake it. The audio is cached for a day, keyed by provider, voice and text, which makes repeated questions, and the closing lines every session shares, cost nothing.
The candidate can also interrupt. If they start answering while the interviewer is still talking, the app stops playback and tells the server, which cancels the rest of that turn's speech, including synthesis already in flight.
You cannot unsay a sentence
Streaming text to a screen is forgiving. A filter can hold back a paragraph, and the final text can replace the draft without the reader noticing. Streaming speech has no undo. Once the first sentence is in someone's ears, it has been said, so whatever checks the output has to happen per sentence, before synthesis, not on the finished response.
An interviewer has something specific it must never say: its own plan. Each topic in a session carries a private objective written for the interviewer, such as "Probe reasoning about concurrency, latency, and throughput." Read aloud, that tells the candidate what is being assessed. So every early sentence is cleaned (plain text, no markdown, bounded length) and checked against the plan's objectives and the prompt's own scaffolding. The check catches verbatim copies, and also paraphrases that reuse a five-word run or most of an objective's words.
On any doubt, early speech stops for that turn. The engine checks the complete question the same way and regenerates it if it leaks, the retry streams nothing early, and only the checked final text is spoken.
The final text raises a second problem: the sentences already spoken are now part of it. So when it arrives, the pipeline compares it with what has been said (lowercased, letters and digits only) and speaks only the remainder. If the final text has diverged from what was already spoken, it speaks nothing more, and the full question is on screen anyway. A voice that stops a sentence short is a small flaw. A voice that repeats itself, or contradicts what it said a second ago, is the thing people remember.
The connection drops, the interview does not
The reconnection gap has a standard answer, and this pipeline uses it unchanged: every replayable event carries a monotonic sequence number, the last 500 per session sit in a Redis stream for two hours, and a reconnecting client sends the last number it saw and gets an exact replay, or a fresh snapshot of session state when the gap reaches further back than what was kept. Voice adds three problems that a text product does not have.
A half-finished answer. If the connection drops while the candidate is mid-answer, the server finalises the open transcription stream and keeps what was said so far for an hour. On reconnect, the app shows the partial answer, and the next stretch of audio continues it rather than starting again. Asking someone to repeat two minutes of an answer because their train went into a tunnel is a reliable way to end a practice session for good.
Audio that must not replay. Events replay. Audio does not. Replaying a missed question's audio after a reconnect would have the interviewer suddenly talking over whatever the candidate is doing now. Audio is live only, and a candidate who missed it taps repeat (up to three times per question), which the cache makes free.
A crash or a deploy in the middle of a step. The interview itself is a LangGraph graph that pauses at an interrupt while it waits for the candidate, checkpointed in Postgres under the session id. A new process picks the graph up at its last checkpoint and finishes the step that was in flight.
The socket is the most fragile part of the system, so nothing that matters lives only on the socket.
Testing without a microphone
Every provider sits behind a small interface (one for the language model, one for transcription, one for synthesis) with a deterministic fake chosen by configuration. That much is ordinary advice. What makes it work for voice is that the fakes model time accurately.
The fake transcriber "hears" the next scripted transcript, but spreads its words across the duration of the audio it actually received (16 kHz, 16-bit mono, so 32,000 bytes a second). Pace and pause metrics therefore behave in tests the way they do in production, and a script entry can pin exact word timings when a test needs a particular pause. Its default transcript even contains an "um". The fake synthesiser returns placeholder bytes shaped like an MP3 stream, in two chunks, and can be told to fail its next call, which is how the outage path gets tested: the captions carry the question, and the turn is marked degraded instead of failing.
With those in place, the integration suite drives real WebSocket sessions, and its test names read like the specification of the voice pipeline: a voice round trip with speech analytics; sentences spoken as they stream, without repeats; reconnecting mid-answer keeps what was said; reconnecting replays missed events; trimmed history gets a snapshot; a new runner resumes from a checkpoint; repeat replays from cache; a speech outage degrades to text. None of them needs an API key or a microphone.
This is the deterministic layer of an AI product, and in a voice product it is most of the code. How good the questions are needs evals. Whether an answer survives a tunnel needs tests, and fakes that respect time are what make them possible.
When text is the better interface
Voice is not free. Transcription is billed by the minute of audio, and in my cost model it is the largest single lever on a voice session: the working target was under $0.40 for a 30-minute session in voice, against under $0.15 in text. Those are targets as well, not measurements.
So text runs on the same socket and the same engine. The app sends the answer as text instead of audio, and a session can switch between the two mid-interview. If voice stops being worth what it costs, it is a mode to turn off, not a rewrite. Voice is the wrong default when:
- The answer is code. The coding round in the same product is typed, in an editor. Nobody should have to dictate a for loop.
- The person cannot talk where they are. An open office, a train, a shared room. A voice-only product loses them; a voice-first one with a text mode keeps them.
- Speaking is not accessible to them. For some people, captions and typed answers are the whole product.
- How it is said carries no information. Voice is worth paying for here because delivery is part of what an interview assesses, which is why pace, fillers and pauses feed the communication score. A product that throws that signal away is paying for transcription to get text the user could have typed.
- A provider is down. This one is not a choice. When transcription is unavailable, the candidate is told to type; when synthesis is unavailable, the captions carry the question. Voice should degrade to text, never to nothing.
The short version
Silence-based endpointing is tuned for conversation, and an interview answer is not a conversation: the pause in the middle sounds the same as the pause at the end. Let the person end the turn. That makes the gap after Done measurable, so give each stage a target, say where it is measured, label targets as targets, and choose your levers before you need them. Keep only the work someone is waiting for on the critical path: a small model's verdict and a decision in code now, the full grading later. Speak the question a sentence at a time, synthesising in parallel and publishing in order, and check each sentence before it is spoken, because speech has no undo and nothing should be said twice. Keep everything that matters off the socket: partial answers, sequenced events and graph checkpoints survive a tunnel, while audio is live only and repeats come from the cache. Test it with fakes that respect time. The pause belongs to the candidate. The gap after Done is yours to engineer.