shipped · production voice agent — eval bank green, full wire-level audit not yet run
The recruiting voice agent I run splits every call across two models. A fast one decides what’s happening and which tool to call. A slower, better one decides what to say. I built it that way to keep the token bill down. That’s the sentence I would have written a year ago, and it’s wrong — not false, just aimed at the wrong reason. The bill is a side effect. The reason the split exists is that it’s the only way I’ve found to keep a phone call from feeling like it’s talking to a website.
What the split actually buys
A phone call has a latency budget that a chat window doesn’t. Silence past a second or two reads as broken, not thoughtful. So every turn has two jobs that compete for the same clock: figure out what the caller needs — which of a ten-tool registry to reach for, which of roughly forty companies and about 5,900 candidates in the reasoning world this turn is even about — and then say something back that sounds like a person, not a form.
One model doing both jobs synchronously means paying the slow model’s latency tax on every single turn, including the trivial ones — “yes,” “what’s the address,” “can you repeat that.” Those don’t need judgment. They need speed. So the architecture is a small model in front doing triage — is this a tool call, a clarifying question, a handoff — and a larger model behind it composing the actual reply, invoked only once the triage step has narrowed what there is to say. The round trip across a live call lands at roughly three to six seconds. (There’s a filler-threshold bug in that number’s history — a wait-indicator that fired on almost every turn because its threshold had never been measured against real calls — and I’ve already written that incident up elsewhere in full; it doesn’t need repeating here, it’s the same latency budget this split is protecting.)
The cost saving is real. Triage-sized prompts run cheap through a fast model instead of expensive through a slow one on every turn. But cost isn’t why the split holds up as an architecture decision. A slow model given the same wall-clock budget for everything would either need to get faster — not available on demand — or the product would need to accept multi-second dead air on every turn, which is the one thing a phone call can’t absorb. The split isn’t cost engineering wearing a latency costume. It’s the other way around: it’s a latency decision that happens to be cheaper, and would still be the right call if it weren’t.
Here’s the test I use to tell the two framings apart: would I keep the split if the slow model got cheap enough to run on every turn for free? If the answer is yes, it was never about cost. And it is yes, because a single model still has to finish reasoning about which of ten tools to reach for, which company and candidate the turn is about, and what tone to answer in, all before it can start producing the words that go to speech synthesis. Collapsing two jobs into one model doesn’t remove either job. It just serializes them behind one slower model’s latency instead of running the cheap one first and only paying the expensive one’s cost on the turns that need judgment about phrasing, not routing.
The eval bank is what makes the split touchable
A dispatcher that mis-routes is worse than a plain slow agent, because it fails silently — it doesn’t crash, it just sends the wrong tool at the wrong candidate or the wrong company, and the call keeps going like nothing happened. That risk is exactly what you’d expect to make people afraid to ever touch the routing logic again once it works. The fix here wasn’t caution. It was a regression-locked eval bank: 34 fixed cases across the tool registry, run before any routing change ships, and a routing change that breaks even one of them doesn’t go out. The 34 is the count on the page I recorded it on; other pages in the same log record 33 and 39, and which of them describes the bank today is UNKNOWN.
shaurya@sps ~ % pnpm eval:dispatch
...
34/34 dispatch evals passed
0 regressions vs last-shipped routing table
That number is the whole point of building the bank at all. Not “the routing works” — routing can look fine in a demo and still be wrong on the case nobody thought to try. “34/34” is a claim you can falsify on every single change, which is a different, much stronger claim than “I tested it and it seemed fine.” The bank turns the dispatcher from a thing you’re afraid to edit into a thing you can edit fast, because the bank tells you within a minute whether the edit broke anything it used to get right.
I want to name the honest limit of that number, because the same corpus that produced this eval bank also produced the sharpest argument against trusting it blindly: a separate incident where live-call caller recognition was silently defeated in the deployed system while the eval suite stayed green the whole time. Passing evals proves the harness agrees with itself. It doesn’t prove the wired, deployed system does the same thing the harness exercised. I haven’t run that class of wire-level audit against the dispatch bank specifically — I don’t have a “34/34 evals green, and separately, a live call confirmed the same routing” pairing to point to yet. So the honest status is: the bank is real; the gate that reads its results was itself found fake-green in an audit on 2026-08-09 (14 of 39 cases crash-passed vacuously, and a deleted baseline had silently disabled regression detection) and fixed; and none of that is proof the live system matches it. Three different claims, and I only have evidence for the first two.
Brevity is a latency constraint, not a manners constraint
The responder side has its own failure mode, and it’s not accuracy — it’s length. A persona instruction told the model to be warm, to let a reply breathe, to go to three sentences when it earned it. Measured against real calls, that produced a median reply of 156 characters, with a tail running out to 492. Every one of those extra characters is extra time: text has to be generated, then synthesized to speech, then played out loud, before the caller can talk again. A verbose responder doesn’t just sound chatty. It adds wall-clock time to the same latency budget the dispatcher was built to protect, on the one turn where the dispatcher has already done its job and handed off.
The fix that held wasn’t a better version of the warm instruction. Persona wording is a request, and a request competes with every other instruction sharing the same prompt — it can lose. So the constraint moved out of the prompt and into the code path the reply has to pass through before it goes to synthesis: a hard cap, checked structurally, not requested politely.
// before: persona prompt only — "warmth," "let it breathe"
// after: a ceiling on the responder's output, enforced outside the persona
// illustrative: the shipped rule is a two-sentence cap. I have no artifact
// for this exact function, so read it as the shape and not as the source.
function enforceReplyBudget(text) {
const words = text.trim().split(/\s+/);
if (words.length > MAX_REPLY_WORDS) {
return truncateToSentenceBoundary(text, MAX_REPLY_WORDS);
}
return text;
}
Under a two-sentence cap — no preamble, at most two names a turn — median reply length dropped from 156 characters to a measured p50 of 25 words, with the outlier tail cut from a maximum of 492 characters down to 41 words. The gap between those two states is the gap between what the persona said and what the code let through — and until someone measured both sides of it, nobody knew the gap existed. A prompt asking a model to be brief is advisory. A cap checked on the output before it reaches synthesis is a fact about the system, and it’s a fact that happens to matter for the same reason the dispatcher split matters: fewer words is less synthesis time, and less synthesis time is latency budget given back to the part of the call that actually needs it.
Closer to instrumentation than to prompting
Put the three pieces together — a routing split chosen for a latency budget, a regression-locked eval bank gating changes to that routing, and a reply-length ceiling enforced in code after measuring the gap between persona and output — and what you get doesn’t look like how most LLM-agent products get built. Most of what I’ve seen ships as a prompt, a vibe check, and a launch. No fixed regression set. No p50/max pulled from real traffic before and after a change. No number that has to stay green for a change to go out at all.
This looks more like how you’d build a piece of a real-time system: instrument the tails, not just the average; lock a regression set so you can prove a change didn’t cost you something you already had; treat “it passed the eval” as a necessary claim, never a sufficient one. None of the individual pieces are exotic — model routing, regression tests, output caps are all standard engineering moves in other domains. What’s unusual is applying all three to a conversational agent, where the default culture still treats “the demo worked” as evidence.
None of this was designed as a system from day one. The dispatcher split, the eval bank, and the reply cap arrived as three separate fixes to three separate complaints — the call felt slow, an untested routing change was one bad edit away from silently misfiring, the agent wouldn’t shut up. They only look like one coherent latency architecture in hindsight, once you line them up next to each other and notice they’re all rationing the same three-to-six-second budget from three different angles. I’d like to claim I planned it that way. I didn’t. What I can claim is that I noticed the pattern after the fact and started treating “does this cost latency budget” as a standing question for every change to this agent since, which is a smaller, more honest claim than “I designed a latency architecture.”
What I don’t know yet
I don’t know how this split ages. The same corpus that produced it already has a newer design on paper — a full-duplex voice architecture built around a different provider’s realtime model — that names the current dispatcher-plus-responder setup as a fallback rung, not the target state, once sub-second interruptible voice is available on that path. I don’t have a timeline for when that supersedes this, and I’m not going to guess one.
I also don’t know whether the eval bank actually catches the routing mistakes that matter most, or only the ones someone thought to write down. Thirty-four fixed cases is thirty-four cases someone anticipated. The wire-level gap between “the harness agrees with itself” and “the deployed system does the same thing on a live call” is real, documented elsewhere in this corpus, and I haven’t closed it for the dispatch bank specifically — I’ve only named that it’s open. Until I run that audit, “34/34” tells me the routing logic hasn’t regressed against what I already thought to test. It doesn’t tell me the routing logic is right.
— Two Models, One Clock