The voice agent I run splits every turn across two models: a fast one that decides what is happening and which tool to reach for, and a slower, better one that decides what to say. That split is usually explained as a cost cut. It isn’t. A phone call has a latency budget a chat window doesn’t — silence past a second or two reads as broken, not thoughtful — and the split is the only way I have found to keep the turn inside it.
Which means the dispatcher is the part that has to be right about time, and the way dispatchers get time wrong is not by being slow. It is by holding a number nobody measured. The filler threshold in that agent had been set to 2500ms because it felt reasonable. When someone finally pulled 173 real turns and computed the median, the threshold turned out to sit below the p50 — so the “let me check” that was supposed to rescue the rare slow turn was firing on roughly 95% of them. It had become a verbal tic. The number was never false. It had just never been checked against the thing it was supposed to bound.
reflexline is that discipline narrowed to one mechanical job: pick the fastest handler whose predicted cost was actually, recently checked.
The mechanism
A Band is a latency tier — a name, a cumulative millisecond ceiling measured from the start of the turn, a handler, and a cost_estimator that is mandatory. The constructor rejects a band built without one. A dispatcher that lets a handler skip declaring how expensive it thinks it is has already given up before the first request.
route() walks the bands tightest-ceiling-first and skips any band whose estimator returned None, whose estimate is older than the router’s TTL, whose prediction doesn’t fit the ceiling, or whose deadline has already gone by. Each skip is recorded with its reason:
shaurya@sps ~ % python examples/demo.py
2. Stale estimates are treated as UNKNOWN, not guessed
'fast' band had a cheap (10ms) estimate -- but it was 5s old against a 1s ttl.
skipped bands: [('fast', 'stale')]
routed to 'slow' instead: [slow-path] hello
5. When every band is UNKNOWN or over budget, routing refuses to guess
NoEligibleBandError (expected): No band could accept this input:
only_band (unknown: cost_estimator returned no estimate)
That last one is the whole posture. A router with no eligible band raises instead of picking the one it likes best, because a fallback chosen without a measurement is the 2500ms threshold again wearing different clothes.
There is a second gate that has nothing to do with latency and everything to do with who gets the last word. VerbosityGate caps a handler’s output after generation, regardless of what the prompt or persona layer asked for. The demo’s persona writes 58 words on purpose and ships 18, because the cap lives in code and the persona has no vote.
A routing policy is a thing you can regress
Routing rules drift the way prompts drift, and they drift silently, because a request that went to the wrong band still returns an answer. So the eval-bank idea gets pointed at dispatch policy instead of model output: a fixed JSONL of (input, expected_band, expected_output_shape) cases that all have to pass before a router change gets to call itself ready.
shaurya@sps ~ % reflexline evalbank check examples/example_router.py examples/cases.jsonl
faq_ping: PASS
faq_hours_case_insensitive: PASS
short_query_goes_to_dispatch: PASS
oversized_query_falls_back_to_respond: PASS
unknown_short_query_still_dispatch: PASS
5/5 cases passed.
READY: all regression cases passed.
Exit 0 when every case passes, 1 when any regressed with a per-case diff, 2 on a usage error. A harness that cannot tell “nothing regressed” from “I could not run” will eventually hand CI an all-clear it didn’t earn.
What the router refuses to know about itself
The package enforces that a measurement was taken and wasn’t stale. It does not close the loop, and three things I measured today say so more precisely than the README does.
The router never times the handler it just ran. I gave a band an estimator promising 5ms and a handler that takes 300ms, and the router dispatched, returned, and recorded nothing:
shaurya@sps ~ % python
>>> res = router.route("x", budget)
>>> res.predicted_ms, round(budget.elapsed_ms(), 1)
(5, 300.1)
>>> res.__dataclass_fields__.keys()
dict_keys(['band', 'output', 'elapsed_ms_at_dispatch', 'predicted_ms', 'skipped'])
There is no field for what it actually cost. A systematically lying estimator is never contradicted by the router’s own data, which means the one instrument positioned to catch a bad estimate declines to look.
Elapsed time is also read once, at the top of route(), before any estimator runs. With three bands whose estimators each take 50ms, the dispatch happened with elapsed_ms_at_dispatch = 0.0 recorded while 100.3ms had genuinely passed inside the estimators. The audit trail is honest about which bands were skipped and wrong about when.
And TimeBudget.total_ms never enters the routing decision at all — route() only ever compares against each band’s own ceiling. A band declaring a 9000ms ceiling, predicted at 4000ms, dispatched cleanly inside a budget whose whole total was 200ms.
What it can’t do yet
- It never measures a handler. No timing around
band.handler(input), no field onRouteResultfor actual cost, no feedback into the estimator. A package about measuring before routing does not itself measure the thing it routed. total_msis decorative during routing. The turn budget is never consulted; only per-band ceilings are. A band whose ceiling exceeds the entire budget is fully eligible, so you can blow a 200ms turn by 4 seconds without the router registering an objection.- Elapsed is sampled once, before the estimators. Every skip decision and the dispatch itself are judged against a clock reading taken before any estimator was called, so expensive estimators are invisible to the very budget they are being consulted about.
- Nothing stops a handler that overruns. The deadline is checked before dispatch and never again. There is no timeout, no cancellation, no interruption — a handler that ignores its prediction runs to completion.
test_adversarial.pyis not run bypytest. It sits at the package root,pyproject.tomlsetstestpaths = ["tests"], and the default run collects 95 tests fromtests/and never sees it. Its three cases pass when invoked by name. A test file outside the collected path reports the same all-clear as a test file that found nothing.- The bundled estimators are stand-ins, and the README says so. “5 + 2ms per character” and a flat 900ms guess exist to make the demo runnable. They are precisely the kind of felt-reasonable number the 2500ms threshold was.
VerbosityGatecounts Unicode code points, not grapheme clusters. It will never split a code point, so emoji and precomposed characters are safe, but a base character followed by a separate combining mark can be cut between the two.
The suite stands at ninety-five tests today, covering band selection across budget-and-estimate combinations, the exact TTL boundary where a fresh estimate becomes UNKNOWN, eval-bank regression detection, the CLI’s three exit codes, and the verbosity gate’s truncation edges. They pass in a quarter of a second as of today, which is a statement about the code in front of me and not a forecast about the next change to it.
— Shaurya