Shaurya Singh

← writing

Nothing Is True Until It Reaches the Ear

A voice agent runs a send clock and a hear clock. Every property that matters is defined on the second one — including a fix I shipped and rolled back.

reverted · the pacing fix I shipped was rolled back before the next call — measurement pointed the other way, and barge-in is still unconfirmed by ear

reverted · the pacing fix I shipped was rolled back before the next call — measurement pointed the other way, and barge-in is still unconfirmed by ear

This corpus has already used “two clocks” three times, for three different splits. One piece means a sub-second budget beside a multi-second one. Another means the code under test beside the code doing the testing. The third is the site’s own standing rule about numbers: don’t derive a duration from two clocks that were never checked against each other. Hold onto that third one. It is the rule this essay breaks, and then gets punished for breaking.

A full-duplex voice agent runs a send clock and a hear clock. The send clock is what the process knows: bytes handed to the carrier’s media stream, position in the utterance it is producing. The hear clock is what the human has received through a speaker, at their ear, in air. Every property anyone cares about is defined on the second clock. Where the interruption landed. What the caller had heard when they spoke over you. Whether the disclosure was disclosed. Not one of those is a fact about what the process sent. All of them are facts about what arrived.

And the agent does not observe the hear clock. It has no sensor on it. It infers it, and the inference is only as good as the pacing assumption underneath — which is a thing I can say with some authority, because I got that assumption wrong in both directions inside three weeks.

six lines of code, and a story that fit

The complaint was plain: the audio broke up badly on the first call the line had taken in months, and the caller hung up after about fifteen seconds. Reading the send loop afterward, the arithmetic looked like a confession. Two lines, next to each other:

chunk_size = 320      # 40ms at 8000hz mulaw
...
time.sleep(0.02)      # 20ms

Eight-bit mulaw at 8 kHz is 8,000 bytes per second of audio. So 320 bytes is exactly 40 milliseconds of sound, and the loop shipped one of those every 20 milliseconds. On paper it sent audio at twice real time, forever, for the whole length of every reply.

The carrier does not drop the surplus. It buffers it and plays it out at the only speed a human ear accepts. So on paper the playback buffer ran ahead of the caller by an amount that grew for as long as the agent kept talking. Six seconds of reply is fully sent after three seconds of wall clock; at that instant the process believes it is finished speaking and the caller has heard half of it. The frontier lag peaks at T/2 for a T-second utterance, at the moment sending ends.

That T/2 is division, not measurement — a number derived from two constants in a loop, where the loop is one clock and the other is a carrier’s playback scheduler I have never instrumented. What I had written down at the time was the qualitative version: the buffer “ran seconds ahead of what the caller actually heard.” I want that flagged here rather than at the end, because the rest of this essay is what happened to the story built on top of it.

The fix was one character.

time.sleep(0.04)      # 320 bytes = 40ms, sent at its true duration

Pace the send clock to the real duration of what you are sending, and the buffer stays shallow. I shipped it. I wrote in the log that this was the root cause and that both symptoms matched it exactly.

and then the measurement went the other way

It was reverted to 0.02 before the next call, on the grounds that it had likely made things worse. That was the estate’s own verdict on my own change, written in the same session that produced the diagnosis: patching blind is how you fix the wrong bug and believe you succeeded, and this session already has an example of it.

Then telemetry landed — bytes, duration, drift and interrupted, logged per turn — and the next call measured the loop directly. The number was ~134ms per send, against a chunk that is 40ms of audio, with drift = +1.59s on 0.56s of audio. That is not a runaway buffer. That is an underrun. Each chunk took 134 milliseconds of wall clock to get out the door: the send loop was not racing ahead of the ear, it was falling seconds behind it. The direction of my entire story was wrong.

Three candidate causes are live and none is a sleep constant. If per-send drops to single-digit milliseconds after disabling Nagle’s algorithm, the loop was waiting on small-packet coalescing. If it is high only on the first chunk, it is TLS warm-up, absorbed by larger 1600-byte chunks. If it stays high throughout, it is neither, and the escalation is replacing the development web server still serving the live line.

Worth naming what else got ruled out, because it is the alternative a skeptic reaches for first and it was settled by measurement, not argument. The standing hypothesis before the pacing story was self-interruption: barge-in fires whenever a final transcript longer than five characters arrives while the agent is speaking, so if the agent’s own output bled back through the line it would cut itself off repeatedly and present as break-up. A classifier was written to separate echo from a genuine interjection and validated against four controls — exact self-echo 1.00/1.00, a mangled partial echo 0.74/1.00, a genuine interjection 0.19/0.00, a new topic clean. Run against a real call transcript, the barge-in in question scored similarity 0.30, word-overlap 0.25: genuine. It was deployed in shadow mode — measuring, not acting — because enabling a guard on a hypothesis the evidence already leaned against would have made the next call uninterpretable.

So the record is: one plausible send-side story, shipped as a fix, rolled back. One plausible echo-side story, largely ruled out without another patch. One measured number that overturned both. That is this essay’s thesis happening to its author, and I would rather print it than the version where I was right.

buffer depth is a dial, and I have now been wrong at both ends

The naive reading of the original bug is that it was a typo and the lesson is to check your constants. The deeper reading is that buffer depth is a dial, and both ends of it are defensible.

A deep buffer buys smoothness. Jitter on the network, a slow synthesis chunk, a GC pause — none of it reaches the caller as a gap, because seconds of audio are already queued in front of the hole. What a deep buffer costs is interruption accuracy, roughly one-for-one: the deeper the buffer, the further the send clock runs ahead of the hear clock, and the wider the window in which the agent’s model of the conversation describes a call that has not happened yet. Only roughly, because clear is not instantaneous and the interrupt detector adds its own latency — the sibling system budgets barge-in cancel at under 150ms as a separate line item for exactly that reason.

And here is what I did not understand when I shipped the one-character fix. Pacing to exactly 1x is not a safe middle. It removes the absorber entirely: every scheduling hiccup in the send path lands directly on the ear, because nothing is queued in front of it. That is not a hypothetical horn of the trade-off — it is the 134ms-per-send measurement, which is what the other end of this dial sounds like. I prescribed the setting whose failure mode I then measured.

There is no value that gets both. You pick a point on the dial knowing one cost is loud in testing and the other is silent. Everyone notices a gap. Almost nobody notices that their interrupt indices are stale — and in my case the drift was printing, in the log line, on every turn. It printed and nobody read it, because it had no target, no threshold and no alert. A metric with no threshold is a metric nobody notices.

one symptom, two inferred mechanisms, one constructed one

I should be precise about what was observed and what was reasoned, because the original write-up was not. There was one symptom: the audio broke up. There were two consequences I derived from reading the loop, both of which matched that symptom, neither of which was independently observed.

Interrupt index. On barge-in, the agent sends a clear event to the media stream. With a deep buffer, clear discards a multi-second backlog mid-word — heard as a heavy, abrupt break-off rather than a clean yield. A guillotine, not a stop.

Context window. Quieter and worse: if playback position and send position have diverged, the interruption never lands where the caller actually spoke. The agent marks its transcript at the send frontier and carries forward a context in which it said several seconds of things the caller never heard. Nothing crashes. The call just stops being about the same conversation the human is in.

There is a third property I want to label honestly, because I have not seen it happen. A required disclosure sitting in a playback buffer when the caller barges in and the agent sends clear was produced, logged, and never delivered. The transcript says it was said. The caller can tell you it was not. I have no incident where that occurred; the arithmetic says it can, and that is the whole claim. Constructed, not observed.

What I cannot claim is the tidy ending the first draft of this had — fix the distance and all three fix at once. No such simultaneous fix was ever observed. The change was rolled back, the audio complaint survived it, the third property is constructed, and as the next section explains, it lives in a different codebase entirely.

There is a static cousin of this already in the corpus: a test fixture pinned to a source-side value that had to be re-derived in delivered terms instead of produced terms. Same word, same shape — but that was a constant someone typed once, sitting still, waiting to be re-pinned. This one is live, time-varying, and set by a number I chose.

the same law, found independently in a build that shares no code

Here is the part that makes the doctrine survive the rollback.

A different agent, in a different lane, on a rented realtime engine — a separate runtime, no shared code with the loop above, no shared audio path — had a consent gate that decided whether a required disclosure had been given before the call could proceed. Version one checked the intent of the first utterance: did the agent set out to say the disclosing thing? That is a question about generation. It passed every test written against it, because in those tests generation and delivery were the same event. (This is the piece that closes a gap another essay in this corpus named and left open: a disclosure gate before content reaches the line.)

An adversarial wave went at it — seven agents, control probe passed. Twelve distinct bypasses landed. Why the control probe matters is a separate essay’s argument, not this one’s.

Twelve is not a tuning problem. Twelve means the gate was scoring the wrong variable, and once you see it, every bypass is the same bypass: get the agent to produce something disclosure-shaped, then prevent it from arriving. Barge in during it. Get it truncated. Get the intent satisfied by an utterance that started right and ended somewhere else. A gate defined on production has an attack surface as wide as the set of ways audio can fail to land, and that set is not enumerable in advance.

Version two grants only when a core-satisfying utterance finishes playing to the ear. Not intended. Not generated. Not sent. Finished arriving. Two supporting rules came with it: the server computes verdicts from delivered transcripts on the live path and refuses client-asserted consent outright — a client reporting that it played something is a client reporting on its own send clock, now on a machine I do not control and with an incentive to be wrong. And barge-in carries transcript-truncation sync, so stored context is cut where the audio was cut, not where generation stopped.

Two builds. No shared code, no shared audio path, no shared sovereign. The same distinction surfaced in both. That is convergent evidence, and it is worth more than a smooth transition would have been.

the rule, stated mechanically

Compute every verdict server-side, from delivered transcripts, at the single throat all speech passes through. The single throat is what makes “delivered” definable: if audio can reach the caller through two paths, one of them will not be instrumented, and the ungated path is the whole bypass.

The narrower claim, and the one I actually learned: carry send-minus-play as a targeted quantity, not as an assumption. This is less novel than I first wrote it. The sibling system already ships an SLO sheet — barge-in cancel under 150ms, disclosure fire rate 100%, an interruption-handled-cleanly rate, all logged per turn. What is missing is specific: the ear-side lag itself has no target, even in the system where it is already printed on every turn. Drift is in the log line. It has no p95, no threshold, no alert, and no owner. That is why a constant I was confident about could be wrong in one direction, get “fixed,” be wrong in the other, and stay undetected across two calls with the instrument running the whole time.

what I don’t know yet

I have never confirmed barge-in by ear on a live call. The code path is there — it resets the speaking flag and sends clear — and it has never been tested by a human deliberately talking over the agent and reporting what happened. That check is still owed.

I do not know which of the three candidate causes of the underrun is the real one. Nagle, TLS warm-up, or the dev server: the next call is designed to discriminate between them by reading per-send off the log, and it has not been placed. Until it is, the current state of the audio bug is UNKNOWN, not fixed.

I do not know whether the T/2 arithmetic ever described reality. It is derived from two constants in code and has never been checked against a measurement of playback, and the one measured frontier number I have points the opposite way. I left the derivation in because the shape of the argument is what the dial section rests on, but the magnitude is not a measurement and I will not let it read as one.

The transcript-truncation sync is designed, not measured. I know what it should do on barge-in. I have no case where I cut a call mid-utterance, pulled the stored context, and confirmed it ends at the same word the caller stopped hearing.

And on the twelve bypasses: version two does have evidence, and it would be dishonest to omit it in the name of modesty. A 100% disclosure fire rate is proven across 25 varied sessions plus 2 negative controls, and the refuter’s own anti-disclosure strings are pinned as permanent test regressions. What that is not is a second seven-agent adversarial wave. “Should close all of them by construction” is the same sentence I would have written about version one, and the count of bypasses an equivalently sized wave would find against version two is UNKNOWN, because that wave has not been run.

— Sent Is Not Heard

← back to writing