Shaurya Singh

← writing

Silence Passes Every Check

A run scored 7 of 7 in 24 ms. Its entire reply said the backend was unreachable. Every check asked what the reply did not do, so silence passed all of them.

measured · 1,324 graded runs from one quality loop · the one-line fix is named, not shipped

measured · 1,324 graded runs, 189 of them non-answers · audit 2026-08-17 · fix specified, not shipped

Here is the best-scoring run in the log.

run 1283 · ms=24 · passed=7 failed=0
reply: "I can't reach my system at the moment — that's on my end, not you."

Seven checks out of seven. Twenty-four milliseconds. A perfect score on a turn where the system told the caller it could not reach its own backend and then stopped talking.

Nothing malfunctioned. The grader did what it was built to do, on a reply that was exactly what it should have been. The degrade line is correct behaviour during an outage — honest, fast, no invention. It scored perfect because every check in the rubric asks what the reply did not do.

the checks are fine; the conjunction is not

This is not the complaint that a check has never been shown it can fail. refuting-verifiers already makes that argument and I am not making it again — a passing report proves nothing until the process has been shown it can go red. These checks go red. The audit records 274 failures off this loop. They are the good kind.

The rubric asks, of each reply: did it invent a role that does not exist, name a person it should not have named, state a count it could not have known, misstate a status, confabulate, or come in over the latency budget? Every one of those is a real failure mode with a reachable red state. I can name six. The run above passed seven. What the seventh check measures is UNKNOWN to me — the audit does not enumerate it and I am not going to guess a check into existence to make a sentence come out even.

The defect is not in any member. It is in the conjunction, and the mechanical way to state it is about failure sets, not about wording. Every check has a set of replies it rejects. Union those six or seven sets and the empty reply is in none of them. That is a property you can test in an afternoon.

The grammar is downstream of it, and it is why the property is so easy to build by accident. Every check here is a prohibition — did not do X — and a set of prohibitions describes the acceptable region by subtraction. The reply that does nothing survives every cut. Another prohibition shrinks the region and leaves the trivial point where it was.

But that framing is a tendency, not the mechanism, and I would rather say so than have it said back to me. reply_is_substantive is rewritable as a prohibition — did not return a non-answer — and would then be a negative check the empty reply fails. Failure sets decide this; wording does not. What the wording explains is why real rubrics drift into this shape: you write prohibitions when your threat model is confabulation, and a threat model made of things the model might wrongly say never thinks to ask whether it said anything.

Either way it is a property of the set, not of any element, which is why reading each check and approving of it — which is what I would have done, and what someone did — finds nothing. The satisfying assignment is not hypothetical and nobody had to engineer it. Normal operation occupies it 189 times in this log.

one graded turn in seven contains no data claim

189 of 1,324 graded runs — 14.3% — are degrade lines. That is what turns this from a cute logic observation into a finding about the instrument. One in seven. Not a tail case, not a rare outage artifact. A seventh of the graded turns contain no claim about the data at all: no role, no count, no status, nothing to be right or wrong about. The grader scored all of them. Not one of them was scored on whether it answered.

This is a different animal from the failure care-does-not-scale names. There, absence was the pass value: cannot-measure rendering as measured-and-clean. Here nothing is missing. The reply is present, honest, and the right thing to say — a fully formed, correct output that is still not an answer, and the rubric has no vocabulary for that distinction because it was never asked to have one.

measured-or-unknown gave facts a third value so a guess could not render as a measurement. a-red-gate-has-two-diseases gave CI gates the same third value, so a gate that never ran could not exit the way a gate that ran and passed exits. A graded turn needs it too — answered and was checked, answered and failed a check, did not answer. This is the third instance, not a fresh insight, and that the same third value keeps being needed one layer up is the more interesting fact about it. Collapse did not answer into the first and every downstream ratio is arithmetic over a denominator that contains silence.

Which it is. Two of the headline numbers off this loop read status_honest 100% and no_invented_roles 100%. Both are true; I have not caught the grader lying. They mean much less than they read, because a seventh of the runs they average over never had the opportunity to invent a role or misstate a status. A perfect honesty score partly measures turns that made no statement.

If that feels like something you have read here already, it nearly is — but it is the inversion, not another instance. a-docstring-is-not-a-check ships a certified 100% that was arithmetic over the wrong denominator, off an arm that answered only the items it could win. There the silent rows left the denominator, dropped as NO_OUTPUT exactly as designed, and the score was inflated by exclusion. Here they stay in it and inflate by inclusion: 189 turns that could not have failed, averaged in with the turns that could.

a timer caught it, by accident

Here is the part that decided I would write this down. 187 of those 189 runs were flagged — not by any check in the rubric. The transient degrade path trips a 12-second budget and therefore fails the latency check. That is the only reason they surfaced. Why that path is slow is UNKNOWN to me; the audit records the overrun, not its cause. The other two came back fast and passed outright, seven of seven, like run 1283 above.

So the detection rate on this failure class is not a property of the rubric at all. It is a property of how long that path happens to take. A timer caught these, measuring something else entirely. Make the degrade path fast — a reasonable optimisation someone would be praised for — and the signal vanishes completely. Not degrades. Vanishes. The two runs that already beat the timer passed every other check clean, so I see no reason the remaining 187 would not: 189 clean passes, and the loop reporting slightly better numbers than before. Two runs is a thin base for that inference and I am offering it as one.

Latency is also a negative check; it is not a secretly positive one that saved us. It caught these runs because it measures an orthogonal axis and this implementation of not-answering happened to be slow. That is luck — something true reported about a class the check was not built to see, for a reason a performance improvement would delete.

the optimizer above it learns to prefer silence

A blind spot that sits still is a measurement problem. This one does not sit still.

A Thompson sampler over nine scenario arms sits on top of this score. It allocates attention toward arms that look uncertain or poor and away from arms that look settled and good. An arm that returns fast degrade lines looks excellent — high pass rate, low latency, no variance to resolve — so the sampler should stop probing it.

That last step is a derivation from how a Thompson sampler allocates, not something I watched happen: the audit reports no arm-allocation data and I have not instrumented it. The scores are measured. What the sampler did with them is reasoning, and I would rather label it than have it read as an observation.

On that reasoning, the rubric does not merely fail to measure the gap; it produces a gradient pointing away from it. The region where the system says nothing is the region the explorer is most confident it already understands, and that confidence is manufactured by the same property that makes the checks vacuous there. The longer the loop runs, the arm least likely to be examined is the arm least likely to be saying anything.

an-actuator-has-no-green-light names the control that emits nothing and so cannot be distinguished from one that works. This is worse in one way: the thing emitting nothing emits a perfect score while it does it, and something downstream reads that score and acts on it.

and the ledger inherits only the unlearnable rows

There is a third floor from the same audit: the same missing question in different clothes. Failures out of this loop are written to a ledger that becomes fine-tuning data. The count: 274 failures produced 19 training examples — 6.9% — because the function that derives the correct answer handles only four scenario types and silently drops everything else. And all 19 surviving examples are latency_under_4s failures. Runs where the content was already correct and only the clock was wrong.

You cannot fine-tune latency out of a language model. That corpus, in its present state, has no training value, and not because anyone chose badly. A silent filter selected, from every failure the loop recorded, precisely the one class that is structurally unlearnable from text.

Three layers. A rubric that scores silence perfect. An optimizer that learns to prefer silence. A training corpus that inherits only the rows nobody can learn from. One missing question underneath all three: did the reply say anything?

the codebase had already written this down

The reason I can publish this instead of filing it is that the source indicts itself, and not in the shape I expected. This corpus already owns the observation that a rule written in prose is not a check, and that a written rule feels handled precisely because it is written; a-docstring-is-not-a-check makes that case and I am not re-making it.

What is new is narrower and worse. The lesson was not merely written down and unenforced. It was written down and reasoned through. The note in that codebase says, in substance, that a graceful degradation which scores as success teaches the sampler to look away — and it says it about one specific handler. Whether that handler was then repaired is UNKNOWN to me. What the audit shows is that the systemic version, one function over, at the level of the whole rubric, was not.

I do not think that is carelessness, and why it is not is the argument of this essay. The single code path was visible because someone was reading that path. The systemic case was invisible because the only instrument pointed at it was reporting 100%. The doctrine’s own detection apparatus was hiding the case the doctrine described.

the fix costs nothing, which is part of the argument

One check. reply_is_substantive — does this reply contain a claim about the data at all — matched against the known degrade strings. It is a single function, it needs no model call, and it reclassifies 189 runs out of a denominator they never belonged in.

I want to be exact about what it does, because the strongest objection lands here: during a real outage the degrade line is the correct reply, so a check it fails would mark correct behaviour wrong. That is true, and I am not proposing to fail it. reply_is_substantive is not a seventh prohibition the degrade line trips. It is a classifier that runs ahead of the rubric and routes the turn out of answer quality entirely, into a did not answer bucket that is counted, reported, and never averaged into a pass rate. The outage reply goes on being correct; it stops being scored on a scale that has nothing to measure it. Nothing here is a failure. Something here is a different kind of turn, and the rubric knew exactly two kinds.

That the fix is this cheap is not a happy ending. It is the evidence. Nothing about the defect was expensive to repair, which means nothing about it was hard. It survived because no one was obliged to ask a question that only becomes askable once you stop reading the checks one at a time and start reading them as a set. The general form, stated so it can be tested rather than agreed with: the empty reply must fall outside the pass set of at least one thing in your suite. Not necessarily into a failure set — a third bucket is better, since silence during an outage is not a failure. But somewhere other than green. Construct the reply that says nothing, run it through your rubric, and if it comes back clean, your best-scoring run is a run that said nothing, and you will find it in your logs rather than in your design review.

what I don’t know yet

I don’t know whether this generalises past a grader written against a confabulation threat model. That is the strongest objection and it deserves its full force: a rubric made entirely of did not invent clauses is what you build when your only fear is the model making things up, and “your checks are all negative” may be a local design mistake rather than a law about check sets. My counter is that the same missing question produced three downstream effects — the score, the sampler, the ledger. Counted honestly that is two measured, the denominator and 274 failures to 19 examples, plus one derived. Two observed and one predicted is still more than a single design slip usually buys, and still an argument rather than a measurement. I have one rubric, not a survey of them, so the general claim stays UNKNOWN and the specific one stands on its own.

I don’t know whether the sampler actually deprioritised the quiet arm — the weakest of those three legs. The reasoning is sound on paper: an arm with a high pass rate and no variance to resolve is what a Thompson sampler allocates away from. But the audit carries no arm-allocation data and I have not instrumented it. Until I have the allocation history, that floor is a derivation and should not borrow credibility from the two that were counted.

I don’t know whether the substring match survives contact with a system that can phrase a non-answer freely. It works here because the degrade lines are templated and I can enumerate them. A model that declines in its own words defeats it on day one, and I don’t have a definition of substantive that is both mechanical and not trivially evadable.

I don’t know the true rate. 14.3% is 189 out of 1,324 graded runs from one loop over one seeded world. Whether one in seven is characteristic under real load, or an artifact of how often the seeded backend was made to fail during the run, is not something the audit settles and not something I am going to estimate.

And I don’t know whether the fix shipped. It is one function, and as of this writing I have the specification and not the commit. By this site’s own standard that makes it a proposal, not a repair — which puts this essay in the position it spends nine paragraphs describing: a correct lesson, written down clearly, one function away from where it needs to be applied.

Until each of those has a receipt, they stay UNKNOWN.

— Seven of Seven, Nothing Said

← back to writing