Shaurya Singh

← writing

Care Doesn't Scale. Refusal Does.

An audit of 313 files found 48 broken guards, each with a correct fix sitting nearby in the same repo. Diligence didn't fail. It just doesn't scale.

ongoing · doctrine

ongoing · doctrine — 48 confirmed instrument defects, 313 files audited, 2026-09-07

I commissioned an audit of my own build lane. Not the model. Not the training runs. The instruments — the scripts that are supposed to stop bad things from happening before they happen: spend brakes, provenance gates, scoring code, backup verifiers. 160 agents, read-only, nine lenses over 313 tracked files, 1,985 tool calls, zero errors. Every finding then went through three independent refuters, each told to default to REFUTED unless the defect survived attack from a different angle. Fifty findings reached that stage. Forty-eight survived. Two didn’t. Nine more got cut by a per-dimension cap before anyone even tried to refute them — flagged as cut, not folded quietly into the count, so the report doesn’t read as more complete than it is.

Forty-eight confirmed defects in code whose entire job is to catch defects.

Here’s the sentence from that audit I can’t get out of my head: the audit paired nine of the defects against a correct twin — the same fix, written by the same person, often days apart, sitting somewhere else in the same repo. Not all forty-eight have a twin on record. But nine confirmed pairs is enough to name the pattern: the knowledge existed. The diligence existed. It just didn’t travel from the file where it was applied to the file where it was needed.

Care doesn’t scale. Refusal does.

The instrument that grades its own input

Start with the sharpest form of the defect, because it explains why “just be more careful” was never going to be the fix.

The pattern is simple to say before it gets technical: a check that only ever confirms something it already knew is not a check. It’s an echo wearing a checkmark. Here’s what that looked like in the one gate that has to hold, because the alternative is a credit card charge nobody authorized.

The spend-brake chain’s first checkpoint sends a request to the billing API to confirm the API key is live before authorizing any spend:

POST <billing api>  {"query": "query { myself { id } }"}
→ 200
→ WORKING+=("runpod-api")

Look at the query. There’s no pod id in it. It asks “does this key name some account,” not “can this key stop this specific pod.” A 200 on that question tells you the key works. It tells you nothing about whether the key can reach the thing it’s about to authorize spending on. The gate can’t fail on the question that matters, because it never asked that question. It only ever had the one answer it was built to accept.

That’s not an isolated typo. The audit found the same shape repeatedly: a check whose pass condition is derived from something the check itself already supplied. One digest routine hashes the same file twice under two names and calls the match a verification. One dedup key asserts that a marker exists in a dictionary the same function wrote forty-some lines earlier — of course it’s there. A refusal that inspects only its own input can’t see the input it’s supposed to be grading. It sees an echo and calls it corroboration.

None of this is a knowledge gap. Whoever wrote verify_brakes.sh knows what a pod id is — the very next script in the chain uses one. The problem is structural: nothing forced the question being asked to be the question that mattered.

The guard that can’t fail

The audit’s proposed fix for this whole family is a construction rule, stated so it can be tested rather than just agreed with: a check must be able to name a reachable state in which it prints red, and that state has to be producible by someone other than the check. If the only way to make the guard fail is to change the guard, it isn’t a guard. It’s a comment that happens to run.

They tested this literally, by mutation. toolread_probe.py ships a self-test that claims to catch a model echoing back a tool’s input operands instead of computing anything with them. Delete the guarded clause — the actual comparison that would catch that failure — and rerun the self-test:

$ git diff toolread_probe.py
-    if result == a or result == b:
-        return "operand echoed, not computed"
$ python toolread_probe.py --self-test
8/8 passed

Eight of eight, green, with the one check that mattered surgically removed. The test suite couldn’t tell the difference between a probe that checks something and a probe that checks nothing, because the corpus generator that feeds it was written to guarantee the result never equals an operand in the first place. The control was vacuously satisfied by construction. It could not have failed even before anyone deleted anything.

That’s the diagnostic the audit uses everywhere: does this check have a reachable red state, and can I produce it without editing the checker? Applied across the lane, it’s how they found that the whole automated-test runner has the same shape one level up. Its own coverage report structurally cannot name a tool whose self-test isn’t being run — having a self-test is the condition that excludes a file from the “missing” list, so a file can carry a self-test that’s never invoked and the runner will never flag it as a gap. Twenty files fit that description. Demonstrated the same way: deliberately break the matcher inside provenance_audit.py, rerun the suite:

$ ./run-tests.sh
31 ok, 0 failed — ALL GREEN

All green, with a known-broken matcher sitting inside it, republished downstream as the lane’s standing test state. Nobody lied. Nobody was careless in a way you could point to and say “fix your habits.” The runner was built to report on the tests it runs, and a test that never runs was never in scope for a red.

Absence is never the pass value

The second construction rule in the audit is shorter: absence is never the pass value. It’s stated as house doctrine in at least four places in this codebase — a provenance module refusing on empty state, a training script’s spend check, a brake gate’s own docstring, all some version of “an unreadable check refuses exactly like an unreadable ledger.” Stated in four places. Violated in at least seven instruments the audit actually opened up.

The tokenizer-provenance gate is the cleanest example. It’s supposed to confirm that a corpus and the tokenizer that will read it actually agree on identity. Zero of the twenty-three shipped manifests in this repo carry the field that gate checks. So the gate takes its unrun branch on every single corpus that exists — not because anything failed, but because the fact it needs was never recorded. The wrapper that calls it discards the detail and keeps only a tally: N of N gates passed. Cannot-measure isn’t supposed to be a pass. Here it renders as one, because nobody wrote the branch that turns “I don’t have the data to check this” into a visible refusal instead of a silent skip.

Same failure, different layer, on the backup that exists to survive a disk wipe: the “already banked” check looks for a JSON sidecar file and stops there. It never confirms the actual checkpoint the sidecar describes still exists on the destination. Delete the real file, leave the sidecar, and the banker will believe forever that a disk-wipe-critical artifact is safe. The sidecar’s presence became the pass value. Its absence — the thing that actually mattered — was never checked at all.

Refuters instructed to default to REFUTED

The one place in this whole lane where the doctrine visibly worked is the audit’s own method, and it’s worth being precise about why. Every finding wasn’t just reviewed — it was handed to three separate refuters, each with a different lens (correctness, reachability, “is this already known”), each told the default verdict is REFUTED unless the finding survives being attacked. That’s not peer review. Peer review agrees more often than it should, because agreement is comfortable. A refuter instructed to default to REFUTED has to do work to let something stand.

The underlying move is the same one the audit demands of every instrument in this lane: don’t trust the checker just because it’s the checker. Assume it’s wrong until something forces it to prove otherwise. There’s an older instance of that exact move, smaller and more concrete, from before this audit existed. An eval script called estate_battery.py checks whether a model knows facts about the estate it runs on — which port a server listens on, who bills for what, who has authorization for what. The first version of that script failed the model that was supposed to pass it. Not a bug in the model — a bug in the checker: its “must never contain” rule fired on a correct answer that named the wrong fact only to reject it, the way “5001, not 8000” correctly contains “8000.” The fix generalized into a rule for every fact-based grader in the lane: a negation check backfires wherever the right answer legitimately references the wrong one; lean on “must contain” there instead. That rule only exists because someone ran a known-good input through a new instrument and treated the failure as the instrument’s fault, not the model’s. The instrument is not exempt from the audit it performs.

The lane’s headline capability claim about argument binding got the same treatment on its own publication day. It shipped with a clean result — one arity of tool call succeeding far more than another, a statistically sharp gap. Same day, a second pass noticed that every tool family in the corpus has exactly one arity: the “argument count” partition and the “which tool” partition were the same partition, wearing two names. The original claim wasn’t wrong about the numbers. It was answering a question adjacent to the one it thought it was answering. It got retracted and re-run against a design built specifically to tell the two explanations apart, and the second design held up under the same adversarial standard the first one never got.

What the fix looked like when it worked

The recurring correct twin in this audit is a script called run_seat_sft_v2.sh. It handles the same unattended pod-termination job that a broken sibling script gets wrong — that one pipes its terminate call to nothing, checks nothing, and falls back to a shutdown command measured to be denied on the machine image it runs on. The working version retries, verifies termination by its actual effect rather than trusting the response, and writes a marker file if it still can’t confirm the pod is dead. The audit’s note on this pair is blunt: nothing new needs inventing. The fix already exists, in the same repository, probably written by the same hands. It just never got copied to the sibling that needed it, because nothing in the repo can enumerate what the siblings even are.

That’s the whole argument in one pairing. The fix and the defect are not separated by skill. They’re separated by the fact that a correct pattern, once written, has no mechanism that propagates it. A person has to remember every place the same shape recurs, forever, across a growing pile of scripts, under deadline, at the end of a long day. That’s not a discipline problem. That’s a scaling problem, and scaling problems don’t get solved by asking people to try harder.

The proposed remedy in the audit is structural for the same reason the defects are structural: a lint rule minted per repaired defect, so the next script that reintroduces the same shape fails a check instead of waiting for the next audit to find it by hand. And a rule that every check named in a docstring has to be named in a test that has been observed to actually fail — not just exist, not just run green, but demonstrated red at least once, the way toolread_probe.py’s self-test was demonstrated to stay green after the guard was deleted. A check nobody has ever watched fail is a check nobody has verified can fail at all.

What I don’t know yet

I don’t know whether the lint rule gets built, or whether it becomes one more correct instrument sitting next to defects it was never wired to catch — the exact failure mode this whole audit exists to name. I don’t know what’s inside the nine findings that got cut by the per-dimension cap before any refuter looked at them; they’re recorded as cut, not cleared, and that’s a different thing from safe. I don’t know whether the refuting-verifier method holds its own defaults as the number of files it has to cover keeps growing, or whether three refuters per finding is itself a diligence cost that stops scaling exactly the way the diligence it was built to replace did. And I don’t know how to build a check for the check that inspects its own input — the pattern generalizes uncomfortably well, and I haven’t yet found the version of “producible by a party other than the check” that closes it for good.

Until I’ve measured that, it stays UNKNOWN.

— Shaurya

← back to writing