Shaurya Singh

← writing

A Docstring Is Not a Check

Its numbers were exact. Four of my sixteen attacks on a grader I did not build found defects, and every one was a rule stated in prose and never run.

verification pass · four defects found and fixed · nothing here graded a model

verification pass · 151 tests inherited, 167 shipped · every number below measured in a venv I built

I was handed a package called echograde and told to assume the person who wrote it was pleased with himself. It is a grader that separates right, wrong and echoed, and refuses to score an item whose answer is already sitting in the prompt. I did not write a line of it.

Its suite went green on the first run, which I expected and which settled nothing, because of what this particular package is. It is an instrument built to catch a specific failure — a measurement that credits the absence of a capability as a demonstration of it. Naming that failure does not confer immunity from it. If anything it does the opposite: the author has written the rule down, and a written rule feels handled. So I did not go looking for sloppiness. I went looking for the package’s own doctrine, applied everywhere except to the package.

I found four. Each one sits in a file where the rule that forbids it is written down somewhere else.

the arithmetic was honest, so start there

Before any of that, the counting, because a report whose totals do not survive a recount is not worth attacking.

The README says 151 passed. It says 148 test functions across six modules, one parametrised four ways, which is where the other three come from. I built a clean virtualenv outside the tree, installed the package the way the README says to install it, and counted.

shaurya@sps ~ % python3 -m venv /tmp/eg && /tmp/eg/bin/pip install -e ".[dev]"
shaurya@sps ~ % /tmp/eg/bin/python -m pytest -q
151 passed in 0.57s
shaurya@sps ~ % grep -h "^def test_" tests/test_*.py | wc -l
148

148 plus three. Exact. The three CLI transcripts quoted in the README — the audit, the grade, the grade under --allow-leaks — reproduce character for character, exit codes included. The demo’s five parts print the numbers the README quotes, down to the accident it says it kept: on a hundred product items built from the index, item i076 is 21 * 4, an exemplar answers 84, and the gold value is sitting in the prompt by coincidence. One item, unplanted, and the audit finds it.

Determinism holds and it holds structurally. There is no clock read and no random draw anywhere in the package — every set is either sorted before it is emitted or used for membership only. Four suite runs, two of them under different hash seeds, and one from a bare PYTHONPATH with nothing installed at all: 151 every time. The demo’s full output is byte-identical across three hash seeds. No build artefacts in the tree, no stray .venv, no egg-info, and no third-party runtime import — the whole thing is stdlib.

So the author counted, and he counted honestly. Every defect below is a defect in a package whose numbers I could not make disagree with themselves.

the module that exists to have one ruler shipped two

normalize.py opens with this, and the sentence is the whole reason the file exists:

String canonicalisation, kept in one place so the audit and the echo detector can never disagree about what “the same string” means.

Two functions live in that file. normalize() decides whether two strings are equal — it strips thousands separators, so 1,234 and 1234 are the same string. tokens() splits text into a token stream, and contains_token_run() uses it to ask whether the gold value appears anywhere in the prompt. That is the check the whole refusal rests on. tokens() split on digit runs, and a comma is not a digit, so 1,234 came apart into 1 and 234.

One file, one stated rule, two rulers.

Here is what that buys you. Write an item whose gold is 1,234. Put the number in the instructions the way anyone writes a worked example — the running total so far is 1234. The model reads it and returns it.

audit leaked: False
outcome for a copy straight out of the prompt: Outcome.CORRECT

Not ECHOED. Not ECHOED_GOLD. Not refused. CORRECT — the absence of the capability scored as a demonstration of it, which is precisely the failure this package was built to make impossible, reproduced inside the fix for it. The audit could not see the leak because its tokenizer had already decided the number ended at the comma. The matcher could see it fine, because that is the ruler it was using.

And the echo detector never had a chance: it deliberately excludes the instructions, on the correct grounds that a model repeating a word from the task statement is not the failure being hunted. Every guard was working exactly as designed. Two of them were measuring in different units.

The fix is one regular expression. A digit run now carries the separators canon_number strips, so a number is one token there for the same reasons it is one string here. Afterwards the audit refuses the item, the strategy is named verbatim, and the demo output does not move by a byte.

I want to be clear about how this was found, because it was not insight. I went looking for a string that normalize and tokens would disagree about, because the file claimed they could not. The claim was the lead.

an override checked once is not checked

Waiver is a frozen dataclass with a __post_init__ that refuses to construct without a written reason and a named approver. Good shape. The grader consumed it like this:

waived = item.waiver is not None

So the validation ran at construction and never again, and the grader asked a question that any object on earth answers correctly. Item(waiver=object()) is a fully valid waiver with no reason and no approver — it walks a refused item straight through the refusal. And frozen is not immutable: build a real waiver, then object.__setattr__(w, "reason", " "), and the recorded justification is gone while the override keeps working.

Both of those are one call: the grader now validates the waiver where it uses it, not where somebody once made it. An override that is only checked when it is written is not an override that is checked.

the floor was measured somewhere else

This is the one I would have shipped, and it is the package’s own retraction happening to the package.

chance_floor scores a degenerate baseline against the real answer key, item by item, and capability_claim compares the arm’s hits to that floor with an exact binomial tail. The doctrine, from compare.py, is stated twice in that file: two arms are only comparable on identical item ids, and same n is not the same sheet.

The floor was measured over every item on the sheet. The arm was measured over the items that actually entered the denominator. Nothing checked that those were the same rows.

So: a hundred items, four categories, forty of them sharing one gold value. MajorityGold — always answer the sheet’s most common gold — hits forty, and the floor prints 40%. Now let an arm answer only those forty items, with that same majority answer, and stay silent on the rest. It gets forty for forty. Sixty rows leave the denominator as NO_OUTPUT, exactly as designed.

observed 40/40 net hits (100.0%)
control majority_gold expects 40/100 (40.0%) against this answer key
capability claim for arm: DEMONSTRATED   p = 1.2e-16

The arm is the control. It emitted the baseline’s own answer and nothing else, and the tool certified it as a demonstrated capability at p = 1.2e-16. A thinner version works too: answer three items, get three right, and three-for-three against a floor measured on a hundred rows comes back DEMONSTRATED at p = 0.039.

Nothing was wrong with the arithmetic. The binomial tail is exact. It was exact about two numbers that were never on the same sheet.

Two fixes. The command-line tool now measures the floor on the items that were actually scored, and capability_claim refuses any pairing where the two denominators disagree — REFUSED, not a worse NOT_DEMONSTRATED, because nothing was demonstrated or denied. The demo output does not change: the control there is already refused for being structurally incapable of scoring on nine of ten categories, which is the worse defect of the two and is reported first.

the rule held in one subcommand

The README is unambiguous. --allow-leaks grades the refused items anyway, for a legacy sheet you have to keep scoring, and it does not buy a clean exit.

echograde grade --allow-leaks honours that. It collects the items that were leaked and scored anyway and exits 2 — CANNOT-SCORE, the instrument rather than the subject. echograde compare --allow-leaks graded the identical items through the identical grader and had no such check. Give it a sheet where a fifth of the items have their own gold planted in the exemplar block, with every band valid and copy rate resolving cleanly, and it exits 0.

The refusal was implemented in a function rather than in the package. Same check, now in both.

what the four have in common

None of these is a seam between two components. Every one of them is a rule that existed as a sentence in one file and as nothing at all in the file that needed it.

The canonicalisation rule was in a module docstring; the tokenizer did not read it. The waiver rule was in __post_init__; the grader did not re-ask. The same-sheet rule was in compare.py’s docstring, two functions above the code that compared a hundred-item floor to a forty-item arm. The --allow-leaks rule was in the README and in one of two subcommands.

That is not carelessness, and it is not a lesson about discipline. It is the specific tax on building a tool whose subject matter is its own failure mode. Writing the doctrine down is the thing that makes you feel you have handled it, and prose is exactly the medium in which a rule cannot be run. The defects clustered where the author had thought hardest, because that is where he had already written the sentence and moved on.

the suite, audited rather than admired

Three things in the inherited 151 are worth naming.

test_stats_are_deterministic_across_repeat_calls reads a pure property off a frozen dataclass five times and asserts the answers are identical. There is no implementation of that property which fails it. test_chance_floor_counts_exact_hits_against_the_key asserts expected_hits == 10 and then expected_rate == 0.10, and expected_rate is defined as hits over n — the second assertion restates the implementation’s own definition. And test_overlap_ratio_is_recorded_but_never_changes_the_outcome has a docstring ending “and still CORRECT”, then asserts the outcome is in {CORRECT, ECHOED_GOLD}, a disjunction that passes under either branch and pins neither.

Against that, most of the ten cases the README calls adversarial really are — the band that raises instead of printing 6.0, the leaked item refused whether the response was right or wrong, "3" not found inside "13". One is mislabelled: rejecting a sheet with no answer key is input validation, not an attack.

The gap that matters is the invariant. The package’s central promise is that on an audited sheet, a copy that lands on gold cannot be silently credited. The only test of it cripples the battery on purpose — Grader(strategies=(ConstantAnswer(),)) — which proves the guard fires when you remove the guard. With the default battery I found a real path to it in an afternoon, and the guard did fire, and it fired by raising rather than printing a number, which is the right behaviour. But nobody had shown it doing that on a battery anyone would actually ship.

Sixteen adversarial tests now ship alongside the original 151. Seven of them failed the day they were written. Each one was checked the only way an assertion can be: the old behaviour was restored and the test was confirmed to fail against it.

shaurya@sps ~ % pytest -q            # fixed tree
167 passed in 1.02s
shaurya@sps ~ % pytest -q            # old semantics restored
7 failed, 160 passed in 1.07s

what I don’t know yet

I broke four things. I have no idea whether four is most of them or a fraction, and the honest reading of the flagship defect is uncomfortable: it was findable in an afternoon by reading a docstring’s claim as a target. If that method works once it probably works again, and I stopped attacking when I ran out of stated claims, not when I ran out of defects.

The tokenizer fix makes two canonicalisations agree about separators. It does not make them the same function, and they still are not. normalize treats the whole string as a possible number; tokens treats each run as one. I closed the gap I could construct a case for. Whether another gap exists between those two is UNKNOWN, and the structural fix — one function, one ruler, no second implementation to drift — is not what shipped.

The capability_claim denominator refusal is now strict: the floor’s item count must equal the arm’s scored count. That is the right rule and it is a coarse one. It refuses a correct pairing that happens to be built two different ways, and I have not run it against a real sheet to find out how often that costs someone a legitimate claim.

And every number in this piece is a property of data this package built for itself. Sixteen attacks, seven landing, four defects, 167 green — all of it measured against constructed sheets, against an answer key the tests wrote. Nothing here called a model. The package’s own honesty tail says it has never caught something its author did not already know was there, and my pass does not change that. It has now caught four things I did not know were there, in itself. That is a claim about a grader. It is still not a claim about any eval it has ever run, because it has not run one.

— Two Rulers, One Number

← back to writing