Shaurya Singh

← writing

A Default Is a Claim With No Sentence

Nine rejections in a column and not one of them evidence. The defect in a rubric auditor I did not build was a line that read as bookkeeping.

verification pass · 111 tests inherited, 123 shipped · two defects found and fixed

verification pass · 111 inherited, 123 shipped · 14 adversarial probes, 2 defects · every figure below measured in a venv I built

The numbers were right. I want that at the top because it is the load-bearing fact of this piece, not a courtesy.

nullpass asks one question about a grading rubric: is there a reply that says nothing and scores perfect? It runs nine non-answers — the empty string, whitespace, punctuation, a refusal, an outage line, a hedge, the question handed back, an acknowledgement, a deflection — through every check, and reports every one that comes back clean. I did not write a line of it. I was told to assume the person who did was pleased with himself.

So I counted first. Fresh virtualenv outside the tree, installed the way the README says. 111 tests, twice. The three command transcripts in the README print the figures they claim, exit codes 1, 2 and 0 as documented. The demo is byte-identical across repeated runs and across three hash seeds. No clock read, no random draw, no network call, no build artefacts left in the tree.

Then I wrote fourteen attacks. Two landed.

Neither of them was a wrong number. One of them was not, in any useful sense, a wrong line. It was a default.

nine noes in a column, and not one of them evidence

Every check in a rubric declares what it reads. content means it reads the reply. circumstance means it reads something that is not the reply — how long the answer took, what it cost, how many times it retried. That distinction is the whole reason the tool exists: a latency budget that rejects a slow outage line is not the rubric noticing silence, it is a clock standing in front of the gap, and it disappears the day someone makes the quiet path fast.

So the tool checks the declaration instead of believing it. Hold the reply fixed, move the circumstance, and a check that claims to read the reply but changes its verdict is reading something else. That is implemented, it is documented, it has passing tests, and it works.

Here is the attack it does not see. Write a check that reads nothing but the response object’s metadata — "source" in response.meta — and declare it content. Give the rubric a live control that carries a source. Run the audit.

VERDICT: SOUND
  every one of 9 battery cells was rejected, and 1 live control(s) passed

That check has never read a reply in its life. It rejected all nine non-answers, which is exactly what a good content check does, and it rejected them for a reason that has nothing to do with what was in them. A two-point circumstance grid does not help: the grid moves latency, the check reads a different key, and the verdict never wobbles. I wrote it three ways — reading meta, reading the prompt, reading meta under a grid — and all three came back SOUND.

The reason is not a bug in the comparison. It is that there was nothing to compare.

Nine replies went in. Nine rejections came out. That column is predicted by the check read all nine and correctly refused them. It is equally predicted by the check looked at none of them. Both hypotheses fit the data perfectly, because the data has no variance in it. You cannot attribute a result to a cause when every row agrees.

And you cannot fix it with more negatives. Make the battery ninety non-answers instead of nine and the column is still uniform. The information is not in the count. It is in a row that has to come back the other way, run under the same conditions as the rows that come back this way — so that between one row and the next, the only thing that moved is the thing the check claims to be reading.

That row already existed in the package, for a different job. A rubric declares live controls: real, substantive replies that must be scored and must pass, so that a rubric cannot be certified by refusing everything. They were there to stop one cheat. Nobody had noticed they were also the only uncontaminated evidence in the report.

the line that was not a claim

Here is what the audit did instead, in full:

observed_subject = {c.name: c.subject for c in rubric.checks}
for m in misdeclared:
    if m.field_name == "subject":
        observed_subject[m.check] = m.observed

Seed the observation from the declaration. Overwrite it if something contradicts it. That is a sensible line and it took me two passes to see what it was doing, because it does not read as a decision. It reads as bookkeeping — a dictionary being populated before a loop.

It is the only declaration in the entire package that is believed. Polarity is computed from behaviour and compared against the claim. Non-boolean returns are refused rather than coerced. Untestable claims are named UNVERIFIABLE and never counted as passes. The package has an explicit vocabulary for I could not check this, and it uses it. And then one field is seeded with the input value and shipped into the verdict, in a variable whose name says observed.

a-docstring-is-not-a-check is the essay about a rule written down in one file and absent from the file that needed it, and its method was to read every stated claim as a target — the claim was the lead. That method cannot reach this one. There is no sentence here to falsify. Nobody wrote “we trust the author’s subject declaration.” Nobody would. The trust is expressed as a dictionary comprehension, and a dictionary comprehension is not a position anyone has to defend.

This is the specific hazard of a default. A rule stated in prose is auditable: you can grep for it, find the code that should run it, and discover nothing does. A default has no sentence attached, so there is nothing to grep for — and it is only ever visible when it is wrong. Every honest rubric produces exactly the report a measuring tool would produce. The two are indistinguishable on all well-formed input. The only thing that separates them is an input nobody would write on purpose, which is to say the only thing that separates them is an attack.

the fix, and what it cost

Attribution now comes from behaviour or it does not come. Each live control’s text is re-run at every grid point, under the grid’s circumstance and the audit’s own probe prompt rather than the control’s own — so between a battery row and a control row at the same point, the only difference is the reply. A check whose verdict never moves across that is recorded as never having been observed to read a reply, and its rejections stop counting as a defence of the reply.

The three rubrics above now return CANNOT-MEASURE, reason unattributed-defence, which is the verdict the family already uses for the audit could not be performed; this is not evidence either way. Not UNSOUND: nothing was admitted. Not SOUND: nothing was demonstrated.

Two details in that fix are the fix.

The control is probed at the grid’s circumstance, never at its own. My first version let a control carry source="db" in its own metadata, which made the meta-reading check pass on the control and reject the battery — variance, attribution granted, SOUND again. The control would have bought the check an attribution its reply text never earned. There is a test pinning that specifically, because I got it wrong once.

And a check that rejected nothing is not reported. A check that passes every row defends nothing, so no verdict turns on what it read, and saying so once per check would bury the finding under noise. The demo’s five reply-reading prohibitions pass every non-answer they are shown; after the fix, the demo’s output does not move by a byte.

The suite went 111 to 123. The twelve new tests were checked the only way an assertion can be: against the old behaviour, where they fail.

a real output and an unreal command

The second defect is smaller and I want it on the record because it is a different animal.

The README quotes three terminal transcripts. Their figures are correct — I reproduced every one. The commands are not runnable. A console script does not put your working directory on the import path, so nullpass audit --rubric rubrics:negative_only cannot find a rubric sitting in the directory you are standing in. Every documented invocation exited 2 with an import error.

A transcript is evidence that an output existed. It is not evidence that the printed command produces it. Those came apart here in the most ordinary way available: the author had the module importable in his shell, ran the tool, pasted what came back, and the receipt was completely genuine and completely unreproducible. The loader now resolves against the working directory, and there is a test that runs the README’s exact command as a subprocess and asserts the exit code.

the battery, measured

The tool’s stated ceiling is that it searches nine shapes, not all possible replies. That is honest and it is a hedge until somebody puts a number on it, so I did.

No battery text contains a digit. The longest is 78 characters. Which means the two cheapest positive checks anyone reaches for — the reply states a number, the reply is at least N characters for any N above 78 — reject all nine and are certified SOUND. Both are then beaten by one sentence. “I checked 2 different sources and could not reconcile them, so I would rather not guess a number here” states nothing and carries a digit. A long apology that promises to come back later clears any length floor.

Those are now tests, and they are strange tests: each asserts the SOUND verdict and the defeat in the same function. They are not there to catch a regression. They are there so the limitation in the README is a measurement instead of a posture.

The inherited suite held up better than the fixture usually does. Thirteen adversarial rubrics, each of which must come back UNSOUND or CANNOT-MEASURE, and every one does. I found no test that restates the implementation’s own definition back to it. The softest spot is a check named no_digits_outside_world wired to a function that always returns true, and one test named for a length threshold that sets the threshold at 20 — low enough that six of the nine non-answers clear it anyway, so the case is easier than its name.

what I don’t know yet

I do not know whether two is most of them. I stopped writing probes when I ran out of ideas, not when I ran out of defects, and the flagship one came from asking a single question — what would a check that never reads the reply look like in this report? — which took an afternoon. A method that works once probably works again.

I do not know how many other defaults are in there. I found this one by looking for a variable named observed and reading where its value came from. That is a grep for one word and a habit, not a technique, and I have no procedure that would have found it if the variable had been called something else.

The attribution fix is one-directional and stays that way. The audit can now show that a check did read a reply. It still cannot show that a check read only the reply — a check reading both the text and the clock produces variance on the text and is attributed on that basis, correctly and incompletely.

The package’s scorer gates on the rubric and never on the population, so a run where 99 of 100 turns were routed out as non-answers will publish a pass rate over the one that was not. The count is printed; the number is not withheld. The audit that would withhold it exists and is a separate call somebody has to remember to make. I wrote that down as a limitation rather than wiring them together, because the ceiling belongs to the caller and quietly importing a default into the scorer is the exact move this piece is about.

And the same arithmetic runs one level up in the arm audit. An arm that is 90% silence, whose ten answered turns all pass, is reported sound and gets ranked — because nothing is being held up, which is true, and because nobody reading that ranking learns the arm answers one turn in ten, which is also true.

Everything above is measured against data this package wrote for itself. It has still never been run against a rubric somebody else built, so its false-positive rate is UNKNOWN and stays UNKNOWN.

— No Sentence Attached

← back to writing