Shaurya Singh

← writing

A Rubric Is a Satisfiability Question

I built the auditor the last essay owed. Three things I had written down as obvious turned out to be design decisions, and one of them became a third verdict.

prototype · built and run · 111 tests, three exit codes observed

prototype · 111 tests, all passing · 3 failed on the first run, all three my error · all three exit codes observed · nothing run against a real grader

The last essay ended owing a fix. One function, specified, not shipped. This is what happened when I stopped specifying it and wrote it.

Not the one function — the general form. Given a rubric, decide whether a reply that says nothing scores perfect. If one does, the rubric measures abstention and no aggregate should come out of it. That is a small enough idea to hold in one hand, and I expected an afternoon of plumbing.

I got three design decisions I had not noticed I was making, and one of them turned into a verdict.

the question has a quantifier, and I had not chosen it

A rubric is a conjunction. Every check has a set of replies it rejects. Union those sets and ask whether the empty reply is in any of them. Stated that way it is a satisfiability question, and satisfiability questions come with a domain attached whether or not you write one down.

I wrote nine. Empty string, whitespace, punctuation only, a refusal, an outage line, a pure hedge, the question handed back, an acknowledgement, a deflection. The tool runs all nine through every check and reports every one that comes back clean.

That is a weaker question than the one the sentence implies, and the weakness is the whole reason the thing runs in milliseconds instead of never. I am not deciding satisfiability over the space of all possible replies. I am asking whether a satisfying assignment exists among nine shapes a person wrote down. A rubric that rejects all nine has been shown weak evidence. It has not been proven sound. It catches the non-answers I thought of.

That limitation is in the README because it is the first thing someone should distrust, and because I would rather name my own ceiling than have it found.

The quantifier over those nine is existential, and that turned out to matter more than the list. One reply, under one set of circumstances, scoring perfect is enough. Not most. One. An optimiser sitting above the rubric does not need the average case to be exploitable; it needs a single point in the space where scoring is free, and it will find that point by doing its job.

the clock is not a defence, and that became a verdict

Here is where the design stopped being plumbing.

A check does not only read the reply. Some checks read the circumstances the reply arrived under — how long it took, what it cost, how many retries. In the loop the last essay measured, almost every non-answer was flagged by a latency budget and by nothing else. A timer caught them, measuring something orthogonal, because that particular quiet path happened to be slow.

So the assignment a rubric gets satisfied by is not just a reply. It is a reply plus its circumstances. I gave rubrics a circumstance grid — a small set of declared points, written into each battery reply before evaluation — and ran the battery at every point.

Two things fell out of that, and the second one I did not expect.

The first is straightforward. A reply rejected at one grid point and clean at another is admitted. Existential quantification, verdict UNSOUND. Fine.

The second needed a new verdict. What about a reply rejected at every declared point, but at every point only by checks that never read it?

That rubric is not sound. Its content checks admit the reply outright. A clock is standing in front of the gap, and how long a quiet path happens to take is not a property of a rubric — it is a property of an implementation, one performance improvement away from changing. Calling that SOUND would be certifying luck.

It is not UNSOUND either. Within the space I was given, nothing scored perfect. I did not observe an admission.

So it gets the third value the whole family already uses. CANNOT-MEASURE, exit 2, reason circumstance-only-defence. The report names every reply whose only defence was a check that does not read replies.

Two rubrics in the demo differ in exactly one place — whether the declared grid contains a fast point:

shaurya@sps ~ % nullpass audit --rubric rubrics:negative_only | tail -2 ; echo $?
VERDICT: UNSOUND
  degenerate-admitted: ... scored perfect in 9 of 18 battery cells
1

shaurya@sps ~ % nullpass audit --rubric rubrics:timer_only | tail -2 ; echo $?
VERDICT: CANNOT-MEASURE
  circumstance-only-defence: ... rejected at every grid point, and only ever by
  checks that do not read the reply
2

Same six checks. Same nine replies. One number different in one declaration, and the exit code moves.

I want to be exact about what that second verdict is worth, because it is easy to oversell. It does not prove the rubric will break. It says the rubric has not been shown to distinguish silence on the reply’s own terms, and that the thing currently doing the distinguishing is outside the rubric’s stated subject. That is a smaller claim than “this will fail,” and it is the honest size of what the tool observed.

I stopped believing the declarations

The first version asked each check to declare itself: NEGATIVE if it fails only when the reply does something, POSITIVE if it fails when the reply omits something. Then it counted the positives and complained when there were none.

That version was a spellchecker. It measured what someone had named their functions. a-detector-that-measured-spelling is the essay about that mistake and I am not making its argument again — I am reporting that I walked into it on the first commit of a tool built specifically to distrust labels.

What killed it is that both directions of the label are wrong in practice, and they are wrong asymmetrically.

A check named as a prohibition — did_not_return_a_non_answer — that genuinely rejects the empty reply is a positive check wearing a negative name, and it works fine. Failure sets decide this. Wording does not.

A check declared POSITIVE that rejects none of the battery is the dangerous one. reply_is_present, implemented as a length comparison that can never be false. It reads like coverage. Its failure set contains no reply that says nothing, so it is a prohibition whatever it is called, and its presence in the list is exactly what makes a review conclude the rubric is covered.

So nothing is believed. Every declaration is checked against behaviour. A POSITIVE check that rejected nothing is reported as misdeclared, with the count. The subject declaration gets the same treatment from the other direction: hold the text fixed and move only the circumstance, and a check that claims to read the reply but changes its mind is reading something else. Hold the circumstance fixed and move only the text, and a check that claims to read the clock but changes its mind is reading the reply.

A declaration that could not be tested — subject verification needs at least two grid points — is named as UNVERIFIABLE and never counted as a pass. That rule is borrowed wholesale from axisproof, which will not certify a scorer whose controls could not be performed, and it is the right rule for the same reason there.

the fix has a cheat in it

The fix the last essay specified is not a seventh prohibition. It is a classifier that runs ahead of the rubric and routes a non-answer out of answer quality entirely, into a bucket that is counted and never averaged in. The outage line goes on being correct. It stops being scored on a scale with nothing to measure it.

Implement that and the attack is immediate: route everything out and the battery has nothing left to admit.

So the tool will not certify a rubric that has never been shown to accept an answer. A rubric declares live controls — real, substantive replies that must be scored and must pass — and a rubric with none can never be SOUND. It exits 2 no matter how well it behaves. Rejecting everything is not soundness, and a router that eats a real answer is a worse disease than the one it was installed to cure.

This is the same shape as a self-identity control in a similarity harness: before you trust anything a scorer says about difference, make it say the right thing about sameness. I did not invent it here. I imported it, because the attack it defeats is the same attack.

The refusal itself is a refusal and not a warning, and that distinction is the point of the package. A warning printed above a number is read by a human once. The number is read by everything downstream, forever. So score() grades every response, reports every outcome, and leaves the aggregate as None when the verdict is not SOUND. There is no flag to override it.

three tests failed, and all three were me

The first full run was 109 tests. Three failed. All three were my expectations, not the code, and I want to be exact about that because the interesting one still taught me something.

Two were bookkeeping. I had written a threshold wrongly in one and picked a check that rejected uniformly in another, so neither test exercised what its name said.

The third I had written to assert CANNOT-MEASURE, and the tool said UNSOUND. A check that crashes on the empty string — reply[0], the most ordinary bug in this domain — cannot be said to have passed that reply, so its cell is an error. But the same check happily passed the refusal, the hedge and the deflection. There was a genuine admission sitting right next to a row that could not be evaluated, and the tool reported the admission.

That is correct, and it is correct because I had copied the ordering from two projects that already argue for it: an admission that really happened is a defect in the subject, a row that could not be evaluated is a gap in the instrument, and the first is more actionable. axisproof ranks REFUSED over VOID on that reasoning. gatehealth refuses to let disease detection cannibalise regression detection on the same one.

So the code was right and I was wrong about my own code, which is a smaller story than a bug — except that the ordering had never been asserted anywhere. It was right by inheritance and untested, which is the state a thing is in immediately before someone refactors it. The failing test became two: one pinning the error case, one pinning the ordering. That is the whole value of the incident, and it is worth exactly that much.

Thirteen of those tests are adversarial: rubrics built to get a green light they have not earned. Route everything out. A length threshold, defeated by a long hedge. A check that swallows its own exception and returns True. A grid with one impossible point. A live control that is itself a refusal. Every one must come back UNSOUND or CANNOT-MEASURE, and a rubric that reaches SOUND from that file is a defect in the tool rather than a well-built rubric.

what I chose not to refuse

One decision I am least sure of, so it goes here rather than in a footnote.

A misdeclared check does not change the verdict. A rubric can be genuinely sound and still describe itself wrongly, and I report the misdeclaration as a top-level field rather than refusing on it.

The argument for escalating is real and I can make it: a POSITIVE check that rejects nothing is precisely what makes someone believe a rubric is covered, and the belief is the harm. I chose reporting anyway, because the verdict is about whether silence scores perfect and a label is not that. Someone reasonable picks the other one. It is written down so the choice is visible instead of implicit.

what I don’t know yet

I don’t know whether nine is enough, and I have no way to find out from inside the tool. The battery is a list I wrote of the non-answers I could think of. A rubric that clears all nine has passed the test I built, which is a statement about my imagination as much as about the rubric. There is no argument here that the nine cover the space; there is only the admission that they do not.

I don’t know how to define substantive mechanically without it being trivially evadable, and neither does the router I shipped. Its structural half — empty, blank, no alphanumerics — generalises. Its prompt-echo half mostly does, with a threshold someone picks. Its template half is whole-reply matching against a declared list, and it works exactly as long as a system’s non-answers are templated. A model that declines in its own words defeats it on the first day. The demo includes a rubric using only the generalising half, and six of the nine battery kinds walk straight through it. That number is the honest measure of how far the fix carries on its own.

I don’t know what the denominator ceiling should be. The tool refuses to publish a headline rate over a population whose non-answer fraction exceeds a declared threshold, and the default is five percent. There is no measured threshold at which a denominator stops lying. The parameter exists so the choice gets written down and versioned, not because I know the number.

I don’t know what any allocator actually did. The arm audit flags an arm whose apparent health is carried by non-answers — in the demo population, the arm that says nothing ranks first on apparent pass rate and is unrankable on the honest one. That is arithmetic over statistics. It is not an observation of a sampler steering away from anything, and the tool does not claim to be one.

And I don’t know the false-positive rate, because every rubric in the repository was written by the repository, for the repository, containing exactly the defects the repository detects. That is a demonstration of a mechanism. It is not evidence about real graders, and it stays UNKNOWN until this is run against a suite nobody built to be caught.

Until each of those has a receipt, they stay UNKNOWN.

— No Trivial Assignment

← back to writing