Shaurya Singh

← work

nullpass

A rubric auditor that checks whether an empty reply can score perfect — and that got caught certifying a check that never read the reply at all.

prototype · audited and fixed 2026-09-18

A rubric is a conjunction of checks, and each was built to catch what a reply does wrong. Invented a role. Named someone. Guessed a count it couldn’t know. Ran past a latency budget. Each can genuinely go red, and each looks sound on its own. None ask what happens when a reply does nothing at all.

nullpass came out of one quality loop where that gap was real: 1,324 graded runs, 189 replies that said nothing. No content check in that rubric ever flagged one; the empty string fails none of them. A 12-second latency check caught 187 of the 189 by accident, because the slow path they came from tripped its budget — the other two came back fast and scored a clean 7 of 7. The conjunction had a trivial satisfying assignment, and normal operation occupied it 189 times before anyone found it in a design review.

The general question, stated so it can be run rather than argued: build the reply that says nothing, run it through the rubric, and check whether it comes back clean. If it does, the rubric measures abstention, not quality, and whatever sits above it — a sampler, a person skimming a pass rate — learns that abstaining scores highest.

The battery and the two claims a check makes

nullpass doesn’t search the whole space of non-answers. It runs a fixed battery of nine replies with no claim about anything — empty string, whitespace, punctuation, a refusal, an outage line, a hedge, the question handed back, an acknowledgement, a deflection — through every check, and reports which score perfect.

Each check declares two things, and the audit doesn’t take either on faith. Polarity: NEGATIVE (fails when the reply does something) or POSITIVE (fails when it omits something). Subject: content (reads the reply) or circumstance (reads latency, retries, cost). A POSITIVE check that rejects none of the nine battery replies is misdeclared — its failure set contains no non-answer, so it’s a prohibition whatever its name says. A content check whose verdict never moves while the reply text changes hasn’t been shown to read the reply at all.

Here’s the audit against a rubric of six well-built negative checks, run on the demo battery:

shaurya@sps ~ % cd nullpass && pip install -e ".[dev]" -q
shaurya@sps nullpass % cd examples && nullpass audit --rubric rubrics:negative_only | tail -4; echo ${pipestatus[1]}
9 perfect · 9 rejected · 0 routed out · 0 errored
VERDICT: UNSOUND
  degenerate-admitted: acknowledgement, deflection, degrade, empty, hedge, punctuation, refusal, restatement, whitespace scored perfect in 9 of 18 battery cells
  This rubric measures abstention. No aggregate score will be emitted from it.
1

Nine reply kinds at two latency points, half the cells clean. The rubric that catches invented roles and bad status has nothing that notices a reply with no content in it.

A clock is not a defense

Make that slow path fast, a change anyone would be praised for, and the rubric’s only defense against silence disappears.

So a rubric can declare a circumstance grid, and the battery runs at every point of it. A degenerate admitted at even one grid point counts as admitted — the question is existential. Point the same six checks — five content checks plus that one latency check — at a grid with only a slow point, and every non-answer gets rejected there too, but the verdict still isn’t SOUND. It comes back CANNOT-MEASURE, reason circumstance-only-defence: every rejection came from the latency check alone, never a content check, so the content checks are shown admitting the same nine replies outright. The thing doing the stopping is a timer, not the rubric.

The router needs its own control

The actual fix isn’t a seventh check — it’s a classifier that runs ahead of the rubric and routes a non-answer out of “answer quality” entirely, into a bucket that’s counted, never averaged into a pass rate. That invites the obvious cheat: route everything out, and the battery has nothing left to admit.

nullpass closes that by requiring at least one live control — a substantive reply the author declares must be scored and must pass. Strip the controls off negative_only and it’s still UNSOUND, exit 1 — its admitted degenerates already convict it first. Strip them off a router rubric that would otherwise route every cell out cleanly, though, and there’s nothing left to convict it, so it drops to CANNOT-MEASURE, reason no-live-control, exit 2: rejecting everything isn’t soundness until something real has survived. A router that eats a declared live control comes back UNSOUND instead.

The scorer declines, and so does the denominator

score() returns a verdict and an aggregate that’s None unless the rubric is SOUND. require_aggregate() raises instead of returning a number computed over abstention — no warning printed above it, because a warning gets read once and a number gets read forever.

The second mechanism, audit_denominator, does the same thing to a headline pass rate — a fraction that lies the moment its bottom half contains silence. The demo ships a synthetic population built to that same 1,324-run, 189-non-answer shape — arithmetic in the fixtures, “constructed, not observed,” never a live log. Pointed at it, the audit counts 189 non-answers against a declared 5% ceiling, prints HEADLINE RATE: withheld, and comes back UNSOUND, reason denominator-contains-silence. Withholding the number without a replacement would just be an obstacle, so substantive_rate — the rate over turns that actually contained an answer, 91.4% here — always comes back regardless.

A third mechanism, audit_arms, points the same idea at a multi-arm setup: an arm that looks best by apparent pass rate can be the one that answered nothing. One demo arm sits at 100% apparent, 100% silence — unranked, not last, because zero answered turns is unmeasured, not disqualified.

The audit that found itself wrong

The first build reported 111 tests, then went through an independent pass that didn’t write it — every source file read, the suite rerun clean, the demo run twice for byte-identity, fourteen new adversarial probes thrown at it. Two real defects came back. A check declared subject=content that actually read the response object, or the prompt, got certified SOUND — the worst wrong answer this tool can give, since the nine battery texts can’t tell “read all nine and rejected them” from “read none of them.” And every documented --rubric invocation in the README exited 2, since the console script didn’t put the working directory on sys.path. Both fixed, both pinned by regression tests. The suite that came out the other side is 123 tests, not 111.

The suite runs 123 passed on Python 3.12.4 (re-run 2026-09-19 UTC), after pip install -e ".[dev]":

shaurya@sps ~ % cd nullpass && pytest -q | tail -1 | cut -d' ' -f1-2
123 passed

What it can’t do yet

  • The battery is a list I wrote, not a proof. Nine reply kinds, not all of them. No battery text contains a digit, and the longest is 78 characters, so the cheapest positive checks anyone reaches for — “states a number”, “is at least N characters” past 78 — both certify SOUND here and both lose to one honest sentence that happens to contain a digit or run long. That defeat is asserted right alongside the SOUND verdict it beats.
  • A SOUND verdict is a claim about nine replies, not the rubric. It means these nine don’t get through — not that nothing empty ever will.
  • The router’s weakest part generalizes least. Structural detection (empty, whitespace, no alphanumerics) holds up. Template matching works only while non-answers stay templated, and a model that declines in its own words walks past it on day one.
  • The 5% degenerate-fraction ceiling is a number I picked, not measured. There’s no threshold at which a denominator provably stops lying; the parameter just makes the choice explicit and versioned instead of silent.
  • The arm audit reads statistics, not an allocator’s real behavior. It can show an arm’s apparent health is propped up by silence, not what any real sampler did about it. It also only flags a carried number: an arm that’s 90% silence whose ten answered turns all pass comes back SOUND and ranked, since nothing about it is propped up — that it barely answers at all is audit_denominator’s job, and the two audits don’t talk to each other.
  • score() gates on the rubric, never the population. A SOUND rubric emits aggregate over whatever survived routing, so a run where 99 of 100 turns were routed out still publishes a number over the one that wasn’t. The routed count prints alongside it, but the number isn’t withheld — that’s audit_denominator’s separate call, and nothing forces it.
  • A misdeclaration doesn’t change the verdict. A sound rubric describing itself wrongly is still reported SOUND, not refused — even though that’s exactly what makes someone believe a rubric is covered when it isn’t.
  • Nothing here has been pointed at a rubric I didn’t build. Every rubric this package has certified or refused was written by this repository, containing exactly the defects it’s built to find. What fraction of real rubrics are UNSOUND stays UNKNOWN until it runs against a suite nobody built to be caught by it.

— Shaurya

← back to work