A composite similarity score is an average of things it claims to measure separately. Jaw, nose, eyes, hair. Clarity, correctness, structure. Each weight is a bet that the feature underneath carries information the others don’t, and nobody tests that bet by looking at the final number — the final number is what hides it. A wrong jaw and a wrong nose cancel into the same total as two right ones.
The only way to find out is to break one axis alone, hold the rest fixed, and watch whether the score moves on that axis alone. If you swap the jaw and every sub-score shifts, you haven’t found seven measurements. You’ve found seven names for one signal.
axisproof generalizes that past faces to any scorer claiming more than one axis: rubric graders, embedding similarity, plagiarism detectors. A scorer that fails a case built specifically to expose it is convicted by its own control — the evidence standard the practice came with.
The mechanism
A manifest declares four thresholds and, per axis, a swap pair that must differ from its base on exactly one axis. The loader rejects a swap pair changing two axes or none, because that invariant is the only thing making the third control mean anything. Then four controls run: self-identity, cross-identity, single-axis independence, and no-free-rider — the last checking that the composite reacts to the swap, since an axis the composite ignores is a declared axis the scorer silently doesn’t have.
Here is the harness catching the demo’s second scorer, one built blind to hair. The demo’s first scorer, the report header and signature, and hair’s three clean sibling axes are all elided:
shaurya@sps ~ % python examples/demo.py
[global] self_identity : PASS -- self-pair scored 1.0000 (>= 0.9000 required) -- held up under its own self-identity control
[global] cross_identity : PASS -- cross-pair scored 0.3625 (<= 0.4000 required) -- correctly told apart
axis 'hair' -- REFUSED (coverage: native)
REFUSED axis_independence:hair: hair sub-score moved 1.0000 -> 1.0000 after swapping hair alone (<= 0.5000 required) -- convicted by its own control: hair sub-score did not drop; scorer may be axis-blind on hair
REFUSED composite_degrade:hair: composite moved 1.0000 -> 1.0000 after swapping hair alone (<= 0.9000 required) -- convicted by its own control: composite ignored the hair swap; hair is a free rider in the composite
PASS axis_stability:hair: other axes stayed within the ±0.0500 stability band while swapping hair
OVERALL: REFUSED
That third line is not padding. The blind scorer is blind, not cross-wired, so nothing else moved when hair did — stability holds, and the two controls that convict it are the two that had something to see.
The nastier fixture is the axis-swapped scorer, which mislabels hair as eyes. It is blind to nothing — both axes are measured, just wired to each other’s names — and the stability half of the control is what names the disease: swapping eyes also moved other axes outside the ±0.0500 stability band (hair 1.0000 -> 0.1500) -- axes are not cleanly separated. A naive independence check alone would have let that through.
The audit
A control can also fail to happen. When a scorer returns a bare composite float and no per-axis breakdown, axisproof falls back to ablation — swap one axis, watch the single composite number move. That infers something, but the stability half of control 3, “the other axes must not drift,” has nothing to observe: under ablation there is nothing per-other-axis to look at.
Until 2026-09-14 that unperformed check had nowhere to go. The axis rollup selected only passed is False as a refusal, so a control that returned None fell through to the same branch as one that ran and held. examples/prefix_rule.py ships that old rollup so the regression stays runnable. Re-applying it to today’s controls, against a scorer with nothing wrong with it:
shaurya@sps ~ % python examples/prefix_rule.py examples/ablation.yaml; echo "exit=$?"
axis verdicts (pre-fix rule): PASS, PASS, PASS, PASS
controls never performed: 4 of 14
OVERALL (pre-fix rule): CERTIFIED
exit=0
A green light over four checks that were never run, emitted by the package whose entire thesis is that a check you have never watched fail is not a control yet. The exact failure this thing exists to condemn, sitting inside the thing.
The fix is a third verdict. An unperformed check is not a pass: any axis carrying one is VOID, any report with a VOID axis is VOID, and REFUSED outranks VOID because a scorer convicted by a control that really ran is a defect in the subject, more actionable than a gap in the instrument. Exit codes are 0/1/2, the same contract gatehealth ships for PASS / FAIL / CANNOT-MEASURE:
shaurya@sps ~ % python -m axisproof.cli certify --manifest examples/fixtures.yaml --scorer tests.fixture_scorers:correct_scorer; echo "exit=$?"
OVERALL: CERTIFIED
exit=0
shaurya@sps ~ % python -m axisproof.cli certify --manifest examples/fixtures.yaml --scorer tests.fixture_scorers:axis_blind_scorer; echo "exit=$?"
OVERALL: REFUSED
exit=1
shaurya@sps ~ % python -m axisproof.cli certify --manifest examples/ablation.yaml --scorer tests.fixture_scorers:composite_only_correct_scorer; echo "exit=$?"
OVERALL: VOID
4 of 14 controls could not be performed and were not counted as passes: axis_stability:jaw, axis_stability:nose, axis_stability:eyes, axis_stability:hair
VOID is not a pass and not a refusal. axisproof did not gather enough evidence to say either; a caller must treat this as uncertified.
exit=2
The list of unperformed controls is a top-level field on the report, not only a line of rendered text, because the reader who most needs to see that a check was skipped is the CI job parsing the JSON. That third scorer is correct on every axis it is asked about, and still cannot be certified — the honest reading, not a bug to soften.
What it can’t do yet
- A scorer with no native breakdown can never be CERTIFIED. Every axis carries one unperformed control, so every ablation run is VOID at exit 2 however well the scorer behaves. Expose a per-axis breakdown, or accept that the verdict is “not measured.”
- Under ablation, controls 3 and 4 are one observation read twice. Both compare the same post-swap composite against a ceiling; only the threshold differs. With the swap threshold at or below the degrade threshold, no-free-rider adds nothing the independence check didn’t already know. Two controls, one instrument.
- The shipped manifest’s thresholds don’t survive the switch to ablation. A single-axis swap moves the composite by roughly
1/num_axesof what it moves that axis’s own sub-score. Pointed at the shipped four-axis fixture, a correct composite-only scorer is REFUSED at exit 1 — composite 0.7875 against a swap threshold of 0.5 tuned for native sub-scores. Loosening it to 0.85 is what produces the VOID run above. - The ablation module isn’t in the certification path.
ablation.pydocuments a sensitivity epsilon separating “no reaction detected” from floating-point noise. Nothing imports it except its own test file; the real ablation checks are inline in the controls module and never consult it. That documented threshold influences no verdict. - Per-axis self/cross overrides are parsed and then ignored. The format lets an axis declare its own self_pair and cross_pair, and the loader validates them; certification only ever evaluates the first axis’s pairs. I planted a self_pair on the second axis that a correct scorer should have failed outright. The run came back
self_identity : PASS,OVERALL: CERTIFIED, exit 0. - The report’s signature is a checksum, not an identity. A sha256 over the report’s own content, proving it wasn’t hand-edited afterward. No key management, no non-repudiation claim.
- Manifests are hand-authored, never searched for. It checks the sabotage cases you declared; it does not go looking for one that breaks you. A scorer that clears every control in a weak manifest has been shown weak evidence.
- It has never been pointed at a real model. Every scorer it has certified or refused is synthetic, built here to have a known defect. Whether these thresholds survive contact with a real embedding model or rubric grader is unmeasured.
A hundred and four tests now cover the three verdicts, each control’s conviction path, the single-axis-diff invariant, ablation coverage, the exit-code contract and the signature. Ninety-four existed before the audit. The ten new ones exist because a harness built to refuse unearned green lights handed out one of its own, and a suite that grows only when you add features has never been told anything it didn’t want to hear. That is a count taken today against this source — including, now, the case that catches the bug ninety-four passing tests were happy to sit beside.
— Shaurya