A right/wrong grader has two bins. A few-shot sheet produces three behaviours: computed and got it right, computed and got it wrong, and did not compute at all. That third one lands in “wrong” usually — except when the string a weak model echoes back happens to equal gold, and then it lands in “right.” A model that computed nothing and one that got it right become the same row.
I wrote about that failure in the sibling essay to this tool. This is not that essay — this is the grader built so the failure has somewhere else to go. Every command below was re-run today, 2026-09-19.
six outcomes, not two
echograde grades every item into one of six states: CORRECT (matched gold, not sitting in the prompt), INCORRECT (a real attempt that missed), ECHOED (a verbatim string came back, nothing computed), ECHOED_GOLD (an echo that happened to equal gold — not a success), NO_OUTPUT (nothing produced), and UNSCORABLE (refused before scoring — gold was reachable from what the model could see). The last two leave the denominator: a sheet that refuses eighteen items reports eighty-two scored, not a hundred with eighteen zeroes folded in.
the refusal runs before any model exists
Instrumenting copying after the fact is a stopgap — it string-matches an answer against exemplars in the prompt, and it’s always one paraphrase behind. So before any response exists, echograde asks a sheet-level question: can gold be obtained from what’s visible, without doing the task? Four built-in strategies check this — a constant every exemplar shares, an exemplar’s own answer, a field in the request, or gold’s token run occurring anywhere in the visible text — plus any transform you declare yourself.
Nothing in this package calls a model; every sheet and response below is one I constructed. I built a three-item sheet where one item’s gold equals its own exemplar’s answer:
shaurya@sps ~ % echograde audit sheet.json
leak audit: 3 items, 1 reachable from the prompt
a2: exemplar_answer: exemplar[0].answer == gold
That item never reaches a grader, refused on the sheet’s shape alone.
an echoed hit is not a hit
Grading needs responses, so I wrote three by hand: a1 correct, a2 echoing its own exemplar’s answer (also gold — why it’s leaked), a3 echoing an exemplar’s answer that isn’t its gold. Refusal on, the leaked item never enters the denominator:
shaurya@sps ~ % echograde grade sheet.json resp.json --arm xl
arm xl: 3 items, 2 scored, 1 refused, 0 no-output
raw accuracy 50.0% (1/2, what right/wrong reports)
net accuracy 50.0% (1/2, echoed hits removed)
copy rate 50.0% (1/2)
REFUSED 1 item(s): gold reachable from the prompt
a2: refused: exemplar_answer: exemplar[0].answer == gold
--allow-leaks scores that item anyway, for a legacy sheet. It doesn’t buy a clean result:
shaurya@sps ~ % echograde grade sheet.json resp.json --arm xl --allow-leaks
arm xl: 3 items, 3 scored, 0 refused, 0 no-output
raw accuracy 66.7% (2/3, what right/wrong reports)
net accuracy 33.3% (1/3, echoed hits removed)
copy rate 66.7% (2/3)
1 hit(s) were echoes of a visible string and are NOT credited
CANNOT SCORE: 1 leaked item(s) were graded anyway; the sheet is not measurable as built
Same three items, same three responses: raw accuracy says two-thirds, net says one-third, and the only difference is whether an echo that landed on gold counts as a hit.
a chance floor has to be able to lose
A margin over a control means nothing if the control can’t fail on most of the sheet. chance_floor scores a degenerate baseline against the real key, item by item, then checks whether it’s capable of scoring in every category. The package’s own demo builds a constructed 100-item, ten-category sheet for exactly this, nine categories with a unique gold value per item, and points a constant-answer baseline at it:
shaurya@sps ~ % python examples/demo.py
[... header, parts 1-3 and part 4's title elided ...]
control constant_answer: 10/100 = 10.0% against this answer key
categories it can ever score on: ['cat0']
categories it is structurally incapable of scoring on: 9
[... the arm's own report elided; it scored 10/90 net, 90 of 100 items ...]
capability claim for arm: REFUSED
observed 10/90 net hits (11.1%)
control constant_answer expects 10/100 (10.0%) against this answer key
control is structurally incapable of scoring on 9 categor(ies) (...); its rate is a property of the answer key, not a chance floor
[... output continues with the majority-gold floor; elided here ...]
That REFUSED fires on degeneracy first. But setting degeneracy aside, the control was measured over all 100 items while the arm scored only 90 — ten refused by the leak audit — and capability_claim would refuse that denominator mismatch too; it just checks degeneracy first, by design. The 10.0% is a property of this demo’s own answer key, not a number from any real model.
a band that knows when it can’t be drawn
I’d been reading a 2σ band computed as 2·SE(accuracy) on a single arm. A difference of two proportions needs the wider standard error of the difference, and below a certain count the normal approximation isn’t valid at all:
shaurya@sps ~ % python3 -c "
from echograde import ProportionDiff
d = ProportionDiff(10, 100, 2, 100)
print(round(d.naive_two_sigma, 4), round(d.two_sigma_unchecked, 4))
d.two_sigma"
0.06 0.0662
Traceback (most recent call last):
[... three lines elided: two frames and a source line; one names an absolute path on this machine ...]
echograde.types.ApproximationInvalid: normal approximation invalid (arm2: n*p = 2 < 5); use fisher_exact_two_sided
The single-arm number and the correct one are close — 6.0 points against 6.6. With n·p = 2 on the control arm, reading a band off the normal approximation at all is the defect, not the size of the miss. stats.py raises instead of printing a number it can’t stand behind, and points you at Fisher’s exact test, which can.
the verifier that found itself wrong
The build was reported at 151 tests. An independent pass reran the suite and threw adversarial probes at it, finding four real bugs — one of them the failure this package exists to forbid: a default path where a pure copy out of the prompt scored CORRECT. Two canonicalisations had drifted apart — the matcher stripped thousands separators, the audit’s tokenizer didn’t, so a gold of 1,234 and a prompt containing 1234 looked identical to one and different to the other. All four got fixed and pinned by regression tests; the suite is 167, not 151. Sixteen of those tests are adversarial, written to break the tool rather than ship it, and seven of the sixteen failed the day they were written — confirmed by restoring the old behaviour and watching the assertion catch it.
shaurya@sps ~ % cd echograde && pip install -e ".[dev]" -q
shaurya@sps echograde % pytest -q | tail -1 | cut -d' ' -f1-2
167 passed
Python 3.12.4, re-run 2026-09-19.
echograde is a standalone package, not wired into the harness that produced the sibling essay’s real numbers — that essay says outright it has no record its owed fix ever shipped there. This is that fix, built as its own tool, not proof it reached the run that motivated it.
what it can’t do yet
- It sees values, not shapes. It catches
89returned verbatim, not a model lifting the right shape of an answer around a value that was never computed. - The audit only covers strategies you declare. A gold value reachable by an undeclared transform passes clean — a test asserts exactly that.
verbatimover-triggers on purpose. A gold of16is “reachable” from a prompt that only mentions a date like2026-03-16. Losing an item to that false alarm costs a row; keeping one whose answer really is in the prompt costs the whole claim.- The degeneracy check is per category. A sheet authored with one category defeats it; the check only sees as much as the category column lets it.
- I don’t know what the refusal costs. On a sheet built without this in mind,
verbatimcould refuse a large fraction of items, and I have no measurement on a sheet someone else built. - An item is trusted to describe its own prompt. Nothing checks that a sheet on disk matches what a model was actually shown — the largest unguarded seam in the design.
capability_claimhas no multiple-comparison correction. Testing across ten categories and reporting the best one can manufacture a false positive, and nothing here stops you.- It can’t see contamination through pretraining. If the model memorised the benchmark, gold was never in the prompt, and this reports a clean sheet regardless.
- Matching is string equality after normalisation, not semantics.
"eighty-nine"and"89"don’t unify, which under-counts both hits and echoes on free-form generation. - Nothing here has been pointed at a sheet I didn’t build. Every sheet in the demo and tests is one I constructed; the only leak it found that I didn’t plant was a coincidence in the demo’s own sheet. What it finds on a sheet somebody else built is unmeasured.
- I haven’t run this against a live eval end to end. It comes from a defect in a harness I’d already written, not from watching
echogradecatch something I didn’t already know was there. Until it surprises me once, it’s a formalisation of a lesson, not an instrument.
— Shaurya