Shaurya Singh

← work

CiteGuard

Legal AI still mis-cites: cases that don’t exist, cases that don’t say what the brief claims, each one a sanction risk with a person’s name attached. CiteGuard is the check that lives outside the model — every citation passes through the rail, the tool can’t route around it, and the skip flag is the one feature it will never ship. It’s a v0.3 prototype with no measured accuracy yet; when a number exists that I can stand behind, it’ll be the first thing on this page.

prototype v0.3 · verify --skip: refused, by design · accuracy: not yet measured

prototype · v0.3

Grounded is not the same as right.

Legal AI has largely moved to grounded, retrieval-backed generation — and it still mis-cites. Cases that do not exist. Cases that exist but do not say what the brief claims they say. In most fields a wrong reference is an embarrassment; in a filing it is a sanction risk with a person’s name attached. That is not an error rate anyone gets to absorb quietly.

And the ground is shifting under the tools: jurisdictions are starting to require that citations be certified before they are filed. The moment that lands, “the model seemed confident” stops being a workflow. Someone has to be able to say, of every citation, this was checked — and mean something specific by it.

A process the tool cannot skip.

The failure I care about is structural, not just statistical. When the check lives inside the same tool that produced the citation, it is a feature — and features get toggled off by whoever is closest to a deadline. CiteGuard’s bet is that verification has to be a rail instead: every citation passes through it, the tool cannot route around it, and the output either carries the certification or visibly does not.

This is the same conviction the correspondent layer’s walls already tested on me: the check has to hold against whoever is closest to the deadline — including, especially, the person who wrote it.

“Does it exist” is the easy question.

The public story about legal AI is fabricated cases — a citation to something that was never decided. That failure is real, and it is also the least interesting one, because it is the only one a database lookup solves. Existence is a solved check. It is not the check that matters.

A citation can be wrong in at least four graded ways, and they get progressively harder to catch:

shaurya@sps — citeguard/failure-modes
fabricatedthe case does not exist — a lookup catches it
misattributedreal case, wrong reporter, year, court or party
stalereal and correctly cited — and overruled, superseded, or narrowed since
unsupportingreal, current, correctly cited, and it does not say what it is cited for

That last one is the whole problem. A citation that is real, current, perfectly formatted and simply does not stand for the proposition it is attached to will pass every mechanical check ever written, and it is the one that loses the argument in front of a judge. Catching it is not a lookup — it requires reading what the case actually held and comparing it to the claim being made.

Which is precisely why the rail sits outside the model. You cannot ask the system that asserted the proposition whether the proposition is supported; it will agree with itself, fluently, and its agreement carries no information. The check has to be performed by something with no stake in the answer — and it has to report a graded verdict rather than a green tick, because "verified" collapsing four distinct failure modes into one boolean is how a tool ends up certifying the exact error it was built to prevent.

No numbers until they’re measured.

You will notice this page has no accuracy percentage, no pilot customers, and no list of jurisdictions won. That is because it is a v0.3 prototype and those things have not happened — a strange omission for a pitch, and exactly the right one for a project whose entire premise is that unchecked claims should not survive contact with a filing. When there is a measured number, it will appear here with the date it was measured on. Until then, the page stays short, because padding is just fabrication with better manners.

← back to work or ask me about it →