Shaurya Singh

← work

worldbattery

A deterministic gate for whether a small model knows its own deployed world — not a benchmark, a gate that refuses to trust itself first.

prototype · 2026-09

A generic multiple-choice benchmark and a domain-specific gate can share the exact same interface and still test completely different things. A model can ace a leaderboard and still get the owner of its own deployed service backwards, or confuse the deploy script with the rollback script, because “knows general facts” and “knows its own deployment” are unrelated skills. worldbattery is the small, ugly, domain-specific version: a library for building a deterministic fine-tuning corpus out of a bounded set of facts, and grading a model’s answers against that same world.

It’s not trying to transfer to a leaderboard. That’s the point.

The shape of a world

You write a world.yaml: facts, their correct terms, and per-context wrong terms that count as confabulation.

facts:
  - id: owner
    statement: "toy-service is owned by the Platform team."
    context_tags: ["ownership"]
    correct_terms: ["Platform team"]
    wrong_terms_by_context:
      ownership: ["Data team", "Security team", "Northwind"]

build expands every fact into N phrasings with zero RNG — same file, same N, byte-identical output, forever, on any machine:

shaurya@sps ~ % worldbattery build --world examples/toy_service/world.yaml -n 8 --out corpus.jsonl
[build] wrote 48 records (6 facts x 8 phrasings) to corpus.jsonl
shaurya@sps ~ % worldbattery build --world examples/toy_service/world.yaml -n 8 --out corpus2.jsonl
shaurya@sps ~ % cmp corpus.jsonl corpus2.jsonl && echo "byte-identical: YES"
byte-identical: YES

The doctrine underneath grading: nothing is a global banned word

A term correct in one context is an error in another, and a fact-grading check doesn’t get to treat that as a special case — it’s the default condition. So worldbattery never builds a global banned-word list. For each fact, the wrong-term set is computed by taking that fact’s own context’s wrong terms, then removing anything that’s actually correct here, anything that’s ambiguous because it’s a correct term for a different fact sharing a context, and anything the fact’s own true statement already says while rejecting it.

The shipped example deliberately entangles a deploy_script and rollback_script fact where each one’s correct answer is the other’s declared wrong answer — the exact collision I hit writing the source article this library generalizes from — and two more facts whose true statement names and rejects their own wrong answer in the same sentence. A correct answer is allowed to say “the running system wins, not world.yaml” without tripping its own confabulation check. must_contain has no such caveat — negation checks assume the flagged string is wrong; a check for presence doesn’t need the same hedge.

The gate refuses to trust itself first

selftest won’t certify a battery until three fixtures and one control probe all pass in the same run: a perfect lookup model scores 100%, a blank-answer model scores 0%, a confused model scores low without crashing the scorer — and a dumb word-overlap retriever, no weights, no learning, has to pass outright. That last one is load-bearing: it isolates “is this corpus even answerable” from “is the real model good,” and if the dumb baseline fails, no real model’s score means anything yet.

shaurya@sps ~ % worldbattery grade --world examples/toy_service/world.yaml --answers answers.jsonl
VOID
[grade] no selftest receipt found — run `worldbattery selftest` first

shaurya@sps ~ % worldbattery selftest --world examples/toy_service/world.yaml
[selftest] perfect                    score=1.00 (expect == 1.0) ... PASS (6/6 facts)
[selftest] blank                      score=0.00 (expect == 0.0) ... PASS (0/6 facts)
[selftest] confused                   score=0.00 (expect <= 0.34) ... PASS (0/6 facts)
[selftest] baseline (control probe)   score=1.00 (expect == 1.0) ... PASS (6/6 facts)
[selftest] ALL FIXTURES PASSED — battery is trustworthy for this world
[selftest] receipt written to examples/toy_service/.worldbattery_selftest.json

grade won’t produce a score without a selftest receipt that exists, matches the current world.yaml byte-for-byte, and recorded a pass. No receipt, a stale receipt, or a failed one — all print VOID. A number from an untrusted instrument is worse than no number.

The context-collision tests weren’t contrived after the fact, either. Writing the shipped example’s world.yaml naturally reproduced the exact bug the doctrine is meant to catch — once across two facts, once within a single fact rejecting its own wrong value — and I found both by running selftest against the first draft and watching the perfect-model fixture fail, not by designing them in from a spec.

What it can’t do yet

  • There’s no fine-tuning here. build produces a corpus shape, not a trained model. grade grades whatever answers you hand it — a JSONL file or a callable — it doesn’t run inference.
  • No coverage/no-confab/no-leakage control battery over the corpus generator itself. Things like “every fact taught at least six times” need a real training pipeline behind them to mean anything; this library stops at the corpus.
  • Only ever run against a single, small, single-owner world. Whether the doctrine holds for a world with multiple disagreeing owners is untested and unknown.

Forty-one tests cover the determinism claim (direct calls, fresh reloads, a real CLI subprocess round-trip), the context-scoping doctrine’s collision cases, all four selftest fixtures — including forcing an instrument bug via monkeypatching to prove selftest actually refuses to certify a broken battery — the VOID rule, the dumb baseline in isolation, and callable-model grading.

— Shaurya

← back to work