I trained a 500M-parameter base model — 7.7B tokens of public web, code and math text, perplexity 14.04 against a bar of 133.83, cost $127.69 on the trainer-time ledger. Those numbers are real and I can reproduce them. The estate-scoped branch they exist to serve hasn’t been trained yet; what follows is about the gate that would certify it. None of those numbers tell you if the model is any good at the one thing the finished model will exist to do.
The thing it will exist to do is know its own world. Not answer trivia. Not pass a general reasoning suite. Know which of its own deployed values is the real one and which is a plausible-sounding wrong one. Know who owns it. Know that exactly one script ships it to production and nothing else does. Know that when the document and the running system disagree, the running system wins. A model that scores well on a public leaderboard and gets any of that wrong has failed at its actual job, and the leaderboard will never tell you.
So the eval that certifies this model isn’t a benchmark. It’s a gate. And the gate has to be built, tested, and re-tested with the same discipline as the model itself, or the whole exercise produces a confident number that means nothing.
Why a generic benchmark can’t see the failure
I have two eval scripts. probe_battery.py is a generic multiple-choice probe — the kind of thing you’d point at any small model to sanity-check it hasn’t collapsed. estate_battery.py is a domain-specific gate that checks the model against this estate’s actual deployment facts: the real running value, not a plausible wrong one; the real owner class; the real voice-synthesis vendor; the fact that exactly one script is the sanctioned path to production. Both scripts consume the same JSON shape. They are drop-in replacements for each other in the harness.
That’s the part worth sitting with. Same interface, same plumbing, same “pass/fail” contract — and only one of them can see the failure modes that matter. A model can ace probe_battery and still get its own deployment facts backwards, because knowing general facts and knowing your own deployment are unrelated skills. The generic probe wasn’t built to test the second thing, so it doesn’t test it, no matter how good it looks doing the first.
This inverts the instinct most people bring to eval design: broader coverage, bigger benchmark, more general capability signal, better guarantee. For a model whose entire job is being coherent about one narrow, bounded world, that instinct is backwards. A benchmark that covers everything in general covers nothing in particular. The domain-specific gate is smaller, uglier, and the only one that certifies anything.
The corpus → train → eval loop, mechanically
None of this works if the training data itself is unreliable, so the corpus generator (estate_corpus.py) has one rule that overrides every other design choice: no randomness. It expands a fixed canon of facts into training examples using template variation only — the same fact rendered through different phrasings, selected by template index, never by a random draw. Feed it the same canon twice, get byte-identical output twice.
That sounds like a hygiene detail. It’s actually the thing that makes the eval’s result mean anything. If the corpus could vary silently between runs, a PASS today and a FAIL tomorrow would be uninterpretable — you wouldn’t know if the model regressed or the data just came out different this time. Determinism turns “run it again” into an actual control, not a hope.
Before any model trains on that corpus, the generator runs four controls against itself:
- coverage — every fact taught at least six times
- no-confab — corrected errors are absent from the generator’s own examples
- no-leakage — eval prompts never appear verbatim in training data
- eval-coverage — every fact the gate checks was actually taught
The rule underneath all four is blunt: you cannot gate what you did not teach. If estate_battery checks a fact the corpus never covered, that’s not a hard eval, it’s a coin flip dressed as a test.
Self-testing the tester
Here’s the part most eval scripts skip. Before I trust estate_battery to grade a real model, I run a known-good “coherent” reference model through it first — a model I already have independent reason to believe gets the facts right. If that model fails the gate, I don’t treat it as a model bug. I treat it as an instrument bug. Something in the eval script itself must be wrong, because I already know this input should pass.
shaurya@sps ~ % python3 estate_battery.py --self-test
[gate] coherent-model self-test ... FAIL (7/8 facts)
[gate] fact D3 flagged as confabulation — investigating harness, not model
That inversion is the whole point. Most eval scripts get written once, pointed at the model under test, and trusted the moment they produce a number. Nobody runs a deliberately-correct input through their own grader to see if the grader can recognize correct when it sees it. This corpus does that first, every time a battery changes, before a single real training run gets graded by it.
It’s how the next bug actually got caught.
The must_contain vs. must_not_contain doctrine
The self-test above wasn’t hypothetical. Running the coherent model through a fresh estate_battery draft produced a real failure, and the failure exposed a design flaw in how the gate was checking for confabulation.
The gate had a no-confabulation check: if the model’s answer contains a specific wrong-answer marker, fail it. That worked fine until a fact whose correct answer legitimately contains the string that another fact treats as its confabulation marker. The correct answer to a deployment-value question is something like “it’s the real value, not the wrong one” — the model correctly naming and rejecting the wrong value in the same sentence, where the wrong value happens to be the literal string another fact’s check is scanning for. A global must_not_contain check flags that as a failure, because it never asked which fact it was grading against. It just pattern-matched a string.
The same shape of bug hit the corpus generator independently: a term that’s the correct answer for one fact (a vendor name tied to one system) is exactly the confabulation marker for a different, unrelated fact in the same estate. Same string, opposite verdict, depending on which fact you’re asking about.
The fix in both places is identical: every check gets scoped to its own fact’s context. There is no global string-match version of “is this true.” must_contain — does the answer include the specific correct marker for this specific fact — turned out to be reliable in a way must_not_contain never fully is, because negation checks assume the flagged string is always wrong, and it isn’t always wrong. It’s only wrong in the context of the fact you didn’t mean to be checking.
# before — global, and false-flags a correct answer
must_not_contain: ["<wrong-value-for-fact-B>"]
# after — scoped to the one fact it actually grades
fact: "deployment_value"
must_contain: ["<correct-value-for-fact-A>"]
A term correct in one context is an error in another. That’s not a special case in this eval. It’s the default condition any fact-grading check has to assume from the start.
The dumb-baseline validity trick
Before trusting the eval to say anything about the real model, I wanted to know the corpus itself was sufficient — that the facts were actually gradeable, independent of whether my model is any good. So I trained a deliberately dumb baseline: a plain word-overlap retriever, no learning, no weights that understand anything, running on the same generated corpus.
It passed. All facts the gate checks for retention came back correct on the dumb retriever. That result isn’t impressive on its own — a word-overlap retriever passing a fact-recall check should be unremarkable. What it proves is narrower and more useful: if the corpus were empty, wrong, or missing coverage, even a dumb retriever would fail, because there’d be nothing correct to overlap with. It passing means the data teaches exactly what the eval checks. That isolates one variable cleanly — data sufficiency — from a completely different question: is the real model any good at using that data. You can’t answer the second question honestly until you’ve closed off the first as a confound.
Wall-1
The training corpus for the judge side of this pipeline includes several thousand refinement rows authored by a frontier model — Claude, specifically, working as a scoring assistant during data refinement. Those rows are walled: they may feed a discriminative judge head, and only the judge head. They never touch the generative trunk. The trunk that actually produces this model’s output stays provably clean of frontier-authored text.
That distinction — judge versus generator — is not a formality. A judge that scores candidate outputs and a model that generates them are different functions even if they share a checkpoint, and keeping frontier-model influence contained to the scoring function is what lets me say the trunk’s outputs are the estate’s own words, trained on the estate’s own facts, not laundered frontier text with a small model’s name on it.
The open risk here is real and I’m stating it plainly rather than pretending it’s closed: best-of-n reranking, where the judge head picks the best of several trunk-generated candidates, could quietly turn the judge back into something closer to a generator. If the judge is doing enough of the actual selection work, “the trunk generates, the judge only scores” stops being a clean line. That needs its own ruling, and I don’t have one yet.
Why this generalizes past one estate
None of this is specific to running a private assistant. Any team fine-tuning a small model on a bounded, specific world — a company’s internal state, a product’s own configuration, a support system’s actual account data — is going to reach for a public benchmark first, because that’s the default instinct and the tooling is already built for it. The public benchmark will tell you the model can do arithmetic and follow instructions. It will not tell you the model knows which environment it’s talking to, which values are current, or which document it should distrust in favor of the running system.
Build the domain gate before you reach for the leaderboard number. It’s smaller, it’s uglier, it doesn’t transfer to a paper, and it’s the only one of the two that’s actually testing the job the model was built to do.
What I don’t know yet
The judge-head wall — frontier text scores but never generates — holds today because reranking is simple. I don’t know if it holds once reranking strategies get more aggressive; that’s an open ruling, not a solved one, and I said so above rather than papering over it.
I also don’t know if “coherent about its own world” scales past one person’s estate. Everything here is built and proven against a single owner, a single deployed system, a single canon of facts. Whether the same corpus-eval-tester discipline holds up for a team’s shared world — more owners, more disagreement about what the canonical fact even is — is UNKNOWN. I haven’t built that version yet, and I’m not going to claim it works until I have.
— From the Estate