prototype · trunk trained and measured — coherence gate proven on a stand-in, not yet run against the real target — 2026-09
Everyone chasing a frontier model is answering the same question: what can this thing do, in general, for anyone. That question is expensive to answer honestly, and I have the retraction to prove it. There’s a second question that almost nobody asks, because it sounds smaller: does this thing know one specific world, for one specific person. That question turned out to be the cheap one, and the honest one, and I think it’s a different kind of artifact entirely.
the trunk, plainly
I trained a base model from scratch: roughly 500M parameters, a 7.7B-token corpus, no chat template, no instruction tuning. Perplexity came in at 14.04 against the 31.6M-parameter baseline’s 133.83 (an untrained model scores ln(vocab) = 9.704 nats, a different number altogether). Total cost: $127.69 on the trainer-time ledger, against a $140 budget — the account’s billing meter, reconciled the same day across five pods, read $164.22 against a trainer ledger of $135.84; that reconciliation is documented elsewhere in this corpus.
params: ~500M
corpus: 7.7B tokens
perplexity: 14.04 (bar: 133.83)
cost: $127.69 (ledger; budget: $140)
Those four lines, with the cost read as what the ledger charged, are the entire uncontested part of this program. Everything past them — what the model can actually do — is where the expensive part starts.
proving general capability is expensive, and it keeps failing anyway
I have a tool-use program running on this trunk: teach it to route arithmetic to a calculator instead of computing by hand, bind the right number of arguments to the right tool, chain three tool calls in sequence, carry a runtime result from one step into the next. All four went from 0% at baseline to 95–100% trained, and when trained together on one checkpoint, none of the four dropped by more than 5 percentage points — a clean composition result.
Getting there required a pre-registered hypothesis test just to find out what was actually being learned. The first published result claimed the wall was argument count — one argument worked, two arguments mostly failed, p = 1.7×10⁻⁶. Refuted the same day: every tool family in the test set happened to have exactly one arity, so “argument count” and “which tool” were the same variable wearing two names. The real test had to isolate them on purpose:
1-arg exact: 100.0% 2-arg exact: 98.0% gap: 2.0pp
P1 (argument count binds, needs gap >=40pp): refuted
P2 (the tool binds): supported
A 2-percentage-point gap versus a 40-point threshold. That’s what it costs to know, instead of guess, which axis a capability actually runs along — and this was one of the simpler questions in the program.
The negative results cost just as much to earn. The trunk cannot generate a computed sum: 0/200 on a+b, fully trained, replicated more than five times. The scale comparison is a separate, single measurement — 0/40 at 1.489B parameters, 0/40 again at 2.945B, confirmed against that run’s own output file when the measurement was re-audited on 2026-09-09. Doubling the parameters bought nothing. The model can compare two numbers at roughly 82% (uncertified: no base score and no recorded control provenance) — it has a sense of magnitude — but it cannot produce one. Scale doesn’t touch that gap. Only routing the arithmetic to an actual calculator does.
And then there’s the result I retracted. A capability bar claimed 10.0% against a 2.0% control, margin comfortably over the bar for “demonstrated.” An independent audit went back and checked whether the control could have scored anything other than 2.0%, and it couldn’t — it was structurally incapable of being right on nine of ten question categories, so the floor wasn’t chance, it was an artifact of how the answer key happened to be shaped.
claimed: 10.0% vs 2.0% control — margin 8.0, "DEMONSTRATED"
audit: 6/10 hits are a 3-way forced choice (chance ~33%, not 2%)
4/10 hits are copied exemplars, miscounted as correct
net: 6/100 vs 9.76 expected by chance — margin +0.24
Margin of +0.24 items is not a result. It’s noise with a press release attached. I didn’t quietly correct the number and move on — the retraction stays on the record, marked retracted, with the reason attached, next to the original claim. A wrong number that gets deleted just gets re-argued in six weeks by someone who never saw the correction.
One more version of the same shape, further back in the same eval lineage: a training run regressed the model, and the multiple-choice probe scored it 20/22 — a pass. The regression only surfaced because a separate generative eval, one that reads the words the model actually writes instead of picking from options it’s handed, caught what the multiple-choice probe structurally could not see. A probe that hands you the right answer as one of four choices is measuring recognition. It was never built to catch a model that has quietly gotten worse at producing anything on its own.
None of this is a story about bad instruments run carelessly. It’s a story about how much verification a general capability claim demands before it’s earned — pre-registered hypotheses, control-gated probes, an eval built specifically to test the eval, and even then, one bad control slipping through turns a headline result into a retraction. General capability is expensive to prove and cheap to overclaim, and the two keep getting confused for each other.
a narrower question, and it’s cheap
There’s a different question that doesn’t need any of that machinery: does the model know this one deployed world. Not general knowledge — this specific estate’s facts. Who owns it. What its own rough size is. What wins when a document and a running system disagree. Which script is the sanctioned way to ship a change.
That question gets its own small corpus and its own small gate, built as a matched pair: ten facts expand deterministically into forty training texts and sixty-six instruction pairs, and the gate checks eight of them back against a held-out model. Before either one is trusted, both run self-tests — coverage, no leaked eval prompts, no reintroduced errors, every gated fact actually taught. And before a single real training run gets graded by the gate, a deliberately dumb baseline — a plain word-overlap retriever with no learning in it at all — gets trained on the corpus and checked against the gate too, purely to confirm the data is sufficient before the model’s competence becomes a variable at all.
facts: 10 -> 40 training texts + 66 instruction pairs
self-test: 4/4 controls, before a token of GPU time is spent
dumb baseline (no learning) passes the gate — the data is sufficient
GPU spent confirming that: $0
That’s the whole cost difference in one line. Confirming that a person-scoped corpus teaches what its own gate checks costs no GPU time at all. Confirming that a general capability claim actually clears its own bar cost an audit, a retraction, and a public correction.
different artifact class, different economics
A frontier-scale claim has to hold across problems nobody hand-picked, for users the builder has never met, against a benchmark someone else designed. Every one of those conditions makes verification harder and cheaper to get wrong — which is exactly what the retraction and the 20/22 pass show. A person-scoped model only has to be coherent about one bounded, already-fully-specified world. The ground truth isn’t a benchmark author’s guess at what matters. It’s a fact list the model’s own operator wrote down and can check by hand.
That’s not a smaller version of the same problem. It’s a different validity criterion. “Does it generalize” and “does it know this” are not the same question scaled down — the first is open-ended and the second is closed, and a closed question is one you can actually finish answering for less than the price of a retracted headline. The estate’s own plan leans on exactly this: it names several intended future branches, one seat per person the model would serve, each meant to get its own small fine-tune off the same trunk, each gated the same way — by whether it knows its one person’s world, not by whether it beats anyone else’s benchmark. None of those branches are trained yet. The pattern is deliberate, not incidental — a family of small, cheap, falsifiable models, not one attempt at a smaller frontier.
what I don’t know yet
The real target model for the person-scoped branch hasn’t been trained. The corpus and the gate are both proven — proven against each other, and against a dumb baseline that has no business passing anything except that the data made it trivial. Whether the actual model, once someone spends the GPU hour on it, passes its own gate is still unmeasured. The loop is validated; the artifact it’s supposed to validate doesn’t exist yet.
I also don’t know whether the gate’s own facts stay true as the trunk it’s grading keeps growing. The gate checks that the model knows its own rough size. The trunk has already been grown well past the number the gate was written to expect, using a technique that duplicates and extends layers rather than retraining from nothing. Nobody has gone back and asked whether the gate’s self-description fact still matches the thing it would be grading. That’s the same failure this whole program keeps finding elsewhere — something true when it was written, quietly outlived by the system it describes — and I haven’t checked whether it’s already happened here.
And I don’t know if the per-seat pattern survives contact with a second real branch. One coherence gate, proven on one estate, is a working method. A family of them, one per person the model serves, each with its own canon and its own gate, is still a plan with no second data point. Until there’s a second one built and gated the same way, “this generalizes past one world” stays exactly where it belongs: UNKNOWN.
— Small on Purpose