Shaurya Singh

← writing

The Box Is Canon

Commit, deployed, and loaded-by-the-running-process are three separate facts. Most coordination arguments are between people who checked none of them.

doctrine · shipped

Three of my agent lanes spent seven hours arguing about which file was the canonical version of a page. Different hashes, different claims, three separate “I verified this one” signatures. The whole argument was about files that were already obsolete. The process actually serving traffic wasn’t loading any of them — it was running off an uncommitted working tree none of the three had checked.

That’s the incident the doctrine comes from. The doctrine is short: commit, deployed, and loaded-by-the-running-process are three different facts. None of them implies the other two. A file existing on disk tells you nothing about whether it’s in git. A commit existing tells you nothing about whether the box has it. The box having it tells you nothing about whether the running process actually imported it. Skip any one check and you can spend an evening being confidently right about a question nobody needed answered.

What “canonical kit” even meant that night

The page was small — an internal reporting view, one route, one template. It went through more versions than a page that size should ever need:

  • An early draft, dead on arrival, never deployed.
  • A first real attempt that cleared review and still shipped with two live bugs a closer read caught later — a parsing mistake and a spot where user-supplied input wasn’t escaped.
  • A fixed version, cleared by three separate lanes running three separate check methods — not the same test signed twice by different names, three genuinely different instruments. This became “the canonical one.”
  • A parallel fork that also fixed one of the two bugs, but got rebuilt on the wrong base by mistake and brought the other bug back with it. Retired.
  • A hash-gated merge script that refuses to run unless the input file’s checksum matches exactly what it expects, so a hand-copied near-duplicate can’t slip in disguised as the real patch.

Every one of those checks was real. Nobody skipped a step at the file level. The problem was one level up: none of the checks asked whether the file they were checking was the one the server would actually load.

The third artifact

The version that actually went live wasn’t any of the above. It existed because the lane that deployed it kept improving the page after pulling it out of the argument — a shared stylesheet, a design-token fork, a type-size floor, a seal include, a CSP nonce for an inline script. Five fixes, all of them things my own deploy gates required, and none of them caught by anyone arguing upstairs about which hash was canonical.

That’s the twist the doctrine is built on: byte-identity with a reviewed kit is worth nothing next to actually passing the gates. A deployer who improves an artifact after extracting it from the argument hasn’t broken the chain of custody — they’ve advanced it. Byte-identity with a kit is worth nothing; passing the gates is worth everything. The lanes holding hashes were measuring the wrong thing. Hash equality answers “is this the exact bytes someone reviewed.” It doesn’t answer “does this pass,” and passing was the only question that mattered.

The receipt that actually explains the seven hours

The reason the argument was possible at all — the reason three lanes could disagree for that long without anyone catching it — is that the module and its template landed in one commit, f8edbde, and the line that wires the route into the running app didn’t — that came later, in 8499940. f8edbde is real. It’s in the log, it’s on the box. And the running process had never heard of it, because nothing imported it. I was serving whatever page had been there before, off a working tree that no commit fully described. Every lane checking “is my kit’s hash correct” was checking a question one layer removed from the one that mattered: is anything importing this file at all.

The doctrine I wrote down afterward names the gap precisely: a file’s presence on disk, its digest, and its commit are three different facts, and none of them establishes that the running process loaded it. Commit is a git fact. Deployed is a filesystem fact. Loaded is a process fact — it lives in memory, not in any of the artifacts you can hash.

Why a status code isn’t a check

Once that gap was visible, I rewrote the acceptance test for any future deploy around it: never accept on a status code, only on authenticated body content. A route returning a redirect tells you a redirect fired. It does not tell you the redirect goes anywhere real — an error handler can shadow a working route and still answer with a clean, boring status that looks identical to success. The only thing that distinguishes “the page is actually there” from “something is answering in its place” is checking for the body content that only the real page produces — a specific class name, a live call the template makes on load, a shape you can’t get by accident. A status code is a proxy for the thing you care about. A proxy is exactly what “loaded” refuses to be inferred from.

The same shape of bug showed up again on a later error-page rollout. The install script for custom 404/403 pages assumed a helper existed in the server’s namespace and appended its route handlers after the line that starts the app — which means, in a script that runs top to bottom, after the point where new routes stop mattering. Run against the live box, the script correctly refused instead of silently wiring in something broken. That’s the other half of the doctrine: build the check so being wrong about the target is inert by construction, not something that depends on whoever’s watching noticing in time. A script that refuses on drift and a body-content check that refuses on a bare redirect are the same discipline pointed at two different failure shapes.

The pages that shipped after that refusal were checked the way the doctrine actually demands: not “does the route respond” but “does the body match a known-good shape” — tested against a fixed positive control and a fixed negative control, and no result accepted until the rig had proven it could also catch a known-bad input. A check that can’t fail is not a check. The interim-reporting page itself didn’t stop moving once it went live, either — a fix landed a few days later that changed how the client polled it, and the version that mattered after that was whatever the box was running post-fix, not the one that was live the day of the seven-hour argument. Canon isn’t a fixed artifact anyone gets to hold up as final. It’s whatever the box is doing at the moment you check, and that moment moves.

The general failure this names

Strip the specifics and the pattern generalizes past one route: any system where documentation, kits, and a live process can drift independently has exactly one authoritative fact, and it’s whatever the live process is doing right now — not what anyone, including the deployer, believes it’s doing. Most coordination failures in a system like that aren’t disagreements about facts. They’re arguments where every party checked a fact adjacent to the one in dispute and none of them checked the actual one. Three lanes each had a real, defensible answer to “which kit is correct.” Nobody had an answer to “what is the box currently loading,” because nobody had asked that question in those words.

This sits next to a broader point about staleness — a fact that was true once doesn’t stay true just because nobody re-checked it — but it’s a narrower and more mechanical claim than that. It’s not “verify more often.” It’s that commit, deployed, and loaded are three specific, independently-checkable facts with a strict order of authority, and an argument that never reaches the third one is an argument about nothing, no matter how carefully the first two were checked.

What I don’t know yet

I don’t know how deep “actually check” has to go before it’s provably enough. Loaded-by-the-running-process beat commit and deployed here, but loaded is itself checked through some interface — a body-content match, a live call — and that interface can in principle be fooled by something that isn’t quite lying, the same way a health endpoint can answer 200 while the thing behind it is still warming up. I don’t have a rule for when to stop descending through proxies and call a check final.

I also don’t know how this doctrine holds up for a lane that structurally can’t reach the box at all. A cloud-side check hitting the box through a fetch has already mangled redirects and choked on error responses in ways that made real box-state read back as a failure that wasn’t one. “The box is canon” is true, but a lane that can only see the box through an unreliable proxy can’t act on that truth directly — it needs a second lane that’s actually on the machine, and I don’t know what the fallback is when no such lane is available.

And I don’t know how the hierarchy extends once there’s more than one box. A single machine running one process has one obvious canon. Two boxes running long-diverged copies of the same product don’t have an obvious tiebreaker just because each one is canon for itself — that’s a different, harder question, and I haven’t seen it answered yet, only avoided by treating each box as its own separate ground truth for now.

← back to writing