Shaurya Singh

← writing

A Red Gate Has at Least Two Diseases

A RED test conflates a real regression with a rotted instrument — and a GREEN one can lie the same way, when a parser silently truncates the input.

shipped · CI doctrine

shipped · CI doctrine — three-exit-code fix closed several silent-death paths in one graphics-lane session, not yet ported past it

A build I work on ran 230 gates in one session. 220 GREEN, 9 RED, 1 INFRA, 5,607 clauses checked. Someone had to close the nine REDs before the session counted as done, so someone did — root-caused all nine, three of them worth writing down. None of the three was a defect in the thing being tested. The subject hadn’t regressed at all. The instrument watching it had.

That’s the whole claim, stated early because it’s short: RED is not one fact. It’s at least two, and the corpus I keep pulling receipts from found a third, then a fourth, in a single afternoon of actually reading the failures instead of re-running them.

Two diseases wearing one color

Disease one is the boring, correct case, and it’s the only one a green-to-red transition is supposed to mean: the code changed, a check that used to pass now legitimately fails, ship nothing until it’s fixed. That’s what RED is for. Most REDs are this. This essay isn’t about most REDs.

Disease two is everything else — the instrument itself decaying independently of the subject it watches. A test suite has two clocks running against it, not one: the code it’s checking, and the code doing the checking. Nobody budgets time for the second clock, because a test file doesn’t look like production code. It doesn’t get code review with the same suspicion, it doesn’t get refactored when the thing around it changes, and it sits there asserting the same thing it asserted the day it was written while everything it depends on keeps moving. Three REDs, three different ways that clock ran out.

The rotted anchor

A planted control called sign_chroma_test existed to prove one thing: that a specific decision-guard sat at a specific code site, a function called pick(...). The control worked by targeting that site directly and checking that the guard’s precondition — a simple inequality, n !== 1 — held there. It never silently passed a bad case; that wasn’t the failure. The failure was that a later, unrelated commit moved the decision itself into a new function, awningCloth(), and the control kept checking the old address. Not a false green. A true report about a place the logic no longer lived.

sign_chroma_test  →  RED
target: pick(...)         (decision relocated to awningCloth())
guard precondition never evaluated: n !== 1 unreachable at old site

The fix wasn’t loosening the control. It was re-deriving it against the function that actually holds the decision now, and re-verifying all eight controls in the same family rather than trusting that the other seven still pointed where they used to. A control isn’t evidence forever just because it was evidence once. Code around a fixed point moves; the fixed point doesn’t follow it on its own.

The gate that cannot report

The second RED was skyline_visibility_test, and it’s the one worth the most attention, because it’s the shape that’s easiest to miss from the outside. The module had nine top-level awaits, no try/catch anywhere around them. Any one rejection kills the process before the first check ever runs. Exit code 1. Zero lines beginning PASS or FAIL. The harness scored that exit code as an ordinary RED and then went looking for the failure by grepping the output for ^FAIL — and found nothing, because there was nothing to find. The check never ran.

That’s the sentence worth keeping: “RED” collapsed two completely different facts into one bit — “the subject failed a check” and “the instrument could not run the check at all.” From the outside, by exit code alone, those are indistinguishable. From the inside, they need opposite fixes. One means go fix the code under test. The other means go fix the test.

The same module had three more silent-death paths in it once someone actually went looking, all closed in the same pass:

  • An error line that didn’t start with PASS or FAIL was still incrementing the fail counter — so a crash and a real failure produced the same tally, no way to tell them apart after the fact.
  • A render harness stamped window.__fatalError on a faulted frame but still set __ready — so a frame that had actually died handed back a full, plausible-looking band of numbers instead of throwing.
  • Math.abs(x - undefined) evaluates to NaN, and NaN > 2 evaluates to false — so a landmark ring that had actually vanished from the frame read as unchanged, because the comparison that was supposed to catch the drop couldn’t fail loud enough to notice it.

Every one of these, one root cause: nothing in the pipeline distinguished “checked and clean” from “never checked.” The fix names a third outcome instead of patching each symptom separately:

exit 0  → PASS
exit 1  → FAIL (named — the line that failed is in the output)
exit 2  → CANNOT MEASURE (the instrument died before it could check anything)

Three outcomes now, three exit codes, none of them silence. This is the direct extension: measured-or-UNKNOWN said a fact that wasn’t just checked shouldn’t be allowed to render as true or false. A CI gate needs the identical third value, for the identical reason. Collapsing “didn’t happen” into the same bucket as “happened and passed” or “happened and failed” makes all three indistinguishable to whoever reads the log next, and whoever reads the log next is going to trust the color, not go read the process’s stderr.

The fixture pinned to the wrong layer

The third RED, post_robustness_test, is a different disease again, and it’s the one most likely to survive a naive fix, because the naive fix looks reasonable. A “killer frame” fixture existed to prove a brightness clamp actually clamps — deliberately over-bright, built to overwhelm the sensor model unless the clamp caught it. It was pinned by a source-side lumen value, ?lamppow=20000. A later, unrelated commit introduced a consumer-side multiplier that both writers of lamp intensity now read, PRACT_NIGHT = 0.35. The fixture kept its original source number. The delivered radiance — the thing actually hitting the sensor — silently dropped to 35% of what it was when the fixture was authored.

R1  29.17   (with clamp, without clamp — identical, unchanged by the bug)
R2  0.00    (unclamped, post-fix)  vs  29.17 (clamped, post-fix)
R3  82.12   (with clamp, without clamp — identical, unchanged by the bug)

R2 is where the rot showed up: before the fix, unclamped and clamped values had converged — the “killer” frame had stopped killing anything, because it was no longer bright enough to need the clamp at all. That reads exactly like a regression in the clamp. It wasn’t. The clamp still worked. The test had gone quiet.

The instinct here is to move the bound — loosen the fixture until it passes again. That would have been exactly wrong, because the bound describes a physical fact, total sensor saturation, not a tunable threshold. The actual fix re-derives the fixture in delivered terms instead of produced terms: 20000 / 0.35 = 57143 lumens reproduces the original delivered radiance exactly, because it’s pinned to the number that’s actually measured downstream, not the number that was true when someone typed it. Same move as pinning a threshold to something checked in the current run instead of a constant frozen at authorship — applied here to a fixture’s input value instead of a pass/fail bound, which is a subtler place for the same bug to hide, because a fixture doesn’t look like a threshold and nobody audits fixtures the way they audit thresholds.

A fifth mode, and it doesn’t even turn red

All three of the above are RED lying about what kind of RED it is. There’s a nastier cousin, from a different lane of the same build, where nothing turns red at all.

A shot-harness parser read generic key:value shot parameters with a regex whose value character class was [\w.|-]+ — word characters, dot, pipe, hyphen. A newly introduced parameter needed a three-part value with a second colon in it, lodforce:kerbVehicle:1. The character class didn’t include colon. The match failed partway through, the token was silently dropped — no error, no warning, no changed exit code — and every render row whose name asserted a forced LOD tier quietly rendered the unforced frame instead. Nothing crashed. Nothing flagged. The output looked like a normal, slightly boring, successful render, because by every signal the harness checked, it was one.

before:  qa/district_shots.mjs:166   value class [\w.|-]+
after:   qa/district_shots.mjs:166   value class [\w.|:-]+
fixed at commit bd5db618

One character. It was caught by a cost signal, not the test suite: the forced-mid and forced-far renders came back byte-equal in cost to the base frame, and a page-state flag that should have reflected the forced setting was simply absent. Nobody wrote an assertion that checks “did the parser actually keep every part of the token it was handed” — because that’s not the kind of thing a key:value parser’s own test suite tends to test. It tests that valid input parses. It doesn’t test that a slightly-more-structured input than the parser was built for fails loud instead of truncating quiet.

This is a fifth exit-code category in spirit, even though it isn’t one in the harness: not FAIL, not CANNOT-MEASURE, but PASSED-ON-THE-WRONG-INPUT. A row that cannot fail loudly, whose name asserts the thing it silently did not do, will clear every gate downstream, because every gate downstream is checking the output of a step that already lied about what it consumed. That’s the same gap “A Passing Report Is a Suspicious Report” names from the other side: a check that has never been shown a case it should fail on hasn’t demonstrated it can fail. This parser had never been fed a value with a second colon in it, so its clean exit code on that input was never evidence of anything — it just hadn’t been asked the question yet.

The fix is a third state, not a better test

Put the four together and a pattern falls out that’s worth stating as a rule instead of four separate incidents. A test suite reporting RED can mean: the subject regressed (fix the subject), the control’s target moved (re-derive the control), the instrument died before it could check anything (fix the instrument, and make sure dying reports as CANNOT-MEASURE instead of borrowing FAIL’s exit code), or the fixture is measuring the wrong layer of a value that used to be one number and is now two (re-pin to what’s actually delivered, not what was originally produced). A fifth thing can happen with no RED at all: the input silently got smaller than it claimed to be, and everything downstream inherited a clean bill of health for work that never happened.

None of these five get fixed by writing a better test. Four of them get fixed by the harness being honest about what it can and can’t currently tell you — PASS, a named FAIL, or CANNOT-MEASURE, with silence banned as an outcome in its own right. The fifth doesn’t even have a harness-level fix yet; it has one instance of a human noticing a number that should have moved and didn’t.

All three REDs closed by re-deriving a control, adding an exit code, or re-pinning a fixture — not by moving a single threshold, predicate, or clause bar to make the numbers work. Deny lists in this build grow. They don’t get trimmed to green.

What I don’t know yet

I don’t know how far the three-exit-code fix generalizes past the one module it shipped in. Nine top-level awaits with no error boundary was a specific, findable bug once someone went looking for silence instead of failure — I don’t know whether every instrument-rot mode has a signature that mechanical, or whether most of them look nothing alike and this taxonomy is really just the four that happened to surface in one session.

I don’t know whether a fixture audit — walking every planted value in a test suite and asking “is this still pinned to something that gets measured in the current run, or to a constant someone typed once” — is cheap enough to run routinely, or whether it only gets done the way this one did: after the symptom already showed up as a false convergence between clamped and unclamped output.

I also don’t know how common the fifth mode is — a step that fails at parsing, produces plausible output anyway, and never trips an exit code at all — outside the one parser it was caught in here. It was found by a cost anomaly a person happened to notice, not by any check built to look for it. I don’t know what fraction of a pipeline’s silent truncations get caught that way versus never get caught, and I don’t have a way to measure the denominator of a bug class defined by not announcing itself.

Until each of those has a receipt, they stay UNKNOWN — the same word this corpus uses everywhere else, applied here to the instruments instead of the facts they’re supposed to be watching.

— Three Exit Codes, None of Them Silence

← back to writing