Shaurya Singh

← writing

An Actuator Has No Green Light

A test that rots reports a wrong answer. A brake that rots reports into a channel nobody opens — and the bill arrives before the alert does.

ongoing · reliability doctrine — one control armed against a stored incident fixture, on a branch, in a session marked unclean; last-fired-at proposed, not shipped

ongoing · reliability doctrine — one control armed against a stored incident fixture, on a branch, in a session marked unclean; last-fired-at proposed, not shipped

A spend ceiling in this estate was named VANCE_MAX_USD. An audit grepped for it and found four hits: two writes, two self-assertions. The variable set itself and asserted its own existence. No consumer anywhere read it to enforce anything.

That is the pure case, and it is narrow. For every day that variable sat there dead it produced what a working cap produces: nothing. Silence from a cap that is holding and silence from a cap never wired are the same silence, and the divergence arrives as a number on an invoice rather than a red light in a pipeline.

But the estate was not running with no cap, and the same audit says so. A sibling check — the one meant to confirm the spend ceiling travels with the pod — verified only that the string VANCE_MAX_USD was a key in a dict the same function had written a few dozen lines earlier. Always true. Nothing read the value. Meanwhile a real per-step brake was armed the whole time on its own default: $140 on one path, $18 on another. Against an operator who had declared $12, up to 11.7 times the ceiling he believed he had set.

So the failure was not an absence. A brake was enforcing a number nobody had declared, and on firing it would leave a record at the wrong threshold. That third state is what the rest of this essay has to survive.

Two classes of control, and the axis is not the one I reached for first

Split the things that are supposed to protect you into two piles. The first pile reports: tests, health checks, metrics, provenance gates, scoring code. Their entire output is a signal, emitted continuously, whether anything is wrong or not. When one rots, the rot surfaces as a wrong signal. This corpus has spent three essays there. I am not re-litigating it.

The second pile acts: brakes, kill switches, watchdogs, rate limiters, self-terminators, backup writers. These emit nothing under normal operation. They are built, correctly, to fire almost never. Their evidence is an event, and the event is the thing you are paying them to prevent.

My own receipts break that split twice, so I will break it first. Acting controls do fail by lying: the $140 cumulative campaign cap charged only while a trainer process was alive, so billing during image pull, pip install, tokenization and OOM gaps all landed in the ledger as $0.00. That brake emitted continuously, and during paid intervals what it emitted was a wrong number. “Reporting controls lie, acting controls go quiet” is too clean to be true.

Sharper still: the real axis is not act versus report. It is runtime versus static. An acting control keeps a perfectly good static observable — you can grep it. The VANCE_MAX_USD finding is a grep with a countable result. The watchdog’s scoping bug was established by reproducing exit 127 in a scratch directory, on demand, without waiting out a 23-hour run. Both are repeatable answers from acting controls, no event required. What an acting control lacks is a runtime observable. Static analysis tells you it is wired. Only a firing record tells you it worked.

The detection channel was never the control

I can name the moment each of these was found. Not one by the control’s own output.

A watchdog meant to stop a 23-hour training run called request_stop inside a bash -c subshell where that function does not exist. Exit code 127, piped to /dev/null. Detection came from an automated audit over 313 tracked files — nine lenses, three adversarial refuters per finding, 160 agents — which did not read the script and infer the bug; it reproduced the failure in a scratch directory. How long the watchdog had been failing is UNKNOWN. Nothing dates its authorship.

The same day that watchdog didn’t fire, spend closed at $31.87 against a $25 cap while the internal ledger read about $9. Two defects belong on this essay’s axis: the ceiling nobody read, and the watchdog exiting 127 into a void. A third belongs to a different essay here — a verifier asking a question with nothing in it. The chain is also not settled. The ledger gap has a separate, reporting-side explanation on record: pod.cost is an hourly rate, not a total, and several instruments mis-logged spend by reading it as cumulative. And the breach has a fourth candidate the register still lists as an open owner decision — two lanes sharing one $25-a-day cap, with a pod from another project billing $1.59 an hour at 0% utilization that day. Detection came from the billing API. The causal story is partly UNKNOWN.

A month earlier, a 500M-parameter run finished at $164.22 metered against a $140 cap. The session note records it in three words — breached invisibly. No alert; a meter reading, after.

And the sharpest case: two pods sat at 0% GPU utilization for roughly 19.6 hours at $3.48 an hour. About $68 burned doing nothing. Both had self-termination brakes, and both brakes had died with the interactive session that launched them, because the brake was a process on a laptop and not a policy on the account. Detection came the next day, when a billing connector was wired up mid-session and the new seat’s first act was to read the meter.

Now the part I nearly cherry-picked. The page supplying the $164.22 number carries a finding that cuts against the tidy version of this section — nine instruments reported success while failing in one night, not one caught by its own author, every catch coming from a second party measuring the same thing with a different instrument. Two of the nine were acting controls — a halt brake denied in-container, which made every cost control decorative for a $128 run, and a watcher keyed on a success string, which cannot see a silent death and waited while about $2.50 idled. Neither was found by an invoice or an outage.

So the honest claim is narrower. Detection never comes from the control’s own output. It comes from an invoice, an audit, an outage, or a second instrument measuring the same quantity differently. That fourth channel most threatens this essay, and has a direct answer: cross-instrument corroboration needs a quantity with a meter. Spend has one. An event that did not happen has no meter, and you cannot point a second instrument at a non-event.

The cost is a rate

A rotted test is expensive at a decision point. You merge the wrong thing once, at the gate, and the cost is bounded by what that decision let through. Bad, and episodic. A dead actuator is expensive per hour. The $68 did not arrive as a decision someone got wrong. It arrived as 19.6 hours times $3.48, accruing the whole time, with no control emitting anything about it. No review meeting catches an accrual.

The second property is worse. When you find a dead brake you do not learn when it died. You can date the commit that introduced the scoping bug, if you’re lucky. You cannot date the last time the control worked, because a working acting control leaves no record of working either. The uncertainty extends backwards, unbounded, to the day it was written.

That is a missing field: no acting control here records its own firing history. Reconstruction is not impossible — a reconciler exists, built for this exact gap the night of the $164.22 breach, and it is what compared a $135.84 ledger against a $164.22 meter, $24.22 past the cap. It has never been invoked; it requires two flags nobody passes. More precise than “nothing exists”: an instrument was written for the hole, and the hole is open, because writing it was not the same as wiring it.

PID liveness versus health is the same bug, wearing a supervisor

The clearest instance of a wrong trigger predicate is not a spend brake. A launcher supervised its server on a ten-second loop and checked that the process existed. A hung server keeps its PID — it holds the port, answers nothing, and the supervisor sees a living process and takes no action, forever. The audit line is exact: supervises PID liveness only, never health. Same disease: nothing emitted while things are fine, nothing emitted while things are catastrophically not fine, because presence was the predicate and presence was still true.

Then the inverted case, which proves firing is not evidence of working either. One log file accumulated 37,844 vance died -- restarting entries in a single week, 4.2 MB of them. That actuator was acting constantly and had no postcondition: nothing checked whether the relaunch produced a serving process. Root cause: UNKNOWN, on two counts. The launcher recorded only that it had died, never why. And the rotation used mv, which swaps the inode while the running process still holds a descriptor on the old one — so after any rotation every diagnostic went to an unlinked file.

That second half corrects this essay’s thesis. An actuator firing into the void is not silent. It is deafening — 4.2 MB in a week, in a channel nobody opens, beside a diagnostic stream written continuously into a file no one could open at all. Both of this estate’s crash-loop logs were found because they got enormous. Emission was never the constraint.

So an acting control needs two things it is rarely built with: a trigger predicate that measures function rather than presence, and a postcondition that measures whether the action changed anything. I will not claim an actuator firing into the void costs more than one that never fires — I have no dollar figure, and inventing the comparison is the self-inflicted wound this site refuses. The measured costs are disk and diagnosability. What it is is more misleading, because it manufactures the appearance of a working supervisor.

The repaired version exists: a health probe with a three-consecutive-failure threshold, a 90-second grace period, an escalating cooldown, and a death path that logs an exit code, a signal, and the last fifteen log lines. Worst case was modelled at 51 restarts in 24 hours, against the 37,844 per week observed. 51 is a design calculation, not a post-deploy measurement. I have the model. I do not have the after.

Arm it against the incident it was built for

The fix I believe in has two halves, and neither is as new as I first wrote it.

First half: every acting control gets armed against a stored fixture of the real incident it exists to prevent — not a synthetic test by the same hand that wrote the control. The actual event, frozen and replayed at the control, which is required to fire.

A draft of this essay claimed that had been done once, properly. False, and the corpus refutes it twice. In mid-July a config-section outage was closed by deleting the section entirely — reproducing the exact outage condition — and confirming the dependent agent still answered, reverified by cold restart. Days later a mutation harness shipped that reintroduces each of six fixed bugs and proves every test goes red against the old behaviour and green against the new: 6 of 6. Both predate the fixture I was about to call the first instance by six weeks. And the rule underneath is already published here, in the essay about care not scaling: every check named in a docstring must be named in a test observed to fail. Firing a control at a stored incident is that rule with a new noun.

What is new in the late-August case is narrower: the fixture is a stored production incident attached to a spend actuator, not a reintroduced bug in a test harness. A policy brake was written the day after the two pods burned $68 idling, and its defining property is one clause: it fires on the 08-28 incident fixture. The design change underneath matters too: it became a policy on the account rather than a process on a laptop, so it can no longer die with the session that started it.

Then the caveat this essay owes. That brake landed on a branch, mutation- and control-verified, in a session whose register row reads Clean handoff: NO and Session status: Active. Whether it is enforcing on the account today is UNKNOWN. By this essay’s own standard it is a claim about having been shown the event, not a record of the firing — and a claim is what I opened by mocking. UNARMED until something other than its author records it firing.

Second half, not shipped: a last-fired-at per acting control, written by real firings and fixture drills alike. That field is the genuinely new part, and new in one dimension only — it makes the evidence durable instead of momentary. A drill proves a control could fire once, in front of whoever ran it. A timestamp proves it to the next reader, who was not in the room.

The report side of this house already has its third outcome, and the red-gate essay owns the reasoning. The act side needs its own pair: ARMED versus UNARMED. Be honest about that claim. Nothing mechanically forces UNARMED as the default for a control with an empty firing record. It is a proposal — a policy someone chooses and pays the drill cost for, which I cannot price.

That proposal covers the one acting control everybody thinks they have: a nightly backup exiting zero emits the same thing whether it wrote real bytes or none. Fire it at a fixture, restore from what it wrote, record the date.

What I don’t know yet

I don’t know how many acting controls exist in this estate, because no registry exists — and one audit is not the enumeration. The 48-defect audit covered 313 tracked files of one repository. The 37,844 restarts, the PID-liveness supervisor and the health-probe repair come from a different July audit of a different codebase. The two idling pods came from a live meter read and no audit at all. Three audits, three systems, no union. I cannot tell you what fraction of acting controls are dead, or claim the incidents above are representative rather than merely found.

I don’t have last-fired-at anywhere. No field, no writer, no reader, no staleness alert. What it would let you see is design reasoning, and that stays a proposal until something runs.

I don’t know whether the brake armed against a stored incident is enforcing anywhere today. Verified on a branch, in a session marked unclean, with no second instrument confirming it. By this essay’s standard, not shipped.

I don’t know how much a fixture buys. Firing on the 08-28 incident proves it catches 08-28. An incident fixture is a rear-view instrument: you can only freeze events you have already paid for.

I don’t know the right drill rate. Arming a brake on purpose costs a genuine termination, kill or restore. Drill too often and you pay for the drills; too rarely and the uncertainty interval reopens. I have no number, not even a bad one.

And I don’t know whether the repaired supervisor is doing better. The health probe shipped 2026-07-19 and nobody has counted since. That gap is most of two months old, which is not a pending item — it is the finding.

Until each has a receipt, they stay UNKNOWN — which, for an acting control, is not a temporary embarrassment. It is the default state of every one you have not deliberately fired.

— Last Fired: Never

← back to writing