A build I work on ran 230 gates in one session: 220 green, 9 red, 1 infra. Someone root-caused all nine reds. Three were worth writing down, and none of the three was a defect in the thing being tested. The subject hadn’t regressed. The instrument watching it had.
One control still pointed at a function the decision had moved out of. One module died in setup before a single check ran and scored an ordinary RED with zero diagnostic lines — the harness went looking for the failure by grepping for FAIL and found nothing, because there was nothing to find. One fixture was pinned to the source side of a value that had quietly grown a consumer-side multiplier, so the “killer frame” built to overwhelm a clamp had stopped being bright enough to need it.
That one infra gate is the tell. Somebody hand-sorted it into a category the harness doesn’t have, because the exit code only knows two. RED collapsed two different facts into one bit: the subject failed a check, and the instrument could not run the check. From outside, by exit code, those are indistinguishable. From inside, they need opposite fixes. gatehealth is that fix as running code for pytest — three outcomes, three exit codes, silence banned as an outcome in its own right.
PASS the check ran and the subject passed it
FAIL the check ran and the subject failed it
CANNOT-MEASURE the check did not really run; this is not evidence
The mechanism
pytest’s outcome vocabulary is subtler than it looks, and that subtlety is most of the bug. A fixture that raises during setup does not produce an “error” outcome — it produces when="setup", outcome="failed", the same word pytest uses for a real assertion failure. Anything reading report.outcome sees one word for two diseases. gatehealth translates that once, at the boundary, then runs a priority table over side-channel diagnostics and its own tracer.
Here is ordinary pytest on a suite built to carry every disease at once:
shaurya@sps ~ % python -m pytest examples/demo_suite --continue-on-collection-errors -q
FAILED examples/demo_suite/test_genuine_regression.py::test_three_part_value_survives_the_parser
FAILED examples/demo_suite/test_stale_fixture.py::test_killer_frame_saturates_the_sensor
FAILED examples/demo_suite/test_swallowed_setup.py::test_night_calibration_applies
ERROR examples/demo_suite/test_collection_error.py
ERROR examples/demo_suite/test_setup_error.py::test_daylight_calibration_applies
3 failed, 7 passed, 2 errors in 0.04s
Seven greens, three reds. Three of those greens are gates gatehealth refuses to accept — one verified nothing at all, one ran the parser and never looked at the answer, one is a planted control that convicted nothing. Two of those reds are not defects in the subject at all. Nothing in that output distinguishes any of it. Same directory, same run, through gatehealth:
shaurya@sps ~ % gatehealth --quiet -- examples/demo_suite; echo "exit=$?"
FAIL test_genuine_regression.py::test_three_part_value_survives_the_parser [-]
-> test_genuine_regression.py:16: AssertionError
CANNOT-MEASURE test_collection_error.py [collection-error]
CANNOT-MEASURE test_rotted_control.py::test_pick_rejects_n_equals_one [canary-did-not-fire]
CANNOT-MEASURE test_setup_error.py::test_daylight_calibration_applies [setup-error]
CANNOT-MEASURE test_stale_fixture.py::test_killer_frame_saturates_the_sensor [fixture-staleness]
CANNOT-MEASURE test_swallowed_setup.py::test_night_calibration_applies [swallowed-exception]
CANNOT-MEASURE test_vacuous.py::test_arithmetic_on_a_literal [zero-assertions]
CANNOT-MEASURE test_zero_assertions.py::test_parser_smoke [no-explicit-check]
PASS test_genuine_pass.py::test_multiple_parameters_parse [-]
PASS test_genuine_pass.py::test_single_parameter_parses [-]
PASS test_raises_only.py::test_empty_shot_name_is_rejected [-]
PASS test_rotted_control.py::test_awning_cloth_rejects_n_equals_one [-]
------------------------------------------------------------
4 PASS · 1 FAIL · 7 CANNOT-MEASURE
exit=1
(Node IDs trimmed of their directory prefix, banner dropped, and every CANNOT-MEASURE line’s follow-up paragraph elided to fit. Those paragraphs are where the tool earns its keep — the staleness one prints pinned=20000.0 live=7000.0 drift=65.0% and tells you to re-derive the fixture in delivered terms, not to move the bound until it goes green.)
Both harnesses exit 1. Only one says that exactly one of the five non-green gates is worth opening the source file for.
FAIL outranks CANNOT-MEASURE for the exit code, even though disease signals outrank FAIL for the per-test label, because disease detection that cannibalizes regression detection is worse than none. An empty run exits 2, not 0: a suite that collected nothing has demonstrated nothing, and calling that a clean bill of health is the same lie as a crashed instrument reporting green.
The green half is the interesting half
Two of the detectors, carrying three of the labels, work on results pytest was perfectly happy with.
A canary is a control test, there to prove a specific guard sits at a specific code site. Mark it @pytest.mark.canary and call gatehealth.mark_fired() from inside the except that proves the guard tripped. If a later commit moves the decision elsewhere, the test keeps passing and stops firing — a true report about a place the logic no longer lives — and gatehealth turns it amber. A canary that failed falls through to plain FAIL: a red canary is already demanding attention, and relabelling it CANNOT-MEASURE would bury a live signal under a maintenance one.
The zero-assertions detector installs a sys.settrace line tracer for the call phase and counts check-shaped evidence that actually executed — not lines that exist, lines that ran, which is the difference a static pass can’t see. It watches two channels: assert statements executed in your code, and calls into a registry of assertion primitives (unittest’s 50 assert*/fail* methods, unittest.mock’s 14 assert_* methods, pytest.raises / warns / deprecated_call / fail). The second channel exists because the first one alone failed against real code; the census below is that story. A green that entered neither channel and never called out of its own test module is zero-assertions; one that did call out, so the only evidence is that nothing raised, is no-explicit-check — a label that reports the shape and declines to decide. Its scoping rule is load-bearing: lines under the stdlib, under site-packages, and inside gatehealth itself never count. pytest’s own internals are full of assert statements, and counting those would let the tracer launder the exact disease it exists to find.
The plugin is deliberately not a pytest11 entry point. A tool that installs a global sys.settrace into every pytest run on the machine because it happened to be in the environment is a worse citizen than one you have to name.
What it can’t do yet
- An
xfailis reported as a skip, and a skip is CANNOT-MEASURE. I ran a test markedxfailthat failed exactly as declared; it came back[skipped], exit 2. An expected failure is a check that ran and did what its author predicted, and gatehealth cannot tell it apart from a gate that never ran. - The tracer matches source text, so a string can pass for a check. A test whose only body is
note = "remember to call pytest.fail( here someday"is classified PASS. The marker match is a substring scan over the executed line, with no awareness the text sits inside a literal — a test that verifies nothing clears the detector by mentioning it. - One thread, one process.
sys.settracesees the thread it was installed on. I ran a test that asserts inside a worker thread and gotzero-assertions— a false positive on a test that checked something real.gatehealth.check()is the escape hatch, and you have to call it by hand. - A dead module counts as one missing gate, not as the gates it was holding. A module with three tests that fails to import produces exactly one CANNOT-MEASURE line. The tally shrinks by three and nothing says by how much; gatehealth knows the module never ran, not what was inside it.
- Teardown contamination is only reported over a green. A finalizer that blows up upgrades a PASS to CANNOT-MEASURE. If the body already failed, the leaked resource goes unnamed — exactly when later tests are most likely running in the contaminated environment.
- Three detectors require you to write calls.
mark_fired,swallow_guardandcheck_pinnedare opt-in, because knowing that anexceptswallowed something that mattered requires knowing intent. Only zero-assertions, no-explicit-check, setup-error, collection-error, teardown-error and skipped work on an unmodified suite. sys.settraceis a single global slot. coverage.py, a debugger or a profiler will fight the plugin for it and whichever installs last wins. Running gatehealth and coverage in one pytest process is not supported.- The false-positive rate is no longer UNKNOWN, and the first answer was bad. A pre-registered census ran the original single-channel detector over 41 real open-source Python packages — 50,991 tests classified — and flagged 37.93% of passing tests as having checked nothing. An adversarial audit re-measured every flag and found 13,095 of 18,690, or 70.06%, were mechanically provable false positives; hand-audited precision on the sample was 21.88%. The study’s own verdict was that the detector was not fit for third-party code in that form. The second channel described above is the fix, and it was re-measured against forty of the same 41 packages at the same commits: the flag rate fell from 38.02% to 11.39%. The false-negative rate was UNKNOWN before the census, stayed UNKNOWN after it, and is still UNKNOWN. Both rounds are written up in the sibling essays.
There is also a fifth failure mode the source incident named and this tool does not detect at all: a parser that silently truncated a three-part token and passed on input it had quietly shrunk — not FAIL, not CANNOT-MEASURE, but passed-on-the-wrong-input. It ships instead as the demo suite’s one genuine FAIL, so the tool has a real regression sitting next to the diseased gates, proving it doesn’t swallow the thing it is supposed to leave alone.
A hundred and twenty tests, up from seventy-three before the census fix, cover both tracer channels and the scoping rule, the diagnostics side-channel, the drift maths, and every branch of the priority table, with integration coverage running real pytest sessions through the real plugin and asserting on the JSON it emits — pytest’s outcome vocabulary is the whole reason this exists, and a mocked report object would have agreed with whatever I assumed. That’s a count against this source today, from a unit suite that has never met a codebase it didn’t already know the answers to; the census above is where the detector met forty-one that it didn’t.
— Shaurya