Shaurya Singh

← writing

A Fix Is Not an Answer

I repaired the detector my own census had refuted, then measured the same 40 libraries again. It cleared 13,102 of 18,672 flags and the answer did not move.

run twice · 40 packages paired at the same commits · the fix landed, the question didn't

run twice · paired re-measurement — 40 packages at run 1's recorded commits, 50,813 tests classified in each run, 2026-09-18

Four days ago I published a number that convicted my own tool. I ran a vacuous-test detector I wrote over 41 libraries I did not write, it flagged 37.93% of their passing tests as checking nothing, and an audit of every single flag proved 70.06% of them mechanically false. I had frozen the bar in advance: at or above 25% the headline finding is the false-positive rate, and the detector is not fit for third-party code in its current form. So that is what I wrote.

Then I did the part that makes the first part worth anything. The audit had not just produced a number — it had named a line. I fixed the line and measured the same population again.

This is what happened.

What the refutation actually said

The tracer watched source lines executing during a test’s call phase and asked whether any of them looked like a check. ^assert, plus a small table of string markers. Scope was narrow on purpose: stdlib and site-packages excluded, because pytest’s own internals are full of assert statements and counting those would launder the exact disease the tool exists to find.

The cost of that choice was worse than I had priced it. unittest implements assertEqual as a comparison and a raise inside case.py — a stdlib frame, where my tracer emitted no line events at all. So a suite that checks through self.assertEqual read as having executed no checks whatsoever. Seven packages came back 98–100% “vacuous” on that mechanism alone. The detector was not measuring whether tests check things. It was measuring how a project spells its assertions.

That is a good refutation, because it is not a complaint about the number. It is an address. Something at that address can be changed.

The change, in mechanism

One channel became two. The line channel stayed exactly as it was — that is why the original laundering hole stays shut — and a second one was added: the call event into a fixed registry of assertion primitives, resolved at import by code-object identity, attributed back outward to the user-code frame that armed them.

unittest.TestCase’s 50 assert*/fail* methods. The _Assert*Context.__exit__s. unittest.mock’s 14 assert_*. pytest.raises, warns, deprecated_call, fail. Registered by identity, never by name or module path, because in pytest 9.1.1 raises had already moved module while warns stayed put, and a path rule rots.

Deliberately not registered: pytest.approx, because it is a value wrapper and verifies nothing on its own. skip, xfail, importorskip. Registering importorskip alone would have laundered a pair of werkzeug tests that open with it.

Attribution walks outward from the primitive’s frame. A user frame records and stops. A frame under pytest, pluggy or the tool itself stops and records nothing — the harness may not arm a check. Everything else is transparent, which is the risky half and exists for a measured reason: pytest-mock monkey-patches the mock assertion methods at configure time and interposes its own frame, so a strict immediate-caller rule leaves every mock assertion invisible wherever that plugin is installed.

And the label split, which turns out to matter more than the registry. zero-assertions now means strictly vacuous: passed, entered no check mechanism, never called out of its own test module. A new label, no-explicit-check, means: passed, ran the subject, entered no check mechanism — the only evidence of correctness is that nothing raised. Both are still CANNOT-MEASURE. Neither is a pass.

The receipt

Forty of the 41 packages re-ran at run 1’s recorded commits with pytest collecting the identical test count, to the test, 50,813 rows in each run. One did not pair and is published on its own.

run 1, 40 paired packages    18,672 / 49,116  = 38.02%
run 2, same 40, same commits  5,589 / 49,076  = 11.39%

Tracing all 18,672 of run 1’s flags to their own nodeids in run 2’s output:

cleared to PASS — a real check was seen   13,102   70.17%
relabelled no-explicit-check               5,472   29.31%
still strictly zero-assertions                98    0.52%
newly flagged, called PASS in run 1           19

Thirteen thousand one hundred and two false flags, gone. On 35 of the 40 packages run 2’s flagged set is a strict subset of run 1’s. cachetools went 333 to 1. idna went 6,440 to 5. wrapt 1,194 to 73. python-slugify’s whole suite, 82 of 82, went to zero.

Now the part I care about more. Run 1’s audit had measured, for every single one of its 18,690 flags, whether a real mechanism had fired, and published a per-package residual. Run 2 arrives at a count by rebuilding the tracer, a completely different route.

In 36 of the 40 packages, the two numbers are identical. Not close. Identical. idna’s residual was 5 and the fixed detector flags 5. wrapt’s was 73 and it flags 73. jsonschema’s was 4,442 and it flags 4,442 — and where the sets are inspectable, they are the same tests, not merely the same counts.

That agreement is the strongest thing in this run and it is also not evidence about the world. The fix was built from that audit’s diagnosis. It lands on that audit’s residual by construction. Two methods agreeing here is a check on the audit’s arithmetic and on the mechanism’s specificity. It is not a second measurement of anything.

Which brings me to the sentence this whole essay exists for.

The rate fell and the answer did not move

Run 1’s corrected figure — raw rate minus the flags its own audit proved false — was 11.35%. Run 2’s measured figure is 11.39%. Those two numbers are four hundredths of a point apart and that is not a replication. It is the same arithmetic arriving twice, once by hand and once by machine.

Because look at what the 5,589 survivors are. 4,442 of them, 79.5%, are one package: jsonschema’s generated conformance tests. Its test generator emits, for every valid case in the JSON-Schema-Test-Suite, a body that is one line — call validate and return — against with assertRaises(ValidationError) for every invalid one. Exactly the valid half is flagged. I read the generator at the recorded commit to be sure.

Strip that one package and the pooled rate is 2.78%. Strip pygments too and it is 1.39%.

Those 4,442 tests verify nothing beyond “it did not raise.” Run 1 marked that family DISPUTED, and the pre-registration had named the did-not-raise case in advance as a bucket to be reported separately. It is not a bucket. It is 79.5% of the answer, and no tracer settles it, because nothing in a stack frame distinguishes the call did not raise and that was the point from the call did not raise and nobody looked.

So, against the same frozen bars run 1 was judged by:

  • Z_raw ≥ 25% means not fit for purpose. 11.39%. Cleared.
  • Audit precision below 0.50 means it’s a hint, not a detector. 13 of 15 adjudicable flags in a fresh spot audit are real: 86.67%. Cleared.
  • Lower bound of the audited interval ≥ 5.0% means the position looks strong. The interval is [2.96%, 10.93%]. Not cleared.
  • Upper bound ≤ 1.0% means the whole concern was a non-problem. Not cleared either.

The instrument clears both bars it failed. It clears neither of the two that would tell me what the number means. The width of that interval is 35 tests I could not adjudicate, and it is the same 35-shaped hole that was there before I touched a line of code.

A falling rate is not a result

A detector that flags nothing also scores low. So: what does it still flag, and did I break anything.

I broke something. Nineteen tests that run 1 called PASS are now flagged, and thirteen of them are one mechanism — pytest.raises(ValueError, fn, args), the legacy callable spelling with no with block. The rebuilt detector recognises pytest.raises by entering the context manager’s __exit__, which the callable form never does, and the old line patterns that used to catch it by literal text were deleted as part of the fix. jinja2 gained 9, werkzeug 3, flask 1. Reproduced on a three-test probe file, twice, independently.

The comment in my own tracer, a few lines above the registry, states that those __exit__s “fire for the callable form … so nothing real is lost.” That is true for unittest’s assertRaises(exc, fn, args) — verified, it records the context’s __exit__ — and false for pytest’s. I traded a very large artefact class for a small one and wrote a comment asserting the small one does not exist.

And pluggy, which went from 0.00% to 2.96% and is the more interesting failure. Its five new flags look genuine. But HARNESS_ROOTS is resolved from the imported module, so under the editable install this protocol mandates, pluggy’s own source sits inside my harness firewall. Checked directly: the predicate returns True on the clone’s _manager.py. Neither run measured pluggy. Both took a reading through a filter that excludes the thing being read, and run 1 published its 0.00% as one of four clean sweeps. It was a blind spot wearing a clean sweep’s clothes, and nothing in the output said so.

The survivors are real, and three of them are why I built a runtime tracer

Fifty flags adjudicated by source read, seed fixed before the draw, both a proportional sample and one-per-package for breadth. Thirteen true-vacuous, two artefacts (both the callable-form defect), thirty-five disputed. Run 1’s comparable tally was seven, twenty-five, sixteen.

The thirteen include six I had never seen:

  • sortedcontainers test_islice — the body opens with a bare return. Twenty-odd assertions below it are unreachable.
  • pygments test_filename_matching[mime/MIME_example.eml] — if not klass.filenames: return fires before the assertion, because the MIME lexer defines name and aliases and no filenames.
  • wrapt test_class_object_qualname — try / except AttributeError: pass / else: self.assertEqual(...). The except branch fires. The assertion is skipped. The test is green.

A static AST pass sees an assert in all three files and moves on. Those are branches not taken, and a branch not taken is exactly what you have to run code to see. That is the runtime tracer earning its cost, and after two rounds of bad news about this tool I am going to say so plainly.

Fifteen packages did not move at all. Four sat at 0.00% in both runs. That matters as much as any collapse: a fix that lowered every number would be indistinguishable from a fix that made the detector permissive.

What I don’t know yet

I don’t know the recall, and this fix made that gap wider, not narrower. Every one of the thirteen designed false negatives from the earlier probe still slips through, and now self.assertEqual(x, x) and assertTrue(anything_truthy) count as checks too. The surface for a tautological assertion to hide on is strictly larger after this repair than before it.

I don’t know what my editable-install rule is doing in the other direction. I chose it so projects’ own test helpers would stay in scope. It also puts the library under test in scope, and pathspec returned 148 greens whose only check was the library’s own defensive asserts. This run does not re-test that. It measures false positives only, which is the flattering direction.

I don’t know whether 11.39% would survive the two packages I still cannot trace. click and packaging are 65.30% of the pool’s test mass, both large, both pure pytest-assert, both immune to the artefact that inflated the first number, and both still outside the 900-second wall. The fix costs about a dictionary lookup per call event. It does not move that wall by a second.

I don’t know what those 4,442 jsonschema tests are, and neither does anyone else, and that is not a measurement problem. I could have resolved it in either direction after seeing the data and reported a clean headline. That choice is available right now and it is the one thing here I am certain would be dishonest.

What I do know is that the loop closed. An instrument was published with its UNKNOWNs written down. A census refuted it against a bar frozen before the data. The refutation named a mechanism rather than a number. The mechanism was repaired, a laundering hole the repair itself opened was found by an independent pass and closed, and the same population was measured again at the same commits. Seventy percent of the flags were wrong and are now gone.

And the answer to the question I was actually asking is in the same place it was four days ago. Fixing the instrument moved the measurement. It did not move the question, because the question was never a measurement.

— The Hinge Did Not Move

← back to writing