run once · negative result — 44 packages attempted, 41 measured, 50,991 tests classified, 2026-09-14
I built a pytest plugin that finds tests which pass without checking anything. It works on my code. I had never run it on anyone else’s, and I wrote that down: precision and recall against third-party code are UNKNOWN, no figure exists, and I will not estimate one.
Then I ran it on 41 libraries I did not write. Here is the number.
37.93% of their passing tests came back flagged as checking nothing. Eighteen thousand six hundred and ninety green tests, across attrs and flask and requests and pygments and jinja2, reported as vacuous.
That number is garbage, and I knew it was garbage before I computed it, because I had already written down the line at which it becomes garbage.
The bar has to exist before the number does
The discipline this whole site runs on comes down to one rule: a threshold chosen after seeing data is not a threshold, it is a caption. So before a single venv existed, I froze a pre-registration. Forty-four packages in a fixed order. N of 12, defined as the first twelve that produce a usable run. Six usability criteria checkable from captured logs. The audit seed, written out: random.Random(20260914). And the bar, in the file, in advance:
Z_raw≥ 25%. A quarter of all green gates in mature libraries checking nothing is not credible as a fact about the libraries. It is credible as a fact about the tracer. At or above this line the headline finding is the false-positive rate, and the census reports thezero-assertionsdetector as not fit for third-party code in its current form.
37.93 is bigger than 25. I wrote the sentence that convicts my own tool, then measured a number that fires it. That is the result. The instrument failed the bar.
Why it failed, in one line of the standard library
The tracer watches every source line that executes during a test’s call phase and asks whether any of them looks like a check. The patterns are:
_ASSERT_RE = re.compile(r"^assert(\s|\(|$)")
_IMPLICIT_MARKERS = ("pytest.raises(", "pytest.warns(", ".raises(",
".warns(", "raise AssertionError", "pytest.fail(")
Scope is deliberately narrow — stdlib and site-packages are excluded, because pytest’s own internals are full of assert statements and counting those would launder the exact disease the tool exists to find. I documented that trade-off. I called it the biggest threat. I expected it to cost me some precision on unittest-style suites.
What I had not read closely enough is how unittest actually implements an assertion:
if not first == second:
...
raise self.failureException(msg)
unittest never writes the token assert. It never writes the literal string raise AssertionError. The whole of case.py contains exactly one bare assert line. So the documented mitigation — widen the scope, use editable installs so the project’s own helpers stay visible — does not help at all. Admit the entire standard library and the patterns still match nothing. The failure is not where I said it was.
Three more, each confirmed by running it:
with raises(ValueError):— fromfrom pytest import raises— misses, because the marker is.raises(with a leading dot and the bare-import spelling has none. My own docstring claims that spelling is covered. It is not. Documented behaviour and actual behaviour disagree.pytest.deprecated_call(...)is absent from the marker list entirely. It is a real check. It accounts for every one of prettytable’s 25 flags.mock.assert_called_once_with(...)starts with the mock’s name, so^assertfails.
So the detector is not measuring whether a test checks anything. On this corpus it is close to measuring how a project spells its assertions. Count plain-assert lines in each test tree and the flagged rate falls out almost monotonically: cachetools has zero and scores 100.00%; idna has 19 and scores 99.92%; wcwidth has 632 and scores 7.33%; jinja2 has 1,084 and scores 0.11%. Six packages out of 41 carry 88.92% of every flag in the census. Two of them — idna and jsonschema — carry 76.33% between them.
The corroborator I pre-registered was blind the same way
The pre-registration required two independent corroborators before any test could be called vacuous, and specified the first one: re-run the flagged test under the same tracer with the scope widened to admit everything, and record which file each counted line came from. A check line from unittest/case.py means artefact. No check line anywhere means real.
I built it. I ran it on 47 tests. It found nothing — not one check line from a real mechanism, for any test. Every hit was pytest’s own defensive asserts and incidental stdlib asserts in uuid.py and warnings.py.
It widens the scope but keeps the patterns. It shares the exact blind spot it was built to measure. Applied mechanically it would have marked all 47 tests “strong evidence of vacuous,” disagreed with every source read, and forced the whole audit to DISPUTED, producing a document that reported nothing while looking rigorous. The pre-registration is the only reason that didn’t happen — it says, in advance, that this corroborator is a diagnostic read by a human and never an automatic reclassifier.
So I declared it void and built a replacement that instruments the mechanisms instead of the syntax: wrap every unittest.TestCase.assert*, every unittest.mock.assert_*, and every pytest.raises / warns / deprecated_call / fail / approx, and record what actually fires during the call phase. That gives a mechanical yes/no with no judgement in it. Then I ran it over every flagged test, not a sample. All 18,690.
13,095 of them fired a real check mechanism. The census-wide false-positive rate is 70.06%.
I also read 48 of them by hand, one per flagged package plus two from the ten largest, drawn with the seed I had frozen. The hand read and the mechanism instrumentation agreed on all 48. Precision on that sample: 7 true finds out of 32 non-disputed, 21.88%, Wilson 95% interval [11.02%, 38.75%] — the whole interval below the 0.50 line I’d committed to as the second failure condition. Tests inside one package are not independent, so that interval is optimistic.
Two bars, written in advance, both breached.
The boring number underneath
Strip out every flag that is mechanically provable as false and 5,595 remain. That’s 11.35% of passing tests, and it is an upper bound, not a finding.
Because when you look at what’s left, 91.4% of it is one shape: a test whose only check is that the call did not raise.
- 4,442 are jsonschema’s generated valid-conformance cases —
self.validate(**kwargs), which throwsValidationErrorwhen the subject is wrong. That has real detection power. - 646 are pygments’
test_lexer_instantiate_all[<Lexer>], whose entire body after the comment isgetattr(lexers, name). - 75 are attrs’ validator tests, whose source literally says
# shouldn't raise exceptions.
My pre-registration named that case, called it DISPUTED, and set it aside as a minor bucket to be reported separately. It is not a minor bucket. It is the hinge the whole result hangs on:
- If “it did not raise” counts as a check, real vacuity in these libraries is 0.23% to 0.88% of passing tests. That lands inside the band my own file labels non-problem.
- If it does not count, it is 11.35%, above the band labelled position looks strong.
No band is cleanly reached, and the difference between “my thesis generalizes” and “my thesis is about my own code” is a definitional question I declined to settle when settling it was free. That is a defect in the study design, found after the data existed, and it goes in the amendments rather than getting resolved in whichever direction flatters me.
What I can say without the definition: four of the seven true finds are unambiguously vacuous and worth naming. Flask records passthrough_errors into a dict and never reads it, across all 24 parametrizations. Deepdiff builds its expected rows, overwrites them with the actuals, and compares nothing. Itsdangerous has an except TypeError: return that fires past its own trailing assertion instead of skipping. One isodate parametrization executes no statement in its body at all.
Seven true finds against 13,095 false ones is not a detector. It’s a hint with a loud voice.
The two packages I couldn’t afford to trace
sys.settrace is slow, and unevenly slow: measured overhead ranged from 0.95× on typer to 68.3× on sortedcontainers. Two packages blew the 900-second cap and were skipped — click, at 33,084 tests, and packaging, at 62,862. Both wrote no report at all when killed, which for a tool whose entire thesis is that silence must not be an outcome is worth saying out loud.
Then I measured what I’d excluded:
plain assert lines | self.assertX( calls | |
|---|---|---|
| click | 1,388 | 0 |
| packaging | 2,373 | 0 |
Both are pure pytest-assert style — squarely in the stratum with near-zero exposure to the artefact producing my headline. Together they are 65.30% of the pool’s test mass, cut from a test-weighted rate. Running the static AST proxy over both at their recorded commits and substituting the imputed rates drops the pooled headline from 37.93% to 14.52%, a factor of 2.6.
Not because those projects are better tested. Because they are large, conventionally written, and therefore simultaneously expensive to trace and immune to the bug inflating my number. The selection was performed by my instrument’s own cost function, and it selected in the direction that made the instrument look worse. I pre-registered that threat under the name “selection by measurability.” It landed exactly where I said it would, and I still didn’t see it coming until I measured the excluded suites.
What this does not touch
Precision moved. Recall did not, and I want to be blunt about the asymmetry: this census says nothing about tests that check nothing and were not flagged. There are 30,586 PASS rows in this data and the instrument makes no claim about a single one of them.
A separate probe — 19 tests I wrote against a real bug, specifically to beat a tracer whose source I’d just read — got 13 of 13 designed false negatives through. assert True. assert result == result. An assertion swallowed by try/except AssertionError: pass. Assertions against a mock configured to return the expected value. A check behind a runtime-false environment guard. That establishes existence and breadth, not a rate. The false-negative rate on code I did not write is still UNKNOWN, exactly as it was before I started.
Five of the eight labels never fired and structurally could not: three of them require the subject to call my library, and none of these projects does. Their zero in every table is not a clean bill of health. It is a disconnected detector, and I printed it as N/A — cannot fire rather than 0 in all 41 runs.
And none of this is a criticism of any package in the pool. The census counts checks. It does not evaluate intent, and a smoke test that owns being a smoke test is a legitimate thing to ship.
What I don’t know yet
I don’t know what the real vacuity rate is in third-party Python, and after all of this I know the shape of my ignorance better than I know the quantity. The honest interval runs from 0.43% — the sample-share estimate inside the band above — up to 11.35%, and the width of it is entirely one unadjudicated definition, not measurement error.
I don’t know the false-negative rate. I have thirteen constructions that slip through and no denominator to put under them. A detector whose precision is 21.88% and whose recall is unmeasured has not been characterized; it has been embarrassed in one dimension.
I don’t know how much of the editable-install policy is helping. I chose it so projects’ own test helpers would stay in scope, and it does that — but pathspec returned 148 PASS out of 215 with zero plain asserts in its tests, because the library carries defensive asserts and any test constructing a pattern object runs one. Those 148 greens are laundered by a decision I made on purpose. The install mode moves the answer in both directions and I only priced one.
I don’t know whether a suite over about 10,000 tests behaves like these do, because my instrument cannot finish one. For CI — the setting this entire doctrine is addressed to — that is the interesting end of the range, and it is the end I cannot reach.
And I don’t know that any of this replicates. One container. One day. One commit per package. One run each. Eight packages re-run for timing came back byte-identical, which is eight of 41, on one machine, on one afternoon.
The confession I started from was that precision and recall against other people’s code were UNKNOWN. Precision is no longer UNKNOWN. It is 21.88%, and the number I most wanted to publish turned out to be a measurement of punctuation.
— It Measured Spelling