Shaurya Singh

← work

windowlint

A linter for one bug: a pass/fail threshold that was true on the day someone typed it, and has been quietly lying ever since.

prototype · 2026-09

A voice agent waits a set number of milliseconds before it says “let me check” mid-turn. That number was 2500. Nobody remembers measuring it. Later someone pulled the real call log and computed the median latency: 3760ms. The threshold was set below the median, which means the filler wasn’t a rare rescue for slow turns — it was firing on almost every turn, and had been since the day it shipped.

Nothing was broken. The gate ran. It printed an answer. It just stopped being an answer to the question anyone thought it was answering.

windowlint is the tool that would have caught that before it shipped. It looks for one shape: a bare numeric literal sitting next to a comparison operator inside a gate, with nothing behind it. A threshold isn’t the number — it’s the number plus whatever it gets compared against, and that second half is where the bug lives.

The mechanism

The static pass walks assert statements, if conditions, and approx(..., abs=) tolerances via ast, and classifies every bare literal it finds into three buckets. derived means the literal carries a # measured YYYY-MM-DD comment or is compared against a same-run measurement call. impossible-to-pass means the literal sits outside the domain the identifier’s own name declares. Everything else is stale, which is the default, and the whole point.

shaurya@sps ~ % windowlint check tests/fixtures
== IMPOSSIBLE-TO-PASS ==
tests/fixtures/density_rule.py:14  same_class_ratio <= 1.35
  reason: threshold 1.35 is above the declared domain's max (ratio in [0, 1]) -- this check can never fail, it protects nothing

== STALE ==
tests/fixtures/filler_threshold.py:16  latency_ms > 2500.0
  reason: threshold 2500.0 is below same-file baseline P50_LATENCY_MS=3760.0 (line 12) -- this fires on the common case, not the tail it's meant to catch
  suggested rewrite: latency_ms > P50_LATENCY_MS * 1.2

summary: 2 impossible-to-pass, 4 stale

Two of the six findings are shown; the other four, and every finding’s suggested-rewrite diff, are elided. Two things in what’s left are doing more work than they look like. The density rule is flagged not because 1.35 is a weird number but because the variable is named ratio, and a ratio cannot exceed 1.0, so the comparison can never come back red — it’s a gate that has never been able to say no. The filler threshold is flagged because the same file already contains a constant named like a measured baseline, and the threshold sits below it on a trigger comparison. That is the exact signature of somebody guessing a number instead of running the query.

windowlint never rewrites your files. Every suggestion prints as a unified diff for a human to accept or reject, because * 1.2 is a template, not a verified fix.

What static analysis structurally cannot see

A threshold read from a config file, or measured three functions away from where it’s compared, looks identical in the source text to a well-derived one. So the runtime pass stops guessing and asks the test to say what it means:

with windowlint.window("p50_latency") as w:
    p50 = compute_p50(real_call_log)
    w.measure(p50)
    assert p50 <= w.threshold(5500)

w.threshold(5500) returns 5500 unchanged — it’s a transparent pass-through — but it records, at the moment of the comparison, whether a same-run measure() already landed in that window. If it didn’t, the comparison had nothing behind it:

shaurya@sps ~ % python examples/demo.py
[static pass elided; absolute paths shortened]
========================================================================
RUNTIME PASS: a threshold with no same-run measurement behind it
========================================================================
== STALE ==
examples/demo.py:95  window('unrelated_config_gate').threshold(20.0)
  reason: window 'unrelated_config_gate' compared against 20.0 with no same-run measurement recorded in this window before the comparison

summary: 1 stale

A pytest plugin resets the recorder before every test, so “no same-run measurement” means this test’s run, not “somebody measured something earlier in the session.” That reset is the load-bearing part. Without it, one well-instrumented test at the top of a file would launder every lazy threshold below it.

Pointed at itself

The README claims windowlint’s own source has no gate assertions of the kind it looks for. The code disagrees, and the code wins:

shaurya@sps ~ % windowlint check src
summary: 3 stale
shaurya@sps ~ % windowlint check tests
summary: 2 impossible-to-pass, 62 stale

Fifty-four of those 64 findings are equality comparisons — assert len(findings) == 1 and its relatives — checks on a test’s own expected shape, which are bare literals in an assert and therefore indistinguishable, to this tool, from a drifted gate. That is not a bug report I want to hide. It’s the honest cost of the scoping decision: windowlint decides what’s gate-shaped by syntax, and a test suite is full of things that are syntactically gates and semantically nothing of the kind.

What it can’t do yet

  • A provenance comment never expires. # measured 2011-01-01 clears a literal exactly as thoroughly as one measured this morning — I ran that case and got provenance comment marks this as measured on 2011-01-01, filed under DERIVED. A linter built to catch constants pinned to an old day currently accepts an old day as proof, and pushes the “is this recent enough” judgment onto whoever reads the report.
  • Equality comparisons get no domain check at all. assert hit_rate == 1.35 is impossible in exactly the same way hit_rate >= 1.35 is, and windowlint reports it as ordinary STALE, because the domain logic only handles <, <=, >, >=. The one bucket that names a definite bug silently skips the operator most tests are written with.
  • The runtime pass checks that a measurement happened, not that it’s the right one. w.measure(0) on something entirely unrelated clears the window. There is no link between the value measured and the value compared — the recorder stores a float and a “did anything land before this line” flag, nothing more.
  • Anything with “measured” in its name counts as a measurement call. A function called measured_last_quarter() clears a threshold outright. The escape hatch is a substring match on an identifier, which means it is exactly as trustworthy as your naming.
  • The baseline check has no units. It flags a threshold for sitting below any same-file constant named like a baseline, comparing raw floats. A millisecond threshold beside a baseline recorded in seconds will match, confidently, and be wrong.
  • JS/TS is regex, not a parser. Single-line conditions only, and the if (...) pattern refuses expressions containing parentheses — so if (percentile(x) > 2500) is invisible. A real TS AST would fix it at the cost of a dependency this package doesn’t take.
  • Nothing here is cross-file. Only same-file NAME = <number> assignments are tracked. A threshold genuinely derived from a constant in another module reports STALE, and this tool has no way to avoid that false positive.

A hundred and thirteen tests cover the three severity buckets, the provenance parser, the JS regex shapes, the runtime recorder, the plugin’s per-test reset, and the fixtures as narratives. That count holds when the package is pip-installed. From a bare PYTHONPATH three of them fail — the pytest11 entry point never registers, so the inner pytest runs don’t load the plugin, and --windowlint-report comes back as an unrecognized argument. 110 passed, 3 failed, and the difference between those two numbers is a packaging fact, not a code fact. Both are measurements taken today against this source, which is the only kind of claim a tool like this has any standing to make.

— Shaurya

← back to work