Past five or so concurrent lanes, my fleet stopped failing on compute and started failing on coordination. The recurring shape was duplication: two agents independently deciding the same thing looked worth fixing, going at it on different branches, and both producing real work that then had to be reconciled or thrown away. It was the complaint that came back most often, and it is not a scheduling problem — nothing was over capacity.
What I wrote for it was a charter: three standing rules and a claim registry that a human chief audited every loop. That held. It also costs a human every loop, and a table nobody reads stops being ground truth and becomes decoration. onelane takes the one clause of that charter that reduces to a fact a computer can settle atomically — is a live, non-expired claim already held for this work_id — and removes the person from it. Everything else the chief does, like deciding which branch is canonical, stays with the chief.
The mechanism
A claim is a lease with a TTL, and the load-bearing part is that liveness is something you keep proving rather than something you were granted once. claim() is atomic and returns either a Lease or a Refused that names who actually holds it:
shaurya@sps ~ % python examples/demo.py
--- 1. Two workers race for the same work_id ---
worker-a claim -> Lease(work_id='render-frame-042', owner='worker-a', ttl=2.0)
worker-b claim -> Refused(work_id='render-frame-042', reason='held', held_by='worker-a')
exactly one winner, as guaranteed: worker-b was refused, not queued behind.
--- 3. worker-a 'crashes' -- stops heartbeating, TTL elapses ---
reap() found 1 stale lease(s):
work_id='render-frame-042' prior_owner='worker-a' held_for=2.22s overdue_by=0.21s
--- 5. Audit trail: StaleClaimRegistry remembers the reap ---
reaped 'render-frame-042': 'worker-a' held it 2.22s, 0.21s past its deadline
Nobody had to notice worker-a was dead. A lock file answers “is this claimed” and answers it forever, so a dead holder’s lock has to be broken by someone who first has to observe the death. A TTL answers “is this claim still alive,” and a claim nobody is refreshing simply stops counting.
A reaped work_id becomes free, not reassigned. Getting a new owner always takes a fresh claim(). Nothing hands work to anybody behind their back, which is the property that makes the audit trail readable afterward.
The registry is the artifact, not the lock
The lease store is already correct the moment it lets a fresh claim through. The registry exists for the question that comes later — who was sitting on this, and how far past their deadline before anyone noticed. That is a governance question, and it is invisible if reap events aren’t kept:
shaurya@sps ~ % onelane status leases.db
Live leases (2):
work_id owner claimed expires_in
---------------- -------- -------- ----------
render-frame-042 worker-a 1.2s ago 4m58s
encode-audio-007 worker-d 0.5s ago 4m59s
Overdue leases, not yet reaped (1):
work_id owner claimed overdue_by
------------- -------- -------- ----------
mux-final-003 worker-e 0.5s ago 0.2s
Reap history (2 total, showing last 2):
work_id prior_owner held_for overdue_by
---------------- ----------- -------- ----------
encode-audio-007 worker-b 0.6s 0.1s
ship-report-019 worker-c 0.6s 0.1s
Three facts, which are the three a chief scans a registry table for, minus the scanning. One lane consistently blowing its TTLs is a signal worth seeing, and it only exists if something wrote the reaps down.
The stress test fails on this machine, and that is the finding
The README says the concurrent-crash test “pins this down under real threading contention, not just asserted.” On a loaded box it does not pin it. It fails it most runs — 14 of 20, invoked on its own, while the machine was running a dozen other jobs. Re-run on the same box an hour later with the load gone, it went 0 of 10. Both numbers are real and they were measured a few hours apart, which is the actual finding: the outcome is a function of contention on the host, not of the code under test:
shaurya@sps ~ % pytest -q
FAILED tests/test_workqueue.py::test_exactly_once_under_concurrent_crashes_sqlite
- AssertionError: a job was processed more than once: ['job-0', 'job-1', ...
1 failed, 94 passed in 6.85s
(pytest prints that as one line and cuts it at the terminal width; it is wrapped here, and the ellipsis is pytest’s own.)
I instrumented the same scenario to find out why, and the number that explains it is not about the lease store at all. Eight threads race twelve jobs on a 40ms TTL. Across sixteen instrumented runs, twelve to forty work items got processed for those twelve jobs, and zero to thirty-five of the complete() calls released nothing — because the lease had already expired and been reaped out from under a worker that was still alive and still working. The spread is the point: there is no single number here, which is why the test is a coin weighted toward red rather than a clean red.
The store told the truth the whole time. release() returned False on every one of those, and heartbeat() would have returned Refused(reason='reaped'). What the store cannot do is reach into the worker and stop it. A lease expiring revokes permission; it does not halt the holder. So WorkQueue’s exactly-once claim is really at-least-once whenever a worker outlives its own TTL, and no amount of atomicity in claim() changes that — it needs a fencing token checked at the point where the work has an effect, which is outside this library.
What it can’t do yet
WorkQueuecannot give you exactly-once, and the test that says it can is the one that fails. A worker whose TTL expires mid-task keeps running. The item gets reaped, requeued, and done a second time. The store refuses the stalerelease()afterwards, which is correct and also far too late to help.- That test is load-dependent, not flaky, and that is worse. It went 14 failed / 6 passed over twenty runs on a heavily loaded host, and 0 failed / 10 passed on the same host once the load cleared. Flaky would at least be honest noise. This is a test whose verdict is set by something it never measures and never reports, so the same code returns green on a quiet laptop and red in a busy CI runner, and neither result knows about the other. It carries no marker, no retry policy, and no record of host load, which means a green run here is not evidence the invariant holds — it is evidence the box was idle.
- An item whose
heartbeat()is refused is dropped from the queue entirely. The queue removes it from both in-flight and known bookkeeping, and the later reap sweep only requeues ids it still considers in-flight — so the work_id ends up in neither pending nor in-flight, and nothing ever re-offers it. Doing the responsible thing and heartbeating is the path that loses work. - Every expiry decision is
expires_at <= time.time()on whoever is asking. No logical clock, no NTP-drift correction, no defense against a clock that jumps. A TTL is exactly as trustworthy as the system clock evaluating it. - This is single-machine locking, not consensus. A
threading.Lockfor threads in one process, stacked on sqlite’sBEGIN IMMEDIATEfile locking for a second process on the same file. That is real mutual exclusion and it is not Raft. Point two machines at a lease database over a network filesystem and the guarantee is gone. - No fairness among refused claimants. Exactly one caller wins a contested work_id, which is tested. Which one wins is whoever reached the lock first.
WorkQueue.pop()is FIFO only because the queue tracks pending order itself; rawclaim()races have no order at all. - A claim that is live but stuck is not this library’s problem.
onelaneknows whether a lease was renewed on time. It has no opinion about whether the work behind it is moving, and a worker that heartbeats faithfully while achieving nothing looks identical to one that is making progress.
Ninety-five tests cover claim refusal, heartbeat reap-and-steal detection, single-delivery reaping under concurrent sweeps, the registry’s audit trail, the CLI, and both backends. Ninety-four passed on the two-core box I measured this on today and the ninety-fifth is the one above — though on a rerun it can be ninety-five, which is the less comfortable half of the same measurement. I would rather leave it failing and write down its flake rate than tune the TTL until it goes quiet.
— Shaurya