There is a failure in small language models that scale does not touch. The trunk can compare two numbers — it has a sense of which one is bigger, it has learned the shape of the relationship — and it still cannot emit the exact value. Doubling the parameters buys nothing, because the gap isn’t a capacity problem. Generating a computed digit sequence left-to-right without carrying real place-value state is a different operation from knowing roughly how large the answer ought to be, and no amount of “be careful with arithmetic” in a prompt converts one into the other.
The architectural response is a hard route. Don’t ask the generator for the answer. Find the expression in what it wrote, compute the value yourself, and substitute. The router isn’t requesting a correct answer — it’s producing one.
arithmetic-wall is the smallest thing that turns that sentence into running code and a number I can stand behind.
What this is not, stated before anything else
This package calls no language model. Nothing here reaches a training run.
The doctrine it implements came out of one, and that run’s figures — the generation failure rates at two model scales, the correlation between trunk output and true magnitude, the before-and-after routing accuracies — do not appear on this page, and they are not what the benchmark below reproduces. I can’t verify them from inside this repository, so they stay out of it. The package’s own README cites versions of those numbers that don’t match the essay they came from, which is its own small argument for keeping them off a page that can’t check either one.
Everything below is synthetic data this package built for itself, specifically so it never has to borrow a number it didn’t measure.
The mechanism
Wrap any text-generation function. Find the arithmetic in its output. Hand the computation to a real evaluator. Put the correct digits back.
shaurya@sps ~ % arithmetic-wall route "Order size: 47 * 83 is 3900 units total."
Order size: 47 * 83 is 3901 units total.
routed: '47 * 83' -> 3900 corrected to 3901
The router only ever rewrites a claim that is both present and wrong. A correct answer is left alone; an expression with no stated answer is left alone, because there is nothing to correct. Only the claimed digits get replaced — the is connective stays put, so the substitution reads like prose rather than like a patch.
The benchmark generates synthetic outputs seeded with error shapes that resemble how models actually miss: off-by-one (a dropped or extra carry), digit transposition (two adjacent digits of the right answer swapped, a token-level slip), and wrong operator precedence (a mixed expression evaluated left-to-right). Uniform random noise would have proven nothing.
shaurya@sps ~ % arithmetic-wall benchmark --n 500 --seed 0 --error-rate 0.9
arithmetic-wall benchmark: n=500, seed=0, error_rate=0.90
before routing: 8.2% of stated answers correct
after routing: 100.0% of stated answers correct
detector found the expression in 100.0% of cases
by seeded error mode (before -> after):
correct n=41 100.0% -> 100.0%
digit_transposition n=198 0.0% -> 100.0%
off_by_one n=219 0.0% -> 100.0%
wrong_precedence n=42 0.0% -> 100.0%
Read that as a shape, not a score. The “before” number is low because --error-rate 0.9 seeds errors into ninety percent of the problems on purpose — it is a harness knob, not an empirical rate at which anything gets arithmetic wrong. Move the knob and the number moves with it:
shaurya@sps ~ % arithmetic-wall benchmark --n 2000 --seed 99 --error-rate 0.3
arithmetic-wall benchmark: n=2000, seed=99, error_rate=0.30
before routing: 69.7% of stated answers correct
after routing: 100.0% of stated answers correct
detector found the expression in 100.0% of cases
[per-mode breakdown elided]
What survives both runs is the thing worth claiming: the after-column is deterministic. It reaches 100% because the router is arithmetic, not inference. That’s the whole argument for routing — the mechanism has no variance to average over.
The evaluator is never eval()
A router that hands model-produced text to Python’s eval() has swapped one untrustworthy generator for a worse one. So safe_eval parses with ast.parse(mode="eval") — which builds a tree and runs nothing — then walks that tree by hand, executing only numeric literals, unary signs, and + - * / // % **. Everything else is refused before a single operator fires.
shaurya@sps ~ % python
>>> safe_eval('__import__("os").system("id")')
UnsafeExpressionError: disallowed syntax node Call in '__import__("os").system("id")'
>>> safe_eval("9**9**9")
UnsafeExpressionError: exponent in '9**9**9' exceeds the safety ceiling
(1000); refusing to compute it
>>> safe_eval("True + True")
UnsafeExpressionError: non-numeric literal in 'True + True': True
That last one matters more than it looks. bool is a subclass of int in Python, so True + True is a perfectly valid arithmetic expression that evaluates to 2 under any whitelist that checks for “is this a number.” It’s rejected explicitly. The exponent ceiling is there because 9**9**9 passes every type check in the whitelist and then takes unbounded time and memory to compute — a denial-of-service that has nothing to do with eval() at all.
What it can’t do yet
- The 100% detection rate is the same project grading its own handwriting. The synthetic generators and the detector were built together, in the same style, so every expression the benchmark produces is one the detector was shaped to find. The rate is reported separately from the accuracy precisely so that circularity stays visible when this is eventually pointed at a real model’s free text — which it never has been.
- A nested parenthesised expression is partially parsed, silently. The candidate scanner supports a flat operator chain and a single number in parens, not
(3 + 4) * 2. GivenSubtotal (3 + 4) * 2 = 15 items.it detects3 + 4, computes 7, ignores the* 2entirely, never associates the claim with anything, and returns the text unchanged. The stated answer is wrong — the correct value is 14 — and the wall lets it through without a word. A clean miss would be better than a confident fragment. - The claim grammar recognises five connectives and nothing else. An answer must follow its expression via
=,is,equals,equal to, or:. Prose like “47 * 83, which comes to 3900 units total” detects the expression and recordsclaimed_value=None, so nothing is corrected, because nothing was understood to have been claimed. Real generator output states answers in the next sentence constantly. - The date guard is a heuristic that cuts both ways.
+and-only count as operators with whitespace touching them, which is what keeps2024-01-15and555-1234from parsing as subtraction chains. It also means genuine subtraction written without that space —3-4— is invisible. Financial notation, other locales’ number formats, and unusual spacing are outside what the tests cover in either direction. - The precedence error mode covers exactly one operator pairing. The generator that seeds it pairs
+/-with*only;/is deliberately excluded to avoid float noise. So the benchmark’swrong_precedencecolumn is a measurement of one specific misordering out of several that a left-to-right evaluator would commit. - Nothing here tests the actual claim. The doctrine is that a model can have magnitude without exact value. This package assumes that and builds the response to it. It does not measure it, cannot measure it from where it sits, and the figures from the run that did are not reproduced here for exactly that reason.
Eighty-three tests cover the whitelist and its refusals, the date and phone-number false-positive guards, the router’s offset tracking across multiple substitutions in one string, each seeded error shape’s arithmetic, the generators round-tripping through the real detector, and the benchmark’s reproducibility from a seed. That’s a count against this source today — and a suite that only ever meets input its own project wrote has been told rather less than a count of eighty-three implies.
— Shaurya