anthill: A Local Code Reviewer That Measures Its Own Precision

anthill is a code reviewer that never stops. It hands every changed version of every tracked file to a Qwen3-8B model running on a Mac mini, and periodically has Claude Opus grade the results. In its first 60.4 hours it completed 1,767 review runs across 705 distinct versions of 457 files, and pushed 5.04 million prompt tokens through the local model at no marginal cost.

Then the paid audit looked at 182 of its findings. It upheld 3 and rejected 179. That is the case study. Local inference solves cost, privacy, and availability; it solves nothing about quality. Running more of it would not have helped, because volume was never the constraint.

What makes the design worth keeping is that the model is allowed to be wrong cheaply, and what the numbers say next is that it is not yet cheap enough. 182 findings reached the layer that bills, and 179 of them should have been stopped before it.

The full argument, with the numbers in context: "What a Local Model Is Actually Good For"

Two Macs, one Ethernet cable. The mini has an M4 and 16 GB; the MacBook has an M5 and 24 GB. On the day this was set up the mini sat at 83% memory free with 1.2 GB of swap, and the MacBook, the larger machine, was at 34% free with 18.3 GB of its 19.5 GB swap in use. The smaller computer had room to think, so it became the inference host.

The cable runs at 1000baseT, roughly 100 MB/s measured. Unified memory moves about a thousand times faster, so pooling RAM across the two machines is not possible and was never the plan. Only text crosses. The model lives where the memory is, the work lives where the code is, and the arrangement removes a distributed queue and a SQLite-over-NFS corruption path in one move. mlx_lm.server holds the model at roughly 16 tokens per second; a real 14.6 KB diff takes 6.2 seconds.

The cost curve is what changes the design. A hosted model charges per unit of work, so every review is a decision. A local model charges for hardware once, so additional reviews are free and time is the only limit. Measured capacity here is about 14,000 reviews a day against a real need of roughly 250. There is no reason to ever throttle it, so the reviewer reads every version of every file rather than only the ones a pull request draws attention to.

Findings pass through 5 filters before anything bills. The order is deliberate: free and deterministic first, expensive and stochastic last. Inverting it means paying frontier prices to discover that a finding quoted code which was never in the file.

The first filter uses no model at all, and it is the one carrying the economics: 85 findings quoted code that was not in the file and died in a string search, for nothing. The middle filters are the measured weak point. They are the next thing to fix, and that is a more useful thing to publish than the token count.

Giving software its own computer forces the safety question on day one. These are constants and refusals in the source rather than intentions in a document.

The first coordination mechanism was a heartbeat table: one row per agent, overwritten. It answers the wrong question. The dangerous state is not that a process holds a lock, it is that a process held one and then died, and an overwritten row keeps no history and cannot say what anyone is holding.

It became a small bus with four primitives: what is true, what is held, an append-only log to tail, and what the human has permitted. Everything expires, so the mechanism that recovers from a dead agent is the same one that grants a live agent permission. The model server's own file lock is enforced by the kernel and visible to no other process, which is exactly how the memory steward would otherwise have frozen the reviewer mid-lock and deadlocked both.

Presence is scoped per machine for the same class of reason. A single global "someone is at the keyboard" flag made the reviewer ease its batch from 25 to 3 because a human was typing on the MacBook, while the work it was easing ran on the mini, where nobody was sitting. An unmeasured machine now returns unknown, and unknown means callers change nothing.

The question this started with was whether a small model on cheap hardware could stand in for a large model on someone else's. It cannot, and 3 of 182 is the number that settles it.

The framing was the mistake. A local model is not a discounted frontier model; it is a different instrument, one that produces candidate volume at no marginal cost on hardware you control, with no quota and no queue. 452 findings that are mostly wrong is noise. It becomes useful only once something can discard it for free, which makes the 85 findings that died in a string search closer to the real result than the five million tokens.

The 3-of-182 number is a measurement, not a verdict. The move after reading it was to find where the loss happened rather than to argue with it, and the answer was already sitting in the vote table. The three lenses meant to challenge every finding had cast 90 votes and refuted 0 of them. A verification stage with a 0% refute rate is not verifying anything. It is a rubber stamp with a latency cost, and it is how 179 findings walked into the layer that bills.

Those lenses were replaced on 2026-08-01 with three that have to cite an artifact before they are allowed to agree. Across 1,010 votes the new set refutes 341, and the spread between them is the part worth having: one refutes at 51.6%, the next at 44.6%, the third at 5%. Three lenses that disagree with each other are asking three different questions. Three that never refuse anything were asking one question and hearing one answer.

What has not moved is the number that matters. The 17 audits since the rebuild have reviewed 159 findings and upheld 0 of them. All 3 upheld findings on this page came from the first audit ever run. Presenting the improved refute rate as a win would repeat the exact error this case study is about, which is mistaking an intermediate metric for the outcome.

It is still the tractable kind of problem, and that is the reason to keep turning the crank rather than start over. The failure lives in prompts, thresholds, and ordering rather than in model weights, so an iteration costs an afternoon instead of a training run. Every rejection is stored with the auditor's reasoning, which means there are 179 labelled negatives sitting in the same database as the findings they describe. The audit layer already writes rules back into the reviewer, and it has written 82 of them so far. The loop that has to close is built and turning. It has not produced a measured gain yet, and until it does, that is what this page will say.

The cross-model judge carries its own warning label. It refutes 46 of the 55 findings it has seen, and a rate that high is watched rather than celebrated: near zero would mean it had started rubber-stamping like the lenses it replaced, and near 100% would mean the same one-directional bias pointing the other way. The question being tracked is not how often it disagrees, but whether it disagrees for reasons that survive being checked.

Adopting a protocol is a decision that can be wrong in both directions, so each was checked against this system's actual threat model and against the current spec rather than against its landing page. 5 decisions came out of it, and only one of them is an adoption. Each carries the trigger that would reopen it, because a decision without a revisit condition is just a preference.

One spec reading settled the largest of these. The 2026-07-28 MCP revision deprecates Sampling and Roots, which forecloses "the second machine joins the bus over MCP" by specification rather than by preference, and the documented replacement, talking to the model API directly, is already what this system does. Reading the changelog was cheaper than discovering that in an implementation.