The headline number is the one that looks bad
The Foundry's keystone model is a ~150M-parameter ModernBERT grounding verifier: a small on-device classifier that decides whether a claim, or an action's parameters, are actually supported by the tool output behind them — before an agent answers or acts.
It scored 97.5% on its own synthetic holdout. On a 95-row human-labeled gold set, it scored 62%. That 35-point gap is overfitting to the corruption taxonomy, and only hand labeling caught it. The model is not in the product, because the promotion gates it has to pass are code. Publishing that failure — instead of the demo that would have hidden it — is what this case study is about.
97.5 → 62%
synthetic vs gold — the caught overfit
95% LCB
paired-bootstrap promotion bar
$0
marginal training data cost
On-device
private by architecture
Why train models when APIs exist
GBuild runs agents against personal, local context: files, messages, calendars. The questions its small models answer — is this contact important, is this claim grounded, is this document sensitive — are exactly the questions whose inputs should never leave the machine. The constraint envelope is explicit: on-device, private, roughly zero marginal cost, inside a RAM and latency budget. A frontier API fails that envelope on every axis, however good its answers.
The first model through the full pipeline is deliberately small: a Tier-0 contact-importance ranker — a small linear model over locally-aggregated communication features that drop-in replaces a hand-weighted heuristic. It ships with a Spearman rank-correlation gate and graceful fallback when no model artifact is present. Its ingest reads chat.db counts only, never message content; a corpus-readiness probe reports whether the local history is even large enough for the eval to mean anything, in counts and a day-span only.
The measurement discipline
Synthetic data, $0 marginal
Training data is generated by programmatically corrupting ground truth along a typed-field taxonomy: swap a number, break a unit, invert a claim. Each corruption is a labeled negative by construction, so the data engine costs nothing and never plateaus.
Gold sets, human-labeled
The synthetic holdout tells you the model learned the corruptions. Only a hand-labeled gold set tells you it learned the task. Both numbers get published, especially when they disagree.
Champion / challenger promotion
A challenger replaces the champion only past a paired-bootstrap 95% lower-confidence-bound significance bar on the margin. The gates are code, not judgment calls, so a persuasive demo cannot argue its way into the product.
Every gold evaluation ships with a provenance dump, data gates reject leaking rows before training, and results files (RESULTS-gold-v0.md, checkpoint discipline docs) live in the repo next to the code that produced them. A number without its command is treated as a rumor.
A Python trainer, a Rust runtime, one contract
The feature vector is a versioned interface, not a convention.
Training happens offline in Python (generate → train → eval → export a JSON artifact). Inference happens in the Rust sidecar. Between them sits a versioned feature contract: the exact features, their order, and their version travel with the model artifact, so a trainer-side feature change cannot silently skew runtime predictions. The runtime model is a trait-based abstraction with additive fallback — no artifact on disk means the heuristic keeps working, not a crash.
Fine-tuning itself targets encoder classifiers (ModernBERT / DeBERTa) rather than generative models: classification is what the product needs, encoders are what the RAM budget affords, and a champion/challenger harness can score them mechanically.
Where it's headed
The charter, eval math, corpus-readiness probe, and the Tier-0 ranker pipeline are built; the grounding verifier is in active training rounds against its gold set, and it stays out of the product until it clears the same bar everything else does. The strategy documents plan a portfolio — make, fork, or resell per niche — but the promotion rule does not bend for the roadmap.
The Foundry is one subsystem of GBuild. Read the GBuild case study
A model earns its place past a significance bar, or it stays out of the product. Both outcomes get published.