Case Study
The G Training Center: A Null Baseline for an Agent Persona
Thirty-three overlays nobody could tell worked. The instrument that makes deletion an evidence-backed answer.
Updated Aug 27, 2026

It started as an impatience problem
A Cloudflare build was taking the better part of an hour. The change waiting on it was a text file: the persona that tells G how to behave. Nothing about editing a paragraph justifies a 45-minute wait to find out whether the paragraph worked, and yet the loop was set by the compiler rather than by the size of the edit.
That mismatch is the whole origin. When the feedback loop is slower than the thought, you stop running the experiment you actually wanted and start running the one you can afford, which is a quiet way to stop measuring anything. The Training Center closes the loop: the lab points the installed application’s own engine at a swappable home directory, so an edit to the persona is read by the very next run. No rebuild, no release, no wait.
The 45-minute figure is the operator’s account of the build he was waiting on, not a measured benchmark. It is recorded here as motivation rather than as a metric.
3 arms
two of them deliberately identical
21
phrases that get a reply rejected
65 of 100
prompts that need a question first
What is being tuned
G is the single assistant inside GBuild, a fork of the Hermes agent with a rewritten identity and a posture that is the actual product decision. Most assistants are permitted to end a turn with a dead end: I can’t do that, or here is what you should go and do. G is not.
The persona names the rule directly. Every “can’t” becomes a path, in order: unlock it yourself when that is reversible and in scope, raise the one card that asks for exactly the missing grant when it needs a human hand, or ask the fewest questions that let you act. Questions gather what is needed to move; they are never a way to hand the work back.
That last clause is why this matters beyond the developer running it. A person who cannot code is not blocked by the task, they are blocked by the instruction. “Open system settings and grant accessibility, then re-save it as a PDF” is a perfectly clear sentence and a wall. An agent that does the thing, or that raises one specific card and then does the thing, removes the wall rather than describing it.

Two tiers, one home directory
The lab is a bash launcher, a zero-dependency Node server on loopback port 4327, and a browser window. It has no package dependencies at all, which is deliberate: an instrument that breaks when an unrelated package changes is an instrument you stop trusting.
Tier A, headless
One prompt through the installed app's own engine against the lab home directory, about 20 seconds a run. Every run appends to a ledger, and each distinct persona content-hash becomes a version, which is what fills the compare matrix.
Tier B, live sim
The real application launched with its home directory pointed at the lab, so the running app and the harness share one persona and one config. The same edit that changed a headless answer is now sitting inside the product, with its own surfaces around it.
The null baseline
GBuild shipped 33 expertise overlays. None of them had ever been validated. The project’s own record of the problem is the sharpest statement of it: they were not broken because nobody built them, they were broken because nobody could tell whether they worked.
The tempting question is which ones to migrate. The honest question comes before that: does the overlay change any outcome at all? If it does not, 33 files delete with no replacement channel needed, which is a far larger simplification than any migration.
Answering it runs into something that quietly defeats most agent evaluation. The engine is stochastic. Ask it the same thing twice and the two answers differ. So a rule of “the arms came out different, therefore the overlay did something” can never reach deletion, because it will fire for a completely useless overlay just as reliably as for a good one. Under that rule the screening always says keep, and the screening is worthless.
overlaytreatmentThe spawn receives the expertise text on the ephemeral system-prompt channel, exactly the way production delivers it.
nonecontrolNo overlay at all. The same prompt, model, and toolset, with the thing under test removed.
none2null baselineAlso no overlay. Treated identically to the control in every respect, which is the entire point: whatever separates these two is nothing but the engine talking to itself differently twice.
The two identical arms are the instrument. The distance between them is the engine’s own noise floor, measured on the same day, on the same machine, under the same conditions as the treatment. An overlay counts as having done something only when it separates from the control by more than the control separates from its own twin. Anything less is overhead: system-prompt tokens on every single turn, buying nothing that can be distinguished from the engine having a different morning.
Repetitions interleave across arms rather than running one arm to completion at a time, so slow drift in the engine or the network spreads evenly instead of landing entirely on whichever arm happened to run last.
The part that is deliberately not automated
Mechanical difference is measured. Whether the difference is an improvement is graded by a person, and an ungraded arm reports pending, never a fabricated pass.
It would be easy to add a model-as-judge arm and report a score. The reason not to is structural: an unvalidated judge grading an unvalidated overlay is a second unmeasured layer scoring the first, which is precisely the stack the entire effort exists to stop. The lab would go from measuring nothing honestly to measuring nothing confidently, and the second state is worse.
So the automated half answers only the question a machine can actually settle: did the output change beyond the noise? The verdict half stays human, is recorded in the ledger next to the run it belongs to, and is missing rather than invented when nobody has looked.
Twenty-one phrases that get an answer thrown out
The posture in the persona is enforceable rather than aspirational, and this is where the novice-user thesis stops being a value statement and becomes a table. Production carries two phrase lists. A reply matching either is rejected and re-run with a directive explaining what it did wrong.
21 homework phrases
“head over to”, “navigate to the”, “go to system settings”, “you will need to install”, “manually add”, “re-save it”. Every one of them hands the work back.
11 hedging phrases
“the tools available in this session”, “cannot truthfully produce”. The operator cannot see a tool list, so blaming one is exposing internal machinery rather than explaining anything.
The homework directive is the clearest single sentence in the system, and it is worth quoting as written:
“You have cards for exactly this. Raise the permission request for a macOS permission, the connect card for an account, a tool request for a missing capability, or an approval otherwise, then do the job yourself. If a surface is worth showing, it is brought to the operator, never pointed at.”
Brought, never pointed at. That is the difference between an assistant that is useful to someone who already knows the software and one that is useful to someone who does not.
Every control on the run bar
The defaults are chosen so that an unmodified run is as close as the lab can get to the real thing: model blank, session fresh, overlay empty, lean toolset, live context on. Each control below is a deliberate departure from that baseline, and the interface says what each departure costs in comparability.
toolsetsMatch the real G chat · Minimal, clarify only · All tools (heavy)Which tools the agent may reach for. The default is the lean set production plain-chat actually uses, so a run is comparable to the shipped drawer rather than to a maximally equipped version of it. The heavy option exists to see what changes when nothing is gated, which is a different question and is labelled as one.
live contexton by defaultReproduces what the real drawer always injects: the application manifest, live time, the operator’s custom instructions, and, on a best-effort basis, connected accounts. It deliberately does not fake live location or device permissions, and the interface links to a written list of what is omitted rather than quietly substituting something plausible.
modelblank for the config defaultLeft empty, the run uses whatever the configuration resolves to, which is the closest match to the real drawer. Naming a model here is how you ask whether an answer survives a model change, since the persona is written to be model-agnostic and that claim is testable.
sessionfresh · continueFresh is a cold start with no prior turns. Continue carries the thread, which is how you catch behavior that only appears on the third turn rather than the first.
max turnsoptional ceilingCaps how far a single run may go on its own. Useful when the question is whether the agent finishes in one move or wanders.
hatblank to auto-routeBlank sends the prompt through production’s own compiled router. Naming one forces it, and the identifier is validated live rather than accepted on faith: an unknown id runs the plain agent instead of silently pretending a route matched.
temporary expertise overlayfree textFills the same slot a hat’s expertise text would occupy, layered underneath the injected context. This is the field the whole overlay A/B is built around, exposed directly so a hypothesis can be typed and run before it is ever committed to a file.
Saved scenarios, one click each
Typing a prompt fresh every time is how a test suite becomes anecdote. The left rail holds saved prompts grouped by the surface they exercise, each runnable individually or as a group, at roughly 20 seconds a run.
The groups map to the surfaces G actually appears on: plain conversation, where voice, warmth, and proactivity are the whole test; maps knowledge through the mapping tools; macOS control, where the interesting moment is the confirm-before-acting pause on anything destructive; and native application work through the same tool surface the product uses. A group carries its own caveat where one applies. The maps group notes that production voice runs an inline path rather than this one, so routing has to be verified separately rather than assumed from a passing run here.
Every run lands in an append-only ledger with its prompt, its settings, and its grade if it has one. Each distinct persona hash becomes a version, and the compare view puts versions against scenarios so a persona edit can be read as a column rather than as a memory of how it used to answer.
Consent cards, previewed live
The right rail renders the real consent card component against edited content, so the wording of an irreversible action can be read in its actual visual form before it is ever put in front of a person. Reversibility is a class rather than a tone, and the class drives the card.
Low stakes
Reversible by construction. Renaming files, where the original names are logged.
Irreversible
Deleting files that do not go to Trash. The card has to say cannot be undone, because it cannot.
Money
Funds leaving the account immediately, with the amount and the recipient on the card rather than in the prose above it.
Publish
Something becoming publicly visible. Reversible in theory and not in practice, which is why it gets its own class.
Each card carries the resources it touches, a plain reason, and a consequence line, plus the option to stop asking, which is the setting that deserves the most scrutiny and gets it here rather than in production.
The instrument cannot drift quietly
A lab that slowly stops matching the product is worse than no lab, because it keeps producing confident results about something that no longer exists. Three mechanisms hold the two together.
The parity manifest is generated from the production Rust source, not hand-written: envelope fragments, tool-gating tables, session and timeout values, the response-guard phrase tables, and configuration defaults are all emitted by a build command and checked fresh by a pre-commit gate. Hand-editing it is explicitly forbidden, which removes the failure where the lab is corrected to match a belief rather than the code.
The persona hash is compared against the shipped file on every load, and any difference raises a visible drift warning rather than being left to be inferred from a surprising answer. The deployment script derives the shipped file set from the actual import graph, so a module that is imported but missing fails the deploy loudly instead of shipping a bundle with a hole in it.
engineThe engine version the lab actually runs, which is the installed application, not the newest one in the repository. When a newer build is staged, it says so in a separate chip rather than letting you assume the number you are reading is the one executing.
personaThe current persona label and content hash, plus a drift warning when the lab file differs from the shipped one. Byte-identical means a run mirrors the real drawer; anything else means the answers will not match, and the interface says that instead of leaving it to be discovered.
backendsWhich speech-to-text and intent backends this machine’s installed application would resolve to. When the lab cannot host the resolved backend, it reports that the card is unavailable rather than quietly substituting a different model and calling the result the same thing.
bug-parityA deliberate switch that reproduces a known production fail-open, where a capability lookup returned the full tool catalog instead of the composed diet. Turning it on makes the lab wrong in exactly the way the shipped build was wrong, which is the only way to reproduce a bug that depends on it.
fixtureRecord captures every served call and the composed envelope; replay serves them back frozen. A replay miss is a hard error rather than a silent fall through to a live call, so a graded run cannot be quietly contaminated by fresh network results.
mcp parityWhether the lab’s tool surface still matches production’s. This is the check that stops the instrument from drifting into measuring something the product no longer does.
One hundred prompts, and the number inside them
Running scenarios someone invented to pass is a way of grading your own homework. So the lab carries a corpus of 100 prompts written by the operator as a person would actually phrase them, classified along lanes, reversibility, consent shape, duration, and risk.
The most useful number to fall out of that classification: 65 of 100 need a clarifying question before the request can be acted on at all. Two thirds of what a real person asks for is underspecified, not because they phrased it badly, but because natural requests carry assumptions the asker never had to make explicit. “Reorder my meetings if traffic changes” does not say whether to move the driving route or the calendar invitations.
That is the measured case for a proactive agent rather than a compliant one. An agent that executes the literal request gets two thirds of them subtly wrong, and an agent that asks the fewest questions that let it act then finishes the job is the only design that fits the data.
Status, stated plainly: the corpus is authored and classified, not reviewed. 15 of 100 entries carry operator review state, and every one of those currently records no verdict, because the earlier two-value interface conflated “the classification is right” with “the reasoning is right” and those verdicts were marked unreliable rather than kept. The prose is the label; the button was not good enough.
What this is not
It is an internal instrument, not a product. It binds to loopback only, it is never publicly reachable, and it has no users in any sense that would make a usage number meaningful. There is no deployment to point at.
The overlay verdict is also not in yet. The instrument exists, the arms run, and the question it was built to answer, whether 33 overlays earn their tokens, is still open. Real runs cost money and are gated behind an explicit environment flag, and the dry-run path uses a deterministic stub precisely so the machinery can be exercised and self-tested without spending anything or pretending a stub result is evidence.
137 commits over 41 days. Roughly 10,953 lines of hand-written JavaScript and HTML, of which 1,846 are its own self-tests, plus 38 contract-test files and 31 local endpoints.
The product this tunes
G lives inside GBuild, an AI-native build environment for macOS. The Training Center exists to change G without shipping GBuild.
Read the GBuild case studyTech Stack
Backend
AI
Tooling
Observability
Frontend
Want to discuss the architecture?
Get in Touch