The Bet
Most "voice AI" is a microphone bolted onto a chatbot: you talk, it types. The bet behind GBuild's voice layer is bigger. Voice is the primary interface to the whole machine: the operating system, its apps, files, and folders, GBuild itself, and the fleet of AI agents working inside it. Say what you want; the computer does it, shows you what it did, and asks before doing anything it cannot undo.
That bet only pays off if the voice stack is engineered like production infrastructure rather than a demo: routed, guarded, observable, and, above all, trainable. This case study covers the architecture, the custom lab built to train it, and the proactive agent pattern that came out of it.
Three Modes, One Router
The voice layer runs in three modes. Dictation is fast, local speech-to-text with automatic cleanup, built on MLX Whisper running on-device. Conversational is turn-based dialogue with the resident agent, spoken back through on-device Kokoro TTS. Realtime is full-duplex live voice over the OpenAI Realtime API and Gemini Live (3.1 Flash Live), where the model listens, speaks, and calls tools while you are still talking.
Underneath all three sits the same spine: an intent classifier and router that decides what an utterance actually is (a command, a question, a dictation, a destructive request) and dispatches it to the right handler. Commands become tool calls, executed through the same registered action surface the rest of GBuild uses, a Rust sidecar speaking JSON-RPC. That is what lets voice open and drive macOS apps, move through files and folders, operate GBuild's own surfaces, and direct the agent fleet: the router is not parsing strings into shell commands, it is calling the same typed actions everything else calls.
The guardrails are first-class. Destructive intents (delete this, send that, anything with an external blast radius) require explicit confirmation before dispatch. Transcripts pass through redaction before they are logged or displayed. The same policy discipline that governs GBuild's coding agents governs what a voice command is allowed to touch.
The Voice Harness Lab
The hardest problem in voice agents is not speech recognition. It is that an agent's behavior lives in its harness, the prompt, tools, and rules it runs inside, and harnesses are usually tuned by vibes. Change a line, redeploy, hope. The Voice Harness Lab replaces hope with a loop.
In the lab, a human holds a real multi-turn conversation with an agent running on a real harness with real tool calls. Every turn gets a verdict: this was right, this was wrong, here is why. Negative feedback does not go into a notebook. The lab analyzes it and proposes a concrete patch to the harness source, which the human approves or hand-edits. The patch is applied to the actual file, backup first, and the very next run picks it up, so the improvement is observable within minutes. An edit that produces invalid TypeScript fails loudly rather than silently degrading the agent. Every session produces an ordered, replayable journal.
Around the live loop sits a training center: scripted voice skits that put a harness through specific situations on demand (the interruptions, ambiguities, and adversarial phrasings you cannot reliably reproduce by improvising), a router bench for measuring intent-routing decisions, and consent-gated dataset export so good sessions can become training data. The goal is blunt: before a voice agent is trusted with your file system, it should have been through every situation you can imagine, on purpose, with a scorecard.
The Proactive Agent
Most agents wait. You prompt, they respond, and the "autonomous" ones are usually cron jobs wearing a trench coat. The proactive agent is a different discipline, and it needs a different kind of harness: one trained to hold a goal, ask the next question, and act on the answers while the conversation is still happening.
The proof it works: in live internal sessions, a proactive voice agent probes the user about their ideas, interests, and goals, and, from the running transcript alone, completes a product requirements document in real time, section by section, while the user just talks. The same pattern handles personal and business CRM activity live: the conversation mentions a person, a commitment, a follow-up, and the record updates before the sentence ends. The result is an agent experience that is more than cron jobs and markdown files. It feels real, interesting, and useful, because it is doing visible work with you, not batch work behind you.
Agents on the Telephone
The outer edge of the stack leaves the desktop entirely: AI agents programmed to place real phone calls, built on Telnyx telephony behind a Cloudflare Workers comms hub with signature-verified webhooks. This is deployed internally in a test environment for internal testers, and labeled exactly that way: it works, it is being exercised, and it has not been released.
Where It Stands
Currently building, inside GBuild's private development. Dictation, conversational, and realtime modes run today; the harness lab and training center are in active daily use tuning them; the proactive agent and phone-calling agents are proven in live internal sessions and internal test deployment respectively. The pattern underneath all of it is the same one that runs through the rest of GBuild: capability first, then guardrails, then a lab that measures whether it actually behaves, and honest labels until it ships.