Gradbot Voice Agent

Gradbot Voice Agent Builder

Build working voice agent apps using the gradbot framework. Output is a complete, runnable app with backend, frontend, prompts, and configuration.

Important

• ALWAYS read references/backend-template.md and references/frontend-guidelines.md before generating code
• The generated app MUST follow the exact patterns from the reference files — these are battle-tested
• Use the frontend-design skill to generate the static/index.html — do NOT write a generic HTML file

Two Complexity Levels

Simple chat (no tools)

For apps that are purely conversational with no actions/side-effects:

• No tools, no tool handler, no state dataclass needed
• Do NOT pass on_tool_call to gradbot.websocket.handle_session() (omit it entirely)
• Prompt can come from a file OR from the frontend (editable textarea)
• No prompts/ directory needed if prompt comes from frontend
• Pass with_voices=True to gradbot.routes.setup() to enable voice selection UI
• See the "Minimal (no tools)" template in references/backend-template.md

Tool-using agent (with tools)

For apps with actions (ordering, searching, game mechanics, etc.):

• Define tools via gradbot.ToolDef and implement tool handlers
• Track state in a dataclass, update it in tool handlers
• System prompts loaded from prompts/ files
• Include on_tool_call callback in gradbot.websocket.handle_session()
• See the "With tools" template in references/backend-template.md

Workflow

Step 1: Understand the Use Case

Before writing code, identify:

1. Domain: What is the voice agent about?
2. Persona: Who is the AI character? What personality?
3. Tools needed: What actions can it perform? (If none, use simple chat pattern)
4. State: What data to track across the conversation? (If none, use simple chat)
5. UI needs: What should the frontend show besides transcript?

Step 2: Create the App Directory

Simple chat (no tools, prompt from frontend):

<app-dir>/
  main.py
  static/
    index.html
  pyproject.toml

Tool-using agent (tools, file-based prompts):

<app-dir>/
  main.py
  game.py          # domain module (state, tools, tool handler)
  prompts/
    main.txt       # (or base.txt + phase1.txt, phase2.txt, etc.)
  static/
    index.html
  pyproject.toml

If the app has static data (menus, inventories, etc.), include it as JSON files.

Step 3: Write the Backend (main.py)

Consult references/backend-template.md for exact code patterns. Choose the right template:

• Minimal (no tools): ~30 lines in a single main.py, voice selection, frontend-editable prompt
• With tools: thin main.py (~20 lines) + domain module (game.py, ~100+ lines) with state, tools, prompts, and tool handler

Key setup rules:

• Use gradbot.config.load(Path(__file__).parent) or gradbot.config.from_env() for config loading — it loads config.yaml from the app directory and any shared parent config automatically. Returns a Config object with .client_kwargs and .session_kwargs properties.
• ALWAYS call gradbot.routes.setup(app, config=cfg, static_dir=...) to serve frontend and bundled JS
• Pass with_voices=True to gradbot.routes.setup() if the frontend needs voice selection (registers /api/voices)
• rewrite_rules enables language-specific text rewriting before TTS. Get it from voice.language.rewrite_rules (returns "en", "fr", etc.). Do NOT use .value — Lang is not a Python enum.
• ALWAYS set silence_timeout_s to 0.0 — the default 5s causes the agent to re-prompt itself with its last message when the user is silent
• Pass config=cfg to handle_session() to auto-set run_kwargs, output_format, and debug from the config. Or pass them individually for custom setups.

Critical rules for tool definitions (when using tools):

• parameters_json must be a JSON string — use json.dumps()
• NEVER use "type": "array" in parameters (some LLMs like Gemma fail). Use "type": "string" with "description": "Comma-separated list" instead
• Tool descriptions should say WHEN to call the tool, not just what it does

Critical rules for tool handlers (when using tools):

• on_tool_call receives 3 args: (handle, input_handle, websocket) where handle is a gradbot.ToolHandle
• handle.name gives the tool name, handle.args gives parsed args (already deserialized dict)
• Send results via handle.send_json({...}) (auto-serializes) or handle.send(json.dumps({...})) for raw JSON
• Send errors via handle.send_error("message")
• Send UI updates via websocket.send_json({"type": "custom_event", ...})
• Use input_handle.send_config(new_config) to swap prompts/tools mid-session

Critical rules for system prompts:

• Keep responses SHORT: "1-2 sentences max" for voice (long text = slow TTS)
• If using tools: include "Call tools silently FIRST, then speak"
• Be explicit about what the agent should NOT do

Step 4: Write System Prompts (tool-using agents only)

Skip this step for simple chat apps where the prompt comes from the frontend.

For tool-using agents, create prompts/base.txt and prompts/main.txt:

• base.txt: personality, speaking style, boundaries, response length
• main.txt: conversation flow, when to call tools, error handling

For multi-phase apps, create phase1.txt, phase2.txt, etc. and swap via input_handle.send_config().

Step 5: Build the Frontend

IMPORTANT: Invoke the frontend-design skill to create static/index.html.

When invoking frontend-design, provide these requirements:

• It's a voice agent UI — primary interaction is speech, not typing
• Must include a mic/call button to start/stop the session
• Must show a live transcript (user bubbles + agent bubbles)
• CRITICAL: Must load audio via three script tags (NOT ES module imports): opus-encoder.js, audio-processor.js, synced-audio-player.js from /static/js/ — then use SyncedAudioPlayer as a global (it is NOT an ES module)
• Must follow the WebSocket protocol in references/frontend-guidelines.md
• Must include echo cancellation checkbox (checked by default) — without it the agent hears its own TTS
• For simple chat: include voice selector grid, editable prompt textarea, speed slider
• For tool-using agents: include domain-specific content panels (menu, inventory, results, etc.)