Building Livekit Agents

Building LiveKit agents

This skill covers how to structure a voice agent. It has no API specifics, because those change; get them from reading-livekit-docs.

It assumes LiveKit Cloud, the recommended path: managed infrastructure, plus LiveKit Inference for models so you don't manage per-provider API keys.

Where the agent runs and which LiveKit the project uses are separate questions. An agent the user self-hosts (on their own servers instead of LiveKit Cloud's agent hosting) still connects to LiveKit Cloud and can still use LiveKit Inference. Inference is a LiveKit Cloud feature, so it's only off the table when the project runs on LiveKit OSS. The architecture advice applies either way; on LiveKit OSS, models come from each provider's own plugin and API keys.

Before you write code

1. Load reading-livekit-docs and look up the APIs you're about to use. Don't write LiveKit code from memory.
2. Confirm the project is connected to a LiveKit Cloud project (or a LiveKit OSS server): LIVEKIT_URL, LIVEKIT_API_KEY, LIVEKIT_API_SECRET, usually in .env. The CLI can set these up.
3. Decide the workflow shape before writing the first agent class (see "Structure" below). Splitting a monolith into handoffs later is much more work than starting with two agents.
4. Plan how you'll verify it. Decide now whether you'll use debugging-livekit-agents (drive a real conversation), testing-livekit-agents (assert on turns), or both, because it affects how you factor the code.

How voice changes the requirements

A voice agent is more than a chat agent with a speaker attached. These constraints drive most design decisions:

Latency. Users expect a reply within a few hundred milliseconds. Context size, tool count, whether a tool call sits on the critical path, and whether responses stream all add to or save from that budget. Plan for network stalls and provider timeouts too; they happen routinely.

Context size. A 10,000-token system prompt with 50 tool definitions feels sluggish on any model, because the model re-reads all of it every turn. Give each phase only the tools it can reach and the instructions it needs.

Listening. Users can't skim or scroll back, and they'll talk over the agent. Long replies are a bug, silence sounds broken, and interruptions are normal.

Structure: handoffs and tasks

The usual failure is one agent that does everything. It collects every tool, instruction, and piece of state until it's slow and unreliable, and by that point splitting it is a rewrite.

Handoffs transfer control from one agent to another. Put them at natural conversation boundaries, like greeting → intake → resolution, or general support → billing specialist. Each agent then carries only its own tools and instructions. Choose a boundary where the context can be summarized for the next agent. If the next agent needs everything the previous one had, the boundary is in the wrong place.

Tasks are tightly scoped prompts aimed at one outcome. Use them for discrete operations that don't need a full agent, or where a focused prompt works better than a general one.

If you can't say in one sentence what an agent is responsible for, split it.

Tools

• Tool descriptions drive behavior. When an agent calls the wrong tool or calls one at the wrong time, check the description before blaming the model. The most common cause is a description that doesn't say when to use the tool.
• Keep tools off the critical path where you can. Users hear every tool call they wait on as latency.
• Plan for tool failure. Decide what the agent says when a backend is down or returns nothing. An agent that makes up an answer when a tool fails is very hard to catch later.

The model interprets; your code owns the state

The model reads the conversation and proposes actions. Application code owns the records, the permission checks, the state transitions, and every external effect. Most agents that "work in the demo and fail in production" have that line blurred somewhere.

• Never classify intent with code. Approval, refusal, correction, cancellation, "next Tuesday" — the runtime model interprets those. A regex, a keyword list, or a phrase whitelist will be wrong in ways you never test, and adding one as a "conservative" second gate has the same defect. Validate structure in code (typed dates, enums, required fields); leave meaning to the model.
• A tool call is the model's interpretation, not proof it was right. Keep message provenance, version checks, ordering, and business rules in code, where they can be checked.
• Tools return facts, not sentences. Compact data, outcomes, and actionable errors; the model chooses the wording. Script exact text only when the task mandates a verbatim disclosure.
• Follow the user, not a form. Accept facts the caller volunteers together, ask only for what's missing or ambiguous, and never demand ritual wording ("say yes to confirm") after a clear answer.
• One authoritative state object per session, and keep model-supplied facts separate from trusted identity, the clock, ids, and receipts.