Writing Livekit Scenarios

Writing simulation scenarios

A scenario is a simulated user's instructions (who they are, what they want) plus agent_expectations (what counts as success). A simulation plays the scenario against the real agent, and an LLM judge grades the transcript.

Runs are disposable. The scenario file is what lasts: it gets reviewed in diffs, re-run for years, and it's what catches a prompt edit that breaks something without anyone noticing.

Before writing a file, look up the exact schema and CLI flags with reading-livekit-docs. Field names and commands change, so this skill doesn't restate them. Conceptually, a file is a named group of scenarios. Each scenario has the simulated user's instructions, the pass criteria, and optionally tags for grouping and per-scenario data the agent can read at runtime. If the CLI offers to add stable per-scenario ids to a file, accept and commit them.

Start from a generated baseline

Don't start from an empty file. LiveKit's generator reads the agent's source and produces a decent first pass much faster than you'd write ten scenarios by hand:

lk agent simulate text -n 10    # confirm the exact flags with --help

Generating from source uploads the code to LiveKit Cloud so the generator can read it, which is why the CLI asks for confirmation first. Tell the user that before running it and let them decline. Some code can't leave the machine, and in that case you write the scenarios by hand. A flag skips the prompt for non-interactive runs.

When the run finishes, the CLI either tells you where it saved the generated scenarios or offers to save them into the project. Either way, get that file into the repo as the starting point.

You can't steer the generator. It infers intent from the code, so it writes plausible conversations instead of the ones this agent's users have, and it doesn't know what the user is worried about. Treat its output as a draft:

1. Read every scenario and delete the ones that don't matter. A file you haven't read isn't a test suite.
2. Sharpen vague expectations. If the judge can't decide an agent_expectations, its verdict flips between runs. This is the biggest cause of flaky scenarios.
3. Add what generation misses. It drifts toward happy paths. references/risk-coverage.md explains how to turn the agent's constraints into a checklist and give each item a scenario.
4. Add what the user is worried about. If they haven't said, ask what they want stress-tested. That input is why a suite a person steers is better than a generated one.

Ask what to probe

The user knows which flow keeps breaking, which customer complained, and which change they're nervous about. Ask, and weight the suite toward that area with more and deeper scenarios.

Focus adds coverage. It never removes coverage of the agent's hard limits. If the user has no preference, generate broadly and tell them that's what you did.

Ground scenarios in what the agent can do

Read the agent's code locally with your normal tools before writing anything. Look for what a user can ask for (capabilities), where requests get blocked (constraints: required steps, unavailable items by name, caps, eligibility), and what the agent must refuse.

Constraints matter most. A scenario that asks for something the agent can't do is only valid if the expectation is that the agent says so. Written the other way, the test fails when the agent behaves correctly.

Scope is the agent reachable from the session entrypoint, plus agents it hands off to and tasks it awaits. Ignore other classes in the directory, unused imports, and example files.

Write instructions the simulated user can follow

There are two shapes, and which one you use depends on what the scenario is for. Details and examples are in references/scenario-craft.md.

• Point-form direction for deterministic flows you'll grade on end state. A structured template (persona, opening line, facts revealed only when asked, steps in order, conditional reactions) keeps the persona model on track and makes the run repeatable.
• A persona paragraph plus goals for open-ended and adversarial scenarios graded only on the conversation, where you want the simulated user to improvise.

In both shapes, goals are requests to the agent, never the agent's own actions. Use real values from the agent's domain and assume no prior state. Vary persona, mood and difficulty across the suite so it isn't ten copies of the same cooperative caller.

Split scenarios into sets, one file per set

The run command takes one scenario file, so a file is a run. Split along the lines you want to run separately, which usually isn't topic.

The most important axis is how the set is graded:

• Scenarios that drive concrete tool flows to a deterministic end state, graded on final state and the conversation (see "Make the agent consume the scenario" below).
• Open-ended and adversarial scenarios graded on agent_expectations alone.

These need different instructions shapes, different agent wiring, and often a different run cadence, so they go in different files. The second axis is cadence: a small set to run before a merge, and the full set before a release.

Give each file a name that describes the set, since it labels the run. Inside a file, use tags to slice. Tag feature so a failure points at the part of the agent that owns it, and add whatever else you filter by (channel, difficulty). Put a header comment on the file recording the set's assumptions: pinned dates, required environment variables, and what its expectations depend on.