Trailblaze Validate Oob

Validating Trailblaze's out-of-box experience

This skill encodes the methodology for honestly evaluating whether Trailblaze's user-facing CLI matches what a fresh agent / new user would intuitively expect. It's the framework's own UX regression test, run by an agent simulating cold-start use.

The goal is to catch counter-intuitive defaults, broken claims in the trailblaze skill, and friction that the team has gone CLI-blind on.

When to run a validation pass

• After any change to the trailblaze CLI surface (commands, flags, default behavior)
• After any change to the trailblaze Claude skill (the sibling trailblaze/SKILL.md) or its references
• Periodically (monthly?) even with no changes, to catch drift
• When a new rung of the adoption ladder gets added to the skill

What "good" looks like

After a validation pass, a fresh agent following only the trailblaze skill should be able to complete the scenario with zero surprises. Specifically:

• Every command the skill says to run actually exists with the documented flags and argument shape
• Every output the skill describes matches what the CLI actually prints
• The basic loop (drive → save → replay → inspect) works end-to-end with no undocumented gotchas
• Failure modes the skill mentions are real failure modes; the recovery steps actually recover
• Errors the user might hit have clear hint: lines pointing at the fix
• No "workaround" paragraphs in the skill that exist to paper over CLI behavior the user wouldn't naturally guess

If the validation surfaces something the agent couldn't have guessed from the skill alone, that's either a skill update OR a CLI fix — and the latter is usually preferable.

Methodology

Step 1: Install the binary fresh

Use the actual end-user install path, NOT ./trailblaze from the repo (which goes through Gradle on every call and isn't what users hit):

./scripts/install-trailblaze-source.sh

The script builds the uber JAR and installs it under ~/.trailblaze/install/ with a symlink onto the system PATH. The specific symlink location depends on your platform (homebrew on macOS, /usr/local/bin/ on Intel macs / Linux, …) — defer to the script's stdout for the exact path rather than memorizing one. Then verify:

which trailblaze         # confirms which binary the shell will pick up
trailblaze --version     # records the version under test (capture this in the report)

If the script fails (missing JDK, Gradle error, symlink-permission failure), that's itself a finding: file it as an "OOB blocker" and spawn a CLI fix chip before trying to run the validation — don't shim around it from inside the skill.

Step 2: Spawn a fresh-context subagent

The subagent must have no prior Trailblaze knowledge. Their only source of truth is the trailblaze skill file. They must not infer commands from the repo's source code or from any other docs.

The right tool for this is a general-purpose agent (read-only is sufficient since the validation only runs --help and discovery commands, never destructive actions).

If the subagent can't proceed at this step — no device connected, target app not installed, network or auth required, etc. — that's a gap too: an OOB experience that requires undocumented setup is itself a finding. Record what setup was needed and what error the subagent hit, file it the same way as an OOB blocker, and resume the validation once the setup gap is closed.

Step 3: Give the subagent a concrete scenario

A scenario must:

• Be a realistic user task ("drive an Android emulator", "save a flow as a trail", "replay a trail and inspect what happened")
• Be doable using ONLY commands the skill describes
• Have a clear "I would now act on a device" stopping point — the validation is read-only; don't actually drive devices or modify state

Scenarios that have worked well:

• Rung 1: "Drive a device. List devices, snapshot one, look at the toolbox to find available actions, identify how you'd act on an element. Stop before actually executing the action."
• Rung 2: "Save the current session as a .trail.yaml. Then replay one. Then generate a report. Then look up past results."
• Rung 3: (when written) "Compose your own agent surface — write a custom typed tool, install a trailmap, curate what tools the agent sees."

Step 4: Subagent prompt template

The validation prompt should always:

1. Tell the subagent it's a Claude Code session that just loaded the skill — no prior Trailblaze knowledge
2. Specify the installed trailblaze binary on PATH as the test surface (NOT ./trailblaze)
3. Give the concrete scenario
4. Instruct them to follow ONLY what the skill says; gaps are gaps
5. Require structured output: Inaccuracies, Gaps, Ambiguities, What works, Suggestions
6. Cite quoted skill text and actual CLI output side-by-side for each finding