Agentic QA

Agentic QA: first-time-user journeys

Assess whether someone can understand and use the product from what it actually shows them, rather than whether someone who knows the implementation can operate it.

You are the QA coordinator, not the novice participant. You may know NodeTool and inspect its repository. The participant must receive neither your knowledge nor this skill. Start each independent cold session as a new process of the participant runner (web/tests/agentic-qa/runParticipant.ts). Never use your own agent tool, a fork, or a browser MCP for a participant. In this repository those inherit the project instructions, memory, git status, and DOM-reading tools. Keep environment setup, blind exploration, and technical diagnosis separate.

This is an engineering QA skill, not a shipped in-product authoring skill. Do not load NodeTool's product skills into the participant.

1. Resolve the run contract

Use the user's target, scope, and permissions. Otherwise use these defaults:

• Inspect the public homepage as an anonymous visitor. Use an explicitly owned, disposable local/staging app for actions that create or modify anything.
• Run the core campaign in journeys.md: website understanding, app first use, and one outcome-led task, each independently cold. Include persistence and recovery inside the task session. Report omissions.
• Use desktop at 1440 × 900 CSS pixels, device scale 1, a fresh browser context, and ordinary browser proficiency but no assumed product knowledge.
• Limit a discovery session to 20 actions and a task session to 50 actions. Limit each to 10 minutes elapsed, including tool/model overhead. Record this overhead separately where possible. Permit at most two reasonable recovery attempts per local obstacle. These are experiment limits, not estimates of human patience.
• Default application/provider spend to zero. Agent-inference cost is separate. Public browsing is read-only. Never buy, publish, invite, send messages to real people, connect personal accounts, or delete existing user data.

Count deliberate browser interactions and explicit waits against the action limit. Screenshot capture and logging do not count. Record the target/build, locale, browser, viewport, persona, task, limits, identity/data state, provider mode, auth omissions, and permitted domains/actions before starting. Do not silently turn a public website CTA into a local-app URL. If no approved app is available, complete public exploration and mark app tasks not run. An unavailable dependency is not a UX finding by itself.

2. Prepare the environment privately

Read runtime.md. Start the disposable app with web/tests/agentic-qa/serveApp.ts. It reuses the journey suite's seeded backend, fake providers, and reset endpoint on separate ports. Reuse infrastructure, not selectors, scripted routes, or returning-user state.

In particular, do not import the journey test fixture unchanged: it calls seedReturningUser and seeds a selected chat model. Do not dismiss onboarding, accept a tour, configure a model, pre-open a document, or write preference storage for a cold session. A reset seeded workspace is not an empty new account: label it seeded-demo. Claim empty-new-account only after verifying that state.

Use separate browser contexts and separate disposable accounts/backends per concurrent participant. Otherwise serialize sessions. A shared reset must never interrupt another session. Capture private logs/trace if useful, but do not feed them to the participant. Never reset a live/production system.

3. Enforce the knowledge boundary

Before launching, audit the actual inputs and permissions, not just the role name.

The participant may receive only:

1. The standalone participant protocol.
2. One neutral task and a generic persona, an entry URL, action/time limits, permitted origins/actions, and any ordinary user-owned input assets.
3. Unannotated viewport screenshots and mechanical browser observations that a person could see, plus neutral operation receipts.

It must not receive parent history, repository files, project instructions, product skills, previous findings, other participants' reports, route names, fixture IDs, feature lists, expected navigation, selectors, source, DOM, an accessibility tree, console output, network responses, or hidden application state. Do not pass this skill, the journey catalogue, or the report's diagnostic rubric.

A new context is necessary but not sufficient. Check auto-loaded instructions, memory, MCP/tool descriptions, hooks, startup git information, and resource access. A worktree isolates edits, not product knowledge. Never use a conversation fork or resume an earlier participant as a new first-time visitor.

Use one of these execution modes:

• Direct (default): run the participant runner. It gives a clean Claude session only the protocol, the packet, and a screenshot/coordinate browser toolset whose responses pass the visual-only contract in runtime.md. It audits the session's actual tool list before the first action.
• Brokered (fallback): a clean participant chooses one coordinate/keyboard action. The coordinator executes it mechanically and returns only the next screenshot and a neutral receipt. Do not correct its target, choose a locator, explain the UI, or suggest another action. Resume the same participant within this session.

If neither mode can preserve the boundary, stop with BLOCKED_ENVIRONMENT and validity: unverified. Do not substitute a knowledgeable solo review and call it first-time-user QA. The skill defines behavior. Host permissions enforce access.

The runner archives the exact launch packet, tool allowlist, configuration evidence, and a hash of the protocol in contract.json and summary.json. If forbidden information reaches the participant, freeze that run as CONTAMINATED. A later clean run is a new record, not a replacement. Do not ask an agent to prove it has forgotten information. Pretraining knowledge cannot be erased: require screenshot provenance for product-specific claims.

4. Run discovery before product-specific tasks

Start the website-understanding participant with no product explanation and no feature-specific goal. Its first observation is the initial viewport, not a full-page capture, extracted copy, a sitemap, or a guided tour of the website.

Have it record a brief first impression before interaction: what the site appears to offer, who it appears to serve, and the most plausible next action. Uncertainty is an acceptable answer. It then chooses which visible pages/links to explore. Do not tell it to find named product areas or visit a preselected route list.

After discovery, preserve its evolving understanding in its own words, together with the screenshots that supplied the information. Compare marketing promises with encountered app behavior only in the coordinator's later report.

For other sessions, pass only a task's ordinary-language outcome, never its internal mapping or expected solution. Product terms become usable only after they are encountered visibly, or when the user's task genuinely supplied them.