Run Smoke Tests

Run Smoke Tests

Execute a broad user journey against a real build and leave enough evidence for a person to audit the judgment. Understand the host project before starting the product or touching its state.

Operating contract

• Treat the Markdown as runtime instructions, not source to compile into code.
• Test through a supported user-facing surface: web UI, API, CLI, desktop, mobile, or a documented combination.
• Separate execution from repair. Do not edit the product, tests, or journey during a run. Finish the report before starting any requested fix.
• Default to local, test, preview, or staging environments. Never mutate production, spend money, send real invitations, or change real user data without explicit authorization.
• Make every pass or fail claim traceable to captured evidence.
• Keep the first failed run. Do not erase evidence by retrying until green.

Workflow

1. Pass the repository-understanding gate

Read the nearest repository instructions, product README, runbook, contribution guide, manifests, launch scripts, existing smoke or end-to-end tests, and the smallest relevant product documentation. Search for auth setup, seed data, test accounts, feature flags, observability, artifact conventions, and cleanup commands. Trace the journey across the relevant UI, service, state, and external-system boundaries so failures can be attributed to the right part of the product.

Before launching anything, be able to state:

product and user surface:
authoritative journey source:
relevant architecture and system boundaries:
build and start commands:
target environment and data boundary:
interaction driver or available tools:
authentication and human checkpoints:
evidence capture capabilities:
safe cleanup path:
known abort conditions:

Do not guess missing commands, URLs, credentials, or permissions. If a missing item blocks safe execution, report it as a preflight blocker.

2. Validate the journey contract

Read references/smoke-format.md. Normalize supplied prose without silently adding product behavior.

Confirm that the journey:

• represents one coherent user goal;
• names its environment and allowed mutations;
• defines critical and noncritical oracles;
• identifies authentication, payment, security, or destructive checkpoints;
• has a cleanup or quarantine plan;
• names conditions that require an immediate abort.

Move small deterministic behaviors into scenario tests. Keep smoke coverage for the larger path and the seams between systems.

3. Preflight without changing product behavior

Build or launch the product with documented commands. Confirm health, test-data availability, driver connectivity, recording support, and enough storage for artifacts.

Prefer the repository's existing artifact directory. Otherwise use .context/smoke-runs/ when .context already exists; if it does not, create a temporary directory and report its absolute path. Do not add generated evidence to version control unless the user asks.

Create a run manifest containing:

journey ID and source path
commit or build identifier
environment and base URL or executable
start time and executor
human checkpoints
recording and log sources

Abort before step one when the build is unhealthy, the environment is wrong, the account or data boundary is unsafe, or required evidence capture is unavailable for a critical oracle.

4. Interpret and execute the Markdown

Follow the journey in order through the real product surface. For each step, record:

timestamp
instruction
action taken
observable result
evidence path or log reference
provisional judgment
deviation, latency, or uncertainty

Use normal user affordances. Do not call internal APIs to bypass UI steps unless the journey explicitly tests an API or uses setup APIs only for isolated fixtures.

Pause at a declared human checkpoint. State the exact action needed and resume from the same state after the human completes it. Never request that a user paste secrets into chat.

If the driver supports video, record the complete run and generate subtitles or a timestamped transcript from the step timeline. Otherwise capture screenshots, terminal output, structured logs, and state snapshots. Do not claim to have a video when the environment cannot produce one.