Auto QA

OpenClaw Auto QA

Run a continuous, current-main OpenClaw product campaign. Treat a reviewer finding as a hypothesis, a passing test as evidence only for its actual head, and a merge as complete only when the canonical repository confirms it. Always prefer a clean, appropriately scoped root-cause refactor over a quick fix or smaller diff. Repair the actual root cause in its canonical owner; a patch is not acceptable when it leaves sibling paths, lifecycle invariants, or the defective abstraction intact.

Set the campaign goal

• Default to 100 distinct, independently verified, actually merged root-cause fixes unless the operator explicitly specifies a different target. Count broken invariants, not pull requests, symptoms, proposed patches, skill updates, or merely passing tests.
• When the operator changes the target during an active campaign, update the existing report, numbered ledger, progress denominator, and live status in place. Preserve every previously verified landing; never reset the investigation or claim that an unfinished target has been reached.
• Exclude security-sensitive, persistent-state, protocol, public-contract, or otherwise review-required work from the accepted count until the appropriate owner explicitly authorizes and verifies its landing.

Prefer clean refactors over quick fixes

• Identify the broken ownership boundary, abstraction, state transition, or dependency contract before choosing an implementation. Compare the canonical owner, callers, callees, and sibling paths; prefer the design that makes their shared invariant obvious and reliable.
• Consolidate decisions and authoritative state in their actual owner. Propagate prepared facts through existing lifecycles, repair all affected siblings, and delete obsolete branches, duplicate policy, dead helpers, and stale abstractions when they are no longer needed.
• Reject symptom-masking guards, one-off exceptions, observed-example literals, parallel code paths, extra caches, fallback stacks, compatibility shims, and tests that merely make a narrow reproduction pass. A smaller change is not safer when it preserves the cause or makes the architecture harder to understand.
• Preserve shipped public contracts and ownership boundaries. If the clean refactor would affect security, persistent state, public configuration, plugin SDK compatibility, a protocol, or a product decision, mark it for maintainer review instead of substituting a tactical patch.
• During independent review, explicitly ask whether the change is the cleanest appropriately bounded root-cause solution. Green tests, a minimal diff, and a plausible local fix are insufficient without that architectural judgment.

Start with the moving source

1. Read all of root AGENTS.md, then each scoped guide for the lanes under inspection. Read the current skill for a specialized workflow instead of reproducing stale instructions:
   • $openclaw-repair-sweep for bug acceptance and duplicate handling.
   • $openclaw-testing for actual test and CI selection.
   • $openclaw-qa-testing for QA Lab, scenario catalogs, and real provider lanes.
   • $control-ui-e2e for browser and Control UI proof.
   • $crabbox for remote, Docker, packaged, cross-platform, and live proof.
   • $autoreview for fresh independent review before publishing or landing.
   • $openclaw-pr-maintainer for authorized maintainer-side GitHub actions.
2. Identify the actual canonical main checkout; a desktop or linked worktree can have a different detached HEAD. Check git -C <verified-canonical-main-checkout> status -sb and record git -C <verified-canonical-main-checkout> rev-parse HEAD. When network access is authorized, let only the orchestrator refresh origin/main, record its full SHA, and prove the canonical checkout matches before inspecting source. Give every read-only reviewer the absolute clean checkout or its own verified exact-head worktree. Before inspection and immediately before accepting results, require both the exact git -C <worker-checkout> rev-parse HEAD and empty output from git --no-optional-locks -C <worker-checkout> status --porcelain=v1 --untracked-files=all --ignore-submodules=none; alternatively, read immutable files directly from the frozen Git object. Discard the complete wave item when either verification fails. Keep intentionally modified implementation worktrees separate from read-only frozen-source review. For offline work, disclose that remote freshness is unverified. Never pull, rebase, or switch another agent's checkout.
3. Make one requested, fresh codex/ worktree per implementation task. Keep reviewer workers read-only. Serialize all shared origin/main refreshes through the orchestrator; pause worker fetches before repository-native PR review, preparation, or landing. After a successful landing, fetch origin/main again, verify the recorded merge commit is an ancestor of that fetched ref, and only then broadcast its full immutable SHA and resume workers. Revalidate candidates against each new head without changing a sibling's active worktree. When offline, disclose that remote freshness is unverified.
4. Initialize or resume the user-requested Markdown report and numbered ledger. Read references/campaign-evidence.md and references/evidence-ledger.md before accepting a finding. If the operator increases the target, update the active denominator and existing numbered ledger without erasing historical time-stamped progress.

Keep at least ten lanes active

When independent worker execution is authorized, keep at least ten materially different subsystem investigations in flight throughout the active campaign. Replace completed, failed, or stale workers promptly. Network authorization separately determines whether a lane may fetch, invoke an externally hosted model, or contact a provider; it does not prohibit authorized local subagents. If independent workers are unavailable or forbidden, record the concurrency requirement as blocked rather than claiming sequential reviews are concurrent. Start with the OpenClaw-specific lane map in references/subsystem-lanes.md. Split a large area into narrow, independent ownership surfaces instead of giving one worker the entire gateway, provider, UI, or app tree.

Use first-class subagents when available and bounded codex exec --sandbox read-only --ephemeral reviewers when agent slots are exhausted; verify the installed CLI's supported options with codex exec --help. Keep CLI workers under an actually surviving parent session or durable supervisor; background children started by a shell that immediately exits are not active workers. Independently check the exact child PIDs and command identity with ps -p <pid-list>, and record the observation time. Give each worker the absolute verified checkout, initial and final exact-SHA and clean-content guards, frozen SHA, one subsystem and its scoped guide, a bounded duration, and the required evidence shape. Record running, successfully completed, failed, timed-out, and stale-guarded workers separately; replace finished workers before claiming continued concurrency. Ask for source, at least one caller and callee, sibling behavior, regression tests, current-main reproduction, upstream dependency proof when relevant, severity, and duplicate references. Do not disclose a proposed diagnosis to an independent verifier. When independent workers are unavailable, disclose the actual limitation; never represent launched, planned, finished, shell-discarded, or stale workers as running.

Observe CPU, memory pressure, disk, open ports, actual worker count, and gateway health between waves. Scale to the machine and operator-authorized load rather than mechanically starting 64 workers or treating a brief load-average spike as failure. Reduce campaign concurrency for sustained memory pressure, gateway failures, process starvation, or an actual operator limit. Keep remote proofs serialized per Testbox lease; never reclaim, sync, or launch another command while that lease has an active command. When a local process session disappears, recover the authoritative remote job and exact exit before retrying or claiming a pass. Use bounded retries and timeouts. Stop only campaign-owned processes.

Prove actual product behavior

Read references/live-proof-routing.md before invoking a provider, private QA build, remote lease, packaged install, or native app.

• Use isolated campaign state and free localhost ports for dev gateways. Never stop, restart, reconfigure, migrate, or bind the operator's running gateway, app, device, state directory, or default gateway port.
• Classify source trust before execution. Do not execute untrusted contributor scripts, hooks, configuration, tests, or package installation on a local or credential-hydrated machine.
• Use the existing service-account or provider credential only through the approved secret-backed workflow. Never print, log, commit, echo, export into an untrusted worker, or include credentials in test artifacts.
• For an OpenAI claim, first verify the gateway's real configured agents.list, selected existing agent, canonical openai/<model>, and a real model response. Prove both the delivered model-final path and the separately persisted transcript or session; one passing route does not establish the other. An unavailable provider, unknown agent, mock, skipped test, fallback response, or an earlier head is not live proof.
• Inspect the actual rendered Control UI and screenshot contents. Reject a page or screenshot containing GatewayRequestError, UNKNOWN_AGENT, or an equivalent gateway error even when browser navigation, the HTTP status, or the image-capture command succeeds.
• Derive QA scenario IDs from current qa/scenarios/index.yaml and scenario YAML. Inspect the actual harness and generated summary. Count a scenario only when the requested run reports a nonzero total, zero failures, and the exact model, provider mode, and relevant behavior.
• Cap an individual live subagent-fanout scenario at 780 seconds after startup. Separately verify setup, timeout recovery, the original parent, child completion, and the final nonzero scenario result.
• For installed-package or plugin bugs, run the real current packaging and user install/update path. Unit tests alone do not prove a built distribution, npm package, Docker image, or Git plugin.
• Test iOS, Android, and macOS only where the current host, device, simulator, signing identity, emulator, or hosted proof actually supports the operation. Record unavailable prerequisites rather than manufacturing mobile or UI coverage.
• Bound stress and soak tests, record the immutable gateway source, owned PID, exact start time, successful/failed/skipped counts, and actual completion time. Preserve an already-running campaign-owned soak at its original source while auditing newer main snapshots; never restart it or describe a ten-hour soak as complete before ten actual elapsed hours.