Fix Nightly Failures

Fix nightly failures

Own the source repair loop: evidence → diagnosis → fix → PR → independent review → required checks → merge → final report. A standalone chat can start from an ongoing or finished suite. Do not require a continuously running coordinator. Carry each fix through targeted verification when the user requests suite repair and the required source/API is available. Approved merged source is a milestone; record deployment prerequisites while continuing other repairs. Framework deployment, a new full-catalog suite and Slack delivery are separate tasks unless authorized. A merged fix does not make the original suite pass.

Start from the suite

Accept a suite URL, suite ID or occurrence/workflow ID, plus any branch, repository, verification or merge constraints. Resolve these to the canonical occurrence and stable result URL; ask only for an identifier or access genuinely needed to read it. Discover provisioned tools, APIs and authentication using operations. Do not require this chat's history, a particular host, coordinator or source SHA.

Example launch:

$fix-nightly-failures <suite URL> — Diagnose failures while remaining members run, fix their owning repositories, open PRs, run codex-pr-review, address findings and merge after approval and required checks. Report conclusions and verification. Do not deploy or start another full suite.

Record the complete selected member list, status, frozen harness/definition revisions and exact app/firmware publication. GitHub accepting a dispatch, an active chat or successful teardown does not prove device execution or a pass. Keep unavailable observations unknown; do not substitute today's build or copy an old success forward.

Preserve independent execution

Diagnose failures as they arrive and let other suite members continue. Ordinary setup, test or teardown failures do not justify cancelling the suite. Let the controller clean the failed run and repair its lane through normal operations; verify the actual repair invocation, conclusion and accepted resume or out-of-service decision. Healthy lanes continue independently.

Separate the original test outcome from later machine repair, cleanup and publication errors. Repairing machine state does not repair source or change the original verdict. If a global framework fault prevents useful execution, diagnose it and use normal operations within the current cancellation authority. Never force-release hardware, edit controller SQLite, change catalog toggles or weaken assertions to manufacture green. Keep live source/configuration unchanged while accepted work depends on them; this skill does not perform deployment.

Diagnose and group on evidence

Keep one small durable ledger in existing task state: routine/run, failing phase/step, original error and evidence links, exact provenance, classification, causal group and supporting evidence, existing fix/job, PR/head/review, next action and verification result. Fetch the first useful framework result and setup/teardown journal before expanding into screenshots, recordings or command logs.

Classify app, routine, framework, machine/fixture state or unknown on evidence. Compare the failing source with current source before fixing it. Reuse the relevant diagnosis mechanics in fix-routine-failure; this suite's completion boundary takes precedence over that skill's deployment/rerun loop.

Group failures when evidence demonstrates the same cause in a shared provider or action at compatible source/build and triggering conditions. Matching wording alone is insufficient. A suspected shared cause can have one diagnosis owner, but keep it provisional and split when evidence differs. Preserve every member's original outcome; grouping never makes an unverified member pass.

Use one coherent fix for a demonstrated cause. Inspect existing PRs, merged fixes and compatible authoring jobs before creating more. Do not mechanically create one fix or job per failed routine. If the error lacks diagnostic information, fix the bounded, redacted observation gap first. Such a PR does not prove the original cause; do not spend hours repeating complete runs to rediscover it.

Implement and finish the PR lifecycle

Source-only investigation and fixes need no lane or activation receipt. Work in a clean isolated worktree from the current destination branch, retaining the failure's channel provenance: dev → dev, staging → staging, an open PR → its recorded repository/branch and base. Reuse existing relevant work. Keep routine behavior in routines/, shared mechanics in framework providers, and ownership or repair in orchestration. Follow repository guidance and proven actions; avoid new compatibility paths, runners or arbitrary sleeps.

Run meaningful source checks, open a focused PR with diagnosis and validation, and use select-pr-routines for applicable coverage. Label product app fixes bug:app, wrong routine actions/expectations/fixtures bug:routine, and shared execution/orchestration/recording/infrastructure bug:framework; use multiple labels only for multiple corrected components, never bug:harness. A routine's Super Mode transition expectation is routine behavior; shared Mac scan-row reveal is framework behavior. Add relevant routine coverage labels separately. Documentation-only changes need no device coverage. Use the canonical independent codex-pr-review skill and its existing launcher on every opened or updated PR. Read its verdict and all relevant requested changes; address real findings, explain evidence-backed disagreements, and rerun review after changing the head. Do not substitute self-review or a bare reviewer agent. See review and merge.

Merge only with an approving canonical review on the exact current head and passing required checks, under the task's merge authority. Reconcile uncertain push/PR/review/merge outcomes before retrying. Continue this lifecycle without asking for authorization already supplied by the task. Keep credentials, private logs and recordings out of public PRs/comments.