Babysit CI

Babysit CI: monitor and fix tests.yaml + lint.yaml + autofix.yaml + docs.yaml + release.yaml binaries

Monitor the tests.yaml, lint.yaml, autofix.yaml, docs.yaml, and release.yaml (Nuitka binary matrix, on projects that enable it) workflows in a fix-verify loop until all stable matrix variations pass and type-checking is clean.

Invocation

This skill involves repeated gh, git, uv run pytest, git commit, and git push calls. Run with --dangerously-skip-permissions to avoid manual approval on each step. Sonnet is recommended: the task is mechanical (fetch logs, match patterns, edit code, commit) and doesn't need deep reasoning:

$ claude --dangerously-skip-permissions --model sonnet /babysit-ci

[!WARNING] --dangerously-skip-permissions bypasses every permission prompt for the whole session: only use it in an environment you trust, ideally a sandbox or disposable checkout, never against an unfamiliar repository or untrusted input.

Because this loop runs autonomously without human review, every commit carries a Co-Authored-By: Claude <noreply@anthropic.com> trailer by default so unattended changes stay traceable, including where a project CLAUDE.md or a global ~/.claude/CLAUDE.md says nothing about commit attribution. The default yields to one thing: an explicit standing rule from the repository's maintainer against AI attribution, which outranks it because the trailer lands in their permanent history. A parent skill (like /repomatic-ship) spawning this loop does not by itself relax the requirement, but an exemption that skill passes down does, and it binds every commit made from that point on.

Keep the message itself short: an imperative subject naming the fix, under 72 characters, and no body at all unless the commit meets one of three cases. It bundles orthogonal work, a public record holds the context, or it resolves a tracked item (Closes #N). A CI fix rarely does: Fix Windows path assertion in `test_cache_paths` is a complete commit message. The case that recurs here is the link: when the red traces to an upstream bug, a dependency release or a linked discussion, put that URL in the body, since it is the only place the next reader will find why the fix looks the way it does. That body is the URL plus at most one clause naming what it holds, and it is capped hard at two lines and 25 words. Never summarize the linked page, and never record what you measured while diagnosing: a failing assertion, a version a probe reported, the flag that turned out to be the cause. Those belong in a code comment beside the fix.

Yield to the orchestrator that spawned you

When /repomatic-ship or another orchestrator spawns this loop as a sub-agent, it may reach in to claim a specific fix, usually one touching a deliberately-kept structure that needs its own judgment. Honor that at once: a message telling you to hold, stop, or stand down on a fix (or on the whole loop) means stop editing the working tree immediately, reply to acknowledge, and neither commit nor push that fix. Mailbox messages are delivered between tool calls, so read yours before every edit and before every commit: a HOLD that landed while you were mid-edit still binds the moment you see it, and you must not race the orchestrator by finishing the edit first. Unless told to stand down entirely, keep polling and reporting the jobs it did not claim, and let it tell you which HEAD to resume on once its fix lands.

Timeline

Three feedback channels run in parallel after every push, each at a different latency. Fix as soon as the fastest channel reports a failure: do not wait for slower channels.

 time   LOCAL (free)              REMOTE (CI minutes)
 ────   ────────────              ───────────────────
 0:00   push
        ├─ pytest ─┐              ├─ lint.yaml ─────────────────────┐
        ├─ mypy ───┤              │   (mypy on all files, YAML,     │
        └─ ruff ───┘              │    secrets, zizmor)             │
                                  │                                 │
                                  └─ tests.yaml ─────────────────┐  │
                                      (12 stable + 6 unstable)   │  │
                                                                 │  │
 0:30   GATE 1: local done                                       │  │
        fail? ─── yes ──► step 5 (fix now, skip CI)              │  │
                   no ──► poll CI                                │  │
                                                                 │  │
 3:30                     GATE 2: lint.yaml done ◄───────────────│──┘
                          mypy fail? ─── yes ──► step 4-5        │
                                          no ──► continue        │
                                                                 │
 5:00                     GATE 3: tests.yaml fast jobs done ◄────┘
                          stable fail? ── yes ──► step 4-5
                                           no ──► early exit if only macOS left (human runs; orchestrator waits)

 8:00                     tests.yaml macOS done (often skippable)

                          all green? ──► DONE

After fixing (step 5-7), the loop restarts from the top: push, run all three channels again.

Loop

1. Detect the repo and branch from the current working directory:

   $ gh repo view --json nameWithOwner --jq '.nameWithOwner'
   $ git branch --show-current

   Use the detected branch for all --branch= flags below.

2. Get the latest runs for the current branch:

   $ gh api 'repos/<OWNER>/<REPO>/actions/runs?head_sha=<FULL_SHA>&per_page=20'

   One call returns every workflow triggered by that commit. Do not reach for gh run list: both its branch-wide form and its per-workflow form have put runs weeks old at the top of the list, on a repository whose workflows had in fact run on every recent commit, which reads as "this workflow has not run in a month". Give head_sha the full 40-character SHA, because an abbreviated one matches nothing and returns an empty list that looks identical to "nothing ran". Quote the whole path: zsh expands the ? as a glob and kills the call with no matches found before gh runs. To list a single workflow across a branch instead, use gh api 'repos/<OWNER>/<REPO>/actions/workflows/<file>/runs?branch=<BRANCH>&created=>=<DATE>', and keep the date filter: without it, that endpoint has returned a page twelve days stale while the branch ran every hour.

   Track all five run IDs (docs.yaml may have none: its paths: filter skips pushes touching nothing docs-relevant). An empty run list for the other workflows is not paths-filtering: GitHub can sit on a push event for hours before materializing any run (a 4-hour lag has been observed), so when a freshly pushed SHA shows no runs, keep re-polling instead of concluding the push was filtered, and measure the wait from run creation, not from the push. The tests.yaml run exercises the full test matrix; lint.yaml runs mypy on every tracked Python file and lints YAML; autofix.yaml runs the mechanical fix jobs (format-*, sync-*, fix-typos, fix-vulnerable-deps) and turns red when one crashes instead of committing a fix; docs.yaml builds and deploys the Sphinx site and runs the broken-links check (an externally cancelled or link-flaky run re-runs cleanly via gh workflow run docs.yaml --ref <BRANCH>, no commit needed); release.yaml runs the Nuitka binary matrix (dev binaries: an ordinary push rebuilds only the [tool.repomatic] nuitka.dev-targets canary subset, while release commits, the weekly schedule and workflow_dispatch build the full target fleet) — but only on projects with [tool.repomatic] nuitka.enabled and a CLI entry point; with Nuitka disabled the per-platform jobs skip on every push, release commits included, and a green release.yaml means package build plus dev pre-release sync only. Even with Nuitka enabled, a single push can still skip the matrix: Metadata.skip_binary_build (repomatic/metadata/git.py) is the authoritative signal, true when the head commit is on a non-code branch, is a user-initiated version-bump commit, or (the common case) its changed files fall entirely outside Metadata.binary_affecting_paths — a docs-only or workflow-only commit, say. Don't infer the reason from the commit message or assume an unrendered/skipped matrix job name means nuitka_matrix itself was empty: the matrix data can be fully populated for that exact commit (verify via the release / 🧬 Project metadata job's logged metadata= JSON output) while the workflow still elects not to build it. Check skip_binary_build's three conditions before asserting why a matrix skipped, since any one of them is enough on its own. All five workflows must pass — see § Autofix job failures and § Nuitka binary build failures in references/failure-patterns.md for how to triage them without stalling the loop.

3. Run local tests while waiting for CI. Don't idle while polling. Start the full test suite and linters locally in the background immediately:

   $ uv run pytest --no-header -q &
   $ uv run --group typing repomatic run mypy &
   $ uv run repomatic run ruff -- check repomatic tests docs &

   Give run mypy no arguments: the tool runner resolves the same file list CI's lint.yaml runs it on, so the two cannot diverge. Naming directories instead is what used to make mypy pass locally and fail in CI.

   Gate 1 (local, ~30s): if any local check fails, you already have the diagnosis: skip straight to step 5 without waiting for CI.

   If local passes, poll CI every 60 seconds with:

   $ uv run repomatic ci-status --branch=<BRANCH> --no-fatal

   It reads every workflow a push can start (derived from .github/workflows/, so its list is wider than the five above), reports each one's latest run, and names the failing jobs that actually gate a merge. Three traps it settles, so no hand-rolled jq has to: a run's own status lags its jobs (every monitored workflow can read queued while a dozen jobs have already finished, which is indistinguishable from the runner-cap saturation a busy account genuinely hits); a continue-on-error probe that crashed hides inside a success run conclusion; and a run whose conclusion is failure with no failed job is a workflow-level error (an invalid strategy.matrix expression, malformed YAML, a missing secret) with no job log to read, which the command flags rather than letting you write off a persistently-red workflow as a known artifact.

   ci-status reads the same endpoints as the gh api calls above, and it has printed a stale table the same way: re-read a surprising red or green once before acting on it.

   Run state does gate the run-scoped log read (step 4): gh run view --log-failed refuses to answer until the parent run reaches a terminal state. A completed job's own log is readable at once.

   A queue is not a hang, and elapsed time alone cannot tell them apart. A long stall invites the theory that some run is stuck and that cancelling it would free the pool, which is a conclusion worth reaching only on evidence, because acting on it destroys work that was progressing fine. Two readings settle it, and neither is the elapsed clock. Get a baseline from that workflow's recent successful runs (gh run list --workflow <wf> --repo <owner/repo> --status success --json createdAt,updatedAt): a suite whose normal duration is two hours is not hung at ninety minutes. Then read per-job completedAt timestamps (gh run view <id> --json jobs) rather than an aggregate: a flat count of in_progress jobs is not evidence of a stall, since jobs finish and others start into the freed slots, holding the count steady while real progress continues. Watch for the self-contradiction that exposes the mistake, a report that the count "dropped from 11 to 9" and that nothing has changed. Two smaller traps live here too: gh reports a pending job's conclusion as "", not null, so a select(.conclusion == null) filter silently matches nothing and reads as "no pending jobs"; and jobs queue against the runner pool their runs-on names, so a stall confined to macOS cells says nothing about Linux capacity. Never cancel another repository's run to free slots without the maintainer's explicit go-ahead, and never ask for that go-ahead on a diagnosis you have not backed with a baseline and timestamps.

   Gate 2 (lint.yaml, ~4 min): lint.yaml finishes before tests.yaml. If "Lint types" (mypy) fails, proceed to step 4 immediately.

   Gate 3 (tests.yaml, ~5-8 min): once the first stable job fails, or all fast platforms (Linux, Windows) pass, proceed.

   Poll in-process; never detach a monitor. Block on gh run watch <RUN_ID> or loop the polls within your own turn. A detached background monitor (a standalone process, a run_in_background: true Bash poller that re-invokes you when it exits, or a Monitor-tool stream that returns control on each tick) makes a parent-resumed run spawn another monitor per tick instead of driving to a terminal state; worse, a spawned sub-agent that detaches this way orphans the poll from its caller the moment it returns. Hold the turn until the run completes: starting a poller and handing back "to be notified" is the early return this loop must never make.

   Every wait between polls must be a sleep, never a busy-wait. A poll loop with no delay (until gh run view ...; do true; done) fires thousands of requests per minute and exhausts the REST quota (5,000/hour) within minutes. The harness blocking a bare foreground sleep is not a reason to drop the delay: put the sleep 60 inside the loop command itself, which runs fine in both foreground and background. Exhaustion does not just blind your own polling — workflows authenticating with the same PAT start failing server-side with misleading errors (see § GitHub API rate-limit exhaustion).

4. On any CI failure, cancel the branch's remaining runs to free runners:

   $ uv run repomatic cancel-runs --branch=<BRANCH>

   The command spares any run whose head commit carries [changelog] Release, mirroring the cancel-in-progress condition in every workflow's concurrency group. Cancelling a release run costs that version its binaries permanently, so never hand-roll the sweep with gh run cancel.

   Then download logs from all failed jobs across the workflows (logs are retained after cancellation):

   # Failed stable test jobs:
   $ gh run view <TESTS_RUN_ID> --json jobs --jq '[.jobs[] | select(.conclusion == "failure" and (.name | contains("⁉️") | not))] | .[].databaseId'
   
   # Failed lint jobs (especially "🛡️ Lint types" for mypy):
   $ gh run view <LINT_RUN_ID> --json jobs --jq '[.jobs[] | select(.conclusion == "failure")] | .[].databaseId'

   The first filter negates the unstable glyph rather than matching a ✅ prefix, mirroring JobStatus.required. tests.yaml runs four required jobs whose names carry no ✅ at all (🧬 Project metadata, 1️⃣ Run-once tests, 📦 Package install, 🖥️ Validate …), and the release engine prefixes the workflow ahead of the glyph, so a prefix test silently drops a real failure from the batch.

   Fetch each failed job's log (gh api repos/<OWNER>/<REPO>/actions/jobs/<JOB_ID>/logs --allow-escape-sequences) and fix them as one batch: different sources surface different issues, and logs survive cancellation. Batch only what has already failed, never what might still fail. Once every harvested failure is root-caused and fixed, push immediately rather than waiting for undrained cells to surface more: the fresh run supersedes the stale one, and serially waiting out each full matrix is the slow path. Analyze following the error triage discipline: stable-job FAILED/AssertionError lines only. Then match each failure against references/failure-patterns.md before fixing anything.

   gh run view --log-failed writes its log cache under ~/.cache/gh. A sandbox that denies that path answers failed to get run log: creating cache entry ... operation not permitted, which masquerades as a gh bug. Run the read in the sandbox first, and disable the sandbox for it only when that error appears.

   A completed job's log is readable while the rest of the run drains, but only with --allow-escape-sequences. The run-scoped reads are gated on the whole run going terminal (gh run view --log-failed answers run <id> is still in progress; logs will be available when it is complete), while the job-scoped gh api repos/<OWNER>/<REPO>/actions/jobs/<JOB_ID>/logs answers a failed cell immediately, twenty minutes into its slowest sibling's build. What makes it look otherwise is a gh guard rather than the API: CI logs carry ANSI colour, so gh refuses to emit them and prints the response contains terminal escape sequences; pass --allow-escape-sequences to output it anyway — one line where a log was expected, indistinguishable from an empty body if the output went to a file. Pass the flag and strip the codes:

   $ gh api repos/<OWNER>/<REPO>/actions/jobs/<JOB_ID>/logs --allow-escape-sequences \
       | sed 's/\x1b\[[0-9;]*m//g' > job.log

   So the diagnosis is minutes away, not an hour: harvest every failed cell as it lands and keep the run alive for the cells still to report. Cancelling to read logs is never the reason — logs survive cancellation, but they never needed it. Make the run terminal only when you have a fix, per the batch-and-push rule above: the stale run has no verification value left once superseded.

5. Fix the root cause using the combined picture from CI logs and local results. Fix the codebase, not the tests, unless the tests are genuinely wrong. Address mypy and ruff failures together (see § mypy/ruff fix oscillation).

   If the root cause is in a third-party dependency, check whether a change this cycle exposed it before treating it as upstream: git log <last-release-tag>..HEAD for a runner/image swap, a dependency bump, or a config change that put the dependency in a context it cannot satisfy (a Rust-built package forced to compile from an sdist on an architecture with no published wheel, say). When a cycle change is the trigger, revert or adjust that change; only a failure independent of everything the cycle touched warrants /file-bug-report for an upstream report.

   After applying fixes, re-run the full local validation:

   $ uv run pytest --no-header -q
   $ uv run --group typing repomatic run mypy
   $ uv run repomatic run ruff -- check repomatic tests docs
   $ uv run repomatic run ruff -- format repomatic tests docs

   Hard gate: all four must come back clean before step 6. If a fix introduces new failures not in the original set, the fix is wrong: revert it and try a different approach rather than layering another fix on top.

   Both ruff commands write in place: read git diff after they run and fold any reformat into the fix. Skipping the format pass does not fail CI — instead the format-python autofix job pushes the reformat as its own commit, a new HEAD that cancels every in-flight run through the shared concurrency group and restarts the whole CI cycle (a wasted Tests + Nuitka round). A parent /repomatic-ship run's format gate only covered the pre-fix tree: this loop's commits are exactly the ones that would skip it.

6. Check autofix status before pushing:

   $ gh api 'repos/<OWNER>/<REPO>/actions/workflows/autofix.yaml/runs?branch=<BRANCH>&created=>=<DATE>&per_page=1'
   $ gh pr list --state=open --json number,title,headRefName,url

   If any open autofix PR already contains your fix — a format-python branch (ruff's own autofixes), a sync-repomatic/sync-workflow-pins branch (a bumped workflow pin, or a spliced-in --exclude-newer-package repomatic=P0D cooldown exemption carrying no version bump at all — that one is the whole fix when the metadata job cannot resolve its own pin), or another sync-*/fix-* branch — prefer merging it over authoring your own commit: GitHub signs the merge commit server-side, so this sidesteps a local hardware-key signing prompt entirely. If it resolves the failure, merge it (gh pr merge <n> --squash --delete-branch), pull, and rebase your fix before pushing — or skip your own commit if the merge is the whole fix. If gh pr merge is denied outright (a standing permissions.deny on the verb, not a retryable prompt), see § PR-merge permission wall.

7. Commit the fix with a clear message describing what changed and why, then git push.

   When the fix corrects a user-facing bug, add a changelog.md entry only when the bug reached a released version. Blame the changed line against the last release tag (git blame, or git log -S): a bug introduced and fixed within the current unreleased cycle never shipped, so it gets no entry; a bug that predates the last tag is a real regression and does. Making this call here keeps a parent /repomatic-ship run from having to add or drop entries afterward.

   Time each push by what its diff rebuilds. A source-affecting fix (repomatic/**, tests/**, pyproject.toml, uv.lock: whatever the repo's test and binary paths: filters name) pushes the moment it clears step 5: the runs it supersedes were verifying an obsolete tree, and its own run rebuilds everything it cancels. A commit those filters skip (changelog-only, docs-only, cosmetic prose) is the opposite case on a binaries-enabled project: release.yaml runs on every push in a per-branch cancel-in-progress group, so pushed mid-drain such a commit cancels the in-flight binary matrix while its own run skips the rebuild (Metadata.skip_binary_build), and the lost verification costs a full re-dispatch. That cost scales with what is actually in flight: an ordinary push builds only the [tool.repomatic] nuitka.dev-targets canary subset, and no push cancels a full fleet at all, since release commits, schedule and workflow_dispatch runs each sit in their own concurrency group. Hold it until the heavy matrices on the current HEAD are terminal, or bundle it into the next source-affecting push; with binaries disabled, only a canary build in flight, or nothing heavy in flight, push freely.

   If commit signing fails, do not loop on it. The sandbox can block the SSH key or socket under ~/.ssh/* (Operation not permitted): fix with dangerouslyDisableSandbox: true for the git commit and git push calls only. A hardware-backed key (Secretive, YubiKey, TPM) then prompts the maintainer per signature, and a refused or missed prompt surfaces as agent refused operation?, indistinguishable from a real failure. Retry once at most after disabling the sandbox; if it still refuses, hand off cleanly: stage the specific files you fixed (never git add -A), return the exact commit message and git push command verbatim, and exit the loop. The fix is done — only the signature is missing. If the block is instead a structural permission deny on gh pr merge (not a signing refusal), the escalation differs — a maintainer's in-chat approval cannot clear a deny rule: see § PR-merge permission wall.

8. Repeat from step 2 until the monitored workflows are green: tests.yaml with all stable (✅) jobs passing, lint.yaml with no mypy failures (test and docs files included). Stop after 5 iterations without progress (the set of distinct failing stable jobs did not shrink): report what was fixed and what remains, and ask for guidance rather than churning. Productive iterations never trip the cap: a release paying down a long test-debt tail legitimately takes more than five pushes.