CI Watch

Watching CI with the GitHub CLI

The pitfall: raw watchers exit early

gh run watch <id> --exit-status and gh pr checks <pr> --watch are not safe to arm right after a push / tag / PR-open:

• Premature exit while QUEUED. With no jobs registered yet, gh sees "nothing in progress" and returns exit 0 even though the run is still queued.
• Transient errors read as failure. A mid-watch HTTP 401: Bad credentials (token refresh) or a 5xx exits non-zero, indistinguishable from a real CI failure.
• No native fix — --interval only tunes the poll cadence.

Never trust a raw watcher's exit code alone. Re-verify terminal status with gh run view <id> --json status,conclusion (done only when status == "completed") or gh pr view <pr> --json statusCheckRollup (done only when every item is terminal). And always fail-fast — act on the first failing check, never wait for all checks to finish.

The blessed tool: scripts/ci-watch.ts

Blocks until the target is truly terminal, then exits with a trustworthy status. Dogfoods nub; runs under plain Node too.

nub  scripts/ci-watch.ts --run <run-id> [--repo o/r] [--timeout <min>]
node scripts/ci-watch.ts --pr  <number> [--repo o/r] [--timeout <min>]

• --run <run-id> — watch a workflow run (polls gh run view --json status,conclusion,jobs).
• --pr <number> — watch a PR's check rollup (polls gh pr view --json statusCheckRollup,…).
• --repo <owner/repo> — defaults to the current repo.
• --timeout <minutes> — wall-clock cap before giving up as pending (default 45).
• --required <names> — comma-separated branch-protection check names to gate on (e.g. --required "CI gate"). Success fires the instant every required check is green; a ghost or a non-required check — pending or failed — never blocks, matching branch-protection semantics. Prefer it when you know the required check name.
• --no-progress <minutes> — how long an unchanged incomplete set (all named checks green, only a ghost left) may sit before exiting 4 STUCK-but-safe (default 8).

What it fixes: waits for the target to EXIST (not-found / no-jobs-yet is "keep polling," never "done"); polls authoritative terminal state; fails fast on the first FAILURE/CANCELLED/TIMED_OUT/ STARTUP_FAILURE; never hangs on a ghost; tolerates transient gh/API errors (retried with exponential jittered backoff, 10s → cap 60s, 90s unauthenticated).

The ghost check — why a strict "all checks terminal" gate hangs

GitHub occasionally registers a check-run that never reports a status: PENDING, nameless, forever. A watcher waiting for every rollup item to be terminal blocks indefinitely even though every real check is green. So a nameless / never-terminating non-required check does not block a green verdict: once every named check is green and the incomplete set has been unchanged for --no-progress minutes, the watcher exits 4 (STUCK-but-safe) and the caller --admin merges. A named pending check is never treated as a ghost, so a real in-flight check is never green-lit early.

Exit-code contract

The final stdout line is a single self-describing summary, e.g. CI-WATCH run 27972328590: SUCCESS (25 job(s) green), CI-WATCH pr 73: FAILURE — check "Test (ubuntu-latest, node 22.13)" → FAILURE (https://…), or CI-WATCH pr 327: STUCK — required/named checks GREEN (51), 1 non-terminal ghost/non-required check(s): (unnamed); safe to --admin merge.

NEVER pipe the watcher — a pipe discards the verdict you gate on

The exit code IS the contract, and a pipeline throws it away: in ci-watch.ts … | tail -20, $? is tail's status, so a FAILED CI reads as SUCCESS. The pipe also buffers stdout until the pipeline ends, so the log stays empty while the watcher runs.

# WRONG — $? is tail's, a red CI looks green, and no interim output
node scripts/ci-watch.ts --pr 604 --repo nubjs/nub | tail -25; echo "exit=$?"

# RIGHT — redirect, then gate on the watcher's OWN status
node scripts/ci-watch.ts --pr 604 --repo nubjs/nub > /tmp/ci604.log 2>&1; rc=$?
tail -15 /tmp/ci604.log; echo "exit=$rc"

main() warns on stderr when it detects a piped stdout (fd 1 is a FIFO for | cmd, not for > file), but redirect by default rather than relying on the warning. Bash ${PIPESTATUS[0]} / zsh ${pipestatus[1]} recover the real status if a pipe is unavoidable.

Cross-check the rollup regardless of what the watcher says. Read gh pr view <pr> --json statusCheckRollup,mergeStateStatus directly and act on any FAILURE. A watcher that has produced nothing for a long stretch is a suspect, not a status.

For a merge-queue drain, prefer scripts/merge-cascade.ts (it gates positively and merges on green); reach for ci-watch.ts when you need to block on one run/PR and branch on the result.