--- name: cb-automation description: The maintenance automation around Circuit Breaker — which bots and scheduled workflows exist (Discord notifications, ledger watch, branch cleanup, Dependabot lockfile sync, the required-checks dispatcher, the post-release follow-up), the GITHUB_TOKEN rules they are built around, how to add a new one safely, and the guardrails for AI agents (Copilot cloud agent, Agentic Workflows, Copilot CLI on the headless box). Use this whenever adding or changing a scheduled or bot workflow, anything under scripts/ci/, a Discord notification, a workflow that pushes commits or opens PRs, a self-hosted runner, or an AI/agent workflow, and when asked why a bot did or did not act. --- # Circuit Breaker — Automation One maintainer, no budget: automation exists to remove toil, and must never become toil itself (a noisy alert, a red check nobody owns, a bot that needs babysitting). Everything that needs a human reaches them through **Discord**. ## What runs on its own | Workflow | When | Does | Talks to Discord | |---|---|---|---| | `notify.yml` | Every watched workflow completes | `scripts/ci/workflow_alert.py` decides; posts failures (with ping) and recoveries | Yes — it is the pager | | `release.yml` | `make release-candidate` | See **cb-release** | "draft staged, waiting for you" (ping), "vX is published" | | `release-followup.yml` | Dispatched by release post-publish | Next-patch PR into `dev`, stale draft cleanup | Via notify.yml on failure | | `ledger-watch.yml` | Nightly 06:23 UTC | `scripts/ci/ledger_watch.py`: one `release-control` issue listing ledger rows expiring within 30 days and risks past `next_review`; closes it when clear | When a new issue opens (ping) | | `branch-cleanup.yml` | Sundays 05:00 UTC; manual dispatch defaults to dry run | `scripts/ci/branch_cleanup.py`: deletes branches fully contained in main/dev, idle > 14 days, not the head or base of an open PR | Via notify.yml on failure | | `dependabot-lockfile-sync.yml` | Dependabot pip PR into dev/main | Regenerates `requirements.txt` from `poetry.lock` with the **base** branch's generator, pushes, then dispatches required checks | Via notify.yml on failure | | `dependabot-automerge.yml` | Dependabot PR opened/updated | Queues `gh pr merge --auto` for patch/minor updates into `dev` (the ruleset's 21 checks still decide); labels majors `major-update` and comments; never touches PRs into `main` | — | | `security.yml`, `codeql.yml` | Weekly + push/PR + dispatch | Scanners | Via notify.yml on failure | | `e2e.yml` | RC tag, agent-path PR, nightly | Calls `composed-e2e.yml`, quarantined (`quarantined: true`, QUAR-001, issue #162) — reports the register row instead of running | Via notify.yml on failure | Squash-merged branches are never cleaned up (their commits are not contained in main), by design of the containment rule. ## GITHUB_TOKEN rules everything here is built around 1. **Events caused by GITHUB_TOKEN start no workflows** — no `push`, `pull_request`, `release`, or `push: tags` run follows a bot's commit, PR, release or tag. The one exception is `workflow_dispatch` (and `repository_dispatch`). So: - a bot that pushes a commit or opens a PR must then run `bash scripts/ci/dispatch_required_checks.sh [dev|main]`, which dispatches `dev-ci.yml`/`ci.yml`, `security.yml` and `codeql.yml` so all 21 required checks land on the head SHA. Without it the PR can never merge. The list of 21 lives only in `tests/build/required_checks.py`. - follow-on work after a release is *dispatched*, never triggered. 2. **Draft releases are invisible without push access.** A job that reads a draft needs `contents: write` even if it only reads. 3. **`workflow_run` and `schedule` only fire from the default branch's copy** of the workflow file. A new watcher or cron does nothing until it reaches `main`. Scheduled workflows carry `# scheduled-ref: default-branch-intentional` or pin a ref (`test_scheduled_workflows_pin_their_ref`). 4. **`pull_request_target` hands out a writable token.** Never check out and execute the PR head in it: restore scripts from the base SHA (see `dependabot-lockfile-sync.yml`) and guard on the actor. 5. Creating PRs with GITHUB_TOKEN requires the repo setting "Allow GitHub Actions to create and approve pull requests". ## Discord - Send only through `scripts/ci/notify_discord.py` (`--level info|success|warning|failure --title … [--body|--body-file] [--url] [--field k=v] [--mention]`). Never `curl` the webhook. - Secrets: `DISCORD_WEBHOOK_URL` (required for anything to send) and `DISCORD_MENTION_USER_ID` (the numeric user id — Developer Mode, right-click your name, Copy User ID — never the username; used only with `--mention`, and a bad value costs only the ping, not the message). Both must be **repository** secrets: environment secrets are invisible to every job without that `environment:`. Pass them through step `env:`, never interpolated into `run:` — a test enforces it. - Unset webhook or a Discord outage = logged no-op, exit 0. A notification is never a reason for a job to fail. - `allowed_mentions` is always explicit: text can never ping `@everyone`, whatever a branch or commit is named. Only `--mention` pings, and only the configured user. - **Notify on state changes, not on every run.** New failure on main/dev or a scheduled/dispatched run → ping. Green after red → recovery, no ping. PR runs, cancellations and green-after-green → silence. A nightly that rewrites an existing issue does not re-notify. - Adding a workflow that should page: add its exact `name:` to `notify.yml`'s `workflows:` list. `test_every_watched_name_is_a_workflow_that_exists` fails if a listed name stops matching. ## Adding an automation — checklist - [ ] Logic in a typed, docstringed stdlib script under `scripts/ci/` with unit tests in `tests/build/` (fixtures, no network). Nothing to register: `make lint` and `scripts/ci/tier0-static.sh` both glob `scripts/*.py scripts/ci/*.py`, so a new script is linted and type-checked the moment it lands. The enumeration these replaced had quietly lost 14 files, and only `make lint` — never CI — ran it. - [ ] **If the job that runs it has no `actions/setup-python` step, the script must run on Python 3.10** — `ubuntu-22.04`'s system `python3`. The dev venv's 3.12 hides the difference, which is how `datetime.UTC` reached `quarantine_notice.py`. `tests/build/test_ci_scripts_match_runner_python.py` fails the build on a newer stdlib name (an `ast` scan plus `vermin`), and `ruff.toml` pins the lint target to `py310` so ruff cannot ask for a 3.11+ alias back. Either stay portable, or add `actions/setup-python`. - [ ] Top-level `permissions: {}` or read-only; grant per job, minimum needed. - [ ] Every `${{ }}` through `env:` and quoted; actions pinned by tag like the rest of the repo (`actions/checkout@v5`), `persist-credentials: false` unless the job pushes. - [ ] Dispatch inputs get `# checkov:skip=CKV_GHA_7` with a reason; run `checkov -f --framework github_actions` locally. - [ ] Anything it pushes or opens is followed by `dispatch_required_checks.sh`. - [ ] Idempotent: a re-run for the same input updates or exits cleanly. - [ ] Destructive actions (delete branch, delete draft, close issue) re-check their precondition immediately before acting, and log each decision. - [ ] Failures reach Discord (add to `notify.yml`); successes usually don't. - [ ] Never `continue-on-error` to make it green, never auto-merge to `main`. ## AI agents Issue and PR text on this public repo is **untrusted input**. An agent that reads it must not also hold write access. - **Copilot cloud agent** (assign an issue, or `gh agent-task create`): for bounded code changes that come back as a PR for review. Good first tasks: QUAR-001 (#162), major Dependabot migrations, a shared sanitiser for the `py/log-injection` alerts. It reads `CLAUDE.md` and these skills (`.claude/skills/` is a supported skills path), so the verification rules apply to it too. - **GitHub Agentic Workflows** (`gh aw`, Copilot engine): for judgement over untrusted text — issue triage, CI failure analysis, release-note drafts, security digests. Keep them read-only; writes go through safe outputs. - **Copilot CLI unattended** (cron on the headless box): explicit `--allow-tool` lists, never `--allow-all`/`--yolo`; always `--deny-tool 'shell(git push)'`, `--secret-env-vars`, `--max-ai-credits`, `--no-ask-user`. Auth via a fine-grained PAT with **Copilot Requests** in `COPILOT_GITHUB_TOKEN` (classic PATs are not supported). Report through `notify_discord.py`. - Nothing an agent produces reaches `main` without a PR, the 21 required checks and a human merge. No agent approves or promotes a release. ## The headless box (Fedora Server, AMD, always on) Runs only what hosted runners cannot: the release soak (install the draft tarball, boot, probe `/readyz`, uninstall), a nightly `make e2e-local` against `main`, the Tier 3 QEMU fleet runner (`fleet.yml`, `[self-hosted, qemu]`, needs KVM) and Copilot CLI report jobs. **Self-hosted runners on a public repo will run fork code** unless restricted: register the box in a runner group limited to named workflows (`fleet.yml` and the soak/nightly ones), make jobs ephemeral (fresh container or VM per job), and never attach it to a `pull_request` trigger. Installer journeys there use rootless podman with `--security-opt label=disable`.