Patch Release Check

patch-release-check

ClickHouse publishes stable patch releases for its supported versions (the last three major releases plus the latest LTS) on a roughly bi-weekly cadence. They are created automatically by the scheduled AutoReleases workflow, but the pipeline can wedge silently and stop releasing. This skill is the every-two-weeks health check: it tells you which targeted versions are missing a recent patch, why the automation failed (classified), and walks gated remediation. There are two distinct failure modes, and they need different fixes:

• Guard failure — AutoReleaseInfo aborts before releasing anything because an open "version bump" PR trips a guard in ci/jobs/auto_release_job.py. Repo/PR side.
• Runner failure — runs can't start at all because no [self-hosted, release-maker] runner is available. Infra side, not a repo change.

Arguments

• $0 (optional): lookback window in days for the workflow failure scan. Default 14.

Hard rules

• Investigation is read-only. Steps 1–4 never change anything.
• Reach the CLI as env -u GH_CONFIG_DIR gh (this is what the script's GH() wrapper does). Dropping GH_CONFIG_DIR avoids a poisoned config dir — one whose token passes gh auth status but 403s on the API — by falling back to the default config / GH_TOKEN. Do not use command gh: command only bypasses an interactive alias (irrelevant in a non-interactive script) and does not strip GH_CONFIG_DIR, so it inherits the poisoned config; and env -u GH_CONFIG_DIR command gh is wrong too — it tries to exec a nonexistent command binary on Linux.
• Working files go in tmp/, never /tmp. (The script also points XDG_CACHE_HOME at a repo-local tmp/gh-cache when the default ~/.cache is not writable — some sandboxes mount a read-only HOME, which would otherwise make gh run view --log-failed fail before reaching GitHub.)
• Never close a PR or dispatch a release without explicit per-action confirmation (use AskUserQuestion). Steps 5–6 are gated.
• Only ever close a robot-clickhouse PR whose head branch is auto/v* AND whose release tag is already superseded. Never a human PR. Fail-close: if you are not certain a guard match is the orphaned robot bump PR, stop and report — do not close.

Steps

1. Run the health check

bash .claude/skills/patch-release-check/scripts/release_health.sh 14

It prints four blocks (each ending in a one-line verdict):

1. Targeted versions — the ✔️ rows of SECURITY.md (supported), what was excluded by config (excluded), and what is actually analyzed.
2. Per-version staleness — latest patch tag (resolved from the complete git/matching-refs tag list, so a quiet/older LTS is never missed), age in days (from the annotated tag's tagger.date, matching auto_release_job.py), release-worthy / first-parent-total commits (rel/tot, reconstructed from the first-parent chain like AutoReleaseInfo), and a ⚠️ MISSING flag when a version is older than STALE_DAYS (default 18) and has release-worthy commits. A supported version with no release tag is a hard failure: the block prints NO TAG and the final verdict is FAILED (exit 3), never green. (A valid git tag with no published GitHub Release is still healthy — the date comes from the tag object.)
3. Failure scan — AutoReleases and CreateRelease runs in the window, each AutoReleases failure classified GUARD / RUNNER / OTHER, plus a tally.
4. Guard re-check — the live result of the exact guard query, i.e. the PR(s) currently wedging the pipeline.

Knobs (env vars): EXCLUDE_VERSIONS (space-separated versions to drop from the analysis; defaults to a hardcoded skip — see Notes), STALE_DAYS (missing threshold).

2. Determine the targeted versions

These are the ✔️ rows in SECURITY.md, generated by utils/security-generator/generate_security.py (the 3 most recent regular versions + up to 2 LTS, months 3 and 8). Read them live — never hardcode; the set rolls forward every release. The script already does this in block 1.

3. Decide which releases are missing

A targeted version is missing (act now) when its latest patch is older than the cadence (~2–2.5 weeks) and its branch has release-worthy commits. That is not raw ahead_by: AutoReleaseInfo drops the post-release version-bump commit (Update autogenerated version ...) before deciding whether a branch has a patch candidate, so the script counts only commits whose message is not a version bump. The rel/tot column shows release-worthy / total — a branch at 0/1 is stale but has only the bump commit and nothing to release (verdict says so), and must not be flagged MISSING. Distinguish "missing" from due — a version that just hit its cadence boundary; once the pipeline is unblocked, automation will release it, so it is not "missing" yet.

4. Diagnose the failures

GUARD — AutoReleaseInfo died in the version-bump-PR guard _assert_no_open_version_bump_prs (raise RuntimeError). This is the guard "check all previous version bump PRs were merged": it runs gh pr list --state open --search "Update version_date.tsv in:title" and aborts if the result is non-empty. The guard runs before any per-branch dispatch, so a guard failure skips every branch — nothing releases.

The script classifies GUARD only when the failed-step log shows both a guard traceback frame and the raise RuntimeError source line — not a line number (which drifts) and not bare RuntimeError. Both frames count: in _assert_no_open_version_bump_prs (ci/jobs/auto_release_job.py) and the legacy in _prepare (tests/ci/auto_release.py), which the 14-day lookback still reaches for two weeks after the praktika migration. Other failures — e.g. a branch reported ERROR for having no release tag — classify as OTHER, so the operator is not sent to hunt version-bump PRs when the guard is clear.

The GitHub Actions log does not name the offending PR — the list is sent to a Slack alert, not stdout. Find it two ways:

• Block 4 of the script re-runs the exact query and prints the live match set.
• Slack: the CIBuddy bot posts Found not merged version bump PRs to #core-ci-info (channel C07FHK0FV5H); the message body is the literal PR JSON and it links the failing run. Search: "Found not merged version bump PRs".

The guard's search is loose full-text, so it also catches unrelated PRs that merely mention version_date.tsv (false positives). The real blocker is the one that recurs in the alert every day — typically an orphaned robot-clickhouse auto/v* bump PR whose release is already superseded.

RUNNER — a completed, cancelled run whose first job never got a runner (no runner_name, zero steps); it sat queued ~24h and was cancelled the instant the next day's cron fired. Means no [self-hosted, release-maker] runner picked it up. This is infra, not a repo change — escalate in #core-ci-info and check the org/repo Actions runner settings. Note: clearing a GUARD blocker does nothing if RUNNER is also failing, and vice versa — both must be healthy for a release to happen. A run that is still queued/in_progress (empty conclusion) is not RUNNER — the script prints its live status instead, since a queued daily run has no runner/steps yet but is not an outage.

OTHER — the run failed for some reason other than the guard or a missing runner. Any unexpected conclusion (startup_failure, timed_out, action_required, …) also counts here, so the tally never reads all-zeros while a run actually failed; startup_failure/timed_out usually point at the release-maker runner or a timeout. Read the failed step directly:

env -u GH_CONFIG_DIR gh run view <run-id> --repo ClickHouse/ClickHouse --log-failed

UNKNOWN — the script could not fetch the required GitHub data to classify the run: either a complete failed-step log (transient API/rate-limit truncation) or, for a cancelled run, its job metadata. It refuses to guess. This is fail-closed: do not assume it was healthy or not a guard failure — re-run the check, or read the run manually with the command above.