Citus Backport

Backport a Citus change to the release branches

This skill encodes how to backport merged main commit(s) to Citus release branches: the release model, the SQL-schema remap recipe, the PG-version compatibility traps, and how to triage the release branches' baseline-red CI. Follow the repository's normal build/test/style conventions (see CONTRIBUTING.md and src/test/regress/README.md) on top of the guidance here.

Reference deep-dives (load on demand)

All live next to this file under references/:

• references/sql-schema-backport.md — the hard case: a commit that changes the SQL schema (new UDF / column / migration). Exact file-by-file remap recipe, the multi_extension.out and upgrade_list_citus_objects.out edits, the install-downgrades trap, and the cross-branch ladder-propagation step (copy the introduction paths UP to every higher major for MVU safety).
• references/ci-triage.md — reading backport CI: the PG-version compile traps (main drops old PGs the release branch keeps), the N-1 mixed-version jobs, and the step-by-step proof that a red job is pre-existing baseline noise vs a real regression you introduced.
• references/manual-upgrade-testing.md — the manual MVU proof: stand up a multi-node cluster at an OLD Citus major, then ALTER EXTENSION citus UPDATE across the backport branches (13.x tail → cross-major → 14.x tail), capturing citus_version() + pg_extension.extversion per node per hop. The make install-all / single-shared-PG requirement.

A C-only backport usually needs only this hub + ci-triage.md. A SQL-schema backport needs all four (add manual-upgrade-testing.md as the final proof once CI is green).

Release model (internalize this first)

• ONE branch per MAJOR. release-13.2 serves the WHOLE 13.x line; release-14.0 serves ALL of 14.x; main is the next major. Citus no longer cuts a branch per minor (this changed ~2026-05).
• Backport FEATURES, not just bugfixes, to the newest two majors (13 & 14 as of this writing).
• Branch names LIE about the version. release-13.2 currently ships 13.4; release-14.0 ships 14.2. NEVER infer the target minor from the branch name — read default_version from src/backend/distributed/citus.control on each target branch. That value drives every SQL edit.
• N-1 compatibility is required and CI-gated. A backport is N-1-safe when it is purely ADDITIVE (new UDF / GUC / column) and does not change an existing UDF signature or wire format used cross-version. If it's additive, it's safe. See references/ci-triage.md for the jobs.

Workflow (per PR, per target branch)

1. Use an isolated build per target major. A dedicated checkout/worktree + its own PostgreSQL install per release branch keeps builds/tests from clobbering each other and your main checkout. Two worktrees of the same clone SHARE one git object store — all branches are visible/pushable from either, and a +-prefixed branch in git branch is checked out in a sibling worktree.
2. Make the SHAs reachable, then git checkout -b bp-<pr>-<release-branch> origin/<release-branch> and git cherry-pick -x <sha> (the -x records (cherry picked from commit <sha>)).
3. Resolve:
   • C-only: usually applies clean or with trivial context conflicts. THEN check PG-version compat (references/ci-triage.md §PG-version) — the #1 non-obvious C-backport failure.
   • SQL-schema: migration / udf / multi_extension.out conflict → remap by hand per references/sql-schema-backport.md. Do NOT just accept main's next-major (e.g. 15.0-1) files.
4. Build in the isolated env: make -sj"$(nproc 2>/dev/null || sysctl -n hw.logicalcpu)" install. For any SQL change ALSO install downgrades (see references/sql-schema-backport.md — plain make install skips them and multi_extension will fail with cascading "no update path" errors otherwise; use make install-all).
5. Test only what's relevant (CI runs the whole suite): the feature's own test + multi_extension for SQL changes. From src/test/regress: python citus_tests/run_test.py <test> (it manages its own cluster). Regenerate any expected/*.out with the runner — never hand-edit .out (column widths / row counts are dynamic).
6. Collapse to ONE commit per PR per branch. Keep the original message; keep a DESCRIPTION: first line iff the change is user-facing (this feeds the changelog). git checkout -- configure if a build regenerated it (configure is never part of a backport). Multiple PRs to the SAME target → ONE branch per target, not per PR (unsquashed: one commit per PR, in main's chronological merge order; SQL-ladder commits sit right after their PR's cherry-pick). Name it bp-release-<major> (e.g. bp-release-13.2). Build cleanly from the SQL-heavy PR's branch, then cherry-pick the C-only PRs on top. Cherry-pick an already-adapted backport commit WITHOUT -x — its message already carries the single (cherry picked from <orig>) trailer; -x would double it.
7. Push to your fork. bp-* branches don't match CI's push trigger (main/release-* only), so kick CI with gh workflow run "Build & Test" --repo <your-fork>/citus --ref <branch>. Open PRs only when the change owner asks.
8. Wait for CI and triage per references/ci-triage.md. Separate baseline-red noise (N-1 / flaky / coverage) from genuinely-new failures you introduced. The two you WILL hit on a release branch: (a) a version-pinned .out that prints a shared helper's changed signature, and (b) a PG-version planner divergence on the OLDER PG the branch keeps but main dropped (PREFER a PG-guarded file like pg16.sql over a <test>_N.out alt file). A third you must NOT chase green by mangling the test: a new GUC/UDF whose test runs under N-1 shows a NEW red in the (non-blocking) N-1 mixed-version jobs — the old lib lacks it and falls back, which IS the N-1 contract. The clean fix is to move that test line into the multi_1_create_citus_schedule placeholder section (schedule mechanics + N-1 label rule in the hard-rules below and ci-triage.md §"NEW N-1 red"). Fix-commit rule (all fixes): pre-review → amend + git push --force-with-lease; post-review → add ONE new fix commit, never rewrite reviewed SHAs (replace a bad remote fix commit with --force-with-lease=<branch>:<badsha>). Report evidence; don't declare done on red.
9. Propagate the ladder UP (SQL-schema backports only). A new SQL object must also be reachable on every HIGHER major's upgrade ladder, or a future 13.x→14.y Major Version Upgrade loses it. As SEPARATE commits (its own PR on main), copy the introduction step/downgrade/udf files byte-for-byte onto release-14.0 (for a 13.x path) and main (for 13.x and 14.x paths, given today's two most recent major versions); main's own top path stays untouched. Full topology + recipe in references/sql-schema-backport.md §Step 5.
10. Add multi_extension.sql ladder-test coverage for every new step. The test only walks the main upgrade path, so a step OFF that path (a 13.x/14.x maintenance-tail detour) is never exercised unless you add a detour block matching the on-walk shape (and the file's existing comment style — no editorial parentheticals): a round-trip no-op + a snapshot at the detour, then retarget the existing next round-trip to bounce <detour> ↔ <next> rather than adding a separate "return" block. That shifts exactly one downstream snapshot; everything after stays byte-identical. A branch's own top/default_version is already covered by its snapshot block. Never hand-edit .out; prefer append-only on reviewed branches. Full pattern + the diff -w header gotcha in references/sql-schema-backport.md.
11. Prove MVU safety on a real cluster (SQL-schema backports, after CI is green). The ladder propagation only claims a future 13→14 Major Version Upgrade keeps the object; prove it. Stand up a multi-node cluster at an OLD major, then ALTER EXTENSION citus UPDATE on EVERY node across the backport branches (13.x tail → cross-major → 14.x tail → cross-major → main tail), capturing citus_version() and pg_extension.extversion per node per hop, plus a functional check that the backported object survived. Build all Citus versions into ONE shared PG with make install-all (plain install omits the downgrade + bridge scripts the cross-major route needs → "no update path"). Full recipe and the single-PG requirement in references/manual-upgrade-testing.md.