Update Contrib

Update a ClickHouse Contrib (Third-Party Library)

Bump a git submodule under contrib/ to a new version, adapting CMake build files and ClickHouse source code as needed.

Arguments

• $0 (required): Library name as it appears under contrib/ (e.g., curl, openssl, arrow)
• $1 (optional): Target version — a git tag, branch, or commit SHA. If omitted, the latest upstream release tag is used.

Two modes

Interactive (a human runs /update-contrib): follow every step below, including the confirmations.

Non-interactive / pipeline mode: the invoking prompt states that the run is automated and passes the target, the branch name, the build directory and often a facts block (submodule URL, current pin, target commit, fork branch, CVE list). In that mode:

• a given target is confirmed — skip step 2's discovery and never call AskUserQuestion;
• never push, never create branches in forks, never open PRs, never use gh; the pipeline pushes and opens the PR;
• do not re-derive facts the prompt already states (URL, pin, tags, build flags) — the pipeline fetched full history and tags into the submodule before the run;
• if the update needs a change inside a ClickHouse fork that does not exist yet, stop and write the reason to the BLOCKED.md path named in the prompt instead of improvising;
• finish by writing the report the prompt asks for (REPORT.md) with an explicit "Open concerns" section ("none" when there are none).

Background

ClickHouse vendors all third-party libraries as git submodules under contrib/. Each library has:

• contrib/<lib>/ — the submodule (upstream repo or a ClickHouse fork)
• contrib/<lib>-cmake/ — ClickHouse's own CMake build files

contrib/update-submodules.sh deletes the upstream CMakeLists.txt files after checkout; ClickHouse relies exclusively on its own *-cmake/ files. Consequences: git -C contrib/<lib> status normally shows deleted upstream CMake files — that is expected, do not restore them and do not re-run the deletion by hand; ninja does not need it either.

Submodule URLs point to either upstream directly (unpatched) or a fork at github.com/ClickHouse/<repo> (when patches are needed; the repo name may differ from the contrib name, e.g. contrib/cassandra → ClickHouse/cpp-driver). Fork branches are named ClickHouse/<version> — usually the bare upstream tag (ClickHouse/v1.52.1, ClickHouse/2.17.1), sometimes with the upstream's own prefix (ClickHouse/openssl-3.5.8, ClickHouse/release-2.14.1) — never ClickHouse/master or ClickHouse/main. A pin can also sit on a fork branch while .gitmodules still names the upstream URL (GitHub serves fork-network objects through the parent repo); check the fork's branches before assuming a pin is a plain upstream commit.

Submodules are cloned shallow in most checkouts. git describe and git log OLD..NEW then report nonsense or fail; run git -C contrib/<lib> fetch --unshallow --tags origin first (the pipeline does this for you).

A few libraries are built through cargo instead of a plain CMake file list — either from rust/workspace/<lib>/ (wasmtime) or from a contrib/<lib>-cmake/ file that drives cargo (chdig, delta-kernel-rs); they export ch_rust::* targets. See rust.md for those.

Process

1. Gather information about the library

Throughout this skill, $LIB refers to the library name passed as $0 and $VERSION to the target version once resolved (step 2). Both must match [A-Za-z0-9._-]+ so they are safe to use unquoted in paths and branch names; reject anything else.

Skip whatever the prompt already states. Otherwise:

git config --file .gitmodules --get "submodule.contrib/$LIB.url"   # upstream or ClickHouse fork?
git -C "contrib/$LIB" rev-parse HEAD                                 # current pin
git -C "contrib/$LIB" fetch --unshallow --tags origin 2>/dev/null || git -C "contrib/$LIB" fetch --tags origin
git -C "contrib/$LIB" describe --tags --abbrev=0 2>/dev/null || echo "no tags"

Then locate the integration file(s) and keep them in $INTEGRATION; every later step works from that variable. Three shapes exist:

INTEGRATION=$(ls contrib/${LIB}-cmake/CMakeLists.txt rust/workspace/${LIB}/CMakeLists.txt 2>/dev/null)
# Shared wrapper: no directory of its own, built by another contrib's CMake file
# (aws-c-auth → contrib/aws-cmake/, the aws-c-* family; individual boost libs → contrib/boost-cmake/).
[ -n "$INTEGRATION" ] || INTEGRATION=$(grep -rl "contrib/${LIB}\b" contrib/*-cmake/CMakeLists.txt)
echo "$INTEGRATION"; [ -n "$INTEGRATION" ] || echo "no integration file found — stop and report"
grep -n "\b${LIB}\b" contrib/CMakeLists.txt rust/workspace/CMakeLists.txt   # where it is added

• plain CMake: contrib/<lib>-cmake/CMakeLists.txt;
• Rust: either rust/workspace/<lib>/CMakeLists.txt (wasmtime) or a contrib/<lib>-cmake/ file that drives cargo (chdig, delta-kernel-rs) — see rust.md;
• shared wrapper: the fallback above; the wrapper's file list and aliases (ch_contrib::aws_s3 for every aws-c-* library) are what steps 8, 9 and 11 must use.

Do not assume the exported target is ch_contrib::${LIB}. Take the actual alias names from $INTEGRATION and grep for those — Rust-backed libraries export ch_rust::* (ch_rust::wasmtime, ch_rust::chdig, ch_rust::delta_kernel_rs), a few keep upstream-style names (OpenSSL::SSL, boost::filesystem), and one library can export several:

ALIASES=$(grep -ho 'add_library *( *[A-Za-z0-9_:.-]*::[A-Za-z0-9_.-]* *ALIAS' $INTEGRATION \
          | sed -E 's/add_library *\( *//; s/ *ALIAS$//' | sort -u)
echo "$ALIASES"
for a in $ALIASES; do grep -rl --include=CMakeLists.txt -F "$a" src programs base rust; done | sort -u   # consumers

Read $INTEGRATION once; it tells you which sources are compiled, which defines and generated headers exist, and whether the library is header-only (add_library(... INTERFACE) — then nothing of it is compiled and only its consumers can break). The consumer list is what steps 9 and 11 work from.

2. Determine the target version

If $1 was provided, use it. Otherwise, find the latest release:

git -C "contrib/$LIB" tag -l --sort=-v:refname | head -20
# For ClickHouse forks, also check the upstream repo: git ls-remote --tags <upstream-url>

Interactive mode only: present the current and target versions to the user with AskUserQuestion for confirmation before proceeding. VERSION is whatever was confirmed (or $1). Verify it matches the charset above before using it in branch names or commits.

3. Check for ClickHouse patches

First decide whether the pin is a plain upstream commit. Do not rely on the .gitmodules URL alone: origin is the upstream repo when the URL names upstream, and ls-remote --heads origin 'ClickHouse/*' then returns nothing even if the pin sits on a fork branch. Check that HEAD is reachable from an upstream ref, and probe the fork separately (pipeline mode: the facts block names the fork URL and branch — use them):

git -C "contrib/$LIB" fetch --tags origin '+refs/heads/*:refs/remotes/origin/*'    # full upstream refs (step 1)
git -C "contrib/$LIB" branch -r --contains HEAD | head -3                          # empty: not an upstream commit
git -C "contrib/$LIB" tag --contains HEAD | head -3

UPSTREAM_URL=$(git config --file .gitmodules --get "submodule.contrib/$LIB.url")
FORK_URL="https://github.com/ClickHouse/$(basename "$UPSTREAM_URL" .git)"        # guess; may differ (cassandra → cpp-driver)
git -C "contrib/$LIB" fetch "$FORK_URL" '+refs/heads/ClickHouse/*:refs/remotes/ch-fork/ClickHouse/*' 2>/dev/null
git -C "contrib/$LIB" branch -r --contains HEAD 'ch-fork/*'                       # non-empty: pin lives on a fork branch

Classify:

• HEAD is on an upstream branch or tag and on no ch-fork/ClickHouse/* branch → unpatched upstream pin; skip the rest of this step.
• HEAD is on a ch-fork/ClickHouse/* branch → forked, patched pin (even when .gitmodules names upstream; step 5 fixes the URL). Continue below.
• HEAD is on neither and the fork guess failed (fetch errored or the repo name differs) → do not conclude "unpatched". Interactive mode: ask the user for the fork URL. Pipeline mode: stop and record the reason in BLOCKED.md.

For a forked pin, identify the patches that must survive the bump:

UPSTREAM_TAG=$(git -C "contrib/$LIB" describe --tags --abbrev=0)     # needs full history (step 1)
git -C "contrib/$LIB" log --oneline "$UPSTREAM_TAG"..HEAD              # ClickHouse-specific commits

Report the patches found. A patch is obsolete only when upstream now contains an equivalent fix — verify in the code, not by commit subject.

4. Create a branch for the update

git checkout -b "bump-${LIB}-${VERSION}"     # pipeline mode: use the branch name given in the prompt

5. Update the fork (if applicable)

If the library uses a ClickHouse fork and has patches, the new branch ClickHouse/<new-version> must exist in the fork with the patches rebased onto the upstream tag before the ClickHouse-side bump:

1. Interactive mode: prepare it in a clone of the fork — branch from the upstream tag, cherry-pick -x the patches in order (authors preserved), resolve conflicts from the new upstream code, drop only patches that upstream has absorbed. Do not push until the build is validated and the user confirms.
2. Pipeline mode: the branch is prepared by a separate job and named in the prompt; point the submodule at exactly that commit and never touch the fork.

If the library has no ClickHouse patches but uses a fork, switch the submodule to the upstream URL instead of creating a new fork branch (update .gitmodules, see step 6). Conversely, if step 3 found the pin on a fork branch while .gitmodules names upstream, fix the URL to the fork in the same commit. Never rewrite a URL to upstream unless step 3 positively classified the pin as an unpatched upstream commit.

6. Update the submodule pointer

git -C "contrib/$LIB" fetch origin              # plus the fork URL / branch when the target lives there
git -C "contrib/$LIB" checkout <target-commit>
git add "contrib/$LIB"

If the submodule URL changes (fork → upstream, or a fork whose URL was wrong), also update .gitmodules and run git submodule sync -- "contrib/$LIB":

git config --file .gitmodules "submodule.contrib/$LIB.url" "<new-url>"
git add .gitmodules

git submodule status must show no + line for the library when you are done (a + means the working tree is not at the recorded commit).

7. Diff the source tree between old and new versions

This is the step that determines what CMake and source changes are needed:

OLD_COMMIT=$(git rev-parse "HEAD:contrib/$LIB"); NEW_COMMIT=$(git -C "contrib/$LIB" rev-parse HEAD)
git -C "contrib/$LIB" diff --stat "$OLD_COMMIT..$NEW_COMMIT"
git -C "contrib/$LIB" diff --name-status "$OLD_COMMIT..$NEW_COMMIT" -- '*.c' '*.cpp' '*.cc' '*.h' '*.hpp' '*.in' 'CMakeLists.txt'

Pay attention to:

• Added/removed/renamed source files — the CMake file list must follow
• New build options or defines — may need new CMake variables or hardcoded config headers
• Changed public headers — may require source adaptation in src/
• Changed *.in templates / generated headers — regenerate the delta, never overwrite a checked-in generated header wholesale (diff generator(old) vs generator(new) and apply only that)

Read the changelog/release notes for API and behaviour changes (stored formats, output changes, dependency version floors that force co-bumps).

8. Update build integration files

Update the file(s) in $INTEGRATION from step 1 (contrib/<lib>-cmake/CMakeLists.txt, a shared wrapper such as contrib/aws-cmake/CMakeLists.txt, or the Rust files — see rust.md) to reflect the new version:

1. Compare source file lists against the actual files in the updated submodule; add new files after the first existing sibling from the same directory, remove deleted ones.
2. Check for new required defines, include paths, or compile options.
3. Verify the result follows ClickHouse CMake conventions — no find_package/find_path/find_library, check_c_compiler_flag, check_cxx_compiler_flag, check_c_source_compiles, check_include_file, check_symbol_exists, check_type_size, cmake_push_check_state, CMAKE_REQUIRED_FLAGS, or any Check* CMake module — these are forbidden by CI style checks (ci/jobs/scripts/check_style/check_cpp.sh) for hermetic, cross-compiled builds. Use hardcoded feature flags instead of runtime detection.
4. If contrib/<lib>-cmake/ carries config headers (config.h.in, hardcoded *_config.h), update them for new version strings, defines or feature flags.
5. Anything executed at build time (a code generator such as protoc or flatc) must be a native host tool: register it with add_native_target, use the IMPORTED_LOCATION under native/ for cross builds, and add its directory from the block in contrib/CMakeLists.txt where disable_dummy_launchers_if_needed restores the real toolchain (clang-tidy builds otherwise produce an empty, non-executable binary).
6. If the version is read from a file at configure time (e.g. liburing's .spec), re-run cmake after the bump; otherwise incremental builds keep a stale generated version header.