--- name: setup-dev description: "Dev environment setup for build tools, PostgreSQL, submodules, and worktrees. Use when setting up or entering a development worktree." --- # Setup Dev Environment Interactive playbook -- follow steps in order, detect current state, skip what is already done, and ask the user when a choice is required. ## Principles - **Global build cache**: ccache in `~/.ccache`, shared across all worktrees. - **Single clone per repo**: one clone each for pg_ducklake and postgres; use git worktrees for parallel work. - **Submodule sharing**: `git worktree add` on each submodule's git repo to share the same object store across worktrees. --- ## Step 1: Detect state Run these checks silently to understand what is already set up: ```bash command -v ccache # compiler cache ls pg-*/bin/pg_config 2>/dev/null # local PG installs in workdir ls ~/.dev/pg-*/configure 2>/dev/null # PG source worktrees ls "$(git rev-parse --show-toplevel)"/duckdb/Makefile 2>/dev/null # duckdb submodule initialized? git worktree list # current worktree situation ``` Skip any step below that is already satisfied. --- ## Step 2: Build tools and compiler cache ### Check ```bash command -v ccache && ccache --version ``` ### If missing, ask user Present these options using the agent's available interaction mechanism: | Option | macOS | Linux (apt) | |--------|-------|-------------| | **brew/apt** | `brew install ccache` | `sudo apt install ccache` | | **Skip** | Continue without cache (slow builds) | | Also check for platform build tools: - macOS: `xcode-select -p` (if missing: `xcode-select --install`) - Linux: `dpkg -l build-essential` (if missing: `sudo apt install build-essential bison flex`) ### Configure ccache for cmake Check the user's shell profile: ```bash grep CMAKE_C_COMPILER_LAUNCHER ~/.zshrc ~/.bashrc 2>/dev/null ``` If not configured, tell the user to add this block to `~/.zshrc` or `~/.bashrc`: ```bash if command -v ccache >/dev/null; then export CC="ccache cc" export CXX="ccache c++" export CMAKE_C_COMPILER_LAUNCHER=ccache export CMAKE_CXX_COMPILER_LAUNCHER=ccache fi ``` **Why**: duckdb and ducklake use cmake and ignore `CC=ccache cc`. Without `CMAKE_C_COMPILER_LAUNCHER`, these heavy builds (~1GB source) recompile from scratch in every worktree. **Note**: if `sccache` is already installed (e.g. via Homebrew), cmake may pick it up automatically. That works fine -- just use `sccache --show-stats` instead of `ccache -s` to verify cache hits. --- ## Step 3: PostgreSQL (build from source) Always build from source, per-worktree. Uses a shared bare repo so multiple PG versions share one clone, and ccache makes rebuilds fast. Ask the user which versions to install (usually the oldest and newest supported, e.g. 14 and 18). ### One-time: clone PG source Shared PG source lives at `~/.dev/pg-src` (bare repo, ~200MB): ```bash git clone --bare https://github.com/postgres/postgres.git ~/.dev/pg-src ``` For each needed version, fetch the branch and create a source worktree: ```bash git -C ~/.dev/pg-src fetch --depth=1 origin refs/heads/REL__STABLE:refs/heads/REL__STABLE git -C ~/.dev/pg-src worktree add ~/.dev/pg- REL__STABLE ``` ### Build and install into current workdir ```bash WT=$(git rev-parse --show-toplevel) NCPU=$(nproc 2>/dev/null || sysctl -n hw.ncpu) for VER in 14 18; do pushd ~/.dev/pg-$VER ./configure --prefix="$WT/pg-$VER" \ --without-icu --without-readline --without-libxml make -j"$NCPU" && make install popd done ``` `PG_CONFIG` will be: `$(pwd)/pg-/bin/pg_config` Add `pg-*/` to `.git/info/exclude` to keep git status clean: ```bash exclude_file=$(git rev-parse --git-dir)/info/exclude grep -qF 'pg-*/' "$exclude_file" 2>/dev/null || echo 'pg-*/' >> "$exclude_file" ``` **Time**: ~15-20 min per version on first run, ~1-2 min on reruns (ccache). --- ## Step 4: Submodules ### duckdb (submodule) pg_ducklake lives in the libpgduckdb monorepo: the libpgddb kernel is plain in-repo source at `libpgduckdb/` (repo root) -- nothing to initialize. The DuckDB source is a submodule at the repo root, `duckdb/`. Initialize it: ```bash git -C "$(git rev-parse --show-toplevel)" submodule update --init --depth=1 duckdb ``` **Time**: 5-10 min on first run (duckdb is large even with shallow clone). ### ducklake (submodule + patches) `third_party/ducklake` is a git **submodule** pinned to a community `duckdb/ducklake` commit. Our divergence from community lives as an ordered series of patch files `third_party/ducklake-NNN-.patch`, applied onto the pristine checkout at build time (apply-once, stamp-guarded -- see the `DUCKLAKE_STAMP` rule in `Makefile`). The build inits the submodule non-recursively (its inner `duckdb` / `extension-ci-tools` submodules are only for ducklake's own CI and are not needed) and applies the patches in numeric order; `make clean` drops the stamp so the next build re-applies from a clean reset. The submodule working tree is therefore intentionally left dirty (patched); the gitlink stays at the pinned community commit. To inspect our customization, read the `ducklake-NNN-*.patch` files. To bump the community base: repoint the submodule gitlink, regenerate/rebase the patches against the new commit, and re-verify (`make ... && make check-regression`). ### Submodule worktrees (automated via hooks) `git worktree add` on a submodule gitdir silently overwrites `core.worktree` to point at the new worktree instead of the main one. Stale worktree entries also break `submodule foreach --recursive`. Two hooks in `.claude/settings.json` handle this automatically: - **PostToolUse `EnterWorktree`** (`.claude/hooks/worktree-setup.sh`): repairs broken gitdirs, then creates submodule worktrees with `core.worktree` preservation. - **PreToolUse `ExitWorktree`** (`.claude/hooks/worktree-cleanup.sh`): removes submodule worktree entries pointing to the current worktree, then repairs any `core.worktree` that got corrupted. Only runs when action is `"remove"`. If the main worktree's submodules are not initialized, the setup hook has nothing to iterate. Fall back to `git submodule update --init --recursive --depth=1` in the main worktree first (slow, downloads from remote). ### PG install for new worktree The PG source worktrees at `~/.dev/pg-` already have compiled object files from the initial build. Reconfigure with the new prefix and rebuild. **`make clean` is required** because PG bakes the `--prefix` path into binaries as rpaths — without it, old objects with the previous worktree's rpath get reused and executables fail at runtime with "Library not loaded". ```bash WT=$(git rev-parse --show-toplevel) NCPU=$(nproc 2>/dev/null || sysctl -n hw.ncpu) for VER in 14 18; do pushd ~/.dev/pg-$VER make clean ./configure --prefix="$WT/pg-$VER" \ --without-icu --without-readline --without-libxml make -j"$NCPU" && make install popd done ``` **Time**: ~2-5 min per version (ccache hits on recompilation). --- ## Step 5: Build and test ```bash PG_CONFIG= make install PG_CONFIG= make installcheck ``` **Time**: first build takes 15-30 min (duckdb cmake build dominates). Subsequent builds with ccache take 1-5 min. A successful `make install` ends with PGXS installing `pg_ducklake.so` and the SQL files into the PG extension directory. --- ## Troubleshooting **ccache not hitting** -- `ccache -s` shows zero hits. Verify: - `echo $CC` contains `ccache` - `echo $CMAKE_C_COMPILER_LAUNCHER` is `ccache` - cmake is picking it up: check build logs for `ccache` in compiler command **Wrong submodule commit** -- reset to what the branch expects: ```bash git -C "$(git rev-parse --show-toplevel)" submodule update duckdb pg_ducklake/third_party/ducklake ``` **PG configure fails** -- missing build deps: - macOS: `xcode-select --install && brew install bison flex` - Ubuntu: `sudo apt install build-essential bison flex libreadline-dev zlib1g-dev` **`pg-*/` in git status** -- re-run Step 3's exclude command.