Paro Benchmark

Paro benchmark

Default to exploration: answer the specific question with the smallest useful experiment. A pilot needs source/binary, SQL/data, settings, timer scope and raw samples, not a preregistered certification campaign. Label conclusions accordingly.

Use the selected checkout and maintained runner

Inspect HEAD/status, preserve user work, and read benchmark README. Discover commands from make -C benchmark help, the intended CLI's --help and its implementation; don't copy stale flags. Inspect recursive Make recipes before a dry run. Reuse the declared Python environment and toolchain.

• Engineering workloads/gates: benchmark/runner.py, harness/, policies/.
• Corpus A/B and cold/warm: corpora/tpcds_compare.py; read CORPORA for data and result contracts.
• Compile diagnosis: corpora/cold_planning.py and EXPLAIN COMPILE.
• Actual operators: EXPLAIN ANALYZE and corpora/d6_execution_profile.py.
• Known-cardinality checks: corpora/plan_quality.py.

Use supported collectors/validators; extend their missing capability instead of building another timer or per-report parser. Full-corpus certification is not required for a targeted diagnostic run. Help must not start a server.

Running the TPC-DS collector

The corpora/ scripts are not importable as a package or runnable by path alone: run them from benchmark/ with PYTHONPATH=.:corpora .venv/bin/python corpora/<script>.py --help. Facts that are easy to miss:

• tpcds_compare.py builds cargo build --release --locked --bin parod in its own checkout and runs <checkout>/target/release/parod; it refuses a source tree that changes during the build. The reported source is that checkout, so the collector revision is the tested revision.
• Normal compile receipts carry compiler_elapsed_us only when the caller exports PARO_COMPILE_WORK_EVIDENCE=1; otherwise compile time is uncovered. Check one cell's compile_receipts[].compile.raw before a long campaign.
• --report runs/<id>/<name>.json creates the RunOutput directory runs/<id>/<name>-run/; each invocation is one campaign over --start..--end.
• --server-data-dir is an immutable seed, cloned per process. It must have a root-relative catalog (storage_dir: ./databases/db-N); older absolute-path seeds are rejected. Rebuild one under the data root: start the tested parod with the empty seed directory as cwd and --data-dir . on an owned port, run corpora/tpcds_setup.py --dsn ... --csv-dir <csv> (declared keys, matching --metadata-track generator-declared), then stop it with SIGTERM. The CSV directory needs all 24 tables plus schema.sql/load.sql, whose COPY paths are absolute. Use the same storage format for every arm.
• Query SQL lives in the DuckDB checkout under extension/tpcds/dsdgen/queries; see CORPORA.

Comparing another revision: add a detached source-only worktree and give it its own Cargo target (CARGO_TARGET_DIR=<worktree>/target); the collector reads <checkout>/target/release/parod. Never let two checkouts share a target. Path-dependency artifact hashes are workspace-relative, so the checkouts overwrite each other's crates, and freshness is judged by source mtimes: a "fresh" build can silently link the other revision's code, and two arms then measure one program. Before collecting, build every arm, confirm distinct binary hashes and one plan- or behavior-level difference per arm, and check each cell's recorded binary_sha256 against its arm. If a shared target was already used, cargo clean --release -p <changed crates> before trusting it again. Use that revision's own collector flags; the single-planner tree has no policy switch, and quality-versus-pipeline Memo comparisons need a revision at or before 5ba483059 (--optimizer-search-policy). Remove the worktree and its target after review.