Rust Bench Criterion

Criterion benchmarks in Rocky

The benches and the CI gate

There are three criterion benches in the engine, and the workflow runs all three:

Decided (#1947, 2026-09-17): dag_execute is not redundant with compile's dag_resolution group — dag_resolution benches the synchronous graph algorithms (topological_sort + execution_layers); dag_execute benches DagExecutor::execute itself (the tokio runtime, the semaphore, JoinSet intra-layer fan-out). Different surfaces, so it's wired in with sample_size(10) to keep it cheap. As with the other two, this buys "it still compiles and runs on CI hardware" — see the no-comparison-step note below — not a perf gate. The correctness property this bench demonstrates (parallel beats sequential) is separately pinned by a real, non-flaky assertion: max_concurrency_bounds_intra_layer_fan_out in rocky-core/src/dag_executor.rs's test module counts peak concurrent nodes via an AtomicUsize high-water mark instead of timing anything, and runs on every cargo test, not just perf-labeled PRs.

compile.rs has four groups:

• cold_compile — full compile() over synthetic projects of 10 / 100 / 1,000 models (plus 10,000 in release mode only — debug builds with DuckDB C++ would take ~30 s per iter and dominate CI).
• dag_resolution — topological_sort + execution_layers on diamond-shaped DAGs of 10 / 100 / 500 / 1000 nodes.
• sql_generation — generate_create_table_as_sql and generate_insert_sql for 10 / 100 / 500 replication plans.
• binary_startup — subprocess startup time for cargo run -p rocky -- --version.

CI wiring — .github/workflows/engine-bench.yml:

on:
  pull_request:
    types: [labeled, synchronize]
    paths:
      - 'engine/crates/**'
      - 'engine/Cargo.toml'
      - 'engine/Cargo.lock'
      - '.github/workflows/engine-bench.yml'

jobs:
  bench:
    if: contains(github.event.pull_request.labels.*.name, 'perf')
    # ...
    steps:
      - name: Run benchmarks
        run: |
          cargo bench --bench compile -p rocky-cli -- --output-format bencher | tee bench-output.txt
          cargo bench --bench state_store -p rocky-core -- --output-format bencher | tee -a bench-output.txt
          cargo bench --bench dag_execute -p rocky-core -- --output-format bencher | tee -a bench-output.txt
      - name: Upload benchmark results
        uses: actions/upload-artifact@...
        with:
          name: bench-output
          path: engine/bench-output.txt
          if-no-files-found: error

There is no comparison step, and there never has been one that worked. The workflow's own comment says why: it runs only on perf-labeled PRs, with no push trigger, so there is no main-branch baseline to compare against. github-action-benchmark needs a gh-pages history branch, and this repo has none. The run uploads raw bencher output as a downloadable artifact and stops there.

The rules you actually need to know:

1. Benches only run when the perf label is on the PR. Adding a bench doesn't automatically run it on PR. To exercise it, add the perf label.
2. The workflow hardcodes three cargo bench invocations — compile -p rocky-cli, state_store -p rocky-core, dag_execute -p rocky-core. A new [[bench]] is not picked up until you add another invocation for it; adding a [[bench]] target to a Cargo.toml alone does nothing in CI.
3. Nothing compares your numbers to anything. No baseline, no alert threshold, no PR comment, no pass/fail. To see a regression you download the bench-output artifact and read it, or you run the bench locally on both revisions yourself. Treat the CI run as evidence the bench still COMPILES AND RUNS, not as a perf gate.
4. Binary startup bench assumes cargo run works from the working directory — it's advisory, not load-bearing; don't panic if it reports noisy numbers.

Adding a new criterion bench to an existing crate

1. Add the dep. In the crate's Cargo.toml:

   [dev-dependencies]
   criterion = { version = "0.5", features = ["html_reports"] }
   tempfile = "3"  # if you need fixtures
   
   [[bench]]
   name = "my_bench"
   harness = false

   harness = false is required for criterion (it supplies its own main).

2. Create crates/<crate>/benches/my_bench.rs. The canonical shape, modelled on benches/compile.rs:

   //! Short doc of what this measures and how to run it.
   //!
   //! Run with: `cargo bench --bench my_bench -p <crate>`
   
   use criterion::{BenchmarkId, Criterion, criterion_group, criterion_main};
   
   fn bench_happy_path(c: &mut Criterion) {
       let mut group = c.benchmark_group("happy_path");
       group.sample_size(10);  // ← drop from the default 100 for expensive iters
   
       for size in [10, 100, 1_000] {
           let input = build_input(size);
           group.bench_with_input(
               BenchmarkId::new("variant_name", size),
               &input,
               |b, input| {
                   b.iter(|| {
                       my_function(input);
                   });
               },
           );
       }
       group.finish();
   }
   
   criterion_group!(benches, bench_happy_path);
   criterion_main!(benches);
   
   fn build_input(size: usize) -> Input { /* ... */ }

3. Decide what scales to measure. Rocky's pattern is to sweep an n dimension (models, rows, nodes) with a few 10× steps (10 / 100 / 1000). Steps > 10× are rare. If one step takes > 30 s on a debug build, gate it behind cfg!(debug_assertions) so it only runs in release — see the 10_000 gate in bench_cold_compile.

4. Reduce sample_size for expensive iters. The default 100 samples × ~30 s per iter = 50 minutes. group.sample_size(10) is the right call for anything that touches DuckDB, the full compiler, or a real filesystem fixture.

5. Name groups and IDs for a reader, not for history. Nothing stores results between runs, so renaming a group loses nothing and costs nothing — there is no baseline to break. Criterion's own target/criterion/ directory is local and per-checkout, so it is not history either: deleting engine/target destroys no CI perf record, because none is kept. Name them clearly because a human reads them in the artifact.

6. Test locally before adding the perf label.

   cd engine
   cargo bench --bench my_bench -p <crate>

   Criterion writes HTML reports to target/criterion/<group>/<id>/report/index.html — open those to confirm the numbers look sane before pushing.