Benchmark

Overview

Scalex has a multi-layered profiling and benchmarking system. Pick the right layer for the situation:

Decision guide

"Where is time spent?" → Start with --timings (Layer 1)

"Is this change faster?" → Use hyperfine before/after (Layer 2), optionally with bench-compare.sh

"Why is parsing slow?" → async-profiler CPU flame graph (Layer 3)

"Why are allocations high?" → async-profiler alloc or JFR ObjectAllocationSample (Layer 3/4)

"Is there GC pressure?" → JFR (Layer 4)

"How fast is extractSymbols on one file?" → Microbenchmark (Layer 5)

"How much memory does indexing use?" → Memory profiling (Layer 6)

"Is there a memory leak or GC regression?" → Memory profiling before/after (Layer 6)

Layer 1: --timings flag

The fastest way to see where time goes. Works in both JVM and native image. Prints to stderr.

# Cold index phase breakdown
rm -rf benchmark/scala3/.scalex
./scalex index benchmark/scala3 --timings

# Warm index
./scalex index benchmark/scala3 --timings

# Query with bloom/text-search breakdown
./scalex refs benchmark/scala3 Compiler --timings

# JVM mode
./mill run index benchmark/scala3 --timings

Phases reported

Index phases: git-ls-files, cache-load, oid-compare, parse, cache-save, and lazy build-* lookup phases

Query phases (refs/imports/coverage): bloom-screen, text-search

Durations are exclusive per thread: nested phases are subtracted from their parent. command and render expose query and output work. request-total measures elapsed application request time, including otherwise uninstrumented work, but not runtime startup before the entry point. Use hyperfine for complete process latency. Concurrent phases can overlap, so summing phases is not a wall-clock measurement. Batch mode reports the shared index load, then a separate request total for each query.

Reading the output

Timings:
  git-ls-files          12.3 ms  ( 1%)
  cache-load            45.2 ms  ( 5%)
  oid-compare            3.1 ms  ( 0%)
  parse                782.0 ms  (80%)
  index-build           89.4 ms  ( 9%)
  cache-save            42.1 ms  ( 4%)
  request-total        974.1 ms

• parse > 70%: Scalameta parsing dominates — look at parallelism, parser options, or reducing parsed file count
• cache-load > 20%: Index deserialization — check bloom skip, file size, buffering
• cache-save > 10%: Index serialization — check if save is running unnecessarily (when parsedCount == 0)
• bloom-screen > text-search: Bloom filter is slow — check expected element count, FPP
• text-search >> bloom-screen: Text scanning dominates — check candidate count (bloom too permissive?)

Layer 2: hyperfine benchmarks (bench.sh)

Reproducible, statistical benchmarks using hyperfine against the Scala 3 compiler repo (~17.7K files).

Prerequisites

• hyperfine installed (brew install hyperfine)
• Native scalex binary built (./build-native.sh)
• The scala3 repo is cloned automatically on first run

Running

# Full suite (cold + warm + query + diverse + timings)
.agents/skills/benchmark/scripts/bench.sh

# Individual modes
.agents/skills/benchmark/scripts/bench.sh cold
.agents/skills/benchmark/scripts/bench.sh warm
.agents/skills/benchmark/scripts/bench.sh query
.agents/skills/benchmark/scripts/bench.sh diverse    # miss, heavy refs, fuzzy, grep, hierarchy
.agents/skills/benchmark/scripts/bench.sh timings    # --timings output for cold/warm/refs
.agents/skills/benchmark/scripts/bench.sh memory     # heap usage, GC pressure (JVM only)

# Custom runs/binary
BENCH_RUNS=10 SCALEX_BIN=./target/scalex .agents/skills/benchmark/scripts/bench.sh

Before/after comparison

# 1. Benchmark current state
BENCH_EXPORT=benchmark/results/before.json .agents/skills/benchmark/scripts/bench.sh

# 2. Make changes, rebuild
./build-native.sh

# 3. Benchmark new state
BENCH_EXPORT=benchmark/results/after.json .agents/skills/benchmark/scripts/bench.sh

# 4. Compare (flags >5% regressions)
.agents/skills/benchmark/scripts/bench-compare.sh benchmark/results/before.json benchmark/results/after.json

bench-compare.sh exits non-zero if any benchmark regressed >5%.

Typical ranges (Apple M3 Max)

Layer 3: async-profiler flame graphs

Reveals call-stack-level CPU hotspots. No code changes needed — JVM agent only.

Prerequisites

brew install async-profiler
# Or set AP_HOME to your installation

Running

# CPU flame graph of cold index
./profiling/profile.sh benchmark/scala3

# Wall-clock (includes I/O wait — useful for parallelStream bottlenecks)
./profiling/profile.sh benchmark/scala3 wall

# Allocation hotspots (where objects are created)
./profiling/profile.sh benchmark/scala3 alloc

# Lock contention (parallelStream synchronization)
./profiling/profile.sh benchmark/scala3 lock

Output: profiling/profile-<event>.html — open in browser for interactive flame graph.

What to look for

• CPU: Wide bars in Scalameta parsing → specific parser methods. Wide bars in parallelStream infrastructure → overhead from small task granularity.
• Alloc: Hot allocation sites → potential for object reuse or structural changes.
• Lock: Contention in ConcurrentLinkedQueue.add() → batch results instead of per-item add.
• Wall: I/O wait in Files.readAllLines or Files.readString → potential for memory-mapped I/O.

Layer 4: JFR (Java Flight Recorder)

Built into the pinned JDK. Near-zero overhead. Best for GC, file I/O, and thread analysis.

Running

# Record with custom config
SCALEX_JAVA_OPTS="-XX:StartFlightRecording=filename=profiling/scalex.jfr,settings=profiling/scalex.jfc,duration=60s" \
  ./mill run index benchmark/scala3

# Quick summary
jfr summary profiling/scalex.jfr

# Specific events
jfr print --events jdk.ObjectAllocationSample profiling/scalex.jfr | head -100
jfr print --events jdk.GarbageCollection profiling/scalex.jfr
jfr print --events jdk.FileRead profiling/scalex.jfr | head -50
jfr print --events jdk.ThreadPark profiling/scalex.jfr | head -50

# GUI analysis
open profiling/scalex.jfr  # Opens in JDK Mission Control

Custom config

profiling/scalex.jfc is tuned for scalex — it enables allocation sampling, GC events, file I/O (>1ms threshold), and thread parking/monitor events.

Layer 5: Microbenchmarks (src/bench.scala)

Isolate per-function costs with warmup and statistical measurement.

Running

# Specific benchmark
./mill bench.run extract-single benchmark/scala3
./mill bench.run bloom-build benchmark/scala3
./mill bench.run persistence-load benchmark/scala3
./mill bench.run search benchmark/scala3
./mill bench.run refs benchmark/scala3

# All benchmarks
./mill bench.run all benchmark/scala3

# Custom warmup/iterations
./mill bench.run extract-single benchmark/scala3 --warmup 3 --iterations 10

Available benchmarks

Reports: mean, median, p99, stddev, min, max per benchmark.