Benchmark Pyrefly

Benchmarking Pyrefly

Use this skill when asked to benchmark pyrefly performance, run a pyrefly bench, compare cold-start vs error-propagation latency, or update the PyTorch benchmark pin. All benches live in pyrefly/pyrefly/benches/ and use Criterion.

Build mode matters: always build optimized, or the numbers are meaningless. Buck uses @fbcode//mode/opt (or @fbcode//mode/opt-clang-thinlto for final numbers). Cargo cargo bench already uses the optimized bench profile; cargo run/cargo build would need --release. Debug builds run 3-10x slower and are not comparable.

Never run two benchmarks in parallel — they compete for CPU and memory and the timings become unreliable.

Micro benchmarks (fast)

Deterministic, single-threaded microbenchmarks of type-checker internals. Cheap to run, good for a quick signal.

• Buck: buck2 run @fbcode//mode/opt fbcode//pyrefly/pyrefly:micro_bench -- --bench
• Cargo: cargo bench --bench micro

Source: pyrefly/pyrefly/benches/micro.rs.

PyTorch benchmarks (heavy, walltime)

Three real-world benchmarks run over a pinned, multi-gigabyte PyTorch checkout (15k+ Python files) across all cores. They are walltime benchmarks (threads, I/O, a long cold start), not deterministic unit tests — distinct from the micro benchmarks. Two drive the actual LSP server (cold_start, error_propagation); one runs a cold batch check (full_check).

All three ship in one target — buck pytorch_bench, cargo bench pytorch — and you select an individual one at runtime with a Criterion name filter rather than picking a separate target. They live under pyrefly/pyrefly/benches/pytorch/, one module file per bench:

• pytorch/main.rs — crate root; declares the modules and calls criterion_main! aggregating the benchmarks' Criterion groups.
• pytorch/common.rs — shared PyTorch-checkout acquisition harness and standard LSP args.
• pytorch/cold_start.rs — the cold-start benchmark. Fresh server per iteration; opens torch/distributed/pipelining/_backward.py and queries go-to-definition of the Parameter import. Proxy for time-to-first-index. Criterion id pytorch/cold_start_go_to_definition.
• pytorch/error_propagation.rs — the error-propagation benchmark. Warm server; edits torch/nn/__init__.py to rebind Parameter to an int and waits for the resulting type error to surface in the distant dependent _backward.py. Proxy for incremental edit-propagation latency. Criterion id pytorch/error_propagation.
• pytorch/full_check.rs — the full-check benchmark. Fresh State per iteration; runs exactly what pyrefly check (project mode, no file args) does from inside the checkout — discovers the project and checks every project file across all cores. Proxy for whole-project batch throughput (not interactive latency). Criterion id pytorch/full_check.

Time cost

These are slow. A cold-start iteration is ~3-5 s on a 64-thread devvm; error-propagation is ~2-3 s and full-check ~1-1.5 s per iteration. Criterion's sample floor is 10, so budget roughly 2-4 minutes per benchmark. Do not run them in CI Sandcastle by default — they are manual/heavy (the PyTorch http_archive dep is labeled manual).

Run commands

Run all benchmarks:

# Buck (internal)
buck2 run @fbcode//mode/opt fbcode//pyrefly/pyrefly:pytorch_bench -- --bench
# Cargo (OSS)
cargo bench --bench pytorch

Run just one, selecting it by Criterion name filter:

# Buck (internal) — cold start
buck2 run @fbcode//mode/opt fbcode//pyrefly/pyrefly:pytorch_bench -- --bench cold_start
# Buck (internal) — error propagation
buck2 run @fbcode//mode/opt fbcode//pyrefly/pyrefly:pytorch_bench -- --bench error_propagation
# Buck (internal) — full check
buck2 run @fbcode//mode/opt fbcode//pyrefly/pyrefly:pytorch_bench -- --bench full_check
# Cargo (OSS)
cargo bench --bench pytorch -- cold_start
cargo bench --bench pytorch -- error_propagation
cargo bench --bench pytorch -- full_check

Useful flags (append after -- for buck; pass directly for cargo):

• --list — list the benches in the binary instead of running them, e.g. buck2 run @fbcode//mode/opt fbcode//pyrefly/pyrefly:pytorch_bench -- --list.
• --quick — quicker, lower-confidence run.
• --noplot — skip Criterion report rendering. Required on a headless devserver with no gnuplot and no usable font: the plotters backend otherwise panics after a bench finishes with BackendError(FontError(FontUnavailable)). Timings are unaffected. Example: buck2 run @fbcode//mode/opt fbcode//pyrefly/pyrefly:pytorch_bench -- --bench --noplot.