Benchmark

Matft vs Numpy benchmark

All measurement is done by a single script, scripts/benchmark.py.

• Matft side: runs Tests/PerformanceTests with swift test -c release and parses the output of XCTest measure {}.
  • Each test is measured with measureWithWarmup {} (Tests/PerformanceTests/PerfFixtures.swift), which works in two stages:
    1. Spin the block for --warmup seconds (default 0.5 s).
    2. Choose the number of calls per sample, N, so that one sample takes about --sample-time seconds (default 20 ms), then run measure {} (the same idea as number in timeit).
  • N is printed as a line MatftBench: -[<Class> <method>] number=N. The script divides each sample by N to get the time per call.
  • With one call per sample, XCTest's own per-iteration overhead dominates fast cases. For example, an operation that really takes 0.14 ms is reported as about 0.5 ms.
• Numpy side: measures the same expression with timeit.
• Result: compares the medians of both and writes benchmarks/results/latest.{json,md}.

On the docs site's Performance page (website/docs/performance.md), everything between <!-- BENCHMARK:START --> and <!-- BENCHMARK:END --> is generated. Do not edit it by hand.

Measure on a local Mac only. CI runners are too noisy, so never use their numbers in the docs.

1. Pre-checks

git status --porcelain
python3 -c "import numpy; print(numpy.__version__)"

• You can measure with uncommitted changes, but the report's commit field becomes -dirty. If the numbers are going into the docs, tell the user to run it after committing.
• If numpy is missing, suggest pip3 install numpy. Do not install it yourself.
• Ask the user to plug in the power and stop heavy work (builds, video in the browser, VMs, etc.). You can check the load with the load average from uptime and ps -Ao pcpu,comm -r | head.

2. Measure

cp benchmarks/results/latest.json /tmp/matft-bench-baseline.json   # to compare with the previous run (if it exists)
python3 scripts/benchmark.py [--baseline /tmp/matft-bench-baseline.json] [--filter Bool]

• It includes a release build, so it takes several minutes. Use a long timeout (10 minutes).
• --filter <regex>: narrow down by case ID, e.g. Bool, Sin.
• --skip-swift / --skip-numpy: reuse the previous JSON, e.g. to re-measure only Numpy.
• --warmup / --sample-time: warm-up seconds and target seconds per sample on the Matft side.
• --repeat / --number: number of samples and calls per sample for Numpy's timeit.
• --configuration debug: measure Matft in a debug build. SwiftPM builds dependencies with the same configuration as the app, so this is the speed seen by an app built without optimization. Generic loops such as initialize(repeating:) become orders of magnitude slower at -Onone, so when you change allocation or per-element loops, measure both release and debug. Cannot be combined with --update-readme.

Measuring the effect of an optimization (A/B comparison)

Using a previously saved JSON as the baseline compares against values taken at a different time under different load, which is very noisy — background load has shifted results by ±100% or more. Measure the effect of an optimization by alternating runs of the pre-change commit with the same measurement harness.

git worktree add /tmp/matft-before <commit before the change>
cp -R Tests/PerformanceTests/. /tmp/matft-before/Tests/PerformanceTests/   # align the harness and cases
cp scripts/benchmark.py /tmp/matft-before/scripts/
# Alternate before → after → before → after (--skip-numpy if only Matft matters)
(cd /tmp/matft-before && python3 scripts/benchmark.py --skip-numpy && cp benchmarks/results/latest.json /tmp/before-1.json)
python3 scripts/benchmark.py --skip-numpy && cp benchmarks/results/latest.json /tmp/after-1.json
# ...repeat for the second round. When done: git worktree remove /tmp/matft-before

• If the cases you did not change stay within a few percent, that round is trustworthy. Use them as a noise control.
• Finally, re-measure Numpy alone with --skip-swift so the Numpy ratios are taken under the same conditions.