Vlm Benchmark

VLM Benchmark — on-device optimization A/B

Benchmark an optimization change (a PR or branch, in the addon or the qvac-fabric) on a specific AWS Device Farm device, using the packages/llm-llamacpp/benchmarks/vlm-benchmark framework. It builds the change from source, runs a baseline vs optimized A/B on the device (N runs each), pairs them by CPU fingerprint, and reports speed and quality — with quality regression as the primary gate. This codifies the QVAC-21320 / QVAC-21297 methodology.

Scope: Qwen3.5-0.8B VLM, projector mmproj CPU-vs-GPU (mmproj=both) by default. The model is selectable (--model) — the framework already ships multiple VLMs (Qwen3.5 f16/q8, Gemma-4-E2B) and a new one can be added to config.cjs; Qwen3.5-q8 is just the default. Device Farm only (no local ADB run yet). Thermal / cross-unit variance is handled by CPU-fingerprint matching over multiple runs, not interleaving.

Source safety (read-only target). This skill never modifies the PR or branch under test. It only reads the target — gh pr view / gh pr diff / git diff — to resolve its ref/commit and classify the change. Every change (harness/config edits and the optimization itself — the fabric overlay REF or cherry-picked addon commits) is made only on the skill's own test/<slug> / test/<slug>-base refs. It never commits to, pushes to, or otherwise alters the source PR's branch, the fabric repo, or main.

Usage

/vlm-benchmark #2921
/vlm-benchmark https://github.com/tetherto/qvac/pull/2921
/vlm-benchmark https://github.com/tetherto/qvac-fabric-llm.cpp/pull/172 --model gemma4-q4 --device pixel9
/vlm-benchmark feat/QVAC-21297-opencl-vision-encoder --device s25 --preset full --runs 2
/vlm-benchmark feat/my-opt --mmproj both --preset base

Arguments

Prerequisites & paths

The commands below use placeholders — resolve each once, up front, then substitute (don't assume hard-coded paths):

• gh CLI authenticated to tetherto/qvac (Actions dispatch + read); node on PATH (for match-devices.js).
• <qvac> — the local qvac main clone (origin = tetherto/qvac). Discover it, don't guess: run git worktree list from any qvac checkout → the first line's path is the main worktree. The session's current directory may be a read-only worktree — never build or push from it.
• <feature-branch> — the resolved target branch from Step 1 (the PR head branch, or the branch argument). The optimized ref is test/<slug>; the baseline ref is test/<slug>-base.
• <pr-repo> — the GitHub repo the PR lives in (resolved in Step 1): tetherto/qvac-fabric-llm.cpp for a fabric PR, tetherto/qvac for an addon PR. Used only to read the PR; Actions dispatch always targets tetherto/qvac.
• <slug> — the feature-branch basename (branch feat/QVAC-21297-opencl-vision-encoder → slug QVAC-21297-opencl-vision-encoder).
• <worktree> — the isolated worktree this run creates in Step 5: <qvac>/../wt-<slug> (a sibling of the clone — the repo's wt-<name> convention). All edits, build, and push happen here.
• <base> — the branch the worktree is cut from; chosen in Step 4 (origin/main, or the qvac feature branch related to the target).

Workflow

Step 1 — Resolve the target

• PR (#<n> or URL): resolve the repo from the target — a qvac-fabric-llm.cpp PR URL → tetherto/qvac-fabric-llm.cpp (a fabric PR); a qvac PR URL or a bare #<n> → tetherto/qvac (an addon PR). Then gh pr view <n> --repo <pr-repo> --json headRefName,commits,files,title,body → the head branch is the optimization. (<pr-repo> is only for reading the PR; all Actions dispatch in Steps 7–8 targets tetherto/qvac, where the workflow lives.)
• Branch: use it directly.
• Parse the flags off the tail (--device / --model / --mmproj / --preset / --runs); unset ones take their defaults, but --device/--model/--mmproj are reconciled against the PR in Step 3 before anything is built.

Step 2 — Detect the change scope (from the diff)

Run gh pr diff <n> --repo <pr-repo> (PR; <pr-repo> from Step 1) or git diff main...<branch> (branch). Classify:

• Addon change — touches packages/*/addon/** or addon JS/config. The optimization is the addon commit(s); baseline is the pre-optimization state (Steps 5–6 build it as <base> + harness edits, addon change omitted; the optimized ref adds the addon commits on top).
• Fabric change — the optimization is a qvac-fabric (tetherto/qvac-fabric-llm.cpp) commit (a fabric PR/commit, or a branch that bumps the fabric). The A/B uses a self-contained vcpkg overlay-port whose REF is the only delta: optimized overlay REF = the fabric PR/branch head commit; baseline overlay REF = the pre-change fabric commit (the PR's merge-base with its base, or the published registry version's commit). Both refs use the overlay — Step 6 covers creating the port and computing its SHA512 directly (no CI round-trip).
• Mixed / dependency — a PR that needs both (addon + a fabric overlay): apply both on the optimized build; the baseline strips only the optimization under test. Keep the optimization the sole delta between the two builds (isolation).

Step 3 — Reconcile the passed options against the PR

Before building, cross-check --device / --model / --mmproj against what the change actually targets, from the PR title/body/linked ticket + the diff-touched code:

• Device / backend — ggml-opencl / "OpenCL" / "Adreno" → an Adreno device (token s25); ggml-vulkan / "Vulkan" / "Mali" → a Mali device (token pixel9); "Metal" → iOS/desktop. Plus any explicit device/SoC mention in the PR (S25, Pixel 9, Adreno 830, Snapdragon) → its token.
• Model — a Qwen / Gemma / other model or projector mention.
• mmproj axis — whether it's a projector (mmproj) change vs an LLM change.

Then, per axis:

• (a) user did not pass it and the PR implies a value → adopt the PR's value and notify.
• (b) user passed it and it conflicts with the PR (e.g. --device pixel9 (Mali/Vulkan) on an Adreno/OpenCL-only fix — the change wouldn't even be exercised) → AskUserQuestion: proceed as passed / switch to the PR's value / cancel.
• (c) no PR signal → keep the passed value or the default.

Branch-mode targets with no PR: best-effort from the branch name / diff, else proceed with passed/defaults.

Step 4 — Choose <base>

• git -C <qvac> fetch origin
• Default: <base> = origin/main. The optimization is layered as the sole delta (fabric overlay REF / cherry-picked addon commits per Step 2), so the baseline stays clean regardless of base — this is the case for fabric PRs (target in tetherto/qvac-fabric-llm.cpp) and standard addon PRs alike.
• Exception — an existing qvac feature branch the target builds on: a branch-mode target that names a qvac branch, or an optimization explicitly stacked on another unmerged qvac feature branch → set <base> = that branch so the A/B builds on the same foundation.

The mmproj=both benchmark harness may not live on <base> yet — until it merges to main it lives on test/QVAC-21257-mmproj-gpu-benchmark-changes (the harness + addon instrumentation grafted cleanly onto main). Step 5 brings it in once the worktree exists. (Do not use the older feat/QVAC-21257-mmproj-gpu-config — it is ~100 commits behind main and does not merge cleanly.)

Step 5 — Create the isolated worktree + bring in the harness

Create the worktree from <qvac> (never the read-only session checkout or other in-use worktrees):

git -C <qvac> worktree add <worktree> -b test/<slug> <base>

(drop -b if test/<slug> already exists.) All later edits, build, and push happen inside <worktree>.

Bring in the mmproj=both harness. Check whether it is already present: git -C <worktree> show HEAD:packages/llm-llamacpp/benchmarks/vlm-benchmark/config.cjs contains mmprojGpu/mmprojModel and benchmarks/vlm-benchmark/match-devices.js exists.

• Present (<base> already carries it — e.g. once the harness has merged to main) → nothing to do.
• Absent → merge the grafted harness branch in — it sits directly on main, so this is a clean merge (no conflicts): git -C <worktree> merge --no-edit origin/test/QVAC-21257-mmproj-gpu-benchmark-changes It brings the mmproj=both axis (mmprojGpu/mmprojModel, legsFor, per-row vision_ms), the addon mmproj-use-gpu key + VisionEncodeMs instrumentation (which vision_ms depends on), match-devices.js, and the matrix_mmproj_gpu input + conditional targeted-android prebuild in benchmark-vlm-model-comparison.yml. (Never merge the old feat/QVAC-21257-mmproj-gpu-config: it trails main by ~100 commits, its addon C++ conflicts with main's ContextSlideOutcome refactor, and its matrix-s25/android_flagship_models wiring predates main's matrix_mobile/device_model.)

Two refs, one delta. The A/B is two refs that share identical harness config and differ only by the optimization. Build them in this order (Step 6 gives the exact edits):

1. baseline = test/<slug>-base = <base> (+ harness) + the harness/config edits, no optimization.
2. optimized = test/<slug> = the baseline plus the optimization only — fabric: point the self-contained packages/llm-llamacpp/vcpkg/ports/qvac-fabric overlay-port REF at the fabric commit with a computed SHA512 (Step 6); addon: cherry-pick / merge the PR/branch's addon commits on top.

(The QVAC-21297 run named these …-s25-stock / …-s25.)

Step 6 — Adjust the harness (notify the user, no approval)

Inside the worktree, make the edits and notify the user of each (file + what changed) — no approval needed, since it's an isolated branch/worktree:

• Optimization — apply the optimization per Step 2 (the sole delta between baseline and optimized):
  • Fabric — use a self-contained overlay-port at packages/llm-llamacpp/vcpkg/ports/qvac-fabric/. Copy portfile.cmake + vcpkg.json + android-vulkan-version.cmake from the registry port tetherto/qvac-registry-vcpkg:ports/qvac-fabric/ into that dir — for a Mali / Android-Vulkan target the port MUST carry android-vulkan-version.cmake + its Vulkan-C++-header provisioning (the port owns the Android vulkan.hpp, so the build works even on a fabric commit that predates a fabric-side Android Vulkan build fix — this is why the baseline builds cleanly). Add "overlay-ports": ["vcpkg/ports"] to packages/llm-llamacpp/vcpkg-configuration.json — main has no overlay-ports key, so add it as the first field (the path is relative to that file's dir → packages/llm-llamacpp/vcpkg/ports). In the portfile, change the registry's REF v${VERSION} to the fabric commit and set SHA512 to the hash computed directly (no CI round-trip; see SKILL-fabric-release.md Phase A). The fabric repo is private, so the archive fetch needs a token. Use curl on the /archive/<ref>.tar.gz URL — not gh api …/tarball/<ref>: the /tarball/ endpoint returns a differently-named top-level dir (owner-repo-shortsha vs repo-fullsha) → a different SHA512 that won't match what vcpkg fetches (and gh api can't write to a file without a redirect anyway). Commit-SHA /archive/ tarballs are byte-stable, so the computed hash equals vcpkg's fetch (validate once against a SHA512 already pinned in a known port):

    curl -fsSL -H "Authorization: Bearer $GH_TOKEN" https://github.com/tetherto/qvac-fabric-llm.cpp/archive/<ref>.tar.gz -o /tmp/f.tgz
    openssl dgst -sha512 /tmp/f.tgz     # or: vcpkg hash /tmp/f.tgz

    vcpkg_from_github(REPO tetherto/qvac-fabric-llm.cpp REF <fabric-commit> SHA512 <computed-sha512>)

    The baseline ref pins the pre-change fabric commit; the optimized ref pins the PR/branch commit — the overlay REF+SHA512 is the only difference between them. (Fallback: if the computed hash is ever wrong, vcpkg prints the expected hash on the first failed fetch.)
  • Addon — cherry-pick / merge the PR/branch's addon commits onto the optimized ref only.
• Model (--model, as reconciled in Step 3) — where it's set depends on the mmproj axis:
  • --mmproj both (mmproj-compare) → the run uses config.mmprojModel, read directly by the compare path (it does not honor matrix_models). Set mmprojModel in config.cjs on the test/ ref (e.g. mmprojModel: GEMMA4_Q4 for gemma4-q4 / Gemma-4-E2B). If the model isn't defined yet, add a {label, name, ctx_size, llm: hf(...), mmproj: hf(...)} literal (match the blob repo/sha/file to the registry/HF source) and point mmprojModel at it.
  • --mmproj gpu (two-models) → pass the model(s) via -f matrix_models=… at dispatch (forwarded to the phone as QVAC_VLM_MODELS); no branch edit.
• Device / preset / samples / mode are DISPATCH INPUTS — not harness edits. Main's benchmark forwards them to the phone via the matrix-mobile job's device_env channel (QVAC_VLM_PRESET ← matrix_preset, QVAC_VLM_SAMPLES ← matrix_samples, QVAC_VLM_MODE ← matrix_mode, and device_model/ device_manufacturer from the matrix_mobile token). Pass them as -f flags in Step 8 — do not edit the branch for these. The --device token fixes the GPU backend (s25→Adreno/OpenCL, pixel9→Mali/Vulkan) → which drives the Step 9 routing-proof grep and which opts are even relevant.
• mmproj (--mmproj) — the one run knob that IS a test-branch config edit for mobile: set mmprojGpu: 'both'|'gpu' in config.cjs (mobile reads config.mmprojGpu; QVAC_VLM_MMPROJ_GPU is desktop-only and not in device_env). both also reads config.mmprojModel (see the Model bullet).

Commit these harness edits on test/<slug> — this commit is the baseline state. Branch the baseline ref at it: git -C <worktree> branch test/<slug>-base. Then apply the optimization on test/<slug> (fabric: set the overlay-port REF+computed SHA512 to the optimized fabric commit; addon: cherry-pick / merge the PR's addon commits) and commit. The two refs now differ by exactly the optimization — nothing else.

Step 7 — Push the two refs (the one approval gate)

Notify the user of the exact pushes and ask for approval — noting they target separate test/<slug> refs (never main), so it's safe. Push both the optimized and baseline refs from Step 5:

git -C <worktree> push -u origin test/<slug>
git -C <worktree> push -u origin test/<slug>-base

Push only on approval.