Qv Mobile Test Dispatch

mobile-test-dispatch

Mobile integration tests run on AWS Device Farm, which is billed per device minute. They do not run automatically on PRs — someone dispatches them by hand, choosing one platform, the device(s), and usually a test filter.

The part that goes wrong is which binary ends up on the phone. A dispatch does not compile the addon; it installs a prebuilt one. Get that wrong and the run is green against code nobody changed.

Canonical reference: docs/ci/MOBILE-ON-DEMAND.md. Read it once per session before answering detailed questions; this skill is the operating procedure, that doc is the source of truth.

When to use this skill

• "Run mobile tests for <addon>"
• "Test my native change on a device / on a phone"
• "Why did my mobile run test the wrong build?"
• "How do I get a run id?"

Safety rules

• Device Farm costs money. Never dispatch the full suite to explore. Once a first run has been done, pass a tests filter and the smallest device set that answers the question.
• The exception is a first run on an addon, which is deliberately the full matrix: every supported device, every test. See Step 3b. Narrow only after it is green.
• llm-llamacpp is sharded (7 Android groups, 13 iOS groups) and an empty tests filter fans each group out as its own Device Farm run — multiplied by the device list. A full first run on a sharded addon is legitimate but expensive, so say what it will cost before dispatching it; outside a first run, always filter for LLM.
• One platform per dispatch. Android and iOS are separate runs.
• A second dispatch of the same workflow on the same branch cancels the first. To cover both platforms, either wait, or use different addons in parallel.
• Never dispatch on someone's behalf without telling them it bills Device Farm.

Step 1 — decide which binary should be tested

prebuild_run_id and package are mutually exclusive — setting both fails with a message telling you to clear one.

The input is named package on most addons but package_spec on asr-ggml, audiogen-ggml, tts-ggml. prebuild_run_id is the same everywhere.

Step 2 — get a run id (only for the prebuild_run_id route)

First: the PR must have built prebuilds at all. The prebuild stage is label-gated by ci-router — it runs only when the PR carries prebuilds, run-desktop-addon-tests, or run-mobile-addon-tests. With none of those there is no bundle and no run id. Add the prebuilds label and let CI re-run.

Then: open the PR's Checks tab, click the run that built the prebuilds, and take the number at the end of its URL.

Do not filter by the addon's own workflow name. Which workflow built the bundle varies — on-pr-nx.yml for most addons, on-pr-<addon>.yml for some, on-merge-nx.yml for a branch build. Scope by the PR's head commit:

PKG=llm-llamacpp   # the package directory name, i.e. packages/<PKG>
PR=1234

SHA=$(gh pr view "$PR" --repo tetherto/qvac --json headRefOid --jq .headRefOid)
for rid in $(gh api "repos/tetherto/qvac/actions/runs?head_sha=$SHA&per_page=100" \
               --jq '.workflow_runs[].id'); do
  gh api "repos/tetherto/qvac/actions/runs/$rid/artifacts?per_page=100" \
    --jq ".artifacts[]|select(.name==\"prebuilds-$PKG\" and .expired==false)|.name" \
    2>/dev/null | grep -q . && { echo "$rid"; break; }
done

# An empty result must not be dispatched: prebuild_run_id="" is the unchanged
# path and quietly resolves @latest, which is the failure this route closes.

Nothing printed means either the label is missing, or — on the nx path — that run only built the addons it considered affected and yours was not one. The dispatch failure message lists which addons a run did build.

Step 3 — pick a valid test filter

tests is a mocha --grep over runner names, not file names. A name that matches nothing is rejected up front by validate-devices, for free, with the list of valid names — so a wrong guess costs nothing but a round trip.

Read the names from the same source validate-devices uses:

# sharded addons (llm-llamacpp, diffusion-cpp, tts-ggml, audiogen-ggml, vla, ...)
jq -r '(.android//{})|[..|strings]|unique|.[]' packages/<PKG>/test/mobile/test-groups.json

# single-spec addons
grep -oE '\brun[A-Z][A-Za-z0-9_]*' packages/<PKG>/test/mobile/integration.auto.cjs | sort -u

If a name is rejected on device with [prestage] FATAL: tests grep /<name>/ matched no known runner, it is in neither the addon's test-groups.json nor its integration.auto.cjs — i.e. a typo. Take a name from the commands above. (That FATAL used to fire for valid runners too, because the prestage generator kept its own list; readKnownRunners() now reads test-groups.json directly.)

Step 3b — which devices to run on

A first run on an addon covers every supported device and every test. That is what says whether the change is good. Narrow only afterwards, when re-running a known failure or iterating on one test.

Not supported — these will schedule and bill, but a failure on one is not acted on: Pixel 8 and older (below the targeted floor) and Pixel 10 (not adopted). Any other fleet device can be added deliberately, e.g. to reproduce a report on specific hardware; say why when you do.

Google Pixel 9 and Google Pixel 9 Pro are different fleet models under the default EQUALS operator. The supported one is the Pro.

Step 4 — dispatch

First run — full matrix, one dispatch per platform, tests left empty. Run them in sequence, not back to back: the concurrency group is keyed on workflow and ref and does NOT include the platform, so dispatching iOS while Android is still running cancels Android. Wait for the first to finish, then fire the second.

gh workflow run integration-mobile-test-<addon>.yml --repo tetherto/qvac --ref <branch> \
  -f platform=Android \
  -f devices_custom="Google Pixel 9 Pro, Samsung Galaxy S25 Ultra, Samsung Galaxy S26 Ultra" \
  -f device_model_operator=EQUALS \
  -f prebuild_run_id=<run id>

# iOS — only after the Android run finishes, or it cancels it
gh workflow run integration-mobile-test-<addon>.yml --repo tetherto/qvac --ref <branch> \
  -f platform=iOS \
  -f devices_custom="Apple iPhone 16 Pro, Apple iPhone 17 Pro" \
  -f device_model_operator=EQUALS \
  -f prebuild_run_id=<run id>

Report a first run as complete only when both platforms actually completed. A cancelled Android leg is not a pass, and on llm-llamacpp it also discards a seed-models step budgeted at up to 120 minutes.

Follow-up — one device, one test, after something fails:

gh workflow run integration-mobile-test-<addon>.yml --repo tetherto/qvac --ref <branch> \
  -f platform=Android \
  -f devices_custom="Samsung Galaxy S26 Ultra" \
  -f device_model_operator=EQUALS \
  -f tests=<runnerName> \
  -f prebuild_run_id=<run id>

• devices_custom takes a comma-separated list and overrides the device dropdown. Names are full fleet names (Google Pixel 9 Pro, Apple iPhone 16 Pro).
• device_model_operator=EQUALS bills exactly that model; CONTAINS may pick a different variant.
• ref selects the JS harness, tests and app — not the native binary. It and the prebuild source are deliberately independent.