Dep Validate

Dep Validate

Run the whole loop yourself, end to end — including the real-hardware stage. The human's job is to answer the judgement questions you escalate, not to run steps. This works the same whether you are local or a headless cloud-launched agent. The only step you cannot drive is the StoreKit sandbox purchase sheet (a system UI), so the device-connected host needs a sandbox Apple ID signed in. If no device is reachable at all, do not hand a human a runbook — escalate that the run must be repeated on a device-connected host (see §4).

CI in this repo (.github/workflows/ci.yml) is build + lint only, so a bump can go green yet break behavior at runtime. Adapty above all: the paywall, purchase, and restore are revenue-critical and only exercise on real hardware. Protection (DNS/VPN start), onboarding, and entitlements are the other surfaces that compile fine but can break in use — and general UI rendering / navigation is one too: an image, widget, theme, or navigation dep can blank out or misdraw a screen with no surface tag to warn you. So the full real-device smoke suite is the required gate for anything that renders on device, not a paywall-only spot-check.

This skill composes two leaf skills — do not reimplement them:

• appium — drive the live UI on a mocked simulator or a real device.
• device-log — pull the iOS share-log / crash report to inspect SDK init, payment, and tunnel errors.

1. Discover the queue

node automation/dep-validate/queue.mjs

Prints JSON: queue (open Dependabot PRs needing validation) and advisories (unscanned ecosystems). Each PR entry carries reasons, majors, highRiskPackages, and surfaces (e.g. payment, platform-bridge, protection, messaging, build-system) precomputed from the bump. Validate one PR per run; take the PR number as input for headless runs.

A queue entry means: the auto-merge workflow would hand it to a human (major bump or non-allowlist file), or it carries a high-risk package (Adapty, Firebase, WireGuard, pigeon, gradle/AGP) at any bump level. Advisories are version-drift notes only — there is no PR to check out, so you surface them, you do not run the loop on them.

2. The validation loop

Run fast to slow. Each stage gates the next: a hard failure stops the loop and you escalate with the evidence you captured. A stage that cannot run (no device, missing account id) is blocked, not passed — a blocked revenue/protection stage forces an escalation.

Stage A — checkout + bootstrap

gh pr checkout <pr> --repo blokadaorg/blokada
git submodule update --init --recursive
# iOS bumps only (Adapty / Firebase / other native deps):
# the host links Firebase/Factory/CodeScanner via SwiftPackageManager (Xcode
# resolves them at build) and embeds the Flutter module as xcframeworks built by:
make -C common build-ios

Record the branch you came from so you can return to it when done.

The dep-validate helpers may not exist on the checked-out PR branch. A Dependabot branch created before this skill merged to main has no automation/dep-validate/ scripts, so node automation/dep-validate/queue.mjs / report.mjs fail there. The helpers only need gh + the PR number, not the PR's own files — so run them from a known-good checkout, not the PR branch: discover the queue before Stage A checkout, and emit the report after you return to your own branch (or run the script via an absolute path / git show <your-branch>:automation/dep-validate/report.mjs). Once the skill is on main and the PR rebases, the scripts are present and this caveat is moot.

Stage B — unit + static (fastest, always)

make test            # full Flutter suite incl. pigeon + build_runner codegen
fvm flutter analyze  # run from common/, or: make -C common analyze

When the bump touches Adapty or common/pubspec.yaml, also run the fallback-consistency check the CI check-adapty-fallback.yml job runs:

node scripts/check-adapty-fallback-version.mjs

Gate: all green to continue. A pigeon or codegen-tool major most often surfaces here as regenerated bridge code that no longer compiles — that is a real failure, capture it. Note the break may not be in the regenerated code: codegen can succeed cleanly yet break the hand-written callers via a renamed generated symbol. Pilot #1082: pigeon 12→26 regenerated all channels fine, then make test failed with Member not found: 'CommandEvents.setup' — pigeon renamed the generated host/event-API setup method setup→setUp (CHANGELOG: "Breaking Change [dart] Changes FlutterApi setup to setUp"), so the regenerated channel.pg.dart emits setUp while the committed caller still calls .setup. Diagnose by grepping the generated symbol's new form against the call site; the fix is a one-line caller rename. The generated *.pg.dart files are gitignored (not committed), so git status won't flag the regen — reproduce by building, and confirm the file is generated (not a tracked edit) with git show origin/main:<path>.

A grouped bump bundles several independent majors — enumerate each one's migration separately, don't report one "build failed". Dependabot groups (e.g. #1082, the flutter-dependencies group, 22 updates) land as a single PR but can carry multiple unrelated majors, each with its own break and its own fix at a different altitude. #1082 bundled three: pigeon 12→26 (platform-bridge, a mechanical 1-line caller rename), adapty_flutter 3.11.4→3.17.0 (payment, needs judgment — withCustomerUserId went non-nullable String?→String, and logShowOnboarding was removed from the API, forcing an onboarding-logging decision), and i18n_extension 12→15 (test-helper only, blocks make test but no runtime surface). The escalation must classify each major on its own — a safe rename and a revenue-path migration in the same PR get different recommendations. And Dependabot cannot split a group, so the recommendation includes the structural choice: un-group so the safe bumps land separately, or do all migrations in one follow-up commit on the branch before re-running on-device validation. Verify the per-major facts against the installed package (git show/grep the new vs old API in ~/.pub-cache or via .dart_tool/package_config.json), not a paraphrased changelog — the strongest evidence is "method X present in 3.11.4 adapty.dart:418, absent in 3.17.0".

Tooling-only bumps don't touch the Flutter app, so make test / analyze is the wrong gate for them. When the diff is confined to a dev-tooling manifest (e.g. automation/appium/wdio/package*.json, a CI-only action), validate the thing that bump can actually break instead:

cd automation/appium/wdio && npm ci          # installs under the bumped lockfile
npx tsc --noEmit -p tsconfig.json            # type-check the harness
npm run test:unit                            # headless .mjs unit tests

Judge by delta, not absolute count: a major like typescript 5→6 may print pre-existing errors that have nothing to do with the bump. Re-run the identical check with the old version pinned (npx -p typescript@<old> tsc --noEmit -p tsconfig.json) on the same tree and diff the output — only new errors are the bump's fault. (The wdio harness runs no tsc in CI and executes specs via ts-node, so a type error there is informational, not a build break.) These bumps carry no app surfaces, so they stop at Stage B → classify; there is no Stage C/D.

Stage C — mocked simulator smoke

Validates UI, onboarding, and settings that do not need real DNS/VPN or StoreKit. The Mocked / FamilyMocked schemes stub NetworkExtension and the store, so this stage cannot validate Adapty purchase or protection start — that is Stage D.

# needs a dev ACCOUNT_ID, or .env.local SIX_DEV_ACCOUNT_ID / FAMILY_DEV_ACCOUNT_ID.
# run-*-mocked is an ios/ target, so invoke it with -C ios from the repo root.
make -C ios run-six-mocked ACCOUNT_ID=<dev-account>      # or run-family-mocked

Then drive it with the appium skill in simulator mode (IOS_USE_SIM=1): start make appium-explore-session IOS_USE_SIM=1, send ui.summary / ui.inspect, walk onboarding and the screens the bump could touch. See ios/SIMULATOR.md for the sim flow. Gate: app launches and the covered flows show no regression.

Stage D — real-device smoke (slowest, covers what mocks stub)

This is the gate for revenue/trust-critical bumps. Drive the physical device with the appium skill (physical mode needs elevated access — see its SKILL.md). A default make appium-explore-session IOS_DEVICE_NAME="<device>" already installs the current workspace build before driving it; use the install-only target below only when you want to install without launching a session:

make -C ios appium-install-six      # or appium-install-family (install-only)

Required first: run the whole smoke suite, not just the surface you think the bump touches. The appium-smoke.yml specs (automation/appium/wdio/src/specs/smoke/) exercise launch → onboarding → home → settings → paywall → protection → activity across the whole app, and a single dep can break a screen far from its stated surface — an image, widget, theme, or navigation library that maps to no payment/protection surface can still blank a screen the classifier would never have flagged. Run the full suite headless the way CI does — make appium-test (or the wdio runner the appium-smoke.yml workflow invokes) — and record it as real-device:smoke. A red smoke suite is a hard failure that stops the loop and forces an escalate, no matter how narrow the bump looked. Only after the full suite is green do the targeted spot-checks below add value.

Distinguish a real regression from an environment fault before you escalate. The physical smoke device is shared and can be left in a state a bump did not cause — an iOS system prompt awaiting a tap (sign-in, trust, software-update, storage), a locked screen, a stale WDA session, an expired provisioning profile. Tells: every spec fails in its before all at the same generic step (e.g. a settings/home element never appears) while the app itself launched, or the failure reproduces on a revert of the suspect commit — an identical failure on a build that no longer contains the change is the environment, not the dep. When you see that, do not revert or hold the bump: report the device fault, and re-run once the device is cleared (a human may need to dismiss the on-device prompt). Reserve the regression verdict for a failure whose signature ties to the bump (a screen the dep renders, an SDK-init error in device-log, a diff that reverting actually fixes).

Then exercise the surfaces the bump implicates, via appium — it is not a human runbook: paywall display, then purchase and restore (Adapty); and protection start (DNS/VPN). The one step you cannot drive is the StoreKit sandbox purchase sheet (a system UI), so the host needs a sandbox Apple ID signed in (Settings → App Store) for the purchase/restore leg.

Record Stage D as per-check stages, not one monolithic stage, so the report can tell a seam from a missing device. Name them real-device:smoke (the full suite, required), real-device:paywall, real-device:protection, real-device:purchase, etc. Mark the ones you drove pass/fail. Mark real-device:smoke blocked (not fail) when the failure is an environment fault per the paragraph above, so the report asks for a device clear + re-run rather than escalating the bump. If you reached the paywall but the sandbox purchase sheet stopped you, mark real-device:purchase blocked while the others stay pass — report.mjs then escalates only that seam (not "re-run on a device"). Only when no device stage ran at all does it emit the re-run escalation.

Pull logs with device-log and scan for SDK-init / payment / tunnel errors:

node automation/device/log.mjs
node automation/device/log.mjs ARTIFACT=crash   # if the app terminated

Capture screenshots (ui.screenshot) of the paywall / purchase result as report artifacts. Treat one flaky device failure as flaky: re-run the stage once before calling it a real regression.

Stage E — post-merge cross-check

After a merge, read the physical-device smoke the appium-smoke.yml workflow runs:

gh run list --workflow appium-smoke.yml --repo blokadaorg/blokada --limit 5
gh run view <run-id> --repo blokadaorg/blokada --log

Pass/fail is in the job log; failure artifacts (screenshots, XML) are under automation/appium/output/ on the run. If post-merge smoke fails, recommend rollback in your report.