CI Adhoc Test

Ad-hoc testing on a real OS/platform via CI

Some behavior is only observable on a real target platform: macOS Seatbelt / sandbox-exec / Gatekeeper / codesigning, Windows cmd.exe / --script-shell selection / .cmd/.bat resolution / Authenticode / SmartScreen, musl-vs-glibc detection, Linux-arm64, a pinned Node floor. Docker closes the Linux corners cheaply but runs Linux containers only — it is not a substitute for macOS or Windows.

The key fact: no PR is required

A GitHub Actions workflow does not need an open pull request. Trigger it on the branch:

on:
  push:
    branches: [<your-probe-branch>]  # THE trigger: runs on every push to THIS branch
    paths:                           # only when the harness itself changes
      - 'tests/<probe-name>/**'
      - '.github/workflows/<wf>.yml'
  workflow_dispatch:                 # INERT until the file lands on the default branch — see below

• Push to the branch → the probe runs. Open no PR, or close one you opened.
• Re-run with another push: git commit --allow-empty -m rerun && git push. Or re-run a finished run with gh run rerun <run-id> (gh run list --branch <branch> for the id).
• push is load-bearing; workflow_dispatch alone will NOT work for a branch-only probe. GitHub only registers a workflow_dispatch workflow if the file is on the default branch, so gh workflow run <wf>.yml --ref <branch> errors ("no workflows found") and it never appears in the Actions UI. Keep the entry for when the probe graduates to main; don't rely on it before then.
• Do NOT open a PR just to get CI. A PR signals "ready to land," which a prototype is not.
• Omit any pull_request: trigger — it ties runs to PR state, which is what you're avoiding.

The harness shape

Keep the probe self-contained under tests/<probe-name>/, mirroring the existing ones:

• A generator / runner (e.g. an SBPL profile generator + a sandbox-exec runner; a .cmd resolver harness).
• Fast unit + smoke tests asserting the enforcement and the bypass/fail-closed cases.
• A README.md (what it validates, how to reproduce locally) and a results.md (findings, plus heavy runs reproduced on demand).
• The branch-scoped workflow .github/workflows/<wf>.yml.

Mirror tests/sandbox-macos-writeconfine/ + .github/workflows/sandbox-macos-writeconfine.yml, or its Windows counterpart tests/sandbox-win-probes/ (each on its own probe branch, not main).

Keep CI lean — fast, deterministic core only: unit tests + the enforcement/bypass smoke matrix, no network, no mega-fixture. Heavy or combinatorial runs are documented in results.md and reproduced on demand. CI capacity is shared; a 22-minute-per-push probe job is already a lot.

Test every candidate FIX in one build-free run, not one per push

When the probe fails and you have several plausible repairs, do not guess-and-push: each loop costs a full build plus queue time. Write a job that tries all candidates at once against a stand-in, with no compile step — a .cmd/shell quoting question needs cmd.exe and a two-line stub, not the real binary. Minutes instead of half an hour, and the losing candidates are results you keep.

Two rules that make the output trustworthy:

• Include the CURRENT broken form as a control. It must fail. If it passes, the reproduction is wrong and every other row is meaningless — say so in the output rather than reading the winners.
• A losing alternative is a result, not a job failure. Exit non-zero only when the control passes or when NO candidate works. Otherwise the job sits permanently red over a candidate you never intended to ship, and readers learn to ignore it.