Executing Distributed System Tests

Executing Distributed-System Tests

Pairs with designing-distributed-system-tests. That skill produces a plan; this skill runs it. The two communicate only through filesystem artifacts: the plan file in and a session directory plus findings report out.

The most common failure mode this skill is built to avoid: a run that produces a green checkmark without anyone having checked that the workload, the fault, and the oracle each did their job. The "green-but-broken" checks are not optional.

Process

1. Load the plan

If a plan file path was supplied, read it. If the user described a plan in conversation, extract the scenario list. If the plan is missing oracles or per-scenario budget tiers (Smoke / Hardening / Release, per the plan template's §7 scenario fields), halt — hand back to the design skill rather than improvise. Improvising an oracle in the moment is how green-but-broken results get produced.

2. Discover the SUT toolbox

Search the repo before writing any new code. Look for:

• tools/, scripts/, bin/ — drivers, workload generators, cluster bring-up scripts
• tests/integration/, tests/stability/, tests/chaos/
• docs/runbooks/, docs/testing/, docs/stability-test-plan.md
• Makefile / justfile / cargo xtask targets that look like cluster commands
• existing CI definitions that already wire fault-injection

Record what you found in the session log under "Toolbox discovered". This is required before any scenario runs — it prevents the skill from re-inventing tools that already exist.

2b. Probe environment capability and guide install if needed

Right after toolbox discovery, before running any scenario, check that the host can actually run the toolbox you just catalogued. If the plan has an "Environment requirements" section (the design skill emits one), treat it as the spec; otherwise infer the list from the selected techniques and the SUT toolbox.

Ask the user first; do not silently probe. Before running any which / --version / docker ps checks, list the capabilities the plan needs and ask the user: "what's available in your environment, and what would you like me to skip?" Most operators already know whether they have Docker, sudo for iptables, a Go toolchain, etc. Asking up front saves a probe round, surfaces substitution options the operator may know about (e.g. "I have podman instead of Docker"), and respects their authority over their own machine. After the user answers, verify with quick probes and reconcile any gap between what they said and what's actually present.

Probe categories:

• container runtime (docker / podman + compose)
• language toolchains at the version floors the plan declares
• backend services (Postgres, MinIO / S3-compatible, message brokers)
• fault-injection facilities (iptables, tc/netem, libfaketime, dm-flakey, Toxiproxy)
• kernel features (network namespaces for asymmetric partition, cgroups for IO throttling)
• observability stack referenced by the plan

For each capability, produce a row: requirement → present? → version → source. Record the full matrix in the session log; the findings report cites it per scenario.

For missing capabilities, do not silently mark INCONCLUSIVE. Two cases:

1. Trivially installable (a package the user can apt-get install / brew install in seconds): surface the install command to the user with a one-line explanation of what it enables, and offer to proceed once they've installed (or to install for them if you have permission and the change is low-blast-radius). Do not run sudo or system-level installs without explicit user approval.

2. Non-trivial to install (requires service setup, license, admin access, or careful configuration): explain what's missing, what scenarios depend on it, and what the user would gain by adding it. Then ask whether to (a) wait while they set it up, (b) proceed and mark dependent scenarios INCONCLUSIVE, or (c) substitute a degraded approximation (and document the substitution honestly in the findings).

Either way, INCONCLUSIVE is only the right verdict after the user has been told what's missing and either declined to install it or the install is genuinely out of reach. "Tried to run and silently no-opped" remains forbidden.

3. Establish a session directory

Create:

{{session_root}}/{{plan_slug}}/{{UTC_timestamp}}/
├── logs/
├── metrics/
├── artifacts/
└── findings/

Default session_root is ./test-sessions/ in the SUT repo, or ./test-sessions/ in the current working directory if the user prefers not to write into the SUT repo. If the caller specified an output root in the request (e.g. "produce a session directory and findings report under /path/X/"), honor that path instead of the default — do not silently relocate output. Place a copy of assets/session-log-template.md at session-log.md inside this directory and fill in the header.

3b. Author mode (opt-in)

By default the execute skill runs scenarios against the SUT's existing test infrastructure plus an ephemeral sibling-package harness under {{session_dir}}/artifacts/harness/. Nothing is written into the SUT repo.

When the operator explicitly asks for author mode (typically "author the tests" or "fill the skeletons" in the prompt, or DIST_TEST_AUTHOR_MODE=1), the skill writes the scenario skeletons into the SUT repo at the Target test file paths declared in the plan's §7. This is the only sanctioned exception to the project-autonomy rule.

In author mode:

1. Pre-flight diff scan. For each scenario, check if its Target test file already exists. If yes, show the operator the existing file and ask whether to overwrite, skip, or regenerate as a sibling (<path>.new.rs etc.). Never silently overwrite operator-authored tests.
2. Write the skeleton. Drop the Skeleton from the plan to the Target test file path. Verify the AUTO-GENERATED header is present; refuse to write a skeleton missing the header. Run the SUT's formatter (cargo fmt, gofmt, black, …) on the new file.
3. Fill the TODOs. Expand the workload / faults / oracle TODO regions from the plan's prose. Use the SUT's existing test fixtures and patterns wherever possible — read at least one sibling test file in the same directory before writing, to match the codebase's idioms.
4. Compile + run. Build the SUT's test binary (e.g. cargo test --no-run -p <crate>) to catch compile errors before running. Then run the test, capture verdict + oracle execution evidence as usual.
5. Stage, do not commit. The skill MUST NOT git commit the generated tests. Surface the diff to the operator and let them review + commit. Author mode produces candidate tests, not approved tests.
6. Provenance. Every auto-written file's header carries the plan path, the scenario id, and the sha of the plan at generation time. Reviewers can diff prose changes against test changes; if the plan's prose drifts from the test, the sha mismatch surfaces.

If a scenario has no Target test file or Skeleton in the plan, author mode is impossible for it — surface that explicitly to the operator and fall back to ephemeral-harness execution for that scenario.

4. Run scenarios in plan order

Checkpoint discipline for long-running scenarios. Distributed test runs routinely involve commands that stay silent for minutes to hours: cold cargo builds, docker compose up, multi-node smoke runs, sustained workload generators. Subagent harnesses and CI runners commonly have watchdogs that kill a task after 5–10 minutes of silence on its tool stream. To keep the watchdog fed and to give the operator visible progress:

• Before starting any command expected to run longer than ~3 minutes, append a one-line "starting" entry to session-log.md (scenario id, command, expected duration). After it finishes, append the result line with elapsed time.
• For commands that take >5 minutes, prefer Bash with run_in_background: true plus periodic status pulls (tail the output file, docker compose ps, curl a health endpoint) every 60–120 seconds rather than blocking on a single foreground call.
• Never let a single foreground command exceed the harness watchdog budget. If a build or run genuinely needs that long, split it (warm a cache first, then time the actual scenario).
• Each scenario writes its own per-scenario findings file as it goes, not only at the end. Partial findings beat a missing report when a scenario times out or the agent is killed mid-run.

For each scenario:

1. Preconditions check. Cluster up cleanly, observability live, baseline metric captured, fault plane responsive.

2. Start workload using the discovered driver.

3. Inject fault per the plan schedule.

4. Capture evidence the fault landed. The plan's §7.M Nemesis + landing evidence field declares which observable signal proves the fault landed (counter, RPC timeout pattern, log marker, partition-status metric). Capture that signal, not a generic one. If the signal is absent or ambiguous, the scenario verdict is INCONCLUSIVE-fault-not-proven (see references/verdict-taxonomy.md) — never PASS.

   For non-serious scenarios that did not declare §7.M (no gated claim category), capture a best-effort generic landing signal from the appropriate row of references/fault-injection-howto.md.

5. Stop / quiesce and collect.

6. Apply oracle — read references/oracle-patterns.md for the right one if the plan didn't fully specify.

7. Record the verdict with the actual oracle execution evidence. The verdict is one of the ten states defined in references/verdict-taxonomy.md: PASS-smoke, PASS-hardening, FAIL-reproducible, FAIL-nondeterministic, INCONCLUSIVE-env, INCONCLUSIVE-oracle-too-weak, INCONCLUSIVE-fault-not-proven, PARTIAL-surface, PARTIAL-model, NOT-RUN. Apply the decision tree in that file to assign the verdict; do not free-form. Record the verdict together with the oracle execution evidence (op count consumed, anomalies found) and the §7.M nemesis landing signal — both are required for any PASS-hardening claim.

Per-arm execution for boundary and fairness scenarios. When the plan's §7.M.S block declares scenario arms (e.g., S5/api, S5/sdk, S5/export, S5/admin):

1. Run each arm as a separate scenario, in plan order. Log entries are tagged with the arm id, not the parent scenario id.
2. Apply the 10-state decision tree per arm — each arm earns its own verdict independently. Arms that the session never reaches earn NOT-RUN (the 10th state added by this iteration; see references/verdict-taxonomy.md).
3. After all arms are scored, compute the scenario-level aggregate verdict via the downgrade rule: any NOT-RUN or PARTIAL-* arm caps the scenario-level verdict at PARTIAL-surface, regardless of which arms passed.
4. Record both the per-arm verdicts and the aggregate in the findings report's Scenario results table and Surface coverage table.

Budget-tier verdict gating. PASS-* verdicts require the run to have actually met the corresponding budget tier declared in the plan, not just produced a clean oracle output:

• Run met only the smoke budget → at best PASS-smoke, never PASS-hardening.
• Run met the hardening budget → eligible for PASS-hardening.
• Run met the release budget → eligible for the release-tier verdict (currently expressed as PASS-hardening with a release annotation; a future iteration may introduce a dedicated state).

Record the budget tier actually met alongside the verdict in the session log so the findings report can confirm the verdict is defensible.