Designing Distributed System Tests

Designing Distributed-System Tests

The default for testing distributed and stateful systems — write a few integration tests and call it done — finds a small fraction of the bugs that actually break these systems in production. This skill enforces an opinionated workflow: scope the change, generate failure-mode hypotheses that cover the categories the literature says matter most, pick techniques from a curated catalog, and emit a structured plan file that the executing-distributed-system-tests skill (or a human) can run.

Plan modes

This skill produces two shapes of plan. Decide which one applies before you start; the steps below branch on it.

• Change-scoped — the default. Use when the caller names a commit, PR, branch-diff, or feature. The plan covers what this change could regress, scoped by its blast radius.
• Project-wide — use when the caller asks for a "release-validation plan", "stability plan for the whole system", "test plan to enough coverage", "what should we be testing", or otherwise frames the request without a specific change. The plan covers what the system should be tested for, with an explicit inventory of existing tests and a gap analysis driving the new-scenario list.

If the framing is ambiguous, ask once before starting — the modes diverge enough that retrofitting one into the other wastes work.

Process

Follow these steps in order. Do not skip; the order matters because later steps depend on artifacts the earlier steps produce.

1. Scope the system

Read the project's entry points: README, AGENTS.md or CLAUDE.md, top-level docs/, any existing test-plan or runbook files. Note:

• Tenancy / isolation model
• Persistence model (what is durable, fsync contract)
• Replication / consensus protocol, quorum, leadership
• Ordering guarantee exposed to clients
• Network boundaries (which RPCs / streams)
• Retry / idempotency contract
• Observability (logs, metrics, traces) available to an oracle

Write this as a one-paragraph SUT model. If anything is ambiguous from the repo, ask the user before proceeding — do not invent guarantees.

1b. Extract claims and guarantees

A good test plan exists to falsify what the product claims. Before generating hypotheses, write down what the SUT promises its users. This is the spine the rest of the plan hangs off — every hypothesis, every scenario, every oracle should be traceable back to a claim it either confirms or refutes.

Sources to mine:

• README "guarantees" / "what we offer" sections
• API docs / reference manuals
• ARCHITECTURE / DESIGN docs (claims about consistency, durability, replication, fault tolerance)
• Public blog posts, talks, marketing material (if any)
• The code itself: function names, doc-comments on public APIs, error types (IdempotencyConflict, StaleRead, etc.) imply guarantees the system claims to enforce
• Existing test names (a test called linearizable_under_partition implies a linearizability claim under partition)

Categorise each claim:

• Safety — "the system never returns a stale read", "no acknowledged write is ever lost", "linearizable per key"
• Liveness — "every accepted operation eventually commits", "leader election completes within N seconds of crash"
• Durability — "fsync'd writes survive crash", "replicated writes survive single-AZ loss"
• Performance / SLO — "p99 append latency ≤ X ms at Y ops/s per session"
• Operational — "rolling upgrade is non-disruptive", "configuration changes are atomic"
• Idempotency / dedup — "same idempotency key never produces two committed effects"
• Isolation — "tenant A's reads never observe tenant B's writes", "no read returns data from a transaction that has not yet committed"
• Ordering — "consumers always see messages in the order the producer sent them", "every reader sees a prefix of the global log order"
• Membership — "a node that fails its liveness probe is removed from the cluster membership view within N seconds", "every joined member appears in the membership table exactly once"
• Boundary — access-boundary semantics: "tenant A's data is never reachable from tenant B on any surface", "a request scoped to namespace X never routes to namespace Y". Subsumes tenancy / authz / namespace / routing / multi-protocol; do not file those as separate categories. Triggers the §7.M.S surface-decomposition discipline (see step 3 and references/boundary-and-isolation-testing.md).
• Fairness — per-group performance and noisy-neighbor isolation: "no tenant can starve another for throughput", "one shard's load does not blow another shard's p99". Group can be tenant, shard, queue, partition, region, priority class, user, table, or workload class. Also triggers §7.M.S.

If the project does NOT explicitly document a claim that appears in the code, write it as an inferred claim and mark it as such — inferred claims are still testable, and surfacing them often catches places where the docs lie or are silent about real guarantees the implementation depends on.

When done, you should have a numbered claims list (C1, C2, …). The hypothesis-generation step (step 3) will reference these by number, the coverage matrix (template §5) tracks claim × hypothesis, and scenarios (template §7) state which claim(s) each is trying to falsify. If a hypothesis cannot be tied back to a claim, either name the missing claim explicitly or drop the hypothesis — untethered hypotheses produce ceremonial scenarios.

Missing claims are a first-class finding. During hypothesis generation (step 3) you will encounter behaviors the implementation relies on that no claim covers — Unicode normalisation policy, specific timeout windows, edge-case error semantics. List these in the plan's "Missing claims discovered" section (template §1c). Surfacing them is one of the highest-value outputs of the whole exercise: it tells the maintainer where docs and implementation have drifted apart.

2. Scope the change OR the project

Change-scoped: Identify the commit, PR, or feature under test. List every file touched and the surfaces (RPCs, on-disk formats, replication messages, public APIs) affected. Build a one-paragraph blast-radius statement.

Project-wide: No specific change. Instead, enumerate the system's externally observable surfaces (public APIs, on-disk formats, wire protocols, replication/consensus, background jobs, operational controls) and the invariants each must preserve. Declare what is in-scope and what is explicitly out-of-scope (adapters, ancillary tools, demo apps) — a project-wide plan that tries to cover everything covers nothing well.

2b. Inventory existing tests (project-wide only)

Walk the SUT's test surface: unit tests, integration tests, fault- injection / stability harnesses, smoke scripts, CI workflows, and any test-plan / runbook docs. For each notable test or harness, capture: what subsystem, what invariant it pins, and what failure modes it would catch. This becomes the left-hand column of the coverage matrix in step 4b.

Do not re-test what is already covered well. The point of the gap analysis is to surface what is NOT covered.

3. Generate failure-mode hypotheses

For each claim from step 1b, ask: under what conditions could the SUT fail to honor this claim? Each hypothesis must be tied to one or more claims by number ("could falsify C3 and C7"). Tests exist to refute claims, not to "check that things work" — a passing test should mean "this claim survived this fault", and a failing test should name the claim it falsified.

Walk the pitfall catalog. Before generating hypotheses from intuition, open references/common-distributed-systems-pitfalls.md. It lists 16 failure modes that recur across the Jepsen analyses corpus, each with a hypothesis template ready to paste-adapt. For every pitfall, decide if it applies to this SUT: y / n / maybe. Every y and most maybes become hypothesis rows. This shortcut prevents the common failure mode of plans that only test what the agent already thought of.

Generate hypotheses for each touched surface (change-scoped) or in-scope surface (project-wide) across these categories: correctness, durability, liveness, partial failure, idempotency / replay, upgrade / rollback, configuration, performance / fairness.

If a category is genuinely not applicable, say so explicitly. The act of writing "N/A because…" surfaces wrong assumptions more often than it sounds like it would.

Boundary and fairness claims trigger §7.M.S. When you encounter a claim about tenant isolation, authz, namespace, routing, multi-protocol access, compatibility across API surfaces, or per-group fairness (noisy-neighbor, queue-group, per-region), tag it with the boundary or fairness category in §1b. Both categories trigger the surface-decomposition discipline in §7.M.S of every scenario that falsifies them — see references/boundary-and-isolation-testing.md for the boundary claim matrix template and surface catalogs.

In project-wide mode the list is typically larger (the system has more surfaces than any single change). Group hypotheses by subsystem so the gap-analysis table stays readable.

4. Select techniques

Open references/catalog-index.md and find the techniques that match your hypotheses. For each technique you pick, open its reference file and write down in the plan: which hypotheses it addresses, what it would catch that other techniques would miss, the typical cost.

For scenarios that will be serious (any claim in {safety, durability, idempotency, isolation, ordering, membership}), also open the executing skill's references/oracle-patterns.md and use the "Checker picker" table at the top to pick the checker(s) matching your model and claim category. The checker choice is part of the plan, not a runtime decision.

A change usually warrants 2–4 techniques in combination. One technique is suspicious — re-check whether you've collapsed multiple distinct hypotheses into one. A project-wide plan typically reaches further across the catalog (5–7 techniques) because the surface is larger.

4b. Map coverage and identify gaps (project-wide only)

Build a table indexed by claim, not just by hypothesis. Each row: the claim (C-number), the hypothesis that would falsify it, the existing test(s) (from step 2b) that exercise it, the verdict (covered / partial / not covered), and the gap kind (no test / shallow test / oracle too weak / no fault-injection variant). Sort by claim severity × gap so the highest-leverage gaps end up at the top.

This table is the heart of the project-wide plan. It tells the maintainer where the product's claims are unverified. Without it the plan is just a wishlist.

For very large systems (50+ claims, 100+ hypotheses), split the matrix. A per-claim summary table (one row per claim with rolled- up verdict) gives the maintainer the at-a-glance view; a per- hypothesis detail table keeps the granular gap-kind information. Without the split, a single matrix where load-bearing claims appear in many rows becomes unreadable.

4c. Declare environment requirements

For each technique you picked, list the runtime dependencies the executing skill will need on the test box: container runtime (docker / podman + compose), language toolchains (Rust, Go, Node, Python at specific minima), database / object-store backends (Postgres N+, MinIO / S3-compatible), fault-injection facilities (iptables, tc/netem, libfaketime, dm-flakey, Toxiproxy), kernel features (network namespaces for asymmetric partitions, cgroups for IO throttling), observability tooling (Prometheus, OTLP collector), and any project-specific binaries.

Put this in the plan's "Environment requirements" section as a checklist with version floors where they matter. The executing skill consults this list at its environment-capability probe step and uses it to either guide the operator through install or mark dependent scenarios INCONCLUSIVE.

5. Design scenarios

For each technique, write concrete scenarios. Each scenario must specify: workload (what generator, rate, distribution, duration); faults (schedule of what is injected when); oracle (the property checked and how); observability required; and the three budget tiers (Smoke / Hardening / Release — see the field list below).

Resist "logs look fine" as an oracle. The oracle must be a machine-checkable property or a metric SLO with a defined threshold.

In project-wide mode, prioritise scenarios that fill the highest- leverage gaps from step 4b, and tag each scenario with both the hypothesis-rows it closes AND the claim(s) it tries to falsify. Long-tail "nice to have" scenarios go into template §9 (open questions / followups), not the actionable scenario list in template §7 — the plan should be actionable, not aspirational.

Every scenario name should encode the claim it targets: linearizable_per_session_under_partition, durability_survives_fsync_loss, idempotent_replay_across_restart. A test named after its claim is harder to weaken; a test named after its setup ("3-node cluster with chaos") tells you nothing about what it actually verifies.

Each scenario is an executable spec. Beyond the prose fields (Workload, Faults, Oracle, Observability, and the three budget tiers), emit two more:

• Target test file — the relative SUT path where this test will live if/when it becomes a permanent regression. Follow the SUT's test conventions: for Rust crates crates/<crate>/tests/auto/<S_id>_<slug>.rs, for Go modules <module>/<pkg>_test.go, for Python pytest tests/auto/test_<slug>.py. The auto/ subdirectory makes generated tests easy to find and review separately from hand-authored ones.
• Skeleton — a language-specific code block with imports, the test function signature, and TODO regions for the workload / faults / oracle bodies. The skeleton MUST include an AUTO-GENERATED header comment with the plan path and scenario id so a reviewer can trace any committed test back to its spec.

The skeleton is what the executing skill (in author mode) writes to the target path and then fills the TODOs from. The plan + the generated test are traceable back to each other; if the plan's prose changes, the test should be regenerated.

Fill §7.M for serious scenarios. If any claim in this scenario's Falsifies if it FAILs row belongs to {safety, durability, idempotency, isolation, ordering, membership}, the scenario is serious and must fill the §7.M sub-block in the plan template. A scenario that decomposes into §7.M.S arms (boundary / fairness) is also serious: each arm is always serious and carries its own §7.M block, regardless of claim category — the executing skill scores every arm through the §7.M checker node of the verdict decision tree. The §7.M sub-block:

• Model under test — pick from the picker in references/history-discipline.md.
• Operation history — which of the default 11 fields the recorder captures; the recording mechanism (in-process / external / server- side / combined).
• Checker — name from the executing skill's references/oracle-patterns.md "Checker picker" table at the top of that file. Or, if no checker, write the justification.
• Nemesis + landing evidence — nemesis from the executing skill's references/fault-injection-howto.md, plus the observable signal that proves the fault landed.
• Ambiguous outcomes — how the recorder treats timeouts, unknowns, retries, duplicates.
• Reduction plan — minimisation recipe + the SUT/harness/checker/ env classification step from the executing skill's references/test-case-reduction.md.

For non-serious scenarios (perf-SLO, liveness, operational), write §7.M: not applicable (no gated claim category falsified) and move on. Do not invent a model just to fill the field.

The §7d confidence statement should lean on the chain ("checker X consumed history Y under nemesis Z with landing evidence E") for every serious scenario. A serious scenario whose §7.M is partially filled cannot contribute to a hardening claim.

Fill §7.M.S for boundary and fairness scenarios. If any claim in this scenario's Falsifies if it FAILs row belongs to {boundary, fairness}, the scenario is surface-decomposition mandatory and must fill the §7.M.S sub-block in the plan template:

• Surfaces — drawn from the catalog in references/boundary-and-isolation-testing.md, or SUT-specific surfaces the catalog does not cover. Minimum three per boundary claim or written justification for fewer.
• Operations — per surface, which operations the scenario exercises.
• Adversarial inputs — confusable identifiers from the catalog in boundary-and-isolation-testing.md.
• Positive controls — what legitimate access must still succeed.
• Negative controls — what illegitimate access must be denied AND not observable in metrics / logs / side channels.
• Delayed / async paths — background jobs, retries, GC, compaction, CDC, exports.
• Observability paths — metrics, traces, audit logs that could themselves leak across the boundary.
• Scenario arms — per-arm IDs (S<n>/api, S<n>/sdk, etc.). Apply the split-into-arms rule: if the scenario spans more than 3 surfaces, more than 3 claim categories, or requires more than 1 independent oracle, split into arms with independent verdicts.

For non-boundary scenarios, write §7.M.S: not applicable (no boundary or fairness claim falsified) and skip the fields. Do not invent surfaces just to fill the field.

The §7.M.S sub-block is a sibling to §7.M (model / history / checker), not a replacement. A scenario falsifying both a consistency claim and a boundary claim fills both blocks.

Every scenario declares three budget tiers. Replace the legacy single Exit criteria field with explicit Smoke / Hardening / Release budgets per scenario:

• Smoke budget — minimum config + duration + faults + seeds required for PASS-smoke.
• Hardening budget — strictly stronger than smoke on every dimension; required for PASS-hardening.
• Release budget — long / repeated / statistical gate, OR an explicit not provided — <reason>. Revisit when: <condition>. declaration. Empty / "TBD" / "see §6b" are explicitly disallowed.

The execute skill uses the budget tier actually met as a verdict precondition; the findings report surfaces every "not provided" release budget in a dedicated Release-budget disclosures section.