Testing

Testing Patterns & Conventions

For VS Code extension E2E fixtures, helpers API, and warm-session rules, see /e2e-vscode-testing.

Closure evidence in CI

Routine CI runs focused guard/validator commands and the normal test lanes once, including commands also cited by historical proof records. It does not rerun commands merely because a record cites them or require current passing/skipped totals to match old transcripts. Adding a test does not require updating closure counts or pins. Recorded transcripts remain internally checked historical evidence; they are not a claim that the current checkout was replayed.

There is no repository-side roadmap, closure register or mutation-replay lane: the program DAG and its decision records are owned by the TAMA controller database, and routine CI is the only executable evidence lane. Do not add a test that loads a repository DAG, ledger or charter tree.

Never use commit SHAs, hashes or commit URLs as durable proof in source, tests, CI, documentation or charters. Squash merges and rebases replace those identities. If historical context is useful, record only the commit or PR landing title (first line) and ISO date. These references are descriptive, not acceptance evidence: check current code and behavior without requiring a historical Git object, ancestry or matching history text. Remove such checks rather than refreshing their pins. Carry this rule into all new or revised charters; it supersedes older requirements. Runtime checkout identities and real dependency integrity pins retain their operational purpose.

Server Cleanup

Always kill dev/preview servers or other long-running test processes when done — stale servers interfere with subsequent runs (e.g., Playwright's reuseExistingServer: true uses old builds).

# After finishing with a server, kill it
# If started in background, use the process ID or port:
kill $(lsof -t -i:4173)   # Unix
taskkill //F //PID <pid>   # Windows

# Or if using pnpm/npm scripts, Ctrl+C the process

Test Output Best Practices

Redirect output to a temp file, then grep — avoids re-running expensive builds:

# Good: capture once, search multiple times
pnpm exec playwright test --project=preview 2>&1 | tee /tmp/e2e-output.log
# Then search as needed:
grep -i "fail\|error" /tmp/e2e-output.log

# Bad: re-running the full test suite each time you need different output
pnpm exec playwright test --project=preview 2>&1 | grep "fail"
pnpm exec playwright test --project=preview 2>&1 | grep "error"  # wasteful re-run

TypeScript Test Patterns

Test locations: Unit tests co-located as *.spec.ts next to source. Type tests in packages/types/ use vitest --typecheck.

Sourcemap testing (see macros.map.spec.ts):

const { s, source, result } = processMacrosForSourcemap(code);
const map = s.generateMap({ source: "test.vue" });

Type testing best practices (packages/types/):

• Use a positive assertion for the changed inference contract. Add a @ts-expect-error negative assertion when it discriminates a plausible widening to any/unknown/never or another public type-boundary regression that existing coverage does not already catch.

it("type is correctly inferred", () => {
  type Result = SomeTypeHelper<Input>;

  // Positive assertion - type matches expected
  assertType<Result>({} as ExpectedType);
  assertType<ExpectedType>({} as Result);

  // @ts-expect-error - Result is not any/unknown/never
  assertType<{ unrelated: true }>({} as Result);
});

Rust Test Patterns

Test File Organization

When a Rust source file's inline #[cfg(test)] mod tests block exceeds ~400 lines, extract tests to a separate sibling file. Two patterns:

For standalone files (e.g., analysis.rs):

// In analysis.rs — replace the inline #[cfg(test)] mod tests { ... } block:
#[cfg(test)]
#[path = "analysis_tests.rs"]
mod analysis_tests;

For mod.rs files (e.g., ide/template/mod.rs):

// In mod.rs — loads tests.rs from the same directory:
#[cfg(test)]
mod tests;

Extracted file contains module contents directly — use super::*;, helpers, and #[test] fns. No wrapping mod tests { } block.

TDD Workflow

Behavioral code changes use TDD. Documentation, generated projections, formatting, and mechanical metadata changes use their owning freshness/validation evidence unless they also change executable behavior.

1. Name the plausible regression or contract boundary and confirm existing coverage does not already discriminate it
2. Write or extend the smallest test and observe it fail before the production change
3. Implement minimum code to pass
4. Run relevant tests, verify pass
5. Refactor while keeping tests green

Durable test vocabulary

Test file/module/test names, comments, fixtures, snapshots, assertion messages, and guard diagnostics describe the lasting behavior or regression boundary. They never name an architecture program/revision, roadmap/DAG, node/block/train identifier, plan phase/stage, implementation sequence, cutover stage, or deletion history. A test for work coordinated by CCA1, for example, names the capability or failing behavior and contains no CCA1 reference.

A comment may supplement that durable explanation with a GitHub issue only when the issue records a specific independently reported product defect and is outside the DAG-controlled issue mappings. Never cite a DAG-managed issue, PR, node, charter, or ledger row in code or tests; the DAG coordinates delivery and is not the defect contract.

Test Economy

Tests are evidence, not a quota. At preflight, map each changed contract to the smallest sufficient proof before proposing new tests:

• Prefer an existing test, then extending or table-driving one existing test, before creating a new test or file.
• Every proposed new test must name a plausible regression or contract boundary not already discriminated by the current suite.
• Existing behavioral coverage, compiler/type/capability enforcement, static validation, canonical gates, bounded inspection, and benchmarks are valid evidence when appropriate; record a terse rationale.
• Do not add prose or formatting assertions unless those exact bytes are a public contract. Do not add implementation mirrors, duplicate permutations, or tests that merely restate the implementation.
• Negative and mutation tests are reserved for plausible critical fail-closed/correctness boundaries or reproduced defects. They are not universal companions to positive tests.
• Incremental, cancellation, stale-publication, counter, allocation, soak, and performance evidence applies only when the change touches the corresponding authority or hot path. Otherwise record it as not applicable with a terse boundary-based rationale.
• New features and bug fixes still require adequate regression evidence. Refactors must keep applicable existing coverage green; they do not earn new tests merely by changing structure.

A declared check must be bound to a lane

A committed test that no command runs is not coverage — it reads as coverage, which is worse than an absent test. A comment saying "checked by X" is not a binding; the binding is a named lane that runs X and fails when X fails. Before citing a check as evidence, find the command.

The recurring trap is a fixture no default runner reaches. Vitest does not evaluate type-level assertions, and the package builds exclude spec files, so type-only contract fixtures compile nowhere unless a dedicated TypeScript project is run explicitly. packages/component-meta/test/*.test-d.ts is bound through tsconfig.contract-tests.json to pnpm --filter @verter/component-meta run test:types, which CI's JS Build & Test lane runs after build:ts (the fixtures assert the built dist declarations as well as the source ones). A new *.test-d.ts under that package's test/ is picked up by pattern; a type contract placed anywhere else needs its own binding.

Same rule for a new lane-external command: real-provider suites, Svelte conformance, compile-fail fixtures, and proto freshness are outside the canonical nextest surface, so a green core gate is not evidence for any of them.

Deletion evidence

Retiring a mechanism, guard, or test requires naming the removed mechanism, the surviving owner of its invariant, and the preserved proof that still fails when the invariant breaks — in the same change, with the mechanism's exclusive helpers, fixtures, allowlists, module wiring, and owning-doc references removed alongside it. Report production, tests, comments, and generated/fixture data as SEPARATE measures; never pool them into one "lines removed" figure and never commit the numbers into the tree (they go stale, then get regenerated instead of re-derived). A test or guard retirement claims a maintenance and build benefit, never shipped runtime speed. Full contributor-facing text: docs/contributing/removing-a-mechanism.md.

Pinned Vue Macro Runtime Oracle

The official Vue macro baseline is generated only from the repository-pinned local @vue/compiler-sfc@3.6.0-rc.5; tests must not fetch compiler output from the network. scripts/vue-macro-runtime-oracle/oracle-lib.mjs parses compiled JavaScript and compares normalized runtime facts instead of carrier formatting. Schema v2 records compiler/profile provenance, constructor order, skipCheck, literal-safe defaults/defaultKind, and typePresent so an omitted type cannot collapse with explicit type: null. Profile fixtures cover development, production, and production custom-element output. A verter-complete-extension row records the official Unknown baseline and may be refined only by a canonical Complete Verter result.

Regenerate and verify with:

node scripts/gen-vue-macro-runtime-oracle.mjs
node scripts/gen-vue-macro-runtime-oracle.mjs --check
node --test scripts/vue-macro-runtime-oracle/oracle.test.mjs

Never hand-edit crates/verter_session/tests/fixtures/vue_macro_runtime_oracle.json; the generator owns it. The canonical gate runs both drift verification and the oracle unit tests.

End-of-change Checks

Outside the orchestration landing-train lifecycle — a local change NOT driven as a train — run the canonical Rust pair after the change, per the repo End-of-change Checks. The full workspace suite is the canonical completeness gate.

Any change driven THROUGH the landing-train lifecycle — including a single substantial train (even a one-slice train) — uses the tiered gating: during slice implementation and fix cycles, targeted runs (changed tests + affected crates + a conservative reverse-dependency closure) are ITERATION EVIDENCE ONLY, and a selector that cannot prove the affected closure MUST fall back to full-workspace coverage for that run (still iteration evidence, never landing evidence); the canonical pair runs at exactly the two lifecycle points — after the final content change on the rebased, landing-frozen train tree, and again independently at post-land confirm. Targeted success is never landing evidence, and the standalone clause above never lets a train-driven change skip the frozen-tree final gate.

The canonical gate's omitted resource defaults are independently measured. Test threads are min(12, available CPUs). Build jobs are additionally clamped by the effective memory ceiling: 12 jobs from 16 GiB, 8 jobs from 12 GiB, and 4 jobs below that, always capped by available CPUs. This keeps the documented 24-GiB host at 8 build jobs under its unchanged 12-GiB default ceiling; the measured 12-job peak was 11.60 GiB and is too close to call portable there. Explicit positive overrides remain exact. Windows uses full nextest capacity; the provider-free core has no provider-specific max-threads = 1 groups. Real-provider tests run through dedicated serial libtest jobs, while compile-fail fixtures run through standalone Cargo binaries rather than nextest test targets.

On Windows, --prepare still warm-lists every archived suite and accepts only an exact status 0. Proc-macro suites are real test suites, not filterable compiler artifacts; their direct first launch prepends the suite's listed rust-build-meta.platforms[build-platform].libdir to the already-sanitized child PATH so the Rust host DLLs resolve. Missing, unavailable, or non-absolute libdir metadata fails setup; no suite is skipped or tolerated.

The canonical full verification pass:

1. node scripts/gate.mjs — CANONICAL provider-free local Rust gate. It builds/lists the SINGLE TEST UNIVERSE once, verifies that core/tsserver/tsgo form a complete disjoint partition, then runs archive-backed Surface 1 without real-provider packages/tests or verter_svelte_conformance. It also runs the shipped-cfg lane (compile check + verter_shipped_cfg_contract nextest run under no-debug-assertions), whose receipt is equally required — a PASS means Surface 1, the wasm JS-boundary lane, and the shipped-cfg guard all passed. Use node scripts/gate.mjs --exhaustive for CI, release, complete failure diagnostics, and comparable benchmarks. See docs/contributing/gate-performance.md.
2. node scripts/provider-ci.mjs run tsserver and node scripts/provider-ci.mjs run tsgo — dedicated serial real-provider lanes. The tsserver lane requires the language-shared and TypeScript-plugin build outputs; the tsgo lane requires the pinned local engine installed by pnpm. CI owns that setup explicitly.
3. node scripts/compile-contracts.mjs — every compile-fail/pass fixture, including feature-specific variants, through standalone non-test Cargo binaries. These builds are absent from nextest discovery.
4. cargo test --no-fail-fast -p verter_svelte_conformance --features conformance-tests --lib --bin verter_svelte_conformance --test main -- --test-threads=4 — the dedicated Svelte conformance lane.
5. pnpm proto:check — proto regeneration/format freshness in the lint-style lane, never as a Rust test.
6. cargo clippy --workspace --all-targets -- -D warnings
7. cargo check --workspace --release — the only thing in the loop that compiles the REAL release profile (opt-level 3 + fat LTO); surface 1 is debug and the shipped-cfg lane uses the cheap no-debug-assertions profile. debug_assert! gates on cfg!, a RUNTIME constant, so its body still name-resolves in release: a #[cfg(debug_assertions)] helper called inside one is an E0425 in every release build (napi and wasm artifacts included) while compiling clean in debug. It is a CHECK and RUNS NO TESTS, so it cannot observe the runtime half of that class — a state mutation written inside a debug_assert! argument compiles fine and silently never executes in a shipped build. verter_shipped_cfg_contract under no-debug-assertions is what covers that half — the ONLY tests in the repo that execute with debug_assertions off — and step 1's gate runs it as its shipped-cfg lane. Mirrored in CI by the rust-build-configs job, which owns the same two commands directly.
8. cargo clippy --target wasm32-unknown-unknown -p verter_wasm -- -D warnings — host clippy cannot see target-gated code, and the wasm32-wasip1/wasip2 clippy jobs cover the SEPARATE extensions/lapce + extensions/zed manifests, not this one. Same rust-build-configs job in CI.
9. cargo fmt --all --check
10. pnpm test for TypeScript changes

Confirm cargo clippy --version reports the rust-toolchain.toml-pinned version before trusting 2–4. Clippy output from a different toolchain is not evidence about the one CI runs.

The gate runner also emits an advisory warning for each non-exempt production Rust source above 1,500 lines, formatted as path (N lines). This scan is informational only: its findings do not enter either surface analyzer, the failure accumulator, or the final gate verdict.

Gate telemetry is report-only. From immediately after mutex acquisition through advisory and teardown, the runner records stable lane-local phase durations/peaks plus the supervisor's highest same-snapshot aggregate live-forest RSS, total process count, and per-lane contributions. It emits a bounded host/tool fingerprint and paired gate-work/gate-telemetry-v1.{log,json} artifacts. complete means every applicable phase was measured and terminal handling ran; partial or watchdog-aborted runs stay partial, while a fully executed red test run may still be measurement-complete. Bounded version/help probes share a separate hard aggregate startup-reporting deadline and hard-terminate their direct child; the canonical build/test deadline starts only after startup collection settles. Failed reporting warns and makes telemetry partial. Cargo timing snapshots require either proven pre-launch absence or a changed pre/post content identity when exact-file deletion fails. These paths never select tests, add a retry/run, alter the failure accumulator, or affect exit status.

A local fail-fast cancellation marks a live shipped phase aborted and any unadmitted remainder not-run, so telemetry is partial; the coverage-aware receipt reducer independently emits FAIL and never consults telemetry. Exhaustive benchmark runs must pass --exhaustive explicitly or their wall time is deliberately truncated and not comparable.

Cargo stable HTML timings are capability-gated on the dev archive, shipped compile check, and shipped contract only, then copied immediately from the producing target's overwrite source to three distinct gate-work/cargo-timings/ files. Archive-backed Surface 1 gets no Cargo timings flag. Nextest reports count final process identities separately from parseable timing identities at total/package(crate)/binary/family levels; legacy count remains the timed-count alias. See docs/contributing/gate-performance.md for phase IDs, fingerprint fields, schema, and artifact names.

Provider prerequisites belong to provider jobs. The core gate neither builds nor probes third-party provider state. The tsserver job installs the workspace and builds exactly @verter/language-shared plus @verter/typescript-plugin before running node scripts/provider-ci.mjs run tsserver with VERTER_REQUIRE_TSSERVER=1. The tsgo job installs the pinned local engine before running its serial lane with VERTER_REQUIRE_TSGO=1. Provider runner failures are hard failures; there is no skip or freshness tolerance in the core gate.

Conformance-harness preflight — real work, before Cargo. Before the archive build, the core gate runs only the harness-owned packages/framework-conformance-harness/bin/gate-smoke.mjs typescript. The separate required BF2 lane realizes its pinned oracle offline and then runs gate-smoke.mjs vapor before exact inventory listing. Vapor calls the real exported ensureVaporRuntimePreloaded() path; TypeScript calls the real exported observeTypeScript() with a multi-file in-memory observation in the workspace domain and asserts its export plus zero relevant diagnostics. No DOM or virtual-host logic is copied into the gate. Each mode runs under process-tree supervision and succeeds only with the exact-key, mode-bound verter-harness-smoke/v1 object receipt emitted after the work completes. Core TypeScript retains the canonical gate's deadline/stall/RSS enforcement and telemetry. BF2 Vapor uses the dedicated lane's 90-minute absolute deadline and teardown without adding a stall, memory, build-job, or test-thread limit. Non-zero exit, applicable supervisor abort, signal, spawn failure, or a missing/invalid/mismatched/extra-key receipt is setup failure 127 with exact HARNESS-SMOKE FAILED [<mode>] attribution; there is no skip, warning, or tolerance. A successful oracle-cache load does not claim that DOM bootstrap or virtual TypeScript observation works.