Counterfact Maintenance

Counterfact Maintenance Skill

When to use this skill

Use this skill when finalizing contributor-facing changes that affect tests, diagnostics/errors, release semantics, docs, or compatibility guarantees.

Files to inspect first

• package.json (canonical scripts)
• AGENTS.md
• packages/counterfact/docs/reference.md
• packages/counterfact/docs/faq.md
• .changeset/*.md (format examples)
• packages/counterfact/test/**/* for existing patterns/fixtures

Existing conventions to follow

• Keep historical study environments under site/study/2026/experiments/environments/ excluded from Renovate; their manifests and lockfiles are reproducibility evidence, not maintained application dependencies. When changing ignorePaths, preserve the exclusions inherited from config:recommended because the array replaces inherited values.

• Use usingTemporaryFiles() for filesystem-heavy tests.

• The Astro site uses compressHTML: true to preserve HTML whitespace around inline prose across source line breaks. Keep that setting explicit when upgrading Astro, and run cd site && npm run test:whitespace for rendering or formatting changes. Check generated text before applying CSS spacing: punctuation and intentional in-word links must remain adjacent.

• Keep single-line command snippets in a focusable horizontal scroll region, with copy controls outside that region. A scrolling flex item needs min-width: 0 so long text cannot widen the whole page. Emitted CSS/HTML checks establish this contract; they do not replace browser checks of scrolling, keyboard access, or viewport overflow.

• In file-watching tests, register the expected event listener before writing the file, then await the saved promise. Close watchers in test teardown so Jest timeouts cannot skip cleanup; restore any changed working directory there as well.

• For source or tsconfig analysis in repository tooling, use the existing TypeScript parser instead of custom tokenizers or JSONC stripping. Preserve the source filename so TypeScript and TSX syntax are parsed correctly.

• Package-boundary export checks validate public subpaths across conditions; they do not replace installed-consumer tests for runtime resolution. Keep regression coverage for null exclusions and overlapping export patterns.

• Keep tests focused by subsystem (packages/counterfact/test/cli, packages/counterfact/test/server, packages/counterfact/test/typescript-generator, packages/counterfact/test/util).

• Preserve documented behavior promises (e.g., regen preserves route edits; types are regenerated).

• When terminal output displays an HTTP message, split its head and body at the first \r\n\r\n separator only: multipart bodies contain additional separators that must remain visible.

• For user-facing behavior changes: add a changeset and update docs under packages/counterfact/docs/.

• After Changesets versions workspace packages, run yarn install --mode skip-build --no-immutable so yarn.lock can match the new internal versions before immutable installation; CI enables Yarn immutability by default, so the explicit override is required in release:version.

• A push to main with no remaining changesets publishes the merged package versions automatically; a manual Release workflow dispatch is the retry and recovery path.

• Keep the npm-publish environment name, OIDC permission, and provenance setting aligned with the npm trusted-publisher configuration.

• Use OS-assigned ephemeral ports for tests that start network servers; fixed high ports can collide on shared CI hosts.

• When an OpenAPI-aware request builder gains an input kind or encoding rule, keep the route-catalog metadata, immutable builder state, required-input diagnostics, help and inspection output, wire serialization, public docs, focused unit tests, and ephemeral-port integration tests aligned. Define and test precedence whenever two builder methods compete for the same HTTP request entity.

• When a status check is required by a merge-queue ruleset, configure its workflow to run on merge_group with checks_requested; a pull_request-only workflow leaves the queue's synthetic commit without that check and eventually times out.

• When a pull-request workflow's decision depends on labels, include labeled and unlabeled activity types so a label change refreshes the required status rather than leaving a stale result.

• For public repository-history audits, record classification confidence separately from chronology confidence. Use an exact origin only when the introducing change is demonstrated; otherwise publish the earliest confirmed affected bound and label it as a bound.

• Keep audit windows, population rules, candidate dispositions, deduplication rules, immutable source identifiers, and derived totals in a checked-in manifest with an offline consistency check. Do not make a headline denominator depend only on hand-maintained page copy or a live search URL.

• For time-to-report quality cohorts, close intake early enough to give every included release the same fixed follow-up. Publish the complete candidate ledger, treat unresolved response cases as censored, make raw event counts primary when exposure units are heterogeneous, and label rate denominators as sensitivity analyses.

Black-box test boundary

Reserve test-black-box/ for implementation-unaware behavioral tests that exercise Counterfact as a complete product through user-facing interfaces. A test is black-box because it controls product inputs and observes product outputs without depending on implementation details, not because it runs in a particular process. The product execution may cross several packages without the harness knowing or selecting which packages are involved. These tests may be more sensitive to subtle regressions than they are helpful at pinpointing their source.

Gherkin feature files are the authoritative inventory of black-box behavior. Each scenario should describe a coherent developer journey and preserve every distinct user-visible claim, while equal or stronger journey coverage should replace duplicate assertions. Python under test-black-box/ is limited to pytest-bdd scenario bindings, step glue, and lifecycle support; do not add standalone pytest test functions as a second behavioral inventory. Create contracts and configurations inside the scenario's temporary project instead of relying on mutable state or shared checked-in fixtures. Reuse one generated project and server within a scenario, but never share mutable state between scenarios. Use dynamic ports, deterministic named examples, bounded polling with complete process diagnostics, and teardown that terminates every child process.

Allowed observation and control surfaces are:

• The shipped counterfact CLI's arguments, stdin, stdout, stderr, and exit status.
• HTTP requests to a server started through that CLI.
• Files generated or changed by that CLI.
• Keystrokes and terminal output from a CLI process attached to a real pseudo-terminal.

Do not put a test in test-black-box/ if it does any of the following:

• Generates or runs a standalone consumer script whose purpose is to exercise counterfact or @counterfact/* package APIs directly.
• Imports package source files, private modules, or dist paths.
• Constructs an application, server, runner, registry, loader, or client directly instead of operating the shipped product.
• Mocks implementation internals or asserts private state, internal call order, or concrete implementation topology.