Night Market Validation And QA

How to add tests

Layout and isolation

• Each plugin owns plugins/<plugin>/tests/ with unit/, integration/ subdirectories and its own conftest.py and pytest config in the plugin's pyproject.toml.
• Root pyproject.toml sets norecursedirs = ["plugins/*", ...]. Plugin tests MUST run per-plugin. Running them from the repo root causes ImportPathMismatchError from duplicate module names (the root conftest.py documents this).

Commands

# Single test file (fastest loop)
cd plugins/imbue
uv run pytest tests/unit/test_deferred_capture.py -x -q

# Full suite for one plugin
cd plugins/<plugin>
uv run python -m pytest tests/ --tb=short -q
# or, where the plugin has a Makefile target:
make -C plugins/<plugin> test

# Everything (what `make test` at root does)
./scripts/run-plugin-tests.sh --all

# Only plugins with staged changes
./scripts/run-plugin-tests.sh --changed

# Mutation testing for one plugin, locally
cd plugins/<plugin>
uv run mutmut run --paths-to-mutate=scripts/,src/ --tests-dir=tests/

Markers

Root pytest runs with --strict-markers, so only registered markers are legal. Registered in root pyproject.toml: unit, integration, e2e, slow, network, plugin, skill, hook, command, bdd, benchmark (skip benchmarks in CI with -m "not benchmark").

Checklist for new code

• Failing test written and observed RED before implementation (Iron Law).
• Test would fail if the fix were reverted (revert-test standard).
• Correct marker applied and test placed in unit/ or integration/ accordingly.
• Plugin suite passes locally with its coverage flag: uv run python -m pytest tests/ --cov-fail-under=<threshold>.
• New skill files have a test_skill_<name>.py structural test.

Failure-mode detection: silent failure in scanners

The most recurrent bug class in this repo is the swallowed error. In scanner-shaped code (anything that walks files and reports findings), the signature is except-continue: a try/except that skips a file on malformed input and reports success on the remainder. The scanner looks healthy while ignoring exactly the inputs most likely to be broken.

The house convention since commit 666171c3 (issue #575): a scanner that cannot process an input emits an ADVISORY finding for it instead of skipping silently. Examples from that commit:

• check_hook_modernization.py emits findings on malformed hooks.json, SyntaxError, or OSError instead of dropping the file.
• pensive harden/scanner.py appends an ADVISORY finding for unreadable files under --strict.
• minister dora_metrics warns and sets a partial flag on malformed tag lines instead of silently classifying the repo Elite.

When reviewing or writing scanner code, grep for the pattern:

rg -n "except .*:\s*$" -A2 plugins/<p>/scripts/ | rg -B1 "continue|pass"

Any hit that discards an error without emitting a finding or an inline "why it is safe to discard" comment violates Constitution rule 10 (errors are not optional).

When NOT to use

• Running the suites, lint, release, or publish mechanics: use night-market-operations (command anatomy lives there).
• Classifying and gating a change, PR review flow, non-negotiables: use night-market-change-control.
• A test is failing and you need triage for a known repo failure mode: use night-market-debugging-playbook.
• History of why a guard exists (incident narratives, reverts): use night-market-failure-archaeology.
• Setting up pytest config or fixtures for a new plugin: Skill(leyline:pytest-config) has the templates, and Skill(leyline:testing-quality-standards) the anti-pattern catalog.

Exit Criteria

• Any completion claim made under this skill cites numbered evidence ([E1]...) with commands and captured output, and carries a COMPLETE, PARTIAL, or BLOCKED status.
• Every new test added in the session fails when its guarded fix is reverted (demonstrated, not assumed).
• New plugin Python code landed with a test observed failing first, and the plugin suite passes at or above its [tool.nightmarket] coverage_threshold.
• Any new skill file has a passing test_skill_<name>.py structural test.
• No scanner code added or reviewed in the session contains an except-continue that drops input without an ADVISORY finding or a stated reason.

Provenance and maintenance

Compiled 2026-07-02 against repo v1.9.15, branch discussions-fix-1.9.14. Volatile facts and how to re-verify them:

• Root coverage floor: rg -n "fail_under" pyproject.toml (85 as of 2026-07-02).
• Per-plugin thresholds: rg -n "coverage_threshold" plugins/*/pyproject.toml (90 everywhere except gauntlet at 85, as of 2026-07-02).
• Threshold parser behavior: read the awk block in scripts/run-plugin-tests.sh (search tool.nightmarket).
• Mutation cadence and plugin list: rg -n "cron|matrix" .github/workflows/mutation-testing.yml.
• Quality gate numbers: cat .claude/quality_gates.json.
• Golden guards still present: ls plugins/leyline/tests/test_python39_compat.py plugins/gauntlet/tests/unit/test_challenges.py plugins/herald/tests/unit/test_double_shot_latte.py plugins/abstract/tests/hooks/test_hook_io.py.
• Cited commits: git log --oneline -1 <hash> for 29081fda, a94240e2, f1cbbcf1, 30e58586, 42f7ce84, 666171c3, 268cff89.
• Marker list: rg -n -A12 "^markers" pyproject.toml.