Test Quality Assurance

Test Quality Assurance Rules

CRITICAL: Follow These Steps EXACTLY - No Deviations

1. Problem Resolution

• Use active file context or user-provided problem name
• If unclear, run: uv run python -m leetcode_py.tools.check_test_cases --threshold=10 --max=1

2. Test Reproducibility Verification Process

MANDATORY 6-Step Process - Execute in Order:

# Step 1: Backup original files
cp -r leetcode/{problem_name} leetcode/{problem_name}_backup

# Step 2: Regenerate from JSON template (use bake, NOT uv run)
bake p-gen -p {problem_name} -f

# Step 3: Restore original solution ONLY
cp leetcode/{problem_name}_backup/solution.py leetcode/{problem_name}/solution.py

# Step 4: Verify linting pass (CRITICAL for CI)
bake lint

# Step 5: Verify tests pass (expected to fail if solution is incomplete)
bake p-test -p {problem_name}
# NOTE: in the batch-problem-creation flow the solution is implemented BEFORE this
# step, so tests MUST pass here. A failure = real defect: wrong expected values in
# the JSON test_cases (most common — see problem-creation.md tree gotchas) or a
# wrong solution. Debug it; do not dismiss it as "incomplete solution".
# SPIN GUARD: 12-20 tiny cases finish in well under a minute. If a pytest run
# exceeds ~1 min wall, it is an INFINITE LOOP in the solution or helper, not
# slowness — kill it (check `ps aux` for a pytest process burning ~100% CPU),
# run the parametrize cases one by one to find the spinner, fix the root cause.
# Never trust a truncated/backgrounded output file as a pass: an exit code
# observed on a run that ended early after logging only the first test is NOT
# a pass (hit with 1265 — a spinning pytest burned ~34 min and an early-truncated
# "exit 0" output masqueraded as green).

# Step 6: Cleanup
rm -rf leetcode/{problem_name}_backup

3. What NOT to Do

• NEVER edit cookiecutter templates ({{cookiecutter.problem_name}}/ files)
• NEVER use uv run python -m leetcode_py.cli.main gen - use bake p-gen instead
• NEVER modify helpers.py manually - let regeneration handle it
• NEVER skip ty verification - this is the main CI issue
• NEVER assume tests will pass - they may fail if solution is incomplete
• NEVER use null in JSON templates - use None for Python None values

4. What to Do

• ALWAYS use bake p-gen -p {problem_name} -f for regeneration
• ALWAYS verify ty passes before considering task complete
• ALWAYS restore original solution after regeneration
• ALWAYS check JSON template if ty fails (look for assert_assert_ bugs)
• ALWAYS use None not null in JSON templates for Python None values

Test Case Standards

Coverage Requirements

• Minimum 12 test cases per problem
• Edge cases: Empty inputs, single elements, boundary values
• Corner cases: Maximum/minimum constraints, duplicates, sorted arrays
• Normal cases: Mixed scenarios with varied complexity

JSON Format

• CRITICAL: Use None NOT null for Python None values in test cases
  • JSON templates use None directly: "[1, None, 2]" NOT "[1, null, 2]"
  • This ensures generated Python code passes linting (ruff/ty check for undefined name null)
• Use single quotes for Python strings: 'hello' not "hello"
• Follow existing parametrize format
• Ensure valid Python list syntax in test_cases field

Quick Commands

CLI Commands (Recommended)

# Generate enhanced problem
uv run lcpy gen -s {problem_name} -o leetcode --force

# Test specific problem
bake p-test -p {problem_name}

# Lint check
bake lint

Development Commands

# Find problems needing enhancement
uv run python -m leetcode_py.tools.check_test_cases --threshold=10

# Check all problems (no limit)
uv run python -m leetcode_py.tools.check_test_cases --threshold=10 --max=none

# Check with custom threshold
uv run python -m leetcode_py.tools.check_test_cases --threshold=12

# Generate from JSON template (uses uv run lcpy internally)
bake p-gen -p {problem_name} -f

Common Issues & Solutions

Issue: `assert_assert_missing_number` Error

Cause: JSON template has `helpersassert_name: "assert_missing_number"` but template adds `assert` prefix Solution: Change JSON to `helpers_assert_name: "missing_number"` so template generates `assert_missing_number`

Issue: ty Import Errors

Cause: Regenerated helpers.py doesn't match test imports Solution: Use `bake p-gen` (not uv run) and verify JSON template is correct

Issue: Tests Fail After Regeneration

Expected: Tests may fail if solution is incomplete (returns 0 or placeholder) Action: This is normal - focus on ty passing, not test results Exception: if the solution is already implemented (batch flow), failures are real defects — fix the JSON test cases or the solution, then re-run the 6 steps

Issue: pre-commit flags solution.py (B905 / E741 / RUF012 / N806 / ty list invariance)

Cause: bake lint during QA runs a narrower ruff config than the pre-commit hook, so zip() without strict= (B905), variable names like l (E741), mutable class-attribute defaults (RUF012), uppercase locals like MOD (N806), and list-invariance type mismatches in ty pass QA and surface only at batch finalization.

• Fix: zip(s, t) → zip(s, t, strict=True); rename l/I/O variables (left_val, etc.); class-level constant lists → tuples (BELOW_20: tuple[str, ...] = (...)) or annotate with typing.ClassVar; MOD = 1_000_000_007 → lowercase mod (N806, recurs on every modulo problem); annotate accumulators with the full return element type (result: list[TreeNode[int] | None], not list[TreeNode[int]] returned against -> list[TreeNode[int] | None] — ty rejects the narrowing under list invariance); a float('-inf') sentinel in an int-returning DP poisons derived values into float (ty invalid-return-type) — prefer a type-pure int sentinel like -1 for non-negative domains (hit with 741 Cherry Pickup); a nested dfs(node: TreeNode[int] | None, ...) whose body touches node.val/node.left/node.right passes bake lint but fails pre-commit ty with one unresolved-attribute per access — ty does not narrow through the call site when root is Optional, so add if node is None: return at the top of the DFS (hit with 988, 7 errors at once). Generated helpers add four more: RUF005 list concatenation in return expressions → unpack instead ([node.val, *left, *right], hit with 889); E731 lambda assigned to a name → rewrite as a def with return type (hit with 894); ty unresolved-attribute when calling .to_list() on a TreeNode | None element — assert all(x is not None ...) does NOT narrow a comprehension variable; filter into a new list (roots = [t for t in result if t is not None]) and assert equal length (hit with 894); ty does NOT narrow subscripts — while vals[i] is not None and even assert vals[i] is not None still yield int | None at the use site (invalid-argument-type in helpers_content); hoist to a local first (v = vals[i]; assert v is not None; ... v, hit with 428). All are fixed at the JSON level (helpers_assert_body / helpers_content) or in solution.py, never in other generated files
• Hit with 273 Integer to English Words: lookup-table lists (BELOW_20 = [...]) at class level flagged RUF012 Mutable default value for class attribute
• Hit with 552/576/629 (MOD → N806) and 652 (ty invalid-return-type on list invariance)
• E501 also fires on COMMENTS: a complexity comment listing per-method costs (# Time: get O(index), add_at_index O(index), ...) overflows col 100 — compress or split it (hit with 707 Design Linked List)
• Solution-defined doubly-linked Node classes (custom linked-list/design problems, e.g. 708 circular list, 716 MaxStack DLL) whose pointer fields are Node | None pass bake p-gen QA and bake lint but fail pre-commit ty with one unresolved-attribute/invalid-assignment error per pointer access — ty cannot narrow attribute fields, and the dead-node marker node.prev = None assignment also fails (invalid-assignment). Fix at the solution level with self-linking non-Optional fields: self.next: Node = next if next is not None else self in __init__, dead marker node.prev = node.next = node plus node.prev is not node liveness checks (hit with 708/716, 12 errors at once). Recurs on linked-list/design problems in the unscrapable queue (919, 428, 431, 510, 369)
• Prevention: write solutions ruff-clean from the start (see problem-creation.md, Batch Flow Notes section)
• Prevention (batch flow, where bake lint is orchestrator-owned and skipped in the QA chain): scoped lint is STILL REQUIRED before reporting PASS, and it must read the repo config — from the repo root run uv run ruff check leetcode/{problem_name}, uv run ruff format --check leetcode/{problem_name}, and uv run ty check leetcode/{problem_name} (uv resolves the pyproject ruff/ty settings, so these mirror what pre-commit's repo-wide run enforces). A bare ruff check outside uv run, or skipping the scoped lint entirely because the step was "not yours", lets SIM110/E501-class issues surface first at batch pre-commit and cost a fix + full pre-commit rerun (hit with 1849, whose agent skipped scoped lint and shipped a SIM110 for-loop-return that pre-commit caught)