Batch Problem Creation

Batch Problem Creation Command

Assistant Workflow

1. Parse arguments: If $ARGUMENTS contains the keyword unscrapable, run the Creating Unscrapable Problems from the Web flow (bottom of this file) instead of the normal loop. Otherwise use $ARGUMENTS as count (valid integer); default 5 if absent
2. Pre-assign problems (main context): find N problems in one pass, scrape each to a /tmp file WITHOUT reading its content (Step 1.1)
3. Per-problem subagent loop: spawn one general-purpose subagent per problem; the agent does JSON creation, tag insert, p-gen, solution, QA. Main context only sees the agent's short report (Step 1.2-1.4)
4. Finalize the whole batch (main context): pre-commit, tag sync check, consistency check (in order — see Batch Finalization)
5. Summarize: batch results + skill improvement suggestions

Why subagents: a problem costs 30-60k context tokens when done inline (scrape dumps, JSON scripts, p-gen/test output). A subagent does the same work in its own context; the orchestrator pays only ~200-500 tokens per problem (spawn prompt + report). This keeps main context near-flat for any batch size.

Step 1: Problem Creation Loop

1.1: Pre-Assign All Problems (main context, ONE pass)

GOTCHA — never hand-copy the number into the scrape. Transcribing #214 as 257 scrapes the wrong problem silently and wastes a full cycle. Chain everything off script output.

next_problem.py --take N is a waterfall: it returns N distinct problems, one per line (NUMBER TAG NAME SOURCE), virtually consuming each pick so nothing repeats. Source priority: (1) unscrapable — the UNSCRAPABLE_QUEUE todo list in unscrapable.py, drained FIRST to keep the queue short; (2) list — registered problem lists via the best-list rule; (3) new — lowest LeetCode number absent from the database entirely (TAG is none, name unknown until scraped). Non-Python (SQL/shell) numbers are never picked. No-argument mode stays single-pick list-only.

Route each manifest line by SOURCE:

Run the lookups + scrapes in ONE bash call. Scrape output goes to files — NEVER print it to main context (it is the single largest context cost; the agent reads the file instead). Print only number, tag, slug, title:

uv run python .claude/.dev/next_problem.py --take {count} | while read -r line; do
  N=${line%% *}          # first token
  SOURCE=${line##* }     # LAST token — never `read -r N TAG NAME SOURCE`: zsh does NOT
  case "$SOURCE" in      # join the remainder into the last var, so a multi-word NAME
    unscrapable)         # ("(unknown until scraped)") silently mangles SOURCE and every
      echo "QUEUE $N none $NAME unscrapable_web_flow"   # no scrape; agent fetches via doocs/curl
      continue ;;
    new|list) ;;
    *) continue ;;       # malformed line — skip rather than scrape garbage
  esac
  uv run lcpy scrape -n "$N" > "/tmp/batch_scrape_${N}.json" 2>&1 < /dev/null
  rc=$?
  if [ $rc -ne 0 ] || head -c 1 "/tmp/batch_scrape_${N}.json" | grep -qv '{'; then
    echo "QUEUE $N none SCRAPE_FAILED $SOURCE"
  else
    META=$(python3 -c "import json; d=json.load(open('/tmp/batch_scrape_${N}.json')); print(d['slug'], '|', d['title'])")
    echo "QUEUE $N none $META $SOURCE"
  fi
done

GOTCHA — zsh read word-splitting. This machine's shell is zsh, and read -r N TAG NAME SOURCE does not join the line remainder into the last variable: NAME (unknown until scraped) leaves SOURCE="until scraped) new", so a [ "$SOURCE" = "new" ] check silently skips EVERY line and the loop no-ops with zero output (2026-09-02 batch 17: 3 no-op scrape loops before root cause). Always key off the FIRST token (${line%% *}) and LAST token (${line##* }) only. If loop-based parsing misbehaves again, fall back to a small python script that reads the raw lines and parses the scrape JSONs (batch 15 precedent — worked first try).

Notes:

• A failed scrape (premium signature Error fetching problem: 'NoneType' object is not iterable, SQL Error: Problem number N not found) does NOT consume the number — handle per Error Handling below (premium → append to the unscrapable queue), then pull a replacement with another --take and scrape it
• GOTCHA — replacement pulls must exclude every number already in the batch manifest, regardless of source. Pick consumption is per-process (state is rebuilt from files each invocation), so a replacement --take called mid-batch re-issues already-picked picks — and any failures you just queued (hit 2026-09-01 batch 9: --take 2 returned already-picked 1133-1152; batch 13, a pure list batch: --take 2 returned the just-queued 1966/2021 and --take 4 returned already-picked 1964/1968). One-pass recipe: --take WIDE in a single process (wide enough to cover queue entries + manifest), filter to the wanted SOURCE, drop manifest numbers, scrape the first survivors. For unscrapable batches an alternative is taking the next unclaimed UNSCRAPABLE_QUEUE lines directly
• GOTCHA — the unscrapable source can hand you SQL problems. The non-Python screen (num not in non_python in next_problem.py) reads NON_PYTHON_PROBLEMS, so it only catches entries ALREADY filed there — a still-queued SQL tuple (1141/1142 in batch 9) flows straight into the manifest. Verify each queue entry's language BEFORE the manifest (doocs folder title / README grep for SQL); if SQL, move the tuple to NON_PYTHON_PROBLEMS and take the next unclaimed queue line. Known SQL already filed: none left queued — new SQL discoveries still surface here
• The QUEUE ... lines are the batch manifest. Keep them; each line becomes one subagent spawn. Subagent prompts for unscrapable lines get the doocs-URL instructions instead of {SCRAPE_PATH} (see Unscrapable Problems Management step 3 for the URL pattern); prompts for new lines say tag is none

1.2: Spawn One Subagent Per Problem (sequential)

For each manifest line, spawn ONE agent with subagent_type: "general-purpose" (fresh empty context — NEVER fork, it inherits the orchestrator's whole conversation and defeats the purpose).

Parallel mode (default 8 concurrent unless the user specifies a number; experiment E4): agents do NOT write tags.json5 — the tag insert moved to finalization gate 2.0 (main-side, from the agent-reported dir names). That removes the only shared-file write, so agents own disjoint files and can run concurrently (p-gen writes only its own problem dir + scoped format/check on new dirs — verified in gen.py). bake p-gen tolerates a JSON absent from tags.json5 — verified 2026-09-01. Hold the flight level by reconciling to target, not one-for-one: on EVERY turn (each report, each spawn, each user message), count running agents (ListAgents) and spawn target - running replacements at once from the manifest. One-for-one refilling lags by 1-3 min per report and lets the flight sag to 3-4 (measured, E3 batch 13) — deficit spawning is what holds it at target. Brief overshoot is fine when several notifications land in one batch. Expect SHORT transient dips even with reconcile (notification latency + silently dropped notifications are real; 3/50 dropped in batch 13) — a sag of 1-2 for one turn is normal, a sag to half target that persists means dropped notifications: run the Step 1.3 fallback check. Ceiling note: ruff/ty/p-test are CPU-bound per agent, so past ~core-count concurrency each slows without wall-time gain — if agent durations inflate without FAILs or races, the cap is CPU, not the flow (E3 measured: durations did NOT inflate at 6 on the batch-13 machine).

GOTCHA — never derive a spawn number from memory. Every spawn prompt must be filled from a MANIFEST LINE read in that same turn (awk '{print $2}' /tmp/batch_manifest.txt to list unspawned numbers). Numbering refills from recall invents problems: 2026-09-02 batch 17 spawned 6 numbers (587-597) that were not in the manifest — 2 landed on SQL (dead spawns), 4 created unplanned problems, and a real manifest line (575) was skipped until the end-of-batch reconciliation caught it. Second consecutive batch with a manifest-discipline failure (batch 16 silently dropped 3 lines). Same rule as the scrape gotcha above, one level up: hand-transcription of numbers is where batches go wrong, whether into a scrape command or a spawn prompt. When refilling, take the LOWEST unspawned manifest number(s) not yet reported — and keep a spawned/set marker (e.g. touch /tmp/batch_spawned_{N}) if multiple reports land in one turn, so the same line is never double-spawned or skipped.

GOTCHA — the manifest has GAPS; never assume contiguous numbering. The new-source pool skips numbers already in the repo, so manifest numbers are non-contiguous (batch 19: ...858, 859, 864... — 853/860-863 absent). "The next number" or "previous + 1" is wrong by construction. Enforcement: touch a spawn marker ONLY for a number that the unspawned-scan (comm of manifest numbers vs spawned markers) listed in that same turn; run the scan FIRST, take its first line, touch, then spawn. Blind neighbor-touching cost batch 19 three stray off-manifest spawns (815/853/860 — all pre-existing problems; see also the pre-existing-problem line in the prompt template).

Subagent prompt template — fill {N}, {TAG}, {SCRAPE_PATH} (/tmp/batch_scrape_{N}.json) from the manifest line, verbatim otherwise:

Create LeetCode problem #{N} in this repo. Work autonomously; report back concisely.

FIRST: read these two files completely before any other action:
- .claude/skills/problem-creation/SKILL.md
- .claude/skills/test-quality-assurance/SKILL.md
Follow them exactly — every gotcha in them is there because it cost a real cycle.

Input data: the problem is already scraped to {SCRAPE_PATH}. Do NOT re-scrape.
Read that file (it has description, examples, constraints, topics, python_code signature, image URLs).
The JSON's _tags should be { "list": ["{TAG}"] } — but do NOT run insert_tag.py; the orchestrator
inserts all tags at finalization (other agents run concurrently and would race on tags.json5).

Do, in order:
1. Build the JSON template at src/leetcode_py/cli/resources/leetcode/json/problems/{problem_name}.json
   per problem-creation.md (images, _tags: { "list": ["{TAG}"] } — if {TAG} is `none` use
   "list": [] and skip all tagging, 12+ test cases, single-quote
   playground strings, None not null). Write it via a python script that machine-verifies every
   test-case expectation with a reference implementation and asserts BEFORE json.dump — never
   hand-transcribe expectations.
2. bake p-gen -p {problem_name}  (fix the JSON, never generated files, re-run until clean)
3. Implement ONE optimal solution in the single Solution class in solution.py (design problems:
   the custom class only), ruff/ty-clean up front per problem-creation.md Batch Flow Notes.
4. Run the QA chain from test-quality-assurance.md step 2 (backup -> p-gen -f -> restore
   solution -> p-test -> cleanup). Repo-wide `bake lint` is NOT yours — the orchestrator runs
   it at finalization; but scoped lint IS yours: `uv run ruff check leetcode/{problem_name}`,
   `uv run ruff format --check leetcode/{problem_name}`, `uv run ty check leetcode/{problem_name}`
   must all be clean before you report PASS. Tests MUST pass (solution is implemented); a
   failure means bad JSON expectations or a wrong solution — fix it, do not dismiss it.
   SPIN GUARD: if p-test exceeds ~1 min on your 12-20 tiny cases, kill it — that is an
   infinite loop in the solution/helper, not slowness (check `ps` for a ~100% CPU pytest);
   find the spinning case by running cases individually and fix the root cause. Never report
   PASS off a truncated or early-ended output file.
5. Write the actual directory name you created (one line) to /tmp/batch_name_{N}.txt, so the
   orchestrator can insert the tag and re-test by real dir names — do NOT assume the scrape
   slug (they can differ, e.g. shorter names chosen to satisfy E501).

IF THE PROBLEM ALREADY EXISTS in the repo (leetcode/{problem_name}/ or its JSON): do NOT
recreate or overwrite from scratch — verify the existing work against these requirements
instead (machine-verified expectations, QA chain, scoped lint), LEAVE `_tags` in the JSON
untouched (the orchestrator owns tags), and report status PASS with a note that the problem
pre-existed.

Boundaries — you own ONLY this problem:
- Touch only: your JSON file, leetcode/{problem_name}/, /tmp scratch files (including
  /tmp/batch_name_{N}.txt)
- Do NOT: edit bakefile.py, tags.json5, other problems' files, problem_lists/*.py,
  unscrapable.py; run `bake lint`, `bake check-consistency`, pre-commit, or any repo-wide
  command; git commit
- This is a subagent task: skip the brainstorming/superpowers skill ceremony; the two skill
  files above are your complete instructions.

Report back EXACTLY this and nothing else:
- problem_name: <name>
- status: PASS | FAIL
- tests: <count> passed
- if FAIL: one paragraph — what failed, the error, what you tried
- any deviation from the skills you had to make and why

MANDATORY before gate 2.0 — spawned-vs-done reconciliation. Never start finalization while any manifest line is unaccounted for. Every spawn must leave a marker (touch /tmp/batch_spawned_{N}), and the batch is fully reported only when this is empty:

comm -23 <(sort -n <(ls /tmp/batch_spawned_* | sed 's/.*batch_spawned_//')) \
         <(sort -n <(ls /tmp/batch_name_*.txt | sed 's/.*batch_name_//;s/\.txt//'))

Non-empty output means a report never arrived (notification drops hit batches 13, 15, 16, and 18 — 2 of 50-111 lines each). Handle per the 1.3 dropped-notification fallback BEFORE proceeding: check /tmp/batch_name_{N}.txt + leetcode/<dir> existence, confirm with a scoped pytest, and respawn only if zero artifacts exist. Batch 18 stalled multiple turns waiting on two reports the user had to rule out manually — the skill had the check but not the when, which is why this is a gate now. Also prune stale markers for numbers you touched but that are NOT manifest lines (off-manifest touches poison this scan into reporting "all spawned" early).

1.3: After Each Agent Report (main context)

• PASS: sed -i '' 's/ problem: str = ".*"/ problem: str = "{problem_name}"/' bakefile.py — keeps bake p-test default pointing at the newest problem.
• MANDATORY every turn: run ListAgents and COUNT running agents, then spawn target - running from the manifest. NEVER infer the flight count from how many reports arrived this turn — notifications clump and lag (spawn latency 1-3 min), so "1 report = 1 slot free, spawn 1" under-spawns persistently and the flight decays (batch 19: sagged to 1 running by batch end; 150-problem batch stretched hours longer than needed). Count the roster, spawn the whole deficit at once.
• The agent also wrote its real dir name to /tmp/batch_name_{N}.txt — authoritative for tag insert (gate 2.0) and Step 3 re-testing; do NOT derive names from the scrape slug (they can differ)
• FAIL: log reason for the summary; decide retry once vs skip per Error Handling. Do not debug inline — a failed agent's context is gone; if retrying, spawn a fresh agent including the failure paragraph in the prompt
• GOTCHA — completion notifications can be silently dropped (3 of 50 in batch 13: the agent left the roster with no notification ever arriving). If an agent seems long-running but you suspect it finished, do NOT respawn (it would collide on the same files) — check /tmp/batch_name_{N}.txt existence + leetcode/<dir> presence, then confirm with a scoped pytest run on that dir before counting it done or scheduling a respawn
• Suppressed-output discipline (the whole point of this design): never print scrape JSON, full p-gen logs, or full test output to main context. | tail -3 or grep a count. If deeper inspection is needed, it belongs in the NEXT agent's prompt, not this conversation

1.4: What Moved Where (vs inline flow)

• Agent-side (its context, not yours): scrape reading, JSON writing + verification, p-gen cycles, solution implementation, QA chain, all gotcha application
• Orchestrator-side (main context): manifest, spawns, bakefile sed, tag insert (gate 2.0), finalization gates, summary
• Unchanged skills: problem-creation.md and test-quality-assurance.md are read BY THE AGENT — do not summarize them into the prompt beyond the pointers above; the files are the source of truth

Step 2: Batch Finalization (MANDATORY — after ALL problems)

Six gates in order. Do NOT skip any.

2.0: Tag Insert (main-side — agents no longer insert their own)

Agents run concurrently and don't touch tags.json5; insert all batch tags here. Keep the name files — the Step 3 re-test still reads them (deleting them here cost a full wasted re-test cycle in one batch: all 30 problems printed MISSING_NAMEFILE and the mapping had to be rebuilt by hand). Cleanup happens at the END of Step 3:

grep '^QUEUE' /tmp/batch_manifest.txt | while read -r _ N TAG _; do
  NAME=$(cat "/tmp/batch_name_${N}.txt")
  if [ "$TAG" != "none" ]; then
    uv run python .claude/.dev/insert_tag.py "$TAG" "$NAME"
  fi
done

• Before inserting, verify each name file exists (ls /tmp/batch_name_*.txt | wc -l equals problem count) — a missing file means that agent never reported; handle per Error Handling before proceeding
• GOTCHA — validate name-file CONTENT before insert, not just existence. An agent wrote its problem NUMBER (1180) instead of the dir name into its name file; insert_tag.py happily bisected the numeric string to a bogus position in tags.json5 and the downstream sed fix preserved that position, costing 2 extra cycles (name fix + position fix + pre-commit rerun). Guard: each name must match ^[a-z][a-z0-9_]*$ — on mismatch, look up the real dir in leetcode/ (or re-ask the agent) before running insert_tag.py. A second variant (453): the agent wrote a plausible but ABBREVIATED dir name (min_moves_...) while the real dir was minimum_moves_... — regex-clean but nonexistent. So also verify the dir and JSON actually exist (the re-test sweep would otherwise false-FAIL on the wrong name):

grep '^QUEUE' /tmp/batch_manifest.txt | while read -r _ N TAG _; do
  NAME=$(cat "/tmp/batch_name_${N}.txt")
  echo "$NAME" | grep -qE '^[a-z][a-z0-9_]*$' || { echo "BAD NAME FILE for $N: '$NAME'"; exit 1; }
  [ -d "leetcode/$NAME" ] || echo "MISSING DIR for $N: $NAME"
  [ -f "src/leetcode_py/cli/resources/leetcode/json/problems/${NAME}.json" ] || echo "MISSING JSON for $N: $NAME"
done

• Gate 2.2 (tag sync) must come out clean afterwards; if a name file's dir differs from the scrape slug, the name file wins

2.1: Pre-Commit (converts notebooks to .py)

pre-commit run -a

• nb-to-py converts every leetcode/**/playground.ipynb to playground.py and deletes the .ipynb; lint runs the full pipeline (sort_tags, check_tag_problems, gen_catalog, lint)
• Fix failures and re-run until clean
• Why first: bake check-consistency diffs playground.py, not playground.ipynb — notebooks MUST be converted before the consistency check

2.2: Tag Sync Check

uv run python .claude/.dev/update_tags.py

• Expect No changes found in any of the specified tags. — that IS the pass state (tags were synced per-problem during creation). Missing/Removed lines only appear on a dirty run
• A clean run creates NO .claude/.dev/update_tags.json — the rm no-ops. Remove the temp file if it exists after a dirty run
• Gate semantics: the gate counts only Missing lines. Pre-existing Removed lines do NOT block it — surface them to the user instead of silently resolving them
• Warning: do not resolve Removed lines on curated lists (neetcode-150, neetcode-250, blind-75, etc.) without the user's call — the _tags.list membership check is circular for batch-created problems and following it once cemented bogus tuples into curated lists. Ground truth = source-list membership by problem number
• If any Missing/Removed lines appear: read and follow @.claude/skills/update-tags.md

2.3: Consistency Check

Each run regenerates ALL problems (minutes on 1300+ trees) — plan to run it as few times as possible:

bake check-consistency > /tmp/consistency.log 2>&1; grep -E 'PASSED|FAILED|Drift' /tmp/consistency.log

• NEVER tail the live output — the Drift: box prints AFTER the summary line, so a tail window hides which file drifted and forces a full multi-minute re-run just to learn the name (batch 19: 3 full runs instead of 1). Redirect to a file and grep it.
• Fix ALL drift files in ONE pass, then re-run once. Re-running after every single fix is the expensive failure mode.
• Regenerates ALL problems from JSON into a temp dir, converts notebooks, lints, then diffs against the working tree (original leetcode/ is restored afterwards — solutions preserved)
• Expect ✅ Consistency check PASSED: all files match JSON source of truth
• On drift (Drift: leetcode/<problem>/<file>): read and follow @.claude/skills/consistency-fix.md
• Loop until PASSED. bake p-gen -f reporting success does NOT mean consistency passes — only this command passing is real

2.4: Test Case Count Check

bake check-test-cases

• Must exit 0 (no problems at or below the default threshold — same command CI runs in test-reproducibility.yml, so local defaults and CI stay in lockstep). Success Criteria says 12+ per problem, but no other gate counts cases — p-test passes with any count, consistency only diffs generated-vs-JSON (a 10-case JSON is self-consistent), and the count otherwise surfaces only in CI. Batch 189's CI failure (campus_bikes, the_maze_iii at exactly 10) was caught nowhere locally. Runs in seconds — cheap fifth gate. On failure: add machine-verified cases to the JSON template AND mirror them into the generated test_solution.py by hand (regenerating clobbers solutions — see the p-gen -f warning below)
• GOTCHA — a FAILED run can leave leetcode/ as regenerated stubs. Restoring original leetcode/... runs at the END of the pipeline; if the run aborts at its lint stage (ruff/ty error mid-batch), the restore never executes and the working tree keeps freshly generated TODO-stub solutions for EVERY problem. After any consistency failure, verify before continuing: grep -l "TODO: Implement" leetcode/*/solution.py — any hit means solutions were wiped (rewritable from session context or a /tmp copy; take the copy BEFORE the first consistency run, not after a failure). Only the PASSED run restores reliably
• GOTCHA — flat cp can silently truncate the safety snapshot. cp leetcode/*/solution.py /tmp/dir/ copied exactly 1 file in one run (glob expansion mangled, likely by the rtk hook) — a silently-empty snapshot defeats its purpose. Use a per-problem loop and verify the count:

mkdir -p /tmp/batch_solutions_$(date +%Y%m%d)
for d in leetcode/*/; do
  [ -f "$d/solution.py" ] && cp "$d/solution.py" "/tmp/batch_solutions_$(date +%Y%m%d)/$(basename "$d")_solution.py"
done
ls /tmp/batch_solutions_$(date +%Y%m%d) | wc -l   # must equal number of problems

• Warning: bake p-gen -p {problem_name} -f overwrites solution.py with the TODO stub — backup and restore around ANY regen:

cp leetcode/{problem_name}/solution.py /tmp/solution_backup.py
bake p-gen -p {problem_name} -f
cp /tmp/solution_backup.py leetcode/{problem_name}/solution.py