Corpus Sweep

Running a corpus sweep

A sweep installs thousands of third-party package-versions and records a verdict for each. The verdicts feed something real — for the build jail, the capability catalog that decides what actual user installs are permitted. So a wrong verdict is not a flaky test, it is a shipped defect.

The single most important thing in this skill: nearly every wrong conclusion comes from a STANDALONE REPRODUCTION THAT SILENTLY TESTED NOTHING. The harness says a package fails. You run it by hand, it passes, and you conclude the harness is wrong. It is almost always the reverse.

⛔ Before you believe any standalone reproduction

Run this checklist. Each line is a measured failure that produced a confident wrong answer.

A clean result where you expected a messy one is a signal to distrust the instrument, not to conclude. Three passes in a row after a real failure means you probably are not running the thing you think you are running.

⛔ Validate any query or filter against a KNOWN POSITIVE

A screen that returns zero hits has two explanations and you cannot tell them apart without a control. Measured: an OSV screen over 2,250 entries returned 0 MAL-* hits. That happened to be correct — but the only way to know was to re-run it against a package known to be flagged (@ctrl/tinycolor@4.1.2 → MAL-2025-47141) and confirm the instrument fires. Do this for every batch API call, grep filter, and classifier before reading a conclusion off it.

Batch APIs deserve a second control: put the known positive in the MIDDLE of a full-size batch and check it is still found at the right index. Silent truncation is real.

⛔ Malicious packages: never execute, never catalog

A package with a MAL-* advisory must never have its scripts run, on this machine or any machine.

• The PM under test may refuse it correctly — but the reference arms (npm, pnpm) have no OSV screen and typically run with --dangerously-allow-all-scripts. If a refusal verdict is computed after the oracle arms, the harness executes the malicious script itself.
• So: detect the refusal and return before any oracle arm runs. Verify the ordering in code, not by assumption — the guarantee must not be incidental to line order.
• Better still, screen the WORKLIST against OSV before the sweep starts and drop hits entirely, so the tarball is never even fetched.
• A refusal is its OWN verdict. Scoring it as a defect of the PM under test blames the tool for working correctly, and lands the package in the wrong bucket of the final report.

Sharding

Round-robin the worklist so each shard gets a mix of heavy head and cheap tail:

python3 -c "
lines=[l.strip() for l in open('worklist.txt') if l.strip()]
for i in range(3):
    open(f'shard{i}.txt','w').write('\n'.join(lines[i::3])+'\n')"

Then launch each shard as its OWN harness-tracked background task — never &, which the harness rejects because a detached job cannot report completion.

• Check fixture isolation first. Concurrent shards are only safe if each process roots its fixtures uniquely (fs.mkdtempSync). Verify before launching, not after.
• 3 shards took throughput from ~0.7 to ~2 runs/min on a 10-core box. More is not obviously better: contention makes packages whose scripts run their OWN installer fail transiently.
• Records must be per-package files so shards never write the same path.

The host will fight you

• Spotlight and Time Machine index the churn. Each cell installs a full dependency tree into a fresh $HOME, so a sweep generates millions of file events. Measured: fseventsd 60%, backupd 30%, mds + spotlightknowledged 47% — about 1.7 cores of pure overhead. Fix with touch <cache>/.metadata_never_index and tmutil addexclusion <cache>, both idempotent.
• High load with slow progress is not always your processes. Check ps -Ao pid,%cpu,comm -r before assuming contention; the answer may be an OS daemon.
• Fixture roots leak on SIGKILL. A harness that cleans up on normal exit does not clean up when you pkill it — and stop-fix-restart is the normal loop when triaging. Measured: 31 orphans, 18.9 GB, on a disk already at 98%. Sweep anything untouched for >30 min at startup; a live shard touches its root continuously so the threshold cannot catch one in use.
• Wall-clock numbers taken during a sweep are untrustworthy. Use load-independent evidence.