Rust Build Hygiene

rust-build-hygiene — launch builds that die with you and clean up after themselves

Orphaned Rust builds outlive their launcher, hold target-dir locks (stalling other builds 30+ min), burn cores, and leave tens of GB of stale target/ behind. Every instance traces to the same root cause: a build launched in a way that survives the process that started it. (cpu-reduction is the mop; this is "don't spill.")

The one rule: NEVER detach a build

A detached build reparents to PID 1 the moment its launcher exits, so it outlives the agent/session/turn, keeps holding the cargo target-dir lock, and TaskStop does NOT reap it (TaskStop kills the agent, not its background bash jobs). This is the most common orphan and the usual cause of Blocking waiting for file lock on artifact directory on the next build.

# BANNED — these orphan to PID 1 and survive TaskStop:
setsid cargo build ... &
nohup cargo build ... &
cargo build ... & disown

How to run a build correctly, by situation

Never fake-wait on a build with a detached shell + a sleep/poll loop.

The fleet, not your build, is what saturates the host

A per-build cap cannot bound N builds. Every build can be individually blameless — --profile fast, QoS-clamped, jobs = 6 — and the machine still dies, because the caps multiply instead of adding. Measured 2026-08-19: 13 concurrent agent builds, every one of them already on --profile fast, produced a 78-way oversubscription of 10 cores — load 464, 0% idle, 36% sys, and a reqwest compile that normally takes ~30s taking 28 minutes. Nothing was misconfigured. There was simply no cap on the SUM.

• make qos-global installs the global governor and is what actually bounds the fleet. It registers scripts/rustc-qos.sh as the machine-wide rustc wrapper, where it does three jobs: clamp QoS; let at most TWO builds compile at a time (NUB_BUILD_SLOTS, default 2 — every other build's first rustc waits in a first-come-first-served queue until a holder's cargo exits, dies, or goes idle for NUB_BUILD_IDLE=120s, so a cargo test running its tests or a cargo blocked on a target lock does not hold the machine); and across the compiling builds, hold one of NUB_RUSTC_LIMIT (default 6) tokens for the life of each rustc. Two builds over six tokens bounds the memory peak to ~6 big-crate compiles (~12 GiB) and keeps a second build's worth of cores busy; strict one-at-a-time was the first cut (2026-08-28) and was measured idling nine cores behind one starved compile while eight builds queued 25 minutes. It needs no cooperation from the caller — which is the point, since the measured failure was builds bypassing the launcher script. A build that queues is not stuck — after 20s it prints one rustc-qos: this build is queued … line on its cargo's stderr, and make build-status shows the holders and the queue. rust-analyzer is exempt so the editor never waits behind agent builds.
• Never blank RUSTC_WRAPPER, and never set NUB_BUILD_FG=1 from an agent. Blanking is cargo's documented "no wrapper", and it opts the build out of the global cap; scripts/rust-build.sh used to do exactly this, which is why 10 of those 13 builds were ungoverned. NUB_BUILD_FG=1 opts a build out of the QoS clamp and the build-slot queue (a bare cargo still takes rustc tokens; through rust-build.sh it blanks both wrapper keys, so out of the tokens too) — it exists for a HUMAN at a terminal whose build must not wait behind the fleet, and an agent that sets it recreates the 2026-08-19 incident. NUB_BUILD_SLOTS=0 disables only the slot layer and NUB_BUILD_SLOTS=1 restores strict one-at-a-time — both are PER-PROCESS environment knobs, read by the wrapper of the cargo that inherits them, not host-wide settings. The host-wide switch is make build-slots-off / build-slots-on (a file every wrapper checks each second, so it also releases builds already queued); it is for an emergency, and it leaves the QoS clamp and the tokens in place.
• make build-status answers "why is this machine saturated?" and "why is my build not starting?" It prints the sum no single session can see: load, which builds hold the compile slots and who is queued behind them (each tagged with its worktree), token occupancy, a STALE WRAPPER line when an older checkout's make install-dev downgraded the governor, and which builds are outside the cap. Run it before concluding your own build is slow or hung — a build whose rustc sits at Compiling for minutes with no CPU is queued, not broken. A foreground Bash call whose cargo goes silent at Compiling is the same thing: relaunching it puts the new cargo at the BACK of the queue.