Deploy Hero Change

Deploy a change to the live hero run

Use manage-hero-run for the run record, DRI, retention, and babysitting. This checklist governs a code cutover and its trial. Single-rack validation does not exercise cross-domain collectives or checkpoint recovery at production scale.

1. Agree the plan

Record the following with the user before changing the live run:

• A complete permanent handoff checkpoint step-N. Use the newest scheduled permanent checkpoint (every 6000 steps, about 26 hours at the hero's pace), or request one from the old run's training-control endpoint with the request-permanent-checkpoint header value. Only a run whose code includes that action can take the request; otherwise schedule the cutover shortly after a scheduled permanent checkpoint. Do not hand off from a temporary checkpoint: it expires three days after it is written.
• A new run ID and checkpoint tree, the unchanged W&B entity/project, and the verified parent history boundary. Follow the launcher procedure.
• A matched control window, 200 steps by default. The old run must record those steps after the handoff checkpoint before it is stopped.
• Explicit gates for loss, throughput/MFU, token drops, router metrics, and the intended effect of the change. Specify tolerances and expected directions; require no unexpected crash, retry, watchdog, or alert.
• Coverage gaps: evals, the child's own save/resume, and retries may fall outside the trial. Include these limitations in the decision.
• A schedule that leaves time for rollback during working hours, a submission owner, and the status issue/communication thread. Report kill, launch, first steps, go/no-go, and rollback transitions.

2. Land the launch record

Land the code change and finalized trigger_hero.sh values on main before cutover. Use a pristine checkout at the verified SHA. The launcher records the new run, retained checkpoint, and W&B fork boundary; its commands and recovery semantics are documented in the linked launcher procedure.

Inventory downstream reports and trackers that pin the run ID or affected metric keys. Update their selection as part of accepting the child.

3. Rehearse preflight and rollback

Use small scripts whose queries fail closed on errors or unrecognized output. Dry-run every guard against the live cluster and obtain an independent review.

Verify:

• Deploy checkout is clean at fetched main; rollback checkout is clean at the old run's recorded SHA. Preserve its original launch command.
• Handoff metadata.json exists, records the expected step, and has the intended retention. Confirm layout and checkpoint lineage.
• Child checkpoint trees are empty for initial cutover. During later recovery, preserve them and verify the newest complete child checkpoint instead.
• No competing coordinator, gang, or hero pods exist apart from the old run. Iris, Kubernetes, object-store, and W&B credentials work.
• Launch refuses a live parent or child coordinator and a dirty checkout. One operator owns submission; capture its output and verify exactly one child coordinator afterward. Resolve an uncertain submission before retrying.
• Rollback cancels the child coordinator, confirms it is terminal, and launches the old revision and run ID as IRIS_USER=marin. Verify its intended resume checkpoint first. Never create another W&B fork for rollback.
• Object-storage headroom, checked again right before requesting the handoff. Temporary checkpoints, which the hero writes hourly, go to hero-checkpoints in US-EAST-08A (100 TiB quota); permanent checkpoints go to marin-us-east-02a in US-EAST-02A, whose quota all Marin work shares. At quota CoreWeave suspends writes for the whole zone, and the hero's next save hangs without an error (#8506, 2026-09-23). Require five checkpoints of free space in 08A (about 21 TB at 4.29 TB each): the old run's newest temporary checkpoint, a restore-smoke copy, the child's temporary checkpoint plus the next one being written (the older is pruned only after the newer commits), and one spare. Require two checkpoints plus a day of recent growth in 02A for the handoff and the child's next permanent checkpoint. Read usage and quota from the Finelog storage.usage namespace; the collector runs every few hours, so if the newest collected_at is more than an hour old, get current values with python -m scripts.ops.storage.coreweave_usage --dry-run.

Distinguish a successful query with no matching jobs from a failed query. Parse Iris CSV headers and CRLF correctly; do not interpret grep -c exit status as a query result. Match pods by task identity because Kubernetes names are sanitized and truncated. Confirm restore from Loading checkpoint from and entry into the training loop.