Run Ferries

Ferries

• experiments/ferries/canary_ferry.py (MoE canary, TPU and GPU via CANARY_ACCELERATOR)
• experiments/ferries/daily.py

Canary is the stable, low-cost health check. Daily exercises a larger envelope with one or two explicit changes.

Shared baseline:

• data: shared nemotron_mix baseline
• default cluster: us-central1 (zone us-central1-a)
• run log: docs/experiments/daily-ferry-log.md

Daily defaults to llama_150m, sequence length 4096, batch size 512, and about 1e19 FLOPs unless its configuration says otherwise.

Inputs Before Proposing (Daily Only)

Canary runs normally do not require a proposal cycle or PR. For daily, collect:

1. Last ferry references: issue URL, PR/commit URL, W&B run URL and Iris job ID
2. Human objective for this interval: standard integration pass, or explicit regression investigation
3. Interval boundary: use "since last ferry run", not fixed wall-clock day boundaries

If objective is ambiguous, ask before editing.

Operating Policy

General

• Hard launch gate: get explicit requester approval before launching any ferry job. Only exception: the requester explicitly says to launch without asking.
• Follow the use-iris skill's job-monitoring workflow until the run reaches a terminal state (SUCCEEDED/FAILED/STOPPED); do not stop early. Full ferry monitoring often takes 4-5 hours.
• Never restart/recreate/mutate cluster without explicit human consent in-thread. Keep cluster mutation guardrails aligned with the Iris monitoring workflow, including the debug exception path.
• Use major-event updates (not spam): launch, first eval, major incident, terminal state.
• Seal each completed daily run with a pushed git tag pointing to the exact launch commit.
• Canonical run-closure PR labels: ferry, ferry-daily, ferry-log-only, ferry-sealed.
• Canonical seal-tag format (daily): ferry/daily/YYYYMMDD/<run_slug>

Canary

• Keep canary stable; only change it for explicit reliability fixes, and only when diagnosing/fixing a concrete failure mode.
• Canary launches usually do not require a PR if the script/config is unchanged.
• If canary fails, treat as urgent infrastructure/training-health triage.
• Canary is run-only by default (W&B + issue updates); no sealing tag or run-closure PR in the normal path.

Daily

• Use daily for bounded evolution, usually 1-2 knobs.
• If daily fails, debug with one bounded fix attempt, then escalate.
• Run-closure PR scope is log-only: update docs/experiments/daily-ferry-log.md, keep detailed debug/run narrative in the issue.

Workflows

Daily lane (proposal + run)

1) Build context since last ferry

Check the latest entries in docs/experiments/daily-ferry-log.md.

LAST_FERRY_SHA=<last_ferry_commit_sha>
LAST_FERRY_DATE=<YYYY-MM-DD>

git log --oneline "${LAST_FERRY_SHA}..HEAD" -- experiments/ lib/ scripts/

gh issue list \
  --label experiment \
  --search "updated:>=${LAST_FERRY_DATE}" \
  --limit 100

Treat GitHub-tagged ferry PRs/issues as source of truth. Use "since last ferry run" rather than fixed wall-clock boundaries.

2) Edit experiments/ferries/daily.py

• Keep edits bounded (typically 1-2 knobs); propose at least one intentional modification each interval.
• If no obvious change emerges from recent commits/issues/ferry history, pick a low-risk tweak (e.g. data-mix adjustment or hyperparameter change) that may improve loss at the same FLOPs budget.
• Pattern-match from the previous daily ferry; avoid high-churn rewrites.
• Update run naming for this interval (e.g. via FERRY_DATE in launch env: daily-125m-YYYY-MM-DD style).

3) Record proposal in issue and push launch commit

In the run issue, record:

• last ferry links (issue + commit + W&B/job link),
• exact config delta and rationale,
• risk level (low/medium/high),
• relaunch fallback note,
• why this run is not literally identical to the previous daily run,
• launch checklist (explicit requester approval or explicit waiver + monitoring started).

Then push the launch commit (no proposal PR by default).

4) Launch

Confirm requester approval in-thread unless they already gave explicit "launch without asking" permission.

uv run iris --cluster=marin job run --no-wait --cpu=1 --memory=2G --extra=cpu \
  -- python -m experiments.ferries.daily

After launch, capture and post to the issue:

• Iris job id (printed by iris job run, form /<user>/iris-run-job-YYYYMMDD-HHMMSS)
• cluster
• launch timestamp
• W&B link(s) when available

Optional deterministic daily rerun name:

uv run iris --cluster=marin job run --no-wait --cpu=1 --memory=2G --extra=cpu \
  -e FERRY_DATE "$(date +%Y%m%d-%H%M%S)-daily-ferry" \
  -- python -m experiments.ferries.daily