Veomni Debug

Quick Path vs Full Protocol

Quick Path

1. Reproduce the error. Read the full traceback.
2. Check .agents/knowledge/constraints.md for known pitfalls.
3. Write a reproducer test if feasible.
4. Minimal fix — root cause only, don't touch surrounding code.
5. Verify: reproducer passes, pytest tests/<module>/ passes, no regressions across modalities.
6. Run make quality, commit. Run /veomni-review before opening the PR or pushing a substantive update, not per commit.

If not resolved in 15 min → switch to Full Protocol.

Full Protocol

Before You Start

Track the phases with whatever todo/plan tool the running agent provides:

Phase 1: Investigate <symptom>       -> in_progress
Phase 2: Pattern analysis            -> pending
Phase 3: Hypothesis & test           -> pending
Phase 4: Implement fix               -> pending
Phase 5: Knowledge capture           -> pending

Phase 1: Root Cause Investigation

1. Read the FULL error message / symptom. Don't skim. Extract 2-3 keywords.
2. Check constraints first: Read .agents/knowledge/constraints.md — many issues are known constraint violations.
3. Reproduce consistently. If you can't reproduce, you don't understand it.
4. git log --oneline -10 — what changed recently?
5. Trace data flow backward through the call stack.
6. Distributed training specifics:
   • Check if error appears on all ranks or just rank 0.
   • FSDP2: verify sharding plan matches model structure (veomni/distributed/parallel_plan.py).
   • Sequence parallel: check that attention inputs are properly split/gathered (veomni/distributed/sequence_parallel/).
   • MoE: verify expert routing and load balancing (veomni/distributed/moe/).

Phase 2: Pattern Analysis

1. Find a working example (previous commit, different config, reference implementation).

2. Compare completely — diff line by line, not skim. Include config YAML, environment vars, and launcher scripts.

3. Identify ALL differences between working and broken code.

4. Check dependencies — different transformers version? Different PyTorch version?

5. If a package version upgrade is suspected, create isolated uv environments to bisect:

   # Env A: the current default pin (the `transformers-stable` group).
   uv venv .venv-a
   VIRTUAL_ENV=.venv-a uv sync --active --extra gpu --dev
   
   # Env B: the same tree with exactly one package moved.
   uv venv .venv-b
   VIRTUAL_ENV=.venv-b uv sync --active --extra gpu --dev
   VIRTUAL_ENV=.venv-b uv pip install "<package>==<other-version>"

   --active is load-bearing. Without it uv sync runs in project mode and targets .venv/, ignoring VIRTUAL_ENV — so both commands would rebuild the main environment instead of the two you just created, which is the opposite of what this is for. (UV_PROJECT_ENVIRONMENT works too.)

   Confirm that installing the alternate version did not change other packages:

   uv pip freeze --python .venv-a/bin/python > /tmp/veomni-bisect-a.freeze
   uv pip freeze --python .venv-b/bin/python > /tmp/veomni-bisect-b.freeze
   diff -u /tmp/veomni-bisect-a.freeze /tmp/veomni-bisect-b.freeze

   Only the target package may differ. Pin or restore every non-target difference in Env B to Env A's version, then compare again before running the reproducer. If the target cannot run with that dependency set, report the compatibility conflict; a multi-package change is not a one-package bisect.

   Then run the same reproducer in both envs, each with its own env activated — the VIRTUAL_ENV= prefixes above apply only to the uv sync lines they are attached to, not to whatever you run next:

   (source .venv-a/bin/activate && <reproducer>)
   (source .venv-b/bin/activate && <reproducer>)

   If the suspect package is transformers, you need two worktrees and two venvs — one venv per worktree. They isolate different things and neither substitutes for the other: a venv isolates the installed packages, a worktree isolates the checkout. generated/ modeling lives in the checkout, so two venvs in one worktree share a single generated/ and regenerating it for Env B silently changes what Env A runs. Two worktrees without separate venvs share one transformers install, which defeats the bisect outright.

   git worktree add ../bisect-a HEAD && (cd ../bisect-a && uv venv .venv && VIRTUAL_ENV=.venv uv sync --active --extra gpu --dev)
   git worktree add ../bisect-b HEAD && (cd ../bisect-b && uv venv .venv && VIRTUAL_ENV=.venv uv sync --active --extra gpu --dev && VIRTUAL_ENV=.venv uv pip install "transformers==<other-version>")

   Regenerate generated/ inside each worktree against its own pin (make patchgen) before running the reproducer — it is produced against the pinned version, and a stale generated/ is itself a source of failures.

   Compare the package sets here too, using uv pip freeze --python with ../bisect-a/.venv/bin/python and ../bisect-b/.venv/bin/python, and reconcile non-transformers dependency version differences as above. The editable VeOmni and patchgen paths must point to their respective worktrees; normalize those corresponding paths only when comparing the freeze output, without changing either environment's editable installs. Run codegen and the reproducer from each worktree with its own environment activated:

   (cd ../bisect-a && source .venv/bin/activate && make patchgen && <reproducer>)
   (cd ../bisect-b && source .venv/bin/activate && make patchgen && <reproducer>)