Nemotron Retrieval Recipes

Safe Workflow

1. Gather only context relevant to the task: recipe family, selected profile, corpus path, existing SDG/training/eval data, target stage range, artifact root, checkpoint path, execution mode, GPU IDs, and whether required secrets are configured. Never ask users to paste secret values.
2. Start with cheap checks before expensive work:
   • uv run nemotron <family> --help
   • uv run nemotron <family> <stage> --help
   • uv run nemotron <family> <stage> -c <profile> -d
   • uv run nemotron <family> run -c <profile> -d --from <stage> --to <stage>
   • run --help may omit inherited -c and -d options even though run -c default -d ... works; validate by running the dry-run when unsure.
   • In an already prepared checkout, uv run --no-sync ... --help or uv run --no-sync ... -d can avoid unexpected dependency sync during read-only checks.
3. Check prerequisites for the requested stage:
   • Repo environment: uv sync --all-extras or the smallest relevant extra if documented by the repo.
   • Stage 0 SDG: NVIDIA_API_KEY.
   • Stages 1–3 GPU work: CUDA/NVIDIA driver availability and enough VRAM.
   • Stage 4 export: the NeMo Export-Deploy container when using TensorRT. Default Nemotron 3 Embed skips this stage.
   • Stage 5 deploy: Docker plus the selected backend's image and artifact contract; default Nemotron 3 may use the checked-in vLLM image without NIM credentials. Load the family reference before requiring NGC credentials.
   • Remote execution: root env.toml profile for --run or --batch; load references/remote.md when remote scheduling, logs, or GPU placement matter.
4. Use dotlist overrides instead of editing defaults unless the user asks for reusable config changes. Keep the selected profile, artifact root, sequence length, prefixes, pooling/normalization, prompt templates, and hard-negative counts consistent across stages.
5. Avoid launching API, GPU, Docker, Slurm, NIM, or long-running jobs unless the user explicitly asked to run them. Offer or run dry-runs, config review, and small pilots first.
6. For local execution, scope requested GPU IDs with CUDA_VISIBLE_DEVICES=<ids>. For --run or --batch, configure scheduler resources such as gpus_per_node in the selected env.toml profile and let the scheduler assign devices; do not assume submit-shell CUDA_VISIBLE_DEVICES propagates remotely.
7. For multi-stage local runs, prefer uv run nemotron <family> run -c <profile> --from <stage> --to <stage>. Use default for rerank. The default run target stops at eval; export and deploy are opt-in.
8. When evaluating quality, compare against the base model on a fixed held-out evaluation set before recommending deployment. Do not substitute a standalone public-benchmark eval for the recipe's own Stage 3 evaluation.
9. For long-running SDG, prep, finetune, or eval work, start the process in a session-safe way and poll at human-scale intervals: roughly 60 seconds for small pilots and 120-300 seconds for larger runs.
10. For failures, localize the failing stage, then inspect the stage config, expected inputs, output directory, and corresponding CLI wrapper or run_uv.py.

References

• references/embed.md: embedding recipe stages, commands, defaults, output paths, and operating patterns.
• references/rerank.md: rerank recipe stages, commands, defaults, output paths, and operating patterns.
• references/evaluation.md: metric interpretation, comparison hygiene, and deployment readiness checks.
• references/remote.md: remote execution profiles, batch/run mode, GPU scoping, logs, and polling.

Examples

User asks: "Recall is decent, but nDCG is poor and the right passage is around rank 40. Should I tune embed or rerank?"

Load references/rerank.md and references/evaluation.md, explain that acceptable recall with poor top-rank ordering points to reranker tuning, then offer a cheap preview before training.

uv run nemotron rerank run -c default -d --from prep --to eval

Troubleshooting

Localize the failing stage, then inspect the stage config, expected inputs, output directory, and corresponding CLI wrapper or run_uv.py.

Limitations

• Bundled references are condensed snapshots; verify commands, flags, defaults, and output paths against the active checkout before execution.
• This skill does not provide datasets, checkpoints, credentials, GPU capacity, Docker images, or NIM services.

Output Style

For planning or debugging recommendations, use this shape when it helps: Decision, Why, Required inputs, Preview command, Execution command, Avoid, and Next step. Omit fields that are irrelevant to a short answer.

Give concrete commands and file paths. State assumptions, expected inputs, expected outputs, and the cheapest validation step that proves the next action is ready. For long-running stages, separate preview commands from execution commands so the user can choose deliberately.

When reporting a dry-run or real run, include a compact run report: command, mode, config, dotlist overrides, input paths, output paths, validation signal or metric file, and next cheapest check. Include the checkout commit when it is available.