Nemotron 3 Ultra Text2sql Lora

Nemotron-3 Ultra Text2SQL LoRA — runbook for a coding agent

This skill helps you run the cookbook in this directory (mbridge_lora_cookbook.ipynb) on the user's behalf. The notebook is generic and ships with placeholders; your job is to gather the user's environment details, fill them in, launch the SLURM jobs, watch them, and report results.

What the tutorial does

Three steps, in order, each a SLURM job:

1. Data prep — builds a BIRD Text2SQL training.jsonl from both the no-reasoning and reasoning splits, formatted with Ultra's tokenizer/chat template. Short CPU job.
2. Convert — distributed import of the Hugging Face base checkpoint into Megatron-Bridge format. A multi-node GPU job (CPU import is not feasible for a 550B model).
3. LoRA fine-tune — packed-sequence LoRA training on the prepared data; saves a LoRA adapter. A multi-node GPU job.

What you must understand before running

• Ultra is a 550B-total / A55B-active hybrid Mamba-Transformer MoE. It does not fit on one node, so every heavy step is a multi-node SLURM job submitted with sbatch and run in a container via Pyxis/enroot. Run everything from a cluster login node where sbatch/squeue/ sacct are available.
• Scale. At the shipped parallel settings, both convert and train need 48 GPUs. Node count is derived automatically as 48 / GPUS_PER_NODE (e.g. 12 nodes at 4 GPUs/node). The user's QOS must permit a job of that size — an interactive or small-node-capped QOS will not work.
• Single config. Everything is driven by one file, config.env, which the notebook's setup cell generates from the values you fill in. Every step and every slurm/*.sbatch script sources it. You can run the notebook cell, or write config.env directly with the same keys.
• One output root. WORKSPACE is the single output root; everything generated lands under $WORKSPACE/{base, dataprep, trained, cache/hf, logs}. The base checkpoint (HF_MODEL_PATH) is the only separate, read-only path.
• The rhythm per step: a launch cell submits the job, a re-runnable check cell shows status (sacct/squeue), and a sanity cell confirms the expected output exists before you move on. Follow this loop; don't skip the sanity check.

Information to gather from the user

Before launching anything, ask the user for the following and confirm the prerequisites. Don't guess these — a wrong value wastes a large multi-node allocation. Prefer asking all of them up front in one batch.

How to reach the cluster

• How do you connect to the login node where SLURM jobs are submitted (e.g. the ssh host)?
• Is there a separate data-transfer host you prefer for large file moves?

SLURM settings

• SLURM account to charge.
• GPU partition and a QOS that allows a multi-node job of 48 / GPUS_PER_NODE nodes (not an interactive or small-node-capped QOS). Confirm the wall-clock limit is enough (convert is short; training is well under a couple of hours by default).
• CPU partition and QOS for the short data-prep job.
• GPUs per node on the target nodes (the tutorial targets GB200 at 4 GPUs/node; the node count derives from this).

Paths (all on a shared filesystem the compute nodes can mount)

• WORKSPACE — the output root to create/use.
• HF_MODEL_PATH — where the already-downloaded Ultra base checkpoint lives (read-only input). The tutorial does not download the base model; confirm it is present.
• The shared-filesystem root to bind-mount into the container (must contain both WORKSPACE and HF_MODEL_PATH).

Container & credentials

• The container image to use (path to a prepared image or a registry reference). The notebook ships a placeholder; this must be filled with a real Ultra-capable image.
• A Hugging Face token so BIRD can be downloaded during data prep. The tutorial expects it at ${WORKSPACE}/cache/hf/token; ask the user to place it there (or provide it so you can), and reference it by path — never print or echo a token.

If the user has an environment-reference document for their cluster, ask for it first and pull these values from there instead of asking one by one.