Nv Segment Ct Finetune

NV-Segment-CT Finetune

Purpose

• Used for smoke or dataset finetuning of NV-Segment-CT VISTA3D on CT NIfTI labels, including the upstream fixed-channel softmax workflow and optional MLflow tracking. Not for clinical validation.
• Wraps the upstream MONAI bundle entrypoint; do not replace it with handwritten training or inference code.
• Manifest inputs are dataset_dir, datalist, target_anatomy, label_mapping, smoke, sanity, auto_seg, softmax, skip_formal_eval, mlflow_tracking_uri, mlflow_experiment_name, and mlflow_run_name.
• Manifest outputs are finetuned_ckpt and schema-checked result_json.

Instructions

• Run only the caller-requested preset, dataset, output directory, and compute budget. Dependency setup and remote tracking require the caller's approval. After reporting the result, stop; further training, publishing, or deployment is a separate request.
• Run scripts/run_finetune.py; do not patch files under bundle/ or upstream checkouts during normal skill use.
• For standalone Bash, include the fresh-environment setup line before the wrapper; benchmark venvs start empty.
• Run the committed script in place from the repo root. Do not copy this skill to a runtime directory, and do not use rm or cleanup commands in generated invocations.
• If a host exposes run_script, use run_script("scripts/run_finetune.py", args=[...]); otherwise run from the repo root.
• For the shortest workflow check, use --smoke; for MSD Task06 Lung Tumor reproduction, use --sanity.
• Choose between the standard and --softmax workflows using the criteria below. Do not combine --softmax with --auto-seg or --sanity.
• Set --mlflow-experiment-name to enable MLflow for the training phase of either workflow. --mlflow-tracking-uri and --mlflow-run-name require an experiment name. Formal pre/post evaluation does not receive MLflow credentials.
• Read references/task06-and-results.md only when you need Task06 reference details, output-field definitions, or manual bundle setup notes.

Choosing the Workflow

Use --softmax only when all of these conditions hold:

• The complete class set is known before training and will not vary between inference requests.
• Labels are mutually exclusive: each voxel is background or exactly one foreground class.
• Every foreground dataset label maps to an existing VISTA3D class ID, and a conventional fixed-channel output is desired.

Keep the standard workflow if point prompts must remain available, classes are selected dynamically at inference, labels can overlap, or the Task06 --sanity reproduction is required.

For --label-mapping '[[1,3],[2,13]]', channel 0 is background, channel 1 represents dataset label 1 initialized from VISTA3D class 3, and channel 2 represents dataset label 2 initialized from VISTA3D class 13. Preserve the entries and their order when using the resulting model_softmax.pt with upstream configs/inference_softmax.json. The nv-segment-ct and nv-segment-ctmr inference skills do not currently expose that fixed-channel inference path.

Available Scripts

Prerequisites

• Python 3.10+ with CUDA-capable Torch for GPU runs.
• Runtime packages from skill_manifest.yaml, especially monai==1.4.0, numpy<2, nibabel, scipy, typer, PyYAML, fire, pytorch-ignite, einops, and huggingface_hub. Install mlflow>=2.10,<4 when MLflow tracking is enabled.
• Optional environment variables: CUDA_VISIBLE_DEVICES restricts visible GPUs; NPROC_PER_NODE overrides GPU count and values >=2 select multi-GPU mode for non-sanity runs; NVSEG_FINETUNE_AUTO_VENV=0 disables the cached MONAI 1.4 compatibility environment. Remote tracking may use DATABRICKS_CONFIG_PROFILE, DATABRICKS_HOST, DATABRICKS_TOKEN, MLFLOW_TRACKING_CLIENT_CERT_PATH, MLFLOW_TRACKING_INSECURE_TLS, MLFLOW_TRACKING_PASSWORD, MLFLOW_TRACKING_SERVER_CERT_PATH, MLFLOW_TRACKING_TOKEN, or MLFLOW_TRACKING_USERNAME; these variables are forwarded only when MLflow is explicitly enabled, and unrelated credentials are not forwarded.
• --softmax also needs the pinned NVIDIA-Medtech source checkout. Set NV_SEGMENT_CT_ROOT to its NV-Segment-CT directory, or set NV_SEGMENT_CTMR_ROOT to the sibling NV-Segment-CTMR directory. The wrapper reads the official softmax config and implementation in place and writes generated overrides only under --output-dir.
• Run outputs: generated bundle configs under skills/nv-segment-ct-finetune/bundle/configs/, including auto_override.json, train_continual_task06_lung.json, and dfw_no_logging.json; checkpoints/evidence under --output-dir; and local tracking data under <output-dir>/mlruns when enabled.
• Dependency cache locations: ~/.cache/nvidia-skills/venvs/nv-segment-ct-finetune-monai14/ for MONAI compatibility packages and ~/.cache/huggingface/ for model assets. These are reusable runtime files, not agent instructions or authorization for another run. Set NVSEG_FINETUNE_AUTO_VENV=0 when compatibility-environment setup is not approved; then use a caller-provided compatible environment.
• Network access: model/config downloads use https://huggingface.co and https://raw.githubusercontent.com; remote tracking contacts only the caller-approved MLflow or Databricks destination when explicitly enabled. The label-dictionary download accepts HTTPS on the pinned source host and rejects redirects.

Fresh environment setup:

python -m pip install "monai==1.4.0" "numpy<2" pytorch-ignite einops nibabel scipy typer PyYAML fire huggingface_hub

When MLflow tracking is enabled, also install:

python -m pip install "mlflow>=2.10,<4"

Known upstream compatibility constraints:

• DFW Task06 reference: Python 3.10.16, MONAI 1.4.0, Torch 2.7.0+cu126.
• Use exact monai==1.4.0 for smoke, sanity, and evidence runs; MONAI 1.5.x can crash the upstream finetune loss on boolean labels.
• Do not float the dependency as monai>=1.4,<1.6 in generated commands.
• The softmax workflow keeps the upstream defaults of 100 epochs and learning rate 1e-4 unless the caller overrides them.

One-time source setup for --softmax:

export NV_SEGMENT_CTMR_COMMIT=cb921f5c58837c0f42a713855d68b32af88e1cdd
export NV_SEGMENT_CTMR_CHECKOUT="$HOME/.cache/nvidia-skills/upstreams/NV-Segment-CTMR-cb921f5"
if [ ! -d "$NV_SEGMENT_CTMR_CHECKOUT/.git" ]; then
  git clone https://github.com/NVIDIA-Medtech/NV-Segment-CTMR.git "$NV_SEGMENT_CTMR_CHECKOUT"
fi
git -C "$NV_SEGMENT_CTMR_CHECKOUT" checkout --detach "$NV_SEGMENT_CTMR_COMMIT"
export NV_SEGMENT_CT_ROOT="$NV_SEGMENT_CTMR_CHECKOUT/NV-Segment-CT"

Usage

Smoke-scale workflow check:

python -m pip install "monai==1.4.0" "numpy<2" pytorch-ignite einops nibabel scipy typer PyYAML fire huggingface_hub && \
python skills/nv-segment-ct-finetune/scripts/run_finetune.py \
  PATH_TO_DATASET \
  --smoke \
  --patch-size '[64,64,64]' \
  --output-dir runs/nvseg_smoke

Use the staged dataset as PATH_TO_DATASET. For the micro fixture, use skills/nv-segment-ct-finetune/fixtures/spleen_micro. Smoke mode proves wiring, config generation, checkpoint loading, and runtime compatibility; it is not a quality bar.

MSD Task06 Lung Tumor sanity reproduction:

python skills/nv-segment-ct-finetune/scripts/run_finetune.py \
  /path/to/Task06 \
  --sanity \
  --output-dir runs/nvseg_task06_sanity

The sanity preset follows the single-GPU DFW recipe: fold-0 validation, label mapping [[1, 23]] for lung tumor, automatic class-prompt segmentation, patch [128,128,128], 5 epochs, and original-spacing configs/evaluate.json scoring before and after training. Expected reference range is pretrained Dice about 0.6697, training-best Dice about 0.6905, and fine-tuned formal Dice about 0.6836.

User-data finetune:

python skills/nv-segment-ct-finetune/scripts/run_finetune.py \
  --dataset-dir /path/to/dataset \
  --datalist /path/to/datalist.json \
  --target-anatomy "lung tumor" \
  --auto-seg \
  --epochs 5 \
  --patch-size '[128,128,128]' \
  --output-dir runs/nvseg_user_finetune

Use --label-mapping '[[1, 23]]' when local label values are custom or the anatomy name is ambiguous.

Optional local MLflow tracking:

python skills/nv-segment-ct-finetune/scripts/run_finetune.py \
  --dataset-dir /path/to/dataset \
  --datalist /path/to/datalist.json \
  --target-anatomy "lung tumor" \
  --epochs 5 \
  --mlflow-experiment-name nvseg-finetune \
  --mlflow-run-name trial-01 \
  --output-dir runs/nvseg_mlflow

This uses MONAI's documented --tracking mlflow path and built-in rank-zero handlers. With no --mlflow-tracking-uri, data stays in <output-dir>/mlruns. Pass a caller-approved remote URI, including databricks, only when remote tracking is intended. MLflow does not change patch size, transforms, optimizer values, DataLoader settings, or other training configuration.

Fixed-channel softmax finetune for mutually exclusive labels:

export NV_SEGMENT_CT_ROOT="$HOME/.cache/nvidia-skills/upstreams/NV-Segment-CTMR-cb921f5/NV-Segment-CT"
python skills/nv-segment-ct-finetune/scripts/run_finetune.py \
  --dataset-dir /path/to/dataset \
  --datalist /path/to/datalist.json \
  --label-mapping '[[1,3],[2,13]]' \
  --softmax \
  --epochs 100 \
  --output-dir runs/nvseg_softmax

This delegates to upstream configs/train_continual_softmax.json. It produces checkpoints/model_softmax.pt; the source model.pt initializes the network but is not compatible with configs/inference_softmax.json. The wrapper therefore recommends the produced softmax checkpoint after a successful run.

Examples

Smoke run on a staged tiny dataset:

python skills/nv-segment-ct-finetune/scripts/run_finetune.py \
  runs/with_vs_without_nv/_inputs/nv_segment_ct_finetune/input_dataset \
  --smoke \
  --patch-size '[64,64,64]' \
  --output-dir runs/nvseg_smoke

Task06 sanity run on a local MSD cache:

python skills/nv-segment-ct-finetune/scripts/run_finetune.py \
  .workbench_data/datasets/Task06_Lung \
  --sanity \
  --output-dir runs/nvseg_task06_sanity