Tao Train Foundation Stereo

Step 3 — Write spec yaml from the spec overrides

Copy the action block from references/spec-overrides-foundation-stereo.md. Replace:

• model.model_type from Step 2 (typically FoundationStereo)
• dataset.<...>.data_sources[*].dataset_name from Step 2
• dataset.<...>.data_sources[*].data_file with the path from Step 1
• For deploy-side evaluate: enforce dataset.test_dataset.batch_size: 1 (see references/tao-deploy-foundation-stereo.md).

Shape consistency: the crop_size in dataset.test_dataset.augmentation.crop_size should match export.input_height / input_width so the trained-model evaluator and the deploy-side TensorRT evaluator operate at the same shape. Note that crop_size is decorative on the pyt evaluate path but authoritative on the deploy evaluate side — see references/troubleshooting-foundation-stereo.md and references/tao-deploy-foundation-stereo.md.

Fresh-install smoke runs are validated at crop_size: [128, 128] with dataset.max_disparity: 128 and model.max_disparity: 128. Avoid 112×112 crops and avoid setting max_disparity smaller than the square crop side for smoke tests: those combinations can fail inside FoundationStereo with feature-map or loss-mask shape mismatches before a checkpoint is produced.

Data source overrides are mandatory for every action. Each data_sources entry is a dict with two mandatory fields: data_file and dataset_name. See references/spec-overrides-foundation-stereo.md for the per-action dataset-requirements table, every action's override block, and the quantize known-issue note.

Step 4 — Run

Create writable home/cache directories inside the mounted output path before using --user. Some TAO containers do not have an /etc/passwd entry for the host UID, and PyTorch / matplotlib need writable cache paths when running as that UID.

mkdir -p <output_dir>/home \
         <output_dir>/.cache/matplotlib \
         <output_dir>/.cache/torchinductor \
         <output_dir>/.cache/xdg

docker run --gpus 'device=0' --shm-size 16G --shm-size=8g \
  --user "$(id -u):$(id -g)" \
  -e USER="$(id -un)" \
  -e LOGNAME="$(id -un)" \
  -e HOME=<output_dir>/home \
  -e MPLCONFIGDIR=<output_dir>/.cache/matplotlib \
  -e TORCHINDUCTOR_CACHE_DIR=<output_dir>/.cache/torchinductor \
  -e XDG_CACHE_HOME=<output_dir>/.cache/xdg \
  -v <data_root>:<data_root>:ro \
  -v <output_dir>:<output_dir> \
  <container> \
  depth_net <action> -e <spec.yaml>

Without --user "$(id -u):$(id -g)" the container writes outputs as nobody:nogroup, blocking host-side cleanup / retry.

Step 5 — Verify

• Container exit code 0
• status.json kpi block populated
• For train: inspect per-step train_loss directly (the entrypoint reports Execution status: PASS even when loss is NaN)
• For evaluate: rely on epe / bp1 / bp2 / bp3 / d1 / rmse (the evaluator also emits abs_rel / sq_rel / rmse_log which are non-meaningful for stereo — see references/parameters-foundation-stereo.md)
• For inference: artifacts under results_dir

For TAO Deploy TensorRT actions (gen_trt_engine, TensorRT evaluate, and TensorRT inference), read references/tao-deploy-foundation-stereo.md first. Deploy spec templates live in this skill's references/ folder with the spec_template_deploy_*.yaml prefix.

Training Requirements

• AutoML metric contract: when recommendations are scored with the packaged evaluate action, use val/epe with minimize direction.
• Training monitoring metric: val/loss Use this only for live training-loss monitoring.
• AutoML/evaluation KPI: val/epe, minimized. The packaged evaluate action writes this key to status.json; using val/loss as the AutoML objective makes trial metric extraction fail. Its Lightning console table labels the same evaluator output test/epe, so compare AutoML's recorded val/epe value with the console's test/epe value when verifying a run.
• Recommended AutoML train parameters: train.optim.lr and train.optim.weight_decay. Do not search validation/test/inference augmentation fields, and do not search model.volume_dim because the current implementation does not consume it.
• Eval dataset: optional. Val dataset configured via dataset.val_dataset.data_sources (each entry needs data_file and dataset_name).

See references/spec-overrides-foundation-stereo.md for the per-action dataset-requirements table and every action's mandatory data-source override block.

Parameters, Metrics, Multi-GPU, Export/TRT, Hardware

See references/parameters-foundation-stereo.md for the full Important Parameters list (incl. model.encoder vits override, model.max_disparity default 416, model.volume_dim no-op note, dataset.baseline, dataset.focal_x, train.precision, export.batch_size), the Evaluation Metrics table, Multi-GPU / Multi-Node launch keys, Export / TRT Defaults (opset_version/on_cpu pairing, NGC 576×960 settings), and Hardware requirements.

Error Patterns and Troubleshooting

See references/troubleshooting-foundation-stereo.md for disparity overflow, smoke-test shape mismatch, missing pretrained paths, the encoder / dataset_name struct errors, the depth_net_stereo: not found entrypoint note, the pyt-vs-deploy crop_size discussion, and the deploy evaluate scalar-conversion failure.

Spec Param / Parent Model Inference

See references/checkpoint-inference-mappings-foundation-stereo.md for the checkpoint-resolution rules (model_epoch_<epoch>_step_<step>.pth, dn_model_latest.pth policy), the absence of parent PyT gen_trt_engine, and the full per-action inference-mapping table from depth_net_stereo.config.json (including parent_model / parent_job_id resolution).

Deployment

• tao-deploy-foundation-stereo