Tao Train Pointpillars

PointPillars

Standalone install? If this session was not initialized by the TAO skill bank plugin, run the tao-setup skill first (host preflight, credentials, cross-skill discovery).

PointPillars for 3D object detection from LiDAR point clouds. Encodes point clouds into a pseudo-image via pillar-based representation, then applies 2D detection. Used in autonomous driving / robotics.

Typically trained from scratch. Provide train.resume_training_checkpoint_path to resume.

For TAO Deploy TensorRT actions (gen_trt_engine, TensorRT evaluate, and TensorRT inference), read references/tao-deploy-pointpillars.md first. Deploy spec templates live in this skill's references/ folder with the spec_template_deploy_*.yaml prefix.

The packaged PyTorch PointPillars CLI supports dataset_convert, train, evaluate, inference, export, and prune. It does not expose a parent-model gen_trt_engine action; TensorRT engine generation is deploy-only. It also does not expose a separate retrain subcommand. Retraining from a pruned model uses pointpillars train -e ... with train.pruned_model_path populated.

Dataclass Schemas

Generated TAO Core schemas are packaged in schemas/<action>.schema.json, with schemas/manifest.json listing available actions. Each generated schema also emits references/spec_template_<action>.yaml from the schema top-level default field. AutoML enablement is declared at the model layer in references/skill_info.yaml via automl_enabled. Runnable AutoML for an action requires schemas/<action>.schema.json and references/spec_template_<action>.yaml to exist and parse. Use the packaged selected-action schema for automl_default_parameters, automl_disabled_parameters, defaults, min/max bounds, enums, option weights, math conditions, dependencies, and popular parameters. Do not expect ~/tao-core at runtime; maintainers regenerate schemas/templates before packaging the skill bank.

Train Action Policy

This model is AutoML-enabled at the model layer. Before handling any train-stage request, read references/skill_info.yaml and resolve the run override from either an explicit automl_policy value or the user's workflow request. Use automl_policy: on by default and only expose on / off in new launch prompts. Treat phrases like "turn off AutoML", "disable AutoML", "no HPO", or "plain training" as automl_policy: off for this run only. When automl_policy: on, automl_enabled: true, and both schemas/train.schema.json and references/spec_template_train.yaml are packaged, route the train action through tao-skill-bank:tao-run-automl by default with this model's skill_dir. Preserve workflow/application overrides for datasets, specs, output directories, GPU/platform settings, parent checkpoints, and automl_policy. Use direct model training only when automl_policy: off or the packaged train schema/template is missing; in the missing-schema case, report that AutoML is enabled but not runnable for this model until schemas are generated.

Non-train actions such as evaluate, inference, export, and deploy flows stay in this model skill. The per-run automl_policy override does not change model metadata.

Training Requirements

• Dataset type: pointpillars
• Formats: default
• AutoML training metric: loss (minimize). This is the per-epoch KPI emitted by the train action and is the metric AutoML must extract for trial ranking.
• Standalone evaluation metrics: the evaluate action emits recall KPIs plus dataset-specific detection metrics. For the general point-cloud dataset, verify bev mAP and 3d mAP; do not substitute an evaluation mAP for AutoML's emitted training loss.

Per-Action Dataset Requirements

Typical Spec Overrides

Data source overrides are mandatory for every action — the agent MUST construct data source paths from the Per-Action Dataset Requirements table above and include them in spec_overrides.

DATA_ROOT = "s3://bucket/data/pointpillars"
DATA_INFO = "/results/{dataset_convert_job_id}/results_dir/data_info"
CHECKPOINT = "/results/{train_job_id}/results_dir/checkpoint_epoch_1.pth"
PRUNED_MODEL = "/results/{prune_job_id}/results_dir/pruned_0.1.tlt"

The raw PointPillars data root must be an extracted folder containing matching train/lidar, train/label, val/lidar, and val/label subfolders before dataset_convert runs. If the source dataset is packaged as separate train/val archives, extract both under the same mounted data root and point dataset.data_path at that root.

train (mandatory data sources):

{
    "train.num_epochs": 30,
    "train.checkpoint_interval": 10,
    "train.validation_interval": 10,
    "train.num_gpus": 1,
    "dataset.data_path": DATA_ROOT,
    "dataset.data_info_path": DATA_INFO,
}

resume train (mandatory checkpoint):

{
    "dataset.data_path": DATA_ROOT,
    "dataset.data_info_path": DATA_INFO,
    "train.resume_training_checkpoint_path": CHECKPOINT,
}

evaluate (mandatory data sources):

{
    "dataset.data_path": DATA_ROOT,
    "dataset.data_info_path": DATA_INFO,
    "evaluate.checkpoint": CHECKPOINT,
}

export (mandatory data sources):

{
    "dataset.data_path": DATA_ROOT,
    "dataset.data_info_path": DATA_INFO,
    "export.checkpoint": CHECKPOINT,
    "export.onnx_file": "/results/{export_job_id}/results_dir/pointpillars.onnx",
}

inference (mandatory data sources):

{
    "dataset.data_path": DATA_ROOT,
    "dataset.data_info_path": DATA_INFO,
    "inference.checkpoint": CHECKPOINT,
}

prune (mandatory data sources):

{
    "dataset.data_path": DATA_ROOT,
    "dataset.data_info_path": DATA_INFO,
    "prune.model": CHECKPOINT,
}

retrain (mandatory data sources):

{
    "dataset.data_path": DATA_ROOT,
    "dataset.data_info_path": DATA_INFO,
    "train.pruned_model_path": PRUNED_MODEL,
}

For local Docker, DATA_INFO must be visible inside every train/evaluate/export/prune/retrain container. Use the dataset_convert job from the same results root, or mount/copy the converted results_dir/data_info folder into the current run and set dataset.data_info_path to that mounted container path. If the host scratch root is mounted at /results and the conversion artifacts live under host scratch/results/<job_id>/results_dir/data_info, the direct-job container path is /results/results/<job_id>/results_dir/data_info. Do not reuse a /results/<job_id>/... path from another run root unless that folder is mounted into the current job.

For AutoML train workflows, perform this as a launch preflight before calling AutoMLRunner.run: create or materialize the dataset_convert output under the current run's RESULTS_ROOT, set dataset.data_info_path to that current-run container path, and verify dbinfos_train.pkl, infos_train.pkl, and infos_val.pkl are present from the train container's point of view. If a runner is cloned or adapted from a prior AutoML algorithm, update the conversion artifact in the new run root; a stale CONVERT_JOB_ID from another results mount is not valid.

Eval Dataset

Optional. Validation data (val.tar.gz) is separate from training. Used for mAP evaluation.

Important Parameters

• train.num_epochs: Default 80 (much higher than other TAO models). PointPillars needs more epochs for convergence on 3D detection.
• train.lr: Learning rate. Default 0.003 (adam_onecycle scheduler).
• dataset.class_names: List of 3D object classes. Default 7 classes (KITTI-style). Modify to match your dataset.
• dataset.data_path: Path to point cloud data directory.
• dataset.data_info_path: Path to data info files from dataset_convert step.
• dataset.point_cloud_range: Spatial extent of the point cloud to consider. Must match your sensor configuration.
• model.dense_head.anchor_generator_config: Anchor configurations per class. Must be tuned for your object sizes and the point cloud range.

Multi-GPU / Multi-Node

Launch method: torchrun (LIGHTNING_EXCLUDED_NETWORK). Uses PyTorch native DistributedDataParallel (NOT Lightning Trainer).

• CUDA_VISIBLE_DEVICES is explicitly set from TAO_VISIBLE_DEVICES
• Uses nn.parallel.DistributedDataParallel directly (not Lightning strategy)
• NODE_RANK is copied to RANK if RANK is unset

Multi-node env vars (set by orchestrator):