Tao Mine Aoi Images

DEFT Mining and Embedding Skill

Standalone install? If this session was not initialized by the TAO skill bank plugin, run the tao-setup skill first (host preflight, credentials, cross-skill discovery).

You are the operator of the DEFT embed-then-mine workflow for VCN AOI. Your job is to take a parquet of weak target images (the gap-analysis or routing output) and a source pool, then produce a deduplicated parquet of mined source images that look similar to the targets — ready to feed into the next training round. Iterative DEFT callers also run the bundled history-aware post-processing step so a sample selected in an earlier iteration is not selected again.

The workflow is fixed and deterministic: embed the targets, embed the source pool, mine nearest neighbours, then (for iterative workflows) remove previously mined samples. Each GPU step's output parquet is the next step's input; history-aware selection is a host-side post-processing step, not another k-NN search. There is no clustering pass or human-in-the-loop selection — depth comes from picking the right encoder and a topn wide enough to leave novel candidates after history filtering.

The whole skill is a thin wrapper around three direct docker run invocations against the pinned TAO data-services image plus one optional bundled host-Python post-processor for iterative history (see versions.yaml, resolved at runtime in Setup). The container's entrypoint takes <category> <action> -e <spec.yaml> [hydra overrides...] — pass embedding image_embeddings -e <embedding_spec.yaml> … for embedding and tmm nearest_neighbors -e <mining_spec.yaml> … for mining. The -e flag points at a YAML that supplies default values for the subtask's schema; anything afterward is a bare Hydra override (key=value) that selectively overrides spec fields per run. (There is no dataset keyword inside the container — that's the TAO launcher's pillar prefix and is dropped here.) Pull the image once if it isn't cached: docker pull "$DS_IMAGE" (after resolving $DS_IMAGE per Setup).

Schema keys can rename between data-services releases (the RCA skill saw inference_csv → inference_results_dir, output_dir → results_dir). When in doubt, introspect the actual schema once per image: docker run --rm "$DS_IMAGE" embedding image_embeddings --cfg=job and ... tmm nearest_neighbors --cfg=job.

Inputs

1. Target parquet — the gap-analysis output, typically mining_gaps.parquet from tao-route-visual-changenet-samples (or kpi_gaps.parquet from tao-analyze-gaps-visual-changenet if routing was skipped). Required column: filepath. If label is also present, label-aware filtering during mining is available; otherwise the mining task silently no-ops the filter.
2. Source pool — a parquet of candidate images to mine against, with a filepath column. If the user only has a CSV, convert it to a parquet with the same columns before Step 2. For label-aware filtering, the pool must also carry a label column.
3. Embedding spec file — a YAML containing model, model_path, batch_size, and (only when model_path is a TAO .pth/.ckpt) model_config_path. Reused across Steps 1 and 2; input_parquet/output_parquet are supplied per run as Hydra overrides. The same spec MUST drive both embedding steps — embeddings from different encoders are not comparable, and mismatched encoders are the most common cause of "the mined images look unrelated" reports.
4. Mining spec file — a YAML containing topn, knn_metric, filter_by_label, and (rarely changed) source_embed_column_name/target_embed_column_name. source_parquet/target_parquet/output_parquet are Hydra overrides at run time. SigLIP and CLIP embeddings should use knn_metric: cosine. When filter_by_label: true but either embedding parquet lacks a label column, the container logs a warning and proceeds without filtering.
5. Mining history (iterative workflows only) — one JSON ledger under the run-level results directory, plus a per-iteration history summary. scripts/filter_mined_history.py owns both files. It identifies a sample by normalized filepath, validates all earlier output hashes, and rejects an iteration gap or duplicate selection. Never hand-edit or reconstruct this ledger on resume.

Setup

Use the pinned TAO data-services URI below, then confirm Docker, the NVIDIA container toolkit, and a GPU are present before doing anything else. A GPU is required for both the encoder forward pass and the cuML/cuDF k-NN search; both steps fail without CUDA.

# Pinned TAO data-services container URI (stamped from the release manifest)
DS_IMAGE=nvcr.io/nvidia/tao/tao-toolkit:7.2.0-data-services  # versions-key: images.tao_toolkit.data_services
echo "DS_IMAGE=$DS_IMAGE"

docker info > /dev/null && echo "OK: docker"
nvidia-smi > /dev/null && echo "OK: GPU"
docker image inspect "$DS_IMAGE" > /dev/null \
  || docker pull "$DS_IMAGE"

Every host path the container reads or writes must be bind-mounted. The most predictable approach mounts the workspace root with identical paths inside and outside the container, then reuses one $DOCKER alias for the three invocations:

WORKSPACE=<absolute path that contains all parquets, outputs, and the source-pool images>
DOCKER="docker run --gpus all --rm --shm-size=8g -v $WORKSPACE:$WORKSPACE -w $WORKSPACE $DS_IMAGE"

Do not pass --user $(id -u):$(id -g) — it triggers a getpwuid() KeyError during the transformers import before any work starts. The container runs as root; chown outputs back to the host UID afterward.

Author the two spec files once per iteration, placing them under $WORKSPACE so the -e argument resolves on both sides of the mount; per-run values stay out of the spec and are passed as Hydra overrides. If the source pool is a CSV, convert it to parquet up front (preserving filepath, and label if present). The default embedding_spec.yaml uses model: SigLIP, model_path: google/siglip-base-patch16-224, batch_size: 64; the default mining_spec.yaml uses topn: 5, knn_metric: cosine, filter_by_label: "false" (quoted — the schema reads it as a string). Iterative DEFT workflows keep the same topn: 5 default; preserve an explicit user value. Increase it only when the history summary shows that the narrow neighborhood is dominated by prior selections.

See references/setup.md for the full environment notes, TAO_SKILL_BANK_PATH handling, the path-mounting rationale, the getpwuid chown workaround, the CSV-to-parquet snippet, and the verbatim spec-file authoring blocks.

Method

Three commands, in order. Each command's output parquet is the next command's input. Run them as plain Bash; the $DOCKER alias from Setup handles the container, GPU, and mounts. Every invocation follows the same shape: -e <spec> for the baked-in defaults, then a handful of Hydra overrides for the run-specific paths.

Step 1 — Embed the target images

$DOCKER embedding image_embeddings \
    -e <embedding_spec.yaml> \
    input_parquet=<target_parquet> \
    output_parquet=<target_embeddings_parquet>

Reads the gap-analysis / routing output and writes a parquet with filepath, embedding, and any extra metadata columns (e.g. label, siamese_score, weakness) carried forward verbatim from the input. Print the output schema (pd.read_parquet(...).columns) to stdout so the script-check hook can confirm the embedding column exists.

If you need to override model / model_path / batch_size for one run without editing the spec, append them as Hydra overrides (e.g. model_path=...).

Step 2 — Embed the source pool

$DOCKER embedding image_embeddings \
    -e <embedding_spec.yaml> \
    input_parquet=<source_pool_parquet> \
    output_parquet=<source_embeddings_parquet>

Same command shape as Step 1, applied to the source pool. Use the identical embedding_spec.yaml as Step 1, and do not override model / model_path / batch_size differently here — mismatched encoder configs across the two steps produce non-comparable embeddings.