Tao Generate Referring Expressions

What it does

When your agent uses it

• The user wants to generate referring-expression annotations from images with KITTI labels
• Build region descriptions
• Produce grouped grounding phrases tied to bboxes
• Run a double-check verification pass on grounding expressions

Example prompts

• “referring expression”
• “region description”
• “KITTI labels”
• “/tao-generate-referring-expressions”

Requirements

• Docker
• A credential in GOOGLE_API_KEY
• Compatibility (from SKILL.md): Requires docker + nvidia-container-toolkit + at least one VLM endpoint (Gemini API key or OpenAI-compatible).
• Pre-approved tools (allowed-tools): Read, Bash, Write

Workflow steps

6 steps, taken from the first numbered list in SKILL.md.

1. 1Images: Ask for data.image_dir, the directory containing .jpg, .jpeg, or .png images.
2. 2KITTI labels: Ask for data.kitti_label_dir, the directory containing one .txt label file per image. Each label line must use KITTI format…
3. 3Resume from existing annotations: If the user already has a unified annotations.jsonl from a previous run, set…
4. 4API access: Ask the user which VLM endpoint they want to use. Present these five options and act on the choice
5. 5Workflow steps: Choose one of
6. 6Output format: Choose one of

What it can do on your machine

Read from SKILL.md and the folder at commit 14a98ae. It shows what the files ask for, not the result of running them.

• Tool permissions

  Pre-approves these tools, so the agent can use them without asking each time:

  • Read
  • Bash
  • Write

  From allowed-tools in the SKILL.md frontmatter.

• Runs code

  No scripts in the folder and no shell commands in SKILL.md (its code samples are bash).

  From the folder's file list and the shell code blocks in SKILL.md.

• Network

  Hosts in commands or code, which the agent is likely to contact:

  • inference-api.nvidia.com

  From URLs in SKILL.md, links to its own repository left out.

• Credentials

  Names these keys or tokens, usually read from environment variables:

  • GOOGLE_API_KEY

  From names ending in _API_KEY, _TOKEN, _SECRET, _KEY or _PASSWORD in SKILL.md.

• Compatibility

  Requires docker + nvidia-container-toolkit + at least one VLM endpoint (Gemini API key or OpenAI-compatible).

  From compatibility in the SKILL.md frontmatter.

Context cost

Tao Generate Referring Expressions loads about 2.6k tokens when it runs, and up to ~7.9k if it reads all its reference files. Until then it costs about 198 tokens; SKILL.md has 990 words of instructions outside code blocks.

Always · name and description, kept in context so the agent knows when to use it

~198

When it runs · the whole SKILL.md, loaded when a task matches

~2.6k

With references · SKILL.md plus every file in references/, read only if the agent opens them

~7.9k

Estimates: characters ÷ 4, the usual rule of thumb; real counts depend on the model's tokenizer. Scripts and assets cost tokens only if the agent reads them.

Safety

• NotePre-approves every shell command (allowed-tools: Bash)SKILL.md

  allowed-tools: Read, Bash, Write

Automated static check — not a guarantee. Review scripts before installing. It scans the text of SKILL.md for risky patterns (piping downloads into a shell, reading credential files, hidden Unicode, destructive commands); files beside SKILL.md are not scanned.

SKILL.md

The full file from NVIDIA/skills at commit 14a98ae, republished under its Apache-2.0 licence (© NVIDIA). 990 words, ~2,586 tokens.

name

tao-generate-referring-expressions

description

Four-step image referring-expression pipeline: turns images plus KITTI bounding-box labels into region descriptions, scene captions, grounded referring expressions, and (optionally) verified expressions via VLM distillation. Use when the user wants to generate referring-expression annotations from images with KITTI labels, build region descriptions, produce grouped grounding phrases tied to bboxes, run a double-check verification pass on grounding expressions, auto-label traffic / scene images for referring datasets, or run the image_referring_expression pipeline. Triggers include 'referring expression', 'region description', 'KITTI labels', 'spatial relationship annotation', 'auto-label image referring expression', 'image_referring_expression'.

allowed-tools

Read, Bash, Write

compatibility

Requires docker + nvidia-container-toolkit + at least one VLM endpoint (Gemini API key or OpenAI-compatible).

license

Apache-2.0

metadata.author

NVIDIA Corporation

metadata.version

0.1.0

tags

image, referring-expression, kitti, bounding-boxes, auto-label, vlm

Step 0: Region expression  ──┐
                              ├──▶  Step 2: Grounding expression  ──▶  [Step 3: Double check]
Step 1: Image caption  ──────┘                                                   (optional)

auto_label generate -e /path/to/spec.yaml \
    results_dir=/results \
    image_referring_expression.data.image_dir=/data/images \
    image_referring_expression.data.kitti_label_dir=/data/labels \
    image_referring_expression.vlm.gemini.api_key=$GOOGLE_API_KEY

© NVIDIA, Apache-2.0. Rendered from Markdown: HTML in the file is shown as text, images as links, and headings moved down two levels. Raw file

Files

SKILL.md and 8 other files (references) in skills/tao-generate-referring-expressions of NVIDIA/skills.

• SKILL.md
• BENCHMARK.md
• config/skillspector-baseline.yaml
• evals/evals.json
• references/configuration.md
• references/skill_info.yaml
• references/vllm_server.md
• skill-card.md
• skill.oms.sig

Open the folder on GitHubat commit 14a98ae