Tao Generate Image Grounding

What it does

When your agent uses it

• The user wants to ground captions to bboxes
• Generate phrase-grounded annotations
• Auto-label images for grounding
• Run the imagegrounding pipeline

Example prompts

• “image grounding”
• “phrase grounding”
• “ground captions”
• “/tao-generate-image-grounding”

Requirements

• Docker
• A credential in GOOGLE_API_KEY
• Compatibility (from SKILL.md): Requires docker + nvidia-container-toolkit + at least one VLM endpoint (Gemini API key or OpenAI-compatible).
• Pre-approved tools (allowed-tools): Read, Bash, Write

Workflow steps

5 steps, taken from the first numbered list in SKILL.md.

1. 1Input JSONL: Ask for the JSONL path. Each line must be one object like {"image_path": "...", "caption": "..."}. image_path can be absolute…
2. 2Image root: If any image_path values are relative, set data.image_root to the directory they should resolve from.
3. 3API access: Ask the user which VLM endpoint they want to use. Present these five options and act on the choice
4. 4Workflow steps: Choose one of
5. 5Resume vs fresh run: By default, the workflow reuses checkpoints and skips completed records. To reprocess everything, set…

What it can do on your machine

Read from SKILL.md and the folder at commit dfdd080. It shows what the files ask for, not the result of running them.

• Tool permissions

  Pre-approves these tools, so the agent can use them without asking each time:

  • Read
  • Bash
  • Write

  From allowed-tools in the SKILL.md frontmatter.

• Runs code

  No scripts in the folder and no shell commands in SKILL.md (its code samples are bash).

  From the folder's file list and the shell code blocks in SKILL.md.

• Network

  Hosts in commands or code, which the agent is likely to contact:

  • inference-api.nvidia.com

  From URLs in SKILL.md, links to its own repository left out.

• Credentials

  Names these keys or tokens, usually read from environment variables:

  • GOOGLE_API_KEY

  From names ending in _API_KEY, _TOKEN, _SECRET, _KEY or _PASSWORD in SKILL.md.

• Compatibility

  Requires docker + nvidia-container-toolkit + at least one VLM endpoint (Gemini API key or OpenAI-compatible).

  From compatibility in the SKILL.md frontmatter.

Context cost

Tao Generate Image Grounding loads about 2k tokens when it runs, and up to ~6.4k if it reads all its reference files. Until then it costs about 117 tokens; SKILL.md has 744 words of instructions outside code blocks.

Always · name and description, kept in context so the agent knows when to use it

~117

When it runs · the whole SKILL.md, loaded when a task matches

~2k

With references · SKILL.md plus every file in references/, read only if the agent opens them

~6.4k

Estimates: characters ÷ 4, the usual rule of thumb; real counts depend on the model's tokenizer. Scripts and assets cost tokens only if the agent reads them.

Safety

• NotePre-approves every shell command (allowed-tools: Bash)SKILL.md

  allowed-tools: Read, Bash, Write

Automated static check — not a guarantee. Review scripts before installing. It scans the text of SKILL.md for risky patterns (piping downloads into a shell, reading credential files, hidden Unicode, destructive commands); files beside SKILL.md are not scanned.

SKILL.md

The full file from NVIDIA/skills at commit dfdd080, republished under its Apache-2.0 licence (© NVIDIA). 744 words, ~2,006 tokens.

name

tao-generate-image-grounding

description

Two-step image grounding pipeline: extracts referring expressions from (image, caption) pairs and grounds them to pixel-space bounding boxes via a VLM. Use when the user wants to ground captions to bboxes, generate phrase-grounded annotations, auto-label images for grounding, or run the image_grounding pipeline. Triggers include 'image grounding', 'phrase grounding', 'ground captions', 'auto-label image grounding', 'image_grounding'.

allowed-tools

Read, Bash, Write

compatibility

Requires docker + nvidia-container-toolkit + at least one VLM endpoint (Gemini API key or OpenAI-compatible).

license

Apache-2.0

metadata.author

NVIDIA Corporation

metadata.version

0.1.0

tags

image, grounding, bounding-boxes, auto-label, vlm, 2d-grounding

Step 0: Expression extraction  → VLM cleans caption, extracts referring expressions + char spans
Step 1: Phrase grounding       → VLM returns pixel bboxes + scores per expression

auto_label generate -e /path/to/spec.yaml \
    results_dir=/results \
    image_grounding.data.input_jsonl=/data/captions.jsonl \
    image_grounding.data.image_root=/data/images \
    image_grounding.vlm.gemini.api_key=$GOOGLE_API_KEY

© NVIDIA, Apache-2.0. Rendered from Markdown: HTML in the file is shown as text, images as links, and headings moved down two levels. Raw file

Files

SKILL.md and 8 other files (references) in skills/tao-generate-image-grounding of NVIDIA/skills.

• SKILL.md
• BENCHMARK.md
• config/skillspector-baseline.yaml
• evals/evals.json
• references/configuration.md
• references/skill_info.yaml
• references/vllm_server.md
• skill-card.md
• skill.oms.sig

Open the folder on GitHubat commit dfdd080