VLM BCQ Gap Analysis

What it does

When your agent uses it

• Pulling failure cases out of a VLM yes/no evaluation
• Preparing inputs for root-cause analysis in a DEFT iteration
• Getting false positive and false negative counts from a predictions JSON

Example prompts

• “Extract the false positives and false negatives from my VLM predictions JSON.”
• “Generate the vlm_bcq spec for results.json and run the gap analysis.”
• “Analyze VLM BCQ gaps for these predictions, where the video IDs are relative to my videos folder.”

Requirements

• Docker, NVIDIA Container Toolkit and one visible GPU
• A predictions JSON from a VLM binary-classification run
• The TAO Toolkit data services container
• Compatibility (from SKILL.md): Requires Docker, NVIDIA Container Toolkit, and one visible GPU.
• Pre-approved tools (allowed-tools): Read, Bash

What it can do on your machine

Read from SKILL.md and the folder at commit 14a98ae. It shows what the files ask for, not the result of running them.

• Tool permissions

  Pre-approves these tools, so the agent can use them without asking each time:

  • Read
  • Bash

  From allowed-tools in the SKILL.md frontmatter.

• Runs code

  Ships 1 file in scripts/ (Python), which the agent can run.

  Shell commands in SKILL.md call:

  • python3

  From the folder's file list and the shell code blocks in SKILL.md.

• Network

  No URLs in SKILL.md.

  From URLs in SKILL.md, links to its own repository left out.

• Credentials

  Names no API keys, tokens, secrets or passwords.

  From names ending in _API_KEY, _TOKEN, _SECRET, _KEY or _PASSWORD in SKILL.md.

• Compatibility

  Requires Docker, NVIDIA Container Toolkit, and one visible GPU.

  From compatibility in the SKILL.md frontmatter.

Context cost

VLM BCQ Gap Analysis loads about 1.3k tokens when it runs, and up to ~1.5k if it reads all its reference files. Until then it costs about 90 tokens; SKILL.md has 523 words of instructions outside code blocks.

Always · name and description, kept in context so the agent knows when to use it

~90

When it runs · the whole SKILL.md, loaded when a task matches

~1.3k

With references · SKILL.md plus every file in references/, read only if the agent opens them

~1.5k

Estimates: characters ÷ 4, the usual rule of thumb; real counts depend on the model's tokenizer. Scripts and assets cost tokens only if the agent reads them.

Safety

• NotePre-approves every shell command (allowed-tools: Bash)SKILL.md

  allowed-tools: Read, Bash

Automated static check — not a guarantee. Review scripts before installing. It scans the text of SKILL.md for risky patterns (piping downloads into a shell, reading credential files, hidden Unicode, destructive commands); the scripts in this folder are not scanned.

SKILL.md

The full file from NVIDIA/skills at commit 14a98ae, republished under its Apache-2.0 licence (© NVIDIA). 523 words, ~1,315 tokens.

name

tao-analyze-gaps-vlm-bcq

description

Extract false-positive and false-negative gaps from VLM binary-classification-question (BCQ, yes/no) predictions. Use when the user asks to "analyze VLM BCQ gaps", "extract VLM false positives and false negatives", or identify failure cases from a predictions JSON for DEFT root-cause analysis on a binary-classification VLM workflow.

allowed-tools

Read, Bash

compatibility

Requires Docker, NVIDIA Container Toolkit, and one visible GPU.

license

Apache-2.0

metadata.author

NVIDIA Corporation

metadata.version

0.1.0

tags

gap-analysis, rcca, vlm, evaluation, false-positive, false-negative

python3 skills/data/tao-analyze-gaps-vlm-bcq/scripts/prepare_vlm_bcq_spec.py \
  --predictions-json /path/to/results.json \
  --videos-dir /path/to/videos/root \
  --results-dir /path/to/output/gaps \
  --output-spec /path/to/output/gaps/vlm_bcq_spec.yaml

predictions_json: /path/to/results.json
videos_dir: ""
results_dir: /path/to/output/gaps

predictions_json: /path/to/results.json
videos_dir: /path/to/videos/root
results_dir: /path/to/output/gaps

gap_analysis vlm_bcq -e /path/to/vlm_bcq_spec.yaml

[
  {
    "video_id": "/path/to/video.mp4",
    "response": "Yes, there is a collision.",
    "gt": "B. No",
    "question": "Is there a collision?"
  }
]

© NVIDIA, Apache-2.0. Rendered from Markdown: HTML in the file is shown as text, images as links, and headings moved down two levels. Raw file

Files

SKILL.md and 8 other files (scripts, references, assets) in skills/tao-analyze-gaps-vlm-bcq of NVIDIA/skills.

• SKILL.md
• BENCHMARK.md
• assets/default_vlm_bcq.yaml
• config/skillspector-baseline.yaml
• evals/evals.json
• references/skill_info.yaml
• scripts/prepare_vlm_bcq_spec.py
• skill-card.md
• skill.oms.sig

Open the folder on GitHubat commit 14a98ae