Vision Skills

vision-skills

Five local CLIs that give a text-only agent eyes. They read one shared vision config (VISION_API_KEY / VISION_BASE_URL / VISION_MODEL / LANG), plus the optional Python-client settings VISION_API_PROTOCOL, VISION_REASONING_EFFORT, and VISION_USER_AGENT — no extra credentials.

Pick the tool by the question you are answering:

glance answers what something is; ground and detect answer where. You give ground a description of a particular thing; you give detect a kind and it enumerates the instances.

Both give real coordinates, but they are not pixel-exact: the box arrives on a 0-1000 grid and is scaled to your image, so the last pixel or few are not reliable. That is accurate enough to crop with, to click, to compare positions against. When a number has to be exact, trace derives it from the actual pixels — offsets, sizes, shapes.

Use the provided tools before hand-rolled pixels

Everything this toolkit ships a tool for, call the tool — do not rewrite it with Pillow in the middle of a task. The CLIs exist so the same pixel work is not hand-coded differently every time:

• cut a box out of an image → crop, not Image.open(...).crop(...)
• sample a region's palette → scripts/dominant_colors.py
• compare two images → scripts/pixel_diff.py
• vectorize to SVG → trace
• locate / inventory elements → ground / detect
• describe / OCR an image → glance
• safely split, OCR, and merge a long screenshot → scripts/long_screenshot_ocr.py
• HTML file to a viewport or full-page screenshot → scripts/html_shot.py

Hand-written Pillow is only for what none of them return: a relation between two things you already located (a gap, a distance), a resize or overlay, drawing. If you catch yourself writing .crop(), .convert(), or histogram code where one of the tools above fits, replace it with the tool call — same coordinates, same box format, and the output feeds the next tool directly.

glance — ask about an image

glance <image>                                 # detailed description
glance <image> -q "<question>"                 # targeted question (qualitative only)
glance <image> --ocr                           # verbatim OCR
glance <image> --region X1,Y1,X2,Y2 -q "..."   # zoom into a crop
glance <img1> <img2> -q "..."                  # compare in ONE call

When you do compare with glance, pass all paths to one call — separate calls cannot see both images, so two descriptions compared afterwards are two hallucination surfaces, not a comparison. --region uploads only the crop, so small text and icons become readable.

But "what changed between these two?" is not a glance question. A one-word badge or a small shift is a rounding error to a vision model and exact to scripts/pixel_diff.py. Diff first to get the box, then glance --region that box to read what the change actually is.

For a tall scrolling screenshot, do not send the whole image through one OCR call and accept the model's downscaling loss. Run the long-screenshot workflow, which finds low-content cut bands, invokes glance on each chunk, uses structured extraction for chat histories, merges only duplicated overlap, and writes a boundary audit:

python3 scripts/long_screenshot_ocr.py work/page.png -o work/page.ocr.md
python3 scripts/long_screenshot_ocr.py work/chat.png --mode chat --resume -o work/chat.ocr.md

Read references/long-screenshot-ocr.md before using it. It defines the verification pass for unsafe cuts and chat-message boundaries.

ground — locate a named target

ground <image> "<target description>"
ground <image> "<target>" --region X1,Y1,X2,Y2

Output: x1: .., y1: .., x2: .., y2: .. in original-image pixels — with --region too (crop hits are mapped back).

Provider-native 0-1000 boxes do not all use the same array order: Gemini uses [y0, x0, y1, x1], while Qwen3-VL, Qwen3.5, and Qwen3.6 use [x0, y0, x1, y1]. Grounding code must select the order by model family (or an explicit override) before scaling to pixels; never parse every provider as Gemini-style yxyx.

If several boxes come back numbered, your description matched more than one element rather than picking out a single thing. Narrow it with what distinguishes the one you mean — its text, its position, the block it sits in — and ask again.

The box is a handle, not just an answer — it feeds the next call:

$ ground screenshot.png "the send button"
x1: 1067, y1: 841, x2: 1108, y2: 881
$ glance screenshot.png --region 1067,841,1108,881 -q "is it enabled or greyed out?"

That two-step is how you inspect anything too small to survive a full-image pass.

detect — find every instance of a kind

detect <image>                        # every UI element
detect <image> "buttons"              # one kind only
detect <image> --region X1,Y1,X2,Y2   # inside one box
detect <image> "text" --fail-on-empty # fail if no text is detected

The default category targets UI screenshots. For photographs or video frames, pass an explicit category such as "objects" or "text". An empty inventory normally prints no elements detected and exits 0; --fail-on-empty instead exits 1 with a stderr diagnostic and no stdout. This reports an empty model result, not proof that the image has no matching elements.

You name a particular thing for ground; you name a kind for detect and it enumerates the instances. Output is a numbered list with each item's visible text and box. A full-screen pass is a fast first draft — counts vary run to run on dense screens. For completeness, detect the layout blocks first, then detect --region each block.

trace — exact shape geometry (local, no vision API)

trace <image>                                  # b/w spline SVG to stdout
trace <image> --polygon                        # boxy diagrams/wireframes
trace <image> --region X1,Y1,X2,Y2 -o out.svg  # crop first

Coordinates come from the actual pixels, not a model's estimate. Flat, high-contrast graphics only; text becomes curves (pair with --ocr when the text matters). Small images are upscaled automatically before tracing, so a 30px icon traces as readily as a screenshot — size is not a reason to skip the tool. Before shipping or reusing a traced SVG, read references/restore-graphic.md — it holds the reuse traps and the ship-vs-hand-write call.

crop — cut a pixel box out of an image (local, no vision API)

crop <image> --region X1,Y1,X2,Y2             # writes <image-stem>.crop.png next to the input
crop <image> --region X1,Y1,X2,Y2 -o out.png
crop <image> --region X1,Y1,X2,Y2 --scale 4   # upscale the cut-out 4x (LANCZOS) first

The same X1,Y1,X2,Y2 pixel boxes ground/detect print, clamped to the image bounds. Once a box is worth keeping — the same crop is about to feed pixel_diff, dominant_colors, and trace in turn — cut it to a file once and reuse it, instead of re-cropping in memory on every call. --scale N upscales the cut-out before writing (default output name becomes <image-stem>.crop@Nx.png): for icons too small for ground/trace to see clearly, crop with --scale 4, then run ground/trace on the upscaled file — coordinates it returns are in the upscaled grid, divide by N to map back to the original image. Requires the optional pillow.

extract_fg — icon foreground as transparent PNG: manual region or auto (local, no vision API)

# manual: you know the region (and optionally the background colour)
python3 scripts/extract_fg.py shot.png --region X1,Y1,X2,Y2 -o icon.png
python3 scripts/extract_fg.py shot.png --region X1,Y1,X2,Y2 --mode dark          # grey/black line logos
python3 scripts/extract_fg.py shot.png --region X1,Y1,X2,Y2 --exclude-color '#E6E6E6'
# auto: `crop --scale` cut-outs with the icon centred — no region needed
crop shot.png --region X1,Y1,X2,Y2 --scale 4 -o d/icon1.png
python3 scripts/extract_fg.py d/icon1.png d/icon2.png       # writes <stem>.clean.png next to each input
python3 scripts/extract_fg.py d/icon1.png --disc-radius 60
python3 scripts/extract_fg.py d/icon1.png --boxes "101,84,184,171"

Manual mode keeps every sufficiently large connected component of the region (separate logo sub-shapes stay together; specks drop out). Auto mode takes a crop --scale cut-out with the icon centred (disc + glyph): the disc centre is the image centre, the disc radius defaults to min(w,h)/2 * 0.6, and the disc colour is sampled from a ring around the centre; that colour is excluded and the glyph is picked as the most saturated among the three largest coloured components (white rings, ripples, and text fall away), output as a 1:1 transparent PNG. When auto inference fails, override the radius with --disc-radius, or pass a ground box (in the upscaled grid) as --boxes to recentre and re-filter by overlap. Multiple images may be passed at once (auto mode). Requires the optional pillow (and numpy for auto mode).