Glmv Grounding

GLMV-Grounding Skill

Extract and visualize grounding results produced by GLM-V. Depending on the user prompt, grounding coordinates in model outputs may appear in different forms, including 2D bounding boxes, Objects Detection JSON, 2D points, 3D bounding boxes, and target-tracking JSON.

Note: GLM-V outputs coordinates where x and y are relative coordinates normalized from pixel coordinates x_pixel and y_pixel using image width W and height H (range 0-1000), i.e., x=round(x_pixel/W1000), y=round(y_pixel/H1000). The origin of the pixel coordinate system is the top-left corner. Note: If the prompt does not explicitly specify a grounding format (for example, "find the location of xxx" or "draw a box around xxx"), treat the request as 2D bounding boxes by default.

When to use

• Use GLM-V to ground targets in images: obtain grounding results in an image for any prompt-described target, with output formats such as 2D bounding box (default), 2D points, and 3D bounding box.
• Use GLM-V to track targets in videos: obtain tracking results in a video for any prompt-described target, with output format like {"0": [{"label": ..., "bbox_2d": ...}, ...], ...}.
• Use utility functions for extraction, conversion, and visualization: extract coordinates, points, and JSON from natural text; normalize and de-normalize coordinates; visualize boxes, points, 3D boxes, and video tracking results.

Setup your API Key

Configure ZHIPU_API_KEY to call the GLM-V API.

1. Get your API key: https://www.bigmodel.cn/usercenter/proj-mgmt/apikeys
2. Configure it with:

python scripts/config_setup.py setup --api-key YOUR_KEY

Security & Transparency

• Primary API key env: ZHIPU_API_KEY (required).
• Timeout env: GLM_GROUNDING_TIMEOUT (optional, seconds, default 60).
• API endpoint: fixed to official Zhipu Chat Completions endpoint in CLI implementation.
• No dynamic key name switching: the skill expects ZHIPU_API_KEY consistently.
• URL/local file handling: the skill can read local files or fetch user-provided URLs for processing/visualization; URL inputs are restricted to public http/https targets (localhost/private network targets are rejected).

Runtime Dependencies

Install dependencies before use:

pip install -r scripts/requirements.txt

Main packages used by this skill:

• requests
• Pillow
• opencv-python
• numpy
• matplotlib
• decord

System dependency for video visualization:

• ffmpeg

General workflow

	Input (image or video + Prompt)
		|
		▼
	Run glm_grounding_cli.py to get grounding results (natural language)
		|
		▼
	Return results (grounding results, visualized image or video)

How to Use

Run glm_grounding_cli.py to get grounding results

• Ground any target in an image

python scripts/glm_grounding_cli.py --image-url "URL provided by user" --prompt "description of target for grounding"

• Track any target in a video

python scripts/glm_grounding_cli.py --video-url /path/to/image.jpg --prompt "description of target for tracking" --visualize --visualization-dir "./vis"

Reply with grounding results

After receiving a grounding prompt from the user, your direct reply should be natural language that includes grounding coordinates. Coordinates $x$ and $y$ are relative values in [0, 1000], computed as:

$$ x = round(x_{pixel} / W * 1000) \ y = round(y_{pixel}/H*1000) $$

where $x_{pixel}, y_{pixel}$ are pixel coordinates with origin (0, 0) at the top-left corner of the image, and W/H are the image width/height.

Unless otherwise specified, grounding results should use the following Python data formats:

• 2D bounding boxes: [[x1, y1, x2, y2], ...], extracted grounding result is a list of boxes, each box has 4 coordinate values
• 2D points: [[x, y], ...], extracted grounding result is a list of points, each point has 2 coordinate values
• 2D polygon: [[x1, y1], [x2, y2], ...], extracted grounding result is a polygon coordinate list, each vertex has 2 coordinate values
• 3D bounding boxes: [{"bbox_3d":[x_center, y_center, z_center, x_size, y_size, z_size, roll, pitch, yaw],"label":"category"}, ...], extracted grounding result is a JSON list where each object contains a category label and one 3D box with 8 coordinate values
• Objects Detection JSON: [{'label': 'category', 'bbox_2d': [x1, y1, x2, y2]}, ...], extracted grounding result is a JSON list where each object contains a category label and one box
• Video Objects Tracking JSON: {0: [{'label': 'car-1', 'bbox_2d': [1,2,3,4]}, {'label': 'car-2', 'bbox_2d': [2,3,4,5]}], 1: [{'label': 'car-2', 'bbox_2d': [4,5,6,7]}, {'label': 'person-1', 'bbox_2d': [10,20,30,40]}]}, extracted grounding result is a JSON object whose keys are video frame indices and values are lists of JSON objects, each containing a category label and one 2D box