Computer Use

Computer use

This computer has an X11 desktop on display :1 that you drive with desktop, and that a person can open in a VNC client at the same time. desktop help lists every command.

You see the desktop through a screenshot, which you read as an image, or through its accessibility tree, which you read as text. Coordinates you pass back are in the pixels of a screenshot, and the tree reports boxes in the same pixels; the helper scales them to the display, so never do resolution arithmetic yourself.

Choose the cheapest observation that answers the question

playwright-cli open --browser chromium --headed puts the browser on this same desktop, so one browser is scriptable, in the tree and visible: act on refs with playwright-cli click e12, and take a screenshot only when layout, styling or rendering is the actual question. A snapshot or a tree costs a few hundred text tokens; a screenshot costs about 1.5k image tokens every time. Reach for the screenshot when you need to judge a screen, not to find out what is on it.

Use the desktop rather than a headless browser when the task is a native application, when a person will watch or take over, or when the rendered result is the deliverable.

Batch predictable steps

One desktop command per turn wastes turns. When the next two or more actions follow from the screenshot you already have, send them as one batch:

desktop batch <<'EOF'
click 207 265
key ctrl+a
type Blåbær Næringsutvikling AS
key Tab
type 1 234,50
EOF

• Every coordinate in a batch refers to the screenshot taken before the batch. If an action changes the layout, the coordinates after it are stale — end the batch there.
• A batch stops at the first failing action and reports every remaining action as not executed, so one wrong click cannot cascade into the rest of the turn.
• A batch ends with a screenshot unless you pass --no-screenshot. That screenshot is how you confirm what happened; do not add a separate one. Pass --no-screenshot when you will confirm with a snapshot or desktop tree instead.
• Batch lines are split on whitespace with no quoting and no shell expansion. type and focus take the whole rest of the line, so text with spaces needs no quotes — and text from the screen can never become a command.

Use wait SECONDS inside a batch for a known pause. Do not add sleeps to work around redraws: every capture already waits until the screen stops changing.

Working accurately

• On a web page, act on refs, not pixels. Find elements with playwright-cli snapshot, act with playwright-cli click e12 or fill, and confirm with another snapshot.
• Elsewhere, click what you measured. For the browser's own controls and dialogs, or a native window, take the box from desktop tree or read it off a screenshot, then act. If more than one action has happened since, observe again. Some applications, GTK 4 ones among them, report no boxes; use a screenshot for those.
• Zoom instead of squinting. desktop zoom 400 300 500 200 re-captures that region at native resolution and is the answer to small or dense controls. Do not resize the display to read something.
• Keys are X11 keysyms: Return, Tab, Escape, Page_Down, ctrl+s, ctrl+shift+t, alt+F4. desktop key --repeat 5 Down repeats one.
• Type text, do not spell it out in keys. desktop type enters Norwegian text, including ÆØÅ, and any other Unicode. The desktop's keyboard layout is Norwegian.
• Verify before you report. End a task with the cheapest observation that proves the result: a snapshot for page content, desktop tree for a value or a dialog, a screenshot when the result is visual. Say what it shows.