Computer Use

This skill provides the computer_use_* action tools for controlling a connected desktop. CU tools run through the main agent loop via HostCuProxy.

The skill is internally preactivated for conversations with a connected desktop client.

Tools in this skill are proxy tools. Execution is forwarded to a connected desktop client or handled by the assistant's virtual desktop.

Observations

Every computer-use step returns the accessibility tree. Every action also returns a screenshot taken after it ran (one at the end of a computer_use_sequence), so check it to see what your action did. An observation comes with a screenshot on a desktop's first look, when it is window-scoped, or when you pass include_screenshot: true. Ask for one whenever the tree is not enough to act on: a canvas, a game, a custom-drawn view, few or unlabeled controls, or a layout question.

The tree is walked to a limited depth to keep steps fast, and says when it was cut off. If the element you need is not in it, call computer_use_observe with full_tree: true.

Coordinates and live editing

Prefer element IDs from the latest observation. Click, scroll and drag coordinates are screen points, not screenshot pixels. For a desktop screenshot, use the per-axis conversion in its observation metadata. If you crop or resize the image, map back to the original screenshot first. Observe again after the layout changes; an older image is not a current target.

During live editing, use the app's zoom and track-height controls, then inspect the fresh screenshot. Avoid installing image libraries or repeatedly cropping and measuring screenshot pixels just to locate a UI control or waveform. Reserve scripted image or media analysis for tasks that actually require it. If an action does not produce the expected visible result, check focus and the coordinate mapping before repeating it.

Waveform pixels do not reveal spoken words. If an edit depends on audio you cannot hear or a boundary you cannot see, ask for a timestamp, a user-positioned playhead, or the source media. Do not guess the cut point or claim an edit succeeded without observing its result.

Scripting apps (macOS)

Reach for computer_use_run_applescript first when an app can be driven by script. It does not take the cursor.

Try the app's own scripting dictionary before anything else. Ask it for the state you want, not for the clicks that would produce that state. Finder, Mail, Music, Notes, Safari, Terminal and many third-party apps have one, and where there is a dictionary the whole task is usually a sentence with no window to open, no tree to read and nothing to click:

tell application "Finder"
  set target of front window to desktop
  set current view of front window to list view
  set sort column of list view options of front window to name column
end tell

System Events menu clicking is the fallback, for apps with no dictionary entry for what you need: click menu item "Split Clip" of menu "Modify" of menu bar 1 inside tell application "System Events" to tell process "iMovie". It is UI automation in a script's clothes: it still depends on the menu sitting where you expect and on the app being frontmost. A menu item that needs a selection or a playhead position does nothing when that context is missing, so set it up first, and read enabled of menu item when unsure.

Click and type for everything a script cannot reach. host_bash is for shell commands, not for driving apps.