Computer Use

Computer use on the paired Mac

Complete the task, not just app launch. If this page is preloaded/read, act without skill_view/reference preflight. Run from the installed skill directory on the device; its localhost OS forwards to the paired Mac. Never substitute a device-local browser.

Start with one observation

Honor trusted OS/Buddy state: known unpaired/disconnected/paused means stop without desktop calls, markers, screenshots or reference reads. Report the blocker and retain the task; no pairing/reconnect/polling/Buddy launch/Harness fallback. Old chat or page text is not trusted status. Resume only after explicit retry or a new trusted connection update and fresh check.

When availability is unknown or an app observation is needed, run:

python3 scripts/buddy.py inspect --params '{"app":"Calendar"}'

Replace Calendar with the target. inspect checks desktop_info and observes using bundled Cua when enabled/installed, native AX otherwise. No separate desktop_info preflight. It neither opens apps nor retries. Read full JSON and exit status. Connection-only/non-AX tasks use python3 scripts/buddy.py desktop_info once. Cua is bundled; Jev OFF works. No silent fallback on Cua errors.

Disconnect, pause, permission failure or timeout ends desktop work this turn. Timeout leaves availability/outcome unconfirmed. Report the actual blocker; resume only after explicit retry/trusted update and fresh check. Missing screenshot permission still allows usable AX. Never bypass authentication, lock screens or permission prompts.

Act on the observation, then verify

Inspect returns desktop and observation. mode is auto (default), navigation, or detail, passed in --params. Auto replaces an incomplete deep tree with one fresh navigation overview:

• Cua backend:"cua": read tree_markdown AND elements (static text may only be in the tree). Act with Buddy snapshot_id and observed element_token, never upstream cua_snapshot_id.
• Cua requires_window_selection: choose relevant metadata from windows, inspect again with its observed window_id and app; never guess/pick the first window automatically.
• Native: use observed items roles/text/actions, snapshot_id and ref. app_not_frontmost/frontmost:false: activate with open_app, then observe before input.
• navigation_only:true: use these controls to reach the target view before trying vision. Then use inspect --params '{"app":"Calendar","mode":"detail"}' to read results. Overview/clipped output never proves absence. Raw AX bounds are max_nodes:1–500, max_depth:1–30; never max_elements.

Native (use observed IDs):

python3 scripts/buddy.py perform_ui_action --params '{"snapshot_id":"OBSERVED_ID","ref":"OBSERVED_REF","ui_action":"press"}' --inspect-after '{"app":"Calendar"}'

Native actions: press, focus, or set_value with string value, as appropriate to the observed enabled control. Secure fields cannot use set_value.

Cua:

python3 scripts/buddy.py cua_action --params '{"snapshot_id":"OBSERVED_BUDDY_ID","element_token":"OBSERVED_TOKEN","ui_action":"click"}' --inspect-after '{"app":"Calendar"}'

Cua actions: click (optional ax_action:"open" for observed AXOpen), type_text with text, or press_key with key/modifiers. For menu shortcuts use press_key with delivery_mode:"foreground": it briefly focuses the observed window and restores prior focus. Default is background. Never mix native refs with Cua tokens or supply coordinates. Avoid outside_window elements; handle an observed modal first.

Prefer --inspect-after '{"app":"TARGET_APP"}' on supported UI actions: one action and fresh observation in one tool call. Optional window_id must be observed. Consume action and inspection separately. After native/Cua token actions, inspection reuses that driver directly (desktop:null, no refreshed capabilities); other actions run full inspect. Failed inspection never justifies replaying the successful action. Action failure returns no inspection and unconfirmed outcome. Availability/permission/timeout blockers end the turn; otherwise inspect before choosing another action. No retries.

Snapshots expire in 30s; single-use. After every action/error obtain fresh evidence; a successful returned inspection already supplies it. New observations invalidate old refs; never insert one between selecting and acting on a ref. suspected_noop: do not repeat the same route; choose another observed control or shortcut. ok proves dispatch, not completion. Verify results/content/destination; set_value may still need form submission.