Computer Use

Computer Use — operate native desktop apps with Cua Driver

Wisp uses the installed Cua Driver as an MCP server. Cua Driver owns the platform-specific desktop integration; this Skill owns the agent workflow and the safety boundary. It can operate native apps, native file dialogs, and browser windows that are visible to the host desktop.

Before the first action

Use this Skill only when the Cua Driver tools are advertised in the current conversation. If they are absent, do not invent tool names or fall back to blind shell input. Tell the user to install Cua Driver, add a stdio MCP connection with:

command: cua-driver
args: mcp

Then ask them to reconnect the MCP service. The driver must run in the interactive user session. An SSH or service-session process cannot see the user's desktop. On macOS, Accessibility and Screen Recording permission must be granted to the Cua Driver app identity. On Windows and Linux, report any interactive-session or display-server refusal as a capability boundary.

If the available Cua Driver tool exposes a health, doctor, or permission status call, use it first. Otherwise the first call the task needs anyway is the connection check: launch_app for an app to open (it returns the pid and window ids), list_windows for one that is already open. list_apps enumerates every installed application; call it only to find an app whose name you do not know. A process starting successfully is not evidence that the desktop is controllable.

Tool selection

Use the exact tool schemas advertised by the connected Cua Driver server. The common names are:

• list_apps and launch_app for application discovery and startup;
• list_windows for exact process/window identity;
• get_window_state for the accessibility tree plus a window screenshot;
• get_desktop_state for the primary desktop screenshot and desktop identity;
• click, type_text, press_key, hotkey, scroll, and drag for input;
• window or session cleanup tools when the driver advertises them.

Cua Driver's schemas arrive through search_mcp_tools, and each result is large. Ask once for everything the task will plausibly need (observation, click, text, keys, scroll) instead of searching again before each new kind of action.

Do not guess a selector, process ID, window ID, element token, or coordinate. Read the current state first. For input, prefer a window target with an exact pid and window_id; use the returned accessibility element_token when the control exposes a semantic action. Use window-local pixel coordinates only when the element is not actionable semantically. Use a desktop target only for deliberate foreground screen actions.

Observe → act → verify

Work in steps. A step is one observation, the actions that observation already justifies, and one verification:

1. Discover the app and select one exact window. If several candidates match, stop and resolve the ambiguity instead of choosing by title alone.
2. Call get_window_state and keep the resulting window identity and fresh element references together. Treat element tokens as stale after a page navigation, dialog transition, window recreation, or material UI change.
3. Act on what that observation shows. Prefer background delivery when the target and platform support it. Request foreground delivery only for that action when the application requires focus and interrupting the user's desktop is acceptable.
4. Read the same target again and verify the application state or external artifact. A successful input dispatch is not proof that the application handled it.
5. If the result is stale, ambiguous, refused, or unverifiable, follow the returned refusal code and re-observe. Do not retry the same blind action.

One observation, several actions

An observation stays valid until the window's layout changes. When the next inputs all go to controls it already shows, and none of them moves, replaces, or removes those controls (a keypad, a toolbar, a row of checkboxes, the fields of one form), send them back to back, as consecutive tool calls in one turn, then observe once and verify the outcome. Wisp runs a turn's tool calls in the order written, and element tokens stay valid until the next observation of that window. Paying a model round trip and a fresh tree for every key press turns a ten-key entry into minutes.

Observe again before acting when an action navigates, opens or closes a dialog or menu, reloads a list, or recreates the window: the next action depends on a layout you have not seen. Keep an action that commits something (send, submit, delete, overwrite) out of a batch: verify what precedes it, get the confirmation it needs, then issue it alone. If a call in a batch returns an error or a refusal, the calls after it still ran; observe and reconcile before sending anything else.

Each observation is large. Do not repeat one to confirm what the previous result already shows. When the schema lets you skip the screenshot or bound the tree, do so whenever the accessibility tree alone answers the question.

Results come from the application

Report what the final observation shows or what reached the disk, never what you expected to see. Do not work out a value yourself (arithmetic, a count, a name the application will choose) and then look for it: read the real value first, and compute a cross-check with a tool if one matters.

For a native save or export, verify the actual path and file existence with a filesystem tool after the application reports completion. For a visual canvas, verify the screenshot and, where possible, an application-owned state or exported artifact. For a browser page, use browser-use page tools when they provide the needed operation; use Cua Driver for browser chrome, native dialogs, or a page surface that the browser bridge cannot access.