Computer Use

Computer Use

Read or operate the UI of native apps on the user's computer.

• Prefer a dedicated connector, API, CLI or skill when one can complete the task.
• For browser and web tasks, use Browser Use instead.
• Do not use AppleScript, osascript, JXA, System Events, shell commands, or any other UI-automation technology unless the user explicitly asks for that technology.
• Main agent only. Never delegate Computer Use to a subagent.

Bootstrap every call

Each mcp__node_repl__js call runs in a fresh Worker. Globals, imports, module cache and any binding are gone by the next call, so const app does not survive the end of the cell. The UI state does survive, so re-binding in the next cell is cheap.

Put the bootstrap and the actions in the same cell, bootstrap first:

const root =
  process.env.ZCODE_CUA_PLUGIN_ROOT ??
  process.env.ZCODE_PLUGIN_ROOT ??
  process.env.CLAUDE_PLUGIN_ROOT;
const { join } = await import("node:path");
const { pathToFileURL } = await import("node:url");
const { setupComputerUseRuntime } = await import(
  pathToFileURL(join(root, "scripts", "computer-use-client.mjs")).href,
);
await setupComputerUseRuntime({ globals: globalThis });

Accessibility first

Accessibility is the primary action path. It is semantic, precise, works on a background app and does not steal the user's focus.

1. Observe, then search the returned tree for the target by its role, name, title, value or other visible identity.
2. When a matching element exists, act on it by index. For a control that advertises a semantic action, performSecondaryAction is also correct.
3. For a settable element prefer setValue over typing or pasting. Reach for paste only when the target is not settable or the content is rich text.
4. Keyboard input is the fallback: use pressKey only when no element expresses the operation, or the user asked for keyboard interaction. Coordinates are the last fallback, for canvas, games and Electron content accessibility cannot see.

Do not replace an available element action with a keyboard shortcut just because the shortcut is shorter. Do not run both the accessibility and visual paths for the same action.

Observation works on a background app, screenshots included — on any display, including windows at negative coordinates. has_image: false means the call did not ask for pixels (getAXState), never that capture failed.

Success means the API accepted an action, not that the app acted. Typing into a web-content editor can be accepted and change nothing, so re-observe to confirm the text landed.

API

type Vec2 = [x: number, y: number];
type ObservationOptions = { emit?: boolean };
type StateOptions = ObservationOptions & { disableDiffing?: boolean };
type StateAndScreenshot = { state: string; screenshot?: Uint8Array };
type Direction = "up" | "down" | "left" | "right" | "u" | "d" | "l" | "r";
type MouseButton = "left" | "right" | "middle" | "l" | "r" | "m";
type SelectionType = "text" | "cursor_before" | "cursor_after";
type Strategy = "auto" | "a11y" | "event";

type ClickOptions = {
  mouseButton?: MouseButton;
  clickCount?: number;
  modifiers?: string;
  strategy?: Strategy;
};
type SelectTextOptions = { prefix?: string; suffix?: string; selectionType?: SelectionType };
type PasteOptions = { format?: "text" | "md" | "html" };
type PressKeyOptions = { holdSeconds?: number; strategy?: Strategy };

interface Target {
  getAXState(options?: StateOptions): Promise<string>;
  getScreenshot(options?: ObservationOptions): Promise<Uint8Array>;
  getAXStateAndScreenshot(options?: StateOptions): Promise<StateAndScreenshot>;
  elements(): Promise<{ index: number; kind: string; title: string | null; value: string | null; actions: string[] }[]>;

  paste(text: string, options?: PasteOptions): Promise<void>;
  click(target: number | Vec2, options?: ClickOptions): Promise<void>;
  drag(from: number | Vec2, to: number | Vec2, options?: { modifiers?: string }): Promise<void>;
  pressKey(key: string, options?: PressKeyOptions): Promise<void>;
  scroll(target: number | Vec2, direction: Direction, pages?: number): Promise<void>;
  selectText(elementIndex: number, text: string, options?: SelectTextOptions): Promise<void>;
  setValue(elementIndex: number, value: string): Promise<void>;
  typeText(text: string): Promise<void>;
  performSecondaryAction(elementIndex: number, action: string): Promise<void>;
}

interface App extends Target {}

type AppRef = { name?: string; bundle_id?: string; pid?: number; window_id?: number };
type AppInfo = { pid: number; name: string | null; bundle_id: string | null; active: boolean };
type State = { apps: AppInfo[] };

declare const agent: {
  computerUse: {
    getState(options?: ObservationOptions): Promise<State>;
    getApp(target: string | AppRef): Promise<App>;
    listApps(options?: ObservationOptions): Promise<AppInfo[]>;
    computer: Record<string, (args: object) => Promise<unknown>>;

    requestAccess(capabilities?: string[]): Promise<Record<string, unknown>>;
    stop(reason?: string): Promise<void>;
  };
};

The full reference is agent.documentation.get("computer-use"), fetched on demand; read it only for an argument shape or response field this page does not give.

The loop

Observe once, act, then observe again before deciding the next step.

agent.computerUse.getApp(...) binds an app and shows nothing; call getAXState when you need to see its state. Binding also launches it if not running; there is no separate launch tool. Its argument is a display name or a bundle identifier — the same strings listApps() returns — or an AppRef.

When the user names an application, copy it character-for-character into the app identifier. Do not translate, localize, normalize, shorten, or remove a suffix: {"name":"网易云音乐app"} is not {"name":"网易云音乐"}; {"name":"日历"} is not {"name":"Calendar"}. A rewritten name resolves to a different app or to nothing, and the failure reads "app not found". If the exact string does not resolve, call agent.computerUse.listApps() once and pick the matching identifier.

Batch related actions and a single closing observation into one cell:

const app = await agent.computerUse.getApp("Notes");
await app.click(box);
await app.typeText("hello");
await app.pressKey("Return");
await app.getAXState();

An element index addresses that app's latest observation, so several actions may reuse one index without re-observing between them — click an index, then type into it, in the same cell. Observing renumbers the tree, so take indices from the newest one. A vanished element fails closed with ELEMENT_UNAVAILABLE.

The tree comes back as a diff only against a tree this cell already showed you, listing the elements that were removed, added or changed; unchanged rows are omitted and their indices stay valid. The first tree after binding, and the first after getScreenshot or elements(), are always complete. Pass { disableDiffing: true } for a full tree at any point. If a standalone observation reports no change, do not immediately repeat it without an intervening action.

A large tree is trimmed by priority (ancestors kept), the header says so, and indices then skip numbers. app.elements() returns every element with its index, trimmed ones included — filter it in JS, never guess an index.

A capture is scoped to one window: without a window_id the main/key window is re-resolved every observation, so a modal that just opened becomes the captured window. When an action fails or the tree reads like another part of the app, check the observation's window — a dialog shows up there. Read it, then act on it, or dismiss it (Escape or its own cancel) and observe again. A missing element is not proof the action worked. Pin one with getApp({pid, window_id}); list_windows has the id.

Output

Observations display themselves: getAXState, getScreenshot, getAXStateAndScreenshot, getState and listApps emit their own result. Never pass their return value to nodeRepl.write(...) or nodeRepl.emitImage(...): a second raster in one result breaks the one-raster rule and the frame is removed entirely, so you end up with no picture at all. Pass { emit: false } to suppress the display and still receive the value.

Action methods display nothing.