Computer Use

node_repl + @qwen-code/cua-sdk (Computer Use)

• Use node_repl (JavaScript) for all Computer Use actions.
• If both servers are available, use node_repl from the desktop-node-repl MCP server; the regular server runs on the Qwen Code host.
• Do not use other technologies besides node_repl for computer interactions, unless specifically requested by the user (e.g. AppleScript, osascript, JXA, System Events, synthesized input).
• Prefer a dedicated plugin or skill when it can complete the task; use Computer Use for app interactions that are not exposed through a more specific interface.
• node_repl state is persistent across calls.
• For text output, use nodeRepl.write(...). It takes a string; use JSON.stringify(...) only for textual metadata. For an observation, write its .text and emit its screenshots as images; do not stringify an observation or driver result containing image bytes.
• Omit yield_time_ms for ordinary UI calls to use the default 10-second wait. A shorter yield does not speed up the action and can add a node_repl_wait round. Use a shorter yield only when you need control back before completion; if a cell is still running, collect its result with node_repl_wait before issuing dependent actions.

Forwarding results in Codex code mode

When calling node_repl through tools.* inside Codex's outer functions.exec, forward each returned content block by its type. nodeRepl.emitImage(...) produces an MCP image block; the outer script must pass that block to image() for the model to receive an image. Prefer the desktop relay tool when it is present; otherwise use the regular node-repl server:

// A code-mode `tools` object throws on an unknown key, so probe with `in`:
// reading an unbound tool would abort the script before any fallback ran.
const DESKTOP_NODE_REPL = 'mcp__desktop_node_repl__node_repl';
const nodeReplTool =
  DESKTOP_NODE_REPL in tools
    ? tools[DESKTOP_NODE_REPL]
    : tools.mcp__node_repl__node_repl;
const result = await nodeReplTool({ code });
for (const block of result.content ?? []) {
  if (block.type === 'text') {
    text(block.text);
  } else if (block.type === 'image') {
    image(block);
  }
}

Here code is the JavaScript to run in the persistent Node REPL. Keep the forwarding loop in the outer code-mode script, outside that code string. Apply the same loop to node_repl_wait results: images may arrive only when a running cell completes. Forward text blocks too, including running-cell IDs and errors. Direct MCP tool calls do not need this outer forwarding loop.

Do not use text(result), text(block), or JSON.stringify(result) to forward an MCP result containing images: this turns image base64 into text, consuming context without showing the image. image() accepts one image block, not the whole result, so do not use image(result) either.

Bootstrap

If desktop-node-repl is connected, skip the installation commands below: it already provides node_repl and the SDK on the connected computer. Continue with the computer initialization below. Otherwise, if node_repl is unavailable, run:

qwen mcp add --scope user node-repl npx -y @qwen-code/node-repl-mcp@0.1.7
npm install --no-save --package-lock=false @qwen-code/cua-sdk@0.20.11

Tell the user to restart Qwen Code, then stop. If only the SDK import is missing, run the second command and retry.

Reuse an existing computer connected to the intended desktop. Otherwise import once per fresh node_repl session. The same App workflow below applies to macOS, Linux and Windows; no additional platform resource is needed:

globalThis.computer = await (
  await import('@qwen-code/cua-sdk/computer-use')
).ComputerUse.create();
var platform = await computer.getPlatform();
nodeRepl.write(`Connected platform: ${platform}`);

Use the connected platform for shortcuts, not the CLI or Node host operating system. When the task identifies an app, combine initialization with computer.getApp() and its first getState() in the same call. Read that state before editing or input.

API surface

type Point = number | { x: number; y: number };
type ComputerUse = {
  getPlatform: () => Promise<'macos' | 'linux' | 'windows'>;
  getApp: (nameOrIdentifierOrPath: string) => Promise<App>;
  listApps: () => Promise<
    Array<{ id: string; displayName: string; isRunning: boolean }>
  >;
  close: () => Promise<void>;
};
type App = {
  getState: (options?: {
    disableDiff?: boolean;
    includeScreenshot?: boolean;
    maxTextChars?: number;
  }) => Promise<State>;
  click: (
    point: Point,
    options?: { button?: 'left' | 'right' | 'middle'; count?: number },
  ) => Promise<object>;
  doubleClick: (point: Point) => Promise<object>;
  rightClick: (
    point: Point,
    options?: { modifier?: string[] },
  ) => Promise<object>;
  setValue: (element: number, value: string) => Promise<object>;
  performSecondaryAction: (element: number, action: string) => Promise<object>;
  typeText: (text: string) => Promise<object>;
  paste: (
    text: string,
    options?: { format?: 'text' | 'md' | 'html'; signal?: AbortSignal },
  ) => Promise<object>;
  selectText: (
    element: number,
    text: string,
    options?: {
      prefix?: string;
      suffix?: string;
      selection?: 'text' | 'cursor_before' | 'cursor_after';
      signal?: AbortSignal;
    },
  ) => Promise<object>;
  pressKey: (
    key: string,
    options?: { modifiers?: string[] },
  ) => Promise<object>;
  hotkey: (keys: string[]) => Promise<object>;
  scroll: (
    point: Point,
    options: { direction: 'up' | 'down' | 'left' | 'right'; amount?: number },
  ) => Promise<object>;
  drag: (options: {
    fromX: number;
    fromY: number;
    toX: number;
    toY: number;
  }) => Promise<object>;
};
type State = {
  app: string;
  window: string;
  mode: 'full' | 'diff' | 'no_change';
  text: string;
  screenshot?: { images: Array<{ mimeType: string; dataBase64: string }> };
};

Workflow

paste and selectText currently require macOS. On Linux/Windows, use the shared typeText, setValue, pressKey and observed actions. Unsupported text methods fail with unsupported_platform; do not retry them as window failures.

1. Initialize

If initialization already bound the task's app and returned its state, reuse that app and observation. Otherwise, bind the app named by the task, then read its state. getApp() binds identity; getState() can open a discovered stopped app. Combine these steps in one Node REPL call:

var app = await computer.getApp('Microsoft Excel');
nodeRepl.write((await app.getState()).text);

The app handle tracks its current window and dialog. Read the returned window title to confirm the intended document. If the app is unknown or its name is ambiguous, discover applications with computer.listApps() and use a matching application id when it distinguishes the app. Two running instances may have the same ID; retrying that ID cannot resolve the ambiguity. Ask the user to keep only the intended instance open rather than guessing a target or retrying it.

AX text uses short numeric IDs, such as [37] TextField "Name". Use IDs from the current observation for element actions. IDs can change when the app's window or session changes. Disabled and static-text rows are observation-only.

For token efficiency, the accessibility tree will be returned as a diff when appropriate. Prefer this default diff output. A full state replaces the previous state; a diff updates it; no-change preserves it. If you need a full replacement, use disableDiff: true only when the previous state is unavailable or no longer useful. Do not disregard the text and then assume that a subsequent diff will reproduce the information you skipped.

Returned text defaults to at most 12,000 characters. Set maxTextChars (minimum 512) to adjust the limit. A truncation notice means some captured rows were omitted; request app.getState({ disableDiff: true, maxTextChars: 24000 }) when you need more full text. An omitted row does not prove an element is absent. Traversal-limited captures cover only the captured nodes: identical captures can return no-change, while changes return full captured state. Use current captured IDs; after a read failure, use only IDs from the latest observation.