Linux Desktop Control

computer-use-linux

Use computer-use-linux when an agent needs to observe or operate a local Linux desktop: inspect the accessibility tree, list/focus windows, take screenshots, click, scroll, type, press keys, or invoke AT-SPI actions.

When to Use

Use this skill when:

• The user wants the agent to control a Linux GUI app.
• You need desktop state from AT-SPI, screenshots, or compositor window metadata.
• You are configuring the computer-use-linux MCP server for your agent.
• A desktop action needs target-aware input instead of blind shell commands.

Do not use this for remote browsers, websites, or headless automation when a browser-specific tool is available. Do not assume desktop actions are safe just because the MCP connection works.

Install

Pick the install that matches how you will run this skill. You can use both.

Claude Code or Codex plugin

claude plugin marketplace add agent-sh/computer-use-linux
claude plugin install computer-use-linux@computer-use-linux
# or, in Codex:
codex plugin marketplace add agent-sh/computer-use-linux
codex plugin add computer-use-linux@computer-use-linux

The plugin registers the MCP server and downloads the pinned, sha256-verified release binaries on first start. It does not put computer-use-linux on PATH; shell commands in this skill need the CLI install below. Set COMPUTER_USE_LINUX_BIN to run a local build instead.

Pi native tools

pi install npm:@agent-sh/computer-use-linux

This enables Pi's computer_use_linux_* tools. It does not put computer-use-linux on PATH. Shell commands in this skill (doctor, setup, setup-window-targeting, guard-accessibility, the MCP command config, and Verification) need the CLI install below.

Shell CLI / MCP server

Use this when you need computer-use-linux on PATH for the commands in this skill.

npm install -g @agent-sh/computer-use-linux
computer-use-linux doctor | jq .readiness

Rust users can install from crates.io:

cargo install computer-use-linux
computer-use-linux doctor | jq .readiness

If doctor reports missing input or accessibility support, run:

computer-use-linux setup
computer-use-linux setup-window-targeting
computer-use-linux doctor | jq .readiness

If doctor selects ydotool as the input backend, also enable its per-user daemon with systemctl --user enable --now ydotoold. Direct uinput, X11 xdotool, and RemoteDesktop portal input do not require ydotoold.

On niri, export the session's NIRI_SOCKET to the server. Window listing and exact focus use niri msg with a direct IPC fallback. Missing positions or unknown/mixed output scaling leave window-relative coordinates unavailable; focus the target and inspect a fresh full-screen screenshot before using desktop coordinates.

On GNOME Wayland, log out and back in after setup-window-targeting if the GNOME Shell extension was newly installed.

For MCP hosts with COMPUTER_USE_LINUX_NOTIFY_ON_COMPLETE=1, call the optional complete_interaction tool once after finishing desktop interaction. A skipped cue is not a task failure. This notification does not guarantee exclusive desktop ownership or that other clients have stopped sending input. This applies only to directly spawned MCP hosts, not the native Pi extension.

setup_accessibility verifies the saved GNOME toolkit-accessibility key separately from runtime AT-SPI. Inspect its warning and readback before assuming new apps can expose trees. Other accessibility tools may change the key later; setup does not hold it enabled continuously.

Optional foreground accessibility guard

Skip unless: the user explicitly wants GNOME's saved toolkit-accessibility setting kept enabled while desktop automation runs.

Run computer-use-linux guard-accessibility in a foreground terminal. It registers a passive AT-SPI window-activation listener and watches/reasserts the saved key with readback. The setting affects all apps for the current user. mcp, setup, and get_app_state never start this guard automatically. Stop with Ctrl-C or SIGTERM before intentionally disabling accessibility. Stopping ends writes and removes its listener without disabling other clients or restoring a previous saved value. Apps launched during a reset/reassertion race may still need restarting; do not claim a complete GNOME toggle fix.

Configure Your Agent

The computer-use-linux binary is an MCP server. Configure it as a stdio MCP server in your agent of choice:

{
  "command": "computer-use-linux",
  "args": ["mcp"]
}

If the binary is not on PATH, use the absolute path (typically ~/.local/bin/computer-use-linux or the npm global bin directory). Pi native tools skip this MCP command config; see Pi setup.

Host-specific guides

• Hermes setup
• Pi coding agent setup

Procedure

1. In Pi, call computer_use_linux_tools with the exact tools or capability you need. Enabled tools use the computer_use_linux_<name> prefix, appear starting on the next model turn, and remain active for the session.
2. Begin every desktop-control turn with get_app_state, scoped to the app you are working in: pass app_name_or_bundle_identifier or a window target (window_id, pid, app_id, wm_class, title). Without a target the result is the whole desktop AT-SPI tree, tree_scoped is false, and message warns; that can flood context. Use include_screenshot: false when the accessibility tree is sufficient. If accessibility_tree_truncated is true, the tree is incomplete: scope to a narrower target and raise max_nodes or max_depth (hard caps 2000 and 64) rather than lowering them. The compact readiness block identifies missing setup.
3. Use doctor only when you need the full diagnostic report.
4. If can_build_accessibility_tree is false, run setup_accessibility and restart the target app.
5. If can_query_windows is false on GNOME Wayland, run setup_window_targeting and ask the user to log out and back in if setup says the shell extension needs a reload.
6. Before targeted input, call list_windows or focused_window and verify the intended window by title, app id, pid, or wm class.
7. Prefer semantic targeting from get_app_state: use element indices or role/name/text/states selectors.
8. Use coordinates only when the UI surface has no useful accessibility tree.
9. For text input, prefer type_text with a target selector (window_id, pid, app_id, wm_class, title, tty, terminal_pid, terminal_command, or terminal_cwd) rather than relying on current focus.
10. After mutating actions, re-check state with get_app_state, focused_window, or an app-specific readback.

Plain left element/index/selector click prefers native AT-SPI click, press, or toggle over toolkit bounds, avoiding coordinate conversion when available. This preference does not replace a coordinate click with an arbitrary action name. Explicit x/y, right clicks, and double/multiple clicks retain pointer semantics. Use perform_action explicitly for entry activate or slider jump; click never substitutes those actions, including when bounds are unavailable.