Ade Computer Use

ADE computer use

This is the one entry point for computer use in ADE. Pick the surface first, then follow that surface's loop.

Pick the surface

Pick App Control for an Electron app whose DOM you need: it acts through the DOM, so labels, selectors, test ids and hover work, and it records and proves the app's own window. That covers your dev build (ade app-control launch) and an installed app such as Slack or a packaged build of the app you work on: start it with a debug port, then attach (ade mac-desktop open "<App>" -- --remote-debugging-port=9333, then ade app-control connect --cdp-port 9333). Otherwise Mac Desktop drives an Electron window by point, like any other app.

ADE's browser is the default for web work. Attach to the user's own browser only when they ask for it, never on your own, and tell them which machine and tab you attached to.

Lane screens (ade screen) exist on macOS and Windows hosts only. On a Linux host, ade screen status says "Lane screens are not supported on Linux yet"; use ade browser and ade app-control there.

Only ADE's surfaces stay off the user's screen. Anything else that clicks, types or captures — an MCP server, osascript or System Events, cliclick, screencapture, a browser-automation CLI, a Chrome you open yourself — acts on the user's real screen, pointer and focus. Do not use them for task work. When an ADE surface cannot do a step, say which step and why, and ask; do not switch tools silently.

Note: ade desktop is a different command. It launches the ADE desktop app.

A device on another machine

ade apple, ade mac-desktop and ade app-control take --machine <name> to drive that machine's simulator, screen or app (--lane <lane there>; use --machine-project to pick the project, since ade apple already uses --project). proof captures there and files the still in YOUR proof drawer. Recordings stay on that machine. The person at that Mac sees "Driven by an agent on <your machine>" and can end your session; stop it yourself when done. If the app needs building, start a subagent on that machine instead (ade chat create --machine …, see ade-cli-control-plane).

The same answer on every surface

Every surface works the same way: observe, act on a handle from the last observation, then check the result. Every acting command (click, type, press, fill, scroll, drag) reports two lines:

• hit: — the element the command actually hit.
• effect: — observed (ADE saw the screen change), unconfirmed (ADE looked and saw no change), not_checked (ADE did not look), or, in the browser, waiting (the action started a navigation that waits for the user to allow this chat to use the ADE browser; observe again after they answer).

When the effect is unconfirmed, observe again before you act again or report the step. Do not repeat the action blindly.

An unconfirmed answer can carry a third line, next:. It names the one input method to try next and why, and often a command to run:

• observe — keep the same method. Look at the new screen, or wait for the label you expect. The action probably applied, or the element is disabled.
• background_input — run the click it prints: the same click at the element's point, delivered to the app alone. No approval, no pointer moves.
• real_input — repeat the same command with --real.
• lease — real input is the fix, but this chat must ask for it first. Run the ade mac-desktop lease command it prints, then repeat with --real.
• browser — the target is a web page. Drive it in the ADE browser, unless the task needs that exact browser.

Follow next: instead of a guess. Today only the Mac Desktop prints it.

Proof per surface

ade proof capture and ade proof record capture the lane's display. They refuse the user's real screen unless you pass --real-screen. Pass it only when the user asks for proof of their own screen. The ade-proof-artifacts skill covers attaching files and confirming a filing.

Each of these commands prints a cite: line, such as ![Settings saved](ade-proof://<id>). Paste it into your answer directly under the claim it proves: the picture shows there, and a video plays there. For a before/after, use a proof-compare block (see ade-proof-artifacts). Caption each item, and say what it does not show.

Show the user

ade ui show mac-desktop     # also: apple, browser, proof, app-control
ade mac-desktop show --text
ade mac-desktop show --floating --text
ade app-control show --floating --text

Mac Desktop

Each lane owns a private macOS screen. ade mac-desktop ... creates it, parks apps on it, and drives them through the Accessibility API. The user's real display and real pointer are never touched, so you never have to ask before clicking — with one exception, the input lease below.

For web tasks, use ADE's built-in browser (ade browser, the ade-browser skill) by default. Open Safari or another browser here only when the user names that app or asks for the Mac Desktop.

macOS runtime hosts only. ade mac-desktop status --text answers everywhere; every other command refuses off macOS with MAC_DESKTOP_UNSUPPORTED_PLATFORM. --lane defaults to ADE_LANE_ID. In a chat, the command is pinned to that chat's lane: --lane naming a different lane is refused, not silently swapped.

Windows Desktop

On a Windows host, ade screen is Windows Desktop (ade windows-desktop and ade mac-desktop are aliases of the same Windows screen; every output says "Windows Desktop"). The verbs are the Mac ones below, with the Windows notes here. Start every task with ade screen status --text: it names the seat, who holds the private screen, whether setup and the saved password are done, whether the PC is locked, and a next line with the exact command or ask.

The two seats

• private (default): a separate Windows session of the user's own account. It shows their wallpaper and taskbar, but it is not their screen. It has its own pointer, keyboard and foreground, so point clicks, drags and --real need no lease. One lane holds it at a time.
• shared: the user's main desktop, only after the user says yes. Actions take over the window the user is using. Real input there is covered by the user's yes; only one shared lane drives at a time (MAC_DESKTOP_LEASE_HELD_BY_OTHER names the other lane: wait and retry).

What needs the user — ask, never do it yourself

start --shared puts an Allow / Don't allow card in your chat and waits; on Allow it starts the shared seat. Only the user's button press counts: typed text is not an answer, and an agent that tries to answer the card itself (or the real-input lease card) is refused. If nobody answers within about three minutes the card is withdrawn and the command fails; ask again later. The same chat is not asked twice. Never pass --consent or --allow-prompt: those are the user's clients' flags and an agent is refused. A command with no chat (a plain shell outside one) cannot act on the main desktop at all; only the user's own ade --role cto can. Never use that role yourself; a holderId you pass is stripped.

Windows specifics

• Long text: on the private seat, and for --real typing, Windows types one character at a time, so type takes about 25 ms per character and refuses more than 4,000 characters in one call; split longer text.
• Browsers: ade screen open chrome (or msedge) on the private seat gets a lane-private profile, not the user's. open prints the windows that appeared; if none did it says why (for example the request was handed to an instance that was already running) and what to run next. Do not invent a --user-data-dir. For web tasks prefer ade browser.
• Editors restore the user's tabs. Notepad reopens the user's open files, which can hold secrets. Create a blank file for the task and open that one: ade screen open notepad -- C:\Users\<you>\AppData\Local\Temp\ade-proof.txt --text. Then observe (or look at the screenshot) before you record or file proof, and stop if anything private is visible.
• Modifier keys: --ctrl, --alt, --shift, --win. --cmd is sent as Ctrl. Example: ade screen press s --ctrl --text.
• Windows: ade screen focus --window <id> raises one of your lane's windows, minimize --window <id> hides it, close --window <id> closes it (it reports closed no when a save prompt kept it open). Only your lane's windows; ids come from ade screen windows --text.
• Paths print with backslashes; quote them. --out must be in the lane worktree or %TEMP%.

Proof on Windows

ade screen observe --text                       # check nothing private shows
ade screen record start --caption "Notepad saves hello" --text
ade proof step "Type hello"
ade screen type "hello" --text
ade screen wait --label "hello" --text
ade screen record stop --text                   # prints duration and a cite line
ade screen proof --caption "hello typed in a blank Notepad file" --text

Paste the cite line under your claim. The shared seat files proof the same way.

Check each step before you report it

A click, type or press that returns ok only means ADE sent the input. It does not mean the app did what you wanted. After each step that matters, confirm it: read the screen the command printed, ade mac-desktop wait --label "<text>", ade mac-desktop observe, or a screenshot. Report only what you confirmed. If a step failed, say which step, and do not describe the result you meant to get. Before record stop, confirm the final state, so the video ends on it.

Common tasks

Use these directly; you do not need --help for them.

Operating loop

1. Start the screen and put an app on it

ade mac-desktop status --text
ade mac-desktop start --text
ade mac-desktop open "Preview" --text
ade mac-desktop windows --text

open takes an app name, a bundle id, a path, or a URL; everything after -- is the app's own argv. Run start first: open and the rest refuse with "Start one first" when the lane has no display. Viewing the screen does not start it, and an idle display with no windows and no viewer closes itself after a while. open starts a separate, blank copy of the app for the lane: no restored windows, tabs or documents, and never a process shared with the user's windows. The copy still shares that app's data with the user (cookies, history, recent files). To adopt a window that is already running, claim it:

ade mac-desktop claim --window <id> --text
ade mac-desktop release --window <id> --text

release gives the window to the user. For an app the lane opened, the user gets the whole app: all its windows move to the user's screen, the lane stops watching it, and stop no longer quits it. A claimed window goes back alone.

Window ids die with their process. Re-run windows rather than caching one.

open answers before the app has a window (watching: yes, no windows yet). Wait for it before you observe — ade mac-desktop wait --label "<window title>" --timeout 8000 --text, or re-run windows. A document app such as TextEdit opens an empty untitled document. An app that opens no window when launched bare (Preview): open a file with it, or press ⌘N (ade mac-desktop press n --cmd --text) once the app is up. MAC_DESKTOP_NO_WINDOW means the display exists but has no window yet: open an app or claim a window; do not run start again.