Resilient Computer Use

Resilient Computer Use

Run UI automation as a repeated observe → identify → act → re-observe → verify loop. Never equate tool context with application reality: a lost binding, empty accessibility tree, or stale element index does not prove the app, window, tab, or user state disappeared.

Chrome default: bind to a tab

The user keeps using Chrome while agents work and runs multiple Codex browser sessions concurrently. Default to tab-bound Chrome control by stable tab ID. Keep observations, screenshots, clicks, typing, navigation, and any network/CDP capture scoped to that tab handle. Do not use the frontmost Chrome window, active tab, window position, or global keyboard shortcuts as the target for ordinary browser work.

• Give each chat its own named browser session and dedicated tab; reuse that chat's binding throughout the task. Claim an existing tab only when it is the intended target and another chat is not operating it. Do not take over another chat's tab.
• A separate tab or window does not isolate cookies, login, or account state within the same Chrome profile. Coordinate account switches and sign-outs that could affect other work; do not change profiles silently.
• If the binding fails, refresh the tab inventory, recover the exact tab, and read the adapter's troubleshooting before switching to desktop control. A failed create-tab command does not prove existing tabs cannot be claimed.
• Native desktop control is a fallback for unsupported native dialogs or a binding that cannot be recovered. Before using it, explain that it requires shared focus and coordinate a brief exclusive interval with the user or other authorized chats. Stop if focus changes, then return to the tab handle as soon as possible. Do not repeatedly steal focus to continue a background task.

Model the layers separately

Diagnose the failing layer before changing anything:

1. Tool transport — Computer Use or a dedicated plugin can communicate.
2. Application process — the intended app is running.
3. Window and focus — the intended window, monitor, sheet, or popup is active.
4. Document or tab — the exact file, record, URL, browser tab, or account is selected.
5. UI snapshot — accessibility text, screenshot, and element indices describe the current state.
6. Operation state — the intended action is pending, in progress, completed, failed, or ambiguous.
7. Persisted state — the external system actually saved the result.

Do not repair a higher layer until the lower layers are known. For example, do not reopen a URL because a tab binding was lost, and do not repeat Submit because the local click response was missing.

Keep a target identity capsule

Before a multi-step workflow or write, retain the strongest available identity:

var uiTarget = {
  app: "Google Chrome",
  bundleId: "com.google.Chrome",
  windowTitle: "Channel content - YouTube Studio",
  tabId: "684841213",
  providerTabId: '["browser-instance","684841213"]',
  urlOrDocument: "https://studio.youtube.com/channel/...",
  resourceId: "channel-or-video-id",
  accountMarker: "expected account or channel",
  expectedState: "editing one unlisted video"
};

Use only fields that exist. Prefer stable resource IDs over titles and titles over window order. Treat tab IDs as stable for the life of that tab/browser instance, not across an actual browser restart.

Select the control surface

Use the narrowest reliable surface that preserves the user's intended session:

1. Use a purpose-built connector, API, or CLI for semantic operations when available and when the user did not explicitly request visible UI interaction.
2. Use a dedicated app/browser plugin when the task depends on its live authenticated context.
3. For Chrome, recover the tab binding first. Use native Computer Use on the same app and profile only under the shared-focus fallback rule above.
4. Prefer accessibility actions. Use screenshots and coordinates only when accessibility is incomplete or incorrect.
5. Do not use AppleScript, shell-driven UI automation, or another automation technology unless the user explicitly requests it.
6. Ask the user to intervene only after supported state refresh, identity recovery, same-app fallback, and tool-specific troubleshooting fail.

Never silently switch browser family, browser profile, desktop app, account, or authenticated destination.

Finish the requested UI outcome

Treat local preparation as an intermediate checkpoint, not completion. If the user asked to attach, upload, save, publish, submit, or otherwise finish a UI workflow, do not stop at “the file is ready,” “select it manually,” or “the browser binding cannot handle the picker” while a supported Computer Use path remains.

• At a native-control boundary, first use the documented tab-bound chooser or download interface when available. If native Computer Use is required, coordinate shared focus as described above before operating it.
• Continue in the same turn through attachment and post-action verification when authorization permits.
• Do not merely promise to use the fallback later. Invoke it and report observed results.
• If a named Computer Use tool is not visible, check the current agent's supported desktop interfaces before calling the capability unavailable. Do not assume another agent's runtime or package is installed.
• If the first fallback attempt fails, re-observe and try the next supported same-app path. Escalate only with the exact attempted methods and current UI state.
• During a stalled or long handoff, provide a concrete progress update within 60 seconds and keep working.

The fallback changes the control surface, not the task scope, destination, account, or confirmation requirements.

Reliable operating loop

1. Initialize or reuse

• Reuse persistent tool and app bindings when valid.
• For Chrome, initialize the documented browser adapter and bind the exact tab first. For native applications, initialize Computer Use and inspect the named app.
• If a display-name lookup fails, retry with the bundle identifier before broader discovery.
• Do not reset the JavaScript session as a first-line recovery; resets destroy useful identity and operation state.

2. Observe fresh state

• Fetch state from the bound tab before browser actions; fetch app state before native actions.
• Use the accessibility tree for identity and controls; inspect the screenshot when layout, focus, overlays, canvas content, native dialogs, or multiple monitors matter.
• If the returned accessibility text is only a diff and earlier state is unavailable or suspect, request a fresh full state.
• Treat empty or partial accessibility text as incomplete evidence, not proof that the UI is blank.

3. Verify the target

Confirm the app, window/document/tab, resource ID or URL, account, and current operation state. When multiple candidates exist, do not choose by first match, visual order, frontmost monitor, or approximate title.

4. Perform the smallest action

• Prefer one semantic accessibility action at a time.
• Re-fetch state after focus changes, navigation, modal transitions, file selection, user interruption, or any action that can rerender the UI.
• Never reuse an element_index after refreshing state.
• Avoid typing text containing newlines into controls where Return may submit. Use set_value when appropriate, or enter multiline content only after verifying the editor behavior.

5. Verify the result

Check an observable postcondition: changed value, URL, selected state, enabled button, progress indicator, saved banner, resulting record, or external readback. A successful click or lack of an error is not completion evidence.

6. Checkpoint long workflows

After each item, record target identity, intended action, completion evidence, and next item. Resume from persisted evidence rather than memory or screen position.

Recovery matrix