Argent Device Interact

Unified tool surface

All interaction tools below accept a udid parameter and auto-dispatch iOS vs Android based on its shape (UUID → iOS simulator, chromium-cdp-<port> → Chromium (CDP) app, anything else → Android adb serial). You use the same tool names on every platform.

Chromium (CDP) app = an Electron app or a Chromium-family browser (Chrome/Brave/Edge) exposing a Chrome DevTools Protocol endpoint. The same describe/tap/keyboard/screenshot surface drives it, but scrolling, tabs, cookies and storage differ — read references/chromium.md before driving a chromium target.

1. Before You Start

If you delegate simulator tasks to sub-agents, make sure they have MCP permissions.

Use list-devices to get a target id. Results are tagged with platform (ios, android, or chromium); booted/ready devices come first. Pick the first entry that matches the platform you need — if none are ready, call boot-device with udid (iOS), avdName (Android), or electronAppPath (boots an Electron app as a chromium device). A Chromium browser already running with a CDP port shows up directly — no boot-device needed. See argent-ios-simulator-setup / argent-android-emulator-setup for full setup flow.

Load tool schemas before first use. Gesture tools (gesture-tap, gesture-swipe, gesture-pinch, gesture-rotate, gesture-custom) may be deferred — their parameter schemas are not loaded until fetched. Always use ToolSearch to load the schemas of all gesture tools you plan to use before calling any of them. If you skip this step, parameters may be coerced to strings instead of numbers, causing validation errors.

2. Best Practices

1. Always refer to tapping_rule from your argent.md rule before tapping.
2. Before performing interactions, consider whether they can be dispatched sequentially - more on that in run-sequence.
3. Use gesture-swipe for lists/scrolling, not gesture-custom, unless you need non-linear movement. On Chromium use gesture-scroll instead — gesture-swipe is touch-only. Consider whether you need multiple swipes, if yes - use run-sequence. Pass momentum: false when the swipe should decelerate before ending for a precise movement.
4. Tap a text field before typing, then use keyboard to enter text.
5. Coordinates are normalized — always 0.0–1.0, not pixels.
6. For app navigation, use the element tree returned after each action (--- Elements after action (describe) ---); call describe only when no fresh tree is available for the current screen. It works on any screen without app restart. Do not navigate from screenshot pixels on regular in-app screens unless the tree failed to expose a reliable target. Use native-describe-screen only when you need app-scoped UIKit properties.

3. Opening Apps

Never navigate to an app by tapping home-screen icons. Use launch-app or open-url — they are instant and reliable.

launch-app — by bundle ID

{ "udid": "<UDID>", "bundleId": "com.apple.MobileSMS" }

Common IDs: com.apple.MobileSMS (Messages), com.apple.mobilesafari (Safari), com.apple.Preferences (Settings), com.apple.Maps, com.apple.Photos, com.apple.mobilemail, com.apple.mobilenotes, com.apple.MobileAddressBook (Contacts)

open-url — by URL scheme

{ "udid": "<UDID>", "url": "messages://" }

Common schemes: messages://, settings://, maps://?q=<query>, tel://<number>, mailto:<address>, https://... (Safari)

4. Choosing the Right Tool

5. Finding Tap Targets

IMPORTANT. When moved to a different screen after an action or do not know the coordinates of component, always perform proper discovery first.

Point follow-up native diagnostics after you already have a candidate point:

• native-user-interactable-view-at-point: deepest native view that would receive touch at a known raw iOS point; requires bundleId
• native-view-at-point: deepest visible native view at a known raw iOS point; requires bundleId

If describe Tool Fails

Read the exact error and choose the action that matches it:

• Error mentions ax-service not available or daemon startup failure: the ax-service daemon could not start. Check that the simulator is booted. Use screenshot as a temporary fallback, or use native-describe-screen with an explicit bundleId if the app has native devtools injected.
• describe returns an empty element list: the screen may be blank, loading, or showing content without accessibility labels. Use screenshot to see what is visible, then retry after the content has loaded.
• describe succeeds but is not detailed enough for a React Native app: use debugger-component-tree next.
• You need app-scoped inspection with full UIKit properties (accessibilityIdentifier, viewClassName): use native-describe-screen with an explicit bundleId. This requires native devtools (dylib) injection.
• You already have a candidate point and want to confirm what would actually receive touch: use native-user-interactable-view-at-point. Use native-view-at-point when you want the visually deepest view instead of the hit-test target.

6. Tool Usage

gesture-tap — Single tap at a point

{ "udid": "<UDID>", "x": 0.5, "y": 0.5 }

Coordinates: 0.0 = left/top, 1.0 = right/bottom.

Before tapping near the bottom of the screen in React Native apps, check that "Open Debugger to View Warnings" banners are not visible — tapping them breaks the debugger connection. Close them with the X icon if present.

gesture-swipe — Straight-line gesture

{ "udid": "<UDID>", "fromX": 0.5, "fromY": 0.7, "toX": 0.5, "toY": 0.3 }

Swipe up (fromY > toY) = scroll content down. Default duration: 300ms. Optional: "durationMs": 500 for slower swipe.

"momentum" defaults to true (a natural flinging swipe). Pass "momentum": false for a momentum-free swipe: the finger decelerates into the end point, resulting in little to no fling. It needs durationMs of at least 150 and is rejected below it.

gesture-pinch — Two-finger pinch

{ "udid": "<UDID>", "centerX": 0.5, "centerY": 0.5, "startDistance": 0.2, "endDistance": 0.6 }

All values are normalized 0.0–1.0 (fractions of screen, not pixels) — same as all other gesture tools. startDistance: 0.2 means fingers start 20% of the screen apart; endDistance: 0.6 means they end 60% apart. startDistance < endDistance = pinch out (zoom in). startDistance > endDistance = pinch in (zoom out). Defaults: angle: 0 (horizontal), durationMs: 300. Optional: "angle": 90 for vertical axis, "durationMs": 500 for slower pinch, "endCenterX"/"endCenterY" to let the centroid drift to a new center over the gesture (omitted = fixed center).

gesture-rotate — Two-finger rotation

{
  "udid": "<UDID>",
  "centerX": 0.5,
  "centerY": 0.5,
  "radius": 0.15,
  "startAngle": 0,
  "endAngle": 90
}

All positions and radii are normalized 0.0–1.0 (fractions of screen, not pixels). radius: 0.15 means each finger is 15% of the screen away from center. endAngle > startAngle = clockwise. Default duration: 300ms. Optional: "durationMs": 500 for slower rotation, and "radiusX"/"radiusY" (fractions of screen width/height; give both — they override radius) with radiusX·width = radiusY·height for a physically circular orbit — a single radius traces a physical ellipse on a non-square screen, coupling a slight pinch into the turn.

gesture-custom — Custom touch sequence

For long-press, drag-and-drop, and other complex sequences, see references/gesture-examples.md. Set "interpolate": 10 to auto-generate smooth intermediate Move events between keyframes.

button — Hardware button press

{ "udid": "<UDID>", "button": "home" }

Values: home, back, power, volumeUp, volumeDown, appSwitch, actionButton

keyboard — Type text or press special keys

{ "udid": "<UDID>", "text": "search query" }

One call does one action. text and key are mutually exclusive, and a call that carries both is rejected with nothing typed. To type and then submit, send two keyboard steps in one run-sequence (§ 8) — { "text": "search query" }, then { "key": "enter" }. Two separate calls do the same work, but cost an extra round-trip.

Special keys: enter, escape, backspace, tab, space, arrow-up, arrow-down, arrow-left, arrow-right, f1–f12. Optional: "delayMs": 100 between keystrokes (default 50ms) — applies to the iOS simulator and Chromium; it is ignored on Android phones/tablets (typed via adb input text, no per-key cadence), on Vega, and on TV targets.

Typing secrets. To enter a credential without its plaintext ever entering your context, transcript, or logs, use a secret placeholder in text (works in keyboard, paste, run-sequence keyboard steps, and flow type steps):

{ "udid": "<UDID>", "text": "{{secret:APP_PASSWORD}}" }

Where the value is read from, and the rules for using a placeholder — including not screenshotting the field afterwards — are in references/secrets.md. Read it before typing any credential.

paste — Paste text into the focused field

{ "udid": "<UDID>", "text": "482913" }

Puts text on the device clipboard (the host clipboard is untouched) and triggers the platform's paste shortcut. iOS simulator and Android emulator only; a TV target, a physical device, Chromium and Vega are rejected.

paste is not a faster keyboard. keyboard types the way a user types and stays the default for every text entry — a search query, a login, a form field. Reach for paste only where a real user would paste: a 2FA / OTP code copied from another app, a long link or token, or to test how the app handles pasted input. It also carries what keyboard can't type on a given platform (multi-line text, non-ASCII on Android), but that alone is not a reason to paste — ask whether the user would.

Tap the field first so it has focus; pasting with no focused field is a silent no-op, as with keyboard. text accepts the same {{secret:<NAME>}} placeholders as keyboard, with the same auto-screenshot skip.