Agent Device

IMPORTANT — agent-device is the ONLY tool for device interaction

ALL simulator/emulator interaction MUST go through agent-device commands. No exceptions, no fallbacks.

Banned tools/commands (never use these for device interaction, even if they seem easier):

• adb for UI interaction — no input tap, input swipe, input text, screencap, etc. Allowed exceptions: adb devices, adb wait-for-device, adb install, adb reverse, adb shell getprop, adb shell am, adb shell pm, adb shell screenrecord + adb pull (see "Android Recording Workaround"), adb shell kill/adb shell pidof (for stopping screenrecord)
• Mobile MCP tools — no mobile_click_on_screen_at_coordinates, mobile_take_screenshot, mobile_list_elements_on_screen, mobile_swipe_on_screen, mobile_type_keys, mobile_press_button, mobile_long_press_on_screen_at_coordinates, or any other mobile_* tool
• xcrun simctl — no simctl io screenshot, simctl openurl, etc. Allowed exceptions: simctl list devices, simctl get_app_container, simctl install, simctl launch
• osascript / AppleScript for simulator control
• Appium or any other automation framework

Why: agent-device manages sessions, coordinate translation, and daemon state. Mixing in other tools causes session conflicts, stale state, and unreliable behavior.

• Keep all delays under 1s. This applies to wait, sleep, and any other waiting mechanism. The app is fast — transitions and network responses complete quickly. The only exception is app launch (open), which may take a few seconds to fully load.

Agent Device Interaction

Control the iOS simulator or Android emulator using agent-device. The primary interaction method is snapshot-based: take an accessibility tree snapshot, find the target element's rect, compute its center, and press.

Prohibited agent-device subcommands

Do NOT use these as standalone subcommands:

• click, find, fill, focus, get text, get attrs, scrollintoview, is, wait text, wait @ref, diff snapshot

Allowed agent-device subcommands: snapshot, screenshot, press, type, scroll, swipe, longpress, back, home, app-switcher, wait <ms>, open, close, keyboard dismiss, appstate, clipboard, alert, settings, record, devices, apps, batch, push, logs, network.

Platform Setup

Device Targeting (Required)

Always target a specific device by name using --device "<name>" to avoid launching the wrong simulator/emulator. At the start of each session:

1. Run agent-device devices to list available devices
2. Identify the correct booted device (or the one the user specified)
3. Probe for an existing session before calling open (see below)
4. If no existing session is found, pass --device "<name>" on open — subsequent commands in the same session inherit it

Probing for Existing Sessions (Do This First)

A previous conversation may have left an active session bound to the device. Calling open with a new session or the default session will fail with a conflict error. Always probe first using appstate (lightweight, no file output):

# Probe: check if a session already owns this device
agent-device appstate --device "iPhone 16"

iOS:

# First: discover available devices
agent-device devices

# Probe for existing session
agent-device appstate --device "iPhone 16"
# If error says "in use by session ios16" → use --session ios16
# If succeeds → default session works, skip open

# Only if no session exists: open with explicit device targeting
agent-device open FlatListPro --device "iPhone 16"

# Subsequent commands — no --device needed (session remembers)
agent-device snapshot -i -c --json   # primary: get elements with rects
agent-device press <x> <y>
agent-device screenshot /tmp/verify.png && sips --resampleHeight 852 /tmp/verify.png >/dev/null  # verification

Android (also requires --session and --platform):

Use agent-device apps --platform android --user-installed to discover the installed package name.

# Probe for existing session
agent-device appstate --device "Android35" --session droid --platform android
# If succeeds → session already exists, skip open

# Only if no session exists: bind session to Android (replace <package> with actual package name)
agent-device open <package> --session droid --platform android \
  --device "Android35" \
  --activity <package>/.MainActivity

# All subsequent commands: just --session droid
agent-device snapshot -i -c --json --session droid  # primary
agent-device press <x> <y> --session droid
agent-device screenshot /tmp/verify.png --session droid && sips --resampleHeight 852 /tmp/verify.png >/dev/null  # verification

Core Workflow — Snapshot-First

The primary method uses the accessibility tree snapshot for exact element coordinates. Screenshots are the fallback for visual verification.

1. Take a snapshot

agent-device snapshot -i -c --json

This returns interactive (-i) elements with their rect coordinates (-c) in JSON format. Each element looks like:

{
  "@ref": "@e25",
  "role": "button",
  "label": "Settings",
  "rect": {"x": 141, "y": 2032, "width": 154, "height": 154}
}

2. Find the target element

Search the snapshot output for your target by matching label, identifier, or value. Example: looking for "Settings" → find the element with "label": "Settings".

3. Compute the center and press

Calculate the center of the element's rect:

• x = rect.x + rect.width / 2
• y = rect.y + rect.height / 2

Then press at those coordinates.

4. Verify with a screenshot

After pressing, take a screenshot to confirm the action worked:

agent-device screenshot /tmp/verify.png && sips --resampleHeight 852 /tmp/verify.png >/dev/null

Then Read /tmp/verify.png to view it.

Fallback: Vision-Based Screenshots

When an element is not in the accessibility tree (e.g., canvas-rendered content, custom drawn views), fall back to screenshots with percentage-based coordinate estimation.

Coordinate System — Device-Agnostic

iOS and Android use different coordinate systems for press. The exact dimensions vary by device. You MUST discover them dynamically at the start of each session.

How press coordinates work

• Android: press takes raw pixel coordinates (same as screenshot dimensions)
• iOS: press takes logical point coordinates (screenshot pixels / scale factor)
  • Modern iPhones (X and later): 3x scale
  • iPhone SE, older models: 2x scale
  • iPads: 2x scale

Session Start: Discover Press Dimensions (Vision Fallback Only)

Snapshot rect values are already in the correct press coordinate space — skip this if you're only using snapshots. This is only needed when estimating coordinates from screenshots. Run once per platform:

agent-device screenshot /tmp/screen.png  # (add --session droid for Android)
sips -g pixelWidth -g pixelHeight /tmp/screen.png

Then compute the press dimensions:

# Android: press coords = screenshot pixels
PRESS_W = RAW_W
PRESS_H = RAW_H

# iOS: press coords = screenshot pixels / scale
PRESS_W = RAW_W / 3    # (use /2 for iPhone SE or iPad)
PRESS_H = RAW_H / 3

Remember these values for the rest of the session. All coordinate calculations use them.

Percentage Method

1. View the screenshot
2. Estimate the element's center as a percentage of screen width and height
3. Multiply by the press dimensions:

x = PRESS_W * (x_percent / 100)
y = PRESS_H * (y_percent / 100)

Command Reference

Navigation

open FlatListPro --device "iPhone 16"    # Launch app (iOS — always specify --device on first open)
open <package> \                          # Launch app (Android — discover package name with `apps --platform android --user-installed`)
  --device "Android35" --session droid --platform android \
  --activity <package>/.MainActivity
close FlatListPro                         # Kill app
back                                  # Navigate back (Android: in-app; iOS: may go to previous app)
home                                  # Device home screen
app-switcher                          # Open app switcher

Element Discovery

snapshot -i -c --json                 # Interactive elements with rects (primary method)

Interactions

press <x> <y>                         # Tap (iOS=logical points, Android=pixels)
press <x> <y> --double-tap            # Double-tap
longpress <x> <y> [durationMs]        # Long press (default 500ms)
type "text"                           # Type into focused field (tap input first)
scroll <up|down|left|right> [0-1]     # Scroll in direction with amount
swipe <x1> <y1> <x2> <y2> [durationMs]  # Precise swipe between coordinates
wait <ms>                             # Wait milliseconds (max 500ms — the app is fast)

Screenshots & Recording

screenshot /tmp/screen.png            # Save screenshot
record start ./recording.mov          # Start video recording (iOS only — see below for Android)
record stop                           # Stop recording (iOS only)

To view a screenshot, downsample and read:

agent-device screenshot /tmp/screen.png && sips --resampleHeight 852 /tmp/screen.png >/dev/null

Then Read /tmp/screen.png.

Android Recording Workaround

agent-device record is broken on Android emulators (API 35+) — it sends SIGINT to the local adb process instead of the on-device screenrecord, producing a corrupt MP4. Use adb directly for Android recording.

First, resolve the serial once per session (store in $SERIAL):

SERIAL=$(adb devices | grep -w device | head -1 | cut -f1)

Then use it for recording:

# Start (run in background)
adb -s $SERIAL shell screenrecord /sdcard/agent-rec.mp4 &

# Stop (SIGINT the on-device process, then pull)
adb -s $SERIAL shell kill -INT $(adb -s $SERIAL shell pidof screenrecord)
sleep 2
adb -s $SERIAL pull /sdcard/agent-rec.mp4 /tmp/recording.mp4
adb -s $SERIAL shell rm -f /sdcard/agent-rec.mp4

Note: screenrecord only encodes frames when the display changes — interact with the UI during recording or you'll get a single-frame file.

Device Info

devices                               # List available devices
apps --platform ios --user-installed  # List installed apps
appstate                              # Show foreground app/activity (useful on Android)
keyboard dismiss                      # Dismiss on-screen keyboard (Android)
clipboard read                        # Read clipboard (iOS only)
clipboard write "text"                # Write to clipboard

Settings (useful for testing)

settings appearance dark              # Switch to dark mode
settings appearance light             # Switch to light mode
settings wifi off                     # Toggle wifi
settings permission grant camera      # Grant camera permission

CI Known Issues

• Session creation may hang on CI runners — agent-device open can hang indefinitely on CI. Set reasonable timeouts and be prepared to fall back to code-only verification with unit tests.
• Use full bundle ID on CI — the app may not be recognized by display name; use the full bundle ID (e.g., org.reactjs.native.example.FlatListPro) instead of FlatListPro.
• Snapshot/session timeouts — CI runners may experience persistent timeouts with snapshots and sessions. If agent-device is unavailable, document it in feedback and proceed with automated tests only.