Phoneagent

PhoneAgent

Use this workflow to drive iOS or Android UI through PhoneAgent's JSON-RPC bridge.

All shell commands below assume you are in the repo root:

cd "$(git rev-parse --show-toplevel)"

Start the RPC bridge

1. Choose a platform bridge (both listen on 127.0.0.1:45678 by default).

# iOS (XCTest-hosted bridge)
./.agents/skills/phoneagent/scripts/start_rpc_bridge_local.sh

# Android (adb bridge; emulator or physical device)
./.agents/skills/phoneagent/scripts/start_android_rpc_bridge_local.sh

Notes:

• start_rpc_bridge_local.sh is interactive and will show a numbered list of iOS devices/simulators. Enter the number for the destination you want.
• start_rpc_bridge_local.sh starts a localhost-only forwarder.
• On Xcode "Connect via network", it uses the CoreDevice tunnel automatically (no extra deps).
• For USB fallback forwarding, install pymobiledevice3 into a local venv: python3 -m venv .venv && ./.venv/bin/python -m pip install -U pip && ./.venv/bin/python -m pip install pymobiledevice3
• start_android_rpc_bridge_local.sh uses adb; if multiple devices are connected it prompts for the serial.

2. Keep the bridge process running.
3. Wait for PHONEAGENT_RPC_READY ... in logs before sending RPC calls.
4. Confirm socket readiness before first RPC:

./.agents/skills/phoneagent/scripts/rpc.py get-tree >/dev/null && echo rpc-ready

Resolve host and port

1. Always use 127.0.0.1:45678 as the RPC endpoint (or rpc.py --port <port> if customized).

Notes:

• Both bridges are localhost-only.
• iOS physical-device flow uses a localhost forwarder.
• If you need to forward manually, first get a device UDID via xcrun devicectl list devices, then run: python3 ./.agents/skills/phoneagent/scripts/forward_rpc_localhost.py --udid <UDID> (binds 127.0.0.1:45678)

Send RPC calls

Use the helper CLI:

# iOS bundle identifier
./.agents/skills/phoneagent/scripts/rpc.py open-app com.apple.Preferences

# Android package name
./.agents/skills/phoneagent/scripts/rpc.py open-app com.android.settings
./.agents/skills/phoneagent/scripts/rpc.py get-tree | head

# Use coordinates copied from the tree (XCUI frame string).
./.agents/skills/phoneagent/scripts/rpc.py enter-text \
  --coordinate '{{33.0, 861.0}, {364.0, 38.0}}' \
  --text 'Display'

./.agents/skills/phoneagent/scripts/rpc.py tap-element \
  --coordinate '{{37.7, 969.7}, {199.7, 29.0}}'

Core operating loop

1. Call get_tree.
2. Identify the best target element in the tree (label/identifier) and copy its frame coordinate string.
3. Prefer coordinate-based actions (tap_element / enter_text).
4. Use the returned tree from the action response to verify the UI changed as expected.
5. Repeat until complete.
6. When the task is complete, always capture a screenshot for the user:
   • Prefer get_context and write result.screenshot_base64 to a PNG (or use ./.agents/skills/phoneagent/scripts/rpc.py get-screen-image, which writes PNG files to /tmp/phoneagent-artifacts).
   • Include the PNG path in your final message so the user can open it.

Use swipe to reveal off-screen content, then use the returned tree (or call get_tree if needed). Use one request at a time per server. Do not fire concurrent batches. Split long keyboard input into chunks; do not send giant enter_text payloads in one call.

RPC method reference

All RPC requests are newline-delimited JSON objects with this shape:

{"id":1,"method":"<method>","params":{...}}

All success responses look like:

{"id":1,"result":{...}}

get_tree

• Does: Returns the accessibility tree of the currently focused app.
• Params: none.
• Returns: {"tree": "<string>"}

Example:

{"id":1,"method":"get_tree","params":{}}

get_screen_image

• Does: Captures the current screen as a base64-encoded PNG plus image dimensions (when available).
• Params: none.
• Returns: {"screenshot_base64":"<base64>","metadata":{"width":<number>,"height":<number>}}

Example:

{"id":2,"method":"get_screen_image","params":{}}

get_context

• Does: Convenience method that returns both the current accessibility tree and the current screen image.
• Params: none.
• Returns: {"tree":"<string>","screenshot_base64":"<base64>","metadata":{"width":<number>,"height":<number>}}

Example:

{"id":3,"method":"get_context","params":{}}

open_app

• Does: Brings the specified app to the foreground (and makes it the focused app for subsequent calls).
• Params: bundle_identifier (string, required).
  • iOS: pass bundle identifier (example com.apple.Preferences).
  • Android: pass package name (example com.android.settings).
• Returns: {"bundle_identifier":"<string>", "tree":"<string>"} (Android also includes package_name).

Example:

{"id":4,"method":"open_app","params":{"bundle_identifier":"com.apple.Preferences"}}

tap

• Does: Taps an absolute point in the current app.
• Params: x (number, required), y (number, required). Coordinates are in absolute screen points as reported by the tree.
• Returns: {"tree":"<string>"}

Example:

{"id":5,"method":"tap","params":{"x":120,"y":300}}