Device Interaction

Device Interaction

TRIGGER when: user asks to verify/test/check if the app works on device, after implementing a UI-affecting feature that needs device verification, user says "does it work", "test this", "check on device", user reports UI doesn't work as expected, need to debug touch/interaction issues. DO NOT TRIGGER when: user asks about unit tests only, build-only requests without device testing, code review without device testing, simulator configuration questions, changes that don't affect UI (e.g. comments, refactors, non-UI logic).

For the Main Agent

This is a SUBAGENT skill. Invoke it via the Agent tool when device verification is needed.

Agent tool:
- subagent_type: "general-purpose"
- description: "Verify login feature works"
- prompt: "Using the device-interaction skill, verify that the login feature works correctly on session <device-interaction-session>. Launch the app, capture screenshot and UI hierarchy, check that the login button is visible and tappable, and report if the implementation is working correctly."

After implementing a UI-affecting feature, invoke this skill to verify the implementation works on a device.

Session Lifecycle

DeviceInteractionStartSession (do this early, runs in the background)
  → DeviceInteractionInstallAndRun (after each code change; includes building)
    → DeviceEventSynthesize (interact + observe, repeatable)
  → DeviceInteractionEndSession (when done — keeping sessions open is resource-heavy)

DeviceInteractionStartSession tool

Device Discovery

When opening a new device interaction session, pass a device identifier to select a device, or omit it to use the current destination. Pass any non-matching value to get a list of available targets.

DeviceInteractionInstallAndRun tool

Optional Parameters

• commandLineArguments — arguments passed to the app at launch. Use $(inherited) as a token to preserve the scheme's existing arguments (e.g. ["$(inherited)", "--reset-state"] to add an extra argument at the end).
• environmentVariables — key/value pairs set in the app's environment at launch. Use "$(inherited)" as a key to preserve the scheme's existing environment variables (e.g. {"$(inherited)": "", "DEBUG_MODE": "1"}).

Omit both parameters to leave the scheme's arguments and environment unchanged.

Prefer these parameters over editing the scheme directly. They are applied only for that one run and have no lasting effect on the user's configuration.

For the Subagent

ALWAYS report UI issues that might be caused by code: overlapping or unreadable text, unexpectedly cropped image/text, wrong colors etc.

DeviceEventSynthesize tool

This tool allows performing an interaction and observing the state of a device.

Reading Hierarchy Files

The hierarchy files include calculated center positions for each element:

UIView {{100, 200}, {50, 30}}, center: {125.0, 215.0}
  UIButton "Login" {{110, 205}, {30, 20}}, center: {125.0, 215.0}

• {100, 200} - origin position
• {50, 30} - width and height
• center: {125.0, 215.0} - calculated center point (best for tapping)

Always prefer the center coordinates for touch events.

Interaction Command Syntax

The interactionCommand parameter accepts a command syntax:

Examples:

• "t 100 200" - Tap at (100, 200)
• "d 200 300" - Double tap at (200, 300)
• "t 200 600 f 200 200 0.3" - Swipe up (scroll to the content below)
• "t 200 200 f 200 600 0.3" - Swipe down (scroll to the content above)
• "b h" - Press home button
• "b h b h" - Press home button twice to go to the app switcher
• "b h w 0 b h" - Wake and unlock a device (non-passcode devices only)
• "sender keyboard kbd hello world" - Type text with spaces
• "sender keyboard kbd hello world" - Type text preserving multiple spaces
• "sender keyboard kbd submit\u{000A}" - Type text then press Return/submit
• "w 0.3" - Wait for 0.3s
• "orientation landscapeLeft" - Rotate device to landscape