Axe Simulator Control

AXe Simulator Control

Use this skill when you need to automate iOS Simulator interactions, including taps, gestures, text input, hardware buttons, or video recording.

Related Skills

• See xcodebuild-testing for running tests that might require simulator interactions
• See screenshot for capturing screenshots from the simulator

Skill Flow

• Example queries:
  • "tap at coordinates (100, 200) in the simulator"
  • "type 'hello world' into the simulator"
  • "press the home button in the simulator"
  • "record a video of the simulator"
  • "perform a scroll up gesture"
• Notes:
  • AXe is installed via Homebrew: brew tap cameroncooke/axe && brew install axe
  • Most commands require a simulator UDID (use axe list-simulators to find it)
  • The simulator must be booted and running for most operations
  • AXe uses the iOS Simulator's accessibility framework
  • For VivaDicta project, the default simulator is: iPhone 17 Pro Max, OS=26.4
  • Save screenshots to llmtemp/screenshots/ directory for temporary test artifacts

1. Determine the Required Action

Touch/Tap Actions if:

• Need to tap at specific coordinates
• Need to add delays before/after taps
• Need to simulate touch gestures

→ Continue with Path A: Touch & Tap (steps 2A-4A)

Text Input if:

• Need to type text into the simulator
• Need to test keyboard input
• Need to simulate user text entry

→ Continue with Path B: Text Input (step 2B)

Gestures if:

• Need to scroll, swipe, or perform system gestures
• Need to test gesture recognizers
• Need to navigate using gestures

→ Continue with Path C: Gestures (steps 2C-3C)

Hardware Buttons if:

• Need to press home, lock, volume buttons
• Need to test app lifecycle events
• Need to simulate device rotation

→ Continue with Path D: Hardware Buttons (steps 2D-3D)

Video Operations if:

• Need to record simulator video
• Need to stream simulator output
• Need to capture simulator for documentation

→ Continue with Path E: Video Operations (steps 2E-3E)

List Simulators if:

• Need to find available simulators
• Need to get a simulator UDID
• First time setup

→ Continue with Path F: List Simulators (step 2F)

Path A: Touch & Tap

2A. Get Simulator UDID

# List available simulators
axe list-simulators

# Store the UDID for the target simulator
UDID="B34FF305-5EA8-412B-943F-1D0371CA17FF"

3A. Get Accurate Coordinates Using describe-ui

IMPORTANT: Always use axe describe-ui to get accurate coordinates instead of guessing from screenshots.

# Get the full UI accessibility tree
axe describe-ui --udid $UDID > ui_tree.json

# Example output for a button:
# {
#   "AXFrame" : "{{175, 741}, {52, 50}}",
#   "AXLabel" : "stop.circle.fill",
#   "type" : "Button",
#   ...
# }

# Calculate center coordinates:
# x_center = x + (width / 2)  →  175 + (52 / 2) = 201
# y_center = y + (height / 2)  →  741 + (50 / 2) = 766

Why this matters:

• Visual appearance in screenshots doesn't always match accessibility frame coordinates
• UI elements that change state (e.g., record → stop button) may have different positions than they appear
• The accessibility frame is what AXe actually uses for tap detection

4A. Perform Tap Action

# Simple tap at coordinates (use coordinates from describe-ui)
axe tap -x 201 -y 766 --udid $UDID

# Tap with delays before and after
axe tap -x 201 -y 766 --pre-delay 1.0 --post-delay 0.5 --udid $UDID

Available tap options:

• -x and -y: Coordinates (required) - get these from describe-ui output
• --pre-delay: Delay in seconds before tapping
• --post-delay: Delay in seconds after tapping
• --udid: Simulator UDID (required)

Note: The tap command does NOT support --duration for long press. Use --pre-delay and --post-delay for timing control.

5A. Verify the Action

• Check the simulator visually or via app logs
• If tap didn't register, ensure:
  • Simulator is in foreground
  • Coordinates are correct (origin is top-left)
  • Target element is interactive
  • You used coordinates from describe-ui, not visual estimates

Path B: Text Input

2B. Type Text into Simulator

# Get simulator UDID first
UDID="B34FF305-5EA8-412B-943F-1D0371CA17FF"

# Type text (ensure a text field is focused first)
axe type 'Hello World!' --udid $UDID

# Type text with special characters (use single quotes)
axe type 'user@example.com' --udid $UDID

Notes:

• The text field must be focused before typing (tap it first if needed)
• Use single quotes to avoid shell interpretation
• Special characters and emojis are supported

Path C: Gestures

2C. Choose Gesture Type

Available gestures:

• scroll-up / scroll-down / scroll-left / scroll-right
• swipe-from-left-edge / swipe-from-right-edge
• swipe-from-top-edge / swipe-from-bottom-edge

3C. Execute Gesture

# Get simulator UDID first
UDID="B34FF305-5EA8-412B-943F-1D0371CA17FF"

# Perform scroll gesture
axe gesture scroll-up --udid $UDID

# Perform edge swipe (for navigation)
axe gesture swipe-from-left-edge --udid $UDID

# Perform swipe gesture
axe gesture swipe-from-bottom-edge --udid $UDID

Path D: Hardware Buttons

2D. Choose Button Action

Available buttons:

• home - Press home button
• lock - Press lock/power button
• volume-up / volume-down
• siri - Activate Siri

3D. Execute Button Press

# Get simulator UDID first
UDID="B34FF305-5EA8-412B-943F-1D0371CA17FF"

# Press home button
axe button home --udid $UDID

# Press lock button with custom duration
axe button lock --duration 2.0 --udid $UDID

# Press volume up
axe button volume-up --udid $UDID

Notes:

• Lock duration determines press-and-hold behavior
• Home button triggers app backgrounding
• Use for testing app lifecycle events