iOS Testing Patterns

iOS Testing Patterns Skill

Comprehensive guide to XCTest and XCUITest execution, analysis, and flaky test detection

Overview

This skill provides patterns and workflows for executing iOS tests using XCTest and XCUITest frameworks. It covers test execution strategies, result analysis, flaky test detection, and troubleshooting common test failures.

Use this skill when:

• Running unit tests or UI tests
• Analyzing test failures
• Detecting and debugging flaky tests
• Optimizing test execution performance
• Setting up CI/CD test pipelines

Quick Reference

When to Use This Skill

Use ios-testing-patterns when:

• Executing XCTest unit tests
• Running XCUITest UI automation tests
• Investigating test failures
• Detecting intermittent test failures (flaky tests)
• Analyzing test performance and execution time
• Setting up test schemes and test plans
• Configuring parallel test execution
• Debugging test-specific simulator issues

Related Skills:

• xcode-workflows: For general build and test operations
• simulator-workflows: For managing test simulators
• ui-automation-workflows: For UI test interaction patterns

Key Concepts

XCTest vs XCUITest

XCTest (Unit/Integration Tests):

• Fast execution (milliseconds to seconds)
• Tests individual components in isolation
• Direct access to app internals
• No UI interaction required
• Runs in same process as app code

XCUITest (UI Tests):

• Slower execution (seconds to minutes)
• Tests user-facing workflows
• Black-box testing (no app internals)
• Requires simulator/device UI
• Runs in separate process

Test Execution Strategies

Sequential Execution:

• Tests run one after another
• Predictable, repeatable results
• Slower overall execution
• Better for debugging

Parallel Execution:

• Tests run simultaneously on multiple simulators
• Faster overall execution
• Requires more system resources
• May expose race conditions

Test Sharding:

• Split test suite across multiple destinations
• Optimal for CI/CD pipelines
• Reduces total execution time
• Requires careful test isolation

Workflows

1. Running Unit Tests

Basic Unit Test Execution:

{
  "operation": "test",
  "scheme": "MyApp",
  "destination": "platform=iOS Simulator,name=iPhone 15,OS=18.0"
}

Note: The destination parameter now supports auto-resolution! You can omit the OS version (e.g., "platform=iOS Simulator,name=iPhone 15") and the tool will automatically select the latest available OS version for that device.

Run Specific Test Class:

{
  "operation": "test",
  "scheme": "MyApp",
  "destination": "platform=iOS Simulator,name=iPhone 15,OS=18.0",
  "options": {
    "only_testing": ["MyAppTests/NetworkTests"]
  }
}

Run Specific Test Method:

{
  "operation": "test",
  "scheme": "MyApp",
  "destination": "platform=iOS Simulator,name=iPhone 15,OS=18.0",
  "options": {
    "only_testing": ["MyAppTests/NetworkTests/testAPIRequest"]
  }
}

2. Running UI Tests

Basic UI Test Execution:

{
  "operation": "test",
  "scheme": "MyAppUITests",
  "destination": "platform=iOS Simulator,name=iPhone 15"
}

UI Tests with Specific Device:

{
  "operation": "test",
  "scheme": "MyAppUITests",
  "destination": "platform=iOS Simulator,name=iPad Pro (12.9-inch)"
}

UI Tests with Test Plan:

{
  "operation": "test",
  "scheme": "MyApp",
  "destination": "platform=iOS Simulator,name=iPhone 15",
  "options": {
    "test_plan": "SmokeTests"
  }
}

3. Parallel Test Execution

Multiple Destinations:

Run tests on multiple simulators simultaneously:

{
  "operation": "test",
  "scheme": "MyApp",
  "destination": [
    "platform=iOS Simulator,name=iPhone 15",
    "platform=iOS Simulator,name=iPhone SE (3rd generation)"
  ],
  "options": {
    "parallel": true
  }
}

Parallel Test Benefits:

• Reduces total execution time
• Tests across multiple device types
• Exposes device-specific issues
• Better CI/CD performance

Parallel Test Considerations:

• Requires multiple simulators
• Higher CPU/memory usage
• May expose race conditions
• Test isolation critical

4. Test Filtering

Skip Specific Tests:

{
  "operation": "test",
  "scheme": "MyApp",
  "destination": "platform=iOS Simulator,name=iPhone 15",
  "options": {
    "skip_testing": [
      "MyAppUITests/SlowTests",
      "MyAppTests/NetworkTests/testLargeDownload"
    ]
  }
}

Run Only Fast Tests:

{
  "operation": "test",
  "scheme": "MyApp",
  "destination": "platform=iOS Simulator,name=iPhone 15",
  "options": {
    "only_testing": ["MyAppTests/UnitTests"],
    "skip_testing": ["MyAppTests/UnitTests/testSlowOperation"]
  }
}

5. Test Result Analysis

Success Response:

{
  "success": true,
  "tests_run": 45,
  "tests_passed": 45,
  "tests_failed": 0,
  "execution_time": "12.4s",
  "scheme": "MyApp"
}

Failure Response:

{
  "success": false,
  "tests_run": 45,
  "tests_passed": 43,
  "tests_failed": 2,
  "failures": [
    {
      "test": "MyAppTests.LoginTests.testInvalidPassword",
      "message": "XCTAssertEqual failed: (\"error\") is not equal to (\"invalid\")",
      "file": "LoginTests.swift",
      "line": 42
    },
    {
      "test": "MyAppUITests.CheckoutTests.testPaymentFlow",
      "message": "Failed to find button \"Confirm\"",
      "file": "CheckoutTests.swift",
      "line": 78
    }
  ],
  "execution_time": "18.7s"
}

Analyzing Test Results:

1. Check success status
2. Review tests_failed count
3. Examine failures array for details
4. Check execution_time for performance issues
5. Look for patterns in failure messages

6. Flaky Test Detection

What are Flaky Tests?

Tests that intermittently pass or fail without code changes:

• Timing-dependent assertions
• Race conditions
• Async operation issues
• Shared state between tests
• Network-dependent tests
• UI animation timing issues

Detection Strategy 1: Multiple Runs

Run tests multiple times to detect flakiness:

# Run tests 5 times and compare results
for i in {1..5}; do
  execute_xcode_command(test, scheme, destination)
  # Record results
done

Detection Strategy 2: Pattern Analysis

Look for common flaky test patterns:

{
  "operation": "test",
  "scheme": "MyApp",
  "destination": "platform=iOS Simulator,name=iPhone 15",
  "options": {
    "only_testing": ["MyAppTests/SuspectedFlakyTest"]
  }
}

Flaky Test Indicators:

1. Timeout failures: "Exceeded timeout of 30 seconds"
2. Element not found: "Failed to find button..." (UI tests)
3. Race conditions: "Expected 5 but got 4"
4. Async timing: "Asynchronous wait failed"
5. Network failures: "Request timed out"

Flaky Test Workflow:

1. Identify suspect test (intermittent failures)
2. Run test 10+ times in isolation
3. Analyze failure patterns
4. Check for:
   - Hard-coded waits (sleep/wait)
   - Expectation timeouts too short
   - Shared mutable state
   - Network dependencies
   - Animation timing assumptions
5. Fix root cause
6. Verify with repeated runs

7. Test Isolation

Clean State Before Tests:

{
  "operation": "test",
  "scheme": "MyApp",
  "destination": "platform=iOS Simulator,name=iPhone 15",
  "options": {
    "clean_before_build": true
  }
}

Reset Simulator State:

# Use simulator-workflows skill
execute_simulator_command(operation: "erase", udid: <device-udid>)

Test Isolation Best Practices:

1. Use setUp() and tearDown() methods
2. Clear app state between tests
3. Use unique test data per test
4. Avoid shared static state
5. Reset UserDefaults in setUp
6. Clear keychain data
7. Reset app permissions

Common Patterns

Test Scheme Configuration

Typical Test Scheme Setup:

• Build Configuration: Debug (faster builds)
• Test Plans: Separate unit/UI/smoke tests
• Coverage: Enable code coverage collection
• Diagnostics: Enable thread sanitizer for threading issues

Multiple Test Plans:

UnitTests.xctestplan       → Fast unit tests only
UITests.xctestplan         → UI automation tests
SmokeTests.xctestplan      → Critical path tests
RegressionTests.xctestplan → Full test suite

Test Data Management

Pattern 1: Test Fixtures

class TestData {
    static let validUser = User(name: "Test", email: "test@example.com")
    static let invalidUser = User(name: "", email: "invalid")
}

Pattern 2: Factory Methods

extension User {
    static func makeTestUser(name: String = "Test") -> User {
        return User(name: name, email: "\(name)@test.com")
    }
}

Pattern 3: Test Database

class TestDatabase {
    func setupTestData() {
        // Create known test data
    }

    func teardownTestData() {
        // Clean up after tests
    }
}

Screenshot Capture on Failure

UI Test Screenshot Pattern:

override func setUp() {
    continueAfterFailure = false
}

override func tearDown() {
    if let testRun = testRun, testRun.failureCount > 0 {
        let screenshot = XCUIScreen.main.screenshot()
        let attachment = XCTAttachment(screenshot: screenshot)
        attachment.lifetime = .keepAlways
        add(attachment)
    }
}

Execute Tests with Screenshots:

{
  "operation": "test",
  "scheme": "MyAppUITests",
  "destination": "platform=iOS Simulator,name=iPhone 15",
  "options": {
    "result_bundle_path": "./TestResults.xcresult"
  }
}

Performance Testing

XCTest Performance Measurement:

func testPerformance() {
    measure {
        // Code to measure
        performExpensiveOperation()
    }
}

Performance Test Execution:

{
  "operation": "test",
  "scheme": "MyApp",
  "destination": "platform=iOS Simulator,name=iPhone 15",
  "options": {
    "only_testing": ["MyAppTests/PerformanceTests"]
  }
}

Troubleshooting

Common Test Failures

1. "Test crashed with signal SIGABRT"

Causes:

• Force unwrapping nil values
• Array index out of bounds
• Precondition/assertion failures
• Fatal errors in app code

Solutions:

1. Check crash logs for stack trace
2. Run test in debugger (Xcode)
3. Add breakpoints at crash location
4. Review force unwraps in test path