Fixing Flaky Tests

Fixing Flaky Tests — Bitwarden iOS

A flaky test passes sometimes and fails other times without any code change. Flakiness usually comes from a small set of root causes. Your job is to find which one applies, fix it surgically, and then prove the fix holds by running the test many times.

Step 1: Locate the test(s)

Search for the function name(s) the user provided:

grep -r "func <testFunctionName>" . --include="*.swift" -l

Note that:

• XCTest functions are named test_<methodName>_<description> (prefix required by XCTest)
• Swift Testing functions use the @Test macro and can have any name — the test_ prefix is conventional but not required

For each test:

• Read the full test function and the class it belongs to
• Read the @testable import at the top of the file — that's the module under test (e.g., @testable import BitwardenShared → module BitwardenShared)
• Read the SUT code being exercised by the test

Step 2: Map the test to a scheme and test plan

When in doubt, search TestPlans/ for the test class name to confirm the plan (and therefore scheme):

grep -r "<TestClassName>" TestPlans/ -l

Always read the simulator config from the project files rather than hardcoding:

DEVICE=$(tr -d '\n' < .test-simulator-device-name)
OS=$(tr -d '\n' < .test-simulator-ios-version)

Step 3: Diagnose the root cause

Read the test carefully and the SUT it exercises. Flaky tests in Swift/iOS typically fall into one of these categories:

Race condition / async ordering The test observes a side effect before the async SUT has finished producing it. Signs: bare Task.sleep or DispatchQueue.asyncAfter in the test, expectations that time out occasionally, @MainActor-isolated code whose scheduling the test doesn't account for.

Shared mutable state State from one test bleeds into the next. Signs: static properties or singletons not reset in tearDown, NotificationCenter observers not removed, Task or DispatchQueue.async work from a previous test still running when the next one starts.

Time / date dependence The SUT reads Date() or Calendar.current inline, so the test result depends on the real clock. Signs: test passes during the day but fails near midnight, or fails only under heavy load when wall-clock timing drifts.

Order dependence Test execution order is randomized in the -Default test plans ("testExecutionOrdering" : "random"), but the -Unit plans used for verification here run sequentially. To reproduce order-dependent flakiness, run against the -Default plan rather than -Unit.

Resource contention CoreData, Keychain, or file system state left behind by a previous run.

Step 4: Fix the test (or plan the SUT change)

Fixes in the test file only — apply and move on

Replacing bare delays with condition polling: Use the project's test helpers instead of a fixed sleep. These are available to all test targets:

• XCTest (subclasses of BaseBitwardenTestCase):

  • waitFor { condition } — spins the run loop; use when the SUT uses Timer.scheduledTimer
  • waitForAsync { condition } — polls asynchronously; use for Swift Concurrency-based SUT code

• Swift Testing (SwiftTestingHelpers.swift):

  • try await waitForAsync { condition } — polls asynchronously in @Test functions
  • waitFor(condition) — spins the run loop in @Test functions using @MainActor
  • await withContinuationTimeout { resume in … } — wraps callback-based async code so the test fails cleanly rather than hanging if the callback is never called

Shared state:

• Move subject and dependency properties from static var to instance var
• Reset all mutable properties in tearDown / addTeardownBlock
• Remove NotificationCenter observers: addTeardownBlock { NotificationCenter.default.removeObserver(token) }

CoreData / Keychain leaks:

• Use .memory store type when constructing AuthenticatorBridgeDataStore or any CoreData stack in tests so nothing persists to disk between runs