Test Reliability

<objective>
A retried test that still flakes will eventually fail 3-of-3 during your most critical release, and a silently auto-repaired test may now verify a different element entirely. This skill builds suites teams can trust: resilient locators, classified flakes, environment-aware healing, data healing, and observable repair with confidence scoring — every automated fix produces evidence a human can review.
</objective>

Quick Route

Discovery Questions

Check .agents/qa-project-context.md first — it carries known flaky areas, selector strategy, and CI environment details. Skip any question it already answers.

• What is your current flaky test rate? Check CI failure stats over the last 30 days. Below 2% is healthy; 2-5% needs attention; above 5% is eroding team trust.
• Where is the pain concentrated? Locator breakage? Timing? Test data? Environment? If unknown, instrument first (see Flake Classification).
• What is your current selector strategy? data-testid everywhere, mixed CSS and role-based, or no strategy (whatever works)? This sets the stability-score baseline.
• How do you handle flaky tests today? Retry and hope, skip and forget, or something structured? Decides how much process you need to add.
• What CI environment runs the tests? Same machine every time or different runners? Consistent or variable resources? Drives the environment-vs-test diagnosis.
• What is your test data strategy? Shared database, per-test fixtures, factory seeding, or external services? Decides whether data healing applies.

Core Principles

1. Prevention over cure. Writing a resilient test costs 1x. Investigating a flaky one costs 10x. Losing team trust in the suite costs 100x.

2. Healing must be observable and reviewable. Every automated repair produces evidence: what broke, what was tried, what worked, the confidence score. Silent fixes erode trust as fast as silent failures.

3. Classify before fixing. The fix for a timing issue is completely different from the fix for a data dependency. Wrong diagnosis wastes effort and can make things worse.

4. Flaky tests are bugs. Not annoyances to tolerate. A flaky test either has a test bug (fix the test), reveals an app bug (fix the app), or exposes an environment issue (fix the environment).

5. Track reliability as a metric, not a feeling. Measure flaky rate, mean time to heal, quarantine age, and selector stability. What gets measured gets fixed.

6. Self-healing is a spectrum. Start with resilient locators (Level 1), add fallback strategies (Level 2), then environment-aware healing (Level 3), then confidence-scored auto-repair (Level 4). Do not jump to Level 4 before mastering Level 1.

Locator Resilience

Multi-Attribute Selectors (Beyond Fallback Chains)

A single locator strategy is a single point of failure. Multi-attribute selectors combine multiple signals for one element lookup — resilience without fallback-chain complexity.

The key insight: instead of "try A, then B, then C," use "find element matching A AND B AND C with tolerance for one signal missing."

// Multi-attribute locator: tries combinations from most specific to least
const submitBtn = await multiAttributeLocator(page, {
  testId: 'checkout-submit',              // most stable signal
  role: 'button',                          // semantic signal
  name: /place order/i,                    // accessible name
  nearText: 'Order Summary',              // visual context
});
// Internally: tries testId+role+name first, then testId alone, then role+name,
// then text, then nearText+role. Returns first visible match.
// Unlike fallback chains, it combines signals for higher confidence.

DOM Similarity / Neighbor Context Matching

When a locator fails, the element may still exist with changed attributes. Use surrounding DOM context to find it:

1. Parent + tag + type: Find the container (by testId), then locate by tag and type within it.
2. Preceding label: Find sibling text (label), then locate the adjacent input/button.
3. Nearby text context: Find visible text near the target, then locate the element type in the same parent.

These are repair candidates scored by the confidence system below — not runtime fallbacks.

Selector Stability Scoring

Rate every selector on a 0-5 scale to prioritize refactoring.

Target: Average score of 3.5+ across the suite. Audit monthly. Prioritize fixing score-0 and score-1 selectors. Emit one score per locator to selector-stability.md (or a CI step) and report the suite average so the 3.5 target is verifiable, not asserted.

Flake Classification Framework

Every flaky test has a root cause category. Classifying correctly determines the fix.

Categories

Classification Decision Tree

Test is flaky
│
├── Does it pass when run alone?
│   ├── YES → ORDER DEPENDENCY or DATA DEPENDENCY
│   │   ├── Does another test create/modify data it needs? → ORDER DEPENDENCY
│   │   └── Does it share a database/file/cache? → DATA DEPENDENCY
│   │
│   └── NO → Not order/data dependent. Continue below.
│
├── Does it fail more often in CI than locally?
│   ├── YES → TIMING or ENVIRONMENT
│   │   ├── Timeout errors? → TIMING (CI is slower)
│   │   ├── Connection errors? → ENVIRONMENT (network latency / service)
│   │   └── Resource errors (OOM, disk)? → ENVIRONMENT (resource contention)
│   │
│   └── NO → Same rate locally and CI. Continue below.
│
├── Does it fail at specific times?
│   ├── YES → TIME SENSITIVITY
│   │   ├── Near midnight? → Date boundary issue
│   │   ├── Near month/year end? → Calendar calculation
│   │   └── Specific hour? → Timezone issue
│   │
│   └── NO → Continue below.
│
├── Does it involve screenshots or visual comparison?
│   ├── YES → VISUAL RENDERING
│   │
│   └── NO → Continue below.
│
├── Does it call external HTTP APIs?
│   ├── YES → EXTERNAL SERVICE
│   │
│   └── NO → TIMING (most likely — default classification)
│       └── Investigate: what async operation is not being awaited?

The fix per category lives in references/flaky-test-runbook.md (Step 4) with full code patterns.

Environment-Aware Healing

Not all test failures are test problems. Some are environment problems. Environment-aware healing distinguishes the two and adapts.

Slow Backend vs True UI Failure

When an action fails, check backend health before blaming the test:

1. Action fails → Hit /api/health.
2. Backend unhealthy (5xx or timeout) → Retry with exponential backoff. Diagnose as backend_down. This is not a UI bug.
3. Backend healthy (2xx) → This is a real UI/test failure. Do not retry.

Return a structured diagnosis: { success: boolean; diagnosis: 'backend_down' | 'ui_failure' | 'backend_slow_recovered' }. This feeds flake classification — backend issues are environment issues, not test bugs.

Resource Contention Detection

Before declaring a test failure in CI, check for resource contention:

• Browser health: Load about:blank. If it takes > 2s (baseline < 500ms), the runner is overloaded.
• API health: Hit /api/health. If it takes > 5s (baseline < 1s), the backend is under pressure.
• Diagnosis: If either check fails, classify as an environment issue and annotate the test result. Do not count resource-contention failures toward flaky test rates.

Data Healing

Test data expires, gets cleaned up, or becomes invalid. Data healing detects and regenerates stale test data.

Common Data Failure Patterns

Self-Healing Test Data Fixture Pattern

Build fixtures that verify data exists and regenerate if stale.

// Pattern: verify → heal → use → cleanup
testUser: async ({ request }, use, testInfo) => {
  // 1. Try to find existing test user by deterministic email
  // 2. Verify auth token is still valid (GET /api/me)
  // 3. If token expired → refresh it (POST /refresh-token), mark as healed
  // 4. If user missing → create new one, mark as healed
  // 5. If healed → annotate testInfo for observability
  // 6. use(user) → run the test
  // 7. Cleanup: delete test user (guaranteed by fixture, even on failure)
}

Key patterns:

• Use testInfo.testId in email/identifiers for per-test uniqueness.
• Annotate testInfo.annotations when healing occurs, for observability.
• Always clean up in the fixture's post-use block, not in afterEach — fixtures guarantee cleanup on failure.

Observable Repair Workflow

Core guardrail: Healing must be observable and reviewable. Every repair follows this flow:

Failure Detected
  │
  ▼
Candidate Repair Generated
  │
  ▼
Confidence Score Computed (0.0 - 1.0)
  │
  ▼
Evidence Diff Produced (what changed, what was tried)
  │
  ▼
Approval Policy Applied
  │ ├── Score >= 0.9   → Auto-apply, log for batch review
  │ ├── Score 0.7-0.89 → Apply in quarantine, flag for individual review
  │ ├── Score 0.5-0.69 → Do NOT apply, open PR with evidence for review
  │ └── Score < 0.5    → Discard, manual investigation required
  │
  ▼
Intent Fidelity Check (does repaired test still test the same thing?)
  │
  ▼
Rollback if intent fidelity drops

Confidence Scoring

Score each repair candidate on six dimensions (weighted sum, 0.0-1.0):

Score thresholds (half-open bands — 0.9 belongs to the auto-apply tier only):

• >= 0.9 — Auto-apply, log for batch review.
• 0.7-0.89 — Apply in quarantine, flag for individual review.
• 0.5-0.69 — Do not apply; open a PR with evidence for review.
• < 0.5 — Discard, manual investigation required.

references/flaky-test-runbook.md (Score Interpretation) uses these exact four bands — keep all three locations identical if you edit them.

Repair Evidence

Every repair produces an evidence record containing: test file, test name, failure type, original locator, candidate replacements (each with confidence, evidence string, and intent-preserved flag), which candidate was selected, timestamp, approval path, rollback trigger, and a screencast.webm recording of the repair run (Playwright 1.59+ page.screencast with showActions annotations — the "agentic video receipt"). Without the screencast, a Level-3/4 healed test asks reviewers to trust a JSON record; with it, the diff and the runtime are both inspectable.