Running Visual Regression Tests

Running Visual Regression Tests

Use this skill when appearance is the thing under test and functional assertions cannot express it: a broken layout at one breakpoint, a theme token that stopped applying, a component that shifted three pixels and swallowed a button.

Visual testing has one failure mode that dominates all others: a suite so noisy that --update-snapshots becomes reflex. At that point the baselines record whatever the code currently does, the diffs are never read, and the suite costs money while proving nothing. Everything below exists to prevent that.

When to Use

• styling and layout regressions reach production while the functional suite stays green
• snapshots pass locally and fail in CI on every run
• baselines get regenerated wholesale because nobody can tell signal from noise
• a design system or theme change needs a blast-radius check
• deciding whether a case belongs in a visual test, an assertion, or an accessibility check

Operating Principles

• A visual test earns its place only if a functional assertion cannot do the job. Text content, disabled state, and element presence are assertions, not pixels.
• Determinism before comparison. Fix fonts, animation, time, data, and viewport first. Threshold tuning applied to an unstable page hides real diffs along with the noise.
• Baselines are generated in one environment. The container that produces them is the container that compares them.
• Every diff is looked at by a human before it becomes a baseline. A bulk update with no review is the failure mode, not a shortcut.
• Snapshot the smallest surface that carries the risk. Component snapshots localize failures; full-page snapshots turn one change into forty red tests.
• A masked region is a documented decision. Masking is how the suite stays quiet, and also how coverage silently disappears.

Workflow

Phase 0: Decide whether pixels are the right tool

Run through this before writing anything.

If most rows land outside the visual column, say so and stop. A visual suite added to compensate for missing assertions makes both worse.

Phase 1: Stabilize the page

Nothing about thresholds until this phase is finished. ./resources/masking-and-stabilization.md has the code for each item.

• Fonts: wait for document.fonts.ready, or self-host and preload. A late webfont swap is the single most common cause of a one-off diff.
• Animation and transitions: disable globally via injected CSS or animations: 'disabled'.
• Time: pin the clock so relative timestamps stop moving. See mocking-network-and-time.
• Data: seed deterministically. A list ordered by an unstable sort key produces a real diff every run.
• Network: stub anything that varies, including avatars, ads, and telemetry.
• Viewport and device scale: set explicitly per project.
• Scrollbars and caret: hide them; they differ across platforms and focus states.
• Randomized content: ids, session tokens, generated names.

The completion criterion for this phase is concrete: the same test run three times in a row on the same commit produces zero diffs. Do not proceed until that holds.

Phase 2: Choose the snapshot surface

Default to region-level. Add full-page snapshots for a small set of layout-critical routes, and know that they will be the noisiest tests in the suite.

Phase 3: Set the baseline policy

Decide and record, using ./resources/baseline-policy.md:

• Where baselines are generated: a container image pinned by digest, matching the CI runner. Locally generated baselines and CI comparison is the second most common cause of permanent redness.
• Which platforms have baselines: one per project (browser, viewport, theme, locale). Every added axis multiplies the file count and the review burden.
• Where baselines live: in the repository with the tests, or in a hosted service. In-repo is simpler and grows the repository; state which you chose.
• Who may update them: the rule that stops reflex updates. A pull request that changes baselines carries the rendered before-and-after in its description.

Phase 4: Configure comparison

Start strict and loosen only against observed noise.

expect: {
  toHaveScreenshot: {
    maxDiffPixelRatio: 0.001,
    threshold: 0.2,          // per-pixel colour tolerance
    animations: 'disabled',
    caret: 'hide',
    scale: 'css',
  },
},

Rules for loosening:

• change one knob at a time and record which noise it addressed
• prefer masking a known-dynamic region over raising the global ratio
• a maxDiffPixelRatio above roughly 0.01 hides a missing button; if you need that much, go back to Phase 1
• never set a per-test tolerance without a comment naming the source of the noise

Config snippets, including the anti-aliasing and platform-difference cases, are in ./resources/visual-config-recipes.md.