QA Verify Console

qa-verify-console — Visual QA Verification for OpenShift Console PRs

Overview

This skill captures before/after screenshots of an OpenShift Console PR running against a live cluster, generates flicker GIFs to highlight visual differences, and posts evidence to the PR. It enables rapid visual QA without manual browser testing.

"Agent" throughout this document refers to the Claude Code agent executing the skill — the LLM-driven process that runs commands, makes decisions about which routes to capture, and interprets results.

Post-PR design: This skill operates on an already-published pull request. It requires a valid PR number so it can fetch both the head and base branches, read the diff to identify affected routes, and post evidence back as a PR comment. It is not designed for pre-PR / local-branch workflows.

The agent:

1. Clones openshift/console, builds both the base and PR branches
2. Runs the console bridge (dev server) against a real OCP cluster
3. Uses Puppeteer to capture screenshots of affected routes
4. Compares baseline vs candidate screenshots and posts evidence to the PR

Prerequisites

Recommended Pod Resources

CPU:               8 cores
Memory:            16Gi
Ephemeral Storage: 40Gi

The console frontend build (yarn install --immutable && yarn run build) is CPU- and memory-intensive. The Go backend build is lighter but still benefits from multiple cores.

Inputs

The agent receives two inputs:

1. PR number — The openshift/console pull request to verify (e.g., 17199)
2. OC login command — A full oc login command string with token and server URL, passed explicitly as the OC_LOGIN_CMD environment variable. This is not an ambient kubeconfig assumption — the caller must provide a complete oc login --token=... --server=... command that the script evals to authenticate against the target cluster.

Optional:

3. Routes — Comma-separated list of console routes to capture. If not provided, the agent analyzes the PR diff to determine affected routes.

Evidence Directory Layout

/workspace/evidence/
  baseline/           # Screenshots from the base branch
    overview.png
    k8s_cluster_nodes.png
    ...
  candidate/          # Screenshots from the PR branch
    overview.png
    k8s_cluster_nodes.png
    ...
  flicker/            # Animated GIFs showing differences
    overview.gif
    k8s_cluster_nodes.gif
    ...
  metadata.json       # PR metadata (branches, author, etc.)
  baseline.json       # Capture summary for baseline
  candidate.json      # Capture summary for candidate

Phases

Phases 0–1 — Validate Inputs & Setup Environment

Fully automated by scripts/setup.sh — the agent just runs it and checks the exit code. No LLM decisions needed.

OC_LOGIN_CMD="oc login --token=... --server=..." \
  bash /workspace/qa-verify-console/scripts/setup.sh "$PR_NUMBER"

On success (exit 0), the script produces:

• /workspace/evidence/metadata.json — PR metadata
• /workspace/console/ — cloned repo with base-branch checked out and pr-branch available
• /workspace/evidence/setup-env.sh — sourceable env vars (LD_LIBRARY_PATH, CHROME_BIN, QA_TOOLS_DIR, branch refs)

The agent should source /workspace/evidence/setup-env.sh before proceeding.

On failure (non-zero exit), setup.sh prints a descriptive error to stderr. The agent should report the error and stop.

Phase 2 — Determine Routes & Capture Baseline

Goal: Decide which console routes to screenshot, then build the base branch and capture them.

Agent decisions (LLM judgement required):

1. Route selection — Analyze the PR diff (gh pr diff "$PR_NUMBER") to determine which console routes are visually affected by the changes. Consider:

   • Which React components are modified and what routes render them
   • Whether the change is cosmetic (CSS/layout) vs. behavioral
   • Common route patterns:
     • / (overview/dashboard)
     • /k8s/cluster/nodes, /k8s/cluster/projects
     • /k8s/ns/<namespace>/pods
     • /monitoring/alerts, /settings/cluster
   • If the diff doesn't clearly map to routes, ask the user

2. Failure triage — If the build or bridge fails, read the error log and decide whether the failure is environmental (retry/report) or a real issue.

Mechanical steps (run these; no decisions needed):

cd /workspace/console && ./build.sh                       # Build backend + frontend
export BRIDGE_USER_AUTH="disabled"                         # Required for headless mode
source ./contrib/oc-environment.sh                         # Sets BRIDGE_K8S_AUTH_BEARER_TOKEN
nohup ./bin/bridge -branding openshift > /tmp/bridge-baseline.log 2>&1 &
# Wait for HTTP 200 on localhost:9000, then capture:
# NOTE: Replace the example routes below with the routes you selected from PR diff analysis in Phase 2.
node /workspace/qa-verify-console/scripts/capture-screenshots.js \
  --routes "/" --routes "/k8s/cluster/nodes" \
  --output-dir /workspace/evidence/baseline \
  --base-url http://localhost:9000
# Kill bridge after capture

Save the JSON summary to /workspace/evidence/baseline.json.

Phase 3 — Capture Candidate (PR Branch)

Goal: Switch to the PR branch, rebuild, and capture the same routes.

Agent decisions: None — use the same route list from Phase 2. The only judgement call is failure triage: if the PR branch fails to build, that likely indicates a real issue with the PR (report it rather than retrying).

Mechanical steps:

cd /workspace/console && git checkout pr-branch
./build.sh
export BRIDGE_USER_AUTH="disabled"
source ./contrib/oc-environment.sh
nohup ./bin/bridge -branding openshift > /tmp/bridge-candidate.log 2>&1 &
# Wait for HTTP 200, then capture same routes as Phase 2
node /workspace/qa-verify-console/scripts/capture-screenshots.js \
  --routes "/" --routes "/k8s/cluster/nodes" \
  --output-dir /workspace/evidence/candidate \
  --base-url http://localhost:9000
# Kill bridge after capture

Save the JSON summary to /workspace/evidence/candidate.json.

Phase 4 — Compare Results & Publish Evidence

Goal: Generate visual comparisons and post evidence to the PR.

Agent decisions (LLM judgement required):

1. Interpret visual differences — Compare baseline vs. candidate screenshots (file size deltas, visual inspection). Decide which routes show meaningful visual changes (>5% file size difference is a useful heuristic) vs. noise.

2. Compose the PR comment — Summarize what changed visually, highlight routes with significant differences, and note any routes that failed to capture. Use console's build-comment-attach.sh if available, otherwise fall back to gh pr comment with embedded image links.

3. Handle partial results — If some screenshots are missing or GIF generation fails, decide what to include in the report and what to note as skipped. Post what is available rather than failing entirely.

Mechanical steps:

• Generate flicker GIFs for each baseline/candidate pair (use console's make-flicker-gif.sh if available, otherwise convert -delay 100 -loop 0)
• Upload images via stage-attachments.sh if available
• Post the PR comment via gh pr comment
• Store evidence summaries in the result store for coordinator access