QA Dashboard

<objective>
Build dashboards that drive decisions, not dashboards that display data. The failure
mode this prevents: a wall of panels showing "5 failures" with no link to the failures,
no target, and no trend — a notification dressed up as a tool that everyone stops opening
by week three. This skill delivers audience-specific QA reporting — rich HTML reports
(Allure), real-time trend panels with regression alerts (Grafana/InfluxDB), self-hosted
AI-assisted aggregation (ReportPortal), and automated stakeholder summaries — each panel
mapped to a question someone actually asks and every red indicator drilling down to the
failing test.
</objective>

Quick Route

Discovery Questions

Check .agents/qa-project-context.md first — if it exists, use it and skip anything answered there. Then:

1. What tool do you use for test reporting today? Console output, JUnit XML, HTML reports, or a dedicated platform? Identifies the starting point and how much plumbing is left.
2. Who will look at the dashboard? Developers need failure details and traces; QA leads need trends and flakiness; leadership needs release confidence and defect rates. Each audience needs a different view.
3. What decisions should the dashboard drive? "Is this build safe to release?" "Which tests need fixing?" "Is quality improving sprint over sprint?" Dashboards without a decision context become shelfware.
4. Where do test results live? GitHub Actions artifacts, S3, a database? The storage location determines which dashboard tool is practical.
5. What CI platform? GitHub Actions, GitLab CI, Jenkins? Each has different artifact and reporting integrations.

Core Principles

1. Dashboards answer questions, they do not display numbers. Every panel must map to a question someone actually asks. "What is the flakiness rate?" is a question. "Total test count" is trivia.

2. Different audiences need different views. A developer debugging a CI failure needs stack traces, screenshots, and traces. A VP needs a single number: "Are we ready to release?" Do not force both through the same dashboard.

3. Real-time for CI, trends for leadership. CI dashboards update on every pipeline run. Leadership dashboards aggregate weekly or per-sprint. Mixing cadences confuses both audiences.

4. Drill-down to action. Every red indicator must link to the specific failing test, the specific flaky test, or the specific coverage gap. A dashboard that shows "5 failures" but does not link to the failures is a notification, not a tool.

5. Automate report generation. Reports that require manual effort (running scripts, copying data, formatting slides) will not survive the first busy sprint. Generate reports from CI pipelines automatically.

Allure Report

Allure generates rich HTML reports from test results with history, categories, and retries built in. It works with Playwright, Jest, Vitest, pytest, and most frameworks.

Allure 2 vs Allure 3. Two paths, and they are easy to mix up because the framework adapters (allure-playwright, allure-vitest, allure-jest) emit the same Allure 2 result files; only the reader differs:

• Allure 2 path — allure-commandline (2.42.1, Jun 2026). brew install allure installs this, not v3. Commands: allure generate / allure open / allure serve; categories via a categories.json dropped into allure-results/. Stable, frozen, dependency-bump-only now.
• Allure 3 path — allure npm package + allurerc.mjs (3.9.0, May 2026). TypeScript rewrite: plugins, single-file config, real-time allure watch, project-wide quality gates, multi-environment reports, and Allure Service for server-side history. Commands: npx allure run / npx allure generate / npx allure watch; categories move into allurerc.mjs (the dropped-in categories.json is a v2 concept).

Choose Allure 3 for new projects. If you follow only the brew install allure / allure generate commands you are on the v2 path — that is fine and fully supported; just know which one you are running.

Minimal Playwright reporter wiring — register the adapter as reporter: ["allure-playwright", { outputFolder, environmentInfo }] so every run drops results into allure-results/ with the environment captured:

// playwright.config.ts
reporter: [
  ["list"],
  ["allure-playwright", {
    outputFolder: "allure-results",
    environmentInfo: {
      Environment: process.env.TEST_ENV ?? "local",
      BaseURL: process.env.BASE_URL ?? "http://localhost:3000",
    },
  }],
],

Add per-test metadata (allure.severity / feature / story / tag) to drive grouping, define failure categories to split product bugs from infra breakage, and preserve history/ across CI runs so trends exist at all. Without history an Allure report is a single snapshot — no trends, no intermittent-failure detection. See references/allure.md for the full Playwright/Vitest configs, the v2 categories.json, the v3 allurerc.mjs + allure run runnable path, and the GitHub Actions history-preservation steps.

Grafana Dashboards

Grafana gives real-time dashboards with alerting. Best when the team already runs Grafana for infrastructure and wants test metrics next to production metrics.

Data pipeline: a post-test CI step parses results (JUnit XML, coverage JSON, timing) and pushes points to a time-series DB (InfluxDB or a Prometheus pushgateway); Grafana queries those. The push script writes two measurements — test_execution (one point per test, tagged by suite/test_name/status/branch/run_id) and test_run_summary (one point per run with pass_rate, total, failed, avg_duration_ms). Tag every point with branch and run_id so panels can filter to main and link back to a specific run.

The script runs under if: always(), so wrap the write loop in try/flush/close — a throw mid-loop otherwise loses every buffered point. See references/grafana.md for the full push-test-metrics.ts (with the flush guard) and the GitHub Actions step.

Recommended panels

Full queries plus Coverage Trend and Duration Distribution panels are in references/grafana.md.

Alerting

Provision alert rules as YAML under provisioning/alerting/ (Grafana 11+). The one the Done When requires: main pass rate drops more than 2 percentage points in a single day. Build it from two queries (mean pass_rate over the last 1d vs the day before) feeding a math expression $yesterday - $today > 2, then route via a Slack contact point (incoming-webhook URL). The full provisioned rule + contact point is in references/grafana.md. Also alert on: pass rate below 95% (10m window), CI duration above 15 min, coverage drop >2% in a week. Dashboards are for investigation; alerts are for detection — a dashboard no one opens catches nothing.

ReportPortal

Self-hosted test reporting platform with ML-powered failure analysis, cross-framework aggregation, and real-time dashboards.

# Pin to a tagged release (current 26.0.x line: 26.0.3). The `master` branch may not
# match the supported 26.x line.
curl -LO https://raw.githubusercontent.com/reportportal/reportportal/26.0.3/docker-compose.yml
docker compose up -d
# Access at http://localhost:8080 (default: superadmin/erebus)
# ML auto-analysis lives in the `service-auto-analyzer` container — confirm it is running.

As of 26.0.3 ReportPortal also ingests agentic test results (launches carry an AGENTIC vs AUTOMATION execution-type badge) — relevant if part of your suite runs through Claude Code or another agent.

Playwright integration

// playwright.config.ts
reporter: [
  ["list"],
  ["@reportportal/agent-js-playwright", {
    apiKey: process.env.RP_API_KEY,
    endpoint: process.env.RP_ENDPOINT ?? "http://localhost:8080/api/v1",
    project: "my-project",
    launch: `E2E Tests - ${process.env.CI ? "CI" : "local"}`,
    attributes: [
      { key: "branch", value: process.env.GITHUB_HEAD_REF ?? "local" },
      { key: "build", value: process.env.GITHUB_RUN_ID ?? "dev" },
    ],
  }],
],

Install with npm i -D @reportportal/agent-js-playwright.

Quality gates are queryable after a run (GET /api/v1/$PROJECT/launch/$LAUNCH_ID/quality-gate) — use the status as a CI gate and fail the pipeline if it is not PASSED.

Allure TestOps (managed alternative)

If self-hosting feels heavy, Allure TestOps (26.2.x line, 2026) is the SaaS path: Allure 3 quality gates, named environments, global attachments, and Allure 3-style flaky detection (flags a test once it shows ≥3 status transitions across its last 10 runs). Its MCP server is in public beta (26.1.1), letting AI agents query launches and quality gates directly — relevant when your QA workflow runs through Claude Code / Cursor.