Aicr Uat Report

Step 2b — Pull debug bundles (only for product-signal failures)

Skip this step entirely for a routine pass-rate report. Run it when the user asks why something failed, or when Step 2 found a test-phase failure worth root-causing:

python3 .agents/skills/aicr-uat-report/uat_report.py --days 3 \
  --download-debug /tmp/uat-debug --max-downloads 3

Prefer --run <id> (repeatable) over raising --max-downloads: usually only the latest failure per failing combo is worth reading.

Bundles land in <dir>/<service>-<gpu>-<intent>-<run_id>/, each with a printed digest — MANIFEST head, failing checks from report.json, and a one-line contents summary. Read that summary for presence, not file names: a missing evidence/ or report.json says the run died before that stage, which is often the whole diagnosis. For a readiness-gate failure the digest also echoes the last --- failed validator output block from readiness-gate.log; its Failed resources: lines name the resources that never became ready, so start with the logs of the operator that owns them. Then open files per the Debug bundles table above.

Cite file_path:line and the run ID for every finding. Leave the download directory in place — the user may want to keep digging.

Step 3 — Render the report

Produce exactly two artifacts, in this order (see Output Format Reference). Sort the table worst-first: lowest pass ratio at the top; bold the Service/GPU/Intent cells of rows with product-signal failures. Include run URLs as links for at least the most recent failure of each failing combo.

Step 4 — Write the RC validation input

A numbered priority list derived from the table:

1. Combos with consistent test-phase failures (0/N or repeated validate-phase or readiness-gate failures) — top manual-validation priority. For readiness failures, name the failing validators and the component that owns them.
2. Combos whose most recent failure is test-phase (even if earlier ones were infra) — deserve a close look.
3. Combos with infra/CI-only failures — noisy, not product signal; note them but rank low.
4. Reservations in bring-up (e.g. GB200 while nightly-intents: [], kind lane) — expected churn, call out separately.
5. Green combos — state them explicitly as lowest priority; a clean bill is information too.

Output Format Reference

## UAT report against `main` (<start>–<end>, N runs)

| Service | GPU | Intent | Pass | Failure nature |
|---|---|---|---|---|
| **AKS** | **H100** | **training** | **0/4** | Real test failures — every run fails at "UAT - validate (all phases)" ([latest](<url>)) |
| EKS | H100 | training | 2/5 | Infra only — 2x bring-up, 1x Buildx CI flake; no test-phase failures |
| GKE | H100 | training | 4/4 | Green |

## RC validation input

1. **<Service>/<GPU>/<intent> is the clear red flag** — <pass ratio,
   failure signature, latest run link, whether it also fails on release
   tags (env issue) or only main (regression candidate)>.
2. ...
N. **<green combos> are solidly green** — lowest manual-testing priority.

Keep failure-nature cells to one sentence; detail beyond that belongs in the RC list, not the table.

Failure Modes

• gh run list returns nothing — window may predate retention or the workflow was renamed; say so rather than reporting "all green".
• Unparsed titles in the Notes line — the run-name format in uat-run.yaml changed; read the workflow's current format string and update NEW_TITLE/OLD_TITLE in uat_report.py in the same PR.
• Unknown reservation (e.g. new cloud) — the script falls back to the uppercased cloud token as the service name; cross-check new rows against infra/uat/reservations.yaml.
• A combo has very few runs (e.g. 0/1) — flag low sample size instead of declaring it broken.
• "no debug artifact" — expected for bring-up/Buildx/ingest failures (see Debug bundles). Report it as corroborating the infra classification; do not present it as a tooling problem.
• "debug artifact expired" — the window exceeds the 30-day retention. Nothing to recover; note it and work from failed step names.
• cluster-debug/ missing or thin — the collector is best-effort and its cloud credentials can expire on a long failure. Say the bundle is incomplete; do not read it as the cluster being healthy.
• Readiness gate failed for an infra reason — the gate talks to the API server on every attempt, so expired cloud credentials or an unreachable control plane also fail it. If the failed-validator block is absent or its messages are connection/auth errors rather than resources that are not ready, reclassify to infra.
• Failing step name disagrees with report.json — trust the bundle. UAT - validate (all phases) + emit signed evidence covers two concerns, so N/N passing checks under a failed step means the evidence leg failed, not the product. Reclassify to infra before ranking it in Step 4.

What This Skill Does NOT Do

• Does not dispatch, re-run, or cancel workflow runs
• Does not download raw job logs (gh run view --log); it reports failed step names and, on request, the uploaded cluster debug bundles
• Does not modify reservations, workflows, or any in-repo file — the only writes are downloaded artifacts under the --download-debug directory