Triage Flaky Test

What it does

When your agent uses it

• : starting from a Sippy link/signal
• A named flaky test across runs — not for diagnosing a single failed Prow job before the failing step is known (use quay-prow-triage)
• A Playwright failure already isolated to one run (use debug-playwright-prow)

Example prompts

• “/triage-flaky-test”

Requirements

• Node.js
• Docker
• Pre-approved tools (allowed-tools): Bash(curl *), Bash(jq *), Bash(gcloud storage ls *), Bash(gcloud storage cp *), Bash(git log *), Bash(git diff *), Bash(grep *), Bash(npx playwright test *), Bash(bash .agents/skills/debug-playwright-prow/scripts/playwright-debug-prow.sh *), Read, Grep, AskUserQuestion

What it can do on your machine

Read from SKILL.md and the folder at commit dc3fedb. It shows what the files ask for, not the result of running them.

• Tool permissions

  Pre-approves these tools, so the agent can use them without asking each time:

  • Bash(curl *)
  • Bash(jq *)
  • Bash(gcloud storage ls *)
  • Bash(gcloud storage cp *)
  • Bash(git log *)
  • Bash(git diff *)
  • Bash(grep *)
  • Bash(npx playwright test *)
  • Bash(bash .agents/skills/debug-playwright-prow/scripts/playwright-debug-prow.sh *)
  • Read

  …and 2 more on the same allowed-tools line.

  From allowed-tools in the SKILL.md frontmatter.

• Runs code

  Shell commands in SKILL.md call:

  • jq
  • git
  • curl
  • bash
  • make

  From the folder's file list and the shell code blocks in SKILL.md.

• Network

  Hosts in commands or code, which the agent is likely to contact:

  • sippy.dptools.openshift.org

  From URLs in SKILL.md, links to its own repository left out.

• Credentials

  Names no API keys, tokens, secrets or passwords.

  From names ending in _API_KEY, _TOKEN, _SECRET, _KEY or _PASSWORD in SKILL.md.

Context cost

Triage Flaky Test loads about 2.6k tokens when it runs, and up to ~5.4k if it reads all its reference files. Until then it costs about 126 tokens; SKILL.md has 1,157 words of instructions outside code blocks.

Always · name and description, kept in context so the agent knows when to use it

~126

When it runs · the whole SKILL.md, loaded when a task matches

~2.6k

With references · SKILL.md plus every file in references/, read only if the agent opens them

~5.4k

Estimates: characters ÷ 4, the usual rule of thumb; real counts depend on the model's tokenizer. Scripts and assets cost tokens only if the agent reads them.

Safety

Automated static check — not a guarantee. Review scripts before installing. It scans the text of SKILL.md for risky patterns (piping downloads into a shell, reading credential files, hidden Unicode, destructive commands); files beside SKILL.md are not scanned.

SKILL.md

The full file from quay/quay at commit dc3fedb, republished under its Apache-2.0 licence (© quay). 1,157 words, ~2,625 tokens.

name

triage-flaky-test

description

Triage a flaky Playwright test end to end, from a Sippy signal to a written fix proposal: Sippy numbers and failing run URLs, Prow artifacts (or the access gap), the spec, a local reproduction, and a proposal in a fixed shape. Use when: starting from a Sippy link/signal or a named flaky test across runs — not for diagnosing a single failed Prow job before the failing step is known (use quay-prow-triage) or a Playwright failure already isolated to one run (use debug-playwright-prow).

allowed-tools

Bash(curl *), Bash(jq *), Bash(gcloud storage ls *), Bash(gcloud storage cp *), Bash(git log *), Bash(git diff *), Bash(grep *), Bash(npx playwright test *), Bash(bash .agents/skills/debug-playwright-prow/scripts/playwright-debug-prow.sh *), Read, Grep, AskUserQuestion

argument-hint

SIPPY_ANALYSIS_URL | "TEST_NAME" RELEASE

RELEASE=quay-3.18
TEST='Theme Switcher › auto theme respects browser color scheme preference'

jq -rn --arg r "$RELEASE" --arg t "$TEST" '
  [{columnField:"name",operatorValue:"equals",value:$t},
   {columnField:"variants",not:true,operatorValue:"has entry",value:"never-stable"},
   {columnField:"variants",not:true,operatorValue:"has entry",value:"aggregated"}]
  | {items:., linkOperator:"and"} | tojson | @uri
  | "https://sippy.dptools.openshift.org/sippy-ng/tests/\($r)/analysis?test=\($t|@uri)&filters=\(.)"'

curl -sS -G --connect-timeout 15 --max-time 60 \
  https://sippy.dptools.openshift.org/api/tests/details \
  --data-urlencode "release=$RELEASE" --data-urlencode "test=$TEST" | jq .

curl -sS -G --connect-timeout 15 --max-time 60 \
  https://sippy.dptools.openshift.org/api/tests/outputs \
  --data-urlencode "release=$RELEASE" --data-urlencode "test=$TEST" | jq .

jq -r '.tests | to_entries[0].value | to_entries[]
  | "\(.key) runs=\(.value.current_runs) flakes=\(.value.current_flakes) fail=\(.value.current_failures)"'

SPEC=web/playwright/e2e/ui/theme-switcher.spec.ts
for BR in origin/master origin/redhat-3.18; do
  git log -5 --date=short --format='%h %ad %s' "$BR" -- "$SPEC"
  git log -5 --date=short --format='%h %ad %s' "$BR" -- web/playwright/fixtures.ts web/playwright.config.ts
done

bash .agents/skills/debug-playwright-prow/scripts/playwright-debug-prow.sh <PROW_URL>

make DOCKER=podman local-dev-down
rm -rf "$ARTIFACTS_DIR"     # if the prow collector ran
rm -rf tmp/prow-artifacts   # if a manual download from references/prow-artifacts.md ran

© quay, Apache-2.0. Rendered from Markdown: HTML in the file is shown as text, images as links, and headings moved down two levels. Raw file

Files

SKILL.md and 3 other files (references) in .agents/skills/triage-flaky-test of quay/quay.

• SKILL.md
• references/proposal-format.md
• references/prow-artifacts.md
• references/reproduction.md

Open the folder on GitHubat commit dc3fedb