Diagnose Job Run Symptoms

Diagnose Job Run Symptoms

Sippy Symptoms are known-failure signatures for OpenShift CI. A symptom is a rule made of a file pattern (a glob over a CI job run's artifact files, e.g. **/build-log.txt) and a matcher (string = substring, regex = regular expression, none = file merely exists, cel = a compound CEL expression over other label names). When a symptom matches a job run's artifacts, Sippy applies one or more Labels — human-readable tags like InfraFailure — to that run. Labels appear in the Sippy UI and Spyglass and help everyone quickly recognize known failure modes without re-debugging them. You do not need any prior Sippy knowledge to use this skill.

This skill takes a Prow job run URL and explains which symptoms matched the run and what each applied label means — including the matched file and text.

When to Use This Skill

Use this skill when:

• A CI job failed and you want to know if it is a known failure mode before debugging it from scratch
• You are about to create a new symptom and want to check what already matched the run (see manage-symptoms)
• You want a plain-language explanation of the labels shown on a run in the Sippy UI or Spyglass

Prerequisites

1. Default mode (already-applied labels): only network access to public GCS (https://storage.googleapis.com) and the public Sippy API (https://sippy.dptools.openshift.org) — no authentication required.

2. Deep mode (--deep, server-side dry-run rescan): a Bearer token from the DPCR cluster.

   • Must be logged into the DPCR cluster via oc login
   • Cluster API: https://api.cr.j7t7.p1.openshiftapps.com:6443
   • Use the oc-auth skill to obtain the token (see the token-acquisition snippet in reevaluate-job-runs/SKILL.md); prefer export SIPPY_TOKEN=... over --token — argv is visible in process listings

3. Python 3: Python 3.6 or later, standard library only.

Implementation Steps

Step 1: Default mode — explain already-applied labels

python3 plugins/ci/skills/diagnose-job-run-symptoms/diagnose_job_run.py \
  "https://prow.ci.openshift.org/view/gs/test-platform-results-public/logs/<job>/<build_id>"

For each applied label the summary output shows: the label title and explanation, links to associated Jira issues, the symptom that applied it (summary, matcher type, file pattern, match string), and the actual matched file and matched text from the run's artifacts.

Step 2: Deep mode — server-side rescan

Use when the run predates the current symptom set or shows no labels. This asks Sippy to re-scan the run with dry_run: true — it writes nothing and reports what would match now:

python3 plugins/ci/skills/diagnose-job-run-symptoms/diagnose_job_run.py \
  "<prow_url>" --deep

Note: deep mode reports label IDs only — the matched file/text detail is only available in default mode (it comes from the GCS artifacts). Deep mode submits an asynchronous batch, polls its same-origin status link through pending, processing, and running, and reports results after the batch reaches complete. A failed or cancelled batch is an error.

Step 3: Nothing matched?

If default mode finds no labels, first suggest --deep: the run may simply never have been scanned, and default mode cannot distinguish that from "scanned, nothing matched". If a deep rescan also finds nothing and the user has identified the failure cause, guide them to the manage-symptoms skill to create a new symptom (and reevaluate-job-runs to apply it retroactively to past runs).

Arguments:

• prow_url: Prow job run URL (https://prow.ci.openshift.org/view/gs/..., positional, required)

Options:

• --deep: Server-side dry-run rescan via the reevaluate API (requires a token)
• --token <token>: Bearer token from the oc-auth skill (only needed with --deep; optional if the SIPPY_TOKEN environment variable is set, which is preferred — argv is visible in process listings; --token takes precedence)
• --format json|summary: Output format (default: summary)