Manage Symptoms

Manage Symptoms

Sippy Symptoms are known-failure signatures for OpenShift CI. A symptom is a rule made of a file pattern (a glob over a CI job run's artifact files, e.g. **/build-log.txt) and a matcher (string = substring, regex = regular expression, none = file merely exists, cel = a compound CEL expression over other label names). When a symptom matches a job run's artifacts, Sippy applies one or more Labels — human-readable tags like InfraFailure — to that run. Labels appear in the Sippy UI and Spyglass and help everyone quickly recognize known failure modes without re-debugging them. You do not need any prior Sippy knowledge to use this skill.

Every symptom must always have at least one label. A symptom exists to label matching runs, so a symptom without a label serves no purpose. Even when the label reads the same string as another field (e.g. the summary), a label is still required — always supply --label-ids. Creating or updating a symptom with no labels is rejected by client-side validation.

Creating or updating a symptom does NOT apply its label to job runs that already finished. Symptom detection only runs automatically as new artifacts arrive, so a brand-new or changed symptom leaves already-completed runs unlabeled until you retroactively reevaluate them. After every create or update you MUST apply the label to the affected runs with the reevaluate-job-runs skill (see Step 8) — otherwise the label you defined never shows up in the Sippy UI or Spyglass, which is the most common reason a symptom "has a label but nothing is labeled."

When to Use This Skill

Use this skill when you need to:

• Auto-label a recurring failure pattern you found in CI logs, so future runs are recognized without re-debugging
• Correct an existing symptom's matcher, file pattern, or match string
• Retire an obsolete symptom that no longer applies

Prerequisites

1. OpenShift CLI Authentication: Required for authenticating to the sippy-auth API

   • Must be logged into the DPCR cluster via oc login
   • Cluster API: https://api.cr.j7t7.p1.openshiftapps.com:6443
   • Use the oc-auth skill to obtain the Bearer token

2. Python 3: Python 3.6 or later

   • Check: python3 --version
   • Uses only standard library (no external dependencies)

Implementation Steps

Step 1: Check for Duplicates

Before creating a symptom, search the existing catalog to avoid duplicates:

python3 plugins/ci/skills/list-symptoms/list_symptoms.py --search "AuthFailure" --format summary

If an equivalent symptom already exists, prefer updating it instead of creating a new one.

Step 2: Verify the Target Labels Exist

Symptoms can only reference labels that already exist:

python3 plugins/ci/skills/list-symptoms/list_symptoms.py --labels --format summary

If a label is missing, create it first with the manage-labels skill. (The script also verifies label IDs against the labels API before submitting; use --skip-label-check only if the labels API is unreachable.)

Step 3: Obtain Authentication Token

Use the oc-auth skill to obtain a Bearer token from the DPCR cluster:

# Get token from the DPCR cluster context
# The oc-auth skill's curl_with_token.sh uses this cluster for sippy-auth
DPCR_CLUSTER="https://api.cr.j7t7.p1.openshiftapps.com:6443"

# Find the oc context for the DPCR cluster and get the token
CONTEXT=$(oc config get-contexts -o name 2>/dev/null | while read -r ctx; do
  server=$(oc config view -o jsonpath="{.clusters[?(@.name=='$(oc config view -o jsonpath="{.contexts[?(@.name=='$ctx')].context.cluster}" 2>/dev/null)')].cluster.server}" 2>/dev/null || echo "")
  server_clean=$(echo "$server" | sed -E 's|^https?://||')
  if [ "$server_clean" = "api.cr.j7t7.p1.openshiftapps.com:6443" ]; then
    echo "$ctx"
    break
  fi
done)

if [ -z "$CONTEXT" ]; then
  echo "Error: Not logged into DPCR cluster. Please run: oc login $DPCR_CLUSTER"
  exit 1
fi

export SIPPY_TOKEN=$(oc whoami -t --context="$CONTEXT" 2>/dev/null)
if [ -z "$SIPPY_TOKEN" ]; then
  echo "Error: Failed to get token. Please re-authenticate to DPCR cluster."
  exit 1
fi

Prefer exporting SIPPY_TOKEN as above rather than passing --token on the command line — command-line arguments are visible in process listings. --token still works and takes precedence over the environment variable.

Step 4: Confirm the Payload with the User

Before any create or update, show the user the full payload that will be sent (summary, matcher type, file pattern, match string, label IDs) and get their confirmation. Before delete, you MUST show the symptom (list-symptoms --id <id> --format summary) and get explicit confirmation — never run delete without the user confirming the specific symptom.

A create or update is only complete once the label is applied to the affected runs (Step 8). Treat create/update + reevaluate as a single workflow, not two optional steps.

Step 5: Create a Symptom

python3 plugins/ci/skills/manage-symptoms/manage_symptoms.py create \
  --summary "AWS could not validate access credentials" \
  --matcher-type string \
  --file-pattern "build-log.txt" \
  --match-string "api error AuthFailure: AWS was not able to validate the provided access credentials" \
  --label-ids InfraFailure

The symptom id is generated by the server from the summary — do not pass --id on create. --label-ids is required: every symptom must apply at least one label. If no suitable label exists yet, create one first with the manage-labels skill.

Step 6: Update a Symptom

Only pass the flags you want to change — the script fetches the existing symptom and merges, because the API's PUT is a full replacement:

python3 plugins/ci/skills/manage-symptoms/manage_symptoms.py update \
  --id AWSCouldNotValidateAccessCredentials \
  --match-string "api error AuthFailure"

To change a symptom's labels on update, pass --label-ids with the new comma-separated list (omitting the flag preserves the existing labels). You cannot remove all labels — a symptom must always keep at least one label, so passing --label-ids "" (or any empty list) is rejected by validation. To retire a symptom entirely, delete it instead (Step 7).

Step 7: Delete a Symptom

Delete is a soft delete on the server side. Requires explicit user confirmation first (see Step 4):

python3 plugins/ci/skills/manage-symptoms/manage_symptoms.py delete \
  --id AWSCouldNotValidateAccessCredentials

Step 8: Apply the Label to Job Runs (Required after Create/Update)

Defining a symptom with a label is not enough — the label is only applied to a run when the symptom is evaluated against it. New/completed runs won't get the label until you retroactively reevaluate them with the reevaluate-job-runs skill. After every create or update, apply the label to the affected runs — do not stop at Step 7.

1. Preview first with --dry-run to confirm the symptom matches the runs you expect (writes nothing):

   python3 plugins/ci/skills/reevaluate-job-runs/reevaluate_job_runs.py \
     https://prow.ci.openshift.org/view/gs/test-platform-results-public/logs/<job>/<build_id> --dry-run --format summary

2. Rerun without --dry-run to actually write the labels, and confirm each run's response shows the label under labels_applied:

   python3 plugins/ci/skills/reevaluate-job-runs/reevaluate_job_runs.py <build_id> [<build_id> ...] --format summary

To label every run behind a triage or regression, collect the prowjob_run_ids (via the fetch-regression-details skill) and pass them all to reevaluate-job-runs — see that skill's "Bulk workflow" section. Reevaluation is idempotent, so it is safe to rerun.