---
name: manage-symptoms
description: Create, update, or delete Sippy Symptoms — known CI failure signatures that automatically label job runs — via the authenticated Sippy API
---

# Manage Symptoms

Sippy Symptoms are known-failure signatures for OpenShift CI. A symptom is a rule made of a file pattern (a glob over a CI job run's artifact files, e.g. `**/build-log.txt`) and a matcher (`string` = substring, `regex` = regular expression, `none` = file merely exists, `cel` = a compound CEL expression over other label names). When a symptom matches a job run's artifacts, Sippy applies one or more **Labels** — human-readable tags like `InfraFailure` — to that run. Labels appear in the Sippy UI and Spyglass and help everyone quickly recognize known failure modes without re-debugging them. You do not need any prior Sippy knowledge to use this skill.

**Every symptom must always have at least one label.** A symptom exists to label matching runs, so a symptom without a label serves no purpose. Even when the label reads the same string as another field (e.g. the summary), a label is still required — always supply `--label-ids`. Creating or updating a symptom with no labels is rejected by client-side validation.

**Creating or updating a symptom does NOT apply its label to job runs that already finished.** Symptom detection only runs automatically as new artifacts arrive, so a brand-new or changed symptom leaves already-completed runs unlabeled until you retroactively reevaluate them. After every create or update you MUST apply the label to the affected runs with the `reevaluate-job-runs` skill (see Step 8) — otherwise the label you defined never shows up in the Sippy UI or Spyglass, which is the most common reason a symptom "has a label but nothing is labeled."

## When to Use This Skill

Use this skill when you need to:

- Auto-label a recurring failure pattern you found in CI logs, so future runs are recognized without re-debugging
- Correct an existing symptom's matcher, file pattern, or match string
- Retire an obsolete symptom that no longer applies

## Prerequisites

1. **OpenShift CLI Authentication**: Required for authenticating to the sippy-auth API
   - Must be logged into the DPCR cluster via `oc login`
   - Cluster API: `https://api.cr.j7t7.p1.openshiftapps.com:6443`
   - Use the `oc-auth` skill to obtain the Bearer token

2. **Python 3**: Python 3.6 or later
   - Check: `python3 --version`
   - Uses only standard library (no external dependencies)

## Implementation Steps

### Step 1: Check for Duplicates

Before creating a symptom, search the existing catalog to avoid duplicates:

```bash
python3 plugins/ci/skills/list-symptoms/list_symptoms.py --search "AuthFailure" --format summary
```

If an equivalent symptom already exists, prefer updating it instead of creating a new one.

### Step 2: Verify the Target Labels Exist

Symptoms can only reference labels that already exist:

```bash
python3 plugins/ci/skills/list-symptoms/list_symptoms.py --labels --format summary
```

If a label is missing, create it first with the `manage-labels` skill. (The script also verifies label IDs against the labels API before submitting; use `--skip-label-check` only if the labels API is unreachable.)

### Step 3: Obtain Authentication Token

Use the `oc-auth` skill to obtain a Bearer token from the DPCR cluster:

```bash
# Get token from the DPCR cluster context
# The oc-auth skill's curl_with_token.sh uses this cluster for sippy-auth
DPCR_CLUSTER="https://api.cr.j7t7.p1.openshiftapps.com:6443"

# Find the oc context for the DPCR cluster and get the token
CONTEXT=$(oc config get-contexts -o name 2>/dev/null | while read -r ctx; do
  server=$(oc config view -o jsonpath="{.clusters[?(@.name=='$(oc config view -o jsonpath="{.contexts[?(@.name=='$ctx')].context.cluster}" 2>/dev/null)')].cluster.server}" 2>/dev/null || echo "")
  server_clean=$(echo "$server" | sed -E 's|^https?://||')
  if [ "$server_clean" = "api.cr.j7t7.p1.openshiftapps.com:6443" ]; then
    echo "$ctx"
    break
  fi
done)

if [ -z "$CONTEXT" ]; then
  echo "Error: Not logged into DPCR cluster. Please run: oc login $DPCR_CLUSTER"
  exit 1
fi

export SIPPY_TOKEN=$(oc whoami -t --context="$CONTEXT" 2>/dev/null)
if [ -z "$SIPPY_TOKEN" ]; then
  echo "Error: Failed to get token. Please re-authenticate to DPCR cluster."
  exit 1
fi
```

Prefer exporting `SIPPY_TOKEN` as above rather than passing `--token` on the command line — command-line arguments are visible in process listings. `--token` still works and takes precedence over the environment variable.

### Step 4: Confirm the Payload with the User

**Before any create or update, show the user the full payload that will be sent (summary, matcher type, file pattern, match string, label IDs) and get their confirmation. Before delete, you MUST show the symptom (`list-symptoms --id <id> --format summary`) and get explicit confirmation — never run delete without the user confirming the specific symptom.**

A create or update is only complete once the label is applied to the affected runs (Step 8). Treat create/update + reevaluate as a single workflow, not two optional steps.

### Step 5: Create a Symptom

```bash
python3 plugins/ci/skills/manage-symptoms/manage_symptoms.py create \
  --summary "AWS could not validate access credentials" \
  --matcher-type string \
  --file-pattern "build-log.txt" \
  --match-string "api error AuthFailure: AWS was not able to validate the provided access credentials" \
  --label-ids InfraFailure
```

The symptom `id` is generated by the server from the summary — do not pass `--id` on create. `--label-ids` is required: every symptom must apply at least one label. If no suitable label exists yet, create one first with the `manage-labels` skill.

### Step 6: Update a Symptom

Only pass the flags you want to change — the script fetches the existing symptom and merges, because the API's PUT is a full replacement:

```bash
python3 plugins/ci/skills/manage-symptoms/manage_symptoms.py update \
  --id AWSCouldNotValidateAccessCredentials \
  --match-string "api error AuthFailure"
```

To change a symptom's labels on update, pass `--label-ids` with the new comma-separated list (omitting the flag preserves the existing labels). You cannot remove all labels — a symptom must always keep at least one label, so passing `--label-ids ""` (or any empty list) is rejected by validation. To retire a symptom entirely, delete it instead (Step 7).

### Step 7: Delete a Symptom

Delete is a soft delete on the server side. Requires explicit user confirmation first (see Step 4):

```bash
python3 plugins/ci/skills/manage-symptoms/manage_symptoms.py delete \
  --id AWSCouldNotValidateAccessCredentials
```

### Step 8: Apply the Label to Job Runs (Required after Create/Update)

Defining a symptom with a label is not enough — the label is only applied to a run when the symptom is evaluated against it. New/completed runs won't get the label until you retroactively reevaluate them with the `reevaluate-job-runs` skill. **After every create or update, apply the label to the affected runs — do not stop at Step 7.**

1. Preview first with `--dry-run` to confirm the symptom matches the runs you expect (writes nothing):

   ```bash
   python3 plugins/ci/skills/reevaluate-job-runs/reevaluate_job_runs.py \
     https://prow.ci.openshift.org/view/gs/test-platform-results-public/logs/<job>/<build_id> --dry-run --format summary
   ```

2. Rerun without `--dry-run` to actually write the labels, and confirm each run's response shows the label under `labels_applied`:

   ```bash
   python3 plugins/ci/skills/reevaluate-job-runs/reevaluate_job_runs.py <build_id> [<build_id> ...] --format summary
   ```

To label every run behind a triage or regression, collect the `prowjob_run_id`s (via the `fetch-regression-details` skill) and pass them all to `reevaluate-job-runs` — see that skill's "Bulk workflow" section. Reevaluation is idempotent, so it is safe to rerun.

### Step 9: Verify

After create/update, verify the symptom itself:

```bash
python3 plugins/ci/skills/list-symptoms/list_symptoms.py --id <new-id> --format summary
```

Then confirm the label actually landed on a run — either from the `labels_applied` field in the Step 8 response, or with the `diagnose-job-run-symptoms` skill on a known-affected run. A symptom that lists a label but shows no `labels_applied` after reevaluation means the matcher/file-pattern is not matching — revisit Step 6.

**Arguments**:
- `action`: `create`, `update`, or `delete` (positional, required)

**Options**:
- `--token <token>`: Bearer token from the oc-auth skill (optional if the `SIPPY_TOKEN` environment variable is set, which is preferred — argv is visible in process listings; `--token` takes precedence)
- `--id <id>`: Symptom ID (required for update/delete; server-generated on create)
- `--summary <text>`: Short unique description (required for create, max 200 characters)
- `--matcher-type string|regex|none|cel`: How the match string is interpreted
- `--file-pattern <glob>`: Artifact glob, e.g. `**/build-log.txt` (required for non-CEL matchers)
- `--match-string <text>`: Substring, regex, or CEL expression
- `--label-ids <list>`: Comma-separated label IDs to apply on match (required — a symptom must always apply at least one label)
- `--skip-label-check`: Skip verifying label IDs against the labels API
- `--format json|summary`: Output format (default: json)

## API Details

**Base URL (writes)**: `https://sippy-auth.dptools.openshift.org/api/jobs/symptoms`

- Create: `POST /api/jobs/symptoms`
- Update: `PUT /api/jobs/symptoms/{id}` (full replacement — the script fetches the existing symptom and merges your changes, so only pass flags you want to change)
- Delete: `DELETE /api/jobs/symptoms/{id}` (soft delete)

**Authentication**: `Authorization: Bearer <token>` from the DPCR cluster.

**Symptom fields**:

| Field | Description |
|-------|-------------|
| `id` | Immutable identifier, generated from the summary on create |
| `summary` | Required, unique, max 200 characters |
| `matcher_type` | One of `string`, `regex`, `none`, `cel` |
| `file_pattern` | Artifact glob; required for all matcher types except `cel` |
| `match_string` | Required for `string`/`regex`/`cel`; not used by `none` (file merely exists) |
| `label_ids` | Label IDs applied on match; required (at least one) and must reference existing labels |
| `created_by`, `updated_by`, timestamps | Metadata set by the server |

**Matcher-type rules**:
- `string` / `regex`: require both `file_pattern` and `match_string`
- `none`: requires only `file_pattern` (matches when the file exists)
- `cel`: requires only `match_string` (a CEL expression over other label names)

## Error Handling

- **Client-side validation**: Missing summary, over-long summary, invalid matcher type, missing file_pattern/match_string for the chosen matcher, or missing labels (every symptom must have at least one) are caught locally before any request (exit 1).
- **Label not found**: If a `--label-ids` value does not exist, validation fails and points you to the `manage-labels` skill to create it first.
- **401/403**: Token missing or expired — refresh it via the `oc-auth` skill.
- **501**: You hit the read-only Sippy instance with a write; make sure the sippy-auth base URL is used (the script already does).
- **400**: Server-side validation failure — the server's message is shown in the `detail` field of the output.
- **Concurrent edits**: The update flow is read-merge-replace with no server-side concurrency control, so near-simultaneous edits can overwrite each other — re-check the symptom after updating if others may be editing.

**Exit Codes**:
- `0`: Success
- `1`: Validation error, API error, or network error

## See Also

- Related Skill: `oc-auth` (provides authentication tokens for sippy-auth)
- Related Skill: `list-symptoms` (search/inspect symptoms and labels, no auth needed)
- Related Skill: `manage-labels` (create labels before symptoms reference them)
- Related Skill: `reevaluate-job-runs` (**required after create/update** to apply the label to already-completed runs; also previews with `--dry-run`)
- Related Skill: `fetch-regression-details` (source of `prowjob_run_id`s when applying a symptom across a triage or regression)
- Related Skill: `diagnose-job-run-symptoms` (explain which symptoms matched a run)