Reevaluate Job Runs

Reevaluate Job Runs

Sippy Symptoms recognize known conditions in OpenShift CI artifacts. A symptom selects artifact files with a glob and uses a string, regex, or file (existence) matcher. A match applies one or more Labels: stable, human-readable tags such as ImagePullNeverCompletes that appear in Sippy, Spyglass, and triage workflows.

Symptom detection normally runs automatically as new job artifacts arrive. Reevaluation asks Sippy to scan completed runs with the current symptom definitions, which is useful when a symptom was created or changed after those runs completed.

When to Use This Skill

Use this skill to:

• Preview which symptoms and labels match existing runs without writing.
• Apply a new or updated symptom to older completed runs.
• Re-scan every job run behind a Component Readiness regression or triage.

Prerequisites

Authentication to the sippy-auth API requires a Bearer token from the DPCR OpenShift cluster (https://api.cr.j7t7.p1.openshiftapps.com:6443). Use the oc-auth skill to select the correct context. Export the token rather than putting it in the process list:

DPCR_CONTEXT="dpcr-context-name"
SIPPY_TOKEN="$(oc whoami -t --context="$DPCR_CONTEXT")"
export SIPPY_TOKEN
test -n "$SIPPY_TOKEN"

The implementation uses Python 3 and the standard library only. The examples that inspect API JSON also use jq and curl.

Preview and Apply

--dry-run uses the same asynchronous API and artifact scan as an applied reevaluation, but reports the labels without changing BigQuery, GCS, or PostgreSQL:

JOB="periodic-ci-example-job"
BUILD_ID="1856789012345678848"
PROW_URL="https://prow.ci.openshift.org/view/gs/test-platform-results-public/logs/$JOB/$BUILD_ID"

python3 plugins/ci/skills/reevaluate-job-runs/reevaluate_job_runs.py \
  "$PROW_URL" --dry-run --format summary

After reviewing the preview, omit --dry-run to apply the current symptom set:

python3 plugins/ci/skills/reevaluate-job-runs/reevaluate_job_runs.py \
  1856789012345678848 1856789012345678849 --format summary

The script deduplicates normalized IDs, submits one asynchronous batch, and polls until it is complete, failed, or cancelled. Sippy accepts at most 10,000 unique numeric build IDs in a request. Full Prow URLs are also accepted; query strings, fragments, trailing slashes, and duplicate inputs are normalized.

Save the batch ID for recovery

Immediately after Sippy accepts and validates a submission, the script prints and flushes the batch ID and same-origin status URL before the first status GET. Summary mode writes this notice to stdout. JSON mode writes it to stderr so stdout remains one parseable document: either the unchanged terminal API response or, if polling fails, an error object containing batch_id, status_url, and error. Save the notice: the server-side batch can keep running if the client loses its connection, exits, or its token expires.

To inspect that same batch later, obtain a fresh token with oc-auth, copy the printed sippy-auth status URL, and make an authenticated GET:

STATUS_URL="https://sippy-auth.dptools.openshift.org/api/jobs/runs/reevaluate/d15dff1f-431c-48db-aa37-628ab42d755e"

curl --fail-with-body --silent --show-error \
  -H "Accept: application/json" \
  -H "Authorization: Bearer $SIPPY_TOKEN" \
  "$STATUS_URL" | jq .

Do not print or echo SIPPY_TOKEN. Only send it to the same https://sippy-auth.dptools.openshift.org origin. The script validates this origin and refuses redirects to a different origin.

The script does not have a resume option and does not accept an existing batch ID. The authenticated GET above observes an existing batch; running the script again creates a new submission.

Triage-wide workflow

Sippy has no triage-level reevaluate endpoint. List the regression IDs attached to the triage, collect every prowjob_run_id with the fetch-regression-details skill, and pass the resulting Bash array to one preview invocation:

REGRESSION_IDS=(34446 34447 34448)
RUN_IDS=()

for REGRESSION_ID in "${REGRESSION_IDS[@]}"; do
  while IFS= read -r RUN_ID; do
    RUN_IDS+=("$RUN_ID")
  done < <(
    python3 plugins/ci/skills/fetch-regression-details/fetch_regression_details.py \
      "$REGRESSION_ID" --format json |
      jq -r '.job_runs[].prowjob_run_id'
  )
done

python3 plugins/ci/skills/reevaluate-job-runs/reevaluate_job_runs.py \
  "${RUN_IDS[@]}" --dry-run --format summary

Review the preview, then repeat the final command without --dry-run. The client deduplicates IDs across regressions and rejects more than 10,000 unique IDs before submission.

Arguments and Options

• runs: One or more Prow build IDs or Prow job URLs (required; maximum 10,000 unique IDs).
• --token TOKEN: Bearer token. Prefer SIPPY_TOKEN because command-line arguments are visible in process listings; --token takes precedence.
• --dry-run: Report matches without writing changes.
• --poll-interval SECONDS: Time between status requests (default 5; must be finite and greater than zero).
• --format json|summary: Output format (default json).