Fetch Test Report

Fetch Test Report

This skill fetches a report for an OpenShift CI test by its full name using the Sippy tests API. It returns test metadata including the BigQuery/Component Readiness test ID, Jira component, pass rates for the current and previous reporting periods, and open bug counts. The open_bugs field counts Jira bugs that mention this test by name, which can help surface bugs that have been filed but not yet triaged in Component Readiness.

When to Use This Skill

Use this skill when you need to:

• Look up a test's Component Readiness / BigQuery test ID (test_id)
• Check current and previous pass rates for a test
• Find the Jira component associated with a test
• Determine if a test is flaking, failing, or stable
• Get run counts and failure/flake breakdowns for a test
• Check if there are open Jira bugs mentioning this test (may not yet be triaged in Component Readiness)
• See a per-variant breakdown of pass rates to identify if the test fails only in certain job types

Prerequisites

1. Network Access: The Sippy API must be accessible at https://sippy.dptools.openshift.org

   • No authentication required
   • Check: curl -s https://sippy.dptools.openshift.org/api/health?release=4.22

2. Python 3: Python 3.6 or later

   • Check: python3 --version
   • Uses only standard library (no external dependencies)

Implementation Steps

Step 1: Determine the Release

If the user did not specify a release, use the fetch-releases skill to determine the latest OCP release:

release=$(python3 plugins/ci/skills/fetch-releases/fetch_releases.py --latest)

If the user specified a release, use that directly.

Step 2: Run the Python Script

The skill uses a Python script to query the Sippy tests API:

# Path to the Python script
script_path="plugins/ci/skills/fetch-test-report/fetch_test_report.py"

# Fetch test report in JSON format (collapsed — one row for the test across all variants)
python3 "$script_path" "<test_name>" --release "$release" --format json

# Or fetch as human-readable summary
python3 "$script_path" "<test_name>" --release "$release" --format summary

# Fetch per-variant breakdown (one row per variant combo — useful for identifying
# if the test is failing only in certain job types, e.g., aws vs metal, upgrade vs new-install)
python3 "$script_path" "<test_name>" --release "$release" --no-collapse --format json

Step 3: Parse the Output

The script outputs a JSON array (typically with one element for an exact name match):

# Store JSON output in a variable for processing
test_data=$(python3 "$script_path" "[sig-api-machinery] Discovery should validate PreferredVersion for each APIGroup [Conformance]" --format json)

# Extract specific fields using jq
test_id=$(echo "$test_data" | jq -r '.[0].test_id')
component=$(echo "$test_data" | jq -r '.[0].jira_component')
pass_rate=$(echo "$test_data" | jq -r '.[0].current_pass_percentage')
runs=$(echo "$test_data" | jq -r '.[0].current_runs')

API Details

Endpoint

GET https://sippy.dptools.openshift.org/api/tests/v2?release={release}&filter={filter_json}
GET https://sippy.dptools.openshift.org/api/tests/v2?release={release}&filter={filter_json}&collapse=false

When collapse=false is specified, the API returns one row per variant combination instead of a single collapsed row. Each row includes a variants array showing the specific variant combo (e.g., ["aws", "ovn", "amd64", "upgrade-micro"]). This helps identify if the test is failing in certain types of jobs and not others.

Filter Format

The filter parameter is a JSON object using Sippy's standard filter syntax:

{
  "items": [
    {
      "columnField": "name",
      "operatorValue": "equals",
      "value": "<test_name>"
    }
  ]
}

Response Schema

The API returns a JSON array of test objects:

[
  {
    "id": 5987,
    "test_id": "openshift-tests:e8f7491a505095b6356dfd8d4cf7218b",
    "name": "[sig-api-machinery] Discovery should validate PreferredVersion for each APIGroup [Conformance]",
    "suite_name": "openshift-tests",
    "variants": null,
    "jira_component": "kube-apiserver",
    "jira_component_id": "12367637",
    "current_successes": 3425,
    "current_failures": 22,
    "current_flakes": 0,
    "current_pass_percentage": 99.36,
    "current_failure_percentage": 0.64,
    "current_flake_percentage": 0,
    "current_working_percentage": 99.36,
    "current_runs": 3447,
    "previous_successes": 2570,
    "previous_failures": 43,
    "previous_flakes": 0,
    "previous_pass_percentage": 98.35,
    "previous_failure_percentage": 1.65,
    "previous_flake_percentage": 0,
    "previous_working_percentage": 98.35,
    "previous_runs": 2613,
    "net_failure_improvement": 1.01,
    "net_flake_improvement": 0,
    "net_working_improvement": 1.01,
    "net_improvement": 1.01,
    "tags": null,
    "open_bugs": 0
  }
]

Key Fields:

• id: Sippy PostgreSQL internal ID (less commonly used)
• test_id: BigQuery / Component Readiness test ID (format: suite:hash). This is the ID used with modern skills like fetch-regression-details and fetch-test-runs.
• name: Full test name
• suite_name: Test suite (e.g., openshift-tests)
• variants: Array of variant strings (only present when --no-collapse / collapse=false is used). Shows the specific variant combo for this row (e.g., ["aws", "ovn", "amd64", "upgrade-micro"]).
• jira_component: Associated OCPBUGS Jira component
• jira_component_id: Jira component numeric ID
• current_*: Metrics for the current 7-day reporting period
• previous_*: Metrics for the 7 days before the current period
• net_working_improvement: Change in working percentage (positive = improving)
• open_bugs: Number of open Jira bugs that mention this test by name. This can surface bugs that have been filed but not yet triaged in Component Readiness. Useful for finding existing work before filing a duplicate.