ONNX Runtime CI Management

ONNX Runtime CI Management

Workflows for triggering, re-running, and unblocking CI checks on an ONNX Runtime PR. The repository is microsoft/onnxruntime. As of 2026-07, nearly all CI runs as GitHub Actions workflows (~80+ checks per PR). Only one required check still runs on Azure Pipelines — Linux Android Emulator QNN CI Pipeline (host aiinfra.visualstudio.com). There is also the bot-driven license/cla status check.

Failures are not all the same. Before touching anything, diagnose each failure (see Triage: Diagnose Before Re-running): most failures need a code change and re-running them just fails again; only genuinely transient (network/disk) failures should be re-run via §1. §4 (Azure Pipelines) applies only to the single QNN pipeline.

Before doing anything, inspect current state so you do not queue duplicate runs.

If you make any change, commit and push it, then stop. A push updates the PR head SHA and automatically starts CI for the new commit. Do not manually re-run failures from the old SHA after pushing a fix; that only queues redundant runs against stale code. Manual re-runs are only for transient failures when the PR head has not changed.

Classify a check's provider

GitHub Actions checks have a non-empty workflowName and a detailsUrl on github.com; the Azure Pipelines check has an empty workflowName and a detailsUrl on aiinfra.visualstudio.com:

gh pr view <number> --repo microsoft/onnxruntime --json statusCheckRollup \
  --jq '[.statusCheckRollup[] | {name, host:(.detailsUrl|split("/")[2])}]
        | group_by(.host) | map({host:.[0].host, count:length})'

Gather Context

# PR metadata + all checks grouped by state
gh pr view <number> --repo microsoft/onnxruntime \
  --json number,title,url,state,isDraft,headRefName,headRefOid,baseRefName,statusCheckRollup

# Head / merge SHAs for external CI
gh api repos/microsoft/onnxruntime/pulls/<number> --jq '{head:.head.sha, merge:.merge_commit_sha}'

Inspect statusCheckRollup and note, for each requested check, whether it is missing, queued, in progress, failed, canceled, skipped, or already successful. Do not re-trigger a check that is already queued/in_progress/SUCCESS unless the user explicitly asks.

Quickly list just the failed/pending checks:

gh pr view <number> --repo microsoft/onnxruntime --json statusCheckRollup \
  --jq '.statusCheckRollup[]
        | {name:(.name//.context), status:(.status//.state), conclusion:.conclusion}
        | select(.conclusion!="SUCCESS" and .status!="SUCCESS")'

Triage: Diagnose Before Re-running

Never blindly re-run failed CI. Most failures need a code change and will fail again identically on re-run. Only transient failures should be re-run. The process is: download the failed job's log, read the actual error, classify it, then fix or re-run case by case.

Step 1 — Download the failed log

For a GitHub Actions check (the vast majority), get the run and read only the failed steps:

HEAD_SHA=$(gh api repos/microsoft/onnxruntime/pulls/<number> --jq .head.sha)

# Map failed checks to their workflow run IDs
gh run list --repo microsoft/onnxruntime --commit "$HEAD_SHA" --limit 100 \
  --json databaseId,workflowName,status,conclusion,url \
  --jq '.[] | select(.conclusion=="failure" or .conclusion=="cancelled")'  # cspell:ignore cancelled -- literal GitHub API value

# Dump just the failed steps of a run (grep for the real error)
gh run view <run_id> --repo microsoft/onnxruntime --log-failed > /tmp/ci_<run_id>.log
grep -nE "error:|FAILED|warning:|Traceback|fatal error|No space left|Could not resolve|timed out" \
  /tmp/ci_<run_id>.log | head -50

For the Azure Pipelines QNN check, open its detailsUrl (a dev.azure.com / aiinfra.visualstudio.com build page) and download the job log, or use the Azure DevOps .../builds/<buildId>/timeline + log APIs (see the ci-failure-retrieval skill for the exact requests).

Step 2 — Classify the failure and act

Rules of thumb:

• A compile error: or a [ FAILED ]/FAILED line means fix the code — re-running reruns the same failing commit and fails identically.
• Only re-run when the log shows a network/disk/agent problem and no compile or assertion error.
• When unsure, download the log and read it; do not guess from the check name alone.
• After any code, test, lint, or generated-file fix, commit and push it. CI starts automatically for the new head SHA. Do not use §1 after pushing a change; the failed runs belong to the old SHA and re-running them would test stale code.

1. Re-run Failed GitHub Actions (transient failures only)

The repo ships a helper that re-runs only the GitHub Actions workflows whose latest run for the PR's current head commit failed/canceled — and skips any workflow that already has a newer run queued or in progress. Use it only after triage confirms the failures are transient (network/disk/agent) — see Triage. It is the safest way to retry those without piling on duplicates. Do not use this helper if you changed anything and pushed a new commit; the push already starts CI for the new head SHA.

Script: tools/scripts/rerun_failed_ci.sh

# Dry run first — shows what would be re-run, triggers nothing
./tools/scripts/rerun_failed_ci.sh <number> --dry-run

# Actually re-run the failed/canceled workflows for the PR's head commit
./tools/scripts/rerun_failed_ci.sh <number>

# Explicit repo (auto-detected from cwd when omitted)
./tools/scripts/rerun_failed_ci.sh <number> microsoft/onnxruntime

It prefers gh run rerun <id> --failed (retry only failed jobs) and falls back to a full rerun for fully canceled runs that have no discrete failed jobs. Requires an authenticated gh. Always run --dry-run first and confirm the list looks right before the real run.

To re-run one specific workflow manually:

gh run list --repo microsoft/onnxruntime --commit <head_sha> --limit 100 \
  --json databaseId,workflowName,status,conclusion,url
gh run rerun <run_id> --repo microsoft/onnxruntime --failed   # only failed jobs
gh run rerun <run_id> --repo microsoft/onnxruntime            # full rerun

2. Unblock license/cla (CLA bot)

The license/cla check is posted by Microsoft's CLA bot, independent of the CI pipelines. When it is stuck as "Expected — Waiting for status to be reported", re-trigger only the bot — no CI jobs are re-run — by posting this comment on the PR:

gh pr comment <number> --repo microsoft/onnxruntime \
  --body "@microsoft-github-policy-service rerun"

Then verify it flips to success:

gh pr view <number> --repo microsoft/onnxruntime --json statusCheckRollup \
  --jq '.statusCheckRollup[] | select((.name//.context)=="license/cla")
        | {status, conclusion}'

Expect {"status":"COMPLETED","conclusion":"SUCCESS"}.