PR Monitor

PR CI Monitor

<!--
SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
SPDX-License-Identifier: CC-BY-4.0
-->

Perform a full health check on a Dynamo pull request. Takes a PR number as argument (e.g., /dynamo:pr-monitor 6554).

Step 1: PR Overview

Gather PR metadata and determine what CI should look like for this PR.

gh pr view $PR_NUMBER --repo ai-dynamo/dynamo --json title,body,author,state,isDraft,additions,deletions,changedFiles,labels,reviewDecision,headRefName,baseRefName

gh pr diff $PR_NUMBER --repo ai-dynamo/dynamo --name-only

Check if full CI should be running. The dynamo repo has two tiers of CI:

• Lightweight (pre-merge.yml): Triggers on all pull_request events. Runs pre-commit, copyright checks, DCO, and optionally rust-clippy/rust-tests (if rust filter matches). This always runs.
• Full pipeline (pr.yaml, container-validation-dynamo.yml): Triggers on push to main or pull-request/[0-9]+ branches ONLY. This includes docker builds, GPU tests, deploy tests. It does NOT trigger on regular PR branches.

To get full CI on a PR, a pull-request/$PR_NUMBER branch must exist (created by copy-pr-bot after an NVIDIA maintainer approves). Check:

gh api repos/ai-dynamo/dynamo/branches/pull-request/$PR_NUMBER 2>/dev/null

If the branch does not exist, full CI has not been triggered. Common reasons:

1. Awaiting approval: An NVIDIA maintainer needs to comment /ok to test <commit_sha> on the PR to create the branch and trigger full CI. This applies to both fork PRs and internal PRs from authors not yet in the approval list.
2. DCO failure: Commits are not signed (Signed-off-by line missing). Check for a DCO bot comment. Fix: author signs their commits (see DCO.md).
3. Unverified commit signature (eligible fork automatic approval): The automatic /ok to test flow requires GitHub to report every PR commit as Verified. A DCO sign-off alone does not meet this requirement. This applies only after the PR has an automatic approval path; signing commits does not create that path. Check the workflow's unsigned-commit comment or inspect .commit.verification for every PR commit; sign each unverified commit, then push again. A maintainer can manually review the current head and comment /ok to test <sha> when automatic approval is unavailable.
4. Draft PR: Some checks may not run until marked ready for review.

For a fork PR that is otherwise eligible for automatic approval, check every commit's verification result:

gh api --paginate "repos/ai-dynamo/dynamo/pulls/$PR_NUMBER/commits?per_page=100" --jq '.[].sha' | while read -r sha; do
  gh api "repos/ai-dynamo/dynamo/commits/$sha" --jq '"\(.sha[0:7]) verified=\(.commit.verification.verified) reason=\(.commit.verification.reason)"'
done

Verify which workflows actually ran against the PR's HEAD commit:

# Get the HEAD SHA
HEAD_SHA=$(gh pr view $PR_NUMBER --repo ai-dynamo/dynamo --json headRefOid --jq '.headRefOid')

# Check which workflow runs exist for this SHA
gh api "repos/ai-dynamo/dynamo/actions/runs?head_sha=$HEAD_SHA" --jq '.workflow_runs[] | {name: .name, status: .status, conclusion: .conclusion}'

Compare the workflows that ran against what's expected. If only Pre Merge, Copyright Checks, DCO Commenter, etc. appear but NOT PR or Dynamo Validation, then full CI was not triggered.

Determine which CI filters are active. If full CI ran, read the changed-files job output from the actual workflow run — this is the authoritative source for which filters are true:

# Find the PR workflow run ID
PR_RUN_ID=$(gh api "repos/ai-dynamo/dynamo/actions/runs?head_sha=$HEAD_SHA" --jq '.workflow_runs[] | select(.name == "PR") | .id')

# Get the changed-files job output (look for the filter results in the logs)
gh api "repos/ai-dynamo/dynamo/actions/runs/$PR_RUN_ID/jobs" --jq '.jobs[] | select(.name == "changed-files") | {id: .id, status: .status, conclusion: .conclusion}'

If full CI hasn't run, fall back to fetching filters.yaml and matching manually:

gh api repos/ai-dynamo/dynamo/contents/.github/filters.yaml --jq '.content' | base64 -d

Key rules for manual matching:

• core being true triggers ALL framework pipelines (vllm, sglang, trtllm).
• Filters use YAML anchors (e.g., *ci) — resolve these when reading.
• Negation patterns like !**/*.md in a filter mean markdown-only changes do NOT trigger that filter.

Jobs that always run in pr.yaml regardless of filters:

• changed-files, deploy-operator, backend-status-check, clean-k8s-builder, cleanup — these run unconditionally or with if: always().

Step 2: CI Status Dashboard

Fetch all check runs. Note: gh pr checks returns non-zero exit codes when checks are pending or failed — this is normal, not an error.

gh pr checks $PR_NUMBER --repo ai-dynamo/dynamo

If no checks are returned at all, refer back to the Step 1 diagnosis (DCO, approval, draft status).

Ignore external CI checks (e.g., GitLab mirror ci/gitlab/*). These are NVIDIA-internal pipelines that cannot be inspected from GitHub. Only analyze GitHub Actions checks.

Distinguish two situations:

1. Full CI triggered (workflow runs include PR, Dynamo Validation): Analyze all jobs normally.
2. Only lightweight CI ran (Pre Merge and utility workflows only): Report this clearly. The filter predictions from Step 1 describe what would run once full CI triggers, but there's nothing to analyze yet beyond the lightweight checks.

If full CI ran, identify the critical path — which checks are most relevant to this PR's changes based on the filter mapping from Step 1.

Produce a concise dashboard grouped by status:

• Failed — needs immediate attention
• Pending/In-progress — still running (note which are critical path)
• Passed — healthy (just count, don't enumerate unless asked)
• Skipped — handled in Step 4

Step 3: Failure Analysis

For each failed GitHub Actions job, drill into the logs to extract root cause.

First, identify the failed jobs within each run:

gh api repos/ai-dynamo/dynamo/actions/runs/$RUN_ID/jobs --jq '.jobs[] | select(.conclusion == "failure") | {name: .name, id: .id, html_url: .html_url}'

Then fetch logs for specific failed jobs. The check URL format from gh pr checks is https://github.com/.../actions/runs/{RUN_ID}/job/{JOB_ID}. Extract the RUN_ID (first number):

gh run view $RUN_ID --repo ai-dynamo/dynamo --log-failed 2>&1 | tail -200

Note: --log-failed concatenates all failed job logs, which can be noisy. For multi-failure runs, prefer fetching per-job to isolate root causes.

For each failure, report:

• Job name and which workflow it belongs to
• Root cause — the actual error (compilation error, test assertion, timeout, infra issue, DCO sign-off failure, etc.)
• Relevant log excerpt — the key lines (max 20 lines), not the full dump
• Suggested fix — concrete action the PR author can take
• If it looks like an infra flake, include: gh run rerun $RUN_ID --repo ai-dynamo/dynamo --failed

If there are no failures, say so and move on.

Step 3b: Main Branch Cross-Reference

For each failed job identified in Step 3, check if the same job also fails on main. This distinguishes PR-caused regressions from pre-existing flakes.

First, check how far behind the PR is from main:

gh api "repos/ai-dynamo/dynamo/compare/main...$HEAD_SHA" --jq '{behind_by: .behind_by, ahead_by: .ahead_by, status: .status}'

If the PR is significantly behind main (>20 commits), note this in the report — some failures may already be fixed on main, and some apparent "regressions" may just be the PR missing recent fixes.

Then, check recent main branch CI results:

# Get last 3 completed PR workflow runs on main
MAIN_RUNS=$(gh api "repos/ai-dynamo/dynamo/actions/workflows/pr.yaml/runs?branch=main&per_page=3&status=completed" --jq '[.workflow_runs[].id] | join(" ")')

# For each failed job name from Step 3, check if it also failed on main
for RUN_ID in $MAIN_RUNS; do
  gh api "repos/ai-dynamo/dynamo/actions/runs/$RUN_ID/jobs?per_page=100&filter=latest" \
    --jq ".jobs[] | select(.name == \"$FAILED_JOB_NAME\") | {run: $RUN_ID, conclusion: .conclusion}"
done

Classification:

• [PR-CAUSED] — Fails on this PR but passes on all recent main runs → Likely a regression introduced by this PR. Needs attention.
• [PRE-EXISTING] — Also fails on recent main runs → Pre-existing flake, not caused by this PR. Suggest rerun.
• [UNCLEAR] — Mixed results on main (sometimes passes, sometimes fails) → Flaky test. Note flakiness and suggest rerun.
• [NEW JOB] — Job doesn't exist on main runs → New CI job added by this PR, can't compare.

Important caveats to include in the report:

• If the PR is behind main by many commits, a [PR-CAUSED] classification may be a false positive — the failure could already be fixed on main. Suggest: "Consider rebasing on main before investigating."
• If the PR is ahead of main (includes main's latest), the classification is reliable.
• Only compare job names, not test-level results. A job passing on main doesn't guarantee the same tests pass — just that the overall job succeeds.