GitHub PR Reviewer

GitHub PR Reviewer Automation

Agent Canvas catalog

For new Agent Canvas installations, use the GitHub code review catalog entry. Its deterministic worker.py delegates each requested exact head to a stable conversation using the selected agent profile. It supports GitHub reviewer-request events and scheduled label scans. The manual upload flow below remains for existing deployments and is deprecated for new installations.

Create a cron automation that watches one or more GitHub repositories for pull requests with a review trigger label, starts an OpenHands review conversation once per label event, and publishes the AI review to GitHub. Windows PowerShell equivalents for the setup, packaging, upload, and API-check shell snippets are in references/windows.md.

The automation script is deterministic: PR discovery, label-event tracking, head eligibility, state persistence, stale-result suppression, the repository checkout, and its removal are all handled in Python. The LLM is invoked only for the review itself.

Before any review conversation is created, a scheduled scan evaluates the current head's GitHub-required checks, read through the GraphQL isRequired signal for the pull request. That signal is the merge policy's own source of truth and is pull-request-scoped, so it stays correct for a stacked PR whose symbolic base branch carries no branch rules of its own:

• Only checks GitHub reports as required decide the scheduled gate. An optional workflow that fails - including one that fails before creating any check run - does not block a head whose required checks pass.
• A completed required run on the exact head whose conclusion is failure, cancelled, or timed_out blocks the review. Any other unrecognized conclusion fails closed as a block rather than silently approving.
• A completed required run whose conclusion is success, neutral, or skipped does not block. A required commit status context is classified alongside required check runs.
• A queued or in_progress required run on the exact head, and a required context that has not reported a current-head check run at all, make the run exit with a waiting on checks outcome. Unfulfilled is never approval. The worker does not hold a slot polling; the next scheduled scan or explicit review request retries.
• When the required-check signal cannot be read or is empty, the gate falls back to every current-head check run and workflow run, so a red head still blocks. In that fallback, workflow runs are read as well as check runs, because a workflow can fail before creating any check run - a workflow-level error, or a pull_request run whose jobs never start. Such a run leaves a failed check suite with no check runs under it, so the commit's check-run rollup and gh pr checks both report success and only the workflow run reveals the red CI. A workflow run whose check suite already reported check runs is left to those runs, so a workflow is never counted twice.
• A scheduled scan considers the trigger label and every open PR, including drafts, that still holds an outstanding all-hands-bot review request. That is what resumes a request made while CI was running: the request is keyed by its own review_requested event, so repeated scans reuse one conversation and one review instead of creating duplicates, and a PR whose request was answered or withdrawn simply drops out of requested_reviewers.
• The scheduled scan also reviews open, non-draft PRs that nobody requested and that carry no trigger label, once their current head has no completed review by this account. That is what keeps reviewing PRs after the outstanding requests are exhausted. The delivery key is the stable scan:{repository}:{number}:{head}, so repeated scans reuse one conversation and one native review, and a changed head becomes eligible again under its new SHA.
• Each scheduled scan classifies the whole open backlog, including unrequested PRs. Blocked, pending, draft, or already-reviewed heads therefore cannot hide an eligible head behind an inspection window. The separate max_new_per_run quota limits only newly created review conversations, not the number of PRs inspected; explicit requests and trigger labels retain priority when eligible candidates are drained.
• A gate stop on an unrequested head posts no managed comment. A PR nobody asked about that is merely red or pending is simply skipped without starting an agent, which is what keeps a scan over a large backlog from posting a gate comment on every PR. The managed comment is the answer to an explicit request, so a requested or labeled head that is red or pending still gets its explanation.
• Workflow runs are read as well as check runs, because a workflow can fail before creating any check run - a workflow-level error, or a pull_request run whose jobs never start. Such a run leaves a failed check suite with no check runs under it, so the commit's check-run rollup and gh pr checks both report success and only the workflow run reveals the red CI. A workflow run whose check suite already reported check runs is left to those runs, so a workflow is never counted twice.
• Runs attributed to any other (obsolete) head SHA are ignored, so a stale failure cannot block the push that fixed it. This applies to workflow runs too.
• Only the latest run of each logical check or workflow counts. A logical check is its name plus the reporting app identity, and a logical workflow is its name plus workflow ID. The latest is chosen by the run ID (the reliable creation sequence) with the start time as a tie-break, so a re-run that fixed a check supersedes its earlier failure on the same SHA while a newer queued or in-progress re-run supersedes an earlier success and makes the head wait. Ordering by the run ID keeps a run whose start time is still absent in its true creation position in both directions.

An explicit all-hands-bot review request is the intake-policy exception: the caller asked for that head by name, so the request dispatches even when required CI is red or pending, and the gate leaves no explanatory comment. Draft, scope, exact-head, and delivery-deduplication safeguards still apply. A blocked or waiting scheduled stop consumes no trigger: once the head's required checks are non-blocking, the scheduled scan, or a new all-hands-bot review request, starts the normal review.

The gate needs no configured list of check names. When it stops a review it leaves one concise explanation on the PR, identified by a hidden marker carrying the head SHA and gate category, so a later run for a different head updates that comment instead of posting another. Only a marker this reviewer account authored counts as its own comment; every other PR comment is untrusted and can neither suppress the explanation nor be edited. If the repository's own workflow already posted a deterministic remediation comment that names the same current-head checks, the gate adds nothing; a disclosure about some other check does not suppress it. The gate applies to scheduled label scans; an explicit all-hands-bot review request bypasses it by design.

Each explicit review request - a re-applied trigger label, or a new reviewer request in event mode - is a fresh review. The conversation for a PR is reused, but its earlier turns must not be trusted as current: before deciding a verdict the reviewer re-fetches the mutable GitHub state (the exact head, the PR body, review comments and threads, review requests, the linked issues' bodies and labels, and the current-head Actions results) and ignores any earlier finding, verdict, or label/priority claim that state no longer supports. Repository analysis the conversation already did, such as reading AGENTS.md, stays useful and is not repeated.

The waiting and blocked explanations name the retry the deployment actually has. A scheduled run proves a scan is configured, so it says the next scan will retry. An event-only run does not, so it tells the reader to remove the outstanding all-hands-bot request and request all-hands-bot again instead of promising a scan that does not exist: GitHub will not accept a second request while the first is still outstanding.

The gate leaves only one explanation per head. When the same head keeps the same hidden marker but the retry wording changes -- an event-only comment on a head whose automation is later switched to a cron scan -- the scheduled run rewrites that managed comment in place, so the comment always names the retry that is actually deployed, and an unchanged body is left untouched.

Bounded intake per scheduled scan

A scheduled scan drains outstanding reviewer requests fairly, but starting an agent for every eligible pull request at once would overload the deployment, so the scan starts at most MAX_NEW_PER_RUN new review conversations, counted across every configured repository rather than per repository. The default is 2, and the rendered config.json overrides it through the max_new_per_run key that github-issue-to-pr and gitlab-issue-to-mr already use.

• Eligible candidates are ordered deterministically: an explicit all-hands-bot request (or a trigger label) is ordered before an unrequested PR, then by the oldest all-hands-bot review_requested event for a requested PR or the PR's own creation time (oldest first) for an unrequested one, then by repository, then by pull-request number, so the oldest work drains first and a later scan reaches the remainder.
• The bound counts the conversations a scan starts. A delivery that only deduplicates or reports an already-running conversation reuses a runtime and consumes no slot, so repeated scans make progress on the backlog instead of re-spending the bound on work already in flight.
• Reaching the bound never cuts the scan short. The scan still evaluates the exact-head checks of the remaining candidates, reconciles completed reviews, and runs the maintainer handoff. A candidate whose checks are pending or failing starts no conversation and consumes no slot, so it cannot block a later green candidate. Only an explicit request or a trigger label gets a waiting or blocked gate comment; an unrequested head that is merely red or pending is skipped silently.
• A dispatch that raises is reported and does not consume a slot or abort the scan, so the candidates behind it are still considered.
• The explicit review_requested event path and the trigger-label scan are unchanged: an explicit request still starts its conversation immediately, and only the scheduled scan's new conversations are bounded.

The review prompt starts with a scope gate: using the repository's own guidance (its scope categories and ownership boundaries, not a list of individual PR numbers), the reviewer decides whether the change belongs in this repository and has the product/architecture direction it needs. When it does not, the review stops with a single event: COMMENT review that says whether the change should move repositories, close, or receive a maintainer decision, and ends with the 🛑 MAINTAINER DECISION REQUIRED verdict. That outcome is neither an approval nor a change request: it does not approve or merge the PR. The completion handler recognizes the verdict and requests one configured maintainer through the same handoff used after an approval. Both paths require a successfully fetched same-repository closing issue labeled priority:medium or priority:high; a missing link or priority label, low/normal priority, unresolved reference, or PR reference does not authorize handoff. One qualifying issue is sufficient even with other low-priority or missing references. Existing manual review requests are unchanged. An in-scope change continues the existing review unchanged.

Before approving a change to user-visible UI behavior, the prompt requires live evidence from a real running application when the repository's guidance demands it: a screenshot, screen recording, or equivalent capture of the running app that exercises the production-facing path. Unit tests, CSS-token or contract assertions, generated mockups, and reconstructed captures may support the review but cannot substitute for that evidence. When the required evidence is missing, the reviewer publishes one event: COMMENT review naming exactly what is missing and ends with 🔄 CHANGES REQUESTED, so approval is withheld and the deterministic maintainer handoff does not fire. Non-UI changes keep the existing requirement of the real command and its observed output, with tests insufficient as sole proof.

The script prepares each review's workspace before the agent starts: the pull request's head commit is downloaded as a tarball and extracted to a directory of its own, which becomes the conversation's working directory. The agent is told not to clone, fetch, check out, or delete anything, and the script removes the checkout once the conversation has stopped. Nothing accumulates between runs.

The script imports shared GitHub transport from scripts/github_client.py, installed with this skill. Include it beside main.py when packaging manually, as shown below; catalog bundles include it automatically.

Prerequisites

Required secret

Verify that the following secret is set in OpenHands Settings -> Secrets:

Pull-request write access is required because the agent publishes a pull request review, not just an issue comment. The Agent Canvas catalog worker may also request a configured human reviewer after approval. A token with only Pull requests: Read will poll happily and then fail at the point of publishing or requesting the handoff.

When several repositories are monitored, the token must cover all of them.

Check with:

curl -s https://api.github.com/user \
  -H "Authorization: Bearer $GITHUB_PERSONAL_ACCESS_TOKEN" \
  | python3 -c "import json,sys; d=json.load(sys.stdin); print(d.get('login') or d.get('message'))"

If the token is missing or invalid, inform the user and stop.

Setup Workflow

Follow these steps in order.