Agent skill

Classify Review Comment

by openshift-eng in openshift-eng/ai-helpers

Classify GitHub PR review comments by severity and topic. An agent skill from openshift-eng/ai-helpers.

Apache-2.0Auto-check passedDevelopment

Install Classify Review Comment

skills CLI
$ npx skills add openshift-eng/ai-helpers --skill classify-review-comment -a claude-code

Project install by default; add -g for ~/.claude/skills/.

GitHub CLI
$ gh skill install openshift-eng/ai-helpers classify-review-comment --agent claude-code

Project scope by default; add --scope user for a personal install. Needs GitHub CLI 2.90.0 or later (public preview).

Manual copy
$ git clone --depth 1 https://github.com/openshift-eng/ai-helpers.git skills-src && mkdir -p .claude/skills && cp -r skills-src/plugins/code-review/skills/classify-review-comment .claude/skills/classify-review-comment && rm -rf skills-src

Use ~/.claude/skills/ instead of .claude/skills for a personal install. The folder must contain SKILL.md.

Claude Code skills documentation · loads skills from .claude/skills/

Facts

Skill name
classify-review-comment
GitHub stars
120
Token cost
~2.8k tokens
SKILL.md length
838 words
Files
2
Skills in repo
118
Repo updated
First seen
Licence
Apache-2.0

At a glance

Classify GitHub PR review comments by severity and topic. An agent skill from openshift-eng/ai-helpers.

  • Works in 3 steps: Single Comment (text) → Comment URL → Full PR
  • The user wants to categorize
  • SKILL.md covers Labels, Input Modes, Classification Approach and Confidence Scoring Rubric, plus 2 more sections
  • Calls gh; reaches github.com

What it does

Classify Review Comment is an agent skill from openshift-eng/ai-helpers. Classify GitHub PR review comments by severity and topic. Use when the user wants to categorize, analyze, or understand patterns in code review feedback — whether for a single comment, a comment URL, or an entire pull request. Triggers on requests like 'classify this comment', 'categorize PR feedback', 'what kind of review comments does this PR have', or 'break down comments by severity'.

Its SKILL.md is about 2.8k tokens, which your agent loads only when the skill is triggered. The skill folder holds 1 other file (for example `config.json`).

It sits in Development, covering Pull requests and Code review. It works with GitHub. The repository describes itself as: Developer productivity tools for Claude Code & other AI assistants. The licence is Apache-2.0.

When your agent uses it

  • The user wants to categorize
  • Understand patterns in code review feedback — whether for a single comment
  • An entire pull request
  • Requests like classify this comment

Example prompts

  • “classify this comment”
  • “categorize PR feedback”
  • “what kind of review comments does this PR have”
  • “/classify-review-comment”

Workflow steps

3 steps, taken from the step headings in SKILL.md.

  1. Single Comment (text)
  2. Comment URL
  3. Full PR

What it can do on your machine

Read from SKILL.md and the folder at commit a627176. It shows what the files ask for, not the result of running them.

  • Tool permissions

    Pre-approves nothing: there is no allowed-tools line, so your agent's usual permission prompts apply.

    From allowed-tools in the SKILL.md frontmatter.

  • Runs code

    Shell commands in SKILL.md call:

    • gh

    From the folder's file list and the shell code blocks in SKILL.md.

  • Network

    Hosts in commands or code, which the agent is likely to contact:

    • github.com

    From URLs in SKILL.md, links to its own repository left out.

  • Credentials

    Names no API keys, tokens, secrets or passwords.

    From names ending in _API_KEY, _TOKEN, _SECRET, _KEY or _PASSWORD in SKILL.md.

Context cost

Classify Review Comment loads about 2.8k tokens when it runs. Until then it costs about 104 tokens; SKILL.md has 838 words of instructions outside code blocks.

Always · name and description, kept in context so the agent knows when to use it
~104
When it runs · the whole SKILL.md, loaded when a task matches
~2.8k

Estimates: characters ÷ 4, the usual rule of thumb; real counts depend on the model's tokenizer. Scripts and assets cost tokens only if the agent reads them.

Safety

Auto-check passed

The automated check found no risky patterns in SKILL.md.

Automated static check — not a guarantee. Review scripts before installing. It scans the text of SKILL.md for risky patterns (piping downloads into a shell, reading credential files, hidden Unicode, destructive commands); files beside SKILL.md are not scanned.

SKILL.md

The full file from openshift-eng/ai-helpers at commit a627176, republished under its Apache-2.0 licence (© openshift-eng). 838 words, ~2,796 tokens.

Download SKILL.mdSave it as .claude/skills/classify-review-comment/SKILL.md (or your agent's skills folder). This skill also uses 1 other file; get the full folder from GitHub.
name
classify-review-comment
description
Classify GitHub PR review comments by severity and topic. Use when the user wants to categorize, analyze, or understand patterns in code review feedback — whether for a single comment, a comment URL, or an entire pull request. Triggers on requests like 'classify this comment', 'categorize PR feedback', 'what kind of review comments does this PR have', or 'break down comments by severity'.

Classify Review Comments

Classify GitHub pull request review comments into severity and topic categories. Works with a single comment (text), a GitHub comment URL, or an entire PR (classifies all comments).

This enables tracking review feedback patterns: what kinds of issues reviewers catch, how severe they are, and where AI-generated code needs the most improvement.

Labels

Read the labels file before classifying any comments:

text
config.json (in the same directory as this skill)

The labels file defines the exact set of valid values for severity and topic. You MUST select from these values — do not invent new labels. Each label includes a description and signal words or examples to guide your selection.

Classification rule: For each comment, find the single best-matching severity and single best-matching topic from the labels file. Match based on the label's description, signals/examples, and the comment content. Use unclassified only when no other label fits.

Input Modes

1. Single Comment (text)

Classify a comment provided directly as text.

Input: The raw comment body. Output: A single classification object.

2. Comment URL

Fetch a specific comment by its GitHub URL and classify it.

URL formats supported:

  • https://github.com/{owner}/{repo}/pull/{number}#issuecomment-{id}
  • https://github.com/{owner}/{repo}/pull/{number}#discussion_r{id}
  • https://github.com/{owner}/{repo}/pull/{number}#pullrequestreview-{id}

Fetch with:

bash
# Issue comment
gh api repos/{owner}/{repo}/issues/comments/{id} --jq '{author: .user.login, body: .body}'

# Review comment (discussion)
gh api repos/{owner}/{repo}/pulls/comments/{id} --jq '{author: .user.login, body: .body}'

# Review body comment
gh api repos/{owner}/{repo}/pulls/{number}/reviews/{id} --jq '{author: .user.login, body: .body}'
3. Full PR

Fetch all comments on a PR, filter out noise, and classify each one.

URL format: https://github.com/{owner}/{repo}/pull/{number}

Fetch with:

bash
# Issue-level conversation comments
gh api repos/{owner}/{repo}/issues/{number}/comments --paginate --jq '.[] | {id: .id, author: .user.login, body: .body}'

# Inline review comments
gh api repos/{owner}/{repo}/pulls/{number}/comments --paginate --jq '.[] | {id: .id, author: .user.login, body: .body}'

# Review body comments (approvals, rejections, general review summaries)
gh api repos/{owner}/{repo}/pulls/{number}/reviews --paginate --jq '.[] | select(.body != null and .body != "") | {id: .id, author: .user.login, body: .body}'

Before classifying, filter out noise comments (these carry no review signal):

  • Pure slash commands: body starts with / followed by a command word (e.g., /lgtm, /test e2e-aws, /approve, /retest, /cc)
  • CI bot notifications: authors like openshift-ci-robot, openshift-ci[bot], cwbotbot, or any *[bot] author not listed in the allowed_bots section of config.json
  • Comments matching any pattern in the noise_patterns section of config.json
  • Auto-CC commands: /auto-cc

Do classify comments from:

  • Human reviewers (all comments, including those directing bots)
  • Any bot listed in allowed_bots in config.json (classify their substantive review comments — code issues, suggestions, questions)

Classification Approach

For each comment:

  1. Read config.json to load the valid severity and topic values
  2. Scan the comment body for signal words and patterns that match label descriptions
  3. Select exactly one severity — match the comment's urgency/tone to the severity descriptions and signals
  4. Select exactly one topic — match the comment's subject matter to the topic descriptions and examples
  5. Score confidence using the rubric below
  6. When ambiguous, prefer the more specific label over unclassified
  7. When a comment spans multiple topics, pick the primary one — what is the reviewer's main concern?
Show full SKILL.md (447 more words)Show less

Confidence Scoring Rubric

Each classification includes a confidence score (0.00–1.00) indicating how certain the classification is. Accumulate the score from these signals:

SignalWeightCriteria
Signal word match+0.25Comment contains signal words/phrases from config.json for the chosen severity or topic label
Unambiguous category+0.25Comment clearly fits one severity and one topic with no viable alternatives
Example pattern match+0.25Comment closely matches a real-world example in this skill or in config.json
Context reinforcement+0.15Multiple independent indicators point to the same classification (e.g., tone + keywords + structure all agree)
Single viable label+0.10No other severity or topic label is a reasonable alternative

Maximum score is 1.00. When multiple signals apply, sum them and cap at 1.00.

Interpretation:

  • >= 0.95 — High confidence: classification can be auto-applied for low-risk labels
  • 0.80–0.94 — Moderate confidence: human confirmation recommended
  • < 0.80 — Low confidence: manual classification required

Additional classification rules:

  • Slash commands mixed with text — if a comment has substantive text before a slash command (e.g., "good analysis\n/override ci/prow/e2e"), classify based on the substantive text, not the slash command
  • Comments directed at bots/AI — classify by the underlying problem, not the fact that the recipient is a bot. "Fix the unit tests" is about test failures (test_gap), "rebase the PR" is a CI/process issue (ci), "push the changes" is a process failure (process). The topic should answer "what went wrong?" not "who is being told to fix it?"

Output Format

Single comment
json
{
  "severity": "<value from config.json severity list>",
  "topic": "<value from config.json topic list>",
  "confidence": 0.95,
  "rationale": "Brief one-line explanation of why this classification was chosen"
}
Full PR
json
{
  "pr": "https://github.com/openshift/hypershift/pull/7620",
  "total_comments": 15,
  "classified": 5,
  "filtered_noise": 10,
  "comments": [
    {
      "id": 2871360513,
      "author": "jparrill",
      "body_preview": "small nit: I would move the vars...",
      "severity": "<value from config.json>",
      "topic": "<value from config.json>",
      "confidence": 0.95,
      "rationale": "Reviewer suggests moving variable declarations for consistency"
    }
  ],
  "summary": {
    "by_severity": {"nitpick": 1, "suggestion": 1, "required_change": 2, "question": 1},
    "by_topic": {"style": 1, "api_design": 1, "logic_bug": 2, "test_gap": 1},
    "by_confidence": {"high": 3, "moderate": 1, "low": 1}
  }
}

Real-World Examples

These are from actual PRs in openshift/hypershift:

Comment: "small nit: I would move the vars key, log and cloudName to var ( section just to be consistent."

json
{"severity": "nitpick", "topic": "style", "confidence": 1.00, "rationale": "Reviewer suggests grouping variables for consistency — cosmetic, not functional"}

Comment: "Why not use NewARMClientOptions here for the clientOptions?"

json
{"severity": "question", "topic": "api_design", "confidence": 0.90, "rationale": "Reviewer asks about API choice for client options construction"}

Comment: "This will panic if items is nil — needs a nil check before the loop"

json
{"severity": "required_change", "topic": "logic_bug", "confidence": 1.00, "rationale": "Nil pointer dereference would cause runtime panic"}

Comment: "failing during hypershift install\n\nClusterRoleBinding is invalid: roleRef.kind: Unsupported value\n"

json
{"severity": "required_change", "topic": "logic_bug", "confidence": 0.90, "rationale": "Installation fails due to missing required roleRef fields"}

Comment: "hypershift-jira-solve-ci - the unit test job is failing and needs fixed"

json
{"severity": "required_change", "topic": "test_gap", "confidence": 0.90, "rationale": "Unit tests are failing — the bot made code changes without ensuring tests pass"}

Comment: "hypershift-jira-solve-ci - rebase the PR to fix the konflux issues"

json
{"severity": "suggestion", "topic": "ci", "confidence": 0.85, "rationale": "PR needs rebasing to resolve CI pipeline issues — classify by the problem (CI), not the recipient (bot)"}

Comment: "hypershift-jira-solve-ci - this still needs fixed since the code did not get pushed"

json
{"severity": "required_change", "topic": "process", "confidence": 0.85, "rationale": "Code changes were not committed/pushed — a process failure, not a code issue"}

Comment: "e2e-aws-4-21 failed on Teardown but due to uncleaned cloud resources, not VPC endpoint blocking the finalizer\n/override ci/prow/e2e-aws-4-21"

json
{"severity": "suggestion", "topic": "ci", "confidence": 0.75, "rationale": "Reviewer explains CI failure root cause and overrides — substantive analysis before the slash command"}

Comment: "Oh no that's ok. I missed that part. No changes requested."

json
{"severity": "unclassified", "topic": "approval", "confidence": 0.70, "rationale": "Reviewer withdrawing their earlier question — acknowledgment"}

Comment: "dup of https://github.com/openshift/hypershift/pull/7727"

json
{"severity": "unclassified", "topic": "process", "confidence": 0.90, "rationale": "Marking PR as duplicate of another — process meta-comment"}

Comment: "The root cause of the CI failure in this PR has been identified. The fix in rejectVpcEndpointConnections doesn't work because of a case mismatch between AWS API responses and SDK v2 enum constants."

json
{"severity": "required_change", "topic": "logic_bug", "confidence": 1.00, "rationale": "Detailed root cause analysis identifying a case mismatch bug"}

Comment: "This controller is doing too much — the reconciler should delegate VPC cleanup to a separate controller instead of inlining it"

json
{"severity": "suggestion", "topic": "architecture_design", "confidence": 0.90, "rationale": "Reviewer identifies a separation-of-concerns issue at the controller level"}

Comment: "The service account token is being logged in plain text here — this needs to be redacted"

json
{"severity": "required_change", "topic": "security", "confidence": 1.00, "rationale": "Sensitive credentials exposed in log output — security vulnerability"}

CodeRabbit issue flagged (starting with _Potential issue_ | _Critical_):

json
{"severity": "required_change", "topic": "logic_bug", "confidence": 0.85, "rationale": "CodeRabbit identified a critical code issue requiring attention"}

© openshift-eng, Apache-2.0. Rendered from Markdown: HTML in the file is shown as text, images as links, and headings moved down two levels. Raw file

Files

SKILL.md and 1 other file in plugins/code-review/skills/classify-review-comment of openshift-eng/ai-helpers.

  • SKILL.md
  • config.json

Open the folder on GitHubat commit a627176

Compare with similar skills

Classify Review Comment next to the 5 skills that share the most tags, products or categories with it. Stars are the repository's; “used in” counts other GitHub owners with a copy.

Classify Review Comment compared with similar skills
SkillStarsUsed inTokensAuto-checkLicenceRepo updated
Classify Review Comment this skillopenshift-eng/ai-helpers120—~2.8kAutomated safety check: PassApache-2.0
PR Babysitteropeninterpreter/openinterpreter69k3 repos~4.2kAutomated safety check: PassApache-2.0
GitHub Review Iterationprisma/orm48k—~2.2kAutomated safety check: PassApache-2.0
PR Review State Fetchprisma/orm48k—~767Automated safety check: PassApache-2.0
PR Finalize Reviewmicrosoft/garnet12k—~3.1kAutomated safety check: PassMIT
Fastlane Pull Request Reviewfastlane/fastlane42k—~550Automated safety check: PassMIT

Similar skills

  • PR Babysitter

    openinterpreter/openinterpreter

    Watches an open GitHub pull request until it merges, handling review comments, diagnosing CI failures and retrying flaky checks along the way.

    69k GitHub starsUsed in 3 repos~4.2k tokens
    DevelopmentAuto-check passed
  • Official

    Runs a loop on a GitHub pull request: fetch review state, triage comments into actions, implement them and resolve threads, repeating until nothing actionable is left.

    48k GitHub stars~2.2k tokensUpdated today
    DevelopmentAuto-check passed
  • Official

    Fetches a pull request's canonical review state as JSON, validates it, and renders markdown, a text summary and triage target files from it using bundled scripts.

    48k GitHub stars~767 tokensUpdated today
    DevelopmentAuto-check passed
  • PR Finalize Review

    microsoft/garnet

    Official

    Checks that a pull request's title and description match its implementation and reviews the code for Garnet best practices, reporting findings without posting them.

    12k GitHub stars~3.1k tokensUpdated today
    DevelopmentAuto-check passed
  • Reviews a fastlane pull request against its linked issue and the project guides, separating blocking from non-blocking findings and handling vulnerabilities privately.

    42k GitHub stars~550 tokensUpdated today
    DevelopmentAuto-check passed
  • Reviews open pull requests in the daisyUI repository using read-only GitHub data and isolated base-versus-PR checks, then writes a merge verdict report.

    43k GitHub stars~766 tokensUpdated 8 days ago
    DevelopmentAuto-check passed

More from openshift-eng/ai-helpers

All 118 skills in this repo
  • Investigate CI Reliability

    openshift-eng/ai-helpers

    Find and independently validate actionable reliability defects across OpenShift release jobs and presubmits, then export portable issue handoffs.

    120 GitHub stars~1.9k tokensUpdated yesterday
    Auto-check passed
  • Address Review PR

    openshift-eng/ai-helpers

    Fetch and address all PR review comments — categorize by priority, make code changes, post replies, and push.

    120 GitHub stars~2.9k tokensUpdated yesterday
    Auto-check passed
  • Categorize Activity Types

    openshift-eng/ai-helpers

    Categorize Jira issues into Red Hat Sankey Activity Type categories using MCP Jira tools.

    120 GitHub stars~2.4k tokensUpdated yesterday
    Auto-check passed
  • Has Review Work

    openshift-eng/ai-helpers

    Decide whether a GitHub PR has unanswered authorized review comments or new required CI failures worth a follow-up agent.

    120 GitHub stars~1.9k tokensUpdated yesterday
    Auto-check passed
  • Must Gather Analyzer

    openshift-eng/ai-helpers

    Analyze OpenShift must-gather diagnostic data including cluster operators, pods, nodes, and network components.

    120 GitHub stars~2.3k tokensUpdated yesterday
    Auto-check passed
  • Payload Autodl JSON

    openshift-eng/ai-helpers

    Schema for the autodl JSON data file produced by payload-analysis for database ingestion — you must use this skill whenever generating the autodl JSON file

    120 GitHub stars~2.6k tokensUpdated yesterday
    Auto-check passed

Works with

Categories

Questions about Classify Review Comment

What does Classify Review Comment do?

Classify GitHub PR review comments by severity and topic. An agent skill from openshift-eng/ai-helpers. Classify Review Comment is an agent skill from openshift-eng/ai-helpers. Classify GitHub PR review comments by severity and topic.

When should I use Classify Review Comment?

Classify Review Comment fits situations like: the user wants to categorize; understand patterns in code review feedback — whether for a single comment; an entire pull request; requests like classify this comment.

How do I install Classify Review Comment in Claude Code?

Run `npx skills add openshift-eng/ai-helpers --skill classify-review-comment -a claude-code`. Or copy the skill folder (plugins/code-review/skills/classify-review-comment in openshift-eng/ai-helpers) into .claude/skills/classify-review-comment in your project. Claude Code loads it when a task matches its description.

How do I install Classify Review Comment in Codex?

Run `npx skills add openshift-eng/ai-helpers --skill classify-review-comment -a codex`. Or copy the skill folder (plugins/code-review/skills/classify-review-comment in openshift-eng/ai-helpers) into .agents/skills/classify-review-comment in your project. Codex loads it when a task matches its description.

Can I use Classify Review Comment in Cursor, Gemini CLI or GitHub Copilot?

Cursor, Gemini CLI, GitHub Copilot and OpenCode also load SKILL.md folders. With the skills CLI, run `npx skills add openshift-eng/ai-helpers --skill classify-review-comment -a cursor` (or -a gemini-cli, github-copilot or opencode for the others). To copy it by hand, put the folder in .cursor/skills/classify-review-comment, .gemini/skills/classify-review-comment, .github/skills/classify-review-comment and .opencode/skills/classify-review-comment in your project.

What does Classify Review Comment need to run?

Going by SKILL.md and its folder, Classify Review Comment needs the command-line tools its instructions call (gh).

Does Classify Review Comment access the network?

SKILL.md names 1 domain. In commands or code: github.com; the agent is likely to contact it when it follows the instructions. This is read from the text; nothing was executed.

Is Classify Review Comment safe to install?

Our automated static check of SKILL.md found no risky patterns, such as piping downloads into a shell, reading credential files or hidden Unicode. It is not a guarantee. Review the folder before installing.

What licence does Classify Review Comment use?

Classify Review Comment is published under the Apache-2.0 licence (the repository's licence). It allows redistribution, so the full SKILL.md is shown on this page.

How many tokens does Classify Review Comment use?

About 2.8k tokens (SKILL.md is roughly 11k characters). Agents keep only the skill's name and description in context until a task matches; then they load SKILL.md in full.

What are the alternatives to Classify Review Comment?

Skills that share tags, products or a category with Classify Review Comment: PR Babysitter (openinterpreter/openinterpreter, 69k stars), GitHub Review Iteration (prisma/orm, 48k stars), PR Review State Fetch (prisma/orm, 48k stars) and PR Finalize Review (microsoft/garnet, 12k stars). The comparison table on this page puts their stars, adoption, token cost, safety result and licence side by side.

Who maintains Classify Review Comment?

openshift-eng (a GitHub organization) maintains it in openshift-eng/ai-helpers, which has 120 GitHub stars. The repository holds 118 skills in this directory. The repository was last updated on October 6, 2026.

Source: openshift-eng/ai-helpers on GitHub. Facts on this page come from the repository at the commit we read; the author's words are quoted as theirs.