Improve Review PR

Improve Review PR

Run a once-per-day outer loop over the automated code-review stage.

The inner loop is already running: review-pr comments on pull requests throughout the day. This skill is the outer loop: read how humans reacted to those comments, extract durable organizational knowledge, and update the review skill so the next inner-loop runs get better.

Goal

Improve future automated reviews by learning from human validation and correction of previous review-pr comments.

Do not re-review product code. Do not restate one-off PR opinions. Only capture knowledge that should change how the review agent behaves on future PRs.

Inputs

• The current checkout of the repository that owns .agents/skills/review-pr/SKILL.md
• GitHub API access via authenticated gh
• Optional lookback window, default last 24 hours
• Optional feedback_corpus.json produced by:

  python3 .agents/skills/improve-review-pr/scripts/collect_review_feedback.py \
    --repo OWNER/REPO \
    --since-hours 24 \
    --output feedback_corpus.json

If feedback_corpus.json is missing, run the collector yourself before analyzing.

Workflow

1. Collect the day's review-agent interactions

Run or read feedback_corpus.json.

The corpus should include, for the lookback window:

• Pull requests that received an automated review from the review agent
• The review agent's top-level review bodies and inline comments
• Human replies to those comments
• Human reactions (for example +1, eyes, confused, thumbs down) when available
• Whether the human accepted a suggestion, dismissed it, edited around it, or explicitly disagreed
• Whether the PR author or another reviewer later fixed the same issue, ignored it, or called it wrong

Identify the review agent by login when possible (github-actions[bot], a bot account, or a configured login). Prefer comments that originated from the review-pr publish path.

2. Score each feedback item

For each human interaction, classify the outcome:

• validated — human agreed, accepted the suggestion, or fixed the issue as recommended
• corrected — human said the finding was wrong, incomplete, too noisy, or the wrong severity
• refined — human mostly agreed but adjusted the guidance, scope, or preferred pattern
• ambiguous — not enough signal to learn from

Ignore pure acknowledgements with no substantive judgment.

3. Synthesize durable organizational knowledge

Look across the day's validated/corrected/refined items for patterns worth remembering. Good candidates:

• Repo conventions the review agent repeatedly misses
• False-positive classes that should be demoted or avoided
• Severity calibration mistakes
• Preferred alternatives to common suggestions
• Missing checks that humans keep adding manually
• Guidance about when not to comment

Reject learnings that are:

• One-off to a single PR or file
• Already covered well by review-pr or a local companion skill
• Product-feature preferences unrelated to review quality
• Temporary project constraints unlikely to recur
• Changes that would break the review-pr output schema, severity labels, safety rules, evidence rules, suggestion-block constraints, or annotated-diff line contract

Prefer a small number of high-confidence learnings over many weak ones.

4. Decide whether to update the skill

Choose exactly one outcome:

• no_changes — no durable learning worth encoding
• update_review_pr — update core .agents/skills/review-pr/SKILL.md
• update_review_pr_local — update or create .agents/skills/review-pr-local/SKILL.md for repository-specific guidance
• both — core and local updates are both warranted

Use review-pr-local for repository-specific conventions. Keep portable review behavior in core review-pr.

If no durable learning exists, stop after writing the synthesis report. Do not open an empty PR.