Verify Vuln Issues

Verify vulnerability issues

This repo's monitoring action files a GitHub issue for every commit it thinks is a security patch. Many are genuine; some are false positives (routine dependency bumps, feature commits, build-tooling changes, AI-misclassified commits). This skill verifies the unverified ones, comments, and labels each.

Scope & disposition

• Target: open issues authored by github-actions[bot] that carry none of the verdict labels — true-positive, false-positive, or dependency. An issue carrying any of those, or one already closed (e.g. a dependency bump closed in an earlier run, or a duplicate), has already been dispositioned — skip it. prepare.mjs applies exactly this filter.

• For every targeted issue: post one analysis comment, then apply the disposition:

  Finding	Action
  Genuine security fix	comment + add true-positive label
  False positive	comment + add false-positive label
  Pure dependency bump	comment + add dependency label + close (not planned)

• Pure dependency bump = the commit's entire diff is package manifests, lockfiles, or version changelogs — no hand-written fix code. Close it regardless of whether the upstream advisory is real; dependency bumps are out of scope for this tracker.

  • Exception — keep it open: if the commit also carries actual vulnerability-fix code (e.g. it backports/cherry-picks specific fix code into vendored sources, not merely a version number/lockfile entry), it is a real finding — treat it as Verified (true-positive), do not close it.

• Duplicate issues: the monitor sometimes files more than one issue for the same commit. prepare.mjs detects these (writes /tmp/vsa/duplicates.json). Keep and verify the canonical one (lowest issue number = first filed); the rest are duplicates — close them as duplicates, or delete with gh issue delete only after the user confirms, since deletion is permanent.

• Labels: add the verdict label. Never remove a label someone else applied — if an issue carries a conflicting one, stop and surface it. You may correct a verdict this skill applied earlier (e.g. true-positive → false-positive) when re-analysis warrants: remove the stale label, add the right one, and post a correction comment explaining why.

• Relay a final summary (counts per repo, false positives, issues closed as dependency bumps, duplicates found).

Prerequisites

gh must be authenticated. gh auth status reporting "not logged in" inside the sandbox is normal for git operations, but gh api/gh issue calls still need a real token. If gh calls fail with HTTP 401/Bad credentials, ask the user to run ! gh auth login in the prompt (the dummy-GH_TOKEN trick does not work — the proxy rejects it).

Ensure the dependency label exists (create once if missing):

gh label create dependency --repo <owner/repo> \
  --description "Issue is only a dependency/version bump — out of scope, ignored" \
  --color ededed

Procedure

1. Fetch & prepare

node .claude/skills/verify-vuln-issues/prepare.mjs

It lists issues, filters to unverified ones, parses each body for its repo + full commit SHA (via scripts/lib/parser.mjs), downloads every commit diff and metadata, and writes /tmp/vsa/:

• todo.json — [{number, state, repository, commitSha, ...}]
• ctx/<n>.txt — per-issue: the AI claim + commit message + file list + diff (lockfiles trimmed, huge files truncated)
• duplicates.json — groups of issues sharing one commit SHA (see "Duplicate issues" above)
• results.json — starts {}; record verdicts here as you go.

The --- FILES CHANGED --- list in each context file is what you use to spot a pure dependency bump (only package.json / *.lock / go.mod / go.sum / CHANGELOG* / doc/changelogs/* / toolchain files).

2. Analyze, in batches

Read ctx/<n>.txt files (~10–15 per turn; read large ones individually). For each issue decide the finding using the rubric below, and record into results.json (verified / false-positive / dependency) so progress survives compaction.

3. Comment + label (+ close)

Write each comment to /tmp/vsa/comments/<n>.md, then per issue:

gh issue comment <n> --repo <owner/repo> --body-file /tmp/vsa/comments/<n>.md
# verified:
gh issue edit <n> --repo <owner/repo> --add-label true-positive
# false positive:
gh issue edit <n> --repo <owner/repo> --add-label false-positive
# pure dependency bump:
gh issue edit <n> --repo <owner/repo> --add-label dependency
gh issue close <n> --repo <owner/repo> --reason "not planned"

Loop over a batch; report any failures.

Verdict rubric

The commit diff is ground truth — the issue body is an LLM guess and is often wrong about the mechanism, severity, or even the vuln class.