Triage Failure Reports

Triage duplicate issues by Failure Data signature

Chatbook posts machine-generated bug reports to GitHub. They pile up because the same underlying bug gets reported many times across different services, versions, and users. This skill turns that pile into a small number of signature-grouped clusters, ties each cluster to the issue or PR that resolves it, and closes the duplicates with a comment that tells the reporter where the real fix lives.

The hard part is not the closing - it's being sure two reports are actually the same bug before you act on someone else's issue. Most of this skill is about earning that confidence.

How a Chatbook failure report is structured

Each report has an auto-generated <details> block. The fields that matter:

• Debug Data table - Version and ReleaseID (the git SHA the user was running). These drive the timeline check; do not skip them.
• Settings table - Model, e.g. <|"Service" -> "OpenAI", "Name" -> Automatic|>. Tells you which service/model triggered it.
• Failure Data - the heart of the signature:
  • "Evaluation" - the failing call, e.g. Wolfram`Chatbook`Common`resolveFullModelSpec[<|"Service" -> "OpenAI", "Name" -> Automatic|>]. The short function name (resolveFullModelSpec) is the primary key.
  • "Expression" - the value that broke a Confirm* check, e.g. Missing["NoModelList"]. (Unhandled-definition failures don't have this field; the bad value is a call argument instead - see the script's fallback.)
  • "Pattern" - what the value failed to match, e.g. _List | Missing["NotConnected"].
  • "Information" - Tag@@path:line,col. Useful for the file; ignore the line number - it drifts between versions for the same bug.
• Stack Data - the call stack, e.g. resolveAutoSettings0 -> EvaluateChatInput.

What makes two reports the same bug

Same failing function + same confirmed expression (and pattern, when present). That's it. Everything else is allowed to vary:

• Different service/model (OpenAI vs Anthropic vs LocalEvaluator) - same bug.
• Different line number in Information - same bug, different version.
• Different version/ReleaseID - same bug reported over time.

The trap to avoid: the same expression surfacing in a different function is a sibling bug, not a duplicate. For example Missing["NoModelList"] thrown inside resolveFullModelSpec (chat evaluation) is a different bug from Missing["NoModelList"] passed into makeServiceModelMenu (the model submenu UI), even though both mention NoModelList. They get fixed by different PRs. Keyword co-occurrence is a candidate filter, never a conclusion.

Workflow

1. Establish the reference signature

If the user named a reference issue, read it (gh issue view <N>) and pull the failing function + expression + pattern from its Failure Data. If they instead handed you a set of reports or a fix, derive the signature from the cluster or from the code the fix touches.

2. Search broadly for candidates

Search the distinctive symbols from the signature - typically the head of the expression (NoModelList) and the failing function (resolveFullModelSpec):

gh issue list --state all --search "NoModelList" --json number,title,state
gh issue list --state all --search "resolveFullModelSpec" --json number,title,state

Two rules that matter:

• Search --state all. You want closed siblings too: the canonical issue may already be closed, and you must not re-close or miss it.
• Use clean alphanumeric tokens. GitHub's search tokenizer garbles backticks, brackets, and quotes - search NoModelList, not Missing["NoModelList"].

Strong candidates appear in every term's results (intersection). The helper script does this intersection for you.

3. Confirm each candidate against the signature

Do not trust keyword co-occurrence. Open each candidate's Failure Data and verify the failing function and confirmed expression actually match. The helper script extracts and groups these for you:

# Intersect the searches and print every candidate grouped by signature:
python .claude/skills/triage-failure-reports/scripts/extract_signatures.py \
  --search NoModelList --search resolveFullModelSpec

# Or inspect a hand-picked set:
python .claude/skills/triage-failure-reports/scripts/extract_signatures.py 1550 1545 547

Run it from the repo working directory so gh resolves the right repository. It prints, per signature group, every issue with its state, version, service, created date, and ReleaseID - exactly the columns you need for the next two steps. Read scripts/extract_signatures.py if you need to adjust parsing for a report format it doesn't recognize.

Confirm the groups make sense: the cluster you care about shares one function + expression; anything in a different function is a sibling to set aside.

4. Identify and verify the resolver

Every cluster needs a single thing it is "resolved by". Two cases:

• Duplicates of another issue. Pick the canonical issue - usually the earliest, or the one with the clearest title/discussion. The rest are duplicates of it.
• Fixed by a merged PR. The user may name it ("these are fixed by #906"), or you find it (gh pr list --search, or the PR that last touched the failing function). Verify it - don't take it on faith. A #NNNN can be an issue or a PR (they share one number space), so confirm with gh pr view <N>, then check the diff actually addresses this signature:

gh pr view 906 --json number,title,state,mergedAt,mergeCommit
gh pr diff 906 | grep -iE 'makeServiceModelMenu|NoModelList'

You want to see the diff add or change handling for the failing function and the expression (e.g. a new makeServiceModelMenu[..., Missing["NoModelList"]] definition). If the diff doesn't touch the signature's code path, it is not the fix.