New: 98,000+ skills indexed from GitHub repos with 100+ stars · every skill safety-checkedBrowse top skillsTop skills
Esc to closeBrowse all skills
Submit a skillBrowse skills
Submit a skillBrowse skills

Agent skill

Triage Agent Eval Failures

by novuhq in novuhq/novu

Triage failing @novu/agent-evals scenarios to decide whether a failure is real or flaky, and whether to fix the playbook/prompt or the test (grader, tape, scenario, or judge).

Custom licenceAuto-check passedAI & LLM Engineering

Install Triage Agent Eval Failures

skills CLI
$ npx skills add novuhq/novu --skill triage-agent-eval-failures -a claude-code

Project install by default; add -g for ~/.claude/skills/.

GitHub CLI
$ gh skill install novuhq/novu triage-agent-eval-failures --agent claude-code

Project scope by default; add --scope user for a personal install. Needs GitHub CLI 2.90.0 or later (public preview).

Manual copy
$ git clone --depth 1 https://github.com/novuhq/novu.git skills-src && mkdir -p .claude/skills && cp -r skills-src/.cursor/skills/triage-agent-eval-failures .claude/skills/triage-agent-eval-failures && rm -rf skills-src

Use ~/.claude/skills/ instead of .claude/skills for a personal install. The folder must contain SKILL.md.

Claude Code skills documentation · loads skills from .claude/skills/

Facts

Skill name
triage-agent-eval-failures
GitHub stars
40k
Token cost
~1.5k tokens
SKILL.md length
625 words
Files
2
Skills in repo
28
Repo updated
First seen
Licence
Custom licence

At a glance

Triage failing @novu/agent-evals scenarios to decide whether a failure is real or flaky, and whether to fix the playbook/prompt or the test (grader, tape, scenario, or judge).

  • Works in 4 steps: identify which grader failed and its kind → read the RunResult evidence → classify the failure → …
  • An agent-evals scenario fails
  • SKILL.md covers Rule 0: rule out flakiness…, Step 1: identify which grader…, Step 2: read the RunResult… and Step 3: classify the failure, plus 3 more sections
  • Calls pnpm

What it does

Triage Agent Eval Failures is an agent skill from novuhq/novu. Triage failing @novu/agent-evals scenarios to decide whether a failure is real or flaky, and whether to fix the playbook/prompt or the test (grader, tape, scenario, or judge). Use when an agent-evals scenario fails, when the user asks why an eval is red, or when deciding whether to fix the test or the prompt.

Its SKILL.md is about 1.5k tokens, which your agent loads only when the skill is triggered. The skill folder holds 1 other file (for example `reference.md`).

It sits in AI & LLM Engineering, covering LLM evaluation. The repository describes itself as: The open-source communication infrastructure for agents and products.

When your agent uses it

  • An agent-evals scenario fails
  • The user asks why an eval is red
  • Deciding whether to fix the test

Example prompts

  • “/triage-agent-eval-failures”

Workflow steps

4 steps, taken from the step headings in SKILL.md.

  1. identify which grader failed and its kind
  2. read the RunResult evidence
  3. classify the failure
  4. apply one bounded fix, then verify

What it can do on your machine

Read from SKILL.md and the folder at commit c4d3eda. It shows what the files ask for, not the result of running them.

  • Tool permissions

    Pre-approves nothing: there is no allowed-tools line, so your agent's usual permission prompts apply.

    From allowed-tools in the SKILL.md frontmatter.

  • Runs code

    Shell commands in SKILL.md call:

    • pnpm

    From the folder's file list and the shell code blocks in SKILL.md.

  • Network

    No URLs in SKILL.md. Its commands use pnpm, which can reach the network depending on how they are called.

    From URLs in SKILL.md, links to its own repository left out.

  • Credentials

    Names no API keys, tokens, secrets or passwords.

    From names ending in _API_KEY, _TOKEN, _SECRET, _KEY or _PASSWORD in SKILL.md.

Context cost

Triage Agent Eval Failures loads about 1.5k tokens when it runs. Until then it costs about 84 tokens; SKILL.md has 625 words of instructions outside code blocks.

Always · name and description, kept in context so the agent knows when to use it
~84
When it runs · the whole SKILL.md, loaded when a task matches
~1.5k

Estimates: characters ÷ 4, the usual rule of thumb; real counts depend on the model's tokenizer. Scripts and assets cost tokens only if the agent reads them.

Safety

Auto-check passed

The automated check found no risky patterns in SKILL.md.

Automated static check — not a guarantee. Review scripts before installing. It scans the text of SKILL.md for risky patterns (piping downloads into a shell, reading credential files, hidden Unicode, destructive commands); files beside SKILL.md are not scanned.

SKILL.md

Its licence (Custom licence) doesn't allow us to republish the file, so here is its outline and opening line. It has 625 words (~1,486 tokens).

“Diagnose a failing scenario in libs/agent-evals and produce a verdict: is the failure real (the playbook under test regressed) or is the test wrong (grader / tape / scenario / judge), or is it just flaky (model non-determinism)?”

— opening of SKILL.md by novuhq, Custom licence
name
triage-agent-eval-failures

Read the full SKILL.md on GitHub

Files

SKILL.md and 1 other file in .cursor/skills/triage-agent-eval-failures of novuhq/novu.

  • SKILL.md
  • reference.md

Open the folder on GitHubat commit c4d3eda

Compare with similar skills

Triage Agent Eval Failures next to the 5 skills that share the most tags, products or categories with it. Stars are the repository's; “used in” counts other GitHub owners with a copy.

Triage Agent Eval Failures compared with similar skills
SkillStarsUsed inTokensAuto-checkLicenceRepo updated
Triage Agent Eval Failures this skillnovuhq/novu40k—~1.5kAutomated safety check: PassCustom licence
LLM Benchmarking with lm-evaluation-harnessOrchestra-Research/AI-Research-SKILLs13k8 repos~3kAutomated safety check: PassMIT
Hugging Face Local Model Evalshuggingface/skills11k2 repos~1.6kAutomated safety check: PassApache-2.0
Looperksimback/looper710—~2.7kAutomated safety check: NotesMIT
Agent Eval Engineeringlangchain-ai/langchain-skills1.3k—~4kAutomated safety check: PassMIT
Quality FlywheelGoogleCloudPlatform/vertex-ai-samples792—~2kAutomated safety check: PassApache-2.0

Similar skills

  • LLM Benchmarking with lm-evaluation-harness

    Orchestra-Research/AI-Research-SKILLs

    Runs lm-evaluation-harness to benchmark language models on academic suites such as MMLU, GSM8K and HumanEval, compare models and track training checkpoints.

    13k GitHub starsUsed in 8 repos~3k tokens
    AI & LLM EngineeringAuto-check passed

  • Hugging Face Local Model Evals

    huggingface/skills

    Official

    Runs evaluations of Hugging Face Hub models on local hardware with inspect-ai or lighteval, and helps choose between vLLM, Transformers and accelerate backends.

    11k GitHub starsUsed in 2 repos~1.6k tokens
    AI & LLM EngineeringAuto-check passed

  • Looper

    ksimback/looper

    Scaffold a well-designed agent loop with best-practice coaching and a cross-model review council.

    710 GitHub stars~2.7k tokensUpdated 2 mo ago
    AI & LLM EngineeringAuto-check: notes

  • Agent Eval Engineering

    langchain-ai/langchain-skills

    Official

    Builds agent evaluations in stages: inspect the repository and traces, agree a Task Spec with you, then build, audit and run a Harbor task with an independent verifier.

    1.3k GitHub stars~4k tokensUpdated yesterday
    AI & LLM EngineeringAuto-check passed

  • Quality Flywheel

    GoogleCloudPlatform/vertex-ai-samples

    Evaluate and improve GenAI models and agents using the Google GenAI Evaluation SDK.

    792 GitHub stars~2k tokensUpdated yesterday
    AI & LLM EngineeringAuto-check passed

  • Eval Harness

    cloudnative-co/claude-code-starter-kit

    Formal evaluation framework for Claude Code sessions implementing eval-driven development (EDD) principles.

    153 GitHub starsUsed in 9 repos~1.3k tokens
    AI & LLM EngineeringAuto-check passed

More from novuhq/novu

All 28 skills in this repo

  • Ink Tui Wizard

    novuhq/novu

    Build terminal user interfaces (TUIs) using Ink (React for CLIs) and @inkjs/ui with a reactive, session-driven wizard pattern.

    40k GitHub stars~1.9k tokensUpdated today
    Auto-check passed

  • Novu Design Workflow

    novuhq/novu

    Design notification workflows the Novu way — choose channels, set severity, decide when a workflow is critical, configure digests, and route based on subscriber state.

    40k GitHub stars~2.6k tokensUpdated today
    Auto-check passed

  • Novu Inbox Integration

    novuhq/novu

    Integrate Novu's in-app notification inbox into web applications.

    40k GitHub stars~5.2k tokensUpdated today
    Auto-check passed

  • Novu Manage Preferences

    novuhq/novu

    Configure notification preferences in Novu at the workflow and subscriber level.

    40k GitHub stars~1.8k tokensUpdated today
    Auto-check passed

  • Novu Manage Subscribers

    novuhq/novu

    Create, update, search, and delete subscribers in Novu. An agent skill from novuhq/novu.

    40k GitHub stars~1.3k tokensUpdated today
    Auto-check passed

  • Novu Trigger Notification

    novuhq/novu

    Trigger Novu notification workflows to send messages across email, SMS, push, chat, and in-app channels.

    40k GitHub stars~1.2k tokensUpdated today
    Auto-check passed

Categories

Questions about Triage Agent Eval Failures

What does Triage Agent Eval Failures do?

Triage failing @novu/agent-evals scenarios to decide whether a failure is real or flaky, and whether to fix the playbook/prompt or the test (grader, tape, scenario, or judge). Triage Agent Eval Failures is an agent skill from novuhq/novu. Triage failing @novu/agent-evals scenarios to decide whether a failure is real or flaky, and whether to fix the playbook/prompt or the test (grader, tape, scenario, or judge).

When should I use Triage Agent Eval Failures?

Triage Agent Eval Failures fits situations like: an agent-evals scenario fails; the user asks why an eval is red; deciding whether to fix the test.

How do I install Triage Agent Eval Failures in Claude Code?

Run `npx skills add novuhq/novu --skill triage-agent-eval-failures -a claude-code`. Or copy the skill folder (.cursor/skills/triage-agent-eval-failures in novuhq/novu) into .claude/skills/triage-agent-eval-failures in your project. Claude Code loads it when a task matches its description.

How do I install Triage Agent Eval Failures in Codex?

Run `npx skills add novuhq/novu --skill triage-agent-eval-failures -a codex`. Or copy the skill folder (.cursor/skills/triage-agent-eval-failures in novuhq/novu) into .agents/skills/triage-agent-eval-failures in your project. Codex loads it when a task matches its description.

Can I use Triage Agent Eval Failures in Cursor, Gemini CLI or GitHub Copilot?

Cursor, Gemini CLI, GitHub Copilot and OpenCode also load SKILL.md folders. With the skills CLI, run `npx skills add novuhq/novu --skill triage-agent-eval-failures -a cursor` (or -a gemini-cli, github-copilot or opencode for the others). To copy it by hand, put the folder in .cursor/skills/triage-agent-eval-failures, .gemini/skills/triage-agent-eval-failures, .github/skills/triage-agent-eval-failures and .opencode/skills/triage-agent-eval-failures in your project.

What does Triage Agent Eval Failures need to run?

Going by SKILL.md and its folder, Triage Agent Eval Failures needs the command-line tools its instructions call (pnpm).

Does Triage Agent Eval Failures access the network?

SKILL.md contains no URLs. Any network use would come from the scripts or tools the agent runs. This is read from the text; nothing was executed.

Is Triage Agent Eval Failures safe to install?

Our automated static check of SKILL.md found no risky patterns, such as piping downloads into a shell, reading credential files or hidden Unicode. It is not a guarantee. Review the folder before installing.

What licence does Triage Agent Eval Failures use?

Triage Agent Eval Failures has a licence file (the repository's licence) that doesn't match a standard licence. Read it on GitHub before reusing the skill.

How many tokens does Triage Agent Eval Failures use?

About 1.5k tokens (SKILL.md is roughly 5.9k characters). Agents keep only the skill's name and description in context until a task matches; then they load SKILL.md in full.

What are the alternatives to Triage Agent Eval Failures?

Skills that share tags, products or a category with Triage Agent Eval Failures: LLM Benchmarking with lm-evaluation-harness (Orchestra-Research/AI-Research-SKILLs, 13k stars), Hugging Face Local Model Evals (huggingface/skills, 11k stars), Looper (ksimback/looper, 710 stars) and Agent Eval Engineering (langchain-ai/langchain-skills, 1.3k stars). The comparison table on this page puts their stars, adoption, token cost, safety result and licence side by side.

Who maintains Triage Agent Eval Failures?

novuhq (a GitHub organization) maintains it in novuhq/novu, which has 40,132 GitHub stars. The repository holds 28 skills in this directory. The repository was last updated on October 10, 2026.

Source: novuhq/novu on GitHub. Facts on this page come from the repository at the commit we read; the author's words are quoted as theirs.