LLM evaluation

Sets up Arize Phoenix to trace, evaluate and monitor LLM applications, with instrumentation for OpenAI, LangChain and LlamaIndex and a self-hosted server.

Orchestra-Research/AI-Research-SKILLs

Investigate a single failing eval from the convex-evals system.

get-convex/convex-evals

Best practices for creating expectations and grader files to evaluate guidance quality.

GoogleChrome/modern-web-guidance-src

Create new skills, modify and improve existing skills, and measure skill performance.

ZS520L/HanakoPro

Add a new evaluator to the amp-evaluation Python library. An agent skill from wso2/agent-manager.

wso2/agent-manager

This skill should be used for advanced LLM evaluation: LLM-as-judge systems, direct scoring, pairwise comparison, rubric calibration, evaluator bias mitigation, confidence scoring, and automated…

guanyang/open-agent-hub

测试 use-self 替身会议的辩论质量。给定 persona + 3 个决策场景，运行完整三阶段辩论并按 5 个维度评分，输出质量报告。

YIKUAIBANZI/forge-skill

A skill your agent uses when the user wants to work on a Claude Code skill file (SKILL.md): writing one from scratch, testing whether an existing one works well, running evals or benchmarks…

avibebuilder/claude-prime

Decide which AI agent behaviors are worth an eval case, then write those cases — harness-, framework-, and language-agnostic.

agentailor/fullstack-langgraph-nextjs-agent

Написание и запуск Claude Code dynamic workflows (JS-оркестратор субагентов).

AnastasiyaW/codex-claude-code-config

Eval enablement accelerator — help customers think through "what does good look like" for their AI agent, then generate a structured eval plan and test cases they can use immediately.

microsoft/eval-guide

Analyze all failures in a convex-evals run, spawning parallel sub-agents to investigate each failure and producing a report with classifications and recommendations.

get-convex/convex-evals

Test a pre-built afm binary at any path — runs pre-flight safety checks, then any combination of unit tests, assertions, smart analysis, promptfoo evals, batch validation, OpenAI compat, GPU…

scouzi1966/maclocal-api

AI quality judge that scores agent responses 0-10 across helpfulness, accuracy, completeness, and clarity.

Atmosphere/atmosphere

Score a Python generative-AI app's outputs on groundedness, relevance, coherence, and safety before it ships.

timothywarner-org/claude-code

Systematic benchmarking framework for Local Brain Search memory system with LLM-as-judge scoring

Abilityai/cornelius

Runs a measurable keep-or-discard improvement loop against a runnable sensor, from framing a goal and KPI through baseline, judging and held-out verification.

bgauryy/octocode

Triage failing @novu/agent-evals scenarios to decide whether a failure is real or flaky, and whether to fix the playbook/prompt or the test (grader, tape, scenario, or judge).

novuhq/novu

Master comprehensive evaluation strategies for LLM applications, from automated metrics to human evaluation and A/B testing.

davila7/claude-code-templates

Run evaluation pipelines on Claude Code skills to test triggering accuracy, workflow correctness, and output quality.

AgriciDaniel/skill-forge

Patterns for continuous autonomous agent loops with quality gates, evals, and recovery controls.

affaan-m/ECC

Design well-structured agent loops with best-practice coaching and cross-model review gates before you run them.

fabricioctelles/skills

Build a repeatable eval loop that grades agent output with an LLM judge, so prompt/skill changes get scored against a baseline instead of eyeballed.

Archive228/loopkit

Conduct a local Publisher evaluation loop in five steps: scrape/run, eval, diagnose, improve, checkpoint.

malloydata/publisher

The step-by-step procedure for running this repo's Promptfoo evals with a generator and a judge over provider APIs.

maplibre/maplibre-agent-skills

Run offline Python (pydanticevals) or Node.js (logfire/evals) evaluations and review them in Logfire.

pydantic/skills

Verify a Nexus change — pick a Jest lane, write a test that can actually fail, run the in-app Obsidian CLI loop, drive the eval harness, or fix a shipped-docs drift failure.

ProfSynapse/nexus

Write or extend an E2E journey spec in evals/specs that proves a PR's change to a human reviewer.

different-ai/openwork

Inspect, query, compare, or explicitly export GitHits agent-eval history in Braintrust using the repository's verified workflow.

githits-com/githits-cli

Run a targeted local React Doctor Evals loop against an uncommitted rule change.

millionco/react-doctor

Help users build robust infrastructure for measuring, monitoring, and iterating on AI product performance using human, code-based, and LLM-as-a-judge methodologies.

RefoundAI/lenny-skills

Scaffold a new LLM evaluation suite package with Playwright config, evaluate fixture, and package files.

elastic/kibana

Trigger an on-demand @kbn/evals Buildkite run by describing what you want in plain English.

elastic/kibana

[Beta] CI-only eval regression runner using gh-aw (GitHub Agentic Workflows).

pskoett/pskoett-ai-skills

Choose, inspect, validate, and report Paperclip Runner or Product E2E evaluations while preserving evidence, provenance, cost, and failure classification.

paperclipai/paperclip

Run skill evals via evals/runner.js — wrapper validates skill names, captures stdout/stderr, persists JSON results.

ww-w-ai/bkit-claude-code

Builds an eval plan for an AI feature - error analysis on real traces, failure-mode ranking, code checks and binary LLM judges validated against human labels, CI tests, production monitoring, and a…

menkesu/awesome-pm-skills

You MUST load this skill before the skill-creator skill AND before making ANY change to, or conducting a review of ANY Agent Skill.

sammcj/agentic-coding

Audit supervised fine-tuning datasets against the behavior and task they are meant to teach.

tokenbender/agent-guides

A skill your agent uses when measuring whether an LLM or agent system actually got better and gating merges on it: golden sets, fixing an inflated LLM-as-judge, scoring RAG (faithfulness, contextual…

ericrisco/rsc-harness

A skill your agent uses when evaluation results need triage and loop-closing — spec failures route to deterministic checks or context rules, generalization failures to the evaluator backlog.

tikalk/adlc-team-skills

Guides LLM fine-tuning with LoRA and QLoRA through Hugging Face PEFT, from dataset validation and training checks to adapter merging, quantization and deployment.

Jeffallan/claude-skills

Designs, tests and refines LLM prompts: zero-shot, few-shot and chain-of-thought patterns, system prompts, structured output schemas and evaluation test suites.

Jeffallan/claude-skills

Designs retrieval-augmented generation systems: document chunking, embeddings, vector store setup, hybrid search, reranking and retrieval evaluation, with checks at each step.

Jeffallan/claude-skills

Set up compliance exports, drift detection, evaluations, scoring, and learning analytics

ucsandman/DashClaw

Guides an interactive error analysis of LLM outputs: studies the dataset, builds a review interface, picks diverse samples and organizes the failure modes you annotate.

ai-evals-course/evals-skills

Run local evaluations for Hugging Face Hub models with inspect-ai or lighteval.

henryalouf/ruflow

Writes fictional but realistic coaching or therapy session transcripts for evals, demos and few-shot examples, in several modalities and export formats.

glebis/claude-skills