LLM evaluation

A skill your agent uses for Codex models/pricing, scheduled tasks, skills, settings, setup, troubleshooting, customization, automations, and self-knowledge—including 'you,' 'your,' 'this app,' or…

theowenyoung/home

Propose an improved version of a prompt registered in a self-hosted AgentX (AgentX-trace-eval) instance, using real low-rated evaluation results as evidence, then publish it as a new version once…

AgentX-ai/AgentX-Trace-Eval

Runs the `autoctx` CLI to improve an approach to a task over several generations, score or refine a single output and inspect what a run produced.

greyhaven-ai/autocontext

Create, refine, and benchmark agent skills. An agent skill from feiskyer/claude-code-settings.

feiskyer/claude-code-settings

Convert an evaluation dataset or leaderboard into the Every Eval Ever (EEE) schema — aggregate .json logs (eval.schema.json) and optional instance samples.jsonl sidecars…

evaleval/every_eval_ever

Install and run a verifiers environment — smoke testing during development and full benchmark evals.

PrimeIntellect-ai/prime-envs

Adds a new task to the bench-swe pipeline from a real GitHub bug-fix issue or pull request, then checks the generated task file and patch.

ory/lumen

Builds diverse synthetic test inputs for LLM pipeline evaluation by defining failure-focused dimensions, drafting tuples with you and turning them into realistic queries.

ai-evals-course/evals-skills

Work on the Nexus LLM eval harness in tests/eval/ — author or fix a scenario fixture, write an eval config, change the executors, assertions or reports, or explain a run that produced nothing…

ProfSynapse/nexus

A skill your agent uses when converting an existing benchmark, rubric, verifier, task YAML/JSON, or domain check into SkillEvaluator BYOG/BYOT custom evaluation.

NVIDIA/SkillEvaluator

Measure whether an RT-VLM configuration change altered caption quality — capture paired baseline and candidate captions for a set of videos, score both against a ground truth with an LLM judge, and…

NVIDIA-AI-Blueprints/video-search-and-summarization

Create new skills, modify and improve existing skills, and measure skill performance.

deepklarity/harness-kit

Build DSPy evaluation harnesses with rich-feedback metrics that are essential for GEPA optimization.

intertwine/dspy-agent-skills

Find the decisions in a pipeline that do not need the expensive model and propose the gate for each: a rule, a classic classifier, or a small model, with fail-closed routing.

undefined-ui/second-brain-os

Design, implement, validate, and calibrate a new eval for the convex-evals suite.

get-convex/convex-evals

Analyze a job description (pasted text OR a URL) and find the AI/ML/GenAI topics it requires that this learn-ai course does NOT yet cover.

starkyru/learn-ai

Benchmarks code generation models with the BigCode Evaluation Harness across HumanEval, MBPP, MultiPL-E and other suites using pass@k metrics.

Orchestra-Research/AI-Research-SKILLs

Evaluates agent skills with a Go CLI that runs YAML-defined benchmarks, compares runs and scores the quality of SKILL.md frontmatter.

microsoft/waza

Checks an LLM judge against human labels using train, dev and test splits, TPR and TNR, and a bias correction applied to production data.

ai-evals-course/evals-skills

Ship and spec AI features, LLM products, agents, copilots, and generative UX — including when to use a model vs.

andreaskelm/pm-brain

Generate synthetic evaluation datasets for the PXI eval harness (evals/pxi/).

Arize-ai/phoenix

Benchmarks AMD's GAIA agent against Claude Code and across models on quality, honesty, steps, tokens, time and real cost, using gaia eval tasks.

amd/gaia

Run a quality benchmark of the /translate skill by selecting stratified test keys, capturing ground truth, translating, judging with sub-agents, and compiling a regression report.

shapeshift/web

Scaffolds evaluation suites for the Axiom AI SDK: eval files, scorers, flag schemas and axiom.config.ts, generated from plain descriptions of an AI capability.

openclaw/clawhub

Test and evaluation harness for AI agents — scenario suites, deterministic replay, regression diffing, cost and latency budgets.

borghei/Claude-Skills

Turn a vague task into a testable definition of done and generate an executable goal-test script for it, optionally with a bounded retry loop around a headless agent.

undefined-ui/second-brain-os

Run agent benchmarks, create tasks, analyze results, and manage agents using BenchFlow.

benchflow-ai/benchflow

Benchmark Claude Code skill performance with variance analysis, tracking pass rate, execution time, and token usage across iterations.

AgriciDaniel/skill-forge

Run a skill's evals and report results. An agent skill from redhat-cop/vault-config-operator.

redhat-cop/vault-config-operator

Add a new model to the convex-evals coding leaderboard, and optionally the decision benchmark, through a PR, then dispatch its baseline runs.

get-convex/convex-evals

Evaluate any output file against a structured evals.yaml assertions file and produce a score report with per-assertion pass/fail results.

digipulse-engineering/GAAI-framework

Create new skills, improve existing skills, and measure skill performance.

SpectrAI-Initiative/InnoClaw

Creates and runs LLM evaluation suites from the omniroute CLI, follows live runs, shows scorecards, compares models and ties eval runs into CI.

diegosouzapw/OmniRoute

Walks you through creating, running and reading waza evals for an agent skill, then proposes concrete fixes when tasks fail or the score is low.

microsoft/waza

Designs a binary Pass/Fail LLM-as-Judge prompt for one subjective failure mode, built from a task statement, clear definitions, labeled examples and a structured output format.

ai-evals-course/evals-skills

Reference for building on Microsoft Foundry with the azure-ai-projects Python SDK: project clients, versioned agents, evaluations, connections, datasets and indexes.

microsoft/skills

Create, evaluate, improve, and benchmark content skills using the local Skill Lab workflow.

ceilf6/FrontAgent

Inspect and debug live streaming agent sessions to understand what the agent did.

agentevals-dev/agentevals

测试 use-persona 的角色扮演一致性。给定 persona + 10 个对话场景，生成回复并按 5 个维度评分，输出一致性报告。

YIKUAIBANZI/forge-skill

Given a list of models (HF name stubs) that have valid agentic ID eval scores in Supabase, build a ranking table: raw per-benchmark accuracy on the 3 ID benchmarks (SWE-Bench-100…

open-thoughts/OpenThoughts-Agent

Checks training code, configs and math against documented framework behavior before an expensive run, citing a knowledge base or official docs for every claim.

Leeroo-AI/superml

為 Twinkle Eval 新增一個評測 benchmark（IFEval、BFCL、RAGAS、Text2SQL、Vision MCQ 之類）。涵蓋 CLAUDE.md §6 的完整強制流程：先建 Milestone 與 6 個 Issue、準備 example dataset、實作 Extractor + Scorer 並註冊 PRESETS、與參考框架做分數與速度對比、撰寫…

ai-twinkle/Eval

Step-by-step guide for adding a new built-in test evaluator provider to Agent Kernel (beyond DeepEval, Opik and JEV).

yaalalabs/agent-kernel

Author and safely run deterministic email-agent evaluation suites with dedicated e2a test agents.

tokencanopy/e2a

Finds top models for a task from official Hugging Face benchmark leaderboards, filters them to what fits your hardware, and returns a comparison table with scores.

huggingface/skills

Build an LLM evaluation and run it against the app, returning an Opik experiment with scores and its link.

comet-ml/opik-mcp

This skill should be used when the user wants to "run an evaluation", "evaluate my ADK agent", "write an eval dataset", "analyze eval failures", "compare eval results", "optimize agent", or needs…

pifferologo/cloud-agents-cli

Evaluates LLMs across 100+ benchmarks from 18+ harnesses (MMLU, HumanEval, GSM8K, safety, VLM) with multi-backend execution.

Orchestra-Research/AI-Research-SKILLs