Best LLM evaluation skills, page 2

Build DSPy 3.2.x programs through spec, program, metric and baseline; extend to optimization and export when requested and justified by task budget.

intertwine/dspy-agent-skills

Prepare the environment and run the LLM-driven agent evals (e2e/agent-evals/) against a chosen sim-use binary.

lycorp-jp/sim-use

Audit an agent's context layout against the four places: system prompt, tools, history, tail.

undefined-ui/second-brain-os

Evaluate and score agent behavior against a golden reference.

agentevals-dev/agentevals

End-to-end GECX/CXAS/CES conversational agent lifecycle -- build agents from requirements (PRD-to-agent), create and run evals (goldens, simulations, tool tests, callback tests), debug failures, and…

GoogleCloudPlatform/cxas-scrapi

A skill your agent uses for Codex models/pricing, scheduled tasks, skills, settings, setup, troubleshooting, customization, automations, and self-knowledge—including 'you,' 'your,' 'this app,' or…

theowenyoung/home

Propose an improved version of a prompt registered in a self-hosted AgentX (AgentX-trace-eval) instance, using real low-rated evaluation results as evidence, then publish it as a new version once…

AgentX-ai/AgentX-Trace-Eval

Runs the `autoctx` CLI to improve an approach to a task over several generations, score or refine a single output and inspect what a run produced.

greyhaven-ai/autocontext

Create, refine, and benchmark agent skills. An agent skill from feiskyer/claude-code-settings.

feiskyer/claude-code-settings

Designs, tests and refines LLM prompts: zero-shot, few-shot and chain-of-thought patterns, system prompts, structured output schemas and evaluation test suites.

Jeffallan/claude-skills

Convert an evaluation dataset or leaderboard into the Every Eval Ever (EEE) schema — aggregate .json logs (eval.schema.json) and optional instance samples.jsonl sidecars…

evaleval/every_eval_ever

Install and run a verifiers environment — smoke testing during development and full benchmark evals.

PrimeIntellect-ai/prime-envs

Adds a new task to the bench-swe pipeline from a real GitHub bug-fix issue or pull request, then checks the generated task file and patch.

ory/lumen

Builds diverse synthetic test inputs for LLM pipeline evaluation by defining failure-focused dimensions, drafting tuples with you and turning them into realistic queries.

ai-evals-course/evals-skills

Work on the Nexus LLM eval harness in tests/eval/ — author or fix a scenario fixture, write an eval config, change the executors, assertions or reports, or explain a run that produced nothing…

ProfSynapse/nexus

Create new skills, modify and improve existing skills, and measure skill performance.

deepklarity/harness-kit

Benchmarks code generation models with the BigCode Evaluation Harness across HumanEval, MBPP, MultiPL-E and other suites using pass@k metrics.

Orchestra-Research/AI-Research-SKILLs

Build DSPy evaluation harnesses with rich-feedback metrics that are essential for GEPA optimization.

intertwine/dspy-agent-skills

Scaffold a first eval suite for an agent: mine real failures into cases, write behavioural checks over traces, and generate the runner.

undefined-ui/second-brain-os

Write and review content for the OpenSEO website (web/) — blog posts, guides, feature pages, FAQs.

Jwuthri/Tracely-ai

Design, implement, validate, and calibrate a new eval for the convex-evals suite.

get-convex/convex-evals

Analyze a job description (pasted text OR a URL) and find the AI/ML/GenAI topics it requires that this learn-ai course does NOT yet cover.

starkyru/learn-ai

Designs retrieval-augmented generation systems: document chunking, embeddings, vector store setup, hybrid search, reranking and retrieval evaluation, with checks at each step.

Jeffallan/claude-skills

Evaluates agent skills with a Go CLI that runs YAML-defined benchmarks, compares runs and scores the quality of SKILL.md frontmatter.

microsoft/waza

Checks an LLM judge against human labels using train, dev and test splits, TPR and TNR, and a bias correction applied to production data.

ai-evals-course/evals-skills

Benchmarks AMD's GAIA agent against Claude Code and across models on quality, honesty, steps, tokens, time and real cost, using gaia eval tasks.

amd/gaia

Generate synthetic evaluation datasets for the PXI eval harness (evals/pxi/).

Arize-ai/phoenix

Run a quality benchmark of the /translate skill by selecting stratified test keys, capturing ground truth, translating, judging with sub-agents, and compiling a regression report.

shapeshift/web

Scaffolds evaluation suites for the Axiom AI SDK: eval files, scorers, flag schemas and axiom.config.ts, generated from plain descriptions of an AI capability.

openclaw/clawhub

Find the decisions in a pipeline that do not need the expensive model and propose the gate for each: a rule, a classic classifier, or a small model, with fail-closed routing.

undefined-ui/second-brain-os

Run agent benchmarks, create tasks, analyze results, and manage agents using BenchFlow.

benchflow-ai/benchflow

Evaluates LLMs across 100+ benchmarks from 18+ harnesses (MMLU, HumanEval, GSM8K, safety, VLM) with multi-backend execution.

Orchestra-Research/AI-Research-SKILLs

Benchmark Claude Code skill performance with variance analysis, tracking pass rate, execution time, and token usage across iterations.

AgriciDaniel/skill-forge

Run a skill's evals and report results. An agent skill from redhat-cop/vault-config-operator.

redhat-cop/vault-config-operator

Add a new model to the convex-evals coding leaderboard, and optionally the decision benchmark, through a PR, then dispatch its baseline runs.

get-convex/convex-evals

Evaluate any output file against a structured evals.yaml assertions file and produce a score report with per-assertion pass/fail results.

digipulse-engineering/GAAI-framework

Create new skills, improve existing skills, and measure skill performance.

SpectrAI-Initiative/InnoClaw

Walks you through creating, running and reading waza evals for an agent skill, then proposes concrete fixes when tasks fail or the score is low.

microsoft/waza

Designs a binary Pass/Fail LLM-as-Judge prompt for one subjective failure mode, built from a task statement, clear definitions, labeled examples and a structured output format.

ai-evals-course/evals-skills

Create, evaluate, improve, and benchmark content skills using the local Skill Lab workflow.

ceilf6/FrontAgent

Inspect and debug live streaming agent sessions to understand what the agent did.

agentevals-dev/agentevals

测试 use-persona 的角色扮演一致性。给定 persona + 10 个对话场景，生成回复并按 5 个维度评分，输出一致性报告。

YIKUAIBANZI/forge-skill

Build custom LLM evaluation pipelines using the OpenJudge framework.

agentscope-ai/OpenJudge

Given a list of models (HF name stubs) that have valid agentic ID eval scores in Supabase, build a ranking table: raw per-benchmark accuracy on the 3 ID benchmarks (SWE-Bench-100…

open-thoughts/OpenThoughts-Agent

Checks training code, configs and math against documented framework behavior before an expensive run, citing a knowledge base or official docs for every claim.

Leeroo-AI/superml

為 Twinkle Eval 新增一個評測 benchmark（IFEval、BFCL、RAGAS、Text2SQL、Vision MCQ 之類）。涵蓋 CLAUDE.md §6 的完整強制流程：先建 Milestone 與 6 個 Issue、準備 example dataset、實作 Extractor + Scorer 並註冊 PRESETS、與參考框架做分數與速度對比、撰寫…

ai-twinkle/Eval

Step-by-step guide for adding a new built-in test evaluator provider to Agent Kernel (beyond DeepEval, Opik and JEV).

yaalalabs/agent-kernel

Author and safely run deterministic email-agent evaluation suites with dedicated e2a test agents.

tokencanopy/e2a