LLM evaluation

Create new skills, modify and improve existing skills, and measure skill performance.

Azure/azqr

Reads the state and results of Caveman Cloud experiments and reports one recommendation or a block, without changing an experiment's lifecycle itself.

JuliusBrussee/caveman

Turns a Caveman report-only optimization observation into one minimal code change and a paired baseline evaluation, after the operator picks which to pursue.

JuliusBrussee/caveman

Provides context about the CoStrict evals system structure in this monorepo.

zgsm-ai/costrict

Runs lm-evaluation-harness to benchmark language models on academic suites such as MMLU, GSM8K and HumanEval, compare models and track training checkpoints.

Orchestra-Research/AI-Research-SKILLs

Create, improve, or evaluate an existing skill from workflows, prompts, SOPs, scripts.

yaojingang/yao-meta-skill

Reference for building on Microsoft Foundry with the azure-ai-projects Python SDK: project clients, versioned agents, evaluations, connections, datasets and indexes.

microsoft/skills

Runs and debugs evaluations of how Chatbox models answer questions about large attached files, using synthetic and real long-document fixtures.

chatboxai/chatbox

Helps instrument a custom Python or TypeScript agent to record events for Failproof AI, verify what gets written, and run an evaluator worker that scores the runs.

FailproofAI/failproofai

A skill your agent uses when creating, improving, finding, or auditing agent skills - the user says 'create a skill', 'do I have a skill for X', 'improve the X skill', 'which skill should I use'…

tripleyak/SkillForge

Guides LLM fine-tuning with LoRA and QLoRA through Hugging Face PEFT, from dataset validation and training checks to adapter merging, quantization and deployment.

Jeffallan/claude-skills

A skill your agent uses when designing, generating an MVP blueprint for, auditing, troubleshooting, refactoring, or explaining an agentic harness for any domain.

DenisSergeevitch/agents-best-practices

Scaffold a well-designed agent loop with best-practice coaching and a cross-model review council.

ksimback/looper

Runs evaluations of Hugging Face Hub models on local hardware with inspect-ai or lighteval, and helps choose between vLLM, Transformers and accelerate backends.

huggingface/skills

Research, create, improve, migrate, evaluate, package, install-check, govern, and safely publish qiaomu-flavored agent skills from workflows, prompts, transcripts, docs, SOPs, runbooks, scripts, or…

joeseesun/qiaomu-meta-skill

Writes and runs black-box benchmark cases for Windmill's flow, app, script, CLI and global AI generation modes, including before-and-after comparisons.

windmill-labs/windmill

Builds agent evaluations in stages: inspect the repository and traces, agree a Task Spec with you, then build, audit and run a Harbor task with an independent verifier.

langchain-ai/langchain-skills

Formal evaluation framework for Claude Code sessions implementing eval-driven development (EDD) principles.

cloudnative-co/claude-code-starter-kit

Evaluate and improve GenAI models and agents using the Google GenAI Evaluation SDK.

GoogleCloudPlatform/vertex-ai-samples

Turn a vague task into a testable definition of done and generate an executable goal-test script for it, optionally with a bounded retry loop around a headless agent.

undefined-ui/second-brain-os

Runs blind pairwise comparisons of Octocode against a gh-based baseline over markdown research questions, scored by total characters through the model rather than self-report.

bgauryy/octocode

Measure whether a change makes Chat's styling replies better, with yarn eval:chat — it runs styling requests on recorded real pages for two versions of Stylebot and measures on the page whether each…

ankit/stylebot

亚马逊卖家专用的 skill 创建器（中文）。当用户想把一个亚马逊运营/自媒体/日常工作流程变成可复用的 skill 时使用。触发场景包括但不限于：用户说"我想做一个 skill""把这个流程变成 skill""帮我写个自动化""优化我已有的 skill""给这个工作流做个自动化"，即使用户没用"skill"这个词，只要在描述"以后每次都这样做"的重复性工作时也应触发。本 skill…

zach22-1999/amazon-skills

Create new skills, modify and improve existing skills, and measure skill performance.

AgentTeam-TaichuAI/ScienceClaw

Write LLM evaluation spec files with datasets, tasks, and evaluators using the @kbn/evals Playwright fixture.

elastic/kibana

SkillsBench task authoring — walk a contributor from idea to submission-ready task following CONTRIBUTING.md and the task-implementation rubric.

benchflow-ai/benchflow

Builds a browser-based annotation page for reviewing LLM traces one at a time with pass/fail labels, notes and saved results, tailored to your data.

ai-evals-course/evals-skills

Create a skill or bring an existing one up to the same standard (validate + asm eval fix loop); run evals, tune triggering.

luongnv89/asm

Create, edit, evaluate, and package agent skills. An agent skill from smixs/skill-conductor.

smixs/skill-conductor

A skill your agent uses to turn an AI idea or existing repository into a credible open-source product and to run evidence-first repository engineering across codebase discovery, context-efficient…

sun461941-hub/ai-project-copilot

Authoring and running behavioral evals for a shopping or merchant agent, covering the case shape, authoring rules, code graders and judges, the run pattern, and poisoned fixtures.

anthropics/commerce-agents

Evaluate verified findings from merge-ready, Greptile, pull-request, CI, security, billing, and other code reviews, then promote durable review gaps into the version-controlled .greptile…

Jwuthri/Tracely-ai

Benchmark a third-party coding-agent harness against FrontierHarness Eval using Runta runtimes.

frontier-harness-eval/eval

Runs LLM-based rubric judging on agent output and loops revise-and-rejudge rounds until a quality threshold is met.

greyhaven-ai/autocontext

A skill your agent uses when converting an existing benchmark, rubric, verifier, task YAML/JSON, or domain check into SkillEvaluator BYOG/BYOT custom evaluation.

NVIDIA/SkillEvaluator

Sends one prompt to Claude, GPT through the Codex CLI and Gemini, then tabulates response time, token use and cost, with an optional judged quality score.

garrytan/gstack

Inspects an LLM evaluation setup for missing error analysis, unvalidated judges and vanity metrics, and ranks the problems by impact with fixes.

ai-evals-course/evals-skills

Evaluates a repository's agent harness (AGENTS.md, rules, skills) for broken paths, redundant instructions and usefulness, and stops at reports.

tech-leads-club/agent-skills

A skill your agent uses when a user wants to create, run, or analyze evaluation suites for Microsoft 365 Copilot declarative agents with the public @microsoft/m365-copilot-eval CLI.

microsoft/work-iq

Turn the current conversation's workflow into a reusable agent skill.

Undertone0809/rudder

Plan a documentation eval for supabase/evals, where a docs guide is the subject under test.

supabase/evals

Guides evaluation of a RAG system by diagnosing failures in traces, building a retrieval test set and scoring retrieval and generation separately.

ai-evals-course/evals-skills

Converts test suites from external eval frameworks into the Margin Eval suite format.

Margin-Lab/evals

Measure whether an RT-VLM configuration change altered caption quality — capture paired baseline and candidate captions for a set of videos, score both against a ground truth with an LLM judge, and…

NVIDIA-AI-Blueprints/video-search-and-summarization

Keeps the engagement record for client work. An agent skill from suboss87/FDEOps.

suboss87/FDEOps

Review a proposed Agent Skill for structural validity and content quality before publishing.

mongodb/agent-skills

Review Benchflow or SkillsBench task-run trajectories and integration-test Benchflow code changes.

benchflow-ai/benchflow

Sets up and runs side-by-side model evaluations with frozen questions, isolated tools and traceable results, using your own question bank or benchmark subsets.

liucongg/liucong-skills