Agent evaluation and testing

Guides the design and implementation of Model Context Protocol servers in TypeScript or Python, from tool naming and error messages to evaluation.

anthropics/skills

Investigates a session where Superpowers went wrong, reads the transcripts on disk and produces an evidence-cited report, optionally prepared as a bug report for the maintainers.

obra/superpowers

Scores SKILL.md files on a nine-dimension rubric, then improves them in a keep-or-revert loop with independent judge agents, test prompts, git history and human checkpoints.

alchaincyf/darwin-skill

Validates OpenHarness features by running real multi-turn agent loops with live LLM calls against an unfamiliar codebase, checking actual tool execution.

HKUDS/OpenHarness

Evaluates an Agent Skill bundle before release for structure, trigger quality, artifact improvement, script correctness, safety, installed-tree integrity and host portability.

rohitg00/ai-engineering-from-scratch

Benchmarks how much CodeGraph helps a coding agent on a real repository, comparing runs with and without it for a chosen local or published version.

colbymchenry/codegraph

Mines local Copilot CLI session logs for dotnet/maui to rank costly or failing runs, tag recurring failure modes, propose repo edits and emit guard evals.

dotnet/maui

Creates, revises, evaluates and publishes skills in the Open-Science app through its native host.skills composer, with optional test prompts and benchmarks.

aipoch/open-science

Evaluates the design quality of an agent skill against official specifications and patterns from existing examples, scoring it and suggesting improvements.

shareAI-lab/Kode-CLI

Scores a skill across description, content organization, writing style and structure, then produces letter grades and a prioritized improvement plan.

Galaxy-Dawn/claude-scholar

Testing framework for evaluating Databricks skills. An agent skill from databricks-solutions/ai-dev-kit.

databricks-solutions/ai-dev-kit

Diagnoses where an agent failed across runs and turns the findings into new skills, system prompt patches and knowledge entries, using the A-Evolve loop.

aiming-lab/AutoResearchClaw

Evaluate skill quality, find the weakest dimension, and apply directed improvements.

Evol-ai/SkillCompass

Call any REST API dynamically. An agent skill from NVIDIA/SkillEvaluator.

NVIDIA/SkillEvaluator

Writes and runs black-box benchmark cases for Windmill's flow, app, script, CLI and global AI generation modes, including before-and-after comparisons.

windmill-labs/windmill

Builds agent evaluations in stages: inspect the repository and traces, agree a Task Spec with you, then build, audit and run a Harbor task with an independent verifier.

langchain-ai/langchain-skills

Lets a Hermes agent run Autocontext scenarios, inspect Hermes curator state, export reusable knowledge and prepare local MLX or CUDA training data through the autoctx CLI.

greyhaven-ai/autocontext

Designs and verifies a deterministic grader that measures whether a GitHub Agentic Workflow run reached its real-world or repository outcome.

github/gh-aw

End-to-end agent evaluation and improvement pipeline. An agent skill from kayba-ai/agentic-context-engine.

kayba-ai/agentic-context-engine

Coordinate multi-agent development pipelines using shared JSON task files.

win4r/team-tasks

Runs blind pairwise comparisons of Octocode against a gh-based baseline over markdown research questions, scored by total characters through the model rather than self-report.

bgauryy/octocode

Checks changes to the senpi coding agent by driving the real CLI from source in an isolated sandbox, over RPC, terminal UI, mock model and CLI smoke channels.

code-yeongyu/senpi

驱动 One-Eval 对 API 或本地模型做端到端评测，覆盖纯文本、多模态、代码生成、函数调用和 Agent benchmark。当用户想评测模型在一个或多个 benchmark 上的表现、比较分数、补充 metric，或生成图文评测报告时使用本 skill。

OpenDCAI/One-Eval

Runs LLM-based rubric judging on agent output and loops revise-and-rejudge rounds until a quality threshold is met.

greyhaven-ai/autocontext

Runs existing Harbor evaluation jobs against a local LobeHub build or LobeHub Cloud, with preflight checks, resume support, and failure triage.

lobehub/lobehub

Operates or analyzes a LoopX-managed benchmark experiment: launching runs, maintaining the experiment board, qualifying integrity, and writing case insights.

loopx-project/loopx

AWS Bedrock AgentCore comprehensive expert for deploying and managing AI agents at scale.

zxkane/aws-skills

Evaluates a repository's agent harness (AGENTS.md, rules, skills) for broken paths, redundant instructions and usefulness, and stops at reports.

tech-leads-club/agent-skills

Runs a reference agent (Codex or Claude Code) and Qwen Code on the same scenario, captures HTTP and terminal traces, and compares them until behavior matches.

QwenLM/qwen-code

Required for 4+ step requests; add tasks at start and update status after each step.

NVIDIA/SkillEvaluator

A skill your agent uses when a user wants to create, run, or analyze evaluation suites for Microsoft 365 Copilot declarative agents with the public @microsoft/m365-copilot-eval CLI.

microsoft/work-iq

Measures whether agents actually follow a skill, rule or agent definition by generating scenarios at three strictness levels and scoring tool-call traces.

affaan-m/ECC

Walks you through drafting, testing, evaluating and tuning a skill for OpenCode, from an intake interview to description optimization.

antongulin/opencode-skill-creator

Agent Script DSL for deterministic Agentforce agents. An agent skill from Jaganpro/sf-skills.

Jaganpro/sf-skills

Runs and interprets a skill's Caliper eval: how often it succeeds over repeated attempts, whether it triggers at all, and whether it beats the agent without it.

edonadei/caliper

Ultimate Claude Code skill creator and architect. An agent skill from AgriciDaniel/skill-forge.

AgriciDaniel/skill-forge

Scores an agent's finished work with a three-stage pipeline: free mechanical checks, an advisory semantic review, and an optional multi-model consensus vote.

Q00/ouroboros

Adds a new task to the bench-swe pipeline from a real GitHub bug-fix issue or pull request, then checks the generated task file and patch.

ory/lumen

Build, deploy, evaluate, optimize, fine-tune, and manage Microsoft Foundry agents, models, and resources end to end.

microsoft/GitHub-Copilot-for-Azure

A skill your agent uses when converting an existing benchmark, rubric, verifier, task YAML/JSON, or domain check into SkillEvaluator BYOG/BYOT custom evaluation.

NVIDIA/SkillEvaluator

Run the weak-agent adversarial test harness against docx-cli.

kklimuk/docx-cli

Mines local Claude Code session transcripts with a deterministic Python pipeline to show what the agent is actually used for, how often it fails and what it costs.

amd/gaia

Runs caliper's smoke evals against the real agent CLIs after a harness or MCP change, with a dry-run plan, failure triage and a report to attach to the PR.

edonadei/caliper

Contrasts successful and failed agent runs of the same task and derives guidelines backed by evidence from transcripts, tool calls and outcome judgments.

AgentToolkit/altk-evolve

Evaluates agent skills with a Go CLI that runs YAML-defined benchmarks, compares runs and scores the quality of SKILL.md frontmatter.

microsoft/waza

Evaluate Expo skills in this repo end-to-end - trigger accuracy, generated code quality, and runtime screenshots on iOS simulator and Android emulator via Expo Go (web optional).

expo/skills

Score one analytical answer against a verified golden, and score which of the entities the golden depends on retrieval delivered to the answerer.

malloydata/publisher

Designs, evaluates and improves the harness around an AI agent: tool permissions, approval gates, state, memory, evals and observability, with phased plans.

NateBJones-Projects/OB1