Python · LLM evaluation

Runs lm-evaluation-harness to benchmark language models on academic suites such as MMLU, GSM8K and HumanEval, compare models and track training checkpoints.

Orchestra-Research/AI-Research-SKILLs

Helps instrument a custom Python or TypeScript agent to record events for Failproof AI, verify what gets written, and run an evaluator worker that scores the runs.

FailproofAI/failproofai

Create a skill or bring an existing one up to the same standard (validate + asm eval fix loop); run evals, tune triggering.

luongnv89/asm

Checks an LLM judge against human labels using train, dev and test splits, TPR and TNR, and a bias correction applied to production data.

ai-evals-course/evals-skills

Reference for building on Microsoft Foundry with the azure-ai-projects Python SDK: project clients, versioned agents, evaluations, connections, datasets and indexes.

microsoft/skills

Add a new evaluator to the amp-evaluation Python library. An agent skill from wso2/agent-manager.

wso2/agent-manager

Score a Python generative-AI app's outputs on groundedness, relevance, coherence, and safety before it ships.

timothywarner-org/claude-code

Run offline Python (pydanticevals) or Node.js (logfire/evals) evaluations and review them in Logfire.

pydantic/skills

Create a new built-in classification evaluator for Phoenix evals.

Arize-ai/phoenix

Bump the next release-please version for a Phoenix Python package (arize-phoenix, arize-phoenix-client, arize-phoenix-evals, arize-phoenix-otel) by opening a PR with a Release-As commit footer.

Arize-ai/phoenix

Audit recent changes to Phoenix's user-facing surfaces (Python clients, TypeScript clients, CLI, REST/GraphQL APIs) and patch the three external-facing agent skills — phoenix-tracing, phoenix-cli…

Arize-ai/phoenix

A skill your agent uses when a published goldset is ready to become executable graders and configs — Python graders and metrics with auto-run unit tests verifying grader correctness.

tikalk/adlc-team-skills

Take a judge live on production traffic — create an Opik online evaluation rule (LLM-as-judge or Python metric) on a project with sampling, filters, variable mapping, and a cost cap, then confirm…

comet-ml/opik-mcp

Build and run evaluators for AI/LLM applications using Phoenix.

github/awesome-copilot

Generates python code that evaluates SageMaker models. An agent skill from awslabs/agent-plugins.

awslabs/agent-plugins

lm-eval-harness: benchmark LLMs (MMLU, GSM8K, etc.). An agent skill from Luciole-Studio/Misaka-Agent.

Luciole-Studio/Misaka-Agent

Improve AI application with evaluation-driven development. An agent skill from github/awesome-copilot.

github/awesome-copilot

Configures and runs LLM evaluation using Promptfoo framework.

daymade/claude-code-skills

Eval-driven skill tuning. An agent skill from ClawBio/ClawBio.

ClawBio/ClawBio

Compile an Anthropic-style skill — a directory with a SKILL.md and optional references/ — into a deterministic, runnable workflow via the rote CLI.

ccplugins/awesome-claude-code-plugins

A skill your agent uses when integrating NVIDIA NeMo Fabric into a consumer application, service, evaluation harness, or platform through the typed Python SDK — translating the consumer's own…

NVIDIA/skills

Best practices for building AI agents with Google's Agent Development Kit (ADK) in Python, covering agent design, tools, sessions, memory, artifacts, evaluation, and deployment.

Mindrally/skills

Bootstrap evaluators from production traces — by default propose online LLM-judge evaluators and, after you confirm, create them in Datadog as disabled drafts (never auto-enabled); on request emit…

datadog-labs/agent-skills

Build type-safe AI agents and graph-based workflows with PydanticAI and PydanticGraph.

magnus919/agent-skills