LLM evaluation

Create a new built-in classification evaluator for Phoenix evals.

Arize-ai/phoenix

Write, extend, and debug PXI Playwright E2E tests for Phoenix.

Arize-ai/phoenix

Bump the next release-please version for a Phoenix Python package (arize-phoenix, arize-phoenix-client, arize-phoenix-evals, arize-phoenix-otel) by opening a PR with a Release-As commit footer.

Arize-ai/phoenix

Audit recent changes to Phoenix's user-facing surfaces (Python clients, TypeScript clients, CLI, REST/GraphQL APIs) and patch the three external-facing agent skills — phoenix-tracing, phoenix-cli…

Arize-ai/phoenix

Maintain the bundled TypeScript package docs that ship inside Phoenix npm packages.

Arize-ai/phoenix

Scaffolds eval.yaml evaluation specs for skills, custom agents, and redistributable gh-aw workflow packages in the dotnet/skills repository.

dotnet/skills

Browse installed bkit skills, agents, and evals via lib/discovery/explorer.js (filesystem scan, no subprocess).

ww-w-ai/bkit-claude-code

Compares a vision-language model's yes/no predictions with ground truth and writes the false-positive and false-negative cases to a JSONL file with a summary report.

NVIDIA/skills

Patterns and techniques for evaluating and improving AI agent outputs.

github/awesome-copilot

Monitor arXiv for new papers relevant to the GLIDE project (prediction-powered inference, active statistical inference, LLM evaluation debiasing, proxy annotation bias correction).

EmertonData/glide

Creates or changes a druxt.js contributor skill in .agents/skills, with its evals and the tests that gate it.

druxt/druxt.js

A skill your agent uses when draft eval criteria need refining, clustering, and acceptance into the published goldset with an isolated holdout split (goldset.md + goldset.json).

tikalk/adlc-team-skills

Improves Agent Skills via validate → rule-based eval cases → plugin-eval → prompt evals → bounded edits with held-out gates.

Arenukvern/mcp_flutter

Find out what is going wrong in LLM or agent traffic by reading sampled Phoenix traces, spans, or sessions, writing free-form notes (open coding), then grouping the notes into a few narrow…

Arize-ai/phoenix

Claude Codeセッションの正式な評価フレームワークで、評価駆動開発（EDD）の原則を実装します. An agent skill from affaan-m/ECC.

affaan-m/ECC

평가 주도 개발(EDD) 원칙을 구현하는 Claude Code 세션용 공식 평가 프레임워크. An agent skill from affaan-m/ECC.

affaan-m/ECC

Patient safety evaluation harness for healthcare application deployments.

affaan-m/ECC

This skill should be used when a specific quality problem (UX, data, architecture, feature) needs systematic diagnosis and iterative fixing toward a defined target.

jacob-dietle/context-os

Entry point for evals. An agent skill from ai-evals-course/evals-skills.

ai-evals-course/evals-skills

A skill your agent uses when writing, editing, or reviewing evalite-scored agent evals in packages/core/compute/assistant-evals/src/evals.

dxos/dxos

Plan standalone — populates the Eval Suite Planning & Logging Template from an Agent Vision or plain-English agent description.

microsoft/eval-guide

A skill your agent uses when a published goldset is ready to become executable graders and configs — Python graders and metrics with auto-run unit tests verifying grader correctness.

tikalk/adlc-team-skills

Take a judge live on production traffic — create an Opik online evaluation rule (LLM-as-judge or Python metric) on a project with sampling, filters, variable mapping, and a cost cap, then confirm…

comet-ml/opik-mcp

eve framework guidance for durable AI agents and agent-powered applications.

vercel/vercel-plugin

Autonomously optimize a Claude Code skill or agent system by running it repeatedly, scoring outputs against evals, mutating owned artifacts (prompt, references, scripts, agent definitions), and…

byungjunjang/jangpm-meta-skills

Scaffold a synthetic gold-standard annotation project for evaluating OpenMed NER and de-identification models — label schema, annotation guidelines, BRAT or Label Studio config, and disjoint…

maziyarpanahi/openmed

Evaluate an OpenMed de-identification or clinical NER model against the leakage-first release gates G1a through G8, which gate releases on residual PHI leakage rather than on F1.

maziyarpanahi/openmed

Build custom LLM evaluation pipelines using the OpenJudge framework.

agentscope-ai/OpenJudge

do e2e tests, run e2e, validate feature, prove it works, PR proof, frame proof, pnpm evals.

Devin-AXIS/iPolloWork

Debug LLM applications using the Phoenix CLI. An agent skill from github/awesome-copilot.

github/awesome-copilot

Run evaluations for Hugging Face Hub models using inspect-ai and lighteval on local hardware.

sickn33/agentic-awesome-skills

Curated upstream guidance for Huggingface Community Evals; use when the workflow matches the user goal.

sickn33/agentic-awesome-skills

UiPath API Workflow assistant — author, run, validate, package, publish, deploy, and troubleshoot JSON workflows for uip api-workflow.

UiPath/skills

Andrej Karpathy 视角顾问。以 Karpathy 的心智模型和工程师式判断，为 AI 产品经理场景做分析。

SpaceZephyr/career.skill

Eval-driven development (EDD) ilkelerini uygulayan Claude Code oturumları için formal değerlendirme çerçevesi

affaan-m/ECC

Monitor and evaluate RAG systems with retrieval quality metrics, groundedness checks, hallucination detection, and continuous regression testing.

sickn33/agentic-awesome-skills

Evaluate retrieval and citation behavior for RAG pipelines from deterministic JSONL fixtures.

davepoon/buildwithclaude

Understand what a Phoenix dataset is and reason well about its examples, outputs, splits, and how it feeds evaluators and experiments.

Arize-ai/phoenix

A skill your agent uses when standing up evals/{system}/ for the first time — scaffolds the EDD directory structure, picks PromptFoo or DeepEval by tech stack, and generates a security baseline.

tikalk/adlc-team-skills

Guardrailed DELETE of auto-registered eval sandboxjobs rows that DID score but FAILED the harvest gate — partial evals (valid-complete <90% or non-benign infra-error 10%).

open-thoughts/OpenThoughts-Agent

克劳德代码会话的正式评估框架，实施评估驱动开发（EDD）原则

affaan-m/ECC

Benchmark video Q&A accuracy and latency of a deployed RT-VLM (Cosmos Reason 3) via vss vlm run, using questions and videos from the DSS vss-devx-base dataset.

NVIDIA-AI-Blueprints/video-search-and-summarization

Validates Terminal User Interface (TUI) output using freeze for screenshot capture and LLM-as-judge for semantic validation.

mikeyobrien/ralph-orchestrator

Handles LLM-as-judge evaluation workflows on Arize including creating/updating evaluators, running evaluations on spans or experiments, managing tasks, trigger-run operations, column mapping, and…

github/awesome-copilot

Run metric-driven iterative optimization loops. An agent skill from leo-kuang-ai/spec-first.

leo-kuang-ai/spec-first

AI agent and LLM system engineering reference covering single-agent dev (ReAct, tool calling, plan-execute), multi-agent coordination (swarm, role decomposition, file locking), LLM security (prompt…

telagod/code-abyss

Build and run evaluators for AI/LLM applications using Phoenix.

github/awesome-copilot

LLM Operations -- RAG, embeddings, vector databases, fine-tuning, prompt engineering avancado, custos de LLM, evals de qualidade e arquiteturas de IA para producao.

davila7/claude-code-templates