By ai-evals-course

Builds a browser-based annotation page for reviewing LLM traces one at a time with pass/fail labels, notes and saved results, tailored to your data.

ai-evals-course/evals-skills

Inspects an LLM evaluation setup for missing error analysis, unvalidated judges and vanity metrics, and ranks the problems by impact with fixes.

ai-evals-course/evals-skills

Guides evaluation of a RAG system by diagnosing failures in traces, building a retrieval test set and scoring retrieval and generation separately.

ai-evals-course/evals-skills

Builds diverse synthetic test inputs for LLM pipeline evaluation by defining failure-focused dimensions, drafting tuples with you and turning them into realistic queries.

ai-evals-course/evals-skills

Checks an LLM judge against human labels using train, dev and test splits, TPR and TNR, and a bias correction applied to production data.

ai-evals-course/evals-skills

Designs a binary Pass/Fail LLM-as-Judge prompt for one subjective failure mode, built from a task statement, clear definitions, labeled examples and a structured output format.

ai-evals-course/evals-skills

Write code evaluators for known failure modes with objective rules.

ai-evals-course/evals-skills

Guides an interactive error analysis of LLM outputs: studies the dataset, builds a review interface, picks diverse samples and organizes the failure modes you annotate.

ai-evals-course/evals-skills

Entry point for evals. An agent skill from ai-evals-course/evals-skills.

ai-evals-course/evals-skills