AI Evals

Templates & Frameworks

• LLM-as-a-Judge Playbook (Building eval systems that improve your AI product) - A systematic three-step process for building, validating, and measuring an LLM judge that provides trusted binary pass/fail metrics for subjective AI quality as
• Three Eval Approaches (Human, Code-based, LLM-based) (Beyond vibe checks: A PM’s complete guide to evals) - A decision framework for choosing the right eval approach based on your use case, with pros and cons for each.
• The Eval Formula (Four-Part Structure) (Beyond vibe checks: A PM’s complete guide to evals) - A four-part formula for writing effective LLM-based eval prompts that any PM can use to construct judge-LLM prompts.
• Open Coding and Axial Coding for AI Error Analysis (Building eval systems that improve your AI product) - A qualitative research methodology adapted for AI product evaluation, used to discover and categorize failure modes from user interaction data.
• RAG Evaluation Framework (Retriever + Generator) (Building eval systems that improve your AI product) - A two-part evaluation approach for RAG systems that separately assesses the retriever and generator components with specific metrics for each.
• AI Eval Improvement Flywheel (Building eval systems that improve your AI product) - The closed-loop process that uses CI safety nets and production discovery engines together to create continuous AI product improvement.
• Reference Dataset Structure (Why your AI product needs a different development lifecycle) - A template for building the initial reference dataset (20-100 examples) to break the cold start and provide a baseline for AI system evaluation.
• Transition Failure Matrix for Agentic Workflows (Building eval systems that improve your AI product) - A diagnostic tool for pinpointing exactly which step in an agent's multi-step workflow breaks down, enabling data-driven debugging.

See references/artifacts.md for the full list with details.

Questions to Help Users

• "What are the top 3 to 5 failure modes your users are currently experiencing in production?"
• "Do you have a single domain expert or benevolent dictator who defines what quality looks like for this feature?"
• "What percentage of your current evaluation process is manual versus automated?"
• "Is your system non-determinism primarily occurring in the retrieval or the generation phase?"
• "Have you established a golden dataset of at least 20 to 50 human-labeled examples yet?"
• "How do you currently measure the performance delta when you switch models or change a system prompt?"

Common Mistakes to Flag

• Relying on vibe checks - Manual and anecdotal testing leads to inconsistent quality and hidden regressions that damage user trust over time.
• Obsessing over prompt engineering - Focusing solely on prompts while neglecting the underlying evaluation system prevents teams from scaling or hill-climbing systematically.
• Using generic metrics for product reporting - Off-the-shelf scores are useful for filtering but often fail to capture the specific value or failures unique to your business logic.
• Ignoring component isolation - Failing to evaluate the retriever separately from the generator in RAG systems makes it impossible to know which part of the stack is failing.
• Neglecting non-determinism - Failing to account for the stochastic nature of LLMs leads to false confidence in results that may not repeat in production.

Deep Dive

For all 33 sourced insights from 11 guests, see references/guest-insights.md

Related Skills

• Ai Product Strategy
• Ai Native Ux