Project Evals

Stage 3: Evaluating guidance for a use case (Needs evals)

This is the third of three stages in creating guidance:

1. Stage 1: Identifying use cases for a feature
2. Stage 2: Authoring guidance for a use case
3. Stage 3: Evaluating guidance for a use case (you are here)

What the eval agent sees vs real-world agents

Real-world coding agents see only guide.md — retrieved automatically via the RAG skills system when a developer asks for help. Every other file in a use case directory is eval infrastructure.

The eval harness runs a separate coding agent in a controlled environment to test whether the guidance works. This eval agent receives the first prompt from tasks/task.md and has access to guide.md via the same RAG system. The harness then runs grader.ts against the eval agent's output.

None of the following are ever seen by real-world coding agents:

How the eval files work together

tasks/task.md, expectations.md, and grader.ts form a tightly coupled pipeline:

1. tasks/task.md — Simulated developer prompts used only by the eval harness. It must start with a YAML frontmatter specifying the base_app, followed by a list of prompts. Each prompt should sound like a real developer request, without naming specific APIs or best practices — the eval agent is expected to discover those by reading guide.md via RAG. The first prompt is the most important: it is used as the default task.

2. expectations.md — The ground truth for what a correct implementation looks like. Each bullet becomes exactly one test in grader.ts. Write expectations assuming the eval agent read guide.md and implemented it faithfully; they describe the observable output, not the implementation approach.

3. grader.ts — A Playwright test file generated 1:1 from expectations.md. Every bullet maps to one test() block. If an expectation cannot be translated into a Playwright assertion (static file check or browser automation), it does not belong in expectations.md.

Writing expectations.md

Write a natural language, bulleted list of assertions that must be true if an agent implements the guide.md correctly (e.g., "The input element is styled with a red border only AFTER a blur event").

• 1:1 with grader tests — Each bullet becomes exactly one test. Write one bullet per assertion. Do not combine multiple checks into a single bullet.
• Plain declarative phrasing — Write each bullet as a plain statement of what is true of a correct implementation (e.g., "The dialog closes when the Escape key is pressed"). Imperative directives such as MUST, MUST NOT, DO, and DO NOT are a guide.md convention for steering coding agents; expectations.md is consumed only by the internal grader generator, so the The implementation MUST… boilerplate adds nothing and should be omitted.
• Concrete, Testable Criteria (No API Facts) — Expectations must be verifiable browser behaviors we can check with Playwright (e.g., computed styles, DOM layout), not just factual statements about an API or code structure.
• Exercised in Demo: Ensure that every expectation written here is actively exercised in the accompanying demo.html. Expectations that aren't covered by the demo lead to unreliable grader calibration.
• Scoped to this use case — Only include expectations that apply to the specific use case being graded. Do not copy generic expectations from other guides if they describe behavior that won't appear in an implementation of this guide (e.g., don't include URL input expectations in a sign-in form grader).
• No external links — The grader generator cannot resolve them.
• Avoid over-constraining — Don't assert implementation details that don't affect correctness (e.g., don't require a direct child relationship if a descendant also works).

Grading Note

• Graders (grader.ts) live within their respective guide folders. These are Playwright test files.
• AVOID using static assertions (like regex or str.includes() on fs.readFileSync) to test CSS or HTML syntax whenever possible. These are extremely brittle and will fail if the agent uses a different class name, semantic element, or formatting.
• Instead, PREFER using Playwright's browser APIs to test computed styles and actual DOM layout. Use element.evaluate((el) => window.getComputedStyle(el).propertyName) to robustly verify that the browser is rendering the feature correctly, regardless of how the agent authored the code.
• A human may manually edit the .ts file if the generator struggles to get it perfectly tailored.

Once a guide has its guide.md, demo.html, and expectations.md completely written, it is ready for the evaluation pipeline.