Docs Eval Planning

Planning a documentation eval

A documentation eval measures a page, not an agent. The prompt sends a user's request plus the page's url, and the checks say whether an agent that read the page produced working code. A gap in the page counts as a failure.

That framing is the whole difficulty. The failure modes are collected in references/flakiness.md. Read it before designing checks, not after.

What this skill produces

A plan. It does not write PROMPT.md, EVAL.ts, or a seed. Implementation and local scoring are a separate job.

The shape it takes

Model a new eval on build-docs-003-api-keys-guide or later.

build-docs-NNN-<subject>, named after the doc rather than the feature. The id is permanent. The published results series is keyed on it, so renaming breaks history. So does renaming a check.

evals/docs/build-docs-NNN-<subject>/
  PROMPT.md      frontmatter and the task the agent sees
  EVAL.ts        the scorer
  README.md      the design rationale, addressed to the next editor
  <helper>.ts    flat beside EVAL.ts, never in a subdirectory
  local/         the seed workspace

CONTRIBUTING.md makes the README.md optional. For a page under test it is required: it is the only place the stripped-word list and the constraints that must not be edited away live. Its sections:

• What this eval measures. The page is the subject, not the agent.
• Do not reintroduce the vocabulary. The stripped words, listed.
• The seed carries the contract. What the seed fixes, and what each fixed thing buys and costs.
• Do not drop the positive controls. Which checks pass for an agent that built nothing, and which ones make them mean something.
• The guide has to actually be read.
• What this eval does not score. Each entry with its reason. An unmeasured rule from the prompt is named here.

EVAL.ts is orchestration. Import named check* functions, assemble one flat array, return { passed: checks.every(c => c.passed), checks }, and wrap it in a try/catch whose catch returns a single self-named failure check. Split the implementations into modules and keep the full check list declared in EVAL.ts. Comment why the phase order is what it is.

Every page-under-test eval carries a check that the referenced page was read with content. Import checkDocsGuideRead from core if it has landed there; otherwise copy the most recent eval's. It resolves the url from the harness's docs result rather than the raw tool call, because a search_docs hit carries the guide's url in its result rather than its request.

Phase 0: read-only, and the repo rules

Write nothing in this phase or the next four. Gathering, choosing the claim, drafting the prompt, and designing the checks are all reading. The output is a plan.

Where the harness offers a planning mode that enforces read-only, enter it first and let it hold the line. In Claude Code that is EnterPlanMode. Where it does not, the discipline is yours to keep.

Then read three things:

1. CONTRIBUTING.md, which carries the repo's rules for any eval: suite selection, the folder shape, the motivation: requirement, prompt discipline, scorer discipline, and how to refresh results in CI. This skill defers to it and does not restate it. Where this skill goes further, it says so.
2. The guide under test, as its .md variant.
3. references/flakiness.md, before designing anything rather than after.

Phase 1: gather the evidence

Work through references/sources.md. Run independent sources in parallel.

The checks come from outside the page. A check list derived from the guide inherits the guide's blind spots, so every check passes and the score reports that the guide is fine when nobody looked. Build the list from the sources, then diff it against the page. Anything the sources treat as essential and the page omits is a candidate check, and it should fail a solution written from the page. That failure is the finding the paired docs ticket acts on.

Several sources need Supabase employee access. Settle which ones you can reach before you start, plan from those, and ignore the rest. Missing access is never a blocker. Record in the plan which sources the inventory came from and which were out of reach.

This phase ends with a failure-point inventory: one bullet per finding, each carrying an issue id, a thread, or a url. A finding with no source does not go in.

Phase 2: pick the one claim, and confirm it can be measured

Name the single thing the guide has to transmit, in the user's terms rather than the product's.

One claim, not three. An eval that measures several unrelated things reports which of them an agent got, and a docs ticket cannot act on that. It also means every check hangs off one subject, which is what makes a saturated check obvious later.

A dense page yields ten candidates, all evidenced. Pick on consequence: the failure that costs a user the most, and among equals the one the page itself warns about. A page that carries a danger callout has already named its own worst case.

Then confirm the harness can measure it, before designing a single check. A claim the harness cannot observe is not the claim, however well evidenced. Three questions, and read the code for the answers rather than assuming:

• Does the end state exist somewhere a scorer can reach? A file in the workspace, a row in the database, a response from a running process. If the correct answer is something the platform supplies at deploy time, decide now where the seed lets it land.
• Can the scorer run whatever has to run? A framework the eval seeds is installed and built inside the sandbox with ctx.exec, not on the host: the host-side build helpers relink the workspace's node_modules to the framework's, which does not carry an eval's own dependencies. Check whether any existing eval runs the same toolchain, and treat a first as a risk to state in the plan.
• What is the time budget? The agent's timeout does not bound the scorer, so a long install and build costs wall-clock rather than the agent's budget. Per-command limits are yours to set.

Write the answers into the plan. A feasibility problem found here is a design change; found in Phase 4 it is a rewrite.