Add Eval

Add a New Eval

Follow these steps whenever the user asks to create a new eval. Read .cursor/skills/add-eval/reference.md for grader helpers, test patterns, and conventions. The eval-writing rules in AGENTS.md ("Authoring New Evals") and README.md ("Writing evals") take precedence over this skill.

Steps 0-2 (gather info, research, design) are collaborative and read-only. Don't create or edit any files until the user has seen the research findings and approved the eval design. Implementation starts at Step 3.

Step 0: Gather Information

Determine the following (ask the user if not provided):

1. Concept to test - what Convex feature or pattern should this eval exercise? (e.g. "vector search", "pagination with joins", "cascade deletes")
2. Specific focus - any edge cases, constraints, or behaviors the user wants to emphasize?

Category Selection

List the existing categories by scanning evals/ top-level directories, then propose the best-fit category. The current categories are:

• If the concept clearly fits one category, propose it with a brief justification.
• If it's ambiguous (e.g. "scheduled mutations" could be fundamentals or mutations), stop and ask the user. Present the candidate categories with reasoning for each.
• If it doesn't fit any existing category, propose creating a new category and ask the user to confirm.

Determine the eval number by listing existing evals in the chosen category and picking the next sequential number.

Step 1: Research

Run these four research tracks. Use sub-agents or parallel tool calls where possible.

A. Convex Docs (source of truth)

1. Fetch https://docs.convex.dev/llms.txt to get the docs table of contents.
2. Identify the 1-3 most relevant doc pages for the concept being tested.
3. Use WebFetch to retrieve those specific pages.
4. Extract the correct API patterns, constraints, and best practices. These are ground truth for designing the eval and answer.

B. Existing Guidelines

1. Read the relevant sections of runner/models/guidelines.md for the concept being tested. It is the source. runner/models/guidelines.ts only loads it.
2. Record:
   • Which existing guidelines are relevant
   • What behavior those guidelines would lead a model to produce
   • Whether the current guidelines would already be expected to make strong models pass
   • Whether any guideline seems to conflict with what the proposed eval is trying to reward
3. Do not treat this as a reason to block the eval automatically. This is context for design and later calibration.
4. If an existing guideline appears to directly contradict the proposed eval, STOP and discuss with the user before proceeding.

C. Existing Eval Patterns

1. Read 2-3 evals in the same or a similar category.
2. Study: TASK.txt style, answer structure, test approach, schema design.
3. Note which grader helpers and test patterns they use. See reference.md for the full catalog.

D. Overlap Check

1. List ALL eval directory names under evals/ to get a high-level view of coverage.
2. For any eval whose name suggests overlap with the proposed concept, read its TASK.txt.
3. If an existing eval tests the same or very similar concept, STOP and warn the user:
   • Name the overlapping eval(s) and explain what they already cover.
   • Ask whether to: differentiate the new eval (narrow its scope), adjust the existing eval, or abandon.
4. Flag even partial overlap, e.g. "002-queries/012-index_and_filter already tests index usage but doesn't cover compound indexes."

Step 2: Design the Eval

Present the full eval design to the user for review. Stay read-only until they approve it.

Why this matters

State what Convex-specific knowledge the eval measures and what silently breaks in production when a model lacks it. Every eval issue and PR needs this section (AGENTS.md). If you can't write it, the eval probably tests trivia.

TASK.txt Draft

Write the complete TASK.txt content. Follow these principles:

• Laser-focused on the concept being tested. Control other variables (keep schema simple, minimize unrelated code).
• Explicit about schema, function names, argument types, return shapes, and which files to create.
• Don't over-specify Convex implementation details that are covered by the guidelines. If a model needs the task to spell out how to use internalMutation or how pagination works, that's a meaningful signal, not a task problem.
• Don't under-specify the problem domain. The model should not need to guess what the feature does, only how to implement it in Convex.
• Include schema as a TypeScript code block when applicable.
• Specify edge cases (empty results, error messages, missing data).

Answer Outline

Describe the files that will be created and the key implementation approach. Don't write the full code yet, just the structure and important decisions.

Test Approach

Describe how the eval will be graded:

• Pick the primary grading primitive first: behavior tests, schema inspection, function-spec comparison, HTTP testing, AI grading, AST analysis, or some combination.
• Pick the pipeline: backend (default), static for selection evals, or module. See "Eval Pipelines" in reference.md.
• Which grader helpers to use (see reference.md for the catalog and decision tree).
• What behaviors to assert on.
• Whether standard unit tests are sufficient, or if you need schema inspection, HTTP testing, AI grading, or something else.

If unit tests cannot fully verify the concept (e.g. testing that a model uses an index rather than a filter, or testing code organization patterns), STOP and discuss with the user. Present the options:

• Schema/index inspection (using getSchema, hasIndexForFields)
• AI grading (createAIGraderTest, currently disabled and requires a repo change to re-enable)
• AST analysis (parse the generated TypeScript files)
• Restructure the eval so the concept can be tested via behavior
• Accept the limitation and test what we can

Let the user decide before proceeding.

Guidelines Hypothesis

Summarize the guideline context before implementation:

• Which existing guidelines are relevant to this eval
• What you would expect guideline-following models to do
• Whether failures on this eval would likely indicate a model gap, an eval/task problem, or a missing/weak guideline
• Any existing guideline that might need to be revised if calibration shows an unexpected result

If a new guideline is likely needed, design it minimal-first. Every token in the guidelines is sent with every prompt, so bloat costs real money. Start with the smallest guideline that teaches the critical pattern (usually one code example), test it, and only expand if models still fail. Avoid pinning specific dependency versions in guidelines as they age quickly. Prefer "always install the latest version" instead. Follow the guideline rules in AGENTS.md "Authoring New Evals": prove the line helps a specific eval, and prefer improving an existing line over adding one.

Push Back

Before presenting the design, critically evaluate it. Warn the user if:

• The eval overlaps heavily with an existing eval (should have been caught in Step 1C, but re-check).
• The task tests too many concepts at once. Each eval should focus on one thing.
• The task is so explicit that it tells the model exactly how to solve it (e.g. specifying .withIndex() calls). We're testing knowledge, not instruction-following.
• The current guidelines suggest a different approach than the eval is rewarding, or make the expected signal unclear.
• The eval seems too easy (every model will pass) or too hard (no model will pass).

Step 3: Implement the Eval

After the user approves the design, implement:

1. Create directory: evals/<category>/<eval_slug>/

2. Write TASK.txt with the approved content. For the static or module pipeline, also add eval.json (see "Eval Pipelines" in reference.md).

3. Create answer directory:

   • answer/package.json:

     {
       "name": "convexbot",
       "version": "1.0.0",
       "dependencies": {
         "convex": "^1.31.2"
       }
     }

     Component evals pin exact versions of convex and the component instead (e.g. "convex": "1.41.0", "@convex-dev/aggregate": "0.2.2"). Copy the pins from a sibling in evals/007-components/. Usage-eval tasks state the same pins. Selection-eval tasks can't name the component, so they pin only convex.
   • answer/convex/schema.ts (if applicable)
   • Implementation files (e.g. answer/convex/index.ts)
   • No returns: validators unless the task tests them (AGENTS.md). Answers are likely training data.

4. Add generated types: bunx convex codegen fails without a deployment ("No CONVEX_DEPLOYMENT set"). Copy _generated and tsconfig.json from a sibling single-module eval with the same files, e.g. convex/index.ts and convex/schema.ts:

   cp -R evals/<sibling>/answer/convex/_generated evals/<sibling>/answer/convex/tsconfig.json evals/<category>/<eval_slug>/answer/convex/

   See "Generated types" in reference.md for when to edit the copied imports.

5. Write grader.test.ts using the approved test approach. Backend-pipeline graders import from grader/index.ts with relative paths:

   import { responseClient, responseAdminClient, addDocuments } from "../../../grader";
   import { api } from "./answer/convex/_generated/api";

   Adjust the depth of ../ based on the eval's nesting level. Static and module graders import from ../../../grader/outputDir instead. grader/index.ts throws "CONVEX_PORT is not set" without a backend.

6. Typecheck:

   bun run typecheck