---
name: run-evals
description: "Run evaluation tests for prompt quality. Use for: run evals, validate prompt changes, check prompt quality, eval recording, evaluation failure, debug eval, use recording."
---

# Run Evaluations

## When to Use
- After modifying any `.prompty` file to validate quality hasn't regressed
- Debugging failing or partial eval results
- Iterating on prompts with cached recordings to avoid LLM costs

## Running Evals

Always activate the virtualenv first: `.venv\Scripts\activate` (Windows) or `source .venv/bin/activate` (Linux/macOS).

### Via CLI
```bash
# Run all workflows
avc test eval

# Run a specific workflow
avc test eval --test-paths evals/tests/mention_action

# Run a single test file
avc test eval --test-paths evals/tests/filter_existing_comment/discard_azure_sdk_repeat_comment.yaml

# Multiple runs (median result kept)
avc test eval --num-runs 5 --test-paths evals/tests/filter_comment_metadata

# Use recordings (cached LLM responses) — first run saves, subsequent runs reuse
avc test eval --use-recording --test-paths evals/tests/mention_action

# Verbose output (show passing tests too)
avc test eval --style verbose
```

### Via run.py directly
```bash
cd evals
python run.py --test-paths tests/mention_action
```

## Existing Workflows

| Workflow directory | Kind | Target function in `_custom.py` | Prompt tested |
|---|---|---|---|
| `mention_action` | `prompt` | `_mention_action_workflow` | `parse_conversation_action.prompty` |
| `mention_summarize` | `summarize_prompt` | `_mention_summarize_workflow` | `summarize_github_actions.prompty` |
| `thread_resolution_action` | `prompt` | `_thread_resolution_action_workflow` | `parse_thread_resolution_action.prompty` |
| `filter_comment_metadata` | `prompt` | `_filter_comment_metadata` | `filter_comment_with_metadata.prompty` |
| `filter_existing_comment` | `prompt` | `_filter_existing_comment` | `filter_existing_comment.prompty` |
| `deduplicate_parser_issue` | `prompt` | `_deduplicate_parser_issue` | `deduplicate_parser_issue.prompty` |
| `deduplicate_guidelines_issue` | `prompt` | `_deduplicate_guidelines_issue` | `deduplicate_guidelines_issue.prompty` |
| `filter_generic_comment` | `prompt` | `_filter_generic_comment` | `filter_generic_comment.prompty` |
| `judge_comment_confidence` | `prompt` | `_judge_comment_confidence` | `judge_comment_confidence.prompty` |
| `merge_comments` | `summarize_prompt` | `_merge_comments` | `merge_comments.prompty` |
| `generate_correlation_ids` | `prompt` | `_generate_correlation_ids` | `generate_correlation_ids.prompty` |

### Evaluator kinds
- **`prompt`** — Action-based. Compares expected vs actual action, then similarity-scores the rationale. Wrong action = 0%.
- **`summarize_prompt`** — Summary-based. Uses `SimilarityEvaluator` on full output. Success threshold: score > 70%.

## Recordings

- Stored in `evals/recordings/<workflow_name>/<testcase_id>.json`
- Gitignored — each dev builds their own cache
- If you change a test file, delete its recording or run without `--use-recording`
- `--use-recording` on first run makes LLM calls and saves; subsequent runs reuse cached responses

## Gotchas

- **Use `python cli.py` not `.\avc`**: The `avc.bat` script calls bare `python` which may resolve to the system Python instead of the venv. Use `.venv\Scripts\activate; python cli.py test eval ...` to ensure the venv Python is used.
- **Field name mismatch**: Test YAML fields must exactly match target function parameter names (excluding `testcase` and `response`)
- **Stale recordings**: After changing a prompt, delete recordings or run without `--use-recording` to get fresh results
- **Testcase uniqueness**: The `testcase` field must be unique across all test files in a workflow — it's the cache key
- **Kind validation**: The `kind` in `test-config.yaml` must be registered in `_config_loader.py` (`prompt` or `summarize_prompt`)