Eval

Eval — Quality & Regression Gate

CRITICAL: Run HARNESS_DIR=$(epic path) first. Never use .harness/ in the project directory.

When to Trigger

• Before /ship creates a PR (automatic if eval.yaml exists)
• After /go completes a feature
• On explicit /eval command
• When user mentions "regression", "baseline", "eval suite", "quality check"
• CI: make eval or epic eval --json

Execution Modes

4 dimensions run in parallel where possible:

1. eval:correctness — Test pass rate, mutation score, assertion density
2. eval:performance — Throughput, latency, memory (opt-in)
3. eval:quality — Lint, code quality, LLM-as-judge
4. eval:regression — Baseline comparison, score deltas

Process

Step 0: Prerequisites

HARNESS_DIR=$(epic path)

If $HARNESS_DIR/eval/eval.yaml does not exist, run scaffold:

epic eval --init

Read the config:

cat $HARNESS_DIR/eval/eval.yaml

Step 0.5: Scaffold benchmarks (when no benchmark infrastructure exists)

If eval.yaml has benchmarks: [] and no benchmark files are found in the project:

1. Generate stub files using the CLI:

   epic eval --scaffold

   Supported stacks (auto-detected from project markers):

   Stack	Detected by	Generated file	Output format
   Rust	Cargo.toml	benches/eval_harness.rs	criterion (exit code)
   Python	pyproject.toml / setup.py	benchmarks/eval_runner.py	JSON composite
   TypeScript	tsconfig.json	benchmarks/eval.ts	JSON composite
   Node.js	package.json	benchmarks/eval.mjs	JSON composite
   Go	go.mod	benchmarks/eval_test.go	JSON composite
   Java	pom.xml / build.gradle	benchmarks/EvalBenchmark.java	exit code
   Kotlin	build.gradle.kts	benchmarks/EvalBenchmark.kt	exit code
   Ruby	Gemfile	benchmarks/eval_benchmark.rb	JSON composite
   PHP	composer.json	benchmarks/eval_benchmark.php	JSON composite
   C#	*.csproj / *.sln	Benchmarks/EvalBenchmark.cs	JSON composite
   Swift	Package.swift	benchmarks/EvalBenchmark.swift	JSON composite
   Elixir	mix.exs	benchmarks/eval_benchmark.exs	JSON composite
   C++	CMakeLists.txt	benchmarks/eval_benchmark.cpp	exit code

2. Customize the generated file — every file has # TODO / // TODO markers:

   • Replace placeholder logic with calls to your actual domain functions
   • Adjust the composite score weights to reflect your domain priorities
   • For precision/recall benchmarks: wire in your real test set and model

3. If --scaffold can't generate a useful stub (domain too complex, custom evaluation logic needed), generate a custom benchmark with LLM assistance:

   • Read the project's main source files to understand the domain
   • Identify the 2–3 most critical quality signals (latency, accuracy, throughput, precision/recall)
   • Write a benchmark that measures those signals and outputs {"composite": 0.0–1.0, ...}
   • Save to benchmarks/eval_runner.{ext} matching the project language

4. Wire into eval.yaml:

   benchmarks:
     - name: eval_runner
       command: python3 benchmarks/eval_runner.py full
       result_type: composite   # parse composite field from JSON stdout

   Use result_type: exit_code for frameworks (criterion, JMH, BenchmarkDotNet) that manage their own output.

Step 1: Run Rust CLI

Execute the structured evaluation via the Rust binary:

epic eval --json

This runs all enabled dimensions and outputs a JSON result. Capture the output.

If the CLI reports llm_judge: SKIPPED (no LLM available in CLI mode), proceed to Step 2 for LLM-as-judge. Otherwise, skip to Step 3.

Step 2: LLM-as-Judge (when llm_judge enabled)

If the quality dimension has llm_judge: true and CLI marked it SKIPPED:

1. Sample 3–5 changed files from the current branch:

   git diff --name-only $(git merge-base HEAD main)

2. For each sampled file, evaluate on a 1-10 rubric:

   • Readability (naming, structure, flow)
   • Correctness (logic, edge cases, error handling)
   • DRY (no unjustified duplication)
   • Security (no obvious vulnerabilities)

3. Average scores across files. Map to 0.0–1.0 scale.

4. Record results alongside CLI output.