Eval Loop

Eval Loop

Generalized quality iteration loop for any dimension — UX, data architecture, code quality, feature completeness, content tone. Traces specific symptoms to structural root causes, defines measurable targets, and iterates with automated backpressure until targets pass.

Meta-Principle: "Don't fix the symptom, fix the class — but verify by measuring the symptom."

Scope-Principle (added): "This loop verifies presence, shape, and content quality. It does NOT verify predictive validity. Scoring/classification problems require statistical gates this skill does not provide — route them to eval-driven-scoring."

When to Use This Skill

Apply this skill when:

• A specific quality complaint surfaces ("I can't click the contact", "this email sounds robotic")
• A feature needs a quality bar before shipping ("what would 10/10 look like?")
• A class of problems keeps recurring (data gaps, UX friction, architectural debt)
• Quality needs to improve but the path from current → target is unclear
• Multiple dimensions (UX + data + code) need coordinated improvement

Do NOT use for:

• Initial architecture design (use /specification-driven-development)
• LLM scoring / classification / predictive ranking — use /eval-driven-scoring. This is a HARD route, not a suggestion.
• Simple bugs with obvious fixes (just fix them)
• Performance optimization (different discipline — profile first)

Relationship to eval-driven-scoring: This skill is the generalized eval loop for deterministic quality (presence, shape, tone, content). eval-driven-scoring is the specialized sibling for predictive quality (classification, ranking). They are NOT interchangeable — predictive problems need holdouts, base rates, and discriminative ratios that this skill does not enforce.

Step 0: Classify the Problem Type (MANDATORY — DO FIRST)

Before Step 1, classify the quality problem. Routing wrong here wastes work and can produce confidently wrong scoring models.

Why routing matters

Backpressure in this skill is assertion-based — "does this link exist", "does this card render", "is the post_url field populated". Those verifications pass when the fix is applied correctly.

Backpressure for predictive problems requires:

• Ground truth provenance audit — is the "correct label" independent of what you're predicting?
• Base rate denominator — what % of the universe fires this rule?
• Discriminative ratio ≥ 2.0 — is the signal real or a base-rate artifact?
• Holdout set — does the rule generalize beyond tuning data?

None of these are in this skill. If you use this skill's backpressure on a scoring problem, you will produce a model that perfectly "passes" on known positives and fails on reality. This is not hypothetical — it has happened and produced client-visible damage.

Routing examples

Core Pattern

SYMPTOM (specific, observed)
    "Lisa can't click the contact to see who they are"
         │
         ▼
GENERALIZE (what class of problem is this?)
    "Signal cards lack provenance — user can't trace to source"
         │
         ▼
ROOT CAUSES (first-principle gaps that create this class)
    1. Agent doesn't include post URLs in evidence
    2. contact_linkedin not mapped to UI component
    3. reasoning_trace field exists but UI doesn't render it
    4. No signal dedup across runs
         │
         ▼
TARGETS (measurable, with verification method)
    T1: "Every signal card has a clickable LinkedIn link" → Playwright assertion
    T2: "Reasoning trace visible on expand"              → Playwright assertion
    T3: "95% of signals include source post URL"         → Unit test on pipeline output
    T4: "No duplicate accounts across consecutive runs"  → Unit test on D1 query
         │
         ▼
FIX (one root cause at a time)
         │
         ▼
VERIFY (run the target's backpressure test)
    Pass → mark target complete, move to next
    Fail → diagnose, adjust fix, re-verify
         │
         ▼
LOOP until all targets pass

The Five Steps

Step 1: Surface and Confirm Symptoms

Collect the specific complaints. Not interpretations — the actual observable problems.

## Symptoms (observed)
1. "I can't click on the contact and see who they are"
2. "There's no way to know if this was previously rejected"
3. "I can't see the LinkedIn post that triggered this signal"
4. "The evidence section just says 'LinkedIn' — that's paper thin"

Ask: "Is this the complete list, or are there more?" Surface ALL symptoms before proceeding. Fixing one while ignoring others wastes iterations.

Step 2: Generalize to Problem Classes

Group symptoms by the structural gap they share.

One symptom can point to multiple classes. Multiple symptoms often share a root.

## Problem Classes
A. PROVENANCE — User cannot trace a signal to its source data
   Symptoms: 1, 3, 4

B. CONTACT INTELLIGENCE — User cannot evaluate who the contact is
   Symptoms: 1, 2

C. SIGNAL DEDUP — System doesn't track cross-run history
   Symptoms: 2

Why generalize? Fixing the class fixes current AND future symptoms in that class. Fixing symptoms one-by-one is whack-a-mole.

Step 3: Identify Root Causes per Class

For each problem class, trace to concrete technical/architectural gaps.

Root causes are things that can be fixed in code, data, prompts, or architecture. They are NOT symptoms restated.

## Root Causes

### Class A: PROVENANCE
A1. Agent prompt doesn't instruct to include LinkedIn post URL in evidence_json
A2. evidence_json schema has no `post_url` field
A3. SignalCard component doesn't render source links
A4. Supabase intelligence_posts VIEW has post_url but pipeline doesn't query it

### Class B: CONTACT INTELLIGENCE
B1. contact_linkedin field exists in D1 but not mapped to UI
B2. No contact card component — just name + email as text
B3. No mailto: link on email
B4. No HubSpot link for the contact (only company)

Test each root cause: "If I fixed ONLY this, would it improve the symptom?" If yes, it's a real root cause. If not, dig deeper.

Step 4: Define Targets with Backpressure

Every root cause fix gets a measurable target. Every target gets a verification method.

The verification method is the backpressure — the thing that tells you the fix actually worked, and catches regressions.

Backpressure Spectrum

Choose the lightest verification that catches the problem:

Rule: If you can write an automated test, write one. If the quality is inherently subjective, use LLM-as-judge with a rubric. Manual checklists are a smell — they don't enable iteration.

Rule (added): Backpressure here verifies this specific instance matches criteria. It does NOT verify the rule generalizes to unseen data. If the target is "rule correctly classifies", Step 0 routing should have sent you elsewhere.

Target Format

## Targets

### T1: Signal cards link to contact LinkedIn profile
- Root cause: B1
- Backpressure: Playwright
- Test: `expect(page.locator('a[href*="linkedin.com/in/"]').first()).toBeVisible()`
- Pass criteria: Link visible, href matches contact_linkedin from API

### T2: 95% of signals include source post URL
- Root cause: A1, A2
- Backpressure: Unit test
- Test: Run pipeline on 20 accounts → `signals.filter(s => s.evidence.post_url).length / signals.length >= 0.95`
- Pass criteria: ≥95% of generated signals have non-null post_url

### T3: Cold emails sound like Shakespeare
- Root cause: Prompt doesn't reference style guide
- Backpressure: LLM-as-judge
- Rubric: references/shakespeare-rubric.md
- Test: Score 10 emails → average ≥ 8/10
- Pass criteria: Mean score ≥ 8, no individual score < 6

Step 5: Iterate

Fix one root cause at a time. Run its backpressure test. Keep or revert.

Fix A1 (add post_url to agent prompt)
  → Run T2 (unit test: 95% have post_url)
  → Result: 60% have post_url (FAIL)
  → Diagnose: agent finds post but doesn't always extract URL
  → Adjust: add explicit instruction "always include post_url field"
  → Re-run T2: 98% (PASS) → KEEP

Fix B1 (map contact_linkedin to SignalCard)
  → Run T1 (Playwright: LinkedIn link visible)
  → Result: PASS
  → KEEP

Iteration Rules:

1. ONE fix per iteration (can't attribute improvement to bundled changes)
2. Run the SPECIFIC target test for that fix (not all tests every time)
3. After ALL fixes applied, run ALL target tests as regression suite
4. Log every iteration — even failures are data
5. After 3 failed iterations on the same root cause, question whether it's the real root cause

State Files

Following the autoresearch pattern, state lives in two files so a fresh agent can continue:

{"timestamp":"2026-03-31T17:00:00Z","target":"T1","method":"playwright","result":"pass","details":"LinkedIn link visible for all 3 signals"}
{"timestamp":"2026-03-31T17:05:00Z","target":"T2","method":"unit_test","result":"fail","details":"12/20 signals have post_url (60%)","iteration":1}
{"timestamp":"2026-03-31T17:30:00Z","target":"T2","method":"unit_test","result":"pass","details":"19/20 signals have post_url (95%)","iteration":2}

Workflow Integration

Reactive (quality complaint)

User: "This is a 3/10, the signal cards are paper thin"
  → Step 0: Classify (presence/content — continue)
  → Step 1: Surface all symptoms (interview user)
  → Step 2-4: Generalize → root causes → targets
  → Step 5: Iterate until targets pass
  → Outcome: "Here's what changed + verification results"