Output Eval Judge Prompt

PASS: The blog post uses formal language throughout. Professional vocabulary,
complete sentences, no slang, no contractions, no first-person casual asides.

FAIL: The blog post contains one or more instances of casual language: slang,
contractions ("don't", "can't"), informal asides ("pretty cool", "super important"),
or conversational filler ("honestly", "basically").

{
  "critique": "Detailed analysis of the output against the criterion...",
  "verdict": "pass"
}

# tests/evals/judge_tone@v1.prompt
---
provider: anthropic
# current as of 2026-05-04 — run output-dev-model-selection for the latest
model: claude-haiku-4-5-20251001
temperature: 0
maxOutputTokens: 1500
---

<system>
You are an evaluation judge. Your task is to determine whether a blog post maintains the requested tone throughout.

## Criterion

Assess whether the blog post consistently uses the requested tone. A single paragraph that breaks tone is a failure.

## Definitions

PASS: The blog post maintains the requested tone in every paragraph. Word choice, sentence structure, and rhetorical style all align with the requested tone.

FAIL: The blog post contains one or more paragraphs where the tone shifts away from what was requested. Common failures include:
- Formal request but casual language appears ("pretty cool", "super important", contractions)
- Professional request but opinionated editorializing appears
- Technical request but oversimplified explanations appear

## Examples

### Example 1: PASS
Requested tone: formal
Blog excerpt: "The implications of quantum computing for cryptographic security are substantial. Current encryption standards rely on the computational infeasibility of factoring large prime numbers, a guarantee that quantum algorithms may undermine."
Critique: The excerpt uses professional vocabulary ("implications", "computational infeasibility"), complete sentences, no contractions, and maintains an academic register. Consistent formal tone throughout.
Verdict: pass

### Example 2: FAIL
Requested tone: formal
Blog excerpt: "Quantum computing is basically going to break all our encryption. It's pretty wild when you think about it — everything we thought was secure might not be."
Critique: The excerpt contains multiple casual markers: "basically", "pretty wild", contractions ("It's", "might not be"), and conversational filler ("when you think about it"). This directly violates the formal tone request.
Verdict: fail

### Example 3: BORDERLINE (fail)
Requested tone: formal
Blog excerpt: "Quantum computing represents a paradigm shift in computational capability. The technology is incredibly promising, though it's important to note the current limitations in qubit stability and error correction."
Critique: Mostly formal, but contains "incredibly promising" (informal intensifier) and "it's" (contraction). While the overall register is professional, these lapses break the formal tone requirement. Even minor inconsistencies constitute a failure.
Verdict: fail

## Output Format

Return a JSON object with exactly two fields:
- "critique": A detailed analysis (3-5 sentences) citing specific evidence from the blog post
- "verdict": Either "pass" or "fail"
</system>

<user>
Requested tone: {{ requested_tone }}

Blog title: {{ blog_title }}

Blog post:
{{ blog_post }}

Evaluate whether this blog post consistently maintains the requested tone.
</user>

// tests/evals/evaluators.ts
import { verify, judgeVerdict } from '@outputai/evals';
import { z } from '@outputai/core';
import { blogInput, blogOutput } from './schemas.js';

export const checkTone = verify(
  {
    name: 'check_tone',
    input: blogInput,
    output: blogOutput
  },
  async ({ input, output, context }) =>
    judgeVerdict({
      prompt: 'judge_tone@v1',
      variables: {
        requested_tone: String(context.ground_truth.expected_tone ?? input.tone ?? 'professional'),
        blog_title: output.title,
        blog_post: output.blog_post
      }
    })
);

// tests/evals/workflow.ts
import { evalWorkflow } from '@outputai/evals';
import { checkTone } from './evaluators.js';

export default evalWorkflow({
  name: 'blog_generator_eval',
  evals: [
    {
      evaluator: checkTone,
      criticality: 'required',
      interpret: { type: 'verdict' }
    }
  ]
});