Metric Design

1. Can a deterministic rule check this?
   → StringMatchGrader / JsonValidatorGrader / FunctionGrader (zero cost, 100% consistent)
   Examples: exact match for classification labels, regex for format checks,
             JSON schema validation, keyword presence/absence

2. Does it require semantic understanding of text quality?
   → LLMGrader with built-in class (low cost, pre-optimized)
   Examples: CorrectnessGrader (factual match), RelevanceGrader (on-topic check),
             HallucinationGrader (faithfulness to context)

3. Does it involve agent behavior (tool calls, planning, memory)?
   → Agent-specific LLMGrader
   Examples: ToolSelectionGrader, TrajectoryAccuracyGrader, MemoryAccuracyGrader

4. Does it involve code execution or syntax?
   → CodeExecutionGrader / SyntaxCheckGrader
   Examples: test case pass rate, syntax validity, code style checks

5. Does it require external tool calls to verify (web search, database lookup)?
   → AgenticGrader (expensive, use only when necessary)
   Examples: fact-checking against live sources, cross-referencing databases

You are evaluating whether a customer support response correctly identifies
and uses the customer's order number from the conversation context.

PASS: The response references the correct order number exactly as it appears
in the context. If multiple orders exist, the response addresses the right one.

FAIL: The response uses a wrong order number, omits the order number when one
was provided, or references an order not present in the context.

Example 1 (PASS):
Context: "Order #12345: shipped May 10"
Response: "Your order #12345 was shipped on May 10 and arrives May 12."
Critique: The response uses the exact order number (#12345) and matches the
ship date from context. No fabrication or omission.
Result: Pass

Example 2 (FAIL):
Context: "Order #12345: shipped May 10"
Response: "Your order #12346 is on its way!"
Critique: The response uses order #12346 but the context only mentions #12345.
This is a fabricated order number, not a typo — #12346 doesn't exist.
Result: Fail

Example 3 (BORDERLINE PASS):
Context: "Orders #12345 (shipped), #12346 (processing)"
Response: "Your recent order has shipped and should arrive soon."
Critique: The response doesn't specify which order, but says "recent order"
which could reasonably refer to either. If the customer only asked about
shipped items, this is fine. If they asked about a specific order, it's
insufficient. Given the generic phrasing, this passes but is weak.
Result: Pass

{
  "critique": "Detailed assessment referencing specific evidence from the response and context",
  "result": "Pass" or "Fail"
}

from openjudge.graders.llm_grader import LLMGrader
from openjudge.graders.schema import GraderMode

order_accuracy_grader = LLMGrader(
    model=model,
    name="order_accuracy",
    mode=GraderMode.POINTWISE,
    template="""
You are evaluating whether a customer support response correctly identifies
and uses the customer's order number from the conversation context.

Context: {context}
Response: {response}

## Pass/Fail Definitions

PASS: The response references the correct order number exactly as it appears
in the context. If multiple orders exist, the response addresses the right one.

FAIL: The response uses a wrong order number, omits the order number when one
was provided, or references an order not present in the context.

## Examples

Example 1 (PASS):
Context: "Order #12345: shipped May 10"
Response: "Your order #12345 was shipped on May 10 and arrives May 12."
Critique: Exact order number match. Ship date matches context. No fabrication.
Result: Pass

Example 2 (FAIL):
Context: "Order #12345: shipped May 10"
Response: "Your order #12346 is on its way!"
Critique: Order #12346 does not exist in context. Fabricated order number.
Result: Fail

Example 3 (BORDERLINE PASS):
Context: "Orders #12345 (shipped), #12346 (processing)"
Response: "Your recent order has shipped and should arrive soon."
Critique: Doesn't specify which order. "Recent order" is ambiguous but not
factually wrong — it acknowledges a shipped order exists.
Result: Pass

## Output Format

Respond in JSON:
{{"critique": "<detailed assessment>", "result": "Pass" or "Fail"}}
""",
)

from openjudge.graders.function_grader import FunctionGrader
from openjudge.graders.schema import GraderScore, GraderMode

def no_competitor_mention(response: str, competitors: list[str] = None) -> GraderScore:
    """Check that response doesn't mention competitor brands."""
    if competitors is None:
        competitors = ["competitor_a", "competitor_b", "rival_co"]
    mentioned = [c for c in competitors if c.lower() in response.lower()]
    if not mentioned:
        return GraderScore(name="no_competitor", score=1.0, reason="No competitor mentions")
    return GraderScore(
        name="no_competitor", score=0.0,
        reason=f"Mentioned competitors: {', '.join(mentioned)}"
    )

competitor_grader = FunctionGrader(
    func=no_competitor_mention,
    name="no_competitor",
    mode=GraderMode.POINTWISE,
)

from openjudge.generator.simple_rubric.generator import (
    SimpleRubricsGenerator,
    SimpleRubricsGeneratorConfig,
)

config = SimpleRubricsGeneratorConfig(
    grader_name="Customer Support Quality",
    model=model,
    task_description="Customer support chatbot for e-commerce: orders, returns, shipping",
    scenario="Customers asking about order status, return policies, and delivery times",
    min_score=0,
    max_score=1,
)

generator = SimpleRubricsGenerator(config)
grader = await generator.generate(
    dataset=[],
    sample_queries=[
        "Where is my order?",
        "How do I return this item?",
        "When will my package arrive?",
    ],
)
# grader is now a ready-to-use LLMGrader

from openjudge.generator.iterative_rubric.generator import (
    IterativeRubricsGenerator,
    IterativePointwiseRubricsGeneratorConfig,
)

config = IterativePointwiseRubricsGeneratorConfig(
    grader_name="E-commerce QA Grader",
    model=model,
    task_description="Evaluate factual answers to e-commerce customer questions",
    min_score=0,
    max_score=1,
    max_epochs=3,
    batch_size=10,
)

train_data = [
    {"query": "What's your return policy?", "response": "30-day returns, free shipping.", "label_score": 1},
    {"query": "What's your return policy?", "response": "We have a policy.", "label_score": 0},
    # ... 20+ examples
]

generator = IterativeRubricsGenerator(config)
grader = await generator.generate(dataset=train_data)