Result Evaluator

Result Evaluator Skill

Overview

This skill evaluates analysis results against predefined criteria and decides ACCEPT_AND_PROCEED or REVISE_AND_RETRY. It follows a 4-phase protocol: criterion alignment → multi-dimensional evaluation with Source Reliability hard gate → statistical methodology audit → overall assessment. Hallucination detected → immediate REVISE; any checklist dimension FAIL → mandatory REVISE (hard gate, overrides scoring).

When to Use This Skill

Always load this skill when:

• User asks to evaluate, audit, score, or quality-check analysis results that another skill — typically code-engineer — has just produced as a Result Package
• User asks for an explicit ACCEPT_AND_PROCEED vs REVISE_AND_RETRY (or CONDITIONAL) decision before the results are used downstream (e.g. fed into a report, shared with stakeholders, or acted on)
• User wants a Source Reliability check on computational or research-style results — to detect hallucinated numbers, fabricated statistics, invented citations, or code–data misalignment
• User asks for a Statistical Methodology Audit covering multiple-testing correction, model-assumption verification, confounder control, sample-size/power, batch effects, outlier/missing-data handling, and reproducibility
• User requests the multi-dimensional quality score (Accuracy / Completeness / Robustness / Relevance / Methodology / Critical Reflection, each 0–10) on a Result Package
• User wants to know whether a result is reproducible from the supplied code and data, or whether the analysis should be re-run before being trusted

Input Sources

This skill evaluates analysis results with methodology documentation. Accepted input formats:

From code-engineer (recommended upstream skill):

• Structured data: --output-file JSON ([{col: val, ...}]) or CSV/MD export — provides the numerical/tabular results
• Methodology documentation: presented in conversation by the agent — includes libraries, statistical methods, method justification
• Data traceability: source file names, sheet/column names, row counts, transformations applied
• Analysis code: the complete code used to produce results (for reproducibility audit in Phase 3)

From other sources: any structured results with accompanying methodology description. Minimum required: results data + method description + data source identification.

If methodology documentation or data traceability is missing, note the gap in evaluation and flag Source Reliability as PARTIALLY_RELIABLE.

Python Package Installation

If you need to install new Python packages, install them through the Tsinghua PyPI mirror for reliability:

pip install [python package] -i https://pypi.tuna.tsinghua.edu.cn/simple

Workflow

Step 1: Understand Evaluation Input

Identify the evaluation context:

• Analysis results to evaluate: Structured output from code execution
• Methodology documentation: How the results were produced (libraries, methods, code)
• Data traceability: Source data identification (file names, column names, row counts)
• Evaluation criteria: What aspects to evaluate and expected quality thresholds. Infer from context if missing (note limitation).
• Analysis plan context: Domain, background information

Prerequisites: results must be available and parseable; methodology documentation and data traceability should be provided (evaluation quality degrades without them); criteria must be specified or inferable.

Step 2: Execute 4-Phase Evaluation Protocol

Phase 1 — Criterion Alignment

Map each result to an evaluation criterion. Flag UNMAPPED results and uncovered criteria. Infer criteria from context if missing (document as inferred).

Phase 2 — Per-Result Evaluation

Source Reliability Hard Gate (check first — hallucination → immediate REVISE_AND_RETRY, skip rest):

For computational-type results (from code-engineer and similar tools):

For research-type results (literature-based, citing external references):

Verdict: RELIABLE (PASS) / PARTIALLY_RELIABLE (FAIL, continue) / UNRELIABLE (REVISE, stop).

Unified Evaluation Matrix — score each dimension 0-10; each dimension also has a PASS/FAIL threshold (score ≥5 → PASS, score <5 → FAIL):

Hard Gate Rule: any dimension FAIL (score <5) → mandatory REVISE_AND_RETRY, regardless of the average score. The scoring average determines the severity grading of the REVISE decision, not whether to REVISE.

Composite Quality Rating (applies only when all dimensions PASS):

Modifiers from Phase 3 RISK items: ≥3 RISK items → downgrade 1 level.

Per-Result Decision (when all dimensions PASS):

When any dimension FAIL: the decision is always REVISE_AND_RETRY. The severity is graded by how many dimensions FAIL and the average score of passing dimensions:

Phase 3 — Statistical Methodology Quality Audit

Domain priorities: Biology → batch, confounders, multiple testing; Chemistry → reproducibility, assumptions; Materials → sample size, uncertainty; Finance → assumptions, confounders, outlier handling.

For each NO: record RISK, assess severity (H/M/L), include in guidance if ≥MEDIUM.

Phase 4 — Overall Assessment

1. Check Phase 2 hard gate: any dimension FAIL → REVISE_AND_RETRY (skip to step 4)
2. If all PASS: compute average score → quality rating → apply RISK modifiers
3. Final decision:
   • ROBUST → ACCEPT_AND_PROCEED
   • ACCEPTABLE → CONDITIONAL — ACCEPT with stated limitations
   • NEEDS_IMPROVEMENT → REVISE_AND_RETRY (MODERATE severity)
4. If REVISE: prioritize guidance (FAIL dimensions > HIGH RISK > low passing scores), limit top 3 actionable items

Step 3: Document Results

Output evaluation results per the Output Schema below.

Output Schema

Every evaluation must produce the following structure:

{
  "verdict": "ACCEPT_AND_PROCEED | CONDITIONAL | REVISE_AND_RETRY",
  "severity": "MODERATE | SIGNIFICANT | CRITICAL",
  "quality_rating": "ROBUST | ACCEPTABLE | NEEDS_IMPROVEMENT",
  "source_reliability": "RELIABLE | PARTIALLY_RELIABLE | UNRELIABLE",
  "dimension_scores": {
    "accuracy": 0-10,
    "completeness": 0-10,
    "robustness": 0-10,
    "relevance": 0-10,
    "methodology": 0-10,
    "critical_reflection": 0-10
  },
  "dimension_status": {
    "accuracy": "PASS | FAIL",
    "completeness": "PASS | FAIL",
    "robustness": "PASS | FAIL",
    "relevance": "PASS | FAIL",
    "methodology": "PASS | FAIL",
    "critical_reflection": "PASS | FAIL"
  },
  "risk_items": [
    {"item": "description", "severity": "H | M | L"}
  ],
  "revision_guidance": ["top 3 prioritized action items"],
  "limitations": ["accepted weaknesses, if CONDITIONAL"]
}

When presenting results to the user, format as a readable summary — not raw JSON. Highlight the verdict, failed dimensions (if any), and revision guidance (if REVISE).