Create Evaluations

Create Evaluations

Generates evaluation documentation for a GAIK component in two mirrored locations following the established pattern of transcription_eval and extraction_eval.

Requires user-provided context — the user must supply the evaluation details (metrics, results, methodology, errors, use cases) as part of their message or as attached content. This skill does not run evaluations itself; it documents them.

What this skill creates

Workflow

Phase 1 — Context Parsing

1. Read the component source at implementation_layer/src/gaik/software_components/{component}/ to understand what the component does, its inputs, outputs, and configuration options.
2. If a stub README exists at evaluation_layer/eval_methods/{component}_eval/README.md, read it.
3. If a stub MDX exists at evaluation-layer/{component}-eval.mdx, read it.
4. Map the user-provided context against the required sections for both output files (see references/readme-template.md and references/mdx-template.md for the full section lists).
5. For each required section where no context was provided, ask the user:

   "No content was provided for [Section Name] (e.g. benchmarking results / error taxonomy / CLI usage). Do you want to supply it now, or continue and mark it N/A?"

   • If the user supplies content → incorporate it before generating files.
   • If the user skips → that section gets _N/A — to be completed._ as its body.
6. Ask once: "Should I also generate Python evaluation script stubs, or README + website content only?"

Phase 2 — Outline

Present a two-column outline to the user:

README sections:          MDX sections:
1. Evaluation Metrics     1. The Problem
2. Tools / Code           2. How We Evaluate
3. Results                3. Benchmarking Results
4. Error Classification   4. Error Classification
5. Improvement Strategies 5. Real-World Applications
6. Reproduction / Usage   6. Quality Considerations
7. Integration snippet    7. Getting Started
8. Installation & Setup
9. Related Resources

Populated: [list sections that have content]
N/A:       [list sections that will be marked N/A]
Scripts:   [yes / no]

Phase 3 — Plan Review (never skip)

Present the outline and wait for explicit approval before writing any files. Adjust based on feedback. Do not start Phase 4 until the user confirms.

Phase 4 — Generate Content

Run all sub-steps in order. Do not commit.

4a — Implementation Layer README

Follow the structure in references/readme-template.md. Key rules:

• Use ## for numbered top-level sections (## 1. Evaluation Metrics, ## 2. Evaluation Tools / Code, etc.)
• Use ### for subsections (### 1.1 List of Metrics, ### 3.2 Performance Comparison Table, etc.)
• Each metric gets: Definition, Formula (code block), Components, Business interpretation, Reference values table
• Results section has a Markdown table with model names as rows and metric columns
• Improvement strategies section has a mapping table: | Performance issue | Improvement strategy |
• Integration section has a code snippet using the actual component's Python API
• Sections with no user-provided content → _N/A — to be completed._

4b — Python Script Stubs (only if user said yes in Phase 1)

Create one stub per logical evaluation step following the pattern below. Add a requirements.txt.

"""
{component}_eval — {purpose of this script}.

Usage:
    python {script_name}.py <arg1> <arg2>
"""

from __future__ import annotations

import argparse
from pathlib import Path


def main(arg1: str, arg2: str) -> None:
    # TODO: implement evaluation logic
    pass


if __name__ == "__main__":
    parser = argparse.ArgumentParser(description=__doc__)
    parser.add_argument("arg1", help="...")
    parser.add_argument("arg2", help="...")
    args = parser.parse_args()
    main(args.arg1, args.arg2)

4c — Website MDX Page

Follow the structure in references/mdx-template.md. Key rules:

• Frontmatter: title (display name) and description (one sentence, matches index entry)
• Section order: The Problem → How We Evaluate → Benchmarking Results → Error Classification → Real-World Applications → Quality Considerations → Getting Started
• Tone: business-first (why it matters), then technical detail
• Challenges in "The Problem" → bullet list with bold lead term
• Results → Markdown table + Key Findings bullet list below it
• Error categories use bold Problem / Impact sub-headings (see extraction-eval pattern)
• Quality Considerations: bold question/statement + one-sentence explanation per item
• Getting Started: numbered list of 4–6 steps, ending with GitHub repo link
• Do not use <Callout type="warn"> — that is only for Coming Soon stubs
• N/A sections: use _N/A — to be completed._