Eee Dataset Conversion

Converting evaluation results into Every Eval Ever (EEE)

Rule you will keep relearning: all records must validate, but validating ≠ correct. Most real defects (answer leakage, double-counted aggregates, hardcoded scorers, non-idempotent ids) pass the schema and are still wrong. Always spot-check content, not just validity.

Written against EEE SCHEMA_VERSION 0.3.0 (import it from every_eval_ever.helpers; never hardcode). If that value has moved, re-verify the field claims in reference/ against the live schema — the schema always wins. tests/test_skill_conversion.py pins this marker and re-validates this skill's templates + frozen reference records, so a schema or validator change fails CI here rather than in your PR. If that test is red, fix the skill, then regenerate the frozen records.

How this runs. A person (the operator) runs you and can answer questions mid-run — you are not fully autonomous. When a choice sets policy (step 7's ask-list), ask the operator instead of deciding silently. Decide and log everything else. Finish with a PR that is ready to merge yet makes every non-obvious decision visible, so the maintainer who reviews it can comment and the skill/schema can improve. Two humans: the operator gates live; the PR informs the maintainer.

When this skill applies

A source has model×benchmark scores (a leaderboard, a paper table, an HF results dataset, a harness dump) and you must emit EEE records. Two artifacts:

• Aggregate .json — one EvaluationLog per model (or per model×benchmark), holding the headline scores. Always produced.
• **Instance _samples.jsonl — one record per example. Only if you have per-item data and want it.

The one rule: canonicalize, never invent

An adapter reshapes information the source already carries into the schema's form; it never manufactures information the source lacks.

• Canonicalization is expected — coercing a boolean pass/fail to 1.0/0.0, resolving a model string to its registry canonical_id, normalizing a metric name, deriving the output dir from the collection. Reshaping a value you hold is always fine, and the aggregate and instance-level paths must reshape the same value the same way, so a result and its per-sample rows agree.
• Invention is not — a score for an unscored sample, bounds for a metric whose range is unknown, a model identity the source never named. Record the gap as absent (a null result id, bounds_status: unknown, a dropped-row failure), never fill it in. This is the "fill" half of the coverage-vs-fill rule in reference/fields.md §sources.

Workflow (do these in order)

1. Inspect the source first — you can't map fields you haven't seen. Establish: distinct models · benchmarks/subtasks · the metric and its range · is there per-item data · the harness · timestamps · provenance (paper + each benchmark's own dataset repo). These facts are usually spread across many surfaces and which lives where varies per dataset, so gather every relevant surface before recording a field as unknown** — see reference/fields.md §sources for the surface checklist, the coverage-vs-fill split, and which wins when they disagree. Filter hygiene junk (.ipynb_checkpoints, *-checkpoint.json) and segregate hand-curated baselines from harness runs.
2. Decide the shape — source_type is set by the artifact you hold, not who ran the compute: raw per-item outputs → evaluation_run (even if a third party ran them); only-aggregate reported numbers → documentation (a leaderboard scrape stays documentation). Then: aggregate-only vs +instances; grain (one log per model = default, or per model×benchmark when a benchmark has its own instance sidecar). See reference/fields.md §shape.
3. Copy a template / reference adapter — templates/aggregate_adapter.py (always) and, for per-item data, templates/instance_sidecar.py (runnable skeletons verified against the live validator). For a fuller real example, mirror every_eval_ever/adapters/llm_stats (aggregate/documentation), .../hfopenllm_v2 (documentation, many models), or .../openeval (aggregate + instance sidecars). Adapters live at every_eval_ever/adapters/<name>/adapter.py and run as uv run python -m every_eval_ever.adapters.<name>.adapter; __init__.py just marks the package. Don't hand-roll the write path or the drop path — the repo owns both: publish through save_evaluation_logs (aggregate-only) or converters.common.publication.publish_evaluation_logs (with instance sidecars), and account for every rejected row via SourceConversionResult + save_failure_report + a non-zero exit. See reference/datastore-gate.md §publish.
4. Fill fields carefully — the field traps are the whole game. Load reference/fields.md (aggregate) and reference/instance-level.md (jsonl).
5. Canonicalize ids — model + benchmark ids must resolve in the eval-card-registry (else they fragment the data). Default: resolve live against the hosted resolver and use canonical_id for the join-key fields, with an opt-out flag + never-fatal fallback to the raw id (marked unverified). But never key evaluation_id on the resolved id — that's a moving join key; the record identity rides the raw source id. See reference/registry.md.
6. Verify — uv run python -m every_eval_ever validate <files> (files/glob, not a dir), an offline unit test, ruff, a live smoke run, and a content spot-check. The validator's semantic checks run only on the CLI, and only when the file sits at its final data/<collection>/<dev>/<model>/ path. They are the merge gate, listed in reference/datastore-gate.md. See reference/verification.md.
7. Ask, then log your decisions. Two channels, don't confuse them:
   • Ask the operator (live) when a choice sets policy: creating a new canonical id · dropping a non-trivial share of the data · an ambiguous metric choice · bounding an unbounded metric · re-hosting large data · the source won't fit without a structural change (a schema field, an edit to a base adapter or a shared converter, relaxing a validator rule). Don't decide these silently — the person running you is there to answer. The structural one is not yours to fold into this PR: its design gets agreed before a PR exists, and carrying it here would hold the adapter behind that discussion.
   • Log (in the PR) every non-obvious choice — not just where it was hard. A confident wrong choice produces no "friction," so log decisions, not pain. Finish with a ready-to-merge PR carrying the decision log below. General gaps (would recur on other datasets) also become a separate skill-labeled PR or a skill-gap issue — you needn't know the fix; flagging where you guessed is enough.