Eee Datastore PR Review

Review and repair an EEE datastore PR

Produce the smallest source-backed change that makes the existing PR both validator-clean and semantically correct. Treat a green validator as necessary, not sufficient.

Operating contract

• Treat the current checkout's schemas and REGISTERED_CHECKS as the local source of truth. Treat the newest bot result for the current PR head as the remote gate.
• Report what the source establishes. Never invent metadata, clamp a score into its declared bounds, or change a value merely to silence the validator.
• Make unknown a researched conclusion, not a default. Record which relevant surfaces were checked before retaining it.
• Distinguish absent record metadata from unavailable source evidence. A missing or null model_info.additional_details object means the record needs investigation; it does not establish either axis as unknown.
• Keep work on the supplied refs/pr/<number> ref. Do not open a replacement PR for another repair round.
• If asked only to review, prepare a patch and findings without uploading or commenting. If asked to fix, update the supplied PR, trigger its validator, and iterate on that same ref.
• Ask the operator before a policy decision: minting a new canonical id, changing a schema/validator rule, dropping non-trivial data, choosing an ambiguous metric or bound, or making another structural change. Do not hide such a choice in a data repair.

Load the live EEE rules

Before editing, read these sibling references:

• ../eee-dataset-conversion/reference/datastore-gate.md
• ../eee-dataset-conversion/reference/fields.md
• ../eee-dataset-conversion/reference/datastore-submission.md
• ../eee-dataset-conversion/reference/verification.md

Read reference/model-deployment.md whenever either model deployment axis is missing, stale, invalid, or suspicious. Read ../eee-dataset-conversion/reference/registry.md when an id is unresolved or a registry update is requested. Load the full eee-dataset-conversion skill when the repair also changes an adapter or regenerated output.

Re-read the allowed deployment values from every_eval_ever/validator/validation_core.py and the live schema. Existing records and old bot comments may use obsolete vocabularies.

Workflow

1. Establish the exact PR state

1. Parse the dataset repo and discussion number from the supplied URL.
2. Fetch the discussion details, commit history, current head, base ref, file diff, conflicts, and every validator comment. Prefer Hugging Face's API or huggingface_hub over scraping rendered HTML.
3. Select only the newest completed bot run whose fingerprint or head matches the current PR. Older green runs describe older data or validator versions.
4. Check out refs/pr/<number> in a dedicated datastore worktree or temporary clone. Preserve the contributor's branch and unrelated changes.
5. Diff the PR head from its merge base with main. Inventory added, modified, renamed, and deleted paths; include aggregate/instance companions even if only one side appears in the diff.

Record the PR head commit and bot schema/compatibility version in the review notes. If the bot and local schema differ, label their disagreement as version skew and investigate it explicitly.

2. Reproduce the gate locally

Run the current EEE CLI against changed .json and .jsonl files at their final data/<collection>/<developer>/<model>/... paths. Pass files or a quoted glob, never a directory. Include companion files required by semantic validation.

Use:

uv run python -m every_eval_ever validate <changed files>
uv run python -m every_eval_ever.check_duplicate_entries <relevant files>

Capture the full output and exit status. Do not rely on Pydantic model construction or validate_file() alone; those can omit semantic checks. If current main and the deployed bot disagree, reproduce both versions when practical and fix toward the current schema without silently degrading data for an old bot.

3. Triage before editing

Group findings by root cause rather than by file. For each group, record:

• affected paths and exact model/result identities;
• local and bot messages;
• whether the issue is mechanical, schema-semantic, or content-semantic;
• the source evidence needed for a correct fix;
• proposed change and confidence.

Inspect content even when the validator omits it. At minimum check suspicious zeroes, score scale and bounds, metric identity, source_data, duplicate overall/subtask aggregates, stable evaluation_id, model identity, answer leakage, and companion pairing. An out-of-range score requires finding the source scale or source value; do not cap, clamp, or round it into validity.

Inspect the raw JSON before constructing an EvaluationLog. The model layer may auto-fill absent deployment keys with unknown, hiding whether the contributor actually supplied additional_details, supplied only one axis, or supplied neither.

4. Research ambiguous metadata

For deployment warnings, apply reference/model-deployment.md to each exact model variant and evaluation run. Determine the two axes independently. Do not infer one from the other, from the developer folder, or from a provider-wide rule.

Search all relevant primary surfaces before choosing unknown: record payload and run config, generating adapter, pinned model card, evaluator methodology, paper and appendix, source repository, and official API/release documentation. Use current web research where facts may have changed, but pin the evidence revision or date relevant to the submitted evaluation.

Batch models only after proving that they share the same evidence. Keep an evidence table with raw model label, canonical model id, both decisions, source URL/revision, and confidence.