Benchmark Runner

## Automated Benchmark Results — `{_skill_name}` 🟡 In Progress

### Run Metadata

| Field | Value |
|---|---|
| **Eval ID** | `{id}` |
| **Run date** | {YYYY-MM-DD HH:MM UTC} |
| **Model** | `{CURRENT_MODEL_NAME}` |
| **Skill version** | `{_skill_sha[:7]}` |
| **Phase** | 1 of 2 complete — Agent A (with skill) finished |

Agent A has completed. Agent B (without skill) will run in the next scheduled window (~5 h).
Results will be updated here automatically.

<!-- BENCHMARK_PARTIAL: {"eval_id":"{id}","model":"{CURRENT_MODEL_NAME}","skill_sha":"{_skill_sha}","issue_number":{N},"blinded_map":{_blinded_scoring_map},"agent_a_asset_url":"{asset_url}","run_date":"{ISO8601}","tokens_a":{tokens_a}} -->

✓ Phase 1 complete — Agent A finished for {eval_id} ({model})
  • Output archived: {asset_url}
  • Partial comment: {comment_url}
  • Tokens used: {tokens_a:,}

NEXT STEP — Phase 2 (Agent B + scoring):
  • If running as a scheduled routine: nothing to do. The next scheduled
    invocation (≥5 h from now, after the rolling usage window resets) will
    detect this partial state automatically and run Phase 2.
  • If running manually: re-invoke this skill any time. It will
    detect the BENCHMARK_PARTIAL marker on issue #{N} and run Phase 2 to
    completion.

import re, json
marker_re = re.compile(r'<!-- BENCHMARK_PARTIAL: ({.*?}) -->', re.DOTALL)
m = marker_re.search(comment_body)
state = json.loads(m.group(1))
# state keys: eval_id, model, skill_sha, issue_number, blinded_map,
#             agent_a_asset_url, run_date, tokens_a

# Send stdout (the eval-case JSON) to the file and stderr (warnings such as
# the >100 KB bundle notice) to a separate log. Do NOT use `2>&1` here — it
# would merge warning lines into the JSON file and break the json.load below.
python3 _automation/benchmark-runner/scripts/get_next_eval.py \
  --model {state["model"]} \
  --priority-issue {state["eval_id"]} \
  > /tmp/eval_case_{id}.json 2>/tmp/eval_case_{id}.log

mkdir -p /tmp/benchmark_{id}/agent_A/output_A

# Option A — gh CLI:
gh release download benchmark-results --repo RConsortium/pharma-skills \
  --pattern "benchmark_agent_a_{eval_id}.zip" --dir /tmp/benchmark_{id}/

# Option B — curl (release assets are public for public repos; token only needed for private):
curl -L "{agent_a_asset_url}" -o /tmp/benchmark_{id}/benchmark_agent_a_{eval_id}.zip

# Then unzip:
cd /tmp/benchmark_{id} && unzip -q benchmark_agent_a_{eval_id}.zip

mkdir -p /tmp/benchmark_{id}/agent_B/output_B

with open("/tmp/benchmark_{id}/agent_B/prompt_B.txt", "w") as f:
    f.write(eval_case["_prompt_b"])

export CLAUDE_CODE_MAX_OUTPUT_TOKENS=64000
cd /tmp/benchmark_{id}/agent_B && \
  cat prompt_B.txt | claude -p \
  --allowedTools "Bash,Read,Write,Edit,Glob" \
  --output-format json > agent_B_run.json 2>&1

d = json.load(open("/tmp/benchmark_{id}/agent_B/agent_B_run.json"))
u = d.get("usage", {})
tokens_b = u.get("input_tokens", 0) + u.get("cache_creation_input_tokens", 0) + u.get("output_tokens", 0)
is_error_b = d.get("is_error", False)

python3 _automation/benchmark-runner/scripts/record_run_result.py \
  --eval-id {state["eval_id"]} --model {state["model"]} \
  --status completed --tokens-b {tokens_b}

mkdir -p /tmp/benchmark_{id}/scoring/candidate_1 /tmp/benchmark_{id}/scoring/candidate_2
# blinded_map: {"candidate_1": "output_B", "candidate_2": "output_A"} (or reversed)
cp -r /tmp/benchmark_{id}/agent_{X}/output_{X}/. /tmp/benchmark_{id}/scoring/candidate_1/
cp -r /tmp/benchmark_{id}/agent_{Y}/output_{Y}/. /tmp/benchmark_{id}/scoring/candidate_2/

## Automated Benchmark Results — `{_skill_name}`

### Run Metadata

| Field | Value |
|---|---|
| **Eval ID** | `{id}` |
| **Run date** | {YYYY-MM-DD HH:MM UTC} |
| **Model** | `{model}` |
| **Skill version** | `{skill_sha[:7]}` |
| **Triggered by** | Scheduled |

### Scorecard

| Metric | With Skill | Without Skill |
|---|---|---|
| **Score** | {score_A} ({pct_A}%) | {score_B} ({pct_B}%) |
| **Assertions** | {pass_A} Pass · {partial_A} Partial · {fail_A} Fail | {pass_B} Pass · {partial_B} Partial · {fail_B} Fail |
| **Skills loaded** | 1 | 0 |
| **Execution time** | {time_A} min | {time_B} min |
| **Token usage** | {tokens_a} | {tokens_b} |
| **{Key Metric 1}** | {value_A1} | {value_B1} |
| **{Key Metric 2}** | {value_A2} | {value_B2} |

### Key Observations

- {2-4 bullet points comparing both agents}

### Verdict

{1-2 sentence overall verdict}

---

## Technical Details & Artifacts

<details>
<summary>View Assertion Breakdown, Code Artifacts, and Logs</summary>

### Assertion Breakdown

| Assertion | With Skill | Without Skill |
|---|---|---|
| {assertion_text_1} | {Pass/Partial/Fail} | {Pass/Partial/Fail} |

### Debugging Information

#### Agent A (With Skill)
- **Total Turns:** {num_turns from agent_A_run.json}
- **Errors/Retries:** {is_error value, or "None"}

#### Agent B (Without Skill)
- **Total Turns:** {num_turns from agent_B_run.json}
- **Errors/Retries:** {is_error value, or "None"}

### Detailed Artifacts

**Agent A Output:** [Download Agent A Archive]({agent_a_asset_url})

#### Agent A (With Skill)
{Key output files — .R, .json, text summaries}

#### Agent B (Without Skill)
{Key output files}

</details>

---
<!-- BENCHMARK_COMPLETE: {"eval_id":"{id}","model":"{model}","skill_sha":"{skill_sha}"} -->
*Posted automatically by `benchmark-runner` · Repo: https://github.com/RConsortium/pharma-skills*

✓ Phase 2 complete — full benchmark posted for {eval_id} ({model})
  • Score: With Skill {pct_A}% · Without Skill {pct_B}%
  • Comment: {comment_url}
  • Tokens — A: {tokens_a:,} · B: {tokens_b:,}

EVERY ROUTINE INVOCATION:

  Phase Detection
    │
    ├─ BENCHMARK_PARTIAL found (no BENCHMARK_COMPLETE for same eval+model) ──► Phase 2
    │     Step 5: load state from comment + restore Agent A output
    │     Step 6: run Agent B (without skill)
    │     Step 7: score blinded
    │     Step 8: format full report
    │     Step 9: post full results comment (with BENCHMARK_COMPLETE marker)
    │     EXIT
    │
    └─ No partial found ──► Phase 1
          Step 0: R pre-flight
          Step 1: get_next_eval.py → if UP_TO_DATE, EXIT
          Step 2: run Agent A (with skill)
          Step 3: archive + upload Agent A output
          Step 4: post partial comment (with BENCHMARK_PARTIAL marker + state JSON)
          EXIT