Analyze Datagen Campaign Summary

analyze-datagen-campaign-summary

Turn a datagen campaign's prose tracker (e.g. ~/Documents/experiments/{active,complete}/<campaign>/tracker.md, whose per-dataset status lives in sentences, not columns) into a clean, auditable per-dataset table/CSV with computed metrics. Built for the qwen3.5-122b-tt 32k campaign but campaign-agnostic — swap the dataset list.

Target columns: Datagen Model | Task Source | Status | N Trials Completed | Mean Turns / Trace | Mean Tok / Trace | Mean Reward | HF Repo Link.

Why this skill exists (the two traps)

1. The tracker rarely has reward / turns / tokens. Trackers record throughput (gen tok/s) + row counts in prose; mean reward, mean turns, and mean tokens are almost never written down. They must be COMPUTED from the uploaded HF trace datasets.
2. Status in prose is stale/ambiguous (a "RUNNING" row that actually finished; a "rescued" row with no clean repo name). Ground-truth Status against HF: if the trace repo exists with rows → COMPLETED (and its row count IS N Trials Completed); otherwise fall back to the tracker's status hint.

The data model (uploaded OT-Agent trace dataset)

Each row of penfever/<slug>-<model>-traces is one trial/trace: {conversations: [{role,content},…], agent, model, model_provider, date, task, episode, run_id, trial_name, result, verifier_output}.

• N Trials Completed = row count of the dataset (one row = one completed trace).
• Mean Turns / Trace = mean count_turns(row) = mean number of conversation messages (canonical definition in scripts/analysis/utils.py; total messages, not just assistant turns — state it in the notes).
• Mean Tok / Trace = mean Qwen3-8B token length of the whole conversation — plain method from the analyze-dataset-token-length skill: tokenizer(extract_conversation_text(row), add_special_tokens=False). Tokenizer is always Qwen/Qwen3-8B for these datasets (their trace-dataset convention), regardless of the served model name (model field is hosted_vllm/<numeric-id>, not a usable tokenizer).
• Mean Reward = Harbor-flat mean of result via mean_reward_per_trial semantics: extract_reward each row (parses the result string, e.g. "0.0" → 0.0), missing/non-numeric counts as 0.0. This matches harbor's <done>/<total> Mean: accuracy exactly — do NOT drop nulls or the number won't reconcile.

Reuse the canonical tools (do NOT reinvent)

/Users/benjaminfeuer/Documents/OpenThoughts-Agent/scripts/analysis/utils.py:

• extract_conversation_text(record) — conversation → full text to tokenize (handles messages/conversations).
• count_turns(record) — turns.
• extract_reward(record) — parses result → float|None. mean_reward_per_trial(rows) — Harbor-flat mean.
• load_hf_trace_dataset(repo_id) — non-streaming loader (fine for small repos; see disk note for large ones).

Token-length details (methods, tokenizer, the metadata-confound trap) → the analyze-dataset-token-length skill. If you'd rather source Mean Reward + trials from the job logs instead of the HF dataset (e.g. the repo was never uploaded), the analyze-job-history-iris skill's analyze_iris_harbor_job.py sidecar carries the harbor Mean: + non_empty_trials per job — but the uploaded dataset is the more reliable ground truth for a COMPLETED row.

⚠ Handling the LARGE trace datasets (disk + bandwidth)

Some campaign datasets are big (tens of thousands of rows / hundreds of MB / dozens of shards). Full load_dataset caches the whole parquet to ~/.cache/huggingface → can blow local disk (a full disk bricks the supervisor — see the disk-health rule in supervisor-init). So:

• STREAM (load_dataset(repo, split="train", streaming=True)) and accumulate in ONE pass — disk stays bounded (shards read on the fly, not cached whole).
• Point HF_HOME / HF_DATASETS_CACHE at the scratchpad and df -h / before launching; bandwidth is unavoidable (the conversations column is the bulk, needed for both turns and tokens) but streaming avoids the disk blowup.
• Batch the tokenizer (e.g. 128 texts) rather than per-row; TOKENIZERS_PARALLELISM=false to avoid the fork-after-tokenizer deadlock when parallelizing.
• Parallelize across datasets with a ProcessPoolExecutor (≈5 workers) — CPU-bound tokenization scales well; each worker streams its own datasets. Checkpoint per-dataset to JSONL so a crash/interrupt resumes instead of recomputing the expensive large ones.