Benchflow Experiment Review

Benchflow Experiment Review

Use this skill to decide whether a Benchflow run trial is clean enough to publish or whether a Benchflow code change is safe to use for new experiments. The standard is: a clean sandbox with only task-needed resources, every agent behavior logged, no verifier leakage, and a final score or healthy failure.

Portability

This skill is intentionally harness-portable. Install or copy the entire benchflow-experiment-review/ directory into the active harness's skill root, keeping SKILL.md, scripts/, references/, evals/, and optional agents/ metadata together. Use the same review procedure regardless of whether the harness loads skills from .claude/skills, .codex/skills, OpenHands/Gemini/pi-agent skill roots, or another compatible SKILL.md directory.

Operating Rule

Do not accept aggregate counts alone. Enumerate the intended matrix by task_id, harness, model, skill mode, trial id, sandbox type, run root, and source/ref. Mark each slot as healthy, missing, duplicate, stale, or unhealthy.

Healthy run outcomes are:

• pass: agent completed and verifier produced a valid score.
• fail: agent completed incorrectly and verifier produced a valid score.
• normal_timeout: agent genuinely ran, timed out, and still produced a complete ACP trajectory, complete LLM trajectory, and reward/scoring metadata.

Infrastructure failures are not healthy failures. A stalled Docker daemon, Daytona transport failure, missing trajectory/acp_trajectory.jsonl, missing trajectory/llm_trajectory.jsonl, malformed or empty trajectory files, missing reward, missing timing, missing token usage for new data, or verifier crash is unhealthy until rerun or explicitly quarantined.

Hard Trajectory Gate

For every current BenchFlow model trial, both trajectory files plus the trainer-facing rollout row are mandatory:

• trajectory/acp_trajectory.jsonl: ACP/tool trace with agent-side events.
• trajectory/llm_trajectory.jsonl: provider LLM request/response trace with token usage evidence.
• results.jsonl: Verifiers / Prime-RL-shaped rollout row derived from the healthy LLM trajectory.

Do not treat ACP alone as sufficient. Do not treat llm_trajectory.jsonl alone as sufficient. Do not treat aggregate result.json fields as a substitute for either trajectory or results.jsonl. A trial with a scored reward, token counts, or a plausible final answer is still unhealthy if either required trajectory file or results.jsonl is missing, empty, truncated, unparsable, or usage-only without recoverable request/response evidence.

One explicit evidence-only exception exists for native ACP subscription auth, where provider HTTP capture is unavailable by construction. With --allow-native-subscription-without-llm, the validator may accept a missing LLM trajectory only when agent_result.usage_source == "agent_native_acp", ACP events and native token usage are healthy, timing and reward are complete, and results.jsonl marks the rollout completed but not training-ready with reason missing_healthy_structured_llm_trajectory. This exception never creates training data, never accepts an empty/malformed present LLM trajectory, and is off by default.

results.jsonl must be reviewed as a training artifact, not just a sidecar. For a healthy model rollout it should contain a parseable row with info.training_ready == true, non-empty prompt, completion, and trajectory, positive token usage, reward/score metadata, task identity, and valid OpenAI-compatible message roles. Token usage may be split into input/output fields or represented by a provider total_tokens value when the provider does not expose a split. If assistant tool_calls appear, the row must carry non-empty tool_defs or tools. Errored, verifier-failed, partial, malformed, or missing-LLM rollouts must fail closed with training_ready == false and an error/training-ready reason instead of being accepted as SFT-ready.

Run the bundled validator before deeper manual review:

python .agents/skills/benchflow-experiment-review/scripts/validate_run_artifacts.py /path/to/rollout-or-jobs-root --json

The validator exits non-zero when any rollout is unhealthy. It is a deterministic fast path, not the whole audit: after it passes, continue with skill-loading, no-skill leakage, verifier-isolation, reward-hacking, and capability-attribution checks below. Oracle/reward-only lanes do not produce model LLM trajectories and must not be mixed into model-run publishability counts; if reviewed, report them as separate non-model evidence.

Completed Trial Review

For each trial, first locate the authoritative run artifacts. Prefer repo validators or existing scripts when available, then inspect raw files. Required evidence includes:

• Run config: task id, harness, model, skill mode, trial id, source commit/ref, sandbox backend, timeout, provider settings, and output paths.
• Trajectories: both trajectory/acp_trajectory.jsonl and trajectory/llm_trajectory.jsonl, parseable from first event/model request through final answer, failure, or timeout.
• Metadata: start/end timestamps, elapsed duration, token usage, tool usage, provider/model response metadata, sandbox metadata, and error fields.
• Results: verifier output, reward/score, result status, rollout-level results.jsonl, and enough provenance to connect the score and training row to the exact trajectory.

Review the trajectory for meaning, not just existence:

• The agent received the intended task prompt and no verifier-only information.
• Tool calls and observations form a plausible task attempt.
• Skill-enabled trials show native skill loading or injection evidence for that harness; no-skill trials do not expose those skill resources.
• For harness-specific skill catalog recovery, load references/harness-skill-catalog-sop.md or run scripts/extract_harness_skills.py against trajectory/llm_trajectory.jsonl. Treat the script as a fast path: if it returns unknown, skill_count: 0, or manual_review_required: true, fall back to the SOP's manual procedure before concluding that no skills were available.
• Check task_skills_loading: with-skill trials should report 1, meaning every expected task-specific skill was recovered in the startup skill catalog; without-skill trials should report 0, meaning task-specific skills were not injected into the agent's startup catalog.
• Timeout trials show real progress or attempts before timeout and still have complete timing, token, ACP trajectory, LLM trajectory, and verifier result metadata.
• The final answer/result and verifier score refer to the same task workspace and trial.
• Failures and timeouts are attributable to agent capability, not missing critical inputs. Audit the task spec, environment variables, API keys, mounted assets, package/runtime availability, network access, compute/memory, and sandbox rendering before labeling a failed or timed-out run as healthy.

Reject or quarantine the trial if any of these appear:

• Missing trajectory/acp_trajectory.jsonl.
• Missing trajectory/llm_trajectory.jsonl.
• Missing rollout-level results.jsonl.
• Empty, truncated, or unparsable trajectory files.
• llm_trajectory.jsonl has no real provider request bodies, no provider response bodies, or no provider token usage in response metadata.
• results.jsonl lacks a Prime-RL-compatible row, marks an unhealthy rollout as training-ready, misses prompt/completion/trajectory/provider token usage, or exposes assistant tool calls without tool definitions.
• Empty transcript, agent never launched, or only setup logs.
• Missing token usage/timing/tool usage metadata for newly generated data.
• The agent lacked required task information, required skills, API keys, credentials, assets, runtime dependencies, or enough compute/resources to make a fair attempt.
• Verifier ran before the agent finished, leaked solution hints, or wrote into agent-visible resources.
• Reward appears hacked, copied from hidden files, or inconsistent with the visible task state.
• Path/root mismatch between task config, sandbox cwd, trajectory paths, and result paths.
• Provider, Daytona, Docker, or filesystem errors that prevented a normal agent attempt.

Integration Tests After Code Changes

Before using a changed Benchflow version for production experiments, run an end-to-end integration check that proves the changed code preserves artifact shape and sandbox behavior.

1. Run focused unit/static checks for changed modules first. Use the repo's normal commands when appropriate, such as targeted pytest, ty, and ruff.
2. Pick representative canary tasks covering with-skill and no-skill modes, success/failure/timeout if feasible, and any changed subsystem.
3. Run the same canaries through Daytona and VM Docker with the same task, harness, model, skill mode, trial id pattern, timeout, and provider settings.
4. Compare artifact schema and semantics, not exact model wording: trajectory files (acp_trajectory.jsonl and llm_trajectory.jsonl), rollout/job results.jsonl, token usage, timing, tool usage, result status, verifier output, and source provenance must be equivalent.
5. Exercise path/root behavior explicitly. Check task root, sandbox cwd, mounted resources, result paths, locked paths, and any previous root-path regression case.
6. Verify skill loading. With-skill runs must show the harness-specific native skill metadata or prompt injection pattern; no-skill runs must not expose skill files or descriptions.
7. Verify verifier isolation and environment hardening. The verifier starts only after agent exit or timeout, cannot leak solution data into the agent phase, and records score provenance against the completed trajectory. Confirm the anti-reward-hacking invariants in references/verifier-hardening-checklist.md: network off by default (proxied and allowlisted when needed), git history scrubbed past the base commit, no answer/hidden-test/scorer files in agent-readable paths, empty patch fails while golden patch passes, and agent test edits are reset before grading.
8. Scale only after the canary passes. Increase concurrency gradually and keep each VM below the Docker/container pressure that makes the daemon unstable.

The pass condition is schema and lifecycle parity between Daytona and Docker: both sandboxes should render the same task-needed resources, log the same classes of agent behavior, and emit complete metadata for the same logical trial outcome.