Datagen Launch

datagen-launch

A datagen job stands up a managed vLLM endpoint serving a teacher model, runs Harbor agent rollouts against a set of tasks in Daytona sandboxes, and uploads the resulting trajectories to an HF dataset repo. There is no model checkpoint — the artifact is the trace dataset.

This skill is cluster-AGNOSTIC — the shared flow, flags, and gotchas. Defer everything cluster-specific (ssh / login node, repo path, conda env, JSC login-node pre-download, which datagen_config (vllm-serve) YAML matches the cluster's GPUs and the model) to .agents/ops/<cluster>/. Do NOT inline cluster-specific values here.

Sibling skills: datagen-launch-iris (Iris TPU path), datagen-job-cleanup (post-run upload + disk cleanup), datagen-reduce-dataset-snapshots (shrink a task dataset's Daytona snapshot count under cap).

Required info (ask if missing)

1. Dataset / tasks — an HF parquet repo id to extract tasks from (e.g. DCAgent/r2egym_sandboxes), or an already-extracted local tasks dir.
2. Teacher model + operating point — determines the datagen_config (vllm-serve) YAML and the trace_harbor_config context-length YAML. Pick the cluster-matched config from .agents/ops/<cluster>/; don't change configs unless asked.
3. Target HF repo — --trace_target_repo (default penfever/ or DCAgent2/ org per the series).

The 2-step flow

Step 1 — extract tasks from the parquet (writes one task dir per row):

python -m scripts.datagen.extract_tasks_from_parquet \
  --parquet <hf-parquet-repo-or-local> \
  --output_dir <tasks dir> \
  --on_exist overwrite

Step 2 — submit the datagen job:

🚧 SUBMIT FROM THE REPO DIR WITH DCFT SET — the sbatch WORKDIR guard hard-fails otherwise. Before launching/relaunching, cd <repo> && export DCFT=$PWD (on Jupiter: /e/scratch/jureap59/feuer1/OpenThoughts-Agent). The generated universal_tracegen.sbatch / universal_taskgen.sbatch resolve WORKDIR from DCFT_PRIVATE → DCFT → $PWD; submitted from $HOME/a scratch subdir with DCFT unset, the guard detects the wrong dir (missing hpc/shell_utils/triton_cache.sh marker) and exit 1s immediately with FATAL: WORKDIR=... is not the OpenThoughts-Agent repo root. If you see that FATAL, cd to the repo, export DCFT=$PWD, resubmit.

python -m hpc.launch \
  --job_type datagen \
  --datagen_config <vllm-serve cfg>.yaml \
  --trace_harbor_config hpc/harbor_yaml/datagen/<ctx>.yaml \
  --tasks_input_path <tasks dir> \
  --trace_target_repo <hf repo> \
  --daytona_api_key "$DAYTONA_DATA_API_KEY" \
  --num_nodes 1 \
  --trace-n-concurrent N

CRITICAL gotchas (get these wrong → silent total failure)

• --daytona_api_key "$DAYTONA_DATA_API_KEY" is MANDATORY. Must be the datagen-org Daytona key, NOT the default RL-org key. The RL-org key rejects declarative sandbox builds, so every trial instant-fails (job looks "running" but produces zero real trajectories). Source the key from your secrets env (see CLAUDE.md "Datagen Daytona Key (CRITICAL)" / .agents/ops/<cluster>/). Do NOT echo the value. (Exact env-var name DAYTONA_DATA_API_KEY — verify against your local secrets env / ops doc.)
• harbor_config needs auto_snapshot: true. Attaches the prebuilt Daytona snapshot instead of doing a slow per-trial declarative build. Use a hpc/harbor_yaml/datagen/<ctx>.yaml that sets it.
• Do NOT export DAYTONA_TARGET. Leaving it set misroutes the run.
• Per-job effective concurrency caps at ~100–128 (per-job ceiling, not aggregate contention). --trace-n-concurrent (and the config's seqs / max_num_seqs) above ~128 is wasted. Parallel datagen jobs are fine; over-fanning a single job is not.
• Daytona snapshot org cap is a HARD, server-side limit (e.g. 40 or 60 per org depending on org). Never raise the cap or convert its ValueError/SnapshotCapExceeded into a warning. If a dataset overflows: clean stale/MISSING snapshots, shrink the dataset's snapshot footprint with datagen-reduce-dataset-snapshots, or reuse a dataset whose snapshots already exist (0 new). If still blocked, ask the user. Snapshots are keyed by sandbox environment, not dataset — they're shared, so do NOT reclaim them per-run. Clean stale at the cap (autonomous): python scripts/daytona/daytona_snapshot_manager.py --api-key-env DAYTONA_DATA_API_KEY --delete-stale --yes — at the stale threshold defined in .agents/projects/daytona/daytona.md § "How to clean stale snapshots" (GT — don't restate the value) (audit-only without --delete-stale; deletes only idle/unprotected harbor__* envs). Full procedure + caveats → .agents/projects/daytona/daytona.md.
• FD-exhaustion / libuv SIGABRT at ~1h on boundary-hugging datasets: model_info.max_input_tokens does nothing when enable_summarize=false (our RL/trace default) — nothing truncates the growing multi-turn prompt → VLLMValidationError: 32769 input tokens overflow. Lowering the token budget is INERT. Real levers: (a) drop n_concurrent_trials (e.g. 900→500), then (b) disable litellm's async logging worker on the hosted_vllm path (its clear_queue exceeded max_time backlog leaks FDs ∝ overflow_rate × concurrency).
• Reward-realness caveat: LLM-judge datasets can score all-0.0 if the judge API key is invalid/unplumbed — inspect the reward distribution at consolidation, not just avg turns.