Kermt Pretrain Scratch

Workflow

Let $KERMT_REPO be the path to your kermt repo checkout.

1. Pre-flight: ensure container + system probe (same as kermt-continue-pretrain step 1). Refuse to proceed if check_system reports gaps.

2. Compute run directory.

   RUN_DIR=$KERMT_REPO/runs/pretrain-scratch_$(date -u +%Y-%m-%dT%H-%M-%SZ)

3. Validate the corpus (no ckpt to validate, so this is the only input check):

   "$SKILL_DIR/scripts/kermt_container.sh" run --data <user-csv> -- \
       "python /skill/scripts/check_data.py --mode pretrain --csv /data/<basename>"

   Abort on ok: false.

4. Prepare the data — no vocab pass-through (we want fresh vocab from corpus):

   "$SKILL_DIR/scripts/kermt_container.sh" run --data <user-csv> --run-dir $RUN_DIR -- \
       "python /skill/scripts/prepare_data.py --mode pretrain \\
            --csv /data/<basename> --out /runs/data \\
            [--val-csv /data/<val-basename>] [--val-frac 0.1] [--seed 0]"

   Outputs land at $RUN_DIR/data/prepare_data.json with vocab_source: "built_fresh".

5. Estimate runtime + warn loudly. This is critical for pretrain-from-scratch:

   • "Pretraining from scratch is days-scale even on multi-GPU; the released KERMT checkpoints were each trained on millions of molecules for hundreds of GPU-hours. If you mainly want to leverage existing knowledge for a downstream task, consider kermt-continue-pretrain from a released ckpt instead, which converges in hours instead of days."
   • Show the corpus size × epochs × GPU count → estimated wall time.
   • Ask for explicit confirmation unless --yes was given.

6. Launch the runner detached.

   "$SKILL_DIR/scripts/kermt_container.sh" run_detached \\
       --name kermt-pretrain-scratch-<ts> \\
       --run-dir $RUN_DIR -- \\
       "python /skill/scripts/run_pretrain_local.py \\
            --from-scratch --pretrain-target-mode <vocab|cmim|hybrid> \\
            --prepare-manifest /runs/data/prepare_data.json \\
            --out /runs \\
            [--epochs N --batch-size N ...]"

   Note: NO --ckpt flag (the runner refuses if both --from-scratch and --ckpt are given). The runner uses the arch group from config/defaults_pretrain.json to size the model.

7. Report to the user. Always include all of the following — do not omit the TensorBoard line under output-length pressure:

   • Container name + id
   • $RUN_DIR/run.json (the manifest with workflow: pretrain-scratch, from_scratch: true, vocab_check: null, arch from defaults, full cmd_replay)
   • Log file: $RUN_DIR/logs/pretrain_ddp.log
   • TensorBoard: $RUN_DIR/logs/tb (open with tensorboard --logdir $RUN_DIR/logs/tb)
   • Suggest kermt-monitor <RUN_DIR> for progress.

Hard rules

• Never accept a --ckpt flag. From-scratch is exclusive with input ckpt — the runner enforces this; the skill should too.
• Never silently default --pretrain-target-mode. This is a significant architectural choice (vocab = lightweight, hybrid = SOTA). Prompt the user if not given on the CLI.
• Strong warning before launching. From-scratch pretrain is the most expensive workflow. The user needs to know what they're committing to.

Common errors

• --pretrain-target-mode is required when --from-scratch is set → user forgot the mode flag. Prompt.
• --from-scratch is incompatible with --ckpt → user provided both; ask which one they meant.
• defaults_pretrain.json has no arch group → repo state issue (should never happen on a fresh clone); points the user at running kermt-setup again.

What's in the manifest after a from-scratch run

Same reproducibility fields as continue-pretrain (repo.commit, kermt_image, cmd_replay, args_applied), plus:

• workflow: "pretrain-scratch"
• from_scratch: true
• inputs.ckpt: null
• ckpt_symlink: null
• vocab_check: null (not verified — vocab built from corpus is authoritative for from-scratch)
• arch: the values pulled from config/defaults_pretrain.json's arch group (with any future CLI overrides applied).

Replayability

Same as continue-pretrain: cmd_replay is a copy-pasteable command. If ok_to_replay: false, the kermt repo working tree was dirty at launch time — check repo.commit and git checkout it first.