Kermt Monitor

kermt-monitor

Companion skill for any KERMT workflow that runs detached: the three pretrain skills (kermt-continue-pretrain, kermt-pretrain-scratch, kermt-add-cmim-pretrain) plus kermt-finetune. kermt-infer and kermt-embed run blocking by default and don't need this skill, but if a user launches them detached on purpose the monitor still works (the workflow-dispatch in step 4 handles unknown workflows by tailing the most-recent log file in the run dir). Reads the run directory's run.json, queries docker for the container's state, surfaces the latest progress, and either tails or follows the log.

Hardware requirements

None. This skill only reads disk + queries docker; no GPU compute.

Inputs

One of:

• <run-dir> — a positional argument pointing at the directory containing run.json (e.g. runs/continue-pretrain_2026-05-17T10-23Z). Preferred.
• --container <name-or-id> — direct container reference; the skill still reads run.json from the run dir referenced inside the container's inspect output if available, but works degraded-mode without it.

Optional:

• --lines N — number of trailing log lines to print (default 50).
• --follow — stream docker logs -f until ^C. Useful for "watch the loss". Without it, the skill is one-shot and exits.
• --json — emit a structured status report instead of human-readable text. Useful when the parent agent wants to take downstream action.

Workflow

Let RUN_DIR=$1 (or whatever path the user supplies).

1. Locate the manifest.

   MANIFEST=$RUN_DIR/run.json

   Refuse to proceed if it doesn't exist; surface a helpful message pointing the user at the run-dir convention (runs/<workflow>_<ts>/).

2. Parse the manifest (Python helper):

   workflow=$(jq -r .workflow $MANIFEST)
   container_name=...   # not directly in run.json today; the skill that
                        # launched stored it in run.json under
                        # container.name during launch (see below note).
   logs_dir=$(jq -r .logs_dir $MANIFEST)
   image_tag=$(jq -r .container.image_tag $MANIFEST)
   started_at=$(jq -r .started_at $MANIFEST)

3. Query docker for container state.

   docker ps --filter "name=$container_name" --format \
       '{{.ID}}\t{{.Status}}\t{{.CreatedAt}}'

   If absent, fall back to docker inspect $container_name --format '{{.State.Status}} (exit {{.State.ExitCode}})' to see whether the container exited (ok or failed) or was removed (--rm after exit).

4. Find the live log file.

   case "$workflow" in
     continue-pretrain|pretrain-scratch)  LOG=$logs_dir/pretrain_ddp.log ;;
     finetune)                            LOG=$logs_dir/finetune.log ;;
     *)                                   LOG=$(ls -1t $logs_dir/*.log 2>/dev/null | head -n 1) ;;
   esac

   The manifest's workflow field disambiguates pretrain (pretrain_ddp.log) from finetune (finetune.log). Other workflows fall back to the most-recently-modified .log in $logs_dir.

5. Show the latest progress.

   • tail -n $LINES $LOG for the raw recent output.
   • Parse the last few progress lines and surface a human-friendly summary. The format differs per workflow:
     • Pretrain: epoch / step / val_loss

       Current epoch: 12/100  step: 4523/9000  val_loss: 0.832 (best 0.821 @ step 4100)

     • Finetune: fold / epoch / val_<metric> (e.g. val_mae for regression, val_auc for classification — read args_applied.metric from run.json)

       Fold 0  epoch 12/30  val_mae 0.187 (best 0.182 @ epoch 9)

     Wall-clock: 1h 23m since started_at; ETA ~6h remaining.

6. Final test-metrics block (finetune, on completion). If workflow is finetune AND the container has exited cleanly (State.Status=exited, ExitCode=0) AND $RUN_DIR/ckpt/fold_*/test_result.csv exists, parse it and emit a per-task metric table:

   Final test metrics (per task):
     Target              MAE
     HLM_clearance       0.187
     RLM_clearance       0.213
     MDR1-MDCK_efflux    0.241
     solubility_pH6.8    0.156

   The metric column matches args_applied.metric (mae for regression, auc for classification, etc.). For multi-fold or ensemble runs, average across folds/models and note ± std if std > 0. Skip silently if no test_result.csv exists (run incomplete or no test split was emitted).

7. If --follow, stream live logs.

   docker logs -f $container_name

   Wraps until ^C.

8. Stop / cleanup hints (printed at end of one-shot mode):

   To stop:        docker stop $container_name
   To remove:     docker rm $container_name
   To re-run:    `$(jq -r .cmd_replay $MANIFEST)`