Kraken Lab Experiment

Step 4: Score and judge

One command produces the honest scorecard and the mechanical verdict:

kraken lab score --session latest --experiment momentum-1 -o json 2>/dev/null

• source: what the session traded against — {kind:"replay", dataset, speed} or {kind:"live", symbols}, derived from the session's contract, not asserted.
• anchor + components[]: the stop totals and the explain-pnl waterfall, carried verbatim — this output can never disagree with kraken explain pnl.
• metrics: return_pct, max_drawdown, max_drawdown_pct, fees, friction, turnover_pct, hit_rate, profit_factor, slippage_sensitivity — every null is an honest "not derivable", never zero (a buys-only DCA run has no closed round trips, so hit_rate and profit_factor read null by design, not by breakage).
• verdict.pass and verdict.checks[]: one check per frozen criterion with threshold, observed, pass. An unknown observation fails its check.
• caveats[]: read them; a verdict with caveats is reported with them.

Digest for the loop log:

kraken lab score --session latest --experiment momentum-1 -o json 2>/dev/null \
  | jq '{session, experiment, pass: .verdict.pass,
         checks: [.verdict.checks[] | {criterion, threshold, observed, pass}],
         caveats}'

The retry cues are in the error envelopes themselves: "still recording — stop it with 'kraken session stop' first" means go back to Step 3; "was aborted before it stopped … only a stopped session can be scored" means that session is dead evidence — start a fresh one; "experiment '…' not found; run 'kraken lab new …' first" means Step 1 never happened in this scope.

Step 5: Conclude — or run again

The verdict is mechanical; the conclusion is yours to report, not adjust:

• pass: state the hypothesis, the checks that carried it, and every caveat. One passing run is evidence, not proof — follow the plan to its end before promoting anything.
• fail: name the failing checks (threshold vs observed) and what the waterfall says the money actually went to (fees? drawdown? never traded?). A failed experiment is a completed experiment.

Promotion: when lab next says plan_complete, assemble the promotion evidence block from kraken lab compare and hand off to the kraken-paper-to-live skill. Its gate is mechanical: at least 2 passing runs with a passing live run among them — replay verdicts count only with their lookahead caveat attached. Two replay passes never promote.

More runs: repeat Steps 2–4 (each start allocates the next ordinal), then read every session side by side:

kraken lab compare momentum-1 -o json 2>/dev/null

One verdict column per session (sessions[].verdict), plus pass_count/total and a named error column for any run that cannot be scored (a still-running run says so — stop it first). Only the verdicts compare across runs: they cover different market windows, so the metric rows are per-session context. Never average scorecards across runs into a blended number the CLI did not produce — compare deliberately never aggregates.

Cold-context verdict reproduction

Any context — including one that never saw the sessions — reproduces a verdict from disk alone:

kraken lab show momentum-1 -o json 2>/dev/null          # the sealed criteria
kraken lab score --session s1 --experiment momentum-1 -o json 2>/dev/null

Same files, same exact-Decimal math, same verdict, byte for byte. If the two disagree with a previously reported verdict, something on disk changed — lab show tells you whether it was the spec.

Running under /loop

Invoke as /loop with this skill and the experiment name. Each firing:

1. kraken lab show <exp> — missing → freeze (Step 1, with --session entries) and return.
2. kraken lab next <exp> -o json and act on its state. If it refuses with a no-run-plan validation error (an experiment frozen before run plans existed), drive the manual state table above instead. start → run its command; running → drive/wait, then stop + score (Steps 3–4) and log the digest; plan_complete → kraken lab compare, conclude (Step 5), and stop the loop with the per-session verdicts (and the promotion evidence block, if concluding toward promotion) as the final report.

Idempotent by construction: a firing that finds nothing to advance does nothing. A killed loop resumes at the same step.

If you hit a mismatch between what you are trying to do and the CLI's interface or responses — including a mismatch between this skill and the installed CLI version's contract — feel free to submit feedback with kraken feedback.