Evaluate ML Pipeline

Evaluate ML Pipeline

Score one learner and persist the report. The pipeline, the splitter, and any non-default score are already declared. skore.evaluate is the entry point. Do not hand-roll cross_val_score, cross_validate, classification_report, or metric prints.

A SkrubLearner does not implement sklearn's fit(X, y). cross_val_score raises. Call skore.evaluate(learner, data={...}) and omit splitter=, unless translation.splitter is prefit.

Human-facing prose

Details: setup-workspace references/human_facing_prose.md. Experiment markdown and # comments describe this evaluation. Questions and the close use the same data-science language: not skill ids, G-* names, or the wrapper CLI. <!-- results-embed: … --> is a site marker.

Procedure

1. python -m skore_skills status. If status.setup.pending is non-empty and status.skills.setup-ml-project is true, load setup-ml-project and stop. Do not start this skill. When it returns, continue. Do not load it again on this turn. If that skill is not installed, name the pending pieces in one line and stop. Do not invent git init, scaffold, or env init. If status.setup.env or status.setup.workspace is declined, stop in one line. A declined git or editable is not asked again; continue. Then python -m skore_skills frame show without --revise. Anything other than proceed loads frame-ml-problem and stops. translation null → stop. Do not write an evaluation call.
2. History-dependent pipeline (backward shift, lag, rolling window, target shift, or a join with side history): if tests/smoke/test_<stem>.py is missing or pytest is red, stop. Route to build-ml-pipeline. Do not write skore.evaluate. Documented n/a only when there is no history-dependent step.
3. python -m skore_skills evaluate consent --stem <stem>.
   • ask, and this turn has not already answered Evaluate from build-ml-pipeline: quote JSON context and present Evaluate (Recommended) / Modify / Stop. Stop. "Run evaluation" is not consent on a first run.
   • The user already answered Evaluate this turn, or JSON is proceed (a real locator already exists): continue.
   • stop — smoke file missing. Route to build.
4. G-SKORE-MODE. Read status.policy.skore_mode. If it is already set, keep it. If unset, ask local (recommended) / hub / mlflow, then python -m skore_skills policy set skore_mode <mode>. local: mkdir reports (exist_ok); do not write reports/README.md. hub or mlflow: do not create reports/. Then load add-python-package if installed; it runs env add-skore --mode <mode> --execute. Do not spell skore[...] here. Do not run env add here. Constructors: references/g_skore_mode.md. A switch or a migration loads sync-ml-reports when that skill is installed.
5. Emit Pre-flight, then 1–3 sentences: this is local full-dataset evaluation with the locked scheme, report metrics, the skore checks, and project.put. The checks are computed in this run and stored with the report. When translation.splitter is prefit, say the model is fitted on the training table and scored on the shipped test table. Name the stem, the fold count when known, and experiments/<stem>.py plus scratch/results/<stem>/. Timing depends on rows, folds, the learner, and the checks. Do not invent minutes. If consent is still pending, preview and stop.
6. Write the call in experiments/NN_*.py only. See the call shapes below. python -m skore_skills style after the edit. The experiment ends at report.checks.summarize(), project.put, and a bare report.
7. After put has stored the report, copy templates/snapshot.py to scratch/results/<stem>/snapshot.py and run it. Then End of turn. Do not add snapshot writes to the experiment file.

Every Python probe goes to scratch/<ts>_<short>.py and runs with the composed-dev Python from env verify. No inline python -c. No warnings.filterwarnings unless the user asks. An unconfirmed signature stays unwritten. End that probe with BLOCKED: <class> signature needs an API lookup that cannot run this turn (<why>). A cache hit is a satisfied lookup.

The call

skore.evaluate in experiments/NN_*.py. Omit splitter= so skore reuses the DataOp cv and split_kwargs, unless translation.splitter is prefit. Passing any other splitter= drops split_kwargs. Omitted splitter= with no DataOp cv is an 80/20 holdout, correct only when translation.report is EstimatorReport and translation.splitter is null. Wiring: references/metadata-routing.md.

When translation.splitter is prefit, fit on the training table only, then score the test table. Confirm fit and evaluate with api get. Do not pass the training table into evaluate. Omitting splitter= here draws a new 80/20 split of the test table. Do not concatenate the two tables.

learner = build_learner()
fitted = learner.fit({"table": train_df})
report = skore.evaluate(
    fitted,
    data={"table": test_df},
    splitter="prefit",
)

An estimator whose fit is (X, y) uses the same rule:

fitted_model = LogisticRegression().fit(X_train, y_train)
report = skore.evaluate(
    fitted_model, X_test, y_test, splitter="prefit"
)

X / y and data stay mutually exclusive.

• SkrubLearner — skore.evaluate(learner, data={...}). Keys are the skrub.var names. Interop: references/skrub_interop.md.
• An estimator whose fit is (X, y) — skore.evaluate(estimator, X, y). Still omit splitter= when the locked cv is already the evaluation scheme. The prefit call above is the exception: the estimator is already fitted, and X and y are the test table.

The cv on mark_as_X is KFold, GroupKFold, or the date-based class from build. If translation names a cv and the marker has none, return to build-ml-pipeline. Do not wire split_kwargs here. Empty split_kwargs plus a possible group column → return to build-ml-pipeline. Do not default to KFold. translation.splitter prefit is not a cv on the marker.

No Stratified* for class imbalance. It compresses across-fold variance.

The headline is translation.metric. A name on the skore default list in build-ml-pipeline needs no scorer. Any other name must already be .skb.with_scoring(...) on the prediction DataOp. If it is not, return to build-ml-pipeline before skore.evaluate. Do not call report.metrics.add. When the scorer is attached, that name is a row in report.metrics.summarize().frame(). skore.evaluate has no scoring= argument (references/custom-metrics.md). If the name is attached and still missing from that frame, the predictor class is wrong: it must be the mixin, then BaseEstimator (RegressorMixin or ClassifierMixin first). Return to build-ml-pipeline. BaseEstimator alone, and BaseEstimator before the mixin, both fail.

An extra check the user asks for after the lock: references/custom-checks.md. Subclass skore.Check at module level in experiments/NN_*.py, then report.checks.add(...) after evaluate and before summarize. add extends SKD checks. Do not invent a check. Do not register one from audit/. Confirm Check and checks.add with api get.

Every evaluation calls report.checks.summarize() with no arguments after evaluate and after any checks.add, then project.put. That stores the check results on the report. Do not pass fast_mode or ignore. Confirm checks.summarize with api get.

report = skore.evaluate(...)
report.checks.summarize()
project.put(STEM, report)
report

Escalate past evaluate only when the dispatcher is too coarse (references/reports.md): EstimatorReport for one held-out fit, CrossValidationReport for per-fold artifacts. Holdout uses EstimatorReport. This loop scores that one learner with one skore.evaluate, one report.checks.summarize(), and one project.put.

CV is necessary but not sufficient for any pipeline with history-dependent features. skore.evaluate materializes the graph once with one env-dict and splits indices. The smoke test exercises a fresh env-dict at predict time. A passing smoke test is still required before the caller may flip status to done. Do not edit JOURNAL.md History or the design-note Status to done from this skill.

skore.evaluate(...) and project.put(...) live only in experiments/NN_*.py. A scratch probe, an audit file, or a notebook that calls them duplicates the report under the same key. Read a stored report with project.summarize() then project.get(id). get(key) raises KeyError because get is by id. Do not re-run evaluate to paper over that.

Plots that skore does not already draw load plot-ml-figure when that skill is installed.