Audit ML Pipeline

Audit ML Pipeline

Per-experiment, human-readable narrative of a skore report. The # %% file is the human notebook and is not executed. materialize.py writes the digest and the HTML viewers in one run. Read-only against the skore Project.

Human-facing prose

Details: setup-workspace references/human_facing_prose.md. Audit markdown and # comments describe this report's findings — not the skills framework, the CLI, or the command that produced an output. Questions, replies, and the close narrative use the same data-science language — not skill ids, G-* names, or the wrapper CLI. Trailing locator tokens stay index strings. Do not put API tutorials, version floors, or locator recipes in audit/<stem>.py. <!-- results-embed: … --> is a site marker. Authoring hints stay in this skill. style is ruff only.

Next-step pointers

The implement loop dispatches audit after evaluate and before record-outcome. The digest carries the checks summary and metrics summary. G-AUDIT-FINDING is a separate normalized finding derived from that digest. manage-ml-backlog record-outcome consumes both and never dispatches audit back.

Where things live — visual map

Mnemonic: audit/ is source (in git); scratch/audit/ and stdout are output. Never copy audit/<stem>.py under scratch/audit/. materialize.py is the gitignored agent script. Never commit anything under scratch/audit/.

Read-only contract

The central rule. Surfaced as the first Stop condition below.

Allowed in audit/<stem>.py:

• skore.Project(...) — open the project this experiment wrote to.
• project.summarize() — list (key, id) pairs.
• project.get(id) — load a specific report by id.
• Every report.* accessor.
• Imports from <pkg> (read-only inspection).

Forbidden in audit/<stem>.py:

• skore.evaluate(...) — duplicates the report under the same key and pollutes summarize().
• project.put(...) — same.
• Snapshot writes (write_text of HTML, locator.txt, report.txt) and help() loops in audit/<stem>.py. Those live in scratch/audit/<stem>/materialize.py, which may write scratch/results/<stem>/ and accessors.txt.
• Writes to data/, reports/, or src/<pkg>/. The audit notebook is a viewer.
• Mutation of the loaded report that survives the cell (e.g. monkey-patching skore symbols).

The runner renders every cell's source + last-expression repr + stdout to the digest. A forbidden call surfaces in the digest (as a put row in a later summarize() cell, or as a **error:** section). The contract is visible, not invisible.

Sibling read-only consumers (different output shapes, same discipline): scratch/<ts>_*.py probes. review-ml-experiment reads the digest as text and does not open the Project. See evaluate-ml-pipeline § Stop conditions for the consumer rule.

Stop conditions — read before anything else

• Setup still pending. Run python -m skore_skills status first. If status.setup.pending is non-empty and status.skills.setup-ml-project is true, load setup-ml-project and stop. Do not start this skill. When it returns, continue. Do not load it again on this turn. If that skill is not installed, name the pending pieces in one line and stop. Do not invent git init, scaffold, or env init. If status.setup.env or status.setup.workspace is declined, stop in one line. A declined git or editable is not asked again; continue.
• No report → STOP. Four-way pairing is hard: run python -m skore_skills design consent --stem <stem> (ask / stop → do not audit) + experiments/NN_*.py + smoke pytest passed + a report under that key in the Project. If the report is missing, explain and stop. Do not skore.evaluate / project.put. Route to evaluate-ml-pipeline or model-ml-pipeline. Direct "audit 02" uses this same gate.
• Read-only against the skore Project. See § Read-only contract. Never skore.evaluate(...) or project.put(...) in an audit file.
• project.get(...) is by id, not key. For hub mode, read the id from the URL printed by project.put(): https://…/<workspace>/<project>/<type-plural>/<N> → id is skore:report:<type-singular>:<N> (URL segment is plural; id uses the singular — drop the trailing s, e.g. cross-validations → cross-validation, estimators → estimator). Hardcode REPORT_ID in the audit file — no summarize() traversal needed. For local mode, read the "id" column of project.summarize() for the matching key row. A KeyError from get("<stem>") means the lookup shape is wrong (get is by id), not that the report is missing.
• Symbol from memory is forbidden. Any skore / skrub / sklearn symbol must come from python -m skore_skills api get this turn. Cache hits under scratch/api/skore/<version>/ count (Shape 0); inline memory does not.
• Agent feature missing → STOP and delegate. If ipython / ipython aren't importable, do NOT fabricate audit outputs by writing print() calls as a workaround. Do NOT type pixi add ... / uv add ... yourself — install is owned by add-python-package § Agent feature. Request via agent tools (ruff / ipython / ipykernel) (binary: install / skip); resume only when add-python-package returns "ready".
• Bare expressions, not print(). The runner captures each cell's last bare expression via result.result and renders its repr. Wrapping in print(repr(...)) lands in stdout instead of the output section; mixed and harder to scan. Use bare expressions; statement-only cells (variable binding) are fine.
• One audit file per experiment stem (four-way pairing). No audit_NN_<short_name>_v2.py. When an experiment is re-run, the audit file is overwritten in place — same stem, same audit.
• Executed artifacts go to scratch/audit/<stem>/, NOT into audit/. Durable artifact is audit/<stem>.py; the rendered digest is ephemeral.
• audit/<stem>.py does not write snapshot files. HTML, locator.txt, and help() trees are materialize.py. No writes to data/ or reports/.
• Don't filter warnings in audit cells. No warnings.filterwarnings(...) unless the user explicitly asks — the runner streams cell stderr into the digest and that's signal. See python -m skore_skills style § Stop conditions.
• Harness "no clarifying questions" hints do NOT waive agent tools (ruff / ipython / ipykernel). Install gate fires regardless.
• Post-hoc audit — required before ending the turn. Walk every pre-flight row; surface unfilled Evidence cells.
• The post-audit gate is mandatory. After the first digest and every refresh, ask Additional report view / Custom query / Custom plot / Close audit in that exact order unless the user already explicitly closed the audit. The one execution may materialize a policy-enabled notebook and a stale-only site preview before this gate. No close-time conversion, git close, record-outcome, or dispatcher return before Close.

Forbidden shortcuts

Pre-flight — emit before any audit-file write or execution

Pre-flight (audit-ml-pipeline):
- [ ] Experiment stem confirmed: <NN_short_name>
      Evidence: journal/NN_<short_name>.md exists AND state is approved
                | "n/a — user invoked re-audit on existing stem"
- [ ] Four-way pairing complete:
        journal/NN_<short_name>.md       — design note (state approved or done)
        experiments/NN_<short_name>.py   — script
        tests/smoke/test_NN_<short_name>.py — smoke test (passing)
        audit/NN_<short_name>.py         — about to be written / refreshed
      Evidence: ls / Glob on each path
- [ ] Report present in skore Project under key=<NN_short_name>
      Evidence: project.summarize() this turn; row with
                key == "<NN_short_name>" appears.
                "Run finished, put() landed" is NOT sufficient.
- [ ] Agent feature available:
        run `ipython -c "print(0)"` and `ipython --version`
        through the project's composed dev environment
      Evidence: tool output of each
                | JOURNAL.md Status `agent feature: installed`
                Missing → STOP, delegate to add-python-package agent tools (ruff / ipython / ipykernel)
- [ ] API CLI consulted for skore symbols used:
      Project, summarize, get, report.checks.summarize, report.metrics.summarize
      Evidence: Read scratch/api/skore/<version>/<topic>.md (this turn)
                | Write the same (this turn)
                | "n/a — cache hit, file already on disk + Read this turn"
- [ ] Template copy + substitution decided:
        <pkg> → package name from src/<pkg>/
        <NN>_<short_name> → experiment stem
        <SKORE_PROJECT_INIT> → literal block copied from experiments/<stem>.py
      Evidence: Read experiments/<stem>.py this turn for the Project init block;
                Read templates/audit.py this turn before Write audit/<stem>.py
- [ ] Read-only contract acknowledged: audit file contains
      summarize / get / report.* only — no evaluate, no put
      Evidence: explicit grep / Read confirmation of the drafted file
- [ ] Execution command shape confirmed:
        python scratch/audit/<stem>/materialize.py
        notebooks true: notebook fill audit/<stem>.py [--html]
      Evidence: command emitted in the response before running
- [ ] G-AUDIT-FINDING derived from the executed digest
      Evidence: issue/tip counts + ordered codes + optional metric
                context | explicit clean-checks value
- [ ] Post-audit gate answered: additional view | query | plot | close
- [ ] Pre-flight re-emitted with evidence before final message.
      Evidence: this checklist appears in the end-of-turn summary.

Before execution

After the four-way pairing, report lookup, agent feature, and API checks are satisfied, emit 1–3 natural sentences immediately before writing or running audit/<stem>.py. Say that this is local read-only report materialization, not retraining: it opens the persisted Skore report, renders checks / metrics / available views, and writes the digest under scratch/audit/<stem>/ plus HTML viewers under scratch/results/<stem>/. Name the exact policy-matched execution command.

Describe cost from facts: report loading and requested view rendering, not model fits or full CV. Unless an observed duration is already available, say timing depends on report size and selected views and do not invent minutes. Emit this preview once for the initial audit; refresh it only when Additional report view / Custom query / Custom plot materially changes the work. The run pauses at the existing post-audit gate before close.

Questions about what the report means are LLM narrative work over the digest. For a fired check, read its documentation URL and apply that page's recommendation. They do not call evaluate or put. If any mandatory gate is pending, preview the possible audit but do not write or execute it.

Audit file contract — overview

The audit file is jupytext percent format (# %%). Filename: audit/NN_<short_name>.py — stem matches the experiment exactly. Template: templates/audit.py.

Substitutions

<SKORE_PROJECT_INIT> and <project-name> are the most error-prone substitutions: the audit must open the same Project the experiment wrote to. Always Read experiments/<stem>.py this turn to lift the literal init block; never reconstruct from memory of the skore mode: decision alone.

Cell sequence (what each cell does)

Brief outline; full anatomy with concrete examples → references/cell_anatomy.md.

1. Module docstring (markdown cell) — what this file is, the read-only rule.
2. Imports (code cell) — import skore, from <pkg> import ....
3. Open the Project (bare-expression cell) — project = skore.Project(...); then project on its own line.
4. List reports — summary = project.summarize(); then summary.
5. Load the report — set REPORT_ID from the URL printed by project.put() (hub: "skore:report:<type-singular>:<N>" — URL path segment is plural, id uses singular, e.g. cross-validations → cross-validation, estimators → estimator; local and mlflow: read summary["id"] for the matching key row), then report = project.get(REPORT_ID), then report as the last expression. Do not write HTML or locator.txt in this cell. materialize.py does that (confirm _repr_html_ with api get).
6. Persisted report — substitute the exact normalized locator from evaluate. For a direct audit, use the selected REPORT_ID and policy.skore_mode: local links ../reports/; Hub uses the exact saved put URL (or a clearly labeled project link); MLflow uses an emitted run URL or records tracking URI + experiment id + run id. If no authoritative locator exists, write n/a — backend did not expose a locator. Do not call put, inspect private storage, or invent a frontend URL.
7. Checks summary — checks = report.checks.summarize(), then checks as the last expression. materialize.py writes checks.html. Its repr groups the walk by severity; every issue / tip line ends with the documentation URL. Read that page and apply its recommendation (custom CSTM* checks may have none).
8. Metrics summary — metrics = report.metrics.summarize().frame(verbose_name=True, flat_index=False), then metrics last. materialize.py writes metrics.html from that frame. That frame is the only metrics table. Do not also paste its values into the design note.
9. Available report accessors — not a notebook cell. Copy templates/materialize.py to scratch/audit/<stem>/materialize.py and run it. It calls help() on report.metrics, report.checks, and any other namespace that exists (inspection, data, …) and writes the trees to accessors.txt. help() prints its tree and returns None. Skip a namespace that is absent. Do not call plot accessors here. Do not put the loop in audit/<stem>.py.

That's the core template. Leave checks as a bare Display: editors render _repr_html_, the runner records the repr, and .frame() drops the severity grouping. Metrics are the frame above, not that Display and not a second markdown table. Extra Display cells are appended only after the user picks Additional report view. Details: → references/cell_anatomy.md.

The digest is the review's canonical source

The rendered digest at scratch/audit/<stem>/audit.md is the single source of truth that review-ml-experiment reads to write journal/ideas/<stem>-<slug>.md and one Ideas row in journal/JOURNAL.md. That skill reads the digest as text, walks Issues: then Tips: in ## Checks summary, and does not re-open the Project, call report.*, or write scratch/<ts>_*.py probes for metric extraction. manage-ml-backlog later triages those files. A promoted idea leaves the Ideas table and becomes a Backlog row.

The contract stays narrow: persisted-report locator + checks + metrics summary. The review must not walk extra Display headings. Do not put ROC / confusion-matrix / importance cells in the core template. After Close-audit extras, those cells use their own ## <title> headings. If the user explicitly asks for a custom figure that accessors.txt did not list, load plot-ml-figure if installed before writing the cell; never replace a skore Display. Save PNG (or HTML) and leave the figure visible; never plt.close in the audit notebook.

G-AUDIT-FINDING

After every successful audit execution, run python -m skore_skills audit finding --stem <stem> (or pass the digest path). Paste JSON finding verbatim. Do not rewrite codes, counts, or the metric clause. Missing digest → JSON finding is n/a — audit digest unavailable.

Extra Display cells do not change G-AUDIT-FINDING (re-run the command; it still reads only checks + metrics).

Return G-AUDIT-FINDING verbatim with the digest and G-REPORT-LOCATOR from python -m skore_skills loop locator --stem <stem>. manage-ml-backlog writes the finding to the design note's Status block. Recompute it after every added view, query, or plot because the digest is overwritten.

Continue auditing

After the initial digest and every successful refresh, unless the user already said to close, AskUserQuestion with one pick in this exact order. None is recommended or preselected.

Gate context. Ahead of the question, state in 2–4 lines what the answer authorizes and the facts it rests on — echoed inline from this turn's digest (the checks and metrics just read, the Display names in accessors.txt) — plus what each option does. A file link is an addition, never the context.

Additional report view / Custom query / Custom plot all edit the same durable audit/<stem>.py, below ## Core audit complete, and the <CALL> block of materialize.py. After an edit: run style, re-run materialize.py once, notebook fill when notebooks are on, derive G-AUDIT-FINDING again with python -m skore_skills audit finding --stem <stem> (from checks + metrics only), then run site build --if-stale when site policy is true and re-present this same gate. Do not run the close-time loop notebooks, run git end-turn, record-outcome, or return to the dispatcher before Close audit. The audit remains read-only: no evaluate, no put, and no workspace-data mutation.