Model Assessment

Model-Assessment Skill

Assess a trained medical-imaging model (in-house, vendor or open-weights; segmentation, classification or detection) in the order the evidence is built: Part A designs and audits the validation study, Part B computes task-correct held-out metrics, Part C adds the uncertainty / OOD / abstention layer a deployment claim needs, and Part D makes an explainability analysis survive review. Run the parts the request needs (a Grad-CAM question starts at Part D), but no metric headline is reported before the Part A split gate is green. Each part ends in a stdlib gate whose verdict is reproduced from a file — never report a pass without running it.

Numbers come only from code executed on the supplied predictions, split table, or the researcher's executed UQ/XAI code; if predictions or ground truth are missing, say so and stop. Integrate MONAI / nnU-Net, MAPIE, captum, pytorch-grad-cam and pretrained OOD scorers by reference — never reimplement them, never build or train the model, never run a model on real patient data.

Elsewhere: building/training → /model-scaffold; choosing or vetting the model → /model-selection; data-stage preprocessing leakage → /imaging-data; paired model comparison / added value over a baseline / decision curves / MRMC / ICC / calibration tables → /analyze-stats (added value: incremental_value.md); AI-vs-expert reader rubric and IRR → /design-ai-benchmarking; LLM/MLLM → /mllm-eval; general validity → /design-study; classical-ML tabular calibration → /radiomics-ml; item-by-item reporting audit → /check-reporting; a finished manuscript → /self-review or /peer-review (the MD0–MD11 model_development.md probe).

Part A — Validation design

The rationale behind Phases 2–7 — the full leakage taxonomy, the internal-vs-external ladder, comparator design, run variance, test-set sizing, the reporting map — is in ${CLAUDE_SKILL_DIR}/references/validation_design.md (load on demand). Verify citations via /search-lit (confirmed DOI/PMID), else mark [UNVERIFIED - NEEDS MANUAL CHECK]; flag an uncertain CLAIM 2024 / TRIPOD+AI / Metrics Reloaded item [VERIFY] and ask.

Phase 1 — Task, intended-use horizon, analysis unit

State the task, the intended-use horizon (screening, triage, pre-procedure, post-hoc), the single headline metric the conclusion leans on, and the analysis unit (per-patient / per-lesion / per-image). Everything downstream is read against this; a per-lesion metric is never reported as per-patient.

Phase 2 — Leakage audit (run first)

Produce the emitted split-assignment table (patient_id,split) and run:

python3 ${CLAUDE_SKILL_DIR}/scripts/check_split_leakage.py \
  --splits <split_assignment.csv> --out qc/split_leakage.json --strict

PATIENT_OVERLAP (a patient in ≥ 2 partitions) and MISSING_SEED (an unreproducible split) are Major, SINGLE_PARTITION Minor — proven by set arithmetic on the ID column the gate prints as id_col (pass --id-col with the patient identifier when it is not the one; an auto-picked column whose name is not patient-level, such as image_id, gets a Minor ID_COL_NOT_PATIENT_LEVEL). A design with patient overlap is never approved. Then walk the leakage the table cannot show (Kapoor & Narayanan, Patterns 2023): preprocessing fit before the split (normalisation, resampling, foundation-model embeddings, ComBat harmonisation over the whole cohort — /imaging-data gates the declared pipeline), site / scanner / burned-in-label shortcuts, and temporal leakage (a random split where future and past coexist). The decisive question: could any value used in training have been computed only with knowledge of a test case?

Phase 3 — Validation tier

Classify honestly: apparent → internal random split → cross-validation → temporal → geographic / external (different site, scanner, vendor) → multi-site external. Cross-validation and bootstrap are development-time optimism corrections, not external validation. Flag a generalisability or deployment claim that outruns the design, and "external validation" where the single external set was used for tuning. Confirm the test set was touched once — no architecture search, hyperparameter sweep, early stopping, or threshold choice read it.

Phase 4 — Comparator

Clinical-only baseline, incremental value over an existing score, or reader comparison (hand the rubric / inter-rater design to /design-ai-benchmarking).

Phase 5 — Test-set sizing

Count events per class in the test set, not the cohort total: a sparse positive set gives a CI spanning much of the usable range, and calibration needs roughly ≥ 100 events. Hand formal sizing to /calc-sample-size.

Phase 6 — Prospective evaluation and deployment-monitoring horizon

Retrospective external validation shows accuracy transfers, not that the model is safe and useful in the workflow. For a clinical-use claim design the higher tier explicitly — silent / shadow deployment → prospective comparative or impact study / RCT on a clinical endpoint → post-deployment monitoring with recalibration-or-withdrawal triggers and subgroup audit (references/validation_design.md §2b). Scope the claim to the tier reached: a retrospective external study never claims deployment readiness or outcome benefit.

Phase 7 — Reporting-guideline fit

Map via /check-reporting: CLAIM 2024 (diagnostic imaging AI), TRIPOD+AI (prediction model), STARD-AI (diagnostic accuracy), PROBAST+AI (risk of bias), and for a prospective/live evaluation DECIDE-AI or CONSORT-AI / SPIRIT-AI.

Part B — Held-out metrics

Phase 8 — Compute task-correct metrics

Generate and execute evaluation code on the held-out predictions (Metrics Reloaded — Maier-Hein & Reinke et al., Nat Methods 2024):

• segmentation — Dice/IoU and a boundary metric (HD95 / NSD), per structure, 95% CIs by patient-level bootstrap (resample patients, not pixels or slices);
• classification — AUROC and AUPRC with patient-level bootstrap CIs, sensitivity/specificity, and PPV/NPV at the deployment prevalence, never bare accuracy on a balanced set. Report AUPRC with the test-set prevalence, which is its no-skill value: AUPRC moves with prevalence, so a value from an enriched or case-control test set does not carry over to deployment or across datasets. For multiclass, state the aggregation (one-vs-rest / macro / micro / pairwise / Obuchowski);
• detection — FROC / mAP with the IoU match criterion stated. Lesions and false positives cluster within patients, so CIs come from a patient-level bootstrap (resample patients, carrying all their lesions and false positives), not a Wilson/binomial interval over lesions, which is too narrow; compare two detectors' FROC curves with JAFROC (RJafroc), not per-lesion tests;
• interactive / promptable segmentation (SAM2 / MedSAM2 / nnInteractive) — the segmentation metrics plus Dice-vs-interactions / number-of-clicks (NoC) to a target threshold, initial-vs-converged (or peak) Dice, and per-case interaction/inference time. With two arms (simulated prompting + human operator), record protocol fidelity — identical prompt types, stopping rule, target threshold, seeds — because arm-to-arm comparability is what lets the human arm validate the simulated one (human-arm design: /design-study);
• generative / synthesis — full-reference (MSE/RMSE/PSNR/SSIM) or no-reference (SNR/CNR, visual scores) quality plus a downstream-task evaluation: image quality is not clinical utility (Park et al., Radiol Med 2024);
• time-to-event discrimination (Harrell's C, time-dependent ROC) → /analyze-stats.

Report the headline as the point estimate with a patient-level bootstrap 95% CI over the test cases: that is the uncertainty of the test-set estimate. Seed-to-seed SD across training runs is a different quantity (training-run variability, usually smaller) — report it separately, over ≥ 5 runs, for a training-recipe or model-comparison claim, and never present it as the CI. A frozen vendor or open-weights model has no training runs to vary; its uncertainty is the test-set CI. Add calibration — for a binary risk or diagnostic output, calibration-in-the-large (intercept), the calibration slope and a flexible (loess) calibration curve, plus the Brier score; ECE only as a supplementary top-label summary for multi-class confidence, with its binning stated — and subgroup slices (the Model Card Factors). Emit results.md (metrics report) and a per-case CSV for /analyze-stats. Load ${CLAUDE_SKILL_DIR}/references/metric_guide.md for the per-task checklist and ${CLAUDE_SKILL_DIR}/references/metric_selection_grounding.md for why each pairing is required and the CLAIM 2024 fit map.