Design Study

Phase 3: Clinical framing

Ask whether the comparator and endpoint support the stated claim:

• Is the model better than current practice, or just another model?
• Is the endpoint clinically meaningful?
• Does performance translate to action?
• incremental value: if the model/marker is framed as adding value beyond / on top of / incremental to an existing tool (a clinical score, a routine test, a baseline model), pre-specify the baseline comparator built from the in-routine-use predictors and how added value will be shown: test it with the likelihood-ratio (or Wald) test of the new term in the development data; report ΔC-index / ΔAUC with a paired CI as the size of the gain, not as a second test (a DeLong test of nested models fitted and evaluated on the same data is invalid; DeLong is for two models frozen before a held-out evaluation); prefer the change in decision-curve net benefit at a pre-specified threshold; NRI only as a pre-specified categorical NRI reported separately for events and non-events, never the continuous NRI (analyze-stats incremental_value.md; Kerr et al., Epidemiology 2014). A standalone discrimination number does not support a "beyond X" claim, and the nested-model comparison cannot be added post hoc without the baseline model.
• fine-tuning contribution baseline: if an NLP/LLM study claims that fine-tuning, LoRA, prompt engineering, or a multi-agent wrapper improves extraction/classification, pre-specify a same-backbone zero-shot or few-shot comparator on the identical input, output schema, and test split; a weaker or unrelated baseline cannot show the adaptation adds value. For imaging:
  • a model built by combining / adapting / fine-tuning existing models → the full ablation ladder (un-adapted base, best single component, direct-train vs transfer), per references/combine_models_ablation_design.md;
  • a head-to-head comparison of several models → comparison fairness: one frozen split/preprocessing through every model, a strong fairly-tuned baseline, a matched (or disclosed) compute budget, and a paired delta test, per references/multi_model_comparison_design.md;
  • a claim that a segmentation model is clinically usable → a pre-specified failure taxonomy, an acceptability endpoint with a named judge and adjudication rule, the tail beside the mean, and edit effort paired against manual-from-scratch, per references/segmentation_failure_characterization_design.md. A mean DSC cannot be converted into a usability claim after the fact.
• endpoint↔conclusion scope: decide up front what kind of conclusion the design can support. A cross-sectional / single-visit / prevalence design cannot support a prognostic or surveillance claim (rescreen interval, disease progression) — that needs longitudinal follow-up. A binary surrogate endpoint (present/absent, >0, dichotomized) is risk stratification, not a patient-care directive (defer/withhold/initiate therapy). At review time /self-review §D + check_scope_coherence.py flag CROSS_SECTIONAL_PROGNOSTIC / SURROGATE_CARE_DIRECTIVE against the conclusion.

Phase 4: Reporting fit

Recommend one primary guideline — TRIPOD+AI, CLAIM, STARD, STROBE (TARGET for a target-trial emulation), PRISMA, CARE, or ARRIVE — plus journal-specific additions if needed.

Frequent Failure Modes

Diagnostic AI

No clinically relevant comparator; exam-level instead of patient-level split; unclear reference standard; AUROC-only reporting without threshold metrics.

Prognostic modeling

Unclear time zero; immortal time bias; feature timing mismatch; no calibration.

Retrospective cohort / screening database

• Time zero misalignment: cohort entry ≠ follow-up start → immortal time bias.
• Interval-censored outcomes treated as exact → underestimated event times.
• Unacknowledged healthy-volunteer bias → inflated external-validity claims.
• Surveillance bias from unequal follow-up frequency between groups.
• Map each threat explicitly to one of the three bias classes (Hernán/Robins): selection bias (who enters), information bias (how measured), confounding (what else differs).
• Comparative / causal question → emulate a target trial. For a treatment-vs-treatment, screening-vs-no-screening, or drug-A-vs-drug-B question on routinely-collected data, specify the seven target-trial components (eligibility, strategies, assignment, time zero, outcome, causal contrast, analysis plan) before extraction. This prevents immortal-time, prevalent-user, and confounding-by-indication bias and turns an association into a defensible causal contrast. New-user + active-comparator design, grace-period clone-censor-weight, and negative controls are in references/target_trial_emulation.md.
• Confounding completeness: pre-specify the adjustment set from a DAG (not a Table-1 p < 0.05 rule), and plan an extended-adjustment sensitivity model to show whether any measured covariate that turns out imbalanced by exposure but lies outside the adjustment set leaves the primary estimate robust. Pre-screen the proposed covariates with scripts/adjustment_set_helper.py (flags mediator / collider / descendant adjustment and omitted confounders, and proposes a candidate backdoor set), then derive the minimal sufficient set with dagitty — see references/dag_adjustment.md. At review time /self-review Phase 2.5e and the O-probes in observational_confounding.md (O1–O18) check this against Table 1 — including O7 over-adjustment, O10 overlapping-subset-gradient discipline, O11 design-based weighting for complex-survey data, O12 data-driven-threshold mining, O13 (a cross-sectional mediation claim cannot order X→M→Y), and O14 (a synergy/joint-effect claim needs the additive interaction scale — RERI/AP/S — not a multiplicative-only test).

Multimodal LLM / report generation

No clear rubric for clinical correctness; benchmark labels derived from noisy reports without adjudication; unsupported claims about safety or workflow benefit; input text containing the target label or diagnosis being predicted; no same-backbone zero-shot/few-shot baseline for a fine-tuning or prompt-engineering claim.

Imaging meta-analysis

Overlapping cohorts; paired modalities analyzed as independent; heterogeneity metrics missing; zero-cell handling unspecified.

Minimal-Fix Principle

Recommend the smallest feasible repair first: clarify the claim, narrow the target population, add a limitation statement, add a clinically relevant baseline, re-run one key sensitivity analysis, or redefine the endpoint more explicitly. Escalate to redesign only when the central claim is not defensible otherwise.

Handoff Rules

• route to analyze-stats when the design is basically sound but analysis details need refinement
• route to check-reporting after the design is locked
• route to self-review when the user wants a pre-submission quality check on their own manuscript
• route back to write-paper only after the main validity risks are documented

This skill does not compute statistics, draft full manuscript prose, resolve raw data-engineering issues, or replace a full peer review when journal-facing tone is required.