Experiment Lab

Experiment Lab Skill

This skill governs MedrixFlow's experiment workflow. It does not replace execution tools. It tells the agent when to use the structured experiment pipeline and how to keep results defensible.

Use This Skill For

• CS/AI experiments on local structured data
• Regression, classification, clustering, PCA/UMAP, and simple ablation tasks
• Empirical social-science or public-health data analysis when paired with /mnt/skills/public/empirical-research-methods/SKILL.md
• Bioinformatics workflows on local expression data, metadata tables, h5ad, or 10x matrix bundles
• QC, differential expression, enrichment attempts, marker-gene summaries
• Requests for manuscript-grade figures or experiment bundles for a report

Core Rules

• Do not invent datasets, metrics, plots, baselines, enrichments, or scientific conclusions.
• Exception: when thread context enables Synthetic Experiment Mode, personal experiment data may be simulated from explicit assumptions to complete the manuscript workflow. Public literature, DOI, baseline, leaderboard, benchmark, license, and dataset-version facts must still be real and verifiable.
• For benchmark-driven work, use dataset_benchmark_discovery first to map candidate datasets, leaderboards, metrics, access limits, and baseline/SOTA hints.
• If the task requires actual experimental output, prefer the experiment_lab tool instead of ad hoc reasoning.
• Ask for missing inputs before running a workflow when the absence would invalidate the result:
  • no dataset
  • no target column for supervised CS/AI work
  • no metadata/group labels for requested differential analysis
• Keep literature evidence separate from experimental evidence. Use academic_research only for related work or method framing.
• Keep candidate benchmark evidence separate from executed experiment evidence. A dataset map can justify a plan, but only experiment_lab or attached result files can support manuscript result claims.
• Default to Python-first execution. Do not ask the user to choose Python or R unless they explicitly request R output.
• For empirical methods such as DID, IV, RDD, PSM/IPW, synthetic control, DML, causal forest, target-trial emulation, TMLE, survival, Table 1, event study, robustness, heterogeneity, or mechanism analysis, first load /mnt/skills/public/empirical-research-methods/SKILL.md and follow its identification gates.

Iterative Experiment Loop

For code-tuning, model-training, ablation, or autonomous experiment requests, use an autoresearch-style loop when the environment supports it:

• Establish a baseline before changing code or hyperparameters.
• Define one primary metric and direction up front, such as lower validation loss or higher AUROC.
• Keep the evaluation harness, dataset split, and comparison budget fixed unless the user explicitly asks to change them.
• Change one coherent idea per trial so the result remains attributable.
• Record each trial with commit or run id, metric, memory/runtime if available, status (keep, discard, or crash), and a short description.
• Keep a change only when it improves the primary metric or clearly simplifies the system without hurting the metric.
• Treat crashes, OOMs, timeout, missing metrics, or incompatible dependencies as failed trials; summarize the failure and move on.
• Do not start an indefinite or overnight loop unless the user explicitly asks for a long-running autonomous run.

CS/AI Routing

• Supervised tabular prediction: run regression or classification through experiment_lab
• Diagnostics:
  • binary classification -> confusion matrix, ROC, PR, feature importance if available
  • regression -> predicted-vs-actual, residual distribution, feature importance if available
• Unsupervised tasks:
  • clustering -> cluster projection + silhouette
  • dimensionality reduction -> PCA/UMAP embedding