Reproducibility Check

Source: https://github.com/aipoch/medical-research-skills

When to Use

Use this skill when you need to assess or improve research reproducibility, for example:

Mode A — Methods Completeness Audit (diagnostic)

1. Pre-submission self-check to ensure the Methods section is complete before journal submission.
2. Replication feasibility review to determine whether another lab/team could repeat the work.
3. Peer review / methodological audit to identify missing details, ambiguities, or under-specified procedures.
4. Internal lab documentation check to improve protocol clarity and reduce tacit knowledge.
5. Meta-research / reproducibility screening to triage papers by reproducibility risk.

Mode B — Open Science Best Practices (prescriptive) 6. Pre-registration guidance for hypotheses, methods, and analysis plans before data collection. 7. FAIR data management to make data findable, accessible, interoperable, and reusable. 8. Code and computational environment sharing (Docker, Binder, GitHub, Zenodo). 9. Replication study design (direct/conceptual replication, safeguard power analysis). 10. Reporting transparency following CONSORT, STROBE, ARRIVE, PRISMA guidelines. 11. Open science practices (badges, registered reports, preprints, open access).

Trigger condition: if the user provides only an abstract/results/discussion without the full Methods section for Mode A, request the complete Methods section first.

Key Features

Mode A — Methods Completeness Audit

• Methods completeness audit focused on replication-critical details.
• Structured missing-items report with clear priority levels (High/Low).
• Ambiguity detection for unclear or under-specified descriptions.
• Reproducibility risk rating (Low/Medium/High) with explicit rationale.
• Actionable supplementation suggestions mapped to specific deficiencies.
• Checklist-driven output using assets/reproducibility_checklist.md when available.

Mode B — Open Science Best Practices

• Pre-registration guidance for OSF Registries, AsPredicted, ClinicalTrials.gov (clinical), PROSPERO (systematic reviews); distinguish confirmatory from exploratory analyses.
• FAIR data management — Findable (persistent identifier, metadata), Accessible (open/controlled access), Interoperable (standard formats, vocabularies), Reusable (license, provenance, data dictionary).
• Code and computational environment sharing — public repositories (GitHub, GitLab, Zenodo DOI), dependency documentation (requirements.txt, renv.lock, conda environment.yml), containerization (Docker, Binder).
• Replication study design — direct replication (match original methods), conceptual replication (different methods, same hypothesis), safeguard power analysis (assume smaller effect), equivalence testing or Bayesian replication factors.
• Reporting transparency — all reporting guidelines (CONSORT, STROBE, ARRIVE, PRISMA), all pre-specified analyses reported, exploratory analyses clearly labeled, supplementary materials shared.
• Open science practices — open science badges (data, materials, pre-registration), registered reports, preprint servers (bioRxiv, medRxiv, arXiv, SSRN), open access routes.

Dependencies

• assets/reproducibility_checklist.md (recommended for structured reporting)
• references/guide.md (recommended for formatting guidance and key checkpoints)

Key Platforms and Tools

• OSF (Open Science Framework) — Project management and pre-registration
• AsPredicted — Streamlined pre-registration
• Zenodo — Data and code archival with DOI
• GitHub / GitLab — Code version control and sharing
• Docker / Binder — Computational environment reproducibility
• FAIR self-assessment tool — Data FAIRness evaluation
• COS (Center for Open Science) — Reproducibility guidelines

Example Usage

Input (user)

Please check whether the following Methods section is complete and reproducible. Identify missing items (high/low priority), unclear descriptions, and provide a reproducibility risk rating with suggestions.

[PASTE FULL METHODS SECTION HERE]

Optional context:
- Equipment list: [models, manufacturers]
- Software: R 4.3.1, Python 3.11, packages: ...
- Statistical plan: ...
- Data processing workflow: ...

Output (assistant)

## Reproducibility Check (Methods Section)

### 1) Missing Information
**High Priority (required for replication)**
1. **Sample size determination**: No power analysis or rationale provided for n=...
   - Location in text: "..."
   - Why it matters: affects statistical validity and replication planning.
2. **Randomization procedure**: Randomization mentioned but method not specified (e.g., block randomization, RNG seed).
   - Location in text: "..."
3. **Reagent/source details**: Antibody X lacks catalog number, clone, vendor, and lot.
   - Location in text: "..."

**Low Priority (improves robustness/clarity)**
1. **Environmental conditions**: Temperature/humidity not reported for assay Y.
   - Location in text: "..."

### 2) Unclear or Ambiguous Descriptions
1. **“Performed as previously described”** without a complete protocol or sufficient parameters.
   - Location in text: "..."
   - Clarification needed: step-by-step parameters, timing, volumes, thresholds.

### 3) Reproducibility Risk Assessment
**Risk Level: Medium**
- Rationale: Core workflow is described, but key replication-critical parameters (randomization, reagent identifiers, preprocessing thresholds) are missing or ambiguous.

### 4) Supplementary Suggestions
1. Add a table listing **all materials/reagents** with vendor, catalog number, lot, and storage conditions.
2. Specify **software versions** and exact packages (including version numbers) used for analysis.
3. Provide **data preprocessing rules** (missing values, outliers, normalization) and decision thresholds.
4. Include **statistical testing details** (assumptions checks, alpha, multiple-comparison correction, effect sizes, CI reporting).

Implementation Details

Inputs

• Required (Mode A): Full text of the Methods section (plain text or file content).
• Optional (Mode A): Materials/equipment list, software and versions, statistical analysis plan, data processing workflow, protocol appendices.
• Optional (Mode B): Research topic, study design, data types, analysis plan, target repository/journal.
• Preferred formats: txt, md, docx (or pasted text). If a file path is provided, the content must be supplied by the user.

Processing Workflow

Mode A — Methods Completeness Audit

1. Method deconstruction
   • Extract and enumerate: materials/reagents, equipment, software, experimental design, procedures, parameters, thresholds, and units.
2. Checklist verification
   • Validate coverage of: sample size/replicates, randomization/blinding, controls, inclusion/exclusion criteria, protocol steps, calibration, preprocessing, statistics, and reporting standards.
   • Prefer structured reporting aligned with assets/reproducibility_checklist.md.
3. Missing information labeling
   • Mark omissions and classify priority:
     • High Priority: required to reproduce results (critical identifiers, parameters, decision rules, analysis details).
     • Low Priority: improves clarity/robustness but not strictly required.
4. Recommendation generation
   • Provide concrete additions (tables, parameter lists, step-by-step clarifications).
   • Assign a Low/Medium/High reproducibility risk rating with explicit reasons.

Mode B — Open Science Best Practices

5. Assess current reproducibility state — Evaluate against three dimensions: methodological (sufficient detail to replicate), computational (code + data + environment = same results), results reproducibility (independent replication yields consistent findings). Identify specific gaps.
6. Pre-registration — Guide pre-registration of hypotheses, methods, and analysis plan BEFORE data collection. Use appropriate platform: OSF Registries, AsPredicted, ClinicalTrials.gov (clinical), or PROSPERO (systematic reviews). Distinguish confirmatory from exploratory analyses.
7. Data management — Apply FAIR principles: Findable (persistent identifier, metadata), Accessible (open or controlled access with clear process), Interoperable (standard formats, vocabularies), Reusable (license, provenance). Create data dictionary documenting every variable. Use tidy data formats.
8. Code and computational environment — Share analysis code in a public repository (GitHub, GitLab, Zenodo for DOI). Document dependencies with requirements.txt, renv.lock, or conda environment.yml. For full reproducibility: containerize with Docker or use Binder. Include README with execution instructions.
9. Replication study design — For direct replication: match original methods as closely as possible. For conceptual replication: test same hypothesis with different methods. Conduct power analysis based on original effect size (use safeguard power: assume smaller effect). Determine sample size for meaningful replication test (use equivalence testing or Bayesian replication factors).
10. Reporting transparency — Follow reporting guidelines (CONSORT, STROBE, ARRIVE, PRISMA). Report all pre-specified analyses regardless of results. Clearly label exploratory analyses. Share full materials (stimuli, protocols, instruments) as supplementary files.
11. Open science practices — Adopt open science badges (data, materials, pre-registration). Consider registered reports format (peer review before results). Use preprint servers (bioRxiv, medRxiv, arXiv, SSRN). Choose open access publication route.