Survey Methodology

6g. Data provenance for re-analysis figures

If your survey reprocesses observational data, notes/datasets.md required:

## <dataset name>
- Source: <URL / DOI>
- Version: <version string>
- Download date: <ISO>
- Processing pipeline: <script path / hash>
- Citation: <DOI of dataset paper>

Without this, your "re-analysis figure" is unauditable — equivalent to PRISMA-name-without-substance for empirical sciences.

Step 7 — Finish-gate checklist

Before brain calls finish(), verify each item below. Adapt to chosen review type per Step 1's relaxations.

1. ☐ notes/scope.md exists, written before any literature load; amendments in notes/scope_amendments.md
2. ☐ Review type declared and committed
3. ☐ Verification floor declared and cleared (anchor exemplar matched in your output)
4. ☐ Topic-ceiling honesty check completed (B-cap acknowledged if applicable)
5. ☐ Question stated in PICO/PECO/PICOS form
6. ☐ Eligibility criteria explicit; measured-but-excluded distinguished from not-measured
7. ☐ Information sources documented with dates and full search strategy
8. ☐ Selection process documented: screener count, dual-screening policy, conflict resolution
9. ☐ Risk-of-bias / quality appraisal at study level (or declared-skipped for scoping reviews)
10. ☐ Synthesis plan pre-specified
11. ☐ notes/adjudication.md exists with disagreement resolution log (universal blind spot — non-empty for any survey covering ≥10 sources)
12. ☐ Certainty-of-evidence rating per load-bearing claim (GRADE / tier)
13. ☐ Flow diagram of identified → screened → eligible → included counts
14. ☐ notes/excluded.md exists with reason per excluded source
15. ☐ Citation budget ≤25/1000 words
16. ☐ All citations carry tier: field per Step 5
17. ☐ notes/datasets.md exists if any re-analysis figure is in report
18. ☐ Open problems concrete (each bound to named missing measurement / method / system)
19. ☐ Funding source / COI / retraction status checked for each cited primary work where applicable
20. ☐ search_current_as_of date noted for each cited SR / review

Anti-patterns — symptoms of B-grade output

Brain should flag in self-review and rewrite. Frequencies validated across the 240-review corpus.

Top 12 anti-patterns by frequency

1. "We comprehensively review..." without a corpus boundary (27/31 CS surveys, ~80% across domains). Replace with bounded N + named selection rule.
2. "We focus on" / "we prioritize" replacing inclusion criteria (22/31 CS surveys). Unfalsifiable scope; replace with explicit yes/no tests.
3. Performance numbers passed through without provenance flagging (20/31 CS surveys). Every quoted score must carry (self-reported / re-run / leaderboard) + setup.
4. Taxonomy figure as primary deliverable (26/31 CS surveys). Removing the figure should leave a contribution; if not, B-grade.
5. Application section without deployment outcome data (18/31 CS surveys). Cut sections that lack measured outcomes.
6. GitHub awesome list as audit substitute (14/31 CS surveys). Curated link list = organize; extracted structured data + reproducible code = audit.
7. PRISMA name-dropped without screening log (1 explicit, ~3 implicit per CS wave). Worse than not invoking PRISMA.
8. "First comprehensive survey of X" for non-disjoint X (6/31 CS surveys). Essentially a noise word in 2024-2026; drop or replace with specific differentiator.
9. Disagreement-handling silently absent (31/31 CS surveys, 12/30 biology). The universal blind spot. Cleanest novelty axis if addressed.
10. "Open questions for human decision" populated with brain's own punts (e.g. "Should we re-weight the matrix?"). Section is for authority-bound questions only. Use escalate_authority_bound tool instead; don't use the section as catch-all.
11. Re-summarizing already-known foundational systems in detail when RESEARCH.md says reader knows them.
12. "Some authors find X, others Y; further work needed" (~10/30 biology, ~9/30 physics). Banned per 6c.

Three failure-mode-specific anti-patterns

13. Bibliometric counting without releasing count code (Renewable Sustainable Energy Reviews modal pattern). The count is the contribution; if it's not reproducible, it's decoration.
14. Dimensional/normalization mixing (Renewable Sustainable Energy Reviews on LCOE; ML surveys on accuracy across benchmark variants). Numbers from different conditions tabulated without reconciliation.
15. Citation laundering through "expert opinion" — quoting Nature Reviews / NEJM Review Articles for quantitative claims as if they were primary research. Per Step 5: these are expert-opinion-tier and require corroboration.

Genre-aware grading

A-grade in math is different from A-grade in CS is different from A-grade in biology. Grade within genre, not uniformly:

• Math A-grade: re-derivation in unified notation OR new short proof OR theoretical-essay with sharp normative argument (Bull AMS allows the essay genre)
• CS/ML A-grade: bounded corpus + author-run measurements OR released extracted-feature data + screening log
• Biology A-grade (rare): original quantitative cross-study re-analysis (e.g. Chang & Prakash dimensionless cellular-acceleration number) OR explicit competing-model adjudication with mechanistic predictions per dataset
• Earth/environment A-grade: dataset reprocessing pipeline documented + new figures from reprocessing
• Engineering A-grade: unified formal architecture subsuming prior approaches OR harmonized device/algorithm spec sheets across primary studies

The rubric varies but the discriminator is invariant: independent verification work present, beyond paper-trust narration.

Mapping to Sisyphus architecture

• Cochrane "two independent reviewers" pattern → already mapped to tool_impl + tool_review blind split (the fix for self-circular impl/test). Survey-time application: screen_dual_<batch> experiment type.
• PRISMA artifact ledger → data/experiments/E*/runs/run_*/results.json
  • notes/experiments.md. Each PRISMA artifact (scope.md, search log, screening table, excluded.md, bias matrix, GRADE summary, adjudication log) is a discrete file, not a prose section.
• GRADE certainty per claim / tier system → tag every load-bearing assertion in notes/literature.md with certainty: {High|Moderate|Low|Very_low}
  • tier: per Step 5. Brain reads when drafting; untagged claims inadmissible in report.
• Authority-bound concern → escalate_authority_bound tool. Use when scope adjudication requires modifying RESEARCH.md. Do not dump punted decisions into a generic "Open questions for human decision" section.
• notes/datasets.md (new artifact for narrative-with-re-analysis type) → tracks observational dataset provenance, versions, processing hashes; required before any re-analysis figure renders.
• notes/adjudication.md (new artifact, Copernicus-inspired) → public disagreement-resolution log; every cross-source contradiction
  • verdict + resolver-agent + evidence path.
• compile_latex hook → can verify presence of finish-gate artifacts (scope.md, datasets.md, adjudication.md, citation count vs body length, tier:-tagging on every cite) and emit warnings to brain pre-finish.

Worked example — what the AI Scientist 2026 survey should have done

The actual run produced 1 audit (E1: AI-Researcher) + 4 synthesis experiments (E2-E5). All 11 other systems were paper-trust. Cost $32, graded B (with 1 element of A-grade in E1).

Under this skill, plan-time decisions:

1. Type: Audit / benchmark survey
2. Floor declared: Counting + Measurement (open-source systems admit both)
3. Topic-ceiling honesty: closed-source tier (Kosmos / Aletheia / Anthropic W2S / Claude Code internals / DeepMind tools) is structurally B-capped — declare in scope.md
4. Plan:

E1-E9: audit_<system> for each open-source: AI-Researcher, RepoAgent,
       LocAgent, RepoAudit, CompileAgent, AIDE, Agent-Lab,
       AutoResearchClaw, OpenHands. Closed-source: documented
       Not_inspectable.

E10-E13: benchmark_sample_<system> for each reported benchmark number:
       - Claude Code on 30-instance SWE-Bench-Verified sample
       - AIDE on 10-instance MLE-Bench-Lite sample
       - AI-Researcher on 5 Scientist-Bench tasks
       - LocAgent file Acc@5 on 30-instance SWE-Bench-Lite sample

E14: cross_paper_reconcile of "GitHub research capability" claims
     across systems (the capability matrix becomes a verified-by-audit
     artifact, not a paper-trust digest).

E15: code_repo_inspect for each of the 9 open-source systems.

E16: disagreement_resolution_log compilation across E1-E15 — the
     universal blind spot, addressed.

Synthesis experiment (E17): the architectural-comparison + failure-mode
     + SOTA chapters, drawing only on E1-E16 evidence.

Estimated cost: $200-400. Estimated grade: A-.

The discriminator isn't writing skill (the existing skills/review/ already covers prose). It's plan-time experiment-type vocabulary. This skill is brain's vocabulary expansion.

References

Detailed evidence and quoted excerpts:

• research/interdisciplinary_protocols.md — PRISMA 2020, PRISMA-ScR, Cochrane Handbook, GRADE, RoB 2, AMSTAR 2, Arksey-O'Malley scoping, Cochrane Rapid Reviews, Greenhalgh narrative-review counterpoint.
• research/bio_med.md — Nature Reviews family editorial framing; Cochrane CDSR; eClinicalMedicine SR worked examples; the 8 things narrative reviews skip.
• research/cs_ml_math.md — BetterBench (NeurIPS 2024), Leak-Cheat-Repeat (EACL 2024), Berkeley RDI benchmark exploits (2025), Acta Numerica unified-derivation pattern, modal LLM-agent surveys as B-grade.
• research/physics_chem.md — RMP Janus collab spin-glass review, Sherrington-Kirkpatrick NRP, Annual Review of Cond Matter exemplars, Chem Soc Rev, J Mater Chem B editorial guidance.
• research/wave2/physics_broad.md — 30 reviews from Living Reviews / RMP / NRP / Annu Rev Nuc Part Sci / ARA&A / Annu Rev Fluid Mech / Reports on Progress in Physics. 70% A-rate (Living Reviews norm).
• research/wave2/chem_materials.md — 30 reviews from Chem Rev / Chem Soc Rev / Nat Rev Chem / Nat Rev Mater / Annu Rev Phys Chem / Annu Rev Mater Res. 60% A-rate; cross-paper benchmark tables in 57%.
• research/wave2/biology_broad.md — 30 reviews from Nature Reviews Immunology / Microbiology / Neuroscience / MCB / Annu Rev Imm / Biochem / Cell-Dev / Neuro / Microbiol / Plant Bio / Trends. 2/30 strict A — the discriminator is hardest to clear in this venue.
• research/wave2/cs_ml_broad.md — 31 surveys from arXiv (cs.LG, cs.CL, cs.CV, cs.AI, cs.CR, cs.SE, cs.IR). 4/31 strict A (~13%). Critical finding: A-grade is topic-determined (open vs closed artifacts).
• research/wave2/math_stats.md — 28 surveys from Acta Numerica / Bull AMS / SIAM Review / Probab Surv / Annu Rev Stat / Stat Surv / FnT TCS. 24/28 A-grade — Acta Numerica norm of unification holds. Genre-aware grading required (essays ≠ unification surveys).
• research/wave2/earth_med.md — 15 earth-environment + 15 clinical medicine reviews. Earth introduces a 3rd methodology mode (narrative-with-embedded-re-analysis); Cochrane is the verbatim template for autonomous-agent SR work. PRISMA adherence item-asymmetric (~7.5% on hard items).
• research/wave2/social_eng.md — 15 economics + 15 engineering reviews. 22/30 A or A- — JEL norm of author re-estimation; AR Econ "stylized-fact tables"; Renewable Sustainable Energy Reviews confirmed B-modal venue.

External references (for brain to fetch on demand):

• PRISMA 2020 statement (Page et al.), https://pmc.ncbi.nlm.nih.gov/articles/PMC8007028/
• Cochrane Handbook ch. 8 (RoB 2), https://www.cochrane.org/authors/handbooks-and-manuals/handbook/current/chapter-08
• PRISMA-ScR (Tricco 2018), https://pubmed.ncbi.nlm.nih.gov/30178033/
• GRADE Working Group, https://www.gradeworkinggroup.org/
• BetterBench (Reuel et al. NeurIPS 2024), https://arxiv.org/abs/2411.12990
• Leak-Cheat-Repeat (Balloccu et al. EACL 2024), https://arxiv.org/abs/2402.03927
• Berkeley RDI benchmark exploits, https://rdi.berkeley.edu/blog/trustworthy-benchmarks-cont/
• VLM-26K bibliometrics, https://arxiv.org/abs/2510.09586
• Lu et al. SLM survey, https://arxiv.org/abs/2409.15790
• OCL-SLR with public artifacts, https://arxiv.org/abs/2501.04897
• Greenhalgh "Time to challenge…", https://pmc.ncbi.nlm.nih.gov/articles/PMC6001568/
• Acta Numerica norm — see Vol 33 (2024), particularly Blanes-Casas-Murua splitting methods (arXiv:2401.01722) for the unified-Lie-series template
• JEL author re-estimation norm — Banerjee-Hanna-Olken-Sverdlin Lisker social protection (DOI 10.1257/jel.20241646) with openICPSR replication