Empirical Paper Writer

Empirical Paper Writer

Use this skill for novel experimental research papers after the topic and contribution have already been framed.

This skill is the empirical counterpart to ../arxiv-paper-writer. It reuses the same high-value paper-engine pieces—citation discipline, LaTeX compilation, source policy, and QA—but changes the paper logic from review to experiment-driven writing.

When to Use

• The user wants a method/experiment paper rather than a review.
• The contribution requires experiments, ablations, or quantitative comparisons.
• The user wants a near-submission draft with explicit placeholders where evidence is not yet verified.

When NOT to Use

• Pure surveys, taxonomies, or literature syntheses.
• Non-academic reports.
• Cases where the user only wants experiment design and no paper draft.

Inputs

• Topic / research direction
• Handoff artifacts from ../paper-from-zero when available:
  • brief/topic-brief.md
  • brief/contribution-map.yaml
  • brief/evidence-matrix.csv
  • plan/outline-contract.md
• Optional user-provided innovation, baselines, datasets, or results

Outputs

• main.tex (placeholder-safe draft)
• ref.bib
• paper.config.yaml (includes runtime.* for experiment execution)
• plan/<timestamp>-<slug>.md
• issues/<timestamp>-<slug>.csv
• Optional notes/literature-notes.md
• Recommended notes/innovation/ (candidates + decision log + evidence links)
• notes/design/ CSV artifacts (baselines.csv, method-components.csv, experiment-matrix.csv)
• Figures/tables/result placeholders in the LaTeX draft (hypothesis-safe)
• Optional experiments/ code scaffold (PyTorch skeleton; not executed by this skill)
• main.pdf after compile/QA when LaTeX is available

Non-Negotiable Rules

1. No prose in main.tex until plan approved and issues CSV exists.
2. Do not fabricate results, numbers, or significance claims.
3. Every experimental claim must map to an evidence item or explicit placeholder.
4. Mark result status explicitly as one of planned, placeholder, or verified.
5. Reuse the existing paper-engine scripts for citation verification, compile, source ranking, and QA whenever possible.

Workflow

Gate 0: Scaffold + Research Contract

1. Confirm venue, page target, datasets/baselines if already known, and any user-provided innovation.
2. Search literature and gather only enough papers to frame the problem, baselines, and closest related methods.
3. Create or refine:
   • contribution map
   • experiment/evidence matrix
   • outline contract
4. Scaffold the project:

   python3 scripts/bootstrap_ieee_empirical_paper.py --stage kickoff --topic "<topic>" --layout project

   --layout project creates <project>/paper/ (LaTeX + issues) and <project>/experiments/ (code scaffold). For a lighter entrypoint with an outline-only plan:

   python3 scripts/bootstrap_ieee_empirical_paper.py --stage outline --topic "<topic>" --layout project

5. Create a skeleton-only main.tex with headings, bullet placeholders, experiment slots, and seed citations.
6. STOP until the user approves the plan.

Phase 0.5: Method & Experiment Design

After the user approves the initial plan, design the method and experiments before creating the issues CSV. This phase produces structured CSV artifacts.

Step 1: Systematic Baseline Identification

1. From the contribution-map primary claim, search for baselines in 3 categories:
   • Direct competitors: SOTA methods on the same task (last 2 years).
   • Foundational methods: well-known classics that anchor the field.
   • Ablation anchors: our method minus its core innovation.
2. Record 10-20 candidates in notes/design/baselines.csv (see assets/baselines-template.csv).
3. Select 4-8 baselines for final comparison. Mark selected=yes with reason.
4. See references/experiment-design.md Section 1 for selection criteria.

Step 2: Innovation Module Design

1. From the gap analysis (nearest prior work weaknesses × improvable directions):
   • Design a minimum viable innovation: one core change that is clearly testable.
   • Make the innovation modular (pluggable component, supports ablation).
   • Define clear input/output interfaces for reproducibility.
2. Record all pipeline components in notes/design/method-components.csv (see assets/method-components-template.csv).
3. Mark is_novel=yes for novel components; define replaceable_by for ablation.

Step 3: Pipeline Architecture Design

1. Design the overall flow: input → preprocessing → core module(s) → postprocessing → output.
2. For each component: function, input/output format, replaceability.
3. Training flow: loss function rationale, optimizer, tuning strategy.
4. Inference flow: runtime cost estimate.
5. Sketch a pipeline architecture diagram placeholder for the Method section.

Step 4: Comparison Experiment Matrix

1. Design the baselines × datasets × metrics matrix.
2. Fair comparison rules:
   • Same data splits, preprocessing, and evaluation protocol for all methods.
   • Use official implementations or paper-reported results (annotate source).
   • Plan ≥3 runs with different seeds; report mean ± std.
3. Record all experiment rows in notes/design/experiment-matrix.csv with type=main_comparison (see assets/experiment-matrix-template.csv).

Step 5: Ablation Experiment Design

1. From method-components.csv, identify factors by ablation_priority:
   • High: core innovation components (must ablate).
   • Medium: architecture choices (ablate if space allows).
   • Low: hyperparameter choices (include only if impact is significant).
2. For each factor: define removal/degradation/random-replacement strategy.
3. Record in experiment-matrix.csv with type=ablation.
4. Minimum 4 ablation factors.

Step 6: Robustness & Efficiency Analysis Plan

1. Robustness: noise levels, distribution shift, domain transfer scenarios.
2. Error analysis: failure case categories, sampling protocol.
3. Efficiency: parameter count, FLOPs, inference latency vs baselines.
4. Record in experiment-matrix.csv with type=robustness and type=efficiency.

Phase 0.5 outputs (saved in notes/design/):

• baselines.csv — baseline & SOTA competitive landscape
• experiment-matrix.csv — full experiment design matrix
• method-components.csv — pipeline component inventory

STOP until the user confirms the design artifacts before proceeding to Gate 1.

Gate 1: Create Issues CSV

1. Check the kickoff gate in the plan.
2. Create issues CSV:

   python3 scripts/bootstrap_ieee_empirical_paper.py --stage issues --topic "<topic>" --with-literature-notes --layout project

3. Validate:

   python3 scripts/validate_empirical_paper_issues.py <paper_dir>/issues/<timestamp>-<slug>.csv

   For --layout project, <paper_dir> is <project>/paper.

Phase 1.5: Literature Enrichment Gate

Before starting the writing loop, ensure citation coverage is adequate.

1. Count check: total unique entries in ref.bib. If < 25, trigger enrichment.
2. Cluster search: for each Related Work sub-area (from the RW taxonomy), search for 3-5 additional relevant papers beyond the initial spine set.
3. Baseline citations: every selected baseline in baselines.csv must have at least one corresponding entry in ref.bib.
4. Method motivation: for each novel component in method-components.csv, find 1-2 papers that motivate the design choice (prior art or the gap it fills).
5. Verify all new citations via the standard verification pipeline.
6. Gate: do not start W1 until ref.bib has ≥ 25 verified entries. Target 30-40 for the finished paper (empirical papers need fewer than reviews, but 14 is universally too low).

Phase 2: Execution Loop

For each issue:

1. Research the exact claim, baseline, or related-work gap.
2. Draft the assigned section or experiment block.
3. Keep results explicit:
   • verified: backed by real evidence
   • placeholder: reserved for future real evidence
   • planned: the experiment is designed but not yet filled in
4. Prefer hypothesis-safe writing for any unverified outcome (e.g., (hypothesis) / [Pending: ...] tags).
5. Never write deterministic superiority claims without verified evidence.
6. Run citation audit / compile / QA before marking DONE.
7. Dependency enforcement: Before marking any issue DONE, verify that ALL issues listed in its Depends_On column are already DONE or SKIP. If any dependency is still TODO or DOING, the current issue MUST NOT be marked DONE. This rule is non-negotiable.

Phase 2.2: Issue Execution Helpers

• Use python3 ../arxiv-paper-writer/scripts/issue_workflow.py --project-dir <paper_dir> render-skeleton --issues <issues.csv> --issue-id <Wx> to render a LaTeX section skeleton for a Writing issue.
• Add --apply-if-missing only when the full section path is entirely absent from main.tex; nested insertion under an existing parent stays manual.
• Before QA or after a batch of edits, run python3 ../arxiv-paper-writer/scripts/issue_workflow.py --project-dir <paper_dir> audit --issues <issues.csv> to check section-path consistency, citation counts, placeholders, and lightweight figure/page signals.

Phase 2.3: Experiment Execution Checkpoint

After all experiment design issues (E0-E4) and experiment code issues (E5-E7) are DONE:

1. Check runability: Verify that the experiment runner script exists and is syntactically valid.
2. STOP and instruct the user:
   • Tell the user to run experiments in their configured environment:

     conda activate <runtime.conda_env>
     cd <project_dir>/experiments
     python run_all.py --config configs/<config>.yaml

   • Tell the user to invoke results-backfill SKILL after experiments complete.
3. Do NOT attempt to run long experiments within the AI session.
4. Mark experiment execution issues (E8-E10) as TODO with note "awaiting user execution".

Phase 2.4: Structural Figure Generation

After all writing issues (W1-W7) reach DONE, resolve structural diagrams. These are non-result figures—architecture, pipeline, formulation diagrams that depend on method design, not on experiment outcomes.

Step 1: Identify required figures Scan main.tex for \fbox{...placeholder...}. Classify each:

• Structural (derivable from method-components.csv / problem formulation): generate now.
• Result-dependent (needs experiment data): keep as placeholder until Phase 2.5.

Step 2: Generate structural TikZ figures For each structural placeholder:

1. Read notes/design/method-components.csv to extract component names, is_novel flags, and data-flow edges.
2. Select a pattern from references/figure-generation-guide.md.
3. Generate a .tikz file under paper/figures/ and replace the \fbox with \input{figures/<name>.tikz}.
4. Standard figures for empirical papers (generate at least 2 of 3):
   • System overview / teaser (fig:teaser): problem setting + where the method fits. Place in Introduction.
   • Method architecture (fig:method): pipeline with components, novel parts highlighted. Place in Method.
   • Formulation diagram (optional, fig:formulation): MDP / state machine / optimization flow. Place in Problem Formulation or Method.

Step 3: Visual issues tracking Use V-prefixed issues (V1, V2, ...) in the issues CSV for each figure. Mark DONE only when the TikZ compiles and is referenced in text.

See references/figure-generation-guide.md for TikZ patterns and style rules.

Gate: All structural \fbox placeholders must be resolved before Phase 2.5. Result-dependent \fbox pass through to Phase 2.5.