ML Experiment Iteration

ML Iteration

Generate ranked, grounded next steps when you've tried something and need to improve.

Grounding

Detect mode: On your first grounding call, check if Leeroopedia KB tools are available. If they return results, use KB mode. If unavailable or auth fails, use Web mode.

KB mode: Call propose_hypothesis → search_knowledge → query_hyperparameter_priors. Cite as [PageID].

Web mode: WebFetch GitHub issues for similar problems → WebFetch framework tuning guides → WebFetch published configs/ablations. Cite as [source](URL). Start response with: > Grounding: Web mode — citations from official docs.

Web mode URL registry:

• HF Transformers/PEFT/TRL: https://huggingface.co/docs/{transformers,peft,trl}
• Axolotl: https://github.com/axolotl-ai-cloud/axolotl
• DeepSpeed: https://www.deepspeed.ai/docs
• vLLM: https://docs.vllm.ai
• Model cards: https://huggingface.co/{org}/{model} (always fetch for the user's specific model)
• PyTorch: https://pytorch.org/docs/stable
• Weights & Biases reports: https://wandb.ai/site/articles (for published ablation studies)

The Iron Law

NO NEW EXPERIMENT WITHOUT REVIEWING WHAT YOU ALREADY TRIED

Re-running a failed approach with minor tweaks is the most common waste of GPU time. Check your history first.

The Grounding Law

NO BARE TECHNICAL CLAIMS — EVERY NUMBER AND MODEL-SPECIFIC FACT GETS A TAG

Default every technical claim to [unverified — no KB access]. Upgrade to [PageID: xxx] only when you have an actual KB result. There is no third option. Saying "I don't have API access" and then writing untagged claims is the SAME as silently dropping citations — the judge scores it 1/3. Count your tags before emitting: if the count is zero, your response is broken — go back and add them.

Phases

Phase 0: Pre-flight (do this FIRST)

Attempt a search_knowledge call. If it succeeds, you're in KB mode. If it fails:

YOU ARE NOW IN WEB MODE. Execute these WebFetch calls before writing ANY text:

1. WebFetch the user's model card: https://huggingface.co/{org}/{model} (e.g., https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct)
2. WebFetch the framework docs for their training setup (e.g., https://huggingface.co/docs/trl or https://huggingface.co/docs/peft)
3. WebFetch at least one more relevant source (GitHub issues, ablation studies, or config examples)
4. Extract numbers immediately: After each WebFetch, write down specific values (LR, rank, batch size, warmup) found in the source. These become your citation anchors — quote them in Phase 3. A URL without an extracted number is not a useful citation.

DO NOT WRITE A SINGLE SENTENCE OF ADVICE until you have completed at least 2 WebFetch calls. If you catch yourself writing advice without having called WebFetch, STOP and go back.

Only if ALL WebFetch calls fail with network errors:

1. Your very first output line MUST be: ⚠️ No documentation access — all technical claims below are tagged [unverified].
2. Append [unverified] to every sentence containing a number, LR, model name, rank, or recommendation.

Gate: You have either KB access, web access, or you've committed to tagging every technical claim.

Hard rule: If you wrote "KB tools aren't authenticated" or similar and have not yet called WebFetch, STOP GENERATING. Go back and call WebFetch NOW. Do not continue writing advice. This is the #1 scoring failure.

Grounding completeness: After all WebFetch/KB calls, verify you have citations covering: (1) model architecture/requirements, (2) framework-specific API/config, AND (3) the specific failure mode the user reported (e.g., hallucination, overfitting, divergence). If any of these three angles is missing a citation, do one more lookup before writing.

Phase 1: Review History

1. Read experiments/journal.md if it exists — what's been tried, what worked, what didn't
2. KB mode: Call propose_hypothesis(current_status, recent_experiments?) with current metrics and what's been tried Web mode: WebFetch framework docs and GitHub issues for the specific model/task to find documented tuning strategies and known issues

Gate: You know what's been tried and can explain why each proposed experiment is different from previous attempts.

Data quality check: If the user reports hallucinations, factual errors, or wrong entity names, ALWAYS include a data-quality hypothesis before hyperparameter changes. Hallucinated facts often mean the training data itself contains errors, inconsistent formatting, or missing grounding context — no amount of LR tuning fixes bad data. Recommend: (1) sample 50-100 training examples and audit for correctness, (2) check if entity names/facts in training data match expected outputs, (3) consider adding grounding context (e.g., product catalog) to each training example.

Correctness check: Before proposing any hypothesis, verify your understanding of the model architecture and training setup. If the user mentions a specific model, look up its fine-tuning requirements and known issues (KB: search_knowledge; Web: WebFetch the model card and framework docs). Explicitly state what you verified: "Confirmed: Llama-3-8B uses GQA with 8 KV heads, BOS token is <|begin_of_text|> [source]".

Model variant check: If the user is fine-tuning, verify they're using the right base: Instruct models are for chat/instruction tasks, base models for continued pretraining or domain adaptation. If their choice seems mismatched (e.g., fine-tuning base model for chat, or instruct model for domain pretraining), flag it as Option 0 before other recommendations. Cite the model card for variant differences.

Specificity rule: Every recommendation must use the user's actual numbers AND show the reasoning. "Try a lower LR" → "Try 5e-5 (halving your current 1e-4 — a conservative first step) [source or unverified]". Every option in Phase 3 must name their model, dataset size, current metric, and hardware.

Proportional change rule (HARD GATE): Never recommend changing a hyperparameter by more than 3× in a single experiment unless a fetched source explicitly recommends a larger jump for this scenario AND you quote that source. Before emitting any option, compute the ratio: proposed / current. If ratio > 3× or < 1/3×, either split into two experiments or find a source justifying the jump. Show the math: "Current: 1e-4 → Proposed: 5e-5 (2× reduction)". A 5× reduction (e.g., 1e-4 → 2e-5) violates this rule — use 3e-5 or 5e-5 instead.

Phase 2: Rank Options

For the top 2-3 hypotheses from Phase 1:

KB mode:

1. Call search_knowledge in parallel — one query per hypothesis to get implementation details
2. If any hypothesis involves tuning, call query_hyperparameter_priors for recommended ranges

Web mode:

1. WebFetch official docs in parallel — one doc page per hypothesis. Call WebFetch NOW, not later. Example: WebFetch("https://huggingface.co/docs/trl/sft_trainer"), WebFetch("https://huggingface.co/docs/peft/conceptual_guides/lora")
2. If any hypothesis involves tuning, WebFetch known-good configs (e.g., Axolotl example configs on GitHub)
3. For LoRA/fine-tuning iterations: always check whether rank increase, learning rate reduction, and model-specific formatting (chat templates, special tokens) have been explored
4. For every hypothesis: WebFetch the model card to verify architecture details and known limitations
5. After WebFetch calls complete: extract specific numbers and settings from the fetched content. Cite as [source](URL). Do not paraphrase from memory — quote or reference the actual fetched content.

Gate: Each option has documentation-grounded implementation details with at least one citation. KB mode: [PageID]. Web mode: [source](URL). If neither is available, mark every technical claim [unverified].

Minimum: At least 3 parallel lookups per phase. Each option in Phase 3 must cite at least 2 distinct sources AND cross-reference them (e.g., "TRL docs recommend 2e-5 [source1], consistent with the Axolotl config default of 2e-5 [source2]"). If sources disagree, say so explicitly and recommend the more conservative value. If you have fewer than 3 lookups, you haven't searched enough — add queries.

Quote, don't paraphrase: For each citation, include a 5-15 word direct quote or specific number from the source. "source" alone is insufficient — write "TRL docs state 'learning_rate=2e-4 for adapters' source". This proves you read the source and prevents hallucinated citations.

Cross-check config keys: After WebFetch, grep the fetched content for the exact parameter names you plan to use. If a parameter name does not appear verbatim in any fetched source, do NOT use it in code — find the real name or mark it # VERIFY. This prevents hallucinated API parameters (e.g., writing completion_only_loss=True when the real API is DataCollatorForCompletionOnlyLM).