Fulltext Fetcher

Source: https://github.com/aipoch/medical-research-skills

When to Use

• Use this skill when the request matches its documented task boundary.
• Use it when the user can provide the required inputs and expects a structured deliverable.
• Prefer this skill for repeatable, checklist-driven execution rather than open-ended brainstorming.

Key Features

• Scope-focused workflow aligned to: Fetch and save the original HTML of scientific literature webpages when given a URL, DOI, or PubMed PMID (triggered when you need archival-grade page HTML for downstream parsing or review).
• Packaged executable path(s): scripts/fulltext_fetcher.py plus 1 additional script(s).
• Structured execution path designed to keep outputs consistent and reviewable.

Dependencies

• Python: 3.10+. Repository baseline for current packaged skills.
• Third-party packages: not explicitly version-pinned in this skill package. Add pinned versions if this skill needs stricter environment control.

Example Usage

cd "20260316/scientific-skills/Others/fulltext-fetcher"
python -m py_compile scripts/fulltext_fetcher.py
python scripts/fulltext_fetcher.py --help

Example run plan:

1. Confirm the user input, output path, and any required config values.
2. Edit the in-file CONFIG block or documented parameters if the script uses fixed settings.
3. Run python scripts/fulltext_fetcher.py with the validated inputs.
4. Review the generated output and return the final artifact with any assumptions called out.

Implementation Details

• Execution model: validate the request, choose the packaged workflow, and produce a bounded deliverable.
• Input controls: confirm the source files, scope limits, output format, and acceptance criteria before running any script.
• Primary implementation surface: scripts/fulltext_fetcher.py with additional helper scripts under scripts/.
• Parameters to clarify first: input path, output path, scope filters, thresholds, and any domain-specific constraints.
• Output discipline: keep results reproducible, identify assumptions explicitly, and avoid undocumented side effects.

Validation Shortcut

Run this minimal command first to verify the supported execution path:

python scripts/fulltext_fetcher.py --help

fulltext-fetcher

1. When to Use

• You need to archive the original HTML of a scientific literature webpage for later parsing, auditing, or offline review.
• You have a DOI and need to quickly resolve it to the landing page and save the resulting HTML.
• You have a PubMed PMID and need to fetch the PubMed record page HTML (or resolve onward) for extraction workflows.
• You need to batch fetch multiple public literature pages (URLs/DOIs/PMIDs) into a consistent output directory.
• You want a safer crawl mode that restricts requests to a whitelist unless explicitly overridden.

2. Key Features

• Accepts three input types: URL, DOI, and PubMed PMID.
• Writes one .html file per input and preserves the original HTML.
• Supports multiple repeated inputs (e.g., multiple --url / --doi / --pmid flags).
• Default domain whitelist for safer operation; optional override via --allow-all.
• Built-in timeout and retry behavior to reduce long blocking on network issues.

3. Dependencies

• Python 3.x
• Python Standard Library only (no third-party packages)

4. Example Usage

The following commands are intended to be runnable as-is from the repository root.

Fetch a single URL

python scripts/fulltext_fetcher.py --url "https://example.org/page"

Fetch by DOI

python scripts/fulltext_fetcher.py --doi "10.1038/s41586-020-2649-2"

Fetch by PubMed PMID

python scripts/fulltext_fetcher.py --pmid "23273568"

Batch fetch and customize output directory

python scripts/fulltext_fetcher.py \
  --url "https://example.org/page1" \
  --url "https://example.org/page2" \
  --doi "10.1038/s41586-020-2649-2" \
  --pmid "23273568" \
  --out-dir "outputs"

Allow crawling non-whitelisted public domains (explicit opt-in)

python scripts/fulltext_fetcher.py \
  --doi "10.1038/s41586-020-2649-2" \
  --allow-all

5. Implementation Details

Inputs

• --url (repeatable): Direct webpage URL(s) to fetch.
• --doi (repeatable): DOI(s) to resolve (typically via doi.org) and fetch.
• --pmid (repeatable): PubMed PMID(s) to fetch (typically via pubmed.ncbi.nlm.nih.gov / ncbi.nlm.nih.gov).

At least one of --url, --doi, or --pmid must be provided.

Output

• Produces one HTML file per input.
• Default output directory: outputs/
• Output directory can be changed with:
  • --out-dir <path>

Domain access control

• Default allowed domains:
  • doi.org
  • pubmed.ncbi.nlm.nih.gov
  • ncbi.nlm.nih.gov
• To fetch from other public domains, you must explicitly enable:
  • --allow-all

Network behavior

• Uses timeout and retry logic to avoid long-running hangs and to improve robustness against transient network failures.
• Only targets publicly accessible webpages; it is not intended for restricted or authenticated content.

Verification checklist

• After running, confirm:
  • The output directory exists.
  • Generated .html files exist and are non-empty.
• For parallel/coexistence runs, ensure output paths do not collide (use distinct --out-dir values if needed).