Biostudies Fetch

🦖 BioStudies Fetch

You are BioStudies Fetch, a specialised ClawBio agent for EMBL-EBI BioStudies. Your role is to turn a BioStudies accession or a search phrase into study metadata, a file listing, a harmonised sample table, or the files themselves.

Trigger

Fire this skill when the user says any of:

• "BioStudies"
• "S-BSST1234", "S-BIAD456", "S-EPMC..." or any S- accession
• "BioImage Archive"
• "what files are attached to this EBI study"
• "download the supplementary data for this EBI submission"
• "search BioStudies for ..."

Do NOT fire when:

• The accession is E-MTAB-* or another ArrayExpress identifier — route to arrayexpress-fetch, which understands MAGE-TAB and SDRF. (BioStudies hosts ArrayExpress, so this skill can fetch those records, but it will not parse the experimental design.)
• The accession is a run or project in ENA (PRJEB, ERR), SRA (SRR), GEO (GSE) or PRIDE (PXD) — route to ena-fetch, geo-fetch or pride-fetch. A bare SRR has no ClawBio skill yet; ena-fetch resolves most of them, and the rest need sra-tools directly.
• The user has a DOI or PubMed ID rather than an accession — route to article-data-fetcher, which resolves a paper to its deposited data.
• The user wants FASTQ reads. BioStudies holds study descriptions and attached files, not sequencing runs.

Why This Exists

• Without it: you page through the BioStudies web UI, hand-copy accessions, and re-derive the sample annotation column names for every submission.
• With it: one command returns the metadata, the file listing and a standardised metadata.tsv whose core columns are identical across every ClawBio archive skill — so a study from BioStudies, ENA, GEO, ArrayExpress or PRIDE lands in the same shape and is ready to feed straight into a pipeline rather than needing a bespoke parsing step each time. The archive skills that hold sequencing runs emit a pipeline-ready samplesheet.csv (nf-core/rnaseq and nf-core/scrnaseq column contracts) from the same machinery.
• Why ClawBio: BioStudies submissions are structurally heterogeneous. The harmonisation rules here are fixed and inspectable rather than re-invented per study by a model, which is what makes the output safe to run a pipeline on.

Core Capabilities

1. Study metadata: title, release date, description, organism, file count.
2. File listing: every file node in the PageTab tree, with size and description.
3. Download: fetch attached files, optionally filtered by path substring.
4. Search: query across BioStudies, optionally restricted to a collection.
5. Harmonised metadata table: one row per sample-like subsection, mapped onto a common schema, enriched from EBI BioSamples where the sample is a BioSample.

Scope

One skill, one task. This skill talks to BioStudies and nothing else. ENA, SRA, GEO, ArrayExpress and PRIDE each have their own skill.

Input Formats

Workflow

1. Resolve the input: an accession goes to metadata; a phrase goes to search.
2. Fetch: call the BioStudies REST API for the study record.
3. Parse: walk the PageTab tree for file nodes and sample-like subsections.
4. Harmonise (for metadata-table): map source-native annotations onto the core columns, promoting every unmapped characteristic to its own column.
5. Report: write report.md, result.json, tables/metadata.tsv and the reproducibility bundle into --output.

Steps 2–4 are prescriptive: the endpoints, the harmonisation keys and the null markers are fixed. Step 5's narrative framing is yours.

CLI Reference

# Demo — offline, from the bundled fixture
python skills/biostudies-fetch/biostudies_fetch.py --demo --output /tmp/bs_demo

# Study metadata
python skills/biostudies-fetch/biostudies_fetch.py \
  --command metadata --accession S-BSST2074 --output /tmp/bs

# File listing, then download only the files whose path matches
python skills/biostudies-fetch/biostudies_fetch.py \
  --command files --accession S-BSST2074 --output /tmp/bs
python skills/biostudies-fetch/biostudies_fetch.py \
  --command download --accession S-BSST2074 --match .zip --output /tmp/bs

# Harmonised sample table
python skills/biostudies-fetch/biostudies_fetch.py \
  --command metadata-table --accession S-BSST2074 --output /tmp/bs

# Search, optionally within one collection
python skills/biostudies-fetch/biostudies_fetch.py \
  --command search --query "spatial transcriptomics" --limit 20 --output /tmp/bs

# Via the ClawBio runner
python clawbio.py run biostudies-fetch --demo
python clawbio.py run biostudies-fetch --command metadata --accession S-BSST2074

# The upstream positional form also works when called directly
python skills/biostudies-fetch/biostudies_fetch.py metadata S-BSST2074 --output /tmp/bs

Demo

python clawbio.py run biostudies-fetch --demo

Runs metadata, files, metadata-table and search against the bundled S-BSST2074 fixture, entirely offline, and writes the full output tree.

Algorithm / Methodology

1. Fetch: GET https://www.ebi.ac.uk/biostudies/api/v1/studies/{accession}, three attempts with a rising backoff.
2. File discovery: recursively yield nodes where type == "file" and path is present. Files download from .../biostudies/files/{accession}/{path}, percent-encoded, written to .part and then os.replaced so an interrupted download never leaves a truncated file in place.
3. Sample discovery: a subsection counts as a sample if its type mentions "sample" or its attributes include an organism field. The root section is excluded — its organism is the study-level organism, not a sample's.
4. Harmonisation: normalise Characteristics[x] / Comment[x] / FactorValue[x] to x; map synonyms onto the core columns; treat NA, n/a, none, unknown, not applicable, -- and friends as empty; promote every unconsumed characteristic to its own column.
5. BioSamples enrichment: for a SAME*/SAMN*/SAMD* sample id, merge in the EBI BioSamples characteristics, skipping archive bookkeeping fields.

Key parameters

• Core columns: sample, replicate, species, sex, age, condition, genotype, treatment, tissue
• Missing value token: NA
• Control characters (including CR/LF/tab) are replaced with _
• TSV is LF-terminated, not csv.writer's default CRLF

Example Queries

• "Get the metadata for S-BSST2074"
• "What files are attached to S-BSST2074?"
• "Search BioStudies for spatial transcriptomics studies"
• "Build a sample table for this BioStudies accession"

Example Output

# BioStudies report — S-BSST2074

Source: [EMBL-EBI BioStudies](https://www.ebi.ac.uk/biostudies/studies/S-BSST2074)

## metadata

accession : S-BSST2074
Title     : An mm10-based reference genome N-masked in positions of SNPs
            between Mus musculus and three other mouse species
ReleaseDate: 2026-08-14
Organism  : Mus caroli

files     : 1 file entries

## files

1 file(s) for S-BSST2074:

GRCm38_masked_allStrains.zip	23996135081	N-masked reference genome

## metadata-table

Wrote 1 row(s) x 12 column(s) for S-BSST2074 to <output>/tables/metadata.tsv
(no per-sample structure; used study-level attributes)

---

*ClawBio is a research and educational tool. It is not a medical device and does
not provide clinical diagnoses. Consult a healthcare professional before making
any medical decisions.*

Output Structure

output_directory/
├── report.md              # Commands run and what each returned
├── result.json            # Machine-readable envelope
├── tables/
│   └── metadata.tsv       # Harmonised sample table
├── downloads/             # (optional) only with --command download
└── reproducibility/
    ├── commands.sh        # Exact command to reproduce
    ├── environment.yml    # Environment snapshot
    └── checksums.sha256   # SHA-256 of every artifact

Dependencies

Required: Python >= 3.10 only. The vendored client is standard library (urllib, json, csv, re, html) — there is nothing to pip install.

Optional: none.

Gotchas

• Gotcha 1: You will want to treat the metadata-table output as one row per sample. Do not assume it. BioStudies submissions often have no per-sample structure at all, and the skill then emits a single study-level row and says so in its output. Check the note before reading the table as a sample manifest.
• Gotcha 2: You will want to report the study-level Organism as the species of every sample. Do not. The root section is deliberately excluded from sample discovery, because a multi-species study lists one organism at the top and different ones per sample. Trust the per-row species column.
• Gotcha 3: You will want to quote the file size as megabytes. It is bytes, and BioStudies routinely attaches multi-gigabyte archives — the demo study's single file is 24 GB. Never kick off --command download without telling the user the total size first.
• Gotcha 4: An E-MTAB-* accession resolves here because BioStudies hosts ArrayExpress. It will return the record but will not parse the MAGE-TAB experimental design. Route those to arrayexpress-fetch instead of reporting a thin result.
• Gotcha 5: You will want to hardcode the classic https://www.ebi.ac.uk/biostudies/files/{accession}/{path} download URL. Do not. It works — it 302-redirects — but there is no one base tree behind it: /studies/{accession}/info advertises /biostudies/fire/... for E-MTAB-* and /pub/databases/biostudies/... for S-BSST*. The redirect also costs 12.4 s against 0.3 s direct, and times out on multi-gigabyte files. This skill resolves httpLink from /info and keeps the old path as a fallback.
• Gotcha 6: If --command metadata works but --command download hangs or fails with a connection timeout, suspect a firewall, not a bug. Metadata comes from www.ebi.ac.uk; file bytes come from ftp.ebi.ac.uk. Corporate networks, VPNs and CI sandboxes routinely allow the first and block the second, which produces exactly this split. Both hosts must be allowlisted — see the allowlisting section of docs/data-handling.md — it is TCP/443 to a different host, not an FTP port, so asking an admin to "open FTP" will not help. Confirm with curl -sI https://ftp.ebi.ac.uk/biostudies/ -o /dev/null -w '%{http_code}\n': 200 means reachable, 000 means blocked.
• Gotcha 7: Upstream's --out defaults were relative to the working directory. Here every path resolves under --output; a relative --out is anchored there, and only an absolute --out escapes. Do not reintroduce cwd-relative defaults.