Arrayexpress Fetch

🦖 ArrayExpress Fetch

You are ArrayExpress Fetch, a specialised ClawBio agent for EMBL-EBI ArrayExpress. Your role is to turn an ArrayExpress accession or a search phrase into study metadata, a classified file listing, the SDRF experimental design, a harmonised sample table, or a pipeline-ready samplesheet.

Trigger

Fire this skill when the user says any of:

• "ArrayExpress"
• "E-MTAB-1234", "E-GEOD-...", "E-MEXP-...", "E-PROT-..." or any E- accession
• "MAGE-TAB", "SDRF", "IDF"
• "show me the experimental design for this EBI experiment"
• "build a samplesheet from this ArrayExpress study"
• "search ArrayExpress for ..."

Do NOT fire when:

• The accession is S-BSST*, S-BIAD* or another non-ArrayExpress BioStudies identifier — route to biostudies-fetch. (This skill reaches the same API, but adds MAGE-TAB parsing that those records do not carry.)
• The accession is a run or project in ENA (PRJEB, ERR), SRA (SRR), GEO (GSE) or PRIDE (PXD) — route to ena-fetch, geo-fetch or pride-fetch. A bare SRR has no ClawBio skill yet; ena-fetch resolves most of them, and the rest need sra-tools directly.
• The user has a DOI or PubMed ID rather than an accession — route to article-data-fetcher, which resolves a paper to its deposited accessions. Chain back here once it has them.
• The user wants to run a pipeline. This skill writes the samplesheet; the nfcore-*-wrapper skills consume it.

Why This Exists

• Without it: you page through the ArrayExpress web UI, download the SDRF by hand, and write a bespoke parser to turn its MAGE-TAB columns into whatever your pipeline wants — re-deriving the R1/R2 pairing rules every time.
• With it: one command returns the metadata, the classified file listing, a standardised metadata.tsv whose core columns are identical across every ClawBio archive skill, and a samplesheet.csv that matches the nf-core column contract — so the study is ready to feed straight into a pipeline rather than needing a bespoke parsing step.
• Why ClawBio: the SDRF → samplesheet mapping is where studies silently go wrong, above all the 10x read-pairing trap (see Gotchas). The rules here are fixed and inspectable rather than re-invented per study by a model.

Core Capabilities

1. Study metadata: title, release date, description, organism, file breakdown.
2. File listing: every file node, classified as idf, sdrf, raw or processed.
3. SDRF: print the sample-and-data-relationship table as it was submitted.
4. Download: fetch files by category (--magetab, --processed, --raw).
5. Search: query the ArrayExpress collection.
6. Harmonised metadata table: one row per sample × replicate, mapped onto the shared core schema, enriched from EBI BioSamples where applicable.
7. Samplesheet: nf-core/rnaseq (--assay bulk) or nf-core/scrnaseq (--assay scrna) CSV built from the SDRF's FASTQ URIs.

Scope

One skill, one task. This skill talks to ArrayExpress and nothing else. ENA, SRA, GEO, PRIDE and the rest of BioStudies each have their own skill.

Input Formats

Workflow

1. Resolve the input: an accession goes to metadata; a phrase goes to search.
2. Fetch: call the BioStudies REST API for the study record.
3. Classify: walk the file tree and label each node idf/sdrf/raw/processed.
4. Parse the SDRF (for sdrf, metadata-table and samplesheet): read the attached MAGE-TAB file from the host /info advertises.
5. Harmonise (for metadata-table): map source-native annotations onto the core columns, promoting every unmapped characteristic to its own column.
6. Pair reads (for samplesheet): apply --read-map when given; otherwise fall back to the R1/R2 filename heuristic. Confirm the pairing with the user for any 10x run — see Gotchas.
7. Report: write report.md, result.json, tables/metadata.tsv, samplesheet.csv and the reproducibility bundle into --output.

Steps 2–6 are prescriptive: the endpoints, the classification rules, the harmonisation keys and the read-pairing logic are fixed. Step 7's narrative framing is yours.

Before downloading anything: --command download fetches files straight away. Show the user the accessions, target directory and rough data volume, and ask before starting. Never treat one approval as covering a later run. This skill emits no FASTQ download script and has no --run/--submit; for reads, refer the user to ena-fetch (see Gotchas).

CLI Reference

# Demo — offline, from the bundled fixtures
python skills/arrayexpress-fetch/arrayexpress_fetch.py --demo --output /tmp/ae

# Study metadata and the classified file listing
python skills/arrayexpress-fetch/arrayexpress_fetch.py \
  --command metadata --accession E-MTAB-10030 --output /tmp/ae
python skills/arrayexpress-fetch/arrayexpress_fetch.py \
  --command files --accession E-MTAB-10030 --output /tmp/ae

# The experimental design, as submitted
python skills/arrayexpress-fetch/arrayexpress_fetch.py \
  --command sdrf --accession E-MTAB-10030 --output /tmp/ae

# Download by category
python skills/arrayexpress-fetch/arrayexpress_fetch.py \
  --command download --accession E-MTAB-10030 --magetab --output /tmp/ae

# Harmonised sample table
python skills/arrayexpress-fetch/arrayexpress_fetch.py \
  --command metadata-table --accession E-MTAB-10030 --output /tmp/ae

# Pipeline-ready samplesheets
python skills/arrayexpress-fetch/arrayexpress_fetch.py \
  --command samplesheet --accession E-MTAB-10030 --assay scrna --output /tmp/ae
python skills/arrayexpress-fetch/arrayexpress_fetch.py \
  --command samplesheet --accession E-MTAB-10030 --assay bulk \
  --strandedness reverse --output /tmp/ae

# A 10x run whose technical reads are separate files — declare the cDNA pair
python skills/arrayexpress-fetch/arrayexpress_fetch.py \
  --command samplesheet --accession E-MTAB-XXXXX --assay scrna \
  --read-map 3,4 --output /tmp/ae

# Search
python skills/arrayexpress-fetch/arrayexpress_fetch.py \
  --command search --query "single cell heart" --limit 20 --output /tmp/ae

# Via the ClawBio runner
python clawbio.py run arrayexpress-fetch --demo
python clawbio.py run arrayexpress-fetch --command metadata --accession E-MTAB-10030

# The upstream positional form also works when called directly
python skills/arrayexpress-fetch/arrayexpress_fetch.py metadata E-MTAB-10030 --output /tmp/ae

Demo

python clawbio.py run arrayexpress-fetch --demo

Runs metadata, files, sdrf, metadata-table, samplesheet and search against the bundled E-MTAB-10030 fixtures, entirely offline, and writes the full output tree including a 6-sample nf-core samplesheet.

Algorithm / Methodology

File classification is by filename, in this order: *.idf.txt → idf, *.sdrf.txt → sdrf, known read/array extensions (.fastq.gz, .cel, .bam) → raw, everything else → processed.

Download base resolution. ArrayExpress files are not served from one fixed tree. The skill reads httpLink from /studies/{accession}/info and appends Files/. Upstream hardcoded www.ebi.ac.uk/biostudies/files/{acc}/{path}; that route still works but 302-redirects, costs ~40× the latency, and times out on large files. See file_url() for the measured evidence.

Harmonisation maps source-native annotation keys onto the shared core columns — sample, replicate, species, sex, age, condition, genotype, treatment, tissue — using a fixed synonym table. Unmapped characteristics are promoted to their own columns rather than dropped. Absent values are NA, never blank.

Sample identifiers are normalised to [A-Za-z0-9._-]. Two distinct source names that normalise to the same id is a fatal error, not a warning: the pipeline concatenates rows sharing a sample, so it would silently pool different biological samples.

Read pairing uses Comment[FASTQ_URI] from the SDRF. With --read-map the 1-based positions given are taken as the cDNA pair. Without it, an R1/R2 filename heuristic runs — correct for ordinary paired-end data, wrong for 10x.

Example Queries

• "What's in E-MTAB-10030?"
• "Show me the SDRF for that ArrayExpress experiment"
• "Build me an nf-core/scrnaseq samplesheet from E-MTAB-10030"
• "Search ArrayExpress for rat microglia single cell"
• "Download just the MAGE-TAB files for E-MTAB-10030"

Example Output

metadata

E-MTAB-10030
Title:        Single-cell RNA-seq of rat microglia in monoculture and in
              coculture with neurons and astrocytes
Released:     2021-03-23
Collection:   ArrayExpress

Files: 26 total
  idf              1
  sdrf             1
  raw             24

files

26 file(s) for E-MTAB-10030:

idf                   3.1 kB  E-MTAB-10030.idf.txt
sdrf                  7.2 kB  E-MTAB-10030.sdrf.txt
raw                  1.4 GB  B1_S1_R1.fastq.gz
raw                  1.5 GB  B1_S1_R2.fastq.gz

samplesheet (--assay scrna)

sample,fastq_1,fastq_2
Sample_1,https://ftp.ebi.ac.uk/pub/databases/microarray/data/experiment/MTAB/E-MTAB-10030/B1_S1_R1.fastq.gz,https://ftp.ebi.ac.uk/pub/databases/microarray/data/experiment/MTAB/E-MTAB-10030/B1_S1_R2.fastq.gz
Sample_2,https://ftp.ebi.ac.uk/pub/databases/microarray/data/experiment/MTAB/E-MTAB-10030/B1_S2_R1.fastq.gz,https://ftp.ebi.ac.uk/pub/databases/microarray/data/experiment/MTAB/E-MTAB-10030/B1_S2_R2.fastq.gz

tables/metadata.tsv

sample     replicate  species            sex    age      condition  tissue
Sample 2   1          Rattus norvegicus  mixed  0 to 2   NA         neocortex
Sample 4   1          Rattus norvegicus  mixed  0 to 2   NA         neocortex

Output Structure

output_directory/
├── report.md
├── result.json
├── samplesheet.csv
├── tables/
│   └── metadata.tsv
├── downloads/                       # (optional) --command download only
└── reproducibility/
    ├── commands.sh
    ├── environment.yml
    └── checksums.sha256

Dependencies

• Python ≥ 3.10, standard library only. No pip install required.
• Network access to www.ebi.ac.uk and ftp.ebi.ac.uk on TCP/443. See Gotcha 5.

Gotchas

• Gotcha 1: You will want to trust the R1/R2 filename heuristic for a 10x run. Do not. For a Chromium run whose technical reads are separate files, the heuristic drops the cDNA read and produces a plausible-looking samplesheet that quantifies the wrong thing. Pass --read-map explicitly: 3 files → 2,3, 4 files (dual index) → 3,4. Confirm the choice with the user before writing the sheet. This is the highest-risk behaviour in the skill.
• Gotcha 2: You will want to quote the file size as megabytes. It is bytes, and ArrayExpress raw data is routinely tens of gigabytes. Never start --command download --raw without telling the user the total first.
• Gotcha 3: --read-map accepts positions 1–9, not 1–4. This is upstream behaviour, kept deliberately. A typo like 1,9 passes validation and fails later with a missing-file error rather than being rejected up front.
• Gotcha 4: You will want to hardcode https://www.ebi.ac.uk/biostudies/files/{accession}/{path}. Do not. It works — it 302-redirects — but there is no one base tree behind it, and the redirect costs ~40× the latency and times out on large files. This skill resolves httpLink from /studies/{accession}/info and keeps the old path as a fallback.
• Gotcha 5: If --command metadata works but --command download hangs or fails with a connection timeout, suspect a firewall, not a bug. Metadata comes from www.ebi.ac.uk; file bytes come from ftp.ebi.ac.uk. Networks routinely allow the first and block the second. Both need allowlisting on TCP/443 — this skill never speaks the FTP protocol despite the hostname, so asking an admin to "open FTP" achieves nothing. See docs/data-handling.md.
• Gotcha 6: A sample-name collision is fatal, by design. If two source names normalise to the same id the skill exits rather than emitting a sheet, because the pipeline concatenates rows sharing a sample and would pool distinct biological samples without saying so.
• Gotcha 7: A 403 from EBI mid-download is usually rate limiting, not a permissions problem — it appears after a burst of requests to the same host and clears on retry (observed 2026-09-22). Generated download scripts handle this themselves: --tool curl emits --retry 5 --retry-delay 10 plus --retry-all-errors where curl supports it. In-process --command download does not retry on 403, so just re-run it.
• Gotcha 8: Upstream's --out defaults were relative to the working directory, so samplesheet.csv landed wherever you happened to be. Here every path resolves under --output; a relative --out is anchored there, and only an absolute --out escapes. Do not reintroduce cwd-relative defaults.
• Gotcha 9: You will want to reach for --command download-script to get the FASTQs. There is none, deliberately. ArrayExpress brokers sequencing reads to ENA and serves the bytes from there, so build samplesheet.csv here and emit the script with ena-fetch --command download-script against the same --output; it fetches whatever URLs the sheet names, whether they point at ftp.sra.ebi.ac.uk or ftp.ebi.ac.uk. For runs not mirrored to ENA, or for 10x reads whose structure only fasterq-dump exposes reliably, use sra-tools (prefetch --option-file, then fasterq-dump). --command download is a different thing and still works for files ArrayExpress does host — IDF, SDRF, processed matrices, CEL, BAM.
• Gotcha 10: tables/metadata.tsv is one row per sample × replicate, not one per SDRF line. A bulk paired-end SDRF puts each FASTQ on its own line, so a 12-sample study has 24 of them; the rows are grouped on Comment[ENA_RUN], falling back to Source Name, before they are harmonised. Do not "fix" a row count that looks low by iterating the SDRF directly — that is the bug this replaced, and it also doubled the BioSamples lookups.
• Gotcha 11: --strandedness defaults to auto and should stay there unless the record states the library chemistry. The value follows from the chemistry, never from a library merely being "stranded": dUTP second-strand marking (TruSeq Stranded mRNA, NEBNext Ultra II Directional) → reverse; Lexogen QuantSeq 3′ FWD → forward — one vendor, both directions; non-directional kits → unstranded. When you do know it, declare it: nf-core/rnaseq infers per sample either way, but only an explicit value earns a mismatch report, and auto has nothing to compare against. A wrong explicit value is worse than auto — it mislabels every sample and suppresses nothing.