Bio Batch Processing

Version Compatibility

Reference examples tested with: BioPython 1.83+ (alternatives: pysam 0.22+, pyfastx 2.0+)

Before using code patterns, verify installed versions match. If versions differ:

• Python: pip show <package> then help(module.function) to check signatures

If code throws ImportError, AttributeError, or TypeError, introspect the installed package and adapt the example to match the actual API rather than retrying.

Batch Processing

"Process all my sequence files in a directory" -> Iterate, merge, split, convert, and summarize across multiple sequence files without loading everything into RAM.

• Python: SeqIO.parse() + Path.glob() (BioPython, pathlib) for streaming
• Python: SeqIO.index_db() (BioPython) for persistent random access across many files
• Python: pysam.FastxFile (pysam) or pyfastx for fast iteration over huge FASTQ

The Governing Principle

list(SeqIO.parse(...)) materializes every SeqRecord in RAM at once. On a directory of large files this causes OOM. SeqIO.parse() itself returns a generator that holds one record at a time, so streaming is the default for batch work: iterate, never list(), unless the file is known-small and needs multiple passes.

For random access across many or huge files, do not load them. SeqIO.index_db() builds one on-disk SQLite index over a list of files that persists across sessions. That, not to_dict(), is the batch random-access tool.

For tens of millions of reads, SeqIO is slow by design: it constructs a full SeqRecord (a Seq, id/name/description, and a letter_annotations dict of per-base qualities) for every read. When the job is plain linear iteration, a thinner reader wins.

Choosing a Reader

pysam.FastxFile exposes .name, .sequence, .comment, .quality, and .get_quality_array() (offset-removed int Phred, but it always subtracts 33, so it is correct only for Phred+33 data - for legacy Phred+64/Solexa stay on SeqIO with the explicit variant string). pyfastx builds a persistent .fxi/.fqi SQLite index and reads random records out of plain or gzipped files without re-bgzipping.

Required Imports

from pathlib import Path
from Bio import SeqIO

Iterate and Count Across Files

Count by iterating, never by building a list. len(list(SeqIO.parse(f))) loads the whole file; sum(1 for _ in ...) holds one record at a time.

for fasta_file in Path('data/').glob('*.fasta'):
    count = sum(1 for _ in SeqIO.parse(fasta_file, 'fasta'))
    print(f'{fasta_file.name}: {count} sequences')

Recursive search uses rglob:

for gb_file in Path('data/').rglob('*.gb'):
    print(f'Found: {gb_file}')

For huge FASTQ where only sequence content matters, skip SeqRecord construction entirely:

import pysam

with pysam.FastxFile('reads.fastq.gz') as fh:
    count = sum(1 for _ in fh)

Random Access Across Many Files

Goal: Look up records by id across a whole directory of files, repeatedly, without holding them in RAM.

Approach: Build one persistent on-disk SQLite index over the file list with index_db. Reopen later with just the index path; lookups reparse single records from disk on demand.

Reference (BioPython 1.83+):

from pathlib import Path
from Bio import SeqIO

files = [str(p) for p in Path('data/').glob('*.fasta')]
records = SeqIO.index_db('combined.idx', files, 'fasta')

print(len(records))            # total across all files
record = records['seq_00042']  # random access by id
records.close()

The index file persists. A later session calls SeqIO.index_db('combined.idx') with no file list and reopens instantly. Ids must be unique across the merged set: a collision raises ValueError: Duplicate key. index_db also indexes BGZF-compressed files; plain gzip is not seekable and cannot be indexed.

Merge Files

Goal: Concatenate sequences from many files into one output without loading them all.

Approach: Chain per-file generators with yield from and stream straight into SeqIO.write, which consumes the generator one record at a time.

Reference (BioPython 1.83+):

def all_records(directory, pattern, format):
    for filepath in Path(directory).glob(pattern):
        yield from SeqIO.parse(filepath, format)

count = SeqIO.write(all_records('data/', '*.fasta', 'fasta'), 'merged.fasta', 'fasta')
print(f'Merged {count} records')

Merge with Source Tracking

Goal: Combine sequences from multiple files, tagging each record with its source filename.

Approach: Stream records through a generator that appends source metadata to the description before writing.

Reference (BioPython 1.83+):

def records_with_source(directory, pattern, format):
    for filepath in Path(directory).glob(pattern):
        for record in SeqIO.parse(filepath, format):
            record.description = f'{record.description} [source={filepath.name}]'
            yield record

SeqIO.write(records_with_source('data/', '*.fasta', 'fasta'), 'merged_tracked.fasta', 'fasta')

When merging files that may share ids, decide upfront: write-then-merge tolerates duplicates (FASTA allows repeated ids), but any later index_db/to_dict over the merged file raises on the duplicate.

Split Files