Bio Filter Sequences

Version Compatibility

Reference examples tested with: BioPython 1.83+

Before using code patterns, verify installed versions match. If versions differ:

• Python: pip show <package> then help(module.function) to check signatures

If code throws ImportError, AttributeError, or TypeError, introspect the installed package and adapt the example to match the actual API rather than retrying.

Filter Sequences

"Filter sequences by length, quality, or content" -> Apply boolean criteria to a stream of sequence records and write survivors to output.

• Python: generator expression with SeqIO.parse() + SeqIO.write() (BioPython)
• CLI: seqkit seq -m 200 (SeqKit) or awk on FASTA

Filter and select sequences based on length, ID, GC content, N content, motifs, regex patterns, and description.

The governing principle

Two traps cause silent, downstream-corrupting errors. Neither raises an exception, so the agent must guard against both up front.

1. Never filter one mate of a paired-end set independently. Aligners (bwa, bowtie2) read R1 and R2 as two parallel streams and pair the i-th record of each, assuming SAME ORDER and SAME COUNT. Dropping a read from R1 alone DESYNCS the files: best case the aligner crashes on a name mismatch, worst case it silently pairs the wrong R1 with the wrong R2, producing mismapped reads and garbage insert sizes with no error. If the input is paired, route to the paired-end-fastq skill (synchronized filtering that writes matched output plus separate orphan/singleton files). Do NOT apply the single-file patterns below to one mate.
2. Filter by STREAMING, not by loading. SeqIO.parse() yields one record at a time; a generator expression into SeqIO.write() holds a single record in RAM regardless of file size. list(SeqIO.parse(...)) materializes every record and OOMs on large FASTQ. Only the patterns that genuinely need all records at once (random sampling, splitting into multiple files) load the file; they say so explicitly.

Required Imports

from Bio import SeqIO
from Bio.SeqUtils import gc_fraction

Core Pattern

Stream records through a generator expression so memory stays flat:

records = SeqIO.parse('input.fasta', 'fasta')
filtered = (rec for rec in records if len(rec.seq) >= 100)
SeqIO.write(filtered, 'output.fasta', 'fasta')

SeqIO.write() consumes the generator lazily and returns the count written.

Filter by Length

Minimum Length

records = SeqIO.parse('input.fasta', 'fasta')
long_seqs = (rec for rec in records if len(rec.seq) >= 500)
SeqIO.write(long_seqs, 'long.fasta', 'fasta')

Length Range

records = SeqIO.parse('input.fasta', 'fasta')
sized = (rec for rec in records if 100 <= len(rec.seq) <= 1000)
SeqIO.write(sized, 'sized.fasta', 'fasta')

Remove Short Sequences

min_length = 200
records = SeqIO.parse('input.fasta', 'fasta')
filtered = (rec for rec in records if len(rec.seq) >= min_length)
count = SeqIO.write(filtered, 'filtered.fasta', 'fasta')

len(rec.seq) counts every base, including soft-masked lowercase (see Case Sensitivity below).

Filter by ID

Select Specific IDs

wanted_ids = {'seq1', 'seq2', 'seq3'}
records = SeqIO.parse('input.fasta', 'fasta')
selected = (rec for rec in records if rec.id in wanted_ids)
SeqIO.write(selected, 'selected.fasta', 'fasta')

rec.id is the first whitespace-delimited token of the header. For CASAVA 1.8+ paired FASTQ, R1 and R2 share this id (the mate number lives after the space), so an id set matches both mates equally - another reason mate-aware filtering belongs in paired-end-fastq.

Select from ID File

Goal: Extract sequences whose IDs appear in an external list file.

Approach: Load IDs into a set for O(1) lookup, then stream-filter and write matches.

Reference (BioPython 1.83+):

with open('ids.txt') as f:
    wanted_ids = {line.strip() for line in f}

records = SeqIO.parse('input.fasta', 'fasta')
selected = (rec for rec in records if rec.id in wanted_ids)
SeqIO.write(selected, 'selected.fasta', 'fasta')

Exclude Specific IDs

exclude_ids = {'bad_seq1', 'bad_seq2'}
records = SeqIO.parse('input.fasta', 'fasta')
kept = (rec for rec in records if rec.id not in exclude_ids)
SeqIO.write(kept, 'kept.fasta', 'fasta')

Filter by ID Pattern

import re

pattern = re.compile(r'^chr\d+$')  # matches chr1, chr2, etc.
records = SeqIO.parse('input.fasta', 'fasta')
chromosomes = (rec for rec in records if pattern.match(rec.id))
SeqIO.write(chromosomes, 'chromosomes.fasta', 'fasta')

Filter by GC Content

Goal: Keep records whose GC fraction falls in a target band.

Approach: Use gc_fraction(), which returns a FRACTION (0-1), NOT a percentage - thresholds must be 0.4, not 40. The ambiguous= mode decides how N and other IUPAC ambiguity codes are counted, and the same sequence yields a different GC value per mode, so set it explicitly rather than relying on the default.

Reference (BioPython 1.83+):

from Bio.SeqUtils import gc_fraction

records = SeqIO.parse('input.fasta', 'fasta')
moderate_gc = (rec for rec in records if 0.4 <= gc_fraction(rec.seq, ambiguous='ignore') <= 0.6)
SeqIO.write(moderate_gc, 'moderate_gc.fasta', 'fasta')

Choosing the ambiguous= mode

gc_fraction(seq, ambiguous='remove') is the default. For the same sequence the three modes give different answers - an N-containing read can pass or fail purely because of the mode:

A naive (G+C)/len silently equals 'ignore' mode and under-reports GC whenever N's are present. The default 'remove' ignores N's entirely, so a heavily-N read can post a misleadingly extreme GC. Pick the mode that matches the intent and pass it explicitly.

High / Low GC bands

records = SeqIO.parse('input.fasta', 'fasta')
high_gc = (rec for rec in records if gc_fraction(rec.seq, ambiguous='ignore') >= 0.6)
SeqIO.write(high_gc, 'high_gc.fasta', 'fasta')