Bio Genome Annotation Repeat Annotation

Version Compatibility

Reference examples tested with: RepeatModeler 2.0.5+, RepeatMasker 4.1.5+, EDTA 2.1+, EarlGrey 4.0+, TEtranscripts 2.2+, matplotlib 3.8+, pandas 2.2+.

Before using code patterns, verify installed versions match. If versions differ:

• CLI: <tool> --version then <tool> --help to confirm flags
• Python: pip show <package> then help(module.function) to check signatures

The library database version matters as much as the binary: RepeatMasker now ships with Dfam (open); RepBase has been paywalled since May 2019, so any pipeline that "requires RepBase" is a reproducibility/access hazard - record the Dfam release and library provenance. If code throws an error, introspect the installed tool and adapt rather than retrying.

Repeat and Transposable Element Annotation

"Mask repeats in my genome assembly" -> Build a de novo repeat-family library, annotate copies genome-wide, and soft-mask them as a prerequisite for gene prediction.

• CLI: RepeatModeler -database mydb -LTRStruct (library), RepeatMasker -lib lib.fa -xsmall assembly.fa (soft-mask), or EarlGrey/EDTA.pl (wrappers)

The Single Most Important Modern Insight -- The Library Is the Experiment, and the Assembly Caps It

Two load-bearing truths the masker hides:

1. De novo library construction is a curation research project, not a button. A RepeatModeler2 run emits mydb-families.fa overnight - a draft of a draft: consensi are routinely 5'-truncated (L1 looks 1.5 kb when the active element is 6 kb), boundary-bled into flanking unique sequence, chimeric (two families merged), and 30-60% "Unknown" on a non-model genome. The dominant error in published TE annotations is the library, not the masker engine. Crucially, masking percentage is robust to a bad library (a chimeric consensus still masks roughly the right real estate), so the headline number survives while everything downstream rots: inflated family counts, wrong classification, distorted age landscapes, and - the killer - host-gene-contaminated consensi that silently mask real genes. Masking + gross % can use an automated library; any per-family biological claim (this family is young/active/novel) needs curation (Goubert 2022 Mob DNA 13:7; TE-Aid; MCHelper).

2. Annotation quality is capped by assembly quality. Short-read de Bruijn assemblers collapse near-identical TE copies and drop the youngest (most identical, most biologically active) ones, so short-read assemblies systematically under-count TEs and bias the age distribution toward "old" - which masquerades as the real signal "this lineage has no recent activity." Software cannot recover what the assembler threw away. Always ask what assembly a "% repeat" came from; HiFi/T2T raised the ceiling (LAI measures it) but T2T satellite/centromere repeats still exceed what the standard TE toolchain can annotate.

Tool Taxonomy

Dfam (open) vs RepBase (paywalled since 2019) is the database schism - modern open pipelines build on Dfam + de novo. Engine -e: rmblast (default, consensus FASTA) vs nhmmer (Dfam profile HMMs, more sensitive for ancient repeats, slower) - the same genome reads a higher % with HMM detection.

Decision Tree by Scenario

RepeatModeler2 -> RepeatMasker (the canonical pair)

# 1. De novo family discovery -> mydb-families.fa
BuildDatabase -name mydb assembly.fa
RepeatModeler -database mydb -threads 16 -LTRStruct   # -LTRStruct enables the LTR structural pipeline

# 2. (Recommended) decontaminate the library against host proteins, then UNION with Dfam clade
#    -> pull any consensus whose best hit is a host gene with no transposase/RT/integrase domain

# 3. Soft-mask against (custom library) for gene prediction
RepeatMasker -lib mydb-families.fa -xsmall -gff -e rmblast -pa 16 -dir rm_out assembly.fa

-xsmall = soft-mask (lowercase) - the key flag, the one people get wrong. Default .masked output hard-masks with N; -x masks with X. -nolow skips low-complexity/simple repeats (often wanted before gene prediction - see below). Outputs: .masked, .out, .tbl (summary %), .align (needed for the landscape).

Soft vs Hard Masking (Critical, and the Over-Masking Massacre)

• Gene prediction needs SOFT-masking. Modern predictors (AUGUSTUS/GeneMark/BRAKER) avoid nucleating models in lowercase but let exons extend into repeats - real genes have TE-derived exons and TE-filled introns. Hard-masking (N) destroys sequence: any gene overlapping a repeat is truncated or never called, and the predictor reports nothing - no error, no log. The gene is simply absent.
• Over-aggressive soft-masking is also a failure. A gene whose promoter sits in an LTR, or a young gene inside a recent TE burst, gets suppressed because the predictor won't start a model in a heavily-lowercased locus. Before gene prediction, soft-mask interspersed repeats only and skip low-complexity (-nolow) - simple repeats overlap real coding microsatellites and low-complexity protein domains.
• The domesticated-gene massacre. A de novo library contains fragments of real multicopy gene families because they look repetitive - and masking them deletes the genome's most interesting genes. Named casualties: RAG1/RAG2 (domesticated Transib transposase; V(D)J recombination), syncytins (captured retroviral env; placentation), SETMAR/Metnase (Hsmar1 mariner + SET domain; DNA repair), CENP-B (pogo transposase; centromere), and the KRAB-ZNF arrays (~350+ primate zinc-finger genes that exist to repress TEs - masking them deletes the genome's anti-TE machinery). Defense: decontaminate the library against a protein DB before masking (EDTA does a version; RepeatModeler2 does not by default). Treat any gene model falling entirely inside a masked "TE" as a flag to investigate, not a finished call.

TE Classification (Wicker-compatible)

• Class I (retrotransposons, copy-and-paste via RNA + RT): LTR-RTs (Ty3/Gypsy, Ty1/Copia - dominate large plant genomes), LINEs (autonomous, often 5'-truncated), SINEs (non-autonomous, e.g. Alu).
• Class II (DNA transposons, mostly cut-and-paste): TIR superfamilies (hAT, Tc1/Mariner, CACTA, PIF/Harbinger, Mutator), Helitrons (rolling-circle, can capture host genes), MITEs (non-autonomous TIR derivatives, EDTA reclassifies ≤600 bp).
• Family vs copy: a family is one consensus in the library (~thousands); a copy/insertion is one genomic locus matching it (millions). "% genome masked" counts copies; classification is family-level; "10,000 TEs" is ambiguous between the two.
• Full-length vs decayed: most copies are dead, truncated, point-mutated relics; only a tiny fraction are intact. Solo-LTR : full-length ratio is real biology (recombinational LTR-RT removal rate), measurable only if the assembly resolved full-length elements. Wicker 2007 Nat Rev Genet is the reference scheme; present Gypsy/Copia and the ICTV Metaviridae/Pseudoviridae names both.

Repeat Statistics and Age Landscape with Python

Goal: Summarize masked content by class and plot the Kimura-divergence landscape (a relative within-genome age readout).

Approach: Parse the RepeatMasker .out file, group by class for bp and genome fraction, then histogram percent divergence stratified by major TE class (x = divergence-from-consensus ~ relative age).

import pandas as pd

def parse_repeatmasker_out(out_file):
    records = []
    with open(out_file) as f:
        for i, line in enumerate(f):
            if i < 3:
                continue
            parts = line.split()
            if len(parts) < 15:
                continue
            records.append({'perc_div': float(parts[1]), 'seqid': parts[4],
                            'repeat_class': parts[10], 'length': int(parts[6]) - int(parts[5]) + 1})
    return pd.DataFrame(records)

def repeat_summary(rm_df, genome_size):
    by_class = rm_df.groupby('repeat_class')['length'].sum().sort_values(ascending=False)
    total = rm_df['length'].sum()
    print(f'Total masked: {total/genome_size:.1%} of genome (a LOWER bound; ancient copies decay past detection)')
    return by_class / genome_size * 100

The landscape is right-censored - the most ancient TEs decayed past alignment detection, so "no old activity" can mean "old activity is invisible." A sharp left (low-divergence) peak is a recent/ongoing burst; treat presence of a recent peak as informative and absence of an old hump cautiously. A truncated/chimeric consensus distorts the whole x-axis (another reason curation matters); never compare landscapes across genomes annotated with different libraries.