Module Authoring

Authoring an annotation module

A module is a directory of authored CSVs plus a YAML header. A compiler turns it into parquet with a content-addressed manifest. You never write parquet by hand and never commit coordinates you looked up yourself — a separate resolution step fills those and records where they came from.

Three packages, three jobs. Keep them straight; most confusion comes from mixing them up.

The compiler is inject-only: it reads a resolution.csv the enricher produced. It will not go and look a coordinate up for you.

The workflow

uv run just-dna-compiler scaffold my_module --name my_module        # 1. skeleton
#                                                                     2. fill the YAML placeholders
#                                                                     3. author the CSVs by hand
uv run just-dna-compiler hint variants.csv --file my_module/variants.csv   # 4. writes nothing
uv run just-dna-enricher  enrich  my_module                         # 5. → resolution.csv
uv run just-dna-enricher  literature my_module                      # 6. → literature.csv (online)
uv run just-dna-compiler validate my_module --strict                # 7. the publish gate
uv run just-dna-compiler compile  my_module my_module/out --strict  # 8. → parquet + manifest.json

Step 2 is not optional. scaffold writes <<REPLACE>> into module.title, description and report_title; leaving any of them fails validation with "unreplaced template placeholder". The same convention appears wherever a tool refuses to invent a value for you.

hint takes the table kind as a positional plus --file (or --row for inline text) — it lints CSV text, not a directory. Its info: lines are worth reading: they name the columns deliberately left to you, because filling them from the same source a later check compares them against would make that check vacuous.

Steps 5 and 6 are the only ones that use the network. Once resolution.csv and literature.csv exist they are the pin: every later compile is offline and reproducible.

A successful compile prints four hashes — digest, content_signature, resolution_signature, and the resolution mode. Recompiling an untouched spec must reproduce all of them.

In this repo the same tiers are mounted on one CLI, which is usually what you want because it loads .env for cache paths: uv run pipelines module …, uv run pipelines enrich ….

Directory layout

my_module/
  module_spec.yaml     # required: identity + display
  variants.csv         # the lead table (or pharm_variants.csv, diplotypes.csv, pgs.csv …)
  studies.csv          # required when variants.csv is present: the grounding
  resolution.csv       # produced by `enrich` — coordinates + VRS ids. Commit it.
  literature.csv       # produced by `literature` — PMID/DOI existence. Commit it.
  licensing.csv        # required when data came from a licence-bearing source
                       # (`sources.csv` is the deprecated 0.5 spelling — read, warned, gone at 1.0)
  logo.png             # optional

One CSV = one concern. A module leads with exactly one primary table. A drug-response module carries pharm_variants.csv and no variants.csv.

module_spec.yaml

schema_version: "1.0"          # always this
module:
  name: my_module              # lowercase alphanumeric + underscores. `my-module` is rejected.
  version: "1.0.0"             # SemVer STRING. Unquoted 1 parses as int and is rejected.
  title: My Module
  description: One sentence a non-specialist can read.
  report_title: What the report section is called
  icon: heart
  color: "#db2828"
genome_build: GRCh38
license: CC0-1.0               # SPDX id; must not contradict licensing.csv
authorship:
  - who: your-name
    role: created
    kind: [human]

module: is extra="forbid" — a typo like colour: is a hard error, not a silent drop.

variants.csv

Always required: genotype, state, conclusion Identity — one of: rsid or chrom + start Optional: ref, alts, weight, negatives, priority, gene, phenotype, category, clinvar, pathogenic, benign, curator, method, direction, stat_significance, effect_size, effect_measure, effect_allele, flags, trait_efo_id, clin_sig, requires_callable, acmg_sf, actionability, callable_from, quality_from, min_quality

Do not author variant_key or authored_ident — the compiler derives them, and variant_key is frozen at load, so an authored one is not overwritten.

rsid,genotype,weight,state,conclusion,gene,clin_sig
rs1801133,A/A,-0.5,risk,Reduced MTHFR activity; homozygous,MTHFR,
rs1801133,A/G,-0.25,risk,Reduced MTHFR activity; heterozygous,MTHFR,
rs1801133,G/G,0.0,neutral,Normal MTHFR activity,MTHFR,

Vocabularies (closed — anything else is rejected)

• state: alt, neutral, protective, ref, risk, significant
• direction: neutral, protective, risk, unknown
• stat_significance: not_significant, significant, suggestive, unknown
• clin_sig: affects, association, benign, conflicting, drug_response, likely_benign, likely_pathogenic, not_provided, other, pathogenic, protective, risk_factor, uncertain_significance
• flags (open list, ;-separated in a cell; these are reserved): conditional, phased, pleiotropic
• chrom: 1–22, X, Y, MT — no chr prefix (chr1 is normalized, NC_… is not)

Genotype rules

1. Alphabetically sorted. A/G, never G/A. An unphased genotype is a set; two spellings of one call would be two rows.
2. Alleles are [ACGT]+, and must be drawn from {ref} ∪ alts at that locus. A genotype whose alleles are not at the locus can never match a VCF.
3. Non-diploid contigs take a single allele. On MT (haploid) and on Y outside the pseudoautosomal regions (hemizygous) write G, not G/G or A/G — a two-allele call there asserts a second copy that does not exist. The compiler warns if you get this wrong, but only warns, and the warning is aggregated, so on a large module it is easy to miss. PAR1/PAR2 on Y are diploid; a mixed mitochondrial population is heteroplasmy and belongs in heteroplasmy.csv, not in a het genotype.
4. Indels are spelled out: A/AG, C/CTT — reference-anchored, VCF convention.
5. ref/alts may only appear with chrom+start. You cannot attach alleles to a bare rsID.

Author by identity, not by coordinate

Prefer rsid alone and let enrich fill the coordinate. Author chrom+start+ref+alts only when there is no rsID (roughly 10% of ClinVar pathogenic variants), or when one rsID names several alleles at a locus and the row must say which.

studies.csv

Always required: pmid. Identity — one of: rsid or chrom (+start, ref). Optional: population, p_value, conclusion, study_design, doi, trait_efo_id, effect_size, effect_measure, stat_significance, provenance_quote, provenance_regex.

• A study must carry the same identity its variant row got. If the variant is keyed by coordinate, the study must be too, or it is an orphan.
• pmid is 1–8 digits. Nine-digit ids are not PubMed ids and are rejected.
• Never invent a PMID. Verify each one resolves before writing it.

pharm_variants.csv (drug response)

Always required: drug, conclusion. Identity — one of: rsid or chrom+start. Optional: ref, gene, genotype, phenotype_category, annotation_id, response, evidence_level, trait_efo_id.

The duplicate key is (variant, drug, genotype, phenotype_category, annotation_id) — one variant and drug legitimately carry separate efficacy, toxicity and pharmacokinetic rows, and they can disagree. phenotype_category is closed: dosage, efficacy, metabolism_pk, other, pd, toxicity. This module type carries no variants.csv and needs no studies.csv.

resolution.csv — produced, committed, never hand-edited

enrich writes one row per resolved locus: variant_key, rsid, chrom, start, ref, alts, genome_build, vrs_id, source, status, …. It is what makes a compile offline and reproducible, and it travels with the module.

• Existing rows are authoritative and merged, never overwritten. To re-resolve after changing the authored table, delete resolution.csv first — otherwise stale rows survive silently.
• A locus whose authored genotype it cannot host is left out and reported. That is deliberate: recording it would hand the compiler a locus it must drop.
• --offline restricts to local caches. Substitution VRS ids mint offline; indels and MNVs need the reference sequence, so an offline run leaves them unminted (expect ~50% coverage on an indel-heavy module, ~99% online).

licensing.csv and licensing

The file is licensing.csv from format 0.6 on. sources.csv is the deprecated spelling: still read, warned about, and removed at 1.0 (RM51). A drafting pass writes whichever copy the module already carries and creates the new name when there is none, so you normally never choose — but if you are hand-editing, use licensing.csv, and never let both exist: two copies of a fact-hashed, hand-editable table are two claims, so the compiler refuses rather than picking a winner. Reach for it in code through just_dna_format.layout (resolve_sidecar, sidecar_write_path), never by name.

The rename stops at the file. The compiled parquet is still sources.parquet and the manifest key is still manifest.sources, both for the whole 0.x tail, because they sit inside artifact.digest or are published keys. So a module reads licensing.csv → sources.parquet → manifest.sources. That is a real legibility cost, taken knowingly; do not "finish" the rename.

Any module built from a licence-bearing source needs a SourceRow recording the terms. Passes that read such a source write it for you. Two rules that bite:

• The compiler refuses to build content from a no-sale source unless declared_use is recorded. Delete the cell and the compile fails — that is the gate working.
• license: in the YAML must not contradict licensing.csv. A ClinVar module declaring CC0-1.0 warns, because the source row says public-domain; they are the same grant, but the check compares spellings. Match the source's spelling.

Verification — and what --strict changes

uv run just-dna-compiler hint variants.csv --file my_module/variants.csv   # rewrites nothing
uv run just-dna-compiler validate  my_module --strict
uv run just-dna-compiler signature my_module    # content signature, no compile
uv run just-dna-compiler compile   my_module out/ --strict

Author against --strict, because that is what a registry runs. The difference is not cosmetic:

A plain compile succeeds through both of the above. So "it compiled" is not evidence the module is correct — a module can compile cleanly and contain rows that will never match a genome.

Check what you shipped, don't assume:

uv run python -c "
import polars as pl; w = pl.read_parquet('out/weights.parquet')
print(w.height, 'rows;', w.filter(pl.col('chrom').is_not_null()).height, 'with a coordinate')"

0 with a coordinate means resolution did not reach the compile — see the trap below.