--- name: module-authoring description: Author, resolve, compile and publish a just-dna annotation module — the spec directory layout, the CSV column contracts and vocabularies, the enrich→compile pipeline, and the checks that decide whether a module will publish. Use when creating a new annotation module, editing an existing spec (variants.csv, pharm_variants.csv, module_spec.yaml), debugging a validate/compile failure, or preparing a module for the registry. --- # Authoring an annotation module A module is a **directory of authored CSVs plus a YAML header**. A compiler turns it into parquet with a content-addressed manifest. You never write parquet by hand and never commit coordinates you looked up yourself — a separate resolution step fills those and records where they came from. Three packages, three jobs. Keep them straight; most confusion comes from mixing them up. | package | CLI | does | never does | |---|---|---|---| | `just-dna-format` | — | the schema: models, vocabularies, identity rules | touch the network | | `just-dna-compiler` | `just-dna-compiler` | spec directory → parquet + `manifest.json` | touch the network | | `just-dna-enricher` | `just-dna-enricher` | resolve rsIDs→coordinates, fill citations, mint VRS ids | decide what a variant *means* | The compiler is **inject-only**: it reads a `resolution.csv` the enricher produced. It will not go and look a coordinate up for you. ## The workflow ```bash uv run just-dna-compiler scaffold my_module --name my_module # 1. skeleton # 2. fill the YAML placeholders # 3. author the CSVs by hand uv run just-dna-compiler hint variants.csv --file my_module/variants.csv # 4. writes nothing uv run just-dna-enricher enrich my_module # 5. → resolution.csv uv run just-dna-enricher literature my_module # 6. → literature.csv (online) uv run just-dna-compiler validate my_module --strict # 7. the publish gate uv run just-dna-compiler compile my_module my_module/out --strict # 8. → parquet + manifest.json ``` **Step 2 is not optional.** `scaffold` writes `<>` into `module.title`, `description` and `report_title`; leaving any of them fails validation with *"unreplaced template placeholder"*. The same convention appears wherever a tool refuses to invent a value for you. `hint` takes the **table kind as a positional** plus `--file` (or `--row` for inline text) — it lints CSV text, not a directory. Its `info:` lines are worth reading: they name the columns deliberately left to you, because filling them from the same source a later check compares them against would make that check vacuous. Steps 5 and 6 are the only ones that use the network. Once `resolution.csv` and `literature.csv` exist they *are* the pin: every later compile is offline and reproducible. A successful compile prints four hashes — `digest`, `content_signature`, `resolution_signature`, and the resolution mode. Recompiling an untouched spec must reproduce all of them. In this repo the same tiers are mounted on one CLI, which is usually what you want because it loads `.env` for cache paths: `uv run pipelines module …`, `uv run pipelines enrich …`. ## Directory layout ``` my_module/ module_spec.yaml # required: identity + display variants.csv # the lead table (or pharm_variants.csv, diplotypes.csv, pgs.csv …) studies.csv # required when variants.csv is present: the grounding resolution.csv # produced by `enrich` — coordinates + VRS ids. Commit it. literature.csv # produced by `literature` — PMID/DOI existence. Commit it. licensing.csv # required when data came from a licence-bearing source # (`sources.csv` is the deprecated 0.5 spelling — read, warned, gone at 1.0) logo.png # optional ``` One CSV = one concern. A module leads with **exactly one** primary table. A drug-response module carries `pharm_variants.csv` and **no** `variants.csv`. ## module_spec.yaml ```yaml schema_version: "1.0" # always this module: name: my_module # lowercase alphanumeric + underscores. `my-module` is rejected. version: "1.0.0" # SemVer STRING. Unquoted 1 parses as int and is rejected. title: My Module description: One sentence a non-specialist can read. report_title: What the report section is called icon: heart color: "#db2828" genome_build: GRCh38 license: CC0-1.0 # SPDX id; must not contradict licensing.csv authorship: - who: your-name role: created kind: [human] ``` `module:` is `extra="forbid"` — a typo like `colour:` is a hard error, not a silent drop. ## variants.csv **Always required:** `genotype`, `state`, `conclusion` **Identity — one of:** `rsid` **or** `chrom` + `start` **Optional:** `ref`, `alts`, `weight`, `negatives`, `priority`, `gene`, `phenotype`, `category`, `clinvar`, `pathogenic`, `benign`, `curator`, `method`, `direction`, `stat_significance`, `effect_size`, `effect_measure`, `effect_allele`, `flags`, `trait_efo_id`, `clin_sig`, `requires_callable`, `acmg_sf`, `actionability`, `callable_from`, `quality_from`, `min_quality` Do **not** author `variant_key` or `authored_ident` — the compiler derives them, and `variant_key` is frozen at load, so an authored one is not overwritten. ```csv rsid,genotype,weight,state,conclusion,gene,clin_sig rs1801133,A/A,-0.5,risk,Reduced MTHFR activity; homozygous,MTHFR, rs1801133,A/G,-0.25,risk,Reduced MTHFR activity; heterozygous,MTHFR, rs1801133,G/G,0.0,neutral,Normal MTHFR activity,MTHFR, ``` ### Vocabularies (closed — anything else is rejected) - **`state`**: `alt`, `neutral`, `protective`, `ref`, `risk`, `significant` - **`direction`**: `neutral`, `protective`, `risk`, `unknown` - **`stat_significance`**: `not_significant`, `significant`, `suggestive`, `unknown` - **`clin_sig`**: `affects`, `association`, `benign`, `conflicting`, `drug_response`, `likely_benign`, `likely_pathogenic`, `not_provided`, `other`, `pathogenic`, `protective`, `risk_factor`, `uncertain_significance` - **`flags`** (open list, `;`-separated in a cell; these are reserved): `conditional`, `phased`, `pleiotropic` - **`chrom`**: `1`–`22`, `X`, `Y`, `MT` — no `chr` prefix (`chr1` is normalized, `NC_…` is not) ### Genotype rules 1. **Alphabetically sorted.** `A/G`, never `G/A`. An unphased genotype is a *set*; two spellings of one call would be two rows. 2. **Alleles are `[ACGT]+`**, and must be drawn from `{ref} ∪ alts` at that locus. A genotype whose alleles are not at the locus can never match a VCF. 3. **Non-diploid contigs take a single allele.** On `MT` (haploid) and on `Y` outside the pseudoautosomal regions (hemizygous) write `G`, not `G/G` or `A/G` — a two-allele call there asserts a second copy that does not exist. The compiler warns if you get this wrong, but only warns, and the warning is aggregated, so on a large module it is easy to miss. PAR1/PAR2 on Y *are* diploid; a mixed mitochondrial population is heteroplasmy and belongs in `heteroplasmy.csv`, not in a het genotype. 4. **Indels are spelled out**: `A/AG`, `C/CTT` — reference-anchored, VCF convention. 5. `ref`/`alts` may only appear **with** `chrom`+`start`. You cannot attach alleles to a bare rsID. ### Author by identity, not by coordinate Prefer `rsid` alone and let `enrich` fill the coordinate. Author `chrom`+`start`+`ref`+`alts` only when there is no rsID (roughly 10% of ClinVar pathogenic variants), or when one rsID names several alleles at a locus and the row must say which. ## studies.csv **Always required:** `pmid`. **Identity — one of:** `rsid` or `chrom` (+`start`, `ref`). Optional: `population`, `p_value`, `conclusion`, `study_design`, `doi`, `trait_efo_id`, `effect_size`, `effect_measure`, `stat_significance`, `provenance_quote`, `provenance_regex`. - A study **must carry the same identity its variant row got.** If the variant is keyed by coordinate, the study must be too, or it is an orphan. - **`pmid` is 1–8 digits.** Nine-digit ids are not PubMed ids and are rejected. - Never invent a PMID. Verify each one resolves before writing it. ## pharm_variants.csv (drug response) **Always required:** `drug`, `conclusion`. **Identity — one of:** `rsid` or `chrom`+`start`. Optional: `ref`, `gene`, `genotype`, `phenotype_category`, `annotation_id`, `response`, `evidence_level`, `trait_efo_id`. The duplicate key is `(variant, drug, genotype, phenotype_category, annotation_id)` — one variant and drug legitimately carry separate efficacy, toxicity and pharmacokinetic rows, and they can disagree. `phenotype_category` is closed: `dosage`, `efficacy`, `metabolism_pk`, `other`, `pd`, `toxicity`. This module type carries **no** `variants.csv` and needs no `studies.csv`. ## resolution.csv — produced, committed, never hand-edited `enrich` writes one row per resolved locus: `variant_key, rsid, chrom, start, ref, alts, genome_build, vrs_id, source, status, …`. It is what makes a compile offline and reproducible, and it travels with the module. - **Existing rows are authoritative and merged, never overwritten.** To re-resolve after changing the authored table, **delete `resolution.csv` first** — otherwise stale rows survive silently. - A locus whose authored genotype it cannot host is **left out** and reported. That is deliberate: recording it would hand the compiler a locus it must drop. - `--offline` restricts to local caches. Substitution VRS ids mint offline; **indels and MNVs need the reference sequence**, so an offline run leaves them unminted (expect ~50% coverage on an indel-heavy module, ~99% online). ## licensing.csv and licensing **The file is `licensing.csv` from format 0.6 on.** `sources.csv` is the deprecated spelling: still read, warned about, and removed at 1.0 (RM51). A drafting pass writes whichever copy the module already carries and creates the new name when there is none, so you normally never choose — but if you are hand-editing, use `licensing.csv`, and **never let both exist**: two copies of a fact-hashed, hand-editable table are two claims, so the compiler refuses rather than picking a winner. Reach for it in code through `just_dna_format.layout` (`resolve_sidecar`, `sidecar_write_path`), never by name. **The rename stops at the file.** The compiled parquet is still `sources.parquet` and the manifest key is still `manifest.sources`, both for the whole 0.x tail, because they sit inside `artifact.digest` or are published keys. So a module reads `licensing.csv` → `sources.parquet` → `manifest.sources`. That is a real legibility cost, taken knowingly; do not "finish" the rename. Any module built from a licence-bearing source needs a `SourceRow` recording the terms. Passes that read such a source write it for you. Two rules that bite: - The compiler **refuses to build** content from a no-sale source unless `declared_use` is recorded. Delete the cell and the compile fails — that is the gate working. - `license:` in the YAML must not contradict `licensing.csv`. A ClinVar module declaring `CC0-1.0` warns, because the source row says `public-domain`; they are the same grant, but the check compares spellings. Match the source's spelling. ## Verification — and what `--strict` changes ```bash uv run just-dna-compiler hint variants.csv --file my_module/variants.csv # rewrites nothing uv run just-dna-compiler validate my_module --strict uv run just-dna-compiler signature my_module # content signature, no compile uv run just-dna-compiler compile my_module out/ --strict ``` **Author against `--strict`, because that is what a registry runs.** The difference is not cosmetic: | condition | plain | `--strict` | |---|---|---| | genotype allele not among the locus's alleles | warning, **valid** | **error, invalid** | | two-allele genotype on `MT`/`Y` | warning | warning | | unresolved rows (no coordinate) | warning | counts against publishability | A plain `compile` **succeeds** through both of the above. So "it compiled" is not evidence the module is correct — a module can compile cleanly and contain rows that will never match a genome. Check what you shipped, don't assume: ```bash uv run python -c " import polars as pl; w = pl.read_parquet('out/weights.parquet') print(w.height, 'rows;', w.filter(pl.col('chrom').is_not_null()).height, 'with a coordinate')" ``` `0 with a coordinate` means resolution did not reach the compile — see the trap below. ## Traps that cost real time Current as of **compiler/enricher 0.5.2**. **`compile_module(resolve_with_ensembl=False)` disables `resolution.csv` too.** The name reads as "don't use Ensembl", which is exactly what a spec carrying its own resolution wants. It is the master switch for *all* resolution: set it False and every row compiles with `chrom=None`, and the compile **succeeds** — it warns, but a script checking only the exit status ships a module that can never match a genome. The correct call is `resolve_with_ensembl=True, ensembl_cache=None`: switch on, no cache, injected-table path. **Deleting `resolution.csv` is part of a rebuild.** Existing rows are authoritative and merged, so a fix that changes an authored allele will not show up until you delete the file first. That is deliberate — the table is a pin, not a cache. **A drafted panel does not need a zygosity decision on every row.** `draft_gene_panel` writes the sole expressible genotype where the contig leaves nothing open — the mitochondrial genome, and chrY outside the pseudoautosomal regions, decided per locus — and keeps `<>` only where a real judgement remains. If you expand placeholders into both zygosities, expand *only* what is still a placeholder; do not key that off the contig yourself. **`licensing.csv` must cover every source your fact tables cite**, including PubMed if you carry studies. A missing row is a warning, not an error, so it is easy to ship without noticing. **A re-draft always changes `artifact.digest`, even when the data is identical.** `licensing.csv` carries a `fetched_at` timestamp stamped when the row is written, and `sources.parquet` is one of the parquets the digest is a Merkle root over — nineteen of them as of 0.6, not four — so two builds of byte-identical content, an hour apart, are two different artifacts. Verified by changing *only* `fetched_at` and recompiling: the digest moves. Consequences worth planning around: - **Recompiling is reproducible; re-drafting is not.** `compile` twice on an untouched spec gives the same digest every time. That is the property to test, and the checklist below says so. - **Do not treat a digest change as evidence that content changed.** Diff the tables. - **Digest-based dedup will miss matches** across rebuilds, so `find-by-hash` cannot recognise a module you rebuilt without editing. If you need a rebuild to be digest-stable, keep the previous `licensing.csv` rather than letting the draft re-stamp it. **Upgrading a module to 0.6 moves its `artifact.digest` on its own**, with no edit at all: the compiler emits new stamped columns (`weights.parquet` went 37 → 39). `content_signature` — the authored-content identity, and the one that claims a dedup slot — does **not** move, measured at 0/11 and 0/16 upstream. So re-pin stored digests at the version boundary and do not read the change as a content change. ## Publishing Version deliberately. A rebuild that changes the compiled shape still moves `artifact.digest`, so it needs a version either way; a rebuild that changes *what variants are in the module* or how they are grounded is a **major**, because someone pinned to the old major would silently receive different content. ```bash uv run pipelines marketplace validate # server-side, no publish uv run pipelines marketplace check --identifiers uv run pipelines marketplace publish --changelog "…" ``` - `check` = validate **plus** network checks (`ref` against the genome, rsID currency, VRS coverage); it returns `would_publish`, the one field to branch on. It has a variant ceiling, so a large module gets `422 too_many_variants` — use `validate`, which has no network tier and decides publishability. - A spec whose raw parts exceed the server's transfer bound needs `--pack` (client-side tar.gz). - Write the changelog as a continuation of the previous one, not a fresh "initial release". There is a **second, separate** destination: the HuggingFace annotator collection, which the app discovers directly. It takes the **compiled** artifacts rather than the spec, and the two are published independently — no command does both, and that is deliberate for now. ```bash uv run pipelines v1-port publish --dry-run # prints the exact file list uv run pipelines v1-port publish # needs HF_TOKEN / `hf auth login` ``` Discovery decides a directory is a module by probing every family in `module_config.LEAD_TABLES`, so a module led by a 0.4 table (`pharm_variants.parquet`, `diplotypes.parquet`, `pgs.parquet`, …) publishes here too — add a new family to that tuple and it becomes discoverable and publishable at once. Verify with `pipelines list-modules`, which answers whether the app can see the module rather than merely whether files landed. A 0.4-led module is joined against the VCF on **rsid + genotype**, not by position: the compiler materializes those families verbatim from their authored CSV and applies `resolution.csv` to `weights.parquet` only, so their `chrom`/`start` arrive null. `validate` and `compile` both warn about this per table, naming how many rows are unplaced and how many `resolution.csv` could place — the warning is expected on an rsid-authored PGx module, is never a `--strict` error, and is not something you can clear by editing the spec. Author the rsid, and expect no matches from a VCF whose `ID` column is empty. Such a module also publishes to the registry as `trusted: false`, which is the facet reporting that same fact rather than a problem with your module. ## Checklist before you call a module done - [ ] `validate --strict` passes - [ ] every weight row has a coordinate (or you can say why not) - [ ] genotypes sorted; single-allele on `MT`/`Y`; alleles drawn from the locus - [ ] every PMID verified to exist, 1–8 digits, and reachable from a weighted variant - [ ] `resolution.csv` and `literature.csv` committed alongside the CSVs - [ ] `licensing.csv` present (not `sources.csv`, and never both) and consistent with `license:` if a licensed source was used - [ ] `module.version` is a quoted SemVer string - [ ] a second **compile** of the untouched spec reproduces the same `artifact.digest` (a re-**draft** will not — `licensing.csv` re-stamps `fetched_at`, which is inside the digest)