Review Tests

Review Tests

Run the review-tests CLI and present the results to the user.

Runs are multi-profile: each component/tier is tested across up to four profiles -- docker (the reference baseline), conda, singularity_galaxy, singularity_pull. A cell is one (component, tier, profile). Most interpretation is a comparison of each profile against docker.

Steps

1. Run bactopia-review-tests via the wrapper script using the default text output (do NOT use --json):

   bash .agents/skills/review-tests/scripts/run-bactopia-review-tests.sh --bactopia-path /home/rpetit3/repos/bactopia/bactopia --silent

   If the user provided a timestamp argument (e.g., /review-tests 20260324_081306), add --run 20260324_081306.

2. Present the text output directly to the user. The CLI produces a clean summary with a "Status Breakdown by Profile" matrix and one section per failing status. Do NOT parse JSON or write extra code to reformat -- just relay the output with your interpretation.

3. Add interpretation and context after showing the output. Interpret by status (see the status reference below), and always frame failures as "which profiles differ from docker, and why." Summarize actionable items and next steps.

4. If the text output is too large for a single response, summarize the key sections (overview, status-by-profile matrix, failures) and note that per-file detail is in summary.json. Use --json/--pretty or read summary.json directly for structured detail.

Status reference (per-cell status)

• passed -- cell matched the committed snapshot / assertions.
• version_drift / output_drift / version+output_drift -- this profile's outputs differ from docker: version_drift = versions.yml (runtime resolved a different tool version than the docker container pin), output_drift = tool output file(s), version+output_drift = both. reason names the divergent fields. When docker passed and only conda/singularity drift, this is genuine dependency-solve divergence, not a bug. Fix = the sccmec-pattern test migration (md5 the profile-stable files, existence-check the divergent ones, versions -> contains('<tool>')) -- NOT snapshot regeneration. See files[] / suggested_edit in summary.json for the exact bucketing. (A pure version_drift may instead warrant updating the container version pin.)
• snapshot_mismatch -- the snapshot didn't match but the files matrix could not attribute it to specific fields (drift not subclassified). Inspect files[] and {profile}/stderr.txt.
• snapshot_stale -- the committed .snap no longer matches the reference runtime (docker); it shows on all profiles including docker. Fix: re-run with --generate under docker to re-baseline. NOT a content or tool problem (on generate=false, docker's own mismatch is promoted to this). Typical cause: a test's snapshot() shape was edited but .snap was never regenerated.
• assertion_failed -- a non-snapshot assertion failed (no output divergence detected). A test logic/assertion issue, not drift -- read {profile}/stderr.txt.
• non_reproducible -- two docker runs produced different snapshots; docker's own output is non-deterministic. Investigate the tool/test; regeneration will not fix it.
• build_failed -- the Conda env or Singularity image failed to build before testing. Infra: build the env/image, then re-run (it blocks triage of that profile).
• no_ground_truth -- docker established no snapshot for the non-docker profiles to validate against (usually docker itself failed to produce one).
• syntax_error -- the Nextflow script failed to compile. Fix the .nf.
• Housekeeping statuses you may also see: skipped, timeout (exceeded the per-run timeout), no_snapshot, n/a.
• undeclared_outputs -- the tool produced files not declared in the module's results, logs, versions, or nf_logs. For each file help the user route it:
  • results: a real tool output users want (report, summary, data file)
  • logs: stdout/stderr from the tool
  • .outputs-ignore: staging artifact, intermediate, version-info side effect, or DB file .outputs-ignore lives at modules/{name}/tests/.outputs-ignore (one glob per line; # comments and blanks allowed; staging/** is ignored by default). NOTE: this check only runs when the tool succeeds, so it is masked on a profile that tool_error'd -- use undeclared_outputs_union to see the full set.
• tool_error -- the tool crashed at runtime. Read error_class:
  • env_dependency -- conda/singularity re-solved a too-new interpreter/dependency (e.g. py>=3.12 pkg_resources, numpy2 newshape, biopython SeqFeature.strand, R readr/lifecycle deprecate_stop). Fix the env/recipe, NOT the test.
  • tool_crash / staging_bug / fs_permission / unknown -- fix the module/upstream or the workspace; when unknown, read the Command error: block. reason carries the real tool error (from Command error:), not the downstream nf-test NullPointerException.

generate gates interpretation (shown in Run Parameters and the # generate=<bool> header of summary.tsv):

• generate=true: the .snap was regenerated under docker first, so docker passing is the re-baseline; any drift shown is genuine (docker vs profile). snapshot_stale cannot occur.
• generate=false: docker also failing => snapshot_stale (run --generate). docker passing while a profile drifts => genuine content drift.

Multi-profile layout & summary.json

Structured results live at logs/run-tests/{ts}/summary.json (plus summary.tsv, whose first line is # generate=<bool>). Prefer summary.json for machine-readable detail; the CLI text is the human summary. Key fields:

• profiles[], reference_profile ("docker").
• results[].cells.{profile}: status, duration, reason, error_class (tool_error only), undeclared_outputs[].
• results[].undeclared_outputs_union: undeclared files unioned across profiles (unmasks profiles that tool_error'd).
• results[].files[]: per output file, the cross-profile md5 matrix -- process, scope (sample/run; subworkflow multi-record), field, name, md5:{profile -> hash|null}, verdict, divergent_profiles[], plus:
  • verdict: stable (equal across all profiles that ran) | divergent | indeterminate (a profile didn't produce it) | skip.
  • comparable: false = intrinsically non-hashable (gz / normalized -> byte md5 is meaningless) => bucket existence-only; verdict:"skip".
  • incomplete[]: profiles that produced no file (e.g. a tool_error'd conda) => verdict is indeterminate, NOT a false stable; re-check after fixing that profile.
  • kind:"versions" + tool_key: a versions.yml -> bucket to contains('<tool_key>'). This matrix is computed from actual runtime outputs, so it is populated even on passing or stale cells -- an always-on divergence diagnostic (also useful for add-* at creation time).
• results[].suggested_edit (module/subworkflow only): the exact test change implied by the verdicts -- snapshot:[fields] (stable), existence:[fields] (divergent content), contains:[{field,value}] (divergent versions). Subworkflow fields are scope-prefixed (sample./run., e.g. sample.blast, run.versions). Directly consumable and self-verifying (diff against the committed test). Workflow tier is intentionally .nftignore-only, so it has no suggested_edit; add the divergent globs to workflows/**/tests/.nftignore instead.