Bio Hi C Analysis Hic Data Io

Version Compatibility

Reference examples tested with: cooler 0.10+, hic2cool 1.0+, hictk 1.0+, bioframe 0.7+

Before using code patterns, verify installed versions match. If versions differ:

• Python: pip show <package> then help(module.function) to check signatures
• CLI: <tool> --version then <tool> --help to confirm flags

If code throws ImportError, AttributeError, or TypeError, introspect the installed package and adapt the example to match the actual API rather than retrying.

Two version boundaries change BEHAVIOUR, not just signatures: hic2cool >= 0.5.0 stores Juicer norms un-inverted (divisive); < 0.5.0 inverted them to multiplicative, so two coolers made from one .hic by different hic2cool versions disagree numerically. cooler standardized creation/balance signatures around 0.8-0.10 (balance_cooler keyword-only after clr; store=False default). Record the converter version in provenance.

Hi-C Data I/O

"Load my Hi-C contact matrix, convert it, and pull out a region." -> Open the cooler at one resolution, fetch raw or balanced pixels, and convert across .hic/.cool/.mcool while knowing what does not survive the round trip.

• Python: cooler.Cooler('file.mcool::/resolutions/10000').matrix(balance=True).fetch('chr1')
• CLI: hic2cool convert in.hic out.mcool -r 0 / hictk convert in.hic out.mcool / cooler cload pairs ...

The Single Most Important Modern Insight -- cooler Stores Observed Counts; .hic Bakes In Norms, So Conversion Is Never Lossless Both Ways

A .cool is a thin, open, HDF5-native store: three tables (chroms, bins, pixels) holding raw observed integer counts in an upper-triangle COO layout, plus optional cached weight bias columns. Balancing, expected, O/E, eigenvectors -- everything else is computed downstream on demand. Juicer's .hic is the opposite philosophy: a sealed binary deliverable with all resolutions, precomputed normalization vectors, and expected vectors welded in. Every footgun in this skill descends from that split:

• Conversion translates between two philosophies, not two encodings. .hic -> .cool loses FRAG (restriction-fragment) matrices (cooler has no FRAG concept) and Juicer's precomputed expected; a norm whose vector is missing arrives as all-NaN. .cool -> .hic loses asymmetric matrices, arbitrary extra pixel value columns, and non-Hi-C labeled arrays.
• The weight column NAME is load-bearing. cooler's own ICE weight is applied MULTIPLICATIVELY by matrix(balance=True). Juicer KR/VC norms are DIVISIVE. Cooler.matrix(divisive_weights=None) (the default) decides by column name: weights named KR, VC, or VC_SQRT are auto-treated as divisive; everything else (including weight) is multiplicative. Import a KR vector under the name weight and it is applied the wrong way -- garbage, no error.
• .mcool is a container of resolutions, not a matrix. Every downstream tool wants a single-resolution URI file.mcool::/resolutions/<bp>, never the bare .mcool. Each resolution is balanced from scratch; the 100kb weight is NOT derivable from the 10kb weight.

Format and Tool Taxonomy

Decision Tree by Scenario

Load a Cooler and Select a Resolution

import cooler

cooler.fileops.list_coolers('matrix.mcool')                      # e.g. ['/resolutions/1000', '/resolutions/10000', ...]
clr = cooler.Cooler('matrix.mcool::/resolutions/10000')          # single-resolution URI, never the bare .mcool
clr.binsize, clr.chromnames, clr.info['sum']                     # info is unvalidated metadata, not a guarantee
'weight' in clr.bins().columns                                   # is this resolution balanced?

clr.matrix().fetch(region) takes a UCSC region string or a bare chrom name; one arg -> symmetric square, two args -> rectangular submatrix (incl. trans). clr.bins().fetch('chr1'), clr.pixels().fetch(region) slice the tables.

Fetch Balanced vs Raw Pixels

Goal: Pull a chromosome submatrix as a dense array, correctly balanced.

Approach: Confirm the file carries a weight column, then matrix(balance=True); on a balanced file, all-NaN rows are masked low-coverage bins (correct), not a bug. For Juicer-imported KR/VC/VC_SQRT weights, name them correctly and the divisive auto-rule fires; force it with divisive_weights= only if a custom column is misnamed.

balanced = clr.matrix(balance=True).fetch('chr1')                # multiplicative cooler weight
raw = clr.matrix(balance=False).fetch('chr1')                    # observed counts
kr = clr.matrix(balance='KR').fetch('chr1')                      # KR/VC/VC_SQRT auto-treated as divisive by name
sparse = clr.matrix(balance=True, sparse=True).fetch('chr1')     # scipy COO for large chromosomes

Convert .hic to cooler (and Back)

hic2cool convert in.hic out.mcool -r 0                           # -r 0 = all resolutions -> .mcool; norm vectors un-inverted (>=0.5.0)
hic2cool convert in.hic out.cool -r 10000                        # single resolution -> .cool
hic2cool extract-norms in.hic out.mcool                          # add Juicer norm vectors to an existing matching cooler
hictk convert in.hic out.mcool                                   # fast C++ path; --resolutions to subset (single -> .cool)
hictk convert in.mcool out.hic                                   # cooler -> .hic v9 ONLY; drops asymmetric/extra columns

Neither converter reads/writes FRAG-binned matrices; a FRAG .hic yields only its BP resolutions. Record the hic2cool version: the 0.5.0 inversion boundary changes weight values.

Build a Cooler from a Matrix (Vectorized)

Goal: Turn an in-memory numpy contact matrix into a cooler without a hand-rolled O(n^2) loop.

Approach: Binnify the chromsizes, take only the upper triangle (cooler stores symmetric_upper), pull the nonzero coordinates with a single vectorized np.triu + np.nonzero, and assemble the pixel DataFrame in one shot.

import cooler
import bioframe
import numpy as np
import pandas as pd

chromsizes = bioframe.fetch_chromsizes('hg38')                   # pin the assembly + chrom naming up front
bins = cooler.binnify(chromsizes, 10000)                         # 10kb bins; bin table identity defines pixel comparability

upper = np.triu(matrix)                                          # cooler stores the upper triangle only
i, j = np.nonzero(upper)                                         # vectorized; never loop over all bin pairs
pixels = pd.DataFrame({'bin1_id': i, 'bin2_id': j, 'count': upper[i, j]})
cooler.create_cooler('new.cool', bins, pixels, assembly='hg38', symmetric_upper=True)

For pairs, prefer the CLI: cooler cload pairs -c1 2 -p1 3 -c2 4 -p2 5 chromsizes.txt:10000 in.pairs out.cool (the pairs must already be flipped/deduped -> contact-pairs).

Coarsen Correctly (Sum Raw, Re-balance Per Resolution)

Goal: Produce a lower-resolution or multi-resolution file whose weights are valid.

Approach: Coarsen the RAW counts then re-run ICE at each new resolution; zoomify does exactly this. Never sum balanced pixels -- weights are resolution-specific and summed balanced values are silently wrong.

cooler.zoomify_cooler('hires.cool', 'out.mcool', resolutions=[10000, 50000, 100000, 500000], chunksize=10_000_000)
cooler.coarsen_cooler('hires.cool', 'lowres.cool', factor=5, chunksize=10_000_000)   # raw sum; re-balance after

cooler zoomify -r 10000,50000,100000,500000 --balance hires.cool   # raw-coarsen then ICE afresh per level

Export, Merge, and Inspect

cooler.merge_coolers('merged.cool', ['rep1.cool', 'rep2.cool'], mergebuf=20_000_000)   # bin tables MUST match
np.save('chr1.npy', clr.matrix(balance=True).fetch('chr1'))

cooler dump -t pixels --join --balanced in.cool > pixels.tsv      # --balanced requires a balanced file
cooler dump -t bins in.cool > bins.tsv
cooler info in.cool ; cooler ls -l in.mcool

Reference, Blacklist, and Chrom-Naming Provenance

The bin table IS the identity of a cooler. Two coolers are pixel-comparable only if their bin tables are byte-identical: same chroms, same order, same contigs present, same binsize. Dropping chrM/scaffolds in one pipeline shifts every bin_id and makes pixel comparison nonsense. chr1 (UCSC) vs 1 (Ensembl) silently zeros every cross-tool join and fetch('1') on a chr-named file returns nothing with no error -- harmonize naming (and the assembly) across the cooler, the genome FASTA, any phasing/annotation track, and any blacklist BED (genome-intervals/bed-file-basics). info['genome-assembly'] is unvalidated metadata, not a checksum. The .pairs #chromsize header ORDER defines the upper-triangle convention and the cooler's assembly/chromsizes must match the order used to flip the pairs upstream (contact-pairs), or bin assignment and the triangle disagree.