Bio Spatial Transcriptomics Image Analysis

Version Compatibility

Reference examples tested with: squidpy 1.7+, scanpy 1.10+, scikit-image 0.22+, numpy 1.26+, pandas 2.2+, cellpose 4.0+ (CLI tools: Baysor 0.6+, proseg 1.0+)

Before using code patterns, verify installed versions match. If versions differ:

• Python: pip show <package> then help(module.function) to check signatures

If code throws ImportError, AttributeError, or TypeError, introspect the installed package and adapt the example to match the actual API rather than retrying.

Image Analysis for Spatial Transcriptomics

"Segment cells from my imaging data" -> Draw cell/nucleus boundaries on an image (or on the transcript point cloud) and assign each molecule to one cell, producing a cell-by-gene matrix.

• Python image-based: cellpose.models.CellposeModel().eval(), StarDist for nuclei, squidpy.im.segment() (watershed baseline)
• CLI transcript-based: Baysor, proseg run on the molecule table (x, y, gene)

"Extract image features for my spots" -> Summarize pixel intensity/texture under each Visium spot (a DIFFERENT operation from segmentation -- no cell boundaries are drawn).

• Python: squidpy.im.calculate_image_features() (summary, histogram, texture/GLCM)

Keep these two operations distinct. squidpy.im feature extraction (texture/summary on H&E) describes the image patch under a spot for domain detection. Segmentation manufactures the cells themselves. This skill leads with segmentation because that is the dominant error source; feature extraction is a downstream convenience.

Governing Principle

In imaging spatial data the cell is a segmentation HYPOTHESIS, not an observation. The raw data are images plus a table of decoded molecules with x, y (and sometimes z); there is no native cell. A cell-by-gene matrix exists only AFTER an algorithm draws boundaries and assigns each molecule to one cell or to background. Every row of that matrix is produced by the segmentation step, which is the dominant, irreducible upstream error source for every imaging platform -- it confounds typing, DE, and communication downstream (Mitchel et al. 2026 Nat Genet 58:434, who find segmentation errors "dominate the results" for context-dependent DE and ligand-receptor inference).

Segmentation fails three ways, and each fabricates a specific downstream lie:

• Over-segmentation splits one cell into many -> inflated cell count; fragments look low-quality and get filtered or mis-typed.
• Under-segmentation merges neighbors into a spatial DOUBLET whose profile is a mixture of two types. Unlike droplet doublets these are spatially structured -- adjacent types merge preferentially -- so standard doublet detectors (Scrublet, DoubletFinder) MISS them, because the synthetic doublets those tools simulate are random pairs, not neighbors.
• Transcript mis-assignment / spillover: a molecule of cell A is assigned to neighbor B (diffusion, optical PSF bleed, z-collapse, boundary error). Spillover is DISTANCE-DEPENDENT and strongest between adjacent heterotypic cells, so it manufactures spatially-structured contamination, not uniform noise.

The cascade is worst for exactly the analyses people prize. A T cell abutting epithelium picks up keratin spillover -> false marker co-expression -> a spurious "transitional/hybrid" state that is pure artifact; rare cells in dense parenchyma (TILs, neutrophils) are swamped by neighbor spillover and lost. Crucially, distance-dependent spillover FABRICATES the short-range co-localization that ligand-receptor and cell-cell communication tools detect -- so an L-R "hit" between two touching types can be a pure segmentation artifact (a circularity; see spatial-communication). Treat the derived matrix as PROVISIONAL and run a contamination QC step before trusting any single-cell-resolution claim.

The decision: choose by available signal, not by reflex

The first question is not "which tool" but "what boundary signal do I have?" A nuclear stain says where the nucleus is; a membrane/boundary stain says where the cell ENDS. Whole-cell segmentation is boundary-finding, and DAPI carries no boundary information -- so DAPI-only whole-cell is always inference (expansion). Adding a membrane/boundary stain converts that inference into measurement, and is the single highest-leverage change available -- it beats swapping algorithms on DAPI-only data.

The ladder of trust runs: DAPI-only StarDist/Cellpose-nuclei (accept nuclear sensitivity loss or expansion bias) < membrane-stain whole-cell Mesmer or 2-channel Cellpose < transcript-aware Baysor/proseg (molecules, not a fixed radius, set boundaries) < segmentation-free SSAM/ClusterMap (best mapping, but the cell unit is lost -- no per-cell matrix, neighbor graph, or QC). Methods evolve fast here; verify current best practice against the latest benchmarks before committing.

Nucleus-only and the expansion trap

Nuclear (DAPI) segmentation is robust because nuclei are round, separated, and high-contrast -- but the nucleus holds only a minority of mRNA, so cytoplasmic transcripts fall OUTSIDE the mask and are discarded (large sensitivity loss) or must be reassigned. The cheap substitute is nucleus expansion: dilate each nuclear mask by a fixed radius until it hits a neighbor. This assumes round, equal-sized, isotropically-arranged cells -- false for almost all tissue. In dense tissue expanded disks collide and partition intercellular space by a Voronoi-like rule unrelated to true membranes (the worst region for spillover); elongated or large-cytoplasm cells (neurons, muscle, glia, macrophages) are badly served -- a fixed disk captures none of their projections and steals neighbors' transcripts. No single radius is correct for a heterogeneous tissue. Expansion is a baseline, not a solution.

Xenium makes this concrete: XOA v1.0-1.9 used DAPI + 15 um nucleus expansion; v2.0+ cut the default to 5 um "for improved accuracy" -- an admission that 15 um over-assigned in dense tissue. The vendor changed the answer, so do not treat any expansion radius as ground truth. Note that Cellpose on Xenium is a community path via Xenium Ranger import-segmentation, NOT the built-in XOA default -- do not conflate them.

Segment nuclei from a DAPI/IF image

Goal: Produce instance masks (one integer label per cell) from a nuclear-stain image as the starting cell hypotheses.

Approach: Run Cellpose's generalist model; in v4 (Cellpose-SAM) there is one model, channels are no longer an input, and diameter is optional because the model is size-invariant. With a membrane channel available, pass it as a second channel for true whole-cell masks instead of nuclei + expansion.

from cellpose import models

model = models.CellposeModel(gpu=False)            # v4 Cellpose-SAM single generalist model; v3 used models.Cellpose(model_type='nuclei')
masks, flows, styles = model.eval(dapi_image, diameter=None)   # v4 returns 3 values + drops channels=; v3 returned masks, flows, styles, diams and took channels=[0,0]
# masks: integer label image; 0 = background, 1..N = cells. This is a HYPOTHESIS, not ground truth.
n_cells = int(masks.max())

DAPI-only masks are nuclei. Approximating whole cells without a membrane stain requires expansion (round-cell bias above) or a transcript-aware method. With a membrane/boundary channel, stacking [nucleus, membrane] and passing both lets Cellpose-SAM use the first channels in any order -- that converts boundary inference into measurement.

Re-segment from the transcript table (membrane-free whole-cell)

Goal: Recover cells that image-based nuclear segmentation drops (small, irregular, immune) by letting transcript composition and density define boundaries.

Approach: Run Baysor or proseg on the per-molecule table (x, y, gene). These are CLI tools; the molecule table is the source of truth and the only object that permits re-segmentation. Optionally seed Baysor with the vendor nuclear masks as a prior.

# Baysor: molecule-table segmentation; -m = min transcripts/cell, -s = expected cell scale (um), :gene names the gene column
baysor run -x x_location -y y_location -g feature_name -m 30 -s 10 \
  --prior-segmentation-confidence 0.5 transcripts.csv nucleus_id

# proseg: transcript-only whole-cell, reads Xenium/CosMx/MERSCOPE molecule tables directly
proseg --xenium transcripts.csv.gz --output-counts counts.csv.gz --output-cell-polygons cells.geojson

Baysor and proseg output per-molecule cell assignments -> rebuild a cell-by-gene matrix from those. Where the panel is sparse and no membrane stain exists, all transcript-only methods become unstable -- check cell-yield and size distributions against the image before trusting them.