Single Cell Data Prep Qc

Single-Cell Data Preparation & QC

Overview

This skill prepares single-cell RNA-seq data from CELLxGENE Census for downstream statistical analysis. It handles the full data engineering pipeline from raw data query to batch-corrected, quality-controlled integrated dataset.

Pipeline:

1. Data Discovery - Query Census, download disease + healthy data, subsample if needed
2. Quality Control - Filter low-quality cells, set gene symbols
3. Harmonization - Standardize cell type annotations across datasets
4. Integration - Batch correction with Harmony, validation, visualization

Output: processed/integrated.h5ad - Clean, batch-corrected dataset ready for DE/pathway analysis

Key Features:

• Enforces ~100K cell limit with two-stage stratified subsampling (donor + cell type)
• Prioritizes donor preservation for robust pseudobulk analysis
• Conservative cell type harmonization
• Harmony batch correction with validation metrics
• Comprehensive QC visualizations at each step

⚠️ CRITICAL: Workspace Initialization

Data Acquisition:

• Data is downloaded via MCP tools (e.g., mcp__single_cell__get_anndata)
• WorkspaceManager can be initialized via direct bash command before writing scripts

Workspace Initialization Pattern:

# Recommended: Initialize workspace before any data operations
python -c "from src.utils.workspace_manager import WorkspaceManager; wm = WorkspaceManager(agent_name='single_cell_analyst'); print(f'DATE={wm.date}'); print(f'RUN_ID={wm.run_id}')"

When writing Python scripts: ❌ NEVER write to project root ✅ ALWAYS write to workspace code directory: workspace/{date}/{run_id}/single_cell_analyst/code/scripts/

See reference/workspace_setup.md for complete pattern.

⚠️ CELLxGENE Census Gene Index Fix: AnnData objects from Census use numeric IDs as var_names (e.g., '0', '1', '2'), NOT gene symbols. Gene symbols are in adata.var['feature_name']. Always run this immediately after any Census download, before any other processing:

adata.var.index = adata.var['feature_name'].astype(str)
adata.var_names_make_unique()

Visualization Requirements

This skill requires publication-quality visualizations at each major step. All figures should:

• Use dpi=300 for publication quality
• Include clear titles and axis labels
• Save to workspace/{date}/{run_id}/single_cell_analyst/results/figures/
• Use informative filenames (e.g., qc_violin_plots.png, batch_correction_umap.png)

Required visualization categories:

1. QC metrics - Distribution plots before/after filtering
2. Cell type composition - Bar plots comparing disease vs healthy
3. Batch correction validation - Before/after UMAPs with metrics
4. Data summary - Cell counts, sample sizes, metadata overview

See workflow documentation for specific visualization requirements at each step.

Procedure Quick Reference

This skill contains 3 procedures located in procedures/:

Mandatory Procedures:

• Subsampling - subsampling_procedure.md - Two-stage stratified sampling to ~100K cells (preserves all donors + cell types; when combined >100K)
• Harmonization - harmonization_procedure.md - Conservative cell type name standardization (when annotations differ)
• Batch Correction - batch_correction_procedure.md - Harmony integration with validation metrics (always required)

⚠️ Forbidden Actions

You Must NEVER:

• Skip the subsampling checkpoint when dataset >100K cells
• Proceed with integration before harmonizing cell types (if needed)
• Skip batch correction validation metrics
• Create analysis scripts without gene symbols properly set
• Write files to project root instead of workspace

See reference/forbidden_actions.md for full details.

Computational Resources

See reference/computational_resources.md for details.

Available resources:

• 650 GB RAM
• Up to 30-minute timeouts (1,800,000 ms)
• Multiple cores for parallel processing

No excuses for shortcuts or skipping steps due to computational constraints.

Two-Stage Workflow

Stage 1: Data Discovery & Size Optimization

Complete workflow: See workflows/stage1_data_discovery.md

Objectives:

• Initialize WorkspaceManager with date/run_id
• Query CELLxGENE Census for disease + healthy tissue data
• Download and save raw data to workspace
• CHECKPOINT: Evaluate size and subsample to ≤100K cells if needed
• Generate initial QC visualizations

Key outputs:

• data/raw/disease_raw_subsampled.h5ad
• data/raw/healthy_raw_subsampled.h5ad
• Initial QC plots

Stage 2: QC, Harmonization & Integration

Complete workflow: See workflows/stage2_qc_integration.md

Objectives:

• Load subsampled data from Stage 1 (≤100K cells guaranteed)
• Perform QC and filtering
• Checkpoint 1: Harmonize cell type annotations if needed
• Checkpoint 2: Perform batch correction with Harmony
• Validate integration quality
• Generate comprehensive visualization suite

Key outputs:

• data/processed/integrated.h5ad - Ready for analysis
• QC, harmonization, integration figures
• Metadata summary JSON

Workflow Completion Checklist

Before declaring data prep complete, verify:

Files present:

# Raw data
ls workspace/{date}/{run_id}/single_cell_analyst/data/raw/*_subsampled.h5ad

# Processed data (MAIN OUTPUT)
ls workspace/{date}/{run_id}/single_cell_analyst/data/processed/integrated.h5ad

# QC figures
ls workspace/{date}/{run_id}/single_cell_analyst/results/figures/qc_*.png

# Integration figures
ls workspace/{date}/{run_id}/single_cell_analyst/results/figures/integration_*.png

Success criteria:

• ✅ Both stages completed in sequence
• ✅ All mandatory checkpoints completed (subsampling, harmonization, batch correction)
• ✅ Gene symbols properly set (not integers)
• ✅ Batch correction validated with metrics
• ✅ All required visualizations generated
• ✅ integrated.h5ad file exists and is valid