opencode-knowledge-work 0.0.0-stage → 0.1.0

package/content/bio-research/skills/bio-research-scvi-tools/references/environment_setup.md

@@ -0,0 +1,254 @@
+# Environment Setup for scvi-tools
+
+This reference covers installation and environment configuration for scvi-tools.
+
+## Installation Options
+
+### Option 1: Conda Environment (Recommended)
+
+```bash
+# Create environment with GPU support
+conda create -n scvi-env python=3.10
+conda activate scvi-env
+
+# Install scvi-tools
+pip install scvi-tools
+
+# For GPU acceleration (recommended for large datasets)
+pip install torch --index-url https://download.pytorch.org/whl/cu118
+
+# Common dependencies
+pip install scanpy leidenalg
+```
+
+### Option 2: Pip Only
+
+```bash
+# Create virtual environment
+python -m venv scvi-env
+source scvi-env/bin/activate  # Linux/Mac
+# scvi-env\Scripts\activate   # Windows
+
+# Install
+pip install scvi-tools scanpy
+```
+
+### Option 3: With Spatial Analysis Support
+
+```bash
+conda create -n scvi-spatial python=3.10
+conda activate scvi-spatial
+
+pip install scvi-tools scanpy squidpy
+```
+
+### Option 4: With MuData Support (Multiome)
+
+```bash
+pip install scvi-tools mudata muon
+```
+
+## Verify Installation
+
+```python
+import scvi
+import torch
+import scanpy as sc
+
+print(f"scvi-tools version: {scvi.__version__}")
+print(f"scanpy version: {sc.__version__}")
+print(f"PyTorch version: {torch.__version__}")
+print(f"GPU available: {torch.cuda.is_available()}")
+
+if torch.cuda.is_available():
+    print(f"GPU device: {torch.cuda.get_device_name(0)}")
+    print(f"GPU memory: {torch.cuda.get_device_properties(0).total_memory / 1e9:.1f} GB")
+```
+
+## GPU Configuration
+
+### Check CUDA Version
+
+```bash
+nvidia-smi
+nvcc --version
+```
+
+### PyTorch CUDA Versions
+
+| CUDA Version | PyTorch Install Command |
+|--------------|------------------------|
+| CUDA 11.8 | `pip install torch --index-url https://download.pytorch.org/whl/cu118` |
+| CUDA 12.1 | `pip install torch --index-url https://download.pytorch.org/whl/cu121` |
+| CPU only | `pip install torch --index-url https://download.pytorch.org/whl/cpu` |
+
+### Memory Management
+
+```python
+import torch
+
+# Clear GPU cache between models
+torch.cuda.empty_cache()
+
+# Monitor memory usage
+print(f"Allocated: {torch.cuda.memory_allocated() / 1e9:.2f} GB")
+print(f"Cached: {torch.cuda.memory_reserved() / 1e9:.2f} GB")
+```
+
+## Common Issues
+
+| Issue | Cause | Solution |
+|-------|-------|----------|
+| `CUDA out of memory` | GPU memory exhausted | Reduce batch_size, use smaller model |
+| `No GPU detected` | CUDA not installed | Install CUDA toolkit matching PyTorch |
+| `Version mismatch` | PyTorch/CUDA incompatibility | Reinstall PyTorch with correct CUDA version |
+| `Import error scvi` | Missing dependencies | `pip install scvi-tools[all]` |
+
+## Jupyter Setup
+
+```bash
+# Install Jupyter kernel
+pip install ipykernel
+python -m ipykernel install --user --name scvi-env --display-name "scvi-tools"
+
+# For interactive plots
+pip install matplotlib seaborn
+```
+
+## Recommended Package Versions
+
+For reproducibility, pin versions:
+
+```bash
+pip install \
+    scvi-tools>=1.0.0 \
+    scanpy>=1.9.0 \
+    anndata>=0.9.0 \
+    torch>=2.0.0
+```
+
+## Version Compatibility Guide
+
+### scvi-tools 1.x vs 0.x API Changes
+
+The 1.x release introduced breaking changes. Key differences:
+
+| Operation | 0.x API (deprecated) | 1.x API (current) |
+|-----------|---------------------|-------------------|
+| Setup data | `scvi.data.setup_anndata(adata, ...)` | `scvi.model.SCVI.setup_anndata(adata, ...)` |
+| Register data | `scvi.data.register_tensor_from_anndata(...)` | Built into `setup_anndata` |
+| View setup | `scvi.data.view_anndata_setup(adata)` | `scvi.model.SCVI.view_anndata_setup(adata)` |
+
+### Migration from 0.x to 1.x
+
+```python
+# OLD (0.x) - DEPRECATED
+import scvi
+scvi.data.setup_anndata(adata, layer="counts", batch_key="batch")
+model = scvi.model.SCVI(adata)
+
+# NEW (1.x) - CURRENT
+import scvi
+scvi.model.SCVI.setup_anndata(adata, layer="counts", batch_key="batch")
+model = scvi.model.SCVI(adata)
+```
+
+### Model-Specific Setup (1.x)
+
+Each model has its own setup method:
+
+```python
+# scVI
+scvi.model.SCVI.setup_anndata(adata, layer="counts", batch_key="batch")
+
+# scANVI
+scvi.model.SCANVI.setup_anndata(adata, layer="counts", batch_key="batch", labels_key="cell_type")
+
+# totalVI
+scvi.model.TOTALVI.setup_anndata(adata, layer="counts", protein_expression_obsm_key="protein")
+
+# MultiVI (uses MuData)
+scvi.model.MULTIVI.setup_mudata(mdata, rna_layer="counts", atac_layer="counts")
+
+# PeakVI
+scvi.model.PEAKVI.setup_anndata(adata, batch_key="batch")
+
+# veloVI
+scvi.external.VELOVI.setup_anndata(adata, spliced_layer="spliced", unspliced_layer="unspliced")
+```
+
+### Minimum Version Requirements
+
+| Package | Minimum Version | Notes |
+|---------|-----------------|-------|
+| scvi-tools | 1.0.0 | Required for current API |
+| scanpy | 1.9.0 | HVG selection improvements |
+| anndata | 0.9.0 | Improved MuData support |
+| torch | 2.0.0 | Performance improvements |
+| mudata | 0.2.0 | Required for MultiVI |
+| scvelo | 0.2.5 | Required for veloVI |
+
+### Check Your Versions
+
+```python
+import scvi
+import scanpy as sc
+import anndata
+import torch
+
+print(f"scvi-tools: {scvi.__version__}")
+print(f"scanpy: {sc.__version__}")
+print(f"anndata: {anndata.__version__}")
+print(f"torch: {torch.__version__}")
+
+# Check if using 1.x API
+if hasattr(scvi.model.SCVI, 'setup_anndata'):
+    print("Using scvi-tools 1.x API")
+else:
+    print("WARNING: Using deprecated 0.x API - please upgrade")
+```
+
+### Known Compatibility Issues
+
+| Issue | Affected Versions | Solution |
+|-------|-------------------|----------|
+| `setup_anndata` not found | scvi-tools < 1.0 | Upgrade to 1.0+ |
+| MuData errors | mudata < 0.2 | `pip install mudata>=0.2.0` |
+| CUDA version mismatch | Any | Reinstall PyTorch for your CUDA |
+| numpy 2.0 issues | Early 2024 builds | `pip install numpy<2.0` |
+
+### Upgrading scvi-tools
+
+```bash
+# Upgrade to latest
+pip install --upgrade scvi-tools
+
+# Upgrade all dependencies
+pip install --upgrade scvi-tools scanpy anndata torch
+
+# If you have issues, clean install
+pip uninstall scvi-tools
+pip cache purge
+pip install scvi-tools
+```
+
+## Testing Installation
+
+```python
+# Quick test with sample data
+import scvi
+import scanpy as sc
+
+# Load test dataset
+adata = scvi.data.heart_cell_atlas_subsampled()
+print(f"Loaded test data: {adata.shape}")
+
+# Setup and create model (quick test)
+scvi.model.SCVI.setup_anndata(adata, layer="counts", batch_key="cell_source")
+model = scvi.model.SCVI(adata, n_latent=10)
+print("Model created successfully")
+
+# Quick training test (1 epoch)
+model.train(max_epochs=1)
+print("Training works!")
+```

package/content/bio-research/skills/bio-research-scvi-tools/references/label_transfer.md

@@ -0,0 +1,373 @@
+# Label Transfer and Reference Mapping with scANVI
+
+This reference covers using scANVI for transferring cell type annotations from a reference atlas to query data.
+
+## Overview
+
+Reference mapping (also called "label transfer") uses a pre-trained model on annotated reference data to predict cell types in new, unannotated query data. This is faster than re-clustering and more consistent across studies.
+
+scANVI excels at this because it:
+- Jointly embeds reference and query in shared space
+- Transfers labels probabilistically
+- Handles batch effects between reference and query
+
+## When to Use Reference Mapping
+
+- Annotating new dataset using existing atlas
+- Consistent annotation across multiple studies
+- Speed: no need to re-cluster and manually annotate
+- Quality: leverage expert-curated reference annotations
+
+## Workflow Options
+
+1. **Train new model**: Train scANVI on reference, then map query
+2. **Use pre-trained model**: Load existing model (e.g., from Model Hub)
+3. **scArches**: Extend existing model with query data (preserves reference)
+
+## Option 1: Train scANVI on Reference
+
+### Step 1: Prepare Reference Data
+
+```python
+import scvi
+import scanpy as sc
+
+# Load reference atlas
+adata_ref = sc.read_h5ad("reference_atlas.h5ad")
+
+# Check annotations
+print(f"Reference cells: {adata_ref.n_obs}")
+print(f"Cell types: {adata_ref.obs['cell_type'].nunique()}")
+print(adata_ref.obs['cell_type'].value_counts())
+
+# Ensure raw counts
+adata_ref.layers["counts"] = adata_ref.raw.X.copy() if adata_ref.raw else adata_ref.X.copy()
+
+# HVG selection
+sc.pp.highly_variable_genes(
+    adata_ref,
+    n_top_genes=3000,
+    flavor="seurat_v3",
+    batch_key="batch" if "batch" in adata_ref.obs else None,
+    layer="counts"
+)
+adata_ref = adata_ref[:, adata_ref.var["highly_variable"]].copy()
+```
+
+### Step 2: Train scANVI on Reference
+
+```python
+# First train scVI (unlabeled)
+scvi.model.SCVI.setup_anndata(
+    adata_ref,
+    layer="counts",
+    batch_key="batch"
+)
+
+scvi_ref = scvi.model.SCVI(adata_ref, n_latent=30)
+scvi_ref.train(max_epochs=200)
+
+# Initialize scANVI from scVI
+scanvi_ref = scvi.model.SCANVI.from_scvi_model(
+    scvi_ref,
+    labels_key="cell_type",
+    unlabeled_category="Unknown"
+)
+
+# Train scANVI
+scanvi_ref.train(max_epochs=50)
+
+# Save for later use
+scanvi_ref.save("scanvi_reference_model/")
+```
+
+### Step 3: Prepare Query Data
+
+```python
+# Load query data
+adata_query = sc.read_h5ad("query_data.h5ad")
+
+# CRITICAL: Use same genes as reference
+common_genes = adata_ref.var_names.intersection(adata_query.var_names)
+print(f"Common genes: {len(common_genes)}")
+
+# Subset query to reference genes
+adata_query = adata_query[:, adata_ref.var_names].copy()
+
+# Handle missing genes (set to 0)
+missing_genes = set(adata_ref.var_names) - set(adata_query.var_names)
+if missing_genes:
+    # Add missing genes with zero expression
+    import numpy as np
+    from scipy.sparse import csr_matrix
+    
+    zero_matrix = csr_matrix((adata_query.n_obs, len(missing_genes)))
+    # ... concat and reorder to match reference
+    
+# Store counts
+adata_query.layers["counts"] = adata_query.X.copy()
+```
+
+### Step 4: Map Query to Reference
+
+```python
+# Prepare query data for mapping
+scvi.model.SCANVI.prepare_query_anndata(adata_query, scanvi_ref)
+
+# Create query model from reference
+scanvi_query = scvi.model.SCANVI.load_query_data(
+    adata_query,
+    scanvi_ref
+)
+
+# Fine-tune on query (optional but recommended)
+scanvi_query.train(
+    max_epochs=100,
+    plan_kwargs={"weight_decay": 0.0}
+)
+
+# Get predictions
+adata_query.obs["predicted_cell_type"] = scanvi_query.predict()
+
+# Get prediction probabilities
+soft_predictions = scanvi_query.predict(soft=True)
+adata_query.obs["prediction_score"] = soft_predictions.max(axis=1)
+```
+
+### Step 5: Evaluate Predictions
+
+```python
+# Confidence scores
+print(f"Mean prediction confidence: {adata_query.obs['prediction_score'].mean():.3f}")
+
+# Low confidence predictions
+low_conf = adata_query.obs['prediction_score'] < 0.5
+print(f"Low confidence cells: {low_conf.sum()} ({low_conf.mean()*100:.1f}%)")
+
+# Visualize
+sc.pp.neighbors(adata_query, use_rep="X_scANVI")
+sc.tl.umap(adata_query)
+sc.pl.umap(adata_query, color=['predicted_cell_type', 'prediction_score'])
+```
+
+## Option 2: Use Pre-Trained Models
+
+### From Model Hub
+
+```python
+# scvi-tools maintains models on HuggingFace
+# Check: https://huggingface.co/scvi-tools
+
+# Example: Load pre-trained model
+from huggingface_hub import hf_hub_download
+
+model_path = hf_hub_download(
+    repo_id="scvi-tools/example-model",
+    filename="model.pt"
+)
+
+# Load model
+model = scvi.model.SCANVI.load(model_path, adata=adata_query)
+```
+
+### From Published Atlas
+
+```python
+# Many atlases provide pre-trained models
+# Example workflow with CellTypist-style model
+
+# Download reference model
+# model = scvi.model.SCANVI.load("atlas_model/", adata=adata_query)
+```
+
+## Option 3: scArches for Incremental Updates
+
+scArches extends a reference model without retraining from scratch:
+
+```python
+# Load existing reference model
+scanvi_ref = scvi.model.SCANVI.load("reference_model/")
+
+# Surgery: prepare for query integration
+scanvi_ref.freeze_layers()
+
+# Map query data
+scvi.model.SCANVI.prepare_query_anndata(adata_query, scanvi_ref)
+scanvi_query = scvi.model.SCANVI.load_query_data(adata_query, scanvi_ref)
+
+# Train only query-specific parameters
+scanvi_query.train(
+    max_epochs=200,
+    plan_kwargs={"weight_decay": 0.0}
+)
+```
+
+## Visualize Reference and Query Together
+
+```python
+# Concatenate for joint visualization
+adata_ref.obs["dataset"] = "reference"
+adata_query.obs["dataset"] = "query"
+
+# Get latent representations
+adata_ref.obsm["X_scANVI"] = scanvi_ref.get_latent_representation()
+adata_query.obsm["X_scANVI"] = scanvi_query.get_latent_representation()
+
+# Combine
+adata_combined = sc.concat([adata_ref, adata_query])
+
+# Compute combined UMAP
+sc.pp.neighbors(adata_combined, use_rep="X_scANVI")
+sc.tl.umap(adata_combined)
+
+# Plot
+sc.pl.umap(
+    adata_combined,
+    color=["dataset", "cell_type", "predicted_cell_type"],
+    ncols=2
+)
+```
+
+## Quality Control for Predictions
+
+### Confidence Filtering
+
+```python
+# Filter predictions by confidence
+confidence_threshold = 0.7
+
+high_conf = adata_query[adata_query.obs['prediction_score'] >= confidence_threshold].copy()
+low_conf = adata_query[adata_query.obs['prediction_score'] < confidence_threshold].copy()
+
+print(f"High confidence: {len(high_conf)} ({len(high_conf)/len(adata_query)*100:.1f}%)")
+print(f"Low confidence: {len(low_conf)} ({len(low_conf)/len(adata_query)*100:.1f}%)")
+```
+
+### Marker Validation
+
+```python
+# Validate predictions with known markers
+markers = {
+    'T cells': ['CD3D', 'CD3E'],
+    'B cells': ['CD19', 'MS4A1'],
+    'Monocytes': ['CD14', 'LYZ']
+}
+
+for ct, genes in markers.items():
+    ct_cells = adata_query[adata_query.obs['predicted_cell_type'] == ct]
+    if len(ct_cells) > 0:
+        for gene in genes:
+            if gene in adata_query.var_names:
+                expr = ct_cells[:, gene].X.mean()
+                print(f"{ct} - {gene}: {expr:.3f}")
+```
+
+## Complete Pipeline
+
+```python
+def transfer_labels(
+    adata_ref,
+    adata_query,
+    cell_type_key="cell_type",
+    batch_key=None,
+    n_top_genes=3000,
+    confidence_threshold=0.5
+):
+    """
+    Transfer cell type labels from reference to query.
+    
+    Parameters
+    ----------
+    adata_ref : AnnData
+        Annotated reference data
+    adata_query : AnnData
+        Unannotated query data
+    cell_type_key : str
+        Column with cell type annotations in reference
+    batch_key : str, optional
+        Batch column
+    n_top_genes : int
+        Number of HVGs
+    confidence_threshold : float
+        Minimum confidence for predictions
+        
+    Returns
+    -------
+    AnnData with predictions
+    """
+    import scvi
+    import scanpy as sc
+    
+    # Prepare reference
+    adata_ref = adata_ref.copy()
+    adata_ref.layers["counts"] = adata_ref.X.copy()
+    
+    sc.pp.highly_variable_genes(
+        adata_ref,
+        n_top_genes=n_top_genes,
+        flavor="seurat_v3",
+        batch_key=batch_key,
+        layer="counts"
+    )
+    adata_ref = adata_ref[:, adata_ref.var["highly_variable"]].copy()
+    
+    # Train reference model
+    scvi.model.SCVI.setup_anndata(adata_ref, layer="counts", batch_key=batch_key)
+    scvi_ref = scvi.model.SCVI(adata_ref, n_latent=30)
+    scvi_ref.train(max_epochs=200)
+    
+    scanvi_ref = scvi.model.SCANVI.from_scvi_model(
+        scvi_ref,
+        labels_key=cell_type_key,
+        unlabeled_category="Unknown"
+    )
+    scanvi_ref.train(max_epochs=50)
+    
+    # Prepare query
+    adata_query = adata_query[:, adata_ref.var_names].copy()
+    adata_query.layers["counts"] = adata_query.X.copy()
+    
+    # Map query
+    scvi.model.SCANVI.prepare_query_anndata(adata_query, scanvi_ref)
+    scanvi_query = scvi.model.SCANVI.load_query_data(adata_query, scanvi_ref)
+    scanvi_query.train(max_epochs=100, plan_kwargs={"weight_decay": 0.0})
+    
+    # Get predictions
+    adata_query.obs["predicted_cell_type"] = scanvi_query.predict()
+    soft = scanvi_query.predict(soft=True)
+    adata_query.obs["prediction_score"] = soft.max(axis=1)
+    
+    # Mark low confidence
+    adata_query.obs["confident_prediction"] = adata_query.obs["prediction_score"] >= confidence_threshold
+    
+    # Add latent representation
+    adata_query.obsm["X_scANVI"] = scanvi_query.get_latent_representation()
+    
+    return adata_query, scanvi_ref, scanvi_query
+
+# Usage
+adata_annotated, ref_model, query_model = transfer_labels(
+    adata_ref,
+    adata_query,
+    cell_type_key="cell_type"
+)
+
+# Visualize
+sc.pp.neighbors(adata_annotated, use_rep="X_scANVI")
+sc.tl.umap(adata_annotated)
+sc.pl.umap(adata_annotated, color=['predicted_cell_type', 'prediction_score'])
+```
+
+## Troubleshooting
+
+| Issue | Cause | Solution |
+|-------|-------|----------|
+| Many low-confidence predictions | Query has novel cell types | Manually annotate low-confidence cells |
+| Wrong predictions | Reference doesn't match tissue | Use tissue-appropriate reference |
+| Gene mismatch | Different gene naming | Convert gene IDs |
+| All same prediction | Query too different | Check data quality, try different reference |
+
+## Key References
+
+- Xu et al. (2021) "Probabilistic harmonization and annotation of single-cell transcriptomics data with deep generative models"
+- Lotfollahi et al. (2022) "Mapping single-cell data to reference atlases by transfer learning"