@bgicli/bgicli 2.1.1 → 2.2.0

package/data/skills/single-preprocessing/SKILL.md

@@ -0,0 +1,184 @@
+---
+name: single-cell-preprocessing-with-omicverse
+title: Single-cell preprocessing with omicverse
+description: Walk through omicverse's single-cell preprocessing tutorials to QC PBMC3k data, normalise counts, detect HVGs, and run PCA/embedding pipelines on CPU, CPU–GPU mixed, or GPU stacks.
+---
+
+# Single-cell preprocessing with omicverse
+
+## Overview
+Follow this skill when a user needs to reproduce the preprocessing workflow from the omicverse notebooks [`t_preprocess.ipynb`](../../omicverse_guide/docs/Tutorials-single/t_preprocess.ipynb), [`t_preprocess_cpu.ipynb`](../../omicverse_guide/docs/Tutorials-single/t_preprocess_cpu.ipynb), and [`t_preprocess_gpu.ipynb`](../../omicverse_guide/docs/Tutorials-single/t_preprocess_gpu.ipynb). The tutorials operate on the 10x PBMC3k dataset and cover QC filtering, normalisation, highly variable gene (HVG) detection, dimensionality reduction, and downstream embeddings.
+
+## Instructions
+1. **Set up the environment**
+   - Import `omicverse as ov` and `scanpy as sc`, then call `ov.plot_set(font_path='Arial')` (or `ov.ov_plot_set()` in legacy notebooks) to standardise figure styling.
+   - Encourage `%load_ext autoreload` and `%autoreload 2` when iterating inside notebooks so code edits propagate without restarting the kernel.
+2. **Prepare input data**
+   - Download the PBMC3k filtered matrix from 10x Genomics (`pbmc3k_filtered_gene_bc_matrices.tar.gz`) and extract it under `data/filtered_gene_bc_matrices/hg19/`.
+   - Load the matrix via `sc.read_10x_mtx(..., var_names='gene_symbols', cache=True)` and keep a writable folder like `write/` for exports.
+3. **Perform quality control (QC)**
+   - Run `ov.pp.qc(adata, tresh={'mito_perc': 0.2, 'nUMIs': 500, 'detected_genes': 250}, doublets_method='scrublet')` for the CPU/CPU–GPU pipelines; omit `doublets_method` on pure GPU where Scrublet is not yet supported.
+   - Review the returned AnnData summary to confirm doublet rates and QC thresholds; advise adjusting cut-offs for different species or sequencing depths.
+4. **Store raw counts before transformations**
+   - Call `ov.utils.store_layers(adata, layers='counts')` immediately after QC so the original counts remain accessible for later recovery and comparison.
+5. **Normalise and select HVGs**
+   - Use `ov.pp.preprocess(adata, mode='shiftlog|pearson', n_HVGs=2000, target_sum=5e5)` to apply shift-log normalisation followed by Pearson residual HVG detection (set `target_sum=None` on GPU, which keeps defaults).
+   - For CPU–GPU mixed runs, demonstrate `ov.pp.recover_counts(...)` to invert normalisation and store reconstructed counts in `adata.layers['recover_counts']`.
+6. **Manage `.raw` and layer recovery**
+   - Snapshot normalised data to `.raw` with `adata.raw = adata` (or `adata.raw = adata.copy()`), and show `ov.utils.retrieve_layers(adata_counts, layers='counts')` to compare normalised vs. raw intensities.
+7. **Scale, reduce, and embed**
+   - Scale features using `ov.pp.scale(adata)` (layers hold scaled matrices) followed by `ov.pp.pca(adata, layer='scaled', n_pcs=50)`.
+   - Construct neighbourhood graphs with:
+     - `sc.pp.neighbors(adata, n_neighbors=15, n_pcs=50, use_rep='scaled|original|X_pca')` for the baseline notebook.
+     - `ov.pp.neighbors(..., use_rep='scaled|original|X_pca')` on CPU–GPU to leverage accelerated routines.
+     - `ov.pp.neighbors(..., method='cagra')` on GPU to call RAPIDS graph primitives.
+   - Generate embeddings via `ov.utils.mde(...)`, `ov.pp.umap(adata)`, `ov.pp.mde(...)`, `ov.pp.tsne(...)`, or `ov.pp.sude(...)` depending on the notebook variant.
+8. **Cluster and annotate**
+   - Run `ov.pp.leiden(adata, resolution=1)` or `ov.single.leiden(adata, resolution=1.0)` after neighbour graph construction; CPU–GPU pipelines also showcase `ov.pp.score_genes_cell_cycle` before clustering.
+   - **IMPORTANT - Defensive checks**: When generating code that plots by clustering results (e.g., `color='leiden'`), always check if the clustering has been performed first:
+     ```python
+     # Check if leiden clustering exists, if not, run it
+     if 'leiden' not in adata.obs:
+         if 'neighbors' not in adata.uns:
+             ov.pp.neighbors(adata, n_neighbors=15, use_rep='X_pca')
+         ov.single.leiden(adata, resolution=1.0)
+     ```
+   - Plot embeddings with `ov.pl.embedding(...)` or `ov.utils.embedding(...)`, colouring by `leiden` clusters and marker genes. Always verify that the column specified in `color=` parameter exists in `adata.obs` before plotting.
+9. **Document outputs**
+   - Encourage saving intermediate AnnData objects (`adata.write('write/pbmc3k_preprocessed.h5ad')`) and figure exports using Matplotlib’s `plt.savefig(...)` to preserve QC summaries and embeddings.
+10. **Notebook-specific notes**
+    - *Baseline (`t_preprocess.ipynb`)*: Focuses on CPU execution with Scanpy neighbours; emphasise storing counts before and after `retrieve_layers` demonstrations.
+    - *CPU–GPU mixed (`t_preprocess_cpu.ipynb`)*: Highlights Omicverse ≥1.7.0 mixed acceleration. Include timing magics (%%time) to showcase speedups and call out `doublets_method='scrublet'` support.
+    - *GPU (`t_preprocess_gpu.ipynb`)*: Requires a CUDA-capable GPU, RAPIDS 24.04 stack, and `rapids-singlecell`. Mention the `ov.pp.anndata_to_GPU`/`ov.pp.anndata_to_CPU` transfers and `method='cagra'` neighbours. Note the current warning that pure-GPU pipelines depend on RAPIDS updates.
+11. **Troubleshooting tips**
+    - If `sc.read_10x_mtx` fails, verify the extracted folder structure and ensure gene symbols are available via `var_names='gene_symbols'`.
+    - Address GPU import errors by confirming the conda environment matches the RAPIDS version for the installed CUDA driver (`nvidia-smi`).
+    - For `ov.pp.preprocess` dimension mismatches, ensure QC filtered out empty barcodes so HVG selection does not encounter zero-variance features.
+    - When embeddings lack expected fields (e.g., `scaled|original|X_pca` missing), re-run `ov.pp.scale` and `ov.pp.pca` to rebuild the cached layers.
+    - **Pipeline dependency errors**: When encountering errors like "Could not find 'leiden' in adata.obs or adata.var_names":
+      - Always check if required preprocessing steps (neighbors, PCA) exist before dependent operations
+      - Check if clustering results exist in `adata.obs` before trying to color plots by them
+      - Use defensive checks in generated code to handle incomplete pipelines gracefully
+    - **Code generation best practice**: Generate robust code with conditional checks for prerequisites rather than assuming perfect sequential execution. Users may run steps in separate sessions or skip intermediate steps.
+
+## Critical API Reference - Batch Column Handling
+
+### Batch Column Validation - REQUIRED Before Batch Operations
+
+**IMPORTANT**: Always validate and prepare the batch column before any batch-aware operations (batch correction, integration, etc.). Missing or NaN values will cause errors.
+
+**CORRECT usage:**
+```python
+# Step 1: Check if batch column exists, create default if not
+if 'batch' not in adata.obs.columns:
+    adata.obs['batch'] = 'batch_1'  # Default single batch
+
+# Step 2: Handle NaN/missing values - CRITICAL!
+adata.obs['batch'] = adata.obs['batch'].fillna('unknown')
+
+# Step 3: Convert to categorical for efficient memory usage
+adata.obs['batch'] = adata.obs['batch'].astype('category')
+
+# Now safe to use in batch-aware operations
+ov.pp.combat(adata, batch='batch')  # or other batch correction methods
+```
+
+**WRONG - DO NOT USE:**
+```python
+# WRONG! Using batch column without validation can cause NaN errors
+# ov.pp.combat(adata, batch='batch')  # May fail if batch has NaN values!
+
+# WRONG! Assuming batch column exists
+# adata.obs['batch'].unique()  # KeyError if column doesn't exist!
+```
+
+### Common Batch-Related Pitfalls
+
+1. **NaN values in batch column**: Always use `fillna()` before batch operations
+2. **Missing batch column**: Always check existence before use
+3. **Non-categorical batch**: Convert to category for memory efficiency
+4. **Mixed data types**: Ensure consistent string type before categorization
+
+```python
+# Complete defensive batch preparation pattern:
+def prepare_batch_column(adata, batch_key='batch', default_batch='batch_1'):
+    """Prepare batch column for batch-aware operations."""
+    if batch_key not in adata.obs.columns:
+        adata.obs[batch_key] = default_batch
+    adata.obs[batch_key] = adata.obs[batch_key].fillna('unknown')
+    adata.obs[batch_key] = adata.obs[batch_key].astype(str).astype('category')
+    return adata
+```
+
+## Highly Variable Genes (HVG) - Small Dataset Handling
+
+### LOESS Failure with Small Batches
+
+**IMPORTANT**: The `seurat_v3` HVG flavor uses LOESS regression which fails on small datasets or small per-batch subsets (<500 cells per batch). This manifests as:
+```
+ValueError: Extrapolation not allowed with blending
+```
+
+**CORRECT - Use try/except fallback pattern:**
+```python
+# Robust HVG selection for any dataset size
+try:
+    sc.pp.highly_variable_genes(
+        adata,
+        flavor='seurat_v3',
+        n_top_genes=2000,
+        batch_key='batch'  # if batch correction is needed
+    )
+except ValueError as e:
+    if 'Extrapolation' in str(e) or 'LOESS' in str(e):
+        # Fallback to simpler method for small datasets
+        sc.pp.highly_variable_genes(
+            adata,
+            flavor='seurat',  # Works with any size
+            n_top_genes=2000
+        )
+    else:
+        raise
+```
+
+**Alternative - Use cell_ranger flavor for batch-aware HVG:**
+```python
+# cell_ranger flavor is more robust for batched data
+sc.pp.highly_variable_genes(
+    adata,
+    flavor='cell_ranger',  # No LOESS, works with batches
+    n_top_genes=2000,
+    batch_key='batch'
+)
+```
+
+### Best Practices for Batch-Aware HVG
+
+1. **Check batch sizes before HVG**: Small batches (<500 cells) will cause LOESS to fail
+2. **Prefer `seurat` or `cell_ranger`** when batch sizes vary significantly
+3. **Use `seurat_v3` only** when all batches have >500 cells
+4. **Always wrap in try/except** when dataset size is unknown
+
+```python
+# Safe batch-aware HVG pattern
+def safe_highly_variable_genes(adata, batch_key='batch', n_top_genes=2000):
+    """Select HVGs with automatic fallback for small batches."""
+    try:
+        sc.pp.highly_variable_genes(
+            adata, flavor='seurat_v3', n_top_genes=n_top_genes, batch_key=batch_key
+        )
+    except ValueError:
+        # Fallback for small batches
+        sc.pp.highly_variable_genes(
+            adata, flavor='seurat', n_top_genes=n_top_genes
+        )
+```
+
+## Examples
+- "Download PBMC3k counts, run QC with Scrublet, normalise with `shiftlog|pearson`, and compute MDE + UMAP embeddings on CPU."
+- "Set up the mixed CPU–GPU workflow in a fresh conda env, recover raw counts after normalisation, and score cell cycle phases before Leiden clustering."
+- "Provision a RAPIDS environment, transfer AnnData to GPU, run `method='cagra'` neighbours, and return embeddings to CPU for plotting."
+
+## References
+- Detailed walkthrough notebooks: [`t_preprocess.ipynb`](../../omicverse_guide/docs/Tutorials-single/t_preprocess.ipynb), [`t_preprocess_cpu.ipynb`](../../omicverse_guide/docs/Tutorials-single/t_preprocess_cpu.ipynb), [`t_preprocess_gpu.ipynb`](../../omicverse_guide/docs/Tutorials-single/t_preprocess_gpu.ipynb)
+- Quick copy/paste commands: [`reference.md`](reference.md)

package/data/skills/single-to-spatial-mapping/SKILL.md

@@ -0,0 +1,48 @@
+---
+name: single2spatial-spatial-mapping
+title: Single2Spatial spatial mapping
+description: Map scRNA-seq atlases onto spatial transcriptomics slides using omicverse's Single2Spatial workflow for deep-forest training, spot-level assessment, and marker visualisation.
+---
+
+# Single2Spatial spatial mapping
+
+## Overview
+Apply this skill when converting single-cell references into spatially resolved profiles. It follows [`t_single2spatial.ipynb`](../../omicverse_guide/docs/Tutorials-bulk2single/t_single2spatial.ipynb), demonstrating how Single2Spatial trains on PDAC scRNA-seq and Visium data, reconstructs spot-level proportions, and visualises marker expression.
+
+## Instructions
+1. **Import dependencies and style**
+   - Load `omicverse as ov`, `scanpy as sc`, `anndata`, `pandas as pd`, `numpy as np`, and `matplotlib.pyplot as plt`.
+   - Call `ov.utils.ov_plot_set()` (or `ov.plot_set()` in older versions) to align plots with omicverse styling.
+2. **Load single-cell and spatial datasets**
+   - Read processed matrices with `pd.read_csv(...)` then create AnnData objects (`anndata.AnnData(raw_df.T)`).
+   - Attach metadata: `single_data.obs = pd.read_csv(...)[['Cell_type']]` and `spatial_data.obs = pd.read_csv(... )` containing coordinates and slide metadata.
+3. **Initialise Single2Spatial**
+   - Instantiate `ov.bulk2single.Single2Spatial(single_data=single_data, spatial_data=spatial_data, celltype_key='Cell_type', spot_key=['xcoord','ycoord'], gpu=0)`.
+   - Note that inputs should be normalised/log-scaled scRNA-seq matrices; ensure `spot_key` matches spatial coordinate columns.
+4. **Train the deep-forest model**
+   - Execute `st_model.train(spot_num=500, cell_num=10, df_save_dir='...', df_save_name='pdac_df', k=10, num_epochs=1000, batch_size=1000, predicted_size=32)` to fit the mapper and generate reconstructed spatial AnnData (`sp_adata`).
+   - Explain that `spot_num` defines sampled pseudo-spots per iteration and `cell_num` controls per-spot cell draws.
+5. **Load pretrained weights**
+   - Use `st_model.load(modelsize=14478, df_load_dir='.../pdac_df.pth', k=10, predicted_size=32)` when checkpoints already exist to skip training.
+6. **Assess spot-level outputs**
+   - Call `st_model.spot_assess()` to compute aggregated spot AnnData (`sp_adata_spot`) for QC.
+   - Plot marker genes with `sc.pl.embedding(sp_adata, basis='X_spatial', color=['REG1A', 'CLDN1', ...], frameon=False, ncols=4)`.
+7. **Visualise proportions and cell-type maps**
+   - Use `sc.pl.embedding(sp_adata_spot, basis='X_spatial', color=['Acinar cells', ...], frameon=False)` to highlight per-spot cell fractions.
+   - Plot `sp_adata` coloured by `Cell_type` with `palette=ov.utils.ov_palette()[11:]` to show reconstructed assignments.
+8. **Export results**
+   - Encourage saving generated AnnData objects (`sp_adata.write_h5ad(...)`, `sp_adata_spot.write_h5ad(...)`) and derived CSV summaries for downstream reporting.
+9. **Troubleshooting tips**
+   - If training diverges, reduce `learning_rate` via keyword arguments or decrease `predicted_size` to stabilise the forest.
+   - Ensure scRNA-seq inputs are log-normalised; raw counts can lead to scale mismatches and poor spatial predictions.
+   - Verify GPU availability when `gpu` is non-zero; fallback to CPU by omitting the argument or setting `gpu=-1`.
+
+## Examples
+- "Train Single2Spatial on PDAC scRNA-seq and Visium slides, then visualise REG1A and CLDN1 spatial expression."
+- "Load a saved Single2Spatial checkpoint to regenerate spot-level cell-type proportions for reporting."
+- "Plot reconstructed cell-type maps with omicverse palettes to compare against histology."
+
+## References
+- Tutorial notebook: [`t_single2spatial.ipynb`](../../omicverse_guide/docs/Tutorials-bulk2single/t_single2spatial.ipynb)
+- Example datasets and models: [`omicverse_guide/docs/Tutorials-bulk2single/data/pdac/`](../../omicverse_guide/docs/Tutorials-bulk2single/data/pdac/)
+- Quick copy/paste commands: [`reference.md`](reference.md)

package/data/skills/single-trajectory/SKILL.md

@@ -0,0 +1,62 @@
+---
+name: single-trajectory-analysis
+title: Single-trajectory analysis
+description: Guide to reproducing OmicVerse trajectory workflows spanning PAGA, Palantir, VIA, velocity coupling, and fate scoring notebooks.
+---
+
+# Single-trajectory analysis skill
+
+## Overview
+This skill describes how to reproduce and extend the single-trajectory analysis workflow in `omicverse`, combining graph-based trajectory inference, RNA velocity coupling, and downstream fate scoring notebooks.
+
+## Trajectory setup
+- **PAGA (Partition-based graph abstraction)**
+  - Build a neighborhood graph (`pp.neighbors`) on the preprocessed AnnData object.
+  - Use `tl.paga` to compute cluster connectivity and `tl.draw_graph` or `tl.umap` with `init_pos='paga'` for embedding.
+  - Interpret edge weights to prioritize branch resolution and seed paths.
+- **Palantir**
+  - Run `Palantir` on diffusion components, seeding with manually selected start cells (e.g., naïve T cells).
+  - Extract pseudotime, branch probabilities, and differentiation potential for subsequent overlays.
+- **VIA**
+  - Execute `via.VIA` on the kNN graph to identify lineage progression with automatic root selection or user-defined roots.
+  - Export terminal states and pseudotime for cross-validation against PAGA and Palantir results.
+
+## Velocity coupling (VIA + scVelo)
+- Use `scv.pp.filter_and_normalize`, `scv.pp.moments`, and `scv.tl.velocity` to generate velocity layers.
+- Provide VIA with `adata.layers['velocity']` to refine lineage directionality (`via.VIA(..., velocity_weight=...)`).
+- Compare VIA pseudotime with scVelo latent time (`scv.tl.latent_time`) to validate directionality and root selection.
+
+## Downstream fate scoring notebooks
+- **`t_cellfate*.ipynb`**: Map lineage probabilities onto T-cell subsets, quantify fate bias, and visualize heatmaps.
+- **`t_metacells.ipynb`**: Aggregate metacell trajectories for robustness checks and meta-state differential expression.
+- **`t_cytotrace.ipynb`**: Integrate CytoTRACE differentiation potential with velocity-informed lineages for maturation scoring.
+
+## Required preprocessing
+1. Quality control: remove low-quality cells/genes, apply doublet filtering.
+2. Normalization & log transformation (`sc.pp.normalize_total`, `sc.pp.log1p`).
+3. Highly variable gene selection tailored to immune datasets (`sc.pp.highly_variable_genes`).
+4. Batch correction if necessary (e.g., `scvi-tools`, `bbknn`).
+5. Compute PCA, neighbor graph, and embedding (UMAP/FA) used by all trajectory methods.
+6. For velocity: compute moments on the same neighbor graph before running VIA coupling.
+
+## Parameter tuning
+- Neighbor graph `n_neighbors` and `n_pcs` should be harmonized across PAGA, VIA, and Palantir to maintain consistency.
+- In VIA, adjust `knn`, `too_big_factor`, and `root_user` for datasets with uneven sampling.
+- Palantir requires careful start cell selection; use marker genes and velocity arrows to confirm.
+- For PAGA, tweak `threshold` to control edge sparsity; ensure connected components reflect biological branches.
+- Velocity estimation: compare `mode='stochastic'` vs `mode='dynamical'` in scVelo; recalibrate if terminal states disagree with VIA.
+
+## Visualization and export
+1. Overlay PAGA edges on UMAP (`scv.pl.paga`) and annotate branch labels.
+2. Plot Palantir pseudotime and branch probabilities on embeddings.
+3. Visualize VIA trajectories using `via.plot_fates` and `via.plot_scatter`.
+4. Export pseudotime tables and fate probabilities to CSV for downstream notebooks.
+5. Save high-resolution figures (PNG/SVG) and notebook artifacts for reproducibility.
+6. Update notebooks with consistent color schemes and metadata columns before sharing.
+
+## Troubleshooting tips
+- **Missing velocity layers**: re-run `scv.pp.moments` and `scv.tl.velocity` ensuring `adata.layers['spliced']`/`['unspliced']` exist; verify loom/H5AD import preserved layers.
+- **Disconnected PAGA graph**: inspect neighbor graph or adjust `n_neighbors`; confirm batch correction didn’t fragment the manifold.
+- **Palantir convergence issues**: reduce diffusion components or reinitialize start cells; ensure no NaN values in data matrix.
+- **VIA terminal states unstable**: increase iterations (`cluster_graph_pruning_iter`), or provide manual terminal state hints based on marker expression.
+- **Notebook kernel memory errors**: downsample cells or precompute summaries (metacells) before rerunning.