pycopro 0.1.0__tar.gz

pycopro-0.1.0/PKG-INFO

@@ -0,0 +1,14 @@
+Metadata-Version: 2.4
+Name: pycopro
+Version: 0.1.0
+Summary: Spatial Kernel-based Reduced Rank CCA for spatial transcriptomics
+Requires-Python: >=3.10
+Requires-Dist: numpy>=1.24
+Requires-Dist: scipy>=1.10
+Requires-Dist: pandas>=2.0
+Requires-Dist: scikit-learn>=1.3
+Requires-Dist: pyarrow>=12
+Requires-Dist: anndata>=0.10
+Provides-Extra: dev
+Requires-Dist: pytest>=7.0; extra == "dev"
+Requires-Dist: pytest-cov; extra == "dev"

pycopro-0.1.0/README.md

@@ -0,0 +1,161 @@
+# CoPro (Python)
+
+**Unsupervised detection of coordinated spatial progressions in spatial transcriptomics**
+
+[![Python](https://img.shields.io/badge/python-≥3.10-blue)](https://www.python.org)
+[![PyPI](https://img.shields.io/badge/pypi-pycopro-orange)](https://pypi.org/project/pycopro/)
+[![GitHub](https://img.shields.io/badge/github-Zhen--Miao%2Fcopro--python-lightgrey)](https://github.com/Zhen-Miao/copro-python)
+[![License: MIT](https://img.shields.io/badge/license-MIT-green)](LICENSE)
+
+CoPro detects **coordinated spatial progressions** between cell types in spatial transcriptomics data. Given the spatial positions and gene expression profiles of cells, CoPro finds a low-dimensional axis along which cells of one type are spatially co-organized with cells of another type — or within a single cell type — revealing continuous tissue structure that discrete clustering misses.
+
+The method is built on **Spatial Kernel-based Reduced Rank CCA (SkrCCA)**: a power-method optimization that maximizes a spatially-weighted cross-covariance between cell type-specific PC score matrices.
+
+> **R package:** The original R implementation is available at [github.com/Zhen-Miao/CoPro](https://github.com/Zhen-Miao/CoPro).
+
+---
+
+## Installation
+
+```bash
+pip install pycopro
+```
+
+Or install from source:
+
+```bash
+git clone https://github.com/Zhen-Miao/copro-python.git
+cd copro-python
+pip install -e .
+```
+
+**Requirements:** Python ≥ 3.10, NumPy, SciPy, pandas, scikit-learn.
+
+---
+
+## Quick start
+
+```python
+import numpy as np
+import pandas as pd
+import copro as cp
+
+# Inputs
+# expr_norm : np.ndarray  (cells × genes, normalized)
+# location  : pd.DataFrame with columns "x", "y"
+# cell_types: np.ndarray  (per-cell type labels)
+
+obj = cp.CoProSingle(
+    normalized_data=expr_norm,
+    location_data=location,
+    meta_data=meta,
+    cell_types=cell_types,
+)
+
+obj = cp.subset_data(obj, ["Cell type A", "Cell type B"])
+obj = cp.compute_pca(obj, n_pca=30)
+obj = cp.compute_distance(obj, normalize=False)
+obj = cp.compute_kernel_matrix(obj, sigma_values=[0.1, 0.2, 0.5])
+obj = cp.run_skr_cca(obj, scale_pcs=True, n_cc=2)
+obj = cp.compute_normalized_correlation(obj)
+obj = cp.compute_gene_and_cell_scores(obj)
+
+print("Selected sigma:", obj.sigma_value_choice)
+
+# Cell scores for the optimal sigma
+sigma = obj.sigma_value_choice
+scores_A = obj.cell_scores[f"cellScores|sigma{sigma}|Cell type A"][:, 0]
+scores_B = obj.cell_scores[f"cellScores|sigma{sigma}|Cell type B"][:, 0]
+```
+
+---
+
+## How it works
+
+CoPro runs a seven-step pipeline:
+
+| Step | Function | Description |
+|------|----------|-------------|
+| 1 | `subset_data` | Filter to cell types of interest |
+| 2 | `compute_pca` | Truncated PCA per cell type (ARPACK, matching R IRLBA) |
+| 3 | `compute_distance` | Pairwise Euclidean distances (within and between types) |
+| 4 | `compute_kernel_matrix` | Gaussian RBF kernel: K = exp(−d²/2σ²) |
+| 5 | `run_skr_cca` | SkrCCA power-method optimization over σ values |
+| 6 | `compute_normalized_correlation` | Spectral-norm normalized CCA correlation per σ and CC |
+| 7 | `compute_gene_and_cell_scores` | Project CCA weights to cell and gene space |
+
+Sigma selection is automatic: the σ that maximizes the mean CC1 normalized correlation is chosen as `obj.sigma_value_choice`.
+
+**Multi-slide support:** Use `CoProMulti` for datasets spanning multiple tissue sections. CCA weights are learned jointly across slides; cell scores are computed per slide.
+
+---
+
+## Tutorials
+
+| Notebook | Dataset | Description |
+|----------|---------|-------------|
+| [`tutorials/tutorial_one_cell_type.ipynb`](tutorials/tutorial_one_cell_type.ipynb) | Intestinal organoid (seqFISH) | Within-type spatial self-organization of epithelial cells along the crypt–villus axis |
+| [`tutorials/tutorial_two_cell_types.ipynb`](tutorials/tutorial_two_cell_types.ipynb) | Mouse brain MERFISH (Zhang et al. *Nature* 2023) | Cross-type co-progression between D1 and D2 striatal neurons |
+
+---
+
+## API reference
+
+### Data containers
+
+**`CoProSingle(normalized_data, location_data, meta_data, cell_types)`**
+Single-slide state container. All pipeline functions modify and return this object.
+
+**`CoProMulti(normalized_data, location_data, meta_data, cell_types)`**
+Multi-slide variant. Requires a `slideID` column in `meta_data`.
+
+### Pipeline functions
+
+```python
+cp.subset_data(obj, cell_types_of_interest)
+cp.compute_pca(obj, n_pca=30, center=True, scale=True)
+cp.compute_distance(obj, normalize=False)
+cp.compute_kernel_matrix(obj, sigma_values, upper_quantile=0.85, lower_limit=5e-7)
+cp.run_skr_cca(obj, scale_pcs=True, n_cc=2, max_iter=500, tol=1e-5)
+cp.compute_normalized_correlation(obj, tol=1e-4)
+cp.compute_gene_and_cell_scores(obj)
+```
+
+### Key output slots
+
+| Attribute | Type | Description |
+|-----------|------|-------------|
+| `obj.sigma_value_choice` | `float` | Automatically selected σ |
+| `obj.normalized_correlation` | `dict[str, DataFrame]` | Normalized correlation per σ and CC |
+| `obj.cell_scores` | `dict[str, ndarray]` | Cell scores, keyed `"cellScores\|sigma{s}\|{ct}"` |
+| `obj.gene_scores` | `dict[str, ndarray]` | Gene scores, keyed `"geneScores\|sigma{s}\|{ct}"` |
+| `obj.kernel_matrices` | `dict[str, ndarray]` | Kernel matrices, keyed `"kernel\|sigma{s}\|{ct_i}\|{ct_j}"` |
+
+---
+
+## Numerical equivalence with R
+
+The Python implementation is numerically validated against the R package on simulation and real datasets. Key design choices ensuring equivalence:
+
+- PCA uses `scipy.sparse.linalg.svds` (ARPACK), the same Krylov-subspace family as R's `irlba::prcomp_irlba`
+- Sign convention matches `prcomp_irlba` via `sklearn.utils.extmath.svd_flip`
+- Distance and kernel computations are algebraically identical to R's `fields::rdist` + Gaussian RBF
+- Kernel matrices agree to machine epsilon (~10⁻¹⁶)
+- Cell score Pearson |r| vs R ≥ 0.9999 across all tested datasets and sigma values
+
+---
+
+## Citation
+
+If you use CoPro in your research, please cite:
+
+> Miao Z. et al. *CoPro: Unsupervised detection of coordinated spatial progressions in spatial transcriptomics* (in preparation).
+
+**MERFISH tutorial data:**
+> Zhang, M., Pan, X., Jung, W. et al. Molecularly defined and spatially resolved cell atlas of the whole mouse brain. *Nature* 624, 343–354 (2023). https://doi.org/10.1038/s41586-023-06808-9
+
+---
+
+## License
+
+MIT

pycopro-0.1.0/copro/__init__.py

@@ -0,0 +1,23 @@
+"""CoPro Python — Spatial Kernel-based Reduced Rank CCA for spatial transcriptomics."""
+
+from .core import CoProSingle, CoProMulti, subset_data
+from .pca import compute_pca
+from .distance import compute_distance
+from .kernel import compute_kernel_matrix
+from .skrcca import run_skr_cca
+from .correlation import compute_normalized_correlation
+from .scores import compute_gene_and_cell_scores
+
+__all__ = [
+    "CoProSingle",
+    "CoProMulti",
+    "subset_data",
+    "compute_pca",
+    "compute_distance",
+    "compute_kernel_matrix",
+    "run_skr_cca",
+    "compute_normalized_correlation",
+    "compute_gene_and_cell_scores",
+]
+
+__version__ = "0.1.0"

pycopro-0.1.0/copro/core.py

@@ -0,0 +1,124 @@
+"""CoProSingle and CoProMulti dataclasses — state containers."""
+
+from __future__ import annotations
+
+from dataclasses import dataclass, field
+from typing import Optional
+
+import numpy as np
+import pandas as pd
+
+
+@dataclass
+class CoProSingle:
+    # Input data
+    normalized_data: np.ndarray          # cells × genes
+    location_data: pd.DataFrame          # cells × {x, y, ...}
+    meta_data: pd.DataFrame
+    cell_types: np.ndarray               # per-cell label vector
+
+    # Set by subset_data
+    cell_types_of_interest: list = field(default_factory=list)
+    normalized_data_sub: Optional[np.ndarray] = None
+    location_data_sub: Optional[pd.DataFrame] = None
+    cell_types_sub: Optional[np.ndarray] = None
+
+    # Computed results (keyed dicts)
+    pca_global: dict = field(default_factory=dict)        # ct → dict with components/scores/sdev
+    distances: dict = field(default_factory=dict)         # flat keys: "dist|A|B"
+    kernel_matrices: dict = field(default_factory=dict)   # flat keys: "kernel|sigma0.1|A|B"
+    sigma_values: list = field(default_factory=list)
+    skr_cca_out: dict = field(default_factory=dict)       # "sigma_0.1" → {ct: w_matrix}
+    normalized_correlation: dict = field(default_factory=dict)
+    sigma_value_choice: Optional[float] = None
+    cell_scores: dict = field(default_factory=dict)
+    gene_scores: dict = field(default_factory=dict)
+    n_cc: int = 2
+    n_pca: int = 30
+    scale_pcs: bool = True
+
+
+@dataclass
+class CoProMulti:
+    """Multi-slide CoPro object. meta_data must have a 'slideID' column."""
+    # Input data
+    normalized_data: np.ndarray
+    location_data: pd.DataFrame
+    meta_data: pd.DataFrame
+    cell_types: np.ndarray
+
+    # Slide list
+    slide_list: list = field(default_factory=list)        # ordered list of slide IDs
+
+    # Set by subset_data
+    cell_types_of_interest: list = field(default_factory=list)
+    normalized_data_sub: Optional[np.ndarray] = None
+    location_data_sub: Optional[pd.DataFrame] = None
+    cell_types_sub: Optional[np.ndarray] = None
+    meta_data_sub: Optional[pd.DataFrame] = None         # subset of meta_data (with slideID)
+
+    # PCA
+    pca_global: dict = field(default_factory=dict)        # ct → global PCA dict (rotation, sdev)
+    pca_results: dict = field(default_factory=dict)       # slide → {ct → scores matrix}
+
+    # Computed results
+    distances: dict = field(default_factory=dict)         # flat keys: "dist|{slide}|A|B"
+    kernel_matrices: dict = field(default_factory=dict)   # flat keys: "kernel|sigma0.1|{slide}|A|B"
+    sigma_values: list = field(default_factory=list)
+    skr_cca_out: dict = field(default_factory=dict)       # "sigma_0.1" → {ct: w_matrix} (shared)
+    normalized_correlation: dict = field(default_factory=dict)
+    sigma_value_choice: Optional[float] = None
+    cell_scores: dict = field(default_factory=dict)       # "cellScores|sigma0.1|{slide}|{ct}"
+    gene_scores: dict = field(default_factory=dict)       # "geneScores|sigma0.1|{ct}" (shared)
+    n_cc: int = 2
+    n_pca: int = 30
+    scale_pcs: bool = True
+
+
+def subset_data(obj, cell_types_of_interest: list, min_cells: int = 10):
+    """Filter data to listed cell types. Works for both CoProSingle and CoProMulti."""
+    if isinstance(obj, CoProMulti):
+        return _subset_data_multi(obj, cell_types_of_interest, min_cells)
+    else:
+        return _subset_data_single(obj, cell_types_of_interest, min_cells)
+
+
+def _subset_data_single(obj: CoProSingle, cell_types_of_interest: list, min_cells: int) -> CoProSingle:
+    for ct in cell_types_of_interest:
+        n = np.sum(obj.cell_types == ct)
+        if n < min_cells:
+            raise ValueError(
+                f"Cell type '{ct}' has only {n} cells (minimum {min_cells} required)."
+            )
+    mask = np.isin(obj.cell_types, cell_types_of_interest)
+    obj.cell_types_of_interest = list(cell_types_of_interest)
+    obj.normalized_data_sub = obj.normalized_data[mask]
+    obj.location_data_sub = obj.location_data.loc[mask].reset_index(drop=True)
+    obj.cell_types_sub = obj.cell_types[mask]
+    return obj
+
+
+def _subset_data_multi(obj: CoProMulti, cell_types_of_interest: list, min_cells: int) -> CoProMulti:
+    """Subset multi-slide object. Checks per-slide cell counts."""
+    if "slideID" not in obj.meta_data.columns:
+        raise ValueError("meta_data must have a 'slideID' column for CoProMulti.")
+
+    # Discover slide list from meta_data if not set
+    if not obj.slide_list:
+        obj.slide_list = sorted(obj.meta_data["slideID"].unique().tolist())
+
+    for ct in cell_types_of_interest:
+        # Check total across all slides
+        n_total = np.sum(obj.cell_types == ct)
+        if n_total < min_cells:
+            raise ValueError(
+                f"Cell type '{ct}' has only {n_total} cells total (minimum {min_cells} required)."
+            )
+
+    mask = np.isin(obj.cell_types, cell_types_of_interest)
+    obj.cell_types_of_interest = list(cell_types_of_interest)
+    obj.normalized_data_sub = obj.normalized_data[mask]
+    obj.location_data_sub = obj.location_data.loc[mask].reset_index(drop=True)
+    obj.cell_types_sub = obj.cell_types[mask]
+    obj.meta_data_sub = obj.meta_data.loc[mask].reset_index(drop=True)
+    return obj

pycopro-0.1.0/copro/correlation.py

@@ -0,0 +1,248 @@
+"""compute_normalized_correlation() — spectral-norm normalized CCA correlation."""
+
+from __future__ import annotations
+
+from itertools import combinations
+
+import numpy as np
+import pandas as pd
+from scipy.sparse.linalg import svds
+
+from .core import CoProSingle
+from .skrcca import _prepare_pc_matrices
+
+
+def _spectral_norm(K: np.ndarray, tol: float = 1e-4) -> float:
+    """Largest singular value of K (spectral norm)."""
+    try:
+        s = svds(K.astype(float), k=1, tol=tol, return_singular_vectors=False)
+        return float(s[0])
+    except Exception:
+        return float(np.linalg.norm(K, ord=2))
+
+
+
+def _get_kernel_for_pair(flat_kernels, sigma, ct_i, ct_j, slide=None):
+    """Retrieve kernel, optionally slide-aware, trying both orderings."""
+    if slide is None:
+        name = f"kernel|sigma{sigma}|{ct_i}|{ct_j}"
+        name_sym = f"kernel|sigma{sigma}|{ct_j}|{ct_i}"
+    else:
+        name = f"kernel|sigma{sigma}|{slide}|{ct_i}|{ct_j}"
+        name_sym = f"kernel|sigma{sigma}|{slide}|{ct_j}|{ct_i}"
+    if name in flat_kernels:
+        return flat_kernels[name]
+    if name_sym in flat_kernels:
+        return flat_kernels[name_sym].T
+    raise KeyError(f"Kernel not found for ({ct_i},{ct_j}) sigma={sigma} slide={slide}")
+
+
+def compute_normalized_correlation(obj, tol: float = 1e-4):
+    """Compute normalized CCA correlation for each sigma × pair × CC.
+
+    Dispatches to multi-slide version for CoProMulti objects.
+
+    Formula:
+        numerator   = (A @ w1)^T K (B @ w2)
+        denominator = ||A @ w1|| * ||B @ w2|| * ||K||_spec
+        norm_corr   = numerator / denominator
+
+    Stores in obj.normalized_correlation[sigma_name] = DataFrame.
+    Chooses obj.sigma_value_choice as sigma maximizing mean CC1 correlation.
+    """
+    from .core import CoProMulti
+    if isinstance(obj, CoProMulti):
+        return _compute_normalized_correlation_multi(obj, tol)
+
+    # --- Single-slide path ---
+    cts = obj.cell_types_of_interest
+    if not cts:
+        raise ValueError("No cell types of interest.")
+    if not obj.skr_cca_out:
+        raise ValueError("CCA results missing. Run run_skr_cca() first.")
+
+    scale_pcs = getattr(obj, "scale_pcs", True)
+    n_cc = obj.n_cc
+
+    # Scaled PC matrices
+    X_dict = _prepare_pc_matrices(obj, scale_pcs, cts)
+
+    # Pairs
+    if len(cts) == 1:
+        pairs = [(cts[0], cts[0])]
+    else:
+        pairs = list(combinations(cts, 2))
+
+    print("Calculating spectral norms (may take a while)...")
+
+    # Precompute spectral norms for each sigma × pair
+    spec_norms = {}
+    for sigma in obj.sigma_values:
+        spec_norms[sigma] = {}
+        for ct_i, ct_j in pairs:
+            try:
+                K = _get_kernel_for_pair(obj.kernel_matrices, sigma, ct_i, ct_j)
+                spec_norms[sigma][(ct_i, ct_j)] = _spectral_norm(K, tol=tol)
+                spec_norms[sigma][(ct_j, ct_i)] = spec_norms[sigma][(ct_i, ct_j)]
+            except KeyError:
+                spec_norms[sigma][(ct_i, ct_j)] = np.nan
+
+    print("Finished calculating spectral norms.")
+
+    correlation_value = {}
+
+    for sigma in obj.sigma_values:
+        sigma_name = f"sigma_{sigma}"
+        w_sigma = obj.skr_cca_out.get(sigma_name)
+        if w_sigma is None:
+            continue
+
+        rows = []
+        for ct_i, ct_j in pairs:
+            A = X_dict[ct_i]
+            B = X_dict[ct_j]
+            try:
+                K = _get_kernel_for_pair(obj.kernel_matrices, sigma, ct_i, ct_j)
+            except KeyError:
+                continue
+            norm_K = spec_norms[sigma].get((ct_i, ct_j), np.nan)
+
+            for cc in range(n_cc):
+                w1 = w_sigma[ct_i][:, cc : cc + 1]
+                w2 = w_sigma[ct_j][:, cc : cc + 1]
+
+                Aw1 = A @ w1
+                Bw2 = B @ w2
+
+                numerator = float((Aw1.T @ K @ Bw2).flat[0])
+                denom = float(np.sqrt(np.sum(Aw1 ** 2))) * float(np.sqrt(np.sum(Bw2 ** 2))) * norm_K
+
+                norm_corr = 0.0 if abs(denom) < 1e-9 else numerator / denom
+
+                rows.append({
+                    "sigma": sigma,
+                    "cell_type_1": ct_i,
+                    "cell_type_2": ct_j,
+                    "CC_index": cc + 1,
+                    "normalized_correlation": norm_corr,
+                })
+
+        correlation_value[sigma_name] = pd.DataFrame(rows)
+
+    obj.normalized_correlation = correlation_value
+
+    # Choose sigma maximizing mean CC1 correlation
+    all_cc1 = []
+    for sigma_name, df in correlation_value.items():
+        if df is not None and len(df) > 0:
+            cc1 = df[df["CC_index"] == 1]
+            mean_corr = cc1["normalized_correlation"].mean()
+            sigma_val = float(sigma_name.replace("sigma_", ""))
+            all_cc1.append((sigma_val, mean_corr))
+
+    if all_cc1:
+        obj.sigma_value_choice = max(all_cc1, key=lambda x: x[1])[0]
+
+    return obj
+
+
+def _compute_normalized_correlation_multi(obj, tol=1e-4):
+    """Multi-slide normalized correlation: per-slide values matching R format.
+
+    R computes normalized correlation independently for each slide using the
+    raw (unscaled) per-slide PCA scores from pcaResults (not scaled by sdev).
+    We replicate this: for each (sigma, slide, pair, CC), compute norm_corr
+    using only that slide's raw PCA scores and per-slide spectral norm.
+    Sigma choice is based on the mean CC1 correlation across slides.
+    """
+    cts = obj.cell_types_of_interest
+    slides = obj.slide_list
+    n_cc = obj.n_cc
+
+    # Use raw (unscaled) per-slide PCA scores — matching R's pcaResults usage
+    X_list_all = {
+        slide: {ct: obj.pca_results[slide][ct].astype(float)
+                for ct in cts if ct in obj.pca_results.get(slide, {})}
+        for slide in slides
+    }
+
+    if len(cts) == 1:
+        pairs = [(cts[0], cts[0])]
+    else:
+        pairs = list(combinations(cts, 2))
+
+    # Precompute per-slide spectral norms for each sigma × pair
+    print("Calculating spectral norms (multi-slide)...")
+    spec_norms = {}  # spec_norms[sigma][(ct_i, ct_j, slide)]
+    for sigma in obj.sigma_values:
+        spec_norms[sigma] = {}
+        for ct_i, ct_j in pairs:
+            for slide in slides:
+                try:
+                    K = _get_kernel_for_pair(obj.kernel_matrices, sigma, ct_i, ct_j, slide)
+                    val = _spectral_norm(K, tol)
+                except KeyError:
+                    val = np.nan
+                spec_norms[sigma][(ct_i, ct_j, slide)] = val
+                spec_norms[sigma][(ct_j, ct_i, slide)] = val
+    print("Finished spectral norms.")
+
+    correlation_value = {}
+
+    for sigma in obj.sigma_values:
+        sigma_name = f"sigma_{sigma}"
+        w_sigma = obj.skr_cca_out.get(sigma_name)
+        if w_sigma is None:
+            continue
+
+        rows = []
+        for ct_i, ct_j in pairs:
+            for cc in range(n_cc):
+                w1 = w_sigma[ct_i][:, cc:cc+1]
+                w2 = w_sigma[ct_j][:, cc:cc+1]
+
+                # Per-slide correlation (matches R format)
+                for slide in slides:
+                    A = X_list_all[slide].get(ct_i)
+                    B = X_list_all[slide].get(ct_j)
+                    if A is None or B is None:
+                        continue
+                    try:
+                        K = _get_kernel_for_pair(obj.kernel_matrices, sigma, ct_i, ct_j, slide)
+                    except KeyError:
+                        continue
+
+                    norm_K = spec_norms[sigma].get((ct_i, ct_j, slide), np.nan)
+                    Aw1 = A @ w1
+                    Bw2 = B @ w2
+                    numerator = float((Aw1.T @ K @ Bw2).flat[0])
+                    denom = (float(np.linalg.norm(Aw1)) *
+                             float(np.linalg.norm(Bw2)) *
+                             norm_K)
+                    norm_corr = 0.0 if abs(denom) < 1e-9 else numerator / denom
+
+                    rows.append({
+                        "sigma": sigma,
+                        "slideID": slide,
+                        "cell_type_1": ct_i,
+                        "cell_type_2": ct_j,
+                        "CC_index": cc + 1,
+                        "normalized_correlation": norm_corr,
+                    })
+
+        correlation_value[sigma_name] = pd.DataFrame(rows)
+
+    obj.normalized_correlation = correlation_value
+
+    # Choose sigma maximizing mean CC1 correlation across slides
+    all_cc1 = []
+    for sigma_name, df in correlation_value.items():
+        if df is not None and len(df) > 0:
+            cc1 = df[df["CC_index"] == 1]
+            mean_corr = cc1["normalized_correlation"].mean()
+            sigma_val = float(sigma_name.replace("sigma_", ""))
+            all_cc1.append((sigma_val, mean_corr))
+    if all_cc1:
+        obj.sigma_value_choice = max(all_cc1, key=lambda x: x[1])[0]
+
+    return obj