dspin 1.3.1__tar.gz → 1.3.3__tar.gz

{dspin-1.3.1/dspin.egg-info → dspin-1.3.3}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: dspin
-Version: 1.3.1
+Version: 1.3.3
 Summary: Regulatory network models from single-cell perturbation profiling
 Home-page: https://github.com/JialongJiang/DSPIN
 Author: Jialong Jiang

dspin-1.3.3/README.md

@@ -0,0 +1,175 @@
+# DSPIN
+
+Tutorial, code and examples of the D-SPIN framework for the preprint "D-SPIN constructs gene regulatory network models from multiplexed scRNA-seq data revealing organizing principles of cellular perturbation response" ([bioRxiv](https://www.biorxiv.org/content/10.1101/2023.04.19.537364))
+
+<div align="center">
+  <img src="figure/readme/Overview_20250523.png" width="800">
+</div>
+
+## Installation
+
+D-SPIN is implemented in Python and MATLAB. The Python code is sufficient for the analysis of most datasets with less than 100k cells and a few hundreds of conditions. The MATLAB code is more efficient on large datasets as MATLAB can be easily deployed on clusters for parallelization with the build-in "parfor" function. 
+
+The python code can be installed with 
+
+`pip install dspin`
+
+The MATLAB code can be downloaded from the folder "DSPIN_matlab", and directly executable in MATLAB after specifying the path to the data. 
+
+## Demos
+
+Two demos of D-SPIN are available on Google Colab. 
+
+The first demo reconstructs the regulatory network of simulated hematopoietic stem cell (HSC) differentiation network with perturbations using the BEELINE framework (Pratapa, Aditya, et al. Nature methods, 2020). 
+
+[Demo1](https://colab.research.google.com/drive/1Nja8AAcP0L7Ag6DGbyn21SxGafsYX4fA?usp=sharing)
+
+The second demo reconstructs regulatory network and response vector in a public single-cell dataset of immune dictionary (Cui, Ang, et al. Nature, 2024), where mouse were treated with different cytokines and lymph nodes were than collected and profiled. 
+
+[Demo2](https://colab.research.google.com/drive/1E8v4awQ-m4D02emG0jp6kmRDmkZ8hjjN?usp=sharing)
+
+## Dependencies
+
+DSPIN package was tested with the following python packages versions:
+
+- python (3.9.18)
+- anndata (0.10.3)
+- matplotlib (3.8.2)
+- scanpy (1.9.6)
+- tqdm (4.65.0)
+- leidenalg (0.10.1)
+- igraph (0.10.8)
+
+Note: other versions may work as well.
+
+## Input data
+
+D‑SPIN can work with many perturbation types **as long as they share the same core regulatory network**:
+
+* **Genetic screens** – Perturb‑seq, CRISPR, RNAi  
+* **Chemical or signaling cues** – drug treatments, growth‑factor changes  
+* **Physiological differences** – healthy vs disease, different patients, time‑course samples  
+* **Spatial niches** – local micro‑environments from spatial transcriptomics  
+
+Because D‑SPIN models the *entire* distribution of transcriptional states, it expects **single‑cell RNA‑seq** data. Bulk data is possible but less informative.
+
+#### AnnData requirements
+
+* `adata.X` — log‑normalized (`log1p`) count matrix **after** QC: remove low‑quality cells, filter high mitochondrial content, keep highly variable genes.  
+* `adata.obs['sample_id']` — condition label; cells with the same value form one perturbation group (≥ 25 cells per group is ideal).  
+* `adata.obs['batch']` — batch identifier for cells processed together; D‑SPIN corrects batch effects by comparing conditions within each batch.  
+* `adata.obs['if_control']` — `True` for controls, `False` otherwise; perturbation effects are measured relative to these controls (or to the overall average if no controls exist).  
+
+## Building network models with D‑SPIN
+
+#### Gene‑level network models
+
+```python
+from dspin.dspin import DSPIN
+
+model = DSPIN(adata, save_path, num_spin=adata.shape[1])
+model.network_inference(
+    sample_id_key='sample_id',
+    method='pseudo_likelihood',
+    directed=True,
+    run_with_matlab=False,
+    params={'stepsz': 0.05, 'lam_l1_j': 0.01}
+)
+model.response_relative_to_control(
+    sample_id_key='sample_id',
+    if_control_key='if_control',
+    batch_key='batch'
+)
+```
+
+* **Results saved to `model`:**
+  * `model.network` – regulatory weights **J**
+  * `model.responses` – gene‑level responses **h**
+  * `model.relative_responses` – responses relative to control (per batch)
+
+* **Key options**
+  * `num_spin=adata.shape[1]` → build a *gene‑level* network  
+  * `directed=True` → only supported with `pseudo_likelihood`  
+  * `run_with_matlab=True` → skip Python inference and write variables to `save_path` for the MATLAB version (speeds up very large data sets)  
+  * `params` → hyper‑parameters such as step size, training epochs, and L1/L2 regularisation  
+
+---
+
+#### Program‑level network models
+
+```python
+from dspin.dspin import DSPIN
+
+model = DSPIN(adata, save_path, num_spin=15)      # 15 gene programs
+model.gene_program_discovery(
+    num_repeat=10,
+    seed=0,
+    cluster_key='cell_type'                       # balance cell types
+)
+model.network_inference(
+    sample_id_key='sample_id',
+    method='mcmc_maximum_likelihood'
+)
+model.response_relative_to_control()
+```
+
+* **Extra outputs**
+  * Gene‑program compositions in `save_path/onmf`
+  * Consensus program gene lists in `save_path`
+
+* **Tips**
+  * Choose `num_spin` ≈ 5 × (number of cell types); keep ≤ 40  
+  * `cluster_key` lets D‑SPIN down‑sample over‑represented cell types before oNMF  
+
+---
+
+#### Finding gene regulators of programs
+
+```python
+# model_program : program‑level DSPIN object
+# model_gene    : gene‑level   DSPIN object
+model_gene.program_regulator_discovery(
+    model_program.program_representation,
+    sample_id_key='sample_id',
+    params={'stepsz': 0.02, 'lam_l1_interaction': 0.01}
+)
+```
+
+* **Prerequisite** – `adata.obs` metadata must match between the two models so each cell’s gene and program states align.
+
+* **Outputs in `model_gene`:**
+  * `model_gene.program_interactions` – regression coefficients **J** (gene ↔ program)
+  * `model_gene.program_activities` – global activity **h** for each program
+
+
+## Application to the T cell population of the immune dictionary dataset
+
+
+<div align="center">
+  <img src="figure/readme/Figure2_immune_dictionary.png" width="600">
+</div>
+
+<div align="left">
+  <em>Figure&nbsp; Overview and program discovery for the immune dictionary dataset. (A) t-distributed stochastic neighbour (t-SNE) embedding for a subsetted cell population from the immune dictionary dataset \cite{cui2024dictionary}. The subset dataset includes CD4, CD8, and regulatory T cells treated by 12 different cytokines, as well as corresponding control samples treated by phosphate-buffered saline (PBS). 
+(B) Heatmaps of gene expression and discretized gene program levels for control and IFN-$\alpha$1-treated samples. The gene programs are weighted averages of single-gene expressions that characterize and denoise the major expression pattern of the gene matrix.</em>
+</div>
+
+<div align="center">
+  <img src="figure/readme/Figure3_program_network.png" width="600">
+</div>
+
+<div align="left">
+  <em>Figure&nbsp; Program/gene-level regulatory network inferred by D-SPIN.
+(A) Diagram of D-SPIN-inferred network model on gene programs. The network is partitioned into 4 modules, each associated with a T cell type in the population.
+(B) Heatmap of the program response of each cytokine. Clustering of the response partitioned the cytokines into 3 major categories. 
+(C) Diagram of the core subnetwork of the D-SPIN-inferred gene network model. The node sizes scale with the number of identified interactions. The network is partitioned into 4 modules, each primarily composed of genes that have elevated expression in one specific T cell type.
+(D) Interaction diagram for the subnetwork of IFN-$\alpha$1/IFN-$\beta$ and IFN-$\gamma$ acting on the program P18-IFN-$\gamma$ response. D-SPIN model shows that Type I and Type II interferons have different effectors to activate the program.</em>
+</div>
+
+## References
+
+1. Jiang, Jialong, et al. "D-SPIN constructs gene regulatory network models from multiplexed scRNA-seq data revealing organizing principles of cellular perturbation response." bioRxiv (2023).
+
+2. Pratapa, Aditya, et al. "Benchmarking algorithms for gene regulatory network inference from single-cell transcriptomic data." Nature methods 17.2 (2020): 147-154.
+
+3. Cui, Ang, et al. "Dictionary of immune responses to cytokines at single-cell resolution." Nature 625.7994 (2024): 377-384.

{dspin-1.3.1 → dspin-1.3.3}/dspin/compute.py

@@ -20,6 +20,7 @@ from tqdm import tqdm
 from typing import Tuple, List, Callable, Any, Dict
 import warnings
 from scipy.sparse import issparse
+from joblib import Parallel, delayed, parallel_backend
 
 
 def category_balance_number(
@@ -290,7 +291,9 @@ def compute_onmf(seed: int,
     return nmf_model
 
 
-def onmf_discretize(onmf_rep_ori: np.array, fig_folder: str = None) -> np.array:
+def onmf_discretize(onmf_rep_ori: np.array, 
+                    num_init: int = 10,
+                    fig_folder: str = None) -> np.array:
     """
     Discretize the ONMF representation using KMeans clustering and visualize the sorted representations.
 
@@ -313,10 +316,11 @@ def onmf_discretize(onmf_rep_ori: np.array, fig_folder: str = None) -> np.array:
             21, num_spin))
 
     onmf_rep_tri = np.zeros(onmf_rep_ori.shape)
-    for ii in range(num_spin):
+    print('Processing gene or program expression levels...')
+    for ii in tqdm(range(num_spin)):
         # Perform KMeans clustering for each spin/component
-        km_fit = KMeans(n_clusters=3, n_init=10).fit(
-            onmf_rep_ori[:, ii].reshape(-1, 1))
+        km_fit = KMeans(n_clusters=3, n_init=num_init).fit(onmf_rep_ori[:, ii].reshape(-1, 1))
+
         onmf_rep_tri[:, ii] = (km_fit.labels_ == np.argsort(km_fit.cluster_centers_.reshape(-1))[0]) * \
             (-1) + (km_fit.labels_ == np.argsort(km_fit.cluster_centers_.reshape(-1))[2]) * 1
 
@@ -324,8 +328,7 @@ def onmf_discretize(onmf_rep_ori: np.array, fig_folder: str = None) -> np.array:
         if ii < 21:
             ax = plt.subplot(grid[ii])
             plt.plot(np.sort(onmf_rep_ori[:, ii]))
-            plt.plot(np.sort(km_fit.cluster_centers_[
-                     km_fit.labels_].reshape(-1)))
+            plt.plot(np.sort(km_fit.cluster_centers_[km_fit.labels_].reshape(-1)))
 
     # Save the visual representation
     if fig_folder is not None:
@@ -524,7 +527,7 @@ def sample_states(samp_full: np.array, onmf_rep_tri: np.array) -> Tuple[np.array
     return state_list, samp_list
 
 
-def para_moments(j_mat: np.array, h_vec: np.array) -> Tuple[np.array, np.array]:
+def para_moments_numpy(j_mat: np.array, h_vec: np.array) -> Tuple[np.array, np.array]:
     """
     Calculate the correlation and mean parameters given a j matrix and an h vector.
 
@@ -563,51 +566,76 @@ def para_moments(j_mat: np.array, h_vec: np.array) -> Tuple[np.array, np.array]:
 
     return corr_para, mean_para
 
-
-@numba.njit
-def np_apply_along_axis(func1d: Callable[[np.ndarray], float], 
-                        axis: int, 
-                        arr: np.ndarray) -> np.ndarray:
+@numba.njit(fastmath=True, cache=True)
+def para_moments(j_mat: np.ndarray,
+                       h_vec: np.ndarray) -> Tuple[np.ndarray, np.ndarray]:
     """
-    Applies a function along the specified axis.
+    Calculate the correlation and mean parameters given an interaction
+    matrix and an external field vector, using a Numba‑accelerated kernel.
 
     Parameters
     ----------
-    func1d : Callable[[np.ndarray], float]
-        A 1-D function to be applied.
-    axis : int
-        Axis along which the function should be applied.
-    arr : np.ndarray
-        Input array.
+    j_mat : np.ndarray
+        Interaction matrix.
+    h_vec : np.ndarray
+        External field vector.
 
     Returns
     -------
-    np.ndarray
-        The result of applying func1d to arr along the specified axis.
+    Tuple[np.ndarray, np.ndarray]
+        The correlation parameter array of shape (n, n).
+        The mean parameter array of shape (n,).
     """
-    assert arr.ndim == 2
-    assert axis in [0, 1]
-    result = np.empty(arr.shape[1]) if axis == 0 else np.empty(arr.shape[0])
+    num_spin: int = j_mat.shape[0]
+
+    # Copy so we can safely modify the diagonal in‑place
+    j_dbl = j_mat.copy()
+    for ii in range(num_spin):
+        j_dbl[ii, ii] += j_mat[ii, ii]
 
-    for i in range(len(result)):
-        result[i] = func1d(arr[:, i]) if axis == 0 else func1d(arr[i, :])
+    num_sample: int = 3 ** num_spin
 
-    return result
+    partition: float = 0.0
+    mean_para = np.zeros(num_spin, dtype=np.float64)
+    corr_para = np.zeros((num_spin, num_spin), dtype=np.float64)
 
+    spins = np.empty(num_spin, dtype=np.int8)
 
-@numba.njit
-def np_mean(array: np.array, axis: int) -> np.array:
-    """
-    Computes the arithmetic mean along the specified axis.
+    for idx in range(num_sample):
+        # Decode `idx` → base‑3 spin configuration in {‑1, 0, 1}
+        tmp = idx
+        for s in range(num_spin - 1, -1, -1):
+            spins[s] = (tmp % 3) - 1
+            tmp //= 3
 
-    Parameters:
-    array (np.array): Input array.
-    axis (int): Axis along which the mean is computed. 1 is for row-wise, 0 is for column-wise.
-    """
-    return np_apply_along_axis(np.mean, axis, array)
+        # Energy: E = −h·s − ½ sᵀJs
+        energy: float = 0.0
+        for ii in range(num_spin):
+            energy -= h_vec[ii] * spins[ii]
+
+        js_term: float = 0.0
+        for ii in range(num_spin):
+            dot: float = 0.0
+            for jj in range(num_spin):
+                dot += j_dbl[ii, jj] * spins[jj]
+            js_term += spins[ii] * dot
+        energy -= 0.5 * js_term
+
+        weight: float = np.exp(-energy)
+
+        partition += weight
+        for ii in range(num_spin):
+            mean_para[ii] += spins[ii] * weight
+            for jj in range(num_spin):
+                corr_para[ii, jj] += spins[ii] * spins[jj] * weight
 
+    mean_para /= partition
+    corr_para /= partition
 
-@numba.jit()
+    return corr_para, mean_para
+
+
+@numba.njit(fastmath=True)
 def pseudol_gradient(cur_j: np.ndarray, 
                      cur_h: np.ndarray, 
                      cur_state: np.ndarray, 
@@ -632,37 +660,43 @@ def pseudol_gradient(cur_j: np.ndarray,
         The gradient of j.
         The gradient of h.
     """
-    num_spin = cur_j.shape[0]
+    num_spin, num_sample = cur_state.shape
+    sample_factor = 1.0 / num_sample     
 
-    cur_j_grad = np.zeros((num_spin, num_spin))
-    cur_h_grad = np.zeros((num_spin, 1))
-
-    # Filtering diagonal elements of cur_j matrix.
-    j_filt = cur_j.copy()
-    np.fill_diagonal(j_filt, 0)
-    effective_h = j_filt.dot(cur_state) + cur_h
+    diag_j = np.empty(num_spin, dtype=cur_j.dtype)
+    for ii in range(num_spin):
+        diag_j[ii] = cur_j[ii, ii]
 
+    j_no_diag = cur_j.copy()
     for ii in range(num_spin):
-        # Calculating gradients for each spin.
-        j_sub = cur_j[ii, ii]
-        h_sub = effective_h[ii, :]
+        j_no_diag[ii, ii] = 0.0
 
-        term1 = np.exp(j_sub + h_sub)
-        term2 = np.exp(j_sub - h_sub)
+    eff_h = j_no_diag @ cur_state + cur_h 
 
-        # Compute gradients.
-        j_sub_grad = cur_state[ii, :] ** 2 - \
-            (term1 + term2) / (term1 + term2 + 1)
-        h_eff_grad = cur_state[ii, :] - (term1 - term2) / (term1 + term2 + 1)
+    exp_plus = np.empty_like(eff_h)
+    exp_minus = np.empty_like(eff_h)
+    denom = np.empty_like(eff_h)
 
-        j_off_sub_grad = h_eff_grad * cur_state
+    for ii in range(num_spin):
+        exp_plus[ii, :] = np.exp(diag_j[ii] + eff_h[ii, :])
+        exp_minus[ii, :] = np.exp(diag_j[ii] - eff_h[ii, :])
+        denom[ii, :] = exp_plus[ii, :] + exp_minus[ii, :] + 1.0
+
+    # Diagonal (self‑coupling) gradients 
+    term_diag = cur_state * cur_state - (exp_plus + exp_minus) / denom
+
+    # Off‑diagonal and field gradients 
+    h_eff_grad = cur_state - (exp_plus - exp_minus) / denom
+    cur_j_grad = (h_eff_grad @ cur_state.T) * sample_factor
+    for ii in range(num_spin):
+        cur_j_grad[ii, ii] = np.sum(term_diag[ii, :]) * sample_factor
 
-        cur_j_grad[ii, :] = np_mean(j_off_sub_grad, axis=1)
-        cur_j_grad[ii, ii] = np.mean(j_sub_grad)
-        cur_h_grad[ii] = np.mean(h_eff_grad)
+    cur_h_grad = np.zeros((num_spin, 1), dtype=cur_j.dtype)
+    for ii in range(num_spin):
+        cur_h_grad[ii, 0] = np.sum(h_eff_grad[ii, :]) * sample_factor
 
-        if not directed:
-            cur_j_grad = (cur_j_grad + cur_j_grad.T) / 2
+    if not directed:
+        cur_j_grad = 0.5 * (cur_j_grad + cur_j_grad.T)
 
     return - cur_j_grad, - cur_h_grad
 
@@ -762,10 +796,10 @@ def samp_moments(j_mat: np.ndarray,
     return corr_para, mean_para
 
 
-def compute_gradient(cur_j: np.ndarray, 
-                     cur_h: np.ndarray, 
-                     raw_data: List[Tuple[np.ndarray, np.ndarray]], 
-                     method: str, train_dat: Dict[str, Any]) -> Tuple[np.ndarray, np.ndarray]:
+def compute_gradient(cur_j: np.ndarray,
+                            cur_h: np.ndarray,
+                            raw_data: List[Tuple[np.ndarray, np.ndarray]],
+                            method: str, train_dat: Dict[str, Any]) -> Tuple[np.ndarray, np.ndarray]:
     """
     Compute the gradient based on the specified method.
 
@@ -798,7 +832,7 @@ def compute_gradient(cur_j: np.ndarray,
         # Computing gradients using the specified method.
         if method == 'pseudo_likelihood':
             j_grad, h_grad = pseudol_gradient(
-                cur_j, cur_h[:, kk].reshape(- 1, 1), raw_data[kk], directed=train_dat['directed'])
+                cur_j, cur_h[:, kk: kk + 1], raw_data[kk], directed=train_dat['directed'])
             h_grad = h_grad.flatten()
         else:
             # Distinguishing between other methods and computing gradients
@@ -807,7 +841,7 @@ def compute_gradient(cur_j: np.ndarray,
                 corr_para, mean_para = para_moments(cur_j, cur_h[:, kk])
             elif method == 'mcmc_maximum_likelihood':
                 corr_para, mean_para = samp_moments(
-                    cur_j, cur_h, train_dat['mcmc_samplingsz'], train_dat['mcmc_samplingmix'], train_dat['mcmc_samplegap'])
+                    cur_j, cur_h[:, kk], train_dat['mcmc_samplingsz'], train_dat['mcmc_samplingmix'], train_dat['mcmc_samplegap'])
             j_grad = corr_para - raw_data[kk][0]
             h_grad = mean_para - raw_data[kk][1].flatten()
 
@@ -817,6 +851,77 @@ def compute_gradient(cur_j: np.ndarray,
     return rec_jgrad, rec_hgrad
 
 
+def _gradient_one_round(kk: int,
+                        cur_j: np.ndarray,
+                        cur_h: np.ndarray,
+                        raw_data_k,
+                        method: str,
+                        train_dat: Dict[str, Any]):
+    """Helper that compute gradient for a single round."""
+    if method == 'pseudo_likelihood':
+        j_grad, h_grad = pseudol_gradient(cur_j, cur_h[:, kk:kk + 1], raw_data_k, directed=train_dat['directed'])
+        h_grad = h_grad.flatten()
+    else:
+        if method == 'maximum_likelihood':
+            corr_para, mean_para = para_moments(cur_j, cur_h[:, kk])
+        elif method == 'mcmc_maximum_likelihood':
+            corr_para, mean_para = samp_moments(
+                    cur_j, cur_h[:, kk], train_dat['mcmc_samplingsz'], train_dat['mcmc_samplingmix'], train_dat['mcmc_samplegap'])
+        j_grad  = corr_para - raw_data_k[0]
+        h_grad  = mean_para  - raw_data_k[1].flatten()
+
+    return kk, j_grad, h_grad
+
+
+def compute_gradient_para(cur_j: np.ndarray,
+                     cur_h: np.ndarray,
+                     raw_data: List[Tuple[np.ndarray, np.ndarray]],
+                     method: str, train_dat: Dict[str, Any]) -> Tuple[np.ndarray, np.ndarray]:
+    """
+    Compute the gradient based on the specified method with parallel processing.
+
+    Parameters
+    ----------
+    cur_j : np.ndarray
+        Current j matrix.
+    cur_h : np.ndarray
+        Current h matrix.
+    raw_data : List[Tuple[np.ndarray, np.ndarray]]
+        Raw data used to calculate the gradient (each element is a tuple of correlation and mean).
+    method : str
+        The method to calculate the gradient; possible values are 'pseudo_likelihood',
+        'maximum_likelihood', and 'mcmc_maximum_likelihood'.
+    train_dat : Dict[str, Any]
+        Training data and hyperparameters.
+
+    Returns
+    -------
+    Tuple[np.array, np.array]
+        The gradient of j.
+        The gradient of h.
+    """
+    num_spin, num_round = cur_h.shape
+    rec_jgrad = np.zeros((num_spin, num_spin, num_round))
+    rec_hgrad = np.zeros((num_spin, num_round))
+
+    n_jobs = train_dat.get('max_workers', min(num_round, os.cpu_count())) 
+    chunk = train_dat.get('chunk_size', 1)          # rounds per task
+    backend = 'loky'                                # process‑based
+
+    # print(f'Processing with {n_jobs} workers...')
+    with parallel_backend(backend, n_jobs=n_jobs):
+        results = Parallel(batch_size=chunk)(
+            delayed(_gradient_one_round)(kk, cur_j, cur_h, raw_data[kk], method, train_dat)
+            for kk in range(num_round)
+        )
+
+    for kk, j_grad, h_grad in results:
+        rec_jgrad[:, :, kk] = j_grad
+        rec_hgrad[:, kk] = h_grad
+
+    return rec_jgrad, rec_hgrad
+
+
 def apply_regularization(rec_jgrad: np.ndarray, 
                          rec_hgrad: np.ndarray, 
                          cur_j: np.ndarray, 
@@ -999,6 +1104,8 @@ def learn_network_adam(raw_data: Any,
 
     backtrack_counter = 0
     counter = 1
+    
+    tqdm._instances.clear()
     pbar = tqdm(total=num_epoch)
 
     while counter <= num_epoch:
@@ -1029,13 +1136,13 @@ def learn_network_adam(raw_data: Any,
 
         # Save training log and print progress
         if counter in list_step:
-            sio.savemat(save_path + 'train_log.mat',
-                        {'list_step': list_step,
-                         'rec_hvec_all': rec_hvec_all,
-                         'rec_jmat_all': rec_jmat_all,
-                         'rec_jgrad_sum_norm': rec_jgrad_sum_norm})
-            # print('Progress: %d, Network gradient: %f' % (
-            #     np.round(100 * counter / num_epoch, 2), rec_jgrad_sum_norm[counter - 1]))
+            if train_dat['save_log']:
+                sio.savemat(save_path + 'train_log.mat',
+                            {'list_step': list_step,
+                            'rec_hvec_all': rec_hvec_all,
+                            'rec_jmat_all': rec_jmat_all,
+                            'rec_jgrad_sum_norm': rec_jgrad_sum_norm})
+
             pbar.update(rec_gap)
             pbar.set_postfix({"Network Gradient": f"{rec_jgrad_sum_norm[counter - 1]:.6f}"})
 

{dspin-1.3.1 → dspin-1.3.3}/dspin/dspin.py

@@ -34,6 +34,7 @@ import matplotlib.patheffects as patheffects
 import warnings
 import itertools
 from typing import List, Dict, Optional, Tuple, Union
+from scipy.stats import skew
 
 
 class AbstractDSPIN(ABC):
@@ -130,7 +131,9 @@ class AbstractDSPIN(ABC):
     def sample_list(self, value):
         self._samp_list = value
 
-    def discretize(self, clip_percentile: float = 100) -> None:
+    def discretize(self, 
+                   clip_percentile: float = 100,
+                   num_init: int = 10) -> None:
         """
         Discretizes the oNMF representation into three states (-1, 0, 1) using K-means clustering.
 
@@ -138,6 +141,8 @@ class AbstractDSPIN(ABC):
         ----------
         clip_percentile : float, optional
             Percentile at which to clip values before discretization (default is 100).
+        num_init : int, optional
+            Number of initializations for K-means clustering (default is 10).
         """
         onmf_rep_ori = self.program_representation_raw.copy()
         fig_folder = self.fig_folder
@@ -147,7 +152,7 @@ class AbstractDSPIN(ABC):
                 onmf_rep_ori[:, ii] = onmf_rep_ori[:, ii] / np.percentile(onmf_rep_ori[:, ii], clip_percentile)
             onmf_rep_ori = onmf_rep_ori.clip(0, 1)
 
-        onmf_rep_tri = onmf_discretize(onmf_rep_ori, fig_folder)
+        onmf_rep_tri = onmf_discretize(onmf_rep_ori, num_init, fig_folder)
         self._onmf_rep_tri = onmf_rep_tri
 
     def raw_data_corr(self, sample_id_key: str) -> None:
@@ -209,7 +214,8 @@ class AbstractDSPIN(ABC):
                   'cur_h': np.zeros((num_spin, num_sample)),
                   'save_path': self.save_path, 
                   'rec_gap': 10, 
-                  'seed': 0}
+                  'seed': 0,
+                  'save_log': False}
         params.update({'lambda_l1_j': 0.01,
                        'lambda_l1_h': 0,
                        'lambda_l2_j': 0,
@@ -332,8 +338,8 @@ class AbstractDSPIN(ABC):
             raise ValueError(f"Column '{batch_key}' not found in adata.obs.")
         
         sample_list = self._samp_list
-        sample_to_control = self.adata.obs.groupby(sample_id_key)[if_control_key].first().to_dict()
-        sample_to_batch = self.adata.obs.groupby(sample_id_key)[batch_key].first().to_dict()
+        sample_to_control = self.adata.obs.groupby(sample_id_key, observed=False)[if_control_key].first().to_dict()
+        sample_to_batch = self.adata.obs.groupby(sample_id_key, observed=False)[batch_key].first().to_dict()
 
         if_control = np.array([sample_to_control[sample] for sample in sample_list])
         batch_index = np.array([sample_to_batch[sample] for sample in sample_list])
@@ -351,7 +357,7 @@ class GeneDSPIN(AbstractDSPIN):
                  adata: ad.AnnData,
                  save_path: str,
                  num_spin: int,
-                 clip_percentile: float = 95):
+                 discretize_params: dict = {}):
         """
         Initialize the GeneDSPIN object.
 
@@ -363,6 +369,8 @@ class GeneDSPIN(AbstractDSPIN):
             Directory path where results will be saved.
         num_spin : int
             Number of spins used in inference.
+        use_default_discretize : bool, optional
+            Whether to use default discretization. Default is True. Otherwise the discretized data should be explicited setted in program_representation.
         clip_percentile : float, optional
             Percentile threshold for clipping values before discretization. Default is 95.
         """
@@ -375,7 +383,26 @@ class GeneDSPIN(AbstractDSPIN):
         else:
             self._onmf_rep_ori = adata.X
 
-        self.discretize(clip_percentile)
+        gene_skew = np.zeros(num_spin)
+
+        for ii in range(num_spin):
+            cur_gene = self._onmf_rep_ori[:, ii]
+            gene_skew[ii] = skew(cur_gene, bias=False)
+
+        if gene_skew.mean() > 2:
+            default_clip = 95
+            print("Input gene expression seem to be heavy-tailed. Using 95 percentile for clipping before discretization.")
+        else:
+            default_clip = 100
+
+        dparams = {'use_default_discretize': True,
+                   'clip_percentile': default_clip,
+                   'num_init': 10}
+        dparams.update(discretize_params)
+        
+        if dparams['use_default_discretize']:
+            self.discretize(clip_percentile=dparams['clip_percentile'], 
+                            num_init=dparams['num_init'])
 
     def program_regulator_discovery(self, 
                                     program_representation_raw: np.ndarray, 
@@ -440,10 +467,7 @@ class ProgramDSPIN(AbstractDSPIN):
     def __init__(self,
                  adata: ad.AnnData,
                  save_path: str,
-                 num_spin: int,
-                 num_onmf_components: int = None,
-                 prior_programs: List[List[str]] = None,
-                 num_repeat: int = 10):
+                 num_spin: int):
         """
         Initialize the ProgramDSPIN object.
 
@@ -455,12 +479,6 @@ class ProgramDSPIN(AbstractDSPIN):
             Directory path for saving results.
         num_spin : int
             Number of spins used in DSPIN.
-        num_onmf_components : int, optional
-            Number of oNMF components, default is None.
-        prior_programs : List[List[str]], optional
-            List of predefined gene programs.
-        num_repeat : int, optional
-            Number of repetitions for oNMF computation. Default is 10.
         """
         super().__init__(adata, save_path, num_spin)
 
@@ -470,8 +488,6 @@ class ProgramDSPIN(AbstractDSPIN):
         self._gene_matrix_large = None
         self._use_data_list = None
         self.gene_program_csv = None
-        self.prior_programs = None
-        self.preprogram_num = len(prior_programs) if prior_programs else 0
 
     @property
     def onmf_decomposition(self):
@@ -546,32 +562,27 @@ class ProgramDSPIN(AbstractDSPIN):
                                cluster_key: str = 'leiden',
                                mode: str = 'compute_summary',
                                prior_programs: List[List[str]] = None,
-                               params: dict = {}) -> None:
+                               params: dict = {},
+                               discretize_params: dict = {}) -> None:
         """
         Discover gene programs by performing oNMF decomposition on the annotated data.
 
         Parameters
         ----------
-        num_onmf_components : int, optional
-            Number of oNMF components. Default is None.
-        num_subsample : int, optional
-            Number of samples to use after subsampling. Default is 10000.
-        num_subsample_large : int, optional
-            Number of samples for large subsampling. Default is None.
         num_repeat : int, optional
             Number of times to repeat oNMF. Default is 10.
         seed : int, optional
             Random seed for reproducibility. Default is 0.
-        balance_obs : str, optional
-            Observation category to balance. Default is None.
-        balance_method : str, optional
-            Method used for balancing categories. Default is None.
-        max_sample_rate : float, optional
-            Maximum sampling rate. Default is 2.
+        cluster_key : str, optional
+            Key in `adata.obs` for clustering labels. Default is 'leiden'.
+        mode: str, optional
+            Mode of operation. Options: 'compute_summary', 'compute_only', 'summary_only'. Default is 'compute_summary'.
         prior_programs : List[List[str]], optional
             List of predefined gene programs. Default is None.
-        summary_method : str, optional
-            Method to summarize oNMF components. Default is 'kmeans'.
+        params : dict, optional
+            Additional parameters for oNMF. Default is an empty dictionary.
+        discretize_params : dict, optional
+            Parameters for discretization of the oNMF representation. Default is an empty dictionary.
         """
 
         if mode not in ['compute_summary', 'compute_only', 'summary_only']:
@@ -662,7 +673,12 @@ class ProgramDSPIN(AbstractDSPIN):
             self._onmf_rep_ori = onmf_summary.transform(
                 gene_matrix / self.matrix_std)
 
-            self.discretize()
+            dparams = {'clip_percentile': 100,
+                       'num_init': 10}
+            dparams.update(discretize_params)
+
+            self.discretize(clip_percentile=dparams['clip_percentile'],
+                            num_init=dparams['num_init'])
 
 
 class DSPIN(object):

{dspin-1.3.1 → dspin-1.3.3/dspin.egg-info}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: dspin
-Version: 1.3.1
+Version: 1.3.3
 Summary: Regulatory network models from single-cell perturbation profiling
 Home-page: https://github.com/JialongJiang/DSPIN
 Author: Jialong Jiang

{dspin-1.3.1 → dspin-1.3.3}/setup.py

@@ -21,7 +21,7 @@ install_requires = [
 # ]
 
 # Managing your version (adjust accordingly)
-version = "1.3.1"  # You can manage versioning in a more sophisticated way if needed
+version = "1.3.3"  # You can manage versioning in a more sophisticated way if needed
 
 # Long description (usually a README file)
 readme = open("README_PyPI.md").read()  # Adjust the file name if necessary

dspin-1.3.1/README.md

@@ -1,134 +0,0 @@
-# DSPIN
-
-Tutorial, code and examples of the D-SPIN framework for the preprint "D-SPIN constructs gene regulatory network models from multiplexed scRNA-seq data revealing organizing principles of cellular perturbation response" ([bioRxiv](https://www.biorxiv.org/content/10.1101/2023.04.19.537364))
-
-![alternativetext](figure/readme/Figure1_20230309_Inna.png)
-
-## Installation
-
-D-SPIN is implemented in Python and Matlab. The Python code is sufficient for analysis of small datasets with around 20k cells and tens of conditions. The Matlab code is only used for network inference on large datasets as Matlab can be easily deployed on clusters for parallelization with the build-in "parfor" function. 
-
-The python code can be installed with 
-
-`pip install dspin`
-
-The Matlab code can be downloaded from the folder "DSPIN_matlab", and directly executable in Matlab after specifying the path to the data. 
-
-## Demos
-
-Two demos of D-SPIN are available on Google Colab. 
-
-The first demo reconstructs the regulatory network of simulated hematopoietic stem cell (HSC) differentiation network with perturbations using the BEELINE framework (Pratapa, Aditya, et al. Nature methods, 2020). 
-
-[Demo1](https://colab.research.google.com/drive/1mjJ09_E9kjEYsGCzf10-aZfuOqHUEV5u?usp=sharing)
-
-The second demo reconstructs regulatory network and response vector in a single-cell dataset collected in the ThomsonLab.In the dataset, human peripheral blood mononuclear cells (PBMCs) were treated with various signaling molecules with different dosages. 
-
-[Demo2](https://colab.research.google.com/drive/1U2WbmKvai3Y_-vlLV0SY1dv1LHdFjg2w?usp=sharing)
-
-## Dependencies
-
-DSPIN package was tested with the following python packages versions:
-
-- python (3.9.18)
-- anndata (0.10.3)
-- matplotlib (3.8.2)
-- scanpy (1.9.6)
-- tqdm (4.65.0)
-- leidenalg (0.10.1)
-- igraph (0.10.8)
-
-Note: other versions may work as well.
-
-## D-SPIN Overview
-
-D-SPIN contains three main steps: 
-* Gene program discovery
-* Network inference
-* Network analysis
-
-The input data should be AnnData object after typical single-cell preprocessing steps including cell counts normalization, log transformation, highly-varible gene selection, and clustering (optional). The input data should contain the following information. The attribute name is subject to the user while the following are defaults of the code:
-* Normalized, log-transformed and filtered gene expression matrix in adata.X
-* Sample information for each cell in adata.obs['sample_id']
-* Batch information for each cell in adata.obs['batch']
-* Clustering information for each cell in adata.obs['leiden']
-* If some samples are control conditions, the relative response vectors will be computed for each batch with respect to the average of control conditions. The control conditions can be indicated in adata.obs['if_control'] with 1 for control and 0 for non-control. Note that all cells in the same sample and batch should have the same value in adata.obs['if_control'].
-
-The D-SPIN is initiallized by model = dspin.DSPIN(adata) with the following major arguments:
-
-* adata: AnnData object with the above information
-* save_path: path to save the results
-* num_spin (default 15): number of gene programs in the model
-* filter_threshold (default 0.02): threshold for minimal proportion of cells that each gene program is expressed in
-
-### Gene program discovery
-
-By default, D-SPIN use orgthogonal non-negative matrix factorization (oNMF) to discover gene programs that coexpress in the data. The user can also specify pre-defined gene programs in full or partial. There are three senarios depending on the num_spin and the input parameter prior_programs.
-* No pre-defined gene programs: empty prior_programs (default)
-* Full pre-defined gene programs: the number of gene programs in prior_programs is equal to num_spin
-* Partial pre-defined gene programs: the number of gene programs in prior_programs is less than num_spin
-
-The gene program discovery contains two stages, the first stage subset and normalize the gene matrix, and the second stage run oNMF multiple times on the subset matrix to obtain consensus gene programs. Subsetting the gene matrix is primarily for computational efficiency, and preferrentially sampling cells with smaller population also helps to avoid overfitting on over-represented cell states in the dataset. As the algorithm for oNMF is stochastic, the algorithm is run multiples times and results from multiple run are combined to obtain a set of consensus gene program. Also, the number of oNMF components can be specified by the user to discover gene programs with finer resolution. For example, run the algorithm with 20 components in each individual run but combine the 20 programs into 15 consensus programs. 
-
-Overall, in the gene program discovery function model.gene_program_discovery() , D-SPIN takes the following major arguments: 
-
-* num_onmf_components (int, optional): The number of oNMF components. Default is infer based on num_spin and pre-assigned programs.
-* num_subsample (int, optional): Number of cells to use in each individual oNMF components computation. Default is 10000.
-* num_subsample_large (int, optional): Number of cells to use in computing consensus oNMF components. Default is 5 times of num_subsample.
-* num_repeat (int, optional): Number of times to repeat the oNMF computation. Default is 10.
-* balance_obs (str, optional): Cell type or clustering label in adata.obs for balanced sampling. Default is None.
-* balance_method (str, optional): Method used for balancing. Options are 'squareroot', 'equal', 'proportional'. Default is 'squareroot'. 
-* max_sample_rate (float, optional): Maximum oversampling rate for underrepresented cell type or clustering categories. Default is 2.
-* prior_programs (List[List[str]], optional): List of pre-assigned gene programs. Default is None.
-* summary_method (str, optional): Method used for computing sensus oNMF components. Options are 'kmeans' and 'leiden'. Default is 'kmeans'.
-
-### Network inference
-
-In the network inference step, D-SPIN automatically choose between three inference methods depending on the number of gene programs:
-
-* Exact maximum-likelihood inference (MLE)
-* Markov-Chain Monte-Carlo (MCMC) maximum-likelihood inference 
-* Pseudo-likelihood inference (PL). 
-
-MLE is exactly solution of the problem but only applies to small network. MCMC is slightly more scalable to networks with 20-40 nodes. PL takes more simplified approximation to the problem which scales to hundreds of nodes, but requires larger cell number for the approximation to be effective. in the gene program discovery function model.network_inference(), D-SPIN takes the following major arguments
-
-* sample_col_name (str, optional): Column name in adata.obs for sample information. Default is 'sample_id'
-* control_col_name (str, optional): Column name in adata.obs for whether the cell is in a control sample. If not provided, all cells are treated as non-control and relative response vector are the same as response vector.
-* batch_col_name (str, optional): Column name in adata.obs for batch information. Relative response are computed by the control response vector in the same batch. If not provided, all cells are treated as one batch. 
-* method (str, optional): Method used for network inference. Options are 'auto', 'maximum_likelihood', 'mcmc_maximum_likelihood', 'pseudo_likelihood'.
-* example_list (List[str], optional): List of samples to use for network inference. Default is using all samples available in adata.obs[sample_col_name].
-* record_step (int, optional): Number of steps to record training progress. Default is 10.
-* params (Dict[str, Any], optional): Hypterparameters for the gradient descent of network inference. Default is None.
-
-
-### Network analysis 
-
-After network inference, D-SPIN provides three outputs:
-
-* Inferred network adjacenty matrix in model.network 
-* Inferred pertrubation response in model.response
-* Relative perturbation response to control in model.relative_response
-
-Typical downstream analysis includes network visualization, module identification, and graph clustering of perturbation response with the network. 
-
-## Application to signaling response data of human PBMCs
-
-D-SPIN takes single-cell sequencing data of multiple perturbation conditions. In the second demo, PBMCs are treated with different signaling molecules such as CD3 antibody, LPS, IL1B, and TGFB1
-
-![alternativetext](figure/thomsonlab_signaling/example_conditions.png)
-
-D-SPIN identifies a set of gene programs that coexpress in the data, and represent each cell as a combination of gene program expression states. 
-
-![alternativetext](figure/thomsonlab_signaling/gene_program_example.png)
-
-D-SPIN uses cross-correlation and mean of each perturbation condition to inferred a unified regulatory network and the response vector of each perturbation condition. The inference can be parallelized across perturbation conditions. The inference code is in Matlab using "parfor", while for demo purpose Python code (without parallelization) is provided.
-
-The inferred regulatory network and perturbations can be jointly analyzed to reveal how perturbations act in the context of the regulatory network.
-
-![alternativetext](figure/thomsonlab_signaling/joint_network_perturbation.png)
-
-## References
-
-1. Jiang, Jialong, et al. "D-SPIN constructs gene regulatory network models from multiplexed scRNA-seq data revealing organizing principles of cellular perturbation response." bioRxiv (2023).
-
-2. Pratapa, Aditya, et al. "Benchmarking algorithms for gene regulatory network inference from single-cell transcriptomic data." Nature methods 17.2 (2020): 147-154.