citegeist 0.1.0__tar.gz

citegeist-0.1.0/LICENSE

@@ -0,0 +1,28 @@
+BSD 3-Clause License
+
+Copyright (c) 2024, Alexander Chang and Brent T. Schlegel
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are met:
+
+1. Redistributions of source code must retain the above copyright notice, this
+   list of conditions and the following disclaimer.
+
+2. Redistributions in binary form must reproduce the above copyright notice,
+   this list of conditions and the following disclaimer in the documentation
+   and/or other materials provided with the distribution.
+
+3. Neither the name of the copyright holder nor the names of its
+   contributors may be used to endorse or promote products derived from
+   this software without specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
+FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
+SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
+CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 

citegeist-0.1.0/PKG-INFO

@@ -0,0 +1,199 @@
+Metadata-Version: 2.2
+Name: citegeist
+Version: 0.1.0
+Summary: A spatial transcriptomics deconvolution tool for cell type identification and gene expression analysis
+Home-page: https://github.com/acc383/CITEgeist
+Author: Alexander Chang, Brent T. Schlegel
+Author-email: acc383@cornell.edu
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: OSI Approved :: BSD License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.8
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Topic :: Scientific/Engineering :: Bio-Informatics
+Requires-Python: >=3.8
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: numpy>=1.20.0
+Requires-Dist: pandas>=1.3.0
+Requires-Dist: scanpy>=1.9.0
+Requires-Dist: scipy>=1.7.0
+Requires-Dist: pyarrow>=5.0.0
+Requires-Dist: gurobipy>=9.5.0
+Requires-Dist: squidpy>=1.6.2
+Provides-Extra: dev
+Requires-Dist: pytest>=7.0.0; extra == "dev"
+Requires-Dist: pytest-cov>=3.0.0; extra == "dev"
+Requires-Dist: black>=22.0.0; extra == "dev"
+Requires-Dist: isort>=5.10.0; extra == "dev"
+Dynamic: author
+Dynamic: author-email
+Dynamic: classifier
+Dynamic: description
+Dynamic: description-content-type
+Dynamic: home-page
+Dynamic: provides-extra
+Dynamic: requires-dist
+Dynamic: requires-python
+Dynamic: summary
+
+# CITEgeist: Cellular Indexing of Transcriptomes and Epitopes for Guided Exploration of Intrinsic Spatial Trends
+
+CITEgeist is a computational method for deconvolving spatial transcriptomics data using spatially-resolved CITE-seq measurements. The pipeline performs both cell-type proportion estimation and gene expression deconvolution in a two-pass approach, leveraging both protein and RNA measurements from the same spatial locations.
+
+## Quick Installation
+
+You can now also install CITEgeist using pip:
+
+```bash
+pip install citegeist
+```
+
+## Table of Contents
+1. [System Requirements](#system-requirements)
+    - [Software Dependencies](#software-dependencies)
+    - [Hardware Requirements](#hardware-requirements)
+2. [Getting Started](#getting-started)
+    - [1. Download the Code and Data](#1-download-the-code-and-data)
+    - [2. Set Up the Environment](#2-set-up-the-environment)
+    - [3. Obtain Gurobi License](#3-obtain-gurobi-license)
+    - [4. Running CITEgeist](#4-running-citegeist)
+3. [Benchmarking and Reproducibility](#benchmarking-and-reproducibility)
+4. [Run the Analysis](#run-the-analysis)
+
+## System Requirements
+
+### Software Dependencies
+
+- **Operating System**:  
+  - Linux  
+  - macOS  
+  - Windows 10 with WSL2  
+
+- **Python**: 3.10
+- **Gurobi** [version > 3.9](https://www.gurobi.com/downloads/gurobi-software/)
+#### Key Python Dependencies
+- scanpy==1.10.4
+- anndata==0.11.3
+- numpy==1.26.4
+- pandas==2.2.3
+- scipy==1.13.1
+- scikit-learn==1.6.1
+- gurobipy==11.0.2 (requires license)
+- matplotlib==3.10.0
+- seaborn==0.13.2
+- h5py==3.12.1
+- squidpy==1.6.2
+- spatialdata==0.2.5.post0
+
+It is recommended to install the dependencies in the `CITEgeist_env.yml` file for running the notebooks.
+
+### Hardware Requirements
+- **RAM**: Minimum 16GB, Recommended 64GB+
+- **Storage**: 16GB minimum for installation and basic analysis
+- **CPU**: Multi-core processor recommended (8+ cores for optimal performance)
+
+---
+
+## Getting Started
+
+### 1. Download the Code and Data (Instructions for Peer Reviewers)
+
+1. Download the code from Figshare: [https://figshare.com/s/34e456fd7786e5211acc](https://figshare.com/s/34e456fd7786e5211acc)
+2. Unzip the downloaded file to your preferred location.
+3. Download the data from GEO (Reviewers: see Data Availability section in the manuscript for the GEO link and Private Access Token).
+4. Run the following code to strip unique identifiers required by GEO:
+
+```bash
+# go to the 'data' directory
+cd data
+
+# untar the raw files
+mkdir -pv ./GEO_data
+tar -xvf GEO_data_RAW.tar -C ./GEO_data
+
+# run the py preprocessing script
+## round 1) aggregate the files by sample
+python3 ./delete_all_but_essential.py --folder GEO_data # select option: 1
+
+## round 2) remove the prefix from necessary files
+python3 ./delete_all_but_essential.py --folder GEO_data # select option: 2
+```
+
+Note: When prompted, select Option 1 or 2 and type 'Yes' to confirm.
+
+### 2. Set Up the Environment
+
+- Install dependencies using the provided environment file:
+
+```bash
+conda env create -f CITEgeist_env.yml
+```
+
+- Activate the environment and set up a Jupyter kernel:
+
+```bash
+conda activate CITEgeist_env
+```
+
+### 3. Obtain Gurobi License
+
+CITEgeist requires a Gurobi license (free for academic use):
+
+1. Sign up for an academic license at: [https://www.gurobi.com/downloads/end-user-license-agreement-academic/](https://www.gurobi.com/downloads/end-user-license-agreement-academic/)
+2. Follow the instructions to download and install your license.
+3. Update the license file path in the notebooks to match your local license location.
+
+### 4. Running CITEgeist
+
+You can run CITEgeist in two ways:
+
+#### A. Using Jupyter Notebooks
+- Update data paths in the top of the notebooks to match your local directory structure.
+- Expected runtime on a standard computer (16 threads, 32GB RAM):
+  - Vignette 1: ~2 hours
+  - Vignette 2: ~2 hours
+  - Vignette 3: ~10 hours
+
+#### Key Parameters:
+- `radius`: Radius for neighbor detection (default: 4)
+- `lambda_reg`: Regularization strength for cell proportion estimation (default: 0.001)
+- `alpha_elastic`: Elastic net mixing parameter for cell proportion estimation (default: 0.7)
+- `max_y_change`: Maximum allowed change in Y values (default: 0.2)
+
+#### Optional Parameters:
+- `profiling_only`: Set for cell-type proportions only.
+- `max_workers`: Number of parallel workers.
+- `checkpoint_interval`: Checkpoint saving interval.
+
+#### B. Using SLURM Distribution
+For large-scale analyses, you can use the provided `CITEgeist/examples/sbatch_sample.sh` script for distributed computing.
+
+---
+
+## Benchmarking and Reproducibility
+
+For specific reproduction of benchmarking tests and detailed methodology, please refer to the 'examples' and 'benchmarking' section in the documentation.
+
+---
+
+## Run the Analysis
+
+You can either:
+
+#### A. Run the notebooks directly:
+- Update data paths in the notebooks to match your local directory structure.
+- Expected runtime: ~2 hours on a standard computer (16 threads, 32GB RAM).
+
+#### B. Use SLURM distribution:
+- Use the provided `examples/sbatch_sample.sh` script for distributed computing.
+
+---
+
+## Additional System Requirements
+- **RAM**: 32GB (minimum)
+- **CPU**: 16 threads (recommended)
+- **Storage**: Sufficient space for the GEO dataset
+- **Operating System**: Linux/Unix recommended (Windows users may need additional configuration)

citegeist-0.1.0/README.md

@@ -0,0 +1,158 @@
+# CITEgeist: Cellular Indexing of Transcriptomes and Epitopes for Guided Exploration of Intrinsic Spatial Trends
+
+CITEgeist is a computational method for deconvolving spatial transcriptomics data using spatially-resolved CITE-seq measurements. The pipeline performs both cell-type proportion estimation and gene expression deconvolution in a two-pass approach, leveraging both protein and RNA measurements from the same spatial locations.
+
+## Quick Installation
+
+You can now also install CITEgeist using pip:
+
+```bash
+pip install citegeist
+```
+
+## Table of Contents
+1. [System Requirements](#system-requirements)
+    - [Software Dependencies](#software-dependencies)
+    - [Hardware Requirements](#hardware-requirements)
+2. [Getting Started](#getting-started)
+    - [1. Download the Code and Data](#1-download-the-code-and-data)
+    - [2. Set Up the Environment](#2-set-up-the-environment)
+    - [3. Obtain Gurobi License](#3-obtain-gurobi-license)
+    - [4. Running CITEgeist](#4-running-citegeist)
+3. [Benchmarking and Reproducibility](#benchmarking-and-reproducibility)
+4. [Run the Analysis](#run-the-analysis)
+
+## System Requirements
+
+### Software Dependencies
+
+- **Operating System**:  
+  - Linux  
+  - macOS  
+  - Windows 10 with WSL2  
+
+- **Python**: 3.10
+- **Gurobi** [version > 3.9](https://www.gurobi.com/downloads/gurobi-software/)
+#### Key Python Dependencies
+- scanpy==1.10.4
+- anndata==0.11.3
+- numpy==1.26.4
+- pandas==2.2.3
+- scipy==1.13.1
+- scikit-learn==1.6.1
+- gurobipy==11.0.2 (requires license)
+- matplotlib==3.10.0
+- seaborn==0.13.2
+- h5py==3.12.1
+- squidpy==1.6.2
+- spatialdata==0.2.5.post0
+
+It is recommended to install the dependencies in the `CITEgeist_env.yml` file for running the notebooks.
+
+### Hardware Requirements
+- **RAM**: Minimum 16GB, Recommended 64GB+
+- **Storage**: 16GB minimum for installation and basic analysis
+- **CPU**: Multi-core processor recommended (8+ cores for optimal performance)
+
+---
+
+## Getting Started
+
+### 1. Download the Code and Data (Instructions for Peer Reviewers)
+
+1. Download the code from Figshare: [https://figshare.com/s/34e456fd7786e5211acc](https://figshare.com/s/34e456fd7786e5211acc)
+2. Unzip the downloaded file to your preferred location.
+3. Download the data from GEO (Reviewers: see Data Availability section in the manuscript for the GEO link and Private Access Token).
+4. Run the following code to strip unique identifiers required by GEO:
+
+```bash
+# go to the 'data' directory
+cd data
+
+# untar the raw files
+mkdir -pv ./GEO_data
+tar -xvf GEO_data_RAW.tar -C ./GEO_data
+
+# run the py preprocessing script
+## round 1) aggregate the files by sample
+python3 ./delete_all_but_essential.py --folder GEO_data # select option: 1
+
+## round 2) remove the prefix from necessary files
+python3 ./delete_all_but_essential.py --folder GEO_data # select option: 2
+```
+
+Note: When prompted, select Option 1 or 2 and type 'Yes' to confirm.
+
+### 2. Set Up the Environment
+
+- Install dependencies using the provided environment file:
+
+```bash
+conda env create -f CITEgeist_env.yml
+```
+
+- Activate the environment and set up a Jupyter kernel:
+
+```bash
+conda activate CITEgeist_env
+```
+
+### 3. Obtain Gurobi License
+
+CITEgeist requires a Gurobi license (free for academic use):
+
+1. Sign up for an academic license at: [https://www.gurobi.com/downloads/end-user-license-agreement-academic/](https://www.gurobi.com/downloads/end-user-license-agreement-academic/)
+2. Follow the instructions to download and install your license.
+3. Update the license file path in the notebooks to match your local license location.
+
+### 4. Running CITEgeist
+
+You can run CITEgeist in two ways:
+
+#### A. Using Jupyter Notebooks
+- Update data paths in the top of the notebooks to match your local directory structure.
+- Expected runtime on a standard computer (16 threads, 32GB RAM):
+  - Vignette 1: ~2 hours
+  - Vignette 2: ~2 hours
+  - Vignette 3: ~10 hours
+
+#### Key Parameters:
+- `radius`: Radius for neighbor detection (default: 4)
+- `lambda_reg`: Regularization strength for cell proportion estimation (default: 0.001)
+- `alpha_elastic`: Elastic net mixing parameter for cell proportion estimation (default: 0.7)
+- `max_y_change`: Maximum allowed change in Y values (default: 0.2)
+
+#### Optional Parameters:
+- `profiling_only`: Set for cell-type proportions only.
+- `max_workers`: Number of parallel workers.
+- `checkpoint_interval`: Checkpoint saving interval.
+
+#### B. Using SLURM Distribution
+For large-scale analyses, you can use the provided `CITEgeist/examples/sbatch_sample.sh` script for distributed computing.
+
+---
+
+## Benchmarking and Reproducibility
+
+For specific reproduction of benchmarking tests and detailed methodology, please refer to the 'examples' and 'benchmarking' section in the documentation.
+
+---
+
+## Run the Analysis
+
+You can either:
+
+#### A. Run the notebooks directly:
+- Update data paths in the notebooks to match your local directory structure.
+- Expected runtime: ~2 hours on a standard computer (16 threads, 32GB RAM).
+
+#### B. Use SLURM distribution:
+- Use the provided `examples/sbatch_sample.sh` script for distributed computing.
+
+---
+
+## Additional System Requirements
+- **RAM**: 32GB (minimum)
+- **CPU**: 16 threads (recommended)
+- **Storage**: Sufficient space for the GEO dataset
+- **Operating System**: Linux/Unix recommended (Windows users may need additional configuration)

citegeist-0.1.0/citegeist/core/__init__.py

@@ -0,0 +1,21 @@
+# Expose key classes and functions for easy access
+
+from .citegeist_model import CitegeistModel
+from .gurobi_impl import optimize_cell_proportions, optimize_gene_expression, map_antibodies_to_profiles
+from .utils import (
+    save_results_to_output,
+    validate_cell_profile_dict,
+    cleanup_memory,
+    setup_logging
+)
+
+__all__ = [
+    "CitegeistModel",
+    "optimize_cell_proportions",
+    "optimize_gene_expression",
+    "map_antibodies_to_profiles",
+    "save_results_to_output",
+    "validate_cell_profile_dict",
+    "cleanup_memory",
+    "setup_logging"
+]

citegeist-0.1.0/citegeist/core/analysis_functions.py

@@ -0,0 +1,42 @@
+
+import scanpy as sc
+import anndata as ad
+
+
+def filter_spots_by_celltype_proportion(adata, celltype, proportion_threshold):
+    """
+    Filter spots by celltype proportion based on if their .obs[celltype] is greater than the proportion_threshold.
+    """
+    return adata[adata.obs[celltype] > proportion_threshold].copy()
+
+
+def expand_prop_gex_adata(prop_gex_adata, celltype_cutoff=0.2, celltype_profile_dict=None):
+    """
+    Expands celltype level spots to have multiple spots for each celltype, but retain spatial information
+    for COMMMOT analysis.
+    """
+    
+    if celltype_profile_dict is None:
+        raise ValueError("celltype_profile_dict is required for this function.")
+    celltypes = celltype_profile_dict.keys()
+
+    all_adatas = []
+    
+    for celltype in celltypes:
+        filtered_adata = filter_spots_by_celltype_proportion(prop_gex_adata, celltype, celltype_cutoff)
+        celltype_label = celltype.replace(" ", "_")
+        layer_name = f"{celltype_label}_genes_pass1"
+        filtered_adata.X = filtered_adata.layers[layer_name]
+        filtered_adata.obs['celltype'] = celltype
+        all_adatas.append(filtered_adata)
+    # Extract the corresponding layer for the filtered_adata, including the spatial information
+    combined_adata = ad.concat(all_adatas, join="outer", keys=celltypes, index_unique = "_", merge="same", uns_merge = "first")
+    return combined_adata
+
+def get_celltype_expression_data(adata, celltype):
+    """
+    Get expression data for a specific celltype.
+    """
+    return adata[adata.obs['celltype'] == celltype].copy()
+
+

citegeist-0.1.0/citegeist/core/checkpoints.py

@@ -0,0 +1,174 @@
+import os
+import logging
+import numpy as np
+from pathlib import Path
+
+class CheckpointManager:
+    """Manages loading and saving of optimization checkpoints."""
+    
+    def __init__(self, output_dir, sample_name):
+        """
+        Initialize checkpoint manager.
+        
+        Args:
+            output_dir (str): Directory for checkpoint files
+            sample_name (str): Unique identifier for this sample
+        """
+        self.output_dir = Path(output_dir)
+        self.sample_name = sample_name
+        self.output_dir.mkdir(parents=True, exist_ok=True)
+    
+    def check_complete_run(self, N, T, M):
+        """
+        Check if a complete run exists.
+        
+        Args:
+            N (int): Number of spots
+            T (int): Number of cell types
+            M (int): Number of genes/features
+            
+        Returns:
+            dict or None: Dictionary of spot profiles if complete run exists, else None
+        """
+        complete_file = self.output_dir / f"{self.sample_name}_gene_expression_complete.npz"
+        
+        if complete_file.exists():
+            try:
+                complete_data = np.load(complete_file)
+                if 'profiles' in complete_data and 'completed_spots' in complete_data:
+                    profiles = complete_data['profiles']
+                    if profiles.shape == (N, T, M):
+                        return {i: profiles[i] for i in range(N)}
+            except Exception as e:
+                logging.error(f"Error loading complete file: {e}")
+                self._cleanup_corrupted_files()
+        return None
+
+    def load_latest_checkpoint(self, N, T, M):
+        """
+        Load the latest valid checkpoint.
+        
+        Args:
+            N (int): Number of spots
+            T (int): Number of cell types
+            M (int): Number of genes/features
+            
+        Returns:
+            tuple: (completed_spots set, spotwise_profiles dict)
+        """
+        checkpoints = list(self.output_dir.glob(
+            f"{self.sample_name}_gene_expression_checkpoint_*.npz"
+        ))
+        
+        if not checkpoints:
+            return set(), {}
+            
+        checkpoint_numbers = [
+            int(f.stem.split('_')[-1]) 
+            for f in checkpoints
+        ]
+        latest_number = max(checkpoint_numbers)
+        latest_checkpoint = self.output_dir / f"{self.sample_name}_gene_expression_checkpoint_{latest_number}.npz"
+        
+        try:
+            checkpoint_data = np.load(latest_checkpoint)
+            if 'profiles' in checkpoint_data and 'completed_spots' in checkpoint_data:
+                profiles = checkpoint_data['profiles']
+                completed_spots = set(checkpoint_data['completed_spots'].tolist())
+                
+                if profiles.shape == (N, T, M):
+                    spotwise_profiles = {
+                        i: profiles[i] 
+                        for i in completed_spots 
+                        if not np.any(np.isnan(profiles[i]))
+                    }
+                    completed_spots = set(spotwise_profiles.keys())
+                    logging.info(f"Loaded {len(completed_spots)} valid profiles from checkpoint")
+                    return completed_spots, spotwise_profiles
+                    
+        except Exception as e:
+            logging.error(f"Error loading checkpoint: {e}")
+            self._cleanup_corrupted_files()
+            
+        return set(), {}
+
+    def save_checkpoint(self, completed_spots, spotwise_profiles, N, T, M):
+        """
+        Save current progress as checkpoint.
+        
+        Args:
+            completed_spots (set): Set of completed spot indices
+            spotwise_profiles (dict): Dictionary of spot profiles
+            N (int): Number of spots
+            T (int): Number of cell types
+            M (int): Number of genes/features
+        """
+        try:
+            n_completed = len(completed_spots)
+            checkpoint_path = self.output_dir / f"{self.sample_name}_gene_expression_checkpoint_{n_completed}.npz"
+            
+            # Convert dictionary to numpy array
+            max_spot = max(spotwise_profiles.keys())
+            profiles_array = np.full((max_spot + 1, T, M), np.nan)
+            
+            for spot_idx, profile in spotwise_profiles.items():
+                profiles_array[spot_idx] = profile
+            
+            # Save checkpoint
+            np.savez_compressed(
+                checkpoint_path,
+                profiles=profiles_array,
+                completed_spots=np.array(list(completed_spots)),
+                n_completed=n_completed
+            )
+            
+            # Cleanup old checkpoints
+            self._cleanup_old_checkpoints(checkpoint_path)
+            
+            logging.info(f"Saved checkpoint after {n_completed} completed spots")
+            
+        except Exception as e:
+            logging.error(f"Failed to save checkpoint: {e}")
+
+    def save_final_results(self, spotwise_profiles, completed_spots, N, T, M):
+        """
+        Save final results.
+        
+        Args:
+            spotwise_profiles (dict): Dictionary of spot profiles
+            completed_spots (set): Set of completed spot indices
+            N (int): Number of spots
+            T (int): Number of cell types
+            M (int): Number of genes/features
+        """
+        final_path = self.output_dir / f"{self.sample_name}_gene_expression_complete.npz"
+        max_spot = max(spotwise_profiles.keys())
+        final_profiles = np.full((max_spot + 1, T, M), np.nan)
+        
+        for spot_idx, profile in spotwise_profiles.items():
+            final_profiles[spot_idx] = profile
+            
+        np.savez_compressed(
+            final_path, 
+            profiles=final_profiles, 
+            completed_spots=np.array(list(completed_spots))
+        )
+        logging.info(f"Saved final results with {len(completed_spots)} completed spots")
+
+    def _cleanup_corrupted_files(self):
+        """Remove all checkpoint files if corruption is detected."""
+        for file in self.output_dir.glob(f"{self.sample_name}_gene_expression*.npz"):
+            try:
+                file.unlink()
+                logging.info(f"Deleted corrupted checkpoint: {file}")
+            except Exception as e:
+                logging.warning(f"Failed to delete {file}: {e}")
+
+    def _cleanup_old_checkpoints(self, current_checkpoint):
+        """Remove old checkpoints, keeping only the latest."""
+        for checkpoint in self.output_dir.glob(f"{self.sample_name}_gene_expression_checkpoint_*.npz"):
+            if checkpoint != current_checkpoint:
+                try:
+                    checkpoint.unlink()
+                except Exception as e:
+                    logging.warning(f"Failed to delete old checkpoint {checkpoint}: {e}")