cystainer 0.1.0__tar.gz

cystainer-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Sysgen lab
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

cystainer-0.1.0/PKG-INFO

@@ -0,0 +1,170 @@
+Metadata-Version: 2.4
+Name: cystainer
+Version: 0.1.0
+Summary: A PyTorch-based tool for predicting missing proteins in cytometry/single-cell data
+Author: Konstantin Ivanov
+Author-email: kivanov@uef.fi
+Classifier: Programming Language :: Python :: 3
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Requires-Python: >=3.11.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: torch>=2.7.1
+Requires-Dist: numpy>=1.26.4
+Requires-Dist: pandas>=2.2.3
+Requires-Dist: anndata>=0.12.4
+Requires-Dist: scipy>=1.14.1
+Requires-Dist: scikit-learn>=1.5.1
+Requires-Dist: tqdm>=4.66.5
+Requires-Dist: matplotlib>=3.9.2
+Dynamic: author
+Dynamic: author-email
+Dynamic: classifier
+Dynamic: description
+Dynamic: description-content-type
+Dynamic: license-file
+Dynamic: requires-dist
+Dynamic: requires-python
+Dynamic: summary
+
+# CyStainer
+CyStainer package for cytometry marker imputation
+
+**CyStainer** is a PyTorch-based deep learning tool for predicting missing proteins and imputing marker expression in cytometry and single-cell data. It utilizes a combination of Variational Autoencoders (VAEs) and Transformer architectures to integrate multiple batches/panels and infer missing markers accurately.
+
+## 📦 Installation
+
+Since CyStainer is built on PyTorch, ensure you have an environment with Python 3.11+ and the appropriate PyTorch version for your hardware (CUDA recommended).
+
+You can install CyStainer directly from the source:
+
+```bash
+git clone https://github.com/sysgen-uef/cystainer_package.git
+cd cystainer_package
+pip install .
+```
+## 🧹 Data Preprocessing (From .fcs to .h5ad)
+
+**CyStainer** expects your input data to be formatted as `AnnData` objects (either passed as a list in memory or saved locally as `.h5ad` files). Before feeding your cytometry data into the model, it must be properly preprocessed. 
+
+**Mandatory Pre-processing:**
+* **Cleaning:** Ensure your data is pre-gated to remove doublets, debris, and dead cells.
+* **Compensation:** Your `.fcs` files should be already compensated.
+
+**Transformation & Scaling (Recommended):**
+You will typically need to transform your fluorescence intensities (e.g., using an arcsinh transformation) and optionally scale them. Removing saturated values (extreme highs or zeros) can also improve model performance depending on your dataset.
+
+Here is a minimal example of how to process a standard `.fcs` file into a ready-to-use `.h5ad` file using `FlowKit` and `AnnData`:
+
+```python
+import flowkit as fk
+import pandas as pd
+import numpy as np
+import anndata as ad
+
+# Load the compensated and cleaned .fcs file
+sample = fk.Sample('path/to/cleaned_sample.fcs')
+df = sample.as_dataframe('raw')
+df.columns = sample.pnn_labels # Set marker names
+
+# Transformation (e.g., arcsinh with a cofactor of 100 or 150)
+cofactor = 100
+non_scatter = ['FSC' not in c and 'SSC' not in c for c in df.columns]
+df.loc[:, non_scatter] = np.arcsinh(df.loc[:, non_scatter] / cofactor)
+
+# Optional: Remove saturation / extreme outliers
+# Useful if your instrument records artificial bounds (e.g., exactly 0 or max value)
+df = df[~np.any(((df <= 0) | (df >= df.max().max())), axis=1)]
+
+# Optional: Scaling (Min-Max scaling to [0, 1] or Z-score normalization)
+# Min-Max Scaling example:
+df = (df - df.min()) / (df.max() - df.min())
+# Z-score Scaling example (alternatively): 
+# df = (df - df.mean()) / df.std()
+
+# Convert to AnnData and save for CyStainer
+adata = ad.AnnData(df)
+adata.write('preprocessed_sample.h5ad', compression='gzip')
+```
+
+Once your `.fcs` files are converted into `.h5ad` objects, you can load them directly into your workflow.
+
+## 🚀 Quick Start Guide
+
+The primary way to interact with the package is through the `CyStainer` wrapper class. The standard workflow consists of initializing the model, loading training data, building the network, and running inference.
+
+### 1. Training a Base Model
+
+You can load your data either from a folder of `.h5ad` files or by passing a list of `AnnData` objects directly in memory.
+
+```python
+from cystainer import CyStainer
+
+# Initialize the stainer (automatically detects CUDA/CPU)
+stainer = CyStainer()
+
+# Load training data
+# Alternatively, use: adata_list=[adata1, adata2]
+# CyStainer includes a built-in utility to visualize how markers overlap across different panels or batches.
+stainer.load_train_data(folder_path='./data_example/train', get_panel_vis=True)
+```
+![image](image/panel_alignment.png)
+
+```python
+# Build the model 
+# You can pass custom hyperparameters here if needed
+stainer.build_model()
+
+# Train the model
+stainer.train()
+
+# By default the model is automatically saved in the same directory
+# as cystainer.pt
+```
+
+### 2. Predicting Missing Markers
+
+Once a model is trained, you can load inference data. The `.load_predict_data()` method ensures your cells are not shuffled so that the output matches your input order.
+
+```python
+# Load prediction data using the base model's reference markers
+stainer.load_predict_data(folder_path='./data_example/test')
+
+# Run predictions and save directly to disk
+stainer.predict(output_path='imputed_cells.h5ad')
+
+# Alternatively, return the predictions as a pandas DataFrame:
+# df_imputed = stainer.predict(return_pred=True)
+```
+
+### 3. Fine-Tuning on New Batches
+
+If you receive new data from a different batch or panel, you don't need to retrain from scratch. CyStainer can freeze the main network and fine-tune only the batch embeddings to align the new data.
+
+```python
+# Load the pre-trained model
+stainer = CyStainer.load('cystainer.pt')
+
+# Load the new data for fine-tuning
+# Note, anndata objects must have batch info column
+stainer.load_finetune_data(folder_path='./data_example/test', batch_info='batch')
+
+# Fine-tune the batch embeddings
+stainer.finetune()
+
+# Predict on the newly fine-tuned data
+stainer.predict(output_path='imputed_new_batch.h5ad')
+```
+
+### 4. Batch Correction
+
+CyStainer allows you to translate cells from their original batch to a specific target batch distribution.
+
+```python
+stainer.predict(
+    output_path='batch_corrected_cells.h5ad',
+    correct_batch=True,
+    target_batch_name='single_batch' # Must match a batch name in stainer.batch_dict
+)
+```

cystainer-0.1.0/README.md

@@ -0,0 +1,140 @@
+# CyStainer
+CyStainer package for cytometry marker imputation
+
+**CyStainer** is a PyTorch-based deep learning tool for predicting missing proteins and imputing marker expression in cytometry and single-cell data. It utilizes a combination of Variational Autoencoders (VAEs) and Transformer architectures to integrate multiple batches/panels and infer missing markers accurately.
+
+## 📦 Installation
+
+Since CyStainer is built on PyTorch, ensure you have an environment with Python 3.11+ and the appropriate PyTorch version for your hardware (CUDA recommended).
+
+You can install CyStainer directly from the source:
+
+```bash
+git clone https://github.com/sysgen-uef/cystainer_package.git
+cd cystainer_package
+pip install .
+```
+## 🧹 Data Preprocessing (From .fcs to .h5ad)
+
+**CyStainer** expects your input data to be formatted as `AnnData` objects (either passed as a list in memory or saved locally as `.h5ad` files). Before feeding your cytometry data into the model, it must be properly preprocessed. 
+
+**Mandatory Pre-processing:**
+* **Cleaning:** Ensure your data is pre-gated to remove doublets, debris, and dead cells.
+* **Compensation:** Your `.fcs` files should be already compensated.
+
+**Transformation & Scaling (Recommended):**
+You will typically need to transform your fluorescence intensities (e.g., using an arcsinh transformation) and optionally scale them. Removing saturated values (extreme highs or zeros) can also improve model performance depending on your dataset.
+
+Here is a minimal example of how to process a standard `.fcs` file into a ready-to-use `.h5ad` file using `FlowKit` and `AnnData`:
+
+```python
+import flowkit as fk
+import pandas as pd
+import numpy as np
+import anndata as ad
+
+# Load the compensated and cleaned .fcs file
+sample = fk.Sample('path/to/cleaned_sample.fcs')
+df = sample.as_dataframe('raw')
+df.columns = sample.pnn_labels # Set marker names
+
+# Transformation (e.g., arcsinh with a cofactor of 100 or 150)
+cofactor = 100
+non_scatter = ['FSC' not in c and 'SSC' not in c for c in df.columns]
+df.loc[:, non_scatter] = np.arcsinh(df.loc[:, non_scatter] / cofactor)
+
+# Optional: Remove saturation / extreme outliers
+# Useful if your instrument records artificial bounds (e.g., exactly 0 or max value)
+df = df[~np.any(((df <= 0) | (df >= df.max().max())), axis=1)]
+
+# Optional: Scaling (Min-Max scaling to [0, 1] or Z-score normalization)
+# Min-Max Scaling example:
+df = (df - df.min()) / (df.max() - df.min())
+# Z-score Scaling example (alternatively): 
+# df = (df - df.mean()) / df.std()
+
+# Convert to AnnData and save for CyStainer
+adata = ad.AnnData(df)
+adata.write('preprocessed_sample.h5ad', compression='gzip')
+```
+
+Once your `.fcs` files are converted into `.h5ad` objects, you can load them directly into your workflow.
+
+## 🚀 Quick Start Guide
+
+The primary way to interact with the package is through the `CyStainer` wrapper class. The standard workflow consists of initializing the model, loading training data, building the network, and running inference.
+
+### 1. Training a Base Model
+
+You can load your data either from a folder of `.h5ad` files or by passing a list of `AnnData` objects directly in memory.
+
+```python
+from cystainer import CyStainer
+
+# Initialize the stainer (automatically detects CUDA/CPU)
+stainer = CyStainer()
+
+# Load training data
+# Alternatively, use: adata_list=[adata1, adata2]
+# CyStainer includes a built-in utility to visualize how markers overlap across different panels or batches.
+stainer.load_train_data(folder_path='./data_example/train', get_panel_vis=True)
+```
+![image](image/panel_alignment.png)
+
+```python
+# Build the model 
+# You can pass custom hyperparameters here if needed
+stainer.build_model()
+
+# Train the model
+stainer.train()
+
+# By default the model is automatically saved in the same directory
+# as cystainer.pt
+```
+
+### 2. Predicting Missing Markers
+
+Once a model is trained, you can load inference data. The `.load_predict_data()` method ensures your cells are not shuffled so that the output matches your input order.
+
+```python
+# Load prediction data using the base model's reference markers
+stainer.load_predict_data(folder_path='./data_example/test')
+
+# Run predictions and save directly to disk
+stainer.predict(output_path='imputed_cells.h5ad')
+
+# Alternatively, return the predictions as a pandas DataFrame:
+# df_imputed = stainer.predict(return_pred=True)
+```
+
+### 3. Fine-Tuning on New Batches
+
+If you receive new data from a different batch or panel, you don't need to retrain from scratch. CyStainer can freeze the main network and fine-tune only the batch embeddings to align the new data.
+
+```python
+# Load the pre-trained model
+stainer = CyStainer.load('cystainer.pt')
+
+# Load the new data for fine-tuning
+# Note, anndata objects must have batch info column
+stainer.load_finetune_data(folder_path='./data_example/test', batch_info='batch')
+
+# Fine-tune the batch embeddings
+stainer.finetune()
+
+# Predict on the newly fine-tuned data
+stainer.predict(output_path='imputed_new_batch.h5ad')
+```
+
+### 4. Batch Correction
+
+CyStainer allows you to translate cells from their original batch to a specific target batch distribution.
+
+```python
+stainer.predict(
+    output_path='batch_corrected_cells.h5ad',
+    correct_batch=True,
+    target_batch_name='single_batch' # Must match a batch name in stainer.batch_dict
+)
+```

cystainer-0.1.0/cystainer/__init__.py

@@ -0,0 +1,16 @@
+# cystainer/__init__.py
+
+# Package version
+__version__ = "0.1.0"
+
+# Imports
+from .runner import CyStainer
+from .data import load_data_from_folder
+from .modules import CyStainerModel
+
+# Imports `from cystainer import *`
+__all__ = [
+    "CyStainerModel",
+    "CyStainer",
+    "load_data_from_folder",
+]

cystainer-0.1.0/cystainer/data.py

@@ -0,0 +1,204 @@
+import os
+import itertools
+import torch
+import numpy as np
+import pandas as pd
+import anndata as ad
+import matplotlib.pyplot as plt
+import matplotlib.patches as patches
+from torch.utils.data import DataLoader
+from .modules import get_marker_list, pad_and_get_pad_mask
+
+def load_data_from_folder(folder_path=None, adata_list=None, exclude_files=[], max_n_cells=int(1e+6), batch_size=1024, is_train=True, drop_markers=None,
+                          reference_markers=None, reference_batch_dict=None, normalize=False, batch_info=None, get_panel_vis=False):
+    """Reads .h5ad files from a folder OR accepts a list of AnnData objects directly."""
+    
+    # Establish the adata_list
+    if adata_list is None:
+        if folder_path is None:
+            raise ValueError("You must provide either 'folder_path' or 'adata_list'.")
+            
+        sample_names = [f for f in os.listdir(folder_path) if f.endswith('.h5ad') and (f not in exclude_files)]
+        if not sample_names:
+            raise ValueError(f"No files found in {folder_path}")
+        
+        # Load adatas from disk
+        loaded_adatas = [ad.read_h5ad(os.path.join(folder_path, f)) for f in sample_names]
+    else:
+        # Use the provided list of AnnData objects
+        loaded_adatas = adata_list
+
+    # Extract DataFrames and Batch Info
+    df_list = [adata.to_df().iloc[:max_n_cells,:] for adata in loaded_adatas]
+    
+    if batch_info is not None:
+        batch_list = [adata.obs[batch_info].tolist() for adata in loaded_adatas]
+        batch_names = list(itertools.chain.from_iterable(batch_list))
+    else:
+        print("No batch info passed, setting 'single_batch' for the data.")
+        batch_names = list(itertools.chain.from_iterable([['single_batch'] * df.shape[0] for df in df_list]))
+
+    # Visualization, Processing, and Normalization
+    if get_panel_vis:
+        visualize_panel_overlap(df_list)
+        
+    if drop_markers is not None:
+        df_list = [df.loc[:,~df.columns.isin(drop_markers)] for df in df_list]
+    
+    if normalize:
+        df_list = [(df - df.mean()) / df.std() for df in df_list]
+
+    # Handle Batches
+    if reference_batch_dict is None:
+        # Sort for deterministic behavior
+        batch_dict = {batch: i for i, batch in enumerate(sorted(list(set(batch_names))))}
+    else:
+        batch_dict = reference_batch_dict.copy()
+        current_max_idx = max(batch_dict.values()) if batch_dict else -1
+        
+        # Find only the new batches and sort them
+        new_batches = sorted(list(set(batch_names) - set(batch_dict.keys())))
+        
+        for batch in new_batches:
+            current_max_idx += 1
+            batch_dict[batch] = current_max_idx
+
+    batch_ids = pd.Series(batch_names).apply(lambda x: batch_dict[x]).values
+
+    # Handle Markers and Padding
+    if reference_markers is None:
+        marker_list, shared_markers, unique_markers = get_marker_list(df_list)
+    else:
+        marker_list = reference_markers['marker_list']
+        shared_markers = reference_markers['shared_markers']
+        unique_markers = reference_markers['unique_markers']
+
+    df_list, pad_mask = pad_and_get_pad_mask(df_list, marker_list)
+    
+    # PyTorch Dataloader
+    data = [{'x': x.astype('float32'), 'batch': b, 'pad': p} 
+            for x, b, p in zip(pd.concat(df_list).values, batch_ids, np.concatenate(pad_mask, axis=0))]
+    
+    dataloader = DataLoader(data, batch_size=batch_size, shuffle=is_train)
+
+    markers_info = {
+        'marker_list': marker_list,
+        'shared_markers': shared_markers,
+        'unique_markers': unique_markers
+    }
+
+    return dataloader, markers_info, batch_dict
+
+def visualize_panel_overlap(df_list, alignment='left'):
+    """
+    Creates a block alignment plot showing shared and unique markers 
+    across different panels.
+    
+    Args:
+        df_list (list): List of pandas DataFrames.
+        panel_names (list): List of panel names as strings.
+        alignment (str): 'center' for a center-outwards pyramid style, 
+                         'left' for a cascading left-aligned style.
+    """
+    # Extract markers and build presence matrix
+    panel_markers = set([frozenset(df.columns) for df in df_list])
+    all_markers = set().union(*panel_markers)
+    num_panels = len(panel_markers)
+    panel_names = [f'Panel {i+1}' for i in range(num_panels)]
+    
+    matrix = pd.DataFrame(index=list(all_markers), columns=panel_names)
+    for name, markers in zip(panel_names, panel_markers):
+        matrix[name] = matrix.index.isin(markers).astype(int)
+        
+    # Sort markers based on requested alignment
+    if alignment == 'left':
+        # Simple cascade sort
+        matrix = matrix.sort_values(by=panel_names, ascending=False)
+        sorted_markers = matrix.index.tolist()
+        
+    elif alignment == 'center':
+        # Center-outwards sort (Frequency and Center of Mass)
+        matrix['freq'] = matrix.sum(axis=1)
+        col_indices = np.arange(num_panels)
+        
+        matrix['com'] = (matrix[panel_names] * col_indices).sum(axis=1) / matrix['freq']
+        matrix['pattern'] = matrix[panel_names].astype(str).agg(''.join, axis=1)
+        
+        grouped = matrix.groupby('pattern')
+        group_stats = grouped.first()[['freq', 'com']].sort_values(by=['freq', 'com'], ascending=[False, True])
+        
+        left_part = []
+        right_part = []
+        center_part = []
+        mid_point = (num_panels - 1) / 2.0
+        
+        for pattern, row in group_stats.iterrows():
+            group_markers = matrix[matrix['pattern'] == pattern].index.tolist()
+            group_markers.sort() # Alphabetical fallback
+            
+            if row['freq'] == num_panels:
+                center_part.extend(group_markers)
+            else:
+                if row['com'] < mid_point:
+                    left_part = group_markers + left_part
+                elif row['com'] > mid_point:
+                    right_part = right_part + group_markers
+                else:
+                    if len(left_part) <= len(right_part):
+                        left_part = group_markers + left_part
+                    else:
+                        right_part = right_part + group_markers
+                        
+        sorted_markers = left_part + center_part + right_part
+        
+    else:
+        raise ValueError("The 'alignment' parameter must be either 'left' or 'center'.")
+
+    # Setup the plot
+    fig, ax = plt.subplots(figsize=(14, len(panel_names) * 1.5))
+    rect_height = 0.6
+    colors = ['#72A0C1', '#D65A61', '#E8B358', '#7CE0C9', '#8291A8', '#C48CB3'] 
+    
+    # Draw the blocks
+    for i, (name, markers) in enumerate(zip(panel_names, panel_markers)):
+        presence = [1 if m in markers else 0 for m in sorted_markers]
+        start_idx = None
+        for j, val in enumerate(presence):
+            if val == 1 and start_idx is None:
+                start_idx = j
+            elif val == 0 and start_idx is not None:
+                rect = patches.Rectangle(
+                    (start_idx, i - rect_height/2), j - start_idx, rect_height,
+                    edgecolor='black', facecolor=colors[i % len(colors)], alpha=0.9, linewidth=0.8
+                )
+                ax.add_patch(rect)
+                start_idx = None
+                
+        # Catch end blocks
+        if start_idx is not None:
+            rect = patches.Rectangle(
+                (start_idx, i - rect_height/2), len(presence) - start_idx, rect_height,
+                edgecolor='black', facecolor=colors[i % len(colors)], alpha=0.9, linewidth=0.8
+            )
+            ax.add_patch(rect)
+            
+        ax.text(-0.8, i, name, va='center', ha='right', fontsize=12, fontweight='bold')
+
+    # Formatting
+    ax.set_xlim(0, len(sorted_markers))
+    ax.set_ylim(-1, len(panel_names))
+    ax.set_yticks([])
+    ax.set_xticks(np.arange(len(sorted_markers)) + 0.5)
+    ax.set_xticklabels(sorted_markers, rotation=90, ha='center', fontsize=10)
+    
+    # Remove borders
+    for spine in ['top', 'right', 'left']:
+        ax.spines[spine].set_visible(False)
+    
+    ax.xaxis.grid(True, linestyle='--', alpha=0.4)
+    ax.set_axisbelow(True)
+    
+    title_align = "Center" if alignment == 'center' else "Left"
+    plt.title(f"Panel Marker Alignment", fontsize=16, pad=25, fontweight='bold')
+    plt.tight_layout()
+    plt.show()