spaceexpress 0.1.5__tar.gz

spaceexpress-0.1.5/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2024 Yeojin Kim
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

spaceexpress-0.1.5/MANIFEST.in

@@ -0,0 +1,2 @@
+include RELEASE_0.1.5.md RELEASE_0.1.5-en.md
+recursive-include examples *.py *.sh

spaceexpress-0.1.5/PKG-INFO

@@ -0,0 +1,81 @@
+Metadata-Version: 2.4
+Name: spaceexpress
+Version: 0.1.5
+Summary: A Python package for spatial transcriptomics
+Author: Yeojin Kim
+Author-email: Yeojin Kim <ykim3030@gatech.edu>
+Project-URL: Repository, https://github.com/YeojinKim220/SpaceExpress
+Keywords: spaceexpress,Python
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: numpy
+Requires-Dist: scipy
+Requires-Dist: pandas>=2.2
+Requires-Dist: anndata>=0.12
+Requires-Dist: scanpy>=1.10
+Requires-Dist: scikit-learn>=1.5
+Requires-Dist: matplotlib>=3.8
+Requires-Dist: torch>=2.1
+Requires-Dist: igraph<1,>=0.11
+Requires-Dist: tqdm
+Requires-Dist: joblib
+Requires-Dist: statsmodels
+Requires-Dist: pygam>=0.9
+Requires-Dist: rpy2<3.6,>=3.5.11
+Provides-Extra: test
+Requires-Dist: pytest; extra == "test"
+Requires-Dist: build; extra == "test"
+Requires-Dist: twine; extra == "test"
+Provides-Extra: notebooks
+Requires-Dist: nbformat; extra == "notebooks"
+Requires-Dist: nbclient; extra == "notebooks"
+Requires-Dist: ipykernel; extra == "notebooks"
+Dynamic: author
+Dynamic: license-file
+
+# SpaceExpress 0.1.5
+
+Korean version: [RELEASE_0.1.5.md](RELEASE_0.1.5.md)
+
+## Changes
+
+- Failed gene-embedding-dimension fits (negative or nonfinite statistics) receive FDR=1. Dimensions without a valid empirical-null fit also return FDR=1.
+- Final FDR values are bounded to [0, 1]. The existing empirical-null estimator is not replaced with another adjustment method.
+- `select_hvg_after_outlier` operates on normalized, log-transformed common genes, setting values at or above the pooled mean + 4 sample standard deviations to 0 before batch-aware HVG selection. It modifies copies, not input objects.
+- Initial preprocessing history is stored in `.uns['spaceexpress_preprocessing']`. DSE skips repeated mean+4SD removal when every input has this history. Unmarked legacy inputs retain the previous behavior; mixed histories require an explicit setting.
+- `SpaceExpress_DSE(..., remove_mean_sd_outliers=False)` also explicitly disables subsequent mean+4SD removal. Existing training-time 95% clipping is unchanged. The separate 99% filter in multi-replicate DSE is also retained.
+- Returned AnnData `.varm` includes `DSE-statistic` and `DSE-fit-failed`, so failures can be inspected without inferring them from 0 predictions.
+
+## Installation
+
+Python 3.11 or newer and R are required. Install R packages `lmtest`, `fitdistrplus`, `dplyr`, and `lme4` first, and configure Python to find the R shared library. Python dependencies are declared in package metadata. Use the PyTorch installation appropriate for your CUDA environment.
+
+```bash
+Rscript -e 'install.packages(c("lmtest", "fitdistrplus", "dplyr", "lme4"), repos="https://cloud.r-project.org")'
+python -m pip install 'spaceexpress[notebooks]==0.1.5'
+```
+
+`examples/install_release.sh BASE_PYTHON NEW_ENV` installs the PyPI release into a new venv sharing an existing scientific/R environment. Package download provenance is recorded in `pip_install_report.json`, with dependencies in `pip_freeze.txt`. This is not a fully isolated installation of all dependencies from scratch.
+
+## Running Data
+
+`examples/public_pair.py` resolves paths relative to its config and does not overwrite originals. It checks raw counts for at least 3 detected genes and finite spatial coordinates, then applies spatial-bin proportional sampling if needed. Common genes detected in at least 3 observations in each sample are normalized to 10,000, log1p-transformed, and processed as above to select 200 HVGs. Selected raw data and QC metrics are also saved.
+
+```bash
+bash examples/submit_public_pair.sh /path/to/env/bin/python /path/to/config_v015.json /path/to/results_v015
+```
+
+Default Slurm requests are 12 CPU/128 GB for preparation, 1 H100 80 GB GPU with 8 CPU/256 GB RAM for training, 12 CPU/128 GB per DSE job, and 4 CPU/64 GB for the notebook. Each job has a 4-hour limit. Set `SE_ACCOUNT`, `SE_QOS`, `SE_CPU_PARTITION`, `SE_GPU_PARTITION`, and `SE_GPU_CONSTRAINT` to configure your cluster. The default CPU partition is `cpu-medium`. For A100, set `SE_GPU_PARTITION=gpu-a100 SE_GPU_CONSTRAINT=A100-80GB`. These are requests, not measured requirements.
+
+Stages run in order: preparation, a 2-epoch GPU probe using all selected observations, main training, k=30/50/100 DSE, and a results notebook. Dependencies require successful prior stages. Main training uses `epochs` and `patience` from the config. Reusing a run directory is rejected; choose a new output path for a new experiment.
+
+Outputs include `sample_summary.csv`, QC and gene diagnostics CSVs, selected-raw/prepared/embedded H5ADs, model state, FDR/statistic/failure-mask CSVs, DSE pickles, per-stage JSON/time/memory logs, and an executed `results.ipynb` with figures. The notebook shows original versus actual observation counts, spatial coverage, counts and detected genes, embeddings, failed fits, k sensitivity, and gene expression and fitted results. Only open pickles from trusted runs.
+
+## Interpretation and Limits
+
+95% clipping can erase a gene's entire training signal in a sample when a sparse gene's upper bound is 0. This can occur after initial 4SD removal and is recorded in `gene_qc_*.csv` and the notebook. This release retains both agreed preprocessing steps and does not automatically substitute another method.
+
+Slide-seq observations may be beads and Stereo-seq observations may be spatial bins; observation counts are not necessarily individual-cell counts. Inputs with only Brain tissue annotations are not assigned inferred detailed cell types for visualization. No additional mitochondrial-percentage or detection thresholds are applied automatically; inspect distributions and sample-specific QC before deciding.
+
+FDR-range, failure-handling, and observation-preservation checks establish numerical behavior, not biological validity. Review k stability and spatial-neighbor recall as well. A comparison with 1 biological sample per condition does not replace replicated population-level inference.

spaceexpress-0.1.5/README.md

@@ -0,0 +1,75 @@
+# SpaceExpress
+
+![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)
+
+SpaceExpress is a tool for differential spatial expression (DSE) analysis using intrinsic coordinate systems of tissues. It leverages spline models and likelihood ratio testing to identify significant spatial gene expression changes across different biological conditions or experimental groups.
+
+## News
+**2024.11.26**  
+- Initial release of SpaceExpress, enabling robust differential spatial expression analysis with support for single and multiple replicates.
+
+## Getting Started
+To learn how to use SpaceExpress for your spatial transcriptomics data, follow the tutorials:
+
+- See **[Tutorials](./docs/source/notebook/)**.
+- **Single Replicate Analysis**: Check [Tutorial 1](./docs/source/notebook/Tutorial_1.ipynb).
+- **Multiple Replicate Analysis**: Check [Tutorial 2](./docs/source/notebook/Tutorial_2.ipynb).
+- **Simulated Data Generation**: Check [Simulated Spatial Transcriptomics Data](./docs/source/notebook/Simulated_Spatial_Transcriptomics_Data.ipynb).
+
+These tutorials will guide you through data preparation, model setup, and interpreting the results of differential spatial expression analysis. The expected run time for Tutorial 1 is approximately 1 hour and 30 minutes, while Tutorial 2 is estimated to take around 2 hours and 30 minutes. 
+
+## Software Dependencies
+To ensure compatibility and optimal performance, please check the required software dependencies specified in the `environment.yml` file. These include key libraries such as `scanpy`, `pandas`, `numpy`, and `rpy2` for seamless integration with R-based statistical models.
+
+## Installation
+Install the SpaceExpress environment and package with the following commands (Expected installation time is approximately 5–10 minutes.):
+
+```bash
+git clone https://github.com/YeojinKim220/SpaceExpress.git
+conda env create -f environment.yml
+conda activate spaceexpress-env
+
+pip install SpaceExpress
+```
+
+This will create and activate a Conda environment with all necessary dependencies and install SpaceExpress in editable mode for local development.
+
+<details>
+  <summary><b>Note:</b> If the installation via the provided <code>environment.yml</code> file fails, you can manually set up the environment using the steps below:</summary>
+
+  ```bash
+  # Step 1: Create and activate a new Conda environment
+  conda create -n spaceexpress-env python=3.11 scikit-learn pandas matplotlib jupyter scanpy rpy2 -y
+  conda activate spaceexpress-env
+
+  # Step 2: Install additional dependencies
+  conda install -y -c conda-forge python-igraph r-lmtest r-fitdistrplus r-dplyr r-lme4
+
+  # Step 3: Install Python packages using pip
+  pip install pygam
+  pip install torch
+
+  # Step 4: Install SpaceExpress
+  pip install SpaceExpress
+  ```
+</details>
+
+## Tested Environments and Compatibility
+- **Linux (Red Hat Enterprise Linux 9.4)**: `python-3.11.11`, `scikit-learn-1.6.0`, `pandas-2.2.3`, `matplotlib-3.10.0`, `jupyter-1.1.1`, `scanpy-1.10.4`, `rpy2-3.5.11`, `python-igraph-0.11.8`, `r-lmtest-0.9_40`, `r-fitdistrplus-1.2_1`, `r-dplyr-1.1.4`, `r-lme4-1.1_35.5`, `pygam-0.9.1`, `torch-2.5.1`.
+- **macOS (Sonoma 14.3)**: `python-3.11.10`, `scikit-learn-1.5.1`, `pandas-2.2.2`, `matplotlib-3.9.2`, `jupyter-1.0.0`, `scanpy-1.10.3`, `rpy2-3.5.11`, `python-igraph-0.11.6`, `r-lmtest-0.9_40`, `r-fitdistrplus-1.1_11`, `r-dplyr-1.1.4`, `r-lme4-1.1_35.5`, `pygam-0.9.1`, `torch-2.5.1`.
+
+## Guidelines for choosing hyperparameters
+We empirically observed that this hyperparameter setting is effective for datasets with sizes ranging from approximately 3k to 10k cells. For datasets with substantially fewer cells, we recommend using smaller values of the "k" parameter in the SpaceExpress_DSE function.
+
+## Citation
+If you use SpaceExpress in your research, please cite:
+```
+@article{kim2024spaceexpress,
+  title={SpaceExpress: a method for comparative spatial transcriptomics based on intrinsic coordinate systems of tissues},
+  author={Kim, Yeojin and Ojha, Abhishek and Schrader, Alex and Lee, Juyeon and Wu, Zijun and Traniello, Ian M and Robinson, Gene E and Han, Hee Sun and Zhao, Sihai D and Sinha, Saurabh},
+  journal={bioRxiv},
+  pages={2024--12},
+  year={2024},
+  publisher={Cold Spring Harbor Laboratory}
+}
+```

spaceexpress-0.1.5/RELEASE_0.1.5-en.md

@@ -0,0 +1,45 @@
+# SpaceExpress 0.1.5
+
+Korean version: [RELEASE_0.1.5.md](RELEASE_0.1.5.md)
+
+## Changes
+
+- Failed gene-embedding-dimension fits (negative or nonfinite statistics) receive FDR=1. Dimensions without a valid empirical-null fit also return FDR=1.
+- Final FDR values are bounded to [0, 1]. The existing empirical-null estimator is not replaced with another adjustment method.
+- `select_hvg_after_outlier` operates on normalized, log-transformed common genes, setting values at or above the pooled mean + 4 sample standard deviations to 0 before batch-aware HVG selection. It modifies copies, not input objects.
+- Initial preprocessing history is stored in `.uns['spaceexpress_preprocessing']`. DSE skips repeated mean+4SD removal when every input has this history. Unmarked legacy inputs retain the previous behavior; mixed histories require an explicit setting.
+- `SpaceExpress_DSE(..., remove_mean_sd_outliers=False)` also explicitly disables subsequent mean+4SD removal. Existing training-time 95% clipping is unchanged. The separate 99% filter in multi-replicate DSE is also retained.
+- Returned AnnData `.varm` includes `DSE-statistic` and `DSE-fit-failed`, so failures can be inspected without inferring them from 0 predictions.
+
+## Installation
+
+Python 3.11 or newer and R are required. Install R packages `lmtest`, `fitdistrplus`, `dplyr`, and `lme4` first, and configure Python to find the R shared library. Python dependencies are declared in package metadata. Use the PyTorch installation appropriate for your CUDA environment.
+
+```bash
+Rscript -e 'install.packages(c("lmtest", "fitdistrplus", "dplyr", "lme4"), repos="https://cloud.r-project.org")'
+python -m pip install 'spaceexpress[notebooks]==0.1.5'
+```
+
+`examples/install_release.sh BASE_PYTHON NEW_ENV` installs the PyPI release into a new venv sharing an existing scientific/R environment. Package download provenance is recorded in `pip_install_report.json`, with dependencies in `pip_freeze.txt`. This is not a fully isolated installation of all dependencies from scratch.
+
+## Running Data
+
+`examples/public_pair.py` resolves paths relative to its config and does not overwrite originals. It checks raw counts for at least 3 detected genes and finite spatial coordinates, then applies spatial-bin proportional sampling if needed. Common genes detected in at least 3 observations in each sample are normalized to 10,000, log1p-transformed, and processed as above to select 200 HVGs. Selected raw data and QC metrics are also saved.
+
+```bash
+bash examples/submit_public_pair.sh /path/to/env/bin/python /path/to/config_v015.json /path/to/results_v015
+```
+
+Default Slurm requests are 12 CPU/128 GB for preparation, 1 H100 80 GB GPU with 8 CPU/256 GB RAM for training, 12 CPU/128 GB per DSE job, and 4 CPU/64 GB for the notebook. Each job has a 4-hour limit. Set `SE_ACCOUNT`, `SE_QOS`, `SE_CPU_PARTITION`, `SE_GPU_PARTITION`, and `SE_GPU_CONSTRAINT` to configure your cluster. The default CPU partition is `cpu-medium`. For A100, set `SE_GPU_PARTITION=gpu-a100 SE_GPU_CONSTRAINT=A100-80GB`. These are requests, not measured requirements.
+
+Stages run in order: preparation, a 2-epoch GPU probe using all selected observations, main training, k=30/50/100 DSE, and a results notebook. Dependencies require successful prior stages. Main training uses `epochs` and `patience` from the config. Reusing a run directory is rejected; choose a new output path for a new experiment.
+
+Outputs include `sample_summary.csv`, QC and gene diagnostics CSVs, selected-raw/prepared/embedded H5ADs, model state, FDR/statistic/failure-mask CSVs, DSE pickles, per-stage JSON/time/memory logs, and an executed `results.ipynb` with figures. The notebook shows original versus actual observation counts, spatial coverage, counts and detected genes, embeddings, failed fits, k sensitivity, and gene expression and fitted results. Only open pickles from trusted runs.
+
+## Interpretation and Limits
+
+95% clipping can erase a gene's entire training signal in a sample when a sparse gene's upper bound is 0. This can occur after initial 4SD removal and is recorded in `gene_qc_*.csv` and the notebook. This release retains both agreed preprocessing steps and does not automatically substitute another method.
+
+Slide-seq observations may be beads and Stereo-seq observations may be spatial bins; observation counts are not necessarily individual-cell counts. Inputs with only Brain tissue annotations are not assigned inferred detailed cell types for visualization. No additional mitochondrial-percentage or detection thresholds are applied automatically; inspect distributions and sample-specific QC before deciding.
+
+FDR-range, failure-handling, and observation-preservation checks establish numerical behavior, not biological validity. Review k stability and spatial-neighbor recall as well. A comparison with 1 biological sample per condition does not replace replicated population-level inference.

spaceexpress-0.1.5/RELEASE_0.1.5.md

@@ -0,0 +1,45 @@
+# SpaceExpress 0.1.5
+
+English version: [RELEASE_0.1.5-en.md](RELEASE_0.1.5-en.md)
+
+## 변경 사항
+
+- 실패한 유전자-embedding 차원 적합(음수 또는 비유한 통계량)의 FDR은 1입니다. 유효한 empirical-null 적합이 불가능한 차원도 FDR=1로 반환합니다.
+- 최종 FDR을 [0, 1]로 제한합니다. 기존 empirical-null 추정식을 다른 보정법으로 교체하지 않습니다.
+- `select_hvg_after_outlier`는 정규화·log 변환된 공통 유전자에서 두 표본을 합친 평균 + 4 표본 표준편차 이상의 값을 0으로 바꾼 뒤 batch-aware HVG를 선택합니다. 입력 객체 대신 복사본을 수정합니다.
+- 초기 전처리 이력을 `.uns['spaceexpress_preprocessing']`에 저장합니다. 모든 입력에 이 이력이 있으면 DSE에서 중복 평균+4SD 제거를 건너뜁니다. 이력이 없는 기존 입력은 기존 동작을 유지하고, 이력이 섞여 있으면 명시적 설정을 요구합니다.
+- `SpaceExpress_DSE(..., remove_mean_sd_outliers=False)`로 후속 평균+4SD 제거를 명시적으로 끌 수도 있습니다. 학습 함수의 기존 95% clipping은 변경하지 않았습니다. 다중 반복 표본 DSE의 별도 99% 필터도 유지합니다.
+- 반환 AnnData의 `.varm`에 `DSE-statistic`, `DSE-fit-failed`를 추가하여 실패를 0 예측값으로 추측하지 않고 확인할 수 있습니다.
+
+## 설치
+
+Python 3.11 이상과 R이 필요합니다. R 패키지 `lmtest`, `fitdistrplus`, `dplyr`, `lme4`를 먼저 설치하고 Python에서 R 공유 라이브러리를 찾을 수 있도록 설정하십시오. Python 의존성은 패키지 메타데이터에 선언했습니다. CUDA에 맞는 PyTorch 설치는 사용하는 GPU 환경을 따릅니다.
+
+```bash
+Rscript -e 'install.packages(c("lmtest", "fitdistrplus", "dplyr", "lme4"), repos="https://cloud.r-project.org")'
+python -m pip install 'spaceexpress[notebooks]==0.1.5'
+```
+
+`examples/install_release.sh BASE_PYTHON NEW_ENV`는 기존 과학 계산·R 환경을 공유하는 새 venv에 PyPI 릴리스를 설치합니다. 패키지 다운로드 출처는 `pip_install_report.json`, 의존성은 `pip_freeze.txt`에 기록합니다. 전체 의존성을 처음부터 새로 설치하는 격리 환경과는 다릅니다.
+
+## 데이터 실행
+
+`examples/public_pair.py`는 config 기준 상대 경로를 사용하며 원본을 덮어쓰지 않습니다. 원본 count에서 최소 3개 검출 유전자·유한 spatial 좌표를 확인하고, 필요하면 공간 구획별 비례 표집합니다. 두 표본에서 각각 3개 이상 관측치에 검출된 공통 유전자를 정규화(총량 10,000)하고 log1p 후 위 전처리로 HVG 200개를 선택합니다. 원본 선택 데이터와 QC 수치도 저장합니다.
+
+```bash
+bash examples/submit_public_pair.sh /path/to/env/bin/python /path/to/config_v015.json /path/to/results_v015
+```
+
+Slurm 기본 자원은 전처리 12 CPU/128 GB, 학습 H100 80 GB 1장·8 CPU/256 GB RAM, DSE 작업당 12 CPU/128 GB, 노트북 4 CPU/64 GB입니다. 각 작업의 시간 한도는 4시간입니다. `SE_ACCOUNT`, `SE_QOS`, `SE_CPU_PARTITION`, `SE_GPU_PARTITION`, `SE_GPU_CONSTRAINT`로 클러스터 설정을 변경할 수 있습니다. CPU 기본 파티션은 `cpu-medium`입니다. A100 사용 시 `SE_GPU_PARTITION=gpu-a100 SE_GPU_CONSTRAINT=A100-80GB`를 설정하십시오. 이는 요청량이며 측정된 필요량이 아닙니다.
+
+전처리, 전체 관측치 2-epoch GPU probe, 본 학습, k=30/50/100 DSE, 결과 노트북 순서로 실행합니다. 앞 단계 성공을 의존 조건으로 사용합니다. 본 학습은 config의 `epochs`, `patience`를 사용합니다. 실행 폴더 재사용은 거부하므로 새 실험에는 새 출력 경로를 지정하십시오.
+
+결과에는 `sample_summary.csv`, QC 및 유전자 진단 CSV, 선택 원본·전처리·embedding H5AD, 모델 상태, FDR·통계량·실패 마스크 CSV, DSE pickle, 단계별 JSON·시간·메모리 로그, 그림이 포함된 실행 완료 `results.ipynb`가 포함됩니다. Notebook에서는 원본 대비 실제 관측치 수, 공간 분포, counts·검출 유전자 수, embedding, 실패 적합, k 민감도, 유전자 발현과 적합 결과를 확인합니다. Pickle은 신뢰하는 실행에서 생성한 파일만 여십시오.
+
+## 해석과 한계
+
+95% clipping은 희소 유전자의 상한이 0이 되면 그 표본에서 해당 유전자의 학습 신호 전체를 없앨 수 있습니다. 초기 4SD 제거 이후에도 발생할 수 있으며, `gene_qc_*.csv`와 notebook에 기록합니다. 이번 릴리스는 합의된 두 전처리를 유지하므로 이를 자동으로 다른 방법으로 바꾸지 않습니다.
+
+Slide-seq는 bead, Stereo-seq는 spatial bin일 수 있으므로 관측치 수를 실제 단일 세포 수와 동일시하지 않습니다. Brain tissue annotation만 있는 입력에서 세부 세포 유형을 추정해 표시하지 않습니다. 미토콘드리아 비율이나 검출량에 추가 임계값을 자동 적용하지 않으며, 분포와 표본별 QC를 검토한 뒤 결정해야 합니다.
+
+FDR 범위·실패 처리·관측치 보존 검사는 수치적 동작 검사이지 생물학적 검증이 아닙니다. k 안정성과 공간 이웃 보존율도 함께 검토하십시오. 조건당 생물학적 표본 1개인 비교는 반복 표본 기반의 집단 수준 추론을 대신하지 못합니다.

spaceexpress-0.1.5/examples/install_release.sh

@@ -0,0 +1,14 @@
+#!/bin/bash
+set -euo pipefail
+BASE_PYTHON=${1:?Pass a Python 3.11+ executable with the scientific/R environment}
+ENV_DIR=${2:?Pass a new environment directory}
+if [[ -e "$ENV_DIR" ]]; then
+  echo "Refusing to overwrite an existing environment: $ENV_DIR" >&2
+  exit 1
+fi
+"$BASE_PYTHON" -m venv --system-site-packages "$ENV_DIR"
+"$ENV_DIR/bin/python" -m pip install --index-url https://pypi.org/simple \
+  --no-cache-dir --report "$ENV_DIR/pip_install_report.json" 'spaceexpress[notebooks]==0.1.5'
+"$ENV_DIR/bin/python" -c 'import importlib.metadata as m, SpaceExpress; assert m.version("spaceexpress") == "0.1.5"; print(m.version("spaceexpress"), SpaceExpress.__file__)'
+"$ENV_DIR/bin/python" -m pip freeze > "$ENV_DIR/pip_freeze.txt"
+"$ENV_DIR/bin/python" -m pip check

spaceexpress-0.1.5/examples/public_pair.py

@@ -0,0 +1,306 @@
+#!/usr/bin/env python3
+"""Reproducible two-sample run; paths in a config are relative to that config."""
+from __future__ import annotations
+
+import argparse
+import csv
+import hashlib
+import importlib.metadata
+import json
+import os
+import pickle
+import subprocess
+import sys
+import time
+import traceback
+from pathlib import Path
+
+import anndata as ad
+import numpy as np
+import pandas as pd
+import scanpy as sc
+import scipy.sparse as sp
+import torch
+from scipy.sparse.csgraph import connected_components
+from sklearn.neighbors import kneighbors_graph
+
+import SpaceExpress as se
+
+
+def write_json(path, value):
+    path = Path(path)
+    temporary = path.with_suffix(path.suffix + '.tmp')
+    temporary.write_text(json.dumps(value, indent=2, sort_keys=True) + '\n')
+    temporary.replace(path)
+
+
+def balanced_positions(frame, n, seed):
+    if len(frame) < n:
+        raise ValueError(f'Requested {n} observations but only {len(frame)} pass QC')
+    if len(frame) == n:
+        return frame.index.to_numpy()
+    work = frame.copy()
+    for axis in ('x', 'y'):
+        values = work[axis].to_numpy(float)
+        edges = np.linspace(values.min(), values.max(), 11)
+        work[f'_{axis}bin'] = np.clip(np.digitize(values, edges[1:-1]), 0, 9)
+    rng = np.random.default_rng(seed)
+    selected = []
+    for indices in work.groupby(['_xbin', '_ybin'], observed=True).groups.values():
+        quota = int(np.floor(n * len(indices) / len(work)))
+        selected.extend(rng.choice(np.asarray(indices), quota, replace=False).tolist())
+    pool = work.index[~work.index.isin(selected)].to_numpy()
+    selected.extend(rng.choice(pool, n-len(selected), replace=False).tolist())
+    return np.asarray(selected)
+
+
+def count_adata(matrix, obs, genes, coords):
+    matrix = sp.csr_matrix(matrix, dtype=np.float32)
+    if not np.isfinite(matrix.data).all() or (matrix.data < 0).any():
+        raise ValueError('Raw counts must be finite and nonnegative')
+    if not np.allclose(matrix.data, np.rint(matrix.data)):
+        raise ValueError('Expected raw integer counts, not normalized expression')
+    result = ad.AnnData(matrix, obs=obs.copy(), var=pd.DataFrame(index=genes))
+    if not result.obs_names.is_unique or not result.var_names.is_unique:
+        raise ValueError('Duplicate observation or gene names in raw input')
+    result.obsm['spatial'] = np.asarray(coords, dtype=np.float32)
+    result.obs['total_counts'] = np.asarray(matrix.sum(axis=1)).ravel()
+    result.obs['n_genes_by_counts'] = np.asarray((matrix > 0).sum(axis=1)).ravel()
+    mt = result.var_names.str.upper().str.startswith('MT-')
+    result.obs['pct_counts_mt'] = 100 * np.asarray(matrix[:, mt].sum(axis=1)).ravel() / np.maximum(result.obs['total_counts'], 1)
+    return result
+
+
+def load_slide(sample):
+    locations = pd.read_csv(sample['locations']).set_index('barcode')
+    labels = pd.read_csv(sample['cell_types']).drop(columns=['Unnamed: 0'], errors='ignore').set_index('barcode')
+    if not locations.index.is_unique or not labels.index.is_unique:
+        raise ValueError('Duplicate barcodes in locations or cell types')
+    with open(sample['dge'], newline='') as handle:
+        genes = next(csv.reader(handle))[1:]
+    matrices, barcodes = [], []
+    for chunk in pd.read_csv(sample['dge'], chunksize=256,
+                             dtype={g: np.float32 for g in genes}):
+        barcodes.extend(chunk['barcode'].astype(str))
+        matrices.append(sp.csr_matrix(chunk[genes].to_numpy(dtype=np.float32)))
+    matrix = sp.vstack(matrices, format='csr')
+    obs = pd.DataFrame(index=pd.Index(barcodes, name='barcode'))
+    obs = obs.join(locations[['x', 'y']]).join(labels[['max_cell_type']])
+    obs['annotation'] = obs['max_cell_type'].astype('string').fillna('Unknown').astype(str)
+    return count_adata(matrix, obs, genes, obs[['x', 'y']].to_numpy())
+
+
+def load_stereo(sample, root, index):
+    backed = ad.read_h5ad(sample['h5ad'], backed='r')
+    try:
+        full_obs = backed.obs.copy()
+        full_obs[['x', 'y']] = np.asarray(backed.obsm['spatial'])[:, :2]
+        full_obs.to_csv(root / f'source_observations_{index}.csv')
+        mask = full_obs['annotation'].astype(str).eq(sample['annotation']).to_numpy()
+        sub = backed[np.flatnonzero(mask), :].to_memory()
+        source_count = backed.n_obs
+    finally:
+        backed.file.close()
+    counts = sub.layers['count']
+    result = count_adata(counts, sub.obs, sub.var_names, sub.obsm['spatial'])
+    return result, source_count
+
+
+def graph_policy(pair):
+    scans, choices = [], []
+    for a in pair:
+        counts = {}
+        for k in range(4, 21):
+            graph = kneighbors_graph(a.obsm['spatial'], k, include_self=False)
+            counts[k] = int(connected_components(graph.maximum(graph.T), directed=False)[0])
+        choices.append(min(counts, key=lambda k: (counts[k], k)))
+        scans.append(counts)
+    k = max(choices)
+    return k, [{'components_by_k': scan, 'components_at_selected_k': scan[k]} for scan in scans]
+
+
+def prepare(config, root, status):
+    pair, summaries = [], []
+    for i, (sample, target) in enumerate(zip(config['samples'], config['target_spots'])):
+        if config['type'] == 'slide':
+            a = load_slide(sample)
+            source_count = a.n_obs
+        else:
+            a, source_count = load_stereo(sample, root, i)
+        selected_tissue_count = a.n_obs
+        coords = a.obsm['spatial']
+        eligible = np.isfinite(coords).all(axis=1) & (a.obs['n_genes_by_counts'].to_numpy() >= 3)
+        frame = a.obs.copy()
+        frame[['x', 'y']] = coords
+        frame['qc_pass'] = eligible
+        candidates = pd.DataFrame(coords[eligible], index=np.flatnonzero(eligible), columns=['x', 'y'])
+        chosen = np.sort(balanced_positions(candidates, target, config['seed'] + i))
+        frame['selected_for_training'] = False
+        frame.iloc[chosen, frame.columns.get_loc('selected_for_training')] = True
+        frame.to_csv(root / f'qc_observations_{i}.csv')
+        a = a[chosen].copy()
+        a.obs['source_sample'] = sample['name']
+        a.obs['condition'] = sample['condition']
+        a.layers['counts'] = a.X.copy()
+        a.write_h5ad(root / f'raw_selected_{i}.h5ad', compression='gzip')
+        summaries.append({'sample': sample['name'], 'original_observations': source_count,
+                          'tissue_observations': selected_tissue_count,
+                          'qc_observations': int(eligible.sum()), 'training_observations': a.n_obs,
+                          'original_genes': a.n_vars})
+        pair.append(a)
+    common = sorted(set(pair[0].var_names).intersection(pair[1].var_names))
+    pair = [a[:, common].copy() for a in pair]
+    keep = np.logical_and.reduce([np.asarray((a.X > 0).sum(axis=0)).ravel() >= 3 for a in pair])
+    pair = [a[:, keep].copy() for a in pair]
+    for a in pair:
+        sc.pp.normalize_total(a, target_sum=1e4)
+        sc.pp.log1p(a)
+    pair, hvg, diagnostics = se.select_hvg_after_outlier(pair, n_top_genes=config['n_hvg'], z_threshold=4)
+    graph_k, graph_scan = graph_policy(pair)
+    clipping = []
+    for i, a in enumerate(pair):
+        dense = a.X.toarray()
+        q95 = np.percentile(dense, 95, axis=0)
+        zeros = a.var_names[(q95 == 0) & (dense.max(axis=0) > 0)].tolist()
+        clipping.append({'sample': config['samples'][i]['name'],
+                         'genes_zeroed_by_embedding_q95': zeros,
+                         'genes_already_all_zero_after_4sd': a.var_names[dense.max(axis=0) == 0].tolist()})
+        pd.DataFrame({'gene': a.var_names, 'embedding_q95': q95,
+                      'positive_observations_after_4sd': (dense > 0).sum(axis=0)}).to_csv(root / f'gene_qc_{i}.csv', index=False)
+        if not np.isfinite(dense).all():
+            raise ValueError('Nonfinite prepared expression')
+        a.write_h5ad(root / f'prepared_condition{i}.h5ad', compression='gzip')
+    pd.Series(hvg, name='gene').to_csv(root / 'hvg_genes.csv', index=False)
+    pd.DataFrame(summaries).to_csv(root / 'sample_summary.csv', index=False)
+    status.update(samples=summaries, hvg_preprocessing=diagnostics, clipping=clipping,
+                  graph_k=graph_k, graph_scan=graph_scan)
+
+
+def embed(config, root, status, probe=False):
+    if not torch.cuda.is_available():
+        raise RuntimeError('CUDA required for the full-size embedding stages')
+    pair = [ad.read_h5ad(root / f'prepared_condition{i}.h5ad') for i in range(2)]
+    if [a.n_obs for a in pair] != config['target_spots']:
+        raise ValueError('Prepared observation counts do not match config')
+    graph_k = json.loads((root / 'prepare_status.json').read_text())['graph_k']
+    shortest = []
+    for i, a in enumerate(pair):
+        path = root / f'shortest_condition{i}.pkl'
+        if not path.exists():
+            temporary = path.with_suffix('.partial')
+            if temporary.exists():
+                raise RuntimeError(f'Incomplete shortest paths exist: {temporary}; inspect before retrying')
+            se.shortest_path(a.obsm['spatial'], str(temporary), k=graph_k)
+            temporary.replace(path)
+        shortest.append(str(path))
+    torch.cuda.reset_peak_memory_stats()
+    start = time.perf_counter()
+    epochs = 2 if probe else config['epochs']
+    embeddings, model = se.train_SpaceExpress_multi(pair, shortest, epochs=epochs,
+        patience=config['patience'], emb_dim=4, hid_dim=32, random_seed=config['seed'], save_model=True)
+    status.update(device=torch.cuda.get_device_name(), epochs_requested=epochs,
+                  training_seconds=time.perf_counter()-start,
+                  peak_cuda_allocated_gib=torch.cuda.max_memory_allocated()/2**30,
+                  peak_cuda_reserved_gib=torch.cuda.max_memory_reserved()/2**30,
+                  device_memory_gib=torch.cuda.get_device_properties(0).total_memory/2**30)
+    for a, emb in zip(pair, embeddings):
+        if emb.shape != (a.n_obs, 4) or not np.isfinite(emb).all() or np.any(emb.std(axis=0) < 1e-8):
+            raise ValueError('Nonfinite, collapsed, or incorrectly sized embedding')
+    status['embedding_shapes'] = [list(e.shape) for e in embeddings]
+    if not probe:
+        for i, (a, emb) in enumerate(zip(pair, embeddings)):
+            a.obsm['SpaceExpress'] = emb
+            a.write_h5ad(root / f'embedded_condition{i}.h5ad', compression='gzip')
+        with (root / 'embedding.pkl').open('wb') as handle:
+            pickle.dump(embeddings, handle)
+        torch.save(model.state_dict(), root / 'model_state.pt')
+
+
+def dse(config, root, status, k):
+    out = root / f'k{k}'
+    out.mkdir(exist_ok=True)
+    pair = [ad.read_h5ad(root / f'embedded_condition{i}.h5ad') for i in range(2)]
+    embeddings = [a.obsm['SpaceExpress'] for a in pair]
+    fdr, fitted = se.SpaceExpress_DSE(embeddings, pair, k=k,
+        n_jobs=int(os.environ.get('SLURM_CPUS_PER_TASK', '4')), multi=False)
+    values = fdr.to_numpy(float)
+    failed = fitted[0].varm['DSE-fit-failed'].to_numpy().T
+    if not np.isfinite(values).all() or ((values < 0) | (values > 1)).any():
+        raise ValueError('Invalid FDR range')
+    if not np.all(values[failed] == 1):
+        raise ValueError('Failed fits did not receive FDR=1')
+    if any(a.uns['spaceexpress_dse']['remove_mean_sd_outliers'] for a in fitted):
+        raise ValueError('Duplicate mean+4SD removal was unexpectedly enabled')
+    if not all(np.isfinite(a.obsm[key]).all() for a in fitted for key in ['DSE-pred', 'DSE-inter']):
+        raise ValueError('Nonfinite predictions')
+    if failed.all():
+        raise ValueError('All DSE fits failed')
+    fdr.to_csv(out / 'fdr.csv')
+    fitted[0].varm['DSE-statistic'].to_csv(out / 'test_statistics.csv')
+    fitted[0].varm['DSE-fit-failed'].to_csv(out / 'fit_failed.csv')
+    with (out / 'result.pkl').open('wb') as handle:
+        pickle.dump([fdr, fitted], handle, protocol=pickle.HIGHEST_PROTOCOL)
+    genes = fdr.columns[(fdr < 0.001).any(axis=0)].tolist()
+    status.update(k=k, fdr_min=float(values.min()), fdr_max=float(values.max()),
+                  failed_gene_dimensions=int(failed.sum()),
+                  failed_genes=fdr.columns[failed.any(axis=0)].tolist(),
+                  dse_gene_count_0_001=len(genes), dse_genes_0_001=genes,
+                  repeated_mean_sd_removal=False)
+
+
+def main():
+    parser = argparse.ArgumentParser()
+    parser.add_argument('--config', type=Path, required=True)
+    parser.add_argument('--output', type=Path, required=True)
+    parser.add_argument('--stage', choices=['prepare', 'probe', 'embed', 'dse', 'report'], required=True)
+    parser.add_argument('--k', type=int, choices=[30, 50, 100], default=30)
+    args = parser.parse_args()
+    config_path = args.config.resolve()
+    config = json.loads(config_path.read_text())
+    for sample in config['samples']:
+        for key in ('dge', 'locations', 'cell_types', 'h5ad'):
+            if key in sample:
+                sample[key] = str((config_path.parent / sample[key]).resolve())
+    root = args.output.resolve()
+    root.mkdir(parents=True, exist_ok=True)
+    stage = f'dse_k{args.k}' if args.stage == 'dse' else args.stage
+    status_path = root / f'{stage}_status.json'
+    if status_path.exists() and json.loads(status_path.read_text())['status'] == 'PASS':
+        raise RuntimeError(f'{stage} already completed; use a new output directory for a fresh run')
+    version = importlib.metadata.version('spaceexpress')
+    if version != '0.1.5':
+        raise RuntimeError(f'Expected installed release 0.1.5, found {version}')
+    snapshot = root / 'config_resolved.json'
+    if snapshot.exists():
+        if json.loads(snapshot.read_text()) != config:
+            raise ValueError('Config changed within a run; choose a new output directory')
+    else:
+        write_json(snapshot, config)
+        (root / 'pip_freeze.txt').write_text(subprocess.check_output([sys.executable, '-m', 'pip', 'freeze'], text=True))
+    sources = {p.name: hashlib.sha256(p.read_bytes()).hexdigest() for p in Path(se.__file__).parent.glob('*.py')}
+    status = {'status': 'RUNNING', 'stage': stage, 'version': version,
+              'python': sys.executable, 'package_path': se.__file__, 'source_sha256': sources,
+              'slurm_job_id': os.environ.get('SLURM_JOB_ID')}
+    write_json(status_path, status)
+    start = time.perf_counter()
+    try:
+        if args.stage == 'prepare':
+            prepare(config, root, status)
+        elif args.stage in ('probe', 'embed'):
+            embed(config, root, status, probe=args.stage == 'probe')
+        elif args.stage == 'dse':
+            dse(config, root, status, args.k)
+        else:
+            from public_pair_notebook import build_and_execute
+            build_and_execute(root)
+        status['status'] = 'PASS'
+    except Exception as exc:
+        status.update(status='ERROR', error=repr(exc), traceback=traceback.format_exc())
+        raise
+    finally:
+        status['elapsed_seconds'] = time.perf_counter()-start
+        write_json(status_path, status)
+
+
+if __name__ == '__main__':
+    main()