cellucid 0.0.1a0__tar.gz

cellucid-0.0.1a0/PKG-INFO

@@ -0,0 +1,105 @@
+Metadata-Version: 2.4
+Name: cellucid
+Version: 0.0.1a0
+Summary: Utilities to export data for the 3D UMAP WebGL viewer.
+Author: cellucid contributors
+License: Proprietary
+Project-URL: Repository, https://github.com/theislab/cellucid-data
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3 :: Only
+Classifier: License :: Other/Proprietary License
+Classifier: Intended Audience :: Science/Research
+Classifier: Topic :: Scientific/Engineering :: Visualization
+Classifier: Topic :: Scientific/Engineering :: Bio-Informatics
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+Requires-Dist: numpy>=1.22
+Requires-Dist: pandas>=1.5
+Requires-Dist: scipy>=1.9
+Requires-Dist: tqdm>=4.66
+Provides-Extra: analysis
+Requires-Dist: anndata>=0.10; extra == "analysis"
+Requires-Dist: scanpy>=1.10; extra == "analysis"
+Provides-Extra: docs
+Requires-Dist: sphinx>=7.2; extra == "docs"
+Requires-Dist: myst-parser>=2.0; extra == "docs"
+Requires-Dist: sphinx-autodoc-typehints>=1.25; extra == "docs"
+Requires-Dist: sphinx_rtd_theme>=1.3; extra == "docs"
+
+<p>
+  <img src="cellucid-logo.svg" alt="Cellucid logo" width="320">
+</p>
+
+[![PyPI version](https://img.shields.io/pypi/v/cellucid.svg)](https://pypi.org/project/cellucid/)
+[![Python versions](https://img.shields.io/pypi/pyversions/cellucid.svg)](https://pypi.org/project/cellucid/)
+[![Documentation Status](https://readthedocs.org/projects/cellucid/badge/?version=latest)](https://cellucid.readthedocs.io/en/latest/)
+
+# cellucid (Python)
+
+Python utilities for exporting AnnData/numpy payloads into the manifest + binary format consumed by the Cellucid WebGL viewer. Everything here is independent of the website; generate data wherever you like, then copy the resulting files into the viewer’s `assets/exports/` directory when you host the site.
+
+## Layout (repo: theislab/cellucid-data)
+- `src/cellucid/prepare_data.py`: `export_data_for_web` writes points/obs/var/connectivity binaries into an `exports/` directory (configurable).
+- `src/cellucid/anndata_variations.py`: UMAP sweep helper for experimentation (uses Scanpy).
+- `src/cellucid/repo_to_text.py`: utility to dump repository contents to a single text file.
+- `notebooks/`: exploratory notebook that demonstrates exporting viewer assets.
+- `data/`: example/raw inputs (h5ad, pickles) and experiment outputs live here; this folder stays untracked by default.
+
+## Installation
+Requires Python 3.10+.
+
+From the repo root:
+
+```bash
+pip install -e python
+# or with optional extras
+pip install -e python[analysis,docs]
+```
+
+The package name is `cellucid` and imports under `cellucid`.
+
+## Usage
+```python
+import scanpy as sc
+from cellucid.prepare_data import export_data_for_web
+
+adata = sc.read_h5ad("path/to/adata.h5ad")
+export_data_for_web(
+    X_umap=adata.obsm["X_umap"],
+    latent_space=adata.X,
+    obs=adata.obs,
+    var=adata.var,
+    gene_expression=adata.X,
+    connectivities=adata.obsp.get("connectivities"),
+    out_dir="exports",  # default; change as needed
+    var_quantization=8,
+    obs_continuous_quantization=8,
+    compression=6,
+)
+```
+
+The exporter defaults to `exports/` under your current working directory. After generation, copy or sync that folder into the website’s `assets/exports/` directory so the viewer can load the files.
+
+To run the UMAP sweep helper:
+
+```bash
+python -m cellucid.anndata_variations
+```
+
+Ensure `python/data/raw` contains the expected h5ad/pickle files or override the module constants before calling. Paths in the notebook and sweep helper assume the `data/` directory inside this `python/` project; point them to your own locations as needed.
+
+## Companion web viewer (theislab/cellucid)
+The static viewer lives in a separate repository (`theislab/cellucid`). After running `export_data_for_web`, copy or sync the resulting `exports/` directory into that repo’s `assets/exports/` folder before hosting. The manifest/binary schema is shared; no build step is required on the web side.
+
+## Documentation
+- Build locally: `pip install -e python[docs] && sphinx-build -b html docs docs/_build/html`
+- Read the Docs: `python/.readthedocs.yaml` targets `docs/conf.py` with the `docs` extra.
+
+## Publishing
+Install packaging tools if needed: `pip install build twine`
+
+1. From `python/`, bump the version in `pyproject.toml`.
+2. Build: `python -m build`
+3. Upload to PyPI (or TestPyPI) with `twine upload dist/*`.
+
+Update the `license`, `authors`, and `Repository` URL in `pyproject.toml` before publishing if needed.

cellucid-0.0.1a0/README.md

@@ -0,0 +1,77 @@
+<p>
+  <img src="cellucid-logo.svg" alt="Cellucid logo" width="320">
+</p>
+
+[![PyPI version](https://img.shields.io/pypi/v/cellucid.svg)](https://pypi.org/project/cellucid/)
+[![Python versions](https://img.shields.io/pypi/pyversions/cellucid.svg)](https://pypi.org/project/cellucid/)
+[![Documentation Status](https://readthedocs.org/projects/cellucid/badge/?version=latest)](https://cellucid.readthedocs.io/en/latest/)
+
+# cellucid (Python)
+
+Python utilities for exporting AnnData/numpy payloads into the manifest + binary format consumed by the Cellucid WebGL viewer. Everything here is independent of the website; generate data wherever you like, then copy the resulting files into the viewer’s `assets/exports/` directory when you host the site.
+
+## Layout (repo: theislab/cellucid-data)
+- `src/cellucid/prepare_data.py`: `export_data_for_web` writes points/obs/var/connectivity binaries into an `exports/` directory (configurable).
+- `src/cellucid/anndata_variations.py`: UMAP sweep helper for experimentation (uses Scanpy).
+- `src/cellucid/repo_to_text.py`: utility to dump repository contents to a single text file.
+- `notebooks/`: exploratory notebook that demonstrates exporting viewer assets.
+- `data/`: example/raw inputs (h5ad, pickles) and experiment outputs live here; this folder stays untracked by default.
+
+## Installation
+Requires Python 3.10+.
+
+From the repo root:
+
+```bash
+pip install -e python
+# or with optional extras
+pip install -e python[analysis,docs]
+```
+
+The package name is `cellucid` and imports under `cellucid`.
+
+## Usage
+```python
+import scanpy as sc
+from cellucid.prepare_data import export_data_for_web
+
+adata = sc.read_h5ad("path/to/adata.h5ad")
+export_data_for_web(
+    X_umap=adata.obsm["X_umap"],
+    latent_space=adata.X,
+    obs=adata.obs,
+    var=adata.var,
+    gene_expression=adata.X,
+    connectivities=adata.obsp.get("connectivities"),
+    out_dir="exports",  # default; change as needed
+    var_quantization=8,
+    obs_continuous_quantization=8,
+    compression=6,
+)
+```
+
+The exporter defaults to `exports/` under your current working directory. After generation, copy or sync that folder into the website’s `assets/exports/` directory so the viewer can load the files.
+
+To run the UMAP sweep helper:
+
+```bash
+python -m cellucid.anndata_variations
+```
+
+Ensure `python/data/raw` contains the expected h5ad/pickle files or override the module constants before calling. Paths in the notebook and sweep helper assume the `data/` directory inside this `python/` project; point them to your own locations as needed.
+
+## Companion web viewer (theislab/cellucid)
+The static viewer lives in a separate repository (`theislab/cellucid`). After running `export_data_for_web`, copy or sync the resulting `exports/` directory into that repo’s `assets/exports/` folder before hosting. The manifest/binary schema is shared; no build step is required on the web side.
+
+## Documentation
+- Build locally: `pip install -e python[docs] && sphinx-build -b html docs docs/_build/html`
+- Read the Docs: `python/.readthedocs.yaml` targets `docs/conf.py` with the `docs` extra.
+
+## Publishing
+Install packaging tools if needed: `pip install build twine`
+
+1. From `python/`, bump the version in `pyproject.toml`.
+2. Build: `python -m build`
+3. Upload to PyPI (or TestPyPI) with `twine upload dist/*`.
+
+Update the `license`, `authors`, and `Repository` URL in `pyproject.toml` before publishing if needed.

cellucid-0.0.1a0/pyproject.toml

@@ -0,0 +1,44 @@
+[build-system]
+requires = ["setuptools>=68"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "cellucid"
+version = "0.0.1a0"
+description = "Utilities to export data for the 3D UMAP WebGL viewer."
+readme = "README.md"
+requires-python = ">=3.10"
+license = { text = "Proprietary" }
+authors = [{ name = "cellucid contributors" }]
+dependencies = [
+    "numpy>=1.22",
+    "pandas>=1.5",
+    "scipy>=1.9",
+    "tqdm>=4.66",
+]
+classifiers = [
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3 :: Only",
+    "License :: Other/Proprietary License",
+    "Intended Audience :: Science/Research",
+    "Topic :: Scientific/Engineering :: Visualization",
+    "Topic :: Scientific/Engineering :: Bio-Informatics",
+]
+
+[project.optional-dependencies]
+analysis = [
+    "anndata>=0.10",
+    "scanpy>=1.10",
+]
+docs = [
+    "sphinx>=7.2",
+    "myst-parser>=2.0",
+    "sphinx-autodoc-typehints>=1.25",
+    "sphinx_rtd_theme>=1.3",
+]
+
+[project.urls]
+Repository = "https://github.com/theislab/cellucid-data"
+
+[tool.setuptools.packages.find]
+where = ["src"]

cellucid-0.0.1a0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

cellucid-0.0.1a0/src/cellucid/__init__.py

@@ -0,0 +1,12 @@
+"""Utilities for exporting AnnData/array data for the 3D UMAP web viewer."""
+
+from importlib import metadata as _metadata
+
+try:
+    __version__ = _metadata.version("cellucid")
+except _metadata.PackageNotFoundError:  # pragma: no cover - handled in dev installs
+    __version__ = "0.0.0"
+
+from .prepare_data import export_data_for_web
+
+__all__ = ["export_data_for_web", "__version__"]

cellucid-0.0.1a0/src/cellucid/anndata_variations.py

@@ -0,0 +1,149 @@
+import gc
+import pandas as pd
+import itertools
+from pathlib import Path
+from tqdm import tqdm
+
+try:
+    import scanpy as sc
+except ImportError as exc:  # pragma: no cover - optional dependency
+    raise ImportError(
+        "scanpy is required for cellucid.anndata_variations; install with "
+        "`pip install cellucid[analysis]` or add scanpy to your environment."
+    ) from exc
+
+def _find_repo_root() -> Path:
+    """Locate the python project root (contains pyproject.toml and data/)."""
+    for candidate in Path(__file__).resolve().parents:
+        if (candidate / "pyproject.toml").exists() and (candidate / "data").exists():
+            return candidate
+    return Path.cwd()
+
+
+REPO_ROOT = _find_repo_root()
+RAW_DATA_DIR = REPO_ROOT / "data" / "raw"
+EXPERIMENT_DIR = REPO_ROOT / "data" / "experiments"
+
+INPUT_H5AD = RAW_DATA_DIR / "adata_latent_scvi-adata_unified_20250925_001_integration-supergpu02.scidom.de-bb886d36409e78a3.h5ad"
+METADATA_PICKLE = RAW_DATA_DIR / "adata_obs_extracted.pickle"
+OUTPUT_DIR = EXPERIMENT_DIR / "umap_parameter_sweep"
+
+MIN_DISTS = [0.1, 0.15, 0.20, 0.25, 0.30, 0.35, 0.40, 0.45, 0.50]
+N_NEIGHBORS_LIST = [5, 10, 15, 30, 50, 90, 120]
+RANDOM_SEED = 42
+
+
+def _print_setup():
+    """Log resolved paths so it's obvious what is being used."""
+    print("==== AnnData UMAP sweep setup ====")
+    print(f"Current working dir : {Path.cwd()}")
+    print(f"Repo root           : {REPO_ROOT}")
+    print(f"Input h5ad          : {INPUT_H5AD} (exists: {INPUT_H5AD.exists()})")
+    print(f"Metadata pickle     : {METADATA_PICKLE} (exists: {METADATA_PICKLE.exists()})")
+    print(f"Output directory    : {OUTPUT_DIR}")
+    print("==================================")
+
+
+def validate_inputs() -> None:
+    """Ensure required files and folders exist before heavy work."""
+    missing = [path for path in (INPUT_H5AD, METADATA_PICKLE) if not path.exists()]
+    if missing:
+        missing_list = "\n - ".join(str(path) for path in missing)
+        raise FileNotFoundError(f"Missing required input files:\n - {missing_list}")
+
+    EXPERIMENT_DIR.mkdir(parents=True, exist_ok=True)
+    OUTPUT_DIR.mkdir(parents=True, exist_ok=True)
+
+
+
+def load_and_preprocess_data():
+    validate_inputs()
+    print("Loading and preparing data (this happens only once)...")
+    adata = sc.read_h5ad(INPUT_H5AD)
+    adata_complete = pd.read_pickle(METADATA_PICKLE)
+
+    if "qc_fail" not in adata_complete.columns:
+        raise KeyError("Metadata pickle missing required column 'qc_fail'.")
+
+    missing_cells = set(adata.obs_names) - set(adata_complete.index)
+    if missing_cells:
+        raise ValueError(f"Metadata missing {len(missing_cells)} cells (e.g., {next(iter(missing_cells))}).")
+
+    adata_complete = adata_complete.loc[adata.obs_names]
+    qc_mask = ~adata_complete["qc_fail"].fillna(False)
+
+    # Apply QC filtering to both the matrix and obs
+    adata = adata[qc_mask].copy()
+    adata_complete = adata_complete.loc[qc_mask]
+    adata.obs = adata_complete
+
+    # Drop heavy derived fields to reduce per-run memory
+    for container in (adata.obsm, adata.obsp):
+        for key in list(container.keys()):
+            del container[key]
+    adata.uns.clear()
+
+    print(f"Data prepared. Final: {adata}")
+    print(f"Cells kept after QC : {adata.n_obs}")
+    print(f"Features            : {adata.n_vars}")
+    return adata
+
+
+def run_parameter_sweep(_adata):
+    """
+    Iterates through parameters, calculating UMAPs and saving results.
+    """
+    OUTPUT_DIR.mkdir(parents=True, exist_ok=True)
+    param_combinations = list(itertools.product(N_NEIGHBORS_LIST, MIN_DISTS))
+    print(f"Starting sweep over {len(param_combinations)} combinations...")
+    print(f"Writing outputs to: {OUTPUT_DIR}")
+
+    existing_before = 0
+    for k, min_dist in param_combinations:
+        dist_str = f"{min_dist:.2f}".replace('.', '_')
+        filename = OUTPUT_DIR / f"adata|k__{k}|min_dist_{dist_str}.h5ad"
+        if filename.exists():
+            existing_before += 1
+    to_compute = len(param_combinations) - existing_before
+    print(f"Found {existing_before} outputs already present; {to_compute} will be computed.")
+
+    if to_compute == 0:
+        print("All combinations already exist. Nothing to do.")
+        return
+
+    skipped = 0
+    created = 0
+
+    for k, min_dist in tqdm(param_combinations, desc="Processing"):
+        dist_str = f"{min_dist:.2f}".replace('.', '_')
+        filename = OUTPUT_DIR / f"adata|k__{k}|min_dist_{dist_str}.h5ad"
+        if filename.exists():
+            skipped += 1
+            tqdm.write(f"Exists, skipping: {filename.name}")
+            continue
+
+        adata = _adata.copy()
+        try:
+            sc.pp.neighbors(adata, n_neighbors=k, random_state=RANDOM_SEED)
+            sc.tl.umap(
+                adata, 
+                n_components=3, 
+                min_dist=min_dist, 
+                random_state=RANDOM_SEED,
+            )
+            adata.write_h5ad(filename)
+            created += 1
+        except Exception as e:
+            tqdm.write(f"Calculation error for {filename.name}: {e}")
+        gc.collect()
+
+    print(f"Finished sweep. Created {created}, skipped {skipped}, total {len(param_combinations)}.")
+
+if __name__ == "__main__":
+    try:
+        _print_setup()
+        clean_adata = load_and_preprocess_data()
+        run_parameter_sweep(clean_adata)
+        print("Processing complete.")
+    except Exception as e:
+        print(f"An error occurred: {e}")