cytome 0.2.2__tar.gz

cytome-0.2.2/LICENSE

@@ -0,0 +1,29 @@
+BSD 3-Clause License
+
+Copyright (c) 2025, Min Dai
+All rights reserved.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are met:
+
+1. Redistributions of source code must retain the above copyright notice, this
+   list of conditions and the following disclaimer.
+
+2. Redistributions in binary form must reproduce the above copyright notice,
+   this list of conditions and the following disclaimer in the documentation
+   and/or other materials provided with the distribution.
+
+3. Neither the name of the copyright holder nor the names of its
+   contributors may be used to endorse or promote products derived from
+   this software without specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
+FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
+SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
+CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

cytome-0.2.2/PKG-INFO

@@ -0,0 +1,270 @@
+Metadata-Version: 2.4
+Name: cytome
+Version: 0.2.2
+Summary: Single-file multi-omics data format for single-cell biology
+Author-email: Min Dai <dai@broadinstitute.org>
+Maintainer-email: Min Dai <dai@broadinstitute.org>
+License: BSD-3-Clause
+Project-URL: Homepage, https://github.com/genecell/cytome
+Project-URL: Source, https://github.com/genecell/cytome
+Project-URL: Documentation, https://github.com/genecell/cytome
+Keywords: single-cell,genomics,bioinformatics,scRNA-seq,scATAC-seq,multiomics,data-format,sqlite,streaming
+Classifier: Development Status :: 4 - Beta
+Classifier: Environment :: Console
+Classifier: Framework :: Jupyter
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: Natural Language :: English
+Classifier: Operating System :: MacOS :: MacOS X
+Classifier: Operating System :: Microsoft :: Windows
+Classifier: Operating System :: POSIX :: Linux
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Scientific/Engineering :: Bio-Informatics
+Classifier: Topic :: Scientific/Engineering :: Information Analysis
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: numpy>=1.24
+Requires-Dist: scipy>=1.11
+Requires-Dist: pandas>=1.5
+Requires-Dist: lz4>=4.0
+Provides-Extra: anndata
+Requires-Dist: anndata>=0.10; extra == "anndata"
+Provides-Extra: mudata
+Requires-Dist: mudata>=0.2; extra == "mudata"
+Provides-Extra: full
+Requires-Dist: anndata>=0.10; extra == "full"
+Requires-Dist: mudata>=0.2; extra == "full"
+Requires-Dist: zstandard>=0.20; extra == "full"
+Requires-Dist: pysam>=0.22; extra == "full"
+Requires-Dist: pyBigWig>=0.3; extra == "full"
+Provides-Extra: dev
+Requires-Dist: pytest>=7.0; extra == "dev"
+Requires-Dist: anndata>=0.10; extra == "dev"
+Requires-Dist: mudata>=0.2; extra == "dev"
+Requires-Dist: zstandard>=0.20; extra == "dev"
+Requires-Dist: pysam>=0.22; extra == "dev"
+Requires-Dist: pyBigWig>=0.3; extra == "dev"
+Requires-Dist: torch>=2.0; extra == "dev"
+Dynamic: license-file
+
+# cytome
+
+**A single-file format for single-cell multi-omics data.**
+
+`cytome` replaces ad-hoc file stacks with one SQLite-backed `.cytome` file that keeps matrices, metadata, embeddings, fragments, and provenance together.
+
+`cytome` stores expression matrices, cell metadata, genomic fragments, embeddings, graphs, and computational provenance in a single SQLite file. It opens from manifest metadata, supports SQL-style metadata filtering, and provides a tested CLI for conversion, merge, subset, export, and validation.
+
+`cytome` is part of the [PIASO](https://piaso.org) toolkit for single-cell analysis. PIASO natively supports cytome datasets for streaming single-cell ATAC-seq and RNA-seq workflows.
+
+## Why cytome?
+
+| Capability | AnnData (`.h5ad`) | BPCells | TileDB-SOMA | cytome (`.cytome`) |
+|---|---:|---:|---:|---:|
+| Single portable file | Yes | No (directory layout) | No (array store) | Yes |
+| Python stdlib core storage dependency | No (`h5py`) | No | No | Yes (`sqlite3`) |
+| SQL metadata queries | No | No | Limited via APIs | Yes |
+| Fragment genomic range queries | No native index | Limited | Yes | Yes (SQLite R-tree) |
+| Native provenance table | No | No | Partial workflow metadata | Yes |
+| Streaming merge API | Partial | Strong compute streaming | Distributed workflows | Yes, chunk-aware |
+| Full matrix load speed | Fast | Fast | Varies | Fast |
+| Cloud-native multi-writer scale-out | No | No | Yes | No (single-writer SQLite) |
+
+## Quick start
+
+No optional dependencies — this runs with `pip install cytome` alone.
+
+```python
+import cytome
+import numpy as np
+import pandas as pd
+import scipy.sparse as sp
+
+rng = np.random.default_rng(0)
+counts = sp.random(1000, 500, density=0.05, format="csr", dtype=np.float32,
+                   random_state=rng)
+
+ds = cytome.create("example.cytome")
+ds.set_entity("cells", pd.DataFrame({
+    "barcode": [f"cell_{i}" for i in range(1000)],
+    "cell_type": rng.choice(["A", "B"], 1000),
+}))
+ds.set_entity("genes", pd.DataFrame({"gene_id": [f"G{i}" for i in range(500)]}))
+ds.add_matrix("RNA_counts", counts)
+ds.flush()
+
+print(ds)                                    # metadata-first summary
+b_cells = ds.cells.query("cell_type == 'B'")  # SQL-style metadata query
+block = ds.RNA.counts[:100, :50]              # random access
+for start, end, chunk in ds.RNA.counts.iter_rows():
+    ...                                       # bounded-memory streaming
+ds.close()
+```
+
+### Marker genes, straight off the file
+
+[COSG](https://github.com/genecell/COSG) reads a `.cytome` in chunks, so peak
+memory does not scale with the number of cells and nothing is converted to
+AnnData first.
+
+```python
+# pip install cosg
+import cosg
+
+markers = cosg.cosg("example.cytome", groupby="cell_type",
+                    modality="RNA", layer="log1p", n_genes_user=10)
+markers["names"]   # per-cell-type marker genes
+```
+
+### Coming from AnnData
+
+```python
+# pip install "cytome[anndata]"
+ds = cytome.from_anndata(adata, modality="RNA", output="rna.cytome")
+```
+
+## Key features
+
+- Instant metadata-first open (`CytomeDataset` initialization reads manifest and table metadata)
+- SQL-queryable entity tables (`cells`, `genes`, `peaks`, `samples`)
+- Merge, subset, and downsample APIs with chunk-aware implementations
+- Fragment storage (chunked, compressed) with genomic range queries and export
+  (bulk fragment **import** is provided by [PIASO](https://piaso.org) —
+  `piaso.pp.importFragments`)
+- Analysis results live on the file: embeddings (`ds.embeddings`), neighbor
+  graphs (`ds.graphs`), per-cell / per-feature columns, and arbitrary analysis
+  artifacts (`ds.metadata`) — PIASO writes results back onto the cytome
+- Provenance logging with parameters, dependency versions, and methods-text export
+- ACID write transactions via SQLite
+- Multi-modal naming convention (`RNA_counts`, `ATAC_counts`, etc.)
+- Optional CSC feature index (`build_feature_index`) for faster column iteration
+- JSON metadata store (`ds.metadata`) for arbitrary nested analysis metadata
+- CLI (`cytome`) for convert, info, merge, subset, downsample, export, validate, provenance, and copy
+
+## Installation
+
+```bash
+pip install cytome
+```
+
+Optional extras:
+
+```bash
+pip install "cytome[full]"
+pip install "cytome[dev]"
+```
+
+## Usage examples
+
+### Convert from AnnData
+
+```python
+import cytome
+ds = cytome.from_anndata(adata, modality="RNA", output="rna.cytome")
+ds.close()
+```
+
+### Open and query
+
+```python
+import cytome
+ds = cytome.open("rna.cytome")
+cd8 = ds.cells.query("cell_type == 'CD8 T'")
+x = ds.RNA.counts[:200, :200]
+ds.close()
+```
+
+### Merge datasets
+
+```python
+import cytome
+merged = cytome.merge(["s1.cytome", "s2.cytome"], output="merged.cytome")
+merged.close()
+```
+
+### Command line
+
+```bash
+cytome info merged.cytome
+cytome merge s1.cytome s2.cytome -o merged.cytome
+cytome subset merged.cytome -o cd8.cytome --query "cell_type == 'CD8 T'"
+```
+
+### ATAC fragments
+
+```python
+ds = cytome.open("multiome.cytome")
+fr = ds.ATAC.fragments.query_region("chr1", 1_000_000, 2_000_000)
+ds.ATAC.fragments.export("fragments.tsv.gz")
+ds.close()
+```
+
+### With PIASO
+
+`cytome` integrates with [PIASO](https://piaso.org) (`pip install piaso-tools`) for
+single-cell analysis. PIASO supports cytome datasets directly — streaming normalization,
+dimensionality reduction, clustering, and visualization without loading full matrices
+into memory.
+
+PIASO functions are **self-contained on a cytome**: they stream from the file
+and write their results back onto it (graphs, embeddings, cluster labels,
+per-cell/per-feature metrics, markers) — they don't return large objects, so you
+read results from the dataset afterwards.
+
+```python
+import cytome
+import piaso
+
+# Open a cytome dataset and run the PIASO ATAC pipeline (results persist on ds)
+ds = cytome.open("atac.cytome")
+
+piaso.pp.calculateCellMetrics(ds)        # per-cell QC -> ds.cells
+piaso.pp.selectPeaks(ds)                  # highly variable peaks -> ds.peaks
+piaso.tl.runSVDLazy(ds)                   # X_svd embedding -> ds.embeddings
+piaso.tl.neighbors(ds)                    # connectivities/distances -> ds.graphs
+piaso.tl.leiden(ds, key_added="leiden")  # labels -> ds.cells["leiden"]
+piaso.tl.umap(ds)                         # X_umap -> ds.embeddings
+piaso.pl.plotUMAP(ds, color="leiden")
+
+# Markers + peak calling can persist to the cytome too
+import cosg
+cosg.run_cosg_cytome(ds, groupby="leiden", modality="ATAC",
+                     write_to_cytome=True)        # -> ds.metadata["cosg"]
+piaso.tl.picco(ds, groupby="leiden", genome="mm10")  # PICCO peak calling
+
+ds.close()
+```
+
+See [piaso.org](https://piaso.org) for documentation and tutorials.
+
+## File format
+
+A `.cytome` file is a SQLite database with:
+
+- Entity tables (`cells`, `genes`, `peaks`, ...)
+- Chunked compressed sparse matrices (`matrix_chunks` + `matrix_meta`)
+- Optional CSC chunks (`matrix_csc_chunks`)
+- Dense chunked embeddings (`dense_chunks` + `embedding_meta`)
+- Fragment storage (chunked compressed BLOBs) + genomic R-tree indices
+- Provenance (`_provenance`) and metadata (`_metadata`)
+
+## Comparison with existing tools
+
+Trade-offs in the current implementation:
+
+- Full matrix materialization can be slower than direct HDF5 reads for some workloads.
+- Merge/subset APIs are chunk-aware, but current merge implementation may still materialize intermediate matrices for gene remapping.
+- SQLite provides strong local reliability and portability but is not a distributed/cloud-native storage engine.
+
+## Citation
+
+If you use `cytome`, please cite the upcoming manuscript.
+
+## License
+
+BSD 3-Clause License. See [LICENSE](LICENSE) for details.

cytome-0.2.2/README.md

@@ -0,0 +1,216 @@
+# cytome
+
+**A single-file format for single-cell multi-omics data.**
+
+`cytome` replaces ad-hoc file stacks with one SQLite-backed `.cytome` file that keeps matrices, metadata, embeddings, fragments, and provenance together.
+
+`cytome` stores expression matrices, cell metadata, genomic fragments, embeddings, graphs, and computational provenance in a single SQLite file. It opens from manifest metadata, supports SQL-style metadata filtering, and provides a tested CLI for conversion, merge, subset, export, and validation.
+
+`cytome` is part of the [PIASO](https://piaso.org) toolkit for single-cell analysis. PIASO natively supports cytome datasets for streaming single-cell ATAC-seq and RNA-seq workflows.
+
+## Why cytome?
+
+| Capability | AnnData (`.h5ad`) | BPCells | TileDB-SOMA | cytome (`.cytome`) |
+|---|---:|---:|---:|---:|
+| Single portable file | Yes | No (directory layout) | No (array store) | Yes |
+| Python stdlib core storage dependency | No (`h5py`) | No | No | Yes (`sqlite3`) |
+| SQL metadata queries | No | No | Limited via APIs | Yes |
+| Fragment genomic range queries | No native index | Limited | Yes | Yes (SQLite R-tree) |
+| Native provenance table | No | No | Partial workflow metadata | Yes |
+| Streaming merge API | Partial | Strong compute streaming | Distributed workflows | Yes, chunk-aware |
+| Full matrix load speed | Fast | Fast | Varies | Fast |
+| Cloud-native multi-writer scale-out | No | No | Yes | No (single-writer SQLite) |
+
+## Quick start
+
+No optional dependencies — this runs with `pip install cytome` alone.
+
+```python
+import cytome
+import numpy as np
+import pandas as pd
+import scipy.sparse as sp
+
+rng = np.random.default_rng(0)
+counts = sp.random(1000, 500, density=0.05, format="csr", dtype=np.float32,
+                   random_state=rng)
+
+ds = cytome.create("example.cytome")
+ds.set_entity("cells", pd.DataFrame({
+    "barcode": [f"cell_{i}" for i in range(1000)],
+    "cell_type": rng.choice(["A", "B"], 1000),
+}))
+ds.set_entity("genes", pd.DataFrame({"gene_id": [f"G{i}" for i in range(500)]}))
+ds.add_matrix("RNA_counts", counts)
+ds.flush()
+
+print(ds)                                    # metadata-first summary
+b_cells = ds.cells.query("cell_type == 'B'")  # SQL-style metadata query
+block = ds.RNA.counts[:100, :50]              # random access
+for start, end, chunk in ds.RNA.counts.iter_rows():
+    ...                                       # bounded-memory streaming
+ds.close()
+```
+
+### Marker genes, straight off the file
+
+[COSG](https://github.com/genecell/COSG) reads a `.cytome` in chunks, so peak
+memory does not scale with the number of cells and nothing is converted to
+AnnData first.
+
+```python
+# pip install cosg
+import cosg
+
+markers = cosg.cosg("example.cytome", groupby="cell_type",
+                    modality="RNA", layer="log1p", n_genes_user=10)
+markers["names"]   # per-cell-type marker genes
+```
+
+### Coming from AnnData
+
+```python
+# pip install "cytome[anndata]"
+ds = cytome.from_anndata(adata, modality="RNA", output="rna.cytome")
+```
+
+## Key features
+
+- Instant metadata-first open (`CytomeDataset` initialization reads manifest and table metadata)
+- SQL-queryable entity tables (`cells`, `genes`, `peaks`, `samples`)
+- Merge, subset, and downsample APIs with chunk-aware implementations
+- Fragment storage (chunked, compressed) with genomic range queries and export
+  (bulk fragment **import** is provided by [PIASO](https://piaso.org) —
+  `piaso.pp.importFragments`)
+- Analysis results live on the file: embeddings (`ds.embeddings`), neighbor
+  graphs (`ds.graphs`), per-cell / per-feature columns, and arbitrary analysis
+  artifacts (`ds.metadata`) — PIASO writes results back onto the cytome
+- Provenance logging with parameters, dependency versions, and methods-text export
+- ACID write transactions via SQLite
+- Multi-modal naming convention (`RNA_counts`, `ATAC_counts`, etc.)
+- Optional CSC feature index (`build_feature_index`) for faster column iteration
+- JSON metadata store (`ds.metadata`) for arbitrary nested analysis metadata
+- CLI (`cytome`) for convert, info, merge, subset, downsample, export, validate, provenance, and copy
+
+## Installation
+
+```bash
+pip install cytome
+```
+
+Optional extras:
+
+```bash
+pip install "cytome[full]"
+pip install "cytome[dev]"
+```
+
+## Usage examples
+
+### Convert from AnnData
+
+```python
+import cytome
+ds = cytome.from_anndata(adata, modality="RNA", output="rna.cytome")
+ds.close()
+```
+
+### Open and query
+
+```python
+import cytome
+ds = cytome.open("rna.cytome")
+cd8 = ds.cells.query("cell_type == 'CD8 T'")
+x = ds.RNA.counts[:200, :200]
+ds.close()
+```
+
+### Merge datasets
+
+```python
+import cytome
+merged = cytome.merge(["s1.cytome", "s2.cytome"], output="merged.cytome")
+merged.close()
+```
+
+### Command line
+
+```bash
+cytome info merged.cytome
+cytome merge s1.cytome s2.cytome -o merged.cytome
+cytome subset merged.cytome -o cd8.cytome --query "cell_type == 'CD8 T'"
+```
+
+### ATAC fragments
+
+```python
+ds = cytome.open("multiome.cytome")
+fr = ds.ATAC.fragments.query_region("chr1", 1_000_000, 2_000_000)
+ds.ATAC.fragments.export("fragments.tsv.gz")
+ds.close()
+```
+
+### With PIASO
+
+`cytome` integrates with [PIASO](https://piaso.org) (`pip install piaso-tools`) for
+single-cell analysis. PIASO supports cytome datasets directly — streaming normalization,
+dimensionality reduction, clustering, and visualization without loading full matrices
+into memory.
+
+PIASO functions are **self-contained on a cytome**: they stream from the file
+and write their results back onto it (graphs, embeddings, cluster labels,
+per-cell/per-feature metrics, markers) — they don't return large objects, so you
+read results from the dataset afterwards.
+
+```python
+import cytome
+import piaso
+
+# Open a cytome dataset and run the PIASO ATAC pipeline (results persist on ds)
+ds = cytome.open("atac.cytome")
+
+piaso.pp.calculateCellMetrics(ds)        # per-cell QC -> ds.cells
+piaso.pp.selectPeaks(ds)                  # highly variable peaks -> ds.peaks
+piaso.tl.runSVDLazy(ds)                   # X_svd embedding -> ds.embeddings
+piaso.tl.neighbors(ds)                    # connectivities/distances -> ds.graphs
+piaso.tl.leiden(ds, key_added="leiden")  # labels -> ds.cells["leiden"]
+piaso.tl.umap(ds)                         # X_umap -> ds.embeddings
+piaso.pl.plotUMAP(ds, color="leiden")
+
+# Markers + peak calling can persist to the cytome too
+import cosg
+cosg.run_cosg_cytome(ds, groupby="leiden", modality="ATAC",
+                     write_to_cytome=True)        # -> ds.metadata["cosg"]
+piaso.tl.picco(ds, groupby="leiden", genome="mm10")  # PICCO peak calling
+
+ds.close()
+```
+
+See [piaso.org](https://piaso.org) for documentation and tutorials.
+
+## File format
+
+A `.cytome` file is a SQLite database with:
+
+- Entity tables (`cells`, `genes`, `peaks`, ...)
+- Chunked compressed sparse matrices (`matrix_chunks` + `matrix_meta`)
+- Optional CSC chunks (`matrix_csc_chunks`)
+- Dense chunked embeddings (`dense_chunks` + `embedding_meta`)
+- Fragment storage (chunked compressed BLOBs) + genomic R-tree indices
+- Provenance (`_provenance`) and metadata (`_metadata`)
+
+## Comparison with existing tools
+
+Trade-offs in the current implementation:
+
+- Full matrix materialization can be slower than direct HDF5 reads for some workloads.
+- Merge/subset APIs are chunk-aware, but current merge implementation may still materialize intermediate matrices for gene remapping.
+- SQLite provides strong local reliability and portability but is not a distributed/cloud-native storage engine.
+
+## Citation
+
+If you use `cytome`, please cite the upcoming manuscript.
+
+## License
+
+BSD 3-Clause License. See [LICENSE](LICENSE) for details.