pyadata-cli 0.5.0__tar.gz → 0.6.0__tar.gz

{pyadata_cli-0.5.0 → pyadata_cli-0.6.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: pyadata-cli
-Version: 0.5.0
+Version: 0.6.0
 Summary: Streaming CLI for exploring and editing large AnnData .h5ad and .zarr stores
 Keywords: anndata,h5ad,zarr,bioinformatics,single-cell,cli
 Author: Aljes Binkevich
@@ -44,7 +44,7 @@ A command-line tool for exploring huge AnnData stores (`.h5ad` and `.zarr`) with
 
 ## Features
 
-- Streaming access to very large `.h5ad` and `.zarr` stores
+- Streaming access to very large `.h5ad` and `.zarr` stores — see [the benchmarks](docs/BENCHMARKS.md) for what that costs in practice, including where loading the file outright is faster
 - Auto-detects `.h5ad` files vs `.zarr` directories
 - Chunked processing for dense and sparse matrices (CSR/CSC)
 - Reads every AnnData on-disk layout, from 0.7.x through the current spec, and always writes the current one
@@ -96,6 +96,7 @@ Run help at any level (e.g. `adata --help`, `adata export --help`).
 - `subset` – stream and write a filtered copy, selected by obs/var name lists (`--obs`/`--var`) or by expression (`--obs-query`/`--var-query`).
 - `split` – write one store per distinct value of an annotation column, with a CSV manifest.
 - `concat` – concatenate stores along the obs axis, with `--join inner|outer` and merge strategies for var and uns.
+- `convert` – change a matrix's dtype, layout (CSR/CSC/dense) or density, streaming; refuses a lossy cast or a large size increase unless forced.
 - `export` – extract data from a store; subcommands: `dataframe` (any dataframe group to CSV), `array` (dense to `.npy`), `sparse` (CSR/CSC to `.mtx`), `dict` (JSON), `image` (PNG). Results go to stdout when no `--output` is given.
 - `import` – write new data into a store at any path; subcommands: `dataframe` (CSV), `array` (`.npy`), `sparse` (`.mtx`), `dict` (JSON), `image` (PNG/JPEG/TIFF).
 
@@ -118,12 +119,26 @@ adata split  data.h5ad --by sample -o per_sample/
 adata concat per_sample/*.h5ad -o merged.h5ad --join outer --label sample
 ```
 
+### Shrinking a store, and making encodings agree
+
+```bash
+adata convert data.h5ad X --inplace --dtype float32
+adata convert data.h5ad --all -o small.h5ad --dtype float32 --indices-dtype int32
+adata convert data.h5ad X -o csc.h5ad --layout csc
+```
+
+Counts written as float64 halve with no loss — and `convert` proves that
+before it writes, by casting every value and casting it back. `concat`
+refuses inputs whose matrices disagree about CSR versus CSC; `--layout` is
+how you make them agree.
+
 ## Documentation
 
 - [Get started](docs/GET_STARTED.md) — a short tutorial
 - [Command reference](docs/COMMANDS.md) — every command and flag
 - [Element spec: HDF5](docs/ELEMENTS_h5ad.md) / [Zarr](docs/ELEMENTS_zarr.md) — the on-disk format, and what this tool does with it
-- [Testing](docs/TESTING.md) — how the suite is organised, and how compatibility is verified against six anndata releases
+- [Testing](docs/TESTING.md) — how the suite is organised, how compatibility is verified against six anndata releases, and the complexity guards that keep cost regressions out
+- [Benchmarks](docs/BENCHMARKS.md) — peak memory and wall time against anndata and scanpy, remeasured on every release
 - [Changelog](CHANGELOG.md)
 
 ## Docker
@@ -131,5 +146,5 @@ adata concat per_sample/*.h5ad -o merged.h5ad --join outer --label sample
 A docker image is available on QUAY: `quay.io/cellgeni/adata-cli:latest`. Pull and run with:
 
 ```bash
-docker run --rm -it -v /path/to/data:/data quay.io/cellgeni/adata-cli:latest view /data/your_file.h5ad
+docker run --rm -it -v /path/to/data:/data quay.io/cellgeni/adata-cli:latest adata view /data/your_file.h5ad
 ```

{pyadata_cli-0.5.0 → pyadata_cli-0.6.0}/README.md

@@ -4,7 +4,7 @@ A command-line tool for exploring huge AnnData stores (`.h5ad` and `.zarr`) with
 
 ## Features
 
-- Streaming access to very large `.h5ad` and `.zarr` stores
+- Streaming access to very large `.h5ad` and `.zarr` stores — see [the benchmarks](docs/BENCHMARKS.md) for what that costs in practice, including where loading the file outright is faster
 - Auto-detects `.h5ad` files vs `.zarr` directories
 - Chunked processing for dense and sparse matrices (CSR/CSC)
 - Reads every AnnData on-disk layout, from 0.7.x through the current spec, and always writes the current one
@@ -56,6 +56,7 @@ Run help at any level (e.g. `adata --help`, `adata export --help`).
 - `subset` – stream and write a filtered copy, selected by obs/var name lists (`--obs`/`--var`) or by expression (`--obs-query`/`--var-query`).
 - `split` – write one store per distinct value of an annotation column, with a CSV manifest.
 - `concat` – concatenate stores along the obs axis, with `--join inner|outer` and merge strategies for var and uns.
+- `convert` – change a matrix's dtype, layout (CSR/CSC/dense) or density, streaming; refuses a lossy cast or a large size increase unless forced.
 - `export` – extract data from a store; subcommands: `dataframe` (any dataframe group to CSV), `array` (dense to `.npy`), `sparse` (CSR/CSC to `.mtx`), `dict` (JSON), `image` (PNG). Results go to stdout when no `--output` is given.
 - `import` – write new data into a store at any path; subcommands: `dataframe` (CSV), `array` (`.npy`), `sparse` (`.mtx`), `dict` (JSON), `image` (PNG/JPEG/TIFF).
 
@@ -78,12 +79,26 @@ adata split  data.h5ad --by sample -o per_sample/
 adata concat per_sample/*.h5ad -o merged.h5ad --join outer --label sample
 ```
 
+### Shrinking a store, and making encodings agree
+
+```bash
+adata convert data.h5ad X --inplace --dtype float32
+adata convert data.h5ad --all -o small.h5ad --dtype float32 --indices-dtype int32
+adata convert data.h5ad X -o csc.h5ad --layout csc
+```
+
+Counts written as float64 halve with no loss — and `convert` proves that
+before it writes, by casting every value and casting it back. `concat`
+refuses inputs whose matrices disagree about CSR versus CSC; `--layout` is
+how you make them agree.
+
 ## Documentation
 
 - [Get started](docs/GET_STARTED.md) — a short tutorial
 - [Command reference](docs/COMMANDS.md) — every command and flag
 - [Element spec: HDF5](docs/ELEMENTS_h5ad.md) / [Zarr](docs/ELEMENTS_zarr.md) — the on-disk format, and what this tool does with it
-- [Testing](docs/TESTING.md) — how the suite is organised, and how compatibility is verified against six anndata releases
+- [Testing](docs/TESTING.md) — how the suite is organised, how compatibility is verified against six anndata releases, and the complexity guards that keep cost regressions out
+- [Benchmarks](docs/BENCHMARKS.md) — peak memory and wall time against anndata and scanpy, remeasured on every release
 - [Changelog](CHANGELOG.md)
 
 ## Docker
@@ -91,5 +106,5 @@ adata concat per_sample/*.h5ad -o merged.h5ad --join outer --label sample
 A docker image is available on QUAY: `quay.io/cellgeni/adata-cli:latest`. Pull and run with:
 
 ```bash
-docker run --rm -it -v /path/to/data:/data quay.io/cellgeni/adata-cli:latest view /data/your_file.h5ad
+docker run --rm -it -v /path/to/data:/data quay.io/cellgeni/adata-cli:latest adata view /data/your_file.h5ad
 ```

{pyadata_cli-0.5.0 → pyadata_cli-0.6.0}/pyproject.toml

@@ -3,7 +3,7 @@
 # so the distribution is published as `pyadata-cli`. The import package
 # and the command are both still `adata`.
 name = "pyadata-cli"
-version = "0.5.0"
+version = "0.6.0"
 description = "Streaming CLI for exploring and editing large AnnData .h5ad and .zarr stores"
 readme = "README.md"
 requires-python = ">=3.12"

{pyadata_cli-0.5.0 → pyadata_cli-0.6.0}/src/adata/cli.py

@@ -6,9 +6,11 @@ from typing import Optional, Sequence, List
 from rich.console import Console
 import typer
 
+from adata.core.convert import LAYOUTS
 from adata.commands import (
-    MERGE_STRATEGIES,
+    MERGE_CHOICES,
     concat_stores,
+    convert_store,
     create_store,
     split_store,
     list_store,
@@ -413,12 +415,14 @@ def concat(
     merge: Optional[str] = typer.Option(
         None,
         "--merge",
-        help="How to reconcile var columns: same, unique, first, only (default: drop)",
+        help="How to reconcile var columns: drop, same, unique, first, only "
+        "(default: drop)",
     ),
     uns_merge: Optional[str] = typer.Option(
         None,
         "--uns-merge",
-        help="How to reconcile uns: same, unique, first, only (default: drop)",
+        help="How to reconcile uns: drop, same, unique, first, only "
+        "(default: drop)",
     ),
     fill_value: float = typer.Option(
         0.0, "--fill-value", help="Value for dense cells introduced by an outer join"
@@ -442,13 +446,18 @@ def concat(
         adata concat a.h5ad b.h5ad -o m.h5ad --keys a,b --index-unique - --uns-merge same
     """
     for name, value in (("--merge", merge), ("--uns-merge", uns_merge)):
-        if value is not None and value not in MERGE_STRATEGIES:
+        if value is not None and value not in MERGE_CHOICES:
             console.print(
                 f"[bold red]Error:[/] {name} must be one of: "
-                f"{', '.join(MERGE_STRATEGIES)}"
+                f"{', '.join(MERGE_CHOICES)}"
             )
             raise typer.Exit(code=1)
 
+    # "drop" is the default, and naming it explicitly has to be allowed: a
+    # config that spells out the default should not be rejected.
+    merge = None if merge == "drop" else merge
+    uns_merge = None if uns_merge == "drop" else uns_merge
+
     try:
         concat_stores(
             files,
@@ -469,6 +478,134 @@ def concat(
         raise typer.Exit(code=1)
 
 
+# ============================================================================
+# CONVERT command
+# ============================================================================
+@app.command("convert")
+def convert(
+    file: Path = typer.Argument(
+        ...,
+        help="Input .h5ad/.zarr",
+        exists=True,
+        readable=True,
+        dir_okay=True,
+        file_okay=True,
+    ),
+    entries: Optional[List[str]] = typer.Argument(
+        None,
+        help="Matrix paths to convert, e.g. 'X', 'layers/counts', 'raw/X'",
+    ),
+    output: Optional[Path] = typer.Option(
+        None,
+        "--output",
+        "-o",
+        help="Output .h5ad/.zarr path. Required unless --inplace.",
+        dir_okay=True,
+        file_okay=True,
+    ),
+    inplace: bool = typer.Option(
+        False,
+        "--inplace",
+        help="Modify source file directly.",
+    ),
+    convert_all: bool = typer.Option(
+        False,
+        "--all",
+        help="Convert X, every layer, and raw/X",
+    ),
+    dtype: Optional[str] = typer.Option(
+        None,
+        "--dtype",
+        help="New dtype for the values, e.g. float32",
+    ),
+    indices_dtype: Optional[str] = typer.Option(
+        None,
+        "--indices-dtype",
+        help="New dtype for sparse indices: int32 or int64",
+    ),
+    layout: Optional[str] = typer.Option(
+        None,
+        "--layout",
+        help="Target layout: csr, csc, dense or sparse",
+    ),
+    force: bool = typer.Option(
+        False,
+        "--force",
+        help="Convert despite a lossy cast or a large size increase",
+    ),
+    in_memory: bool = typer.Option(
+        False,
+        "--in-memory",
+        help="Transpose in memory instead of streaming (faster if it fits)",
+    ),
+    chunk_rows: int = typer.Option(
+        1024,
+        "--chunk",
+        "-C",
+        help="Row chunk size for dense matrices",
+    ),
+    zarr_format: Optional[int] = typer.Option(
+        None,
+        "--zarr-format",
+        help="Zarr spec version to write (defaults to the source store's)",
+    ),
+) -> None:
+    """
+    Change a matrix's dtype, layout or density.
+
+    Counts held as float64 cost twice the disk and twice the read for no
+    information; a tool that wants CSC cannot use a CSR store; and concat
+    refuses inputs whose encodings disagree. All three are this command.
+
+    A cast that would not round-trip is refused before anything is written,
+    as is a densification that would inflate the store; --force overrides
+    both. Transposing streams by default, so it works on matrices too large
+    to load.
+
+    Examples:
+        adata convert data.h5ad X -o out.h5ad --dtype float32
+        adata convert data.h5ad X -o out.h5ad --layout csc
+        adata convert data.h5ad X --inplace --dtype float32 --indices-dtype int32
+        adata convert data.h5ad --all -o out.h5ad --dtype float32
+    """
+    if not inplace and output is None:
+        console.print(
+            "[bold red]Error:[/] Output file is required. "
+            "Use --output/-o or --inplace.",
+        )
+        raise typer.Exit(code=1)
+
+    if layout is not None and layout not in LAYOUTS:
+        console.print(
+            f"[bold red]Error:[/] --layout must be one of: {', '.join(LAYOUTS)}."
+        )
+        raise typer.Exit(code=1)
+
+    if zarr_format is not None and zarr_format not in (2, 3):
+        console.print("[bold red]Error:[/] --zarr-format must be 2 or 3.")
+        raise typer.Exit(code=1)
+
+    try:
+        convert_store(
+            file=file,
+            entries=list(entries) if entries else None,
+            output=output,
+            console=console,
+            dtype=dtype,
+            indices_dtype=indices_dtype,
+            layout=layout,
+            convert_all=convert_all,
+            inplace=inplace,
+            chunk_rows=chunk_rows,
+            in_memory=in_memory,
+            force=force,
+            zarr_format=zarr_format,
+        )
+    except Exception as e:
+        console.print(f"[bold red]Error:[/] {e}")
+        raise typer.Exit(code=1)
+
+
 # ============================================================================
 # SPLIT command
 # ============================================================================

{pyadata_cli-0.5.0 → pyadata_cli-0.6.0}/src/adata/commands/__init__.py

@@ -5,4 +5,5 @@ from adata.commands.import_data import import_object
 from adata.commands.ls import list_store
 from adata.commands.create import create_store
 from adata.commands.split import split_store
-from adata.commands.concat import MERGE_STRATEGIES, concat_stores
+from adata.commands.concat import MERGE_CHOICES, MERGE_STRATEGIES, concat_stores
+from adata.commands.convert import convert_store

{pyadata_cli-0.5.0 → pyadata_cli-0.6.0}/src/adata/commands/concat.py

@@ -7,9 +7,9 @@ from typing import Optional, Sequence
 
 from rich.console import Console
 
-from adata.core.concat import MERGE_STRATEGIES, concat_on_disk
+from adata.core.concat import MERGE_CHOICES, MERGE_STRATEGIES, concat_on_disk
 
-__all__ = ["MERGE_STRATEGIES", "concat_stores"]
+__all__ = ["MERGE_CHOICES", "MERGE_STRATEGIES", "concat_stores"]
 
 
 def concat_stores(

pyadata_cli-0.6.0/src/adata/commands/convert.py

@@ -0,0 +1,218 @@
+"""The `convert` command: rewrite matrices, copy everything else.
+
+Writes a whole new store rather than editing one in place, for the same
+reason `subset` does: a conversion that fails half way leaves the original
+untouched, and `--inplace` becomes an atomic swap of a finished file rather
+than a partial edit of a live one.
+"""
+
+from __future__ import annotations
+
+import shutil
+from pathlib import Path
+from typing import Any, List, Optional
+
+from rich.console import Console
+
+from adata.core.convert import (
+    DEFAULT_CHUNK,
+    LAYOUTS,
+    Plan,
+    convert_matrix,
+    parse_dtype,
+    plan_conversion,
+)
+from adata.elements import spec
+from adata.elements.write import ensure_anndata_skeleton
+from adata.storage import copy_tree, detect_backend, is_group, open_store
+
+#: What `--all` means. The matrices an AnnData object is built from, not
+#: every 2-D array in the file: obsm/varm hold embeddings, whose dtype is
+#: rarely what anyone is trying to shrink. Those still convert by path.
+ALL_PREFIXES = ("X", "layers", "raw/X")
+
+
+def discover_matrices(root: Any) -> List[str]:
+    """Entry paths `--all` resolves to, in a stable order."""
+    found: List[str] = []
+    if "X" in root:
+        found.append("X")
+    if "layers" in root and is_group(root["layers"]):
+        found.extend(f"layers/{key}" for key in sorted(root["layers"].keys()))
+    if "raw" in root and is_group(root["raw"]) and "X" in root["raw"]:
+        found.append("raw/X")
+    return found
+
+
+def _same_path(left: Path, right: Path) -> bool:
+    """Do these name the same store, through symlinks and `..` alike?"""
+    try:
+        return left.resolve() == right.resolve()
+    except OSError:  # pragma: no cover - unresolvable path
+        return left.absolute() == right.absolute()
+
+
+def _resolve(root: Any, path: str) -> Any:
+    obj = root
+    for part in path.split("/"):
+        if part not in obj:
+            raise KeyError(f"{path!r} not found in the store.")
+        obj = obj[part]
+    return obj
+
+
+def convert_store(
+    file: Path,
+    entries: Optional[List[str]],
+    output: Optional[Path],
+    console: Console,
+    *,
+    dtype: Optional[str] = None,
+    indices_dtype: Optional[str] = None,
+    layout: Optional[str] = None,
+    convert_all: bool = False,
+    inplace: bool = False,
+    chunk: int = DEFAULT_CHUNK,
+    chunk_rows: int = 1024,
+    in_memory: bool = False,
+    force: bool = False,
+    zarr_format: Optional[int] = None,
+) -> None:
+    """Write a copy of `file` with the named matrices converted."""
+    if not inplace and output is None:
+        raise ValueError("Output file is required unless --inplace is specified.")
+    if dtype is None and indices_dtype is None and layout is None:
+        raise ValueError(
+            "Nothing to do: pass at least one of --dtype, --indices-dtype "
+            "or --layout."
+        )
+    if layout is not None and layout not in LAYOUTS:
+        raise ValueError(f"--layout must be one of: {', '.join(LAYOUTS)}.")
+
+    data_dtype = parse_dtype(dtype) if dtype else None
+    index_dtype = (
+        parse_dtype(indices_dtype, allowed=("int32", "int64"))
+        if indices_dtype
+        else None
+    )
+
+    if output is not None and not inplace and _same_path(file, output):
+        # Opening the destination "w" clears it while the source is still
+        # being read from it. HDF5 refuses; Zarr does not, and quietly
+        # produced an empty store where the data used to be.
+        raise ValueError(
+            f"Output path is the input: {output}. Use --inplace to replace "
+            "it, which writes to a temporary file first."
+        )
+
+    if inplace:
+        backend = detect_backend(file)
+        if backend == "zarr":
+            base = file.stem if file.suffix else file.name
+            dst_path = file.with_name(f"{base}.convert-tmp.zarr")
+        else:
+            dst_path = file.with_name(f"{file.name}.convert-tmp")
+        if dst_path.exists():
+            raise FileExistsError(f"Temporary path already exists: {dst_path}")
+    else:
+        dst_path = output
+
+    if zarr_format is None and detect_backend(file) == "zarr":
+        with open_store(file, "r") as probe:
+            zarr_format = probe.zarr_format
+
+    with open_store(file, "r") as src_store:
+        src = src_store.root
+        targets = discover_matrices(src) if convert_all else list(entries or [])
+        if not targets:
+            raise ValueError(
+                "No matrices selected. Name one (e.g. `X`) or pass --all."
+            )
+        # Every check first, before the destination exists. A refusal must
+        # not leave behind a store holding a copy of obs and var and nothing
+        # else -- the caller cannot tell that from a finished conversion.
+        plans = {
+            path: plan_conversion(
+                _resolve(src, path),
+                path,
+                dtype=data_dtype,
+                index_dtype=index_dtype,
+                layout=layout,
+                chunk=chunk,
+                force=force,
+                console=console,
+            )
+            for path in targets
+        }
+
+        console.print(
+            f"[cyan]Converting {len(targets)} matrix/matrices:[/] "
+            + ", ".join(targets)
+        )
+
+        with open_store(dst_path, "w", zarr_format=zarr_format) as dst_store:
+            dst = dst_store.root
+            _write(
+                src, dst, plans,
+                chunk=chunk, chunk_rows=chunk_rows, in_memory=in_memory,
+                console=console,
+            )
+            ensure_anndata_skeleton(dst)
+
+    if inplace:
+        if file.is_dir():
+            shutil.rmtree(file)
+        elif file.exists():
+            file.unlink()
+        if dst_path.is_dir():
+            shutil.move(str(dst_path), str(file))
+        else:
+            dst_path.replace(file)
+        console.print(f"[green]Converted[/] {file}")
+    else:
+        console.print(f"[green]Wrote[/] {dst_path}")
+
+
+def _write(src: Any, dst: Any, plans: dict, **options: Any) -> None:
+    """Copy the store across, converting the targeted entries as they pass."""
+    by_parent: dict = {}
+    for path in plans:
+        parent, _, leaf = path.rpartition("/")
+        by_parent.setdefault(parent, {})[leaf] = plans[path]
+
+    for key in src.keys():
+        if key in by_parent.get("", {}):
+            convert_matrix(
+                src[key], dst, key, plan=by_parent[""][key], **options
+            )
+        elif key in by_parent:
+            # Any parent, not just layers and raw. Restricting it to those
+            # two meant an explicitly named `obsm/X_pca` was copied
+            # unconverted and the command still reported success.
+            _write_group(src[key], dst, key, by_parent, **options)
+        else:
+            copy_tree(src[key], dst, key)
+
+
+def _write_group(
+    group: Any, dst: Any, name: str, by_parent: dict, **options: Any
+) -> None:
+    """Recreate one container, converting the members that were named."""
+    from adata.storage import copy_attrs, is_zarr_group
+
+    out = dst.create_group(name)
+    copy_attrs(
+        group.attrs,
+        out.attrs,
+        target_backend="zarr" if is_zarr_group(dst) else "hdf5",
+    )
+    if not spec.encoding_type(out):
+        spec.set_encoding(out, spec.RAW if name == "raw" else spec.DICT)
+
+
+    wanted = by_parent.get(name, {})
+    for key in group.keys():
+        if key in wanted:
+            convert_matrix(group[key], out, key, plan=wanted[key], **options)
+        else:
+            copy_tree(group[key], out, key)