nps-cli 0.1.2__tar.gz → 0.1.5__tar.gz

nps_cli-0.1.5/PKG-INFO

@@ -0,0 +1,119 @@
+Metadata-Version: 2.4
+Name: nps-cli
+Version: 0.1.5
+Summary: A simple CLI for sampling point clouds from large volumetric datasets
+Author: Jakob Troidl
+License-Expression: MIT
+Classifier: Programming Language :: Python :: 3.11
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown
+Requires-Dist: click
+Requires-Dist: cloud-volume!=12.13.*
+Requires-Dist: volara>=2.0.1
+Requires-Dist: pocaduck
+Provides-Extra: dev
+Requires-Dist: pytest; extra == "dev"
+Requires-Dist: black; extra == "dev"
+Requires-Dist: build; extra == "dev"
+Requires-Dist: twine; extra == "dev"
+Requires-Dist: ruff; extra == "dev"
+
+[![pypi](https://badge.fury.io/py/nps-cli.svg)](https://badge.fury.io/py/nps-cli)
+[![Python Version](https://img.shields.io/pypi/pyversions/nps-cli.svg)](https://pypi.org/project/nps-cli/)
+
+# nps: distributed point cloud sampling
+
+`nps` samples point clouds of every segment in a large segmentation volumes
+* Compatible with [CloudVolume](https://github.com/seung-lab/cloud-volume) (precomputed or graphene).
+* It processes the volume blockwise on local, LSF or Slurm workers and writes the points in [pocaduck](https://github.com/JaneliaSciComp/pocaduck) format.
+
+```bash
+pip install nps-cli
+nps --help
+```
+
+## Usage
+
+```bash
+# Whole volume
+nps --cv-path precomputed://gs://neuroglancer-janelia-flyem-hemibrain/v1.0/segmentation
+
+# Subvolume (begin_x begin_y begin_z end_x end_y end_z, in voxels), 1% of each segment
+nps --cv-path precomputed://gs://neuroglancer-janelia-flyem-hemibrain/v1.0/segmentation \
+    --bbox 15347 19712 18606 15859 20224 19118 --fraction 0.01
+
+# Also store the supervoxel ID of every point, looked up in a separate volume
+# that shares the voxel grid of --cv-path
+nps --cv-path precomputed://gs://<bucket>/agglomerated \
+    --svid-cv-path precomputed://gs://<bucket>/supervoxels
+
+# Graphene: read supervoxels from the same volume (agglomerate=False)
+nps --cv-path graphene://https://<server>/segmentation/table/<table> --sample_svids
+```
+
+Key options (run `nps --help` for all of them):
+
+| Option | Default | Meaning |
+| --- | --- | --- |
+| `--fraction` | `0.001` | Fraction of each segment's voxels to sample (at least 1 per segment and block). |
+| `--mip` | `0` | Resolution level to sample at. Coordinates are voxels of this mip. |
+| `--bbox` | whole volume | ROI in voxels. |
+| `--block-size` | `128 128 128` | Block size per task, in voxels. |
+| `--worker-type` / `--num-workers` / `--cpus-per-worker` / `--queue` | `LocalWorker` / `8` / `4` / `local` | Where and how the blocks are processed. |
+| `--fill-missing` | off | Treat missing chunks as empty instead of raising an error. |
+| `--overwrite` | off | Delete points of an earlier run in `--output-dir` instead of refusing to start. |
+| `--skip-empty` / `--block-mask` / `--mask-mip` | on | Skipping empty blocks, see below. |
+
+### Output
+
+```
+<output-dir>/
+  block_mask.npy            # occupancy mask from the empty-block pre-scan
+  points/
+    unified_index.db        # pocaduck index over all workers
+    worker_<id>/...         # parquet files + per-worker index
+```
+
+Each point is `[x, y, z]`, or `[x, y, z, svid]` when SVIDs are sampled, stored under its segment label.
+
+`nps` refuses to write into an `--output-dir` that already holds points, since every
+run re-samples all blocks and old points would show up as duplicates. Pass
+`--overwrite` to delete them first.
+
+### Skipping empty blocks
+
+Before sampling, `nps` scans the ROI at a coarse mip (`--mask-mip`, default the
+coarsest one) and never dispatches blocks without labels. The mask is dilated by
+one block, so thin structures lost in downsampling are still sampled. Use
+`--no-skip-empty` to disable the scan, or `--block-mask <path>` to reuse a
+`block_mask.npy` from an earlier run with the same `--bbox`, `--block-size` and `--mip`.
+
+## Reading point clouds
+
+```python
+from pocaduck import Query, StorageConfig
+
+query = Query(storage_config=StorageConfig(base_path="<output-dir>/points"))
+labels = query.get_labels()
+points = query.get_points(label=12345)  # all points of one segment
+query.close()
+```
+
+See the [pocaduck docs](https://github.com/JaneliaSciComp/pocaduck?tab=readme-ov-file#querying-point-clouds)
+for more, including [optimizing](https://github.com/JaneliaSciComp/pocaduck?tab=readme-ov-file#running-the-optimization-pipeline) the storage for fast reads.
+
+## Example scripts
+
+| Script | What it does |
+| --- | --- |
+| [`liconn_local.sh`](scripts/liconn_local.sh) | Points only, local workers |
+| [`liconn_svids_local.sh`](scripts/liconn_svids_local.sh) | Points + SVIDs from a separate precomputed volume |
+| [`flywire_cluster.sh`](scripts/flywire_cluster.sh) | Points + SVIDs for FlyWire on LSF (`LABEL_NAME=flywire_v141_m783` for the 2023 snapshot) |
+
+## Release
+
+```bash
+pip install -e ".[dev]"
+python -m build
+twine upload dist/*
+```

nps_cli-0.1.5/README.md

@@ -0,0 +1,99 @@
+[![pypi](https://badge.fury.io/py/nps-cli.svg)](https://badge.fury.io/py/nps-cli)
+[![Python Version](https://img.shields.io/pypi/pyversions/nps-cli.svg)](https://pypi.org/project/nps-cli/)
+
+# nps: distributed point cloud sampling
+
+`nps` samples point clouds of every segment in a large segmentation volumes
+* Compatible with [CloudVolume](https://github.com/seung-lab/cloud-volume) (precomputed or graphene).
+* It processes the volume blockwise on local, LSF or Slurm workers and writes the points in [pocaduck](https://github.com/JaneliaSciComp/pocaduck) format.
+
+```bash
+pip install nps-cli
+nps --help
+```
+
+## Usage
+
+```bash
+# Whole volume
+nps --cv-path precomputed://gs://neuroglancer-janelia-flyem-hemibrain/v1.0/segmentation
+
+# Subvolume (begin_x begin_y begin_z end_x end_y end_z, in voxels), 1% of each segment
+nps --cv-path precomputed://gs://neuroglancer-janelia-flyem-hemibrain/v1.0/segmentation \
+    --bbox 15347 19712 18606 15859 20224 19118 --fraction 0.01
+
+# Also store the supervoxel ID of every point, looked up in a separate volume
+# that shares the voxel grid of --cv-path
+nps --cv-path precomputed://gs://<bucket>/agglomerated \
+    --svid-cv-path precomputed://gs://<bucket>/supervoxels
+
+# Graphene: read supervoxels from the same volume (agglomerate=False)
+nps --cv-path graphene://https://<server>/segmentation/table/<table> --sample_svids
+```
+
+Key options (run `nps --help` for all of them):
+
+| Option | Default | Meaning |
+| --- | --- | --- |
+| `--fraction` | `0.001` | Fraction of each segment's voxels to sample (at least 1 per segment and block). |
+| `--mip` | `0` | Resolution level to sample at. Coordinates are voxels of this mip. |
+| `--bbox` | whole volume | ROI in voxels. |
+| `--block-size` | `128 128 128` | Block size per task, in voxels. |
+| `--worker-type` / `--num-workers` / `--cpus-per-worker` / `--queue` | `LocalWorker` / `8` / `4` / `local` | Where and how the blocks are processed. |
+| `--fill-missing` | off | Treat missing chunks as empty instead of raising an error. |
+| `--overwrite` | off | Delete points of an earlier run in `--output-dir` instead of refusing to start. |
+| `--skip-empty` / `--block-mask` / `--mask-mip` | on | Skipping empty blocks, see below. |
+
+### Output
+
+```
+<output-dir>/
+  block_mask.npy            # occupancy mask from the empty-block pre-scan
+  points/
+    unified_index.db        # pocaduck index over all workers
+    worker_<id>/...         # parquet files + per-worker index
+```
+
+Each point is `[x, y, z]`, or `[x, y, z, svid]` when SVIDs are sampled, stored under its segment label.
+
+`nps` refuses to write into an `--output-dir` that already holds points, since every
+run re-samples all blocks and old points would show up as duplicates. Pass
+`--overwrite` to delete them first.
+
+### Skipping empty blocks
+
+Before sampling, `nps` scans the ROI at a coarse mip (`--mask-mip`, default the
+coarsest one) and never dispatches blocks without labels. The mask is dilated by
+one block, so thin structures lost in downsampling are still sampled. Use
+`--no-skip-empty` to disable the scan, or `--block-mask <path>` to reuse a
+`block_mask.npy` from an earlier run with the same `--bbox`, `--block-size` and `--mip`.
+
+## Reading point clouds
+
+```python
+from pocaduck import Query, StorageConfig
+
+query = Query(storage_config=StorageConfig(base_path="<output-dir>/points"))
+labels = query.get_labels()
+points = query.get_points(label=12345)  # all points of one segment
+query.close()
+```
+
+See the [pocaduck docs](https://github.com/JaneliaSciComp/pocaduck?tab=readme-ov-file#querying-point-clouds)
+for more, including [optimizing](https://github.com/JaneliaSciComp/pocaduck?tab=readme-ov-file#running-the-optimization-pipeline) the storage for fast reads.
+
+## Example scripts
+
+| Script | What it does |
+| --- | --- |
+| [`liconn_local.sh`](scripts/liconn_local.sh) | Points only, local workers |
+| [`liconn_svids_local.sh`](scripts/liconn_svids_local.sh) | Points + SVIDs from a separate precomputed volume |
+| [`flywire_cluster.sh`](scripts/flywire_cluster.sh) | Points + SVIDs for FlyWire on LSF (`LABEL_NAME=flywire_v141_m783` for the 2023 snapshot) |
+
+## Release
+
+```bash
+pip install -e ".[dev]"
+python -m build
+twine upload dist/*
+```

nps_cli-0.1.5/nps/blockwise/__init__.py

@@ -0,0 +1,2 @@
+from .sample_points import SamplePoints as SamplePoints
+from .sample_points import consolidate_indexes as consolidate_indexes

nps_cli-0.1.5/nps/blockwise/block_mask.py

@@ -0,0 +1,141 @@
+"""
+Per-block occupancy mask computed from a coarse mip level.
+
+Reading a chunk from a (sharded) precomputed volume costs roughly the same
+regardless of how many voxels are requested, so checking each block for
+emptiness at sampling time does not pay off. Instead we make one pass over the
+ROI at the coarsest available mip, where a single chunk covers hundreds of
+sampling blocks, and record for every block whether it contains any label.
+The mask is then used by :class:`SamplePoints` to skip empty blocks inside the
+daisy scheduler, so workers never touch them.
+"""
+
+from concurrent.futures import ThreadPoolExecutor
+from itertools import product
+
+import numpy as np
+from cloudvolume import CloudVolume
+from funlib.geometry import Coordinate, Roi
+from volara.datasets import CloudVolumeWrapper
+
+# Approximate edge length, in coarse voxels, of the region read per job.
+_TARGET_TILE_VOXELS = 128
+
+
+def _open_coarse(store: str, mip: int, timestamp: int, agglomerate: bool) -> CloudVolume:
+    return CloudVolume(
+        store,
+        mip=mip,
+        use_https=True,
+        agglomerate=agglomerate,
+        timestamp=timestamp,
+        fill_missing=True,  # missing chunks are, by definition, empty
+        bounded=False,  # tiles at the ROI edge may poke out of the volume
+    )
+
+
+def _tile_job(args):
+    """Read one coarse tile and return which of its blocks are non-empty."""
+    vol, tile_index, block_lo, block_hi, roi_begin, roi_end, block_size, factor = args
+    block_lo, block_hi = np.array(block_lo), np.array(block_hi)
+    roi_begin, roi_end = np.array(roi_begin), np.array(roi_end)
+    block_size, factor = np.array(block_size), np.array(factor)
+
+    # Coarse voxel range covering all blocks in this tile (clipped to the ROI).
+    fine_lo = roi_begin + block_lo * block_size
+    fine_hi = np.minimum(roi_begin + block_hi * block_size, roi_end)
+    coarse_lo = fine_lo // factor
+    coarse_hi = -(-fine_hi // factor)  # ceil
+    data = np.asarray(
+        vol[tuple(slice(int(a), int(b)) for a, b in zip(coarse_lo, coarse_hi))]
+    ).squeeze(axis=-1)
+
+    out = np.zeros(block_hi - block_lo, dtype=bool)
+    for idx in product(*(range(n) for n in out.shape)):
+        b = block_lo + np.array(idx)
+        lo = (roi_begin + b * block_size) // factor - coarse_lo
+        hi = -(-(np.minimum(roi_begin + (b + 1) * block_size, roi_end)) // factor) - coarse_lo
+        out[idx] = data[tuple(slice(int(a), int(c)) for a, c in zip(lo, hi))].any()
+    return tile_index, out
+
+
+def _dilate(mask: np.ndarray, radius: int) -> np.ndarray:
+    """Binary dilation with a cube of the given radius (no scipy dependency)."""
+    if radius <= 0:
+        return mask
+    padded = np.pad(mask, radius)
+    out = np.zeros_like(mask)
+    for shift in product(range(2 * radius + 1), repeat=mask.ndim):
+        sl = tuple(slice(s, s + n) for s, n in zip(shift, mask.shape))
+        out |= padded[sl]
+    return out
+
+
+def compute_block_mask(
+    labels: CloudVolumeWrapper,
+    roi: Roi,
+    block_size: Coordinate,
+    mask_mip: int | None = None,
+    num_workers: int = 8,
+    dilate: int = 1,
+) -> np.ndarray:
+    """
+    Compute a boolean array, indexed by block index within ``roi``, that is
+    True for blocks containing at least one non-zero label.
+
+    Args:
+        labels: The label volume that will be sampled.
+        roi: ROI to be processed, in voxel coordinates of ``labels.mip``
+            (i.e. the frame daisy blocks are expressed in).
+        block_size: Sampling block size in the same frame.
+        mask_mip: Mip level to read for the occupancy check. Defaults to the
+            coarsest mip available.
+        num_workers: Number of threads reading coarse tiles. Reads are I/O and
+            decompression bound and release the GIL, so threads scale as well
+            as processes here while avoiding multiprocessing entirely (forked
+            and spawned pools both died with BrokenProcessPool on LSF nodes).
+        dilate: Grow the mask by this many blocks in every direction. Guards
+            against thin structures that vanish when downsampled.
+    """
+    base = CloudVolume(str(labels.store), mip=labels.mip, use_https=True)
+    if mask_mip is None:
+        mask_mip = max(base.available_mips)
+    factor = np.asarray(base.mip_resolution(mask_mip)) / np.asarray(
+        base.mip_resolution(labels.mip)
+    )
+    assert np.allclose(factor, np.round(factor)), (
+        f"mip {mask_mip} resolution is not an integer multiple of mip {labels.mip}"
+    )
+    factor = np.round(factor).astype(int)
+
+    roi_begin = np.array(roi.begin)
+    roi_end = np.array(roi.end)
+    block_size = np.array(block_size)
+    n_blocks = -(-(roi_end - roi_begin) // block_size)
+
+    # Blocks per tile per axis: enough that a tile is ~_TARGET_TILE_VOXELS wide.
+    blocks_per_tile = np.maximum(
+        1, _TARGET_TILE_VOXELS // np.maximum(1, block_size // factor)
+    )
+    n_tiles = -(-n_blocks // blocks_per_tile)
+
+    coarse = _open_coarse(
+        str(labels.store), mask_mip, labels.timestamp, labels.agglomerate
+    )
+    jobs = []
+    for t in product(*(range(n) for n in n_tiles)):
+        t = np.array(t)
+        lo = t * blocks_per_tile
+        hi = np.minimum(lo + blocks_per_tile, n_blocks)
+        jobs.append(
+            (coarse, tuple(t), tuple(lo), tuple(hi), tuple(roi_begin),
+             tuple(roi_end), tuple(block_size), tuple(factor))
+        )
+
+    mask = np.zeros(n_blocks, dtype=bool)
+    with ThreadPoolExecutor(max_workers=num_workers) as pool:
+        for tile_index, sub in pool.map(_tile_job, jobs):
+            lo = np.array(tile_index) * blocks_per_tile
+            mask[tuple(slice(a, a + n) for a, n in zip(lo, sub.shape))] = sub
+
+    return _dilate(mask, dilate)

nps_cli-0.1.5/nps/blockwise/sample_points.py

@@ -0,0 +1,248 @@
+import glob
+import logging
+import os
+import shutil
+import uuid
+from contextlib import contextmanager
+from typing import Literal
+
+import daisy
+import duckdb
+import numpy as np
+import pandas as pd
+import pyarrow as pa
+import pyarrow.parquet as pq
+from daisy import Block
+from funlib.geometry import Coordinate, Roi
+from volara.blockwise import BlockwiseTask
+from volara.datasets import CloudVolumeWrapper, Dataset
+from volara.utils import PydanticCoordinate
+
+_INDEX_SCHEMA = """
+    CREATE TABLE IF NOT EXISTS {name} (
+        label UBIGINT,
+        block_id VARCHAR,
+        file_path VARCHAR,
+        point_count UBIGINT
+    )
+"""
+
+
+class SamplePoints(BlockwiseTask):
+    task_type: Literal["sample_pc"] = "sample_pc"
+    out_dir: str
+    labels: CloudVolumeWrapper
+    svids: CloudVolumeWrapper | None = None
+    block_mask: str | None = None
+    block_size: PydanticCoordinate
+    fraction: float
+    fit: Literal["shrink"] = "shrink"
+    read_write_conflict: Literal[False] = False
+
+    @property
+    def task_name(self) -> str:
+        return f"{self.labels.name}-{self.task_type}"
+
+    @property
+    def write_roi(self) -> Roi:
+        total_roi = self.labels.array("r").roi
+        if self.roi is not None:
+            total_roi = total_roi.intersect(self.roi)
+        return total_roi
+
+    @property
+    def voxel_size(self) -> Coordinate:
+        return self.labels.voxel_size
+
+    @property
+    def write_size(self) -> Coordinate:
+        return self.block_size
+
+    @property
+    def context_size(self) -> Coordinate:
+        return Coordinate((0,) * self.write_size.dims)
+
+    def drop_artifacts(self):
+        # init() recreates the directory when the task is run.
+        shutil.rmtree(self.out_dir, ignore_errors=True)
+
+    def check_block_func(self):
+        """
+        Daisy calls this in the scheduler before dispatching a block; returning
+        True means "already done". If a block mask is given, report empty blocks
+        as done so they are never sent to a worker.
+        """
+        base = super().check_block_func()
+        if self.block_mask is None:
+            return base
+
+        mask = np.load(self.block_mask)
+        roi_begin = np.array(self.write_roi.begin)
+        block_size = np.array(self.block_size)
+
+        def check_block(block: Block) -> bool:
+            index = (np.array(block.write_roi.begin) - roi_begin) // block_size
+            if not mask[tuple(index)]:
+                return True
+            return base(block)
+
+        return check_block
+
+    @property
+    def output_datasets(self) -> list[Dataset]:
+        return []
+
+    def sample_pc_in_block(self, block: Block, labels: np.ndarray, svids: np.ndarray | None, offset: Coordinate, writer: "_PointWriter"):
+        block_id = block.block_id[1]
+
+        seg_ids, pts = self.sample_segment_points(labels, self.fraction)
+        logging.info(f"sampled {len(pts)} points in block {block_id}")
+
+        packed_pts = (pts + np.array(offset)).astype(np.uint64)
+        if self.svids is not None:
+            sampled_svids = svids[pts[:, 0], pts[:, 1], pts[:, 2]].reshape(-1, 1)
+            packed_pts = np.concatenate((packed_pts, sampled_svids.astype(np.uint64)), axis=1)
+
+        writer.write(block_id, seg_ids, packed_pts)
+
+    def init(self):
+        os.makedirs(self.out_dir, exist_ok=True)
+
+    @contextmanager
+    def process_block_func(self):
+        data = self.labels.array("r")
+        if self.svids is not None:
+            supervoxels = self.svids.array("r")
+
+        try:
+            worker_id = daisy.Context.from_env()["worker_id"]
+        except KeyError:
+            worker_id = 0
+        # One writer per worker process instead of a pocaduck Ingestor per
+        # block, which re-read and re-wrote the worker's whole parquet file
+        # and ran two DuckDB queries per segment on every block.
+        writer = _PointWriter(self.out_dir, worker_id)
+
+        def process_block(block: Block):
+            # Index with the Roi (world/absolute voxel coords), not with
+            # to_slices(): numpy-style keys on a funlib Array are zero-based and
+            # would read from the wrong place on volumes with a voxel offset.
+            # Only squeeze the channel axis; edge blocks can be 1 voxel thick.
+            labels = np.asarray(data[block.write_roi]).squeeze(axis=-1)
+
+            offset = block.write_roi.get_begin()
+
+            if self.svids is not None:
+                s = np.asarray(supervoxels[block.write_roi]).squeeze(axis=-1)
+                assert s.shape == labels.shape, (
+                    f"label block {labels.shape} and svid block {s.shape} differ in "
+                    f"{block.write_roi}; the label and supervoxel volumes must share "
+                    "resolution and offset at the chosen mip"
+                )
+            else:
+                s = None
+
+            self.sample_pc_in_block(block, labels, s, offset, writer)
+
+        # Index consolidation happens once in the driver after all blocks are
+        # done (see consolidate_indexes), not concurrently in every worker.
+        yield process_block
+
+    def sample_segment_points(self, labels, fraction, background=0):
+        """
+        Sample max(1, int(n * fraction)) voxels without replacement from every
+        segment with n voxels. Returns the segment ID of every sampled point
+        and the (N, 3) voxel coordinates within ``labels``.
+
+        Vectorized: one sort over the foreground voxels instead of one full
+        ``labels == seg`` scan per segment.
+        """
+        flat = labels.ravel()
+        idx = np.flatnonzero(flat != background) if background is not None else np.arange(flat.size)
+        vals = flat[idx]
+
+        # Shuffle, then stable-sort by segment: groups segments together with a
+        # random order inside each group, so the first k of a group are a
+        # uniform sample without replacement.
+        perm = np.random.permutation(len(idx))
+        perm = perm[np.argsort(vals[perm], kind="stable")]
+        idx, vals = idx[perm], vals[perm]
+
+        _, starts, counts = np.unique(vals, return_index=True, return_counts=True)
+        k = np.maximum(1, (counts * fraction).astype(np.int64))
+        rank = np.arange(len(vals)) - np.repeat(starts, counts)
+        keep = rank < np.repeat(k, counts)
+
+        coords = np.column_stack(np.unravel_index(idx[keep], labels.shape))
+        return vals[keep], coords
+
+
+class _PointWriter:
+    """
+    Writes sampled points of one worker in the layout of pocaduck's Ingestor
+    (``worker_<id>/data/*.parquet`` plus ``worker_<id>/index_<id>.db``), so
+    :func:`consolidate_indexes` and ``pocaduck.Query`` work unchanged.
+
+    Every block is written (parquet first, then its index rows) before the
+    block returns: daisy marks a block done as soon as process_block returns
+    and does not wait for worker teardown, so buffering across blocks would
+    lose data.
+    """
+
+    def __init__(self, out_dir: str, worker_id):
+        worker_dir = os.path.join(out_dir, f"worker_{worker_id}")
+        self.data_dir = os.path.join(worker_dir, "data")
+        self.db_path = os.path.join(worker_dir, f"index_{worker_id}.db")
+        os.makedirs(self.data_dir, exist_ok=True)
+        # Unique prefix: a restarted worker can reuse a worker_id and must not
+        # overwrite files of a previous run.
+        self.prefix = f"{worker_id}-{uuid.uuid4().hex[:8]}"
+
+    def write(self, block_id, seg_ids: np.ndarray, points: np.ndarray):
+        if len(points) == 0:
+            return
+        file_path = os.path.join(self.data_dir, f"{self.prefix}-{block_id}.parquet")
+        data = pa.FixedSizeListArray.from_arrays(pa.array(points.ravel()), points.shape[1])
+        table = pa.table({
+            "label": seg_ids.astype(np.int64),
+            "block_id": np.full(len(points), block_id, dtype=np.int64),
+            # Same list<uint64> column the pocaduck Ingestor writes.
+            "data": data.cast(pa.list_(pa.uint64())),
+        })
+        # Write-then-rename so a killed worker never leaves a truncated file.
+        pq.write_table(table, file_path + ".tmp")
+        os.replace(file_path + ".tmp", file_path)
+
+        segs, counts = np.unique(seg_ids, return_counts=True)
+        index = pd.DataFrame({
+            "label": segs.astype(np.uint64),
+            "block_id": str(block_id),
+            "file_path": file_path,
+            "point_count": counts.astype(np.uint64),
+        })
+        # Short-lived connection: the driver attaches these files for
+        # consolidation while worker processes may still be alive.
+        con = duckdb.connect(self.db_path)
+        con.execute(_INDEX_SCHEMA.format(name="point_cloud_index"))
+        con.register("new_rows", index)
+        con.execute("INSERT INTO point_cloud_index SELECT * FROM new_rows")
+        con.close()
+
+
+def consolidate_indexes(out_dir: str) -> str:
+    """
+    Merge all worker indexes into ``unified_index.db`` (what pocaduck's Query
+    reads). Replaces ``Ingestor.consolidate_indexes``, which pulls every row
+    through Python and was run concurrently by every worker.
+    """
+    output_path = os.path.join(out_dir, "unified_index.db")
+    if os.path.exists(output_path):
+        os.remove(output_path)
+    con = duckdb.connect(output_path)
+    con.execute(_INDEX_SCHEMA.format(name="point_cloud_index"))
+    for path in sorted(glob.glob(os.path.join(out_dir, "worker_*", "index_*.db"))):
+        con.execute(f"ATTACH '{path}' AS w (READ_ONLY)")
+        con.execute("INSERT INTO point_cloud_index SELECT * FROM w.point_cloud_index")
+        con.execute("DETACH w")
+    con.close()
+    return output_path

nps_cli-0.1.5/nps/main.py

@@ -0,0 +1,117 @@
+import faulthandler
+import os
+import time
+
+import click
+import numpy as np
+from funlib.geometry import Coordinate, Roi
+from volara.datasets import CloudVolumeWrapper
+from volara.workers import LSFWorker, LocalWorker, SlurmWorker
+
+from nps.blockwise.block_mask import compute_block_mask
+from nps.blockwise.sample_points import SamplePoints, consolidate_indexes
+
+
+@click.command(context_settings=dict(help_option_names=['-h', '--help']))
+@click.option('--cv-path', required=True, help='Path to CloudVolume data.')
+@click.option('--svid-cv-path', default=None, help='Separate CloudVolume path holding supervoxel IDs (e.g. a precomputed watershed volume). Implies SVID sampling. Must share the voxel grid of --cv-path at the chosen mip.')
+@click.option('--mip', default=0, type=int, show_default=True, help='MIP level to use.')
+@click.option('--timestamp', default=int(time.time()), help='Optional timestamp for the dataset version (graphene only).')
+@click.option('--sample_svids', is_flag=True, default=False, help='Sample SVIDs in addition to points (default: False). Without --svid-cv-path, supervoxels are read from --cv-path with agglomerate=False (graphene only).')
+@click.option('--fill-missing', is_flag=True, default=False, help='Return 0 for missing chunks instead of raising an error (default: False).')
+@click.option('--output-dir', '-o', default='./nps_output', show_default=True, type=click.Path(file_okay=False, writable=True), help='Output directory.')
+@click.option('--overwrite', is_flag=True, default=False, help='Delete points from an earlier run in --output-dir before sampling (default: False).')
+@click.option('--worker-type', default='LocalWorker', show_default=True, type=click.Choice(['LocalWorker', 'LSFWorker', 'SlurmWorker'], case_sensitive=True), help='Type of worker to use for sampling.')
+@click.option('--num-workers', default=8, show_default=True, type=int, help='Number of workers for blockwise sampling.')
+@click.option('--cpus-per-worker', default=4, show_default=True, type=int, help='Number of CPUs per worker.')
+@click.option('--queue', default='local', show_default=True, help='Queue name (LSF/Slurm backends).')
+@click.option('--fraction', default=0.001, show_default=True, type=float, help='Fraction of points to sample [0.0, 1.0].')
+@click.option('--block-size', nargs=3, type=int, default=(128, 128, 128), show_default=True, help='Block size in voxels (X Y Z).')
+@click.option('--bbox', nargs=6, type=int, default=None, help='Bounding box: begin_x begin_y begin_z end_x end_y end_z (in voxels).')
+@click.option('--skip-empty/--no-skip-empty', default=True, show_default=True, help='Pre-scan a coarse mip to find blocks without labels and skip them.')
+@click.option('--mask-mip', default=None, type=int, help='Mip level used for the empty-block pre-scan. [default: coarsest available]')
+@click.option('--block-mask', default=None, type=click.Path(exists=True, dir_okay=False), help='Reuse an existing block_mask.npy from a previous run with identical --bbox, --block-size and --mip instead of re-scanning.')
+def main(cv_path, svid_cv_path, mip, timestamp, output_dir, num_workers, cpus_per_worker, queue, fraction, block_size, sample_svids, worker_type, bbox, fill_missing, skip_empty, mask_mip, block_mask, overwrite):
+
+    # On a native crash (segfault) dump every thread's Python stack to stderr so
+    # the cause shows up in the job log instead of a bare "exit code 139".
+    faulthandler.enable()
+
+    output_dir = os.path.abspath(output_dir)
+    points_dir = os.path.join(output_dir, 'points')
+    # Every run re-samples all blocks, so points left over from an earlier run
+    # would end up as duplicates. Check before the (slow) empty-block pre-scan.
+    if os.path.isdir(points_dir) and os.listdir(points_dir):
+        if not overwrite:
+            raise click.UsageError(
+                f"{points_dir} already holds points from an earlier run. "
+                "Pass --overwrite to delete them or choose another --output-dir."
+            )
+        click.echo(f"--overwrite: deleting points from an earlier run in {points_dir}")
+    os.makedirs(points_dir, exist_ok=True)
+
+    click.echo(f"Reading CloudVolume at {cv_path} (mip={mip}, timestamp={timestamp})")
+
+    base = os.path.basename(output_dir)
+    common = dict(mip=mip, timestamp=timestamp, fill_missing=fill_missing)
+    labels = CloudVolumeWrapper(data_name=base + "_labels", store=cv_path, **common)
+
+    if sample_svids or svid_cv_path:
+        svid_store = svid_cv_path or cv_path
+        click.echo(f"Sampling SVIDs in addition to points from {svid_store}...")
+        # agglomerate=False selects the watershed layer for graphene and is
+        # ignored by precomputed volumes, so one code path serves both cases.
+        svids = CloudVolumeWrapper(data_name=base + "_svids", store=svid_store, agglomerate=False, **common)
+    else:
+        click.echo("Sampling points only (no SVIDs)...")
+        svids = None
+
+    if worker_type == 'LocalWorker':
+        click.echo("Using LocalWorker for sampling.")
+        worker_config = LocalWorker()
+    elif worker_type == 'LSFWorker':
+        click.echo(f"Using LSFWorker with queue '{queue}' and {cpus_per_worker} CPUs per worker.")
+        worker_config = LSFWorker(queue=queue, num_cpus=cpus_per_worker)
+    elif worker_type == 'SlurmWorker':
+        click.echo(f"Using SlurmWorker with queue '{queue}' and {cpus_per_worker} CPUs per worker.")
+        worker_config = SlurmWorker(queue=queue, num_cpus=cpus_per_worker)
+
+    roi = None
+    if bbox is not None:
+        begin, end = bbox[:3], bbox[3:]
+        roi = (begin, tuple(e - b for b, e in zip(begin, end)))
+
+    if block_mask is None and skip_empty:
+        total_roi = labels.array("r").roi
+        if roi is not None:
+            total_roi = total_roi.intersect(Roi(*roi))
+        click.echo(f"Scanning mip {mask_mip if mask_mip is not None else 'coarsest'} for empty blocks in {total_roi}...")
+        mask = compute_block_mask(labels, total_roi, Coordinate(*block_size), mask_mip=mask_mip, num_workers=num_workers)
+        block_mask = os.path.join(output_dir, "block_mask.npy")
+        np.save(block_mask, mask)
+        click.echo(f"{mask.sum():,} of {mask.size:,} blocks ({mask.mean():.0%}) contain labels; the rest will be skipped. Mask saved to {block_mask}")
+    elif block_mask is not None:
+        click.echo(f"Using block mask {block_mask}")
+
+    task = SamplePoints(
+        labels=labels,
+        svids=svids,
+        block_mask=block_mask,
+        block_size=np.array(block_size),
+        num_workers=num_workers,
+        out_dir=points_dir,
+        fraction=fraction,
+        worker_config=worker_config,
+        roi=roi,
+    )
+
+    click.echo("Running task...")
+    task.drop()  # also deletes points_dir (see SamplePoints.drop_artifacts)
+    task.run_blockwise(multiprocessing=True)
+    click.echo("Consolidating worker indexes...")
+    consolidate_indexes(points_dir)
+    click.secho("✅ Done!", fg='green')
+
+
+if __name__ == "__main__":
+    main()

nps_cli-0.1.5/nps_cli.egg-info/PKG-INFO

@@ -0,0 +1,119 @@
+Metadata-Version: 2.4
+Name: nps-cli
+Version: 0.1.5
+Summary: A simple CLI for sampling point clouds from large volumetric datasets
+Author: Jakob Troidl
+License-Expression: MIT
+Classifier: Programming Language :: Python :: 3.11
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown
+Requires-Dist: click
+Requires-Dist: cloud-volume!=12.13.*
+Requires-Dist: volara>=2.0.1
+Requires-Dist: pocaduck
+Provides-Extra: dev
+Requires-Dist: pytest; extra == "dev"
+Requires-Dist: black; extra == "dev"
+Requires-Dist: build; extra == "dev"
+Requires-Dist: twine; extra == "dev"
+Requires-Dist: ruff; extra == "dev"
+
+[![pypi](https://badge.fury.io/py/nps-cli.svg)](https://badge.fury.io/py/nps-cli)
+[![Python Version](https://img.shields.io/pypi/pyversions/nps-cli.svg)](https://pypi.org/project/nps-cli/)
+
+# nps: distributed point cloud sampling
+
+`nps` samples point clouds of every segment in a large segmentation volumes
+* Compatible with [CloudVolume](https://github.com/seung-lab/cloud-volume) (precomputed or graphene).
+* It processes the volume blockwise on local, LSF or Slurm workers and writes the points in [pocaduck](https://github.com/JaneliaSciComp/pocaduck) format.
+
+```bash
+pip install nps-cli
+nps --help
+```
+
+## Usage
+
+```bash
+# Whole volume
+nps --cv-path precomputed://gs://neuroglancer-janelia-flyem-hemibrain/v1.0/segmentation
+
+# Subvolume (begin_x begin_y begin_z end_x end_y end_z, in voxels), 1% of each segment
+nps --cv-path precomputed://gs://neuroglancer-janelia-flyem-hemibrain/v1.0/segmentation \
+    --bbox 15347 19712 18606 15859 20224 19118 --fraction 0.01
+
+# Also store the supervoxel ID of every point, looked up in a separate volume
+# that shares the voxel grid of --cv-path
+nps --cv-path precomputed://gs://<bucket>/agglomerated \
+    --svid-cv-path precomputed://gs://<bucket>/supervoxels
+
+# Graphene: read supervoxels from the same volume (agglomerate=False)
+nps --cv-path graphene://https://<server>/segmentation/table/<table> --sample_svids
+```
+
+Key options (run `nps --help` for all of them):
+
+| Option | Default | Meaning |
+| --- | --- | --- |
+| `--fraction` | `0.001` | Fraction of each segment's voxels to sample (at least 1 per segment and block). |
+| `--mip` | `0` | Resolution level to sample at. Coordinates are voxels of this mip. |
+| `--bbox` | whole volume | ROI in voxels. |
+| `--block-size` | `128 128 128` | Block size per task, in voxels. |
+| `--worker-type` / `--num-workers` / `--cpus-per-worker` / `--queue` | `LocalWorker` / `8` / `4` / `local` | Where and how the blocks are processed. |
+| `--fill-missing` | off | Treat missing chunks as empty instead of raising an error. |
+| `--overwrite` | off | Delete points of an earlier run in `--output-dir` instead of refusing to start. |
+| `--skip-empty` / `--block-mask` / `--mask-mip` | on | Skipping empty blocks, see below. |
+
+### Output
+
+```
+<output-dir>/
+  block_mask.npy            # occupancy mask from the empty-block pre-scan
+  points/
+    unified_index.db        # pocaduck index over all workers
+    worker_<id>/...         # parquet files + per-worker index
+```
+
+Each point is `[x, y, z]`, or `[x, y, z, svid]` when SVIDs are sampled, stored under its segment label.
+
+`nps` refuses to write into an `--output-dir` that already holds points, since every
+run re-samples all blocks and old points would show up as duplicates. Pass
+`--overwrite` to delete them first.
+
+### Skipping empty blocks
+
+Before sampling, `nps` scans the ROI at a coarse mip (`--mask-mip`, default the
+coarsest one) and never dispatches blocks without labels. The mask is dilated by
+one block, so thin structures lost in downsampling are still sampled. Use
+`--no-skip-empty` to disable the scan, or `--block-mask <path>` to reuse a
+`block_mask.npy` from an earlier run with the same `--bbox`, `--block-size` and `--mip`.
+
+## Reading point clouds
+
+```python
+from pocaduck import Query, StorageConfig
+
+query = Query(storage_config=StorageConfig(base_path="<output-dir>/points"))
+labels = query.get_labels()
+points = query.get_points(label=12345)  # all points of one segment
+query.close()
+```
+
+See the [pocaduck docs](https://github.com/JaneliaSciComp/pocaduck?tab=readme-ov-file#querying-point-clouds)
+for more, including [optimizing](https://github.com/JaneliaSciComp/pocaduck?tab=readme-ov-file#running-the-optimization-pipeline) the storage for fast reads.
+
+## Example scripts
+
+| Script | What it does |
+| --- | --- |
+| [`liconn_local.sh`](scripts/liconn_local.sh) | Points only, local workers |
+| [`liconn_svids_local.sh`](scripts/liconn_svids_local.sh) | Points + SVIDs from a separate precomputed volume |
+| [`flywire_cluster.sh`](scripts/flywire_cluster.sh) | Points + SVIDs for FlyWire on LSF (`LABEL_NAME=flywire_v141_m783` for the 2023 snapshot) |
+
+## Release
+
+```bash
+pip install -e ".[dev]"
+python -m build
+twine upload dist/*
+```

{nps_cli-0.1.2 → nps_cli-0.1.5}/nps_cli.egg-info/SOURCES.txt

@@ -2,6 +2,9 @@ README.md
 pyproject.toml
 nps/__init__.py
 nps/main.py
+nps/blockwise/__init__.py
+nps/blockwise/block_mask.py
+nps/blockwise/sample_points.py
 nps_cli.egg-info/PKG-INFO
 nps_cli.egg-info/SOURCES.txt
 nps_cli.egg-info/dependency_links.txt

{nps_cli-0.1.2 → nps_cli-0.1.5}/nps_cli.egg-info/requires.txt

@@ -1,5 +1,6 @@
 click
-volara
+cloud-volume!=12.13.*
+volara>=2.0.1
 pocaduck
 
 [dev]

nps_cli-0.1.5/nps_cli.egg-info/top_level.txt

@@ -0,0 +1,2 @@
+nps
+nps_output

{nps_cli-0.1.2 → nps_cli-0.1.5}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "nps-cli"
-version = "0.1.2"
+version = "0.1.5"
 description = "A simple CLI for sampling point clouds from large volumetric datasets"
 readme = { file = "README.md", content-type = "text/markdown" }
 requires-python = ">=3.11"
@@ -12,7 +12,11 @@ license = "MIT"
 authors = [{ name = "Jakob Troidl" }]
 dependencies = [
     "click",
-    "volara",
+    # 12.13.x passes the nominal chunk size instead of the real cutout size to
+    # the compressed_segmentation decoder for sharded chunks; partial chunks at
+    # the volume edge then segfault (fixed again in 12.14).
+    "cloud-volume!=12.13.*",
+    "volara>=2.0.1",
     "pocaduck",
 ]
 
@@ -36,7 +40,7 @@ sample-points = "nps.blockwise:SamplePoints"
 nps = "nps.main:main"
 
 [tool.setuptools.packages.find]
-include = ["nps"]
+include = ["nps*"]
 
 [tool.ruff]
 exclude = ["dist", "build", "nps.egg-info", "volara_logs", "nps_output"]

nps_cli-0.1.2/PKG-INFO

@@ -1,100 +0,0 @@
-Metadata-Version: 2.4
-Name: nps-cli
-Version: 0.1.2
-Summary: A simple CLI for sampling point clouds from large volumetric datasets
-Author: Jakob Troidl
-License-Expression: MIT
-Classifier: Programming Language :: Python :: 3.11
-Requires-Python: >=3.11
-Description-Content-Type: text/markdown
-Requires-Dist: click
-Requires-Dist: volara
-Requires-Dist: pocaduck
-Provides-Extra: dev
-Requires-Dist: pytest; extra == "dev"
-Requires-Dist: black; extra == "dev"
-Requires-Dist: build; extra == "dev"
-Requires-Dist: twine; extra == "dev"
-Requires-Dist: ruff; extra == "dev"
-
-[![pypi](https://badge.fury.io/py/nps-cli.svg)](https://badge.fury.io/py/nps-cli)
-[![Python Version](https://img.shields.io/pypi/pyversions/nps-cli.svg)](https://pypi.org/project/nps-cli/)
-
-# CLI for Distributed Point Cloud Sampling
-
-```
-pip install nps-cli
-````
-### Get started
-
-```zsh
-nps --help
-Usage: nps [OPTIONS]
-
-Options:
-  --cv-path TEXT                  Path to CloudVolume data.  [required]
-  --mip INTEGER                   MIP level to use.  [default: 0]
-  --timestamp INTEGER             Optional timestamp for the dataset version
-                                  (graphene only).
-  --sample_svids                  Sample SVIDs in addition to points (default:
-                                  False) Graphene only.
-  -o, --output-dir DIRECTORY      Output directory.  [default: ./nps_output]
-  --worker-type [LocalWorker|LSFWorker|SlurmWorker]
-                                  Type of worker to use for sampling.
-                                  [default: LocalWorker]
-  --num-workers INTEGER           Number of workers for blockwise sampling.
-                                  [default: 8]
-  --cpus-per-worker INTEGER       Number of CPUs per worker.  [default: 4]
-  --queue TEXT                    Queue name (for LSF backend).  [default:
-                                  local]
-  --fraction FLOAT                Fraction of points to sample [0.0, 1.0].
-                                  [default: 0.001]
-  --bbox INTEGER...               Bounding box: begin_x begin_y begin_z
-                                  end_x end_y end_z (in voxels).
-  --block-size INTEGER...         Block size in voxels (X Y Z).  [default:
-                                  128, 128, 128]
-  -h, --help                      Show this message and exit.
-```
-
-### Example usage
-
-```bash
-nps --cv-path precomputed://gs://neuroglancer-janelia-flyem-hemibrain/v1.0/segmentation
-```
-
-Sample point clouds within a FlyEM Hemibrain subvolume: 
-
-```bash
-nps --cv-path precomputed://gs://neuroglancer-janelia-flyem-hemibrain/v1.0/segmentation --bbox 15347 19712 18606 15859 20224 19118 --fraction 0.01
-```
-
-### Reading Point Clouds
-
-Please refer to the [pocaduck](https://github.com/JaneliaSciComp/pocaduck?tab=readme-ov-file#querying-point-clouds) repo on how to read point clouds from the output directory:
-
-```python
-from pocaduck import Query
-
-# Create a query object
-query = Query(storage_config=<PATH>) # path to folder where nps output is stored
-
-# Get all available labels
-labels = query.get_labels()
-print(f"Available labels: {labels}")
-
-# Get all points for a label (aggregated across all blocks)
-points = query.get_points(label=12345)
-print(f"Retrieved {points.shape[0]} points for label 12345")
-
-# Close the query connection when done
-query.close()
-```
-
-For optimized point cloud reading, consider [this](https://github.com/JaneliaSciComp/pocaduck?tab=readme-ov-file#running-the-optimization-pipeline).
-
-
-### Deploy
-```python
-python -m build
-twine upload dist/*
-```

nps_cli-0.1.2/README.md

@@ -1,81 +0,0 @@
-[![pypi](https://badge.fury.io/py/nps-cli.svg)](https://badge.fury.io/py/nps-cli)
-[![Python Version](https://img.shields.io/pypi/pyversions/nps-cli.svg)](https://pypi.org/project/nps-cli/)
-
-# CLI for Distributed Point Cloud Sampling
-
-```
-pip install nps-cli
-````
-### Get started
-
-```zsh
-nps --help
-Usage: nps [OPTIONS]
-
-Options:
-  --cv-path TEXT                  Path to CloudVolume data.  [required]
-  --mip INTEGER                   MIP level to use.  [default: 0]
-  --timestamp INTEGER             Optional timestamp for the dataset version
-                                  (graphene only).
-  --sample_svids                  Sample SVIDs in addition to points (default:
-                                  False) Graphene only.
-  -o, --output-dir DIRECTORY      Output directory.  [default: ./nps_output]
-  --worker-type [LocalWorker|LSFWorker|SlurmWorker]
-                                  Type of worker to use for sampling.
-                                  [default: LocalWorker]
-  --num-workers INTEGER           Number of workers for blockwise sampling.
-                                  [default: 8]
-  --cpus-per-worker INTEGER       Number of CPUs per worker.  [default: 4]
-  --queue TEXT                    Queue name (for LSF backend).  [default:
-                                  local]
-  --fraction FLOAT                Fraction of points to sample [0.0, 1.0].
-                                  [default: 0.001]
-  --bbox INTEGER...               Bounding box: begin_x begin_y begin_z
-                                  end_x end_y end_z (in voxels).
-  --block-size INTEGER...         Block size in voxels (X Y Z).  [default:
-                                  128, 128, 128]
-  -h, --help                      Show this message and exit.
-```
-
-### Example usage
-
-```bash
-nps --cv-path precomputed://gs://neuroglancer-janelia-flyem-hemibrain/v1.0/segmentation
-```
-
-Sample point clouds within a FlyEM Hemibrain subvolume: 
-
-```bash
-nps --cv-path precomputed://gs://neuroglancer-janelia-flyem-hemibrain/v1.0/segmentation --bbox 15347 19712 18606 15859 20224 19118 --fraction 0.01
-```
-
-### Reading Point Clouds
-
-Please refer to the [pocaduck](https://github.com/JaneliaSciComp/pocaduck?tab=readme-ov-file#querying-point-clouds) repo on how to read point clouds from the output directory:
-
-```python
-from pocaduck import Query
-
-# Create a query object
-query = Query(storage_config=<PATH>) # path to folder where nps output is stored
-
-# Get all available labels
-labels = query.get_labels()
-print(f"Available labels: {labels}")
-
-# Get all points for a label (aggregated across all blocks)
-points = query.get_points(label=12345)
-print(f"Retrieved {points.shape[0]} points for label 12345")
-
-# Close the query connection when done
-query.close()
-```
-
-For optimized point cloud reading, consider [this](https://github.com/JaneliaSciComp/pocaduck?tab=readme-ov-file#running-the-optimization-pipeline).
-
-
-### Deploy
-```python
-python -m build
-twine upload dist/*
-```

nps_cli-0.1.2/nps/main.py

@@ -1,77 +0,0 @@
-import os
-import numpy as np
-import click
-import time
-
-from nps.blockwise.sample_points import SamplePoints
-from volara.datasets import CloudVolumeWrapper
-from volara.workers import LSFWorker, LocalWorker, SlurmWorker
-
-@click.command(context_settings=dict(help_option_names=['-h', '--help']))
-@click.option('--cv-path', required=True, help='Path to CloudVolume data.')
-@click.option('--mip', default=0, type=int, show_default=True, help='MIP level to use.')
-@click.option('--timestamp', default=int(time.time()), help='Optional timestamp for the dataset version (graphene only).')
-@click.option('--sample_svids', is_flag=True, default=False, help='Sample SVIDs in addition to points (default: False).')
-@click.option('--output-dir', '-o', default='./nps_output', show_default=True, type=click.Path(file_okay=False, writable=True), help='Output directory.')
-@click.option('--worker-type', default='LocalWorker', show_default=True, type=click.Choice(['LocalWorker', 'LSFWorker', 'SlurmWorker'], case_sensitive=True), help='Type of worker to use for sampling.')
-@click.option('--num-workers', default=8, show_default=True, type=int, help='Number of workers for blockwise sampling.')
-@click.option('--cpus-per-worker', default=4, show_default=True, type=int, help='Number of CPUs per worker.')
-@click.option('--queue', default='local', show_default=True, help='Queue name (for LSF backend).')
-@click.option('--fraction', default=0.001, show_default=True, type=float, help='Fraction of points to sample [0.0, 1.0].')
-@click.option('--block-size', nargs=3, type=int, default=(128, 128, 128), show_default=True, help='Block size in voxels (X Y Z).')
-@click.option('--bbox', nargs=6, type=int, default=None, help='Bounding box: begin_x begin_y begin_z end_x end_y end_z (in voxels).')
-def main(cv_path, mip, timestamp, output_dir, num_workers, cpus_per_worker, queue, fraction, block_size, sample_svids, worker_type, bbox):
-
-    click.echo(f"Reading CloudVolume at {cv_path} (mip={mip}, timestamp={timestamp})")
-
-    labels_name = os.path.dirname(output_dir) + "_labels"
-    labels = CloudVolumeWrapper(data_name=labels_name, store=cv_path, mip=mip, timestamp=timestamp)
-
-    if sample_svids:
-        click.echo("Sampling SVIDs in addition to points...")
-        svids_name = os.path.dirname(output_dir) + "_svids"
-        svids = CloudVolumeWrapper(data_name=svids_name, store=cv_path, mip=mip, timestamp=timestamp, agglomerate=False)
-    else:
-        click.echo("Sampling points only (no SVIDs)...")
-        svids = None
-
-    points_dir = os.path.join(output_dir, 'points')
-    if not os.path.exists(points_dir):
-        os.makedirs(points_dir)
-
-    if worker_type == 'LocalWorker':
-        click.echo("Using LocalWorker for sampling.")
-        worker_config = LocalWorker()
-    elif worker_type == 'LSFWorker':
-        click.echo(f"Using LSFWorker with queue '{queue}' and {cpus_per_worker} CPUs per worker.")
-        worker_config = LSFWorker(queue=queue, num_cpus=cpus_per_worker)
-    elif worker_type == 'SlurmWorker':
-        click.echo(f"Using SlurmWorker with queue '{queue}' and {cpus_per_worker} CPUs per worker.")
-        worker_config = SlurmWorker(queue=queue, num_cpus=cpus_per_worker)
-
-    roi = None
-    if bbox is not None:
-        begin = bbox[:3]
-        end = bbox[3:]
-        shape = tuple(e - b for b, e in zip(begin, end))
-        roi = (begin, shape)
-
-    task = SamplePoints(
-        labels=labels,
-        svids=svids,
-        block_size=np.array(block_size),
-        num_workers=num_workers,
-        out_dir=points_dir,
-        fraction=fraction,
-        worker_config=worker_config,
-        roi=roi,
-    )
-
-    click.echo("Running task...")
-    task.drop()
-    task.run_blockwise(multiprocessing=True)
-    click.secho("✅ Done!", fg='green')
-
-
-if __name__ == "__main__":
-    main()

nps_cli-0.1.2/nps_cli.egg-info/PKG-INFO

@@ -1,100 +0,0 @@
-Metadata-Version: 2.4
-Name: nps-cli
-Version: 0.1.2
-Summary: A simple CLI for sampling point clouds from large volumetric datasets
-Author: Jakob Troidl
-License-Expression: MIT
-Classifier: Programming Language :: Python :: 3.11
-Requires-Python: >=3.11
-Description-Content-Type: text/markdown
-Requires-Dist: click
-Requires-Dist: volara
-Requires-Dist: pocaduck
-Provides-Extra: dev
-Requires-Dist: pytest; extra == "dev"
-Requires-Dist: black; extra == "dev"
-Requires-Dist: build; extra == "dev"
-Requires-Dist: twine; extra == "dev"
-Requires-Dist: ruff; extra == "dev"
-
-[![pypi](https://badge.fury.io/py/nps-cli.svg)](https://badge.fury.io/py/nps-cli)
-[![Python Version](https://img.shields.io/pypi/pyversions/nps-cli.svg)](https://pypi.org/project/nps-cli/)
-
-# CLI for Distributed Point Cloud Sampling
-
-```
-pip install nps-cli
-````
-### Get started
-
-```zsh
-nps --help
-Usage: nps [OPTIONS]
-
-Options:
-  --cv-path TEXT                  Path to CloudVolume data.  [required]
-  --mip INTEGER                   MIP level to use.  [default: 0]
-  --timestamp INTEGER             Optional timestamp for the dataset version
-                                  (graphene only).
-  --sample_svids                  Sample SVIDs in addition to points (default:
-                                  False) Graphene only.
-  -o, --output-dir DIRECTORY      Output directory.  [default: ./nps_output]
-  --worker-type [LocalWorker|LSFWorker|SlurmWorker]
-                                  Type of worker to use for sampling.
-                                  [default: LocalWorker]
-  --num-workers INTEGER           Number of workers for blockwise sampling.
-                                  [default: 8]
-  --cpus-per-worker INTEGER       Number of CPUs per worker.  [default: 4]
-  --queue TEXT                    Queue name (for LSF backend).  [default:
-                                  local]
-  --fraction FLOAT                Fraction of points to sample [0.0, 1.0].
-                                  [default: 0.001]
-  --bbox INTEGER...               Bounding box: begin_x begin_y begin_z
-                                  end_x end_y end_z (in voxels).
-  --block-size INTEGER...         Block size in voxels (X Y Z).  [default:
-                                  128, 128, 128]
-  -h, --help                      Show this message and exit.
-```
-
-### Example usage
-
-```bash
-nps --cv-path precomputed://gs://neuroglancer-janelia-flyem-hemibrain/v1.0/segmentation
-```
-
-Sample point clouds within a FlyEM Hemibrain subvolume: 
-
-```bash
-nps --cv-path precomputed://gs://neuroglancer-janelia-flyem-hemibrain/v1.0/segmentation --bbox 15347 19712 18606 15859 20224 19118 --fraction 0.01
-```
-
-### Reading Point Clouds
-
-Please refer to the [pocaduck](https://github.com/JaneliaSciComp/pocaduck?tab=readme-ov-file#querying-point-clouds) repo on how to read point clouds from the output directory:
-
-```python
-from pocaduck import Query
-
-# Create a query object
-query = Query(storage_config=<PATH>) # path to folder where nps output is stored
-
-# Get all available labels
-labels = query.get_labels()
-print(f"Available labels: {labels}")
-
-# Get all points for a label (aggregated across all blocks)
-points = query.get_points(label=12345)
-print(f"Retrieved {points.shape[0]} points for label 12345")
-
-# Close the query connection when done
-query.close()
-```
-
-For optimized point cloud reading, consider [this](https://github.com/JaneliaSciComp/pocaduck?tab=readme-ov-file#running-the-optimization-pipeline).
-
-
-### Deploy
-```python
-python -m build
-twine upload dist/*
-```

nps_cli-0.1.2/nps_cli.egg-info/top_level.txt

@@ -1 +0,0 @@
-nps