nps-cli 0.1.3__tar.gz → 0.1.5__tar.gz

nps_cli-0.1.5/PKG-INFO

@@ -0,0 +1,119 @@
+Metadata-Version: 2.4
+Name: nps-cli
+Version: 0.1.5
+Summary: A simple CLI for sampling point clouds from large volumetric datasets
+Author: Jakob Troidl
+License-Expression: MIT
+Classifier: Programming Language :: Python :: 3.11
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown
+Requires-Dist: click
+Requires-Dist: cloud-volume!=12.13.*
+Requires-Dist: volara>=2.0.1
+Requires-Dist: pocaduck
+Provides-Extra: dev
+Requires-Dist: pytest; extra == "dev"
+Requires-Dist: black; extra == "dev"
+Requires-Dist: build; extra == "dev"
+Requires-Dist: twine; extra == "dev"
+Requires-Dist: ruff; extra == "dev"
+
+[![pypi](https://badge.fury.io/py/nps-cli.svg)](https://badge.fury.io/py/nps-cli)
+[![Python Version](https://img.shields.io/pypi/pyversions/nps-cli.svg)](https://pypi.org/project/nps-cli/)
+
+# nps: distributed point cloud sampling
+
+`nps` samples point clouds of every segment in a large segmentation volumes
+* Compatible with [CloudVolume](https://github.com/seung-lab/cloud-volume) (precomputed or graphene).
+* It processes the volume blockwise on local, LSF or Slurm workers and writes the points in [pocaduck](https://github.com/JaneliaSciComp/pocaduck) format.
+
+```bash
+pip install nps-cli
+nps --help
+```
+
+## Usage
+
+```bash
+# Whole volume
+nps --cv-path precomputed://gs://neuroglancer-janelia-flyem-hemibrain/v1.0/segmentation
+
+# Subvolume (begin_x begin_y begin_z end_x end_y end_z, in voxels), 1% of each segment
+nps --cv-path precomputed://gs://neuroglancer-janelia-flyem-hemibrain/v1.0/segmentation \
+    --bbox 15347 19712 18606 15859 20224 19118 --fraction 0.01
+
+# Also store the supervoxel ID of every point, looked up in a separate volume
+# that shares the voxel grid of --cv-path
+nps --cv-path precomputed://gs://<bucket>/agglomerated \
+    --svid-cv-path precomputed://gs://<bucket>/supervoxels
+
+# Graphene: read supervoxels from the same volume (agglomerate=False)
+nps --cv-path graphene://https://<server>/segmentation/table/<table> --sample_svids
+```
+
+Key options (run `nps --help` for all of them):
+
+| Option | Default | Meaning |
+| --- | --- | --- |
+| `--fraction` | `0.001` | Fraction of each segment's voxels to sample (at least 1 per segment and block). |
+| `--mip` | `0` | Resolution level to sample at. Coordinates are voxels of this mip. |
+| `--bbox` | whole volume | ROI in voxels. |
+| `--block-size` | `128 128 128` | Block size per task, in voxels. |
+| `--worker-type` / `--num-workers` / `--cpus-per-worker` / `--queue` | `LocalWorker` / `8` / `4` / `local` | Where and how the blocks are processed. |
+| `--fill-missing` | off | Treat missing chunks as empty instead of raising an error. |
+| `--overwrite` | off | Delete points of an earlier run in `--output-dir` instead of refusing to start. |
+| `--skip-empty` / `--block-mask` / `--mask-mip` | on | Skipping empty blocks, see below. |
+
+### Output
+
+```
+<output-dir>/
+  block_mask.npy            # occupancy mask from the empty-block pre-scan
+  points/
+    unified_index.db        # pocaduck index over all workers
+    worker_<id>/...         # parquet files + per-worker index
+```
+
+Each point is `[x, y, z]`, or `[x, y, z, svid]` when SVIDs are sampled, stored under its segment label.
+
+`nps` refuses to write into an `--output-dir` that already holds points, since every
+run re-samples all blocks and old points would show up as duplicates. Pass
+`--overwrite` to delete them first.
+
+### Skipping empty blocks
+
+Before sampling, `nps` scans the ROI at a coarse mip (`--mask-mip`, default the
+coarsest one) and never dispatches blocks without labels. The mask is dilated by
+one block, so thin structures lost in downsampling are still sampled. Use
+`--no-skip-empty` to disable the scan, or `--block-mask <path>` to reuse a
+`block_mask.npy` from an earlier run with the same `--bbox`, `--block-size` and `--mip`.
+
+## Reading point clouds
+
+```python
+from pocaduck import Query, StorageConfig
+
+query = Query(storage_config=StorageConfig(base_path="<output-dir>/points"))
+labels = query.get_labels()
+points = query.get_points(label=12345)  # all points of one segment
+query.close()
+```
+
+See the [pocaduck docs](https://github.com/JaneliaSciComp/pocaduck?tab=readme-ov-file#querying-point-clouds)
+for more, including [optimizing](https://github.com/JaneliaSciComp/pocaduck?tab=readme-ov-file#running-the-optimization-pipeline) the storage for fast reads.
+
+## Example scripts
+
+| Script | What it does |
+| --- | --- |
+| [`liconn_local.sh`](scripts/liconn_local.sh) | Points only, local workers |
+| [`liconn_svids_local.sh`](scripts/liconn_svids_local.sh) | Points + SVIDs from a separate precomputed volume |
+| [`flywire_cluster.sh`](scripts/flywire_cluster.sh) | Points + SVIDs for FlyWire on LSF (`LABEL_NAME=flywire_v141_m783` for the 2023 snapshot) |
+
+## Release
+
+```bash
+pip install -e ".[dev]"
+python -m build
+twine upload dist/*
+```

nps_cli-0.1.5/README.md

@@ -0,0 +1,99 @@
+[![pypi](https://badge.fury.io/py/nps-cli.svg)](https://badge.fury.io/py/nps-cli)
+[![Python Version](https://img.shields.io/pypi/pyversions/nps-cli.svg)](https://pypi.org/project/nps-cli/)
+
+# nps: distributed point cloud sampling
+
+`nps` samples point clouds of every segment in a large segmentation volumes
+* Compatible with [CloudVolume](https://github.com/seung-lab/cloud-volume) (precomputed or graphene).
+* It processes the volume blockwise on local, LSF or Slurm workers and writes the points in [pocaduck](https://github.com/JaneliaSciComp/pocaduck) format.
+
+```bash
+pip install nps-cli
+nps --help
+```
+
+## Usage
+
+```bash
+# Whole volume
+nps --cv-path precomputed://gs://neuroglancer-janelia-flyem-hemibrain/v1.0/segmentation
+
+# Subvolume (begin_x begin_y begin_z end_x end_y end_z, in voxels), 1% of each segment
+nps --cv-path precomputed://gs://neuroglancer-janelia-flyem-hemibrain/v1.0/segmentation \
+    --bbox 15347 19712 18606 15859 20224 19118 --fraction 0.01
+
+# Also store the supervoxel ID of every point, looked up in a separate volume
+# that shares the voxel grid of --cv-path
+nps --cv-path precomputed://gs://<bucket>/agglomerated \
+    --svid-cv-path precomputed://gs://<bucket>/supervoxels
+
+# Graphene: read supervoxels from the same volume (agglomerate=False)
+nps --cv-path graphene://https://<server>/segmentation/table/<table> --sample_svids
+```
+
+Key options (run `nps --help` for all of them):
+
+| Option | Default | Meaning |
+| --- | --- | --- |
+| `--fraction` | `0.001` | Fraction of each segment's voxels to sample (at least 1 per segment and block). |
+| `--mip` | `0` | Resolution level to sample at. Coordinates are voxels of this mip. |
+| `--bbox` | whole volume | ROI in voxels. |
+| `--block-size` | `128 128 128` | Block size per task, in voxels. |
+| `--worker-type` / `--num-workers` / `--cpus-per-worker` / `--queue` | `LocalWorker` / `8` / `4` / `local` | Where and how the blocks are processed. |
+| `--fill-missing` | off | Treat missing chunks as empty instead of raising an error. |
+| `--overwrite` | off | Delete points of an earlier run in `--output-dir` instead of refusing to start. |
+| `--skip-empty` / `--block-mask` / `--mask-mip` | on | Skipping empty blocks, see below. |
+
+### Output
+
+```
+<output-dir>/
+  block_mask.npy            # occupancy mask from the empty-block pre-scan
+  points/
+    unified_index.db        # pocaduck index over all workers
+    worker_<id>/...         # parquet files + per-worker index
+```
+
+Each point is `[x, y, z]`, or `[x, y, z, svid]` when SVIDs are sampled, stored under its segment label.
+
+`nps` refuses to write into an `--output-dir` that already holds points, since every
+run re-samples all blocks and old points would show up as duplicates. Pass
+`--overwrite` to delete them first.
+
+### Skipping empty blocks
+
+Before sampling, `nps` scans the ROI at a coarse mip (`--mask-mip`, default the
+coarsest one) and never dispatches blocks without labels. The mask is dilated by
+one block, so thin structures lost in downsampling are still sampled. Use
+`--no-skip-empty` to disable the scan, or `--block-mask <path>` to reuse a
+`block_mask.npy` from an earlier run with the same `--bbox`, `--block-size` and `--mip`.
+
+## Reading point clouds
+
+```python
+from pocaduck import Query, StorageConfig
+
+query = Query(storage_config=StorageConfig(base_path="<output-dir>/points"))
+labels = query.get_labels()
+points = query.get_points(label=12345)  # all points of one segment
+query.close()
+```
+
+See the [pocaduck docs](https://github.com/JaneliaSciComp/pocaduck?tab=readme-ov-file#querying-point-clouds)
+for more, including [optimizing](https://github.com/JaneliaSciComp/pocaduck?tab=readme-ov-file#running-the-optimization-pipeline) the storage for fast reads.
+
+## Example scripts
+
+| Script | What it does |
+| --- | --- |
+| [`liconn_local.sh`](scripts/liconn_local.sh) | Points only, local workers |
+| [`liconn_svids_local.sh`](scripts/liconn_svids_local.sh) | Points + SVIDs from a separate precomputed volume |
+| [`flywire_cluster.sh`](scripts/flywire_cluster.sh) | Points + SVIDs for FlyWire on LSF (`LABEL_NAME=flywire_v141_m783` for the 2023 snapshot) |
+
+## Release
+
+```bash
+pip install -e ".[dev]"
+python -m build
+twine upload dist/*
+```

nps_cli-0.1.5/nps/blockwise/__init__.py

@@ -0,0 +1,2 @@
+from .sample_points import SamplePoints as SamplePoints
+from .sample_points import consolidate_indexes as consolidate_indexes

nps_cli-0.1.5/nps/blockwise/block_mask.py

@@ -0,0 +1,141 @@
+"""
+Per-block occupancy mask computed from a coarse mip level.
+
+Reading a chunk from a (sharded) precomputed volume costs roughly the same
+regardless of how many voxels are requested, so checking each block for
+emptiness at sampling time does not pay off. Instead we make one pass over the
+ROI at the coarsest available mip, where a single chunk covers hundreds of
+sampling blocks, and record for every block whether it contains any label.
+The mask is then used by :class:`SamplePoints` to skip empty blocks inside the
+daisy scheduler, so workers never touch them.
+"""
+
+from concurrent.futures import ThreadPoolExecutor
+from itertools import product
+
+import numpy as np
+from cloudvolume import CloudVolume
+from funlib.geometry import Coordinate, Roi
+from volara.datasets import CloudVolumeWrapper
+
+# Approximate edge length, in coarse voxels, of the region read per job.
+_TARGET_TILE_VOXELS = 128
+
+
+def _open_coarse(store: str, mip: int, timestamp: int, agglomerate: bool) -> CloudVolume:
+    return CloudVolume(
+        store,
+        mip=mip,
+        use_https=True,
+        agglomerate=agglomerate,
+        timestamp=timestamp,
+        fill_missing=True,  # missing chunks are, by definition, empty
+        bounded=False,  # tiles at the ROI edge may poke out of the volume
+    )
+
+
+def _tile_job(args):
+    """Read one coarse tile and return which of its blocks are non-empty."""
+    vol, tile_index, block_lo, block_hi, roi_begin, roi_end, block_size, factor = args
+    block_lo, block_hi = np.array(block_lo), np.array(block_hi)
+    roi_begin, roi_end = np.array(roi_begin), np.array(roi_end)
+    block_size, factor = np.array(block_size), np.array(factor)
+
+    # Coarse voxel range covering all blocks in this tile (clipped to the ROI).
+    fine_lo = roi_begin + block_lo * block_size
+    fine_hi = np.minimum(roi_begin + block_hi * block_size, roi_end)
+    coarse_lo = fine_lo // factor
+    coarse_hi = -(-fine_hi // factor)  # ceil
+    data = np.asarray(
+        vol[tuple(slice(int(a), int(b)) for a, b in zip(coarse_lo, coarse_hi))]
+    ).squeeze(axis=-1)
+
+    out = np.zeros(block_hi - block_lo, dtype=bool)
+    for idx in product(*(range(n) for n in out.shape)):
+        b = block_lo + np.array(idx)
+        lo = (roi_begin + b * block_size) // factor - coarse_lo
+        hi = -(-(np.minimum(roi_begin + (b + 1) * block_size, roi_end)) // factor) - coarse_lo
+        out[idx] = data[tuple(slice(int(a), int(c)) for a, c in zip(lo, hi))].any()
+    return tile_index, out
+
+
+def _dilate(mask: np.ndarray, radius: int) -> np.ndarray:
+    """Binary dilation with a cube of the given radius (no scipy dependency)."""
+    if radius <= 0:
+        return mask
+    padded = np.pad(mask, radius)
+    out = np.zeros_like(mask)
+    for shift in product(range(2 * radius + 1), repeat=mask.ndim):
+        sl = tuple(slice(s, s + n) for s, n in zip(shift, mask.shape))
+        out |= padded[sl]
+    return out
+
+
+def compute_block_mask(
+    labels: CloudVolumeWrapper,
+    roi: Roi,
+    block_size: Coordinate,
+    mask_mip: int | None = None,
+    num_workers: int = 8,
+    dilate: int = 1,
+) -> np.ndarray:
+    """
+    Compute a boolean array, indexed by block index within ``roi``, that is
+    True for blocks containing at least one non-zero label.
+
+    Args:
+        labels: The label volume that will be sampled.
+        roi: ROI to be processed, in voxel coordinates of ``labels.mip``
+            (i.e. the frame daisy blocks are expressed in).
+        block_size: Sampling block size in the same frame.
+        mask_mip: Mip level to read for the occupancy check. Defaults to the
+            coarsest mip available.
+        num_workers: Number of threads reading coarse tiles. Reads are I/O and
+            decompression bound and release the GIL, so threads scale as well
+            as processes here while avoiding multiprocessing entirely (forked
+            and spawned pools both died with BrokenProcessPool on LSF nodes).
+        dilate: Grow the mask by this many blocks in every direction. Guards
+            against thin structures that vanish when downsampled.
+    """
+    base = CloudVolume(str(labels.store), mip=labels.mip, use_https=True)
+    if mask_mip is None:
+        mask_mip = max(base.available_mips)
+    factor = np.asarray(base.mip_resolution(mask_mip)) / np.asarray(
+        base.mip_resolution(labels.mip)
+    )
+    assert np.allclose(factor, np.round(factor)), (
+        f"mip {mask_mip} resolution is not an integer multiple of mip {labels.mip}"
+    )
+    factor = np.round(factor).astype(int)
+
+    roi_begin = np.array(roi.begin)
+    roi_end = np.array(roi.end)
+    block_size = np.array(block_size)
+    n_blocks = -(-(roi_end - roi_begin) // block_size)
+
+    # Blocks per tile per axis: enough that a tile is ~_TARGET_TILE_VOXELS wide.
+    blocks_per_tile = np.maximum(
+        1, _TARGET_TILE_VOXELS // np.maximum(1, block_size // factor)
+    )
+    n_tiles = -(-n_blocks // blocks_per_tile)
+
+    coarse = _open_coarse(
+        str(labels.store), mask_mip, labels.timestamp, labels.agglomerate
+    )
+    jobs = []
+    for t in product(*(range(n) for n in n_tiles)):
+        t = np.array(t)
+        lo = t * blocks_per_tile
+        hi = np.minimum(lo + blocks_per_tile, n_blocks)
+        jobs.append(
+            (coarse, tuple(t), tuple(lo), tuple(hi), tuple(roi_begin),
+             tuple(roi_end), tuple(block_size), tuple(factor))
+        )
+
+    mask = np.zeros(n_blocks, dtype=bool)
+    with ThreadPoolExecutor(max_workers=num_workers) as pool:
+        for tile_index, sub in pool.map(_tile_job, jobs):
+            lo = np.array(tile_index) * blocks_per_tile
+            mask[tuple(slice(a, a + n) for a, n in zip(lo, sub.shape))] = sub
+
+    return _dilate(mask, dilate)

nps_cli-0.1.5/nps/blockwise/sample_points.py

@@ -0,0 +1,248 @@
+import glob
+import logging
+import os
+import shutil
+import uuid
+from contextlib import contextmanager
+from typing import Literal
+
+import daisy
+import duckdb
+import numpy as np
+import pandas as pd
+import pyarrow as pa
+import pyarrow.parquet as pq
+from daisy import Block
+from funlib.geometry import Coordinate, Roi
+from volara.blockwise import BlockwiseTask
+from volara.datasets import CloudVolumeWrapper, Dataset
+from volara.utils import PydanticCoordinate
+
+_INDEX_SCHEMA = """
+    CREATE TABLE IF NOT EXISTS {name} (
+        label UBIGINT,
+        block_id VARCHAR,
+        file_path VARCHAR,
+        point_count UBIGINT
+    )
+"""
+
+
+class SamplePoints(BlockwiseTask):
+    task_type: Literal["sample_pc"] = "sample_pc"
+    out_dir: str
+    labels: CloudVolumeWrapper
+    svids: CloudVolumeWrapper | None = None
+    block_mask: str | None = None
+    block_size: PydanticCoordinate
+    fraction: float
+    fit: Literal["shrink"] = "shrink"
+    read_write_conflict: Literal[False] = False
+
+    @property
+    def task_name(self) -> str:
+        return f"{self.labels.name}-{self.task_type}"
+
+    @property
+    def write_roi(self) -> Roi:
+        total_roi = self.labels.array("r").roi
+        if self.roi is not None:
+            total_roi = total_roi.intersect(self.roi)
+        return total_roi
+
+    @property
+    def voxel_size(self) -> Coordinate:
+        return self.labels.voxel_size
+
+    @property
+    def write_size(self) -> Coordinate:
+        return self.block_size
+
+    @property
+    def context_size(self) -> Coordinate:
+        return Coordinate((0,) * self.write_size.dims)
+
+    def drop_artifacts(self):
+        # init() recreates the directory when the task is run.
+        shutil.rmtree(self.out_dir, ignore_errors=True)
+
+    def check_block_func(self):
+        """
+        Daisy calls this in the scheduler before dispatching a block; returning
+        True means "already done". If a block mask is given, report empty blocks
+        as done so they are never sent to a worker.
+        """
+        base = super().check_block_func()
+        if self.block_mask is None:
+            return base
+
+        mask = np.load(self.block_mask)
+        roi_begin = np.array(self.write_roi.begin)
+        block_size = np.array(self.block_size)
+
+        def check_block(block: Block) -> bool:
+            index = (np.array(block.write_roi.begin) - roi_begin) // block_size
+            if not mask[tuple(index)]:
+                return True
+            return base(block)
+
+        return check_block
+
+    @property
+    def output_datasets(self) -> list[Dataset]:
+        return []
+
+    def sample_pc_in_block(self, block: Block, labels: np.ndarray, svids: np.ndarray | None, offset: Coordinate, writer: "_PointWriter"):
+        block_id = block.block_id[1]
+
+        seg_ids, pts = self.sample_segment_points(labels, self.fraction)
+        logging.info(f"sampled {len(pts)} points in block {block_id}")
+
+        packed_pts = (pts + np.array(offset)).astype(np.uint64)
+        if self.svids is not None:
+            sampled_svids = svids[pts[:, 0], pts[:, 1], pts[:, 2]].reshape(-1, 1)
+            packed_pts = np.concatenate((packed_pts, sampled_svids.astype(np.uint64)), axis=1)
+
+        writer.write(block_id, seg_ids, packed_pts)
+
+    def init(self):
+        os.makedirs(self.out_dir, exist_ok=True)
+
+    @contextmanager
+    def process_block_func(self):
+        data = self.labels.array("r")
+        if self.svids is not None:
+            supervoxels = self.svids.array("r")
+
+        try:
+            worker_id = daisy.Context.from_env()["worker_id"]
+        except KeyError:
+            worker_id = 0
+        # One writer per worker process instead of a pocaduck Ingestor per
+        # block, which re-read and re-wrote the worker's whole parquet file
+        # and ran two DuckDB queries per segment on every block.
+        writer = _PointWriter(self.out_dir, worker_id)
+
+        def process_block(block: Block):
+            # Index with the Roi (world/absolute voxel coords), not with
+            # to_slices(): numpy-style keys on a funlib Array are zero-based and
+            # would read from the wrong place on volumes with a voxel offset.
+            # Only squeeze the channel axis; edge blocks can be 1 voxel thick.
+            labels = np.asarray(data[block.write_roi]).squeeze(axis=-1)
+
+            offset = block.write_roi.get_begin()
+
+            if self.svids is not None:
+                s = np.asarray(supervoxels[block.write_roi]).squeeze(axis=-1)
+                assert s.shape == labels.shape, (
+                    f"label block {labels.shape} and svid block {s.shape} differ in "
+                    f"{block.write_roi}; the label and supervoxel volumes must share "
+                    "resolution and offset at the chosen mip"
+                )
+            else:
+                s = None
+
+            self.sample_pc_in_block(block, labels, s, offset, writer)
+
+        # Index consolidation happens once in the driver after all blocks are
+        # done (see consolidate_indexes), not concurrently in every worker.
+        yield process_block
+
+    def sample_segment_points(self, labels, fraction, background=0):
+        """
+        Sample max(1, int(n * fraction)) voxels without replacement from every
+        segment with n voxels. Returns the segment ID of every sampled point
+        and the (N, 3) voxel coordinates within ``labels``.
+
+        Vectorized: one sort over the foreground voxels instead of one full
+        ``labels == seg`` scan per segment.
+        """
+        flat = labels.ravel()
+        idx = np.flatnonzero(flat != background) if background is not None else np.arange(flat.size)
+        vals = flat[idx]
+
+        # Shuffle, then stable-sort by segment: groups segments together with a
+        # random order inside each group, so the first k of a group are a
+        # uniform sample without replacement.
+        perm = np.random.permutation(len(idx))
+        perm = perm[np.argsort(vals[perm], kind="stable")]
+        idx, vals = idx[perm], vals[perm]
+
+        _, starts, counts = np.unique(vals, return_index=True, return_counts=True)
+        k = np.maximum(1, (counts * fraction).astype(np.int64))
+        rank = np.arange(len(vals)) - np.repeat(starts, counts)
+        keep = rank < np.repeat(k, counts)
+
+        coords = np.column_stack(np.unravel_index(idx[keep], labels.shape))
+        return vals[keep], coords
+
+
+class _PointWriter:
+    """
+    Writes sampled points of one worker in the layout of pocaduck's Ingestor
+    (``worker_<id>/data/*.parquet`` plus ``worker_<id>/index_<id>.db``), so
+    :func:`consolidate_indexes` and ``pocaduck.Query`` work unchanged.
+
+    Every block is written (parquet first, then its index rows) before the
+    block returns: daisy marks a block done as soon as process_block returns
+    and does not wait for worker teardown, so buffering across blocks would
+    lose data.
+    """
+
+    def __init__(self, out_dir: str, worker_id):
+        worker_dir = os.path.join(out_dir, f"worker_{worker_id}")
+        self.data_dir = os.path.join(worker_dir, "data")
+        self.db_path = os.path.join(worker_dir, f"index_{worker_id}.db")
+        os.makedirs(self.data_dir, exist_ok=True)
+        # Unique prefix: a restarted worker can reuse a worker_id and must not
+        # overwrite files of a previous run.
+        self.prefix = f"{worker_id}-{uuid.uuid4().hex[:8]}"
+
+    def write(self, block_id, seg_ids: np.ndarray, points: np.ndarray):
+        if len(points) == 0:
+            return
+        file_path = os.path.join(self.data_dir, f"{self.prefix}-{block_id}.parquet")
+        data = pa.FixedSizeListArray.from_arrays(pa.array(points.ravel()), points.shape[1])
+        table = pa.table({
+            "label": seg_ids.astype(np.int64),
+            "block_id": np.full(len(points), block_id, dtype=np.int64),
+            # Same list<uint64> column the pocaduck Ingestor writes.
+            "data": data.cast(pa.list_(pa.uint64())),
+        })
+        # Write-then-rename so a killed worker never leaves a truncated file.
+        pq.write_table(table, file_path + ".tmp")
+        os.replace(file_path + ".tmp", file_path)
+
+        segs, counts = np.unique(seg_ids, return_counts=True)
+        index = pd.DataFrame({
+            "label": segs.astype(np.uint64),
+            "block_id": str(block_id),
+            "file_path": file_path,
+            "point_count": counts.astype(np.uint64),
+        })
+        # Short-lived connection: the driver attaches these files for
+        # consolidation while worker processes may still be alive.
+        con = duckdb.connect(self.db_path)
+        con.execute(_INDEX_SCHEMA.format(name="point_cloud_index"))
+        con.register("new_rows", index)
+        con.execute("INSERT INTO point_cloud_index SELECT * FROM new_rows")
+        con.close()
+
+
+def consolidate_indexes(out_dir: str) -> str:
+    """
+    Merge all worker indexes into ``unified_index.db`` (what pocaduck's Query
+    reads). Replaces ``Ingestor.consolidate_indexes``, which pulls every row
+    through Python and was run concurrently by every worker.
+    """
+    output_path = os.path.join(out_dir, "unified_index.db")
+    if os.path.exists(output_path):
+        os.remove(output_path)
+    con = duckdb.connect(output_path)
+    con.execute(_INDEX_SCHEMA.format(name="point_cloud_index"))
+    for path in sorted(glob.glob(os.path.join(out_dir, "worker_*", "index_*.db"))):
+        con.execute(f"ATTACH '{path}' AS w (READ_ONLY)")
+        con.execute("INSERT INTO point_cloud_index SELECT * FROM w.point_cloud_index")
+        con.execute("DETACH w")
+    con.close()
+    return output_path