tilewise-ccl 0.0.2__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- tilewise_ccl-0.0.2/LICENSE +21 -0
- tilewise_ccl-0.0.2/PKG-INFO +472 -0
- tilewise_ccl-0.0.2/README.md +431 -0
- tilewise_ccl-0.0.2/pyproject.toml +65 -0
- tilewise_ccl-0.0.2/setup.cfg +4 -0
- tilewise_ccl-0.0.2/src/tilewise_ccl/__init__.py +19 -0
- tilewise_ccl-0.0.2/src/tilewise_ccl/core.py +1083 -0
- tilewise_ccl-0.0.2/src/tilewise_ccl/legacy.py +271 -0
- tilewise_ccl-0.0.2/src/tilewise_ccl.egg-info/PKG-INFO +472 -0
- tilewise_ccl-0.0.2/src/tilewise_ccl.egg-info/SOURCES.txt +15 -0
- tilewise_ccl-0.0.2/src/tilewise_ccl.egg-info/dependency_links.txt +1 -0
- tilewise_ccl-0.0.2/src/tilewise_ccl.egg-info/requires.txt +17 -0
- tilewise_ccl-0.0.2/src/tilewise_ccl.egg-info/top_level.txt +1 -0
- tilewise_ccl-0.0.2/tests/test_backend_dyna.py +171 -0
- tilewise_ccl-0.0.2/tests/test_default_tile_shape.py +82 -0
- tilewise_ccl-0.0.2/tests/test_label_array.py +137 -0
- tilewise_ccl-0.0.2/tests/test_label_legacy.py +95 -0
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 Bugra Oezdemir
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
|
@@ -0,0 +1,472 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: tilewise-ccl
|
|
3
|
+
Version: 0.0.2
|
|
4
|
+
Summary: Fast tile-wise connected-components labeling of large N-D arrays, returning a lazy dask array
|
|
5
|
+
Author-email: Bugra Oezdemir <bugraa.ozdemir@gmail.com>
|
|
6
|
+
License: MIT
|
|
7
|
+
Project-URL: Homepage, https://github.com/bugraoezdemir/tilewise-ccl
|
|
8
|
+
Keywords: connected-components,labeling,segmentation,ndimage,out-of-core,memory-bounded,chunked-array,image-processing,bioimage,microscopy
|
|
9
|
+
Classifier: Development Status :: 3 - Alpha
|
|
10
|
+
Classifier: Intended Audience :: Science/Research
|
|
11
|
+
Classifier: Intended Audience :: Developers
|
|
12
|
+
Classifier: License :: OSI Approved :: MIT License
|
|
13
|
+
Classifier: Operating System :: OS Independent
|
|
14
|
+
Classifier: Programming Language :: Python :: 3
|
|
15
|
+
Classifier: Programming Language :: Python :: 3.11
|
|
16
|
+
Classifier: Programming Language :: Python :: 3.12
|
|
17
|
+
Classifier: Programming Language :: Python :: 3.13
|
|
18
|
+
Classifier: Topic :: Scientific/Engineering
|
|
19
|
+
Classifier: Topic :: Scientific/Engineering :: Bio-Informatics
|
|
20
|
+
Classifier: Topic :: Scientific/Engineering :: Image Processing
|
|
21
|
+
Classifier: Topic :: Scientific/Engineering :: Visualization
|
|
22
|
+
Classifier: Topic :: Software Development :: Libraries :: Python Modules
|
|
23
|
+
Classifier: Topic :: System :: Distributed Computing
|
|
24
|
+
Requires-Python: >=3.11
|
|
25
|
+
Description-Content-Type: text/markdown
|
|
26
|
+
License-File: LICENSE
|
|
27
|
+
Requires-Dist: numpy>=1.20.0
|
|
28
|
+
Requires-Dist: scipy>=1.6.0
|
|
29
|
+
Requires-Dist: dask
|
|
30
|
+
Provides-Extra: dyna
|
|
31
|
+
Requires-Dist: dyna-zarr>=0.0.5; extra == "dyna"
|
|
32
|
+
Provides-Extra: cc3d
|
|
33
|
+
Requires-Dist: connected-components-3d; extra == "cc3d"
|
|
34
|
+
Provides-Extra: all
|
|
35
|
+
Requires-Dist: dyna-zarr>=0.0.5; extra == "all"
|
|
36
|
+
Requires-Dist: connected-components-3d; extra == "all"
|
|
37
|
+
Provides-Extra: dev
|
|
38
|
+
Requires-Dist: pytest>=7.0; extra == "dev"
|
|
39
|
+
Requires-Dist: zarr>=3.0.0; extra == "dev"
|
|
40
|
+
Dynamic: license-file
|
|
41
|
+
|
|
42
|
+
# tilewise-ccl
|
|
43
|
+
|
|
44
|
+
Fast tile-wise **connected-components labeling** for large N-dimensional arrays,
|
|
45
|
+
returning a lazy array of labels.
|
|
46
|
+
|
|
47
|
+
`tilewise_ccl.label_array` takes any large N-D binary mask (NumPy, Zarr, or Dask
|
|
48
|
+
backed) and returns a **lazy dask array** of connected-component labels, computed
|
|
49
|
+
with a tile-local labeling pass plus a small global boundary-piece graph. It is:
|
|
50
|
+
|
|
51
|
+
- **Fast**: only the pieces that touch a tile border are reconciled, on a small graph, so the
|
|
52
|
+
cost stays low even as the number of tiles grows.
|
|
53
|
+
- **N-dimensional**: 2D, 3D, or higher; connectivity is configurable.
|
|
54
|
+
- **Storage- and domain-agnostic**: no OME-Zarr / image-format dependency; just
|
|
55
|
+
NumPy + SciPy + Dask.
|
|
56
|
+
- **Dask-optional**: the `dyna` backend (see [Backends](#backends)) runs the whole
|
|
57
|
+
thing through [dyna-zarr](https://github.com/bugraoezdemir/dyna_zarr)'s pull model
|
|
58
|
+
instead - no task graph, and it fuses a preceding threshold into each tile read.
|
|
59
|
+
- **Memory-bounded**: peak memory scales with `(tile size) × (concurrency)`, not
|
|
60
|
+
with the array size, so it labels arrays far larger than RAM.
|
|
61
|
+
- **Composable**: the result is a genuine lazy dask array, so it plugs straight
|
|
62
|
+
into other dask operations, such as `da.unique`, slicing, arithmetic, `.to_zarr(...)`, region-property
|
|
63
|
+
computation, etc.
|
|
64
|
+
|
|
65
|
+
Labels are **dense integers `1..N`** (background `0`), `int32` when `N < 2**31`
|
|
66
|
+
(essentially always) else `int64`.
|
|
67
|
+
|
|
68
|
+
---
|
|
69
|
+
|
|
70
|
+
## Installation
|
|
71
|
+
|
|
72
|
+
```bash
|
|
73
|
+
pip install tilewise-ccl
|
|
74
|
+
# or, from a checkout:
|
|
75
|
+
pip install -e .
|
|
76
|
+
```
|
|
77
|
+
|
|
78
|
+
Requires Python ≥ 3.11 and `numpy`, `scipy`, `dask`.
|
|
79
|
+
|
|
80
|
+
Optional extras, each needed only by the feature that names it:
|
|
81
|
+
|
|
82
|
+
```bash
|
|
83
|
+
pip install "tilewise-ccl[dyna]" # backend="dyna": pull-model tile reads, no dask graph
|
|
84
|
+
pip install "tilewise-ccl[cc3d]" # labeler="cc3d": a faster per-tile CCL kernel (3-D)
|
|
85
|
+
pip install "tilewise-ccl[all]" # both
|
|
86
|
+
```
|
|
87
|
+
|
|
88
|
+
Each is imported lazily at the point of use, so a plain install stays light and
|
|
89
|
+
only raises - naming the extra - if you reach for that feature without it.
|
|
90
|
+
|
|
91
|
+
---
|
|
92
|
+
|
|
93
|
+
## Quick start
|
|
94
|
+
|
|
95
|
+
`label_array` expects a **binary mask** (anything truthy is foreground). Threshold
|
|
96
|
+
your data first, then label:
|
|
97
|
+
|
|
98
|
+
```python
|
|
99
|
+
import numpy as np
|
|
100
|
+
from tilewise_ccl import label_array
|
|
101
|
+
|
|
102
|
+
# any N-D array; truthy = foreground
|
|
103
|
+
image = np.random.default_rng(0).integers(0, 100, size=(512, 512), dtype=np.uint8)
|
|
104
|
+
mask = image > 80
|
|
105
|
+
|
|
106
|
+
labels = label_array(mask, tile_shape=(256, 256), connectivity=2)
|
|
107
|
+
|
|
108
|
+
labels # lazy dask.array.Array, dtype int32, shape (512, 512)
|
|
109
|
+
result = labels.compute() # materialize to a NumPy array of labels 1..N
|
|
110
|
+
|
|
111
|
+
# opt in to diagnostics (object/graph counts, timings) -> returns a tuple
|
|
112
|
+
labels, diag = label_array(mask, tile_shape=(256, 256), connectivity=2, diagnostics=True)
|
|
113
|
+
print(diag["n_final_objects"])
|
|
114
|
+
```
|
|
115
|
+
|
|
116
|
+
### On a Dask array (e.g. a large Zarr / OME-Zarr level)
|
|
117
|
+
|
|
118
|
+
The input can be lazy. `label_array` reads it tile by tile, so nothing needs to
|
|
119
|
+
fit in memory at once. Note the labeling is *not* forced until you compute or
|
|
120
|
+
write the result.
|
|
121
|
+
|
|
122
|
+
```python
|
|
123
|
+
import dask.array as da
|
|
124
|
+
from tilewise_ccl import label_array
|
|
125
|
+
|
|
126
|
+
# read one array directly from a zarr store (no OME-Zarr machinery needed)
|
|
127
|
+
arr = da.from_zarr("dataset.zarr/0") # e.g. a huge 3D volume, uint8/uint16
|
|
128
|
+
mask = arr > 128 # lazy boolean mask
|
|
129
|
+
|
|
130
|
+
labels = label_array(mask, tile_shape=(384, 384, 384), connectivity=2, n_workers=8)
|
|
131
|
+
|
|
132
|
+
# the result is lazy. Stream it straight to disk (each chunk written once)
|
|
133
|
+
labels.to_zarr("labels.zarr", overwrite=True)
|
|
134
|
+
```
|
|
135
|
+
|
|
136
|
+
### The same, without dask (the `dyna` backend)
|
|
137
|
+
|
|
138
|
+
The identical job through [dyna-zarr](https://github.com/bugraoezdemir/dyna_zarr)'s
|
|
139
|
+
pull model - no task graph anywhere. Install the `[dyna]` extra. Labels are
|
|
140
|
+
byte-identical to the dask path; only the execution differs. See
|
|
141
|
+
[Backends](#backends) for what changes and why.
|
|
142
|
+
|
|
143
|
+
```python
|
|
144
|
+
from dyna_zarr.io import io as dio
|
|
145
|
+
from tilewise_ccl import label_array
|
|
146
|
+
|
|
147
|
+
# read one array directly from a zarr store, exactly as above
|
|
148
|
+
arr = dio.read("dataset.zarr/0") # a DynamicArray, not a dask array
|
|
149
|
+
mask = arr > 128 # lazy - and FUSED into each tile read
|
|
150
|
+
|
|
151
|
+
# backend is inferred from the mask type - a DynamicArray selects the dyna path
|
|
152
|
+
labels = label_array(mask, tile_shape=(384, 384, 384), connectivity=2, n_workers=8)
|
|
153
|
+
|
|
154
|
+
# the result is a lazy DynamicArray - write it with dyna-zarr's writer, which
|
|
155
|
+
# picks up the tile grid automatically (see the note below).
|
|
156
|
+
dio.write(labels, "labels.zarr", chunks=(96, 96, 96),
|
|
157
|
+
max_workers=8, overwrite=True)
|
|
158
|
+
```
|
|
159
|
+
|
|
160
|
+
Two differences worth noting against the dask version above:
|
|
161
|
+
|
|
162
|
+
- **The threshold costs nothing extra.** `arr > 128` is never materialized - each
|
|
163
|
+
tile applies the comparison as it is pulled, so there is no intermediate mask
|
|
164
|
+
array on disk and no per-tile scheduler overhead.
|
|
165
|
+
- **The writer follows the tile grid on its own.** `label_array` records its
|
|
166
|
+
`tile_shape` on the returned array, and `dio.write` reads it, so the write
|
|
167
|
+
regions line up with the tiles and each tile is read and labeled exactly once.
|
|
168
|
+
You do not have to restate the tiling. (`chunks` is separate and can stay small
|
|
169
|
+
for fast downstream reads.)
|
|
170
|
+
|
|
171
|
+
### Downstream composability
|
|
172
|
+
|
|
173
|
+
The labeling is lazy, and so is everything you build on it. A useful consequence:
|
|
174
|
+
`properties=True` gives you every object's size and bounding box as **metadata**,
|
|
175
|
+
without a pass over the data - so you can find an object of interest and then
|
|
176
|
+
read back only the box it occupies.
|
|
177
|
+
|
|
178
|
+
```python
|
|
179
|
+
import numpy as np
|
|
180
|
+
|
|
181
|
+
labels, diag = label_array(mask, tile_shape=(128, 128, 128), connectivity=2,
|
|
182
|
+
n_workers=8, properties=True)
|
|
183
|
+
|
|
184
|
+
# area / bbox come from the per-tile pass - no extra read of the array
|
|
185
|
+
i = int(np.argmax(diag["area"])) # the largest object
|
|
186
|
+
lbl = int(diag["label_values"][i])
|
|
187
|
+
lo, hi = diag["bbox_start"][i], diag["bbox_stop"][i]
|
|
188
|
+
|
|
189
|
+
# read back ONLY that object's bounding box (.compute() works on both backends)
|
|
190
|
+
crop = labels[tuple(slice(int(a), int(b)) for a, b in zip(lo, hi))].compute()
|
|
191
|
+
obj = crop == lbl # the object itself, isolated
|
|
192
|
+
```
|
|
193
|
+
|
|
194
|
+
Only the box is ever materialized. The rest of the array is never touched.
|
|
195
|
+
|
|
196
|
+
The same laziness applies to ordinary array work:
|
|
197
|
+
|
|
198
|
+
```python
|
|
199
|
+
import dask.array as da
|
|
200
|
+
|
|
201
|
+
# count objects an independent way (excludes background 0)
|
|
202
|
+
n_objects = int(np.count_nonzero(da.unique(labels).compute()))
|
|
203
|
+
|
|
204
|
+
# match the storage chunking before writing (cheap when tile_shape is a
|
|
205
|
+
# multiple of the target chunk, e.g. 384 = 4 * 96)
|
|
206
|
+
labels.rechunk((96, 96, 96)).to_zarr("labels.zarr", overwrite=True)
|
|
207
|
+
```
|
|
208
|
+
|
|
209
|
+
---
|
|
210
|
+
|
|
211
|
+
## API
|
|
212
|
+
|
|
213
|
+
### `label_array(mask, tile_shape=None, connectivity=2, n_workers=1, diagnostics=False, properties=False, verbose=False, backend="auto", executor="thread", labeler="scipy")`
|
|
214
|
+
|
|
215
|
+
Label connected components of a binary N-D mask into a lazy dask array.
|
|
216
|
+
|
|
217
|
+
**Returns:**
|
|
218
|
+
|
|
219
|
+
- **By default** (`diagnostics=False`): just `output_labels`, a lazy array of
|
|
220
|
+
dense labels `1..N` (background `0`), same shape as `mask`. dtype is `int32`
|
|
221
|
+
when `N < 2**31`, else `int64`. It is a `dask.array.Array` with the default
|
|
222
|
+
`backend="dask"`, and a `dyna_zarr` `DynamicArray` with `backend="dyna"`.
|
|
223
|
+
- **With `diagnostics=True`**: a tuple `(output_labels, diag)`, where `diag` is a
|
|
224
|
+
`dict` of object/graph counts and timings (see below).
|
|
225
|
+
|
|
226
|
+
```python
|
|
227
|
+
labels = label_array(mask) # -> dask.array.Array
|
|
228
|
+
labels, diag = label_array(mask, diagnostics=True) # -> (dask.array.Array, dict)
|
|
229
|
+
```
|
|
230
|
+
|
|
231
|
+
#### Parameters
|
|
232
|
+
|
|
233
|
+
| Parameter | Type | Default | Description |
|
|
234
|
+
|---|---|---|---|
|
|
235
|
+
| `mask` | `np.ndarray` \| `dask.array.Array` \| `zarr.Array` | *required* | N-D array; truthy elements are foreground. Read tile by tile, so it may be far larger than RAM. |
|
|
236
|
+
| `tile_shape` | sequence of int | `None` | Size of each processing tile, **one entry per dimension** (its length must equal `mask.ndim`). `None` derives one from the mask: a ~256 MiB working set over the trailing 3 (spatial) axes, snapped to whole storage chunks when the mask exposes `.chunks`, leading (t/c-like) axes left at 1. This is an *execution hyperparameter* and does not change the result. See [Choosing `tile_shape`](#choosing-tile_shape). |
|
|
237
|
+
| `connectivity` | int | `2` | Passed to `scipy.ndimage.generate_binary_structure(ndim, connectivity)`. Higher = more diagonal neighbors are considered connected (see [Connectivity](#connectivity)). |
|
|
238
|
+
| `n_workers` | int | `1` | If `> 1`, the eager metadata pass (Phase A) runs over tiles via a thread pool. Each tile touches only its own region, so this is safe. Phase B (the lazy graph) is scheduled by Dask when you compute/write the result. |
|
|
239
|
+
| `diagnostics` | bool | `False` | If `True`, also return a `diag` dict (the function returns a `(labels, diag)` tuple instead of just `labels`). |
|
|
240
|
+
| `properties` | bool | `False` | Also compute per-object `area` (voxel count) and `bbox_start`/`bbox_stop` (half-open global bounding box) into `diag` - implies returning the tuple. Aggregated from per-tile partials, so memory stays bounded by tile size no matter how large an object is. |
|
|
241
|
+
| `backend` | `"auto"` \| `"dask"` \| `"dyna"` | `"auto"` | Execution backend. `"auto"` infers it from the mask type - a `DynamicArray` gets the dyna path, anything else gets dask - so you rarely pass this. See [Backends](#backends). |
|
|
242
|
+
| `executor` | `"thread"` \| `"process"` | `"thread"` | Pool type for Phase A. Which one wins depends entirely on `backend` - see [Backends](#backends). |
|
|
243
|
+
| `labeler` | `"scipy"` \| `"cc3d"` | `"scipy"` | Per-tile CCL kernel. `"cc3d"` (needs the `[cc3d]` extra, 3-D only) is ~2x faster per call but **holds the GIL**, so it only pays off with `executor="process"`. Both emit identical labels. |
|
|
244
|
+
| `verbose` | bool | `False` | Print per-tile progress for Phase A (immediately) and Phase B (when the returned array is computed/written). |
|
|
245
|
+
|
|
246
|
+
> **Note.** `tile_shape` must have the same length as `mask.ndim`; a mismatch
|
|
247
|
+
> raises `ValueError`.
|
|
248
|
+
|
|
249
|
+
#### `diag` fields (returned only when `diagnostics=True`)
|
|
250
|
+
|
|
251
|
+
| Key | Meaning |
|
|
252
|
+
|---|---|
|
|
253
|
+
| `n_tiles`, `grid_shape` | Number of tiles and the tile-grid shape. |
|
|
254
|
+
| `n_final_objects` | Total number of connected components (== `N`). |
|
|
255
|
+
| `n_interior_objects` | Objects fully contained in a single tile (resolved locally, never entered the graph). |
|
|
256
|
+
| `n_boundary_pieces`, `n_edges`, `n_boundary_groups` | Size of the boundary-reconciliation graph: pieces touching a tile border, edges between them, and the resulting merged groups. |
|
|
257
|
+
| `n_objects_single_tile` | Objects occupying exactly one tile (do **not** cross a tile border). |
|
|
258
|
+
| `n_objects_crossing` | Objects that cross into ≥2 tiles. |
|
|
259
|
+
| `n_objects_crossing_gt3_tiles` | Objects spanning >3 tiles (large objects). |
|
|
260
|
+
| `max_tiles_spanned` | Maximum number of tiles any single object spans. |
|
|
261
|
+
| `output_dtype` | dtype of `output_labels` (`"int32"` or `"int64"`). |
|
|
262
|
+
| `time_phaseA_s`, `time_reconcile_s`, `time_total_s` | Eager-phase timings (seconds). Phase B's cost is incurred later, on compute/write. |
|
|
263
|
+
|
|
264
|
+
---
|
|
265
|
+
|
|
266
|
+
## Connectivity
|
|
267
|
+
|
|
268
|
+
`connectivity` selects the neighborhood via
|
|
269
|
+
`scipy.ndimage.generate_binary_structure(ndim, connectivity)`. An offset counts
|
|
270
|
+
as a neighbor iff its number of nonzero components is `≤ connectivity`:
|
|
271
|
+
|
|
272
|
+
| ndim | `connectivity=1` | `connectivity=2` | `connectivity=3` |
|
|
273
|
+
|---|---|---|---|
|
|
274
|
+
| 2D | 4-connected (faces) | 8-connected (+ corners) | n/a |
|
|
275
|
+
| 3D | 6-connected (faces) | 18-connected (+ edges) | 26-connected (+ corners) |
|
|
276
|
+
|
|
277
|
+
Cross-tile connections (including diagonal corner/edge crossings between tiles)
|
|
278
|
+
are handled correctly for every connectivity, so the label result is identical to
|
|
279
|
+
a single whole-array `scipy.ndimage.label`, independent of `tile_shape`.
|
|
280
|
+
|
|
281
|
+
---
|
|
282
|
+
|
|
283
|
+
## Choosing `tile_shape`
|
|
284
|
+
|
|
285
|
+
`tile_shape` is a pure performance/parallelism knob and does **not** affect the
|
|
286
|
+
result. Guidance:
|
|
287
|
+
|
|
288
|
+
- **Bigger tiles** → more objects fit entirely inside one tile → fewer boundary
|
|
289
|
+
pieces → smaller reconciliation graph and less per-tile overhead. The cost is
|
|
290
|
+
more memory per tile (one `scipy.ndimage.label` call on a haloed tile) and
|
|
291
|
+
coarser parallelism.
|
|
292
|
+
- **The default is usually fine.** With `tile_shape=None`, `label_array` targets a
|
|
293
|
+
~256 MiB int32 working set over the trailing three axes and snaps it to whole
|
|
294
|
+
storage chunks, so the tile is always a chunk multiple and no chunk is read
|
|
295
|
+
twice. On a 3-D array chunked at `96³` that lands on `384³`; on a 5-D
|
|
296
|
+
`(t, c, z, y, x)` it tiles `z/y/x` only and leaves `t`/`c` at 1.
|
|
297
|
+
- Override it when you have memory to spare. Peak RSS is roughly **2x the tile's
|
|
298
|
+
int32 size, per worker** (measured), so a `512³` tile costs ~1 GiB per worker -
|
|
299
|
+
8 GiB at `n_workers=8`. Bigger tiles mean fewer boundary pieces and a smaller
|
|
300
|
+
reconciliation graph, at that cost.
|
|
301
|
+
- It is **decoupled from storage chunking**: `label_array` reads whatever tile
|
|
302
|
+
size you ask for directly, regardless of how the data is chunked on disk, so
|
|
303
|
+
there is no need to `rechunk` the input first. This keeps it efficient even on
|
|
304
|
+
natively small-chunked data.
|
|
305
|
+
- If you write the result with `.to_zarr`, keep the output chunk shape a divisor
|
|
306
|
+
of `tile_shape` (e.g. tile `384`, chunks `96`) so the pre-write `rechunk` is a
|
|
307
|
+
cheap sub-slice rather than a cross-block shuffle.
|
|
308
|
+
|
|
309
|
+
Peak memory is roughly `n_workers × (tile voxels) × (a few bytes)`. Set
|
|
310
|
+
`tile_shape` and `n_workers` to fit your memory budget.
|
|
311
|
+
|
|
312
|
+
---
|
|
313
|
+
|
|
314
|
+
## How it works: interior–boundary reconciliation
|
|
315
|
+
|
|
316
|
+
`tilewise-ccl` is a **tile-wise** connected-components labeler: label each tile on its
|
|
317
|
+
own, then merge the components that meet across tile borders. It adapts that
|
|
318
|
+
classic idea with two moves that keep it fast and memory-bounded:
|
|
319
|
+
|
|
320
|
+
**1. Interior–boundary separation** shrinks what must be reconciled. A
|
|
321
|
+
component touching none of its tile's borders is already a finished object and is
|
|
322
|
+
set aside at once. Only components touching a border (the ones that might continue
|
|
323
|
+
into a neighbour) are reconciled. So the global step is a small graph over
|
|
324
|
+
**border pieces**, whose size tracks the tiles' seam area, not the object count,
|
|
325
|
+
and stays flat as the number of tiles grows.
|
|
326
|
+
|
|
327
|
+
**2. Plan first, materialize once** defers ever touching the data. The first
|
|
328
|
+
pass reads each tile but keeps only compact **metadata** (which pieces merge, plus
|
|
329
|
+
a per-tile lookup table) and allocates **no output array**. The labels are a lazy
|
|
330
|
+
dask graph that materializes exactly once, on `.compute()` / `.to_zarr(...)`,
|
|
331
|
+
relabelling each block in a single pass. Peak memory scales with tile size ×
|
|
332
|
+
concurrency (a handful of tiles), never the whole array, and **no voxel is written
|
|
333
|
+
twice**.
|
|
334
|
+
|
|
335
|
+
In short: most objects sit inside a single tile and are finished there (cheap);
|
|
336
|
+
only the few that straddle a **seam** between tiles ever need the reconciliation
|
|
337
|
+
graph.
|
|
338
|
+
|
|
339
|
+
### The two phases
|
|
340
|
+
|
|
341
|
+
```mermaid
|
|
342
|
+
flowchart TB
|
|
343
|
+
T["Tile the N-D mask"] --> L
|
|
344
|
+
|
|
345
|
+
subgraph PA["① Phase A · per tile, in parallel · metadata only, no array built"]
|
|
346
|
+
L["label a 1-voxel-haloed tile<br/>(scipy.ndimage.label)"] --> Q{"touches a<br/>tile border?"}
|
|
347
|
+
Q -->|no| I["<b>interior</b> piece<br/>→ a finished object"]
|
|
348
|
+
Q -->|yes| Bd["<b>boundary</b> piece<br/>→ cache its border slabs"]
|
|
349
|
+
end
|
|
350
|
+
|
|
351
|
+
subgraph RC["② Reconcile · metadata only (the whole plan)"]
|
|
352
|
+
Bd --> M["match facing border slabs<br/>of adjacent tiles → edges"]
|
|
353
|
+
M --> U["union-find over<br/>boundary pieces"]
|
|
354
|
+
end
|
|
355
|
+
|
|
356
|
+
I --> LUT["per-tile lookup table:<br/>local id → dense final id 1..N"]
|
|
357
|
+
U --> LUT
|
|
358
|
+
LUT --> PB["③ Phase B · lazy dask map_overlap<br/>relabel each block once + apply its LUT<br/>(runs only on .compute() / .to_zarr())"]
|
|
359
|
+
```
|
|
360
|
+
|
|
361
|
+
1. **Phase A** (eager, one tile at a time, in parallel; metadata only). Read a
|
|
362
|
+
1-voxel-haloed tile and label it with `scipy.ndimage.label`. Split its
|
|
363
|
+
components into **interior** (touch no border → finished) and **boundary**
|
|
364
|
+
(touch a border → deferred, their border slabs cached). Only small per-tile
|
|
365
|
+
metadata is kept (never voxel data), so memory is bounded by
|
|
366
|
+
`tile size × n_workers` and the pass runs safely across a thread pool. **No
|
|
367
|
+
output array is allocated.**
|
|
368
|
+
|
|
369
|
+
2. **Reconcile** (metadata only). Compare the facing border slabs of adjacent
|
|
370
|
+
tiles (faces, plus edge/corner diagonals for higher connectivity); where
|
|
371
|
+
foreground meets foreground, link the two boundary pieces. A union-find over
|
|
372
|
+
those pieces merges them into whole objects, and each tile gets a lookup table
|
|
373
|
+
from its local ids to dense final ids `1..N`. This little plan is the *entire*
|
|
374
|
+
global result.
|
|
375
|
+
|
|
376
|
+
3. **Phase B** (lazy; materialize once). A `dask.array.map_overlap` relabels each
|
|
377
|
+
block and applies that tile's lookup table, emitting final labels directly, **with no
|
|
378
|
+
read-modify-write of an output buffer**. Nothing runs until you `.compute()` or
|
|
379
|
+
`.to_zarr(...)`, and each chunk is produced exactly once.
|
|
380
|
+
|
|
381
|
+
Because most objects are interior, the reconciliation graph is pure metadata over
|
|
382
|
+
border pieces, and the array is only ever built lazily, cost stays low even for
|
|
383
|
+
pathological inputs: an object spanning hundreds of tiles still reconciles in well
|
|
384
|
+
under a second.
|
|
385
|
+
|
|
386
|
+
---
|
|
387
|
+
|
|
388
|
+
## Backends
|
|
389
|
+
|
|
390
|
+
`backend` selects how tiles are READ and how Phase B is expressed. Both produce
|
|
391
|
+
byte-identical labels - it is purely an execution choice.
|
|
392
|
+
|
|
393
|
+
### `backend="auto"` (default)
|
|
394
|
+
|
|
395
|
+
Picks `"dyna"` when `mask` is a `dyna_zarr` `DynamicArray`, `"dask"` otherwise. The
|
|
396
|
+
mask type already determines which path is viable - the dyna path accepts nothing
|
|
397
|
+
else - so the backend is normally not worth stating. Pass it explicitly only to
|
|
398
|
+
force the dask path for a `DynamicArray` mask (which works: each tile is
|
|
399
|
+
materialized as it is pulled).
|
|
400
|
+
|
|
401
|
+
### `backend="dask"`
|
|
402
|
+
|
|
403
|
+
`mask` may be a NumPy, Zarr, or Dask array. Phase B is a `da.map_blocks` over the
|
|
404
|
+
mask, so the result is a genuine `dask.array.Array` that composes with the rest of
|
|
405
|
+
dask (see [Downstream composability](#downstream-composability)).
|
|
406
|
+
|
|
407
|
+
### `backend="dyna"`
|
|
408
|
+
|
|
409
|
+
Selected automatically when `mask` is a [dyna-zarr](https://github.com/bugraoezdemir/dyna_zarr)
|
|
410
|
+
`DynamicArray` (install the `[dyna]` extra). Phase A pulls each tile directly
|
|
411
|
+
through dyna's pull model, and Phase B is a native dyna `map_overlap` - **no dask
|
|
412
|
+
graph anywhere**. Two things make this fast:
|
|
413
|
+
|
|
414
|
+
- A lazy threshold is **fused into the tile read**. `io.read(path) > 128` is not
|
|
415
|
+
materialized; each tile applies the comparison as it is pulled, so there is no
|
|
416
|
+
separate mask array on disk and no scheduler overhead per tile.
|
|
417
|
+
- Reads are memory-bounded by construction: a tile pull reads exactly the region
|
|
418
|
+
asked for.
|
|
419
|
+
|
|
420
|
+
The returned array is a lazy `DynamicArray`, so write it with dyna-zarr's writer
|
|
421
|
+
rather than `.to_zarr()` - see the [worked example above](#the-same-without-dask-the-dyna-backend).
|
|
422
|
+
|
|
423
|
+
Any `DynamicArray` works as the mask - however you obtained it, and with any
|
|
424
|
+
chain of lazy dyna operations already applied.
|
|
425
|
+
|
|
426
|
+
### Picking `executor`
|
|
427
|
+
|
|
428
|
+
`executor` only affects Phase A, and the right answer depends on the backend,
|
|
429
|
+
because the two differ in how much GIL-bound Python work a tile read costs.
|
|
430
|
+
|
|
431
|
+
Measured end to end on a 1024³ uint8 volume (1 GiB), 256³ tiles, 64³ output
|
|
432
|
+
chunks, 8 workers, best of 2. **Both backends write the same zarr v3 array with
|
|
433
|
+
the same blosc/zstd-5/bitshuffle codec** and produce byte-identical output
|
|
434
|
+
(82.6 MiB on disk in every row). Phase B is the lazy half, timed by writing:
|
|
435
|
+
|
|
436
|
+
| backend / executor | Phase A | Phase B (write) | total |
|
|
437
|
+
|---|---|---|---|
|
|
438
|
+
| `dask` / `thread` | 7.0 s | 12.8 s | 19.7 s |
|
|
439
|
+
| `dask` / `process` | **5.1 s** | 12.7 s | 17.8 s |
|
|
440
|
+
| `dyna` / `thread` | **2.8 s** | **8.2 s** | **11.0 s** |
|
|
441
|
+
| `dyna` / `process` | 4.4 s | 8.1 s | 12.5 s |
|
|
442
|
+
|
|
443
|
+
Reading it:
|
|
444
|
+
|
|
445
|
+
- **Phase B dominates** - roughly 65-75% of the run. It is mostly I/O and
|
|
446
|
+
compression, and `executor` does not touch it.
|
|
447
|
+
- Within Phase A the rule holds: `process` helps `dask` (1.4x, its read goes
|
|
448
|
+
through dask's graph and zarr's Python-level chunk assembly, which is
|
|
449
|
+
GIL-bound), while `thread` wins for `dyna` (1.6x, it pulls tiles directly, so
|
|
450
|
+
little GIL contention remains and per-tile pickling costs more than it saves).
|
|
451
|
+
- **The backend matters more than the executor.** `dyna`/`thread` is **1.8x** the
|
|
452
|
+
`dask`/`thread` baseline overall, while switching executor moves the total by
|
|
453
|
+
~10%. Choose the backend first.
|
|
454
|
+
|
|
455
|
+
One machine's numbers; the direction generalizes, the ratios may not.
|
|
456
|
+
|
|
457
|
+
The mask is shipped to each worker **once** via the pool initializer, and lazy
|
|
458
|
+
masks (zarr / dask / dyna) pickle as small references, so no array data is copied.
|
|
459
|
+
Process pools fall back to threads automatically for an in-memory NumPy mask
|
|
460
|
+
(which *would* be copied per worker) and for grids too small to amortize start-up.
|
|
461
|
+
|
|
462
|
+
`labeler="cc3d"` follows the same logic: it is faster per call but holds the GIL,
|
|
463
|
+
so pair it with `executor="process"` or not at all.
|
|
464
|
+
|
|
465
|
+
---
|
|
466
|
+
|
|
467
|
+
## Also available
|
|
468
|
+
|
|
469
|
+
- `tilewise_ccl.label_array_legacy(mask, output_labels, ...)`: an earlier *eager*
|
|
470
|
+
implementation that writes into a pre-allocated output buffer (NumPy/Zarr)
|
|
471
|
+
instead of returning a lazy array. Kept for comparison/benchmarking; prefer
|
|
472
|
+
`label_array` for new code.
|