worldarray 0.1.1__tar.gz → 0.3.0__tar.gz

worldarray-0.3.0/CHANGELOG.md

@@ -0,0 +1,103 @@
+# Changelog
+
+## 0.3.0 (2026-10-22)
+
+Grids and blocks, built to spec draft 3 (Behaviours 17 to 19). Nothing on the read or
+write side changes.
+
+- `blocks(shape, block_shape, context=0, box=None, clip=True)`: the array (or a voxel
+  box, which may reach past the array) cut into write boxes that tile it with no overlap
+  and no gap, each with a read box grown by `context` (one int, one per axis, or
+  `(before, after)` pairs) and clipped to the array when `clip`; C order, `len`,
+  `blocks[id]` in O(1), `index_of(begin)`; a `Block` carries `index`, `id`, `begin`,
+  `end`, `read_begin`, `read_end`, `at_start` and `at_end` per axis and `inner`, the write
+  box's slices inside the read box. Plain picklable records; nothing is scheduled.
+  `vol.blocks()` defaults the block to `write_shape`, so default blocks are aligned writes.
+  `vol.write_chunk(index, data)` writes one whole unit.
+- `Grid(shape, voxel_size, translation, axes, units, chunks, shards)`: a `WorldArray`
+  without a store, with `box`, `corner`, `index_box`, `chunks_covering`, `write_shape`,
+  `blocks` and `meta(dtype)`; `Grid.from_meta(meta, voxel_size=)` is an existing array's
+  grid or the resample target grid over it. `create(path, grid=g, dtype=)` creates from it.
+- `level_factors(voxel_size, ratio=1.5, ...)`: the anisotropy-aware pyramid schedule
+  mesh-n-bone and cellmap-analyze share, per level.
+- `units.abbreviation`, and `store.join`, `store.parent`, `store.basename` as public
+  URL-aware path helpers.
+
+## 0.2.0 (2026-10-22)
+
+Metadata and placed creation, built to spec draft 3 (Behaviours 10 to 16). Nothing on the
+read side changes.
+
+- `ome_multiscales(axes, units, levels, version)` builds the complete OME-NGFF 0.4 or 0.5
+  group document with one explicit placement per level; `level_placement` and
+  `level_shape` derive a level from factors and an origin (Convention 1's rule inverted,
+    so a level built from a ROI origin is placed from it; `level_shape` absorbs float-noise
+  factors such as `0.3 / 0.1`). Units are written in OME's names (`"nm"` becomes
+  `"nanometer"`; `units.ome_name`), an unknown unit as given; the multiscale's `type` is
+  written when given. Expected documents are pinned in `tests/fixtures/write.json`, shared
+  with chunkmirage's TypeScript builder, each case with its resolved per-axis units and
+  types, fifteen cases.
+- `create(path, shape, dtype, voxel_size, translation, chunks=..., ...)` writes the group
+  document (made when absent, else its one dataset entry added or replaced, every other
+  attribute kept) and then the array, as zarr v2 with OME 0.4 or zarr v3 with OME 0.5
+  following the group's format. `chunks` is required, codecs are explicit with a zstd
+  level 3 default, shards are a zarr v3 option. `exists` is `"error"` or `"adopt"` (an
+  identical array is opened for writing, a different one raises naming every difference,
+  a creation lost to another process is adopted after a short backoff). Nothing is ever
+    deleted, truncated or overwritten. Write units equal to the fill value are not stored
+  unless `store_fill=True`, and a stored unit written so loses its key, as zarr-python's
+  `write_empty_chunks=False` does. `create` also writes a bare group document for every
+  missing ancestor down from the `.zarr` root and follows the nearest existing ancestor's
+  zarr version, so a level made in a zarr v2 container is zarr v2 with OME 0.4 and the
+  hierarchy browses (cellmap-analyze, cellmap-flow). Adopting an identical array writes no
+  group document; after a write the document is read again and a lost entry is written
+  again, then raises (registration-station's concurrent block jobs).
+- `ensure_group_entry` is public: list an array the package did not create, such as a
+  symlinked `s0` (mesh-n-bone), or a level filled before it is listed.
+- `open(path, mode="r+")` opens an existing array for writing; a `WorldArray` pickles by
+  its arguments and its context's spec and reopens in the receiving process, a capped
+  context staying capped (mesh-n-bone's workers). `write.codecs_of` takes a path.
+- `write_index(begin, end, data)` writes whole write units only (the shard when sharded,
+  else the chunk; the clipped last unit allowed), so writers in different processes need
+  no coordination; `partial=True` does tensorstore's read-modify-write. `write(box, data)`
+  maps a world box by the centre rule first. No cast, no padding, no resampling on write.
+- `open_pyramid(path, mode="r+").add_level(name, factors=, origin=)` creates a level's
+  entry and empty array, placed and sized from the level below with its dtype, chunks,
+  shards, codecs and fill value copied; the downsampling itself stays with the caller.
+- `chunk_key` and `parse_chunk_key` for zarr v2, zarr v3 (both encodings, sharded arrays
+  keyed by shard), N5 and precomputed, as free functions and on `WorldArray`;
+  `list_chunks` uses the same parser. `write_shape`, `ArrayMeta.shard_shape`,
+  `chunks_covering(write=True)`.
+## 0.1.1 (2026-10-22)
+
+- funlib `voxel_size`/`offset` on the parent group place an array that carries none, for
+  zarr and N5 alike; funlib's keys on an N5 array are read in the array's C order, as
+  zarr-python's N5 store writes them (both from mesh-n-bone's review of its PR #48).
+- A precomputed path that names a scale is that scale under `level=0` or `"first"` too;
+  other levels warn and are ignored, as on an array path.
+- The GCE metadata guard is applied only off Google Cloud (decided from the DMI product
+  name and the `GCE_METADATA_*` variables), so metadata-server credentials keep working on
+  a Google VM.
+- `Pyramid.placements()`, `voxel_sizes()` and `translations()` read the group document when
+  the pyramid was opened through one of its level paths too; they used to open every level
+  from there, and a missing level made them raise.
+- **Placement change.** An N5 level with `downsamplingFactors` and no offset now sits with
+  its corner at the origin at every level, as Neuroglancer draws N5: translation
+  `f / 2 * base`. 0.1.0 followed BigDataViewer for `pixelResolution` pyramids, centring the
+  base's voxel 0 on the origin (`(f - 1) / 2 * base`, half a base voxel lower), while
+  placing `resolution` pyramids with the corner at the origin, so the two disagreed. Two
+  shared fixture cases move and are renamed: `n5_group_pixelresolution_and_downsampling_factors_corner_at_origin`
+  (translation 4, 4, 4) and `n5_bigstitcher_resolution_with_downsampling_factors_corner_at_origin`
+  (0.5, 0.157, 0.157).
+
+## 0.1.0 (2026-10-21)
+
+- First version, built to `docs/spec.md`: open zarr v2, zarr v3, N5 and Neuroglancer
+  precomputed arrays and pyramids through tensorstore; one metadata precedence per format
+  with the voxel-centre rule for translations; index-space and world-box reads with
+  padding; level selection; nearest and linear resampling; retries; shared and capped
+  tensorstore contexts.
+  Built and trialled in one day against the readers it replaces in cellmap-flow,
+  cellmap-analyze, mesh-n-bone and chunkmirage, and checked on their own data by the
+  CineMap, liconn-err, registration-station and salamander repos; see `docs/spec.md`,
+  "Trial status". Hot-path reads cost 0.1 to 0.2 ms over a raw tensorstore read.

{worldarray-0.1.1 → worldarray-0.3.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: worldarray
-Version: 0.1.1
+Version: 0.3.0
 Summary: Read zarr, N5 and Neuroglancer precomputed volumes in world coordinates through tensorstore: correct voxel placement, pyramid levels, padded region reads, resampling. numpy + tensorstore only.
 Project-URL: Homepage, https://janeliascicomp.github.io/worldarray/
 Project-URL: Documentation, https://janeliascicomp.github.io/worldarray/
@@ -80,7 +80,10 @@ resampled when no level matches, fast enough for a model server or a thousand cl
 workers, with retries for flaky object stores and thread caps for shared hosts. No one
 package did all of that, so every tool wrote its own reader, and the readers drifted apart. worldarray is that reader, written once, with the rules stated and tested.
 
-Its dependencies are `tensorstore` and `numpy`. It is read-only in this version. The
+Its dependencies are `tensorstore` and `numpy`. Since 0.2 it also writes the one thing
+every tool wrote for itself and got wrong in its own way: the OME-NGFF metadata of a placed
+array or pyramid level, and the array itself, created once and filled by aligned writes
+from any number of jobs. It never deletes, overwrites or downsamples. The
 [specification](https://janeliascicomp.github.io/worldarray/spec/) is the contract it is
 built to; every rule below is a test.
 
@@ -115,6 +118,13 @@ vol.read_index((0, 0, 0), (64, 64, 64), padding=0)
 
 pyr = wa.open_pyramid(path)             # every level, opened on first use
 pyr.select((16, 16, 16), mode="floor")  # the level to read at 16 nm
+
+# writing: a placed level, its OME-NGFF group document written for it (zarr v3 + OME 0.5 here)
+out = wa.create("out.zarr/s0", shape=(512, 512, 512), dtype="uint64",
+                voxel_size=(8, 8, 8), translation=(4, 4, 4), chunks=(64, 64, 64))
+out.write_index((0, 0, 0), (64, 64, 64), block)       # whole chunks only: many jobs, no locks
+out.write(box, block)                                  # or by world box
+s1 = wa.open_pyramid("out.zarr", mode="r+").add_level("s1", factors=2)   # placed and sized; you fill it
 ```
 
 ## What it does
@@ -139,6 +149,18 @@ pyr.select((16, 16, 16), mode="floor")  # the level to read at 16 nm
   cancelled future on timeout.
 - **Carries the geometry**: `Box` with `intersect`, `union`, `contains`, `grow`, `shift`,
   `snap` to a grid, and the voxel indices a world box selects.
+- **Writes placed arrays** (0.2): the OME-NGFF 0.4 or 0.5 document from voxel size, centre
+    translation, axes and units, with each level's placement explicit or derived from
+  factors and an origin; `create` following the container's format and writing the group
+  documents an output needs down from its `.zarr` root, with explicit chunks and codecs,
+  and only two answers to "it exists": raise, or adopt an identical array after checking;
+  whole-unit writes from many processes with no coordination; `add_level` as metadata plus
+  an empty array; chunk keys for every format. Nothing is ever deleted or overwritten.
+- **Plans outputs and blocks** (0.3): a store-less `Grid` with the same geometry as an
+  array, so an output is planned, cut into blocks and only then created; `blocks` tiling an
+  array or a box into write boxes with context-grown read boxes, in C order, as plain
+  picklable records a cluster job takes by id; the anisotropy-aware pyramid schedule.
+  Nothing is scheduled or run.
 
 ## The placement rule
 
@@ -152,7 +174,7 @@ moved half a voxel on reading:
 | OME-NGFF `translation` | centre | as is |
 | COSEM `transform.translate` | centre | as is |
 | `offset` beside `resolution` / `voxel_size` (`offset` missing: 0) | corner | `offset + voxel_size / 2` |
-| N5 `pixelResolution` or `resolution` with `downsamplingFactors f`, no offset | BigDataViewer | `(f - 1) / 2 * base` |
+| N5 `pixelResolution` or `resolution` with `downsamplingFactors f`, no offset | corner at the origin at every level, as Neuroglancer draws N5 | `f / 2 * base` |
 | precomputed `voxel_offset` | corner | `(voxel_offset + 1/2) * resolution` |
 | nothing | a plain array | voxels of 1, corner at the origin |
 
@@ -176,13 +198,32 @@ wa.open_tensorstore(path, level=None, context=None)
 wa.shared_context(cache_bytes=0) ; wa.capped_context(cache_bytes=0, concurrency=1, network=True) ; wa.Retry(...)
 wa.list_chunk_keys(path, separator=".")        # chunk indices from keys alone, no metadata needed
 wa.Box ; wa.units.TABLE
+
+# writing (0.2)
+wa.create(path, shape, dtype, voxel_size, translation, *, axes=None, units="nanometer", chunks=..., shards=None,
+          fill_value=0, codecs="zstd", format=None, attrs=None, group_attrs=None, exists="error", store_fill=False) -> WorldArray
+wa.open(path, mode="r+") ; wa.open_pyramid(path, mode="r+").add_level(name, factors=, origin=0, ...)
+wa.ome_multiscales(axes, units, levels, version="0.5", type=None) ; wa.level_placement(voxel_size, translation, factors, origin=0)
+wa.level_shape(shape, factors, origin=0) ; wa.chunk_key(index, format, ...) ; wa.parse_chunk_key(key, format, ...)
+wa.ensure_group_entry(group_path, name, voxel_size, translation, axes, units="nanometer")   # list an array you made yourself
+wa.units.ome_name("nm") -> "nanometer" ; wa.write.codecs_of(path) -> (codecs, shards)
+
+# grids and blocks (0.3)
+g = wa.Grid(shape, voxel_size, translation, axes=None, units="nanometer", chunks=None, shards=None)   # or Grid.from_meta(meta, voxel_size=)
+wa.create(path, grid=g, dtype="uint64")
+for b in wa.blocks(shape, block_shape, context=0, box=None, clip=True):   # also vol.blocks() and g.blocks()
+    b.begin, b.end, b.read_begin, b.read_end, b.inner, b.at_end, b.id    # plain records; blocks[id] is O(1)
+wa.level_factors(voxel_size, ratio=1.5, max_levels=None, stop_shape=None, shape=None)
+wa.units.abbreviation("nanometer") ; wa.store.join(path, "s0") ; wa.store.parent(path) ; wa.store.basename(path)
 ```
 
 `WorldArray`: `ts`, `meta`, `grid`, `shape`, `dtype`, `axes`, `voxel_size`, `translation`,
-`corner`, `box`, `levels`, `level_name`, `level_path`, `source_voxel_size`, `resampled`;
+`corner`, `box`, `levels`, `level_name`, `level_path`, `source_voxel_size`, `resampled`,
+`writable`, `write_shape`;
 `read(box, voxel_size=, padding=, method=, mask=, anchor=, transform=, post_transform=, select=, strict=)`,
 `read_index(begin, end, padding=, mask=, select=, transform=)`, `index_box(box)`,
-`chunks_covering(begin, end)`, `list_chunks()`, `reopen(context=)`.
+`chunks_covering(begin, end, write=)`, `list_chunks()`, `chunk_key(index)`, `parse_chunk_key(key)`,
+`write_index(begin, end, data, partial=)`, `write(box, data, partial=)`, `reopen(context=)`.
 
 A box is a `(begin, end)` pair of floats in your units, half-open per axis, over the
 spatial axes or every axis; any object with `.begin`/`.end` or `.offset`/`.shape` also

{worldarray-0.1.1 → worldarray-0.3.0}/README.md

@@ -21,7 +21,10 @@ resampled when no level matches, fast enough for a model server or a thousand cl
 workers, with retries for flaky object stores and thread caps for shared hosts. No one
 package did all of that, so every tool wrote its own reader, and the readers drifted apart. worldarray is that reader, written once, with the rules stated and tested.
 
-Its dependencies are `tensorstore` and `numpy`. It is read-only in this version. The
+Its dependencies are `tensorstore` and `numpy`. Since 0.2 it also writes the one thing
+every tool wrote for itself and got wrong in its own way: the OME-NGFF metadata of a placed
+array or pyramid level, and the array itself, created once and filled by aligned writes
+from any number of jobs. It never deletes, overwrites or downsamples. The
 [specification](https://janeliascicomp.github.io/worldarray/spec/) is the contract it is
 built to; every rule below is a test.
 
@@ -56,6 +59,13 @@ vol.read_index((0, 0, 0), (64, 64, 64), padding=0)
 
 pyr = wa.open_pyramid(path)             # every level, opened on first use
 pyr.select((16, 16, 16), mode="floor")  # the level to read at 16 nm
+
+# writing: a placed level, its OME-NGFF group document written for it (zarr v3 + OME 0.5 here)
+out = wa.create("out.zarr/s0", shape=(512, 512, 512), dtype="uint64",
+                voxel_size=(8, 8, 8), translation=(4, 4, 4), chunks=(64, 64, 64))
+out.write_index((0, 0, 0), (64, 64, 64), block)       # whole chunks only: many jobs, no locks
+out.write(box, block)                                  # or by world box
+s1 = wa.open_pyramid("out.zarr", mode="r+").add_level("s1", factors=2)   # placed and sized; you fill it
 ```
 
 ## What it does
@@ -80,6 +90,18 @@ pyr.select((16, 16, 16), mode="floor")  # the level to read at 16 nm
   cancelled future on timeout.
 - **Carries the geometry**: `Box` with `intersect`, `union`, `contains`, `grow`, `shift`,
   `snap` to a grid, and the voxel indices a world box selects.
+- **Writes placed arrays** (0.2): the OME-NGFF 0.4 or 0.5 document from voxel size, centre
+    translation, axes and units, with each level's placement explicit or derived from
+  factors and an origin; `create` following the container's format and writing the group
+  documents an output needs down from its `.zarr` root, with explicit chunks and codecs,
+  and only two answers to "it exists": raise, or adopt an identical array after checking;
+  whole-unit writes from many processes with no coordination; `add_level` as metadata plus
+  an empty array; chunk keys for every format. Nothing is ever deleted or overwritten.
+- **Plans outputs and blocks** (0.3): a store-less `Grid` with the same geometry as an
+  array, so an output is planned, cut into blocks and only then created; `blocks` tiling an
+  array or a box into write boxes with context-grown read boxes, in C order, as plain
+  picklable records a cluster job takes by id; the anisotropy-aware pyramid schedule.
+  Nothing is scheduled or run.
 
 ## The placement rule
 
@@ -93,7 +115,7 @@ moved half a voxel on reading:
 | OME-NGFF `translation` | centre | as is |
 | COSEM `transform.translate` | centre | as is |
 | `offset` beside `resolution` / `voxel_size` (`offset` missing: 0) | corner | `offset + voxel_size / 2` |
-| N5 `pixelResolution` or `resolution` with `downsamplingFactors f`, no offset | BigDataViewer | `(f - 1) / 2 * base` |
+| N5 `pixelResolution` or `resolution` with `downsamplingFactors f`, no offset | corner at the origin at every level, as Neuroglancer draws N5 | `f / 2 * base` |
 | precomputed `voxel_offset` | corner | `(voxel_offset + 1/2) * resolution` |
 | nothing | a plain array | voxels of 1, corner at the origin |
 
@@ -117,13 +139,32 @@ wa.open_tensorstore(path, level=None, context=None)
 wa.shared_context(cache_bytes=0) ; wa.capped_context(cache_bytes=0, concurrency=1, network=True) ; wa.Retry(...)
 wa.list_chunk_keys(path, separator=".")        # chunk indices from keys alone, no metadata needed
 wa.Box ; wa.units.TABLE
+
+# writing (0.2)
+wa.create(path, shape, dtype, voxel_size, translation, *, axes=None, units="nanometer", chunks=..., shards=None,
+          fill_value=0, codecs="zstd", format=None, attrs=None, group_attrs=None, exists="error", store_fill=False) -> WorldArray
+wa.open(path, mode="r+") ; wa.open_pyramid(path, mode="r+").add_level(name, factors=, origin=0, ...)
+wa.ome_multiscales(axes, units, levels, version="0.5", type=None) ; wa.level_placement(voxel_size, translation, factors, origin=0)
+wa.level_shape(shape, factors, origin=0) ; wa.chunk_key(index, format, ...) ; wa.parse_chunk_key(key, format, ...)
+wa.ensure_group_entry(group_path, name, voxel_size, translation, axes, units="nanometer")   # list an array you made yourself
+wa.units.ome_name("nm") -> "nanometer" ; wa.write.codecs_of(path) -> (codecs, shards)
+
+# grids and blocks (0.3)
+g = wa.Grid(shape, voxel_size, translation, axes=None, units="nanometer", chunks=None, shards=None)   # or Grid.from_meta(meta, voxel_size=)
+wa.create(path, grid=g, dtype="uint64")
+for b in wa.blocks(shape, block_shape, context=0, box=None, clip=True):   # also vol.blocks() and g.blocks()
+    b.begin, b.end, b.read_begin, b.read_end, b.inner, b.at_end, b.id    # plain records; blocks[id] is O(1)
+wa.level_factors(voxel_size, ratio=1.5, max_levels=None, stop_shape=None, shape=None)
+wa.units.abbreviation("nanometer") ; wa.store.join(path, "s0") ; wa.store.parent(path) ; wa.store.basename(path)
 ```
 
 `WorldArray`: `ts`, `meta`, `grid`, `shape`, `dtype`, `axes`, `voxel_size`, `translation`,
-`corner`, `box`, `levels`, `level_name`, `level_path`, `source_voxel_size`, `resampled`;
+`corner`, `box`, `levels`, `level_name`, `level_path`, `source_voxel_size`, `resampled`,
+`writable`, `write_shape`;
 `read(box, voxel_size=, padding=, method=, mask=, anchor=, transform=, post_transform=, select=, strict=)`,
 `read_index(begin, end, padding=, mask=, select=, transform=)`, `index_box(box)`,
-`chunks_covering(begin, end)`, `list_chunks()`, `reopen(context=)`.
+`chunks_covering(begin, end, write=)`, `list_chunks()`, `chunk_key(index)`, `parse_chunk_key(key)`,
+`write_index(begin, end, data, partial=)`, `write(box, data, partial=)`, `reopen(context=)`.
 
 A box is a `(begin, end)` pair of floats in your units, half-open per axis, over the
 spatial axes or every axis; any object with `.begin`/`.end` or `.offset`/`.shape` also