worldarray 0.1.0__tar.gz → 0.2.0__tar.gz

worldarray-0.2.0/CHANGELOG.md

@@ -0,0 +1,80 @@
+# Changelog
+
+## 0.2.0 (2026-10-22)
+
+Metadata and placed creation, built to spec draft 3 (Behaviours 10 to 16). Nothing on the
+read side changes.
+
+- `ome_multiscales(axes, units, levels, version)` builds the complete OME-NGFF 0.4 or 0.5
+  group document with one explicit placement per level; `level_placement` and
+  `level_shape` derive a level from factors and an origin (Convention 1's rule inverted,
+    so a level built from a ROI origin is placed from it; `level_shape` absorbs float-noise
+  factors such as `0.3 / 0.1`). Units are written in OME's names (`"nm"` becomes
+  `"nanometer"`; `units.ome_name`), an unknown unit as given; the multiscale's `type` is
+  written when given. Expected documents are pinned in `tests/fixtures/write.json`, shared
+  with chunkmirage's TypeScript builder, each case with its resolved per-axis units and
+  types, fifteen cases.
+- `create(path, shape, dtype, voxel_size, translation, chunks=..., ...)` writes the group
+  document (made when absent, else its one dataset entry added or replaced, every other
+  attribute kept) and then the array, as zarr v2 with OME 0.4 or zarr v3 with OME 0.5
+  following the group's format. `chunks` is required, codecs are explicit with a zstd
+  level 3 default, shards are a zarr v3 option. `exists` is `"error"` or `"adopt"` (an
+  identical array is opened for writing, a different one raises naming every difference,
+  a creation lost to another process is adopted after a short backoff). Nothing is ever
+    deleted, truncated or overwritten. Write units equal to the fill value are not stored
+  unless `store_fill=True`, and a stored unit written so loses its key, as zarr-python's
+  `write_empty_chunks=False` does. `create` also writes a bare group document for every
+  missing ancestor down from the `.zarr` root and follows the nearest existing ancestor's
+  zarr version, so a level made in a zarr v2 container is zarr v2 with OME 0.4 and the
+  hierarchy browses (cellmap-analyze, cellmap-flow). Adopting an identical array writes no
+  group document; after a write the document is read again and a lost entry is written
+  again, then raises (registration-station's concurrent block jobs).
+- `ensure_group_entry` is public: list an array the package did not create, such as a
+  symlinked `s0` (mesh-n-bone), or a level filled before it is listed.
+- `open(path, mode="r+")` opens an existing array for writing; a `WorldArray` pickles by
+  its arguments and its context's spec and reopens in the receiving process, a capped
+  context staying capped (mesh-n-bone's workers). `write.codecs_of` takes a path.
+- `write_index(begin, end, data)` writes whole write units only (the shard when sharded,
+  else the chunk; the clipped last unit allowed), so writers in different processes need
+  no coordination; `partial=True` does tensorstore's read-modify-write. `write(box, data)`
+  maps a world box by the centre rule first. No cast, no padding, no resampling on write.
+- `open_pyramid(path, mode="r+").add_level(name, factors=, origin=)` creates a level's
+  entry and empty array, placed and sized from the level below with its dtype, chunks,
+  shards, codecs and fill value copied; the downsampling itself stays with the caller.
+- `chunk_key` and `parse_chunk_key` for zarr v2, zarr v3 (both encodings, sharded arrays
+  keyed by shard), N5 and precomputed, as free functions and on `WorldArray`;
+  `list_chunks` uses the same parser. `write_shape`, `ArrayMeta.shard_shape`,
+  `chunks_covering(write=True)`.
+## 0.1.1 (2026-10-22)
+
+- funlib `voxel_size`/`offset` on the parent group place an array that carries none, for
+  zarr and N5 alike; funlib's keys on an N5 array are read in the array's C order, as
+  zarr-python's N5 store writes them (both from mesh-n-bone's review of its PR #48).
+- A precomputed path that names a scale is that scale under `level=0` or `"first"` too;
+  other levels warn and are ignored, as on an array path.
+- The GCE metadata guard is applied only off Google Cloud (decided from the DMI product
+  name and the `GCE_METADATA_*` variables), so metadata-server credentials keep working on
+  a Google VM.
+- `Pyramid.placements()`, `voxel_sizes()` and `translations()` read the group document when
+  the pyramid was opened through one of its level paths too; they used to open every level
+  from there, and a missing level made them raise.
+- **Placement change.** An N5 level with `downsamplingFactors` and no offset now sits with
+  its corner at the origin at every level, as Neuroglancer draws N5: translation
+  `f / 2 * base`. 0.1.0 followed BigDataViewer for `pixelResolution` pyramids, centring the
+  base's voxel 0 on the origin (`(f - 1) / 2 * base`, half a base voxel lower), while
+  placing `resolution` pyramids with the corner at the origin, so the two disagreed. Two
+  shared fixture cases move and are renamed: `n5_group_pixelresolution_and_downsampling_factors_corner_at_origin`
+  (translation 4, 4, 4) and `n5_bigstitcher_resolution_with_downsampling_factors_corner_at_origin`
+  (0.5, 0.157, 0.157).
+
+## 0.1.0 (2026-10-21)
+
+- First version, built to `docs/spec.md`: open zarr v2, zarr v3, N5 and Neuroglancer
+  precomputed arrays and pyramids through tensorstore; one metadata precedence per format
+  with the voxel-centre rule for translations; index-space and world-box reads with
+  padding; level selection; nearest and linear resampling; retries; shared and capped
+  tensorstore contexts.
+  Built and trialled in one day against the readers it replaces in cellmap-flow,
+  cellmap-analyze, mesh-n-bone and chunkmirage, and checked on their own data by the
+  CineMap, liconn-err, registration-station and salamander repos; see `docs/spec.md`,
+  "Trial status". Hot-path reads cost 0.1 to 0.2 ms over a raw tensorstore read.

{worldarray-0.1.0 → worldarray-0.2.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: worldarray
-Version: 0.1.0
+Version: 0.2.0
 Summary: Read zarr, N5 and Neuroglancer precomputed volumes in world coordinates through tensorstore: correct voxel placement, pyramid levels, padded region reads, resampling. numpy + tensorstore only.
 Project-URL: Homepage, https://janeliascicomp.github.io/worldarray/
 Project-URL: Documentation, https://janeliascicomp.github.io/worldarray/
@@ -67,8 +67,7 @@ pip install worldarray
 
 ## Why
 
-Volumetric data at Janelia lives in zarr v2, zarr v3, N5 and Neuroglancer precomputed
-stores, on local disks, S3, GCS and HTTP. Each format records *where the array sits*,
+Volumetric data lives in zarr v2, zarr v3, N5 and Neuroglancer precomputed stores, on local disks, S3, GCS and HTTP. Each format records *where the array sits*,
 its voxel size, origin, units and axes, in its own dialect, and the dialects disagree on
 the one thing that matters most: whether a stored origin means the centre of the first
 voxel or its corner. Get that wrong and every object is half a voxel out of place, which
@@ -79,10 +78,12 @@ rule for each, picking a pyramid level for a wanted resolution, and reading a re
 given in physical units as a plain numpy block, padded where the region leaves the data,
 resampled when no level matches, fast enough for a model server or a thousand cluster
 workers, with retries for flaky object stores and thread caps for shared hosts. No one
-package did all of that, so every tool here wrote its own reader, and the readers drifted
-apart. worldarray is that reader, written once, with the rules stated and tested.
+package did all of that, so every tool wrote its own reader, and the readers drifted apart. worldarray is that reader, written once, with the rules stated and tested.
 
-Its dependencies are `tensorstore` and `numpy`. It is read-only in this version. The
+Its dependencies are `tensorstore` and `numpy`. Since 0.2 it also writes the one thing
+every tool wrote for itself and got wrong in its own way: the OME-NGFF metadata of a placed
+array or pyramid level, and the array itself, created once and filled by aligned writes
+from any number of jobs. It never deletes, overwrites or downsamples. The
 [specification](https://janeliascicomp.github.io/worldarray/spec/) is the contract it is
 built to; every rule below is a test.
 
@@ -117,6 +118,13 @@ vol.read_index((0, 0, 0), (64, 64, 64), padding=0)
 
 pyr = wa.open_pyramid(path)             # every level, opened on first use
 pyr.select((16, 16, 16), mode="floor")  # the level to read at 16 nm
+
+# writing: a placed level, its OME-NGFF group document written for it (zarr v3 + OME 0.5 here)
+out = wa.create("out.zarr/s0", shape=(512, 512, 512), dtype="uint64",
+                voxel_size=(8, 8, 8), translation=(4, 4, 4), chunks=(64, 64, 64))
+out.write_index((0, 0, 0), (64, 64, 64), block)       # whole chunks only: many jobs, no locks
+out.write(box, block)                                  # or by world box
+s1 = wa.open_pyramid("out.zarr", mode="r+").add_level("s1", factors=2)   # placed and sized; you fill it
 ```
 
 ## What it does
@@ -141,6 +149,13 @@ pyr.select((16, 16, 16), mode="floor")  # the level to read at 16 nm
   cancelled future on timeout.
 - **Carries the geometry**: `Box` with `intersect`, `union`, `contains`, `grow`, `shift`,
   `snap` to a grid, and the voxel indices a world box selects.
+- **Writes placed arrays** (0.2): the OME-NGFF 0.4 or 0.5 document from voxel size, centre
+    translation, axes and units, with each level's placement explicit or derived from
+  factors and an origin; `create` following the container's format and writing the group
+  documents an output needs down from its `.zarr` root, with explicit chunks and codecs,
+  and only two answers to "it exists": raise, or adopt an identical array after checking;
+  whole-unit writes from many processes with no coordination; `add_level` as metadata plus
+  an empty array; chunk keys for every format. Nothing is ever deleted or overwritten.
 
 ## The placement rule
 
@@ -154,7 +169,7 @@ moved half a voxel on reading:
 | OME-NGFF `translation` | centre | as is |
 | COSEM `transform.translate` | centre | as is |
 | `offset` beside `resolution` / `voxel_size` (`offset` missing: 0) | corner | `offset + voxel_size / 2` |
-| N5 `pixelResolution` or `resolution` with `downsamplingFactors f`, no offset | BigDataViewer | `(f - 1) / 2 * base` |
+| N5 `pixelResolution` or `resolution` with `downsamplingFactors f`, no offset | corner at the origin at every level, as Neuroglancer draws N5 | `f / 2 * base` |
 | precomputed `voxel_offset` | corner | `(voxel_offset + 1/2) * resolution` |
 | nothing | a plain array | voxels of 1, corner at the origin |
 
@@ -178,13 +193,24 @@ wa.open_tensorstore(path, level=None, context=None)
 wa.shared_context(cache_bytes=0) ; wa.capped_context(cache_bytes=0, concurrency=1, network=True) ; wa.Retry(...)
 wa.list_chunk_keys(path, separator=".")        # chunk indices from keys alone, no metadata needed
 wa.Box ; wa.units.TABLE
+
+# writing (0.2)
+wa.create(path, shape, dtype, voxel_size, translation, *, axes=None, units="nanometer", chunks=..., shards=None,
+          fill_value=0, codecs="zstd", format=None, attrs=None, group_attrs=None, exists="error", store_fill=False) -> WorldArray
+wa.open(path, mode="r+") ; wa.open_pyramid(path, mode="r+").add_level(name, factors=, origin=0, ...)
+wa.ome_multiscales(axes, units, levels, version="0.5", type=None) ; wa.level_placement(voxel_size, translation, factors, origin=0)
+wa.level_shape(shape, factors, origin=0) ; wa.chunk_key(index, format, ...) ; wa.parse_chunk_key(key, format, ...)
+wa.ensure_group_entry(group_path, name, voxel_size, translation, axes, units="nanometer")   # list an array you made yourself
+wa.units.ome_name("nm") -> "nanometer" ; wa.write.codecs_of(path) -> (codecs, shards)
 ```
 
 `WorldArray`: `ts`, `meta`, `grid`, `shape`, `dtype`, `axes`, `voxel_size`, `translation`,
-`corner`, `box`, `levels`, `level_name`, `level_path`, `source_voxel_size`, `resampled`;
+`corner`, `box`, `levels`, `level_name`, `level_path`, `source_voxel_size`, `resampled`,
+`writable`, `write_shape`;
 `read(box, voxel_size=, padding=, method=, mask=, anchor=, transform=, post_transform=, select=, strict=)`,
 `read_index(begin, end, padding=, mask=, select=, transform=)`, `index_box(box)`,
-`chunks_covering(begin, end)`, `list_chunks()`, `reopen(context=)`.
+`chunks_covering(begin, end, write=)`, `list_chunks()`, `chunk_key(index)`, `parse_chunk_key(key)`,
+`write_index(begin, end, data, partial=)`, `write(box, data, partial=)`, `reopen(context=)`.
 
 A box is a `(begin, end)` pair of floats in your units, half-open per axis, over the
 spatial axes or every axis; any object with `.begin`/`.end` or `.offset`/`.shape` also

{worldarray-0.1.0 → worldarray-0.2.0}/README.md

@@ -8,8 +8,7 @@ pip install worldarray
 
 ## Why
 
-Volumetric data at Janelia lives in zarr v2, zarr v3, N5 and Neuroglancer precomputed
-stores, on local disks, S3, GCS and HTTP. Each format records *where the array sits*,
+Volumetric data lives in zarr v2, zarr v3, N5 and Neuroglancer precomputed stores, on local disks, S3, GCS and HTTP. Each format records *where the array sits*,
 its voxel size, origin, units and axes, in its own dialect, and the dialects disagree on
 the one thing that matters most: whether a stored origin means the centre of the first
 voxel or its corner. Get that wrong and every object is half a voxel out of place, which
@@ -20,10 +19,12 @@ rule for each, picking a pyramid level for a wanted resolution, and reading a re
 given in physical units as a plain numpy block, padded where the region leaves the data,
 resampled when no level matches, fast enough for a model server or a thousand cluster
 workers, with retries for flaky object stores and thread caps for shared hosts. No one
-package did all of that, so every tool here wrote its own reader, and the readers drifted
-apart. worldarray is that reader, written once, with the rules stated and tested.
+package did all of that, so every tool wrote its own reader, and the readers drifted apart. worldarray is that reader, written once, with the rules stated and tested.
 
-Its dependencies are `tensorstore` and `numpy`. It is read-only in this version. The
+Its dependencies are `tensorstore` and `numpy`. Since 0.2 it also writes the one thing
+every tool wrote for itself and got wrong in its own way: the OME-NGFF metadata of a placed
+array or pyramid level, and the array itself, created once and filled by aligned writes
+from any number of jobs. It never deletes, overwrites or downsamples. The
 [specification](https://janeliascicomp.github.io/worldarray/spec/) is the contract it is
 built to; every rule below is a test.
 
@@ -58,6 +59,13 @@ vol.read_index((0, 0, 0), (64, 64, 64), padding=0)
 
 pyr = wa.open_pyramid(path)             # every level, opened on first use
 pyr.select((16, 16, 16), mode="floor")  # the level to read at 16 nm
+
+# writing: a placed level, its OME-NGFF group document written for it (zarr v3 + OME 0.5 here)
+out = wa.create("out.zarr/s0", shape=(512, 512, 512), dtype="uint64",
+                voxel_size=(8, 8, 8), translation=(4, 4, 4), chunks=(64, 64, 64))
+out.write_index((0, 0, 0), (64, 64, 64), block)       # whole chunks only: many jobs, no locks
+out.write(box, block)                                  # or by world box
+s1 = wa.open_pyramid("out.zarr", mode="r+").add_level("s1", factors=2)   # placed and sized; you fill it
 ```
 
 ## What it does
@@ -82,6 +90,13 @@ pyr.select((16, 16, 16), mode="floor")  # the level to read at 16 nm
   cancelled future on timeout.
 - **Carries the geometry**: `Box` with `intersect`, `union`, `contains`, `grow`, `shift`,
   `snap` to a grid, and the voxel indices a world box selects.
+- **Writes placed arrays** (0.2): the OME-NGFF 0.4 or 0.5 document from voxel size, centre
+    translation, axes and units, with each level's placement explicit or derived from
+  factors and an origin; `create` following the container's format and writing the group
+  documents an output needs down from its `.zarr` root, with explicit chunks and codecs,
+  and only two answers to "it exists": raise, or adopt an identical array after checking;
+  whole-unit writes from many processes with no coordination; `add_level` as metadata plus
+  an empty array; chunk keys for every format. Nothing is ever deleted or overwritten.
 
 ## The placement rule
 
@@ -95,7 +110,7 @@ moved half a voxel on reading:
 | OME-NGFF `translation` | centre | as is |
 | COSEM `transform.translate` | centre | as is |
 | `offset` beside `resolution` / `voxel_size` (`offset` missing: 0) | corner | `offset + voxel_size / 2` |
-| N5 `pixelResolution` or `resolution` with `downsamplingFactors f`, no offset | BigDataViewer | `(f - 1) / 2 * base` |
+| N5 `pixelResolution` or `resolution` with `downsamplingFactors f`, no offset | corner at the origin at every level, as Neuroglancer draws N5 | `f / 2 * base` |
 | precomputed `voxel_offset` | corner | `(voxel_offset + 1/2) * resolution` |
 | nothing | a plain array | voxels of 1, corner at the origin |
 
@@ -119,13 +134,24 @@ wa.open_tensorstore(path, level=None, context=None)
 wa.shared_context(cache_bytes=0) ; wa.capped_context(cache_bytes=0, concurrency=1, network=True) ; wa.Retry(...)
 wa.list_chunk_keys(path, separator=".")        # chunk indices from keys alone, no metadata needed
 wa.Box ; wa.units.TABLE
+
+# writing (0.2)
+wa.create(path, shape, dtype, voxel_size, translation, *, axes=None, units="nanometer", chunks=..., shards=None,
+          fill_value=0, codecs="zstd", format=None, attrs=None, group_attrs=None, exists="error", store_fill=False) -> WorldArray
+wa.open(path, mode="r+") ; wa.open_pyramid(path, mode="r+").add_level(name, factors=, origin=0, ...)
+wa.ome_multiscales(axes, units, levels, version="0.5", type=None) ; wa.level_placement(voxel_size, translation, factors, origin=0)
+wa.level_shape(shape, factors, origin=0) ; wa.chunk_key(index, format, ...) ; wa.parse_chunk_key(key, format, ...)
+wa.ensure_group_entry(group_path, name, voxel_size, translation, axes, units="nanometer")   # list an array you made yourself
+wa.units.ome_name("nm") -> "nanometer" ; wa.write.codecs_of(path) -> (codecs, shards)
 ```
 
 `WorldArray`: `ts`, `meta`, `grid`, `shape`, `dtype`, `axes`, `voxel_size`, `translation`,
-`corner`, `box`, `levels`, `level_name`, `level_path`, `source_voxel_size`, `resampled`;
+`corner`, `box`, `levels`, `level_name`, `level_path`, `source_voxel_size`, `resampled`,
+`writable`, `write_shape`;
 `read(box, voxel_size=, padding=, method=, mask=, anchor=, transform=, post_transform=, select=, strict=)`,
 `read_index(begin, end, padding=, mask=, select=, transform=)`, `index_box(box)`,
-`chunks_covering(begin, end)`, `list_chunks()`, `reopen(context=)`.
+`chunks_covering(begin, end, write=)`, `list_chunks()`, `chunk_key(index)`, `parse_chunk_key(key)`,
+`write_index(begin, end, data, partial=)`, `write(box, data, partial=)`, `reopen(context=)`.
 
 A box is a `(begin, end)` pair of floats in your units, half-open per axis, over the
 spatial axes or every axis; any object with `.begin`/`.end` or `.offset`/`.shape` also