sharedbox 0.7.0__tar.gz → 0.8.0__tar.gz

{sharedbox-0.7.0 → sharedbox-0.8.0}/CHANGELOG.md

@@ -9,6 +9,108 @@ Dates are marked as `DD-MM-YYYY`
 
 <!-- towncrier release notes start -->
 
+## [0.8.0](https://github.com/jacopoabramo/sharedbox/releases/tag/v0.8.0) - 30-10-2026
+
+### Breaking
+
+- `sharedbox.Shape` takes its dimensions as one tuple, `Shape((480, 640))`, in
+  place of one argument per dimension. See
+  [How to migrate from 0.7](https://jacopoabramo.github.io/sharedbox/how-to/migrate-from-0.7/). ([#88](https://github.com/jacopoabramo/sharedbox/pull/88))
+- `SharedBox` reserves the names `metadata`, `metadata_version` and
+  `metadata_version_of`: a field or `InitVar` with one of them raises
+  `TypeError` when the class is defined. Rename the field:
+
+  ```python
+  # 0.7
+  class Parcel(SharedBox):
+      metadata: Annotated[str, Capacity(64)] = ""
+
+  # 0.8
+  class Parcel(SharedBox):
+      label: Annotated[str, Capacity(64)] = ""
+  ```
+
+  `SharedBox.update` takes the keyword `metadata`, so a type checker refuses
+  `update(**values)` when the values of the dictionary have a type that the
+  box's metadata type does not accept, such as `str`:
+
+  ```python
+  # 0.7
+  def restock(parcel: Parcel, values: dict[str, str]) -> None:
+      parcel.update(**values)
+
+  # 0.8
+  def restock(parcel: Parcel, values: dict[str, Any]) -> None:
+      parcel.update(**values)
+  ```
+
+  See
+  [How to migrate from 0.7](https://jacopoabramo.github.io/sharedbox/how-to/migrate-from-0.7/). ([#91](https://github.com/jacopoabramo/sharedbox/pull/91))
+
+### Added
+
+- `MaxShape`: an array of a stream item declared with a maximum shape, as in
+  `Annotated[npt.NDArray[np.uint16], MaxShape((2048, 2048))]`, stores its own
+  shape with each item, as kind 75 of stream layout 1.2. `writing()` views the
+  slot in the shape of the template's array, `slot.send` publishes the first
+  rows of that view without a copy, and `receive_into` returns a view of the
+  stored shape over `out`. ([#88](https://github.com/jacopoabramo/sharedbox/pull/88))
+- `SharedStream.sender(on_full="release")`: a send that has waited its whole
+  `timeout` releases the lossless readers a full ring behind and sends, as in
+  `stream.sender(on_full="release").send(item, timeout=0.1)`. Each released
+  reader raises `ReaderReleased`, whose `missed` counts the items it lost, and
+  goes on at the newest item; `ReceiveEvents.released` reports the count to
+  callbacks. `SharedSendStream.asend` and `awriting` take a `timeout`. In
+  `stream.hpp`, `send`, `send_with` and `begin` take an `on_full` policy, and a
+  released reader's `receive` gives `status::released`, in stream layout 1.3. ([#89](https://github.com/jacopoabramo/sharedbox/pull/89))
+- `MetadataFullError` and `MetadataGoneError`: raised when an add finds no
+  room for a version, and when the version a read asks for is no longer held. ([#91](https://github.com/jacopoabramo/sharedbox/pull/91))
+- `SharedStream.metadata`: a stream keeps metadata the same way.
+  `SharedStream.create(Reading, capacity=8, metadata={"unit": "kg"})` stores
+  version 1. `SharedSendStream.send`, `asend`, `send_nowait` and
+  `SendSlot.send` take `metadata=` and add a version that applies from that
+  item. `SharedReceiveStream.metadata` and `metadata_version` give the version
+  of the item the reader received last. `SharedStream.create` takes
+  `metadata_capacity` and `metadata_versions`, and `create` and `attach` take
+  `metadata_type`. `sharedbox` 0.7.0 and earlier cannot open a stream created
+  with metadata or a `metadata_capacity`. ([#91](https://github.com/jacopoabramo/sharedbox/pull/91))
+- `box.hpp`, `stream.hpp` and `sharedbox_c.h`: box layout 3.2 and stream
+  layout 1.4 hold metadata after the record or the description table.
+  `handle` and `stream` gain `metadata_head` and `read_metadata`, and
+  `stream_reader` gains `metadata_version_at`. `handle` and `stream_sender`
+  gain `add_metadata`. `handle::write(values, doc, lock_timeout)` and
+  `write_large` with the same arguments write fields and add a version in one
+  write, and
+  `stream_sender::can_add_metadata` checks an add without making it.
+  `sharedbox_c.h` adds `sbx_metadata_head`, `sbx_read_metadata`,
+  `sbx_add_metadata` and `SBX_E_METADATA_FULL`. ([#91](https://github.com/jacopoabramo/sharedbox/pull/91))
+- `SharedBox.metadata`: a box keeps metadata, JSON-like values stored as CBOR
+  in its segment, in numbered versions. `Scale.create(name, metadata={"unit":
+  "kg"})` stores version 1, and `SharedBox.update(metadata=...)` adds the next
+  one from any process, with or without field values, in one write.
+  `metadata`, `metadata_version` and `metadata_version_of` read the versions,
+  a map as a read-only mapping and a list as a tuple. `BoxEvents.metadata` is
+  emitted with each new version. The class keywords `metadata_capacity` and
+  `metadata_versions` size the room for versions, and
+  `class Scale(SharedBox[Calibration])` gives the metadata's type to type
+  checkers. ([#91](https://github.com/jacopoabramo/sharedbox/pull/91))
+
+### Changed
+
+- `sharedbox.DType` is a `StrEnum` with one member per element type:
+  `DType(numpy.uint8)` returns `DType.uint8`, which equals `"uint8"`. ([#88](https://github.com/jacopoabramo/sharedbox/pull/88))
+
+### Fixed
+
+- `SharedBox` and `SharedStream` at exit: the `atexit` hook of `sharedbox`
+  closes every box and stream the process still has open and releases the
+  native objects of `SharedBox` subclasses, their watches and events groups,
+  so nanobind reports no leaked instances when a box class defined at module
+  level outlives shutdown. A box or stream used after the hook raises
+  `BoxClosedError` or `StreamClosedError`, and creating or attaching a box
+  raises `BoxClosedError`. ([#90](https://github.com/jacopoabramo/sharedbox/pull/90))
+
+
 ## [0.7.0](https://github.com/jacopoabramo/sharedbox/releases/tag/v0.7.0) - 28-10-2026
 
 ### Added

{sharedbox-0.7.0 → sharedbox-0.8.0}/CLAUDE.md

@@ -13,11 +13,11 @@ the layout and the native module converts values.
 ```text
 sharedbox/
 |-- include/sharedbox/
-|   |-- core.hpp               what every segment kind uses: result and error, liveness, names, mappings, the open step, waiter slots, the type codec
-|   |-- box.hpp                box layout 3.1 and its protocols: create, publish, open, lock, read, write, capsule handle
-|   |-- stream.hpp             stream layout 1.1: create, open, the sender, readers, the gate, dead ends
+|   |-- core.hpp               what every segment kind uses: result and error, liveness, names, mappings, the open step, waiter slots, the type codec, the metadata area and its CBOR check
+|   |-- box.hpp                box layout 3.2 and its protocols: create, publish, open, lock, read, write, metadata, capsule handle
+|   |-- stream.hpp             stream layout 1.4: create, open, the sender, readers, the gate, dead ends, metadata
 |   |-- sharedbox.hpp          includes every kind; the header consumers include
-|   |-- sharedbox_c.h          minimal C interface: sbx_open, sbx_import, sbx_read, sbx_write, sbx_schema_hash, sbx_release, sbx_field_desc, typed sbx_read_* and sbx_write_*
+|   |-- sharedbox_c.h          minimal C interface: sbx_open, sbx_import, sbx_read, sbx_write, sbx_schema_hash, sbx_release, sbx_field_desc, typed sbx_read_* and sbx_write_*, sbx_metadata_head, sbx_read_metadata, sbx_add_metadata
 |   `-- sharedbox_c.cpp        its implementation, compiled by the consumer (CMake target sharedbox::c)
 |-- cmake/
 |   |-- sharedbox-config.cmake       find_package(sharedbox) from an installed wheel; the build writes
@@ -28,11 +28,14 @@ sharedbox/
 |   |-- _box.py                SharedBox: class keywords, fields, fields(), create/attach, __post_init__, update, snapshot, unlink, __sharedbox_box__
 |   |-- _layout.py             Capacity, field() and Field, field offsets and kind codes, schema hash
 |   |-- _types.py              annotation parsing into TypeSpec, the description table
-|   |-- _arrays.py             Shape, DType, SupportsDLPack, register_array_type
+|   |-- _arrays.py             Shape, MaxShape, DType, SupportsDLPack, register_array_type
 |   |-- _events.py             BoxEvents, FieldWatch, the watcher thread
 |   |-- _follow.py             BoxEvents.follow and unfollow: forwarding the events of boxes that references reach
 |   |-- _stream.py             SharedStream, SharedSendStream, SharedReceiveStream, ReceiveEvents, the statistics classes, the worker threads
+|   |-- _metadata.py           metadata documents: the type variable M, CBOR encoding, read-only decoding, the cache of decoded versions
 |   |-- _refs.py               reference fields: BoxRef, the class registry, BrokenReferenceError, UnknownBoxClassError
+|   |-- _exit.py               the exit handler: stops the watcher and stream threads, closes boxes, streams and ends,
+|   |                          replaces the native objects of watchers and box classes
 |   |-- _native.pyi            hand-written stub for the extension
 |   |-- py.typed
 |   |-- benchmarks/            the benchbox command (extra: benchmarks)
@@ -67,7 +70,9 @@ sharedbox/
 |   |-- test_follow.py         BoxEvents.follow, unfollow and nested
 |   |-- test_native_waiters.py waiter slots, dead owners, interrupt
 |   |-- test_box_sync_async.py a sync and an async watch on one box in two threads
+|   |-- test_metadata.py       metadata of a box: documents, versions, events, typed access, reserved names
 |   |-- test_stream.py         SharedStream, sender and reader, blocking and non-blocking calls
+|   |-- test_stream_metadata.py  metadata of a stream: the version of each item, in one process and across processes
 |   |-- test_stream_async.py   asyncio calls on a stream, cancellation
 |   |-- test_stream_events.py  ReceiveEvents delivery
 |   |-- test_stream_processes.py  streams across processes, dead senders and readers
@@ -91,7 +96,7 @@ sharedbox/
 |   |-- examples/              one script per how-to guide, included by the guide
 |   |-- javascripts/charts.js  draws each div of class sbx-chart from a plotly figure JSON file
 |   |-- assets/benchmarks/     the figure JSON files of the performance page, from benchbox plot
-|   `-- reference/segment-layout.md  core 1.0, box layout 3.1, names and protocols
+|   `-- reference/segment-layout.md  core 1.0, box layout 3.2, stream layout 1.4, names and protocols
 |-- includes/abbreviations.md  acronym tooltips appended to every page
 |-- zensical.toml              site configuration and navigation
 |-- .github/workflows/ci.yaml  lint, docs check, C++ tests, cibuildwheel wheels, tests, stress,
@@ -135,18 +140,19 @@ class's identity (the `identity` class keyword, by default
 always asks for a new name and raises `SegmentExistsError` if it is taken.
 
 Every segment starts with the same 64-byte line, whatever its kind. Core
-version 1.0, box layout 3.1, from offset 0:
+version 1.0, box layout 3.2, from offset 0:
 
 - Common line 0, written once at creation: `magic` (`SBX_BOX_` for a box,
   `SBX_STRM` for a stream), `core_major` 1, `core_minor` 0, `kind_major` 3,
-  `kind_minor` 1, `schema_hash`, `create_id`, `creator_start`,
+  `kind_minor` 2, `schema_hash`, `create_id`, `creator_start`,
   `creator_pidns`, `creator_pid`, `waiter_slots`, `size`.
 - Box line 1, at offset 64: `seq` (sequence lock; the generation is
   `seq >> 1`, there is no generation field), `writer_pid`, `wake_word`,
   `waiters` and `sleepers`, which writes and waits change; then written
   once at creation `field_count`, `record_size`, `record`, `tail` (always
-  128) and `types_size`, followed by reserved zero bytes. The header is
-  128 bytes.
+  128), `types_size`, `metadata_size` and `metadata` (the offset of the
+  metadata area, 0 for none); last `metadata_head`, the newest metadata
+  version, 0 for none, which each add changes. The header is 128 bytes.
 - The tail: `field_count` field table entries of 8 bytes (`u32 offset`,
   `u32 capacity_and_kind`: top 8 bits the kind code, the low 24 the
   capacity or size, or for kinds 64 and up the offset of the field's
@@ -155,10 +161,15 @@ version 1.0, box layout 3.1, from offset 0:
   `asleep_on`: the offset of the count the slot's current wait added 1 to,
   which freeing a dead owner's slot takes back), then the description
   table of `types_size` bytes. Kinds are 0 to 5, 6 to 12 (fixed-size
-  scalars), 13 to 23 (fixed-width numbers and `datetime64`) and 64 to 74
-  (described types).
-- The record, at a multiple of 64. The mapping size is rounded up to 4 KiB.
-  `record + record_size <= 2**32 - 4096`.
+  scalars), 13 to 23 (fixed-width numbers and `datetime64`) and 64 to 75
+  (described types; 75, an array up to a maximum shape, only in a stream).
+- The record, at a multiple of 64. `record + record_size <= 2**32 - 4096`.
+- The metadata area, when `metadata` is not 0, at the first multiple of 64
+  after the record, `metadata + metadata_size <= 2**32 - 4096`: a 64-byte
+  line (`entries`, `ring_size`), `entries` version entries of 32 bytes
+  (`seq`, `version`, `position`, `offset`, `length`), then a ring of
+  `ring_size` bytes holding one CBOR document per version. The mapping size
+  is rounded up to 4 KiB.
 
 `magic` is stored last on create, with release ordering, after the initial
 values are in the record; attach waits for it. Attach copies line 0 and the
@@ -174,8 +185,11 @@ layout 1.0 or 2.0, which 0.5 and earlier wrote.
 A stream's segment starts with a 256-byte header: the common line, a line
 of geometry written once, the sender's line and the readers' line. Then
 come a 64-byte entry per reader, the waiter slots, `capacity` slots each
-holding a `seq` and one item, and the description table. Every offset is
-64-bit. The full specification is in `docs/reference/segment-layout.md`.
+holding a `seq` and one item, the description table, and the metadata
+area laid out as a box's. Stream layout 1.4 keeps `metadata_size` and
+`metadata` in the geometry line and `metadata_head` in the sender's line.
+Every offset is 64-bit. The full specification is in
+`docs/reference/segment-layout.md`.
 
 ### Record encoding
 

{sharedbox-0.7.0 → sharedbox-0.8.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: sharedbox
-Version: 0.7.0
+Version: 0.8.0
 Summary: Python containers using shared memory.
 Author-Email: Jacopo Abramo <jacopo.abramo@gmail.com>
 License-Expression: Apache-2.0
@@ -14,7 +14,9 @@ Classifier: Programming Language :: Python :: Free Threading :: 2 - Beta
 Classifier: Programming Language :: C++
 Classifier: Topic :: Software Development :: Libraries :: Python Modules
 Requires-Python: >=3.11
+Requires-Dist: cbor2>=6.1.5
 Requires-Dist: psygnal>=0.16.1
+Requires-Dist: typing-extensions>=4.16.0; python_full_version < "3.13"
 Provides-Extra: benchmarks
 Requires-Dist: numpy>=2.4.6; extra == "benchmarks"
 Requires-Dist: plotly>=7.1.0; extra == "benchmarks"

sharedbox-0.8.0/docs/examples/attach_metadata.py

@@ -0,0 +1,78 @@
+"""The script of the guide "How to attach metadata to a box or stream"."""
+
+# --8<-- [start:box]
+from typing import TypedDict
+
+from sharedbox import SharedBox, SharedStream
+
+
+# --8<-- [start:calibration]
+class Calibration(TypedDict):
+    unit: str
+    offset: float
+
+
+# --8<-- [end:calibration]
+class Scale(SharedBox[Calibration]):
+    weight: float = 0.0
+
+
+scale = Scale.create("example-scale", metadata={"unit": "kg", "offset": 0.0})
+print(scale.metadata_version)  # 1
+print(scale.metadata)  # {'unit': 'kg', 'offset': 0.0}
+# --8<-- [end:box]
+
+# --8<-- [start:add]
+scale.update(weight=1.2, metadata={"unit": "kg", "offset": 0.25})
+print(scale.weight, scale.metadata_version)  # 1.2 2
+print(scale.metadata_version_of(1)["offset"])  # 0.0
+print(scale.metadata_version_of(2)["offset"])  # 0.25
+scale.update(metadata={"unit": "kg", "offset": 0.3})
+print(scale.weight, scale.metadata_version)  # 1.2 3
+# --8<-- [end:add]
+
+
+# --8<-- [start:capacity]
+class Display(SharedBox[Calibration], metadata_capacity=4096, metadata_versions=8):
+    weight: float = 0.0
+
+
+display = Display.create("example-display")
+print(display.metadata)  # None
+display.update(metadata={"unit": "g", "offset": 0.0})
+# --8<-- [end:capacity]
+
+# --8<-- [start:read-only]
+stored = scale.metadata_version_of(2)
+try:
+    stored["offset"] = 0.5
+except TypeError as error:
+    print(error)  # 'mappingproxy' object does not support item assignment
+scale.update(metadata={**stored, "offset": 0.5})
+# --8<-- [end:read-only]
+
+
+# --8<-- [start:stream]
+with SharedStream.create(
+    float,
+    "example-scale:readings",
+    capacity=8,
+    metadata={"unit": "kg", "offset": 0.0},
+    metadata_type=Calibration,
+) as stream:
+    reader = stream.reader()
+    with stream.sender() as sender:
+        sender.send(1.5)
+        sender.send(1.75, metadata={"unit": "kg", "offset": 0.25})
+    for weight in reader:
+        calibration = reader.metadata
+        if calibration is not None:
+            print(reader.metadata_version, weight - calibration["offset"])
+            # prints 1 1.5, then 2 1.5
+SharedStream.unlink("example-scale:readings")
+# --8<-- [end:stream]
+
+scale.close()
+display.close()
+Scale.unlink("example-scale")
+Display.unlink("example-display")

{sharedbox-0.7.0 → sharedbox-0.8.0}/docs/examples/send_items_through_a_stream.py

@@ -12,6 +12,8 @@ from sharedbox import (
     Capacity,
     DType,
     EndOfStream,
+    MaxShape,
+    ReaderReleased,
     Shape,
     SharedReceiveStream,
     SharedSendStream,
@@ -23,7 +25,7 @@ from sharedbox import (
 class Letter:
     index: int
     recipient: Annotated[str, Capacity(32)]
-    scan: Annotated[np.ndarray, Shape(32, 32), DType("uint8")]
+    scan: Annotated[np.ndarray, Shape((32, 32)), DType("uint8")]
 
 
 # --8<-- [end:types]
@@ -93,6 +95,50 @@ def send_filled(sender: SharedSendStream[Letter], count: int) -> None:
 # --8<-- [end:writing]
 
 
+# --8<-- [start:shapes]
+@dataclass(frozen=True)
+class Sack:
+    index: int
+    weights: Annotated[np.ndarray, MaxShape((64,)), DType("float32")]
+
+
+def send_sacks(sender: SharedSendStream[Sack]) -> None:
+    sender.send(Sack(0, np.array([1.5, 0.25, 4.0], np.float32)))
+    template = Sack(1, np.empty(64, np.float32))
+    with sender.writing(template) as slot:
+        weights = slot.item.weights
+        weights[:5] = 2.5
+        slot.send(replace(slot.item, weights=weights[:5]))
+
+
+def read_sacks(reader: SharedReceiveStream[Sack]) -> None:
+    weights = np.empty(64, np.float32)
+    for sack in reader.iter_into({"weights": weights}):
+        print(sack.index, sack.weights.shape)
+
+
+# --8<-- [end:shapes]
+
+
+# --8<-- [start:release]
+def sort_without_stalling(sender: SharedSendStream[Letter], count: int) -> None:
+    for i in range(count):
+        sender.send(Letter(i, "Ada", np.full((32, 32), i, np.uint8)), timeout=0.1)
+
+
+def file_letters(reader: SharedReceiveStream[Letter]) -> None:
+    while True:
+        try:
+            for letter in reader:
+                print("filed", letter.index)
+            return
+        except ReaderReleased as released:
+            print("missed", released.missed)
+
+
+# --8<-- [end:release]
+
+
 # --8<-- [start:main]
 if __name__ == "__main__":
     context = mp.get_context("spawn")
@@ -127,3 +173,15 @@ if __name__ == "__main__":
         for letter in reader:
             print(letter.index, int(letter.scan[0, 0]))
     SharedStream.unlink("example-sorting:filled")
+    with SharedStream.create(Sack, "example-sorting:sacks", capacity=4) as sacks:
+        sack_reader = sacks.reader()
+        with sacks.sender() as sack_sender:
+            send_sacks(sack_sender)
+        read_sacks(sack_reader)
+    SharedStream.unlink("example-sorting:sacks")
+    with SharedStream.create(Letter, "example-sorting:release", capacity=4) as stream:
+        reader = stream.reader()
+        with stream.sender(on_full="release") as sender:
+            sort_without_stalling(sender, 6)
+        file_letters(reader)
+    SharedStream.unlink("example-sorting:release")

{sharedbox-0.7.0 → sharedbox-0.8.0}/docs/examples/store_arrays.py

@@ -13,7 +13,9 @@ from sharedbox import DType, Shape, SharedBox, SupportsDLPack, field
 class Scanner(SharedBox, name="example-scanner"):
     scan: npt.NDArray[np.uint8] = np.zeros((4, 6), np.uint8)
     gain: npt.NDArray[np.float64] = field(shape=(2,), default=np.ones(2))
-    raw: Annotated[SupportsDLPack, Shape(3), DType("float32")] = np.zeros(3, np.float32)
+    raw: Annotated[SupportsDLPack, Shape((3,)), DType("float32")] = np.zeros(
+        3, np.float32
+    )
 
 
 # --8<-- [end:declare]

{sharedbox-0.7.0 → sharedbox-0.8.0}/docs/explanation/glossary.md

@@ -88,6 +88,27 @@ Lossy is a [reader](#reader) mode in which the reader skips the items the
 [`missed`][sharedbox.SharedReceiveStream.missed]. See
 [`reader`][sharedbox.SharedStream.reader].
 
+### Metadata
+
+Metadata is a value a [box](#box) or [stream](#stream) keeps in its
+[segment](#segment) to say what its fields or items mean, such as their
+units or the calibration they were taken with. Every process reads it from
+the segment, and adding a new value adds a
+[metadata version](#metadata-version). The `metadata` of
+[`field`][sharedbox.field] is a different thing, kept in each process's
+class. See
+[How to attach metadata to a box or stream](../how-to/attach-metadata.md).
+
+### Metadata version
+
+A metadata version is one value of a box's or stream's
+[metadata](#metadata), numbered from 1. Adding metadata adds a version and
+leaves the earlier ones as they were, until the box or stream lets the
+oldest go to make room for new ones. A stream item belongs to the newest
+version its [sender](#sender) added with that item or before it. See
+[`update`][sharedbox.SharedBox.update] and
+[`send`][sharedbox.SharedSendStream.send].
+
 ### Reader
 
 A reader is an object that receives the items of a [stream](#stream) in one
@@ -95,6 +116,15 @@ process. Each reader has its own position in the stream and a mode:
 [lossless](#lossless), [lossy](#lossy) or [latest](#latest). See
 [`SharedReceiveStream`][sharedbox.SharedReceiveStream].
 
+### Released reader
+
+A released reader is a [lossless](#lossless) [reader](#reader) that the
+[sender](#sender) stopped waiting for: the sender was opened with
+`on_full="release"`, and a send waited its whole timeout. Its next receive raises
+[`ReaderReleased`][sharedbox.ReaderReleased] with the number of items it
+lost, and it goes on at the newest item. See
+[`sender`][sharedbox.SharedStream.sender].
+
 ### Reference field
 
 A reference field is a [field](#field) annotated with another `SharedBox`

sharedbox-0.8.0/docs/how-to/attach-metadata.md

@@ -0,0 +1,192 @@
+---
+icon: lucide/wrench
+---
+
+# How to attach metadata to a box or stream
+
+A [box](../explanation/glossary.md#box) or a
+[stream](../explanation/glossary.md#stream) carries values, but not what
+they mean, such as the unit of a weight or the calibration a reading was
+taken with.
+[Metadata](../explanation/glossary.md#metadata) carries that, in the same
+[segment](../explanation/glossary.md#segment), so every process that opens
+the box or stream reads it from there, in Python, C or C++. Each change
+adds a new [metadata version](../explanation/glossary.md#metadata-version)
+and keeps the older ones for a while, so a reader can tell which version
+an item was sent under.
+
+The `metadata` you pass to [`field`][sharedbox.field] is a different
+thing: a mapping kept in each process's class and never stored in the
+segment, from which `sharedbox` reads only its own keys, such as `"shape"`.
+
+## Create a box with metadata
+
+Give the type of the metadata as the type argument of `SharedBox`, and pass
+the first value as `metadata` when you create the box:
+
+```{.python}
+--8<-- "docs/examples/attach_metadata.py:box"
+```
+
+Here the metadata is a `TypedDict`, so a type checker knows what
+[`metadata`][sharedbox.SharedBox.metadata] holds and checks every value you
+add. Without a type argument, as in `class Scale(SharedBox)`, the metadata
+is a `Mapping[str, Any]`. `metadata` gives the value of the newest version,
+or `None` for a box that has none, and
+[`metadata_version`][sharedbox.SharedBox.metadata_version] gives its number,
+counting from 1.
+
+A value is `None`, a `bool`, an `int`, a `float`, a `str`, `bytes`, a list
+or tuple, or a mapping with `str` keys, nested up to 32 levels;
+[`update`][sharedbox.SharedBox.update] lists the limits. The
+type argument must be a mapping type, but only a type checker holds you to
+that: at run time a value that is not a mapping is stored all the same.
+Calling the class takes the `metadata` keyword too, but a type checker
+accepts it only in `create`, where it does not check the value.
+
+## Add a version
+
+Any process that has the box open adds a version by passing `metadata` to
+[`update`][sharedbox.SharedBox.update]. The call may also name fields, and
+then the new values and the new version go in as one write, so a reader
+that reads the fields and then `metadata` gets the version written with
+them, or a newer one. Without fields, the call only adds the version:
+
+```{.python}
+--8<-- "docs/examples/attach_metadata.py:add"
+```
+
+The first call writes `weight` and version 2 together, and the second adds
+version 3. An `update` that raises, because a value is out of range or the
+box has no room for the version, writes neither the fields nor the version,
+and `metadata=None`, the default, adds no version. Every process that
+listens to the `metadata` signal of the box's
+[`events`][sharedbox.SharedBox.events] gets the new version's number, once
+for several versions added close together, and gets it before the signals
+of the fields written with it:
+
+```python
+scale.events.metadata.connect(lambda version: print("calibration", version))
+```
+
+[`metadata_version_of`][sharedbox.SharedBox.metadata_version_of] reads an
+older version for as long as the box keeps it. The box keeps the last
+`metadata_versions` versions, 64 unless the class sets that keyword, or
+fewer when they no longer fit in its room for metadata. Reading a version
+it has let go raises [`MetadataGoneError`][sharedbox.MetadataGoneError].
+
+## Leave room for metadata added later
+
+A box created with `metadata` gets 65536 bytes of room for its metadata,
+and one created without it gets none, so `update(metadata=...)` on it raises
+[`MetadataFullError`][sharedbox.MetadataFullError]. When the first version
+comes after the box is created, give the class a `metadata_capacity`:
+
+```{.python}
+--8<-- "docs/examples/attach_metadata.py:capacity"
+```
+
+The capacity is the number of bytes all the kept versions share, 0 or at
+least 24. One version can take up to a third of it, rounded down to a
+multiple of 8, and a longer value raises `ValueError` and adds nothing. A
+value is stored as CBOR, a compact binary form of JSON-like values, so
+`{"unit": "kg", "offset": 0.0}` takes 19 bytes. `metadata_versions`, 2 to 65535, limits
+how many versions the box keeps. A stream takes both as keywords of
+[`create`][sharedbox.SharedStream.create].
+
+## Attach metadata to a stream
+
+A stream takes its first version as `metadata` in
+[`create`][sharedbox.SharedStream.create], and its sender adds each later
+one with the `metadata` of [`send`][sharedbox.SharedSendStream.send].
+[`asend`][sharedbox.SharedSendStream.asend],
+[`send_nowait`][sharedbox.SharedSendStream.send_nowait] and the `send` of
+the slot that [`writing`][sharedbox.SharedSendStream.writing] gives take
+the keyword too. A version applies from the item sent with it, and a
+reader's [`metadata`][sharedbox.SharedReceiveStream.metadata] and
+[`metadata_version`][sharedbox.SharedReceiveStream.metadata_version]
+describe the item it received last, so the reader can tell which version
+each item was sent under. The metadata of this stream is a `TypedDict` too:
+
+```{.python}
+--8<-- "docs/examples/attach_metadata.py:calibration"
+```
+
+```{.python}
+--8<-- "docs/examples/attach_metadata.py:stream"
+```
+
+The reader prints version 1 for the first reading, which was sent before
+version 2 existed, and version 2 for the reading sent with it. Before its
+first item, a reader's `metadata_version` is 0
+and its `metadata` is `None`, and the same holds for an item sent before the
+stream had any version. The stream's own
+[`metadata`][sharedbox.SharedStream.metadata] gives the newest version, as
+a box's does.
+
+A type checker takes the stream's metadata type from `metadata_type`,
+given to `create` as above or to `attach` in another process, as in
+`SharedStream.attach(float, name, metadata_type=Calibration)`. Without
+it the metadata is a `Mapping[str, Any]`, whatever `metadata` you pass.
+
+A stream keeps every version that an item still in its ring was sent
+under, and keeps version 1 while an item sent before it is in the ring,
+so that a reader can tell such an item from one whose version is gone. A
+send with `metadata` that would let such a version go raises
+`MetadataFullError` before it waits for room in the ring, so it sends
+nothing and releases no reader; the slot's `send` in a `writing` block
+raises it the same way and leaves the slot reserved. The same send
+succeeds once the sender has sent enough further items that none of
+those items is left in the ring.
+Once a reader's last item has left the ring, though, its version can go,
+and the reader's `metadata` then raises `MetadataGoneError`. A lossy or
+latest reader meets this first, because the sender may overwrite its item
+right after the reader receives it.
+
+!!! warning "A callback on the main thread can see a later item's version"
+    The reader's `metadata` follows the item the reader received last,
+    which may be later than the item a callback was given. A callback of
+    [`reader.events`][sharedbox.SharedReceiveStream.events] connected with
+    `thread="main"` runs after the reader's thread has received its item,
+    and that thread may have received the next one by then. Read
+    `metadata` in a callback that runs on the reader's thread, and pass it
+    on with the item.
+
+!!! warning "sharedbox 0.7 cannot open a stream with metadata"
+    `sharedbox` 0.7.0 and earlier refuse a stream created with `metadata`
+    or a `metadata_capacity`, and raise
+    [`SchemaMismatchError`][sharedbox.SchemaMismatchError] saying its
+    header is corrupt. They open a box with metadata, and a stream created
+    without it, and ignore the metadata. Create a stream without metadata
+    while a process on 0.7 still opens it.
+
+## Change a value you read
+
+A value you read is a read-only view, because every read of a version
+through the same box or stream shares one decoded copy. A mapping reads
+back as a `types.MappingProxyType` and a list as a tuple, nested the same
+way. A type checker still sees the metadata's type, though, so it accepts
+a change that fails when it runs. To change a key, build a new value and
+add it as the next version:
+
+```{.python}
+--8<-- "docs/examples/attach_metadata.py:read-only"
+```
+
+## Read metadata from C or C++
+
+A C program reads a box's versions with `sbx_metadata_head` and
+`sbx_read_metadata`, and adds one with `sbx_add_metadata`. In C++, `handle`
+and `stream` read versions with `metadata_head` and `read_metadata`,
+`handle` and `stream_sender` add them with `add_metadata`, and
+`stream_reader::metadata_version_at` finds the version an item was sent
+under; streams have no C calls. Each version is one CBOR value, which your
+program decodes with a CBOR library of its own.
+[Segment layout](../reference/segment-layout.md#metadata) lists the calls
+and the CBOR a version may hold.
+
+??? example "The whole script"
+
+    ```{.python}
+    --8<-- "docs/examples/attach_metadata.py"
+    ```

{sharedbox-0.7.0 → sharedbox-0.8.0}/docs/how-to/index.md

@@ -12,6 +12,8 @@ you're trying to do; if `sharedbox` is new to you, the
 
 - [How to install sharedbox](install-sharedbox.md): the wheels, the
   platforms and the `benchmarks` extra
+- [How to migrate from 0.7](migrate-from-0.7.md): the changes code written
+  for 0.7 needs
 
 ### Use a box or a stream
 
@@ -42,6 +44,9 @@ items through a [stream](../explanation/glossary.md#stream).
   one callback for changes in every box a box refers to
 - [How to send items through a stream](send-items-through-a-stream.md): one
   sender, readers that receive every item or only the newest, and asyncio
+- [How to attach metadata to a box or stream](attach-metadata.md): units or
+  calibration that every process reads, and the version each item was sent
+  under
 
 ### C and C++