slick-stream-buffer-py 0.1.0__tar.gz → 2.0.0__tar.gz

{slick_stream_buffer_py-0.1.0 → slick_stream_buffer_py-2.0.0}/API_DIFFERENCES.md

@@ -1,8 +1,8 @@
 # API Differences: Python vs C++
 
 The Python implementation is binary-compatible with the C++ `slick::SlickStreamBuffer`
-at the shared memory level — every byte, offset, and memory ordering matches. The
-*language-level* API differs in the following ways.
+(`slick/stream_buffer.hpp` v2.0.0) at the shared memory level — every byte, offset, and
+memory ordering matches. The *language-level* API differs in the following ways.
 
 ## Construction
 
@@ -11,9 +11,16 @@ at the shared memory level — every byte, offset, and memory ordering matches.
 | `stream_buffer buf(capacity, control_size)` | `SlickStreamBuffer(capacity=..., control_size=...)` |
 | `stream_buffer buf(capacity, control_size, "name")` | `SlickStreamBuffer(capacity=..., control_size=..., name="name")` |
 | `stream_buffer buf("name")` | `SlickStreamBuffer(name="name")` |
+| `stream_buffer::remove("name")` | `SlickStreamBuffer.remove("name")` (static, returns `bool`) |
 | throws `std::invalid_argument` (bad sizes) | raises `ValueError` |
+| throws `std::length_error` (geometry past the address space) | raises `ValueError` |
 | throws `std::runtime_error` (shm failures) | raises `RuntimeError` (or `FileNotFoundError` when opening a missing segment) |
 
+Both implementations take initialization authority from the OS, not from the in-band
+init-state slot: C++ uses `slick::shm`'s `is_creator()`, Python uses whether
+`SharedMemory(create=True)` succeeded. Both reduce to `O_CREAT|O_EXCL` on POSIX and
+`ERROR_ALREADY_EXISTS` on Windows, so neither ever initializes a segment it did not create.
+
 ## Consumer cursor is passed by value
 
 C++ updates the cursor through a reference; Python has no by-reference ints, so the
@@ -34,6 +41,31 @@ data, length, cursor = buf.read(cursor)   # updated cursor returned
 `read()` returns `(None, 0, cursor)` where C++ returns `{nullptr, 0}`.
 `read_last()` returns `(None, 0)` / `(bytes, length)`.
 
+## Read traits are a runtime argument, not a template parameter
+
+C++ configures `read()` with a template parameter, so the flags are compile-time constants
+and a misconfiguration cannot cross a translation unit. Python has no equivalent, so the
+same traits class is passed as an ordinary second argument:
+
+```cpp
+// C++
+struct counting : slick::read_traits { static constexpr bool count_loss = true; };
+auto [data, len] = buf.read<counting>(cursor);
+```
+
+```python
+# Python
+from slick_stream_buffer_py import counting_read_traits
+data, length, cursor = buf.read(cursor, counting_read_traits)
+```
+
+The flags and their defaults are identical (`detect_reset = True`, `count_loss = False`),
+and subclassing `read_traits` works the same way as deriving from `slick::read_traits`.
+Python also exports `ReadTraits` / `CountingReadTraits` as PEP 8 spellings of the same two
+classes. The performance argument for making `count_loss` opt-in is a C++ one (~20% of
+`read()` on MSVC); in Python the cost is two attribute lookups per call, but the *semantics*
+match so that `loss_count()` means the same thing on both sides.
+
 ## Zero-copy vs copies
 
 | Operation | C++ | Python |
@@ -52,11 +84,12 @@ Python API safe by default.
 `prepare()`/`data()` memoryviews are invalidated by the next `prepare()` (which may
 relocate the region) — same semantics as C++ flat_buffer-style invalidation.
 
-## prepare() overflow
+## prepare() / consume() overflow
 
-C++ throws `std::length_error` when `size() + n > capacity()`; Python raises
-`ValueError`. C++ `consume()` asserts messages are < 4 GiB (debug builds); Python
-always raises `ValueError`.
+C++ throws `std::length_error` when `size() + n > capacity()`; Python raises `ValueError`.
+`consume()` behaves the same on both sides — a message of 4 GiB or more is rejected before
+any state moves, so nothing is published and the bytes stay readable — but C++ throws
+`std::length_error` and Python raises `ValueError`.
 
 ## consume() return value
 
@@ -66,9 +99,10 @@ published (`data is None`), matching C++ `operator bool`.
 
 ## loss_count()
 
-C++ counts losses only when `SLICK_STREAM_BUFFER_ENABLE_LOSS_DETECTION` is enabled
-(default: debug builds only). Python always counts. The counter is per-instance in
-both implementations, not shared through the segment.
+Identical semantics on both sides: losses are counted only for reads made with a traits
+whose `count_loss` is true, so `loss_count()` reads 0 under the defaults. Records are still
+skipped correctly either way. The counter is per-instance in both implementations, not
+shared through the segment.
 
 ## Memory orderings
 
@@ -84,10 +118,19 @@ release/acquire — strictly stronger, therefore still correct.
 Python adds explicit lifecycle management that C++ handles via RAII:
 
 - `close()` — detach from the segment (does not delete it)
-- `unlink()` — delete the segment (call once, from the owner, after all users closed)
+- `unlink()` — delete the segment this instance is attached to (call once, from the owner,
+  after all users closed - including this instance, so `close()` then `unlink()` is fine)
 - context manager support: `with SlickStreamBuffer(...) as buf: ...` (closes on exit)
 - `get_shm_name()` — the exact segment name to pass to C++ (`/`-prefixed on POSIX)
 
+`remove(name)` is the one lifecycle call both sides share, with the same contract: unlink a
+segment by name without attaching to it, returning `True` on success and, on Windows, always
+`True` because a section dies with its last handle. Python routes it through
+`_posixshmem.shm_unlink`, then drops the name from `multiprocessing`'s resource tracker if -
+and only if - this process is what registered it. A name registered elsewhere stays that
+owner's to account for: the tracker answers an unregister for a name it never saw by raising
+`KeyError` and printing a traceback from its own process.
+
 On POSIX with Python >= 3.13, attaching to an existing segment uses
 `SharedMemory(track=False)` so Python's resource tracker never unlinks a segment the
 process does not own. On older Pythons the tracker may log a harmless warning at exit

{slick_stream_buffer_py-0.1.0 → slick_stream_buffer_py-2.0.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: slick-stream-buffer-py
-Version: 0.1.0
+Version: 2.0.0
 Summary: Lock-free SPMC byte stream buffer with C++ interoperability via shared memory
 Home-page: https://github.com/SlickQuant/slick-stream-buffer-py
 Author: Slick Quant
@@ -43,6 +43,10 @@ a lock-free single-producer multi-consumer (SPMC) byte stream buffer with shared
 consumer (or vice versa) communicate seamlessly through the same shared memory segment, using
 the same `std::atomic` synchronization primitives.
 
+Tracks C++ `slick/stream_buffer.hpp` **v2.0.0** (header magic `'SSB1'`); the two carry the same
+version number. The shared memory format is unchanged from v1.x, so a segment written by an
+older process is still readable.
+
 ## Features
 
 - **Byte stream, message records**: the producer writes raw bytes (`prepare`/`commit`) and
@@ -51,10 +55,13 @@ the same `std::atomic` synchronization primitives.
 - **Lock-free SPMC**: one producer, any number of consumers, no locks anywhere
 - **Binary-compatible with C++**: identical 64-byte header, 32-byte control records, and
   acquire/release memory orderings (`slick/stream_buffer.hpp`, header magic `'SSB1'`)
+- **Never touches a segment it does not own**: only the process that actually creates a
+  segment initializes it, and an existing one is validated (magic, geometry, mapped size)
+  before a byte is written
 - **Shared memory IPC**: built on `multiprocessing.shared_memory`, matching the C++
   slick-shm naming conventions on Windows, Linux, and macOS
-- **Lossy by design**: slow consumers skip overwritten data and count the loss instead of
-  blocking the producer
+- **Lossy by design**: slow consumers skip overwritten data instead of blocking the producer,
+  and count it when you ask them to (see [Read traits](#read-traits))
 - **Local memory mode**: same API without shared memory for single-process use
 - **No runtime dependencies**: Python 3.8+ standard library plus a small bundled C++
   extension for the atomics
@@ -108,6 +115,33 @@ while True:
     handle_package(data, length)
 ```
 
+### Read traits
+
+`read()` takes a per-call configuration, mirroring the C++ template parameter:
+
+```python
+class read_traits:
+    detect_reset = True    # resync a cursor that outlived a reset()
+    count_loss   = False   # accumulate loss_count()
+```
+
+Ordinary callers write `buf.read(cursor)` and never name a traits class. Loss counting is
+opt-in - in C++ it measured ~20% of `read()` even in a run that lost nothing - so subclass, or
+use the bundled `counting_read_traits`, when you need it:
+
+```python
+from slick_stream_buffer_py import counting_read_traits
+
+data, length, cursor = stream.read(cursor, counting_read_traits)
+print(stream.loss_count())
+```
+
+So `loss_count()` stays 0 unless you read through traits that ask for it - records are still
+skipped correctly, only the counter is silent. `detect_reset` is on by default; turn it off
+if you never call `reset()`.
+
+`ReadTraits` / `CountingReadTraits` are PEP 8 aliases of the same two classes.
+
 ### Local memory mode (single process)
 
 ```python
@@ -143,6 +177,44 @@ Shared memory naming: use `get_shm_name()` to obtain the exact name to pass to C
 (on POSIX it includes the leading `/` that `shm_open()` requires; on Windows it is the
 raw name).
 
+### Shared-memory lifecycle
+
+Only the process that actually creates a segment initializes it - `SharedMemory(create=True)`
+is the OS-backed answer (`O_CREAT|O_EXCL` on POSIX, `ERROR_ALREADY_EXISTS` on Windows), so a
+name that collides with another application's segment is never overwritten. Constructing with
+a geometry over a live segment attaches to it instead: `own_buffer` is then `False` and the
+existing cursors are left alone.
+
+A crashed creator can leave a segment behind. On POSIX the name survives the process, so:
+
+- if it died **part-way through initialization**, the segment is stuck in the `INITIALIZING`
+  state and every later creator and opener fails construction after a 2-second wait;
+- if it died **after initialization**, the segment is intact and gets attached with the dead
+  producer's cursors still in it.
+
+Neither is recovered automatically: nothing portable can distinguish a dead creator from a
+slow one - a PID stamped in the header means nothing across PID namespaces and is unreliable
+under PID reuse - so taking a segment over is an explicit decision. The wedged-segment error
+names the step. Call `remove()` once at startup, when the application knows no other process
+is using the segment:
+
+```python
+SlickStreamBuffer.remove("my_stream")        # clear anything a previous run left
+buf = SlickStreamBuffer(capacity=1 << 20, control_size=4096, name="my_stream")
+```
+
+On POSIX `remove()` unlinks immediately: the next creator gets a fresh segment, while any
+process still mapped to the old one keeps reading the orphaned copy. On Windows it is a no-op
+returning `True` - a section there disappears once the last handle closes.
+
+One Python-only caveat: on POSIX before Python 3.13, `multiprocessing`'s resource tracker
+unlinks every segment the process mapped when it exits, including ones it only attached to.
+Python 3.13+ attaches with `SharedMemory(track=False)`, which is why 3.13 or newer is
+recommended when several processes share a segment. `remove()` unlinks the segment either way,
+and retracts the tracker's registration only when this process is the one that registered it -
+a name registered by some other process, or by a `SharedMemory` you built yourself, stays that
+owner's to account for. See [API_DIFFERENCES.md](API_DIFFERENCES.md).
+
 ### Shared memory layout
 
 ```
@@ -172,12 +244,13 @@ Total segment size: `64 + 32 * control_size + capacity`.
 | `consume(n) -> PublishedRecord` | producer | publish the first n readable bytes as ONE record (clamped) |
 | `discard()` | producer | drop unconsumed + prepared bytes without publishing |
 | `data() -> memoryview` / `size()` | producer | the committed-but-unconsumed region |
-| `read(cursor) -> (data, length, cursor)` | consumer | next record, or `(None, 0, cursor)`; skips lost records |
+| `read(cursor, traits=read_traits) -> (data, length, cursor)` | consumer | next record, or `(None, 0, cursor)`; skips lost records |
 | `read_last() -> (data, length)` | consumer | newest record without a cursor |
 | `initial_reading_index()` | consumer | cursor for a late joiner (skips history) |
-| `loss_count()` | consumer | records skipped by this instance due to overwrite |
+| `loss_count()` | consumer | records skipped by this instance, counting only reads made with `count_loss` traits |
 | `reset()` | owner | clear all state (not thread-safe) |
-| `close()` / `unlink()` | lifecycle | detach / delete the segment |
+| `close()` / `unlink()` | lifecycle | detach / delete this instance's segment |
+| `SlickStreamBuffer.remove(name)` | lifecycle | unlink a segment by name - the stale-segment recovery step |
 
 See [API_DIFFERENCES.md](API_DIFFERENCES.md) for the exact deviations from the C++ API
 (cursor passed by value, bytes copies vs pointers, exceptions).
@@ -187,10 +260,13 @@ See [API_DIFFERENCES.md](API_DIFFERENCES.md) for the exact deviations from the C
 - Producer methods (`prepare`/`commit`/`consume`/`discard`/`data`/`size`/`reset`) must be
   called from a single thread.
 - The buffer is lossy: if the producer outruns a consumer by more than the control ring or
-  data ring size, the consumer skips ahead and the loss is counted.
+  data ring size, the consumer skips ahead. The loss is counted only for reads made with
+  traits whose `count_loss` is true (see [Read traits](#read-traits)).
 - `prepare()` may relocate the readable region: memoryviews previously returned by `data()`
   or `prepare()` are invalidated.
-- A single message (one `consume()` call) is limited to < 4 GiB.
+- A single message (one `consume()` call) is limited to < 4 GiB: `consume()` raises
+  `ValueError` rather than publishing a truncated record length. It raises before touching any
+  state, so the bytes stay readable and can go out as several smaller records.
 
 ## Building and Testing
 
@@ -203,7 +279,7 @@ python tests/test_atomic_ops.py
 python tests/test_local_mode.py
 python tests/test_shm_mode.py
 
-# 3. C++ interop tests (requires CMake + a C++20 compiler)
+# 3. C++ interop tests (requires CMake 3.21+ and a C++20 compiler)
 cmake -S . -B build
 cmake --build build --config Debug
 cd build && ctest -C Debug --output-on-failure
@@ -220,7 +296,14 @@ cmake -S . -B build \
   -DFETCHCONTENT_SOURCE_DIR_SLICK-SHM=/path/to/slick-shm
 ```
 
-If a failed test run leaves segments behind: `python tests/cleanup_shm.py`
+The 4 GiB record-limit test needs a 4 GiB ring, so it is skipped unless you opt in:
+
+```bash
+SSB_PY_TEST_4GIB=1 python tests/test_local_mode.py
+```
+
+If a failed test run leaves segments behind: `python tests/cleanup_shm.py`, or
+`SlickStreamBuffer.remove(name)` for one you know by name.
 
 ## Related Projects
 

{slick_stream_buffer_py-0.1.0 → slick_stream_buffer_py-2.0.0}/README.md

@@ -7,6 +7,10 @@ a lock-free single-producer multi-consumer (SPMC) byte stream buffer with shared
 consumer (or vice versa) communicate seamlessly through the same shared memory segment, using
 the same `std::atomic` synchronization primitives.
 
+Tracks C++ `slick/stream_buffer.hpp` **v2.0.0** (header magic `'SSB1'`); the two carry the same
+version number. The shared memory format is unchanged from v1.x, so a segment written by an
+older process is still readable.
+
 ## Features
 
 - **Byte stream, message records**: the producer writes raw bytes (`prepare`/`commit`) and
@@ -15,10 +19,13 @@ the same `std::atomic` synchronization primitives.
 - **Lock-free SPMC**: one producer, any number of consumers, no locks anywhere
 - **Binary-compatible with C++**: identical 64-byte header, 32-byte control records, and
   acquire/release memory orderings (`slick/stream_buffer.hpp`, header magic `'SSB1'`)
+- **Never touches a segment it does not own**: only the process that actually creates a
+  segment initializes it, and an existing one is validated (magic, geometry, mapped size)
+  before a byte is written
 - **Shared memory IPC**: built on `multiprocessing.shared_memory`, matching the C++
   slick-shm naming conventions on Windows, Linux, and macOS
-- **Lossy by design**: slow consumers skip overwritten data and count the loss instead of
-  blocking the producer
+- **Lossy by design**: slow consumers skip overwritten data instead of blocking the producer,
+  and count it when you ask them to (see [Read traits](#read-traits))
 - **Local memory mode**: same API without shared memory for single-process use
 - **No runtime dependencies**: Python 3.8+ standard library plus a small bundled C++
   extension for the atomics
@@ -72,6 +79,33 @@ while True:
     handle_package(data, length)
 ```
 
+### Read traits
+
+`read()` takes a per-call configuration, mirroring the C++ template parameter:
+
+```python
+class read_traits:
+    detect_reset = True    # resync a cursor that outlived a reset()
+    count_loss   = False   # accumulate loss_count()
+```
+
+Ordinary callers write `buf.read(cursor)` and never name a traits class. Loss counting is
+opt-in - in C++ it measured ~20% of `read()` even in a run that lost nothing - so subclass, or
+use the bundled `counting_read_traits`, when you need it:
+
+```python
+from slick_stream_buffer_py import counting_read_traits
+
+data, length, cursor = stream.read(cursor, counting_read_traits)
+print(stream.loss_count())
+```
+
+So `loss_count()` stays 0 unless you read through traits that ask for it - records are still
+skipped correctly, only the counter is silent. `detect_reset` is on by default; turn it off
+if you never call `reset()`.
+
+`ReadTraits` / `CountingReadTraits` are PEP 8 aliases of the same two classes.
+
 ### Local memory mode (single process)
 
 ```python
@@ -107,6 +141,44 @@ Shared memory naming: use `get_shm_name()` to obtain the exact name to pass to C
 (on POSIX it includes the leading `/` that `shm_open()` requires; on Windows it is the
 raw name).
 
+### Shared-memory lifecycle
+
+Only the process that actually creates a segment initializes it - `SharedMemory(create=True)`
+is the OS-backed answer (`O_CREAT|O_EXCL` on POSIX, `ERROR_ALREADY_EXISTS` on Windows), so a
+name that collides with another application's segment is never overwritten. Constructing with
+a geometry over a live segment attaches to it instead: `own_buffer` is then `False` and the
+existing cursors are left alone.
+
+A crashed creator can leave a segment behind. On POSIX the name survives the process, so:
+
+- if it died **part-way through initialization**, the segment is stuck in the `INITIALIZING`
+  state and every later creator and opener fails construction after a 2-second wait;
+- if it died **after initialization**, the segment is intact and gets attached with the dead
+  producer's cursors still in it.
+
+Neither is recovered automatically: nothing portable can distinguish a dead creator from a
+slow one - a PID stamped in the header means nothing across PID namespaces and is unreliable
+under PID reuse - so taking a segment over is an explicit decision. The wedged-segment error
+names the step. Call `remove()` once at startup, when the application knows no other process
+is using the segment:
+
+```python
+SlickStreamBuffer.remove("my_stream")        # clear anything a previous run left
+buf = SlickStreamBuffer(capacity=1 << 20, control_size=4096, name="my_stream")
+```
+
+On POSIX `remove()` unlinks immediately: the next creator gets a fresh segment, while any
+process still mapped to the old one keeps reading the orphaned copy. On Windows it is a no-op
+returning `True` - a section there disappears once the last handle closes.
+
+One Python-only caveat: on POSIX before Python 3.13, `multiprocessing`'s resource tracker
+unlinks every segment the process mapped when it exits, including ones it only attached to.
+Python 3.13+ attaches with `SharedMemory(track=False)`, which is why 3.13 or newer is
+recommended when several processes share a segment. `remove()` unlinks the segment either way,
+and retracts the tracker's registration only when this process is the one that registered it -
+a name registered by some other process, or by a `SharedMemory` you built yourself, stays that
+owner's to account for. See [API_DIFFERENCES.md](API_DIFFERENCES.md).
+
 ### Shared memory layout
 
 ```
@@ -136,12 +208,13 @@ Total segment size: `64 + 32 * control_size + capacity`.
 | `consume(n) -> PublishedRecord` | producer | publish the first n readable bytes as ONE record (clamped) |
 | `discard()` | producer | drop unconsumed + prepared bytes without publishing |
 | `data() -> memoryview` / `size()` | producer | the committed-but-unconsumed region |
-| `read(cursor) -> (data, length, cursor)` | consumer | next record, or `(None, 0, cursor)`; skips lost records |
+| `read(cursor, traits=read_traits) -> (data, length, cursor)` | consumer | next record, or `(None, 0, cursor)`; skips lost records |
 | `read_last() -> (data, length)` | consumer | newest record without a cursor |
 | `initial_reading_index()` | consumer | cursor for a late joiner (skips history) |
-| `loss_count()` | consumer | records skipped by this instance due to overwrite |
+| `loss_count()` | consumer | records skipped by this instance, counting only reads made with `count_loss` traits |
 | `reset()` | owner | clear all state (not thread-safe) |
-| `close()` / `unlink()` | lifecycle | detach / delete the segment |
+| `close()` / `unlink()` | lifecycle | detach / delete this instance's segment |
+| `SlickStreamBuffer.remove(name)` | lifecycle | unlink a segment by name - the stale-segment recovery step |
 
 See [API_DIFFERENCES.md](API_DIFFERENCES.md) for the exact deviations from the C++ API
 (cursor passed by value, bytes copies vs pointers, exceptions).
@@ -151,10 +224,13 @@ See [API_DIFFERENCES.md](API_DIFFERENCES.md) for the exact deviations from the C
 - Producer methods (`prepare`/`commit`/`consume`/`discard`/`data`/`size`/`reset`) must be
   called from a single thread.
 - The buffer is lossy: if the producer outruns a consumer by more than the control ring or
-  data ring size, the consumer skips ahead and the loss is counted.
+  data ring size, the consumer skips ahead. The loss is counted only for reads made with
+  traits whose `count_loss` is true (see [Read traits](#read-traits)).
 - `prepare()` may relocate the readable region: memoryviews previously returned by `data()`
   or `prepare()` are invalidated.
-- A single message (one `consume()` call) is limited to < 4 GiB.
+- A single message (one `consume()` call) is limited to < 4 GiB: `consume()` raises
+  `ValueError` rather than publishing a truncated record length. It raises before touching any
+  state, so the bytes stay readable and can go out as several smaller records.
 
 ## Building and Testing
 
@@ -167,7 +243,7 @@ python tests/test_atomic_ops.py
 python tests/test_local_mode.py
 python tests/test_shm_mode.py
 
-# 3. C++ interop tests (requires CMake + a C++20 compiler)
+# 3. C++ interop tests (requires CMake 3.21+ and a C++20 compiler)
 cmake -S . -B build
 cmake --build build --config Debug
 cd build && ctest -C Debug --output-on-failure
@@ -184,7 +260,14 @@ cmake -S . -B build \
   -DFETCHCONTENT_SOURCE_DIR_SLICK-SHM=/path/to/slick-shm
 ```
 
-If a failed test run leaves segments behind: `python tests/cleanup_shm.py`
+The 4 GiB record-limit test needs a 4 GiB ring, so it is skipped unless you opt in:
+
+```bash
+SSB_PY_TEST_4GIB=1 python tests/test_local_mode.py
+```
+
+If a failed test run leaves segments behind: `python tests/cleanup_shm.py`, or
+`SlickStreamBuffer.remove(name)` for one you know by name.
 
 ## Related Projects