slick-stream-buffer-multiplexer-py 0.1.0__tar.gz → 2.0.0__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (16) hide show
  1. slick_stream_buffer_multiplexer_py-2.0.0/API_DIFFERENCES.md +146 -0
  2. {slick_stream_buffer_multiplexer_py-0.1.0 → slick_stream_buffer_multiplexer_py-2.0.0}/PKG-INFO +127 -11
  3. {slick_stream_buffer_multiplexer_py-0.1.0 → slick_stream_buffer_multiplexer_py-2.0.0}/README.md +124 -8
  4. {slick_stream_buffer_multiplexer_py-0.1.0 → slick_stream_buffer_multiplexer_py-2.0.0}/pyproject.toml +2 -2
  5. {slick_stream_buffer_multiplexer_py-0.1.0 → slick_stream_buffer_multiplexer_py-2.0.0}/requirements.txt +2 -2
  6. {slick_stream_buffer_multiplexer_py-0.1.0 → slick_stream_buffer_multiplexer_py-2.0.0}/setup.py +2 -2
  7. {slick_stream_buffer_multiplexer_py-0.1.0 → slick_stream_buffer_multiplexer_py-2.0.0}/slick_stream_buffer_multiplexer_py.py +235 -53
  8. {slick_stream_buffer_multiplexer_py-0.1.0 → slick_stream_buffer_multiplexer_py-2.0.0}/tests/test_interop.py +106 -1
  9. {slick_stream_buffer_multiplexer_py-0.1.0 → slick_stream_buffer_multiplexer_py-2.0.0}/tests/test_multiplexer.py +120 -0
  10. {slick_stream_buffer_multiplexer_py-0.1.0 → slick_stream_buffer_multiplexer_py-2.0.0}/tests/test_multiplexer_mpmc.py +80 -1
  11. {slick_stream_buffer_multiplexer_py-0.1.0 → slick_stream_buffer_multiplexer_py-2.0.0}/tests/test_multiplexer_shm.py +47 -11
  12. slick_stream_buffer_multiplexer_py-0.1.0/API_DIFFERENCES.md +0 -78
  13. {slick_stream_buffer_multiplexer_py-0.1.0 → slick_stream_buffer_multiplexer_py-2.0.0}/LICENSE +0 -0
  14. {slick_stream_buffer_multiplexer_py-0.1.0 → slick_stream_buffer_multiplexer_py-2.0.0}/MANIFEST.in +0 -0
  15. {slick_stream_buffer_multiplexer_py-0.1.0 → slick_stream_buffer_multiplexer_py-2.0.0}/setup.cfg +0 -0
  16. {slick_stream_buffer_multiplexer_py-0.1.0 → slick_stream_buffer_multiplexer_py-2.0.0}/slick_stream_buffer_multiplexer_py.egg-info/SOURCES.txt +0 -0
@@ -0,0 +1,146 @@
1
+ # API Differences: Python vs C++
2
+
3
+ The Python implementation is binary-compatible with the C++
4
+ `slick::stream_buffer_multiplexer` at the shared memory level — the shared record queue
5
+ ('SLQ1', or 'SLQ0' with `enable_read_last` off; 16-byte elements) and every producer segment
6
+ ('SSB1') match byte-for-byte. Peers must agree on `enable_read_last`, the one trait recorded in
7
+ the queue's header (see [Traits](#traits-are-a-constructor-argument-not-a-template-parameter)).
8
+ The *language-level* API differs in the following ways.
9
+
10
+ ## Construction
11
+
12
+ | C++ | Python |
13
+ |---|---|
14
+ | `stream_buffer_multiplexer mux(size)` | `StreamBufferMultiplexer(shared_queue_size=size)` |
15
+ | `stream_buffer_multiplexer mux(size, "name")` | `StreamBufferMultiplexer(shared_queue_size=size, name="name")` |
16
+ | `stream_buffer_multiplexer mux("name")` | `StreamBufferMultiplexer(name="name")` |
17
+ | `add_producer(id, cap, ctl)` | `add_producer(id, capacity=cap, control_size=ctl)` |
18
+ | `add_producer(id, cap, ctl, "name")` | `add_producer(id, capacity=cap, control_size=ctl, name="name")` |
19
+ | `add_producer(id, "name")` | `add_producer(id, name="name")` |
20
+ | throws `std::invalid_argument` (duplicate id, bad sizes) | raises `ValueError` |
21
+ | `uint32_t producer_id` (range enforced by the type) | raises `ValueError` outside `0..0xFFFFFFFF`, `TypeError` for a non-integer |
22
+ | throws `std::runtime_error` (shm failures) | raises `RuntimeError`, `ValueError` (queue geometry mismatch), or `FileNotFoundError` (missing segment) |
23
+
24
+ ## Consumer cursor is passed by value
25
+
26
+ ```cpp
27
+ // C++
28
+ uint64_t cursor = 0;
29
+ auto rec = mux.read(cursor); // cursor updated in place
30
+ ```
31
+
32
+ ```python
33
+ # Python
34
+ cursor = 0
35
+ rec, cursor = mux.read(cursor) # updated cursor returned
36
+ ```
37
+
38
+ For the work-stealing overload, pass an `AtomicCursor` (from slick-queue-py); the second
39
+ return value is then the claimed shared-queue index (`-1` when nothing was available):
40
+
41
+ ```python
42
+ rec, idx = mux.read(shared_cursor)
43
+ ```
44
+
45
+ ## Zero-copy vs copies
46
+
47
+ Same policy as slick-stream-buffer-py: `prepare()` and `data()` return zero-copy
48
+ memoryviews into the producer's ring; `read()` results (`MultiplexRecord.data`) and
49
+ `consume()` results are **bytes copies**, safe by default under the lossy overwrite
50
+ semantics.
51
+
52
+ ## Producer buffer accessors
53
+
54
+ C++ exposes both `stream_buffer()` (reference) and `stream_buffer_ptr()`
55
+ (`std::shared_ptr`); Python has no owning/non-owning distinction, so there is a single
56
+ `producer.stream_buffer` property returning the underlying `SlickStreamBuffer`.
57
+ Similarly, `get_producer_buffer()` and `find_producer()` both return the same
58
+ `ProducerBuffer` object (or `None`).
59
+
60
+ ## Producer lookup
61
+
62
+ C++ uses a dense vector for producer ids < 4096 with a hash-map fallback; Python uses a
63
+ plain dict for all ids (already O(1) — the dense-path micro-optimization is meaningless
64
+ in Python).
65
+
66
+ `get_producer_buffers()` returns a `const` map reference in C++; Python returns a cached
67
+ read-only `types.MappingProxyType` over the same dict — mutation raises, later
68
+ `add_producer()` calls still show through, and the call allocates nothing. Iteration order
69
+ is insertion order, which C++'s `unordered_map` does not promise.
70
+
71
+ ## Traits are a constructor argument, not a template parameter
72
+
73
+ C++ spells the configuration `basic_stream_buffer_multiplexer<Traits = default_queue_traits>`;
74
+ Python passes the same `QueueTraits` class as a `traits=` keyword argument, matching how
75
+ slick-queue-py does it:
76
+
77
+ ```cpp
78
+ // C++
79
+ slick::basic_stream_buffer_multiplexer<slick::debug_queue_traits> mux(1024);
80
+ ```
81
+
82
+ ```python
83
+ # Python
84
+ mux = StreamBufferMultiplexer(shared_queue_size=1024, traits=MyTraits)
85
+ ```
86
+
87
+ `mux.traits` returns the shared queue's frozen snapshot (C++ spells this
88
+ `using traits = Traits`). One behavioural difference, inherited from slick-queue-py:
89
+ C++'s `default_queue_traits` follows `NDEBUG`, so loss counting is off in release builds;
90
+ Python has no debug/release split and defaults `enable_loss_detection` to on.
91
+
92
+ That difference is harmless across processes because only one trait reaches shared memory:
93
+ `enable_read_last`. The queue's creator records it in the layout marker (`'SLQ1'` on, `'SLQ0'`
94
+ off) and every attacher in either language must be configured the same way, or construction fails
95
+ with "Shared memory feature mismatch" (`RuntimeError` / `std::runtime_error`). Both sides default
96
+ it to on; a peer that turns it off needs its counterparts to do the same:
97
+
98
+ ```cpp
99
+ // C++
100
+ struct no_read_last_traits : slick::default_queue_traits {
101
+ static constexpr bool enable_read_last = false;
102
+ };
103
+ slick::basic_stream_buffer_multiplexer<no_read_last_traits> mux("mux_records");
104
+ ```
105
+
106
+ ```python
107
+ # Python
108
+ class NoReadLast(QueueTraits):
109
+ enable_read_last = False
110
+
111
+ mux = StreamBufferMultiplexer(name="mux_records", traits=NoReadLast)
112
+ ```
113
+
114
+ ## loss_count()
115
+
116
+ Same semantics on both sides: `traits.enable_loss_detection` switches the shared queue's
117
+ wrap-loss term and the multiplexer-level term together, and both read 0 when it is off. Only
118
+ the multiplexer-level term is filtered by producer registration. Records are skipped correctly
119
+ either way. All counters are per-instance, not shared through the segments.
120
+
121
+ `ProducerBuffer.loss_count()` is likewise identical: the multiplexer dereferences through
122
+ traits with `count_loss` off, so that counter only moves for reads the caller makes directly
123
+ through `producer.stream_buffer.read(cursor, traits)`.
124
+
125
+ ## producer_buffer is not movable in C++
126
+
127
+ C++ deletes `producer_buffer`'s copy and move operations, because the multiplexer keeps a raw
128
+ pointer to the same object its `shared_ptr` names, and moving one out would leave both pointing
129
+ at an emptied object. Python has no moves and rebinding a name copies a reference, so there is
130
+ nothing to delete — `ProducerBuffer` has no equivalent restriction.
131
+
132
+ ## Lifecycle
133
+
134
+ Python adds explicit lifecycle management that C++ handles via RAII:
135
+
136
+ - `close()` — detach from all producer segments and the queue segment; the registration table is
137
+ kept, so a later `unlink()` still reaches every producer
138
+ - `unlink()` — delete the queue segment and all registered producers' segments; valid before or
139
+ after `close()` (close every handle, then unlink once), and a no-op when repeated
140
+ - context manager support: `with StreamBufferMultiplexer(...) as mux: ...`
141
+ - `get_shm_name()` — on the multiplexer (queue segment) and on each `ProducerBuffer`,
142
+ the exact names to pass to C++ (`/`-prefixed on POSIX)
143
+
144
+ `remove(name)` is the one lifecycle call both sides share, with the same contract: unlink a
145
+ segment by name without attaching to it, returning `True` on success and, on Windows, always
146
+ `True` because a section dies with its last handle.
@@ -1,6 +1,6 @@
1
1
  Metadata-Version: 2.4
2
2
  Name: slick-stream-buffer-multiplexer-py
3
- Version: 0.1.0
3
+ Version: 2.0.0
4
4
  Summary: Lock-free MPMC byte stream multiplexer with C++ interoperability via shared memory
5
5
  Home-page: https://github.com/SlickQuant/slick-stream-buffer-multiplexer-py
6
6
  Author: Slick Quant
@@ -28,8 +28,8 @@ Classifier: Operating System :: MacOS
28
28
  Requires-Python: >=3.8
29
29
  Description-Content-Type: text/markdown
30
30
  License-File: LICENSE
31
- Requires-Dist: slick-queue-py>=1.2.0
32
- Requires-Dist: slick-stream-buffer-py>=0.1.0
31
+ Requires-Dist: slick-queue-py>=2.0.0
32
+ Requires-Dist: slick-stream-buffer-py>=2.0.0
33
33
  Dynamic: author
34
34
  Dynamic: home-page
35
35
  Dynamic: license-file
@@ -41,7 +41,14 @@ A Python implementation of [slick-stream-buffer-multiplexer](https://github.com/
41
41
  a lock-free multi-producer multi-consumer (MPMC) byte stream multiplexer with shared memory support.
42
42
 
43
43
  **Maintains exact binary compatibility with the C++ version**: Python and C++ producers and
44
- consumers can be mixed freely across processes, fanning into the same shared memory segments.
44
+ consumers can be mixed across processes, fanning into the same shared memory segments — provided
45
+ every peer of a shared queue agrees on `enable_read_last` (see
46
+ [`enable_read_last` must match on every peer](#enable_read_last-must-match-on-every-peer)).
47
+
48
+ Tracks C++ `slick/stream_buffer_multiplexer.hpp` **v2.0.0**, and requires slick-queue-py >=
49
+ 2.0.0 and slick-stream-buffer-py >= 2.0.0. The shared-memory formats are unchanged
50
+ (`'SLQ1'` / `'SSB1'`), so segments written by v0.1.0 or by a C++ v1.x process are still
51
+ readable with the default traits.
45
52
 
46
53
  ## Concept
47
54
 
@@ -70,8 +77,8 @@ records referencing other producer ids are silently skipped (not counted as loss
70
77
  ## Requirements
71
78
 
72
79
  - Python 3.8+
73
- - [slick-queue-py](https://github.com/SlickQuant/slick-queue-py) >= 1.2.0
74
- - [slick-stream-buffer-py](https://github.com/SlickQuant/slick-stream-buffer-py) >= 0.1.0
80
+ - [slick-queue-py](https://github.com/SlickQuant/slick-queue-py) >= 2.0.0
81
+ - [slick-stream-buffer-py](https://github.com/SlickQuant/slick-stream-buffer-py) >= 2.0.0
75
82
 
76
83
  Both dependencies bundle a small C++ extension for `std::atomic` operations; this package itself
77
84
  is pure Python.
@@ -130,14 +137,116 @@ shared_cursor = AtomicCursor(cursor_shm.buf, 0)
130
137
  rec, idx = mux.read(shared_cursor) # atomically claims the next message
131
138
  ```
132
139
 
140
+ ## Traits
141
+
142
+ Feature configuration is a `traits` constructor argument — and it is slick-queue-py's
143
+ `QueueTraits`, the traits of the shared record queue the multiplexer is built on, not a traits
144
+ type of its own:
145
+
146
+ ```python
147
+ from slick_stream_buffer_multiplexer_py import StreamBufferMultiplexer, QueueTraits
148
+
149
+ class Silent(QueueTraits):
150
+ enable_loss_detection = False # both loss terms go quiet together
151
+
152
+ mux = StreamBufferMultiplexer(shared_queue_size=1024, traits=Silent)
153
+ ```
154
+
155
+ Every knob the multiplexer has beyond that queue is a passthrough, and its one real tunable
156
+ asks the same question `enable_loss_detection` already asks — so a separate flag would only let
157
+ the two disagree. That matters because `loss_count()` **sums** the two counters: a
158
+ half-configured pair would return a partial total that reads like a complete one. One flag, one
159
+ number. `mux.traits` is the queue's own frozen snapshot, not a second copy.
160
+
161
+ Unlike C++ — where `default_queue_traits` follows `NDEBUG` — slick-queue-py defaults
162
+ `enable_loss_detection` to **on**, so the counters work out of the box.
163
+
164
+ ### `enable_read_last` must match on every peer
165
+
166
+ The other traits are local to one instance and may differ freely between processes, but
167
+ `enable_read_last` is part of the shared record queue's header protocol. The queue's creator
168
+ stamps it into the segment's layout marker — `'SLQ1'` when the last-published index is maintained,
169
+ `'SLQ0'` when it is not — and every attacher, Python or C++, checks the marker against its own
170
+ traits. A peer that disagrees is refused at construction (`RuntimeError` in Python,
171
+ `std::runtime_error` in C++, both reading *"Shared memory feature mismatch"*) rather than
172
+ misreading the header: a Python queue created with `enable_read_last = False` cannot be opened by a
173
+ C++ `slick::stream_buffer_multiplexer`, whose default traits have it on.
174
+
175
+ Both languages default it to on, so only turning it off needs coordinating — do it identically on
176
+ every peer of the queue:
177
+
178
+ ```python
179
+ class NoReadLast(QueueTraits):
180
+ enable_read_last = False
181
+
182
+ mux = StreamBufferMultiplexer(shared_queue_size=1 << 16, name="mux_records", traits=NoReadLast)
183
+ ```
184
+
185
+ ```cpp
186
+ struct no_read_last_traits : slick::default_queue_traits {
187
+ static constexpr bool enable_read_last = false;
188
+ };
189
+ slick::basic_stream_buffer_multiplexer<no_read_last_traits> mux("mux_records");
190
+ ```
191
+
192
+ The multiplexer itself never calls `read_last()` (`initial_reading_index()` reads the reservation
193
+ counter instead), so turning it off only drops one CAS from each `consume()`'s publish into the
194
+ shared queue. Producer segments (`'SSB1'`) carry no such flag.
195
+
133
196
  ## Three Loss Counters
134
197
 
135
198
  | Counter | Meaning |
136
199
  |---|---|
137
- | `mux.loss_count()` | shared-queue wrap loss **plus** multiplexer-level loss (registered producers lapped before dereference) |
138
- | `producer.loss_count()` | that producer's own inner-ring loss |
200
+ | `mux.loss_count()` | shared-queue wrap loss **plus** multiplexer-level loss (registered producers lapped before dereference); both terms switched by `traits.enable_loss_detection` |
201
+ | `producer.loss_count()` | that producer's own inner-ring loss — **only** for reads the caller makes through `producer.stream_buffer.read(cursor, traits)` with counting traits |
139
202
  | unregistered producer ids | **never counted** — silently skipped |
140
203
 
204
+ Only the multiplexer-level term is filtered by registration. The wrap-loss term counts entries
205
+ the shared queue dropped by wrapping, and a lapped slot has already been overwritten, so nothing
206
+ is left to say which producer it named — an instance registering a subset of producers therefore
207
+ sees wrap loss for producers it ignores, making the total an upper bound on what it missed. Size
208
+ the shared queue so it does not wrap and that term goes to 0.
209
+
210
+ `producer.loss_count()` stays at 0 under multiplexer traffic by design: the multiplexer
211
+ dereferences through traits with `count_loss` off, because `read()`'s counter measures how far
212
+ a *sequential scan* jumped, which for a jump-read is a distance every later lapped dereference
213
+ would add all over again. The multiplexer counts one loss per lapped record instead.
214
+
215
+ `mux.get_producer_buffers()` returns every registered producer keyed by `producer_id` — a
216
+ read-only live view of the registration table, cheap enough for a monitoring loop:
217
+
218
+ ```python
219
+ inner_loss = sum(p.loss_count() for p in mux.get_producer_buffers().values())
220
+ ```
221
+
222
+ ## Shutdown
223
+
224
+ Close every handle, then unlink once — the usual `shared_memory` idiom:
225
+
226
+ ```python
227
+ mux.close() # or leave a `with StreamBufferMultiplexer(...) as mux:` block
228
+ mux.unlink() # deletes the record queue and every registered producer's segment
229
+ ```
230
+
231
+ `close()` keeps the registration table so `unlink()` can still reach each producer's segment, and
232
+ `unlink()` runs once: repeating it never deletes a segment a later run has created under the same
233
+ name. On Windows `unlink()` is a no-op — a section disappears once its last handle closes.
234
+
235
+ ## Stale-segment recovery
236
+
237
+ `StreamBufferMultiplexer.remove(name)` unlinks a segment by name — it works for both kinds this
238
+ class creates, a shared record queue's and a producer stream buffer's, since both are plain
239
+ shared memory segments underneath. Use it when a previous run died mid-initialization and left a
240
+ segment wedged, which construction over that name reports by raising:
241
+
242
+ ```python
243
+ StreamBufferMultiplexer.remove("md_records") # clear what a dead run left
244
+ mux = StreamBufferMultiplexer(shared_queue_size=1024, name="md_records")
245
+ ```
246
+
247
+ Only call it when no process is using the segment. On Windows it is a no-op returning `True` —
248
+ a section there disappears once the last handle closes.
249
+
141
250
  ## C++ Interoperability
142
251
 
143
252
  The C++ side composes the identical primitives — either language can create any segment:
@@ -163,16 +272,23 @@ exact names to pass to C++ (POSIX names include the required leading `/`).
163
272
 
164
273
  The multiplexer adds no shared-memory structures of its own:
165
274
 
166
- - The shared queue is a standard slick-queue segment (`'SLQ1'`, `element_size=16`) whose
167
- elements are `uint64 sequence | uint32 producer_id | uint32 pad0`
275
+ - The shared queue is a standard slick-queue segment (`'SLQ1'`, or `'SLQ0'` when created with
276
+ `enable_read_last` off; `element_size=16`) whose elements are
277
+ `uint64 sequence | uint32 producer_id | uint32 pad0`
168
278
  - Each producer buffer is a standard slick-stream-buffer segment (`'SSB1'`)
169
279
 
170
280
  ## Caveats (same as C++)
171
281
 
172
282
  - `add_producer()` is single-threaded setup: call it before producer/consumer threads start.
283
+ - A `producer_id` must fit the record's `uint32` field: `add_producer()` raises `ValueError`
284
+ for one outside `0..0xFFFFFFFF` (and `TypeError` for a non-integer), before any segment is
285
+ created.
173
286
  - Each producer's `prepare/commit/consume/...` must be called from a single thread.
174
287
  - Lossy semantics: slow consumers skip overwritten data (see the loss counters above).
175
- - A single message (one `consume()` call) is limited to < 4 GiB.
288
+ Records are skipped correctly whatever the traits say; only the counters go quiet.
289
+ - A single message (one `consume()` call) is limited to < 4 GiB: `consume()` raises
290
+ `ValueError` rather than truncating one that is not, before any state moves — so nothing is
291
+ published to either the producer's ring or the shared record queue.
176
292
 
177
293
  See [API_DIFFERENCES.md](API_DIFFERENCES.md) for exact deviations from the C++ API.
178
294
 
@@ -4,7 +4,14 @@ A Python implementation of [slick-stream-buffer-multiplexer](https://github.com/
4
4
  a lock-free multi-producer multi-consumer (MPMC) byte stream multiplexer with shared memory support.
5
5
 
6
6
  **Maintains exact binary compatibility with the C++ version**: Python and C++ producers and
7
- consumers can be mixed freely across processes, fanning into the same shared memory segments.
7
+ consumers can be mixed across processes, fanning into the same shared memory segments — provided
8
+ every peer of a shared queue agrees on `enable_read_last` (see
9
+ [`enable_read_last` must match on every peer](#enable_read_last-must-match-on-every-peer)).
10
+
11
+ Tracks C++ `slick/stream_buffer_multiplexer.hpp` **v2.0.0**, and requires slick-queue-py >=
12
+ 2.0.0 and slick-stream-buffer-py >= 2.0.0. The shared-memory formats are unchanged
13
+ (`'SLQ1'` / `'SSB1'`), so segments written by v0.1.0 or by a C++ v1.x process are still
14
+ readable with the default traits.
8
15
 
9
16
  ## Concept
10
17
 
@@ -33,8 +40,8 @@ records referencing other producer ids are silently skipped (not counted as loss
33
40
  ## Requirements
34
41
 
35
42
  - Python 3.8+
36
- - [slick-queue-py](https://github.com/SlickQuant/slick-queue-py) >= 1.2.0
37
- - [slick-stream-buffer-py](https://github.com/SlickQuant/slick-stream-buffer-py) >= 0.1.0
43
+ - [slick-queue-py](https://github.com/SlickQuant/slick-queue-py) >= 2.0.0
44
+ - [slick-stream-buffer-py](https://github.com/SlickQuant/slick-stream-buffer-py) >= 2.0.0
38
45
 
39
46
  Both dependencies bundle a small C++ extension for `std::atomic` operations; this package itself
40
47
  is pure Python.
@@ -93,14 +100,116 @@ shared_cursor = AtomicCursor(cursor_shm.buf, 0)
93
100
  rec, idx = mux.read(shared_cursor) # atomically claims the next message
94
101
  ```
95
102
 
103
+ ## Traits
104
+
105
+ Feature configuration is a `traits` constructor argument — and it is slick-queue-py's
106
+ `QueueTraits`, the traits of the shared record queue the multiplexer is built on, not a traits
107
+ type of its own:
108
+
109
+ ```python
110
+ from slick_stream_buffer_multiplexer_py import StreamBufferMultiplexer, QueueTraits
111
+
112
+ class Silent(QueueTraits):
113
+ enable_loss_detection = False # both loss terms go quiet together
114
+
115
+ mux = StreamBufferMultiplexer(shared_queue_size=1024, traits=Silent)
116
+ ```
117
+
118
+ Every knob the multiplexer has beyond that queue is a passthrough, and its one real tunable
119
+ asks the same question `enable_loss_detection` already asks — so a separate flag would only let
120
+ the two disagree. That matters because `loss_count()` **sums** the two counters: a
121
+ half-configured pair would return a partial total that reads like a complete one. One flag, one
122
+ number. `mux.traits` is the queue's own frozen snapshot, not a second copy.
123
+
124
+ Unlike C++ — where `default_queue_traits` follows `NDEBUG` — slick-queue-py defaults
125
+ `enable_loss_detection` to **on**, so the counters work out of the box.
126
+
127
+ ### `enable_read_last` must match on every peer
128
+
129
+ The other traits are local to one instance and may differ freely between processes, but
130
+ `enable_read_last` is part of the shared record queue's header protocol. The queue's creator
131
+ stamps it into the segment's layout marker — `'SLQ1'` when the last-published index is maintained,
132
+ `'SLQ0'` when it is not — and every attacher, Python or C++, checks the marker against its own
133
+ traits. A peer that disagrees is refused at construction (`RuntimeError` in Python,
134
+ `std::runtime_error` in C++, both reading *"Shared memory feature mismatch"*) rather than
135
+ misreading the header: a Python queue created with `enable_read_last = False` cannot be opened by a
136
+ C++ `slick::stream_buffer_multiplexer`, whose default traits have it on.
137
+
138
+ Both languages default it to on, so only turning it off needs coordinating — do it identically on
139
+ every peer of the queue:
140
+
141
+ ```python
142
+ class NoReadLast(QueueTraits):
143
+ enable_read_last = False
144
+
145
+ mux = StreamBufferMultiplexer(shared_queue_size=1 << 16, name="mux_records", traits=NoReadLast)
146
+ ```
147
+
148
+ ```cpp
149
+ struct no_read_last_traits : slick::default_queue_traits {
150
+ static constexpr bool enable_read_last = false;
151
+ };
152
+ slick::basic_stream_buffer_multiplexer<no_read_last_traits> mux("mux_records");
153
+ ```
154
+
155
+ The multiplexer itself never calls `read_last()` (`initial_reading_index()` reads the reservation
156
+ counter instead), so turning it off only drops one CAS from each `consume()`'s publish into the
157
+ shared queue. Producer segments (`'SSB1'`) carry no such flag.
158
+
96
159
  ## Three Loss Counters
97
160
 
98
161
  | Counter | Meaning |
99
162
  |---|---|
100
- | `mux.loss_count()` | shared-queue wrap loss **plus** multiplexer-level loss (registered producers lapped before dereference) |
101
- | `producer.loss_count()` | that producer's own inner-ring loss |
163
+ | `mux.loss_count()` | shared-queue wrap loss **plus** multiplexer-level loss (registered producers lapped before dereference); both terms switched by `traits.enable_loss_detection` |
164
+ | `producer.loss_count()` | that producer's own inner-ring loss — **only** for reads the caller makes through `producer.stream_buffer.read(cursor, traits)` with counting traits |
102
165
  | unregistered producer ids | **never counted** — silently skipped |
103
166
 
167
+ Only the multiplexer-level term is filtered by registration. The wrap-loss term counts entries
168
+ the shared queue dropped by wrapping, and a lapped slot has already been overwritten, so nothing
169
+ is left to say which producer it named — an instance registering a subset of producers therefore
170
+ sees wrap loss for producers it ignores, making the total an upper bound on what it missed. Size
171
+ the shared queue so it does not wrap and that term goes to 0.
172
+
173
+ `producer.loss_count()` stays at 0 under multiplexer traffic by design: the multiplexer
174
+ dereferences through traits with `count_loss` off, because `read()`'s counter measures how far
175
+ a *sequential scan* jumped, which for a jump-read is a distance every later lapped dereference
176
+ would add all over again. The multiplexer counts one loss per lapped record instead.
177
+
178
+ `mux.get_producer_buffers()` returns every registered producer keyed by `producer_id` — a
179
+ read-only live view of the registration table, cheap enough for a monitoring loop:
180
+
181
+ ```python
182
+ inner_loss = sum(p.loss_count() for p in mux.get_producer_buffers().values())
183
+ ```
184
+
185
+ ## Shutdown
186
+
187
+ Close every handle, then unlink once — the usual `shared_memory` idiom:
188
+
189
+ ```python
190
+ mux.close() # or leave a `with StreamBufferMultiplexer(...) as mux:` block
191
+ mux.unlink() # deletes the record queue and every registered producer's segment
192
+ ```
193
+
194
+ `close()` keeps the registration table so `unlink()` can still reach each producer's segment, and
195
+ `unlink()` runs once: repeating it never deletes a segment a later run has created under the same
196
+ name. On Windows `unlink()` is a no-op — a section disappears once its last handle closes.
197
+
198
+ ## Stale-segment recovery
199
+
200
+ `StreamBufferMultiplexer.remove(name)` unlinks a segment by name — it works for both kinds this
201
+ class creates, a shared record queue's and a producer stream buffer's, since both are plain
202
+ shared memory segments underneath. Use it when a previous run died mid-initialization and left a
203
+ segment wedged, which construction over that name reports by raising:
204
+
205
+ ```python
206
+ StreamBufferMultiplexer.remove("md_records") # clear what a dead run left
207
+ mux = StreamBufferMultiplexer(shared_queue_size=1024, name="md_records")
208
+ ```
209
+
210
+ Only call it when no process is using the segment. On Windows it is a no-op returning `True` —
211
+ a section there disappears once the last handle closes.
212
+
104
213
  ## C++ Interoperability
105
214
 
106
215
  The C++ side composes the identical primitives — either language can create any segment:
@@ -126,16 +235,23 @@ exact names to pass to C++ (POSIX names include the required leading `/`).
126
235
 
127
236
  The multiplexer adds no shared-memory structures of its own:
128
237
 
129
- - The shared queue is a standard slick-queue segment (`'SLQ1'`, `element_size=16`) whose
130
- elements are `uint64 sequence | uint32 producer_id | uint32 pad0`
238
+ - The shared queue is a standard slick-queue segment (`'SLQ1'`, or `'SLQ0'` when created with
239
+ `enable_read_last` off; `element_size=16`) whose elements are
240
+ `uint64 sequence | uint32 producer_id | uint32 pad0`
131
241
  - Each producer buffer is a standard slick-stream-buffer segment (`'SSB1'`)
132
242
 
133
243
  ## Caveats (same as C++)
134
244
 
135
245
  - `add_producer()` is single-threaded setup: call it before producer/consumer threads start.
246
+ - A `producer_id` must fit the record's `uint32` field: `add_producer()` raises `ValueError`
247
+ for one outside `0..0xFFFFFFFF` (and `TypeError` for a non-integer), before any segment is
248
+ created.
136
249
  - Each producer's `prepare/commit/consume/...` must be called from a single thread.
137
250
  - Lossy semantics: slow consumers skip overwritten data (see the loss counters above).
138
- - A single message (one `consume()` call) is limited to < 4 GiB.
251
+ Records are skipped correctly whatever the traits say; only the counters go quiet.
252
+ - A single message (one `consume()` call) is limited to < 4 GiB: `consume()` raises
253
+ `ValueError` rather than truncating one that is not, before any state moves — so nothing is
254
+ published to either the producer's ring or the shared record queue.
139
255
 
140
256
  See [API_DIFFERENCES.md](API_DIFFERENCES.md) for exact deviations from the C++ API.
141
257
 
@@ -30,8 +30,8 @@ classifiers = [
30
30
  "Operating System :: MacOS",
31
31
  ]
32
32
  dependencies = [
33
- "slick-queue-py>=1.2.0",
34
- "slick-stream-buffer-py>=0.1.0",
33
+ "slick-queue-py>=2.0.0",
34
+ "slick-stream-buffer-py>=2.0.0",
35
35
  ]
36
36
 
37
37
  [project.urls]
@@ -1,3 +1,3 @@
1
1
  # Runtime dependencies (binary-compatible Python ports of the C++ dependencies)
2
- slick-queue-py>=1.2.0
3
- slick-stream-buffer-py>=0.1.0
2
+ slick-queue-py>=2.0.0
3
+ slick-stream-buffer-py>=2.0.0
@@ -52,8 +52,8 @@ setup(
52
52
  keywords='stream-buffer, multiplexer, lock-free, atomic, shared-memory, ipc, multiprocessing, mpmc',
53
53
  py_modules=['slick_stream_buffer_multiplexer_py'],
54
54
  install_requires=[
55
- 'slick-queue-py>=1.2.0',
56
- 'slick-stream-buffer-py>=0.1.0',
55
+ 'slick-queue-py>=2.0.0',
56
+ 'slick-stream-buffer-py>=2.0.0',
57
57
  ],
58
58
  python_requires='>=3.8',
59
59
  license='MIT',