slick-stream-buffer-multiplexer-py 0.1.0__tar.gz → 2.0.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- slick_stream_buffer_multiplexer_py-2.0.0/API_DIFFERENCES.md +146 -0
- {slick_stream_buffer_multiplexer_py-0.1.0 → slick_stream_buffer_multiplexer_py-2.0.0}/PKG-INFO +127 -11
- {slick_stream_buffer_multiplexer_py-0.1.0 → slick_stream_buffer_multiplexer_py-2.0.0}/README.md +124 -8
- {slick_stream_buffer_multiplexer_py-0.1.0 → slick_stream_buffer_multiplexer_py-2.0.0}/pyproject.toml +2 -2
- {slick_stream_buffer_multiplexer_py-0.1.0 → slick_stream_buffer_multiplexer_py-2.0.0}/requirements.txt +2 -2
- {slick_stream_buffer_multiplexer_py-0.1.0 → slick_stream_buffer_multiplexer_py-2.0.0}/setup.py +2 -2
- {slick_stream_buffer_multiplexer_py-0.1.0 → slick_stream_buffer_multiplexer_py-2.0.0}/slick_stream_buffer_multiplexer_py.py +235 -53
- {slick_stream_buffer_multiplexer_py-0.1.0 → slick_stream_buffer_multiplexer_py-2.0.0}/tests/test_interop.py +106 -1
- {slick_stream_buffer_multiplexer_py-0.1.0 → slick_stream_buffer_multiplexer_py-2.0.0}/tests/test_multiplexer.py +120 -0
- {slick_stream_buffer_multiplexer_py-0.1.0 → slick_stream_buffer_multiplexer_py-2.0.0}/tests/test_multiplexer_mpmc.py +80 -1
- {slick_stream_buffer_multiplexer_py-0.1.0 → slick_stream_buffer_multiplexer_py-2.0.0}/tests/test_multiplexer_shm.py +47 -11
- slick_stream_buffer_multiplexer_py-0.1.0/API_DIFFERENCES.md +0 -78
- {slick_stream_buffer_multiplexer_py-0.1.0 → slick_stream_buffer_multiplexer_py-2.0.0}/LICENSE +0 -0
- {slick_stream_buffer_multiplexer_py-0.1.0 → slick_stream_buffer_multiplexer_py-2.0.0}/MANIFEST.in +0 -0
- {slick_stream_buffer_multiplexer_py-0.1.0 → slick_stream_buffer_multiplexer_py-2.0.0}/setup.cfg +0 -0
- {slick_stream_buffer_multiplexer_py-0.1.0 → slick_stream_buffer_multiplexer_py-2.0.0}/slick_stream_buffer_multiplexer_py.egg-info/SOURCES.txt +0 -0
|
@@ -0,0 +1,146 @@
|
|
|
1
|
+
# API Differences: Python vs C++
|
|
2
|
+
|
|
3
|
+
The Python implementation is binary-compatible with the C++
|
|
4
|
+
`slick::stream_buffer_multiplexer` at the shared memory level — the shared record queue
|
|
5
|
+
('SLQ1', or 'SLQ0' with `enable_read_last` off; 16-byte elements) and every producer segment
|
|
6
|
+
('SSB1') match byte-for-byte. Peers must agree on `enable_read_last`, the one trait recorded in
|
|
7
|
+
the queue's header (see [Traits](#traits-are-a-constructor-argument-not-a-template-parameter)).
|
|
8
|
+
The *language-level* API differs in the following ways.
|
|
9
|
+
|
|
10
|
+
## Construction
|
|
11
|
+
|
|
12
|
+
| C++ | Python |
|
|
13
|
+
|---|---|
|
|
14
|
+
| `stream_buffer_multiplexer mux(size)` | `StreamBufferMultiplexer(shared_queue_size=size)` |
|
|
15
|
+
| `stream_buffer_multiplexer mux(size, "name")` | `StreamBufferMultiplexer(shared_queue_size=size, name="name")` |
|
|
16
|
+
| `stream_buffer_multiplexer mux("name")` | `StreamBufferMultiplexer(name="name")` |
|
|
17
|
+
| `add_producer(id, cap, ctl)` | `add_producer(id, capacity=cap, control_size=ctl)` |
|
|
18
|
+
| `add_producer(id, cap, ctl, "name")` | `add_producer(id, capacity=cap, control_size=ctl, name="name")` |
|
|
19
|
+
| `add_producer(id, "name")` | `add_producer(id, name="name")` |
|
|
20
|
+
| throws `std::invalid_argument` (duplicate id, bad sizes) | raises `ValueError` |
|
|
21
|
+
| `uint32_t producer_id` (range enforced by the type) | raises `ValueError` outside `0..0xFFFFFFFF`, `TypeError` for a non-integer |
|
|
22
|
+
| throws `std::runtime_error` (shm failures) | raises `RuntimeError`, `ValueError` (queue geometry mismatch), or `FileNotFoundError` (missing segment) |
|
|
23
|
+
|
|
24
|
+
## Consumer cursor is passed by value
|
|
25
|
+
|
|
26
|
+
```cpp
|
|
27
|
+
// C++
|
|
28
|
+
uint64_t cursor = 0;
|
|
29
|
+
auto rec = mux.read(cursor); // cursor updated in place
|
|
30
|
+
```
|
|
31
|
+
|
|
32
|
+
```python
|
|
33
|
+
# Python
|
|
34
|
+
cursor = 0
|
|
35
|
+
rec, cursor = mux.read(cursor) # updated cursor returned
|
|
36
|
+
```
|
|
37
|
+
|
|
38
|
+
For the work-stealing overload, pass an `AtomicCursor` (from slick-queue-py); the second
|
|
39
|
+
return value is then the claimed shared-queue index (`-1` when nothing was available):
|
|
40
|
+
|
|
41
|
+
```python
|
|
42
|
+
rec, idx = mux.read(shared_cursor)
|
|
43
|
+
```
|
|
44
|
+
|
|
45
|
+
## Zero-copy vs copies
|
|
46
|
+
|
|
47
|
+
Same policy as slick-stream-buffer-py: `prepare()` and `data()` return zero-copy
|
|
48
|
+
memoryviews into the producer's ring; `read()` results (`MultiplexRecord.data`) and
|
|
49
|
+
`consume()` results are **bytes copies**, safe by default under the lossy overwrite
|
|
50
|
+
semantics.
|
|
51
|
+
|
|
52
|
+
## Producer buffer accessors
|
|
53
|
+
|
|
54
|
+
C++ exposes both `stream_buffer()` (reference) and `stream_buffer_ptr()`
|
|
55
|
+
(`std::shared_ptr`); Python has no owning/non-owning distinction, so there is a single
|
|
56
|
+
`producer.stream_buffer` property returning the underlying `SlickStreamBuffer`.
|
|
57
|
+
Similarly, `get_producer_buffer()` and `find_producer()` both return the same
|
|
58
|
+
`ProducerBuffer` object (or `None`).
|
|
59
|
+
|
|
60
|
+
## Producer lookup
|
|
61
|
+
|
|
62
|
+
C++ uses a dense vector for producer ids < 4096 with a hash-map fallback; Python uses a
|
|
63
|
+
plain dict for all ids (already O(1) — the dense-path micro-optimization is meaningless
|
|
64
|
+
in Python).
|
|
65
|
+
|
|
66
|
+
`get_producer_buffers()` returns a `const` map reference in C++; Python returns a cached
|
|
67
|
+
read-only `types.MappingProxyType` over the same dict — mutation raises, later
|
|
68
|
+
`add_producer()` calls still show through, and the call allocates nothing. Iteration order
|
|
69
|
+
is insertion order, which C++'s `unordered_map` does not promise.
|
|
70
|
+
|
|
71
|
+
## Traits are a constructor argument, not a template parameter
|
|
72
|
+
|
|
73
|
+
C++ spells the configuration `basic_stream_buffer_multiplexer<Traits = default_queue_traits>`;
|
|
74
|
+
Python passes the same `QueueTraits` class as a `traits=` keyword argument, matching how
|
|
75
|
+
slick-queue-py does it:
|
|
76
|
+
|
|
77
|
+
```cpp
|
|
78
|
+
// C++
|
|
79
|
+
slick::basic_stream_buffer_multiplexer<slick::debug_queue_traits> mux(1024);
|
|
80
|
+
```
|
|
81
|
+
|
|
82
|
+
```python
|
|
83
|
+
# Python
|
|
84
|
+
mux = StreamBufferMultiplexer(shared_queue_size=1024, traits=MyTraits)
|
|
85
|
+
```
|
|
86
|
+
|
|
87
|
+
`mux.traits` returns the shared queue's frozen snapshot (C++ spells this
|
|
88
|
+
`using traits = Traits`). One behavioural difference, inherited from slick-queue-py:
|
|
89
|
+
C++'s `default_queue_traits` follows `NDEBUG`, so loss counting is off in release builds;
|
|
90
|
+
Python has no debug/release split and defaults `enable_loss_detection` to on.
|
|
91
|
+
|
|
92
|
+
That difference is harmless across processes because only one trait reaches shared memory:
|
|
93
|
+
`enable_read_last`. The queue's creator records it in the layout marker (`'SLQ1'` on, `'SLQ0'`
|
|
94
|
+
off) and every attacher in either language must be configured the same way, or construction fails
|
|
95
|
+
with "Shared memory feature mismatch" (`RuntimeError` / `std::runtime_error`). Both sides default
|
|
96
|
+
it to on; a peer that turns it off needs its counterparts to do the same:
|
|
97
|
+
|
|
98
|
+
```cpp
|
|
99
|
+
// C++
|
|
100
|
+
struct no_read_last_traits : slick::default_queue_traits {
|
|
101
|
+
static constexpr bool enable_read_last = false;
|
|
102
|
+
};
|
|
103
|
+
slick::basic_stream_buffer_multiplexer<no_read_last_traits> mux("mux_records");
|
|
104
|
+
```
|
|
105
|
+
|
|
106
|
+
```python
|
|
107
|
+
# Python
|
|
108
|
+
class NoReadLast(QueueTraits):
|
|
109
|
+
enable_read_last = False
|
|
110
|
+
|
|
111
|
+
mux = StreamBufferMultiplexer(name="mux_records", traits=NoReadLast)
|
|
112
|
+
```
|
|
113
|
+
|
|
114
|
+
## loss_count()
|
|
115
|
+
|
|
116
|
+
Same semantics on both sides: `traits.enable_loss_detection` switches the shared queue's
|
|
117
|
+
wrap-loss term and the multiplexer-level term together, and both read 0 when it is off. Only
|
|
118
|
+
the multiplexer-level term is filtered by producer registration. Records are skipped correctly
|
|
119
|
+
either way. All counters are per-instance, not shared through the segments.
|
|
120
|
+
|
|
121
|
+
`ProducerBuffer.loss_count()` is likewise identical: the multiplexer dereferences through
|
|
122
|
+
traits with `count_loss` off, so that counter only moves for reads the caller makes directly
|
|
123
|
+
through `producer.stream_buffer.read(cursor, traits)`.
|
|
124
|
+
|
|
125
|
+
## producer_buffer is not movable in C++
|
|
126
|
+
|
|
127
|
+
C++ deletes `producer_buffer`'s copy and move operations, because the multiplexer keeps a raw
|
|
128
|
+
pointer to the same object its `shared_ptr` names, and moving one out would leave both pointing
|
|
129
|
+
at an emptied object. Python has no moves and rebinding a name copies a reference, so there is
|
|
130
|
+
nothing to delete — `ProducerBuffer` has no equivalent restriction.
|
|
131
|
+
|
|
132
|
+
## Lifecycle
|
|
133
|
+
|
|
134
|
+
Python adds explicit lifecycle management that C++ handles via RAII:
|
|
135
|
+
|
|
136
|
+
- `close()` — detach from all producer segments and the queue segment; the registration table is
|
|
137
|
+
kept, so a later `unlink()` still reaches every producer
|
|
138
|
+
- `unlink()` — delete the queue segment and all registered producers' segments; valid before or
|
|
139
|
+
after `close()` (close every handle, then unlink once), and a no-op when repeated
|
|
140
|
+
- context manager support: `with StreamBufferMultiplexer(...) as mux: ...`
|
|
141
|
+
- `get_shm_name()` — on the multiplexer (queue segment) and on each `ProducerBuffer`,
|
|
142
|
+
the exact names to pass to C++ (`/`-prefixed on POSIX)
|
|
143
|
+
|
|
144
|
+
`remove(name)` is the one lifecycle call both sides share, with the same contract: unlink a
|
|
145
|
+
segment by name without attaching to it, returning `True` on success and, on Windows, always
|
|
146
|
+
`True` because a section dies with its last handle.
|
{slick_stream_buffer_multiplexer_py-0.1.0 → slick_stream_buffer_multiplexer_py-2.0.0}/PKG-INFO
RENAMED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
Metadata-Version: 2.4
|
|
2
2
|
Name: slick-stream-buffer-multiplexer-py
|
|
3
|
-
Version: 0.
|
|
3
|
+
Version: 2.0.0
|
|
4
4
|
Summary: Lock-free MPMC byte stream multiplexer with C++ interoperability via shared memory
|
|
5
5
|
Home-page: https://github.com/SlickQuant/slick-stream-buffer-multiplexer-py
|
|
6
6
|
Author: Slick Quant
|
|
@@ -28,8 +28,8 @@ Classifier: Operating System :: MacOS
|
|
|
28
28
|
Requires-Python: >=3.8
|
|
29
29
|
Description-Content-Type: text/markdown
|
|
30
30
|
License-File: LICENSE
|
|
31
|
-
Requires-Dist: slick-queue-py>=
|
|
32
|
-
Requires-Dist: slick-stream-buffer-py>=0.
|
|
31
|
+
Requires-Dist: slick-queue-py>=2.0.0
|
|
32
|
+
Requires-Dist: slick-stream-buffer-py>=2.0.0
|
|
33
33
|
Dynamic: author
|
|
34
34
|
Dynamic: home-page
|
|
35
35
|
Dynamic: license-file
|
|
@@ -41,7 +41,14 @@ A Python implementation of [slick-stream-buffer-multiplexer](https://github.com/
|
|
|
41
41
|
a lock-free multi-producer multi-consumer (MPMC) byte stream multiplexer with shared memory support.
|
|
42
42
|
|
|
43
43
|
**Maintains exact binary compatibility with the C++ version**: Python and C++ producers and
|
|
44
|
-
consumers can be mixed
|
|
44
|
+
consumers can be mixed across processes, fanning into the same shared memory segments — provided
|
|
45
|
+
every peer of a shared queue agrees on `enable_read_last` (see
|
|
46
|
+
[`enable_read_last` must match on every peer](#enable_read_last-must-match-on-every-peer)).
|
|
47
|
+
|
|
48
|
+
Tracks C++ `slick/stream_buffer_multiplexer.hpp` **v2.0.0**, and requires slick-queue-py >=
|
|
49
|
+
2.0.0 and slick-stream-buffer-py >= 2.0.0. The shared-memory formats are unchanged
|
|
50
|
+
(`'SLQ1'` / `'SSB1'`), so segments written by v0.1.0 or by a C++ v1.x process are still
|
|
51
|
+
readable with the default traits.
|
|
45
52
|
|
|
46
53
|
## Concept
|
|
47
54
|
|
|
@@ -70,8 +77,8 @@ records referencing other producer ids are silently skipped (not counted as loss
|
|
|
70
77
|
## Requirements
|
|
71
78
|
|
|
72
79
|
- Python 3.8+
|
|
73
|
-
- [slick-queue-py](https://github.com/SlickQuant/slick-queue-py) >=
|
|
74
|
-
- [slick-stream-buffer-py](https://github.com/SlickQuant/slick-stream-buffer-py) >= 0.
|
|
80
|
+
- [slick-queue-py](https://github.com/SlickQuant/slick-queue-py) >= 2.0.0
|
|
81
|
+
- [slick-stream-buffer-py](https://github.com/SlickQuant/slick-stream-buffer-py) >= 2.0.0
|
|
75
82
|
|
|
76
83
|
Both dependencies bundle a small C++ extension for `std::atomic` operations; this package itself
|
|
77
84
|
is pure Python.
|
|
@@ -130,14 +137,116 @@ shared_cursor = AtomicCursor(cursor_shm.buf, 0)
|
|
|
130
137
|
rec, idx = mux.read(shared_cursor) # atomically claims the next message
|
|
131
138
|
```
|
|
132
139
|
|
|
140
|
+
## Traits
|
|
141
|
+
|
|
142
|
+
Feature configuration is a `traits` constructor argument — and it is slick-queue-py's
|
|
143
|
+
`QueueTraits`, the traits of the shared record queue the multiplexer is built on, not a traits
|
|
144
|
+
type of its own:
|
|
145
|
+
|
|
146
|
+
```python
|
|
147
|
+
from slick_stream_buffer_multiplexer_py import StreamBufferMultiplexer, QueueTraits
|
|
148
|
+
|
|
149
|
+
class Silent(QueueTraits):
|
|
150
|
+
enable_loss_detection = False # both loss terms go quiet together
|
|
151
|
+
|
|
152
|
+
mux = StreamBufferMultiplexer(shared_queue_size=1024, traits=Silent)
|
|
153
|
+
```
|
|
154
|
+
|
|
155
|
+
Every knob the multiplexer has beyond that queue is a passthrough, and its one real tunable
|
|
156
|
+
asks the same question `enable_loss_detection` already asks — so a separate flag would only let
|
|
157
|
+
the two disagree. That matters because `loss_count()` **sums** the two counters: a
|
|
158
|
+
half-configured pair would return a partial total that reads like a complete one. One flag, one
|
|
159
|
+
number. `mux.traits` is the queue's own frozen snapshot, not a second copy.
|
|
160
|
+
|
|
161
|
+
Unlike C++ — where `default_queue_traits` follows `NDEBUG` — slick-queue-py defaults
|
|
162
|
+
`enable_loss_detection` to **on**, so the counters work out of the box.
|
|
163
|
+
|
|
164
|
+
### `enable_read_last` must match on every peer
|
|
165
|
+
|
|
166
|
+
The other traits are local to one instance and may differ freely between processes, but
|
|
167
|
+
`enable_read_last` is part of the shared record queue's header protocol. The queue's creator
|
|
168
|
+
stamps it into the segment's layout marker — `'SLQ1'` when the last-published index is maintained,
|
|
169
|
+
`'SLQ0'` when it is not — and every attacher, Python or C++, checks the marker against its own
|
|
170
|
+
traits. A peer that disagrees is refused at construction (`RuntimeError` in Python,
|
|
171
|
+
`std::runtime_error` in C++, both reading *"Shared memory feature mismatch"*) rather than
|
|
172
|
+
misreading the header: a Python queue created with `enable_read_last = False` cannot be opened by a
|
|
173
|
+
C++ `slick::stream_buffer_multiplexer`, whose default traits have it on.
|
|
174
|
+
|
|
175
|
+
Both languages default it to on, so only turning it off needs coordinating — do it identically on
|
|
176
|
+
every peer of the queue:
|
|
177
|
+
|
|
178
|
+
```python
|
|
179
|
+
class NoReadLast(QueueTraits):
|
|
180
|
+
enable_read_last = False
|
|
181
|
+
|
|
182
|
+
mux = StreamBufferMultiplexer(shared_queue_size=1 << 16, name="mux_records", traits=NoReadLast)
|
|
183
|
+
```
|
|
184
|
+
|
|
185
|
+
```cpp
|
|
186
|
+
struct no_read_last_traits : slick::default_queue_traits {
|
|
187
|
+
static constexpr bool enable_read_last = false;
|
|
188
|
+
};
|
|
189
|
+
slick::basic_stream_buffer_multiplexer<no_read_last_traits> mux("mux_records");
|
|
190
|
+
```
|
|
191
|
+
|
|
192
|
+
The multiplexer itself never calls `read_last()` (`initial_reading_index()` reads the reservation
|
|
193
|
+
counter instead), so turning it off only drops one CAS from each `consume()`'s publish into the
|
|
194
|
+
shared queue. Producer segments (`'SSB1'`) carry no such flag.
|
|
195
|
+
|
|
133
196
|
## Three Loss Counters
|
|
134
197
|
|
|
135
198
|
| Counter | Meaning |
|
|
136
199
|
|---|---|
|
|
137
|
-
| `mux.loss_count()` | shared-queue wrap loss **plus** multiplexer-level loss (registered producers lapped before dereference) |
|
|
138
|
-
| `producer.loss_count()` | that producer's own inner-ring loss |
|
|
200
|
+
| `mux.loss_count()` | shared-queue wrap loss **plus** multiplexer-level loss (registered producers lapped before dereference); both terms switched by `traits.enable_loss_detection` |
|
|
201
|
+
| `producer.loss_count()` | that producer's own inner-ring loss — **only** for reads the caller makes through `producer.stream_buffer.read(cursor, traits)` with counting traits |
|
|
139
202
|
| unregistered producer ids | **never counted** — silently skipped |
|
|
140
203
|
|
|
204
|
+
Only the multiplexer-level term is filtered by registration. The wrap-loss term counts entries
|
|
205
|
+
the shared queue dropped by wrapping, and a lapped slot has already been overwritten, so nothing
|
|
206
|
+
is left to say which producer it named — an instance registering a subset of producers therefore
|
|
207
|
+
sees wrap loss for producers it ignores, making the total an upper bound on what it missed. Size
|
|
208
|
+
the shared queue so it does not wrap and that term goes to 0.
|
|
209
|
+
|
|
210
|
+
`producer.loss_count()` stays at 0 under multiplexer traffic by design: the multiplexer
|
|
211
|
+
dereferences through traits with `count_loss` off, because `read()`'s counter measures how far
|
|
212
|
+
a *sequential scan* jumped, which for a jump-read is a distance every later lapped dereference
|
|
213
|
+
would add all over again. The multiplexer counts one loss per lapped record instead.
|
|
214
|
+
|
|
215
|
+
`mux.get_producer_buffers()` returns every registered producer keyed by `producer_id` — a
|
|
216
|
+
read-only live view of the registration table, cheap enough for a monitoring loop:
|
|
217
|
+
|
|
218
|
+
```python
|
|
219
|
+
inner_loss = sum(p.loss_count() for p in mux.get_producer_buffers().values())
|
|
220
|
+
```
|
|
221
|
+
|
|
222
|
+
## Shutdown
|
|
223
|
+
|
|
224
|
+
Close every handle, then unlink once — the usual `shared_memory` idiom:
|
|
225
|
+
|
|
226
|
+
```python
|
|
227
|
+
mux.close() # or leave a `with StreamBufferMultiplexer(...) as mux:` block
|
|
228
|
+
mux.unlink() # deletes the record queue and every registered producer's segment
|
|
229
|
+
```
|
|
230
|
+
|
|
231
|
+
`close()` keeps the registration table so `unlink()` can still reach each producer's segment, and
|
|
232
|
+
`unlink()` runs once: repeating it never deletes a segment a later run has created under the same
|
|
233
|
+
name. On Windows `unlink()` is a no-op — a section disappears once its last handle closes.
|
|
234
|
+
|
|
235
|
+
## Stale-segment recovery
|
|
236
|
+
|
|
237
|
+
`StreamBufferMultiplexer.remove(name)` unlinks a segment by name — it works for both kinds this
|
|
238
|
+
class creates, a shared record queue's and a producer stream buffer's, since both are plain
|
|
239
|
+
shared memory segments underneath. Use it when a previous run died mid-initialization and left a
|
|
240
|
+
segment wedged, which construction over that name reports by raising:
|
|
241
|
+
|
|
242
|
+
```python
|
|
243
|
+
StreamBufferMultiplexer.remove("md_records") # clear what a dead run left
|
|
244
|
+
mux = StreamBufferMultiplexer(shared_queue_size=1024, name="md_records")
|
|
245
|
+
```
|
|
246
|
+
|
|
247
|
+
Only call it when no process is using the segment. On Windows it is a no-op returning `True` —
|
|
248
|
+
a section there disappears once the last handle closes.
|
|
249
|
+
|
|
141
250
|
## C++ Interoperability
|
|
142
251
|
|
|
143
252
|
The C++ side composes the identical primitives — either language can create any segment:
|
|
@@ -163,16 +272,23 @@ exact names to pass to C++ (POSIX names include the required leading `/`).
|
|
|
163
272
|
|
|
164
273
|
The multiplexer adds no shared-memory structures of its own:
|
|
165
274
|
|
|
166
|
-
- The shared queue is a standard slick-queue segment (`'SLQ1'`, `
|
|
167
|
-
|
|
275
|
+
- The shared queue is a standard slick-queue segment (`'SLQ1'`, or `'SLQ0'` when created with
|
|
276
|
+
`enable_read_last` off; `element_size=16`) whose elements are
|
|
277
|
+
`uint64 sequence | uint32 producer_id | uint32 pad0`
|
|
168
278
|
- Each producer buffer is a standard slick-stream-buffer segment (`'SSB1'`)
|
|
169
279
|
|
|
170
280
|
## Caveats (same as C++)
|
|
171
281
|
|
|
172
282
|
- `add_producer()` is single-threaded setup: call it before producer/consumer threads start.
|
|
283
|
+
- A `producer_id` must fit the record's `uint32` field: `add_producer()` raises `ValueError`
|
|
284
|
+
for one outside `0..0xFFFFFFFF` (and `TypeError` for a non-integer), before any segment is
|
|
285
|
+
created.
|
|
173
286
|
- Each producer's `prepare/commit/consume/...` must be called from a single thread.
|
|
174
287
|
- Lossy semantics: slow consumers skip overwritten data (see the loss counters above).
|
|
175
|
-
|
|
288
|
+
Records are skipped correctly whatever the traits say; only the counters go quiet.
|
|
289
|
+
- A single message (one `consume()` call) is limited to < 4 GiB: `consume()` raises
|
|
290
|
+
`ValueError` rather than truncating one that is not, before any state moves — so nothing is
|
|
291
|
+
published to either the producer's ring or the shared record queue.
|
|
176
292
|
|
|
177
293
|
See [API_DIFFERENCES.md](API_DIFFERENCES.md) for exact deviations from the C++ API.
|
|
178
294
|
|
{slick_stream_buffer_multiplexer_py-0.1.0 → slick_stream_buffer_multiplexer_py-2.0.0}/README.md
RENAMED
|
@@ -4,7 +4,14 @@ A Python implementation of [slick-stream-buffer-multiplexer](https://github.com/
|
|
|
4
4
|
a lock-free multi-producer multi-consumer (MPMC) byte stream multiplexer with shared memory support.
|
|
5
5
|
|
|
6
6
|
**Maintains exact binary compatibility with the C++ version**: Python and C++ producers and
|
|
7
|
-
consumers can be mixed
|
|
7
|
+
consumers can be mixed across processes, fanning into the same shared memory segments — provided
|
|
8
|
+
every peer of a shared queue agrees on `enable_read_last` (see
|
|
9
|
+
[`enable_read_last` must match on every peer](#enable_read_last-must-match-on-every-peer)).
|
|
10
|
+
|
|
11
|
+
Tracks C++ `slick/stream_buffer_multiplexer.hpp` **v2.0.0**, and requires slick-queue-py >=
|
|
12
|
+
2.0.0 and slick-stream-buffer-py >= 2.0.0. The shared-memory formats are unchanged
|
|
13
|
+
(`'SLQ1'` / `'SSB1'`), so segments written by v0.1.0 or by a C++ v1.x process are still
|
|
14
|
+
readable with the default traits.
|
|
8
15
|
|
|
9
16
|
## Concept
|
|
10
17
|
|
|
@@ -33,8 +40,8 @@ records referencing other producer ids are silently skipped (not counted as loss
|
|
|
33
40
|
## Requirements
|
|
34
41
|
|
|
35
42
|
- Python 3.8+
|
|
36
|
-
- [slick-queue-py](https://github.com/SlickQuant/slick-queue-py) >=
|
|
37
|
-
- [slick-stream-buffer-py](https://github.com/SlickQuant/slick-stream-buffer-py) >= 0.
|
|
43
|
+
- [slick-queue-py](https://github.com/SlickQuant/slick-queue-py) >= 2.0.0
|
|
44
|
+
- [slick-stream-buffer-py](https://github.com/SlickQuant/slick-stream-buffer-py) >= 2.0.0
|
|
38
45
|
|
|
39
46
|
Both dependencies bundle a small C++ extension for `std::atomic` operations; this package itself
|
|
40
47
|
is pure Python.
|
|
@@ -93,14 +100,116 @@ shared_cursor = AtomicCursor(cursor_shm.buf, 0)
|
|
|
93
100
|
rec, idx = mux.read(shared_cursor) # atomically claims the next message
|
|
94
101
|
```
|
|
95
102
|
|
|
103
|
+
## Traits
|
|
104
|
+
|
|
105
|
+
Feature configuration is a `traits` constructor argument — and it is slick-queue-py's
|
|
106
|
+
`QueueTraits`, the traits of the shared record queue the multiplexer is built on, not a traits
|
|
107
|
+
type of its own:
|
|
108
|
+
|
|
109
|
+
```python
|
|
110
|
+
from slick_stream_buffer_multiplexer_py import StreamBufferMultiplexer, QueueTraits
|
|
111
|
+
|
|
112
|
+
class Silent(QueueTraits):
|
|
113
|
+
enable_loss_detection = False # both loss terms go quiet together
|
|
114
|
+
|
|
115
|
+
mux = StreamBufferMultiplexer(shared_queue_size=1024, traits=Silent)
|
|
116
|
+
```
|
|
117
|
+
|
|
118
|
+
Every knob the multiplexer has beyond that queue is a passthrough, and its one real tunable
|
|
119
|
+
asks the same question `enable_loss_detection` already asks — so a separate flag would only let
|
|
120
|
+
the two disagree. That matters because `loss_count()` **sums** the two counters: a
|
|
121
|
+
half-configured pair would return a partial total that reads like a complete one. One flag, one
|
|
122
|
+
number. `mux.traits` is the queue's own frozen snapshot, not a second copy.
|
|
123
|
+
|
|
124
|
+
Unlike C++ — where `default_queue_traits` follows `NDEBUG` — slick-queue-py defaults
|
|
125
|
+
`enable_loss_detection` to **on**, so the counters work out of the box.
|
|
126
|
+
|
|
127
|
+
### `enable_read_last` must match on every peer
|
|
128
|
+
|
|
129
|
+
The other traits are local to one instance and may differ freely between processes, but
|
|
130
|
+
`enable_read_last` is part of the shared record queue's header protocol. The queue's creator
|
|
131
|
+
stamps it into the segment's layout marker — `'SLQ1'` when the last-published index is maintained,
|
|
132
|
+
`'SLQ0'` when it is not — and every attacher, Python or C++, checks the marker against its own
|
|
133
|
+
traits. A peer that disagrees is refused at construction (`RuntimeError` in Python,
|
|
134
|
+
`std::runtime_error` in C++, both reading *"Shared memory feature mismatch"*) rather than
|
|
135
|
+
misreading the header: a Python queue created with `enable_read_last = False` cannot be opened by a
|
|
136
|
+
C++ `slick::stream_buffer_multiplexer`, whose default traits have it on.
|
|
137
|
+
|
|
138
|
+
Both languages default it to on, so only turning it off needs coordinating — do it identically on
|
|
139
|
+
every peer of the queue:
|
|
140
|
+
|
|
141
|
+
```python
|
|
142
|
+
class NoReadLast(QueueTraits):
|
|
143
|
+
enable_read_last = False
|
|
144
|
+
|
|
145
|
+
mux = StreamBufferMultiplexer(shared_queue_size=1 << 16, name="mux_records", traits=NoReadLast)
|
|
146
|
+
```
|
|
147
|
+
|
|
148
|
+
```cpp
|
|
149
|
+
struct no_read_last_traits : slick::default_queue_traits {
|
|
150
|
+
static constexpr bool enable_read_last = false;
|
|
151
|
+
};
|
|
152
|
+
slick::basic_stream_buffer_multiplexer<no_read_last_traits> mux("mux_records");
|
|
153
|
+
```
|
|
154
|
+
|
|
155
|
+
The multiplexer itself never calls `read_last()` (`initial_reading_index()` reads the reservation
|
|
156
|
+
counter instead), so turning it off only drops one CAS from each `consume()`'s publish into the
|
|
157
|
+
shared queue. Producer segments (`'SSB1'`) carry no such flag.
|
|
158
|
+
|
|
96
159
|
## Three Loss Counters
|
|
97
160
|
|
|
98
161
|
| Counter | Meaning |
|
|
99
162
|
|---|---|
|
|
100
|
-
| `mux.loss_count()` | shared-queue wrap loss **plus** multiplexer-level loss (registered producers lapped before dereference) |
|
|
101
|
-
| `producer.loss_count()` | that producer's own inner-ring loss |
|
|
163
|
+
| `mux.loss_count()` | shared-queue wrap loss **plus** multiplexer-level loss (registered producers lapped before dereference); both terms switched by `traits.enable_loss_detection` |
|
|
164
|
+
| `producer.loss_count()` | that producer's own inner-ring loss — **only** for reads the caller makes through `producer.stream_buffer.read(cursor, traits)` with counting traits |
|
|
102
165
|
| unregistered producer ids | **never counted** — silently skipped |
|
|
103
166
|
|
|
167
|
+
Only the multiplexer-level term is filtered by registration. The wrap-loss term counts entries
|
|
168
|
+
the shared queue dropped by wrapping, and a lapped slot has already been overwritten, so nothing
|
|
169
|
+
is left to say which producer it named — an instance registering a subset of producers therefore
|
|
170
|
+
sees wrap loss for producers it ignores, making the total an upper bound on what it missed. Size
|
|
171
|
+
the shared queue so it does not wrap and that term goes to 0.
|
|
172
|
+
|
|
173
|
+
`producer.loss_count()` stays at 0 under multiplexer traffic by design: the multiplexer
|
|
174
|
+
dereferences through traits with `count_loss` off, because `read()`'s counter measures how far
|
|
175
|
+
a *sequential scan* jumped, which for a jump-read is a distance every later lapped dereference
|
|
176
|
+
would add all over again. The multiplexer counts one loss per lapped record instead.
|
|
177
|
+
|
|
178
|
+
`mux.get_producer_buffers()` returns every registered producer keyed by `producer_id` — a
|
|
179
|
+
read-only live view of the registration table, cheap enough for a monitoring loop:
|
|
180
|
+
|
|
181
|
+
```python
|
|
182
|
+
inner_loss = sum(p.loss_count() for p in mux.get_producer_buffers().values())
|
|
183
|
+
```
|
|
184
|
+
|
|
185
|
+
## Shutdown
|
|
186
|
+
|
|
187
|
+
Close every handle, then unlink once — the usual `shared_memory` idiom:
|
|
188
|
+
|
|
189
|
+
```python
|
|
190
|
+
mux.close() # or leave a `with StreamBufferMultiplexer(...) as mux:` block
|
|
191
|
+
mux.unlink() # deletes the record queue and every registered producer's segment
|
|
192
|
+
```
|
|
193
|
+
|
|
194
|
+
`close()` keeps the registration table so `unlink()` can still reach each producer's segment, and
|
|
195
|
+
`unlink()` runs once: repeating it never deletes a segment a later run has created under the same
|
|
196
|
+
name. On Windows `unlink()` is a no-op — a section disappears once its last handle closes.
|
|
197
|
+
|
|
198
|
+
## Stale-segment recovery
|
|
199
|
+
|
|
200
|
+
`StreamBufferMultiplexer.remove(name)` unlinks a segment by name — it works for both kinds this
|
|
201
|
+
class creates, a shared record queue's and a producer stream buffer's, since both are plain
|
|
202
|
+
shared memory segments underneath. Use it when a previous run died mid-initialization and left a
|
|
203
|
+
segment wedged, which construction over that name reports by raising:
|
|
204
|
+
|
|
205
|
+
```python
|
|
206
|
+
StreamBufferMultiplexer.remove("md_records") # clear what a dead run left
|
|
207
|
+
mux = StreamBufferMultiplexer(shared_queue_size=1024, name="md_records")
|
|
208
|
+
```
|
|
209
|
+
|
|
210
|
+
Only call it when no process is using the segment. On Windows it is a no-op returning `True` —
|
|
211
|
+
a section there disappears once the last handle closes.
|
|
212
|
+
|
|
104
213
|
## C++ Interoperability
|
|
105
214
|
|
|
106
215
|
The C++ side composes the identical primitives — either language can create any segment:
|
|
@@ -126,16 +235,23 @@ exact names to pass to C++ (POSIX names include the required leading `/`).
|
|
|
126
235
|
|
|
127
236
|
The multiplexer adds no shared-memory structures of its own:
|
|
128
237
|
|
|
129
|
-
- The shared queue is a standard slick-queue segment (`'SLQ1'`, `
|
|
130
|
-
|
|
238
|
+
- The shared queue is a standard slick-queue segment (`'SLQ1'`, or `'SLQ0'` when created with
|
|
239
|
+
`enable_read_last` off; `element_size=16`) whose elements are
|
|
240
|
+
`uint64 sequence | uint32 producer_id | uint32 pad0`
|
|
131
241
|
- Each producer buffer is a standard slick-stream-buffer segment (`'SSB1'`)
|
|
132
242
|
|
|
133
243
|
## Caveats (same as C++)
|
|
134
244
|
|
|
135
245
|
- `add_producer()` is single-threaded setup: call it before producer/consumer threads start.
|
|
246
|
+
- A `producer_id` must fit the record's `uint32` field: `add_producer()` raises `ValueError`
|
|
247
|
+
for one outside `0..0xFFFFFFFF` (and `TypeError` for a non-integer), before any segment is
|
|
248
|
+
created.
|
|
136
249
|
- Each producer's `prepare/commit/consume/...` must be called from a single thread.
|
|
137
250
|
- Lossy semantics: slow consumers skip overwritten data (see the loss counters above).
|
|
138
|
-
|
|
251
|
+
Records are skipped correctly whatever the traits say; only the counters go quiet.
|
|
252
|
+
- A single message (one `consume()` call) is limited to < 4 GiB: `consume()` raises
|
|
253
|
+
`ValueError` rather than truncating one that is not, before any state moves — so nothing is
|
|
254
|
+
published to either the producer's ring or the shared record queue.
|
|
139
255
|
|
|
140
256
|
See [API_DIFFERENCES.md](API_DIFFERENCES.md) for exact deviations from the C++ API.
|
|
141
257
|
|
|
@@ -1,3 +1,3 @@
|
|
|
1
1
|
# Runtime dependencies (binary-compatible Python ports of the C++ dependencies)
|
|
2
|
-
slick-queue-py>=
|
|
3
|
-
slick-stream-buffer-py>=0.
|
|
2
|
+
slick-queue-py>=2.0.0
|
|
3
|
+
slick-stream-buffer-py>=2.0.0
|
{slick_stream_buffer_multiplexer_py-0.1.0 → slick_stream_buffer_multiplexer_py-2.0.0}/setup.py
RENAMED
|
@@ -52,8 +52,8 @@ setup(
|
|
|
52
52
|
keywords='stream-buffer, multiplexer, lock-free, atomic, shared-memory, ipc, multiprocessing, mpmc',
|
|
53
53
|
py_modules=['slick_stream_buffer_multiplexer_py'],
|
|
54
54
|
install_requires=[
|
|
55
|
-
'slick-queue-py>=
|
|
56
|
-
'slick-stream-buffer-py>=0.
|
|
55
|
+
'slick-queue-py>=2.0.0',
|
|
56
|
+
'slick-stream-buffer-py>=2.0.0',
|
|
57
57
|
],
|
|
58
58
|
python_requires='>=3.8',
|
|
59
59
|
license='MIT',
|