slick-queue-py 1.1.0__tar.gz → 1.2.0__tar.gz

{slick_queue_py-1.1.0 → slick_queue_py-1.2.0}/CHANGELOG.md

@@ -1,80 +1,97 @@
-# Changelogs
-
-## [v1.1.0] - 2026-02-04
-
-### Breaking Changes
-- **BREAKING:** `read_last()` return signature changed from `Optional[bytes]` to `Tuple[Optional[bytes], int]`
-  - Now returns `(data, size)` tuple where `size` indicates the number of slots the item occupies
-  - Migration: Change `data = q.read_last()` to `data, size = q.read_last()`
-  - Provides actual slot size information instead of always returning full `element_size`
-
-### Added
-- Modern format support with efficient `last_published_` atomic tracking
-  - New `last_published_` atomic field at offset 16 for O(1) `read_last()` performance
-  - Header magic number `0x534C5131` ('SLQ1') at offset 24 for format detection
-  - Init state atomic at offset 48 for CAS-based ownership detection
-  - Automatic format detection distinguishes modern vs legacy queues
-- CAS-based queue ownership detection matching C++ implementation (queue.h:618-648)
-- Helper methods `_wait_for_shared_memory_ready()` and `_detect_format_version()`
-- Comprehensive test suites:
-  - `test_read_last.py`: 10 tests covering empty queue, single/multiple items, multi-slot publishes, concurrent publishes, reset, shared memory, wraparound, variable sizes
-  - `test_read_last_interop.py`: 5 tests for Python ↔ C++ interoperability
-  - `test_modern_format.py`: 6 tests for format compatibility and memory layout verification
-- C++ test program `cpp_read_last_tester.cpp` for read_last() interoperability testing
-- New constants: `SIZE_OFFSET`, `ELEMENT_SIZE_OFFSET`, `LAST_PUBLISHED_OFFSET`, `HEADER_MAGIC_OFFSET`, `HEADER_MAGIC`, `INIT_STATE_OFFSET`, `INIT_STATE_*`, `K_INVALID_INDEX`
-- Documentation: `READ_LAST_INTEROP.md` comprehensive interoperability guide
-
-### Changed
-- Updated shared memory header layout to match C++ slick-queue v1.x:
-  - Reserved info: 8 bytes at offset 0-7 (compact from previous 32-byte padded format)
-  - Size: 4 bytes at offset 8-11 (moved from offset 32)
-  - Element size: 4 bytes at offset 12-15 (moved from offset 36)
-  - Last published: 8 bytes at offset 16-23 (NEW)
-  - Header magic: 4 bytes at offset 24-27 (NEW)
-  - Init state: 4 bytes at offset 48-51 (NEW)
-- `publish()` now updates `last_published_` atomic using CAS loop in modern format (queue.h:331-337)
-- `reset()` now resets `last_published_` to `K_INVALID_INDEX` when in modern format
-- `close()` properly releases `_atomic_last_published` wrapper
-- `AtomicReservedInfo.RESERVED_INFO_FMT` simplified from 32-byte to 8-byte format
-- Repository name changed from slick_queue_py to slick-queue-py
-- Updated README.md with new `read_last()` API documentation
-- Enhanced CMakeLists.txt with new test targets
-
-### Fixed
-- `read_last()` now returns actual slot size instead of always returning full `element_size`
-- Test queue names shortened to comply with macOS 31-character shared memory name limit
-
-### Performance
-- O(1) `read_last()` via direct atomic load (improved from reserved_info calculation)
-- Reduced header size: 8-byte reserved_info vs 32-byte padded (24 bytes saved)
-- Lock-free `last_published_` tracking enables efficient concurrent access
-
-### Compatibility
-- **Backward Compatible:** Automatically detects and supports legacy queue format
-- **C++ Interop:** Full compatibility with C++ slick-queue modern format
-- Python and C++ can share queues created by either language
-- All existing tests continue to pass (atomic ops, cursors, local mode, multi-producer, interop)
-
-## [v1.0.1] - 2025-12-27
-
-### Fixed
-- Fixed extra `/` prefix being added to shared memory names on POSIX systems (Linux/macOS)
-- Fixed test failures on Linux and macOS platforms
-
-### Added
-- macOS shared memory name length validation (31 character limit including `/` prefix)
-- C++ extension support for Linux/macOS platforms with automatic fallback to native methods if unavailable
-- `atomic_store_64` and `atomic_cas_64` functions to C++ extension for improved cross-platform consistency
-
-### Changed
-- Linux/macOS now prioritize C++ extension for atomic operations, falling back to native methods (`__sync_val_compare_and_swap` or `libatomic`) if extension is not available
-- Improved atomic operation reliability across all platforms
-- Enhanced test suite for better cross-platform compatibility
-- Publishing: Updated PyPI builds to use `cibuildwheel` for proper manylinux wheel generation
-- Ignore conda publish failing for now
-
-## [v1.0.0] - 2025-12-26
-
-- Initial release
-- Python implementation of SlickQueue - a lock-free multi-producer multi-consumer (MPMC) queue with C++ interoperability through shared memory.
+# Changelogs
+
+## [v1.2.0] - 2026-07-09
+
+### Added
+- `loss_count()`: per-instance count of items skipped due to overwrite, matching
+  C++ slick-queue v1.5.0 (queue.h:230-236). Counted in both the single-consumer
+  read (slot overwritten by a newer generation at the same position) and the
+  AtomicCursor work-stealing read (overrun attributed to the claiming consumer).
+  Cleared by `reset()`. Note: Python always counts; C++ counts only when
+  `SLICK_QUEUE_ENABLE_LOSS_DETECTION` is enabled (debug builds by default).
+- `initial_reading_index()`: cursor for a late-joining consumer - 0 for a newly
+  created queue, or the current writing index of an opened queue, matching C++
+  slick-queue v1.5.0 (queue.h:242-244).
+- `tests/test_loss_count.py`: 7 tests covering both new methods.
+
+These are per-instance API additions only - the shared memory layout is unchanged
+and remains fully compatible with C++ slick-queue v1.x ('SLQ1' format).
+
+## [v1.1.0] - 2026-02-04
+
+### Breaking Changes
+- **BREAKING:** `read_last()` return signature changed from `Optional[bytes]` to `Tuple[Optional[bytes], int]`
+  - Now returns `(data, size)` tuple where `size` indicates the number of slots the item occupies
+  - Migration: Change `data = q.read_last()` to `data, size = q.read_last()`
+  - Provides actual slot size information instead of always returning full `element_size`
+
+### Added
+- Modern format support with efficient `last_published_` atomic tracking
+  - New `last_published_` atomic field at offset 16 for O(1) `read_last()` performance
+  - Header magic number `0x534C5131` ('SLQ1') at offset 24 for format detection
+  - Init state atomic at offset 48 for CAS-based ownership detection
+  - Automatic format detection distinguishes modern vs legacy queues
+- CAS-based queue ownership detection matching C++ implementation (queue.h:618-648)
+- Helper methods `_wait_for_shared_memory_ready()` and `_detect_format_version()`
+- Comprehensive test suites:
+  - `test_read_last.py`: 10 tests covering empty queue, single/multiple items, multi-slot publishes, concurrent publishes, reset, shared memory, wraparound, variable sizes
+  - `test_read_last_interop.py`: 5 tests for Python ↔ C++ interoperability
+  - `test_modern_format.py`: 6 tests for format compatibility and memory layout verification
+- C++ test program `cpp_read_last_tester.cpp` for read_last() interoperability testing
+- New constants: `SIZE_OFFSET`, `ELEMENT_SIZE_OFFSET`, `LAST_PUBLISHED_OFFSET`, `HEADER_MAGIC_OFFSET`, `HEADER_MAGIC`, `INIT_STATE_OFFSET`, `INIT_STATE_*`, `K_INVALID_INDEX`
+- Documentation: `READ_LAST_INTEROP.md` comprehensive interoperability guide
+
+### Changed
+- Updated shared memory header layout to match C++ slick-queue v1.x:
+  - Reserved info: 8 bytes at offset 0-7 (compact from previous 32-byte padded format)
+  - Size: 4 bytes at offset 8-11 (moved from offset 32)
+  - Element size: 4 bytes at offset 12-15 (moved from offset 36)
+  - Last published: 8 bytes at offset 16-23 (NEW)
+  - Header magic: 4 bytes at offset 24-27 (NEW)
+  - Init state: 4 bytes at offset 48-51 (NEW)
+- `publish()` now updates `last_published_` atomic using CAS loop in modern format (queue.h:331-337)
+- `reset()` now resets `last_published_` to `K_INVALID_INDEX` when in modern format
+- `close()` properly releases `_atomic_last_published` wrapper
+- `AtomicReservedInfo.RESERVED_INFO_FMT` simplified from 32-byte to 8-byte format
+- Repository name changed from slick_queue_py to slick-queue-py
+- Updated README.md with new `read_last()` API documentation
+- Enhanced CMakeLists.txt with new test targets
+
+### Fixed
+- `read_last()` now returns actual slot size instead of always returning full `element_size`
+- Test queue names shortened to comply with macOS 31-character shared memory name limit
+
+### Performance
+- O(1) `read_last()` via direct atomic load (improved from reserved_info calculation)
+- Reduced header size: 8-byte reserved_info vs 32-byte padded (24 bytes saved)
+- Lock-free `last_published_` tracking enables efficient concurrent access
+
+### Compatibility
+- **Backward Compatible:** Automatically detects and supports legacy queue format
+- **C++ Interop:** Full compatibility with C++ slick-queue modern format
+- Python and C++ can share queues created by either language
+- All existing tests continue to pass (atomic ops, cursors, local mode, multi-producer, interop)
+
+## [v1.0.1] - 2025-12-27
+
+### Fixed
+- Fixed extra `/` prefix being added to shared memory names on POSIX systems (Linux/macOS)
+- Fixed test failures on Linux and macOS platforms
+
+### Added
+- macOS shared memory name length validation (31 character limit including `/` prefix)
+- C++ extension support for Linux/macOS platforms with automatic fallback to native methods if unavailable
+- `atomic_store_64` and `atomic_cas_64` functions to C++ extension for improved cross-platform consistency
+
+### Changed
+- Linux/macOS now prioritize C++ extension for atomic operations, falling back to native methods (`__sync_val_compare_and_swap` or `libatomic`) if extension is not available
+- Improved atomic operation reliability across all platforms
+- Enhanced test suite for better cross-platform compatibility
+- Publishing: Updated PyPI builds to use `cibuildwheel` for proper manylinux wheel generation
+- Ignore conda publish failing for now
+
+## [v1.0.0] - 2025-12-26
+
+- Initial release
+- Python implementation of SlickQueue - a lock-free multi-producer multi-consumer (MPMC) queue with C++ interoperability through shared memory.
 - Windows, Linux, and macOS support

{slick_queue_py-1.1.0 → slick_queue_py-1.2.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: slick-queue-py
-Version: 1.1.0
+Version: 1.2.0
 Summary: Lock-free MPMC queue with C++ interoperability via shared memory
 Home-page: https://github.com/SlickQuant/slick-queue-py
 Author: Slick Quant

{slick_queue_py-1.1.0 → slick_queue_py-1.2.0}/slick_queue_py.egg-info/SOURCES.txt

@@ -31,6 +31,7 @@ tests/test_atomic_cursor.py
 tests/test_atomic_ops.py
 tests/test_interop.py
 tests/test_local_mode.py
+tests/test_loss_count.py
 tests/test_modern_format.py
 tests/test_multi_producer.py
 tests/test_queue.py

{slick_queue_py-1.1.0 → slick_queue_py-1.2.0}/slick_queue_py.py

@@ -20,7 +20,7 @@ Supported on Python 3.8+ (uses multiprocessing.shared_memory).
 """
 from __future__ import annotations
 
-__version__ = '1.1.0'
+__version__ = '1.2.0'
 
 import struct
 import sys
@@ -104,6 +104,8 @@ class SlickQueue:
         self._own = False
         self._last_published_valid = False
         self._atomic_last_published = None
+        # per-instance loss counter (matches C++ queue.h:98 loss_count_; local, not shared)
+        self._loss_count = 0
 
         # Validate parameters
         if size is not None:
@@ -529,6 +531,12 @@ class SlickQueue:
                 read_index = 0
                 continue
 
+            # Loss detection (C++ queue.h:357-361): the slot holds a newer
+            # generation at the same position - the items in between were
+            # overwritten before this consumer read them
+            if data_index != (2**64 - 1) and data_index > read_index and ((data_index & self.mask) == idx):
+                self._loss_count += data_index - read_index
+
             # Check if data is ready (C++ lines 258-261)
             if data_index == (2**64 - 1) or data_index < read_index:
                 return None, 0, read_index
@@ -583,6 +591,12 @@ class SlickQueue:
             if data_index == (2**64 - 1) or data_index < current_index:
                 return None, 0, -1
 
+            # Loss detection (C++ queue.h:406-411): the slot holds a newer
+            # generation at the same position; count only if we claim it below
+            overrun = 0
+            if data_index > current_index and ((data_index & self.mask) == idx):
+                overrun = data_index - current_index
+
             # Check for wrap (C++ lines 300-304)
             if data_index > current_index and ((data_index & self.mask) != idx):
                 # Try to atomically update cursor to skip wrapped slots
@@ -595,12 +609,42 @@ class SlickQueue:
 
             if success:
                 # Successfully claimed the item, read and return it
+                # (loss attributed to the claiming consumer, C++ queue.h:422-426)
+                if overrun != 0:
+                    self._loss_count += overrun
                 data_off = self._data_offset + (current_index & self.mask) * self.element_size
                 data = bytes(self._buf[data_off: data_off + slot_size * self.element_size])
                 return data, slot_size, current_index
 
             # CAS failed, another consumer claimed it, retry
 
+    def loss_count(self) -> int:
+        """
+        Get the number of items skipped due to overwrite observed by this
+        queue instance.
+
+        Matches C++ queue.h:230-236 (added in slick-queue v1.5.0). The counter
+        is per-instance (not shared through the segment). Note: Python always
+        counts; C++ counts only when SLICK_QUEUE_ENABLE_LOSS_DETECTION is
+        enabled (debug builds by default).
+
+        Returns:
+            Count of skipped items observed by this instance.
+        """
+        return self._loss_count
+
+    def initial_reading_index(self) -> int:
+        """
+        Get the initial reading index for a late-joining consumer: 0 if the
+        queue is newly created, or the current writing index if opened existing.
+
+        Matches C++ queue.h:242-244 (added in slick-queue v1.5.0).
+
+        Returns:
+            Initial reading index.
+        """
+        return self._atomic_reserved.load()[0]
+
     def read_last(self) -> Tuple[Optional[bytes], int]:
         """
         Read the last published data in the queue.
@@ -655,6 +699,9 @@ class SlickQueue:
         if self._last_published_valid:
             self._atomic_last_published.store_release(K_INVALID_INDEX)
 
+        # Reset loss counter (C++ queue.h:474-476)
+        self._loss_count = 0
+
     def close(self) -> None:
         """Close the queue connection.
 

slick_queue_py-1.2.0/tests/test_loss_count.py

@@ -0,0 +1,185 @@
+"""
+Tests for loss_count() and initial_reading_index() (added in v1.2.0,
+matching C++ slick-queue v1.5.0 queue.h:230-244).
+"""
+import sys
+from pathlib import Path
+
+sys.path.insert(0, str(Path(__file__).parent.parent))
+
+from slick_queue_py import SlickQueue
+from atomic_ops import AtomicCursor
+
+
+def test_initial_reading_index_new_queue():
+    """A newly created queue starts writing at index 0."""
+    q = SlickQueue(size=16, element_size=8)
+    assert q.initial_reading_index() == 0
+    q.close()
+
+
+def test_initial_reading_index_after_publish():
+    """initial_reading_index equals the current writing index."""
+    q = SlickQueue(size=16, element_size=8)
+    for i in range(5):
+        idx = q.reserve()
+        q[idx][:8] = i.to_bytes(8, 'little')
+        q.publish(idx)
+    assert q.initial_reading_index() == 5
+
+    # a late joiner starting there sees nothing until a new publish
+    cursor = q.initial_reading_index()
+    data, size, cursor = q.read(cursor)
+    assert data is None
+
+    idx = q.reserve()
+    q[idx][:8] = (99).to_bytes(8, 'little')
+    q.publish(idx)
+    data, size, cursor = q.read(cursor)
+    assert data is not None
+    assert int.from_bytes(data[:8], 'little') == 99
+    q.close()
+
+
+def test_initial_reading_index_shm_opener():
+    """An opener of an existing shm queue sees the creator's write index."""
+    creator = SlickQueue(name="slq_test_init_idx", size=16, element_size=8)
+    try:
+        for i in range(3):
+            idx = creator.reserve()
+            creator[idx][:8] = i.to_bytes(8, 'little')
+            creator.publish(idx)
+
+        opener = SlickQueue(name="slq_test_init_idx", element_size=8)
+        try:
+            assert opener.initial_reading_index() == 3
+        finally:
+            opener.close()
+    finally:
+        creator.close()
+        creator.unlink()
+
+
+def test_no_loss_on_normal_reads():
+    """Sequential reads with no overwrite never count loss."""
+    q = SlickQueue(size=16, element_size=8)
+    for i in range(10):
+        idx = q.reserve()
+        q[idx][:8] = i.to_bytes(8, 'little')
+        q.publish(idx)
+
+    cursor = 0
+    for i in range(10):
+        data, size, cursor = q.read(cursor)
+        assert int.from_bytes(data[:8], 'little') == i
+    assert q.loss_count() == 0
+    q.close()
+
+
+def test_single_consumer_loss_on_lap():
+    """A lapped consumer skips overwritten items and counts them as loss
+    (C++ queue.h:357-361)."""
+    q = SlickQueue(size=4, element_size=8)
+    for i in range(8):  # laps the 4-slot queue once
+        idx = q.reserve()
+        q[idx][:8] = i.to_bytes(8, 'little')
+        q.publish(idx)
+
+    # slot 0 now holds item 4; a cursor at 0 must return item 4 and count 4 lost
+    data, size, cursor = q.read(0)
+    assert data is not None
+    assert int.from_bytes(data[:8], 'little') == 4
+    assert q.loss_count() == 4, f"expected loss 4, got {q.loss_count()}"
+    assert cursor == 5
+
+    for i in range(5, 8):
+        data, size, cursor = q.read(cursor)
+        assert int.from_bytes(data[:8], 'little') == i
+    assert q.loss_count() == 4  # no further loss
+    q.close()
+
+
+def test_atomic_cursor_loss_on_lap():
+    """Work-stealing consumers count loss when claiming a lapped slot
+    (C++ queue.h:406-426)."""
+    q = SlickQueue(size=4, element_size=8)
+    for i in range(8):
+        idx = q.reserve()
+        q[idx][:8] = i.to_bytes(8, 'little')
+        q.publish(idx)
+
+    cursor_buf = bytearray(8)
+    cursor = AtomicCursor(cursor_buf, 0)
+    cursor.store(0)
+
+    data, size, index = q.read(cursor)
+    assert data is not None
+    assert int.from_bytes(data[:8], 'little') == 4
+    assert q.loss_count() == 4, f"expected loss 4, got {q.loss_count()}"
+
+    remaining = []
+    while True:
+        data, size, index = q.read(cursor)
+        if data is None:
+            break
+        remaining.append(int.from_bytes(data[:8], 'little'))
+    assert remaining == [5, 6, 7]
+    assert q.loss_count() == 4
+    cursor.release()
+    q.close()
+
+
+def test_reset_clears_loss_count():
+    """reset() zeroes the loss counter (C++ queue.h:474-476)."""
+    q = SlickQueue(size=4, element_size=8)
+    for i in range(8):
+        idx = q.reserve()
+        q[idx][:8] = i.to_bytes(8, 'little')
+        q.publish(idx)
+    q.read(0)
+    assert q.loss_count() == 4
+
+    q.reset()
+    assert q.loss_count() == 0
+    q.close()
+
+
+def run_all_tests():
+    tests = [
+        ("InitialReadingIndexNewQueue", test_initial_reading_index_new_queue),
+        ("InitialReadingIndexAfterPublish", test_initial_reading_index_after_publish),
+        ("InitialReadingIndexShmOpener", test_initial_reading_index_shm_opener),
+        ("NoLossOnNormalReads", test_no_loss_on_normal_reads),
+        ("SingleConsumerLossOnLap", test_single_consumer_loss_on_lap),
+        ("AtomicCursorLossOnLap", test_atomic_cursor_loss_on_lap),
+        ("ResetClearsLossCount", test_reset_clears_loss_count),
+    ]
+
+    print("=" * 70)
+    print("Running Loss Count / Initial Reading Index Tests")
+    print("=" * 70)
+
+    passed = 0
+    failed = 0
+    for name, test_func in tests:
+        try:
+            print(f"\n[TEST] {name}...", end=" ")
+            test_func()
+            print("[PASSED]")
+            passed += 1
+        except Exception as e:
+            print("[FAILED]")
+            print(f"  Error: {e}")
+            import traceback
+            traceback.print_exc()
+            failed += 1
+
+    print("\n" + "=" * 70)
+    print(f"Results: {passed} passed, {failed} failed")
+    print("=" * 70)
+    return failed == 0
+
+
+if __name__ == '__main__':
+    success = run_all_tests()
+    sys.exit(0 if success else 1)