shared-multiprocess-queue 0.1.1__py3-none-any.whl

shared_multiprocess_queue-0.1.1.dist-info/METADATA

@@ -0,0 +1,327 @@
+Metadata-Version: 2.4
+Name: shared-multiprocess-queue
+Version: 0.1.1
+Summary: A shared-memory based multiprocessing queue for cross-process communication
+Author-email: Raymond Chastain <RaymondLC92@protonmail.com>
+License: MIT
+Keywords: python,package
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3.13
+Requires-Python: >=3.13.7
+Description-Content-Type: text/markdown
+Requires-Dist: cloudpickle>=3.0.0
+Requires-Dist: shared_memory_lock>=0.1.1
+
+# Shared Queue
+
+A high-performance shared-memory based multiprocessing queue for cross-process communication. Unlike `multiprocessing.Queue`, this queue is fully picklable, can be accessed by name from any process, and provides advanced features like batch operations and metrics.
+
+## Features
+
+- **Picklable**: Can be safely passed between processes via pickle
+- **Named queues**: Multiple processes can connect to the same queue by name
+- **Ring buffer design**: Efficient circular buffer with atomic head/tail pointers
+- **Batch operations**: `put_batch()` for high-throughput scenarios
+- **Shared memory backed**: Uses `multiprocessing.shared_memory` for zero-copy IPC
+- **Thread and process safe**: Synchronized access with `shared_memory_lock`
+- **Observable**: Built-in metrics for monitoring queue utilization
+- **Run namespacing**: Isolate queues by run_id to prevent collisions
+
+## Installation
+
+```bash
+pip install shared-multiprocess-queue
+```
+
+Or for development:
+
+```bash
+git clone <repo-url>
+cd shared_queue
+uv sync
+uv pip install -e .
+```
+
+## Quick Start
+
+```python
+from shared_queue import SharedMemoryQueue
+from multiprocessing import Process
+import time
+
+def worker(queue_name: str, run_id: str, worker_id: int):
+    # Connect to existing queue by name
+    queue = SharedMemoryQueue(name=queue_name, create=False, run_id=run_id)
+
+    while True:
+        try:
+            task = queue.get(timeout=1.0)
+            if task == "STOP":
+                break
+            print(f"Worker {worker_id} processing: {task}")
+            time.sleep(0.1)  # Simulate work
+        except queue.Empty:
+            continue
+
+    queue.close()
+
+def main():
+    run_id = "task_processor"
+
+    # Create the queue in main process
+    work_queue = SharedMemoryQueue(
+        name="tasks",
+        create=True,
+        capacity=1000,
+        item_size=4096,
+        run_id=run_id
+    )
+
+    # Start worker processes
+    workers = []
+    for i in range(3):
+        p = Process(target=worker, args=("tasks", run_id, i))
+        p.start()
+        workers.append(p)
+
+    # Add tasks to queue
+    for i in range(20):
+        work_queue.put(f"task_{i}")
+
+    # Signal workers to stop
+    for _ in workers:
+        work_queue.put("STOP")
+
+    # Wait for workers to finish
+    for p in workers:
+        p.join()
+
+    print(f"Final queue size: {work_queue.qsize()}")
+
+    # Cleanup
+    work_queue.unlink()
+
+if __name__ == "__main__":
+    main()
+```
+
+## API Reference
+
+### `SharedMemoryQueue(name, create=False, capacity=1000, item_size=4096, run_id="", lock=None)`
+
+Creates or connects to a shared memory queue.
+
+**Parameters:**
+- `name` (str): Queue identifier
+- `create` (bool): Whether to create new queue (`True`) or connect to existing (`False`)
+- `capacity` (int): Maximum number of items the queue can hold
+- `item_size` (int): Maximum size of each serialized item in bytes
+- `run_id` (str): Optional run identifier for namespacing queues
+- `lock` (Any): Ignored - `SharedMemoryLock` is always used internally
+
+**Methods:**
+- `put(item, block=True, timeout=None)`: Add item to queue
+- `get(block=True, timeout=None)`: Remove and return item from queue
+- `put_nowait(item)`: Add item without blocking (raises `Full` if queue is full)
+- `get_nowait()`: Get item without blocking (raises `Empty` if queue is empty)
+- `put_batch(items)`: Add multiple items atomically
+- `empty()`: Check if queue is empty
+- `full()`: Check if queue is full
+- `qsize()`: Get approximate queue size
+- `get_metrics()`: Return queue metrics dictionary
+- `close()`: Close connection to shared memory
+- `unlink()`: Delete the shared memory segment (call from creator process)
+
+**Exceptions:**
+- `Empty`: Raised when queue is empty and non-blocking operation requested
+- `Full`: Raised when queue is full and non-blocking operation requested
+
+## Advanced Usage
+
+### Batch Operations
+
+For high-throughput scenarios, use `put_batch()` to add multiple items atomically:
+
+```python
+queue = SharedMemoryQueue("batch_queue", create=True, capacity=10000)
+
+# Process items in batches for better performance
+batch = []
+for i in range(100):
+    batch.append(f"item_{i}")
+    if len(batch) >= 50:
+        queue.put_batch(batch)
+        batch = []
+
+# Don't forget remaining items
+if batch:
+    queue.put_batch(batch)
+```
+
+### Queue Metrics
+
+Monitor queue performance with built-in metrics:
+
+```python
+queue = SharedMemoryQueue("monitored_queue", create=True, capacity=1000)
+
+# Add some items
+for i in range(100):
+    queue.put(f"item_{i}")
+
+metrics = queue.get_metrics()
+print(f"Queue size: {metrics['queue_size']}")
+print(f"Capacity: {metrics['queue_capacity']}")
+print(f"Utilization: {metrics['queue_utilization_percent']:.1f}%")
+print(f"Item size limit: {metrics['queue_item_size_bytes']} bytes")
+```
+
+### Cross-Process Communication
+
+```python
+# Producer process
+def producer(queue_name, run_id):
+    queue = SharedMemoryQueue(queue_name, create=False, run_id=run_id)
+    for i in range(1000):
+        queue.put({"data": f"item_{i}", "timestamp": time.time()})
+    queue.put(None)  # Sentinel value
+
+# Consumer process
+def consumer(queue_name, run_id):
+    queue = SharedMemoryQueue(queue_name, create=False, run_id=run_id)
+    results = []
+    while True:
+        item = queue.get()
+        if item is None:  # Sentinel value
+            break
+        results.append(item)
+    return results
+
+# Main process
+run_id = "data_pipeline"
+queue = SharedMemoryQueue("data_queue", create=True, capacity=2000, run_id=run_id)
+
+# Start processes
+p1 = Process(target=producer, args=("data_queue", run_id))
+p2 = Process(target=consumer, args=("data_queue", run_id))
+
+p1.start()
+p2.start()
+
+p1.join()
+p2.join()
+
+queue.unlink()
+```
+
+### Timeout Handling
+
+```python
+queue = SharedMemoryQueue("timeout_queue", create=True, capacity=10)
+
+try:
+    # Will wait up to 5 seconds for space
+    queue.put("item", block=True, timeout=5.0)
+except Full:
+    print("Queue was full for 5 seconds")
+
+try:
+    # Will wait up to 2 seconds for an item
+    item = queue.get(block=True, timeout=2.0)
+except Empty:
+    print("No items available for 2 seconds")
+```
+
+## Performance Characteristics
+
+- **Ring buffer**: O(1) put/get operations
+- **Zero-copy**: Items stored directly in shared memory
+- **Atomic operations**: Thread and process-safe without Python GIL limitations
+- **Batch operations**: Minimize lock contention for high-throughput scenarios
+- **Configurable capacity**: Balance memory usage vs queue depth
+
+## Use Cases
+
+### 1. High-Throughput Task Distribution
+
+```python
+# Distribute work across multiple worker processes
+task_queue = SharedMemoryQueue("tasks", create=True, capacity=50000, item_size=8192)
+
+# Producer adds tasks
+tasks = generate_large_task_list()
+task_queue.put_batch(tasks)  # Efficient batch insertion
+
+# Multiple workers consume tasks
+def worker():
+    while True:
+        try:
+            task = task_queue.get_nowait()
+            process_task(task)
+        except Empty:
+            time.sleep(0.01)
+```
+
+### 2. Inter-Service Communication
+
+```python
+# Service A publishes events
+event_queue = SharedMemoryQueue("events", create=True, capacity=10000, run_id="system")
+
+# Service B subscribes to events
+event_queue = SharedMemoryQueue("events", create=False, run_id="system")
+while True:
+    event = event_queue.get()
+    handle_event(event)
+```
+
+### 3. Real-Time Data Processing
+
+```python
+# High-frequency data ingestion
+data_queue = SharedMemoryQueue("sensor_data", create=True, capacity=100000, item_size=1024)
+
+# Sensor data producer
+def collect_sensor_data():
+    batch = []
+    while True:
+        reading = read_sensor()
+        batch.append(reading)
+        if len(batch) >= 100:  # Batch for efficiency
+            data_queue.put_batch(batch)
+            batch = []
+```
+
+## Implementation Details
+
+- Uses `multiprocessing.shared_memory` for the underlying storage
+- Ring buffer layout: `[head][tail][capacity][item_size][data_slots...]`
+- Items serialized with `cloudpickle` for maximum compatibility
+- Synchronized with `shared_memory_lock` for cross-process safety
+- Header stores 64-bit integers for head/tail pointers
+- Each slot prefixed with 32-bit length for variable-size items
+
+## Development
+
+```bash
+# Install dependencies
+uv sync
+
+# Run tests
+uv run pytest
+
+# Run tests with coverage
+uv run pytest --cov=shared_queue
+
+# Type checking
+uv run mypy .
+
+# Linting
+uv run ruff check .
+```
+
+## License
+
+MIT

shared_multiprocess_queue-0.1.1.dist-info/RECORD

@@ -0,0 +1,6 @@
+shared_queue/__init__.py,sha256=n7Ol-4ftiIarTO4vZpBMV-V1aAkxcjpHtpqzVFDMJPY,204
+shared_queue/queue.py,sha256=tt2IA5jckivnOMY06Vrs4fEUDyCBWbJV-H4dnIi4470,24354
+shared_multiprocess_queue-0.1.1.dist-info/METADATA,sha256=lQdVctNFcI1DgCL1Yo1u5qWCZ58jA97FwlCgtX1NxeU,8969
+shared_multiprocess_queue-0.1.1.dist-info/WHEEL,sha256=_zCd3N1l69ArxyTb8rzEoP9TpbYXkqRFSNOD5OuxnTs,91
+shared_multiprocess_queue-0.1.1.dist-info/top_level.txt,sha256=jyPOO-FyMPquhn_5tpGt4WS9V0Aa0JjoYgt-aAH8eyw,13
+shared_multiprocess_queue-0.1.1.dist-info/RECORD,,

shared_multiprocess_queue-0.1.1.dist-info/WHEEL

@@ -0,0 +1,5 @@
+Wheel-Version: 1.0
+Generator: setuptools (80.9.0)
+Root-Is-Purelib: true
+Tag: py3-none-any
+

shared_multiprocess_queue-0.1.1.dist-info/top_level.txt

@@ -0,0 +1 @@
+shared_queue

shared_queue/__init__.py

@@ -0,0 +1,6 @@
+"""A shared-memory based multiprocessing queue for cross-process communication"""
+
+from .queue import SharedMemoryQueue, Empty, Full
+
+__version__ = "0.1.1"
+__all__ = ["SharedMemoryQueue", "Empty", "Full"]

shared_queue/queue.py

@@ -0,0 +1,628 @@
+"""
+Shared memory queue implementation for cross-process communication.
+
+This queue can be created dynamically at runtime and accessed by name
+from any process without requiring pickling or process restarts.
+"""
+
+from __future__ import annotations
+
+import asyncio
+import struct
+import time
+from multiprocessing import shared_memory
+from typing import Any, Protocol, Awaitable
+
+import cloudpickle
+
+from shared_memory_lock import SharedMemoryLock
+
+# Queue layout in shared memory:
+# [head: u64][tail: u64][capacity: u64][item_size: u64][data...]
+_HEADER_FMT = "!QQQQ"  # head, tail, capacity, item_size
+_HEADER_SIZE = struct.calcsize(_HEADER_FMT)
+
+
+class Observable(Protocol):
+    """Protocol for objects that can provide metrics."""
+
+    def get_metrics(self) -> dict[str, Any]:
+        """Return metrics dictionary."""
+        ...
+
+
+class SharedMemoryQueue:
+    """
+    A FIFO queue backed by shared memory that can be created and accessed by name.
+
+    This queue uses a ring buffer design with atomic head/tail pointers.
+    Items are serialized using cloudpickle and stored in fixed-size slots.
+
+    Usage:
+        # Process A creates a queue
+        queue = SharedMemoryQueue(name="work_queue", create=True, capacity=1000, run_id="app1")
+        queue.put({"task": "data"})
+
+        # Process B connects to it by name
+        queue = SharedMemoryQueue(name="work_queue", create=False, run_id="app1")
+        item = queue.get()
+    """
+
+    def __init__(
+        self,
+        name: str,
+        create: bool = False,
+        capacity: int = 1000,
+        item_size: int = 4096,
+        run_id: str = "",
+        lock: Any = None,
+    ):
+        """
+        Initialize a shared memory queue.
+
+        Args:
+            name: Queue name (used to identify the shared memory segment)
+            create: Whether to create a new queue or connect to existing
+            capacity: Maximum number of items the queue can hold
+            item_size: Maximum size of each serialized item in bytes
+            run_id: Run identifier for namespacing
+            lock: Ignored - SharedMemoryLock is always used
+        """
+        self.name = name
+        self.run_id = run_id
+        self.capacity = capacity
+        self.item_size = item_size
+
+        # Create or connect to shared memory lock
+        self._lock = SharedMemoryLock(
+            name=f"{name}_queue_lock",
+            create=create,
+            run_id=run_id
+        )
+
+        # Calculate total size needed
+        data_size = capacity * item_size
+        total_size = _HEADER_SIZE + data_size
+
+        # Create or connect to shared memory
+        shm_name = f"{run_id}-{name}-queue" if run_id else f"{name}-queue"
+        self._shm = shared_memory.SharedMemory(name=shm_name, create=create, size=total_size)
+
+        if create:
+            # Initialize header: head=0, tail=0, capacity, item_size
+            self._write_header(0, 0, capacity, item_size)
+        else:
+            # Read existing header to get queue parameters
+            head, tail, cap, isize = self._read_header()
+            self.capacity = cap
+            self.item_size = isize
+
+    def _read_header(self) -> tuple[int, int, int, int]:
+        """Read header: (head, tail, capacity, item_size)"""
+        header_bytes = bytes(self._shm.buf[:_HEADER_SIZE])
+        head, tail, capacity, item_size = struct.unpack(_HEADER_FMT, header_bytes)
+        return int(head), int(tail), int(capacity), int(item_size)
+
+    def _write_header(self, head: int, tail: int, capacity: int, item_size: int) -> None:
+        """Write header: (head, tail, capacity, item_size)"""
+        header_bytes = struct.pack(_HEADER_FMT, head, tail, capacity, item_size)
+        self._shm.buf[:_HEADER_SIZE] = header_bytes
+
+    def _get_slot_offset(self, index: int) -> int:
+        """Calculate the offset for a given slot index"""
+        slot_index = index % self.capacity
+        return _HEADER_SIZE + (slot_index * self.item_size)
+
+    def put(self, item: Any, block: bool = True, timeout: float | None = None) -> None:
+        """
+        Put an item into the queue.
+
+        Args:
+            item: The item to enqueue
+            block: Whether to block if queue is full
+            timeout: Maximum time to wait if blocking
+
+        Raises:
+            Full: If queue is full and block=False or timeout expires
+        """
+        start_time = time.time() if timeout else None
+
+        while True:
+            if self._lock:
+                with self._lock:
+                    head, tail, capacity, item_size = self._read_header()
+
+                    # Check if queue is full
+                    if (tail - head) >= capacity:
+                        if not block:
+                            raise Full("Queue is full")
+                        if timeout and (time.time() - start_time) >= timeout:
+                            raise Full("Queue is full (timeout)")
+                        # Release lock before sleeping
+                    else:
+                        # Serialize the item
+                        try:
+                            item_bytes = cloudpickle.dumps(item)
+                        except Exception as e:
+                            raise ValueError(f"Failed to serialize item: {e}")
+
+                        if len(item_bytes) > item_size:
+                            raise ValueError(f"Item too large: {len(item_bytes)} bytes > {item_size} bytes")
+
+                        # Write item to tail slot
+                        slot_offset = self._get_slot_offset(tail)
+                        # Write item size first (4 bytes), then item data
+                        size_bytes = struct.pack("!I", len(item_bytes))
+                        self._shm.buf[slot_offset : slot_offset + 4] = size_bytes
+                        self._shm.buf[slot_offset + 4 : slot_offset + 4 + len(item_bytes)] = item_bytes
+
+                        # Update tail
+                        self._write_header(head, tail + 1, capacity, item_size)
+                        return
+            else:
+                head, tail, capacity, item_size = self._read_header()
+
+                # Check if queue is full
+                if (tail - head) >= capacity:
+                    if not block:
+                        raise Full("Queue is full")
+                else:
+                    # Serialize the item
+                    try:
+                        item_bytes = cloudpickle.dumps(item)
+                    except Exception as e:
+                        raise ValueError(f"Failed to serialize item: {e}")
+
+                    if len(item_bytes) > item_size:
+                        raise ValueError(f"Item too large: {len(item_bytes)} bytes > {item_size} bytes")
+
+                    # Write item to tail slot
+                    slot_offset = self._get_slot_offset(tail)
+                    # Write item size first (4 bytes), then item data
+                    size_bytes = struct.pack("!I", len(item_bytes))
+                    self._shm.buf[slot_offset : slot_offset + 4] = size_bytes
+                    self._shm.buf[slot_offset + 4 : slot_offset + 4 + len(item_bytes)] = item_bytes
+
+                    # Update tail
+                    self._write_header(head, tail + 1, capacity, item_size)
+                    return
+
+            # Queue was full - check timeout and sleep
+            if timeout and (time.time() - start_time) > timeout:
+                raise Full("Queue is full (timeout)")
+            time.sleep(0.001)  # Brief sleep before retry
+
+    def get(self, block: bool = True, timeout: float | None = None) -> Any:
+        """
+        Get an item from the queue.
+
+        Args:
+            block: Whether to block if queue is empty
+            timeout: Maximum time to wait if blocking
+
+        Returns:
+            The dequeued item
+
+        Raises:
+            Empty: If queue is empty and block=False or timeout expires
+        """
+        start_time = time.time() if timeout else None
+
+        while True:
+            if self._lock:
+                with self._lock:
+                    head, tail, capacity, item_size = self._read_header()
+
+                    # Check if queue is empty
+                    if head >= tail:
+                        if not block:
+                            raise Empty("Queue is empty")
+                        if timeout and (time.time() - start_time) >= timeout:
+                            raise Empty("Queue is empty (timeout)")
+                        # Release lock before sleeping
+                    else:
+                        # Read item from head slot
+                        slot_offset = self._get_slot_offset(head)
+                        # Read item size first
+                        size_bytes = bytes(self._shm.buf[slot_offset : slot_offset + 4])
+                        item_len = struct.unpack("!I", size_bytes)[0]
+
+                        # Read item data
+                        item_bytes = bytes(self._shm.buf[slot_offset + 4 : slot_offset + 4 + item_len])
+
+                        # Update head
+                        self._write_header(head + 1, tail, capacity, item_size)
+
+                        # Deserialize and return
+                        try:
+                            return cloudpickle.loads(item_bytes)
+                        except Exception as e:
+                            raise ValueError(f"Failed to deserialize item: {e}")
+            else:
+                head, tail, capacity, item_size = self._read_header()
+
+                # Check if queue is empty
+                if head >= tail:
+                    if not block:
+                        raise Empty("Queue is empty")
+                else:
+                    # Read item from head slot
+                    slot_offset = self._get_slot_offset(head)
+                    # Read item size first
+                    size_bytes = bytes(self._shm.buf[slot_offset : slot_offset + 4])
+                    item_len = struct.unpack("!I", size_bytes)[0]
+
+                    # Read item data
+                    item_bytes = bytes(self._shm.buf[slot_offset + 4 : slot_offset + 4 + item_len])
+
+                    # Update head
+                    self._write_header(head + 1, tail, capacity, item_size)
+
+                    # Deserialize and return
+                    try:
+                        return cloudpickle.loads(item_bytes)
+                    except Exception as e:
+                        raise ValueError(f"Failed to deserialize item: {e}")
+
+            # Queue was empty - check timeout and sleep
+            if timeout and (time.time() - start_time) > timeout:
+                raise Empty("Queue is empty (timeout)")
+            time.sleep(0.001)  # Brief sleep before retry
+
+    def put_nowait(self, item: Any) -> None:
+        """Put an item without blocking"""
+        self.put(item, block=False)
+
+    async def put_async(self, item: Any, timeout: float | None = None) -> None:
+        """
+        Async put an item into the queue.
+
+        Args:
+            item: The item to enqueue
+            timeout: Maximum time to wait
+
+        Raises:
+            Full: If queue is full and timeout expires
+        """
+        start_time = time.time() if timeout else None
+
+        while True:
+            if self._lock:
+                async with self._lock:
+                    head, tail, capacity, item_size = self._read_header()
+
+                    # Check if queue is full
+                    if (tail - head) >= capacity:
+                        if timeout and (time.time() - start_time) >= timeout:
+                            raise Full("Queue is full (timeout)")
+                        # Release lock before sleeping
+                    else:
+                        # Serialize the item
+                        try:
+                            item_bytes = cloudpickle.dumps(item)
+                        except Exception as e:
+                            raise ValueError(f"Failed to serialize item: {e}")
+
+                        if len(item_bytes) > item_size:
+                            raise ValueError(f"Item too large: {len(item_bytes)} bytes > {item_size} bytes")
+
+                        # Write item to tail slot
+                        slot_offset = self._get_slot_offset(tail)
+                        # Write item size first (4 bytes), then item data
+                        size_bytes = struct.pack("!I", len(item_bytes))
+                        self._shm.buf[slot_offset : slot_offset + 4] = size_bytes
+                        self._shm.buf[slot_offset + 4 : slot_offset + 4 + len(item_bytes)] = item_bytes
+
+                        # Update tail
+                        self._write_header(head, tail + 1, capacity, item_size)
+                        return
+            else:
+                # No lock mode
+                head, tail, capacity, item_size = self._read_header()
+
+                # Check if queue is full
+                if (tail - head) >= capacity:
+                    if timeout and (time.time() - start_time) >= timeout:
+                        raise Full("Queue is full (timeout)")
+                else:
+                    # Serialize the item
+                    try:
+                        item_bytes = cloudpickle.dumps(item)
+                    except Exception as e:
+                        raise ValueError(f"Failed to serialize item: {e}")
+
+                    if len(item_bytes) > item_size:
+                        raise ValueError(f"Item too large: {len(item_bytes)} bytes > {item_size} bytes")
+
+                    # Write item to tail slot
+                    slot_offset = self._get_slot_offset(tail)
+                    # Write item size first (4 bytes), then item data
+                    size_bytes = struct.pack("!I", len(item_bytes))
+                    self._shm.buf[slot_offset : slot_offset + 4] = size_bytes
+                    self._shm.buf[slot_offset + 4 : slot_offset + 4 + len(item_bytes)] = item_bytes
+
+                    # Update tail
+                    self._write_header(head, tail + 1, capacity, item_size)
+                    return
+
+            # Queue was full - check timeout and sleep
+            if timeout and (time.time() - start_time) > timeout:
+                raise Full("Queue is full (timeout)")
+            await asyncio.sleep(0.001)  # Brief async sleep before retry
+
+    async def get_async(self, timeout: float | None = None) -> Any:
+        """
+        Async get an item from the queue.
+
+        Args:
+            timeout: Maximum time to wait
+
+        Returns:
+            The dequeued item
+
+        Raises:
+            Empty: If queue is empty and timeout expires
+        """
+        start_time = time.time() if timeout else None
+
+        while True:
+            if self._lock:
+                async with self._lock:
+                    head, tail, capacity, item_size = self._read_header()
+
+                    # Check if queue is empty
+                    if head >= tail:
+                        if timeout and (time.time() - start_time) >= timeout:
+                            raise Empty("Queue is empty (timeout)")
+                        # Release lock before sleeping
+                    else:
+                        # Read item from head slot
+                        slot_offset = self._get_slot_offset(head)
+                        # Read item size first
+                        size_bytes = bytes(self._shm.buf[slot_offset : slot_offset + 4])
+                        item_len = struct.unpack("!I", size_bytes)[0]
+
+                        # Read item data
+                        item_bytes = bytes(self._shm.buf[slot_offset + 4 : slot_offset + 4 + item_len])
+
+                        # Update head
+                        self._write_header(head + 1, tail, capacity, item_size)
+
+                        # Deserialize and return
+                        try:
+                            return cloudpickle.loads(item_bytes)
+                        except Exception as e:
+                            raise ValueError(f"Failed to deserialize item: {e}")
+            else:
+                # No lock mode
+                head, tail, capacity, item_size = self._read_header()
+
+                # Check if queue is empty
+                if head >= tail:
+                    if timeout and (time.time() - start_time) >= timeout:
+                        raise Empty("Queue is empty (timeout)")
+                else:
+                    # Read item from head slot
+                    slot_offset = self._get_slot_offset(head)
+                    # Read item size first
+                    size_bytes = bytes(self._shm.buf[slot_offset : slot_offset + 4])
+                    item_len = struct.unpack("!I", size_bytes)[0]
+
+                    # Read item data
+                    item_bytes = bytes(self._shm.buf[slot_offset + 4 : slot_offset + 4 + item_len])
+
+                    # Update head
+                    self._write_header(head + 1, tail, capacity, item_size)
+
+                    # Deserialize and return
+                    try:
+                        return cloudpickle.loads(item_bytes)
+                    except Exception as e:
+                        raise ValueError(f"Failed to deserialize item: {e}")
+
+            # Queue was empty - check timeout and sleep
+            if timeout and (time.time() - start_time) > timeout:
+                raise Empty("Queue is empty (timeout)")
+            await asyncio.sleep(0.001)  # Brief async sleep before retry
+
+
+    def put_batch(self, items: list[Any]) -> None:
+        """
+        Put multiple items atomically under a single lock acquisition.
+
+        Writes all items directly to the ring buffer in one operation.
+
+        Args:
+            items: List of items to enqueue
+
+        Raises:
+            Full: If queue doesn't have space for all items
+        """
+        if not items:
+            return
+
+        if self._lock:
+            with self._lock:
+                # Read current state
+                head, tail, capacity, item_size = self._read_header()
+
+                # Check if we have space for all items
+                current_size = tail - head
+                if current_size + len(items) > capacity:
+                    raise Full(f"Queue full: {current_size}/{capacity}, cannot add {len(items)} items")
+
+                # Serialize all items first
+                serialized_items = []
+                for item in items:
+                    try:
+                        item_bytes = cloudpickle.dumps(item)
+                    except Exception as e:
+                        raise ValueError(f"Failed to serialize item: {e}")
+
+                    if len(item_bytes) > item_size:
+                        raise ValueError(f"Item too large: {len(item_bytes)} bytes > {item_size} bytes")
+
+                    serialized_items.append(item_bytes)
+
+                # Write all items to consecutive slots
+                for item_bytes in serialized_items:
+                    slot_offset = self._get_slot_offset(tail)
+                    # Write item size first (4 bytes), then item data
+                    size_bytes = struct.pack("!I", len(item_bytes))
+                    self._shm.buf[slot_offset : slot_offset + 4] = size_bytes
+                    self._shm.buf[slot_offset + 4 : slot_offset + 4 + len(item_bytes)] = item_bytes
+                    tail += 1
+
+                # Update header once with new tail
+                self._write_header(head, tail, capacity, item_size)
+        else:
+            # Read current state
+            head, tail, capacity, item_size = self._read_header()
+
+            # Check if we have space for all items
+            current_size = tail - head
+            if current_size + len(items) > capacity:
+                raise Full(f"Queue full: {current_size}/{capacity}, cannot add {len(items)} items")
+
+            # Serialize all items first
+            serialized_items = []
+            for item in items:
+                try:
+                    item_bytes = cloudpickle.dumps(item)
+                except Exception as e:
+                    raise ValueError(f"Failed to serialize item: {e}")
+
+                if len(item_bytes) > item_size:
+                    raise ValueError(f"Item too large: {len(item_bytes)} bytes > {item_size} bytes")
+
+                serialized_items.append(item_bytes)
+
+            # Write all items to consecutive slots
+            for item_bytes in serialized_items:
+                slot_offset = self._get_slot_offset(tail)
+                # Write item size first (4 bytes), then item data
+                size_bytes = struct.pack("!I", len(item_bytes))
+                self._shm.buf[slot_offset : slot_offset + 4] = size_bytes
+                self._shm.buf[slot_offset + 4 : slot_offset + 4 + len(item_bytes)] = item_bytes
+                tail += 1
+
+            # Update header once with new tail
+            self._write_header(head, tail, capacity, item_size)
+
+    def get_nowait(self) -> Any:
+        """Get an item without blocking"""
+        return self.get(block=False)
+
+    def empty(self) -> bool:
+        """Check if queue is empty"""
+        head, tail, _, _ = self._read_header()
+        return head >= tail
+
+    def full(self) -> bool:
+        """Check if queue is full"""
+        head, tail, capacity, _ = self._read_header()
+        return (tail - head) >= capacity
+
+    def qsize(self) -> int:
+        """Get approximate queue size"""
+        head, tail, _, _ = self._read_header()
+        return max(0, tail - head)
+
+    def close(self) -> None:
+        """Close the shared memory connection"""
+        if hasattr(self, "_shm") and self._shm:
+            try:
+                self._shm.close()
+            except Exception:
+                pass
+        if hasattr(self, "_lock") and self._lock:
+            try:
+                self._lock.close()
+            except Exception:
+                pass
+
+    def unlink(self) -> None:
+        """Unlink (delete) the shared memory segment"""
+        self.close()
+        try:
+            shm_name = f"{self.run_id}-{self.name}-queue" if self.run_id else f"{self.name}-queue"
+            temp_shm = shared_memory.SharedMemory(name=shm_name)
+            temp_shm.unlink()
+            temp_shm.close()
+        except FileNotFoundError:
+            pass
+        except Exception:
+            pass
+
+        # Unlink the lock
+        if hasattr(self, "_lock") and self._lock:
+            try:
+                self._lock.unlink()
+            except Exception:
+                pass
+
+    def get_metrics(self) -> dict[str, Any]:
+        """
+        Returns metrics for this queue.
+
+        Implements Observable protocol.
+        """
+        try:
+            head, tail, capacity, item_size = self._read_header()
+            size = max(0, tail - head)
+            return {
+                "queue_size": size,
+                "queue_capacity": capacity,
+                "queue_utilization_percent": (size / capacity * 100) if capacity > 0 else 0.0,
+                "queue_item_size_bytes": item_size,
+            }
+        except Exception:
+            return {}
+
+    def __getstate__(self) -> dict[str, Any]:
+        """
+        Prepare for pickling - return only connection info, not the lock.
+
+        The lock cannot be pickled and must be created fresh in each process.
+        """
+        return {
+            "name": self.name,
+            "run_id": self.run_id,
+            "capacity": self.capacity,
+            "item_size": self.item_size,
+        }
+
+    def __setstate__(self, state: dict[str, Any]) -> None:
+        """
+        Reconnect after unpickling in a new process.
+
+        Reconnects to the same shared memory lock and data segment.
+        """
+        self.name = state["name"]
+        self.run_id = state["run_id"]
+        self.capacity = state["capacity"]
+        self.item_size = state["item_size"]
+
+        # Reconnect to the same shared memory lock
+        self._lock = SharedMemoryLock(
+            name=f"{self.name}_queue_lock",
+            create=False,
+            run_id=self.run_id
+        )
+
+        # Reconnect to existing shared memory
+        shm_name = f"{self.run_id}-{self.name}-queue" if self.run_id else f"{self.name}-queue"
+        total_size = _HEADER_SIZE + (self.capacity * self.item_size)
+        self._shm = shared_memory.SharedMemory(name=shm_name, create=False, size=total_size)
+
+
+class Empty(Exception):
+    """Raised when queue is empty"""
+
+    pass
+
+
+class Full(Exception):
+    """Raised when queue is full"""
+
+    pass