py-sharedmemory 4.1.1__tar.gz → 4.2.1__tar.gz

{py_sharedmemory-4.1.1 → py_sharedmemory-4.2.1}/CHANGELOG.md

@@ -1,5 +1,37 @@
 # Changelog
 
+## 4.2.1
+
+- `put` copies large `bytes` and `bytearray` objects (64 KiB and up) straight into shared
+  memory, wherever they appear in a message. It used to build the complete pickle in
+  memory first: an extra copy of the payload that briefly needed up to 1.5 times its size
+  (48 MiB for a 32 MiB `bytearray`). Strided NumPy arrays, which NumPy pickles as `bytes`,
+  skip that copy too. The message format is unchanged.
+- Resizing a `bytearray` while its message is being pickled, from a `__reduce__` method,
+  now raises `BufferError` instead of risking a corrupted message.
+
+## 4.2.0
+
+- Reuse received shared-memory blocks instead of freeing them. Writing into fresh pages
+  cost far more than the copy itself, so large `put` calls are now much faster. A block
+  is reused for a message that fills at least half of it, or for any message that fits a
+  block of up to 1 MiB. The sender keeps at most 8 blocks and 256 MiB for reuse, never
+  more blocks than the capacity allows, and frees them on `close()`, so peak memory is
+  unchanged. On Linux and macOS at most 8 unread blocks stay mapped (two file
+  descriptors each).
+- Acknowledgements carry a per-message sequence number, so an unknown or repeated
+  acknowledgement can never release a reused block that holds a newer message.
+- Fix receivers hanging forever in `close()` or at process exit after reading many
+  messages that a closed sender had already freed: such messages are no longer
+  acknowledged, so the acknowledgement pipe can no longer fill up.
+- Fix a Windows race where a message freed while a receiver attached to it raised a
+  confusing `UnpicklingError`; it now raises the usual "freed before it was received"
+  `FileNotFoundError`.
+- Received NumPy arrays are always aligned. Arrays that followed an odd-sized buffer in the
+  same message used to arrive unaligned, which slows NumPy and breaks libraries that need
+  aligned memory. Each array now owns its memory, so keeping one no longer keeps the
+  whole message alive.
+
 ## 4.1.1
 
 - Unread messages no longer hold file descriptors on Linux and macOS: the sender unmaps

{py_sharedmemory-4.1.1/py_sharedmemory.egg-info → py_sharedmemory-4.2.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: py-sharedmemory
-Version: 4.1.1
+Version: 4.2.1
 Summary: Fast multiprocessing.Queue alternative that moves large Python objects and NumPy arrays through shared memory
 Author: Tobias Würth
 License-Expression: MIT
@@ -68,10 +68,11 @@ images, and simulation states arrive up to 2× faster, and transfers too large f
 
 - **Familiar API:** `put` / `get` with `block`, `timeout`, `put_nowait` / `get_nowait`,
   `queue.Full`, and `queue.Empty`.
-- **Any picklable object:** NumPy arrays and other buffers use pickle protocol 5
-  out-of-band buffers, so they are copied straight into shared memory.
-- **Bounded memory:** a capacity limit applies backpressure, and each shared-memory
-  block is freed once its message is received.
+- **Any picklable object:** NumPy arrays, other protocol-5 buffers, and large `bytes` and
+  `bytearray` objects are copied straight into shared memory, without an intermediate
+  pickle copy.
+- **Bounded memory:** a capacity limit applies backpressure, and received blocks are
+  reused for later messages, so shared memory stays within `capacity × message size`.
 - **One producer, many consumers:** consumer processes and threads share one receiver.
 - **Portable:** Linux, macOS, and Windows; `spawn`, `fork`, and `forkserver`; Python 3.11–3.15.
 - **Lightweight:** no dependencies, fully typed (PEP 561), and 100% test coverage.
@@ -131,8 +132,8 @@ arrays, image or video frames, RL rollouts, large `bytes`, or big nested objects
 helps where large `multiprocessing.Queue` transfers cause heavy pipe I/O or uneven
 throughput, which is especially noticeable on Windows.
 
-Prefer `multiprocessing.Queue` for small messages. Below about 100 kB, creating and
-freeing a shared-memory block (about 0.09 ms per message) costs more than the pipe.
+Prefer `multiprocessing.Queue` for small messages. Below about 100 kB, the fixed cost of
+mapping a shared-memory block for every message outweighs the pipe.
 
 To let several processes read and write **the same** array in place, use
 [`multiprocessing.shared_memory`](https://docs.python.org/3/library/multiprocessing.shared_memory.html)
@@ -155,20 +156,25 @@ Core i9-14900K, 191.8 GiB RAM, Windows 11, Python 3.12.10, NumPy 2.5.3, measured
 2026-10-08. Contiguous float32 arrays, `spawn`, capacity 2, median of 5 runs excluding
 startup and warmup; up to 1,000 messages or 512 MiB per run, at least one message.
 1 GB = 10⁹ bytes. See the [raw results](https://github.com/tobiaswuerth/python_shared_memory_queue/blob/main/benchmarks/results/windows-numpy.json).
-These numbers show transfer speed; I/O load and stability depend on the workload.
+These numbers show transfer speed; I/O load and stability depend on the workload. They
+predate version 4.2, which reuses received blocks and so no longer pays to fault in fresh
+pages on every `put`.
 
 ## How it works
 
-1. `put()` pickles the object with protocol 5. Large buffers, such as NumPy array data,
-   stay out-of-band, and everything is copied into one new shared-memory block.
-2. Only the block's name and buffer sizes travel through a `multiprocessing.Queue`.
+1. `put()` pickles the object with protocol 5. Large buffers, such as NumPy array data and
+   `bytes` or `bytearray` objects of 64 KiB or more, never enter the pickle; everything is copied once,
+   from the original objects into one shared-memory block: a block from an earlier,
+   received message when one fits, otherwise a new one.
+2. Only the block's name, buffer sizes, and a sequence number travel through a
+   `multiprocessing.Queue`.
 3. `get()` attaches to the block, copies the data into local memory, unpickles it, and
    sends an acknowledgement back.
-4. A background thread in the producer frees the block when the acknowledgement
-   arrives. `capacity` limits how many blocks can be outstanding at once.
+4. A background thread in the producer makes the block available for reuse when the
+   acknowledgement arrives. `capacity` limits how many messages can be outstanding.
 
 Received objects never point into shared memory, so they stay valid after the block is
-freed and can be modified freely.
+reused or freed and can be modified freely.
 
 ## Usage guide
 
@@ -188,9 +194,15 @@ consumer = ctx.Process(target=consume, args=(receiver,))
 
 ### Capacity and backpressure
 
-`capacity` counts messages that were sent but not yet received and freed. When it is
-reached, `put()` blocks, so a fast producer cannot exhaust memory. Shared-memory use is
-at most about `capacity × message size`. `capacity=0` or `None` means unbounded.
+`capacity` counts messages that were sent but not yet received. When it is reached,
+`put()` blocks, so a fast producer cannot exhaust memory. Shared-memory use is at most
+about `capacity × message size`. `capacity=0` or `None` means unbounded.
+
+Received blocks are reused rather than freed, because writing into memory that is
+already mapped is much faster than faulting in fresh pages. A block is reused for a
+message that fills at least half of it, or for any message that fits a block of up to
+1 MiB. The sender keeps at most 8 such blocks and 256 MiB, never more blocks than the
+capacity allows, and frees them on `close()`. Blocks over 256 MiB are freed once received.
 
 ### Timeouts and non-blocking calls
 
@@ -241,7 +253,9 @@ such as `None` is a simple way to tell consumers that the stream has ended.
 Arrays keep their dtype, shape, and writability. Read-only arrays stay read-only, and
 writable ones arrive as writable local copies. C- and Fortran-contiguous arrays keep
 their layout and are copied straight into shared memory without an intermediate pickle
-copy; other views, such as strided slices, arrive as contiguous copies.
+copy; other views, such as strided slices, arrive as contiguous copies. Each received
+array has its own aligned memory, so keeping one array does not keep the rest of the
+message alive.
 
 ### Errors
 
@@ -250,8 +264,9 @@ copy; other views, such as strided slices, arrive as contiguous copies.
 | `queue.Full` | `put` found no capacity within `timeout` or with `block=False`. |
 | `queue.Empty` | `get` found no message within `timeout` or with `block=False`. |
 | `BrokenPipeError` | The endpoint is closed, or the producer's acknowledgement channel failed. |
+| `FileNotFoundError` | `get` received a message that the producer freed by closing before it was read. |
 | `RuntimeError` | A second process tries to produce, or a used sender is passed to another process. |
-| Pickling errors | The object cannot be pickled or unpickled. Both endpoints stay usable, and failed messages are still freed. |
+| Pickling errors | The object cannot be pickled or unpickled. Both endpoints stay usable, and failed messages are still released. |
 | `RuntimeWarning` (warning) | A `with sender:` block or the producing process ended normally and discarded unacknowledged messages. |
 
 ## API reference
@@ -266,10 +281,10 @@ for the underlying queues; it defaults to `multiprocessing.get_context()`.
 
 | Member | Description |
 | --- | --- |
-| `put(obj, block=True, timeout=None)` | Copy `obj` into a new shared-memory block and send it. |
+| `put(obj, block=True, timeout=None)` | Copy `obj` into a shared-memory block and send it. |
 | `put_nowait(obj)` | `put(obj, block=False)`. |
-| `wait_for_all_ack(timeout=None) -> bool` | Wait until every message is received and freed; `False` on timeout. |
-| `close()` | Free unread messages silently and release resources. Runs automatically at normal process exit. |
+| `wait_for_all_ack(timeout=None) -> bool` | Wait until every message is received and its block released; `False` on timeout. |
+| `close()` | Free unread messages and reusable blocks silently and release resources. Runs automatically at normal process exit. |
 | `closed` | Whether the sender is closed. |
 | `with sender:` | Closes the sender on exit; warns if a clean exit discards unacknowledged messages. |
 
@@ -308,8 +323,9 @@ the last handle closes, so nothing can leak after a crash.
 producer closed or exited before this message was read. Call `wait_for_all_ack()` before
 closing the sender; the sender also warns when this happens.
 
-**`OSError: [Errno 24] Too many open files` with many unread messages:** fixed in 4.1.1,
-where unread messages no longer hold file descriptors.
+**`OSError: [Errno 24] Too many open files` with many unread messages:** fixed in 4.1.1.
+On Linux and macOS the sender keeps at most 8 blocks mapped for reuse, two file
+descriptors each; other unread messages hold none.
 
 **`RuntimeError: A shared-memory pair supports one producing process`:** a second
 process called `put`. Create a separate pair for each producer process.

{py_sharedmemory-4.1.1 → py_sharedmemory-4.2.1}/README.md

@@ -18,10 +18,11 @@ images, and simulation states arrive up to 2× faster, and transfers too large f
 
 - **Familiar API:** `put` / `get` with `block`, `timeout`, `put_nowait` / `get_nowait`,
   `queue.Full`, and `queue.Empty`.
-- **Any picklable object:** NumPy arrays and other buffers use pickle protocol 5
-  out-of-band buffers, so they are copied straight into shared memory.
-- **Bounded memory:** a capacity limit applies backpressure, and each shared-memory
-  block is freed once its message is received.
+- **Any picklable object:** NumPy arrays, other protocol-5 buffers, and large `bytes` and
+  `bytearray` objects are copied straight into shared memory, without an intermediate
+  pickle copy.
+- **Bounded memory:** a capacity limit applies backpressure, and received blocks are
+  reused for later messages, so shared memory stays within `capacity × message size`.
 - **One producer, many consumers:** consumer processes and threads share one receiver.
 - **Portable:** Linux, macOS, and Windows; `spawn`, `fork`, and `forkserver`; Python 3.11–3.15.
 - **Lightweight:** no dependencies, fully typed (PEP 561), and 100% test coverage.
@@ -81,8 +82,8 @@ arrays, image or video frames, RL rollouts, large `bytes`, or big nested objects
 helps where large `multiprocessing.Queue` transfers cause heavy pipe I/O or uneven
 throughput, which is especially noticeable on Windows.
 
-Prefer `multiprocessing.Queue` for small messages. Below about 100 kB, creating and
-freeing a shared-memory block (about 0.09 ms per message) costs more than the pipe.
+Prefer `multiprocessing.Queue` for small messages. Below about 100 kB, the fixed cost of
+mapping a shared-memory block for every message outweighs the pipe.
 
 To let several processes read and write **the same** array in place, use
 [`multiprocessing.shared_memory`](https://docs.python.org/3/library/multiprocessing.shared_memory.html)
@@ -105,20 +106,25 @@ Core i9-14900K, 191.8 GiB RAM, Windows 11, Python 3.12.10, NumPy 2.5.3, measured
 2026-10-08. Contiguous float32 arrays, `spawn`, capacity 2, median of 5 runs excluding
 startup and warmup; up to 1,000 messages or 512 MiB per run, at least one message.
 1 GB = 10⁹ bytes. See the [raw results](https://github.com/tobiaswuerth/python_shared_memory_queue/blob/main/benchmarks/results/windows-numpy.json).
-These numbers show transfer speed; I/O load and stability depend on the workload.
+These numbers show transfer speed; I/O load and stability depend on the workload. They
+predate version 4.2, which reuses received blocks and so no longer pays to fault in fresh
+pages on every `put`.
 
 ## How it works
 
-1. `put()` pickles the object with protocol 5. Large buffers, such as NumPy array data,
-   stay out-of-band, and everything is copied into one new shared-memory block.
-2. Only the block's name and buffer sizes travel through a `multiprocessing.Queue`.
+1. `put()` pickles the object with protocol 5. Large buffers, such as NumPy array data and
+   `bytes` or `bytearray` objects of 64 KiB or more, never enter the pickle; everything is copied once,
+   from the original objects into one shared-memory block: a block from an earlier,
+   received message when one fits, otherwise a new one.
+2. Only the block's name, buffer sizes, and a sequence number travel through a
+   `multiprocessing.Queue`.
 3. `get()` attaches to the block, copies the data into local memory, unpickles it, and
    sends an acknowledgement back.
-4. A background thread in the producer frees the block when the acknowledgement
-   arrives. `capacity` limits how many blocks can be outstanding at once.
+4. A background thread in the producer makes the block available for reuse when the
+   acknowledgement arrives. `capacity` limits how many messages can be outstanding.
 
 Received objects never point into shared memory, so they stay valid after the block is
-freed and can be modified freely.
+reused or freed and can be modified freely.
 
 ## Usage guide
 
@@ -138,9 +144,15 @@ consumer = ctx.Process(target=consume, args=(receiver,))
 
 ### Capacity and backpressure
 
-`capacity` counts messages that were sent but not yet received and freed. When it is
-reached, `put()` blocks, so a fast producer cannot exhaust memory. Shared-memory use is
-at most about `capacity × message size`. `capacity=0` or `None` means unbounded.
+`capacity` counts messages that were sent but not yet received. When it is reached,
+`put()` blocks, so a fast producer cannot exhaust memory. Shared-memory use is at most
+about `capacity × message size`. `capacity=0` or `None` means unbounded.
+
+Received blocks are reused rather than freed, because writing into memory that is
+already mapped is much faster than faulting in fresh pages. A block is reused for a
+message that fills at least half of it, or for any message that fits a block of up to
+1 MiB. The sender keeps at most 8 such blocks and 256 MiB, never more blocks than the
+capacity allows, and frees them on `close()`. Blocks over 256 MiB are freed once received.
 
 ### Timeouts and non-blocking calls
 
@@ -191,7 +203,9 @@ such as `None` is a simple way to tell consumers that the stream has ended.
 Arrays keep their dtype, shape, and writability. Read-only arrays stay read-only, and
 writable ones arrive as writable local copies. C- and Fortran-contiguous arrays keep
 their layout and are copied straight into shared memory without an intermediate pickle
-copy; other views, such as strided slices, arrive as contiguous copies.
+copy; other views, such as strided slices, arrive as contiguous copies. Each received
+array has its own aligned memory, so keeping one array does not keep the rest of the
+message alive.
 
 ### Errors
 
@@ -200,8 +214,9 @@ copy; other views, such as strided slices, arrive as contiguous copies.
 | `queue.Full` | `put` found no capacity within `timeout` or with `block=False`. |
 | `queue.Empty` | `get` found no message within `timeout` or with `block=False`. |
 | `BrokenPipeError` | The endpoint is closed, or the producer's acknowledgement channel failed. |
+| `FileNotFoundError` | `get` received a message that the producer freed by closing before it was read. |
 | `RuntimeError` | A second process tries to produce, or a used sender is passed to another process. |
-| Pickling errors | The object cannot be pickled or unpickled. Both endpoints stay usable, and failed messages are still freed. |
+| Pickling errors | The object cannot be pickled or unpickled. Both endpoints stay usable, and failed messages are still released. |
 | `RuntimeWarning` (warning) | A `with sender:` block or the producing process ended normally and discarded unacknowledged messages. |
 
 ## API reference
@@ -216,10 +231,10 @@ for the underlying queues; it defaults to `multiprocessing.get_context()`.
 
 | Member | Description |
 | --- | --- |
-| `put(obj, block=True, timeout=None)` | Copy `obj` into a new shared-memory block and send it. |
+| `put(obj, block=True, timeout=None)` | Copy `obj` into a shared-memory block and send it. |
 | `put_nowait(obj)` | `put(obj, block=False)`. |
-| `wait_for_all_ack(timeout=None) -> bool` | Wait until every message is received and freed; `False` on timeout. |
-| `close()` | Free unread messages silently and release resources. Runs automatically at normal process exit. |
+| `wait_for_all_ack(timeout=None) -> bool` | Wait until every message is received and its block released; `False` on timeout. |
+| `close()` | Free unread messages and reusable blocks silently and release resources. Runs automatically at normal process exit. |
 | `closed` | Whether the sender is closed. |
 | `with sender:` | Closes the sender on exit; warns if a clean exit discards unacknowledged messages. |
 
@@ -258,8 +273,9 @@ the last handle closes, so nothing can leak after a crash.
 producer closed or exited before this message was read. Call `wait_for_all_ack()` before
 closing the sender; the sender also warns when this happens.
 
-**`OSError: [Errno 24] Too many open files` with many unread messages:** fixed in 4.1.1,
-where unread messages no longer hold file descriptors.
+**`OSError: [Errno 24] Too many open files` with many unread messages:** fixed in 4.1.1.
+On Linux and macOS the sender keeps at most 8 blocks mapped for reuse, two file
+descriptors each; other unread messages hold none.
 
 **`RuntimeError: A shared-memory pair supports one producing process`:** a second
 process called `put`. Create a separate pair for each producer process.

{py_sharedmemory-4.1.1 → py_sharedmemory-4.2.1}/py_sharedmemory/__init__.py

@@ -25,5 +25,5 @@ from .factory import create_shared_memory_pair
 from .receiver import SharedMemoryReceiver
 from .sender import SharedMemorySender
 
-__version__ = "4.1.1"
+__version__ = "4.2.1"
 __all__ = ["SharedMemoryReceiver", "SharedMemorySender", "__version__", "create_shared_memory_pair"]

py_sharedmemory-4.2.1/py_sharedmemory/convert.py

@@ -0,0 +1,155 @@
+"""Serialize protocol-5 pickle data through a single shared-memory block."""
+
+import errno
+import pickle
+import sys
+from collections.abc import Callable
+from contextlib import suppress
+from multiprocessing.shared_memory import SharedMemory
+from typing import Any, NamedTuple, cast
+
+# Receivers never unlink. Tracking their attachments would leak tracker entries and
+# warnings when a receiver does not share the sender's resource tracker.
+_ATTACH_OPTIONS: dict[str, Any] = {"track": False} if sys.version_info >= (3, 13) else {}
+
+
+class SharedMemoryMessage(NamedTuple):
+    name: str
+    buffer_sizes: tuple[int, ...]
+    # Unique per sender. Blocks are reused, so acknowledgements name the message, not the block.
+    sequence: int = 0
+
+    @property
+    def size(self) -> int:
+        return sum(self.buffer_sizes)
+
+
+def create_shared_memory(size: int) -> SharedMemory:
+    return SharedMemory(create=True, size=size)
+
+
+def release_shared_memory(allocation: SharedMemory) -> None:
+    """Release both the mapping and its name, including on error paths."""
+    try:
+        allocation.close()
+    finally:
+        with suppress(FileNotFoundError):
+            allocation.unlink()
+
+
+class _PickleChunks(list[memoryview]):
+    """Pickle output kept as views of what pickle wrote rather than one joined copy."""
+
+    def write(self, data: Any) -> None:
+        # Pickle passes bytes and bytearrays of 64 KiB or more as the objects themselves.
+        # A view also stops a bytearray from being resized before it is copied, which would
+        # leave its length header in the stream wrong.
+        self.append(memoryview(data))
+
+
+def serialize(
+    data: Any,
+    sequence: int = 0,
+    allocate: Callable[[int], SharedMemory] = create_shared_memory,
+) -> tuple[SharedMemory, SharedMemoryMessage]:
+    """Copy a pickle and its out-of-band buffers into an owned block of at least that size.
+
+    Large bytes, bytearrays, and contiguous arrays are copied once, straight from the
+    original objects into shared memory. `allocate` returns a new or reused block; this
+    function frees it if copying fails.
+    """
+    chunks = _PickleChunks()
+    buffers: list[memoryview] = []
+    allocation = None
+
+    def append_buffer(buffer: pickle.PickleBuffer) -> None:
+        buffers.append(buffer.raw())
+
+    try:
+        pickle.Pickler(chunks, protocol=5, buffer_callback=append_buffer).dump(data)
+        pickle_size = sum(chunk.nbytes for chunk in chunks)
+        buffer_sizes = (pickle_size, *(buffer.nbytes for buffer in buffers))
+        allocation = allocate(sum(buffer_sizes))
+        shared = cast(memoryview, allocation.buf)
+        offset = 0
+        for view in (*chunks, *buffers):
+            shared[offset : offset + view.nbytes] = view
+            offset += view.nbytes
+        return allocation, SharedMemoryMessage(allocation.name, buffer_sizes, sequence)
+    except BaseException:
+        if allocation is not None:
+            release_shared_memory(allocation)
+        raise
+    finally:
+        for view in (*chunks, *buffers):
+            view.release()
+
+
+def deserialize(message: SharedMemoryMessage) -> Any:
+    """Unpickle with owned local buffers and always close the shared mapping."""
+    return load(message, attach(message))
+
+
+def attach(message: SharedMemoryMessage) -> SharedMemory:
+    """Map the block that valid metadata names, or explain why it no longer exists."""
+    _validate_message(message)
+    try:
+        allocation = SharedMemory(name=message.name, **_ATTACH_OPTIONS)
+    except FileNotFoundError as error:
+        raise _freed_block_error(message) from error
+    # Attaching on Windows is not atomic: a name freed in between is recreated as a new,
+    # zero-filled block. Every pickle starts with a nonzero opcode.
+    if cast(memoryview, allocation.buf)[0] == 0:
+        allocation.close()
+        raise _freed_block_error(message)
+    return allocation
+
+
+def _freed_block_error(message: SharedMemoryMessage) -> FileNotFoundError:
+    return FileNotFoundError(
+        errno.ENOENT,
+        "Shared-memory block was freed before it was received; the sender probably "
+        "closed without calling wait_for_all_ack()",
+        message.name,
+    )
+
+
+def load(message: SharedMemoryMessage, allocation: SharedMemory) -> Any:
+    """Unpickle an attached block with owned local buffers, then close the mapping."""
+    try:
+        if message.size > allocation.size:
+            raise ValueError("Shared-memory block is smaller than its metadata")
+        shared = cast(memoryview, allocation.buf)
+        pickle_size = message.buffer_sizes[0]
+        buffers = _copy_buffers(shared, pickle_size, message.buffer_sizes[1:])
+        with shared[:pickle_size] as pickle_data:
+            return pickle.loads(pickle_data, buffers=buffers)
+    finally:
+        allocation.close()
+
+
+def _validate_message(message: SharedMemoryMessage) -> None:
+    if not isinstance(message, SharedMemoryMessage) or not isinstance(message.name, str):
+        raise ValueError("Invalid shared-memory metadata")
+    sizes = message.buffer_sizes
+    if (
+        not message.name
+        or type(message.sequence) is not int
+        or message.sequence < 0
+        or not isinstance(sizes, tuple)
+        or not sizes
+        or any(type(size) is not int or size < 0 for size in sizes)
+        or sizes[0] == 0
+    ):
+        raise ValueError("Invalid shared-memory metadata")
+
+
+def _copy_buffers(shared: memoryview, offset: int, sizes: tuple[int, ...]) -> list[memoryview]:
+    # A separate writable copy per buffer starts aligned for any dtype and is freed on its own,
+    # so a small array cannot keep a large neighbour alive.
+    buffers = []
+    for size in sizes:
+        with shared[offset : offset + size] as shared_buffer:
+            buffers.append(memoryview(bytearray(shared_buffer)))
+        offset += size
+    return buffers

{py_sharedmemory-4.1.1 → py_sharedmemory-4.2.1}/py_sharedmemory/receiver.py

@@ -6,18 +6,18 @@ from multiprocessing.queues import Queue
 from types import TracebackType
 from typing import Any, Self
 
-from .convert import SharedMemoryMessage, deserialize
+from .convert import SharedMemoryMessage, attach, load
 
 
 class SharedMemoryReceiver:
     """Consuming endpoint of a pair; create it with `create_shared_memory_pair`.
 
     Several threads and processes may share one receiver; each message is delivered to
-    exactly one of them. Every message is acknowledged, even when decoding fails, so
-    the sender can free its memory.
+    exactly one of them. Every received block is acknowledged, even when decoding fails,
+    so the sender can free its memory.
     """
 
-    def __init__(self, data_queue: Queue[SharedMemoryMessage], ack_queue: Queue[str]) -> None:
+    def __init__(self, data_queue: Queue[SharedMemoryMessage], ack_queue: Queue[int]) -> None:
         self._data_queue = data_queue
         self._ack_queue = ack_queue
         self._closed = False
@@ -35,24 +35,26 @@ class SharedMemoryReceiver:
             timeout: Maximum seconds to wait; `None` waits forever.
 
         Returns:
-            The object, which stays valid after the shared memory is freed. Arrays keep
+            The object, which stays valid after the block is reused or freed. Arrays keep
             their original writability.
 
         Raises:
             queue.Empty: No message arrived in time.
             BrokenPipeError: The receiver is closed.
+            FileNotFoundError: The sender freed the message before it was received.
 
         Unpickling errors propagate unchanged; the message is still acknowledged.
         """
         if self._closed:
             raise BrokenPipeError("Receiver is closed")
         message = self._data_queue.get(block, timeout)
+        # A block that is gone was already freed by its sender, which expects no acknowledgement.
+        allocation = attach(message)
         try:
-            return deserialize(message)
+            return load(message, allocation)
         finally:
             # Failed decoding must still release the sender's memory and capacity.
-            if isinstance(message, SharedMemoryMessage):
-                self._ack_queue.put_nowait(message.name)
+            self._ack_queue.put_nowait(message.sequence)
 
     def get_nowait(self) -> Any:
         """Equivalent to `get(block=False)`."""