py-sharedmemory 4.1.0__tar.gz → 4.2.0__tar.gz

py_sharedmemory-4.2.0/CHANGELOG.md

@@ -0,0 +1,73 @@
+# Changelog
+
+## 4.2.0
+
+- Reuse received shared-memory blocks instead of freeing them. Writing into fresh pages
+  cost far more than the copy itself, so large `put` calls are now much faster. A block
+  is reused for a message that fills at least half of it, or for any message that fits a
+  block of up to 1 MiB. The sender keeps at most 8 blocks and 256 MiB for reuse, never
+  more blocks than the capacity allows, and frees them on `close()`, so peak memory is
+  unchanged. On Linux and macOS at most 8 unread blocks stay mapped (two file
+  descriptors each).
+- Acknowledgements carry a per-message sequence number, so an unknown or repeated
+  acknowledgement can never release a reused block that holds a newer message.
+- Fix receivers hanging forever in `close()` or at process exit after reading many
+  messages that a closed sender had already freed: such messages are no longer
+  acknowledged, so the acknowledgement pipe can no longer fill up.
+- Fix a Windows race where a message freed while a receiver attached to it raised a
+  confusing `UnpicklingError`; it now raises the usual "freed before it was received"
+  `FileNotFoundError`.
+- Received NumPy arrays are always aligned. Arrays that followed an odd-sized buffer in the
+  same message used to arrive unaligned, which slows NumPy and breaks libraries that need
+  aligned memory. Each array now owns its memory, so keeping one no longer keeps the
+  whole message alive.
+
+## 4.1.1
+
+- Unread messages no longer hold file descriptors on Linux and macOS: the sender unmaps
+  each block right after writing it. Unbounded pairs previously failed with
+  `Too many open files` after about 120 unread messages under macOS's default limit.
+- Cap outstanding messages at the metadata queue's limit (32,767 on macOS), so a blocking
+  `put` on an unbounded pair waits instead of raising `queue.Full`.
+- Explain a freed block in the receiver's `FileNotFoundError`, which happens when the
+  sender closes before a message is read.
+
+## 4.1.0
+
+- Warn with a `RuntimeWarning` when leaving a `with sender:` block or exiting the
+  producing process discards unacknowledged messages, so a forgotten
+  `wait_for_all_ack()` no longer loses data silently. Explicit `close()` calls and
+  exits caused by exceptions stay silent, and closing still never blocks.
+- Collect garbage between tests to avoid Python 3.12 resource-tracker reentrancy errors,
+  and add a stress test that checks every byte across consumer processes.
+
+## 4.0.0
+
+- **Breaking:** the import package is renamed from `memory` to `py_sharedmemory` to match the
+  distribution name. `memory` clashed with the unrelated `memory` package on PyPI and with
+  projects' own `memory` modules. Upgrade by replacing `from memory import ...` with
+  `from py_sharedmemory import ...`; the API is unchanged and no alias is kept.
+- Rewrite the README as a user guide with an API reference, performance table,
+  Docker, macOS, open-file-limit, and troubleshooting notes, and absolute links so
+  images render on PyPI.
+- Document every public class and method; add NumPy streaming and worker-pool examples.
+- Add contributing and security guides, issue forms, and a pull request template.
+
+## 3.0.1
+
+- Fix resource-tracker leaks on Linux and macOS: receivers no longer register blocks they
+  attach to, so `fork` workers no longer accumulate tracker entries or print
+  "leaked shared_memory" warnings at exit.
+- Ship type hints (`py.typed`), checked with strict mypy; expose `memory.__version__`.
+- Support and test Python 3.15; test Linux, macOS, and Windows for every Python version.
+- Require 100% branch coverage; add pre-commit hooks, Dependabot, and a trusted-publishing
+  release workflow.
+
+## 3.0.0
+
+- Python 3.11+, standard-library-only runtime.
+- Fix acknowledgement races, capacity leaks, failed-put recovery, and process cleanup.
+- Preserve NumPy mutability; remove redundant copies and retained pickle storage.
+- Simplify core ownership and sender flow; use immutable metadata and recover from partial startup.
+- Add explicit close, context managers, context selection, and timed acknowledgement waits.
+- Replace legacy tests and benchmarks; add platform CI and recorded machine results.

{py_sharedmemory-4.1.0/py_sharedmemory.egg-info → py_sharedmemory-4.2.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: py-sharedmemory
-Version: 4.1.0
+Version: 4.2.0
 Summary: Fast multiprocessing.Queue alternative that moves large Python objects and NumPy arrays through shared memory
 Author: Tobias Würth
 License-Expression: MIT
@@ -70,8 +70,8 @@ images, and simulation states arrive up to 2× faster, and transfers too large f
   `queue.Full`, and `queue.Empty`.
 - **Any picklable object:** NumPy arrays and other buffers use pickle protocol 5
   out-of-band buffers, so they are copied straight into shared memory.
-- **Bounded memory:** a capacity limit applies backpressure, and each shared-memory
-  block is freed once its message is received.
+- **Bounded memory:** a capacity limit applies backpressure, and received blocks are
+  reused for later messages, so shared memory stays within `capacity × message size`.
 - **One producer, many consumers:** consumer processes and threads share one receiver.
 - **Portable:** Linux, macOS, and Windows; `spawn`, `fork`, and `forkserver`; Python 3.11–3.15.
 - **Lightweight:** no dependencies, fully typed (PEP 561), and 100% test coverage.
@@ -131,8 +131,8 @@ arrays, image or video frames, RL rollouts, large `bytes`, or big nested objects
 helps where large `multiprocessing.Queue` transfers cause heavy pipe I/O or uneven
 throughput, which is especially noticeable on Windows.
 
-Prefer `multiprocessing.Queue` for small messages. Below about 100 kB, creating and
-freeing a shared-memory block (about 0.09 ms per message) costs more than the pipe.
+Prefer `multiprocessing.Queue` for small messages. Below about 100 kB, the fixed cost of
+mapping a shared-memory block for every message outweighs the pipe.
 
 To let several processes read and write **the same** array in place, use
 [`multiprocessing.shared_memory`](https://docs.python.org/3/library/multiprocessing.shared_memory.html)
@@ -155,20 +155,24 @@ Core i9-14900K, 191.8 GiB RAM, Windows 11, Python 3.12.10, NumPy 2.5.3, measured
 2026-10-08. Contiguous float32 arrays, `spawn`, capacity 2, median of 5 runs excluding
 startup and warmup; up to 1,000 messages or 512 MiB per run, at least one message.
 1 GB = 10⁹ bytes. See the [raw results](https://github.com/tobiaswuerth/python_shared_memory_queue/blob/main/benchmarks/results/windows-numpy.json).
-These numbers show transfer speed; I/O load and stability depend on the workload.
+These numbers show transfer speed; I/O load and stability depend on the workload. They
+predate version 4.2, which reuses received blocks and so no longer pays to fault in fresh
+pages on every `put`.
 
 ## How it works
 
 1. `put()` pickles the object with protocol 5. Large buffers, such as NumPy array data,
-   stay out-of-band, and everything is copied into one new shared-memory block.
-2. Only the block's name and buffer sizes travel through a `multiprocessing.Queue`.
+   stay out-of-band, and everything is copied into one shared-memory block: a block from
+   an earlier, received message when one fits, otherwise a new one.
+2. Only the block's name, buffer sizes, and a sequence number travel through a
+   `multiprocessing.Queue`.
 3. `get()` attaches to the block, copies the data into local memory, unpickles it, and
    sends an acknowledgement back.
-4. A background thread in the producer frees the block when the acknowledgement
-   arrives. `capacity` limits how many blocks can be outstanding at once.
+4. A background thread in the producer makes the block available for reuse when the
+   acknowledgement arrives. `capacity` limits how many messages can be outstanding.
 
 Received objects never point into shared memory, so they stay valid after the block is
-freed and can be modified freely.
+reused or freed and can be modified freely.
 
 ## Usage guide
 
@@ -188,11 +192,15 @@ consumer = ctx.Process(target=consume, args=(receiver,))
 
 ### Capacity and backpressure
 
-`capacity` counts messages that were sent but not yet received and freed. When it is
-reached, `put()` blocks, so a fast producer cannot exhaust memory. Shared-memory use is
-at most about `capacity × message size`. `capacity=0` or `None` means unbounded; prefer
-a limit, because unread messages also hold open files on Linux and macOS (see the
-platform notes below).
+`capacity` counts messages that were sent but not yet received. When it is reached,
+`put()` blocks, so a fast producer cannot exhaust memory. Shared-memory use is at most
+about `capacity × message size`. `capacity=0` or `None` means unbounded.
+
+Received blocks are reused rather than freed, because writing into memory that is
+already mapped is much faster than faulting in fresh pages. A block is reused for a
+message that fills at least half of it, or for any message that fits a block of up to
+1 MiB. The sender keeps at most 8 such blocks and 256 MiB, never more blocks than the
+capacity allows, and frees them on `close()`. Blocks over 256 MiB are freed once received.
 
 ### Timeouts and non-blocking calls
 
@@ -243,7 +251,9 @@ such as `None` is a simple way to tell consumers that the stream has ended.
 Arrays keep their dtype, shape, and writability. Read-only arrays stay read-only, and
 writable ones arrive as writable local copies. C- and Fortran-contiguous arrays keep
 their layout and are copied straight into shared memory without an intermediate pickle
-copy; other views, such as strided slices, arrive as contiguous copies.
+copy; other views, such as strided slices, arrive as contiguous copies. Each received
+array has its own aligned memory, so keeping one array does not keep the rest of the
+message alive.
 
 ### Errors
 
@@ -252,8 +262,9 @@ copy; other views, such as strided slices, arrive as contiguous copies.
 | `queue.Full` | `put` found no capacity within `timeout` or with `block=False`. |
 | `queue.Empty` | `get` found no message within `timeout` or with `block=False`. |
 | `BrokenPipeError` | The endpoint is closed, or the producer's acknowledgement channel failed. |
+| `FileNotFoundError` | `get` received a message that the producer freed by closing before it was read. |
 | `RuntimeError` | A second process tries to produce, or a used sender is passed to another process. |
-| Pickling errors | The object cannot be pickled or unpickled. Both endpoints stay usable, and failed messages are still freed. |
+| Pickling errors | The object cannot be pickled or unpickled. Both endpoints stay usable, and failed messages are still released. |
 | `RuntimeWarning` (warning) | A `with sender:` block or the producing process ended normally and discarded unacknowledged messages. |
 
 ## API reference
@@ -268,10 +279,10 @@ for the underlying queues; it defaults to `multiprocessing.get_context()`.
 
 | Member | Description |
 | --- | --- |
-| `put(obj, block=True, timeout=None)` | Copy `obj` into a new shared-memory block and send it. |
+| `put(obj, block=True, timeout=None)` | Copy `obj` into a shared-memory block and send it. |
 | `put_nowait(obj)` | `put(obj, block=False)`. |
-| `wait_for_all_ack(timeout=None) -> bool` | Wait until every message is received and freed; `False` on timeout. |
-| `close()` | Free unread messages silently and release resources. Runs automatically at normal process exit. |
+| `wait_for_all_ack(timeout=None) -> bool` | Wait until every message is received and its block released; `False` on timeout. |
+| `close()` | Free unread messages and reusable blocks silently and release resources. Runs automatically at normal process exit. |
 | `closed` | Whether the sender is closed. |
 | `with sender:` | Closes the sender on exit; warns if a clean exit discards unacknowledged messages. |
 
@@ -296,13 +307,9 @@ with `OSError: [Errno 28] No space left on device` or crash with `SIGBUS`.
 **Linux:** outside containers, `/dev/shm` usually defaults to half of physical RAM.
 
 **macOS:** fully supported and tested on Apple Silicon. The default start method is
-`spawn`, so guard the entry point with `if __name__ == "__main__":`.
-
-**Open-file limit (Linux and macOS):** each unread message keeps two file descriptors
-open in the producer until it is received. macOS allows 256 per process by default and
-Linux usually 1,024, so an unbounded pair fails with `OSError: [Errno 24] Too many open
-files` after roughly 120 or 500 unread messages. Set a `capacity` well below that, or
-raise the limit with `ulimit -n`. The sender stays usable after this error.
+`spawn`, so guard the entry point with `if __name__ == "__main__":`. An unbounded pair
+holds at most 32,767 unread messages, the limit of macOS semaphores; beyond that `put`
+waits or raises `queue.Full` like a full bounded pair.
 
 **Windows:** shared memory is backed by the page file, and each block disappears once
 the last handle closes, so nothing can leak after a crash.
@@ -310,6 +317,14 @@ the last handle closes, so nothing can leak after a crash.
 **"resource_tracker: There appear to be N leaked shared_memory objects":** fixed in
 3.0.1. Upgrade if you see this warning with the `fork` start method.
 
+**`FileNotFoundError: Shared-memory block was freed before it was received`:** the
+producer closed or exited before this message was read. Call `wait_for_all_ack()` before
+closing the sender; the sender also warns when this happens.
+
+**`OSError: [Errno 24] Too many open files` with many unread messages:** fixed in 4.1.1.
+On Linux and macOS the sender keeps at most 8 blocks mapped for reuse, two file
+descriptors each; other unread messages hold none.
+
 **`RuntimeError: A shared-memory pair supports one producing process`:** a second
 process called `put`. Create a separate pair for each producer process.
 

{py_sharedmemory-4.1.0 → py_sharedmemory-4.2.0}/README.md

@@ -20,8 +20,8 @@ images, and simulation states arrive up to 2× faster, and transfers too large f
   `queue.Full`, and `queue.Empty`.
 - **Any picklable object:** NumPy arrays and other buffers use pickle protocol 5
   out-of-band buffers, so they are copied straight into shared memory.
-- **Bounded memory:** a capacity limit applies backpressure, and each shared-memory
-  block is freed once its message is received.
+- **Bounded memory:** a capacity limit applies backpressure, and received blocks are
+  reused for later messages, so shared memory stays within `capacity × message size`.
 - **One producer, many consumers:** consumer processes and threads share one receiver.
 - **Portable:** Linux, macOS, and Windows; `spawn`, `fork`, and `forkserver`; Python 3.11–3.15.
 - **Lightweight:** no dependencies, fully typed (PEP 561), and 100% test coverage.
@@ -81,8 +81,8 @@ arrays, image or video frames, RL rollouts, large `bytes`, or big nested objects
 helps where large `multiprocessing.Queue` transfers cause heavy pipe I/O or uneven
 throughput, which is especially noticeable on Windows.
 
-Prefer `multiprocessing.Queue` for small messages. Below about 100 kB, creating and
-freeing a shared-memory block (about 0.09 ms per message) costs more than the pipe.
+Prefer `multiprocessing.Queue` for small messages. Below about 100 kB, the fixed cost of
+mapping a shared-memory block for every message outweighs the pipe.
 
 To let several processes read and write **the same** array in place, use
 [`multiprocessing.shared_memory`](https://docs.python.org/3/library/multiprocessing.shared_memory.html)
@@ -105,20 +105,24 @@ Core i9-14900K, 191.8 GiB RAM, Windows 11, Python 3.12.10, NumPy 2.5.3, measured
 2026-10-08. Contiguous float32 arrays, `spawn`, capacity 2, median of 5 runs excluding
 startup and warmup; up to 1,000 messages or 512 MiB per run, at least one message.
 1 GB = 10⁹ bytes. See the [raw results](https://github.com/tobiaswuerth/python_shared_memory_queue/blob/main/benchmarks/results/windows-numpy.json).
-These numbers show transfer speed; I/O load and stability depend on the workload.
+These numbers show transfer speed; I/O load and stability depend on the workload. They
+predate version 4.2, which reuses received blocks and so no longer pays to fault in fresh
+pages on every `put`.
 
 ## How it works
 
 1. `put()` pickles the object with protocol 5. Large buffers, such as NumPy array data,
-   stay out-of-band, and everything is copied into one new shared-memory block.
-2. Only the block's name and buffer sizes travel through a `multiprocessing.Queue`.
+   stay out-of-band, and everything is copied into one shared-memory block: a block from
+   an earlier, received message when one fits, otherwise a new one.
+2. Only the block's name, buffer sizes, and a sequence number travel through a
+   `multiprocessing.Queue`.
 3. `get()` attaches to the block, copies the data into local memory, unpickles it, and
    sends an acknowledgement back.
-4. A background thread in the producer frees the block when the acknowledgement
-   arrives. `capacity` limits how many blocks can be outstanding at once.
+4. A background thread in the producer makes the block available for reuse when the
+   acknowledgement arrives. `capacity` limits how many messages can be outstanding.
 
 Received objects never point into shared memory, so they stay valid after the block is
-freed and can be modified freely.
+reused or freed and can be modified freely.
 
 ## Usage guide
 
@@ -138,11 +142,15 @@ consumer = ctx.Process(target=consume, args=(receiver,))
 
 ### Capacity and backpressure
 
-`capacity` counts messages that were sent but not yet received and freed. When it is
-reached, `put()` blocks, so a fast producer cannot exhaust memory. Shared-memory use is
-at most about `capacity × message size`. `capacity=0` or `None` means unbounded; prefer
-a limit, because unread messages also hold open files on Linux and macOS (see the
-platform notes below).
+`capacity` counts messages that were sent but not yet received. When it is reached,
+`put()` blocks, so a fast producer cannot exhaust memory. Shared-memory use is at most
+about `capacity × message size`. `capacity=0` or `None` means unbounded.
+
+Received blocks are reused rather than freed, because writing into memory that is
+already mapped is much faster than faulting in fresh pages. A block is reused for a
+message that fills at least half of it, or for any message that fits a block of up to
+1 MiB. The sender keeps at most 8 such blocks and 256 MiB, never more blocks than the
+capacity allows, and frees them on `close()`. Blocks over 256 MiB are freed once received.
 
 ### Timeouts and non-blocking calls
 
@@ -193,7 +201,9 @@ such as `None` is a simple way to tell consumers that the stream has ended.
 Arrays keep their dtype, shape, and writability. Read-only arrays stay read-only, and
 writable ones arrive as writable local copies. C- and Fortran-contiguous arrays keep
 their layout and are copied straight into shared memory without an intermediate pickle
-copy; other views, such as strided slices, arrive as contiguous copies.
+copy; other views, such as strided slices, arrive as contiguous copies. Each received
+array has its own aligned memory, so keeping one array does not keep the rest of the
+message alive.
 
 ### Errors
 
@@ -202,8 +212,9 @@ copy; other views, such as strided slices, arrive as contiguous copies.
 | `queue.Full` | `put` found no capacity within `timeout` or with `block=False`. |
 | `queue.Empty` | `get` found no message within `timeout` or with `block=False`. |
 | `BrokenPipeError` | The endpoint is closed, or the producer's acknowledgement channel failed. |
+| `FileNotFoundError` | `get` received a message that the producer freed by closing before it was read. |
 | `RuntimeError` | A second process tries to produce, or a used sender is passed to another process. |
-| Pickling errors | The object cannot be pickled or unpickled. Both endpoints stay usable, and failed messages are still freed. |
+| Pickling errors | The object cannot be pickled or unpickled. Both endpoints stay usable, and failed messages are still released. |
 | `RuntimeWarning` (warning) | A `with sender:` block or the producing process ended normally and discarded unacknowledged messages. |
 
 ## API reference
@@ -218,10 +229,10 @@ for the underlying queues; it defaults to `multiprocessing.get_context()`.
 
 | Member | Description |
 | --- | --- |
-| `put(obj, block=True, timeout=None)` | Copy `obj` into a new shared-memory block and send it. |
+| `put(obj, block=True, timeout=None)` | Copy `obj` into a shared-memory block and send it. |
 | `put_nowait(obj)` | `put(obj, block=False)`. |
-| `wait_for_all_ack(timeout=None) -> bool` | Wait until every message is received and freed; `False` on timeout. |
-| `close()` | Free unread messages silently and release resources. Runs automatically at normal process exit. |
+| `wait_for_all_ack(timeout=None) -> bool` | Wait until every message is received and its block released; `False` on timeout. |
+| `close()` | Free unread messages and reusable blocks silently and release resources. Runs automatically at normal process exit. |
 | `closed` | Whether the sender is closed. |
 | `with sender:` | Closes the sender on exit; warns if a clean exit discards unacknowledged messages. |
 
@@ -246,13 +257,9 @@ with `OSError: [Errno 28] No space left on device` or crash with `SIGBUS`.
 **Linux:** outside containers, `/dev/shm` usually defaults to half of physical RAM.
 
 **macOS:** fully supported and tested on Apple Silicon. The default start method is
-`spawn`, so guard the entry point with `if __name__ == "__main__":`.
-
-**Open-file limit (Linux and macOS):** each unread message keeps two file descriptors
-open in the producer until it is received. macOS allows 256 per process by default and
-Linux usually 1,024, so an unbounded pair fails with `OSError: [Errno 24] Too many open
-files` after roughly 120 or 500 unread messages. Set a `capacity` well below that, or
-raise the limit with `ulimit -n`. The sender stays usable after this error.
+`spawn`, so guard the entry point with `if __name__ == "__main__":`. An unbounded pair
+holds at most 32,767 unread messages, the limit of macOS semaphores; beyond that `put`
+waits or raises `queue.Full` like a full bounded pair.
 
 **Windows:** shared memory is backed by the page file, and each block disappears once
 the last handle closes, so nothing can leak after a crash.
@@ -260,6 +267,14 @@ the last handle closes, so nothing can leak after a crash.
 **"resource_tracker: There appear to be N leaked shared_memory objects":** fixed in
 3.0.1. Upgrade if you see this warning with the `fork` start method.
 
+**`FileNotFoundError: Shared-memory block was freed before it was received`:** the
+producer closed or exited before this message was read. Call `wait_for_all_ack()` before
+closing the sender; the sender also warns when this happens.
+
+**`OSError: [Errno 24] Too many open files` with many unread messages:** fixed in 4.1.1.
+On Linux and macOS the sender keeps at most 8 blocks mapped for reuse, two file
+descriptors each; other unread messages hold none.
+
 **`RuntimeError: A shared-memory pair supports one producing process`:** a second
 process called `put`. Create a separate pair for each producer process.
 

{py_sharedmemory-4.1.0 → py_sharedmemory-4.2.0}/py_sharedmemory/__init__.py

@@ -25,5 +25,5 @@ from .factory import create_shared_memory_pair
 from .receiver import SharedMemoryReceiver
 from .sender import SharedMemorySender
 
-__version__ = "4.1.0"
+__version__ = "4.2.0"
 __all__ = ["SharedMemoryReceiver", "SharedMemorySender", "__version__", "create_shared_memory_pair"]

{py_sharedmemory-4.1.0 → py_sharedmemory-4.2.0}/py_sharedmemory/convert.py

@@ -1,7 +1,9 @@
 """Serialize protocol-5 pickle data through a single shared-memory block."""
 
+import errno
 import pickle
 import sys
+from collections.abc import Callable
 from contextlib import suppress
 from multiprocessing.shared_memory import SharedMemory
 from typing import Any, NamedTuple, cast
@@ -14,12 +16,18 @@ _ATTACH_OPTIONS: dict[str, Any] = {"track": False} if sys.version_info >= (3, 13
 class SharedMemoryMessage(NamedTuple):
     name: str
     buffer_sizes: tuple[int, ...]
+    # Unique per sender. Blocks are reused, so acknowledgements name the message, not the block.
+    sequence: int = 0
 
     @property
     def size(self) -> int:
         return sum(self.buffer_sizes)
 
 
+def create_shared_memory(size: int) -> SharedMemory:
+    return SharedMemory(create=True, size=size)
+
+
 def release_shared_memory(allocation: SharedMemory) -> None:
     """Release both the mapping and its name, including on error paths."""
     try:
@@ -29,8 +37,15 @@ def release_shared_memory(allocation: SharedMemory) -> None:
             allocation.unlink()
 
 
-def serialize(data: Any) -> tuple[SharedMemory, SharedMemoryMessage]:
-    """Copy a pickle and its out-of-band buffers into owned shared memory."""
+def serialize(
+    data: Any,
+    sequence: int = 0,
+    allocate: Callable[[int], SharedMemory] = create_shared_memory,
+) -> tuple[SharedMemory, SharedMemoryMessage]:
+    """Copy a pickle and its out-of-band buffers into an owned block of at least that size.
+
+    `allocate` returns a new or reused block; this function frees it if copying fails.
+    """
     buffers: list[memoryview] = []
     allocation = None
 
@@ -40,14 +55,14 @@ def serialize(data: Any) -> tuple[SharedMemory, SharedMemoryMessage]:
     try:
         pickle_data = pickle.dumps(data, protocol=5, buffer_callback=append_buffer)
         buffer_sizes = (len(pickle_data), *(buffer.nbytes for buffer in buffers))
-        allocation = SharedMemory(create=True, size=sum(buffer_sizes))
+        allocation = allocate(sum(buffer_sizes))
         shared = cast(memoryview, allocation.buf)
         shared[: len(pickle_data)] = pickle_data
         offset = len(pickle_data)
         for buffer in buffers:
             shared[offset : offset + buffer.nbytes] = buffer
             offset += buffer.nbytes
-        return allocation, SharedMemoryMessage(allocation.name, buffer_sizes)
+        return allocation, SharedMemoryMessage(allocation.name, buffer_sizes, sequence)
     except BaseException:
         if allocation is not None:
             release_shared_memory(allocation)
@@ -59,17 +74,41 @@ def serialize(data: Any) -> tuple[SharedMemory, SharedMemoryMessage]:
 
 def deserialize(message: SharedMemoryMessage) -> Any:
     """Unpickle with owned local buffers and always close the shared mapping."""
+    return load(message, attach(message))
+
+
+def attach(message: SharedMemoryMessage) -> SharedMemory:
+    """Map the block that valid metadata names, or explain why it no longer exists."""
     _validate_message(message)
-    allocation = SharedMemory(name=message.name, **_ATTACH_OPTIONS)
+    try:
+        allocation = SharedMemory(name=message.name, **_ATTACH_OPTIONS)
+    except FileNotFoundError as error:
+        raise _freed_block_error(message) from error
+    # Attaching on Windows is not atomic: a name freed in between is recreated as a new,
+    # zero-filled block. Every pickle starts with a nonzero opcode.
+    if cast(memoryview, allocation.buf)[0] == 0:
+        allocation.close()
+        raise _freed_block_error(message)
+    return allocation
+
+
+def _freed_block_error(message: SharedMemoryMessage) -> FileNotFoundError:
+    return FileNotFoundError(
+        errno.ENOENT,
+        "Shared-memory block was freed before it was received; the sender probably "
+        "closed without calling wait_for_all_ack()",
+        message.name,
+    )
+
+
+def load(message: SharedMemoryMessage, allocation: SharedMemory) -> Any:
+    """Unpickle an attached block with owned local buffers, then close the mapping."""
     try:
         if message.size > allocation.size:
             raise ValueError("Shared-memory block is smaller than its metadata")
         shared = cast(memoryview, allocation.buf)
         pickle_size = message.buffer_sizes[0]
-        buffers: list[memoryview] = []
-        if len(message.buffer_sizes) > 1:
-            with shared[pickle_size : message.size] as shared_buffer:
-                buffers = _copy_buffers(shared_buffer, message.buffer_sizes[1:])
+        buffers = _copy_buffers(shared, pickle_size, message.buffer_sizes[1:])
         with shared[:pickle_size] as pickle_data:
             return pickle.loads(pickle_data, buffers=buffers)
     finally:
@@ -82,6 +121,8 @@ def _validate_message(message: SharedMemoryMessage) -> None:
     sizes = message.buffer_sizes
     if (
         not message.name
+        or type(message.sequence) is not int
+        or message.sequence < 0
         or not isinstance(sizes, tuple)
         or not sizes
         or any(type(size) is not int or size < 0 for size in sizes)
@@ -90,12 +131,12 @@ def _validate_message(message: SharedMemoryMessage) -> None:
         raise ValueError("Invalid shared-memory metadata")
 
 
-def _copy_buffers(shared_buffer: memoryview, sizes: tuple[int, ...]) -> list[memoryview]:
-    # One writable copy; each returned view keeps its local storage alive.
+def _copy_buffers(shared: memoryview, offset: int, sizes: tuple[int, ...]) -> list[memoryview]:
+    # A separate writable copy per buffer starts aligned for any dtype and is freed on its own,
+    # so a small array cannot keep a large neighbour alive.
     buffers = []
-    with memoryview(bytearray(shared_buffer)) as local_buffer:
-        offset = 0
-        for size in sizes:
-            buffers.append(local_buffer[offset : offset + size])
-            offset += size
+    for size in sizes:
+        with shared[offset : offset + size] as shared_buffer:
+            buffers.append(memoryview(bytearray(shared_buffer)))
+        offset += size
     return buffers

{py_sharedmemory-4.1.0 → py_sharedmemory-4.2.0}/py_sharedmemory/factory.py

@@ -21,8 +21,7 @@ def create_shared_memory_pair(
 
     Args:
         capacity: Maximum messages sent but not yet received and freed; `put` blocks
-            when it is reached. `0` or `None` means unbounded, but on Linux and macOS
-            every unread message also holds two open file descriptors.
+            when it is reached. `0` or `None` means unbounded.
         ctx: Multiprocessing context for the underlying queues, such as
             `multiprocessing.get_context("spawn")`. Defaults to the default context;
             use the same context to start the worker processes.

{py_sharedmemory-4.1.0 → py_sharedmemory-4.2.0}/py_sharedmemory/receiver.py

@@ -6,18 +6,18 @@ from multiprocessing.queues import Queue
 from types import TracebackType
 from typing import Any, Self
 
-from .convert import SharedMemoryMessage, deserialize
+from .convert import SharedMemoryMessage, attach, load
 
 
 class SharedMemoryReceiver:
     """Consuming endpoint of a pair; create it with `create_shared_memory_pair`.
 
     Several threads and processes may share one receiver; each message is delivered to
-    exactly one of them. Every message is acknowledged, even when decoding fails, so
-    the sender can free its memory.
+    exactly one of them. Every received block is acknowledged, even when decoding fails,
+    so the sender can free its memory.
     """
 
-    def __init__(self, data_queue: Queue[SharedMemoryMessage], ack_queue: Queue[str]) -> None:
+    def __init__(self, data_queue: Queue[SharedMemoryMessage], ack_queue: Queue[int]) -> None:
         self._data_queue = data_queue
         self._ack_queue = ack_queue
         self._closed = False
@@ -35,24 +35,26 @@ class SharedMemoryReceiver:
             timeout: Maximum seconds to wait; `None` waits forever.
 
         Returns:
-            The object, which stays valid after the shared memory is freed. Arrays keep
+            The object, which stays valid after the block is reused or freed. Arrays keep
             their original writability.
 
         Raises:
             queue.Empty: No message arrived in time.
             BrokenPipeError: The receiver is closed.
+            FileNotFoundError: The sender freed the message before it was received.
 
         Unpickling errors propagate unchanged; the message is still acknowledged.
         """
         if self._closed:
             raise BrokenPipeError("Receiver is closed")
         message = self._data_queue.get(block, timeout)
+        # A block that is gone was already freed by its sender, which expects no acknowledgement.
+        allocation = attach(message)
         try:
-            return deserialize(message)
+            return load(message, allocation)
         finally:
             # Failed decoding must still release the sender's memory and capacity.
-            if isinstance(message, SharedMemoryMessage):
-                self._ack_queue.put_nowait(message.name)
+            self._ack_queue.put_nowait(message.sequence)
 
     def get_nowait(self) -> Any:
         """Equivalent to `get(block=False)`."""