py-sharedmemory 4.1.0__tar.gz → 4.1.1__tar.gz

{py_sharedmemory-4.1.0 → py_sharedmemory-4.1.1}/CHANGELOG.md

@@ -1,11 +1,23 @@
 # Changelog
 
+## 4.1.1
+
+- Unread messages no longer hold file descriptors on Linux and macOS: the sender unmaps
+  each block right after writing it. Unbounded pairs previously failed with
+  `Too many open files` after about 120 unread messages under macOS's default limit.
+- Cap outstanding messages at the metadata queue's limit (32,767 on macOS), so a blocking
+  `put` on an unbounded pair waits instead of raising `queue.Full`.
+- Explain a freed block in the receiver's `FileNotFoundError`, which happens when the
+  sender closes before a message is read.
+
 ## 4.1.0
 
 - Warn with a `RuntimeWarning` when leaving a `with sender:` block or exiting the
   producing process discards unacknowledged messages, so a forgotten
   `wait_for_all_ack()` no longer loses data silently. Explicit `close()` calls and
   exits caused by exceptions stay silent, and closing still never blocks.
+- Collect garbage between tests to avoid Python 3.12 resource-tracker reentrancy errors,
+  and add a stress test that checks every byte across consumer processes.
 
 ## 4.0.0
 

{py_sharedmemory-4.1.0/py_sharedmemory.egg-info → py_sharedmemory-4.1.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: py-sharedmemory
-Version: 4.1.0
+Version: 4.1.1
 Summary: Fast multiprocessing.Queue alternative that moves large Python objects and NumPy arrays through shared memory
 Author: Tobias Würth
 License-Expression: MIT
@@ -190,9 +190,7 @@ consumer = ctx.Process(target=consume, args=(receiver,))
 
 `capacity` counts messages that were sent but not yet received and freed. When it is
 reached, `put()` blocks, so a fast producer cannot exhaust memory. Shared-memory use is
-at most about `capacity × message size`. `capacity=0` or `None` means unbounded; prefer
-a limit, because unread messages also hold open files on Linux and macOS (see the
-platform notes below).
+at most about `capacity × message size`. `capacity=0` or `None` means unbounded.
 
 ### Timeouts and non-blocking calls
 
@@ -296,13 +294,9 @@ with `OSError: [Errno 28] No space left on device` or crash with `SIGBUS`.
 **Linux:** outside containers, `/dev/shm` usually defaults to half of physical RAM.
 
 **macOS:** fully supported and tested on Apple Silicon. The default start method is
-`spawn`, so guard the entry point with `if __name__ == "__main__":`.
-
-**Open-file limit (Linux and macOS):** each unread message keeps two file descriptors
-open in the producer until it is received. macOS allows 256 per process by default and
-Linux usually 1,024, so an unbounded pair fails with `OSError: [Errno 24] Too many open
-files` after roughly 120 or 500 unread messages. Set a `capacity` well below that, or
-raise the limit with `ulimit -n`. The sender stays usable after this error.
+`spawn`, so guard the entry point with `if __name__ == "__main__":`. An unbounded pair
+holds at most 32,767 unread messages, the limit of macOS semaphores; beyond that `put`
+waits or raises `queue.Full` like a full bounded pair.
 
 **Windows:** shared memory is backed by the page file, and each block disappears once
 the last handle closes, so nothing can leak after a crash.
@@ -310,6 +304,13 @@ the last handle closes, so nothing can leak after a crash.
 **"resource_tracker: There appear to be N leaked shared_memory objects":** fixed in
 3.0.1. Upgrade if you see this warning with the `fork` start method.
 
+**`FileNotFoundError: Shared-memory block was freed before it was received`:** the
+producer closed or exited before this message was read. Call `wait_for_all_ack()` before
+closing the sender; the sender also warns when this happens.
+
+**`OSError: [Errno 24] Too many open files` with many unread messages:** fixed in 4.1.1,
+where unread messages no longer hold file descriptors.
+
 **`RuntimeError: A shared-memory pair supports one producing process`:** a second
 process called `put`. Create a separate pair for each producer process.
 

{py_sharedmemory-4.1.0 → py_sharedmemory-4.1.1}/README.md

@@ -140,9 +140,7 @@ consumer = ctx.Process(target=consume, args=(receiver,))
 
 `capacity` counts messages that were sent but not yet received and freed. When it is
 reached, `put()` blocks, so a fast producer cannot exhaust memory. Shared-memory use is
-at most about `capacity × message size`. `capacity=0` or `None` means unbounded; prefer
-a limit, because unread messages also hold open files on Linux and macOS (see the
-platform notes below).
+at most about `capacity × message size`. `capacity=0` or `None` means unbounded.
 
 ### Timeouts and non-blocking calls
 
@@ -246,13 +244,9 @@ with `OSError: [Errno 28] No space left on device` or crash with `SIGBUS`.
 **Linux:** outside containers, `/dev/shm` usually defaults to half of physical RAM.
 
 **macOS:** fully supported and tested on Apple Silicon. The default start method is
-`spawn`, so guard the entry point with `if __name__ == "__main__":`.
-
-**Open-file limit (Linux and macOS):** each unread message keeps two file descriptors
-open in the producer until it is received. macOS allows 256 per process by default and
-Linux usually 1,024, so an unbounded pair fails with `OSError: [Errno 24] Too many open
-files` after roughly 120 or 500 unread messages. Set a `capacity` well below that, or
-raise the limit with `ulimit -n`. The sender stays usable after this error.
+`spawn`, so guard the entry point with `if __name__ == "__main__":`. An unbounded pair
+holds at most 32,767 unread messages, the limit of macOS semaphores; beyond that `put`
+waits or raises `queue.Full` like a full bounded pair.
 
 **Windows:** shared memory is backed by the page file, and each block disappears once
 the last handle closes, so nothing can leak after a crash.
@@ -260,6 +254,13 @@ the last handle closes, so nothing can leak after a crash.
 **"resource_tracker: There appear to be N leaked shared_memory objects":** fixed in
 3.0.1. Upgrade if you see this warning with the `fork` start method.
 
+**`FileNotFoundError: Shared-memory block was freed before it was received`:** the
+producer closed or exited before this message was read. Call `wait_for_all_ack()` before
+closing the sender; the sender also warns when this happens.
+
+**`OSError: [Errno 24] Too many open files` with many unread messages:** fixed in 4.1.1,
+where unread messages no longer hold file descriptors.
+
 **`RuntimeError: A shared-memory pair supports one producing process`:** a second
 process called `put`. Create a separate pair for each producer process.
 

{py_sharedmemory-4.1.0 → py_sharedmemory-4.1.1}/py_sharedmemory/__init__.py

@@ -25,5 +25,5 @@ from .factory import create_shared_memory_pair
 from .receiver import SharedMemoryReceiver
 from .sender import SharedMemorySender
 
-__version__ = "4.1.0"
+__version__ = "4.1.1"
 __all__ = ["SharedMemoryReceiver", "SharedMemorySender", "__version__", "create_shared_memory_pair"]

{py_sharedmemory-4.1.0 → py_sharedmemory-4.1.1}/py_sharedmemory/convert.py

@@ -1,5 +1,6 @@
 """Serialize protocol-5 pickle data through a single shared-memory block."""
 
+import errno
 import pickle
 import sys
 from contextlib import suppress
@@ -60,7 +61,15 @@ def serialize(data: Any) -> tuple[SharedMemory, SharedMemoryMessage]:
 def deserialize(message: SharedMemoryMessage) -> Any:
     """Unpickle with owned local buffers and always close the shared mapping."""
     _validate_message(message)
-    allocation = SharedMemory(name=message.name, **_ATTACH_OPTIONS)
+    try:
+        allocation = SharedMemory(name=message.name, **_ATTACH_OPTIONS)
+    except FileNotFoundError as error:
+        raise FileNotFoundError(
+            errno.ENOENT,
+            "Shared-memory block was freed before it was received; the sender probably "
+            "closed without calling wait_for_all_ack()",
+            message.name,
+        ) from error
     try:
         if message.size > allocation.size:
             raise ValueError("Shared-memory block is smaller than its metadata")

{py_sharedmemory-4.1.0 → py_sharedmemory-4.1.1}/py_sharedmemory/factory.py

@@ -21,8 +21,7 @@ def create_shared_memory_pair(
 
     Args:
         capacity: Maximum messages sent but not yet received and freed; `put` blocks
-            when it is reached. `0` or `None` means unbounded, but on Linux and macOS
-            every unread message also holds two open file descriptors.
+            when it is reached. `0` or `None` means unbounded.
         ctx: Multiprocessing context for the underlying queues, such as
             `multiprocessing.get_context("spawn")`. Defaults to the default context;
             use the same context to start the worker processes.

{py_sharedmemory-4.1.0 → py_sharedmemory-4.1.1}/py_sharedmemory/sender.py

@@ -3,11 +3,13 @@
 from __future__ import annotations
 
 import os
+import sys
 import threading
 import time
 import warnings
 from multiprocessing.queues import Queue
 from multiprocessing.shared_memory import SharedMemory
+from multiprocessing.synchronize import SEM_VALUE_MAX  # type: ignore[attr-defined]  # not in typeshed
 from multiprocessing.util import Finalize
 from queue import Empty, Full
 from types import TracebackType
@@ -18,6 +20,14 @@ from .convert import SharedMemoryMessage, release_shared_memory, serialize
 if TYPE_CHECKING:
     from multiprocessing.sharedctypes import Synchronized
 
+# A POSIX block lives on by name until it is unlinked, so the sender can drop its mapping
+# and file descriptors right after writing. Windows frees a block with its last handle.
+_UNMAP_AFTER_WRITE = sys.platform != "win32"
+
+# The metadata queue holds at most SEM_VALUE_MAX unread items (32,767 on macOS); beyond
+# that its put would raise queue.Full even for a blocking put on an unbounded sender.
+_MAX_OUTSTANDING = SEM_VALUE_MAX
+
 
 def _validate_capacity(capacity: int | None) -> None:
     if capacity is None:
@@ -215,7 +225,8 @@ class SharedMemorySender:
         with self._condition:
             self._ensure_open()
             self._start_ack_thread()
-            while self.capacity and self._pending_count >= self.capacity:
+            limit = min(self.capacity or _MAX_OUTSTANDING, _MAX_OUTSTANDING)
+            while self._pending_count >= limit:
                 if not block:
                     raise Full
                 remaining = None if deadline is None else deadline - time.monotonic()
@@ -235,6 +246,8 @@ class SharedMemorySender:
                 raise
             try:
                 self._ensure_open()
+                if _UNMAP_AFTER_WRITE:
+                    allocation.close()
                 self._data_queue.put_nowait(message)
             except BaseException:
                 self._release_message(allocation.name)

{py_sharedmemory-4.1.0 → py_sharedmemory-4.1.1/py_sharedmemory.egg-info}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: py-sharedmemory
-Version: 4.1.0
+Version: 4.1.1
 Summary: Fast multiprocessing.Queue alternative that moves large Python objects and NumPy arrays through shared memory
 Author: Tobias Würth
 License-Expression: MIT
@@ -190,9 +190,7 @@ consumer = ctx.Process(target=consume, args=(receiver,))
 
 `capacity` counts messages that were sent but not yet received and freed. When it is
 reached, `put()` blocks, so a fast producer cannot exhaust memory. Shared-memory use is
-at most about `capacity × message size`. `capacity=0` or `None` means unbounded; prefer
-a limit, because unread messages also hold open files on Linux and macOS (see the
-platform notes below).
+at most about `capacity × message size`. `capacity=0` or `None` means unbounded.
 
 ### Timeouts and non-blocking calls
 
@@ -296,13 +294,9 @@ with `OSError: [Errno 28] No space left on device` or crash with `SIGBUS`.
 **Linux:** outside containers, `/dev/shm` usually defaults to half of physical RAM.
 
 **macOS:** fully supported and tested on Apple Silicon. The default start method is
-`spawn`, so guard the entry point with `if __name__ == "__main__":`.
-
-**Open-file limit (Linux and macOS):** each unread message keeps two file descriptors
-open in the producer until it is received. macOS allows 256 per process by default and
-Linux usually 1,024, so an unbounded pair fails with `OSError: [Errno 24] Too many open
-files` after roughly 120 or 500 unread messages. Set a `capacity` well below that, or
-raise the limit with `ulimit -n`. The sender stays usable after this error.
+`spawn`, so guard the entry point with `if __name__ == "__main__":`. An unbounded pair
+holds at most 32,767 unread messages, the limit of macOS semaphores; beyond that `put`
+waits or raises `queue.Full` like a full bounded pair.
 
 **Windows:** shared memory is backed by the page file, and each block disappears once
 the last handle closes, so nothing can leak after a crash.
@@ -310,6 +304,13 @@ the last handle closes, so nothing can leak after a crash.
 **"resource_tracker: There appear to be N leaked shared_memory objects":** fixed in
 3.0.1. Upgrade if you see this warning with the `fork` start method.
 
+**`FileNotFoundError: Shared-memory block was freed before it was received`:** the
+producer closed or exited before this message was read. Call `wait_for_all_ack()` before
+closing the sender; the sender also warns when this happens.
+
+**`OSError: [Errno 24] Too many open files` with many unread messages:** fixed in 4.1.1,
+where unread messages no longer hold file descriptors.
+
 **`RuntimeError: A shared-memory pair supports one producing process`:** a second
 process called `put`. Create a separate pair for each producer process.
 

{py_sharedmemory-4.1.0 → py_sharedmemory-4.1.1}/tests/test_convert.py

@@ -242,6 +242,15 @@ def test_cleanup_tolerates_already_unlinked_blocks():
     assert_unlinked(info.name)
 
 
+def test_freed_block_reports_a_clear_error():
+    handle, info = serialize(42)
+    release_shared_memory(handle)
+    with pytest.raises(FileNotFoundError, match="freed before it was received") as caught:
+        deserialize(info)
+    assert caught.value.filename == info.name
+    assert isinstance(caught.value.__cause__, FileNotFoundError)
+
+
 def test_cleanup_ignores_names_that_are_already_gone():
     # Windows never raises here because unlink is a no-op, so simulate POSIX.
     handle = Mock()

{py_sharedmemory-4.1.0 → py_sharedmemory-4.1.1}/tests/test_processes.py

@@ -193,6 +193,35 @@ raise SystemExit(process.exitcode)
 """
 
 
+LOW_FILE_LIMIT = """
+import resource
+
+from py_sharedmemory import create_shared_memory_pair
+
+# macOS defaults to 256 open files; unread messages must not hold descriptors.
+resource.setrlimit(resource.RLIMIT_NOFILE, (256, 256))
+sender, receiver = create_shared_memory_pair(0)
+for index in range(1_000):
+    sender.put(index)
+assert [receiver.get(timeout=10) for _ in range(1_000)] == list(range(1_000))
+assert sender.wait_for_all_ack(timeout=10)
+sender.close()
+receiver.close()
+"""
+
+
+@pytest.mark.skipif(sys.platform == "win32", reason="POSIX file-descriptor limits")
+def test_unread_messages_hold_no_file_descriptors():
+    result = subprocess.run(
+        [sys.executable, "-c", LOW_FILE_LIMIT],
+        cwd=Path(__file__).resolve().parents[1],
+        capture_output=True,
+        text=True,
+        timeout=60,
+    )
+    assert result.returncode == 0, result.stderr
+
+
 UNREAD_AT_EXIT = """
 from py_sharedmemory import create_shared_memory_pair
 

{py_sharedmemory-4.1.0 → py_sharedmemory-4.1.1}/tests/test_queue.py

@@ -1,11 +1,13 @@
 import multiprocessing as mp
 import os
 import pickle
+import sys
 import threading
 import time
 import warnings
 from concurrent.futures import ThreadPoolExecutor
 from contextlib import suppress
+from multiprocessing.shared_memory import SharedMemory
 from queue import Empty, Full
 from unittest.mock import Mock
 
@@ -93,6 +95,38 @@ def test_full_nonblocking_and_immediate_timeouts(make_pair, kwargs):
     assert receiver.get(timeout=3) == 3
 
 
+@pytest.mark.parametrize("capacity", [0, None, 5])
+def test_outstanding_messages_never_exceed_the_metadata_queue_limit(
+    make_pair, monkeypatch, capacity
+):
+    # macOS limits a multiprocessing queue to 32,767 unread items; simulate a tiny limit.
+    monkeypatch.setattr(sender_module, "_MAX_OUTSTANDING", 2)
+    sender, receiver = make_pair(capacity)
+    sender.put(1)
+    sender.put(2)
+    with pytest.raises(Full):
+        sender.put_nowait(3)
+    with pytest.raises(Full):
+        sender.put(3, timeout=0.05)
+    assert receiver.get(timeout=3) == 1
+    sender.put(3, timeout=3)
+    assert [receiver.get(timeout=3) for _ in range(2)] == [2, 3]
+    assert sender.wait_for_all_ack(timeout=3)
+
+
+@pytest.mark.parametrize("unmap", [False, True])
+def test_sender_unmaps_blocks_only_where_names_keep_them_alive(make_pair, monkeypatch, unmap):
+    monkeypatch.setattr(sender_module, "_UNMAP_AFTER_WRITE", unmap)
+    sender, receiver = make_pair()
+    sender.put(42)
+    allocation = next(iter(sender._blocks.values()))
+    assert (allocation.buf is None) is unmap
+    if not (unmap and sys.platform == "win32"):
+        # Windows frees an unmapped block at once, which is why it never unmaps.
+        assert receiver.get(timeout=3) == 42
+        assert sender.wait_for_all_ack(timeout=3)
+
+
 def test_put_timeout_waits_for_capacity(make_pair):
     sender, _ = make_pair()
     sender.put(1)
@@ -204,7 +238,9 @@ def test_failed_decode_is_acknowledged_and_next_message_succeeds(make_pair):
     sender, receiver = make_pair()
     sender.put(42)
     name = next(iter(sender._blocks))
-    sender._blocks[name].buf[0] = ord("!")
+    corrupt = SharedMemory(name=name)
+    corrupt.buf[0] = ord("!")
+    corrupt.close()
     with pytest.raises(pickle.UnpicklingError):
         receiver.get(timeout=3)
     assert sender.wait_for_all_ack(timeout=3)