py-sharedmemory 2.0.1__tar.gz → 3.0.0__tar.gz

py_sharedmemory-3.0.0/CHANGELOG.md

@@ -0,0 +1,10 @@
+# Changelog
+
+## 3.0.0
+
+- Python 3.11+, standard-library-only runtime.
+- Fix acknowledgement races, capacity leaks, failed-put recovery, and process cleanup.
+- Preserve NumPy mutability; remove redundant copies and retained pickle storage.
+- Simplify core ownership and sender flow; use immutable metadata and recover from partial startup.
+- Add explicit close, context managers, context selection, and timed acknowledgement waits.
+- Replace legacy tests and benchmarks; add platform CI and recorded machine results.

{py_sharedmemory-2.0.1 → py_sharedmemory-3.0.0}/LICENSE

@@ -1,6 +1,6 @@
 MIT License
 
-Copyright (c) 2025 товіаѕ
+Copyright (c) 2026 Tobias Würth
 
 Permission is hereby granted, free of charge, to any person obtaining a copy
 of this software and associated documentation files (the "Software"), to deal

py_sharedmemory-3.0.0/MANIFEST.in

@@ -0,0 +1,4 @@
+include CHANGELOG.md
+recursive-include tests *.py
+recursive-include benchmarks *.py *.json *.png
+recursive-include examples *.py

py_sharedmemory-3.0.0/PKG-INFO

@@ -0,0 +1,98 @@
+Metadata-Version: 2.4
+Name: py-sharedmemory
+Version: 3.0.0
+Summary: Shared-memory queues for large Python objects with less pipe I/O
+Author: Tobias Würth
+License-Expression: MIT
+Project-URL: Repository, https://github.com/tobiaswuerth/python_shared_memory_queue
+Project-URL: Bug Tracker, https://github.com/tobiaswuerth/python_shared_memory_queue/issues
+Keywords: queue,multiprocessing,numpy,inter-process-communication,shared-memory
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3 :: Only
+Classifier: Operating System :: OS Independent
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Provides-Extra: test
+Requires-Dist: pytest>=7.4; extra == "test"
+Requires-Dist: pytest-cov>=4.1; extra == "test"
+Requires-Dist: pytest-timeout>=2.3; extra == "test"
+Requires-Dist: numpy>=1.26; extra == "test"
+Provides-Extra: benchmark
+Requires-Dist: numpy>=1.26; extra == "benchmark"
+Requires-Dist: matplotlib>=3.8; extra == "benchmark"
+Dynamic: license-file
+
+# py-sharedmemory
+
+Send large Python objects between processes with less pipe I/O.
+
+Originally built to reduce the heavy Windows pipe I/O and uneven throughput caused by
+large `multiprocessing.Queue` transfers. Payloads travel through shared memory;
+queues carry only small metadata, avoiding bulk transfers through the pipe.
+
+Useful for **ML pipelines, reinforcement-learning actors and replay buffers,
+scientific computing, and image/video processing**. Supports pickleable Python
+objects with familiar `put/get` calls. No runtime dependencies. Python 3.11+.
+
+```sh
+pip install py-sharedmemory
+```
+
+## Measured performance
+
+![Transfer speed relative to multiprocessing.Queue, from 16 bytes to 10 GB](benchmarks/results/windows-numpy.png)
+
+About **2× faster at 1–10 MB**; **10 GB transferred in 8.0 s**.
+`mp.Queue` failed at 10 GB (Windows error 87). Small messages favor standard queues.
+
+Core i9-14900K, 191.8 GiB RAM, Windows 11, Python 3.12.10, NumPy 2.5.3.
+2026-10-08; contiguous float32 arrays; `spawn`; capacity 2; median of 5 runs,
+excluding startup and warmup. Up to 1,000 messages or 512 MiB/run, at least one
+message. GB = 10⁹ bytes. [Raw results](benchmarks/results/windows-numpy.json).
+These measurements show transfer speed; I/O and stability depend on the workload.
+
+## Usage
+
+```python
+import multiprocessing as mp
+from memory import create_shared_memory_pair
+
+
+def produce(sender):
+    with sender:
+        sender.put({"step": 42, "frames": bytearray(16_000_000)})
+        sender.wait_for_all_ack()
+
+
+if __name__ == "__main__":
+    ctx = mp.get_context("spawn")
+    sender, receiver = create_shared_memory_pair(capacity=2, ctx=ctx)
+    worker = ctx.Process(target=produce, args=(sender,))
+    worker.start()
+    with receiver:
+        batch = receiver.get(timeout=30)
+    worker.join()
+    sender.close()
+```
+
+One producing process per pair; multiple threads and consumers are supported.
+Pass endpoints to workers before use. `put/get` support `block`, `timeout`, and
+`*_nowait`. Capacity counts outstanding messages; 0/`None` is unbounded.
+
+**Drain before producer close or exit:** `wait_for_all_ack(timeout=...)` returns
+`False` on timeout. Close discards unread data. Within one process, drain before
+closing either endpoint. Received data stays valid after cleanup; arrays preserve mutability.
+
+A [complete example](examples/basic.py) includes error handling.
+
+## Test and benchmark
+
+```sh
+python -m pip install -e ".[test,benchmark]"
+python -m pytest --cov=memory
+python -m benchmarks.benchmark --sizes 16 100 1000 10000 100000 1000000 10000000 100000000 1000000000 10000000000 --repeats 5 --json benchmarks/results/windows-numpy.json
+python -m benchmarks.plot benchmarks/results/windows-numpy.json benchmarks/results/windows-numpy.png
+```
+
+The full benchmark needs enough RAM for several payload copies.

py_sharedmemory-3.0.0/README.md

@@ -0,0 +1,73 @@
+# py-sharedmemory
+
+Send large Python objects between processes with less pipe I/O.
+
+Originally built to reduce the heavy Windows pipe I/O and uneven throughput caused by
+large `multiprocessing.Queue` transfers. Payloads travel through shared memory;
+queues carry only small metadata, avoiding bulk transfers through the pipe.
+
+Useful for **ML pipelines, reinforcement-learning actors and replay buffers,
+scientific computing, and image/video processing**. Supports pickleable Python
+objects with familiar `put/get` calls. No runtime dependencies. Python 3.11+.
+
+```sh
+pip install py-sharedmemory
+```
+
+## Measured performance
+
+![Transfer speed relative to multiprocessing.Queue, from 16 bytes to 10 GB](benchmarks/results/windows-numpy.png)
+
+About **2× faster at 1–10 MB**; **10 GB transferred in 8.0 s**.
+`mp.Queue` failed at 10 GB (Windows error 87). Small messages favor standard queues.
+
+Core i9-14900K, 191.8 GiB RAM, Windows 11, Python 3.12.10, NumPy 2.5.3.
+2026-10-08; contiguous float32 arrays; `spawn`; capacity 2; median of 5 runs,
+excluding startup and warmup. Up to 1,000 messages or 512 MiB/run, at least one
+message. GB = 10⁹ bytes. [Raw results](benchmarks/results/windows-numpy.json).
+These measurements show transfer speed; I/O and stability depend on the workload.
+
+## Usage
+
+```python
+import multiprocessing as mp
+from memory import create_shared_memory_pair
+
+
+def produce(sender):
+    with sender:
+        sender.put({"step": 42, "frames": bytearray(16_000_000)})
+        sender.wait_for_all_ack()
+
+
+if __name__ == "__main__":
+    ctx = mp.get_context("spawn")
+    sender, receiver = create_shared_memory_pair(capacity=2, ctx=ctx)
+    worker = ctx.Process(target=produce, args=(sender,))
+    worker.start()
+    with receiver:
+        batch = receiver.get(timeout=30)
+    worker.join()
+    sender.close()
+```
+
+One producing process per pair; multiple threads and consumers are supported.
+Pass endpoints to workers before use. `put/get` support `block`, `timeout`, and
+`*_nowait`. Capacity counts outstanding messages; 0/`None` is unbounded.
+
+**Drain before producer close or exit:** `wait_for_all_ack(timeout=...)` returns
+`False` on timeout. Close discards unread data. Within one process, drain before
+closing either endpoint. Received data stays valid after cleanup; arrays preserve mutability.
+
+A [complete example](examples/basic.py) includes error handling.
+
+## Test and benchmark
+
+```sh
+python -m pip install -e ".[test,benchmark]"
+python -m pytest --cov=memory
+python -m benchmarks.benchmark --sizes 16 100 1000 10000 100000 1000000 10000000 100000000 1000000000 10000000000 --repeats 5 --json benchmarks/results/windows-numpy.json
+python -m benchmarks.plot benchmarks/results/windows-numpy.json benchmarks/results/windows-numpy.png
+```
+
+The full benchmark needs enough RAM for several payload copies.

py_sharedmemory-3.0.0/benchmarks/__init__.py

@@ -0,0 +1 @@
+"""Opt-in throughput measurements; excluded from the installed package."""

py_sharedmemory-3.0.0/benchmarks/benchmark.py

@@ -0,0 +1,168 @@
+"""Compare real process-to-process throughput with multiprocessing.Queue."""
+
+import argparse
+import io
+import json
+import multiprocessing as mp
+import platform
+import statistics
+import time
+from contextlib import redirect_stderr
+from datetime import UTC, datetime
+from pathlib import Path
+
+from memory import create_shared_memory_pair
+
+
+def _receive(receiver, count, warmup, control, size):
+    try:
+        for _ in range(warmup):
+            receiver.get(timeout=30)
+        control.send("ready")
+        if control.recv() != "start":
+            raise RuntimeError("Invalid benchmark handshake")
+        for _ in range(count):
+            payload = receiver.get(timeout=30)
+            actual_size = payload.nbytes if hasattr(payload, "nbytes") else len(payload)
+            if actual_size != size:
+                raise ValueError("Received the wrong payload size")
+        control.send("done")
+    except Exception as error:
+        control.send({"error": f"{type(error).__name__}: {error}"})
+    finally:
+        receiver.close()
+        control.close()
+
+
+def _expect(control, expected):
+    if not control.poll(30):
+        raise TimeoutError(f"Benchmark consumer did not signal {expected}")
+    message = control.recv()
+    if isinstance(message, dict) and "error" in message:
+        raise RuntimeError(f"Consumer failed: {message['error']}")
+    if message != expected:
+        raise RuntimeError("Invalid benchmark handshake")
+
+
+def measure(transport, payload, count, capacity, ctx):
+    """Measure after process startup and warmup, through the last received item."""
+    size = payload.nbytes if hasattr(payload, "nbytes") else len(payload)
+    if transport == "shared":
+        sender, receiver = create_shared_memory_pair(capacity, ctx=ctx)
+    else:
+        sender = receiver = ctx.Queue(maxsize=capacity)
+    control, child_control = ctx.Pipe()
+    warmup = min(3, count)
+    process = ctx.Process(target=_receive, args=(receiver, count, warmup, child_control, size))
+    process.start()
+    child_control.close()
+    try:
+        for _ in range(warmup):
+            sender.put(payload, timeout=30)
+        _expect(control, "ready")
+        if transport == "shared" and not sender.wait_for_all_ack(timeout=30):
+            raise TimeoutError("Warmup messages were not acknowledged")
+        start = time.perf_counter()
+        control.send("start")
+        for _ in range(count):
+            sender.put(payload, timeout=30)
+        _expect(control, "done")
+        if transport == "shared" and not sender.wait_for_all_ack(timeout=30):
+            raise TimeoutError("Benchmark messages were not acknowledged")
+        duration = time.perf_counter() - start
+        process.join(10)
+        if process.is_alive() or process.exitcode != 0:
+            raise RuntimeError("Benchmark consumer failed")
+        return duration
+    finally:
+        if process.is_alive():
+            process.terminate()
+            process.join(5)
+        control.close()
+        if transport == "shared":
+            sender.close()
+            receiver.close()
+        else:
+            sender.cancel_join_thread()
+            sender.close()
+
+
+def _positive_int(value):
+    number = int(value)
+    if number <= 0:
+        raise argparse.ArgumentTypeError("must be positive")
+    return number
+
+
+def main(argv=None):
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("--sizes", nargs="+", type=_positive_int, default=[1_024, 2**20, 2**24])
+    parser.add_argument(
+        "--messages", type=_positive_int, help="default: up to 1,000 or 512 MiB/run"
+    )
+    parser.add_argument("--capacity", type=_positive_int, default=2)
+    parser.add_argument("--repeats", type=_positive_int, default=3)
+    parser.add_argument("--payload", choices=["numpy", "bytes"], default="numpy")
+    parser.add_argument("--start-method", choices=mp.get_all_start_methods(), default="spawn")
+    parser.add_argument("--json", type=Path, help="optional result file")
+    args = parser.parse_args(argv)
+    if args.payload == "numpy" and any(size % 4 for size in args.sizes):
+        parser.error("NumPy sizes must be multiples of 4 for float32 payloads")
+    if args.payload == "numpy":
+        import numpy as np
+
+    ctx = mp.get_context(args.start_method)
+    results = []
+    report = {
+        "generated_at_utc": datetime.now(UTC).isoformat(timespec="seconds"),
+        "python": platform.python_version(),
+        "platform": platform.platform(),
+        "processor": platform.processor(),
+        "start_method": args.start_method,
+        "capacity": args.capacity,
+        "payload": args.payload,
+        "numpy": np.__version__ if args.payload == "numpy" else None,
+        "repeats": args.repeats,
+        "results": results,
+    }
+    print(f"Payload: {args.payload}; start method: {args.start_method}; capacity: {args.capacity}")
+    print(f"{'Bytes':>12}  {'Transport':>9}  {'MiB/s':>10}  {'ms/message':>12}")
+    for size in args.sizes:
+        count = args.messages or min(1_000, max(1, 2**29 // size))
+        if args.payload == "numpy":
+            payload = np.ones(size // 4, dtype=np.float32)
+        else:
+            payload = b"x" * size
+        for transport in ("standard", "shared"):
+            row = {"transport": transport, "size": size, "messages": count}
+            durations = []
+            diagnostics = io.StringIO()
+            try:
+                with redirect_stderr(diagnostics):
+                    for _ in range(args.repeats):
+                        durations.append(measure(transport, payload, count, args.capacity, ctx))
+            except Exception as error:
+                row.update(status="failed", error=f"{type(error).__name__}: {error}")
+                if diagnostics.getvalue():
+                    row["stderr"] = diagnostics.getvalue()
+                print(f"{size:12,d}  {transport:>9}  FAILED: {row['error']}", flush=True)
+            else:
+                duration = statistics.median(durations)
+                throughput = size * count / duration / 2**20
+                latency = duration * 1_000 / count
+                row.update(
+                    status="ok",
+                    median_mib_per_second=throughput,
+                    median_ms_per_message=latency,
+                )
+                print(
+                    f"{size:12,d}  {transport:>9}  {throughput:10.1f}  {latency:12.3f}", flush=True
+                )
+            row["durations_seconds"] = durations
+            results.append(row)
+            if args.json:
+                args.json.write_text(json.dumps(report, indent=2) + "\n", encoding="utf-8")
+
+
+if __name__ == "__main__":
+    main()

py_sharedmemory-3.0.0/benchmarks/plot.py

@@ -0,0 +1,80 @@
+"""Plot the recorded benchmark without inventing throughput for failed transfers."""
+
+import argparse
+import json
+from pathlib import Path
+
+import matplotlib
+
+matplotlib.use("Agg")
+import matplotlib.pyplot as plt  # noqa: E402
+
+
+def plot(report, output):
+    plt.rcParams.update({"font.size": 11, "font.family": "DejaVu Sans"})
+    shared = {r["size"]: r for r in report["results"] if r["transport"] == "shared"}
+    standard = {r["size"]: r for r in report["results"] if r["transport"] == "standard"}
+    sizes = sorted(s for s in shared if shared[s]["status"] == standard[s]["status"] == "ok")
+    ratios = [
+        shared[s]["median_mib_per_second"] / standard[s]["median_mib_per_second"] for s in sizes
+    ]
+    failures = [
+        s for s in shared if shared[s]["status"] == "ok" and standard[s]["status"] == "failed"
+    ]
+    fig, axes = plt.subplots(
+        1,
+        2 if failures else 1,
+        figsize=(11, 4.5),
+        gridspec_kw={"width_ratios": [4, 1]} if failures else None,
+        layout="constrained",
+        squeeze=False,
+    )
+    ax = axes[0, 0]
+    bars = ax.bar(
+        range(len(sizes)),
+        ratios,
+        width=0.7,
+        color=["#0d9488" if ratio >= 1 else "#94a3b8" for ratio in ratios],
+    )
+    ax.bar_label(bars, labels=[f"{ratio:.2f}×" for ratio in ratios], padding=5, fontsize=10)
+    ax.axhline(1, color="#64748b", linestyle="--", linewidth=1.2)
+    ax.text(0.01, 1.05, "Standard queue (1×)", transform=ax.get_yaxis_transform(), fontsize=9)
+    ax.set_xticks(range(len(sizes)))
+    ax.set_xticklabels([_size_label(size) for size in sizes])
+    ax.set_ylim(0, max(2.5, max(ratios, default=1) * 1.2))
+    ax.set_xlabel("Payload per message")
+    ax.set_ylabel("Speedup · higher is better")
+    ax.set_title("Transfer speed vs. multiprocessing.Queue", loc="left", fontweight="bold", pad=18)
+    ax.grid(axis="y", color="#e2e8f0", linewidth=0.8)
+    ax.set_axisbelow(True)
+    ax.spines[["top", "right"]].set_visible(False)
+    if failures:
+        panel = axes[0, 1]
+        panel.axis("off")
+        size = max(failures)
+        seconds = shared[size]["median_ms_per_message"] / 1_000
+        panel.text(0.5, 0.86, f"{_size_label(size)} payload", ha="center", fontweight="bold")
+        panel.text(0.5, 0.63, f"{seconds:.1f} s", ha="center", fontsize=30, color="#0d9488")
+        panel.text(0.5, 0.5, "py-sharedmemory", ha="center")
+        panel.text(0.5, 0.25, "mp.Queue failed", ha="center", color="#b91c1c")
+    fig.savefig(output, dpi=200, facecolor="white")
+    plt.close(fig)
+
+
+def _size_label(size):
+    for divisor, unit in ((10**9, "GB"), (10**6, "MB"), (10**3, "kB")):
+        if size >= divisor:
+            return f"{size / divisor:g} {unit}"
+    return f"{size} B"
+
+
+def main():
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("results", type=Path)
+    parser.add_argument("output", type=Path)
+    args = parser.parse_args()
+    plot(json.loads(args.results.read_text(encoding="utf-8")), args.output)
+
+
+if __name__ == "__main__":
+    main()