deadpool-executor 2025.2.1__tar.gz → 2025.2.2__tar.gz

{deadpool_executor-2025.2.1 → deadpool_executor-2025.2.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.3
 Name: deadpool-executor
-Version: 2025.2.1
+Version: 2025.2.2
 Summary: Deadpool
 Author-email: Caleb Hattingh <caleb.hattingh@gmail.com>
 Description-Content-Type: text/x-rst
@@ -13,6 +13,7 @@ Classifier: Programming Language :: Python :: 3.9
 Classifier: Programming Language :: Python :: 3.10
 Classifier: Programming Language :: Python :: 3.11
 Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
 Classifier: Programming Language :: Python :: Implementation
 Classifier: Programming Language :: Python :: Implementation :: CPython
 Requires-Dist: psutil
@@ -247,6 +248,119 @@ stdlib pool:
   that modification will get the new vars. One example use-case
   is dynamically changing the logging level within subprocesses.
 
+Minimum and Maximum Workers
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+``Deadpool`` has a ``min_workers`` and ``max_workers`` parameter.
+While ``max_workers`` is the same as the stdlib pool, ``min_workers``
+is a new feature.
+
+The ``min_workers`` parameter allows deadpool to "scale down" the
+pool when it is idle. This is another strategy alongside other
+features like ``max_tasks_per_child`` and ``max_worker_memory_bytes``
+to help deal with memory bloat in long-running pools.
+
+Statistics
+~~~~~~~~~~
+
+Here is a very simple example of how to get statistics from the
+executor:
+
+.. code-block:: python
+
+    with deadpool.Deadpool() as exe:
+        fut = exe.submit(...)
+        stats = exe.get_statistics()
+
+The call must be made while the executor is still alive. It
+will succeed after the executor is shut down or closed, but
+some of the statistics will be zeroed out.
+
+The call to ``get_statistics`` will return a dictionary with the
+following keys:
+
+- ``tasks_received``: The total number of tasks submitted to the
+  executor. Does not mean that they started running, only that they
+  were successfully submitted.
+- ``tasks_launched``: The total number of tasks that were launched
+  on a subprocess. This records the count of all tasks that were
+  successfully scheduled to run. These tasks were picked up from
+  the submit backlog and given to a worker process to execute.
+- ``tasks_failed``: The total number of tasks that failed. This
+  includes tasks that raised an exception, and tasks that were
+  killed due to a timeout, and really any other reason that a task
+  failed.
+- ``worker_processes_created``: The total number of subprocesses that
+  were ever created by the executor. This can be, and often will be
+  greater than the `max_workers` setting because there are many options
+  that can cause workers to be discarded and replaced. Examples of these
+  might be the ``max_tasks_per_child`` setting, or the ``min_workers``
+  setting, or the memory thresholds and so on.
+- ``max_workers_busy_concurrently``: The maximum number of workers that
+  were ever busy at the same time. This is a useful statistic to
+  decide whether you might consider increasing or decreasing the size
+  of the pool. For example, if your ``max_workers`` is set to 100, but
+  after running for, say, a week, you see that ``max_workers_busy_concurrently``
+  is only 50, then you might consider reducing the pool size to 50.
+  The system memory manager on linux likes to hold onto heap memory.
+  If your have more workers than you need, you'll see that the system
+  memory usage over time is going to be higher than it needs to be
+  because even when the pool is fully idle, you will still observe
+  the persistent worker processes having a large memory allocation
+  even though no jobs are running. This is a symptom of malloc
+  retention behaviour.
+- ``worker_processes_still_alive``: The number of worker processes that
+  are still alive. This includes both idle and busy worker processes.
+  This is mainly a debugging statistic that I can use to check whether
+  worker processes are "leaking" somehow and not being cleaned up
+  correctly. This number should not be greater than the ``max_workers``.
+  (It could be, temporarily, depending on the exact timing and strategy
+  in the inner workings of the executor, but on average it should not)
+- ``worker_processes_idle``: The number of worker processes that are idle.
+- ``worker_processes_busy``: The number of worker processes that are busy.
+
+
+Here is an example from the tests to explain what each of the
+statistics mean:
+
+.. code-block:: python
+
+    with deadpool.Deadpool(min_workers=5, max_workers=10) as exe:
+        futs = []
+        for _ in range(50):
+            futs.append(exe.submit(t, 0.05))
+            futs.append(exe.submit(f_err, Exception))
+
+        results = []
+        for fut in deadpool.as_completed(futs):
+            try:
+                results.append(fut.result())
+            except Exception:
+                pass
+
+        time.sleep(0.5)
+        stats = exe.get_statistics()
+
+    assert results == [0.05] * 50
+    print(f"{stats=}")
+    assert stats == {
+        "tasks_received": 100,
+        "tasks_launched": 100,
+        "tasks_failed": 50,
+        "worker_processes_created": 10,
+        "max_workers_busy_concurrently": 10,
+        "worker_processes_still_alive": 5,
+        "worker_processes_idle": 5,
+        "worker_processes_busy": 0,
+    }
+
+In this example, we submit 100 tasks, 50 of which will raise an
+exception. The executor will create 10 worker processes, and
+will have a maximum of 10 workers busy at the same time. After
+all the tasks are completed, we wait for a short time to allow
+the executor to clean up any worker processes that are no longer
+needed. The statistics should show that 5 worker processes are
+still alive, and all of them are idle.
 
 Show me some code
 =================

{deadpool_executor-2025.2.1 → deadpool_executor-2025.2.2}/README.rst

@@ -220,6 +220,119 @@ stdlib pool:
   that modification will get the new vars. One example use-case
   is dynamically changing the logging level within subprocesses.
 
+Minimum and Maximum Workers
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+``Deadpool`` has a ``min_workers`` and ``max_workers`` parameter.
+While ``max_workers`` is the same as the stdlib pool, ``min_workers``
+is a new feature.
+
+The ``min_workers`` parameter allows deadpool to "scale down" the
+pool when it is idle. This is another strategy alongside other
+features like ``max_tasks_per_child`` and ``max_worker_memory_bytes``
+to help deal with memory bloat in long-running pools.
+
+Statistics
+~~~~~~~~~~
+
+Here is a very simple example of how to get statistics from the
+executor:
+
+.. code-block:: python
+
+    with deadpool.Deadpool() as exe:
+        fut = exe.submit(...)
+        stats = exe.get_statistics()
+
+The call must be made while the executor is still alive. It
+will succeed after the executor is shut down or closed, but
+some of the statistics will be zeroed out.
+
+The call to ``get_statistics`` will return a dictionary with the
+following keys:
+
+- ``tasks_received``: The total number of tasks submitted to the
+  executor. Does not mean that they started running, only that they
+  were successfully submitted.
+- ``tasks_launched``: The total number of tasks that were launched
+  on a subprocess. This records the count of all tasks that were
+  successfully scheduled to run. These tasks were picked up from
+  the submit backlog and given to a worker process to execute.
+- ``tasks_failed``: The total number of tasks that failed. This
+  includes tasks that raised an exception, and tasks that were
+  killed due to a timeout, and really any other reason that a task
+  failed.
+- ``worker_processes_created``: The total number of subprocesses that
+  were ever created by the executor. This can be, and often will be
+  greater than the `max_workers` setting because there are many options
+  that can cause workers to be discarded and replaced. Examples of these
+  might be the ``max_tasks_per_child`` setting, or the ``min_workers``
+  setting, or the memory thresholds and so on.
+- ``max_workers_busy_concurrently``: The maximum number of workers that
+  were ever busy at the same time. This is a useful statistic to
+  decide whether you might consider increasing or decreasing the size
+  of the pool. For example, if your ``max_workers`` is set to 100, but
+  after running for, say, a week, you see that ``max_workers_busy_concurrently``
+  is only 50, then you might consider reducing the pool size to 50.
+  The system memory manager on linux likes to hold onto heap memory.
+  If your have more workers than you need, you'll see that the system
+  memory usage over time is going to be higher than it needs to be
+  because even when the pool is fully idle, you will still observe
+  the persistent worker processes having a large memory allocation
+  even though no jobs are running. This is a symptom of malloc
+  retention behaviour.
+- ``worker_processes_still_alive``: The number of worker processes that
+  are still alive. This includes both idle and busy worker processes.
+  This is mainly a debugging statistic that I can use to check whether
+  worker processes are "leaking" somehow and not being cleaned up
+  correctly. This number should not be greater than the ``max_workers``.
+  (It could be, temporarily, depending on the exact timing and strategy
+  in the inner workings of the executor, but on average it should not)
+- ``worker_processes_idle``: The number of worker processes that are idle.
+- ``worker_processes_busy``: The number of worker processes that are busy.
+
+
+Here is an example from the tests to explain what each of the
+statistics mean:
+
+.. code-block:: python
+
+    with deadpool.Deadpool(min_workers=5, max_workers=10) as exe:
+        futs = []
+        for _ in range(50):
+            futs.append(exe.submit(t, 0.05))
+            futs.append(exe.submit(f_err, Exception))
+
+        results = []
+        for fut in deadpool.as_completed(futs):
+            try:
+                results.append(fut.result())
+            except Exception:
+                pass
+
+        time.sleep(0.5)
+        stats = exe.get_statistics()
+
+    assert results == [0.05] * 50
+    print(f"{stats=}")
+    assert stats == {
+        "tasks_received": 100,
+        "tasks_launched": 100,
+        "tasks_failed": 50,
+        "worker_processes_created": 10,
+        "max_workers_busy_concurrently": 10,
+        "worker_processes_still_alive": 5,
+        "worker_processes_idle": 5,
+        "worker_processes_busy": 0,
+    }
+
+In this example, we submit 100 tasks, 50 of which will raise an
+exception. The executor will create 10 worker processes, and
+will have a maximum of 10 workers busy at the same time. After
+all the tasks are completed, we wait for a short time to allow
+the executor to clean up any worker processes that are no longer
+needed. The statistics should show that 5 worker processes are
+still alive, and all of them are idle.
 
 Show me some code
 =================

{deadpool_executor-2025.2.1 → deadpool_executor-2025.2.2}/deadpool.py

@@ -32,6 +32,7 @@ import traceback
 import typing
 import weakref
 import atexit
+import json
 from concurrent.futures import CancelledError, Executor, InvalidStateError, as_completed
 from dataclasses import dataclass, field
 from multiprocessing.connection import Connection
@@ -42,7 +43,7 @@ from functools import partial
 
 import psutil
 
-__version__ = "2025.2.1"
+__version__ = "2025.2.2"
 __all__ = [
     "Deadpool",
     "Future",
@@ -71,6 +72,46 @@ logger = logging.getLogger("deadpool")
 #         logger.info("Error stopping the multiprocessing resource tracker")
 
 
+@dataclass
+class Stat:
+    lock: threading.Lock
+    value: int = 0
+
+    def increment(self, value: int = 1):
+        with self.lock:
+            self.value += value
+
+    def set(self, value: int = 0):
+        self.value = value
+
+
+class Statistics:
+    def __init__(self):
+        self._lock = threading.Lock()
+
+        self.tasks_received = Stat(self._lock, 0)
+        self.tasks_launched = Stat(self._lock, 0)
+        self.tasks_failed = Stat(self._lock, 0)
+        self.worker_processes_created = Stat(self._lock, 0)
+        self.max_workers_busy_concurrently = Stat(self._lock, 0)
+
+    def reset_counters(self):
+        self.tasks_received.set()
+        self.tasks_launched.set()
+        self.tasks_failed.set()
+        self.worker_processes_created.set()
+        self.max_workers_busy_concurrently.set()
+
+    def to_dict(self) -> dict[str, typing.Any]:
+        return {
+            "tasks_received": self.tasks_received.value,
+            "tasks_launched": self.tasks_launched.value,
+            "tasks_failed": self.tasks_failed.value,
+            "worker_processes_created": self.worker_processes_created.value,
+            "max_workers_busy_concurrently": self.max_workers_busy_concurrently.value,
+        }
+
+
 @dataclass(order=True)
 class PrioritizedItem:
     priority: int
@@ -277,6 +318,7 @@ class Deadpool(Executor):
         )
         self.running_jobs = Queue(maxsize=self.pool_size)
         self.running_futs = weakref.WeakSet()
+        self.existing_workers = weakref.WeakSet()
         self.closed = False
         self.shutdown_wait = shutdown_wait
         self.shutdown_cancel_futures = shutdown_cancel_futures
@@ -284,6 +326,7 @@ class Deadpool(Executor):
         self.malloc_trim_rss_memory_threshold_bytes = (
             malloc_trim_rss_memory_threshold_bytes
         )
+        self._statistics = Statistics()
 
         # TODO: overcommit
         self.workers: SimpleQueue[WorkerProcess] = SimpleQueue()
@@ -303,6 +346,17 @@ class Deadpool(Executor):
         )
         self.runner_thread.start()
 
+    def get_statistics(self) -> dict[str, typing.Any]:
+        stats = self._statistics.to_dict()
+
+        # These are not counters; they are determined at the time of the
+        # call based on the state of the worker processes.
+        stats["worker_processes_still_alive"] = len(self.existing_workers)
+        stats["worker_processes_idle"] = self.workers.qsize()
+        stats["worker_processes_busy"] = len(self.busy_workers)
+
+        return stats
+
     def add_worker_to_pool(self):
         if self.propagate_environ:
             # By constructing here, late, we allow the user to make
@@ -332,6 +386,8 @@ class Deadpool(Executor):
             malloc_trim_rss_memory_threshold_bytes=self.malloc_trim_rss_memory_threshold_bytes,
         )
         self.workers.put(worker)
+        self._statistics.worker_processes_created.increment()
+        self.existing_workers.add(worker)
 
     def clear_workers(self):
         """Clear all workers from the pool.
@@ -373,6 +429,7 @@ class Deadpool(Executor):
                 continue
 
             t = threading.Thread(target=self.run_task, args=job, daemon=True)
+            self._statistics.tasks_launched.increment()
             t.start()
 
     def get_process(self) -> WorkerProcess:
@@ -386,17 +443,35 @@ class Deadpool(Executor):
 
         wp = self.workers.get()
         self.busy_workers.add(wp)
+        if (
+            len(self.busy_workers)
+            > self._statistics.max_workers_busy_concurrently.value
+        ):
+            self._statistics.max_workers_busy_concurrently.value = len(
+                self.busy_workers
+            )
+
         return wp
 
     def done_with_process(self, wp: WorkerProcess):
+        # This worker is done with its job and is no longer busy.
         self.busy_workers.remove(wp)
 
-        bw = len(self.busy_workers)
-        mw = self.min_workers
-        qs = self.workers.qsize()
-
-        total_workers = bw + qs
-        if total_workers > mw and qs > 0:
+        count_workers_busy = len(self.busy_workers)
+        count_workers_idle = self.workers.qsize()
+        backlog_size = self.submitted_jobs.qsize()
+
+        # The `1` is for `wp` itself.
+        total_workers = count_workers_busy + count_workers_idle + 1
+        there_are_more_workers_than_min = total_workers > self.min_workers
+        task_backlog_is_empty = backlog_size == 0
+
+        # if there_are_more_workers_than_min and (there_are_idle_workers or task_backlog_is_empty):
+        if there_are_more_workers_than_min and task_backlog_is_empty:
+            # We have more workers than the minimum, and there is no backlog of
+            # tasks. This implies any tasks currently in play have already been picked
+            # up by workers in the pool, or the pool is idle. We can safely remove
+            # this worker from the pool.
             wp.shutdown(wait=False)
             return
 
@@ -454,9 +529,12 @@ class Deadpool(Executor):
                     #  where the worker process often OOMs. As such, not sure
                     #  whether logging at warning level is appropriate.
                     logger.warning(f"BrokenPipeError on {worker.pid}, retrying.")
+                    worker.ok = False
                     self.done_with_process(worker)
+                    # TODO: probably this should be moved into the `done_with_process`
+                    #  and can act on the `worker.ok` flag.
                     kill_proc_tree(worker.pid, sig=signal.SIGKILL)
-            else:
+            else:  # pragma: no cover
                 # If we get here, we've tried to submit the job to a worker
                 # process multiple times and failed each time. We're giving
                 # up.
@@ -472,19 +550,23 @@ class Deadpool(Executor):
                     try:
                         results = worker.get_results()
                     except EOFError:
+                        self._statistics.tasks_failed.increment()
                         fut.set_exception(
                             ProcessError("Worker process died unexpectedly")
                         )
                     except BaseException as e:
+                        self._statistics.tasks_failed.increment()
                         logger.debug(f"Unexpected exception from worker: {e}")
                         fut.set_exception(e)
                     else:
                         if isinstance(results, BaseException):
+                            self._statistics.tasks_failed.increment()
                             fut.set_exception(results)
                         else:
                             fut.set_result(results)
 
                         if isinstance(results, TimeoutError):
+                            self._statistics.tasks_failed.increment()
                             logger.debug(
                                 f"TimeoutError on {worker.pid}, setting ok=False"
                             )
@@ -492,6 +574,7 @@ class Deadpool(Executor):
                     finally:
                         break
                 elif not worker.is_alive():
+                    self._statistics.tasks_failed.increment()
                     logger.debug(f"p is no longer alive: {worker.process}")
                     try:
                         signame = signal.strsignal(-worker.process.exitcode)
@@ -554,6 +637,7 @@ class Deadpool(Executor):
                 item=(fn, args, kwargs, deadpool_timeout, fut),
             )
         )
+        self._statistics.tasks_received.increment()
         return fut
 
     def shutdown(self, wait: bool = True, *, cancel_futures: bool = False) -> None:

{deadpool_executor-2025.2.1 → deadpool_executor-2025.2.2}/examples/leftover.py

@@ -28,7 +28,9 @@ def main():
             time.sleep(random.randrange(0, 10) / 10)
             w = random.randrange(0, 50) / 10
             exe.submit(worker, results, w, deadpool_timeout=4)
-            print(f"{exe.workers.qsize()=} {len(exe.busy_workers)=}")
+            print(
+                f"{exe.workers.qsize()=} {len(exe.busy_workers)=} {len(exe.existing_workers)=}"
+            )
 
     outcome = []
     while not results.empty():

{deadpool_executor-2025.2.1 → deadpool_executor-2025.2.2}/pyproject.toml

@@ -20,6 +20,7 @@ classifiers = [
     "Programming Language :: Python :: 3.10",
     "Programming Language :: Python :: 3.11",
     "Programming Language :: Python :: 3.12",
+    "Programming Language :: Python :: 3.13",
     "Programming Language :: Python :: Implementation",
     "Programming Language :: Python :: Implementation :: CPython",
 ]

{deadpool_executor-2025.2.1 → deadpool_executor-2025.2.2}/tests/test_deadpool.py

@@ -5,6 +5,7 @@ import queue
 import signal
 import sys
 import time
+import multiprocessing as mp
 from concurrent.futures import CancelledError, as_completed
 from contextlib import contextmanager
 from functools import partial
@@ -19,11 +20,6 @@ def logging_initializer():
     return partial(logging.basicConfig, level=logging.DEBUG)
 
 
-async def test_func():
-    await asyncio.sleep(0.1)
-    return 42
-
-
 def f():
     return 123
 
@@ -54,7 +50,7 @@ def test_cancel_all_futures():
 
 @pytest.mark.parametrize("malloc_threshold", [None, 0, 1_000_000])
 @pytest.mark.parametrize("daemon", [True, False])
-@pytest.mark.parametrize("min_workers", [None, 0])
+@pytest.mark.parametrize("min_workers", [None, 10])
 def test_simple(malloc_threshold, daemon, min_workers):
     with deadpool.Deadpool(
         malloc_trim_rss_memory_threshold_bytes=malloc_threshold,
@@ -65,7 +61,24 @@ def test_simple(malloc_threshold, daemon, min_workers):
         fut = exe.submit(t, 0.05)
         result = fut.result()
 
+        time.sleep(0.5)
+        stats = exe.get_statistics()
+        # To exercise the reset path
+        exe._statistics.reset_counters()
+
     assert result == 0.05
+    print(f"{stats=}")
+
+    assert stats == {
+        "tasks_received": 1,
+        "tasks_launched": 1,
+        "tasks_failed": 0,
+        "worker_processes_created": 10,
+        "max_workers_busy_concurrently": 1,
+        "worker_processes_still_alive": 10,
+        "worker_processes_idle": 10,
+        "worker_processes_busy": 0,
+    }
 
     # Outside the context manager, no new tasks
     # can be submitted.
@@ -73,6 +86,68 @@ def test_simple(malloc_threshold, daemon, min_workers):
         exe.submit(f)
 
 
+def test_stats():
+    with deadpool.Deadpool(
+        min_workers=5,
+        max_workers=10,
+    ) as exe:
+        futs = []
+        for _ in range(6):
+            futs.append(exe.submit(t, 0.05))
+
+        results = [fut.result() for fut in deadpool.as_completed(futs)]
+        time.sleep(0.5)
+        stats = exe.get_statistics()
+
+    assert results == [0.05] * 6
+    print(f"{stats=}")
+    assert stats == {
+        "tasks_received": 6,
+        "tasks_launched": 6,
+        "tasks_failed": 0,
+        "worker_processes_created": 10,
+        # This is an important one.
+        "max_workers_busy_concurrently": 6,
+        "worker_processes_still_alive": 5,
+        "worker_processes_idle": 5,
+        "worker_processes_busy": 0,
+    }
+
+
+def test_stats_with_errors():
+    with deadpool.Deadpool(
+        min_workers=5,
+        max_workers=10,
+    ) as exe:
+        futs = []
+        for _ in range(50):
+            futs.append(exe.submit(t, 0.05))
+            futs.append(exe.submit(f_err, Exception))
+
+        results = []
+        for fut in deadpool.as_completed(futs):
+            try:
+                results.append(fut.result())
+            except Exception:
+                pass
+
+        time.sleep(0.5)
+        stats = exe.get_statistics()
+
+    assert results == [0.05] * 50
+    print(f"{stats=}")
+    assert stats == {
+        "tasks_received": 100,
+        "tasks_launched": 100,
+        "tasks_failed": 50,
+        "worker_processes_created": 10,
+        "max_workers_busy_concurrently": 10,
+        "worker_processes_still_alive": 5,
+        "worker_processes_idle": 5,
+        "worker_processes_busy": 0,
+    }
+
+
 ##### Env var propagation #####
 
 
@@ -222,7 +297,10 @@ def test_simple_batch(logging_initializer):
     assert results == [0.1] * 2
 
 
-@pytest.mark.parametrize("ctx", ["spawn", "forkserver"])
+@pytest.mark.parametrize(
+    "ctx",
+    ["spawn", "forkserver", mp.get_context("spawn"), mp.get_context("forkserver")],
+)
 def test_ctx(logging_initializer, ctx):
     with deadpool.Deadpool(mp_context=ctx, initializer=logging_initializer) as exe:
         fut = exe.submit(t, 0.05)