mpengine 0.3.1__tar.gz → 0.5.0__tar.gz

{mpengine-0.3.1/src/mpengine.egg-info → mpengine-0.5.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: mpengine
-Version: 0.3.1
+Version: 0.5.0
 Summary: A small, general-purpose multiprocessing engine: dispatch any callable across processes with run manifests, per-worker logs, on-disk outputs and failure isolation
 Author: Amandeep Singh
 License-Expression: MIT
@@ -19,16 +19,23 @@ Classifier: Programming Language :: Python :: 3.13
 Classifier: Programming Language :: Python :: 3.14
 Classifier: Topic :: Software Development :: Libraries :: Python Modules
 Classifier: Topic :: System :: Distributed Computing
+Classifier: Typing :: Typed
 Requires-Python: >=3.10
 Description-Content-Type: text/markdown
 License-File: LICENSE
 Requires-Dist: numpy>=1.26
 Requires-Dist: cloudpickle>=3.0
 Requires-Dist: tqdm>=4.60
+Requires-Dist: threadpoolctl>=3.0
+Provides-Extra: test
+Requires-Dist: pytest>=7.0; extra == "test"
+Requires-Dist: joblib>=1.3; extra == "test"
 Dynamic: license-file
 
 # mpengine
 
+[![CI](https://github.com/singhamandeep-kgp/multiprocessor/actions/workflows/ci.yml/badge.svg)](https://github.com/singhamandeep-kgp/multiprocessor/actions/workflows/ci.yml)
+
 *(This repo is named `multiprocessor`; the package it ships is `mpengine` — see below.)*
 
 A small, general-purpose multiprocessing engine. Give it any callable and a list
@@ -39,7 +46,7 @@ Every run produces three things:
 
 | what | where |
 |---|---|
-| a manifest of exactly what was launched, and how it ended | `<base>/manifests/<run_id>.txt` |
+| human-readable and versioned JSON manifests | `<base>/manifests/<run_id>.txt` and `.json` |
 | each job's result, saved to disk | `<base>/outputs/<run_id>/<label>` |
 | one log file per **worker process** | `<base>/logs/<run_id>/worker_<pid>.log` |
 | a log of the run as a whole | `<base>/logs/<run_id>/run.log` |
@@ -126,11 +133,92 @@ run with a custom `save_fn`, pass the matching reader as
 | `n_workers` | worker *process* count (not threads — see below); defaults to `os.cpu_count()`, clamped down to the number of jobs if there are fewer jobs than that |
 | `debug` | run sequentially in-process — real tracebacks you can attach a debugger to, no pool |
 | `show_progress` | live terminal display: one overall bar for the whole run, plus a live rate number per worker process. Ignored when `debug=True` |
+| `retries`, `retry_delay` | retry transient callback failures, optionally pausing between attempts |
+| `failure_policy` | `continue` (default), `raise` after all jobs, or `fail_fast` |
+| `timeout` | wall-clock limit for the complete parallel dispatch |
+| `cancel_token` | a thread-safe `CancellationToken` another thread can use to abort the run |
 
 Develop with `debug=True`, then flip it off. Chasing a bug through a process
 pool means reading a traceback re-raised from a worker that has already exited,
 and it never tells you which job dict was at fault.
 
+### Performance
+
+The defaults are already tuned; these are the knobs for when they aren't
+enough. Measured on 8 cores with OpenBLAS against joblib 1.5.3 — reproduce
+with `python benchmarks/bench.py`, which keeps each "before" path runnable
+rather than quoting a remembered number.
+
+| argument | meaning | measured |
+|---|---|---|
+| `blas_threads` | threads each worker's native BLAS may use for one numpy call. `'auto'` = `cpu_count // workers`; an int to set it; `None` to disable | 24 SVD jobs **8.26s → 1.20s** |
+| `chunksize` | jobs per submission. `'auto'` batches large runs and collapses to 1 on small ones | 20,000 tiny jobs **5.55s → 1.01s** |
+| `broadcast` | a `dict` of values shipped once per *worker* instead of once per job, delivered to your function as keyword arguments | 200 jobs + 80 MB panel **7.97s → 0.55s** |
+| `reuse_pool` | keep the pool (and its broadcast payload) alive between calls instead of rebuilding it. Off by default | 10 dispatches **9.86s → 0.02s** |
+| `initializer`, `initargs` | run once per worker before its first job — and unlike the stdlib's, may be a closure | — |
+| `max_tasks_per_child` | recycle a worker every N user jobs, so a slow leak can't end the run; disables batching to keep the count exact (Python 3.11+) | — |
+
+**BLAS threads** is the one that catches people out. numpy hands matrix work to
+a native library that is itself multi-threaded, and each worker process loads
+its own copy believing it owns the machine — so eight workers each spawn eight
+BLAS threads and 64 threads fight over 8 cores. Capping them means parallelism
+comes from mpengine, one job per core, instead. This is on by default.
+
+**Broadcast** is for the panel or fitted model every job needs:
+
+```python
+summary = run(score, param_sets, base_dir="runs",
+              broadcast={"panel": panel}, reuse_pool=True)
+```
+
+`score` is then called as `score(**params, panel=panel)`. Worth being precise
+about when it pays: batching alone already reduces a closure-captured payload
+to one copy per batch, so on a single cold call with a payload under ~10 MB,
+broadcast costs slightly more than it saves. Above that, or paired with
+`reuse_pool=True` where the payload is delivered once for the life of the pool,
+it is a different order of magnitude — 0.55s on the 80 MB case, level with
+joblib's memmapped 0.58s. Call `shutdown_pools()` when you're done with a
+reused pool, or let `atexit` do it.
+
+For a large NumPy array, wrap it in `Shared` to copy it once into OS shared
+memory and let every worker read the same physical bytes:
+
+```python
+from mpengine import Shared
+
+summary = run(score, param_sets, base_dir="runs",
+              broadcast={"panel": Shared(panel)})
+```
+
+Shared arrays are read-only in workers and must be C-contiguous with a
+non-object dtype. Keep the original array alive while a reused pool needs it.
+
+### Reliability controls
+
+```python
+from mpengine import CancellationToken, RunFailedError, run
+
+token = CancellationToken()
+try:
+    summary = run(
+        score,
+        param_sets,
+        base_dir="runs",
+        retries=2,
+        retry_delay=0.25,
+        failure_policy="raise",
+        timeout=300,
+        cancel_token=token,
+    )
+except RunFailedError as exc:
+    summary = exc.summary  # outputs and both manifests are already durable
+```
+
+`failure_policy="continue"` preserves the original behavior. `"raise"` runs
+everything and then raises if any job failed. `"fail_fast"` terminates the
+pool after the first failed result. Timeout and cancellation also terminate
+the pool and raise exceptions carrying a partial `summary`.
+
 ### Logging
 
 The library logs its whole lifecycle through the standard `logging` module and
@@ -182,10 +270,13 @@ That's a cost trade-off to be aware of, not a limitation on what's allowed.
 you want to drive the pool yourself:
 
 ```python
-from mpengine import expand_call, process_jobs, process_jobs_
+from mpengine import expand_call, iter_jobs, process_jobs, process_jobs_
 
 jobs = [{"func": my_task, "x": 1, "y": 2}, {"func": other_task, "n": 5}]
 results = process_jobs(jobs)     # n_workers defaults to os.cpu_count(); or process_jobs_ to stay sequential
+
+for result in iter_jobs(jobs):   # consume each result as soon as it completes
+    store(result)
 ```
 
 A job is just a dict carrying its own callback plus that callback's kwargs, so a
@@ -203,6 +294,25 @@ for triangular workloads — where item `i` costs `O(i)`, such as an
 expanding-window computation — which keeps workers from idling while one
 overloaded worker finishes.
 
+## Tests
+
+```bash
+pip install -e ".[test]"
+pytest                    # everything, ~30s
+pytest -m "not slow"      # skip the process-pool tests, under a second
+```
+
+576 tests. The ones that spawn real pools are marked `slow` and dominate the
+runtime, so the marker exists to keep a fast inner loop available - but they
+run by default, because the behaviour they cover (a dead worker surfacing
+instead of hanging, per-job failure attribution inside a batch, BLAS budgets
+read back from inside a worker) is exactly the behaviour worth guarding.
+
+CI runs the suite on Linux, Windows and macOS across Python 3.10, 3.12 and
+3.14. The platform spread matters here: Windows and macOS start workers with
+`spawn`, Linux does not, and mpengine caps BLAS threads by a different
+mechanism in each case.
+
 ## Changelog
 
 See [CHANGELOG.md](CHANGELOG.md). Versions below 1.0 may carry breaking changes