mpengine 0.2.0__tar.gz → 0.3.0__tar.gz

{mpengine-0.2.0/src/mpengine.egg-info → mpengine-0.3.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: mpengine
-Version: 0.2.0
+Version: 0.3.0
 Summary: A small, general-purpose multiprocessing engine: dispatch any callable across processes with run manifests, per-worker logs, on-disk outputs and failure isolation
 Author: Amandeep Singh
 License-Expression: MIT
@@ -42,6 +42,7 @@ Every run produces three things:
 | a manifest of exactly what was launched, and how it ended | `<base>/manifests/<run_id>.txt` |
 | each job's result, saved to disk | `<base>/outputs/<run_id>/<label>` |
 | one log file per **worker process** | `<base>/logs/<run_id>/worker_<pid>.log` |
+| a log of the run as a whole | `<base>/logs/<run_id>/run.log` |
 
 A single job failing does **not** abort the run — it is recorded as a failed
 job and the rest continue.
@@ -127,9 +128,43 @@ run with a custom `save_fn`, pass the matching reader as
 | `show_progress` | live terminal display: one overall bar for the whole run, plus a live rate number per worker process. Ignored when `debug=True` |
 
 Develop with `debug=True`, then flip it off. Chasing a bug through a process
-pool means reading a `RemoteTraceback` from a worker that has already exited,
+pool means reading a traceback re-raised from a worker that has already exited,
 and it never tells you which job dict was at fault.
 
+### Logging
+
+The library logs its whole lifecycle through the standard `logging` module and
+writes nothing to stdout except the banner, so embedding it never pollutes a
+host application's output. It installs only a `NullHandler` — to see anything,
+configure logging yourself:
+
+```python
+import logging
+logging.basicConfig(level=logging.INFO)
+```
+
+Two loggers, so either half can be tuned or silenced independently:
+
+| logger | emits |
+|---|---|
+| `mpengine.orchestrator` | run start/done, resolved dirs, per-job outcomes, worker ranking |
+| `mpengine.engine` | dispatch start/done, worker-count clamping, progress milestones, worker death |
+
+Levels: **INFO** for run lifecycle, **DEBUG** for per-job success (a 10,000-job
+sweep would otherwise be 10,000 INFO lines), **WARNING** for a failed job or a
+clamped worker count, **ERROR** for a dead worker.
+
+Every run also writes `<log_dir>/<run_id>/run.log` containing the parent's
+whole view of that run — dispatch, milestones, every job outcome, the summary —
+alongside the existing per-worker log files. That happens regardless of how you
+configure logging, so a run is always a self-contained durable record.
+
+**Progress display is terminal-aware.** Interactively you get the tqdm bars and
+per-worker lines as before. When stderr is not a terminal — piped, redirected to
+a log file, running under CI or a scheduler — those are suppressed (they render
+as unreadable escape-sequence noise in a file) and periodic completion
+milestones are logged at INFO instead.
+
 ### Closures and lambdas
 
 `func`, and any custom `save_fn`, can be a closure or a lambda — not just a
@@ -168,6 +203,11 @@ for triangular workloads — where item `i` costs `O(i)`, such as an
 expanding-window computation — which keeps workers from idling while one
 overloaded worker finishes.
 
+## Changelog
+
+See [CHANGELOG.md](CHANGELOG.md). Versions below 1.0 may carry breaking changes
+in a minor release; each is listed there with the migration needed.
+
 ## What else is in the repo
 
 The `src/learning/` package holds demo and exercise scripts used while

{mpengine-0.2.0 → mpengine-0.3.0}/README.md

@@ -13,6 +13,7 @@ Every run produces three things:
 | a manifest of exactly what was launched, and how it ended | `<base>/manifests/<run_id>.txt` |
 | each job's result, saved to disk | `<base>/outputs/<run_id>/<label>` |
 | one log file per **worker process** | `<base>/logs/<run_id>/worker_<pid>.log` |
+| a log of the run as a whole | `<base>/logs/<run_id>/run.log` |
 
 A single job failing does **not** abort the run — it is recorded as a failed
 job and the rest continue.
@@ -98,9 +99,43 @@ run with a custom `save_fn`, pass the matching reader as
 | `show_progress` | live terminal display: one overall bar for the whole run, plus a live rate number per worker process. Ignored when `debug=True` |
 
 Develop with `debug=True`, then flip it off. Chasing a bug through a process
-pool means reading a `RemoteTraceback` from a worker that has already exited,
+pool means reading a traceback re-raised from a worker that has already exited,
 and it never tells you which job dict was at fault.
 
+### Logging
+
+The library logs its whole lifecycle through the standard `logging` module and
+writes nothing to stdout except the banner, so embedding it never pollutes a
+host application's output. It installs only a `NullHandler` — to see anything,
+configure logging yourself:
+
+```python
+import logging
+logging.basicConfig(level=logging.INFO)
+```
+
+Two loggers, so either half can be tuned or silenced independently:
+
+| logger | emits |
+|---|---|
+| `mpengine.orchestrator` | run start/done, resolved dirs, per-job outcomes, worker ranking |
+| `mpengine.engine` | dispatch start/done, worker-count clamping, progress milestones, worker death |
+
+Levels: **INFO** for run lifecycle, **DEBUG** for per-job success (a 10,000-job
+sweep would otherwise be 10,000 INFO lines), **WARNING** for a failed job or a
+clamped worker count, **ERROR** for a dead worker.
+
+Every run also writes `<log_dir>/<run_id>/run.log` containing the parent's
+whole view of that run — dispatch, milestones, every job outcome, the summary —
+alongside the existing per-worker log files. That happens regardless of how you
+configure logging, so a run is always a self-contained durable record.
+
+**Progress display is terminal-aware.** Interactively you get the tqdm bars and
+per-worker lines as before. When stderr is not a terminal — piped, redirected to
+a log file, running under CI or a scheduler — those are suppressed (they render
+as unreadable escape-sequence noise in a file) and periodic completion
+milestones are logged at INFO instead.
+
 ### Closures and lambdas
 
 `func`, and any custom `save_fn`, can be a closure or a lambda — not just a
@@ -139,6 +174,11 @@ for triangular workloads — where item `i` costs `O(i)`, such as an
 expanding-window computation — which keeps workers from idling while one
 overloaded worker finishes.
 
+## Changelog
+
+See [CHANGELOG.md](CHANGELOG.md). Versions below 1.0 may carry breaking changes
+in a minor release; each is listed there with the migration needed.
+
 ## What else is in the repo
 
 The `src/learning/` package holds demo and exercise scripts used while

{mpengine-0.2.0 → mpengine-0.3.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "mpengine"
-version = "0.2.0"
+version = "0.3.0"
 description = "A small, general-purpose multiprocessing engine: dispatch any callable across processes with run manifests, per-worker logs, on-disk outputs and failure isolation"
 readme = "README.md"
 license = "MIT"

{mpengine-0.2.0 → mpengine-0.3.0}/src/mpengine/__init__.py

@@ -39,6 +39,16 @@ Learning*, Ch.20 - the docstrings name the specific snippets - but nothing here
 is finance-specific; it parallelizes any callable.
 """
 
+import logging as _logging
+
+# A library must never configure logging for its host application - it only
+# names its loggers and attaches a no-op handler so that a caller who has set
+# nothing up sees no "No handlers could be found" noise. To actually see any
+# of it: logging.basicConfig(level=logging.INFO). Loggers are
+# "mpengine.engine" and "mpengine.orchestrator", so either half can be tuned
+# or silenced on its own.
+_logging.getLogger("mpengine").addHandler(_logging.NullHandler())
+
 from mpengine.engine import expand_call, process_jobs, process_jobs_, report_progress
 from mpengine.orchestrator import (
     JobResult,
@@ -73,4 +83,4 @@ __all__ = [
     "parts_to_molecules",
 ]
 
-__version__ = "0.2.0"
+__version__ = "0.3.0"

mpengine-0.3.0/src/mpengine/engine.py

@@ -0,0 +1,258 @@
+"""Ch.20 SS20.5 - the multiprocessing engine (exercises 6+7).
+
+expand_call / process_jobs_ / process_jobs / report_progress - a generic
+job-dict-to-function-call pipeline, reusable by any exercise rather than
+tied to one atom/molecule shape. This is the book's own point: stop writing
+a bespoke parallelization wrapper per function, and build one library that
+can parallelize unknown functions regardless of their arguments or output.
+
+A "job" here is just a dict: one 'func' entry (the callback) plus whatever
+kwargs that callback needs. `expand_call` turns such a dict into a call,
+which is the hinge the whole engine turns on.
+"""
+
+from __future__ import annotations
+
+import datetime as dt
+import logging
+import os
+import sys
+import time
+from concurrent.futures import ProcessPoolExecutor, as_completed
+from concurrent.futures.process import BrokenProcessPool
+from typing import Any, Callable
+
+import cloudpickle
+
+_log = logging.getLogger("mpengine.engine")
+
+
+def expand_call(kargs: dict[str, Any]) -> Any:
+    """Ch.20 Snippet 20.10 (`expandCall`) - turn a job dict into a call.
+
+    Deviation from the book: `kargs` is copied before popping. The book's
+    literal `kargs.pop('func')` mutates the *caller's* dict in place, so the
+    same job list can only ever be run once - a second pass raises
+    KeyError('func'). ex04 deliberately runs one job list through both
+    `process_jobs_` and `process_jobs` to compare them, so the copy matters.
+    """
+    kargs = dict(kargs)
+    func = kargs.pop("func")
+    return func(**kargs)
+
+
+def process_jobs_(jobs: list[dict[str, Any]]) -> list[Any]:
+    """Ch.20 Snippet 20.8 (`processJobs_`) - sequential fallback, for debugging.
+
+    Runs every job in-process, one at a time, in submission order. No pool,
+    no pickling, no spawn - so a failing job raises immediately, at its exact
+    position in the list, with an ordinary live traceback you can attach a
+    debugger to. That clarity is the entire reason this mode exists.
+    """
+    return [expand_call(job) for job in jobs]
+
+
+def report_progress(job_num: int, num_jobs: int, time0: float, task: str) -> None:
+    """Ch.20 Snippet 20.9's `reportProgress`, modernized.
+
+    Overwrites its own line via '\\r' until the final job, then emits '\\n'.
+    Units stay the book's minutes even though a fast demo run always shows
+    "0.00 minutes" - the snippet is sized for real multi-minute workloads.
+    """
+    frac = job_num / num_jobs
+    minutes_elapsed = (time.time() - time0) / 60.0
+    minutes_remaining = minutes_elapsed * (1 / frac - 1) if frac > 0 else 0.0
+    timestamp = dt.datetime.now().isoformat(sep=" ", timespec="seconds")
+    msg = (
+        f"{timestamp} {frac * 100:5.1f}% {task} done after {minutes_elapsed:.2f} "
+        f"minutes. Remaining {minutes_remaining:.2f} minutes."
+    )
+    print(msg, end="\n" if job_num >= num_jobs else "\r", file=sys.stderr, flush=True)
+
+
+def _run_from_blob(blob: bytes) -> tuple[int, float, Any]:
+    """Executor target: undo the cloudpickle wrapping `process_jobs` applies
+    before submission, then dispatch exactly as `expand_call` always has.
+
+    Returns `(pid, atom_seconds, result)`. The pid attributes the completion to
+    a specific worker process; `atom_seconds` is measured around the call
+    itself, *inside* the worker, so it is pure compute time - it cannot include
+    queueing, spawn cost, or the idle gap between this atom finishing and the
+    rest of the run finishing. That distinction is the whole point: timing the
+    same thing from the parent would measure when the result *arrived*, not how
+    long the work actually took.
+    """
+    job = cloudpickle.loads(blob)
+    t0 = time.perf_counter()
+    result = expand_call(job)
+    return os.getpid(), time.perf_counter() - t0, result
+
+
+def _infer_task_name(jobs: list[dict[str, Any]]) -> str:
+    """Best-effort display name for a job list.
+
+    The obvious `jobs[0]["func"].__name__` crashes with AttributeError on the
+    very callables this engine goes out of its way to support: a
+    `functools.partial` (a natural way to pin one big fitted object once) and
+    any class instance implementing `__call__` have no `__name__`. Degrade to
+    the type name, then to a constant, rather than refusing to run.
+    """
+    func = jobs[0].get("func")
+    return getattr(func, "__name__", None) or type(func).__name__ or "job"
+
+
+def process_jobs(
+    jobs: list[dict[str, Any]],
+    task: str | None = None,
+    n_workers: int = os.cpu_count() or 4,
+    on_progress: Callable[[int, float, Any], None] | None = None,
+    on_job_error: Callable[[int, BaseException], None] | None = None,
+    text_progress: bool | None = None,
+) -> list[Any]:
+    """Ch.20 Snippet 20.9 (`processJobs`) - parallel dispatch over a process pool.
+
+    Named `n_workers`, not `n_threads`: this spawns OS *processes*, each with
+    its own interpreter and memory space, not threads sharing one. That
+    distinction matters here specifically - ex02 measured that CPython's GIL
+    makes threads add nothing for CPU-bound work, which is exactly what this
+    engine dispatches.
+
+    Results arrive in *completion* order, not submission order, which is what
+    lets `report_progress` report honestly as each job lands - with uneven job
+    costs (see ex03), an early-submitted heavy job can finish long after
+    several later-submitted light ones. (`orchestrator.run` re-sorts back to
+    submission order before returning, since its results are label-addressed.)
+
+    Each job is serialized with `cloudpickle` before submission (rather than
+    relying on the executor's own stdlib-`pickle` handling), so a job's 'func'
+    - or a nested callable, like `orchestrator.run`'s `save_fn` - can be a
+    closure or a lambda, not just a module-level function. `_run_from_blob` is
+    itself a plain module-level function, so it still pickles by reference with
+    no dependence on `__main__` spawn fixup.
+
+    Deliberately built on `concurrent.futures.ProcessPoolExecutor` rather than
+    the book's literal `mp.Pool` + `imap_unordered`. `mp.Pool` cannot detect a
+    worker that DIES mid-job (OOM-killed, or segfaulting inside native code
+    such as BLAS): CPython only resolves a task slot when a result arrives on
+    the output queue, and a dead process posts nothing, so the run blocks
+    forever with no exception and no log line. `ProcessPoolExecutor` watches
+    its workers and raises `BrokenProcessPool` instead - a hang that never
+    surfaces is far worse for an unattended run than a loud failure. The
+    `with` block here is also genuinely graceful, unlike `mp.Pool.__exit__`,
+    which calls `terminate()`.
+
+    `on_progress`, if given, is called as `on_progress(pid, atom_seconds,
+    result)` for every completed job - `pid` identifies which worker process
+    produced it and `atom_seconds` is that job's pure compute time as measured
+    inside the worker, which is what lets a caller (see `orchestrator.run`'s
+    `show_progress`) drive a per-worker live display with real timings. The
+    return value here is unaffected either way - still the plain `list[Any]`
+    of results, never the `(pid, atom_seconds, result)` triples.
+    `text_progress` controls the book's own `\\r`-overwriting stderr line.
+    Left at None it is automatic: emitted only when no `on_progress` display
+    is running AND stderr is a real terminal. That distinction matters in
+    production - redirected to a log file, a `\\r` line per job is unreadable
+    noise, so when the text display is off this logs periodic completion
+    milestones at INFO instead. Pass True/False to force it either way.
+
+    `on_job_error`, if given, is called as `on_job_error(index, exc)` for any
+    job that cannot be *serialized* for submission, and that job is skipped
+    instead of sinking the batch. Jobs are cloudpickled up front, so without
+    this one unpicklable payload (a lock, a live socket, an open file handle
+    captured by a closure) raises before any job runs at all - job 517 of 1000
+    taking down the 999 that were perfectly runnable. Left as None, that
+    original fail-fast behaviour is preserved, which keeps this layer
+    book-faithful; `orchestrator.run` opts in to turn such a failure into one
+    failed job, matching the per-job isolation it already promises.
+    """
+    if not jobs:
+        return []
+    if task is None:
+        task = _infer_task_name(jobs)
+
+    # Serialize per job rather than in one comprehension, so a single
+    # unpicklable payload can be attributed and skipped instead of aborting
+    # the whole submission (see `on_job_error`).
+    blobs: list[bytes] = []
+    for i, job in enumerate(jobs):
+        try:
+            blobs.append(cloudpickle.dumps(job))
+        except Exception as exc:
+            if on_job_error is None:
+                raise
+            _log.warning(
+                "job could not be serialized for dispatch, skipping "
+                "task=%s job_index=%d error=%s", task, i, exc,
+            )
+            on_job_error(i, exc)
+    if not blobs:
+        _log.warning("nothing dispatchable task=%s n_jobs=%d", task, len(jobs))
+        return []
+
+    # Never spin up more workers than there are jobs to hand them: the
+    # executor starts its processes eagerly, so surplus workers pay full spawn
+    # cost to do nothing. This only ever clamps downward - when jobs outnumber
+    # n_workers no clamp is needed, since the same fixed pool keeps pulling
+    # jobs until the list is done. Worker count is sized to hardware, not to
+    # job count.
+    out: list[Any] = []
+    time0 = time.time()
+    n_submitted = len(blobs)
+    n_pool = min(n_workers, n_submitted)
+    if n_pool < n_workers:
+        _log.warning(
+            "n_workers=%d clamped to %d task=%s reason=fewer jobs than workers",
+            n_workers, n_pool, task,
+        )
+
+    # Auto: the book's carriage-return line is a terminal affordance, so only emit it when
+    # a terminal is actually there and nothing else owns the display.
+    if text_progress is None:
+        text_progress = on_progress is None and sys.stderr.isatty()
+    # When nothing is drawing live, log milestones so a redirected run still
+    # shows movement - roughly ten lines, whatever the job count.
+    milestone = max(1, n_submitted // 10) if not text_progress else 0
+
+    _log.info(
+        "dispatch start task=%s n_jobs=%d n_workers=%d text_progress=%s",
+        task, n_submitted, n_pool, text_progress,
+    )
+    with ProcessPoolExecutor(max_workers=n_pool) as executor:
+        futures = [executor.submit(_run_from_blob, blob) for blob in blobs]
+        try:
+            for i, future in enumerate(as_completed(futures), 1):
+                pid, atom_s, out_ = future.result()
+                out.append(out_)
+                if on_progress is not None:
+                    on_progress(pid, atom_s, out_)
+                if text_progress:
+                    report_progress(i, n_submitted, time0, task)
+                elif milestone and (i % milestone == 0 or i == n_submitted):
+                    _log.info(
+                        "progress task=%s done=%d/%d pct=%.0f elapsed_s=%.2f",
+                        task, i, n_submitted, 100.0 * i / n_submitted,
+                        time.time() - time0,
+                    )
+        except BrokenProcessPool as exc:
+            # Cancel whatever has not started so shutdown does not block on
+            # work that can never complete, then re-raise with the context the
+            # bare stdlib error lacks: how far the run actually got.
+            for future in futures:
+                future.cancel()
+            _log.error(
+                "worker process died task=%s completed=%d/%d - run aborted",
+                task, len(out), n_submitted,
+            )
+            raise BrokenProcessPool(
+                f"a worker process died during task {task!r} after "
+                f"{len(out)}/{n_submitted} jobs completed - it was most likely "
+                f"OOM-killed or crashed inside native code (e.g. BLAS). The "
+                f"completed jobs' results were lost with the pool; if you need "
+                f"per-job durability use orchestrator.run, which saves each "
+                f"result to disk as it lands."
+            ) from exc
+    _log.info(
+        "dispatch done task=%s completed=%d/%d elapsed_s=%.2f",
+        task, len(out), n_submitted, time.time() - time0,
+    )
+    return out

{mpengine-0.2.0 → mpengine-0.3.0}/src/mpengine/orchestrator.py

@@ -42,6 +42,13 @@ from mpengine.engine import expand_call, process_jobs, process_jobs_
 
 N_WORKERS = os.cpu_count() or 4
 
+# Run reporting goes through logging rather than print, so embedding mpengine
+# in a larger app or a CI job does not pollute stdout. The library adds no
+# handler of its own (a library must not configure the root logger); callers
+# who want to see this call logging.basicConfig(level=logging.INFO). The
+# spider banner is the one deliberate exception and still prints.
+_log = logging.getLogger("mpengine.orchestrator")
+
 # one FileHandler-backed logger per worker *process*, lazily created on that
 # process's first job and reused for every job after - keyed by pid, which is
 # always fresh in a newly spawned worker, so no cross-run coordination needed
@@ -111,7 +118,98 @@ def load_run_outputs(
     path = Path(run_dir)
     if not path.is_dir():
         raise NotADirectoryError(f"not a run output directory: {path}")
-    return {p.name: load_fn(p) for p in sorted(path.iterdir()) if p.is_file()}
+    # Skip `.<label>.partial` leftovers: those are in-flight or abandoned
+    # writes from `_run_and_save_job`, never a finished result.
+    return {
+        p.name: load_fn(p)
+        for p in sorted(path.iterdir())
+        if p.is_file() and not (p.name.startswith(".") and p.name.endswith(".partial"))
+    }
+
+
+@contextmanager
+def _run_log_file(log_path: Path) -> Iterator[None]:
+    """Capture the parent's whole view of one run into `<log_dir>/run.log`.
+
+    The per-worker files record what each worker did; this records what the
+    run as a whole did - dispatch, milestones, per-job outcomes, the summary -
+    so one run is one self-contained, durable record. That is the audit trail
+    this library is actually differentiated on.
+
+    The handler is attached to the shared "mpengine" parent logger, so records
+    from both `mpengine.engine` and `mpengine.orchestrator` land in it.
+
+    Level handling is deliberately restrained. A logger's level decides whether
+    a record is created at all, and that decision is shared by every handler
+    downstream - so forcing DEBUG here to enrich the file would also push
+    DEBUG records into the caller's own handlers (a per-job line for all
+    10,000 jobs appearing in someone's console who asked for INFO). Instead the
+    level is only lowered as far as INFO, and only when it was coarser than
+    that, so the file always captures the full lifecycle and every failure
+    without ever making the caller's output noisier than they configured. A
+    caller who genuinely wants per-job DEBUG detail in the file just sets DEBUG
+    themselves. Everything is restored on the way out, including if the run
+    raises.
+    """
+    parent = logging.getLogger("mpengine")
+    handler = logging.FileHandler(log_path / "run.log", encoding="utf-8")
+    handler.setLevel(logging.DEBUG)
+    handler.setFormatter(
+        logging.Formatter("%(asctime)s %(levelname)-7s %(name)s: %(message)s")
+    )
+    previous_level = parent.level
+    if not parent.isEnabledFor(logging.INFO):
+        parent.setLevel(logging.INFO)
+    parent.addHandler(handler)
+    try:
+        yield
+    finally:
+        parent.removeHandler(handler)
+        handler.close()
+        parent.setLevel(previous_level)
+
+
+def _validate_labels(labels: list[str]) -> None:
+    """Reject labels that are unsafe or ambiguous as filenames.
+
+    Each label is used directly as a filename (`output_dir / label`), and
+    pathlib's `/` is unforgiving about what that permits: an absolute label
+    discards the base directory entirely, `..` walks out of the run tree, and
+    an empty label collapses onto the run directory itself so `save_fn` is
+    handed a directory to write. None of that is reachable from hostile input
+    - a caller only ever passes its own labels - but all of it silently writes
+    somewhere other than the run directory, which is worth a loud error rather
+    than a debugging session.
+
+    Duplicates are rejected for a different reason: two jobs sharing a label
+    write to one path, both report `ok` with that same `output_path`, and
+    whichever finishes last silently wins. The other result is simply gone.
+    """
+    bad: list[str] = []
+    for label in labels:
+        if not isinstance(label, str) or not label.strip():
+            bad.append(f"{label!r} (empty)")
+        elif Path(label).is_absolute() or (len(label) > 1 and label[1] == ":"):
+            bad.append(f"{label!r} (absolute path)")
+        elif "/" in label or "\\" in label:
+            bad.append(f"{label!r} (contains a path separator)")
+        elif ".." in Path(label).parts:
+            bad.append(f"{label!r} (contains '..')")
+    if bad:
+        raise ValueError(
+            "labels are used directly as output filenames, so these are not usable: "
+            + ", ".join(bad)
+        )
+
+    seen: dict[str, int] = {}
+    for label in labels:
+        seen[label] = seen.get(label, 0) + 1
+    dupes = sorted(lbl for lbl, n in seen.items() if n > 1)
+    if dupes:
+        raise ValueError(
+            "labels must be unique - each one names an output file, so duplicates "
+            f"would overwrite each other: {', '.join(repr(d) for d in dupes)}"
+        )
 
 
 def _get_worker_logger(log_dir: Path) -> logging.Logger:
@@ -133,7 +231,7 @@ def _get_worker_logger(log_dir: Path) -> logging.Logger:
 
 
 def _run_and_save_job(job: dict) -> JobResult:
-    """The actual Pool/sequential target. Unwraps its own bookkeeping fields,
+    """The actual worker/sequential target. Unwraps its own bookkeeping fields,
     calls the caller's function via `expand_call` (reusing engine.py rather
     than reimplementing the dict-to-call trick), saves the result, and always
     returns a JobResult - exceptions are caught here, never re-raised, so one
@@ -153,11 +251,24 @@ def _run_and_save_job(job: dict) -> JobResult:
         logger.error("failed %s: %s\n%s", label, exc, traceback.format_exc())
         return JobResult(label=label, status="error", error=str(exc))
 
+    # Write to a sibling temp path and only then rename into place. A save_fn
+    # that dies partway through (plausible for a large or custom serializer)
+    # would otherwise leave partial bytes at the real output path while the job
+    # is correctly recorded as failed - and a later `load_run_outputs`, which
+    # loads every file it finds, would then choke on that corpse and take down
+    # the read-back of an otherwise healthy run. os.replace is atomic on both
+    # POSIX and Windows, so the final path only ever holds a complete result.
     output_path = output_dir / label
+    tmp_path = output_dir / f".{label}.partial"
     try:
-        save_fn(result, output_path)
+        save_fn(result, tmp_path)
+        os.replace(tmp_path, output_path)
     except Exception as exc:
         logger.error("save failed %s: %s\n%s", label, exc, traceback.format_exc())
+        try:
+            tmp_path.unlink(missing_ok=True)
+        except OSError:
+            pass
         return JobResult(label=label, status="error", error=f"save failed: {exc}")
 
     logger.info("completed %s -> %s", label, output_path)
@@ -235,7 +346,11 @@ def _progress_renderer(
     overall = tqdm(total=n_jobs, desc=task, position=0, file=sys.stderr, mininterval=0)
     worker_bars: dict[int, tqdm] = {}
     available_names = _WORKER_NAMES.copy()
-    random.shuffle(available_names)
+    # A private Random instance, not `random.shuffle`. Shuffling via the module
+    # function advances the interpreter-global RNG, so picking cosmetic worker
+    # codenames would perturb any caller relying on `random` for a reproducible
+    # sequence - a real hazard for the Monte Carlo work this engine is aimed at.
+    random.Random().shuffle(available_names)
 
     def _next_worker_name() -> str:
         if available_names:
@@ -270,26 +385,34 @@ def _progress_renderer(
             bar.close()
 
 
-def _print_worker_ranking(stats: dict[int, WorkerStats]) -> None:
+def _log_worker_ranking(stats: dict[int, WorkerStats]) -> None:
     """Rank workers fastest-to-slowest by seconds per atom.
 
-    Note the caveat printed alongside: with uneven atom sizes, a low
-    s/atom can mean small atoms rather than a genuinely faster worker.
+    Emitted through `logging`, not `print`: this is run reporting, and a
+    library that writes it unconditionally to stdout breaks piping, notebooks
+    and log aggregation for anything embedding mpengine. Callers who want to
+    see it configure logging (e.g. `logging.basicConfig(level=logging.INFO)`);
+    callers who do not are no longer forced to.
+
+    Note the caveat logged alongside: with uneven atom sizes - the normal case
+    in quant work - a low s/atom can mean small atoms rather than a genuinely
+    faster worker, so the fastest/slowest tags are a hint, not a measurement.
     """
     if not stats:
         return
 
     ranked = sorted(stats.values(), key=lambda s: s.avg_atom_s)
-    print("\nworkers, fastest to slowest (by avg seconds per atom):")
+    lines = ["workers, fastest to slowest (by avg seconds per atom):"]
     for i, s in enumerate(ranked):
         tag = ""
         if len(ranked) > 1:
             tag = "  <- fastest" if i == 0 else ("  <- slowest" if i == len(ranked) - 1 else "")
-        print(
+        lines.append(
             f"  {s.name:<12} (PID {s.pid:>6})  {s.n_atoms:>3} atoms  "
             f"avg {s.avg_atom_s:6.2f}s/atom  {s.atoms_per_s:6.2f} atoms/s{tag}"
         )
-    print("  (uneven atom sizes: a low s/atom can mean small atoms, not a faster worker)")
+    lines.append("  (uneven atom sizes: a low s/atom can mean small atoms, not a faster worker)")
+    _log.info("\n".join(lines))
 
 
 def run(
@@ -353,8 +476,25 @@ def run(
             f"pass base_dir, or all of output_dir/log_dir/manifest_dir (missing: {', '.join(missing)})"
         )
 
-    task = task or func.__name__
-    run_id = f"{task}_{dt.datetime.now():%Y%m%d_%H%M%S}"
+    # `func.__name__` is not safe: a functools.partial or a callable object -
+    # both of which this engine explicitly supports - has no __name__.
+    func_name = getattr(func, "__name__", None) or type(func).__name__ or "job"
+    task = task or func_name
+
+    # Everything above and below this point is validation; no directory,
+    # manifest or banner is produced until it all passes, so a rejected call
+    # cannot leave orphaned run artefacts behind.
+    labels = labels or [f"job_{i:04d}" for i in range(len(param_sets))]
+    if len(labels) != len(param_sets):
+        raise ValueError(f"got {len(labels)} labels for {len(param_sets)} param sets")
+    _validate_labels(labels)
+
+    # Microseconds, not seconds. At second resolution two runs of the same task
+    # inside one second produced an identical run_id, and because the
+    # directories are made with exist_ok=True they were silently *shared*: the
+    # second run's manifest, opened 'w', truncated the first run's manifest
+    # outright, and default job_NNNN labels overwrote its outputs.
+    run_id = f"{task}_{dt.datetime.now():%Y%m%d_%H%M%S_%f}"
 
     output_path = Path(output_dir) / run_id
     log_path = Path(log_dir) / run_id
@@ -363,10 +503,6 @@ def run(
     Path(manifest_dir).mkdir(parents=True, exist_ok=True)
     manifest_path = Path(manifest_dir) / f"{run_id}.txt"
 
-    labels = labels or [f"job_{i:04d}" for i in range(len(param_sets))]
-    if len(labels) != len(param_sets):
-        raise ValueError(f"got {len(labels)} labels for {len(param_sets)} param sets")
-
     jobs = [
         {
             "func": _run_and_save_job,
@@ -384,7 +520,7 @@ def run(
     with open(manifest_path, "w") as f:
         f.write(f"run_id: {run_id}\n")
         f.write(f"launched: {dt.datetime.now().isoformat(sep=' ', timespec='seconds')}\n")
-        f.write(f"func: {func.__name__}\n")
+        f.write(f"func: {func_name}\n")
         f.write(f"task: {task}\n")
         f.write(f"n_workers: {n_workers}\n")
         f.write(f"debug: {debug}\n")
@@ -396,50 +532,134 @@ def run(
             f.write(f"  {label}: {params}\n")
         f.write("\n")
 
-    worker_stats: dict[int, WorkerStats] = {}
-
-    print_banner(task, n_workers, debug)
-
-    t0 = time.perf_counter()
-    if debug:
-        raw_results = process_jobs_(jobs)
-    elif show_progress:
-        with _progress_renderer(len(jobs), task, worker_stats) as on_progress:
-            raw_results = process_jobs(jobs, task=task, n_workers=n_workers, on_progress=on_progress)
-    else:
-        raw_results = process_jobs(jobs, task=task, n_workers=n_workers)
-    elapsed_s = time.perf_counter() - t0
-
-    if show_progress and not debug:
-        _print_worker_ranking(worker_stats)
+    # One durable record per run: everything from here down - dispatch,
+    # milestones, every job outcome, the summary - is captured to
+    # <log_dir>/run.log as well as going to the caller's own handlers.
+    with _run_log_file(log_path):
+        worker_stats: dict[int, WorkerStats] = {}
+
+        print_banner(task, n_workers, debug)
+        _log.info(
+            "run start run_id=%s task=%s func=%s n_jobs=%d n_workers=%d "
+            "debug=%s show_progress=%s",
+            run_id, task, func_name, len(jobs), n_workers, debug, show_progress,
+        )
+        _log.info(
+            "run dirs run_id=%s output_dir=%s log_dir=%s manifest=%s",
+            run_id, output_path, log_path, manifest_path,
+        )
 
-    n_ok = sum(1 for r in raw_results if r.status == "ok")
-    n_failed = len(raw_results) - n_ok
+        # A job that cannot even be cloudpickled for submission becomes one failed
+        # JobResult rather than sinking the whole batch - the same per-job
+        # isolation this layer already promises for jobs that fail while running.
+        unsendable: list[JobResult] = []
+
+        def on_job_error(index: int, exc: BaseException) -> None:
+            unsendable.append(
+                JobResult(
+                    label=labels[index],
+                    status="error",
+                    error=f"could not be serialized for dispatch: {exc}",
+                )
+            )
 
-    with open(manifest_path, "a") as f:
-        f.write("results:\n")
-        for r in raw_results:
-            if r.status == "ok":
-                f.write(f"  {r.label}: ok -> {r.output_path}\n")
+        def log_job(pid: int, atom_s: float, result: Any) -> None:
+            """Surface every job outcome in the PARENT log as it lands.
+
+            Successes are DEBUG - a 10,000-job sweep should not write 10,000 INFO
+            lines - but failures are WARNING, because until now a failed job was
+            only ever written to that worker's own file and an operator watching
+            the run's logs saw nothing at all until the manifest was read.
+            """
+            label = getattr(result, "label", "?")
+            if getattr(result, "status", None) == "ok":
+                _log.debug("job ok label=%s pid=%s atom_s=%.3f", label, pid, atom_s)
             else:
-                f.write(f"  {r.label}: ERROR - {r.error}\n")
-        f.write(f"\nn_ok: {n_ok}\n")
-        f.write(f"n_failed: {n_failed}\n")
-        f.write(f"elapsed_s: {elapsed_s:.3f}\n")
-
-    print(f"Logs stored here     - {log_path}")
-    print(f"Output stored here   - {output_path}")
-    print(f"Manifest stored here - {manifest_path}")
-
-    return RunSummary(
-        run_id=run_id,
-        manifest_path=str(manifest_path),
-        output_dir=str(output_path),
-        log_dir=str(log_path),
-        n_jobs=len(jobs),
-        n_ok=n_ok,
-        n_failed=n_failed,
-        elapsed_s=elapsed_s,
-        results=raw_results,
-        worker_stats=worker_stats,
-    )
+                _log.warning(
+                    "job FAILED label=%s pid=%s atom_s=%.3f error=%s",
+                    label, pid, atom_s, getattr(result, "error", None),
+                )
+
+        # The live display is a terminal affordance. Interactively it is the whole
+        # point; redirected to a log file tqdm's redraws are unreadable noise, so
+        # in prod we fall back to the engine's periodic INFO milestones instead.
+        interactive = sys.stderr.isatty()
+        live_display = show_progress and not debug and interactive
+        if show_progress and not debug and not interactive:
+            _log.info(
+                "progress display disabled run_id=%s reason=stderr is not a terminal "
+                "(logging milestones instead)", run_id,
+            )
+
+        t0 = time.perf_counter()
+        if debug:
+            raw_results = process_jobs_(jobs)
+            for r in raw_results:
+                log_job(os.getpid(), 0.0, r)
+        elif live_display:
+            with _progress_renderer(len(jobs), task, worker_stats) as render:
+                def on_progress(pid: int, atom_s: float, result: Any) -> None:
+                    log_job(pid, atom_s, result)
+                    render(pid, atom_s, result)
+
+                raw_results = process_jobs(
+                    jobs, task=task, n_workers=n_workers,
+                    on_progress=on_progress, on_job_error=on_job_error,
+                    text_progress=False,
+                )
+        else:
+            raw_results = process_jobs(
+                jobs, task=task, n_workers=n_workers,
+                on_progress=log_job, on_job_error=on_job_error,
+                text_progress=(not show_progress) and interactive,
+            )
+        elapsed_s = time.perf_counter() - t0
+
+        if worker_stats:
+            _log_worker_ranking(worker_stats)
+
+        # Restore submission order. `process_jobs` yields in completion order (by
+        # design - that is what makes its progress reporting honest), but these
+        # results are label-addressed, so a caller zipping them against the
+        # param_sets it passed in would silently mismatch. Jobs that never made it
+        # off the ground are folded back into their original positions too.
+        raw_results = raw_results + unsendable
+        label_order = {label: i for i, label in enumerate(labels)}
+        raw_results.sort(key=lambda r: label_order.get(r.label, len(label_order)))
+
+        n_ok = sum(1 for r in raw_results if r.status == "ok")
+        n_failed = len(raw_results) - n_ok
+
+        with open(manifest_path, "a") as f:
+            f.write("results:\n")
+            for r in raw_results:
+                if r.status == "ok":
+                    f.write(f"  {r.label}: ok -> {r.output_path}\n")
+                else:
+                    f.write(f"  {r.label}: ERROR - {r.error}\n")
+            f.write(f"\nn_ok: {n_ok}\n")
+            f.write(f"n_failed: {n_failed}\n")
+            f.write(f"elapsed_s: {elapsed_s:.3f}\n")
+
+        _log.info(
+            "run done run_id=%s task=%s n_ok=%d n_failed=%d elapsed_s=%.2f "
+            "throughput_jobs_s=%.2f",
+            run_id, task, n_ok, n_failed, elapsed_s,
+            (len(raw_results) / elapsed_s) if elapsed_s > 0 else 0.0,
+        )
+        _log.info("Logs stored here     - %s", log_path)
+        _log.info("Output stored here   - %s", output_path)
+        _log.info("Manifest stored here - %s", manifest_path)
+
+        return RunSummary(
+            run_id=run_id,
+            manifest_path=str(manifest_path),
+            output_dir=str(output_path),
+            log_dir=str(log_path),
+            n_jobs=len(jobs),
+            n_ok=n_ok,
+            n_failed=n_failed,
+            elapsed_s=elapsed_s,
+            results=raw_results,
+            worker_stats=worker_stats,
+        )

{mpengine-0.2.0 → mpengine-0.3.0}/src/mpengine/partition.py

@@ -39,6 +39,48 @@ def lin_parts(num_atoms: int, num_threads: int) -> np.ndarray:
     return np.ceil(parts).astype(int)
 
 
+def _integer_boundaries(float_parts: list[float], num_atoms: int, num_molecules: int) -> np.ndarray:
+    """Turn exact float boundaries into integer ones without collapsing any
+    molecule to zero width.
+
+    Rounding each boundary independently (the obvious `np.round(parts)`) lets
+    two adjacent float boundaries land on the same integer, which yields an
+    empty molecule - i.e. a dispatched job with no work in it. That is not
+    rare: for `nested_parts` it happened for ~68% of (num_atoms, num_threads)
+    pairs under 80, including such ordinary cases as (10, 8).
+
+    Instead, allocate integer *widths* by largest remainder: floor each float
+    width, force a floor of 1 atom per molecule, then reconcile the total back
+    to `num_atoms` by handing surplus atoms to the largest fractional
+    remainders (or reclaiming from the smallest). A valid all-non-empty
+    assignment always exists here because callers clamp
+    `num_molecules <= num_atoms` first.
+    """
+    widths = np.diff(np.asarray(float_parts, dtype=float))
+    base = np.floor(widths).astype(np.int64)
+    frac = widths - np.floor(widths)
+    base = np.maximum(base, 1)
+
+    shortfall = int(num_atoms) - int(base.sum())
+    if shortfall > 0:
+        order = np.argsort(-frac, kind="stable")
+        for k in range(shortfall):
+            base[order[k % num_molecules]] += 1
+    elif shortfall < 0:
+        order = np.argsort(frac, kind="stable")
+        k = 0
+        guard = 0
+        while shortfall < 0 and guard < 100 * num_molecules:
+            idx = order[k % num_molecules]
+            if base[idx] > 1:
+                base[idx] -= 1
+                shortfall += 1
+            k += 1
+            guard += 1
+
+    return np.concatenate(([0], np.cumsum(base))).astype(int)
+
+
 def nested_parts(num_atoms: int, num_threads: int, upper_triang: bool = False) -> np.ndarray:
     """Ch.20 Snippet 20.6 (`nestedParts`) - boundary indices for up to
     `num_threads` equal-*work* partitions of a triangular-cost workload where
@@ -52,15 +94,22 @@ def nested_parts(num_atoms: int, num_threads: int, upper_triang: bool = False) -
 
     `upper_triang=True` reverses which end carries the heavy rows - use it
     when atom cost *decreases* with atom index instead of increasing.
+
+    The float boundaries are solved exactly and converted to integers by
+    `_integer_boundaries`, which guarantees no molecule comes back empty -
+    rounding each boundary independently silently produced zero-width
+    molecules for the majority of input pairs.
     """
     num_threads_ = min(num_threads, num_atoms)
+    if num_atoms <= 0 or num_threads_ <= 0:
+        return np.array([0], dtype=int)
     parts = [0.0]
     for _ in range(num_threads_):
         prev = parts[-1]
         part = 1 + 4 * (prev**2 + prev + num_atoms * (num_atoms + 1) / num_threads_)
         part = (-1 + part**0.5) / 2
         parts.append(part)
-    parts_arr = np.round(parts).astype(int)
+    parts_arr = _integer_boundaries(parts, num_atoms, num_threads_)
     if upper_triang:
         parts_arr = np.concatenate(([0], np.cumsum(np.diff(parts_arr)[::-1])))
     return parts_arr

{mpengine-0.2.0 → mpengine-0.3.0/src/mpengine.egg-info}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: mpengine
-Version: 0.2.0
+Version: 0.3.0
 Summary: A small, general-purpose multiprocessing engine: dispatch any callable across processes with run manifests, per-worker logs, on-disk outputs and failure isolation
 Author: Amandeep Singh
 License-Expression: MIT
@@ -42,6 +42,7 @@ Every run produces three things:
 | a manifest of exactly what was launched, and how it ended | `<base>/manifests/<run_id>.txt` |
 | each job's result, saved to disk | `<base>/outputs/<run_id>/<label>` |
 | one log file per **worker process** | `<base>/logs/<run_id>/worker_<pid>.log` |
+| a log of the run as a whole | `<base>/logs/<run_id>/run.log` |
 
 A single job failing does **not** abort the run — it is recorded as a failed
 job and the rest continue.
@@ -127,9 +128,43 @@ run with a custom `save_fn`, pass the matching reader as
 | `show_progress` | live terminal display: one overall bar for the whole run, plus a live rate number per worker process. Ignored when `debug=True` |
 
 Develop with `debug=True`, then flip it off. Chasing a bug through a process
-pool means reading a `RemoteTraceback` from a worker that has already exited,
+pool means reading a traceback re-raised from a worker that has already exited,
 and it never tells you which job dict was at fault.
 
+### Logging
+
+The library logs its whole lifecycle through the standard `logging` module and
+writes nothing to stdout except the banner, so embedding it never pollutes a
+host application's output. It installs only a `NullHandler` — to see anything,
+configure logging yourself:
+
+```python
+import logging
+logging.basicConfig(level=logging.INFO)
+```
+
+Two loggers, so either half can be tuned or silenced independently:
+
+| logger | emits |
+|---|---|
+| `mpengine.orchestrator` | run start/done, resolved dirs, per-job outcomes, worker ranking |
+| `mpengine.engine` | dispatch start/done, worker-count clamping, progress milestones, worker death |
+
+Levels: **INFO** for run lifecycle, **DEBUG** for per-job success (a 10,000-job
+sweep would otherwise be 10,000 INFO lines), **WARNING** for a failed job or a
+clamped worker count, **ERROR** for a dead worker.
+
+Every run also writes `<log_dir>/<run_id>/run.log` containing the parent's
+whole view of that run — dispatch, milestones, every job outcome, the summary —
+alongside the existing per-worker log files. That happens regardless of how you
+configure logging, so a run is always a self-contained durable record.
+
+**Progress display is terminal-aware.** Interactively you get the tqdm bars and
+per-worker lines as before. When stderr is not a terminal — piped, redirected to
+a log file, running under CI or a scheduler — those are suppressed (they render
+as unreadable escape-sequence noise in a file) and periodic completion
+milestones are logged at INFO instead.
+
 ### Closures and lambdas
 
 `func`, and any custom `save_fn`, can be a closure or a lambda — not just a
@@ -168,6 +203,11 @@ for triangular workloads — where item `i` costs `O(i)`, such as an
 expanding-window computation — which keeps workers from idling while one
 overloaded worker finishes.
 
+## Changelog
+
+See [CHANGELOG.md](CHANGELOG.md). Versions below 1.0 may carry breaking changes
+in a minor release; each is listed there with the migration needed.
+
 ## What else is in the repo
 
 The `src/learning/` package holds demo and exercise scripts used while

mpengine-0.2.0/src/mpengine/engine.py

@@ -1,152 +0,0 @@
-"""Ch.20 SS20.5 - the multiprocessing engine (exercises 6+7).
-
-expand_call / process_jobs_ / process_jobs / report_progress - a generic
-job-dict-to-function-call pipeline, reusable by any exercise rather than
-tied to one atom/molecule shape. This is the book's own point: stop writing
-a bespoke parallelization wrapper per function, and build one library that
-can parallelize unknown functions regardless of their arguments or output.
-
-A "job" here is just a dict: one 'func' entry (the callback) plus whatever
-kwargs that callback needs. `expand_call` turns such a dict into a call,
-which is the hinge the whole engine turns on.
-"""
-
-from __future__ import annotations
-
-import datetime as dt
-import multiprocessing as mp
-import os
-import sys
-import time
-from typing import Any, Callable
-
-import cloudpickle
-
-
-def expand_call(kargs: dict[str, Any]) -> Any:
-    """Ch.20 Snippet 20.10 (`expandCall`) - turn a job dict into a call.
-
-    Deviation from the book: `kargs` is copied before popping. The book's
-    literal `kargs.pop('func')` mutates the *caller's* dict in place, so the
-    same job list can only ever be run once - a second pass raises
-    KeyError('func'). ex04 deliberately runs one job list through both
-    `process_jobs_` and `process_jobs` to compare them, so the copy matters.
-    """
-    kargs = dict(kargs)
-    func = kargs.pop("func")
-    return func(**kargs)
-
-
-def process_jobs_(jobs: list[dict[str, Any]]) -> list[Any]:
-    """Ch.20 Snippet 20.8 (`processJobs_`) - sequential fallback, for debugging.
-
-    Runs every job in-process, one at a time, in submission order. No pool,
-    no pickling, no spawn - so a failing job raises immediately, at its exact
-    position in the list, with an ordinary live traceback you can attach a
-    debugger to. That clarity is the entire reason this mode exists.
-    """
-    return [expand_call(job) for job in jobs]
-
-
-def report_progress(job_num: int, num_jobs: int, time0: float, task: str) -> None:
-    """Ch.20 Snippet 20.9's `reportProgress`, modernized.
-
-    Overwrites its own line via '\\r' until the final job, then emits '\\n'.
-    Units stay the book's minutes even though a fast demo run always shows
-    "0.00 minutes" - the snippet is sized for real multi-minute workloads.
-    """
-    frac = job_num / num_jobs
-    minutes_elapsed = (time.time() - time0) / 60.0
-    minutes_remaining = minutes_elapsed * (1 / frac - 1) if frac > 0 else 0.0
-    timestamp = dt.datetime.now().isoformat(sep=" ", timespec="seconds")
-    msg = (
-        f"{timestamp} {frac * 100:5.1f}% {task} done after {minutes_elapsed:.2f} "
-        f"minutes. Remaining {minutes_remaining:.2f} minutes."
-    )
-    print(msg, end="\n" if job_num >= num_jobs else "\r", file=sys.stderr, flush=True)
-
-
-def _run_from_blob(blob: bytes) -> tuple[int, float, Any]:
-    """Pool target: undo the cloudpickle wrapping `process_jobs` applies before
-    submission, then dispatch exactly as `expand_call` always has.
-
-    Returns `(pid, atom_seconds, result)`. The pid attributes the completion to
-    a specific worker process; `atom_seconds` is measured around the call
-    itself, *inside* the worker, so it is pure compute time - it cannot include
-    queueing, spawn cost, or the idle gap between this atom finishing and the
-    rest of the run finishing. That distinction is the whole point: timing the
-    same thing from the parent would measure when the result *arrived*, not how
-    long the work actually took.
-    """
-    job = cloudpickle.loads(blob)
-    t0 = time.perf_counter()
-    result = expand_call(job)
-    return os.getpid(), time.perf_counter() - t0, result
-
-
-def process_jobs(
-    jobs: list[dict[str, Any]],
-    task: str | None = None,
-    n_workers: int = os.cpu_count() or 4,
-    on_progress: Callable[[int, float, Any], None] | None = None,
-) -> list[Any]:
-    """Ch.20 Snippet 20.9 (`processJobs`) - real `mp.Pool` + `imap_unordered`.
-
-    Named `n_workers`, not `n_threads`: this spawns OS *processes* via
-    `mp.Pool`, each with its own interpreter and memory space, not threads
-    sharing one. That distinction matters here specifically - ex02 measured
-    that CPython's GIL makes threads add nothing for CPU-bound work, which is
-    exactly what this engine dispatches.
-
-    Results arrive in *completion* order, not submission order, which is what
-    lets `report_progress` report honestly as each job lands - with uneven job
-    costs (see ex03), an early-submitted heavy job can finish long after
-    several later-submitted light ones.
-
-    Each job is serialized with `cloudpickle` before submission (rather than
-    relying on `Pool`'s own stdlib-`pickle` handling of `jobs`), so a job's
-    'func' - or a nested callable, like `orchestrator.run`'s `save_fn` - can be
-    a closure or a lambda, not just a module-level function. `_run_from_blob`
-    is itself a plain module-level function, so it still pickles by reference
-    with no dependence on `__main__` spawn fixup.
-
-    `on_progress`, if given, is called as `on_progress(pid, atom_seconds,
-    result)` for every completed job - `pid` identifies which worker process
-    produced it and `atom_seconds` is that job's pure compute time as measured
-    inside the worker, which is what lets a caller (see `orchestrator.run`'s
-    `show_progress`) drive a per-worker live display with real timings. The
-    return value here is unaffected either way - still the plain `list[Any]`
-    of results, never the `(pid, atom_seconds, result)` triples.
-    Supplying `on_progress` means the caller is rendering its own progress
-    display, so the book's own text-based `report_progress` is skipped for
-    that call - the two would otherwise both write to stderr and garble each
-    other's redraws.
-
-    Uses explicit close()+join() rather than `with mp.Pool(...) as pool:`,
-    whose __exit__ calls terminate() - an abrupt kill, not the book's graceful
-    shutdown. The try/finally still lets a job's exception propagate while
-    guaranteeing the workers are torn down.
-    """
-    if task is None:
-        task = jobs[0]["func"].__name__
-    blobs = [cloudpickle.dumps(job) for job in jobs]
-    # `Pool(processes=n)` spawns exactly n OS processes immediately, whether
-    # or not there's n jobs' worth of work - never spin up more workers than
-    # there are jobs to hand them. This only ever clamps downward: when jobs
-    # outnumber n_workers, no clamp is needed at all - the same fixed pool
-    # keeps pulling jobs off the queue, one per worker at a time, until the
-    # whole list is done. Worker count is sized to hardware, not job count.
-    pool = mp.Pool(processes=min(n_workers, len(jobs)))
-    out: list[Any] = []
-    time0 = time.time()
-    try:
-        for i, (pid, atom_s, out_) in enumerate(pool.imap_unordered(_run_from_blob, blobs), 1):
-            out.append(out_)
-            if on_progress is not None:
-                on_progress(pid, atom_s, out_)
-            else:
-                report_progress(i, len(jobs), time0, task)
-    finally:
-        pool.close()
-        pool.join()
-    return out