processes 2.0.1__tar.gz → 3.0.1__tar.gz

processes-3.0.1/.github/PULL_REQUEST_TEMPLATE.md

@@ -0,0 +1,10 @@
+## What and why
+<!-- Describe the change and the motivation behind it -->
+
+## Type of change
+- [ ] `feat` — new feature
+- [ ] `fix` — bug fix
+- [ ] `refactor` — no behavior change
+- [ ] `docs` — documentation only
+- [ ] `test` — tests only
+- [ ] `chore` / `ci` / `build`

{processes-2.0.1 → processes-3.0.1}/CHANGELOG.md

@@ -1,3 +1,24 @@
+## v3.0.1 (2026-06-13)
+
+### Refactor
+
+- **process**: use bare raise and extract _is_done helper
+
+## v3.0.0 (2026-06-13)
+
+### BREAKING CHANGE
+
+- html_mail_handler parameter removed from Task; use
+smtp_config and email_style instead. HTMLSMTPHandler removed from public API.
+
+### Feat
+
+- **html_logging**: expose last_path_traced_vars on HTMLSMTPHandler
+
+### Refactor
+
+- **email**: replace HTMLSMTPHandler with SMTPConfig/HTMLEmailStyle
+
 ## v2.0.1 (2026-06-12)
 
 ### Fix

processes-3.0.1/CONTRIBUTING.md

@@ -0,0 +1,135 @@
+# Contributing to Processes
+
+Thanks for your interest in contributing! This document covers the two
+contributor paths, project conventions, and how to submit a change.
+
+## Contributor paths
+
+**Maintainer** (direct push access): clone the repository directly.
+
+```bash
+git clone https://github.com/oliverm91/processes.git
+cd processes
+uv sync
+```
+
+**External contributor** (everyone else): fork first, then clone your fork.
+
+```bash
+# 1. Fork on GitHub (button top-right of the repo page)
+# 2. Clone your fork
+git clone https://github.com/<your-username>/processes.git
+cd processes
+uv sync
+```
+
+Both paths use [uv](https://docs.astral.sh/uv/) for dependency management.
+`uv sync` installs all dev dependencies (pytest, mypy, ruff, mkdocs, commitizen).
+
+## Project layout
+
+```
+src/processes/   # library source (the public package)
+tests/           # pytest test suite
+examples/        # runnable usage examples
+docs/            # mkdocs documentation source
+```
+
+Keep new public symbols in `src/processes/` and re-export them from
+`src/processes/__init__.py` if they are part of the stable API.
+
+## Running checks locally
+
+Before opening a pull request, make sure the following pass:
+
+```bash
+uv run pytest          # test suite
+uv run mypy            # strict type checking
+uv run ruff check .    # linting
+uv run ruff format .   # formatting
+```
+
+CI runs the same checks on every push and pull request.
+
+## Style guide
+
+- Python >= 3.10 syntax.
+- Type hints on all new code. `mypy` is configured with `strict = true` and
+  `disallow_untyped_defs = true`.
+- Ruff rules: `F`, `E`, `W`, `I`, `B`, `UP`. Line length is 100.
+- Prefer the standard library — the library has zero runtime dependencies.
+- No comments that just restate the code. A comment is welcome only when it
+  captures a non-obvious *why*.
+
+## Tests
+
+- Add or update tests in `tests/` for any behavior change.
+- Keep tests focused and deterministic; avoid sleeps where possible.
+- If you fix a bug, add a regression test that fails before the fix.
+
+## Commit messages
+
+This project follows [Conventional Commits](https://www.conventionalcommits.org/),
+enforced by [commitizen](https://commitizen-tools.github.io/commitizen/). Use
+`uv run cz commit` to be prompted through an interactive commit, or write the
+message by hand:
+
+```
+<type>(<scope>): <short description>
+
+<optional body>
+
+<optional footer>
+```
+
+Common types: `feat`, `fix`, `docs`, `style`, `refactor`, `test`, `chore`,
+`build`, `ci`.
+
+## Submitting a pull request
+
+1. Create a feature branch: `git checkout -b feat/my-change`.
+2. Make your changes, add tests, and confirm all checks pass.
+3. Push the branch to your fork (or directly, if you're a maintainer) and open
+   a PR against `main` on the upstream repository.
+4. Fill in a clear description of the *what* and the *why*.
+5. Make sure CI is green and the changelog preview looks reasonable for
+   user-visible changes.
+6. Be ready to revise — code review is a normal part of the process.
+
+Prefer several small, focused PRs over one large one.
+
+## Documentation
+
+User-facing changes should be reflected in the docs under `docs/`. To preview
+locally:
+
+```bash
+uv run mkdocs serve
+```
+
+## Versioning and releases
+
+Versions are derived from git tags via `hatch-vcs` (`v$version`, e.g. `v2.0.1`).
+Do not edit the version in `pyproject.toml` manually. To cut a release, the
+maintainer runs:
+
+```bash
+uv run cz bump
+```
+
+This updates the version, regenerates the changelog, creates a tag, and
+triggers the `publish.yml` workflow to push the release to PyPI.
+
+## Reporting issues
+
+Use the GitHub issue tracker. Please include:
+
+- A minimal, self-contained reproduction.
+- The Python version, library version, and operating system.
+- The full traceback for crashes, and the smallest possible task graph that
+  triggers the problem.
+
+## License
+
+By contributing, you agree that your contributions will be licensed under the
+MIT License.

processes-3.0.1/PKG-INFO

@@ -0,0 +1,407 @@
+Metadata-Version: 2.4
+Name: processes
+Version: 3.0.1
+Summary: A Python library for managing and executing dependent tasks in parallel or sequential order with automatic dependency resolution and topological sorting
+Project-URL: Homepage, https://github.com/oliverm91/processes
+Project-URL: Documentation, https://oliverm91.github.io/processes/
+Project-URL: Repository, https://github.com/oliverm91/processes
+Project-URL: Issues, https://github.com/oliverm91/processes/issues
+Project-URL: Changelog, https://github.com/oliverm91/processes/blob/main/CHANGELOG.md
+Author-email: Oliver Mohr Bonometti <oliver.mohr.b@gmail.com>
+License-Expression: MIT
+License-File: LICENSE
+Keywords: dag,dependencies,etl,parallel,process,tasks,topological-sort,workflow
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+
+<div align="center">
+  <img src="https://raw.githubusercontent.com/oliverm91/processes/refs/heads/main/assets/banner.svg" width="100%" alt="Processes - Smart Task Orchestration">
+</div>
+
+# Processes: Smart Task Orchestration
+
+[![Python Version](https://img.shields.io/badge/python-3.10%2B-blue.svg)](https://www.python.org/)
+![Fast & Lightweight](https://img.shields.io/badge/Library-Pure%20Python-green.svg)
+[![Documentation](https://img.shields.io/badge/docs-GitHub%20Pages-blue.svg)](https://oliverm91.github.io/processes/)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
+[![PyPI version](https://img.shields.io/pypi/v/processes.svg)](https://pypi.org/project/processes/)
+
+[![Python Tests Status](https://github.com/oliverm91/processes/actions/workflows/tests.yml/badge.svg?branch=main)](https://github.com/oliverm91/processes/actions/workflows/tests.yml)
+[![Ruff Lint Status](https://github.com/oliverm91/processes/actions/workflows/lint.yml/badge.svg?branch=main)](https://github.com/oliverm91/processes/actions/workflows/lint.yml)
+[![mypy-check](https://github.com/oliverm91/processes/actions/workflows/mypy.yml/badge.svg)](https://github.com/oliverm91/processes/actions/workflows/mypy.yml)
+
+---
+
+**Run a list of Python callables that depend on each other — in parallel when possible, with per-task log files and optional HTML email notification on failure. Zero dependencies. Pure Python 3.10+.**
+
+---
+
+## ✨ Why Processes?
+
+- 🔗 **Declare what depends on what** — write your tasks in any order; the runtime sorts them so every dependency runs first.
+- ⚡ **Run in parallel when you can** — independent tasks run together on a thread pool; the runtime switches on automatically for jobs with 10+ tasks.
+- 🛡️ **One failure doesn't stop the rest** — a failed task skips only the jobs that depend on it, and **every other part of the workflow keeps running**.
+- 📝 **One log file per task** — share a single log across the whole run, or keep them separate for easier debugging.
+- 📧 **Email alerts when something breaks** — pass an `SMTPConfig` to a task and get a styled HTML email (with traceback, task context, and the list of jobs that were skipped) the instant it raises.
+- 🧰 **Modern, strictly-typed Python 3.10+** — `from __future__ import annotations`, full `mypy --strict` clean, `dict[str, TaskResult]`, `set[str]`, `|` unions.
+
+---
+
+## ⚙️ How it works
+
+A `Process` holds a list of `Task`s. At construction it validates names, types, dependency references, and detects cycles — raising before anything runs. 
+
+When you call `process.run()`, tasks are topologically sorted and scheduled: dependencies first, independent tasks in parallel.
+
+A `TaskDependency` can forward an upstream result directly into a downstream function, as a positional or keyword argument. The result is a `ProcessResult` with `passed_tasks_results` and `failed_tasks` for inspection.
+
+---
+
+## 🚀 Quick start
+
+A 15-line "hello pipeline" — one upstream task feeding a downstream one, run in parallel.
+
+```python
+from processes import Process, Task, TaskDependency
+
+
+def load_users() -> list[dict]:
+    return [{"id": 1}, {"id": 2}, {"id": 3}]
+
+
+def enrich(users: list[dict]) -> list[dict]:
+    return [{**u, "name": f"user-{u['id']}"} for u in users]
+
+
+tasks = [
+    Task("load", "run.log", load_users),
+    Task(
+        "enrich",
+        "run.log",
+        enrich,
+        dependencies=[TaskDependency("load", use_result_as_additional_args=True)],
+    ),
+]
+
+with Process(tasks) as p:
+    result = p.run(parallel=True)
+
+print(result.passed_tasks_results["enrich"].result)
+# [{'id': 1, 'name': 'user-1'}, {'id': 2, 'name': 'user-2'}, {'id': 3, 'name': 'user-3'}]
+```
+
+---
+
+## 🧪 End-to-end example
+
+A realistic mini-pipeline: fetch two sources **in parallel**, transform them, aggregate, and notify — with per-task log files, result piping, and one task deliberately failing to show fault isolation.
+
+<details>
+<summary>Show the full end-to-end example</summary>
+
+```python
+import logging
+from pathlib import Path
+
+from processes import HTMLEmailStyle, Process, SMTPConfig, Task, TaskDependency
+
+LOG_DIR = Path("logs")
+LOG_DIR.mkdir(exist_ok=True)
+
+
+# --- 1. Two independent "fetch" tasks that run in parallel -----------------
+def fetch_orders() -> list[dict]:
+    logging.info("querying orders API")
+    return [{"order_id": 1, "amount": 42.0}, {"order_id": 2, "amount": 17.5}]
+
+
+def fetch_inventory() -> list[dict]:
+    logging.info("querying inventory API")
+    return [{"sku": "A-1", "qty": 12}, {"sku": "B-2", "qty": 3}]
+
+
+# --- 2. Two transforms that consume the upstream results -------------------
+def total_revenue(orders: list[dict]) -> float:
+    total = sum(o["amount"] for o in orders)
+    logging.info("revenue computed: %s", total)
+    return total
+
+
+def stock_value(inventory: list[dict], *, price_per_unit: float = 10.0) -> float:
+    value = sum(i["qty"] for i in inventory) * price_per_unit
+    logging.info("stock value: %s", value)
+    return value
+
+
+# --- 3. An aggregator that joins the two branches --------------------------
+def build_report(*, revenue: float, stock: float) -> str:
+    return f"daily-report | revenue={revenue:.2f} stock={stock:.2f}"
+
+
+# --- 4. A flaky notifier that ALWAYS fails — to show fault isolation -------
+def notify_slack(report: str) -> None:
+    raise RuntimeError("slack webhook returned 503")
+
+
+# --- 5. A sibling task that does NOT depend on notify and still runs -------
+def archive_report(report: str) -> str:
+    out = LOG_DIR / "report.txt"
+    out.write_text(report)
+    return str(out)
+
+
+# --- 6. Optional: SMTP config so failures page on-call --------------------
+smtp = SMTPConfig(
+    mailhost=("smtp.example.com", 587),
+    fromaddr="alerts@example.com",
+    toaddrs=["oncall@example.com"],
+    credentials=("user", "pass"),
+    secure=(),
+)
+
+tasks = [
+    Task("fetch_orders",   LOG_DIR / "fetch_orders.log",   fetch_orders),
+    Task("fetch_inventory", LOG_DIR / "fetch_inventory.log", fetch_inventory),
+
+    Task(
+        "compute_revenue",
+        LOG_DIR / "compute_revenue.log",
+        total_revenue,
+        dependencies=[TaskDependency("fetch_orders", use_result_as_additional_args=True)],
+    ),
+    Task(
+        "compute_stock",
+        LOG_DIR / "compute_stock.log",
+        stock_value,
+        kwargs={"price_per_unit": 7.25},
+        dependencies=[
+            TaskDependency(
+                "fetch_inventory",
+                use_result_as_additional_kwargs=True,
+                additional_kwarg_name="inventory",
+            )
+        ],
+    ),
+
+    Task(
+        "build_report",
+        LOG_DIR / "build_report.log",
+        build_report,
+        dependencies=[
+            TaskDependency("compute_revenue", use_result_as_additional_kwargs=True,
+                           additional_kwarg_name="revenue"),
+            TaskDependency("compute_stock",    use_result_as_additional_kwargs=True,
+                           additional_kwarg_name="stock"),
+        ],
+    ),
+
+    # notify_slack fails on purpose. archive_report is a *sibling*
+    # of notify_slack (both depend on build_report), so it has no
+    # dependency on the failed task and runs normally — the rest of
+    # the workflow is not blackholed by one broken step.
+    Task(
+        "notify_slack",
+        LOG_DIR / "notify_slack.log",
+        notify_slack,
+        dependencies=[TaskDependency("build_report", use_result_as_additional_args=True)],
+        smtp_config=smtp,
+    ),
+    Task(
+        "archive_report",
+        LOG_DIR / "archive_report.log",
+        archive_report,
+        dependencies=[TaskDependency("build_report", use_result_as_additional_args=True)],
+    ),
+]
+
+with Process(tasks) as process:
+    result = process.run(parallel=True)
+
+print("passed:", sorted(result.passed_tasks_results))
+# archive_report, build_report, compute_revenue, compute_stock, fetch_inventory, fetch_orders
+print("failed:", sorted(result.failed_tasks))
+# notify_slack
+print("report:", result.passed_tasks_results["build_report"].result)
+# daily-report | revenue=59.50 stock=262.50
+```
+
+The failing `notify_slack` task does **not** abort the run. `archive_report` is a sibling of the failed task (both depend on the successful `build_report`), so it runs unaffected — the rest of the workflow is not blackholed by one broken step. The HTML email handler also fires on the `notify_slack` task, paging on-call with the full traceback and the list of downstream tasks that were skipped because of it.
+
+</details>
+
+---
+
+## 📚 API Reference
+
+<details>
+<summary>Show API reference</summary>
+
+### `Task`
+
+```python
+Task(
+    name: str,
+    log_path: str | os.PathLike,
+    func: Callable[..., Any],
+    args: tuple = (),
+    kwargs: dict | None = None,
+    dependencies: list[TaskDependency] | None = None,
+    smtp_config: SMTPConfig | None = None,
+    email_style: HTMLEmailStyle | None = None,
+    timeout: float | None = None,
+    retries: int | None = 0,
+    retry_on: tuple[type[Exception], ...] | None = None,
+)
+```
+
+- `name` — unique within the `Process`; no spaces.
+- `log_path` — the file this task logs to (INFO level, format `%(asctime)s - %(name)s - %(levelname)s - %(message)s`).
+- `func` — the callable; receives `func(*args, **kwargs)` after result-injection.
+- `smtp_config` — when set, fires an HTML email on `logging.ERROR`; body includes `task_name`, `function`, `args`, `kwargs`, and `downstream_impact`.
+- `email_style` — optional presentation override; defaults to `HTMLEmailStyle()` (modern, neutral, English) when `smtp_config` is set.
+- `timeout` — seconds allowed per attempt; `None` means no limit. When the timeout fires the underlying thread is detached (Python threading limitation).
+- `retries` — additional attempts after the first failure; `0` or `None` means a single attempt. Defaults to `0`.
+- `retry_on` — tuple of exception types that trigger a retry. When `retries >= 1` and `retry_on` is `None`, defaults to `(ConnectionError, TimeoutError)` at call time.
+
+### `TaskDependency`
+
+```python
+TaskDependency(
+    task_name: str,
+    use_result_as_additional_args: bool = False,
+    use_result_as_additional_kwargs: bool = False,
+    additional_kwarg_name: str = "",
+)
+```
+
+- `use_result_as_additional_args=True` — upstream result appended as the next **positional** arg.
+- `use_result_as_additional_kwargs=True` with a non-empty `additional_kwarg_name` — upstream result injected as a **keyword** arg.
+- Both flags can be combined (positional first, then kwarg).
+
+### `Process`
+
+```python
+Process(tasks: list[Task])  # validates types, names, deps, cycles
+
+process.run(parallel: bool | None = None, max_workers: int = 4) -> ProcessResult
+```
+
+- Raises `DependencyNotFoundError`, `CircularDependencyError`, `TypeError`, `ValueError` on construction if the workflow is malformed.
+- `parallel=None` auto-parallelises when `len(tasks) >= 10`; `max_workers=1` is always sequential.
+- Use as a context manager — it cleans up `FileHandler`s on exit.
+
+### `ProcessResult`
+
+```python
+result.passed_tasks_results  # dict[str, TaskResult] — name → TaskResult for every task that succeeded
+result.failed_tasks          # set[str] — all tasks that did not produce a result (errored + skipped)
+result.errored_tasks         # set[str] — tasks whose function actually raised
+result.skipped_tasks         # set[str] — tasks skipped because an upstream dependency failed
+
+TaskResult(worked: bool, result: Any, exception: Exception | None)
+```
+
+### `SMTPConfig`
+
+```python
+SMTPConfig(
+    mailhost,                      # (host, port)
+    fromaddr,
+    toaddrs,                       # list[str]
+    credentials=None,              # (username, password) | None
+    secure=None,                   # () = STARTTLS; omit for no encryption
+    timeout=5,
+)
+```
+
+### `HTMLEmailStyle`
+
+```python
+HTMLEmailStyle(
+    style="modern",                # classic | modern | compact
+    palette="neutral",             # neutral | catppuccin | neobones | slate
+    language="en",                 # en | es | pt | fr | de | it
+    traced_vars_frame_filter=None, # substring to pick the traced frame | None
+)
+```
+
+All fields are optional — omit `HTMLEmailStyle` entirely to use the defaults.
+
+</details>
+
+<details>
+<summary>Show fault-tolerance rules in detail</summary>
+
+When a task raises:
+
+1. The exception is caught and stored in `TaskResult.exception`; the task name goes into `failed_tasks` and `errored_tasks`.
+2. **Every task that depends on it (directly or indirectly) is skipped** — added to `failed_tasks` and `skipped_tasks` without running.
+3. **Every other independent part of the workflow keeps running.** With `parallel=True` they keep running concurrently on the worker pool.
+4. After `run()` returns, `ProcessResult.errored_tasks` and `ProcessResult.skipped_tasks` let you distinguish root failures from cascade skips for triage or alerting.
+
+When a task has `retries >= 1`, a failure matching `retry_on` triggers another attempt before the task is declared failed and its dependants are skipped. This gives transient errors (network blips, connection resets) a chance to resolve without aborting downstream work.
+
+This makes the library a good fit for fan-out / fan-in pipelines, "best-effort" notifications, and any workflow where one broken step should not blackhole the rest.
+
+</details>
+
+<details>
+<summary>Show comparison with other libraries</summary>
+
+| | **Processes** | Airflow | Celery | Luigi |
+|---|---|---|---|---|
+| External dependencies | **None** | many | broker (Redis/RabbitMQ) | few |
+| Setup cost | `pip install` | cluster | broker + workers | task + config |
+| Parallelism | built-in | via executors | via workers | via workers |
+| Per-task file logs | **yes (built-in)** | via handlers | via signals | partial |
+| HTML email on failure | **yes (built-in)** | via callbacks | via signals | manual |
+| DAG validation at construction | **yes** | yes (DAG file) | n/a | partial |
+| Strict typing (`mypy --strict`) | **yes** | partial | partial | no |
+
+`Processes` is **not** a distributed scheduler — there are no workers on remote machines, no SLA monitoring, no web UI. If you need any of those, you need Airflow or a similar orchestrator. If you want a small, fast, dependency-aware pipeline that *just runs* in a single process, this is it.
+
+</details>
+
+<details>
+<summary>Show advanced configuration</summary>
+
+- **Shared log file** — pass the same `log_path` to every `Task` for a single combined run.log; pass distinct paths for per-task isolation.
+- **Auto-parallel** — `Process.run()` with no argument runs sequentially for small workflows and switches to parallel for `len(tasks) >= 10`. Pass `parallel=True` or `parallel=False` to force the mode.
+- **Result inspection** — iterate `result.passed_tasks_results.items()` to log or post-process every successful task; iterate `result.failed_tasks` for triage.
+- **Re-raising** — wrap `process.run()` in `try/except` if you need a non-zero exit code on any failure; the library itself does not raise on partial failure.
+
+</details>
+
+---
+
+## 📦 Installation
+
+From PyPI:
+
+```bash
+pip install processes
+```
+
+Or straight from the repository (pure Python, no build step):
+
+```bash
+pip install git+https://github.com/oliverm91/processes.git
+```
+
+Requires **Python 3.10+**.
+
+---
+
+## 📄 License & contributing
+
+Released under the **MIT License** — see [docs](https://oliverm91.github.io/processes/) for full API details.
+
+Contributions welcome — see [CONTRIBUTING.md](CONTRIBUTING.md) for the workflow, style, and commit-message conventions used by this project.