processes 2.0.1__tar.gz → 3.1.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- processes-3.1.0/.github/PULL_REQUEST_TEMPLATE.md +10 -0
- {processes-2.0.1 → processes-3.1.0}/.github/workflows/docs.yml +2 -2
- {processes-2.0.1 → processes-3.1.0}/.github/workflows/lint.yml +2 -2
- {processes-2.0.1 → processes-3.1.0}/.github/workflows/mypy.yml +2 -2
- {processes-2.0.1 → processes-3.1.0}/.github/workflows/publish.yml +4 -4
- {processes-2.0.1 → processes-3.1.0}/.github/workflows/tags.yml +2 -2
- {processes-2.0.1 → processes-3.1.0}/.github/workflows/tests.yml +2 -2
- {processes-2.0.1 → processes-3.1.0}/CHANGELOG.md +33 -0
- processes-3.1.0/CONTRIBUTING.md +135 -0
- processes-3.1.0/PKG-INFO +408 -0
- processes-3.1.0/README.md +383 -0
- processes-3.1.0/RELEASE_NOTES.md +36 -0
- {processes-2.0.1 → processes-3.1.0}/docs/examples/advanced.md +53 -11
- {processes-2.0.1 → processes-3.1.0}/docs/index.md +75 -5
- {processes-2.0.1 → processes-3.1.0}/docs/reference.md +2 -1
- {processes-2.0.1 → processes-3.1.0}/mkdocs.yml +10 -0
- {processes-2.0.1 → processes-3.1.0}/pyproject.toml +3 -3
- {processes-2.0.1 → processes-3.1.0}/src/processes/__init__.py +6 -7
- processes-3.1.0/src/processes/_email_internals.py +168 -0
- processes-3.1.0/src/processes/_error_data.py +38 -0
- processes-3.1.0/src/processes/_log_formatting.py +44 -0
- processes-3.1.0/src/processes/_tb_utils.py +79 -0
- processes-3.1.0/src/processes/email_config.py +91 -0
- processes-3.1.0/src/processes/exceptions.py +36 -0
- {processes-2.0.1 → processes-3.1.0}/src/processes/process.py +71 -51
- processes-3.1.0/src/processes/task.py +421 -0
- processes-3.1.0/tests/base_test.py +62 -0
- processes-3.1.0/tests/manual_tests/__init__.py +0 -0
- {processes-2.0.1 → processes-3.1.0}/tests/manual_tests/manual_pipeline_inspect.py +8 -8
- {processes-2.0.1 → processes-3.1.0}/tests/manual_tests/manual_themed_tracebacks.py +15 -13
- processes-3.1.0/tests/test_args_kwargs.py +125 -0
- {processes-2.0.1 → processes-3.1.0}/tests/test_complex_dag_failures.py +29 -98
- processes-3.1.0/tests/test_dependencies.py +232 -0
- processes-3.1.0/tests/test_email_themes.py +531 -0
- processes-3.1.0/tests/test_logfiles.py +84 -0
- processes-3.1.0/tests/test_normal_run_no_errors.py +137 -0
- {processes-2.0.1 → processes-3.1.0}/tests/test_parallel_race_conditions.py +21 -33
- processes-3.1.0/tests/test_timeout_retry.py +338 -0
- processes-3.1.0/tests/test_unique_name.py +40 -0
- processes-2.0.1/PKG-INFO +0 -234
- processes-2.0.1/README.md +0 -210
- processes-2.0.1/src/processes/exception_html_formatter.py +0 -351
- processes-2.0.1/src/processes/html_logging.py +0 -160
- processes-2.0.1/src/processes/task.py +0 -307
- processes-2.0.1/tests/conftest.py +0 -9
- processes-2.0.1/tests/log_cleaner.py +0 -8
- processes-2.0.1/tests/test_args_kwargs.py +0 -166
- processes-2.0.1/tests/test_dependencies.py +0 -266
- processes-2.0.1/tests/test_email_themes.py +0 -489
- processes-2.0.1/tests/test_examples.py +0 -48
- processes-2.0.1/tests/test_logfiles.py +0 -130
- processes-2.0.1/tests/test_normal_run_no_errors.py +0 -174
- processes-2.0.1/tests/test_unique_name.py +0 -45
- {processes-2.0.1 → processes-3.1.0}/.github/ISSUE_TEMPLATE/bug_report.md +0 -0
- {processes-2.0.1 → processes-3.1.0}/.github/ISSUE_TEMPLATE/custom.md +0 -0
- {processes-2.0.1 → processes-3.1.0}/.github/ISSUE_TEMPLATE/feature_request.md +0 -0
- {processes-2.0.1 → processes-3.1.0}/.github/workflows/lint-pr.yml +0 -0
- {processes-2.0.1 → processes-3.1.0}/.gitignore +0 -0
- {processes-2.0.1 → processes-3.1.0}/LICENSE +0 -0
- {processes-2.0.1 → processes-3.1.0}/assets/banner.svg +0 -0
- {processes-2.0.1 → processes-3.1.0}/assets/logo.png +0 -0
- {processes-2.0.1 → processes-3.1.0}/assets/logo.svg +0 -0
- {processes-2.0.1 → processes-3.1.0}/assets/social_banner.png +0 -0
- {processes-2.0.1 → processes-3.1.0}/assets/social_banner.svg +0 -0
- {processes-2.0.1 → processes-3.1.0}/docs/assets/favicon.svg +0 -0
- {processes-2.0.1 → processes-3.1.0}/docs/examples/basic.md +0 -0
- {processes-2.0.1 → processes-3.1.0}/docs/examples/intro.md +0 -0
- {processes-2.0.1 → processes-3.1.0}/examples/01_basic_tasks_and_dependencies/README.md +0 -0
- {processes-2.0.1 → processes-3.1.0}/examples/01_basic_tasks_and_dependencies/example1.py +0 -0
- {processes-2.0.1 → processes-3.1.0}/examples/02_task_dependencies_result_passing/README.md +0 -0
- {processes-2.0.1 → processes-3.1.0}/examples/02_task_dependencies_result_passing/example2.py +0 -0
- {processes-2.0.1 → processes-3.1.0}/examples/README.md +0 -0
- {processes-2.0.1 → processes-3.1.0}/pytest.ini +0 -0
- /processes-2.0.1/tests/__init__.py → /processes-3.1.0/src/processes/py.typed +0 -0
- {processes-2.0.1 → processes-3.1.0}/src/processes/themes/languages/de.json +0 -0
- {processes-2.0.1 → processes-3.1.0}/src/processes/themes/languages/en.json +0 -0
- {processes-2.0.1 → processes-3.1.0}/src/processes/themes/languages/es.json +0 -0
- {processes-2.0.1 → processes-3.1.0}/src/processes/themes/languages/fr.json +0 -0
- {processes-2.0.1 → processes-3.1.0}/src/processes/themes/languages/it.json +0 -0
- {processes-2.0.1 → processes-3.1.0}/src/processes/themes/languages/pt.json +0 -0
- {processes-2.0.1 → processes-3.1.0}/src/processes/themes/palettes/catppuccin.css +0 -0
- {processes-2.0.1 → processes-3.1.0}/src/processes/themes/palettes/neobones.css +0 -0
- {processes-2.0.1 → processes-3.1.0}/src/processes/themes/palettes/neutral.css +0 -0
- {processes-2.0.1 → processes-3.1.0}/src/processes/themes/palettes/slate.css +0 -0
- {processes-2.0.1 → processes-3.1.0}/src/processes/themes/styles/classic.html +0 -0
- {processes-2.0.1 → processes-3.1.0}/src/processes/themes/styles/compact.html +0 -0
- {processes-2.0.1 → processes-3.1.0}/src/processes/themes/styles/modern.html +0 -0
- {processes-2.0.1/tests/manual_tests → processes-3.1.0/tests}/__init__.py +0 -0
- {processes-2.0.1 → processes-3.1.0}/uv.lock +0 -0
|
@@ -0,0 +1,10 @@
|
|
|
1
|
+
## What and why
|
|
2
|
+
<!-- Describe the change and the motivation behind it -->
|
|
3
|
+
|
|
4
|
+
## Type of change
|
|
5
|
+
- [ ] `feat` — new feature
|
|
6
|
+
- [ ] `fix` — bug fix
|
|
7
|
+
- [ ] `refactor` — no behavior change
|
|
8
|
+
- [ ] `docs` — documentation only
|
|
9
|
+
- [ ] `test` — tests only
|
|
10
|
+
- [ ] `chore` / `ci` / `build`
|
|
@@ -20,8 +20,8 @@ jobs:
|
|
|
20
20
|
deploy:
|
|
21
21
|
runs-on: ubuntu-latest
|
|
22
22
|
steps:
|
|
23
|
-
- uses: actions/checkout@
|
|
24
|
-
- uses: astral-sh/setup-uv@
|
|
23
|
+
- uses: actions/checkout@v6
|
|
24
|
+
- uses: astral-sh/setup-uv@v6
|
|
25
25
|
- name: Install dependencies
|
|
26
26
|
run: uv sync --all-groups
|
|
27
27
|
- name: Build and Deploy
|
|
@@ -14,10 +14,10 @@ jobs:
|
|
|
14
14
|
lint:
|
|
15
15
|
runs-on: ubuntu-latest
|
|
16
16
|
steps:
|
|
17
|
-
- uses: actions/checkout@
|
|
17
|
+
- uses: actions/checkout@v6
|
|
18
18
|
|
|
19
19
|
- name: Install uv
|
|
20
|
-
uses: astral-sh/setup-uv@
|
|
20
|
+
uses: astral-sh/setup-uv@v6
|
|
21
21
|
with:
|
|
22
22
|
enable-cache: true # Speeds up future runs
|
|
23
23
|
|
|
@@ -14,10 +14,10 @@ jobs:
|
|
|
14
14
|
lint:
|
|
15
15
|
runs-on: ubuntu-latest
|
|
16
16
|
steps:
|
|
17
|
-
- uses: actions/checkout@
|
|
17
|
+
- uses: actions/checkout@v6
|
|
18
18
|
|
|
19
19
|
- name: Install uv
|
|
20
|
-
uses: astral-sh/setup-uv@
|
|
20
|
+
uses: astral-sh/setup-uv@v6
|
|
21
21
|
with:
|
|
22
22
|
enable-cache: true # Speeds up future runs
|
|
23
23
|
|
|
@@ -8,9 +8,9 @@ jobs:
|
|
|
8
8
|
quality-gate:
|
|
9
9
|
runs-on: ubuntu-latest
|
|
10
10
|
steps:
|
|
11
|
-
- uses: actions/checkout@
|
|
11
|
+
- uses: actions/checkout@v6
|
|
12
12
|
- name: Install uv
|
|
13
|
-
uses: astral-sh/setup-uv@
|
|
13
|
+
uses: astral-sh/setup-uv@v6
|
|
14
14
|
|
|
15
15
|
- name: Install dependencies
|
|
16
16
|
run: uv sync --all-extras --dev
|
|
@@ -33,9 +33,9 @@ jobs:
|
|
|
33
33
|
contents: read
|
|
34
34
|
|
|
35
35
|
steps:
|
|
36
|
-
- uses: actions/checkout@
|
|
36
|
+
- uses: actions/checkout@v6
|
|
37
37
|
- name: Install uv
|
|
38
|
-
uses: astral-sh/setup-uv@
|
|
38
|
+
uses: astral-sh/setup-uv@v6
|
|
39
39
|
|
|
40
40
|
- name: Build package
|
|
41
41
|
run: uv build
|
|
@@ -1,3 +1,36 @@
|
|
|
1
|
+
## v3.1.0 (2026-06-13)
|
|
2
|
+
|
|
3
|
+
### Feat
|
|
4
|
+
|
|
5
|
+
- **logging**: include failure context in task logfiles
|
|
6
|
+
|
|
7
|
+
### Refactor
|
|
8
|
+
|
|
9
|
+
- extract typed _ErrorData from task_context via shared base formatter
|
|
10
|
+
- inline single-use _task_context_lines helper
|
|
11
|
+
- move task logfile formatting out of _email_internals
|
|
12
|
+
|
|
13
|
+
## v3.0.1 (2026-06-13)
|
|
14
|
+
|
|
15
|
+
### Refactor
|
|
16
|
+
|
|
17
|
+
- **process**: use bare raise and extract _is_done helper
|
|
18
|
+
|
|
19
|
+
## v3.0.0 (2026-06-13)
|
|
20
|
+
|
|
21
|
+
### BREAKING CHANGE
|
|
22
|
+
|
|
23
|
+
- html_mail_handler parameter removed from Task; use
|
|
24
|
+
smtp_config and email_style instead. HTMLSMTPHandler removed from public API.
|
|
25
|
+
|
|
26
|
+
### Feat
|
|
27
|
+
|
|
28
|
+
- **html_logging**: expose last_path_traced_vars on HTMLSMTPHandler
|
|
29
|
+
|
|
30
|
+
### Refactor
|
|
31
|
+
|
|
32
|
+
- **email**: replace HTMLSMTPHandler with SMTPConfig/HTMLEmailStyle
|
|
33
|
+
|
|
1
34
|
## v2.0.1 (2026-06-12)
|
|
2
35
|
|
|
3
36
|
### Fix
|
|
@@ -0,0 +1,135 @@
|
|
|
1
|
+
# Contributing to Processes
|
|
2
|
+
|
|
3
|
+
Thanks for your interest in contributing! This document covers the two
|
|
4
|
+
contributor paths, project conventions, and how to submit a change.
|
|
5
|
+
|
|
6
|
+
## Contributor paths
|
|
7
|
+
|
|
8
|
+
**Maintainer** (direct push access): clone the repository directly.
|
|
9
|
+
|
|
10
|
+
```bash
|
|
11
|
+
git clone https://github.com/oliverm91/processes.git
|
|
12
|
+
cd processes
|
|
13
|
+
uv sync
|
|
14
|
+
```
|
|
15
|
+
|
|
16
|
+
**External contributor** (everyone else): fork first, then clone your fork.
|
|
17
|
+
|
|
18
|
+
```bash
|
|
19
|
+
# 1. Fork on GitHub (button top-right of the repo page)
|
|
20
|
+
# 2. Clone your fork
|
|
21
|
+
git clone https://github.com/<your-username>/processes.git
|
|
22
|
+
cd processes
|
|
23
|
+
uv sync
|
|
24
|
+
```
|
|
25
|
+
|
|
26
|
+
Both paths use [uv](https://docs.astral.sh/uv/) for dependency management.
|
|
27
|
+
`uv sync` installs all dev dependencies (pytest, mypy, ruff, mkdocs, commitizen).
|
|
28
|
+
|
|
29
|
+
## Project layout
|
|
30
|
+
|
|
31
|
+
```
|
|
32
|
+
src/processes/ # library source (the public package)
|
|
33
|
+
tests/ # pytest test suite
|
|
34
|
+
examples/ # runnable usage examples
|
|
35
|
+
docs/ # mkdocs documentation source
|
|
36
|
+
```
|
|
37
|
+
|
|
38
|
+
Keep new public symbols in `src/processes/` and re-export them from
|
|
39
|
+
`src/processes/__init__.py` if they are part of the stable API.
|
|
40
|
+
|
|
41
|
+
## Running checks locally
|
|
42
|
+
|
|
43
|
+
Before opening a pull request, make sure the following pass:
|
|
44
|
+
|
|
45
|
+
```bash
|
|
46
|
+
uv run pytest # test suite
|
|
47
|
+
uv run mypy # strict type checking
|
|
48
|
+
uv run ruff check . # linting
|
|
49
|
+
uv run ruff format . # formatting
|
|
50
|
+
```
|
|
51
|
+
|
|
52
|
+
CI runs the same checks on every push and pull request.
|
|
53
|
+
|
|
54
|
+
## Style guide
|
|
55
|
+
|
|
56
|
+
- Python >= 3.10 syntax.
|
|
57
|
+
- Type hints on all new code. `mypy` is configured with `strict = true` and
|
|
58
|
+
`disallow_untyped_defs = true`.
|
|
59
|
+
- Ruff rules: `F`, `E`, `W`, `I`, `B`, `UP`. Line length is 100.
|
|
60
|
+
- Prefer the standard library — the library has zero runtime dependencies.
|
|
61
|
+
- No comments that just restate the code. A comment is welcome only when it
|
|
62
|
+
captures a non-obvious *why*.
|
|
63
|
+
|
|
64
|
+
## Tests
|
|
65
|
+
|
|
66
|
+
- Add or update tests in `tests/` for any behavior change.
|
|
67
|
+
- Keep tests focused and deterministic; avoid sleeps where possible.
|
|
68
|
+
- If you fix a bug, add a regression test that fails before the fix.
|
|
69
|
+
|
|
70
|
+
## Commit messages
|
|
71
|
+
|
|
72
|
+
This project follows [Conventional Commits](https://www.conventionalcommits.org/),
|
|
73
|
+
enforced by [commitizen](https://commitizen-tools.github.io/commitizen/). Use
|
|
74
|
+
`uv run cz commit` to be prompted through an interactive commit, or write the
|
|
75
|
+
message by hand:
|
|
76
|
+
|
|
77
|
+
```
|
|
78
|
+
<type>(<scope>): <short description>
|
|
79
|
+
|
|
80
|
+
<optional body>
|
|
81
|
+
|
|
82
|
+
<optional footer>
|
|
83
|
+
```
|
|
84
|
+
|
|
85
|
+
Common types: `feat`, `fix`, `docs`, `style`, `refactor`, `test`, `chore`,
|
|
86
|
+
`build`, `ci`.
|
|
87
|
+
|
|
88
|
+
## Submitting a pull request
|
|
89
|
+
|
|
90
|
+
1. Create a feature branch: `git checkout -b feat/my-change`.
|
|
91
|
+
2. Make your changes, add tests, and confirm all checks pass.
|
|
92
|
+
3. Push the branch to your fork (or directly, if you're a maintainer) and open
|
|
93
|
+
a PR against `main` on the upstream repository.
|
|
94
|
+
4. Fill in a clear description of the *what* and the *why*.
|
|
95
|
+
5. Make sure CI is green and the changelog preview looks reasonable for
|
|
96
|
+
user-visible changes.
|
|
97
|
+
6. Be ready to revise — code review is a normal part of the process.
|
|
98
|
+
|
|
99
|
+
Prefer several small, focused PRs over one large one.
|
|
100
|
+
|
|
101
|
+
## Documentation
|
|
102
|
+
|
|
103
|
+
User-facing changes should be reflected in the docs under `docs/`. To preview
|
|
104
|
+
locally:
|
|
105
|
+
|
|
106
|
+
```bash
|
|
107
|
+
uv run mkdocs serve
|
|
108
|
+
```
|
|
109
|
+
|
|
110
|
+
## Versioning and releases
|
|
111
|
+
|
|
112
|
+
Versions are derived from git tags via `hatch-vcs` (`v$version`, e.g. `v2.0.1`).
|
|
113
|
+
Do not edit the version in `pyproject.toml` manually. To cut a release, the
|
|
114
|
+
maintainer runs:
|
|
115
|
+
|
|
116
|
+
```bash
|
|
117
|
+
uv run cz bump
|
|
118
|
+
```
|
|
119
|
+
|
|
120
|
+
This updates the version, regenerates the changelog, creates a tag, and
|
|
121
|
+
triggers the `publish.yml` workflow to push the release to PyPI.
|
|
122
|
+
|
|
123
|
+
## Reporting issues
|
|
124
|
+
|
|
125
|
+
Use the GitHub issue tracker. Please include:
|
|
126
|
+
|
|
127
|
+
- A minimal, self-contained reproduction.
|
|
128
|
+
- The Python version, library version, and operating system.
|
|
129
|
+
- The full traceback for crashes, and the smallest possible task graph that
|
|
130
|
+
triggers the problem.
|
|
131
|
+
|
|
132
|
+
## License
|
|
133
|
+
|
|
134
|
+
By contributing, you agree that your contributions will be licensed under the
|
|
135
|
+
MIT License.
|
processes-3.1.0/PKG-INFO
ADDED
|
@@ -0,0 +1,408 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: processes
|
|
3
|
+
Version: 3.1.0
|
|
4
|
+
Summary: Orchestrate graphs of callables in Python with automatic dependency resolution, parallel execution, retries, timeouts, and HTML email alerts on failure — zero dependencies
|
|
5
|
+
Project-URL: Homepage, https://github.com/oliverm91/processes
|
|
6
|
+
Project-URL: Documentation, https://oliverm91.github.io/processes/
|
|
7
|
+
Project-URL: Repository, https://github.com/oliverm91/processes
|
|
8
|
+
Project-URL: Issues, https://github.com/oliverm91/processes/issues
|
|
9
|
+
Project-URL: Changelog, https://github.com/oliverm91/processes/blob/main/CHANGELOG.md
|
|
10
|
+
Author-email: Oliver Mohr Bonometti <oliver.mohr.b@gmail.com>
|
|
11
|
+
License-Expression: MIT
|
|
12
|
+
License-File: LICENSE
|
|
13
|
+
Keywords: dag,dependencies,etl,parallel,process,tasks,topological-sort,workflow
|
|
14
|
+
Classifier: Development Status :: 4 - Beta
|
|
15
|
+
Classifier: Intended Audience :: Developers
|
|
16
|
+
Classifier: Programming Language :: Python :: 3
|
|
17
|
+
Classifier: Programming Language :: Python :: 3.10
|
|
18
|
+
Classifier: Programming Language :: Python :: 3.11
|
|
19
|
+
Classifier: Programming Language :: Python :: 3.12
|
|
20
|
+
Classifier: Programming Language :: Python :: 3.13
|
|
21
|
+
Classifier: Programming Language :: Python :: 3.14
|
|
22
|
+
Classifier: Typing :: Typed
|
|
23
|
+
Requires-Python: >=3.10
|
|
24
|
+
Description-Content-Type: text/markdown
|
|
25
|
+
|
|
26
|
+
<div align="center">
|
|
27
|
+
<img src="https://raw.githubusercontent.com/oliverm91/processes/refs/heads/main/assets/banner.svg" width="100%" alt="Processes - Smart Task Orchestration">
|
|
28
|
+
</div>
|
|
29
|
+
|
|
30
|
+
# Processes: Smart Task Orchestration
|
|
31
|
+
|
|
32
|
+
[](https://www.python.org/)
|
|
33
|
+

|
|
34
|
+
[](https://oliverm91.github.io/processes/)
|
|
35
|
+
[](https://opensource.org/licenses/MIT)
|
|
36
|
+
[](https://pypi.org/project/processes/)
|
|
37
|
+
|
|
38
|
+
[](https://github.com/oliverm91/processes/actions/workflows/tests.yml)
|
|
39
|
+
[](https://github.com/oliverm91/processes/actions/workflows/lint.yml)
|
|
40
|
+
[](https://github.com/oliverm91/processes/actions/workflows/mypy.yml)
|
|
41
|
+
|
|
42
|
+
---
|
|
43
|
+
|
|
44
|
+
**Run a list of Python callables that depend on each other — in parallel when possible, with per-task log files and optional HTML email notification on failure. Zero dependencies. Pure Python 3.10+.**
|
|
45
|
+
|
|
46
|
+
---
|
|
47
|
+
|
|
48
|
+
## ✨ Why Processes?
|
|
49
|
+
|
|
50
|
+
- 🔗 **Declare what depends on what** — write your tasks in any order; the runtime sorts them so every dependency runs first.
|
|
51
|
+
- ⚡ **Run in parallel when you can** — independent tasks run together on a thread pool; the runtime switches on automatically for jobs with 10+ tasks.
|
|
52
|
+
- 🛡️ **One failure doesn't stop the rest** — a failed task skips only the jobs that depend on it, and **every other part of the workflow keeps running**.
|
|
53
|
+
- 📝 **One log file per task** — share a single log across the whole run, or keep them separate for easier debugging.
|
|
54
|
+
- 📧 **Email alerts when something breaks** — pass an `SMTPConfig` to a task and get a styled HTML email (with traceback, task context, and the list of jobs that were skipped) the instant it raises.
|
|
55
|
+
- 🧰 **Modern, strictly-typed Python 3.10+** — `from __future__ import annotations`, full `mypy --strict` clean, `dict[str, TaskResult]`, `set[str]`, `|` unions.
|
|
56
|
+
|
|
57
|
+
---
|
|
58
|
+
|
|
59
|
+
## ⚙️ How it works
|
|
60
|
+
|
|
61
|
+
A `Process` holds a list of `Task`s. At construction it validates names, types, dependency references, and detects cycles — raising before anything runs.
|
|
62
|
+
|
|
63
|
+
When you call `process.run()`, tasks are topologically sorted and scheduled: dependencies first, independent tasks in parallel.
|
|
64
|
+
|
|
65
|
+
A `TaskDependency` can forward an upstream result directly into a downstream function, as a positional or keyword argument. The result is a `ProcessResult` with `passed_tasks_results` and `failed_tasks` for inspection.
|
|
66
|
+
|
|
67
|
+
---
|
|
68
|
+
|
|
69
|
+
## 🚀 Quick start
|
|
70
|
+
|
|
71
|
+
A 15-line "hello pipeline" — one upstream task feeding a downstream one, run in parallel.
|
|
72
|
+
|
|
73
|
+
```python
|
|
74
|
+
from processes import Process, Task, TaskDependency
|
|
75
|
+
|
|
76
|
+
|
|
77
|
+
def load_users() -> list[dict]:
|
|
78
|
+
return [{"id": 1}, {"id": 2}, {"id": 3}]
|
|
79
|
+
|
|
80
|
+
|
|
81
|
+
def enrich(users: list[dict]) -> list[dict]:
|
|
82
|
+
return [{**u, "name": f"user-{u['id']}"} for u in users]
|
|
83
|
+
|
|
84
|
+
|
|
85
|
+
tasks = [
|
|
86
|
+
Task("load", "run.log", load_users),
|
|
87
|
+
Task(
|
|
88
|
+
"enrich",
|
|
89
|
+
"run.log",
|
|
90
|
+
enrich,
|
|
91
|
+
dependencies=[TaskDependency("load", use_result_as_additional_args=True)],
|
|
92
|
+
),
|
|
93
|
+
]
|
|
94
|
+
|
|
95
|
+
with Process(tasks) as p:
|
|
96
|
+
result = p.run(parallel=True)
|
|
97
|
+
|
|
98
|
+
print(result.passed_tasks_results["enrich"].result)
|
|
99
|
+
# [{'id': 1, 'name': 'user-1'}, {'id': 2, 'name': 'user-2'}, {'id': 3, 'name': 'user-3'}]
|
|
100
|
+
```
|
|
101
|
+
|
|
102
|
+
---
|
|
103
|
+
|
|
104
|
+
## 🧪 End-to-end example
|
|
105
|
+
|
|
106
|
+
A realistic mini-pipeline: fetch two sources **in parallel**, transform them, aggregate, and notify — with per-task log files, result piping, and one task deliberately failing to show fault isolation.
|
|
107
|
+
|
|
108
|
+
<details>
|
|
109
|
+
<summary>Show the full end-to-end example</summary>
|
|
110
|
+
|
|
111
|
+
```python
|
|
112
|
+
import logging
|
|
113
|
+
from pathlib import Path
|
|
114
|
+
|
|
115
|
+
from processes import HTMLEmailStyle, Process, SMTPConfig, Task, TaskDependency
|
|
116
|
+
|
|
117
|
+
LOG_DIR = Path("logs")
|
|
118
|
+
LOG_DIR.mkdir(exist_ok=True)
|
|
119
|
+
|
|
120
|
+
|
|
121
|
+
# --- 1. Two independent "fetch" tasks that run in parallel -----------------
|
|
122
|
+
def fetch_orders() -> list[dict]:
|
|
123
|
+
logging.info("querying orders API")
|
|
124
|
+
return [{"order_id": 1, "amount": 42.0}, {"order_id": 2, "amount": 17.5}]
|
|
125
|
+
|
|
126
|
+
|
|
127
|
+
def fetch_inventory() -> list[dict]:
|
|
128
|
+
logging.info("querying inventory API")
|
|
129
|
+
return [{"sku": "A-1", "qty": 12}, {"sku": "B-2", "qty": 3}]
|
|
130
|
+
|
|
131
|
+
|
|
132
|
+
# --- 2. Two transforms that consume the upstream results -------------------
|
|
133
|
+
def total_revenue(orders: list[dict]) -> float:
|
|
134
|
+
total = sum(o["amount"] for o in orders)
|
|
135
|
+
logging.info("revenue computed: %s", total)
|
|
136
|
+
return total
|
|
137
|
+
|
|
138
|
+
|
|
139
|
+
def stock_value(inventory: list[dict], *, price_per_unit: float = 10.0) -> float:
|
|
140
|
+
value = sum(i["qty"] for i in inventory) * price_per_unit
|
|
141
|
+
logging.info("stock value: %s", value)
|
|
142
|
+
return value
|
|
143
|
+
|
|
144
|
+
|
|
145
|
+
# --- 3. An aggregator that joins the two branches --------------------------
|
|
146
|
+
def build_report(*, revenue: float, stock: float) -> str:
|
|
147
|
+
return f"daily-report | revenue={revenue:.2f} stock={stock:.2f}"
|
|
148
|
+
|
|
149
|
+
|
|
150
|
+
# --- 4. A flaky notifier that ALWAYS fails — to show fault isolation -------
|
|
151
|
+
def notify_slack(report: str) -> None:
|
|
152
|
+
raise RuntimeError("slack webhook returned 503")
|
|
153
|
+
|
|
154
|
+
|
|
155
|
+
# --- 5. A sibling task that does NOT depend on notify and still runs -------
|
|
156
|
+
def archive_report(report: str) -> str:
|
|
157
|
+
out = LOG_DIR / "report.txt"
|
|
158
|
+
out.write_text(report)
|
|
159
|
+
return str(out)
|
|
160
|
+
|
|
161
|
+
|
|
162
|
+
# --- 6. Optional: SMTP config so failures page on-call --------------------
|
|
163
|
+
smtp = SMTPConfig(
|
|
164
|
+
mailhost=("smtp.example.com", 587),
|
|
165
|
+
fromaddr="alerts@example.com",
|
|
166
|
+
toaddrs=["oncall@example.com"],
|
|
167
|
+
credentials=("user", "pass"),
|
|
168
|
+
secure=(),
|
|
169
|
+
)
|
|
170
|
+
|
|
171
|
+
tasks = [
|
|
172
|
+
Task("fetch_orders", LOG_DIR / "fetch_orders.log", fetch_orders),
|
|
173
|
+
Task("fetch_inventory", LOG_DIR / "fetch_inventory.log", fetch_inventory),
|
|
174
|
+
|
|
175
|
+
Task(
|
|
176
|
+
"compute_revenue",
|
|
177
|
+
LOG_DIR / "compute_revenue.log",
|
|
178
|
+
total_revenue,
|
|
179
|
+
dependencies=[TaskDependency("fetch_orders", use_result_as_additional_args=True)],
|
|
180
|
+
),
|
|
181
|
+
Task(
|
|
182
|
+
"compute_stock",
|
|
183
|
+
LOG_DIR / "compute_stock.log",
|
|
184
|
+
stock_value,
|
|
185
|
+
kwargs={"price_per_unit": 7.25},
|
|
186
|
+
dependencies=[
|
|
187
|
+
TaskDependency(
|
|
188
|
+
"fetch_inventory",
|
|
189
|
+
use_result_as_additional_kwargs=True,
|
|
190
|
+
additional_kwarg_name="inventory",
|
|
191
|
+
)
|
|
192
|
+
],
|
|
193
|
+
),
|
|
194
|
+
|
|
195
|
+
Task(
|
|
196
|
+
"build_report",
|
|
197
|
+
LOG_DIR / "build_report.log",
|
|
198
|
+
build_report,
|
|
199
|
+
dependencies=[
|
|
200
|
+
TaskDependency("compute_revenue", use_result_as_additional_kwargs=True,
|
|
201
|
+
additional_kwarg_name="revenue"),
|
|
202
|
+
TaskDependency("compute_stock", use_result_as_additional_kwargs=True,
|
|
203
|
+
additional_kwarg_name="stock"),
|
|
204
|
+
],
|
|
205
|
+
),
|
|
206
|
+
|
|
207
|
+
# notify_slack fails on purpose. archive_report is a *sibling*
|
|
208
|
+
# of notify_slack (both depend on build_report), so it has no
|
|
209
|
+
# dependency on the failed task and runs normally — the rest of
|
|
210
|
+
# the workflow is not blackholed by one broken step.
|
|
211
|
+
Task(
|
|
212
|
+
"notify_slack",
|
|
213
|
+
LOG_DIR / "notify_slack.log",
|
|
214
|
+
notify_slack,
|
|
215
|
+
dependencies=[TaskDependency("build_report", use_result_as_additional_args=True)],
|
|
216
|
+
smtp_config=smtp,
|
|
217
|
+
),
|
|
218
|
+
Task(
|
|
219
|
+
"archive_report",
|
|
220
|
+
LOG_DIR / "archive_report.log",
|
|
221
|
+
archive_report,
|
|
222
|
+
dependencies=[TaskDependency("build_report", use_result_as_additional_args=True)],
|
|
223
|
+
),
|
|
224
|
+
]
|
|
225
|
+
|
|
226
|
+
with Process(tasks) as process:
|
|
227
|
+
result = process.run(parallel=True)
|
|
228
|
+
|
|
229
|
+
print("passed:", sorted(result.passed_tasks_results))
|
|
230
|
+
# archive_report, build_report, compute_revenue, compute_stock, fetch_inventory, fetch_orders
|
|
231
|
+
print("failed:", sorted(result.failed_tasks))
|
|
232
|
+
# notify_slack
|
|
233
|
+
print("report:", result.passed_tasks_results["build_report"].result)
|
|
234
|
+
# daily-report | revenue=59.50 stock=262.50
|
|
235
|
+
```
|
|
236
|
+
|
|
237
|
+
The failing `notify_slack` task does **not** abort the run. `archive_report` is a sibling of the failed task (both depend on the successful `build_report`), so it runs unaffected — the rest of the workflow is not blackholed by one broken step. The HTML email handler also fires on the `notify_slack` task, paging on-call with the full traceback and the list of downstream tasks that were skipped because of it.
|
|
238
|
+
|
|
239
|
+
</details>
|
|
240
|
+
|
|
241
|
+
---
|
|
242
|
+
|
|
243
|
+
## 📚 API Reference
|
|
244
|
+
|
|
245
|
+
<details>
|
|
246
|
+
<summary>Show API reference</summary>
|
|
247
|
+
|
|
248
|
+
### `Task`
|
|
249
|
+
|
|
250
|
+
```python
|
|
251
|
+
Task(
|
|
252
|
+
name: str,
|
|
253
|
+
log_path: str | os.PathLike,
|
|
254
|
+
func: Callable[..., Any],
|
|
255
|
+
args: tuple = (),
|
|
256
|
+
kwargs: dict | None = None,
|
|
257
|
+
dependencies: list[TaskDependency] | None = None,
|
|
258
|
+
smtp_config: SMTPConfig | None = None,
|
|
259
|
+
email_style: HTMLEmailStyle | None = None,
|
|
260
|
+
timeout: float | None = None,
|
|
261
|
+
retries: int | None = 0,
|
|
262
|
+
retry_on: tuple[type[Exception], ...] | None = None,
|
|
263
|
+
)
|
|
264
|
+
```
|
|
265
|
+
|
|
266
|
+
- `name` — unique within the `Process`; no spaces.
|
|
267
|
+
- `log_path` — the file this task logs to (INFO level, format `%(asctime)s - %(name)s - %(levelname)s - %(message)s`).
|
|
268
|
+
- `func` — the callable; receives `func(*args, **kwargs)` after result-injection.
|
|
269
|
+
- `smtp_config` — when set, fires an HTML email on `logging.ERROR`; body includes `task_name`, `function`, `args`, `kwargs`, and `downstream_impact`.
|
|
270
|
+
- `email_style` — optional presentation override; defaults to `HTMLEmailStyle()` (modern, neutral, English) when `smtp_config` is set.
|
|
271
|
+
- `timeout` — seconds allowed per attempt; `None` means no limit. When the timeout fires the underlying thread is detached (Python threading limitation).
|
|
272
|
+
- `retries` — additional attempts after the first failure; `0` or `None` means a single attempt. Defaults to `0`.
|
|
273
|
+
- `retry_on` — tuple of exception types that trigger a retry. When `retries >= 1` and `retry_on` is `None`, defaults to `(ConnectionError, TimeoutError)` at call time.
|
|
274
|
+
|
|
275
|
+
### `TaskDependency`
|
|
276
|
+
|
|
277
|
+
```python
|
|
278
|
+
TaskDependency(
|
|
279
|
+
task_name: str,
|
|
280
|
+
use_result_as_additional_args: bool = False,
|
|
281
|
+
use_result_as_additional_kwargs: bool = False,
|
|
282
|
+
additional_kwarg_name: str = "",
|
|
283
|
+
)
|
|
284
|
+
```
|
|
285
|
+
|
|
286
|
+
- `use_result_as_additional_args=True` — upstream result appended as the next **positional** arg.
|
|
287
|
+
- `use_result_as_additional_kwargs=True` with a non-empty `additional_kwarg_name` — upstream result injected as a **keyword** arg.
|
|
288
|
+
- Both flags can be combined (positional first, then kwarg).
|
|
289
|
+
|
|
290
|
+
### `Process`
|
|
291
|
+
|
|
292
|
+
```python
|
|
293
|
+
Process(tasks: list[Task]) # validates types, names, deps, cycles
|
|
294
|
+
|
|
295
|
+
process.run(parallel: bool | None = None, max_workers: int = 4) -> ProcessResult
|
|
296
|
+
```
|
|
297
|
+
|
|
298
|
+
- Raises `DependencyNotFoundError`, `CircularDependencyError`, `TypeError`, `ValueError` on construction if the workflow is malformed.
|
|
299
|
+
- `parallel=None` auto-parallelises when `len(tasks) >= 10`; `max_workers=1` is always sequential.
|
|
300
|
+
- Use as a context manager — it cleans up `FileHandler`s on exit.
|
|
301
|
+
|
|
302
|
+
### `ProcessResult`
|
|
303
|
+
|
|
304
|
+
```python
|
|
305
|
+
result.passed_tasks_results # dict[str, TaskResult] — name → TaskResult for every task that succeeded
|
|
306
|
+
result.failed_tasks # set[str] — all tasks that did not produce a result (errored + skipped)
|
|
307
|
+
result.errored_tasks # set[str] — tasks whose function actually raised
|
|
308
|
+
result.skipped_tasks # set[str] — tasks skipped because an upstream dependency failed
|
|
309
|
+
|
|
310
|
+
TaskResult(worked: bool, result: Any, exception: Exception | None)
|
|
311
|
+
```
|
|
312
|
+
|
|
313
|
+
### `SMTPConfig`
|
|
314
|
+
|
|
315
|
+
```python
|
|
316
|
+
SMTPConfig(
|
|
317
|
+
mailhost, # (host, port)
|
|
318
|
+
fromaddr,
|
|
319
|
+
toaddrs, # list[str]
|
|
320
|
+
credentials=None, # (username, password) | None
|
|
321
|
+
secure=None, # () = STARTTLS; omit for no encryption
|
|
322
|
+
timeout=5,
|
|
323
|
+
)
|
|
324
|
+
```
|
|
325
|
+
|
|
326
|
+
### `HTMLEmailStyle`
|
|
327
|
+
|
|
328
|
+
```python
|
|
329
|
+
HTMLEmailStyle(
|
|
330
|
+
style="modern", # classic | modern | compact
|
|
331
|
+
palette="neutral", # neutral | catppuccin | neobones | slate
|
|
332
|
+
language="en", # en | es | pt | fr | de | it
|
|
333
|
+
traced_vars_frame_filter=None, # substring to pick the traced frame | None
|
|
334
|
+
)
|
|
335
|
+
```
|
|
336
|
+
|
|
337
|
+
All fields are optional — omit `HTMLEmailStyle` entirely to use the defaults.
|
|
338
|
+
|
|
339
|
+
</details>
|
|
340
|
+
|
|
341
|
+
<details>
|
|
342
|
+
<summary>Show fault-tolerance rules in detail</summary>
|
|
343
|
+
|
|
344
|
+
When a task raises:
|
|
345
|
+
|
|
346
|
+
1. The exception is caught and stored in `TaskResult.exception`; the task name goes into `failed_tasks` and `errored_tasks`.
|
|
347
|
+
2. **Every task that depends on it (directly or indirectly) is skipped** — added to `failed_tasks` and `skipped_tasks` without running.
|
|
348
|
+
3. **Every other independent part of the workflow keeps running.** With `parallel=True` they keep running concurrently on the worker pool.
|
|
349
|
+
4. After `run()` returns, `ProcessResult.errored_tasks` and `ProcessResult.skipped_tasks` let you distinguish root failures from cascade skips for triage or alerting.
|
|
350
|
+
|
|
351
|
+
When a task has `retries >= 1`, a failure matching `retry_on` triggers another attempt before the task is declared failed and its dependants are skipped. This gives transient errors (network blips, connection resets) a chance to resolve without aborting downstream work.
|
|
352
|
+
|
|
353
|
+
This makes the library a good fit for fan-out / fan-in pipelines, "best-effort" notifications, and any workflow where one broken step should not blackhole the rest.
|
|
354
|
+
|
|
355
|
+
</details>
|
|
356
|
+
|
|
357
|
+
<details>
|
|
358
|
+
<summary>Show comparison with other libraries</summary>
|
|
359
|
+
|
|
360
|
+
| | **Processes** | Airflow | Celery | Luigi |
|
|
361
|
+
|---|---|---|---|---|
|
|
362
|
+
| External dependencies | **None** | many | broker (Redis/RabbitMQ) | few |
|
|
363
|
+
| Setup cost | `pip install` | cluster | broker + workers | task + config |
|
|
364
|
+
| Parallelism | built-in | via executors | via workers | via workers |
|
|
365
|
+
| Per-task file logs | **yes (built-in)** | via handlers | via signals | partial |
|
|
366
|
+
| HTML email on failure | **yes (built-in)** | via callbacks | via signals | manual |
|
|
367
|
+
| DAG validation at construction | **yes** | yes (DAG file) | n/a | partial |
|
|
368
|
+
| Strict typing (`mypy --strict`) | **yes** | partial | partial | no |
|
|
369
|
+
|
|
370
|
+
`Processes` is **not** a distributed scheduler — there are no workers on remote machines, no SLA monitoring, no web UI. If you need any of those, you need Airflow or a similar orchestrator. If you want a small, fast, dependency-aware pipeline that *just runs* in a single process, this is it.
|
|
371
|
+
|
|
372
|
+
</details>
|
|
373
|
+
|
|
374
|
+
<details>
|
|
375
|
+
<summary>Show advanced configuration</summary>
|
|
376
|
+
|
|
377
|
+
- **Shared log file** — pass the same `log_path` to every `Task` for a single combined run.log; pass distinct paths for per-task isolation.
|
|
378
|
+
- **Auto-parallel** — `Process.run()` with no argument runs sequentially for small workflows and switches to parallel for `len(tasks) >= 10`. Pass `parallel=True` or `parallel=False` to force the mode.
|
|
379
|
+
- **Result inspection** — iterate `result.passed_tasks_results.items()` to log or post-process every successful task; iterate `result.failed_tasks` for triage.
|
|
380
|
+
- **Re-raising** — wrap `process.run()` in `try/except` if you need a non-zero exit code on any failure; the library itself does not raise on partial failure.
|
|
381
|
+
|
|
382
|
+
</details>
|
|
383
|
+
|
|
384
|
+
---
|
|
385
|
+
|
|
386
|
+
## 📦 Installation
|
|
387
|
+
|
|
388
|
+
From PyPI:
|
|
389
|
+
|
|
390
|
+
```bash
|
|
391
|
+
pip install processes
|
|
392
|
+
```
|
|
393
|
+
|
|
394
|
+
Or straight from the repository (pure Python, no build step):
|
|
395
|
+
|
|
396
|
+
```bash
|
|
397
|
+
pip install git+https://github.com/oliverm91/processes.git
|
|
398
|
+
```
|
|
399
|
+
|
|
400
|
+
Requires **Python 3.10+**.
|
|
401
|
+
|
|
402
|
+
---
|
|
403
|
+
|
|
404
|
+
## 📄 License & contributing
|
|
405
|
+
|
|
406
|
+
Released under the **MIT License** — see [docs](https://oliverm91.github.io/processes/) for full API details.
|
|
407
|
+
|
|
408
|
+
Contributions welcome — see [CONTRIBUTING.md](CONTRIBUTING.md) for the workflow, style, and commit-message conventions used by this project.
|