processes 3.1.0__tar.gz → 7.0.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- processes-7.0.0/.github/workflows/lint-pr.yml +31 -0
- {processes-3.1.0 → processes-7.0.0}/.github/workflows/tests.yml +1 -1
- {processes-3.1.0 → processes-7.0.0}/.gitignore +5 -1
- processes-7.0.0/CHANGELOG.md +204 -0
- {processes-3.1.0 → processes-7.0.0}/PKG-INFO +183 -56
- {processes-3.1.0 → processes-7.0.0}/README.md +181 -53
- {processes-3.1.0 → processes-7.0.0}/docs/examples/advanced.md +17 -15
- {processes-3.1.0 → processes-7.0.0}/docs/examples/basic.md +10 -10
- processes-7.0.0/docs/examples/reports.md +128 -0
- {processes-3.1.0 → processes-7.0.0}/docs/index.md +48 -42
- processes-7.0.0/docs/reference.md +14 -0
- {processes-3.1.0 → processes-7.0.0}/examples/01_basic_tasks_and_dependencies/README.md +10 -10
- {processes-3.1.0 → processes-7.0.0}/examples/01_basic_tasks_and_dependencies/example1.py +4 -4
- {processes-3.1.0 → processes-7.0.0}/examples/02_task_dependencies_result_passing/README.md +7 -7
- {processes-3.1.0 → processes-7.0.0}/examples/02_task_dependencies_result_passing/example2.py +6 -6
- processes-7.0.0/examples/03_reports_and_notifications/README.md +32 -0
- processes-7.0.0/examples/03_reports_and_notifications/example3.py +181 -0
- {processes-3.1.0 → processes-7.0.0}/examples/README.md +12 -1
- {processes-3.1.0 → processes-7.0.0}/mkdocs.yml +1 -0
- {processes-3.1.0 → processes-7.0.0}/pyproject.toml +2 -2
- processes-7.0.0/src/processes/__init__.py +32 -0
- processes-7.0.0/src/processes/_logfile.py +100 -0
- processes-7.0.0/src/processes/_tb_utils.py +155 -0
- processes-7.0.0/src/processes/comms/__init__.py +15 -0
- processes-7.0.0/src/processes/comms/_email.py +304 -0
- processes-7.0.0/src/processes/comms/_webhook.py +126 -0
- processes-7.0.0/src/processes/comms/base.py +57 -0
- processes-7.0.0/src/processes/comms/channels.py +112 -0
- {processes-3.1.0/src/processes → processes-7.0.0/src/processes/comms}/email_config.py +14 -28
- processes-7.0.0/src/processes/comms/themes/languages/de.json +20 -0
- processes-7.0.0/src/processes/comms/themes/languages/en.json +20 -0
- processes-7.0.0/src/processes/comms/themes/languages/es.json +20 -0
- processes-7.0.0/src/processes/comms/themes/languages/fr.json +20 -0
- processes-7.0.0/src/processes/comms/themes/languages/it.json +20 -0
- processes-7.0.0/src/processes/comms/themes/languages/pt.json +20 -0
- processes-7.0.0/src/processes/comms/themes/styles/report.html +178 -0
- processes-7.0.0/src/processes/comms/webhook_config.py +44 -0
- processes-7.0.0/src/processes/error_data.py +42 -0
- processes-7.0.0/src/processes/execution_report.py +221 -0
- {processes-3.1.0 → processes-7.0.0}/src/processes/process.py +203 -94
- {processes-3.1.0 → processes-7.0.0}/src/processes/task.py +103 -154
- processes-7.0.0/src/processes/task_types.py +202 -0
- processes-7.0.0/tests/conftest.py +75 -0
- processes-7.0.0/tests/manual_tests/manual_report_notify.py +331 -0
- {processes-3.1.0 → processes-7.0.0}/tests/test_args_kwargs.py +25 -23
- {processes-3.1.0 → processes-7.0.0}/tests/test_complex_dag_failures.py +30 -117
- {processes-3.1.0 → processes-7.0.0}/tests/test_dependencies.py +14 -14
- processes-7.0.0/tests/test_execution_report.py +120 -0
- {processes-3.1.0 → processes-7.0.0}/tests/test_logfiles.py +4 -4
- {processes-3.1.0 → processes-7.0.0}/tests/test_normal_run_no_errors.py +22 -27
- {processes-3.1.0 → processes-7.0.0}/tests/test_parallel_race_conditions.py +19 -18
- processes-7.0.0/tests/test_process_mutation.py +137 -0
- processes-7.0.0/tests/test_report_notify_dispatch.py +155 -0
- processes-7.0.0/tests/test_report_rendering.py +149 -0
- processes-7.0.0/tests/test_report_send.py +344 -0
- processes-7.0.0/tests/test_report_to_json.py +98 -0
- processes-7.0.0/tests/test_resolve_args_failure.py +52 -0
- {processes-3.1.0 → processes-7.0.0}/tests/test_timeout_retry.py +40 -38
- processes-7.0.0/tests/test_unique_name.py +71 -0
- {processes-3.1.0 → processes-7.0.0}/uv.lock +34 -171
- processes-3.1.0/.github/workflows/lint-pr.yml +0 -31
- processes-3.1.0/CHANGELOG.md +0 -98
- processes-3.1.0/RELEASE_NOTES.md +0 -36
- processes-3.1.0/docs/reference.md +0 -10
- processes-3.1.0/src/processes/__init__.py +0 -23
- processes-3.1.0/src/processes/_email_internals.py +0 -168
- processes-3.1.0/src/processes/_error_data.py +0 -38
- processes-3.1.0/src/processes/_log_formatting.py +0 -44
- processes-3.1.0/src/processes/_tb_utils.py +0 -79
- processes-3.1.0/src/processes/themes/languages/de.json +0 -15
- processes-3.1.0/src/processes/themes/languages/en.json +0 -15
- processes-3.1.0/src/processes/themes/languages/es.json +0 -15
- processes-3.1.0/src/processes/themes/languages/fr.json +0 -15
- processes-3.1.0/src/processes/themes/languages/it.json +0 -15
- processes-3.1.0/src/processes/themes/languages/pt.json +0 -15
- processes-3.1.0/src/processes/themes/styles/classic.html +0 -49
- processes-3.1.0/src/processes/themes/styles/compact.html +0 -67
- processes-3.1.0/src/processes/themes/styles/modern.html +0 -115
- processes-3.1.0/tests/manual_tests/manual_pipeline_inspect.py +0 -423
- processes-3.1.0/tests/manual_tests/manual_themed_tracebacks.py +0 -304
- processes-3.1.0/tests/test_email_themes.py +0 -531
- processes-3.1.0/tests/test_unique_name.py +0 -40
- {processes-3.1.0 → processes-7.0.0}/.github/ISSUE_TEMPLATE/bug_report.md +0 -0
- {processes-3.1.0 → processes-7.0.0}/.github/ISSUE_TEMPLATE/custom.md +0 -0
- {processes-3.1.0 → processes-7.0.0}/.github/ISSUE_TEMPLATE/feature_request.md +0 -0
- {processes-3.1.0 → processes-7.0.0}/.github/PULL_REQUEST_TEMPLATE.md +0 -0
- {processes-3.1.0 → processes-7.0.0}/.github/workflows/docs.yml +0 -0
- {processes-3.1.0 → processes-7.0.0}/.github/workflows/lint.yml +0 -0
- {processes-3.1.0 → processes-7.0.0}/.github/workflows/mypy.yml +0 -0
- {processes-3.1.0 → processes-7.0.0}/.github/workflows/publish.yml +0 -0
- {processes-3.1.0 → processes-7.0.0}/.github/workflows/tags.yml +0 -0
- {processes-3.1.0 → processes-7.0.0}/CONTRIBUTING.md +0 -0
- {processes-3.1.0 → processes-7.0.0}/LICENSE +0 -0
- {processes-3.1.0 → processes-7.0.0}/assets/banner.svg +0 -0
- {processes-3.1.0 → processes-7.0.0}/assets/logo.png +0 -0
- {processes-3.1.0 → processes-7.0.0}/assets/logo.svg +0 -0
- {processes-3.1.0 → processes-7.0.0}/assets/social_banner.png +0 -0
- {processes-3.1.0 → processes-7.0.0}/assets/social_banner.svg +0 -0
- {processes-3.1.0 → processes-7.0.0}/docs/assets/favicon.svg +0 -0
- {processes-3.1.0 → processes-7.0.0}/docs/examples/intro.md +0 -0
- {processes-3.1.0 → processes-7.0.0}/pytest.ini +0 -0
- {processes-3.1.0/src/processes → processes-7.0.0/src/processes/comms}/themes/palettes/catppuccin.css +0 -0
- {processes-3.1.0/src/processes → processes-7.0.0/src/processes/comms}/themes/palettes/neobones.css +0 -0
- {processes-3.1.0/src/processes → processes-7.0.0/src/processes/comms}/themes/palettes/neutral.css +0 -0
- {processes-3.1.0/src/processes → processes-7.0.0/src/processes/comms}/themes/palettes/slate.css +0 -0
- {processes-3.1.0 → processes-7.0.0}/src/processes/exceptions.py +0 -0
- {processes-3.1.0 → processes-7.0.0}/src/processes/py.typed +0 -0
- {processes-3.1.0 → processes-7.0.0}/tests/__init__.py +0 -0
- {processes-3.1.0 → processes-7.0.0}/tests/base_test.py +0 -0
- {processes-3.1.0 → processes-7.0.0}/tests/manual_tests/__init__.py +0 -0
|
@@ -0,0 +1,31 @@
|
|
|
1
|
+
name: "Lint PR"
|
|
2
|
+
|
|
3
|
+
on:
|
|
4
|
+
pull_request:
|
|
5
|
+
types:
|
|
6
|
+
- opened
|
|
7
|
+
- edited
|
|
8
|
+
- synchronize
|
|
9
|
+
|
|
10
|
+
permissions:
|
|
11
|
+
pull-requests: read
|
|
12
|
+
|
|
13
|
+
jobs:
|
|
14
|
+
main:
|
|
15
|
+
name: Validate PR title
|
|
16
|
+
runs-on: ubuntu-latest
|
|
17
|
+
steps:
|
|
18
|
+
- uses: amannn/action-semantic-pull-request@v6
|
|
19
|
+
env:
|
|
20
|
+
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
|
21
|
+
with:
|
|
22
|
+
types: |
|
|
23
|
+
fix
|
|
24
|
+
feat
|
|
25
|
+
docs
|
|
26
|
+
style
|
|
27
|
+
refactor
|
|
28
|
+
perf
|
|
29
|
+
test
|
|
30
|
+
build
|
|
31
|
+
ci
|
|
@@ -0,0 +1,204 @@
|
|
|
1
|
+
## v7.0.0 (2026-06-20)
|
|
2
|
+
|
|
3
|
+
### BREAKING CHANGE
|
|
4
|
+
|
|
5
|
+
- Task(channels=...) and the NotificationChannel API are removed;
|
|
6
|
+
HTMLEmailStyle.style and traced_vars_frame_filter are removed (the latter moves
|
|
7
|
+
to Task). Per-task email/webhook alerts are replaced by report.notify.
|
|
8
|
+
|
|
9
|
+
### Feat
|
|
10
|
+
|
|
11
|
+
- add process name to report, email subject, and webhook payload
|
|
12
|
+
- normalize task names to lowercase for case-insensitive matching
|
|
13
|
+
- delegate notifications to ProcessExecutionReport, drop per-task channels
|
|
14
|
+
- replace notify_errors with notify(only_errors=...) and add task filter
|
|
15
|
+
- implement WebhookChannel and EmailChannel send_report
|
|
16
|
+
- add ReportChannel architecture for report notifications
|
|
17
|
+
- add ProcessExecutionReport.notify/.notify_errors stubs
|
|
18
|
+
- add ProcessExecutionReport.to_json
|
|
19
|
+
- add graph mutation methods to Process
|
|
20
|
+
|
|
21
|
+
### Refactor
|
|
22
|
+
|
|
23
|
+
- group communication code into a comms/ package
|
|
24
|
+
- unify SMTP and webhook transport behind single classes
|
|
25
|
+
- extract task value types to leaf module, break import cycle
|
|
26
|
+
|
|
27
|
+
### Perf
|
|
28
|
+
|
|
29
|
+
- cache palette CSS and report template loading
|
|
30
|
+
- cache language string loading
|
|
31
|
+
- add slots=True to frozen value types
|
|
32
|
+
|
|
33
|
+
## v6.0.0 (2026-06-15)
|
|
34
|
+
|
|
35
|
+
### BREAKING CHANGE
|
|
36
|
+
|
|
37
|
+
- Process.run() returns ProcessExecutionReport instead of
|
|
38
|
+
ProcessResult. Use report.successes / report.errored / report.skipped
|
|
39
|
+
instead of passed_tasks_results / errored_tasks / skipped_tasks, and
|
|
40
|
+
report.entries[name].result / .error instead of
|
|
41
|
+
passed_tasks_results[name] / failed_tasks_results[name].
|
|
42
|
+
|
|
43
|
+
### Feat
|
|
44
|
+
|
|
45
|
+
- remove ProcessResult, have Process.run() return ProcessExecutionReport
|
|
46
|
+
- expand TaskStatus with PENDING and add status field to TaskResult
|
|
47
|
+
- add ProcessExecutionReport for per-task execution summaries
|
|
48
|
+
- track failed task results in ProcessRunner and ProcessResult
|
|
49
|
+
- promote ErrorData to public API and add timing/attempts to TaskResult
|
|
50
|
+
|
|
51
|
+
### Fix
|
|
52
|
+
|
|
53
|
+
- make Task.run never propagate dependency-resolution failures
|
|
54
|
+
|
|
55
|
+
### Refactor
|
|
56
|
+
|
|
57
|
+
- drop _is_unrunnable side effect and reuse TaskResult helpers
|
|
58
|
+
|
|
59
|
+
## v5.0.0 (2026-06-14)
|
|
60
|
+
|
|
61
|
+
### BREAKING CHANGE
|
|
62
|
+
|
|
63
|
+
- log_path and func swap positions; existing positional
|
|
64
|
+
Task(name, log_path, func, ...) calls must be updated to
|
|
65
|
+
Task(name, func, log_path, ...).
|
|
66
|
+
|
|
67
|
+
### Feat
|
|
68
|
+
|
|
69
|
+
- add nest_under option to WebhookConfig for nested payload envelopes
|
|
70
|
+
- make Task.log_path optional, reorder constructor signature
|
|
71
|
+
|
|
72
|
+
## v4.1.0 (2026-06-14)
|
|
73
|
+
|
|
74
|
+
### Feat
|
|
75
|
+
|
|
76
|
+
- add extra_payload to WebhookConfig for service-specific routing keys
|
|
77
|
+
- include traced variables in task logfiles
|
|
78
|
+
- add generic WebhookChannel with optional HMAC body signing
|
|
79
|
+
|
|
80
|
+
### Refactor
|
|
81
|
+
|
|
82
|
+
- make traced_vars a plain {name: repr(value)} dict
|
|
83
|
+
|
|
84
|
+
## v4.0.0 (2026-06-14)
|
|
85
|
+
|
|
86
|
+
### BREAKING CHANGE
|
|
87
|
+
|
|
88
|
+
- Task no longer accepts smtp_config or email_style.
|
|
89
|
+
Pass channels=[EmailChannel(smtp_config, style)] instead.
|
|
90
|
+
|
|
91
|
+
### Feat
|
|
92
|
+
|
|
93
|
+
- add NotificationChannel abstraction with file and email channels
|
|
94
|
+
|
|
95
|
+
### Refactor
|
|
96
|
+
|
|
97
|
+
- configure email alerts via channels instead of Task smtp_config
|
|
98
|
+
- keep file and email channel implementations internal
|
|
99
|
+
- build task handlers through notification channels
|
|
100
|
+
|
|
101
|
+
## v3.1.1 (2026-06-13)
|
|
102
|
+
|
|
103
|
+
### Refactor
|
|
104
|
+
|
|
105
|
+
- rename _log_formatting to _logfile_formatting and _TaskLogFormatter to _TaskLogfileFormatter
|
|
106
|
+
|
|
107
|
+
## v3.1.0 (2026-06-13)
|
|
108
|
+
|
|
109
|
+
### Feat
|
|
110
|
+
|
|
111
|
+
- **logging**: include failure context in task logfiles
|
|
112
|
+
|
|
113
|
+
### Refactor
|
|
114
|
+
|
|
115
|
+
- extract typed _ErrorData from task_context via shared base formatter
|
|
116
|
+
- inline single-use _task_context_lines helper
|
|
117
|
+
- move task logfile formatting out of _email_internals
|
|
118
|
+
|
|
119
|
+
## v3.0.1 (2026-06-13)
|
|
120
|
+
|
|
121
|
+
### Refactor
|
|
122
|
+
|
|
123
|
+
- **process**: use bare raise and extract _is_done helper
|
|
124
|
+
|
|
125
|
+
## v3.0.0 (2026-06-13)
|
|
126
|
+
|
|
127
|
+
### BREAKING CHANGE
|
|
128
|
+
|
|
129
|
+
- html_mail_handler parameter removed from Task; use
|
|
130
|
+
smtp_config and email_style instead. HTMLSMTPHandler removed from public API.
|
|
131
|
+
|
|
132
|
+
### Feat
|
|
133
|
+
|
|
134
|
+
- **html_logging**: expose last_path_traced_vars on HTMLSMTPHandler
|
|
135
|
+
|
|
136
|
+
### Refactor
|
|
137
|
+
|
|
138
|
+
- **email**: replace HTMLSMTPHandler with SMTPConfig/HTMLEmailStyle
|
|
139
|
+
|
|
140
|
+
## v2.0.1 (2026-06-12)
|
|
141
|
+
|
|
142
|
+
### Fix
|
|
143
|
+
|
|
144
|
+
- **publish.yml**: remove unneeded step ruff format check in quality-gate
|
|
145
|
+
|
|
146
|
+
## v2.0.0 (2026-06-12)
|
|
147
|
+
|
|
148
|
+
### BREAKING CHANGE
|
|
149
|
+
|
|
150
|
+
- log records no longer carry
|
|
151
|
+
``post_traceback_html_body``. Consumers introspecting that attribute
|
|
152
|
+
must read ``record.task_context`` instead.
|
|
153
|
+
|
|
154
|
+
### Feat
|
|
155
|
+
|
|
156
|
+
- **email_alerting**: Traced Variables section with file:line reference in email body
|
|
157
|
+
- **email_alerting**: language alternatives for HTML email body
|
|
158
|
+
- **email_alerting**: template-driven HTML body from pure metadata
|
|
159
|
+
|
|
160
|
+
### Fix
|
|
161
|
+
|
|
162
|
+
- **process**: parallel runner no longer raises on unrunnable tail
|
|
163
|
+
|
|
164
|
+
## v1.0.5 (2026-01-19)
|
|
165
|
+
|
|
166
|
+
### Fix
|
|
167
|
+
|
|
168
|
+
- **docs**: added urls to pyproject file
|
|
169
|
+
|
|
170
|
+
## v1.0.4 (2026-01-19)
|
|
171
|
+
|
|
172
|
+
### Fix
|
|
173
|
+
|
|
174
|
+
- **docs**: added pypi badge and pip install instruction to readme
|
|
175
|
+
|
|
176
|
+
## v1.0.3 (2026-01-19)
|
|
177
|
+
|
|
178
|
+
### Fix
|
|
179
|
+
|
|
180
|
+
- **docs**: changes in readme and index of docs to show banner
|
|
181
|
+
|
|
182
|
+
## v1.0.2 (2026-01-19)
|
|
183
|
+
|
|
184
|
+
### Fix
|
|
185
|
+
|
|
186
|
+
- **ci**: added a workflow for publishing to pypi
|
|
187
|
+
|
|
188
|
+
## v1.0.1 (2026-01-19)
|
|
189
|
+
|
|
190
|
+
### Fix
|
|
191
|
+
|
|
192
|
+
- **lint**: fix ruff formatting
|
|
193
|
+
|
|
194
|
+
## v1.0.0 (2026-01-19)
|
|
195
|
+
|
|
196
|
+
### BREAKING CHANGE
|
|
197
|
+
|
|
198
|
+
- Task.run method had one of its kwargs removed
|
|
199
|
+
|
|
200
|
+
### Fix
|
|
201
|
+
|
|
202
|
+
- **Task-can-no-longer-pass-logger-to-its-function.-Changed-some-examples,-documentations-and-better-type-hints**: Task.run method no longer can pass logger to its function as kwarg
|
|
203
|
+
|
|
204
|
+
## v0.1.0 (2026-01-18)
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
Metadata-Version: 2.4
|
|
2
2
|
Name: processes
|
|
3
|
-
Version:
|
|
3
|
+
Version: 7.0.0
|
|
4
4
|
Summary: Orchestrate graphs of callables in Python with automatic dependency resolution, parallel execution, retries, timeouts, and HTML email alerts on failure — zero dependencies
|
|
5
5
|
Project-URL: Homepage, https://github.com/oliverm91/processes
|
|
6
6
|
Project-URL: Documentation, https://oliverm91.github.io/processes/
|
|
@@ -14,13 +14,12 @@ Keywords: dag,dependencies,etl,parallel,process,tasks,topological-sort,workflow
|
|
|
14
14
|
Classifier: Development Status :: 4 - Beta
|
|
15
15
|
Classifier: Intended Audience :: Developers
|
|
16
16
|
Classifier: Programming Language :: Python :: 3
|
|
17
|
-
Classifier: Programming Language :: Python :: 3.10
|
|
18
17
|
Classifier: Programming Language :: Python :: 3.11
|
|
19
18
|
Classifier: Programming Language :: Python :: 3.12
|
|
20
19
|
Classifier: Programming Language :: Python :: 3.13
|
|
21
20
|
Classifier: Programming Language :: Python :: 3.14
|
|
22
21
|
Classifier: Typing :: Typed
|
|
23
|
-
Requires-Python: >=3.
|
|
22
|
+
Requires-Python: >=3.11
|
|
24
23
|
Description-Content-Type: text/markdown
|
|
25
24
|
|
|
26
25
|
<div align="center">
|
|
@@ -29,7 +28,7 @@ Description-Content-Type: text/markdown
|
|
|
29
28
|
|
|
30
29
|
# Processes: Smart Task Orchestration
|
|
31
30
|
|
|
32
|
-
[](https://www.python.org/)
|
|
33
32
|

|
|
34
33
|
[](https://oliverm91.github.io/processes/)
|
|
35
34
|
[](https://opensource.org/licenses/MIT)
|
|
@@ -41,7 +40,7 @@ Description-Content-Type: text/markdown
|
|
|
41
40
|
|
|
42
41
|
---
|
|
43
42
|
|
|
44
|
-
**Run a list of Python callables that depend on each other — in parallel when possible, with per-task log files and optional HTML email notification on failure. Zero dependencies. Pure Python 3.
|
|
43
|
+
**Run a list of Python callables that depend on each other — in parallel when possible, with per-task log files and optional HTML email notification on failure. Zero dependencies. Pure Python 3.11+.**
|
|
45
44
|
|
|
46
45
|
---
|
|
47
46
|
|
|
@@ -52,7 +51,7 @@ Description-Content-Type: text/markdown
|
|
|
52
51
|
- 🛡️ **One failure doesn't stop the rest** — a failed task skips only the jobs that depend on it, and **every other part of the workflow keeps running**.
|
|
53
52
|
- 📝 **One log file per task** — share a single log across the whole run, or keep them separate for easier debugging.
|
|
54
53
|
- 📧 **Email alerts when something breaks** — pass an `SMTPConfig` to a task and get a styled HTML email (with traceback, task context, and the list of jobs that were skipped) the instant it raises.
|
|
55
|
-
- 🧰 **Modern, strictly-typed Python 3.
|
|
54
|
+
- 🧰 **Modern, strictly-typed Python 3.11+** — `from __future__ import annotations`, full `mypy --strict` clean, `dict[str, TaskResult]`, `set[str]`, `|` unions.
|
|
56
55
|
|
|
57
56
|
---
|
|
58
57
|
|
|
@@ -62,7 +61,7 @@ A `Process` holds a list of `Task`s. At construction it validates names, types,
|
|
|
62
61
|
|
|
63
62
|
When you call `process.run()`, tasks are topologically sorted and scheduled: dependencies first, independent tasks in parallel.
|
|
64
63
|
|
|
65
|
-
A `TaskDependency` can forward an upstream result directly into a downstream function, as a positional or keyword argument. The result is a `
|
|
64
|
+
A `TaskDependency` can forward an upstream result directly into a downstream function, as a positional or keyword argument. The result is a `ProcessExecutionReport` with `successes`, `errored`, and `skipped` for inspection.
|
|
66
65
|
|
|
67
66
|
---
|
|
68
67
|
|
|
@@ -83,11 +82,11 @@ def enrich(users: list[dict]) -> list[dict]:
|
|
|
83
82
|
|
|
84
83
|
|
|
85
84
|
tasks = [
|
|
86
|
-
Task("load", "run.log"
|
|
85
|
+
Task("load", load_users, "run.log"),
|
|
87
86
|
Task(
|
|
88
87
|
"enrich",
|
|
89
|
-
"run.log",
|
|
90
88
|
enrich,
|
|
89
|
+
"run.log",
|
|
91
90
|
dependencies=[TaskDependency("load", use_result_as_additional_args=True)],
|
|
92
91
|
),
|
|
93
92
|
]
|
|
@@ -95,7 +94,7 @@ tasks = [
|
|
|
95
94
|
with Process(tasks) as p:
|
|
96
95
|
result = p.run(parallel=True)
|
|
97
96
|
|
|
98
|
-
print(result.
|
|
97
|
+
print(result.successes["enrich"].result)
|
|
99
98
|
# [{'id': 1, 'name': 'user-1'}, {'id': 2, 'name': 'user-2'}, {'id': 3, 'name': 'user-3'}]
|
|
100
99
|
```
|
|
101
100
|
|
|
@@ -112,7 +111,7 @@ A realistic mini-pipeline: fetch two sources **in parallel**, transform them, ag
|
|
|
112
111
|
import logging
|
|
113
112
|
from pathlib import Path
|
|
114
113
|
|
|
115
|
-
from processes import HTMLEmailStyle, Process, SMTPConfig, Task, TaskDependency
|
|
114
|
+
from processes import EmailChannel, HTMLEmailStyle, Process, SMTPConfig, Task, TaskDependency
|
|
116
115
|
|
|
117
116
|
LOG_DIR = Path("logs")
|
|
118
117
|
LOG_DIR.mkdir(exist_ok=True)
|
|
@@ -169,19 +168,19 @@ smtp = SMTPConfig(
|
|
|
169
168
|
)
|
|
170
169
|
|
|
171
170
|
tasks = [
|
|
172
|
-
Task("fetch_orders", LOG_DIR / "fetch_orders.log"
|
|
173
|
-
Task("fetch_inventory", LOG_DIR / "fetch_inventory.log"
|
|
171
|
+
Task("fetch_orders", fetch_orders, LOG_DIR / "fetch_orders.log"),
|
|
172
|
+
Task("fetch_inventory", fetch_inventory, LOG_DIR / "fetch_inventory.log"),
|
|
174
173
|
|
|
175
174
|
Task(
|
|
176
175
|
"compute_revenue",
|
|
177
|
-
LOG_DIR / "compute_revenue.log",
|
|
178
176
|
total_revenue,
|
|
177
|
+
LOG_DIR / "compute_revenue.log",
|
|
179
178
|
dependencies=[TaskDependency("fetch_orders", use_result_as_additional_args=True)],
|
|
180
179
|
),
|
|
181
180
|
Task(
|
|
182
181
|
"compute_stock",
|
|
183
|
-
LOG_DIR / "compute_stock.log",
|
|
184
182
|
stock_value,
|
|
183
|
+
LOG_DIR / "compute_stock.log",
|
|
185
184
|
kwargs={"price_per_unit": 7.25},
|
|
186
185
|
dependencies=[
|
|
187
186
|
TaskDependency(
|
|
@@ -194,8 +193,8 @@ tasks = [
|
|
|
194
193
|
|
|
195
194
|
Task(
|
|
196
195
|
"build_report",
|
|
197
|
-
LOG_DIR / "build_report.log",
|
|
198
196
|
build_report,
|
|
197
|
+
LOG_DIR / "build_report.log",
|
|
199
198
|
dependencies=[
|
|
200
199
|
TaskDependency("compute_revenue", use_result_as_additional_kwargs=True,
|
|
201
200
|
additional_kwarg_name="revenue"),
|
|
@@ -210,31 +209,31 @@ tasks = [
|
|
|
210
209
|
# the workflow is not blackholed by one broken step.
|
|
211
210
|
Task(
|
|
212
211
|
"notify_slack",
|
|
213
|
-
LOG_DIR / "notify_slack.log",
|
|
214
212
|
notify_slack,
|
|
213
|
+
LOG_DIR / "notify_slack.log",
|
|
215
214
|
dependencies=[TaskDependency("build_report", use_result_as_additional_args=True)],
|
|
216
|
-
smtp_config=smtp,
|
|
217
215
|
),
|
|
218
216
|
Task(
|
|
219
217
|
"archive_report",
|
|
220
|
-
LOG_DIR / "archive_report.log",
|
|
221
218
|
archive_report,
|
|
219
|
+
LOG_DIR / "archive_report.log",
|
|
222
220
|
dependencies=[TaskDependency("build_report", use_result_as_additional_args=True)],
|
|
223
221
|
),
|
|
224
222
|
]
|
|
225
223
|
|
|
226
224
|
with Process(tasks) as process:
|
|
227
225
|
result = process.run(parallel=True)
|
|
226
|
+
result.notify(EmailChannel(smtp), only_errors=True) # one report email for the failed task(s)
|
|
228
227
|
|
|
229
|
-
print("passed:", sorted(result.
|
|
228
|
+
print("passed:", sorted(result.successes))
|
|
230
229
|
# archive_report, build_report, compute_revenue, compute_stock, fetch_inventory, fetch_orders
|
|
231
|
-
print("failed:", sorted(result.
|
|
230
|
+
print("failed:", sorted(set(result.errored) | set(result.skipped)))
|
|
232
231
|
# notify_slack
|
|
233
|
-
print("report:", result.
|
|
232
|
+
print("report:", result.successes["build_report"].result)
|
|
234
233
|
# daily-report | revenue=59.50 stock=262.50
|
|
235
234
|
```
|
|
236
235
|
|
|
237
|
-
The failing `notify_slack` task does **not** abort the run. `archive_report` is a sibling of the failed task (both depend on the successful `build_report`), so it runs unaffected — the rest of the workflow is not blackholed by one broken step.
|
|
236
|
+
The failing `notify_slack` task does **not** abort the run. `archive_report` is a sibling of the failed task (both depend on the successful `build_report`), so it runs unaffected — the rest of the workflow is not blackholed by one broken step. Calling `result.notify(EmailChannel(smtp), only_errors=True)` then delivers a single report email covering the failed task with its traceback and the downstream tasks that were skipped because of it.
|
|
238
237
|
|
|
239
238
|
</details>
|
|
240
239
|
|
|
@@ -250,13 +249,12 @@ The failing `notify_slack` task does **not** abort the run. `archive_report` is
|
|
|
250
249
|
```python
|
|
251
250
|
Task(
|
|
252
251
|
name: str,
|
|
253
|
-
log_path: str | os.PathLike,
|
|
254
252
|
func: Callable[..., Any],
|
|
253
|
+
log_path: str | None = None,
|
|
255
254
|
args: tuple = (),
|
|
256
255
|
kwargs: dict | None = None,
|
|
257
256
|
dependencies: list[TaskDependency] | None = None,
|
|
258
|
-
|
|
259
|
-
email_style: HTMLEmailStyle | None = None,
|
|
257
|
+
traced_vars_frame_filter: str | None = None,
|
|
260
258
|
timeout: float | None = None,
|
|
261
259
|
retries: int | None = 0,
|
|
262
260
|
retry_on: tuple[type[Exception], ...] | None = None,
|
|
@@ -264,10 +262,9 @@ Task(
|
|
|
264
262
|
```
|
|
265
263
|
|
|
266
264
|
- `name` — unique within the `Process`; no spaces.
|
|
267
|
-
- `log_path` — the file this task logs to (INFO level, format `%(asctime)s - %(name)s - %(levelname)s - %(message)s`).
|
|
265
|
+
- `log_path` — the file this task logs to (INFO level, format `%(asctime)s - %(name)s - %(levelname)s - %(message)s`), with the structured failure context appended on error. `None` (the default) means no file logging; a `NullHandler` is attached instead. A `Task` does not send notifications — error notification is delegated to `ProcessExecutionReport.notify`.
|
|
268
266
|
- `func` — the callable; receives `func(*args, **kwargs)` after result-injection.
|
|
269
|
-
- `
|
|
270
|
-
- `email_style` — optional presentation override; defaults to `HTMLEmailStyle()` (modern, neutral, English) when `smtp_config` is set.
|
|
267
|
+
- `traced_vars_frame_filter` — substring selecting which traceback frame's locals are captured into the failure context (and thus into both the logfile and any report notification). `None` (default) captures the outermost user frame.
|
|
271
268
|
- `timeout` — seconds allowed per attempt; `None` means no limit. When the timeout fires the underlying thread is detached (Python threading limitation).
|
|
272
269
|
- `retries` — additional attempts after the first failure; `0` or `None` means a single attempt. Defaults to `0`.
|
|
273
270
|
- `retry_on` — tuple of exception types that trigger a retry. When `retries >= 1` and `retry_on` is `None`, defaults to `(ConnectionError, TimeoutError)` at call time.
|
|
@@ -292,24 +289,86 @@ TaskDependency(
|
|
|
292
289
|
```python
|
|
293
290
|
Process(tasks: list[Task]) # validates types, names, deps, cycles
|
|
294
291
|
|
|
295
|
-
process.run(parallel: bool | None = None, max_workers: int = 4) ->
|
|
292
|
+
process.run(parallel: bool | None = None, max_workers: int = 4) -> ProcessExecutionReport
|
|
296
293
|
```
|
|
297
294
|
|
|
298
295
|
- Raises `DependencyNotFoundError`, `CircularDependencyError`, `TypeError`, `ValueError` on construction if the workflow is malformed.
|
|
299
296
|
- `parallel=None` auto-parallelises when `len(tasks) >= 10`; `max_workers=1` is always sequential.
|
|
300
297
|
- Use as a context manager — it cleans up `FileHandler`s on exit.
|
|
301
298
|
|
|
302
|
-
### `
|
|
299
|
+
### `TaskResult`
|
|
300
|
+
|
|
301
|
+
```python
|
|
302
|
+
TaskResult(
|
|
303
|
+
status: TaskStatus,
|
|
304
|
+
result: Any,
|
|
305
|
+
exception: Exception | None,
|
|
306
|
+
error_data: ErrorData | None = None,
|
|
307
|
+
elapsed_seconds: float = 0.0, # wall-clock time across all attempts
|
|
308
|
+
attempts: int = 0, # attempts actually executed (0 if never run)
|
|
309
|
+
)
|
|
310
|
+
```
|
|
311
|
+
|
|
312
|
+
- `status` — `TaskStatus.PENDING | SUCCESS | ERRORED | SKIPPED`.
|
|
313
|
+
- `worked` — `True` if `status == TaskStatus.SUCCESS`.
|
|
314
|
+
|
|
315
|
+
### `ProcessExecutionReport`
|
|
316
|
+
|
|
317
|
+
```python
|
|
318
|
+
report.entries # dict[str, TaskReportEntry] — one entry per task, in topological order
|
|
319
|
+
report.successes # dict[str, TaskReportEntry] — entries with status == TaskStatus.SUCCESS
|
|
320
|
+
report.errored # dict[str, TaskReportEntry] — entries with status == TaskStatus.ERRORED
|
|
321
|
+
report.skipped # dict[str, TaskReportEntry] — entries with status == TaskStatus.SKIPPED
|
|
322
|
+
```
|
|
323
|
+
|
|
324
|
+
A `TaskReportEntry` carries `name`, `function`, `args`, `kwargs`, `status`
|
|
325
|
+
(`TaskStatus.SUCCESS | ERRORED | SKIPPED`), `elapsed_seconds`, `attempts`,
|
|
326
|
+
plus `result` (set when `SUCCESS`) and `error: ErrorData | None` (set when
|
|
327
|
+
`ERRORED`).
|
|
303
328
|
|
|
304
329
|
```python
|
|
305
|
-
|
|
306
|
-
|
|
307
|
-
|
|
308
|
-
|
|
330
|
+
with Process([load_task, apply_task, notify_task]) as process:
|
|
331
|
+
report = process.run()
|
|
332
|
+
|
|
333
|
+
for name, entry in report.entries.items():
|
|
334
|
+
if entry.status is TaskStatus.ERRORED:
|
|
335
|
+
print(f"{name} failed after {entry.attempts} attempt(s): {entry.error.exception}")
|
|
336
|
+
```
|
|
309
337
|
|
|
310
|
-
|
|
338
|
+
Deliver the report through one or more channels with `notify`:
|
|
339
|
+
|
|
340
|
+
```python
|
|
341
|
+
report.notify(
|
|
342
|
+
*channels: ReportChannel,
|
|
343
|
+
only_errors: bool = False, # restrict the payload to ERRORED tasks
|
|
344
|
+
tasks: list[str] | None = None, # restrict to these task names (case-insensitive)
|
|
345
|
+
show_warnings: bool = True, # warn (not raise) if a channel fails
|
|
346
|
+
)
|
|
311
347
|
```
|
|
312
348
|
|
|
349
|
+
`only_errors` and `tasks` compose (both filters apply). A failing channel never
|
|
350
|
+
aborts the others. Built-in channels: `EmailChannel` (HTML email) and
|
|
351
|
+
`WebhookChannel` (JSON POST).
|
|
352
|
+
|
|
353
|
+
### `ErrorData`
|
|
354
|
+
|
|
355
|
+
```python
|
|
356
|
+
ErrorData(
|
|
357
|
+
task_name: str,
|
|
358
|
+
function: str,
|
|
359
|
+
args: tuple[Any, ...],
|
|
360
|
+
kwargs: dict[str, Any],
|
|
361
|
+
downstream_impact: list[str],
|
|
362
|
+
exception: str,
|
|
363
|
+
traceback_str: str,
|
|
364
|
+
traced_vars: dict[str, str],
|
|
365
|
+
traced_vars_location: str,
|
|
366
|
+
)
|
|
367
|
+
```
|
|
368
|
+
|
|
369
|
+
Structured failure context for a single task, available via
|
|
370
|
+
`TaskResult.error_data` and `TaskReportEntry.error`.
|
|
371
|
+
|
|
313
372
|
### `SMTPConfig`
|
|
314
373
|
|
|
315
374
|
```python
|
|
@@ -327,14 +386,92 @@ SMTPConfig(
|
|
|
327
386
|
|
|
328
387
|
```python
|
|
329
388
|
HTMLEmailStyle(
|
|
330
|
-
style="modern", # classic | modern | compact
|
|
331
389
|
palette="neutral", # neutral | catppuccin | neobones | slate
|
|
332
390
|
language="en", # en | es | pt | fr | de | it
|
|
333
|
-
traced_vars_frame_filter=None, # substring to pick the traced frame | None
|
|
334
391
|
)
|
|
335
392
|
```
|
|
336
393
|
|
|
337
|
-
|
|
394
|
+
### `ReportContent`
|
|
395
|
+
|
|
396
|
+
```python
|
|
397
|
+
ReportContent(
|
|
398
|
+
show_traceback=True, # include each failure's full traceback
|
|
399
|
+
show_traced_vars=True, # include each failure's traced local variables
|
|
400
|
+
)
|
|
401
|
+
```
|
|
402
|
+
|
|
403
|
+
Per-channel selection of how much per-task detail a report notification includes.
|
|
404
|
+
Pass the same instance to several channels for uniform content, or give each its
|
|
405
|
+
own.
|
|
406
|
+
|
|
407
|
+
### `EmailChannel`
|
|
408
|
+
|
|
409
|
+
```python
|
|
410
|
+
EmailChannel(
|
|
411
|
+
smtp_config: SMTPConfig,
|
|
412
|
+
style: HTMLEmailStyle | None = None, # defaults to HTMLEmailStyle()
|
|
413
|
+
content: ReportContent | None = None, # defaults to ReportContent()
|
|
414
|
+
)
|
|
415
|
+
```
|
|
416
|
+
|
|
417
|
+
A `ReportChannel` that delivers a finished report as a styled HTML email when
|
|
418
|
+
passed to `report.notify(...)`. The body lists every task with its status and,
|
|
419
|
+
for each failure, the exception, traceback, downstream impact, and traced
|
|
420
|
+
variables (subject to `content`).
|
|
421
|
+
|
|
422
|
+
#### Traced Variables
|
|
423
|
+
|
|
424
|
+
On failure, each task captures the local variables of the **outermost user
|
|
425
|
+
frame in the traceback** — i.e. the last frame that is not inside
|
|
426
|
+
`site-packages` or your virtualenv — into both its logfile and the report email.
|
|
427
|
+
A `file:line` reference next to the section shows exactly where those values
|
|
428
|
+
were captured. Point this at a different frame per task with
|
|
429
|
+
`Task(traced_vars_frame_filter=…)`.
|
|
430
|
+
|
|
431
|
+
### `WebhookChannel`
|
|
432
|
+
|
|
433
|
+
```python
|
|
434
|
+
WebhookChannel(
|
|
435
|
+
webhook_config: WebhookConfig,
|
|
436
|
+
content: ReportContent | None = None, # defaults to ReportContent()
|
|
437
|
+
)
|
|
438
|
+
```
|
|
439
|
+
|
|
440
|
+
A `ReportChannel` that POSTs the finished report as JSON when passed to
|
|
441
|
+
`report.notify(...)`.
|
|
442
|
+
|
|
443
|
+
```python
|
|
444
|
+
WebhookConfig(
|
|
445
|
+
url: str,
|
|
446
|
+
headers: dict[str, str] = {}, # merged with default Content-Type: application/json
|
|
447
|
+
timeout: int = 5,
|
|
448
|
+
secret: str | None = None, # HMAC-SHA256 signs the body when set
|
|
449
|
+
extra_payload: dict[str, Any] = {}, # extra top-level keys merged into the JSON body
|
|
450
|
+
nest_under: str | None = None, # nest the generic fields under this key, e.g. "data"
|
|
451
|
+
)
|
|
452
|
+
```
|
|
453
|
+
|
|
454
|
+
Transport configuration for `WebhookChannel`. When the report is delivered,
|
|
455
|
+
POSTs a generic JSON payload to `url` — an `entries` object mapping each task
|
|
456
|
+
name to its `status`, `function`, `elapsed_seconds`, `attempts`, and (for
|
|
457
|
+
failures) an `error` block with `exception`, `traceback`, `downstream_impact`,
|
|
458
|
+
and `traced_vars` (subject to `ReportContent`). Not coupled to any specific
|
|
459
|
+
service (Slack, Discord, etc.).
|
|
460
|
+
|
|
461
|
+
`extra_payload` keys are merged into the JSON body and take precedence over
|
|
462
|
+
the generic fields on collision — useful for service-specific routing data
|
|
463
|
+
(e.g. a Telegram `chat_id` or a Slack `channel`/`username` override) without
|
|
464
|
+
subclassing.
|
|
465
|
+
|
|
466
|
+
`nest_under` nests the generic fields under a single key instead of leaving
|
|
467
|
+
them top-level, e.g. `nest_under="data"` produces
|
|
468
|
+
`{"data": {"task_name": ..., ...}, "chat_id": "..."}`. `extra_payload` keys
|
|
469
|
+
always stay top-level and still take precedence on collision. `None`
|
|
470
|
+
(the default) keeps the flat payload shape.
|
|
471
|
+
|
|
472
|
+
If `secret` is set, the request carries an `X-Signature-SHA256` header with
|
|
473
|
+
the hex-encoded `hmac.new(secret, body, hashlib.sha256)` digest of the JSON
|
|
474
|
+
body, so receivers can verify the payload wasn't tampered with.
|
|
338
475
|
|
|
339
476
|
</details>
|
|
340
477
|
|
|
@@ -343,10 +480,10 @@ All fields are optional — omit `HTMLEmailStyle` entirely to use the defaults.
|
|
|
343
480
|
|
|
344
481
|
When a task raises:
|
|
345
482
|
|
|
346
|
-
1. The exception is caught and stored in `TaskResult.exception`; the task
|
|
347
|
-
2. **Every task that depends on it (directly or indirectly) is skipped** —
|
|
483
|
+
1. The exception is caught and stored in `TaskResult.exception`; the task's entry in the report gets `status == TaskStatus.ERRORED`.
|
|
484
|
+
2. **Every task that depends on it (directly or indirectly) is skipped** — its entry gets `status == TaskStatus.SKIPPED` without running.
|
|
348
485
|
3. **Every other independent part of the workflow keeps running.** With `parallel=True` they keep running concurrently on the worker pool.
|
|
349
|
-
4. After `run()` returns, `
|
|
486
|
+
4. After `run()` returns, `ProcessExecutionReport.errored` and `ProcessExecutionReport.skipped` let you distinguish root failures from cascade skips for triage or alerting.
|
|
350
487
|
|
|
351
488
|
When a task has `retries >= 1`, a failure matching `retry_on` triggers another attempt before the task is declared failed and its dependants are skipped. This gives transient errors (network blips, connection resets) a chance to resolve without aborting downstream work.
|
|
352
489
|
|
|
@@ -355,19 +492,9 @@ This makes the library a good fit for fan-out / fan-in pipelines, "best-effort"
|
|
|
355
492
|
</details>
|
|
356
493
|
|
|
357
494
|
<details>
|
|
358
|
-
<summary>Show
|
|
359
|
-
|
|
360
|
-
| | **Processes** | Airflow | Celery | Luigi |
|
|
361
|
-
|---|---|---|---|---|
|
|
362
|
-
| External dependencies | **None** | many | broker (Redis/RabbitMQ) | few |
|
|
363
|
-
| Setup cost | `pip install` | cluster | broker + workers | task + config |
|
|
364
|
-
| Parallelism | built-in | via executors | via workers | via workers |
|
|
365
|
-
| Per-task file logs | **yes (built-in)** | via handlers | via signals | partial |
|
|
366
|
-
| HTML email on failure | **yes (built-in)** | via callbacks | via signals | manual |
|
|
367
|
-
| DAG validation at construction | **yes** | yes (DAG file) | n/a | partial |
|
|
368
|
-
| Strict typing (`mypy --strict`) | **yes** | partial | partial | no |
|
|
495
|
+
<summary>Show scope & limitations</summary>
|
|
369
496
|
|
|
370
|
-
`Processes` is **not** a distributed scheduler — there are no workers on remote machines, no SLA monitoring, no web UI. If you need any of those,
|
|
497
|
+
`Processes` is **not** a distributed scheduler — there are no workers on remote machines, no SLA monitoring, no web UI. If you need any of those, reach for a full orchestrator. If you want a small, fast, dependency-aware pipeline that *just runs* in a single process, this is it.
|
|
371
498
|
|
|
372
499
|
</details>
|
|
373
500
|
|
|
@@ -376,7 +503,7 @@ This makes the library a good fit for fan-out / fan-in pipelines, "best-effort"
|
|
|
376
503
|
|
|
377
504
|
- **Shared log file** — pass the same `log_path` to every `Task` for a single combined run.log; pass distinct paths for per-task isolation.
|
|
378
505
|
- **Auto-parallel** — `Process.run()` with no argument runs sequentially for small workflows and switches to parallel for `len(tasks) >= 10`. Pass `parallel=True` or `parallel=False` to force the mode.
|
|
379
|
-
- **Result inspection** — iterate `
|
|
506
|
+
- **Result inspection** — iterate `report.successes.items()` to log or post-process every successful task; iterate `set(report.errored) | set(report.skipped)` for triage.
|
|
380
507
|
- **Re-raising** — wrap `process.run()` in `try/except` if you need a non-zero exit code on any failure; the library itself does not raise on partial failure.
|
|
381
508
|
|
|
382
509
|
</details>
|
|
@@ -397,7 +524,7 @@ Or straight from the repository (pure Python, no build step):
|
|
|
397
524
|
pip install git+https://github.com/oliverm91/processes.git
|
|
398
525
|
```
|
|
399
526
|
|
|
400
|
-
Requires **Python 3.
|
|
527
|
+
Requires **Python 3.11+**.
|
|
401
528
|
|
|
402
529
|
---
|
|
403
530
|
|