dotflow 0.15.0.dev2__tar.gz → 0.15.0.dev4__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev4}/PKG-INFO +243 -211
- {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev4}/README.md +238 -210
- {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev4}/dotflow/__init__.py +1 -1
- {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev4}/dotflow/abc/flow.py +19 -0
- dotflow-0.15.0.dev4/dotflow/abc/server.py +30 -0
- {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev4}/dotflow/cli/commands/cloud.py +9 -1
- dotflow-0.15.0.dev4/dotflow/cli/commands/deploy.py +171 -0
- {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev4}/dotflow/cli/commands/init.py +1 -3
- {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev4}/dotflow/cli/commands/log.py +2 -1
- dotflow-0.15.0.dev4/dotflow/cli/commands/storage.py +55 -0
- {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev4}/dotflow/cli/setup.py +5 -2
- dotflow-0.15.0.dev4/dotflow/cloud/alibaba/__init__.py +11 -0
- dotflow-0.15.0.dev4/dotflow/cloud/alibaba/constants.py +22 -0
- dotflow-0.15.0.dev4/dotflow/cloud/alibaba/deployers/__init__.py +13 -0
- dotflow-0.15.0.dev4/dotflow/cloud/alibaba/deployers/fc.py +134 -0
- dotflow-0.15.0.dev4/dotflow/cloud/alibaba/deployers/fc_scheduled.py +78 -0
- dotflow-0.15.0.dev4/dotflow/cloud/alibaba/services/__init__.py +1 -0
- dotflow-0.15.0.dev4/dotflow/cloud/alibaba/services/cr.py +99 -0
- dotflow-0.15.0.dev4/dotflow/cloud/aws/__init__.py +19 -0
- dotflow-0.15.0.dev4/dotflow/cloud/aws/constants.py +23 -0
- dotflow-0.15.0.dev4/dotflow/cloud/aws/deployers/__init__.py +19 -0
- dotflow-0.15.0.dev2/dotflow/cloud/aws/base_lambda_deployer.py → dotflow-0.15.0.dev4/dotflow/cloud/aws/deployers/base_lambda.py +6 -9
- dotflow-0.15.0.dev2/dotflow/cloud/aws/ecs_deployer.py → dotflow-0.15.0.dev4/dotflow/cloud/aws/deployers/ecs.py +11 -3
- dotflow-0.15.0.dev4/dotflow/cloud/aws/deployers/ecs_scheduled.py +154 -0
- dotflow-0.15.0.dev2/dotflow/cloud/aws/lambda_deployer.py → dotflow-0.15.0.dev4/dotflow/cloud/aws/deployers/lambda_.py +9 -3
- dotflow-0.15.0.dev2/dotflow/cloud/aws/lambda_api_deployer.py → dotflow-0.15.0.dev4/dotflow/cloud/aws/deployers/lambda_api.py +6 -3
- dotflow-0.15.0.dev2/dotflow/cloud/aws/lambda_s3_deployer.py → dotflow-0.15.0.dev4/dotflow/cloud/aws/deployers/lambda_s3.py +10 -4
- dotflow-0.15.0.dev2/dotflow/cloud/aws/lambda_sqs_deployer.py → dotflow-0.15.0.dev4/dotflow/cloud/aws/deployers/lambda_sqs.py +19 -4
- dotflow-0.15.0.dev4/dotflow/cloud/aws/schedule.py +66 -0
- {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev4}/dotflow/cloud/aws/services/cloudwatch.py +4 -1
- {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev4}/dotflow/cloud/aws/services/ecr.py +8 -5
- {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev4}/dotflow/cloud/aws/services/iam.py +4 -1
- {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev4}/dotflow/cloud/core.py +8 -2
- dotflow-0.15.0.dev4/dotflow/cloud/gcp/__init__.py +5 -0
- {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev4}/dotflow/cloud/gcp/constants.py +9 -0
- dotflow-0.15.0.dev4/dotflow/cloud/gcp/deployers/__init__.py +5 -0
- dotflow-0.15.0.dev2/dotflow/cloud/gcp/cloudrun_deployer.py → dotflow-0.15.0.dev4/dotflow/cloud/gcp/deployers/cloudrun.py +12 -10
- {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev4}/dotflow/cloud/gcp/services/apis.py +11 -3
- {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev4}/dotflow/cloud/gcp/services/artifact_registry.py +16 -5
- dotflow-0.15.0.dev4/dotflow/cloud/github/__init__.py +5 -0
- dotflow-0.15.0.dev4/dotflow/cloud/github/deployers/__init__.py +5 -0
- dotflow-0.15.0.dev2/dotflow/cloud/github/actions_deployer.py → dotflow-0.15.0.dev4/dotflow/cloud/github/deployers/actions.py +24 -26
- {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev4}/dotflow/core/action.py +40 -79
- {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev4}/dotflow/core/config.py +5 -5
- {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev4}/dotflow/core/context.py +17 -4
- {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev4}/dotflow/core/dotflow.py +1 -1
- dotflow-0.15.0.dev4/dotflow/core/engine.py +280 -0
- dotflow-0.15.0.dev4/dotflow/core/module.py +30 -0
- {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev4}/dotflow/core/serializers/task.py +51 -35
- {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev4}/dotflow/core/serializers/transport.py +2 -2
- dotflow-0.15.0.dev4/dotflow/core/serializers/workflow.py +14 -0
- {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev4}/dotflow/core/task.py +10 -2
- {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev4}/dotflow/core/types/workflow.py +1 -0
- {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev4}/dotflow/core/workflow.py +69 -67
- {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev4}/dotflow/providers/__init__.py +2 -0
- dotflow-0.15.0.dev4/dotflow/providers/server_default.py +112 -0
- dotflow-0.15.0.dev4/dotflow/providers/storage_default.py +22 -0
- {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev4}/dotflow/providers/storage_file.py +11 -8
- {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev4}/dotflow/settings.py +3 -1
- dotflow-0.15.0.dev4/dotflow/utils/error_handler.py +13 -0
- {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev4}/dotflow/utils/tools.py +0 -10
- {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev4}/pyproject.toml +4 -3
- dotflow-0.15.0.dev2/dotflow/abc/api.py +0 -24
- dotflow-0.15.0.dev2/dotflow/cli/commands/deploy.py +0 -110
- dotflow-0.15.0.dev2/dotflow/cloud/aws/__init__.py +0 -15
- dotflow-0.15.0.dev2/dotflow/cloud/aws/constants.py +0 -9
- dotflow-0.15.0.dev2/dotflow/cloud/gcp/__init__.py +0 -5
- dotflow-0.15.0.dev2/dotflow/cloud/github/__init__.py +0 -5
- dotflow-0.15.0.dev2/dotflow/core/execution.py +0 -176
- dotflow-0.15.0.dev2/dotflow/core/module.py +0 -36
- dotflow-0.15.0.dev2/dotflow/core/serializers/workflow.py +0 -23
- dotflow-0.15.0.dev2/dotflow/providers/api_default.py +0 -158
- dotflow-0.15.0.dev2/dotflow/providers/storage_default.py +0 -20
- dotflow-0.15.0.dev2/dotflow/storage.py +0 -22
- dotflow-0.15.0.dev2/dotflow/utils/error_handler.py +0 -22
- {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev4}/LICENSE +0 -0
- {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev4}/dotflow/abc/__init__.py +0 -0
- {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev4}/dotflow/abc/file.py +0 -0
- {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev4}/dotflow/abc/http.py +0 -0
- {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev4}/dotflow/abc/log.py +0 -0
- {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev4}/dotflow/abc/metrics.py +0 -0
- {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev4}/dotflow/abc/notify.py +0 -0
- {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev4}/dotflow/abc/scheduler.py +0 -0
- {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev4}/dotflow/abc/storage.py +0 -0
- {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev4}/dotflow/abc/tcp.py +0 -0
- {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev4}/dotflow/abc/tracer.py +0 -0
- {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev4}/dotflow/cli/__init__.py +0 -0
- {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev4}/dotflow/cli/command.py +0 -0
- {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev4}/dotflow/cli/commands/__init__.py +0 -0
- {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev4}/dotflow/cli/commands/schedule.py +0 -0
- {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev4}/dotflow/cli/commands/start.py +0 -0
- {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev4}/dotflow/cli/validators/__init__.py +0 -0
- {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev4}/dotflow/cli/validators/start.py +0 -0
- {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev4}/dotflow/cloud/__init__.py +0 -0
- {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev4}/dotflow/cloud/aws/services/__init__.py +0 -0
- {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev4}/dotflow/cloud/aws/services/s3.py +0 -0
- {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev4}/dotflow/cloud/gcp/services/__init__.py +0 -0
- {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev4}/dotflow/cloud/gcp/services/gcs.py +0 -0
- {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev4}/dotflow/cloud/github/constants.py +0 -0
- {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev4}/dotflow/core/__init__.py +0 -0
- {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev4}/dotflow/core/decorators/__init__.py +0 -0
- {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev4}/dotflow/core/decorators/time.py +0 -0
- {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev4}/dotflow/core/exception.py +0 -0
- {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev4}/dotflow/core/serializers/__init__.py +0 -0
- {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev4}/dotflow/core/types/__init__.py +0 -0
- {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev4}/dotflow/core/types/execution.py +0 -0
- {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev4}/dotflow/core/types/overlap.py +0 -0
- {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev4}/dotflow/core/types/status.py +0 -0
- {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev4}/dotflow/core/types/storage.py +0 -0
- {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev4}/dotflow/logging.py +0 -0
- {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev4}/dotflow/main.py +0 -0
- {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev4}/dotflow/providers/log_default.py +0 -0
- {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev4}/dotflow/providers/log_opentelemetry.py +0 -0
- {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev4}/dotflow/providers/log_sentry.py +0 -0
- {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev4}/dotflow/providers/metrics_default.py +0 -0
- {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev4}/dotflow/providers/metrics_opentelemetry.py +0 -0
- {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev4}/dotflow/providers/notify_default.py +0 -0
- {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev4}/dotflow/providers/notify_discord.py +0 -0
- {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev4}/dotflow/providers/notify_telegram.py +0 -0
- {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev4}/dotflow/providers/scheduler_cron.py +0 -0
- {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev4}/dotflow/providers/scheduler_default.py +0 -0
- {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev4}/dotflow/providers/storage_gcs.py +0 -0
- {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev4}/dotflow/providers/storage_s3.py +0 -0
- {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev4}/dotflow/providers/tracer_default.py +0 -0
- {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev4}/dotflow/providers/tracer_opentelemetry.py +0 -0
- {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev4}/dotflow/providers/tracer_sentry.py +0 -0
- {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev4}/dotflow/types.py +0 -0
- {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev4}/dotflow/utils/__init__.py +0 -0
- {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev4}/dotflow/utils/basic_functions.py +0 -0
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
Metadata-Version: 2.4
|
|
2
2
|
Name: dotflow
|
|
3
|
-
Version: 0.15.0.
|
|
3
|
+
Version: 0.15.0.dev4
|
|
4
4
|
Summary: 🎲 Dotflow turns an idea into flow! Lightweight Python library for execution pipelines with retry, parallel, cron and async support.
|
|
5
5
|
License: MIT License
|
|
6
6
|
|
|
@@ -42,6 +42,7 @@ Classifier: Programming Language :: Python :: 3.11
|
|
|
42
42
|
Classifier: Programming Language :: Python :: 3.12
|
|
43
43
|
Classifier: Programming Language :: Python :: 3.13
|
|
44
44
|
Provides-Extra: aws
|
|
45
|
+
Provides-Extra: deploy-alibaba
|
|
45
46
|
Provides-Extra: deploy-aws
|
|
46
47
|
Provides-Extra: deploy-gcp
|
|
47
48
|
Provides-Extra: deploy-github
|
|
@@ -49,7 +50,10 @@ Provides-Extra: gcp
|
|
|
49
50
|
Provides-Extra: otel
|
|
50
51
|
Provides-Extra: scheduler
|
|
51
52
|
Provides-Extra: sentry
|
|
53
|
+
Requires-Dist: GitPython ; extra == "deploy-github"
|
|
52
54
|
Requires-Dist: PyGithub ; extra == "deploy-github"
|
|
55
|
+
Requires-Dist: alibabacloud-fc-open20210406 ; extra == "deploy-alibaba"
|
|
56
|
+
Requires-Dist: alibabacloud-tea-openapi ; extra == "deploy-alibaba"
|
|
53
57
|
Requires-Dist: boto3 ; extra == "aws"
|
|
54
58
|
Requires-Dist: boto3 ; extra == "deploy-aws"
|
|
55
59
|
Requires-Dist: cookiecutter (>=2.0)
|
|
@@ -76,86 +80,40 @@ Project-URL: Repository, https://github.com/dotflow-io/dotflow
|
|
|
76
80
|
Description-Content-Type: text/markdown
|
|
77
81
|
|
|
78
82
|
<div align="center">
|
|
79
|
-
<a aria-label="Serverless.com" href="https://dotflow.io">Website</a>
|
|
80
|
-
•
|
|
81
|
-
<a aria-label="Dotflow Documentation" href="https://dotflow-io.github.io/dotflow/">Documentation</a>
|
|
82
|
-
•
|
|
83
|
-
<a aria-label="Pypi" href="https://pypi.org/project/dotflow/">Pypi</a>
|
|
84
|
-
</div>
|
|
85
83
|
|
|
86
|
-
|
|
84
|
+

|
|
87
85
|
|
|
88
|
-
|
|
86
|
+
**import dotflow. @action. deploy. Done.**
|
|
89
87
|
|
|
90
|
-
](https://pypi.org/project/dotflow/)
|
|
89
|
+
[](https://pypi.org/project/dotflow/)
|
|
90
|
+
[](https://pypi.org/project/dotflow/)
|
|
91
|
+
[](https://github.com/dotflow-io/dotflow)
|
|
91
92
|
|
|
92
|
-
|
|
93
|
-

|
|
94
|
-

|
|
95
|
-

|
|
96
|
-

|
|
93
|
+
[Website](https://dotflow.io) · [Documentation](https://dotflow-io.github.io/dotflow/) · [PyPI](https://pypi.org/project/dotflow/)
|
|
97
94
|
|
|
98
95
|
</div>
|
|
99
96
|
|
|
100
|
-
|
|
101
|
-
|
|
102
|
-
Dotflow is a lightweight Python library for building execution pipelines. Define tasks with decorators, chain them together, and run workflows in sequential, parallel, or background mode — with built-in retry, timeout, storage, notifications, and more.
|
|
103
|
-
|
|
104
|
-
> **[Read the full documentation](https://dotflow-io.github.io/dotflow/)**
|
|
105
|
-
|
|
106
|
-
## Table of Contents
|
|
97
|
+
---
|
|
107
98
|
|
|
108
|
-
|
|
109
|
-
<summary>Click to expand</summary>
|
|
110
|
-
|
|
111
|
-
- [Getting Help](#getting-help)
|
|
112
|
-
- [Installation](#installation)
|
|
113
|
-
- [Quick Start](#quick-start)
|
|
114
|
-
- [Features](#features)
|
|
115
|
-
- [Execution Modes](#execution-modes)
|
|
116
|
-
- [Retry, Timeout & Backoff](#retry-timeout--backoff)
|
|
117
|
-
- [Context System](#context-system)
|
|
118
|
-
- [Checkpoint & Resume](#checkpoint--resume)
|
|
119
|
-
- [Storage Providers](#storage-providers)
|
|
120
|
-
- [Notifications](#notifications)
|
|
121
|
-
- [Class-Based Steps](#class-based-steps)
|
|
122
|
-
- [Task Groups](#task-groups)
|
|
123
|
-
- [Callbacks](#callbacks)
|
|
124
|
-
- [Error Handling](#error-handling)
|
|
125
|
-
- [Async Support](#async-support)
|
|
126
|
-
- [Scheduler / Cron](#scheduler--cron)
|
|
127
|
-
- [CLI](#cli)
|
|
128
|
-
- [Dependency Injection via Config](#dependency-injection-via-config)
|
|
129
|
-
- [More Examples](#more-examples)
|
|
130
|
-
- [Commit Style](#commit-style)
|
|
131
|
-
- [License](#license)
|
|
99
|
+
# Dotflow
|
|
132
100
|
|
|
133
|
-
|
|
101
|
+
Dotflow is a lightweight Python library for execution pipelines. Define tasks with decorators, chain them together, and deploy to any cloud — with built-in retry, parallel execution, storage, observability, and cloud deployment.
|
|
134
102
|
|
|
135
|
-
##
|
|
103
|
+
## Why Dotflow?
|
|
136
104
|
|
|
137
|
-
|
|
105
|
+
- **Simple** — `@action` decorator + `workflow.start()`. That's it.
|
|
106
|
+
- **Resilient** — Retry, backoff, timeout, checkpoints, and error handling out of the box.
|
|
107
|
+
- **Observable** — OpenTelemetry traces, metrics, and logs. Sentry error tracking.
|
|
108
|
+
- **Deployable** — `dotflow deploy --platform lambda` ships your pipeline to AWS in one command.
|
|
109
|
+
- **Portable** — Same code runs on Lambda, ECS, Cloud Run, Alibaba FC, Kubernetes, Docker, or GitHub Actions.
|
|
138
110
|
|
|
139
|
-
|
|
140
|
-
- [Documentation](https://github.com/dotflow-io/dotflow/issues/new/choose)
|
|
141
|
-
- [Feature Request](https://github.com/dotflow-io/dotflow/issues/new/choose)
|
|
142
|
-
- [Security Issue](https://github.com/dotflow-io/dotflow/issues/new/choose)
|
|
143
|
-
- [General Question](https://github.com/dotflow-io/dotflow/issues/new/choose)
|
|
144
|
-
|
|
145
|
-
## Installation
|
|
111
|
+
## Install
|
|
146
112
|
|
|
147
113
|
```bash
|
|
148
114
|
pip install dotflow
|
|
149
115
|
```
|
|
150
116
|
|
|
151
|
-
**Optional extras:**
|
|
152
|
-
|
|
153
|
-
```bash
|
|
154
|
-
pip install dotflow[aws] # AWS S3 storage
|
|
155
|
-
pip install dotflow[gcp] # Google Cloud Storage
|
|
156
|
-
pip install dotflow[scheduler] # Cron-based scheduler
|
|
157
|
-
```
|
|
158
|
-
|
|
159
117
|
## Quick Start
|
|
160
118
|
|
|
161
119
|
```python
|
|
@@ -182,9 +140,99 @@ workflow.task.add(step=load)
|
|
|
182
140
|
workflow.start()
|
|
183
141
|
```
|
|
184
142
|
|
|
143
|
+
## Deploy anywhere
|
|
144
|
+
|
|
145
|
+
Write your pipeline once. Deploy to any cloud with a single command.
|
|
146
|
+
|
|
147
|
+
```bash
|
|
148
|
+
dotflow init
|
|
149
|
+
dotflow deploy --platform lambda --project my_pipeline
|
|
150
|
+
```
|
|
151
|
+
|
|
152
|
+
### Supported platforms
|
|
153
|
+
|
|
154
|
+
| Platform | Deploy method |
|
|
155
|
+
|----------|---------------|
|
|
156
|
+
| <img src="https://raw.githubusercontent.com/dotflow-io/dotflow/develop/docs/assets/icon-docker.svg" width="16" /> **Docker** | `docker compose up` |
|
|
157
|
+
| <img src="https://raw.githubusercontent.com/dotflow-io/dotflow/develop/docs/assets/icon-aws-lambda.svg" width="16" /> **AWS Lambda** | `dotflow deploy` |
|
|
158
|
+
| <img src="https://raw.githubusercontent.com/dotflow-io/dotflow/develop/docs/assets/icon-aws-lambda.svg" width="16" /> <img src="https://raw.githubusercontent.com/dotflow-io/dotflow/develop/docs/assets/icon-aws-eventbridge.svg" width="16" /> **AWS Lambda + EventBridge** | `dotflow deploy --schedule` |
|
|
159
|
+
| <img src="https://raw.githubusercontent.com/dotflow-io/dotflow/develop/docs/assets/icon-aws-lambda.svg" width="16" /> <img src="https://raw.githubusercontent.com/dotflow-io/dotflow/develop/docs/assets/icon-aws-s3.svg" width="16" /> **AWS Lambda + S3 Trigger** | `dotflow deploy` |
|
|
160
|
+
| <img src="https://raw.githubusercontent.com/dotflow-io/dotflow/develop/docs/assets/icon-aws-lambda.svg" width="16" /> <img src="https://raw.githubusercontent.com/dotflow-io/dotflow/develop/docs/assets/icon-aws-sqs.svg" width="16" /> **AWS Lambda + SQS Trigger** | `dotflow deploy` |
|
|
161
|
+
| <img src="https://raw.githubusercontent.com/dotflow-io/dotflow/develop/docs/assets/icon-aws-lambda.svg" width="16" /> <img src="https://raw.githubusercontent.com/dotflow-io/dotflow/develop/docs/assets/icon-aws-api-gateway.svg" width="16" /> **AWS Lambda + API Gateway** | `dotflow deploy` |
|
|
162
|
+
| <img src="https://raw.githubusercontent.com/dotflow-io/dotflow/develop/docs/assets/icon-aws-ecs.svg" width="16" /> **AWS ECS Fargate** | `dotflow deploy` |
|
|
163
|
+
| <img src="https://raw.githubusercontent.com/dotflow-io/dotflow/develop/docs/assets/icon-aws-ecs.svg" width="16" /> <img src="https://raw.githubusercontent.com/dotflow-io/dotflow/develop/docs/assets/icon-aws-eventbridge.svg" width="16" /> **AWS ECS + EventBridge** | `dotflow deploy --schedule` |
|
|
164
|
+
| <img src="https://raw.githubusercontent.com/dotflow-io/dotflow/develop/docs/assets/icon-google-cloud.svg" width="16" /> **Google Cloud Run** | `dotflow deploy` |
|
|
165
|
+
| <img src="https://raw.githubusercontent.com/dotflow-io/dotflow/develop/docs/assets/icon-google-cloud.svg" width="16" /> <img src="https://raw.githubusercontent.com/dotflow-io/dotflow/develop/docs/assets/icon-google-cloud-scheduler.svg" width="16" /> **Cloud Run + Scheduler** | `dotflow deploy --schedule` |
|
|
166
|
+
| <img src="https://raw.githubusercontent.com/dotflow-io/dotflow/develop/docs/assets/icon-kubernetes.svg" width="16" /> **Kubernetes** | `kubectl apply` |
|
|
167
|
+
| <img src="https://raw.githubusercontent.com/dotflow-io/dotflow/develop/docs/assets/icon-alibaba-cloud.svg" width="16" /> **Alibaba Cloud FC** | `dotflow deploy` |
|
|
168
|
+
| <img src="https://raw.githubusercontent.com/dotflow-io/dotflow/develop/docs/assets/icon-alibaba-cloud.svg" width="16" /> **Alibaba Cloud FC + Timer** | `dotflow deploy --schedule` |
|
|
169
|
+
| <img src="https://raw.githubusercontent.com/dotflow-io/dotflow/develop/docs/assets/icon-github-actions.svg" width="16" /> **GitHub Actions** | `dotflow deploy` |
|
|
170
|
+
|
|
171
|
+
> [See all 34+ platforms →](https://dotflow-io.github.io/dotflow/nav/cloud/)
|
|
172
|
+
|
|
173
|
+
## Optional extras
|
|
174
|
+
|
|
175
|
+
```bash
|
|
176
|
+
pip install dotflow[aws] # S3 storage
|
|
177
|
+
pip install dotflow[gcp] # Google Cloud Storage
|
|
178
|
+
pip install dotflow[scheduler] # Cron scheduler
|
|
179
|
+
pip install dotflow[otel] # OpenTelemetry
|
|
180
|
+
pip install dotflow[sentry] # Sentry error tracking
|
|
181
|
+
pip install dotflow[deploy-aws] # AWS deploy (Lambda, ECS)
|
|
182
|
+
pip install dotflow[deploy-gcp] # GCP deploy (Cloud Run)
|
|
183
|
+
pip install dotflow[deploy-alibaba] # Alibaba Cloud deploy (FC)
|
|
184
|
+
pip install dotflow[deploy-github] # GitHub Actions deploy
|
|
185
|
+
```
|
|
186
|
+
|
|
187
|
+
> **[Read the full documentation →](https://dotflow-io.github.io/dotflow/)**
|
|
188
|
+
|
|
189
|
+
## Documentation
|
|
190
|
+
|
|
191
|
+
| Section | Description |
|
|
192
|
+
|---------|-------------|
|
|
193
|
+
| [Concepts](https://dotflow-io.github.io/dotflow/nav/concepts/concept-workflow-and-tasks/) | Workflows, tasks, context, providers, process modes |
|
|
194
|
+
| [How-to Guides](https://dotflow-io.github.io/dotflow/nav/how-to/) | Step-by-step tutorials for workflows, tasks, and CLI |
|
|
195
|
+
| [Cloud Deployment](https://dotflow-io.github.io/dotflow/nav/cloud/) | Deploy to AWS, GCP, Alibaba, Kubernetes, Docker, GitHub Actions |
|
|
196
|
+
| [Integrations](https://dotflow-io.github.io/dotflow/nav/integrations/) | OpenTelemetry, Sentry, Telegram, Discord, S3, GCS, Server |
|
|
197
|
+
| [Examples](https://dotflow-io.github.io/dotflow/nav/examples/) | Real-world pipelines: ETL, health checks, async, scheduler |
|
|
198
|
+
| [Reference](https://dotflow-io.github.io/dotflow/nav/reference/dotflow/) | API reference for all classes and providers |
|
|
199
|
+
| [Custom Providers](https://dotflow-io.github.io/dotflow/nav/development/custom-providers/) | Build your own storage, notify, log, tracer, or metrics provider |
|
|
200
|
+
|
|
185
201
|
## Features
|
|
186
202
|
|
|
187
|
-
|
|
203
|
+
<details>
|
|
204
|
+
<summary><strong>Observability</strong></summary>
|
|
205
|
+
|
|
206
|
+
> [OpenTelemetry docs](https://dotflow-io.github.io/dotflow/nav/tutorial/log-opentelemetry/) | [Sentry docs](https://dotflow-io.github.io/dotflow/nav/tutorial/log-sentry/) | [Tracer docs](https://dotflow-io.github.io/dotflow/nav/tutorial/tracer-opentelemetry/) | [Metrics docs](https://dotflow-io.github.io/dotflow/nav/tutorial/metrics-opentelemetry/)
|
|
207
|
+
|
|
208
|
+
Built-in support for OpenTelemetry and Sentry:
|
|
209
|
+
|
|
210
|
+
```python
|
|
211
|
+
from dotflow import Config
|
|
212
|
+
from dotflow.providers import LogOpenTelemetry, TracerOpenTelemetry, MetricsOpenTelemetry
|
|
213
|
+
|
|
214
|
+
# OpenTelemetry: traces, metrics, and structured logs
|
|
215
|
+
config = Config(
|
|
216
|
+
log=LogOpenTelemetry(service_name="my-pipeline"),
|
|
217
|
+
tracer=TracerOpenTelemetry(service_name="my-pipeline"),
|
|
218
|
+
metrics=MetricsOpenTelemetry(service_name="my-pipeline"),
|
|
219
|
+
)
|
|
220
|
+
```
|
|
221
|
+
|
|
222
|
+
```python
|
|
223
|
+
from dotflow.providers import LogSentry, TracerSentry
|
|
224
|
+
|
|
225
|
+
# Sentry: error tracking + performance monitoring
|
|
226
|
+
config = Config(
|
|
227
|
+
log=LogSentry(dsn="https://xxx@sentry.io/123"),
|
|
228
|
+
tracer=TracerSentry(),
|
|
229
|
+
)
|
|
230
|
+
```
|
|
231
|
+
|
|
232
|
+
</details>
|
|
233
|
+
|
|
234
|
+
<details>
|
|
235
|
+
<summary><strong>Execution Modes</strong></summary>
|
|
188
236
|
|
|
189
237
|
> [Process Mode docs](https://dotflow-io.github.io/dotflow/nav/concepts/process-mode-sequential/)
|
|
190
238
|
|
|
@@ -255,7 +303,10 @@ flowchart TD
|
|
|
255
303
|
|
|
256
304
|
---
|
|
257
305
|
|
|
258
|
-
|
|
306
|
+
</details>
|
|
307
|
+
|
|
308
|
+
<details>
|
|
309
|
+
<summary><strong>Retry, Timeout & Backoff</strong></summary>
|
|
259
310
|
|
|
260
311
|
> [Retry docs](https://dotflow-io.github.io/dotflow/nav/tutorial/task-retry/) | [Backoff docs](https://dotflow-io.github.io/dotflow/nav/tutorial/task-backoff/) | [Timeout docs](https://dotflow-io.github.io/dotflow/nav/tutorial/task-timeout/)
|
|
261
312
|
|
|
@@ -278,7 +329,10 @@ def unreliable_api_call():
|
|
|
278
329
|
|
|
279
330
|
---
|
|
280
331
|
|
|
281
|
-
|
|
332
|
+
</details>
|
|
333
|
+
|
|
334
|
+
<details>
|
|
335
|
+
<summary><strong>Context System</strong></summary>
|
|
282
336
|
|
|
283
337
|
> [Context docs](https://dotflow-io.github.io/dotflow/nav/tutorial/initial-context/) | [Previous Context](https://dotflow-io.github.io/dotflow/nav/tutorial/previous-context/) | [Many Contexts](https://dotflow-io.github.io/dotflow/nav/tutorial/many-contexts/)
|
|
284
338
|
|
|
@@ -309,7 +363,10 @@ Each `Context` object contains:
|
|
|
309
363
|
|
|
310
364
|
---
|
|
311
365
|
|
|
312
|
-
|
|
366
|
+
</details>
|
|
367
|
+
|
|
368
|
+
<details>
|
|
369
|
+
<summary><strong>Checkpoint & Resume</strong></summary>
|
|
313
370
|
|
|
314
371
|
> [Checkpoint docs](https://dotflow-io.github.io/dotflow/nav/tutorial/checkpoint/)
|
|
315
372
|
|
|
@@ -335,7 +392,10 @@ workflow.start(resume=True)
|
|
|
335
392
|
|
|
336
393
|
---
|
|
337
394
|
|
|
338
|
-
|
|
395
|
+
</details>
|
|
396
|
+
|
|
397
|
+
<details>
|
|
398
|
+
<summary><strong>Storage Providers</strong></summary>
|
|
339
399
|
|
|
340
400
|
> [Storage docs](https://dotflow-io.github.io/dotflow/nav/tutorial/storage-default/)
|
|
341
401
|
|
|
@@ -389,32 +449,39 @@ workflow = DotFlow(config=config)
|
|
|
389
449
|
|
|
390
450
|
---
|
|
391
451
|
|
|
392
|
-
|
|
452
|
+
</details>
|
|
453
|
+
|
|
454
|
+
<details>
|
|
455
|
+
<summary><strong>Notifications</strong></summary>
|
|
393
456
|
|
|
394
|
-
> [Telegram docs](https://dotflow-io.github.io/dotflow/nav/tutorial/notify-telegram/)
|
|
457
|
+
> [Telegram docs](https://dotflow-io.github.io/dotflow/nav/tutorial/notify-telegram/) | [Discord docs](https://dotflow-io.github.io/dotflow/nav/tutorial/notify-discord/)
|
|
395
458
|
|
|
396
|
-
Get notified about task status changes via Telegram.
|
|
459
|
+
Get notified about task status changes via Telegram or Discord.
|
|
397
460
|
|
|
398
461
|
```python
|
|
399
|
-
from dotflow import
|
|
462
|
+
from dotflow import Config
|
|
400
463
|
from dotflow.providers import NotifyTelegram
|
|
401
|
-
from dotflow.core.types.status import TypeStatus
|
|
402
464
|
|
|
403
|
-
|
|
465
|
+
config = Config(notify=NotifyTelegram(
|
|
404
466
|
token="YOUR_BOT_TOKEN",
|
|
405
467
|
chat_id=123456789,
|
|
406
|
-
|
|
407
|
-
)
|
|
408
|
-
|
|
409
|
-
config = Config(notify=notify)
|
|
410
|
-
workflow = DotFlow(config=config)
|
|
468
|
+
))
|
|
411
469
|
```
|
|
412
470
|
|
|
413
|
-
|
|
471
|
+
```python
|
|
472
|
+
from dotflow.providers import NotifyDiscord
|
|
473
|
+
|
|
474
|
+
config = Config(notify=NotifyDiscord(
|
|
475
|
+
webhook_url="https://discord.com/api/webhooks/...",
|
|
476
|
+
))
|
|
477
|
+
```
|
|
414
478
|
|
|
415
479
|
---
|
|
416
480
|
|
|
417
|
-
|
|
481
|
+
</details>
|
|
482
|
+
|
|
483
|
+
<details>
|
|
484
|
+
<summary><strong>Class-Based Steps</strong></summary>
|
|
418
485
|
|
|
419
486
|
Return a class instance from a task, and Dotflow will automatically discover and execute all `@action`-decorated methods in source order.
|
|
420
487
|
|
|
@@ -446,7 +513,10 @@ workflow.start()
|
|
|
446
513
|
|
|
447
514
|
---
|
|
448
515
|
|
|
449
|
-
|
|
516
|
+
</details>
|
|
517
|
+
|
|
518
|
+
<details>
|
|
519
|
+
<summary><strong>Task Groups</strong></summary>
|
|
450
520
|
|
|
451
521
|
> [Groups docs](https://dotflow-io.github.io/dotflow/nav/tutorial/groups/)
|
|
452
522
|
|
|
@@ -463,7 +533,10 @@ workflow.start() # groups run in parallel, tasks within each group run sequenti
|
|
|
463
533
|
|
|
464
534
|
---
|
|
465
535
|
|
|
466
|
-
|
|
536
|
+
</details>
|
|
537
|
+
|
|
538
|
+
<details>
|
|
539
|
+
<summary><strong>Callbacks</strong></summary>
|
|
467
540
|
|
|
468
541
|
> [Task Callback docs](https://dotflow-io.github.io/dotflow/nav/tutorial/task-callback/) | [Workflow Callback docs](https://dotflow-io.github.io/dotflow/nav/tutorial/workflow-callback/)
|
|
469
542
|
|
|
@@ -490,7 +563,10 @@ workflow.start(on_success=on_success, on_failure=on_failure)
|
|
|
490
563
|
|
|
491
564
|
---
|
|
492
565
|
|
|
493
|
-
|
|
566
|
+
</details>
|
|
567
|
+
|
|
568
|
+
<details>
|
|
569
|
+
<summary><strong>Error Handling</strong></summary>
|
|
494
570
|
|
|
495
571
|
> [Error Handling docs](https://dotflow-io.github.io/dotflow/nav/tutorial/error-handling/) | [Keep Going docs](https://dotflow-io.github.io/dotflow/nav/tutorial/keep-going/)
|
|
496
572
|
|
|
@@ -520,7 +596,10 @@ for task in workflow.result_task():
|
|
|
520
596
|
|
|
521
597
|
---
|
|
522
598
|
|
|
523
|
-
|
|
599
|
+
</details>
|
|
600
|
+
|
|
601
|
+
<details>
|
|
602
|
+
<summary><strong>Async Support</strong></summary>
|
|
524
603
|
|
|
525
604
|
> [Async docs](https://dotflow-io.github.io/dotflow/nav/tutorial/async-actions/)
|
|
526
605
|
|
|
@@ -543,7 +622,10 @@ workflow.start()
|
|
|
543
622
|
|
|
544
623
|
---
|
|
545
624
|
|
|
546
|
-
|
|
625
|
+
</details>
|
|
626
|
+
|
|
627
|
+
<details>
|
|
628
|
+
<summary><strong>Scheduler / Cron</strong></summary>
|
|
547
629
|
|
|
548
630
|
> [Cron scheduler docs](https://dotflow-io.github.io/dotflow/nav/tutorial/scheduler-cron/) | [Default scheduler](https://dotflow-io.github.io/dotflow/nav/tutorial/scheduler-default/) | [Cron overlap (concepts)](https://dotflow-io.github.io/dotflow/nav/concepts/concept-cron-overlap/)
|
|
549
631
|
|
|
@@ -592,7 +674,10 @@ The scheduler handles graceful shutdown via `SIGINT`/`SIGTERM` signals automatic
|
|
|
592
674
|
|
|
593
675
|
---
|
|
594
676
|
|
|
595
|
-
|
|
677
|
+
</details>
|
|
678
|
+
|
|
679
|
+
<details>
|
|
680
|
+
<summary><strong>CLI</strong></summary>
|
|
596
681
|
|
|
597
682
|
> [CLI docs](https://dotflow-io.github.io/dotflow/nav/how-to/cli/simple-start/)
|
|
598
683
|
|
|
@@ -634,26 +719,76 @@ Available CLI commands:
|
|
|
634
719
|
|
|
635
720
|
| Command | Description |
|
|
636
721
|
|---------|-------------|
|
|
637
|
-
| `dotflow init` |
|
|
722
|
+
| `dotflow init` | Scaffold a new project with cloud support |
|
|
638
723
|
| `dotflow start` | Run a workflow |
|
|
639
724
|
| `dotflow schedule` | Run a workflow on a cron schedule |
|
|
640
|
-
| `dotflow
|
|
725
|
+
| `dotflow logs` | View execution logs |
|
|
726
|
+
| `dotflow cloud list` | Show available cloud platforms |
|
|
727
|
+
| `dotflow cloud generate --platform <name>` | Generate deployment files |
|
|
728
|
+
| `dotflow deploy --platform <name> --project <name>` | Deploy to cloud |
|
|
729
|
+
|
|
730
|
+
---
|
|
731
|
+
|
|
732
|
+
</details>
|
|
733
|
+
|
|
734
|
+
<details>
|
|
735
|
+
<summary><strong>Server Provider</strong></summary>
|
|
736
|
+
|
|
737
|
+
> [Server docs](https://dotflow-io.github.io/dotflow/nav/tutorial/server-default/)
|
|
738
|
+
|
|
739
|
+
Send workflow and task execution data to a remote API (e.g. dotflow-api) in real time.
|
|
740
|
+
|
|
741
|
+
```python
|
|
742
|
+
from dotflow import DotFlow, Config, action
|
|
743
|
+
from dotflow.providers import ServerDefault
|
|
744
|
+
|
|
745
|
+
@action
|
|
746
|
+
def my_task():
|
|
747
|
+
return {"result": "ok"}
|
|
748
|
+
|
|
749
|
+
config = Config(
|
|
750
|
+
server=ServerDefault(
|
|
751
|
+
base_url="http://localhost:8000/api/v1",
|
|
752
|
+
user_token="your-api-token",
|
|
753
|
+
)
|
|
754
|
+
)
|
|
755
|
+
|
|
756
|
+
workflow = DotFlow(config=config)
|
|
757
|
+
workflow.task.add(step=my_task)
|
|
758
|
+
workflow.start()
|
|
759
|
+
```
|
|
760
|
+
|
|
761
|
+
| Parameter | Type | Default | Description |
|
|
762
|
+
|-----------|------|---------|-------------|
|
|
763
|
+
| `base_url` | `str` | `""` | API base URL |
|
|
764
|
+
| `user_token` | `str` | `""` | API token (`X-User-Token` header) |
|
|
765
|
+
| `timeout` | `float` | `5.0` | HTTP request timeout in seconds |
|
|
766
|
+
|
|
767
|
+
The server provider automatically:
|
|
768
|
+
- Creates the workflow on `DotFlow()` init
|
|
769
|
+
- Creates each task on `task.add()`
|
|
770
|
+
- Updates task status on each transition (In progress, Completed, Failed, Retry)
|
|
771
|
+
- Updates workflow status on completion (In progress → Completed)
|
|
641
772
|
|
|
642
773
|
---
|
|
643
774
|
|
|
644
|
-
|
|
775
|
+
</details>
|
|
776
|
+
|
|
777
|
+
<details>
|
|
778
|
+
<summary><strong>Dependency Injection via Config</strong></summary>
|
|
645
779
|
|
|
646
|
-
The `Config` class lets you swap providers for storage, notifications, logging, and
|
|
780
|
+
The `Config` class lets you swap providers for storage, notifications, logging, scheduling, and server:
|
|
647
781
|
|
|
648
782
|
```python
|
|
649
783
|
from dotflow import DotFlow, Config
|
|
650
|
-
from dotflow.providers import StorageFile, NotifyTelegram, LogDefault, SchedulerCron
|
|
784
|
+
from dotflow.providers import StorageFile, NotifyTelegram, LogDefault, SchedulerCron, ServerDefault
|
|
651
785
|
|
|
652
786
|
config = Config(
|
|
653
787
|
storage=StorageFile(path=".output"),
|
|
654
788
|
notify=NotifyTelegram(token="...", chat_id=123),
|
|
655
789
|
log=LogDefault(),
|
|
656
790
|
scheduler=SchedulerCron(cron="0 * * * *"),
|
|
791
|
+
server=ServerDefault(base_url="...", user_token="..."),
|
|
657
792
|
)
|
|
658
793
|
|
|
659
794
|
workflow = DotFlow(config=config)
|
|
@@ -665,12 +800,18 @@ Extend Dotflow by implementing the abstract base classes:
|
|
|
665
800
|
|-----|---------|---------|
|
|
666
801
|
| `Storage` | `post`, `get`, `key` | Custom storage backends |
|
|
667
802
|
| `Notify` | `hook_status_task` | Custom notification channels |
|
|
668
|
-
| `Log` | `info`, `error` | Custom logging |
|
|
803
|
+
| `Log` | `info`, `error`, `warning`, `debug` | Custom logging |
|
|
669
804
|
| `Scheduler` | `start`, `stop` | Custom scheduling strategies |
|
|
805
|
+
| `Tracer` | `start_workflow`, `end_workflow`, `start_task`, `end_task` | Distributed tracing |
|
|
806
|
+
| `Metrics` | `workflow_started`, `workflow_completed`, `workflow_failed`, `task_completed`, `task_failed`, `task_retried` | Counters and histograms |
|
|
807
|
+
| `Server` | `create_workflow`, `update_workflow`, `create_task`, `update_task` | Remote API communication |
|
|
670
808
|
|
|
671
809
|
---
|
|
672
810
|
|
|
673
|
-
|
|
811
|
+
</details>
|
|
812
|
+
|
|
813
|
+
<details>
|
|
814
|
+
<summary><strong>Results & Inspection</strong></summary>
|
|
674
815
|
|
|
675
816
|
After execution, inspect results directly from the workflow object:
|
|
676
817
|
|
|
@@ -701,7 +842,10 @@ workflow.task.schema() # Pydantic schema of the workflow
|
|
|
701
842
|
|
|
702
843
|
---
|
|
703
844
|
|
|
704
|
-
|
|
845
|
+
</details>
|
|
846
|
+
|
|
847
|
+
<details>
|
|
848
|
+
<summary><strong>Dynamic Module Import</strong></summary>
|
|
705
849
|
|
|
706
850
|
Reference tasks and callbacks by their module path string instead of importing them directly:
|
|
707
851
|
|
|
@@ -712,124 +856,12 @@ workflow.task.add(step="my_package.tasks.save_results", callback="my_package.cal
|
|
|
712
856
|
|
|
713
857
|
---
|
|
714
858
|
|
|
715
|
-
|
|
716
|
-
|
|
717
|
-
All examples are available in the [`docs_src/`](https://github.com/dotflow-io/dotflow/tree/develop/docs_src) directory.
|
|
718
|
-
|
|
719
|
-
#### Basic
|
|
720
|
-
|
|
721
|
-
| Example | Description | Command |
|
|
722
|
-
|---------|-------------|---------|
|
|
723
|
-
| [first_steps](https://github.com/dotflow-io/dotflow/blob/develop/docs_src/first_steps/first_steps.py) | Minimal workflow with callback | `python docs_src/first_steps/first_steps.py` |
|
|
724
|
-
| [simple_function_workflow](https://github.com/dotflow-io/dotflow/blob/develop/docs_src/basic/simple_function_workflow.py) | Simple function-based workflow | `python docs_src/basic/simple_function_workflow.py` |
|
|
725
|
-
| [simple_class_workflow](https://github.com/dotflow-io/dotflow/blob/develop/docs_src/basic/simple_class_workflow.py) | Class-based step with retry | `python docs_src/basic/simple_class_workflow.py` |
|
|
726
|
-
| [simple_function_workflow_with_error](https://github.com/dotflow-io/dotflow/blob/develop/docs_src/basic/simple_function_workflow_with_error.py) | Error inspection after failure | `python docs_src/basic/simple_function_workflow_with_error.py` |
|
|
727
|
-
|
|
728
|
-
#### Async
|
|
729
|
-
|
|
730
|
-
| Example | Description | Command |
|
|
731
|
-
|---------|-------------|---------|
|
|
732
|
-
| [async_action](https://github.com/dotflow-io/dotflow/blob/develop/docs_src/async/async_action.py) | Async task functions | `python docs_src/async/async_action.py` |
|
|
733
|
-
|
|
734
|
-
#### Context
|
|
735
|
-
|
|
736
|
-
| Example | Description | Command |
|
|
737
|
-
|---------|-------------|---------|
|
|
738
|
-
| [context](https://github.com/dotflow-io/dotflow/blob/develop/docs_src/context/context.py) | Creating and inspecting a Context | `python docs_src/context/context.py` |
|
|
739
|
-
| [initial_context](https://github.com/dotflow-io/dotflow/blob/develop/docs_src/initial_context/initial_context.py) | Passing initial context per task | `python docs_src/initial_context/initial_context.py` |
|
|
740
|
-
| [previous_context](https://github.com/dotflow-io/dotflow/blob/develop/docs_src/previous_context/previous_context.py) | Chaining context between tasks | `python docs_src/previous_context/previous_context.py` |
|
|
741
|
-
| [many_contexts](https://github.com/dotflow-io/dotflow/blob/develop/docs_src/context/many_contexts.py) | Using both initial and previous context | `python docs_src/context/many_contexts.py` |
|
|
742
|
-
|
|
743
|
-
#### Process Modes
|
|
744
|
-
|
|
745
|
-
| Example | Description | Command |
|
|
746
|
-
|---------|-------------|---------|
|
|
747
|
-
| [sequential](https://github.com/dotflow-io/dotflow/blob/develop/docs_src/process_mode/sequential.py) | Sequential execution | `python docs_src/process_mode/sequential.py` |
|
|
748
|
-
| [background](https://github.com/dotflow-io/dotflow/blob/develop/docs_src/process_mode/background.py) | Background (non-blocking) execution | `python docs_src/process_mode/background.py` |
|
|
749
|
-
| [parallel](https://github.com/dotflow-io/dotflow/blob/develop/docs_src/process_mode/parallel.py) | Parallel execution | `python docs_src/process_mode/parallel.py` |
|
|
750
|
-
| [parallel_group](https://github.com/dotflow-io/dotflow/blob/develop/docs_src/process_mode/parallel_group.py) | Parallel groups execution | `python docs_src/process_mode/parallel_group.py` |
|
|
751
|
-
| [sequential_group_mode](https://github.com/dotflow-io/dotflow/blob/develop/docs_src/workflow/sequential_group_mode.py) | Sequential with named groups | `python docs_src/workflow/sequential_group_mode.py` |
|
|
752
|
-
|
|
753
|
-
#### Resilience (Retry, Backoff, Timeout)
|
|
754
|
-
|
|
755
|
-
| Example | Description | Command |
|
|
756
|
-
|---------|-------------|---------|
|
|
757
|
-
| [retry](https://github.com/dotflow-io/dotflow/blob/develop/docs_src/retry/retry.py) | Retry on function and class steps | `python docs_src/retry/retry.py` |
|
|
758
|
-
| [retry_delay](https://github.com/dotflow-io/dotflow/blob/develop/docs_src/retry/retry_delay.py) | Retry with delay between attempts | `python docs_src/retry/retry_delay.py` |
|
|
759
|
-
| [backoff](https://github.com/dotflow-io/dotflow/blob/develop/docs_src/backoff/backoff.py) | Exponential backoff on retries | `python docs_src/backoff/backoff.py` |
|
|
760
|
-
| [timeout](https://github.com/dotflow-io/dotflow/blob/develop/docs_src/timeout/timeout.py) | Timeout per task execution | `python docs_src/timeout/timeout.py` |
|
|
761
|
-
|
|
762
|
-
#### Callbacks
|
|
763
|
-
|
|
764
|
-
| Example | Description | Command |
|
|
765
|
-
|---------|-------------|---------|
|
|
766
|
-
| [task_callback](https://github.com/dotflow-io/dotflow/blob/develop/docs_src/callback/task_callback.py) | Per-task callback on completion | `python docs_src/callback/task_callback.py` |
|
|
767
|
-
| [workflow_callback_success](https://github.com/dotflow-io/dotflow/blob/develop/docs_src/callback/workflow_callback_success.py) | Workflow-level success callback | `python docs_src/callback/workflow_callback_success.py` |
|
|
768
|
-
| [workflow_callback_failure](https://github.com/dotflow-io/dotflow/blob/develop/docs_src/callback/workflow_callback_failure.py) | Workflow-level failure callback | `python docs_src/callback/workflow_callback_failure.py` |
|
|
769
|
-
|
|
770
|
-
#### Error Handling
|
|
771
|
-
|
|
772
|
-
| Example | Description | Command |
|
|
773
|
-
|---------|-------------|---------|
|
|
774
|
-
| [errors](https://github.com/dotflow-io/dotflow/blob/develop/docs_src/errors/errors.py) | Inspecting task errors and retry count | `python docs_src/errors/errors.py` |
|
|
775
|
-
| [keep_going_true](https://github.com/dotflow-io/dotflow/blob/develop/docs_src/workflow/keep_going_true.py) | Continue workflow after task failure | `python docs_src/workflow/keep_going_true.py` |
|
|
776
|
-
|
|
777
|
-
#### Groups
|
|
778
|
-
|
|
779
|
-
| Example | Description | Command |
|
|
780
|
-
|---------|-------------|---------|
|
|
781
|
-
| [step_with_groups](https://github.com/dotflow-io/dotflow/blob/develop/docs_src/group/step_with_groups.py) | Tasks in named parallel groups | `python docs_src/group/step_with_groups.py` |
|
|
782
|
-
|
|
783
|
-
#### Storage
|
|
784
|
-
|
|
785
|
-
| Example | Description | Command |
|
|
786
|
-
|---------|-------------|---------|
|
|
787
|
-
| [storage_file](https://github.com/dotflow-io/dotflow/blob/develop/docs_src/storage/storage_file.py) | File-based JSON storage | `python docs_src/storage/storage_file.py` |
|
|
788
|
-
| [storage_s3](https://github.com/dotflow-io/dotflow/blob/develop/docs_src/storage/storage_s3.py) | AWS S3 storage | `python docs_src/storage/storage_s3.py` |
|
|
789
|
-
| [storage_gcs](https://github.com/dotflow-io/dotflow/blob/develop/docs_src/storage/storage_gcs.py) | Google Cloud Storage | `python docs_src/storage/storage_gcs.py` |
|
|
790
|
-
|
|
791
|
-
#### Checkpoint & Resume
|
|
792
|
-
|
|
793
|
-
| Example | Description | Command |
|
|
794
|
-
|---------|-------------|---------|
|
|
795
|
-
| [checkpoint](https://github.com/dotflow-io/dotflow/blob/develop/docs_src/checkpoint/checkpoint.py) | Resume workflow from last checkpoint | `python docs_src/checkpoint/checkpoint.py` |
|
|
796
|
-
|
|
797
|
-
#### Notifications
|
|
798
|
-
|
|
799
|
-
| Example | Description | Command |
|
|
800
|
-
|---------|-------------|---------|
|
|
801
|
-
| [notify_telegram](https://github.com/dotflow-io/dotflow/blob/develop/docs_src/notify/notify_telegram.py) | Telegram notifications on failure | `python docs_src/notify/notify_telegram.py` |
|
|
802
|
-
|
|
803
|
-
#### Config & Providers
|
|
859
|
+
</details>
|
|
804
860
|
|
|
805
|
-
| Example | Description | Command |
|
|
806
|
-
|---------|-------------|---------|
|
|
807
|
-
| [config](https://github.com/dotflow-io/dotflow/blob/develop/docs_src/config/config.py) | Full Config with storage, notify, log | `python docs_src/config/config.py` |
|
|
808
|
-
| [storage_provider](https://github.com/dotflow-io/dotflow/blob/develop/docs_src/config/storage_provider.py) | Swapping storage providers | `python docs_src/config/storage_provider.py` |
|
|
809
|
-
| [notify_provider](https://github.com/dotflow-io/dotflow/blob/develop/docs_src/config/notify_provider.py) | Swapping notification providers | `python docs_src/config/notify_provider.py` |
|
|
810
|
-
| [log_provider](https://github.com/dotflow-io/dotflow/blob/develop/docs_src/config/log_provider.py) | Custom log provider | `python docs_src/config/log_provider.py` |
|
|
811
861
|
|
|
812
|
-
|
|
862
|
+
## More Examples
|
|
813
863
|
|
|
814
|
-
|
|
815
|
-
|---------|-------------|---------|
|
|
816
|
-
| [step_function_result_task](https://github.com/dotflow-io/dotflow/blob/develop/docs_src/output/step_function_result_task.py) | Inspect task results (function) | `python docs_src/output/step_function_result_task.py` |
|
|
817
|
-
| [step_function_result_context](https://github.com/dotflow-io/dotflow/blob/develop/docs_src/output/step_function_result_context.py) | Inspect context results (function) | `python docs_src/output/step_function_result_context.py` |
|
|
818
|
-
| [step_function_result_storage](https://github.com/dotflow-io/dotflow/blob/develop/docs_src/output/step_function_result_storage.py) | Inspect storage results (function) | `python docs_src/output/step_function_result_storage.py` |
|
|
819
|
-
| [step_class_result_task](https://github.com/dotflow-io/dotflow/blob/develop/docs_src/output/step_class_result_task.py) | Inspect task results (class) | `python docs_src/output/step_class_result_task.py` |
|
|
820
|
-
| [step_class_result_context](https://github.com/dotflow-io/dotflow/blob/develop/docs_src/output/step_class_result_context.py) | Inspect context results (class) | `python docs_src/output/step_class_result_context.py` |
|
|
821
|
-
| [step_class_result_storage](https://github.com/dotflow-io/dotflow/blob/develop/docs_src/output/step_class_result_storage.py) | Inspect storage results (class) | `python docs_src/output/step_class_result_storage.py` |
|
|
822
|
-
|
|
823
|
-
#### CLI
|
|
824
|
-
|
|
825
|
-
| Example | Description | Command |
|
|
826
|
-
|---------|-------------|---------|
|
|
827
|
-
| [simple_cli](https://github.com/dotflow-io/dotflow/blob/develop/docs_src/basic/simple_cli.py) | Basic CLI execution | `dotflow start --step docs_src.basic.simple_cli.simple_step` |
|
|
828
|
-
| [cli_with_callback](https://github.com/dotflow-io/dotflow/blob/develop/docs_src/cli/cli_with_callback.py) | CLI with callback function | `dotflow start --step docs_src.cli.cli_with_callback.simple_step --callback docs_src.cli.cli_with_callback.callback` |
|
|
829
|
-
| [cli_with_initial_context](https://github.com/dotflow-io/dotflow/blob/develop/docs_src/cli/cli_with_initial_context.py) | CLI with initial context | `dotflow start --step docs_src.cli.cli_with_initial_context.simple_step --initial-context abc` |
|
|
830
|
-
| [cli_with_mode](https://github.com/dotflow-io/dotflow/blob/develop/docs_src/cli/cli_with_mode.py) | CLI with execution mode | `dotflow start --step docs_src.cli.cli_with_mode.simple_step --mode sequential` |
|
|
831
|
-
| [cli_with_output_context](https://github.com/dotflow-io/dotflow/blob/develop/docs_src/cli/cli_with_output_context.py) | CLI with file storage output | `dotflow start --step docs_src.cli.cli_with_output_context.simple_step --storage file` |
|
|
832
|
-
| [cli_with_path](https://github.com/dotflow-io/dotflow/blob/develop/docs_src/cli/cli_with_path.py) | CLI with custom storage path | `dotflow start --step docs_src.cli.cli_with_path.simple_step --path .storage --storage file` |
|
|
864
|
+
All examples are available in the [`docs_src/`](https://github.com/dotflow-io/dotflow/tree/develop/docs_src) directory.
|
|
833
865
|
|
|
834
866
|
## Commit Style
|
|
835
867
|
|