dotflow 0.15.0.dev2__tar.gz → 0.15.0.dev3__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (109) hide show
  1. {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev3}/PKG-INFO +185 -208
  2. {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev3}/README.md +183 -207
  3. {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev3}/dotflow/__init__.py +1 -1
  4. {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev3}/dotflow/abc/flow.py +19 -0
  5. {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev3}/dotflow/cli/commands/cloud.py +8 -1
  6. {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev3}/dotflow/cli/commands/init.py +1 -3
  7. dotflow-0.15.0.dev3/dotflow/cli/commands/storage.py +55 -0
  8. {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev3}/dotflow/cloud/gcp/services/apis.py +5 -3
  9. {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev3}/dotflow/cloud/github/actions_deployer.py +14 -11
  10. {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev3}/dotflow/core/action.py +40 -79
  11. {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev3}/dotflow/core/context.py +17 -4
  12. dotflow-0.15.0.dev3/dotflow/core/engine.py +277 -0
  13. dotflow-0.15.0.dev3/dotflow/core/module.py +30 -0
  14. {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev3}/dotflow/core/serializers/task.py +7 -9
  15. {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev3}/dotflow/core/workflow.py +46 -62
  16. dotflow-0.15.0.dev3/dotflow/providers/storage_default.py +22 -0
  17. {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev3}/dotflow/providers/storage_file.py +11 -8
  18. {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev3}/dotflow/settings.py +1 -1
  19. dotflow-0.15.0.dev3/dotflow/utils/error_handler.py +13 -0
  20. {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev3}/dotflow/utils/tools.py +0 -10
  21. {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev3}/pyproject.toml +3 -3
  22. dotflow-0.15.0.dev2/dotflow/core/execution.py +0 -176
  23. dotflow-0.15.0.dev2/dotflow/core/module.py +0 -36
  24. dotflow-0.15.0.dev2/dotflow/providers/storage_default.py +0 -20
  25. dotflow-0.15.0.dev2/dotflow/storage.py +0 -22
  26. dotflow-0.15.0.dev2/dotflow/utils/error_handler.py +0 -22
  27. {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev3}/LICENSE +0 -0
  28. {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev3}/dotflow/abc/__init__.py +0 -0
  29. {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev3}/dotflow/abc/api.py +0 -0
  30. {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev3}/dotflow/abc/file.py +0 -0
  31. {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev3}/dotflow/abc/http.py +0 -0
  32. {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev3}/dotflow/abc/log.py +0 -0
  33. {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev3}/dotflow/abc/metrics.py +0 -0
  34. {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev3}/dotflow/abc/notify.py +0 -0
  35. {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev3}/dotflow/abc/scheduler.py +0 -0
  36. {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev3}/dotflow/abc/storage.py +0 -0
  37. {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev3}/dotflow/abc/tcp.py +0 -0
  38. {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev3}/dotflow/abc/tracer.py +0 -0
  39. {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev3}/dotflow/cli/__init__.py +0 -0
  40. {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev3}/dotflow/cli/command.py +0 -0
  41. {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev3}/dotflow/cli/commands/__init__.py +0 -0
  42. {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev3}/dotflow/cli/commands/deploy.py +0 -0
  43. {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev3}/dotflow/cli/commands/log.py +0 -0
  44. {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev3}/dotflow/cli/commands/schedule.py +0 -0
  45. {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev3}/dotflow/cli/commands/start.py +0 -0
  46. {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev3}/dotflow/cli/setup.py +0 -0
  47. {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev3}/dotflow/cli/validators/__init__.py +0 -0
  48. {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev3}/dotflow/cli/validators/start.py +0 -0
  49. {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev3}/dotflow/cloud/__init__.py +0 -0
  50. {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev3}/dotflow/cloud/aws/__init__.py +0 -0
  51. {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev3}/dotflow/cloud/aws/base_lambda_deployer.py +0 -0
  52. {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev3}/dotflow/cloud/aws/constants.py +0 -0
  53. {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev3}/dotflow/cloud/aws/ecs_deployer.py +0 -0
  54. {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev3}/dotflow/cloud/aws/lambda_api_deployer.py +0 -0
  55. {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev3}/dotflow/cloud/aws/lambda_deployer.py +0 -0
  56. {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev3}/dotflow/cloud/aws/lambda_s3_deployer.py +0 -0
  57. {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev3}/dotflow/cloud/aws/lambda_sqs_deployer.py +0 -0
  58. {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev3}/dotflow/cloud/aws/services/__init__.py +0 -0
  59. {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev3}/dotflow/cloud/aws/services/cloudwatch.py +0 -0
  60. {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev3}/dotflow/cloud/aws/services/ecr.py +0 -0
  61. {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev3}/dotflow/cloud/aws/services/iam.py +0 -0
  62. {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev3}/dotflow/cloud/aws/services/s3.py +0 -0
  63. {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev3}/dotflow/cloud/core.py +0 -0
  64. {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev3}/dotflow/cloud/gcp/__init__.py +0 -0
  65. {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev3}/dotflow/cloud/gcp/cloudrun_deployer.py +0 -0
  66. {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev3}/dotflow/cloud/gcp/constants.py +0 -0
  67. {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev3}/dotflow/cloud/gcp/services/__init__.py +0 -0
  68. {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev3}/dotflow/cloud/gcp/services/artifact_registry.py +0 -0
  69. {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev3}/dotflow/cloud/gcp/services/gcs.py +0 -0
  70. {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev3}/dotflow/cloud/github/__init__.py +0 -0
  71. {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev3}/dotflow/cloud/github/constants.py +0 -0
  72. {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev3}/dotflow/core/__init__.py +0 -0
  73. {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev3}/dotflow/core/config.py +0 -0
  74. {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev3}/dotflow/core/decorators/__init__.py +0 -0
  75. {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev3}/dotflow/core/decorators/time.py +0 -0
  76. {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev3}/dotflow/core/dotflow.py +0 -0
  77. {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev3}/dotflow/core/exception.py +0 -0
  78. {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev3}/dotflow/core/serializers/__init__.py +0 -0
  79. {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev3}/dotflow/core/serializers/transport.py +0 -0
  80. {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev3}/dotflow/core/serializers/workflow.py +0 -0
  81. {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev3}/dotflow/core/task.py +0 -0
  82. {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev3}/dotflow/core/types/__init__.py +0 -0
  83. {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev3}/dotflow/core/types/execution.py +0 -0
  84. {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev3}/dotflow/core/types/overlap.py +0 -0
  85. {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev3}/dotflow/core/types/status.py +0 -0
  86. {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev3}/dotflow/core/types/storage.py +0 -0
  87. {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev3}/dotflow/core/types/workflow.py +0 -0
  88. {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev3}/dotflow/logging.py +0 -0
  89. {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev3}/dotflow/main.py +0 -0
  90. {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev3}/dotflow/providers/__init__.py +0 -0
  91. {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev3}/dotflow/providers/api_default.py +0 -0
  92. {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev3}/dotflow/providers/log_default.py +0 -0
  93. {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev3}/dotflow/providers/log_opentelemetry.py +0 -0
  94. {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev3}/dotflow/providers/log_sentry.py +0 -0
  95. {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev3}/dotflow/providers/metrics_default.py +0 -0
  96. {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev3}/dotflow/providers/metrics_opentelemetry.py +0 -0
  97. {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev3}/dotflow/providers/notify_default.py +0 -0
  98. {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev3}/dotflow/providers/notify_discord.py +0 -0
  99. {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev3}/dotflow/providers/notify_telegram.py +0 -0
  100. {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev3}/dotflow/providers/scheduler_cron.py +0 -0
  101. {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev3}/dotflow/providers/scheduler_default.py +0 -0
  102. {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev3}/dotflow/providers/storage_gcs.py +0 -0
  103. {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev3}/dotflow/providers/storage_s3.py +0 -0
  104. {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev3}/dotflow/providers/tracer_default.py +0 -0
  105. {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev3}/dotflow/providers/tracer_opentelemetry.py +0 -0
  106. {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev3}/dotflow/providers/tracer_sentry.py +0 -0
  107. {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev3}/dotflow/types.py +0 -0
  108. {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev3}/dotflow/utils/__init__.py +0 -0
  109. {dotflow-0.15.0.dev2 → dotflow-0.15.0.dev3}/dotflow/utils/basic_functions.py +0 -0
@@ -1,6 +1,6 @@
1
1
  Metadata-Version: 2.4
2
2
  Name: dotflow
3
- Version: 0.15.0.dev2
3
+ Version: 0.15.0.dev3
4
4
  Summary: 🎲 Dotflow turns an idea into flow! Lightweight Python library for execution pipelines with retry, parallel, cron and async support.
5
5
  License: MIT License
6
6
 
@@ -49,6 +49,7 @@ Provides-Extra: gcp
49
49
  Provides-Extra: otel
50
50
  Provides-Extra: scheduler
51
51
  Provides-Extra: sentry
52
+ Requires-Dist: GitPython ; extra == "deploy-github"
52
53
  Requires-Dist: PyGithub ; extra == "deploy-github"
53
54
  Requires-Dist: boto3 ; extra == "aws"
54
55
  Requires-Dist: boto3 ; extra == "deploy-aws"
@@ -76,86 +77,40 @@ Project-URL: Repository, https://github.com/dotflow-io/dotflow
76
77
  Description-Content-Type: text/markdown
77
78
 
78
79
  <div align="center">
79
- <a aria-label="Serverless.com" href="https://dotflow.io">Website</a>
80
- &nbsp;•&nbsp;
81
- <a aria-label="Dotflow Documentation" href="https://dotflow-io.github.io/dotflow/">Documentation</a>
82
- &nbsp;•&nbsp;
83
- <a aria-label="Pypi" href="https://pypi.org/project/dotflow/">Pypi</a>
84
- </div>
85
80
 
86
- <br/>
81
+ ![](docs/assets/dotflow.png)
87
82
 
88
- <div align="center">
83
+ **Build production-ready data pipelines in minutes, not days.**
89
84
 
90
- ![](https://raw.githubusercontent.com/FernandoCelmer/dotflow/master/docs/assets/dotflow.gif)
85
+ [![PyPI](https://img.shields.io/pypi/v/dotflow?style=flat-square)](https://pypi.org/project/dotflow/)
86
+ [![Python](https://img.shields.io/pypi/pyversions/dotflow?style=flat-square)](https://pypi.org/project/dotflow/)
87
+ [![Downloads](https://img.shields.io/pypi/dm/dotflow?style=flat-square)](https://pypi.org/project/dotflow/)
88
+ [![Stars](https://img.shields.io/github/stars/dotflow-io?label=Stars&style=flat-square)](https://github.com/dotflow-io/dotflow)
91
89
 
92
- ![GitHub Org's stars](https://img.shields.io/github/stars/dotflow-io?label=Dotflow&style=flat-square)
93
- ![GitHub last commit](https://img.shields.io/github/last-commit/dotflow-io/dotflow?style=flat-square)
94
- ![PyPI](https://img.shields.io/pypi/v/dotflow?style=flat-square)
95
- ![PyPI - Python Version](https://img.shields.io/pypi/pyversions/dotflow?style=flat-square)
96
- ![PyPI - Downloads](https://img.shields.io/pypi/dm/dotflow?style=flat-square)
90
+ [Website](https://dotflow.io) · [Documentation](https://dotflow-io.github.io/dotflow/) · [PyPI](https://pypi.org/project/dotflow/)
97
91
 
98
92
  </div>
99
93
 
100
- # Welcome to Dotflow
101
-
102
- Dotflow is a lightweight Python library for building execution pipelines. Define tasks with decorators, chain them together, and run workflows in sequential, parallel, or background mode — with built-in retry, timeout, storage, notifications, and more.
103
-
104
- > **[Read the full documentation](https://dotflow-io.github.io/dotflow/)**
105
-
106
- ## Table of Contents
107
-
108
- <details>
109
- <summary>Click to expand</summary>
110
-
111
- - [Getting Help](#getting-help)
112
- - [Installation](#installation)
113
- - [Quick Start](#quick-start)
114
- - [Features](#features)
115
- - [Execution Modes](#execution-modes)
116
- - [Retry, Timeout & Backoff](#retry-timeout--backoff)
117
- - [Context System](#context-system)
118
- - [Checkpoint & Resume](#checkpoint--resume)
119
- - [Storage Providers](#storage-providers)
120
- - [Notifications](#notifications)
121
- - [Class-Based Steps](#class-based-steps)
122
- - [Task Groups](#task-groups)
123
- - [Callbacks](#callbacks)
124
- - [Error Handling](#error-handling)
125
- - [Async Support](#async-support)
126
- - [Scheduler / Cron](#scheduler--cron)
127
- - [CLI](#cli)
128
- - [Dependency Injection via Config](#dependency-injection-via-config)
129
- - [More Examples](#more-examples)
130
- - [Commit Style](#commit-style)
131
- - [License](#license)
94
+ ---
132
95
 
133
- </details>
96
+ # Dotflow
134
97
 
135
- ## Getting Help
98
+ Dotflow is a lightweight Python library for execution pipelines. Define tasks with decorators, chain them together, and deploy to any cloud — with built-in retry, parallel execution, storage, observability, and cloud deployment.
136
99
 
137
- We use GitHub issues for tracking bugs and feature requests.
100
+ ## Why Dotflow?
138
101
 
139
- - [Bug Report](https://github.com/dotflow-io/dotflow/issues/new/choose)
140
- - [Documentation](https://github.com/dotflow-io/dotflow/issues/new/choose)
141
- - [Feature Request](https://github.com/dotflow-io/dotflow/issues/new/choose)
142
- - [Security Issue](https://github.com/dotflow-io/dotflow/issues/new/choose)
143
- - [General Question](https://github.com/dotflow-io/dotflow/issues/new/choose)
102
+ - **Simple** — `@action` decorator + `workflow.start()`. That's it.
103
+ - **Resilient** — Retry, backoff, timeout, checkpoints, and error handling out of the box.
104
+ - **Observable** — OpenTelemetry traces, metrics, and logs. Sentry error tracking.
105
+ - **Deployable** — `dotflow deploy --platform lambda` ships your pipeline to AWS in one command.
106
+ - **Portable** — Same code runs on Lambda, ECS, Cloud Run, Kubernetes, Docker, or GitHub Actions.
144
107
 
145
- ## Installation
108
+ ## Install
146
109
 
147
110
  ```bash
148
111
  pip install dotflow
149
112
  ```
150
113
 
151
- **Optional extras:**
152
-
153
- ```bash
154
- pip install dotflow[aws] # AWS S3 storage
155
- pip install dotflow[gcp] # Google Cloud Storage
156
- pip install dotflow[scheduler] # Cron-based scheduler
157
- ```
158
-
159
114
  ## Quick Start
160
115
 
161
116
  ```python
@@ -182,9 +137,94 @@ workflow.task.add(step=load)
182
137
  workflow.start()
183
138
  ```
184
139
 
140
+ ## Deploy anywhere
141
+
142
+ Write your pipeline once. Deploy to any cloud with a single command.
143
+
144
+ ```bash
145
+ dotflow init
146
+ dotflow deploy --platform lambda --project my_pipeline
147
+ ```
148
+
149
+ ### Supported platforms
150
+
151
+ | Platform | Deploy method |
152
+ |----------|---------------|
153
+ | <img src="https://cdn.simpleicons.org/docker" width="16" /> **Docker** | `docker compose up` |
154
+ | <img src="https://www.vectorlogo.zone/logos/amazon_awslambda/amazon_awslambda-icon.svg" width="16" /> **AWS Lambda** | `dotflow deploy` |
155
+ | <img src="https://www.vectorlogo.zone/logos/amazon_awslambda/amazon_awslambda-icon.svg" width="16" /> **AWS Lambda + EventBridge** | `dotflow deploy --schedule` |
156
+ | <img src="https://www.vectorlogo.zone/logos/amazon_awslambda/amazon_awslambda-icon.svg" width="16" /> **AWS Lambda + S3 Trigger** | `dotflow deploy` |
157
+ | <img src="https://www.vectorlogo.zone/logos/amazon_awslambda/amazon_awslambda-icon.svg" width="16" /> **AWS Lambda + SQS Trigger** | `dotflow deploy` |
158
+ | <img src="https://www.vectorlogo.zone/logos/amazon_awslambda/amazon_awslambda-icon.svg" width="16" /> **AWS Lambda + API Gateway** | `dotflow deploy` |
159
+ | <img src="https://www.vectorlogo.zone/logos/amazon_ecs/amazon_ecs-icon.svg" width="16" /> **AWS ECS Fargate** | `dotflow deploy` |
160
+ | <img src="https://cdn.simpleicons.org/googlecloud" width="16" /> **Google Cloud Run** | `gcloud run deploy` |
161
+ | <img src="https://cdn.simpleicons.org/kubernetes" width="16" /> **Kubernetes** | `kubectl apply` |
162
+ | <img src="https://cdn.simpleicons.org/githubactions" width="16" /> **GitHub Actions** | `dotflow deploy` |
163
+
164
+ > [See all 34+ platforms →](https://dotflow-io.github.io/dotflow/nav/cloud/)
165
+
166
+ ## Optional extras
167
+
168
+ ```bash
169
+ pip install dotflow[aws] # S3 storage
170
+ pip install dotflow[gcp] # Google Cloud Storage
171
+ pip install dotflow[scheduler] # Cron scheduler
172
+ pip install dotflow[otel] # OpenTelemetry
173
+ pip install dotflow[sentry] # Sentry error tracking
174
+ pip install dotflow[deploy-aws] # AWS deploy (Lambda, ECS)
175
+ pip install dotflow[deploy-gcp] # GCP deploy (Cloud Run)
176
+ pip install dotflow[deploy-github] # GitHub Actions deploy
177
+ ```
178
+
179
+ > **[Read the full documentation →](https://dotflow-io.github.io/dotflow/)**
180
+
181
+ ## Documentation
182
+
183
+ | Section | Description |
184
+ |---------|-------------|
185
+ | [Concepts](https://dotflow-io.github.io/dotflow/nav/concepts/concept-workflow-and-tasks/) | Workflows, tasks, context, providers, process modes |
186
+ | [How-to Guides](https://dotflow-io.github.io/dotflow/nav/how-to/) | Step-by-step tutorials for workflows, tasks, and CLI |
187
+ | [Cloud Deployment](https://dotflow-io.github.io/dotflow/nav/cloud/) | Deploy to AWS, GCP, Kubernetes, Docker, GitHub Actions |
188
+ | [Integrations](https://dotflow-io.github.io/dotflow/nav/integrations/) | OpenTelemetry, Sentry, Telegram, Discord, S3, GCS |
189
+ | [Examples](https://dotflow-io.github.io/dotflow/nav/examples/) | Real-world pipelines: ETL, health checks, async, scheduler |
190
+ | [Reference](https://dotflow-io.github.io/dotflow/nav/reference/dotflow/) | API reference for all classes and providers |
191
+ | [Custom Providers](https://dotflow-io.github.io/dotflow/nav/development/custom-providers/) | Build your own storage, notify, log, tracer, or metrics provider |
192
+
185
193
  ## Features
186
194
 
187
- ### Execution Modes
195
+ <details>
196
+ <summary><strong>Observability</strong></summary>
197
+
198
+ > [OpenTelemetry docs](https://dotflow-io.github.io/dotflow/nav/tutorial/log-opentelemetry/) | [Sentry docs](https://dotflow-io.github.io/dotflow/nav/tutorial/log-sentry/) | [Tracer docs](https://dotflow-io.github.io/dotflow/nav/tutorial/tracer-opentelemetry/) | [Metrics docs](https://dotflow-io.github.io/dotflow/nav/tutorial/metrics-opentelemetry/)
199
+
200
+ Built-in support for OpenTelemetry and Sentry:
201
+
202
+ ```python
203
+ from dotflow import Config
204
+ from dotflow.providers import LogOpenTelemetry, TracerOpenTelemetry, MetricsOpenTelemetry
205
+
206
+ # OpenTelemetry: traces, metrics, and structured logs
207
+ config = Config(
208
+ log=LogOpenTelemetry(service_name="my-pipeline"),
209
+ tracer=TracerOpenTelemetry(service_name="my-pipeline"),
210
+ metrics=MetricsOpenTelemetry(service_name="my-pipeline"),
211
+ )
212
+ ```
213
+
214
+ ```python
215
+ from dotflow.providers import LogSentry, TracerSentry
216
+
217
+ # Sentry: error tracking + performance monitoring
218
+ config = Config(
219
+ log=LogSentry(dsn="https://xxx@sentry.io/123"),
220
+ tracer=TracerSentry(),
221
+ )
222
+ ```
223
+
224
+ </details>
225
+
226
+ <details>
227
+ <summary><strong>Execution Modes</strong></summary>
188
228
 
189
229
  > [Process Mode docs](https://dotflow-io.github.io/dotflow/nav/concepts/process-mode-sequential/)
190
230
 
@@ -255,7 +295,10 @@ flowchart TD
255
295
 
256
296
  ---
257
297
 
258
- ### Retry, Timeout & Backoff
298
+ </details>
299
+
300
+ <details>
301
+ <summary><strong>Retry, Timeout & Backoff</strong></summary>
259
302
 
260
303
  > [Retry docs](https://dotflow-io.github.io/dotflow/nav/tutorial/task-retry/) | [Backoff docs](https://dotflow-io.github.io/dotflow/nav/tutorial/task-backoff/) | [Timeout docs](https://dotflow-io.github.io/dotflow/nav/tutorial/task-timeout/)
261
304
 
@@ -278,7 +321,10 @@ def unreliable_api_call():
278
321
 
279
322
  ---
280
323
 
281
- ### Context System
324
+ </details>
325
+
326
+ <details>
327
+ <summary><strong>Context System</strong></summary>
282
328
 
283
329
  > [Context docs](https://dotflow-io.github.io/dotflow/nav/tutorial/initial-context/) | [Previous Context](https://dotflow-io.github.io/dotflow/nav/tutorial/previous-context/) | [Many Contexts](https://dotflow-io.github.io/dotflow/nav/tutorial/many-contexts/)
284
330
 
@@ -309,7 +355,10 @@ Each `Context` object contains:
309
355
 
310
356
  ---
311
357
 
312
- ### Checkpoint & Resume
358
+ </details>
359
+
360
+ <details>
361
+ <summary><strong>Checkpoint & Resume</strong></summary>
313
362
 
314
363
  > [Checkpoint docs](https://dotflow-io.github.io/dotflow/nav/tutorial/checkpoint/)
315
364
 
@@ -335,7 +384,10 @@ workflow.start(resume=True)
335
384
 
336
385
  ---
337
386
 
338
- ### Storage Providers
387
+ </details>
388
+
389
+ <details>
390
+ <summary><strong>Storage Providers</strong></summary>
339
391
 
340
392
  > [Storage docs](https://dotflow-io.github.io/dotflow/nav/tutorial/storage-default/)
341
393
 
@@ -389,32 +441,39 @@ workflow = DotFlow(config=config)
389
441
 
390
442
  ---
391
443
 
392
- ### Notifications
444
+ </details>
445
+
446
+ <details>
447
+ <summary><strong>Notifications</strong></summary>
393
448
 
394
- > [Telegram docs](https://dotflow-io.github.io/dotflow/nav/tutorial/notify-telegram/)
449
+ > [Telegram docs](https://dotflow-io.github.io/dotflow/nav/tutorial/notify-telegram/) | [Discord docs](https://dotflow-io.github.io/dotflow/nav/tutorial/notify-discord/)
395
450
 
396
- Get notified about task status changes via Telegram.
451
+ Get notified about task status changes via Telegram or Discord.
397
452
 
398
453
  ```python
399
- from dotflow import DotFlow, Config
454
+ from dotflow import Config
400
455
  from dotflow.providers import NotifyTelegram
401
- from dotflow.core.types.status import TypeStatus
402
456
 
403
- notify = NotifyTelegram(
457
+ config = Config(notify=NotifyTelegram(
404
458
  token="YOUR_BOT_TOKEN",
405
459
  chat_id=123456789,
406
- notification_type=TypeStatus.FAILED, # only notify on failures (optional)
407
- )
408
-
409
- config = Config(notify=notify)
410
- workflow = DotFlow(config=config)
460
+ ))
411
461
  ```
412
462
 
413
- Status types: `NOT_STARTED`, `IN_PROGRESS`, `COMPLETED`, `PAUSED`, `RETRY`, `FAILED`
463
+ ```python
464
+ from dotflow.providers import NotifyDiscord
465
+
466
+ config = Config(notify=NotifyDiscord(
467
+ webhook_url="https://discord.com/api/webhooks/...",
468
+ ))
469
+ ```
414
470
 
415
471
  ---
416
472
 
417
- ### Class-Based Steps
473
+ </details>
474
+
475
+ <details>
476
+ <summary><strong>Class-Based Steps</strong></summary>
418
477
 
419
478
  Return a class instance from a task, and Dotflow will automatically discover and execute all `@action`-decorated methods in source order.
420
479
 
@@ -446,7 +505,10 @@ workflow.start()
446
505
 
447
506
  ---
448
507
 
449
- ### Task Groups
508
+ </details>
509
+
510
+ <details>
511
+ <summary><strong>Task Groups</strong></summary>
450
512
 
451
513
  > [Groups docs](https://dotflow-io.github.io/dotflow/nav/tutorial/groups/)
452
514
 
@@ -463,7 +525,10 @@ workflow.start() # groups run in parallel, tasks within each group run sequenti
463
525
 
464
526
  ---
465
527
 
466
- ### Callbacks
528
+ </details>
529
+
530
+ <details>
531
+ <summary><strong>Callbacks</strong></summary>
467
532
 
468
533
  > [Task Callback docs](https://dotflow-io.github.io/dotflow/nav/tutorial/task-callback/) | [Workflow Callback docs](https://dotflow-io.github.io/dotflow/nav/tutorial/workflow-callback/)
469
534
 
@@ -490,7 +555,10 @@ workflow.start(on_success=on_success, on_failure=on_failure)
490
555
 
491
556
  ---
492
557
 
493
- ### Error Handling
558
+ </details>
559
+
560
+ <details>
561
+ <summary><strong>Error Handling</strong></summary>
494
562
 
495
563
  > [Error Handling docs](https://dotflow-io.github.io/dotflow/nav/tutorial/error-handling/) | [Keep Going docs](https://dotflow-io.github.io/dotflow/nav/tutorial/keep-going/)
496
564
 
@@ -520,7 +588,10 @@ for task in workflow.result_task():
520
588
 
521
589
  ---
522
590
 
523
- ### Async Support
591
+ </details>
592
+
593
+ <details>
594
+ <summary><strong>Async Support</strong></summary>
524
595
 
525
596
  > [Async docs](https://dotflow-io.github.io/dotflow/nav/tutorial/async-actions/)
526
597
 
@@ -543,7 +614,10 @@ workflow.start()
543
614
 
544
615
  ---
545
616
 
546
- ### Scheduler / Cron
617
+ </details>
618
+
619
+ <details>
620
+ <summary><strong>Scheduler / Cron</strong></summary>
547
621
 
548
622
  > [Cron scheduler docs](https://dotflow-io.github.io/dotflow/nav/tutorial/scheduler-cron/) | [Default scheduler](https://dotflow-io.github.io/dotflow/nav/tutorial/scheduler-default/) | [Cron overlap (concepts)](https://dotflow-io.github.io/dotflow/nav/concepts/concept-cron-overlap/)
549
623
 
@@ -592,7 +666,10 @@ The scheduler handles graceful shutdown via `SIGINT`/`SIGTERM` signals automatic
592
666
 
593
667
  ---
594
668
 
595
- ### CLI
669
+ </details>
670
+
671
+ <details>
672
+ <summary><strong>CLI</strong></summary>
596
673
 
597
674
  > [CLI docs](https://dotflow-io.github.io/dotflow/nav/how-to/cli/simple-start/)
598
675
 
@@ -634,14 +711,20 @@ Available CLI commands:
634
711
 
635
712
  | Command | Description |
636
713
  |---------|-------------|
637
- | `dotflow init` | Initialize a new Dotflow project |
714
+ | `dotflow init` | Scaffold a new project with cloud support |
638
715
  | `dotflow start` | Run a workflow |
639
716
  | `dotflow schedule` | Run a workflow on a cron schedule |
640
- | `dotflow log` | View execution logs |
717
+ | `dotflow logs` | View execution logs |
718
+ | `dotflow cloud list` | Show available cloud platforms |
719
+ | `dotflow cloud generate --platform <name>` | Generate deployment files |
720
+ | `dotflow deploy --platform <name> --project <name>` | Deploy to cloud |
641
721
 
642
722
  ---
643
723
 
644
- ### Dependency Injection via Config
724
+ </details>
725
+
726
+ <details>
727
+ <summary><strong>Dependency Injection via Config</strong></summary>
645
728
 
646
729
  The `Config` class lets you swap providers for storage, notifications, logging, and scheduling:
647
730
 
@@ -670,7 +753,10 @@ Extend Dotflow by implementing the abstract base classes:
670
753
 
671
754
  ---
672
755
 
673
- ### Results & Inspection
756
+ </details>
757
+
758
+ <details>
759
+ <summary><strong>Results & Inspection</strong></summary>
674
760
 
675
761
  After execution, inspect results directly from the workflow object:
676
762
 
@@ -701,7 +787,10 @@ workflow.task.schema() # Pydantic schema of the workflow
701
787
 
702
788
  ---
703
789
 
704
- ### Dynamic Module Import
790
+ </details>
791
+
792
+ <details>
793
+ <summary><strong>Dynamic Module Import</strong></summary>
705
794
 
706
795
  Reference tasks and callbacks by their module path string instead of importing them directly:
707
796
 
@@ -712,124 +801,12 @@ workflow.task.add(step="my_package.tasks.save_results", callback="my_package.cal
712
801
 
713
802
  ---
714
803
 
715
- ## More Examples
716
-
717
- All examples are available in the [`docs_src/`](https://github.com/dotflow-io/dotflow/tree/develop/docs_src) directory.
718
-
719
- #### Basic
720
-
721
- | Example | Description | Command |
722
- |---------|-------------|---------|
723
- | [first_steps](https://github.com/dotflow-io/dotflow/blob/develop/docs_src/first_steps/first_steps.py) | Minimal workflow with callback | `python docs_src/first_steps/first_steps.py` |
724
- | [simple_function_workflow](https://github.com/dotflow-io/dotflow/blob/develop/docs_src/basic/simple_function_workflow.py) | Simple function-based workflow | `python docs_src/basic/simple_function_workflow.py` |
725
- | [simple_class_workflow](https://github.com/dotflow-io/dotflow/blob/develop/docs_src/basic/simple_class_workflow.py) | Class-based step with retry | `python docs_src/basic/simple_class_workflow.py` |
726
- | [simple_function_workflow_with_error](https://github.com/dotflow-io/dotflow/blob/develop/docs_src/basic/simple_function_workflow_with_error.py) | Error inspection after failure | `python docs_src/basic/simple_function_workflow_with_error.py` |
727
-
728
- #### Async
729
-
730
- | Example | Description | Command |
731
- |---------|-------------|---------|
732
- | [async_action](https://github.com/dotflow-io/dotflow/blob/develop/docs_src/async/async_action.py) | Async task functions | `python docs_src/async/async_action.py` |
733
-
734
- #### Context
735
-
736
- | Example | Description | Command |
737
- |---------|-------------|---------|
738
- | [context](https://github.com/dotflow-io/dotflow/blob/develop/docs_src/context/context.py) | Creating and inspecting a Context | `python docs_src/context/context.py` |
739
- | [initial_context](https://github.com/dotflow-io/dotflow/blob/develop/docs_src/initial_context/initial_context.py) | Passing initial context per task | `python docs_src/initial_context/initial_context.py` |
740
- | [previous_context](https://github.com/dotflow-io/dotflow/blob/develop/docs_src/previous_context/previous_context.py) | Chaining context between tasks | `python docs_src/previous_context/previous_context.py` |
741
- | [many_contexts](https://github.com/dotflow-io/dotflow/blob/develop/docs_src/context/many_contexts.py) | Using both initial and previous context | `python docs_src/context/many_contexts.py` |
742
-
743
- #### Process Modes
744
-
745
- | Example | Description | Command |
746
- |---------|-------------|---------|
747
- | [sequential](https://github.com/dotflow-io/dotflow/blob/develop/docs_src/process_mode/sequential.py) | Sequential execution | `python docs_src/process_mode/sequential.py` |
748
- | [background](https://github.com/dotflow-io/dotflow/blob/develop/docs_src/process_mode/background.py) | Background (non-blocking) execution | `python docs_src/process_mode/background.py` |
749
- | [parallel](https://github.com/dotflow-io/dotflow/blob/develop/docs_src/process_mode/parallel.py) | Parallel execution | `python docs_src/process_mode/parallel.py` |
750
- | [parallel_group](https://github.com/dotflow-io/dotflow/blob/develop/docs_src/process_mode/parallel_group.py) | Parallel groups execution | `python docs_src/process_mode/parallel_group.py` |
751
- | [sequential_group_mode](https://github.com/dotflow-io/dotflow/blob/develop/docs_src/workflow/sequential_group_mode.py) | Sequential with named groups | `python docs_src/workflow/sequential_group_mode.py` |
752
-
753
- #### Resilience (Retry, Backoff, Timeout)
754
-
755
- | Example | Description | Command |
756
- |---------|-------------|---------|
757
- | [retry](https://github.com/dotflow-io/dotflow/blob/develop/docs_src/retry/retry.py) | Retry on function and class steps | `python docs_src/retry/retry.py` |
758
- | [retry_delay](https://github.com/dotflow-io/dotflow/blob/develop/docs_src/retry/retry_delay.py) | Retry with delay between attempts | `python docs_src/retry/retry_delay.py` |
759
- | [backoff](https://github.com/dotflow-io/dotflow/blob/develop/docs_src/backoff/backoff.py) | Exponential backoff on retries | `python docs_src/backoff/backoff.py` |
760
- | [timeout](https://github.com/dotflow-io/dotflow/blob/develop/docs_src/timeout/timeout.py) | Timeout per task execution | `python docs_src/timeout/timeout.py` |
761
-
762
- #### Callbacks
763
-
764
- | Example | Description | Command |
765
- |---------|-------------|---------|
766
- | [task_callback](https://github.com/dotflow-io/dotflow/blob/develop/docs_src/callback/task_callback.py) | Per-task callback on completion | `python docs_src/callback/task_callback.py` |
767
- | [workflow_callback_success](https://github.com/dotflow-io/dotflow/blob/develop/docs_src/callback/workflow_callback_success.py) | Workflow-level success callback | `python docs_src/callback/workflow_callback_success.py` |
768
- | [workflow_callback_failure](https://github.com/dotflow-io/dotflow/blob/develop/docs_src/callback/workflow_callback_failure.py) | Workflow-level failure callback | `python docs_src/callback/workflow_callback_failure.py` |
769
-
770
- #### Error Handling
771
-
772
- | Example | Description | Command |
773
- |---------|-------------|---------|
774
- | [errors](https://github.com/dotflow-io/dotflow/blob/develop/docs_src/errors/errors.py) | Inspecting task errors and retry count | `python docs_src/errors/errors.py` |
775
- | [keep_going_true](https://github.com/dotflow-io/dotflow/blob/develop/docs_src/workflow/keep_going_true.py) | Continue workflow after task failure | `python docs_src/workflow/keep_going_true.py` |
776
-
777
- #### Groups
778
-
779
- | Example | Description | Command |
780
- |---------|-------------|---------|
781
- | [step_with_groups](https://github.com/dotflow-io/dotflow/blob/develop/docs_src/group/step_with_groups.py) | Tasks in named parallel groups | `python docs_src/group/step_with_groups.py` |
782
-
783
- #### Storage
784
-
785
- | Example | Description | Command |
786
- |---------|-------------|---------|
787
- | [storage_file](https://github.com/dotflow-io/dotflow/blob/develop/docs_src/storage/storage_file.py) | File-based JSON storage | `python docs_src/storage/storage_file.py` |
788
- | [storage_s3](https://github.com/dotflow-io/dotflow/blob/develop/docs_src/storage/storage_s3.py) | AWS S3 storage | `python docs_src/storage/storage_s3.py` |
789
- | [storage_gcs](https://github.com/dotflow-io/dotflow/blob/develop/docs_src/storage/storage_gcs.py) | Google Cloud Storage | `python docs_src/storage/storage_gcs.py` |
790
-
791
- #### Checkpoint & Resume
792
-
793
- | Example | Description | Command |
794
- |---------|-------------|---------|
795
- | [checkpoint](https://github.com/dotflow-io/dotflow/blob/develop/docs_src/checkpoint/checkpoint.py) | Resume workflow from last checkpoint | `python docs_src/checkpoint/checkpoint.py` |
796
-
797
- #### Notifications
798
-
799
- | Example | Description | Command |
800
- |---------|-------------|---------|
801
- | [notify_telegram](https://github.com/dotflow-io/dotflow/blob/develop/docs_src/notify/notify_telegram.py) | Telegram notifications on failure | `python docs_src/notify/notify_telegram.py` |
802
-
803
- #### Config & Providers
804
+ </details>
804
805
 
805
- | Example | Description | Command |
806
- |---------|-------------|---------|
807
- | [config](https://github.com/dotflow-io/dotflow/blob/develop/docs_src/config/config.py) | Full Config with storage, notify, log | `python docs_src/config/config.py` |
808
- | [storage_provider](https://github.com/dotflow-io/dotflow/blob/develop/docs_src/config/storage_provider.py) | Swapping storage providers | `python docs_src/config/storage_provider.py` |
809
- | [notify_provider](https://github.com/dotflow-io/dotflow/blob/develop/docs_src/config/notify_provider.py) | Swapping notification providers | `python docs_src/config/notify_provider.py` |
810
- | [log_provider](https://github.com/dotflow-io/dotflow/blob/develop/docs_src/config/log_provider.py) | Custom log provider | `python docs_src/config/log_provider.py` |
811
806
 
812
- #### Results & Output
807
+ ## More Examples
813
808
 
814
- | Example | Description | Command |
815
- |---------|-------------|---------|
816
- | [step_function_result_task](https://github.com/dotflow-io/dotflow/blob/develop/docs_src/output/step_function_result_task.py) | Inspect task results (function) | `python docs_src/output/step_function_result_task.py` |
817
- | [step_function_result_context](https://github.com/dotflow-io/dotflow/blob/develop/docs_src/output/step_function_result_context.py) | Inspect context results (function) | `python docs_src/output/step_function_result_context.py` |
818
- | [step_function_result_storage](https://github.com/dotflow-io/dotflow/blob/develop/docs_src/output/step_function_result_storage.py) | Inspect storage results (function) | `python docs_src/output/step_function_result_storage.py` |
819
- | [step_class_result_task](https://github.com/dotflow-io/dotflow/blob/develop/docs_src/output/step_class_result_task.py) | Inspect task results (class) | `python docs_src/output/step_class_result_task.py` |
820
- | [step_class_result_context](https://github.com/dotflow-io/dotflow/blob/develop/docs_src/output/step_class_result_context.py) | Inspect context results (class) | `python docs_src/output/step_class_result_context.py` |
821
- | [step_class_result_storage](https://github.com/dotflow-io/dotflow/blob/develop/docs_src/output/step_class_result_storage.py) | Inspect storage results (class) | `python docs_src/output/step_class_result_storage.py` |
822
-
823
- #### CLI
824
-
825
- | Example | Description | Command |
826
- |---------|-------------|---------|
827
- | [simple_cli](https://github.com/dotflow-io/dotflow/blob/develop/docs_src/basic/simple_cli.py) | Basic CLI execution | `dotflow start --step docs_src.basic.simple_cli.simple_step` |
828
- | [cli_with_callback](https://github.com/dotflow-io/dotflow/blob/develop/docs_src/cli/cli_with_callback.py) | CLI with callback function | `dotflow start --step docs_src.cli.cli_with_callback.simple_step --callback docs_src.cli.cli_with_callback.callback` |
829
- | [cli_with_initial_context](https://github.com/dotflow-io/dotflow/blob/develop/docs_src/cli/cli_with_initial_context.py) | CLI with initial context | `dotflow start --step docs_src.cli.cli_with_initial_context.simple_step --initial-context abc` |
830
- | [cli_with_mode](https://github.com/dotflow-io/dotflow/blob/develop/docs_src/cli/cli_with_mode.py) | CLI with execution mode | `dotflow start --step docs_src.cli.cli_with_mode.simple_step --mode sequential` |
831
- | [cli_with_output_context](https://github.com/dotflow-io/dotflow/blob/develop/docs_src/cli/cli_with_output_context.py) | CLI with file storage output | `dotflow start --step docs_src.cli.cli_with_output_context.simple_step --storage file` |
832
- | [cli_with_path](https://github.com/dotflow-io/dotflow/blob/develop/docs_src/cli/cli_with_path.py) | CLI with custom storage path | `dotflow start --step docs_src.cli.cli_with_path.simple_step --path .storage --storage file` |
809
+ All examples are available in the [`docs_src/`](https://github.com/dotflow-io/dotflow/tree/develop/docs_src) directory.
833
810
 
834
811
  ## Commit Style
835
812