ddeutil-workflow 0.0.33__py3-none-any.whl → 0.0.35__py3-none-any.whl

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (24) hide show
  1. ddeutil/workflow/__about__.py +1 -1
  2. ddeutil/workflow/__init__.py +19 -10
  3. ddeutil/workflow/api/api.py +13 -8
  4. ddeutil/workflow/api/routes/__init__.py +8 -0
  5. ddeutil/workflow/api/routes/logs.py +36 -0
  6. ddeutil/workflow/api/{route.py → routes/schedules.py} +2 -131
  7. ddeutil/workflow/api/routes/workflows.py +137 -0
  8. ddeutil/workflow/audit.py +28 -37
  9. ddeutil/workflow/{hook.py → caller.py} +27 -27
  10. ddeutil/workflow/conf.py +47 -12
  11. ddeutil/workflow/job.py +149 -138
  12. ddeutil/workflow/logs.py +214 -0
  13. ddeutil/workflow/params.py +40 -12
  14. ddeutil/workflow/result.py +40 -61
  15. ddeutil/workflow/scheduler.py +185 -163
  16. ddeutil/workflow/{stage.py → stages.py} +105 -42
  17. ddeutil/workflow/utils.py +20 -2
  18. ddeutil/workflow/workflow.py +142 -117
  19. {ddeutil_workflow-0.0.33.dist-info → ddeutil_workflow-0.0.35.dist-info}/METADATA +36 -32
  20. ddeutil_workflow-0.0.35.dist-info/RECORD +30 -0
  21. {ddeutil_workflow-0.0.33.dist-info → ddeutil_workflow-0.0.35.dist-info}/WHEEL +1 -1
  22. ddeutil_workflow-0.0.33.dist-info/RECORD +0 -26
  23. {ddeutil_workflow-0.0.33.dist-info → ddeutil_workflow-0.0.35.dist-info}/LICENSE +0 -0
  24. {ddeutil_workflow-0.0.33.dist-info → ddeutil_workflow-0.0.35.dist-info}/top_level.txt +0 -0
{ddeutil_workflow-0.0.33.dist-info → ddeutil_workflow-0.0.35.dist-info}/METADATA RENAMED
@@ -1,6 +1,6 @@
1
1
  Metadata-Version: 2.2
2
2
  Name: ddeutil-workflow
3
- Version: 0.0.33
3
+ Version: 0.0.35
4
4
  Summary: Lightweight workflow orchestration
5
5
  Author-email: ddeutils <korawich.anu@gmail.com>
6
6
  License: MIT
@@ -62,7 +62,7 @@ configuration. It called **Metadata Driven Data Workflow**.
62
62
  1. The Minimum frequency unit of scheduling is **1 minute** :warning:
63
63
  2. Can not re-run only failed stage and its pending downstream :rotating_light:
64
64
  3. All parallel tasks inside workflow engine use Multi-Threading
65
- (Python 3.13 unlock GIL :unlock:)
65
+ (🐍 Python 3.13 unlock GIL :unlock:)
66
66
 
67
67
  ---
68
68
 
@@ -74,49 +74,52 @@ you should to set the data layer separate this core program before run this appl
74
74
 
75
75
  ```mermaid
76
76
  flowchart LR
77
- subgraph Interface
78
- A((User))
79
- subgraph Docker Container
77
+ A((fa:fa-user User))
78
+
79
+ subgraph Docker Container
80
+ direction TB
80
81
  G@{ shape: rounded, label: "Observe<br>Application" }
81
- end
82
82
  end
83
83
 
84
- A --->|action| B(Workflow<br>Application)
85
- B ---> |response| A
86
- B -..-> |response| G
87
- G -..-> |request| B
88
-
89
84
  subgraph Docker Container
90
- B
85
+ direction TB
86
+ B@{ shape: rounded, label: "Workflow<br>Application" }
91
87
  end
92
88
 
89
+ A <-->|action &<br>response| B
90
+ B -...-> |response| G
91
+ G -...-> |request| B
92
+
93
93
  subgraph Data Context
94
- D@{ shape: processes, label: "Logs" }
95
- E@{ shape: lin-cyl, label: "Audit<br>Logs" }
94
+ D@{ shape: processes, label: "Logs" }
95
+ E@{ shape: lin-cyl, label: "Audit<br>Logs" }
96
96
  end
97
97
 
98
98
  subgraph Git Context
99
- F@{ shape: tag-rect, label: "YAML<br>files" }
99
+ F@{ shape: tag-rect, label: "YAML<br>files" }
100
100
  end
101
101
 
102
- B --->|disable| F
103
- F --->|read| B
102
+ A ---> |push| H(Repo)
103
+ H -.-> |pull| F
104
+
105
+ B <-->|disable &<br>read| F
104
106
 
105
- B --->|write| E
106
- E --->|read| B
107
- B --->|write| D
107
+ B <-->|read &<br>write| E
108
+
109
+ B -->|write| D
108
110
 
109
111
  D -.->|read| G
110
112
  E -.->|read| G
111
113
  ```
112
114
 
113
115
  > [!WARNING]
114
- > _Disclaimer_: I inspire the dynamic statement from the [**GitHub Action**](https://github.com/features/actions)
115
- > with `.yml` files and all configs file from several data orchestration framework
116
- > tools from my experience on Data Engineer. :grimacing:
117
- >
118
- > Other workflow tools that I interest on them and pick some interested feature
119
- > implement to this package:
116
+ > _**Disclaimer**_: I inspire the dynamic YAML statement from the [**GitHub Action**](https://github.com/features/actions),
117
+ > and all configs pattern from several data orchestration framework tools from
118
+ > my data engineering experience. :grimacing:
119
+
120
+ > [!NOTE]
121
+ > Other workflow orchestration tools that I interest and pick them to be inspiration
122
+ > some for this package:
120
123
  >
121
124
  > - [Google **Workflows**](https://cloud.google.com/workflows)
122
125
  > - [AWS **Step Functions**](https://aws.amazon.com/step-functions/)
@@ -138,9 +141,9 @@ This is examples that use workflow file for running common Data Engineering
138
141
  use-case.
139
142
 
140
143
  > [!IMPORTANT]
141
- > I recommend you to use the `hook` stage for all actions that you want to do
144
+ > I recommend you to use the `call` stage for all actions that you want to do
142
145
  > with workflow activity that you want to orchestrate. Because it is able to
143
- > dynamic an input argument with the same hook function that make you use less
146
+ > dynamic an input argument with the same call function that make you use less
144
147
  > time to maintenance your data workflows.
145
148
 
146
149
  ```yaml
@@ -182,7 +185,7 @@ run-py-local:
182
185
  writing_mode: flatten
183
186
  aws_s3_path: my-data/open-data/${{ params.source-extract }}
184
187
 
185
- # This Authentication code should implement with your custom hook
188
+ # This Authentication code should implement with your custom call
186
189
  # function. The template allow you to use environment variable.
187
190
  aws_access_client_id: ${AWS_ACCESS_CLIENT_ID}
188
191
  aws_access_client_secret: ${AWS_ACCESS_CLIENT_SECRET}
@@ -247,7 +250,7 @@ it will use default value and do not raise any error to you.
247
250
  | Name | Component | Default | Description |
248
251
  |:-----------------------------|:---------:|:--------------------------------------------------------------------------------------------------------------------------------|:-------------------------------------------------------------------------------------------------------------------|
249
252
  | **ROOT_PATH** | Core | `.` | The root path of the workflow application. |
250
- | **REGISTRY** | Core | `.` | List of importable string for the hook stage. |
253
+ | **REGISTRY** | Core | `.` | List of importable string for the call stage. |
251
254
  | **REGISTRY_FILTER** | Core | `ddeutil.workflow.templates` | List of importable string for the filter template. |
252
255
  | **CONF_PATH** | Core | `conf` | The config path that keep all template `.yaml` files. |
253
256
  | **TIMEZONE** | Core | `Asia/Bangkok` | A Timezone string value that will pass to `ZoneInfo` object. |
@@ -264,9 +267,10 @@ it will use default value and do not raise any error to you.
264
267
  | **PATH** | Log | `./logs` | The log path of the workflow saving log. |
265
268
  | **DEBUG_MODE** | Log | `true` | A flag that enable logging with debug level mode. |
266
269
  | **FORMAT** | Log | `%(asctime)s.%(msecs)03d (%(name)-10s, %(process)-5d,%(thread)-5d) [%(levelname)-7s] %(message)-120s (%(filename)s:%(lineno)s)` | |
270
+ | **FORMAT_FILE** | Log | `{datetime} ({process:5d}, {thread:5d}) {message:120s} ({filename}:{lineno})` | |
267
271
  | **DATETIME_FORMAT** | Log | `%Y-%m-%d %H:%M:%S` | |
268
- | **ENABLE_ROTATED_FILE** | Log | `false` | |
269
- | **PATH** | Audit | `./logs` | |
272
+ | **ENABLE_WRITE** | Log | `false` | |
273
+ | **PATH** | Audit | `./audits` | |
270
274
  | **ENABLE_WRITE** | Audit | `true` | A flag that enable logging object saving log to its destination. |
271
275
  | **MAX_PROCESS** | App | `2` | The maximum process worker number that run in scheduler app module. |
272
276
  | **MAX_SCHEDULE_PER_PROCESS** | App | `100` | A schedule per process that run parallel. |
ddeutil_workflow-0.0.35.dist-info/RECORD ADDED
@@ -0,0 +1,30 @@
1
+ ddeutil/workflow/__about__.py,sha256=nntH3Ja8ABeB5HcYg4Fy7-Z6jDBj67mjILrr2_dJHiw,28
2
+ ddeutil/workflow/__cron.py,sha256=3i-wmjTlh0ADCzN9pLKaWHzJkXzC72aIBmVEQSbyCCE,26895
3
+ ddeutil/workflow/__init__.py,sha256=SX48GHuEMoxJiKKXbBWT8IxcIk9AnibB-T8cUj1Dx1k,1818
4
+ ddeutil/workflow/__types.py,sha256=CK1jfzyHP9P-MB0ElhpJZ59ZFGJC9MkQuAop5739_9k,4304
5
+ ddeutil/workflow/audit.py,sha256=kWymGFEy4WBUXDez1QLRU3FvqLMYQhKJHAzAvIms_Vk,8077
6
+ ddeutil/workflow/caller.py,sha256=qNfrr0B2ykLm6Y8JfhU1rnf5XGmp-usXrPmXM5uGRnM,5473
7
+ ddeutil/workflow/conf.py,sha256=cFc2cd_SGXg9PMrkvCT7WWE85a5UN-DdH53_JIbFyzs,14031
8
+ ddeutil/workflow/cron.py,sha256=j8EeoHst70toRfnD_frix41vrI-eLYVJkZ9yeJtpfnI,8871
9
+ ddeutil/workflow/exceptions.py,sha256=5ghT443VLq0IeU87loHNEqqrrrctklP7YfxwJ51ImWU,949
10
+ ddeutil/workflow/job.py,sha256=iUrmNbpBCpxMJyK_hAxpnyLBtrVhyDCoua4X12CH_BI,24702
11
+ ddeutil/workflow/logs.py,sha256=fmtZl7lQYhKre_auADN64jqD2cqN4aMkNUZdMdhKioM,6398
12
+ ddeutil/workflow/params.py,sha256=bqFDOcfAGcZ-HYa2n_5tO31uw3KXc66wHpUZ5cgpsTk,6702
13
+ ddeutil/workflow/result.py,sha256=iuMvKv5OAzKWDbQf3yfrMHNEse6VdqvsKjv-DXKk-aQ,4349
14
+ ddeutil/workflow/scheduler.py,sha256=DTD5HbCB8oQaxvAQ51-hvpec-NmDIYpyAXsMTODWuoc,25401
15
+ ddeutil/workflow/stages.py,sha256=4SJcseBw4hv9WahJIXfaPdgn9-1Rbti_awMxO-AiP8s,26438
16
+ ddeutil/workflow/templates.py,sha256=A0JgZFGkBv-AX-EskZj656nG5zFd3j1PpLpyXihf6Xg,10967
17
+ ddeutil/workflow/utils.py,sha256=MctJmvklTYtiqZ-nZ7fazQeDoe77UvU0YUEqQZSlbCs,7225
18
+ ddeutil/workflow/workflow.py,sha256=-n-8C0P-SY0BRc1ak6YhP_J2tT924O6ZkCnDJQFu-z8,45156
19
+ ddeutil/workflow/api/__init__.py,sha256=F53NMBWtb9IKaDWkPU5KvybGGfKAcbehgn6TLBwHuuM,21
20
+ ddeutil/workflow/api/api.py,sha256=q7wE0CDPCzpwmhnqKO5qSVO2Mjl1folqwF5E4W-l_UE,4042
21
+ ddeutil/workflow/api/repeat.py,sha256=zyvsrXKk-3-_N8ZRZSki0Mueshugum2jtqctEOp9QSc,4927
22
+ ddeutil/workflow/api/routes/__init__.py,sha256=HRUg4yB3023Iml8FQKuY0X6u9FavJe-HqEl6V8N_4hs,450
23
+ ddeutil/workflow/api/routes/logs.py,sha256=uC8daeCrH_vpVJ-9Um5dMGfHR8JVtwwrDH5I1-Vp6Sw,971
24
+ ddeutil/workflow/api/routes/schedules.py,sha256=9Q4cPYQWOyiZ1lnanpwtIaLWQtLRX1Vwx7uE30cd_1w,4644
25
+ ddeutil/workflow/api/routes/workflows.py,sha256=LQiLlB3tESUtOZWcuLUKIHKvnit_yX3fKzYB4FqhavI,4354
26
+ ddeutil_workflow-0.0.35.dist-info/LICENSE,sha256=nGFZ1QEhhhWeMHf9n99_fdt4vQaXS29xWKxt-OcLywk,1085
27
+ ddeutil_workflow-0.0.35.dist-info/METADATA,sha256=FGRFLyia45WdWsiJin8fuTmLoD5-MqKDO63tflCJCzI,19269
28
+ ddeutil_workflow-0.0.35.dist-info/WHEEL,sha256=nn6H5-ilmfVryoAQl3ZQ2l8SH5imPWFpm1A5FgEuFV4,91
29
+ ddeutil_workflow-0.0.35.dist-info/top_level.txt,sha256=m9M6XeSWDwt_yMsmH6gcOjHZVK5O0-vgtNBuncHjzW4,8
30
+ ddeutil_workflow-0.0.35.dist-info/RECORD,,
{ddeutil_workflow-0.0.33.dist-info → ddeutil_workflow-0.0.35.dist-info}/WHEEL RENAMED
@@ -1,5 +1,5 @@
1
1
  Wheel-Version: 1.0
2
- Generator: setuptools (75.8.0)
2
+ Generator: setuptools (75.8.1)
3
3
  Root-Is-Purelib: true
4
4
  Tag: py3-none-any
5
5
 
ddeutil_workflow-0.0.33.dist-info/RECORD DELETED
@@ -1,26 +0,0 @@
1
- ddeutil/workflow/__about__.py,sha256=3CTb0QhYG3rUiN5ms0KRCjX1IdXetGlrmdY7SfsFOQQ,28
2
- ddeutil/workflow/__cron.py,sha256=3i-wmjTlh0ADCzN9pLKaWHzJkXzC72aIBmVEQSbyCCE,26895
3
- ddeutil/workflow/__init__.py,sha256=Mo7M7-PU_brXSZJ9JSylPoNi7-njZxD3jr3BenTaB9k,1705
4
- ddeutil/workflow/__types.py,sha256=CK1jfzyHP9P-MB0ElhpJZ59ZFGJC9MkQuAop5739_9k,4304
5
- ddeutil/workflow/audit.py,sha256=i1He3T44nPrUabpof_ACQB0azEKucXxjlG3lwOpN3hA,8289
6
- ddeutil/workflow/conf.py,sha256=4ndpWsdvLS7_Ae0udxU0MX3Uow2PpTGxV4F06GZ5-KQ,12985
7
- ddeutil/workflow/cron.py,sha256=j8EeoHst70toRfnD_frix41vrI-eLYVJkZ9yeJtpfnI,8871
8
- ddeutil/workflow/exceptions.py,sha256=5ghT443VLq0IeU87loHNEqqrrrctklP7YfxwJ51ImWU,949
9
- ddeutil/workflow/hook.py,sha256=MgZFlTGvaRSBrTouZGlxwYpKQoKDOT26PNhESeL3LY0,5469
10
- ddeutil/workflow/job.py,sha256=QhB1fw17EgAyltLLComkmD4Ao3mHKHrF2h6A1He1TSQ,24584
11
- ddeutil/workflow/params.py,sha256=LKR7jXyxTb5NVrFav_fl2y9xo3p7qL1S9h-i6CtvNwE,5851
12
- ddeutil/workflow/result.py,sha256=XOs4VzZb3HJr2hXzA3rPPAB1fG2cWGMR-Btjo-qiFYE,4567
13
- ddeutil/workflow/scheduler.py,sha256=MODsAYg0aP3AJX45O_PT7_XzIDA3_dGuR_8Q1LETv60,24430
14
- ddeutil/workflow/stage.py,sha256=ct9kqFKxnS2eJZv-ZMjKP1LROivooBDVwcMm8dlGMjk,24705
15
- ddeutil/workflow/templates.py,sha256=A0JgZFGkBv-AX-EskZj656nG5zFd3j1PpLpyXihf6Xg,10967
16
- ddeutil/workflow/utils.py,sha256=rTDQKaaber7cRqzJjWpCP9OTbarti1UMKdLgH6VRjFM,6709
17
- ddeutil/workflow/workflow.py,sha256=HfUKAZo7LMs6Mn1mGSw8-D8To4os-sbNuux9RjhSyCY,43687
18
- ddeutil/workflow/api/__init__.py,sha256=F53NMBWtb9IKaDWkPU5KvybGGfKAcbehgn6TLBwHuuM,21
19
- ddeutil/workflow/api/api.py,sha256=TfgJtu-yREsrRveLcqTjxoJszuhq21qKkl4oyQpSzvQ,3959
20
- ddeutil/workflow/api/repeat.py,sha256=zyvsrXKk-3-_N8ZRZSki0Mueshugum2jtqctEOp9QSc,4927
21
- ddeutil/workflow/api/route.py,sha256=XHPw9IKiVLPWYl937u09s0_Kd6rolWK9TbuIN0RFkfA,8625
22
- ddeutil_workflow-0.0.33.dist-info/LICENSE,sha256=nGFZ1QEhhhWeMHf9n99_fdt4vQaXS29xWKxt-OcLywk,1085
23
- ddeutil_workflow-0.0.33.dist-info/METADATA,sha256=syJ1Sk2H6sKxsVTLxJeQr5oIMb4oozPFQ_lfz5INEiU,18887
24
- ddeutil_workflow-0.0.33.dist-info/WHEEL,sha256=In9FTNxeP60KnTkGw7wk6mJPYd_dQSjEZmXdBdMCI-8,91
25
- ddeutil_workflow-0.0.33.dist-info/top_level.txt,sha256=m9M6XeSWDwt_yMsmH6gcOjHZVK5O0-vgtNBuncHjzW4,8
26
- ddeutil_workflow-0.0.33.dist-info/RECORD,,