ddeutil-workflow 0.0.41__py3-none-any.whl → 0.0.43__py3-none-any.whl

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (24) hide show
  1. ddeutil/workflow/__about__.py +1 -1
  2. ddeutil/workflow/__init__.py +5 -1
  3. ddeutil/workflow/api/api.py +7 -7
  4. ddeutil/workflow/api/routes/schedules.py +5 -5
  5. ddeutil/workflow/api/routes/workflows.py +2 -2
  6. ddeutil/workflow/conf.py +39 -28
  7. ddeutil/workflow/cron.py +12 -13
  8. ddeutil/workflow/exceptions.py +13 -3
  9. ddeutil/workflow/job.py +40 -42
  10. ddeutil/workflow/logs.py +33 -6
  11. ddeutil/workflow/params.py +77 -18
  12. ddeutil/workflow/result.py +36 -8
  13. ddeutil/workflow/reusables.py +16 -13
  14. ddeutil/workflow/scheduler.py +32 -37
  15. ddeutil/workflow/stages.py +285 -120
  16. ddeutil/workflow/utils.py +0 -1
  17. ddeutil/workflow/workflow.py +127 -90
  18. {ddeutil_workflow-0.0.41.dist-info → ddeutil_workflow-0.0.43.dist-info}/METADATA +29 -32
  19. ddeutil_workflow-0.0.43.dist-info/RECORD +30 -0
  20. ddeutil/workflow/context.py +0 -61
  21. ddeutil_workflow-0.0.41.dist-info/RECORD +0 -31
  22. {ddeutil_workflow-0.0.41.dist-info → ddeutil_workflow-0.0.43.dist-info}/WHEEL +0 -0
  23. {ddeutil_workflow-0.0.41.dist-info → ddeutil_workflow-0.0.43.dist-info}/licenses/LICENSE +0 -0
  24. {ddeutil_workflow-0.0.41.dist-info → ddeutil_workflow-0.0.43.dist-info}/top_level.txt +0 -0
{ddeutil_workflow-0.0.41.dist-info → ddeutil_workflow-0.0.43.dist-info}/METADATA RENAMED
@@ -1,6 +1,6 @@
1
1
  Metadata-Version: 2.4
2
2
  Name: ddeutil-workflow
3
- Version: 0.0.41
3
+ Version: 0.0.43
4
4
  Summary: Lightweight workflow orchestration
5
5
  Author-email: ddeutils <korawich.anu@gmail.com>
6
6
  License: MIT
@@ -22,7 +22,7 @@ Classifier: Programming Language :: Python :: 3.13
22
22
  Requires-Python: >=3.9.13
23
23
  Description-Content-Type: text/markdown
24
24
  License-File: LICENSE
25
- Requires-Dist: ddeutil>=0.4.6
25
+ Requires-Dist: ddeutil[checksum]>=0.4.6
26
26
  Requires-Dist: ddeutil-io[toml,yaml]>=0.2.10
27
27
  Requires-Dist: pydantic==2.11.1
28
28
  Requires-Dist: python-dotenv==1.1.0
@@ -212,7 +212,7 @@ execution time such as `run-date` should change base on that workflow running da
212
212
  ```python
213
213
  from ddeutil.workflow import Workflow, Result
214
214
 
215
- workflow: Workflow = Workflow.from_loader('run-py-local')
215
+ workflow: Workflow = Workflow.from_conf('run-py-local')
216
216
  result: Result = workflow.execute(
217
217
  params={"source-extract": "USD-THB", "asat-dt": "2024-01-01"}
218
218
  )
@@ -246,7 +246,7 @@ from ddeutil.workflow import Schedule
246
246
 
247
247
  (
248
248
  Schedule
249
- .from_loader("schedule-run-local-wf")
249
+ .from_conf("schedule-run-local-wf")
250
250
  .pending(stop=None)
251
251
  )
252
252
  ```
@@ -261,34 +261,31 @@ it will use default value and do not raise any error to you.
261
261
  > The config value that you will set on the environment should combine with
262
262
  > prefix, component, and name which is `WORKFLOW_{component}_{name}` (Upper case).
263
263
 
264
- | Name | Component | Default | Description |
265
- |:-----------------------------|:---------:|:--------------------------------------------------------------------------------------------------------------------------------|:-------------------------------------------------------------------------------------------------------------------|
266
- | **ROOT_PATH** | Core | `.` | The root path of the workflow application. |
267
- | **REGISTRY** | Core | `.` | List of importable string for the call stage. |
268
- | **REGISTRY_FILTER** | Core | `ddeutil.workflow.templates` | List of importable string for the filter template. |
269
- | **CONF_PATH** | Core | `conf` | The config path that keep all template `.yaml` files. |
270
- | **TIMEZONE** | Core | `Asia/Bangkok` | A Timezone string value that will pass to `ZoneInfo` object. |
271
- | **STAGE_DEFAULT_ID** | Core | `true` | A flag that enable default stage ID that use for catch an execution output. |
272
- | **STAGE_RAISE_ERROR** | Core | `false` | A flag that all stage raise StageException from stage execution. |
273
- | **JOB_DEFAULT_ID** | Core | `false` | A flag that enable default job ID that use for catch an execution output. The ID that use will be sequence number. |
274
- | **JOB_RAISE_ERROR** | Core | `true` | A flag that all job raise JobException from job strategy execution. |
275
- | **MAX_NUM_POKING** | Core | `4` | . |
276
- | **MAX_JOB_PARALLEL** | Core | `2` | The maximum job number that able to run parallel in workflow executor. |
277
- | **MAX_JOB_EXEC_TIMEOUT** | Core | `600` | |
278
- | **MAX_CRON_PER_WORKFLOW** | Core | `5` | |
279
- | **MAX_QUEUE_COMPLETE_HIST** | Core | `16` | |
280
- | **GENERATE_ID_SIMPLE_MODE** | Core | `true` | A flog that enable generating ID with `md5` algorithm. |
281
- | **PATH** | Log | `./logs` | The log path of the workflow saving log. |
282
- | **DEBUG_MODE** | Log | `true` | A flag that enable logging with debug level mode. |
283
- | **FORMAT** | Log | `%(asctime)s.%(msecs)03d (%(name)-10s, %(process)-5d,%(thread)-5d) [%(levelname)-7s] %(message)-120s (%(filename)s:%(lineno)s)` | |
284
- | **FORMAT_FILE** | Log | `{datetime} ({process:5d}, {thread:5d}) {message:120s} ({filename}:{lineno})` | |
285
- | **DATETIME_FORMAT** | Log | `%Y-%m-%d %H:%M:%S` | |
286
- | **ENABLE_WRITE** | Log | `false` | |
287
- | **PATH** | Audit | `./audits` | |
288
- | **ENABLE_WRITE** | Audit | `true` | A flag that enable logging object saving log to its destination. |
289
- | **MAX_PROCESS** | App | `2` | The maximum process worker number that run in scheduler app module. |
290
- | **MAX_SCHEDULE_PER_PROCESS** | App | `100` | A schedule per process that run parallel. |
291
- | **STOP_BOUNDARY_DELTA** | App | `'{"minutes": 5, "seconds": 20}'` | A time delta value that use to stop scheduler app in json string format. |
264
+ | Name | Component | Default | Override | Description |
265
+ |:-----------------------------|:---------:|:--------------------------------------------------------------------------------------------------------------------------------|:--------:|:-------------------------------------------------------------------------------------------------------------------|
266
+ | **ROOT_PATH** | Core | `.` | No | The root path of the workflow application. |
267
+ | **REGISTRY_CALLER** | Core | `.` | Yes | List of importable string for the call stage. |
268
+ | **REGISTRY_FILTER** | Core | `ddeutil.workflow.templates` | Yes | List of importable string for the filter template. |
269
+ | **CONF_PATH** | Core | `conf` | No | The config path that keep all template `.yaml` files. |
270
+ | **TIMEZONE** | Core | `Asia/Bangkok` | No | A Timezone string value that will pass to `ZoneInfo` object. |
271
+ | **STAGE_DEFAULT_ID** | Core | `true` | No | A flag that enable default stage ID that use for catch an execution output. |
272
+ | **STAGE_RAISE_ERROR** | Core | `false` | Yes | A flag that all stage raise StageException from stage execution. |
273
+ | **JOB_DEFAULT_ID** | Core | `false` | No | A flag that enable default job ID that use for catch an execution output. The ID that use will be sequence number. |
274
+ | **JOB_RAISE_ERROR** | Core | `true` | Yes | A flag that all job raise JobException from job strategy execution. |
275
+ | **MAX_CRON_PER_WORKFLOW** | Core | `5` | No | |
276
+ | **MAX_QUEUE_COMPLETE_HIST** | Core | `16` | No | |
277
+ | **GENERATE_ID_SIMPLE_MODE** | Core | `true` | No | A flog that enable generating ID with `md5` algorithm. |
278
+ | **DEBUG_MODE** | Log | `true` | No | A flag that enable logging with debug level mode. |
279
+ | **FORMAT** | Log | `%(asctime)s.%(msecs)03d (%(name)-10s, %(process)-5d,%(thread)-5d) [%(levelname)-7s] %(message)-120s (%(filename)s:%(lineno)s)` | No | |
280
+ | **FORMAT_FILE** | Log | `{datetime} ({process:5d}, {thread:5d}) {message:120s} ({filename}:{lineno})` | No | |
281
+ | **DATETIME_FORMAT** | Log | `%Y-%m-%d %H:%M:%S` | No | |
282
+ | **TRACE_PATH** | Log | `./logs` | No | The log path of the workflow saving log. |
283
+ | **TRACE_ENABLE_WRITE** | Log | `false` | No | |
284
+ | **AUDIT_PATH** | Log | `./audits` | No | |
285
+ | **AUDIT_ENABLE_WRITE** | Log | `true` | No | A flag that enable logging object saving log to its destination. |
286
+ | **MAX_PROCESS** | App | `2` | No | The maximum process worker number that run in scheduler app module. |
287
+ | **MAX_SCHEDULE_PER_PROCESS** | App | `100` | No | A schedule per process that run parallel. |
288
+ | **STOP_BOUNDARY_DELTA** | App | `'{"minutes": 5, "seconds": 20}'` | No | A time delta value that use to stop scheduler app in json string format. |
292
289
 
293
290
  **API Application**:
294
291
 
ddeutil_workflow-0.0.43.dist-info/RECORD ADDED
@@ -0,0 +1,30 @@
1
+ ddeutil/workflow/__about__.py,sha256=feByUkEJfWx0d1Lb9oGvd2f3clHHuGXDT7T7Ki5zoEA,28
2
+ ddeutil/workflow/__cron.py,sha256=h8rLeIUAAEB2SdZ4Jhch7LU1Yl3bbJ-iNNJ3tQ0eYVM,28095
3
+ ddeutil/workflow/__init__.py,sha256=m7ZTCuUOarcTKJuXOyuaXd5WTIO7NTkqCeCrNX3d5i8,1943
4
+ ddeutil/workflow/__types.py,sha256=8jBdbfb3aZSetjz0mvNrpGHwwxJff7mK8_4v41cLqlc,4316
5
+ ddeutil/workflow/conf.py,sha256=lDzWiVSNlNAhTzxbNIhIbQAIF1ggbmetAp0yn2fgnsc,12385
6
+ ddeutil/workflow/cron.py,sha256=80SijzMdDOBxTWRsiF-Fmuz7Ym7leY0XT2lzRAPGdXc,8781
7
+ ddeutil/workflow/exceptions.py,sha256=uLNxzav3HRcr4vaZnvbUIF_eTR6UXXZNaxroMWFOUL4,1418
8
+ ddeutil/workflow/job.py,sha256=6CqLb1F_z3mHU_rOVRY7Z2V2B1tY3p7FZIJgb8DM1YE,29969
9
+ ddeutil/workflow/logs.py,sha256=RkM5o_JPoWhFY7NrbYAARZQWjLC62YB_FYzTTcyDp8U,19816
10
+ ddeutil/workflow/params.py,sha256=xCtFEh0-G-G-f8y_SXxyf31bU6Ox5p5Z-WbBFXrjy8M,9960
11
+ ddeutil/workflow/result.py,sha256=iwkUzOubxhLCuO-ngWEWL6t-CpYBpINIIO_ubg4kz14,4701
12
+ ddeutil/workflow/reusables.py,sha256=AtZO83HDFu1uK_azUinv5d8jsA36f2i3n_tqMrolbvc,17529
13
+ ddeutil/workflow/scheduler.py,sha256=_MDsEHbBVOeF-381U8DfIMDyca_nG3XNXmgX4229_EU,27437
14
+ ddeutil/workflow/stages.py,sha256=RMyOU9KqPhS-49kQduT6-iysGJZFwX0Cbgo9o8kzIAY,47304
15
+ ddeutil/workflow/utils.py,sha256=sblje9qOtejCHVt8EVrbC0KY98vKqvxccaR5HIkRiTA,7363
16
+ ddeutil/workflow/workflow.py,sha256=LPZzGNf55Tca0XXQZytTiyWbocC1T47cB_XsWpiXdyc,50482
17
+ ddeutil/workflow/api/__init__.py,sha256=F53NMBWtb9IKaDWkPU5KvybGGfKAcbehgn6TLBwHuuM,21
18
+ ddeutil/workflow/api/api.py,sha256=b-bMg0aRsEqt8Qb2hNUtamEt2Fq2CgNotF2oXSAdDu8,5226
19
+ ddeutil/workflow/api/log.py,sha256=NMTnOnsBrDB5129329xF2myLdrb-z9k1MQrmrP7qXJw,1818
20
+ ddeutil/workflow/api/repeat.py,sha256=cycd1-91j-4v6uY1SkrZHd9l95e-YgVC4UCSNNFuGJ8,5277
21
+ ddeutil/workflow/api/routes/__init__.py,sha256=qoGtOMyVgQ5nTUc8J8wH27A8isaxl3IFCX8qoyibeCY,484
22
+ ddeutil/workflow/api/routes/job.py,sha256=YVta083i8vU8-o4WdKFwDpfdC9vN1dZ6goZSmNlQXHA,1954
23
+ ddeutil/workflow/api/routes/logs.py,sha256=TeRDrEelbKS2Hu_EovgLh0bOdmSv9mfnrIZsrE7uPD4,5353
24
+ ddeutil/workflow/api/routes/schedules.py,sha256=rUWBm5RgLS1PNBHSWwWXJ0l-c5mYWfl9os0BA9_OTEw,4810
25
+ ddeutil/workflow/api/routes/workflows.py,sha256=ctgQGxXfpIV6bHFDM9IQ1_qaQHT6n5-HjJ1-D4GKWpc,4527
26
+ ddeutil_workflow-0.0.43.dist-info/licenses/LICENSE,sha256=nGFZ1QEhhhWeMHf9n99_fdt4vQaXS29xWKxt-OcLywk,1085
27
+ ddeutil_workflow-0.0.43.dist-info/METADATA,sha256=eOwkhUh-w_bSW0LOlOCielBQTg0mvf2gEkkNMlbwIU4,19134
28
+ ddeutil_workflow-0.0.43.dist-info/WHEEL,sha256=CmyFI0kx5cdEMTLiONQRbGQwjIoR1aIYB7eCAQ4KPJ0,91
29
+ ddeutil_workflow-0.0.43.dist-info/top_level.txt,sha256=m9M6XeSWDwt_yMsmH6gcOjHZVK5O0-vgtNBuncHjzW4,8
30
+ ddeutil_workflow-0.0.43.dist-info/RECORD,,
ddeutil/workflow/context.py DELETED
@@ -1,61 +0,0 @@
1
- from __future__ import annotations
2
-
3
- from typing import Optional, Union
4
-
5
- from pydantic import BaseModel, ConfigDict, Field
6
-
7
- from .__types import DictData
8
-
9
-
10
- class ErrorContext(BaseModel): # pragma: no cov
11
- model_config = ConfigDict(arbitrary_types_allowed=True)
12
-
13
- obj: Exception = Field(alias="class")
14
- name: str = Field(description="A name of exception class.")
15
- message: str = Field(description="A exception message.")
16
-
17
-
18
- class OutputContext(BaseModel): # pragma: no cov
19
- outputs: DictData = Field(default_factory=dict)
20
- errors: Optional[ErrorContext] = Field(default=None)
21
- skipped: bool = Field(default=False)
22
-
23
- def is_exception(self) -> bool:
24
- return self.errors is not None
25
-
26
-
27
- class StageContext(BaseModel): # pragma: no cov
28
- stages: dict[str, OutputContext]
29
- errors: Optional[ErrorContext] = Field(default=None)
30
-
31
- def is_exception(self) -> bool:
32
- return self.errors is not None
33
-
34
-
35
- class MatrixContext(StageContext): # pragma: no cov
36
- matrix: DictData = Field(default_factory=dict)
37
-
38
-
39
- MatrixStageContext = dict[
40
- str, Union[MatrixContext, StageContext]
41
- ] # pragma: no cov
42
-
43
-
44
- class StrategyContext(BaseModel): # pragma: no cov
45
- strategies: MatrixStageContext
46
- errors: Optional[ErrorContext] = Field(default=None)
47
-
48
- def is_exception(self) -> bool:
49
- return self.errors is not None
50
-
51
-
52
- StrategyMatrixContext = Union[
53
- StrategyContext, MatrixStageContext
54
- ] # pragma: no cov
55
-
56
-
57
- class JobContext(BaseModel): # pragma: no cov
58
- params: DictData = Field(description="A parameterize value")
59
- jobs: dict[str, StrategyMatrixContext]
60
- errors: Optional[ErrorContext] = Field(default=None)
61
- skipped: bool = Field(default=False)
ddeutil_workflow-0.0.41.dist-info/RECORD DELETED
@@ -1,31 +0,0 @@
1
- ddeutil/workflow/__about__.py,sha256=chYiprva6VKQg6XFgrwBoMIWSlSshsb-yyHtFoiipvc,28
2
- ddeutil/workflow/__cron.py,sha256=h8rLeIUAAEB2SdZ4Jhch7LU1Yl3bbJ-iNNJ3tQ0eYVM,28095
3
- ddeutil/workflow/__init__.py,sha256=cYWwG2utpsYvdwqvkFSRWi_Q6gylDgNQBcIWcF5NFs4,1861
4
- ddeutil/workflow/__types.py,sha256=8jBdbfb3aZSetjz0mvNrpGHwwxJff7mK8_4v41cLqlc,4316
5
- ddeutil/workflow/conf.py,sha256=wQXL5bfEUx8DbS6cEIRtsDjvQZzw7sGPvs-g5r2zOeM,12095
6
- ddeutil/workflow/context.py,sha256=vsk4JQL7t3KsnKPfshw3O7YrPFo2h4rnnNd3B-G9Kj4,1700
7
- ddeutil/workflow/cron.py,sha256=j8EeoHst70toRfnD_frix41vrI-eLYVJkZ9yeJtpfnI,8871
8
- ddeutil/workflow/exceptions.py,sha256=fO37f9p7lOjIJgVOpKE_1X44yJTwBepyukZV9a7NNm4,1241
9
- ddeutil/workflow/job.py,sha256=TAtOFOFWHGqXlc6QWvtum1KpfWFDs5AfNbYMtwvrIE8,30102
10
- ddeutil/workflow/logs.py,sha256=nbyoUONqSZ4QpowgMs962m-Qs-UDfcyMHs4bxIAT470,18782
11
- ddeutil/workflow/params.py,sha256=Mv-D2DY5inm1ug0lsgCPDkO5wT_AUhc5XEF5jxgDx6U,8036
12
- ddeutil/workflow/result.py,sha256=ynZB0g_vEEXn24034J-hatjNWDBmRAj38S8SqGRM-8I,4029
13
- ddeutil/workflow/reusables.py,sha256=Rw7qS2cQM4SBxuIyqiyJN5yZNBfObISXWRIgDUMC2fY,17449
14
- ddeutil/workflow/scheduler.py,sha256=Y4VuVDIHz64l3IN-7tKP76qrMKShpgXXjX64mKjSWLo,27651
15
- ddeutil/workflow/stages.py,sha256=Kp3VLhKMMlgQyMLEYWJfBdR8DoCD0nuzSY0-r-P8eS8,41574
16
- ddeutil/workflow/utils.py,sha256=XwkxOpPaHbvjCKCGA3kVriEYabKyZ_P6pTbkYPnK704,7380
17
- ddeutil/workflow/workflow.py,sha256=s2z_QMTmfsRjiw25CuKFQIqY9smCbKjFwbBwCZkikfc,49615
18
- ddeutil/workflow/api/__init__.py,sha256=F53NMBWtb9IKaDWkPU5KvybGGfKAcbehgn6TLBwHuuM,21
19
- ddeutil/workflow/api/api.py,sha256=C9f6w11zE1SHz8lwjRxVOwrO90pCMr9REj2WLQsO0lI,5190
20
- ddeutil/workflow/api/log.py,sha256=NMTnOnsBrDB5129329xF2myLdrb-z9k1MQrmrP7qXJw,1818
21
- ddeutil/workflow/api/repeat.py,sha256=cycd1-91j-4v6uY1SkrZHd9l95e-YgVC4UCSNNFuGJ8,5277
22
- ddeutil/workflow/api/routes/__init__.py,sha256=qoGtOMyVgQ5nTUc8J8wH27A8isaxl3IFCX8qoyibeCY,484
23
- ddeutil/workflow/api/routes/job.py,sha256=YVta083i8vU8-o4WdKFwDpfdC9vN1dZ6goZSmNlQXHA,1954
24
- ddeutil/workflow/api/routes/logs.py,sha256=TeRDrEelbKS2Hu_EovgLh0bOdmSv9mfnrIZsrE7uPD4,5353
25
- ddeutil/workflow/api/routes/schedules.py,sha256=uWYDOwlV8w56hKQmfkQFwdZ6t2gZSJeCdBIzMmJenAQ,4824
26
- ddeutil/workflow/api/routes/workflows.py,sha256=TiGlwosHucccVmGv_SE9mvs3_BR2c6QZWQhw--aAv5w,4537
27
- ddeutil_workflow-0.0.41.dist-info/licenses/LICENSE,sha256=nGFZ1QEhhhWeMHf9n99_fdt4vQaXS29xWKxt-OcLywk,1085
28
- ddeutil_workflow-0.0.41.dist-info/METADATA,sha256=ACsWRIhGZuaRo_d400dlAFsSRlNcha8Vy0PzVMkE3NU,19729
29
- ddeutil_workflow-0.0.41.dist-info/WHEEL,sha256=CmyFI0kx5cdEMTLiONQRbGQwjIoR1aIYB7eCAQ4KPJ0,91
30
- ddeutil_workflow-0.0.41.dist-info/top_level.txt,sha256=m9M6XeSWDwt_yMsmH6gcOjHZVK5O0-vgtNBuncHjzW4,8
31
- ddeutil_workflow-0.0.41.dist-info/RECORD,,