ddeutil-workflow 0.0.49__py3-none-any.whl → 0.0.51__py3-none-any.whl

{ddeutil_workflow-0.0.49.dist-info → ddeutil_workflow-0.0.51.dist-info}/METADATA

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: ddeutil-workflow
-Version: 0.0.49
+Version: 0.0.51
 Summary: Lightweight workflow orchestration
 Author-email: ddeutils <korawich.anu@gmail.com>
 License: MIT
@@ -22,8 +22,8 @@ Classifier: Programming Language :: Python :: 3.13
 Requires-Python: >=3.9.13
 Description-Content-Type: text/markdown
 License-File: LICENSE
-Requires-Dist: ddeutil[checksum]>=0.4.6
-Requires-Dist: ddeutil-io[toml,yaml]>=0.2.10
+Requires-Dist: ddeutil[checksum]>=0.4.7
+Requires-Dist: ddeutil-io[toml,yaml]>=0.2.11
 Requires-Dist: pydantic==2.11.1
 Requires-Dist: python-dotenv==1.1.0
 Requires-Dist: schedule<2.0.0,==1.2.2
@@ -39,6 +39,8 @@ Requires-Dist: ujson; extra == "api"
 Provides-Extra: async
 Requires-Dist: aiofiles; extra == "async"
 Requires-Dist: aiohttp; extra == "async"
+Provides-Extra: docker
+Requires-Dist: docker==7.1.0; extra == "docker"
 Dynamic: license-file
 
 # Workflow Orchestration
@@ -137,7 +139,8 @@ flowchart LR
 
 ## 📦 Installation
 
-This project need `ddeutil` and `ddeutil-io` extension namespace packages.
+This project need `ddeutil` and `ddeutil-io` extension namespace packages to be
+the base deps.
 If you want to install this package with application add-ons, you should add
 `app` in installation;
 
@@ -146,7 +149,7 @@ If you want to install this package with application add-ons, you should add
 | Python         | `ddeutil-workflow`       | :heavy_check_mark:  |
 | FastAPI Server | `ddeutil-workflow[api]`  | :heavy_check_mark:  |
 
-## :beers: Usage
+## 🎯 Usage
 
 This is examples that use workflow file for running common Data Engineering
 use-case.
@@ -196,12 +199,54 @@ run-py-local:
 
                  # Arguments of target data that want to land.
                  writing_mode: flatten
-                 aws_s3_path: my-data/open-data/${{ params.source-extract }}
+                 aws:
+                    path: my-data/open-data/${{ params.source-extract }}
 
-                 # This Authentication code should implement with your custom call
-                 # function. The template allow you to use environment variable.
-                 aws_access_client_id: ${AWS_ACCESS_CLIENT_ID}
-                 aws_access_client_secret: ${AWS_ACCESS_CLIENT_SECRET}
+                    # This Authentication code should implement with your custom call
+                    # function. The template allow you to use environment variable.
+                    access_client_id: ${AWS_ACCESS_CLIENT_ID}
+                    access_client_secret: ${AWS_ACCESS_CLIENT_SECRET}
+```
+
+Before execute this workflow, you should implement caller function first.
+
+```text
+registry-caller/
+  ╰─ tasks.py
+```
+
+This function will store as module that will import from `WORKFLOW_CORE_REGISTRY_CALLER`
+value (This config can override by extra parameters with `registry_caller` key).
+
+```python
+from ddeutil.workflow import Result, tag
+from ddeutil.workflow.exceptions import StageException
+from pydantic import BaseModel, SecretStr
+
+class AwsCredential(BaseModel):
+    path: str
+    access_client_id: str
+    access_client_secret: SecretStr
+
+class RestAuth(BaseModel):
+    type: str
+    keys: SecretStr
+
+@tag("requests", alias="get-api-with-oauth-to-s3")
+def get_api_with_oauth_to_s3(
+    method: str,
+    url: str,
+    body: dict[str, str],
+    auth: RestAuth,
+    writing_node: str,
+    aws: AwsCredential,
+    result: Result,
+) -> dict[str, int]:
+    result.trace.info("[CALLER]: Start get data via RestAPI to S3.")
+    result.trace.info(f"... {method}: {url}")
+    if method != "post":
+       raise StageException(f"RestAPI does not support for {method} action.")
+    return {"records": 1000}
 ```
 
 The above workflow template is main executor pipeline that you want to do. If you
@@ -268,7 +313,6 @@ it will use default value and do not raise any error to you.
 | **TIMEZONE**                 |   Core    | `Asia/Bangkok`                                                                                                                  | A Timezone string value that will pass to `ZoneInfo` object.                                                       |
 | **STAGE_DEFAULT_ID**         |   Core    | `false`                                                                                                                         | A flag that enable default stage ID that use for catch an execution output.                                        |
 | **STAGE_RAISE_ERROR**        |   Core    | `false`                                                                                                                         | A flag that all stage raise StageException from stage execution.                                                   |
-| **JOB_RAISE_ERROR**          |   Core    | `true`                                                                                                                          | A flag that all job raise JobException from job strategy execution.                                                |
 | **MAX_CRON_PER_WORKFLOW**    |   Core    | `5`                                                                                                                             |                                                                                                                    |
 | **MAX_QUEUE_COMPLETE_HIST**  |   Core    | `16`                                                                                                                            |                                                                                                                    |
 | **GENERATE_ID_SIMPLE_MODE**  |   Core    | `true`                                                                                                                          | A flog that enable generating ID with `md5` algorithm.                                                             |
@@ -297,13 +341,15 @@ only.
 ## :rocket: Deployment
 
 This package able to run as an application service for receive manual trigger
-from the master node via RestAPI or use to be Scheduler background service
-like crontab job but via Python API.
+from any node via RestAPI or use to be Scheduler background application
+like crontab job but via Python API or FastAPI app.
 
 ### API Server
 
+This server use FastAPI package to be the base application.
+
 ```shell
-(venv) $ uvicorn ddeutil.workflow.api:app \
+(.venv) $ uvicorn ddeutil.workflow.api:app \
   --host 127.0.0.1 \
   --port 80 \
   --no-access-log
@@ -313,8 +359,19 @@ like crontab job but via Python API.
 > If this package already deploy, it is able to use multiprocess;
 > `uvicorn ddeutil.workflow.api:app --host 127.0.0.1 --port 80 --workers 4`
 
+### Local Schedule
+
+> [!WARNING]
+> This CLI does not implement yet.
+
+```shell
+(.venv) $ ddeutil-workflow schedule
+```
+
 ### Docker Container
 
+Build a Docker container from this package.
+
 ```shell
 $ docker build -t ddeutil-workflow:latest -f .container/Dockerfile .
 $ docker run -i ddeutil-workflow:latest ddeutil-workflow

ddeutil_workflow-0.0.51.dist-info/RECORD

@@ -0,0 +1,31 @@
+ddeutil/workflow/__about__.py,sha256=lV_4ewekJpV5o1pQoqnMK3g475fbmyO855YKAFNFKF8,28
+ddeutil/workflow/__cron.py,sha256=h8rLeIUAAEB2SdZ4Jhch7LU1Yl3bbJ-iNNJ3tQ0eYVM,28095
+ddeutil/workflow/__init__.py,sha256=AMmM7Cq_jrO_MzIZjzQyFvcFkVj9ad7V1f3PtLrI030,1736
+ddeutil/workflow/__main__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
+ddeutil/workflow/__types.py,sha256=8jBdbfb3aZSetjz0mvNrpGHwwxJff7mK8_4v41cLqlc,4316
+ddeutil/workflow/conf.py,sha256=80rgmJKFU7BlH5xTLnghGzGhE8C6LFAQykd9mjHSjo8,12528
+ddeutil/workflow/cron.py,sha256=WS2MInn0Sp5DKlZDZH5VFZ5AA0Q3_AnBnYEU4lZSv4I,9779
+ddeutil/workflow/exceptions.py,sha256=r4Jrf9qtVPALU4wh4bnb_OYqC-StqSQJEmFC-_QK934,1408
+ddeutil/workflow/job.py,sha256=LydncguMUTTdcG_fImjoPFZBulmZT_PD3jYmnmcYskk,35041
+ddeutil/workflow/logs.py,sha256=rsoBrUGQrooou18fg2yvPsB8NOaXnUA5ThQpBr_WVMg,26598
+ddeutil/workflow/params.py,sha256=FKY4Oo1Ze4QZKRfAk7rqKsi44YaJQAbqAtXM6vlO2hI,11392
+ddeutil/workflow/result.py,sha256=27nPQq9CETLCVczv4vvFEF9w2TllHZ_ROfyDoLFxRWM,5647
+ddeutil/workflow/reusables.py,sha256=iXcS7Gg-71qVX4ln0ILTDx03cTtUnj_rNoXHTVdVrxc,17636
+ddeutil/workflow/scheduler.py,sha256=4G5AogkmnsTKe7jKYSfU35qjubR82WQ8CLtEe9kqPTE,28304
+ddeutil/workflow/stages.py,sha256=Ywwky6rJJ5cMvridG-1Y4rv-wJJ_WMbXysViR3WxQrY,69108
+ddeutil/workflow/utils.py,sha256=zbVttaMFMRLuuBJdSJf7D9qtz8bOnQIBq-rHI3Eqy4M,7821
+ddeutil/workflow/workflow.py,sha256=v_O3Gm4J6SCN_1NfGd3_ClpJkatFCyoqXR3nAIMAoF4,50575
+ddeutil/workflow/api/__init__.py,sha256=F53NMBWtb9IKaDWkPU5KvybGGfKAcbehgn6TLBwHuuM,21
+ddeutil/workflow/api/api.py,sha256=CWtPLgOv2Jus9E7nzG5mG2Z32ZEkUK3JWQ2htZyMRpA,5244
+ddeutil/workflow/api/log.py,sha256=NMTnOnsBrDB5129329xF2myLdrb-z9k1MQrmrP7qXJw,1818
+ddeutil/workflow/api/repeat.py,sha256=uTtUFVLpiYYahXvCVx8sueRQ03K2Xw1id_gW3IMmX1U,5295
+ddeutil/workflow/api/routes/__init__.py,sha256=qoGtOMyVgQ5nTUc8J8wH27A8isaxl3IFCX8qoyibeCY,484
+ddeutil/workflow/api/routes/job.py,sha256=oPwBVP0Mxwxv-bGPlfmxQQ9PcVl0ev9HoPzndpYDCCQ,1954
+ddeutil/workflow/api/routes/logs.py,sha256=U6vOni3wd-ZTOwd3yVdSOpgyRmNdcgfngU5KlLM3Cww,5383
+ddeutil/workflow/api/routes/schedules.py,sha256=EgUjyRGhsm6UNaMj5luh6TcY6l571sCHcla-BL1iOfY,4829
+ddeutil/workflow/api/routes/workflows.py,sha256=JcDOrn1deK8ztFRcMTNATQejG6KMA7JxZLVc4QeBsP4,4527
+ddeutil_workflow-0.0.51.dist-info/licenses/LICENSE,sha256=nGFZ1QEhhhWeMHf9n99_fdt4vQaXS29xWKxt-OcLywk,1085
+ddeutil_workflow-0.0.51.dist-info/METADATA,sha256=xlAbUVZ9djYf1GImH9v0N-mR7XpxcQIkvwGuX5XmSrs,19425
+ddeutil_workflow-0.0.51.dist-info/WHEEL,sha256=CmyFI0kx5cdEMTLiONQRbGQwjIoR1aIYB7eCAQ4KPJ0,91
+ddeutil_workflow-0.0.51.dist-info/top_level.txt,sha256=m9M6XeSWDwt_yMsmH6gcOjHZVK5O0-vgtNBuncHjzW4,8
+ddeutil_workflow-0.0.51.dist-info/RECORD,,

ddeutil_workflow-0.0.49.dist-info/RECORD

@@ -1,31 +0,0 @@
-ddeutil/workflow/__about__.py,sha256=8c8KBEXeEOskazR5AlLYEjCpyi54xsoTaaqRY8pXUJY,28
-ddeutil/workflow/__cron.py,sha256=h8rLeIUAAEB2SdZ4Jhch7LU1Yl3bbJ-iNNJ3tQ0eYVM,28095
-ddeutil/workflow/__init__.py,sha256=EpqROKwAX76Wea-37hi0R_oVQ2jm2cc9ML9USxRAsak,1971
-ddeutil/workflow/__main__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
-ddeutil/workflow/__types.py,sha256=8jBdbfb3aZSetjz0mvNrpGHwwxJff7mK8_4v41cLqlc,4316
-ddeutil/workflow/conf.py,sha256=lviP7bFsOCJtD8S1VyJK6aaSL9Nj_vfC2Kkpe1z2Zec,12444
-ddeutil/workflow/cron.py,sha256=80SijzMdDOBxTWRsiF-Fmuz7Ym7leY0XT2lzRAPGdXc,8781
-ddeutil/workflow/exceptions.py,sha256=uLNxzav3HRcr4vaZnvbUIF_eTR6UXXZNaxroMWFOUL4,1418
-ddeutil/workflow/job.py,sha256=y2q_md2nUp1jfgjaQdjDZqrHR541ENrWB__S1-Eoyss,30830
-ddeutil/workflow/logs.py,sha256=JghQawGd16ysf-5y7ZFtSGFY82uwgb9oMKNYy9eGI-o,26468
-ddeutil/workflow/params.py,sha256=xCtFEh0-G-G-f8y_SXxyf31bU6Ox5p5Z-WbBFXrjy8M,9960
-ddeutil/workflow/result.py,sha256=kizTEP6DY9ewDQQR17YgfrtMXRW-wF8vRzG26wzAqUM,5439
-ddeutil/workflow/reusables.py,sha256=hIpehea6J4OWeXX55kjYzo-c9-_Cc0YRwLRRbcaUkZs,17539
-ddeutil/workflow/scheduler.py,sha256=F783QaJfPg8tvYyvJvkwl8Sa42vsJzj6BzzROZFvm9I,28153
-ddeutil/workflow/stages.py,sha256=2vO5pdUgKHeGsS762fdYrBkWfvby_Q3Z71ptHajEp8k,55904
-ddeutil/workflow/utils.py,sha256=CtFUrP_4m6xaJooc9RbE4ulTBE-OkICg-MPHqzCuJ0I,7392
-ddeutil/workflow/workflow.py,sha256=mCpsAY0Su-pMJ_xZ-qF6lDEXn-Ih7myUODZ0ZEuyVew,50804
-ddeutil/workflow/api/__init__.py,sha256=F53NMBWtb9IKaDWkPU5KvybGGfKAcbehgn6TLBwHuuM,21
-ddeutil/workflow/api/api.py,sha256=CWtPLgOv2Jus9E7nzG5mG2Z32ZEkUK3JWQ2htZyMRpA,5244
-ddeutil/workflow/api/log.py,sha256=NMTnOnsBrDB5129329xF2myLdrb-z9k1MQrmrP7qXJw,1818
-ddeutil/workflow/api/repeat.py,sha256=uTtUFVLpiYYahXvCVx8sueRQ03K2Xw1id_gW3IMmX1U,5295
-ddeutil/workflow/api/routes/__init__.py,sha256=qoGtOMyVgQ5nTUc8J8wH27A8isaxl3IFCX8qoyibeCY,484
-ddeutil/workflow/api/routes/job.py,sha256=oPwBVP0Mxwxv-bGPlfmxQQ9PcVl0ev9HoPzndpYDCCQ,1954
-ddeutil/workflow/api/routes/logs.py,sha256=U6vOni3wd-ZTOwd3yVdSOpgyRmNdcgfngU5KlLM3Cww,5383
-ddeutil/workflow/api/routes/schedules.py,sha256=EgUjyRGhsm6UNaMj5luh6TcY6l571sCHcla-BL1iOfY,4829
-ddeutil/workflow/api/routes/workflows.py,sha256=JcDOrn1deK8ztFRcMTNATQejG6KMA7JxZLVc4QeBsP4,4527
-ddeutil_workflow-0.0.49.dist-info/licenses/LICENSE,sha256=nGFZ1QEhhhWeMHf9n99_fdt4vQaXS29xWKxt-OcLywk,1085
-ddeutil_workflow-0.0.49.dist-info/METADATA,sha256=U847vy3oZ6ZEvngEe-9W3wA1vkKGr15UQa73v9Dsy7k,18257
-ddeutil_workflow-0.0.49.dist-info/WHEEL,sha256=CmyFI0kx5cdEMTLiONQRbGQwjIoR1aIYB7eCAQ4KPJ0,91
-ddeutil_workflow-0.0.49.dist-info/top_level.txt,sha256=m9M6XeSWDwt_yMsmH6gcOjHZVK5O0-vgtNBuncHjzW4,8
-ddeutil_workflow-0.0.49.dist-info/RECORD,,