ddeutil-workflow 0.0.12__py3-none-any.whl → 0.0.14__py3-none-any.whl

{ddeutil_workflow-0.0.12.dist-info → ddeutil_workflow-0.0.14.dist-info}/METADATA

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: ddeutil-workflow
-Version: 0.0.12
+Version: 0.0.14
 Summary: Lightweight workflow orchestration with less dependencies
 Author-email: ddeutils <korawich.anu@gmail.com>
 License: MIT
@@ -18,19 +18,22 @@ Classifier: Programming Language :: Python :: 3.9
 Classifier: Programming Language :: Python :: 3.10
 Classifier: Programming Language :: Python :: 3.11
 Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
 Requires-Python: >=3.9.13
 Description-Content-Type: text/markdown
 License-File: LICENSE
-Requires-Dist: ddeutil-io
+Requires-Dist: ddeutil >=0.4.0
+Requires-Dist: ddeutil-io >=0.1.13
 Requires-Dist: python-dotenv ==1.0.1
 Requires-Dist: typer <1.0.0,==0.12.5
 Requires-Dist: schedule <2.0.0,==1.2.2
 Provides-Extra: api
-Requires-Dist: fastapi <1.0.0,==0.112.2 ; extra == 'api'
+Requires-Dist: fastapi <1.0.0,>=0.115.0 ; extra == 'api'
 
 # Workflow
 
 [![test](https://github.com/ddeutils/ddeutil-workflow/actions/workflows/tests.yml/badge.svg?branch=main)](https://github.com/ddeutils/ddeutil-workflow/actions/workflows/tests.yml)
+[![pypi version](https://img.shields.io/pypi/v/ddeutil-workflow)](https://pypi.org/project/ddeutil-workflow/)
 [![python support version](https://img.shields.io/pypi/pyversions/ddeutil-workflow)](https://pypi.org/project/ddeutil-workflow/)
 [![size](https://img.shields.io/github/languages/code-size/ddeutils/ddeutil-workflow)](https://github.com/ddeutils/ddeutil-workflow)
 [![gh license](https://img.shields.io/github/license/ddeutils/ddeutil-workflow)](https://github.com/ddeutils/ddeutil-workflow/blob/main/LICENSE)
@@ -41,8 +44,9 @@ for easy to make a simple metadata driven for data workflow orchestration.
 It can to use for data operator by a `.yaml` template.
 
 > [!WARNING]
-> This package provide only orchestration workload. That mean you should not use
-> workflow stage to process any large data which use lot of compute usecase.
+> This package provide only orchestration workload task. That mean you should not
+> use the workflow stage to process any large volume data which use lot of compute
+> resource. :cold_sweat:
 
 In my opinion, I think it should not create duplicate workflow codes if I can
 write with dynamic input parameters on the one template workflow that just change
@@ -50,23 +54,25 @@ the input parameters per use-case instead.
 This way I can handle a lot of logical workflows in our orgs with only metadata
 configuration. It called **Metadata Driven Data Workflow**.
 
-Next, we should get some monitoring tools for manage logging that return from
-workflow running. Because it not show us what is a use-case that running data
-workflow.
+**:pushpin: <u>Rules of This Workflow engine</u>**:
 
-> [!NOTE]
-> _Disclaimer_: I inspire the dynamic statement from the GitHub Action `.yml` files
-> and all of config file from several data orchestration framework tools from my
-> experience on Data Engineer.
-
-**Rules of This Workflow engine**:
+1. Minimum frequency unit of scheduling is **1 minute** :warning:
+2. Can not re-run only failed stage and its pending downstream :rotating_light:
+3. All parallel tasks inside workflow engine use Multi-Threading
+   (Because Python 3.13 unlock GIL :unlock:)
 
-1. Minimum unit of scheduling is 1 minute
-2. Cannot re-run only failed stage and its pending downstream
-3. All parallel tasks inside workflow engine use Threading
-   (Because Python 3.13 unlock GIL)
+> [!NOTE]
+> _Disclaimer_: I inspire the dynamic statement from the [**GitHub Action**](https://github.com/features/actions)
+> with `.yml` files and all of config file from several data orchestration framework
+> tools from my experience on Data Engineer. :grimacing:
+>
+> Other workflow that I interest on them and pick some interested feature to this
+> package:
+>
+> - [Google **Workflows**](https://cloud.google.com/workflows)
+> - [AWS **Step Functions**](https://aws.amazon.com/step-functions/)
 
-## Installation
+## :round_pushpin: Installation
 
 This project need `ddeutil-io` extension namespace packages. If you want to install
 this package with application add-ons, you should add `app` in installation;
@@ -79,7 +85,7 @@ this package with application add-ons, you should add `app` in installation;
 
 > I added this feature to the main milestone.
 >
-> **Docker Images** supported:
+> :egg: **Docker Images** supported:
 >
 > | Docker Image                | Python Version | Support |
 > |-----------------------------|----------------|---------|
@@ -87,8 +93,9 @@ this package with application add-ons, you should add `app` in installation;
 > | ddeutil-workflow:python3.10 | `3.10`         | :x:     |
 > | ddeutil-workflow:python3.11 | `3.11`         | :x:     |
 > | ddeutil-workflow:python3.12 | `3.12`         | :x:     |
+> | ddeutil-workflow:python3.12 | `3.13`         | :x:     |
 
-## Usage
+## :beers: Usage
 
 This is examples that use workflow file for running common Data Engineering
 use-case.
@@ -100,8 +107,10 @@ use-case.
 > maintenance your data workflows.
 
 ```yaml
-run_py_local:
-   type: Workflow
+run-py-local:
+
+   # Validate model that use to parsing exists for template file
+   type: ddeutil.workflow.Workflow
    on:
       # If workflow deploy to schedule, it will running every 5 minutes
       # with Asia/Bangkok timezone.
@@ -110,7 +119,7 @@ run_py_local:
    params:
       # Incoming execution parameters will validate with this type. It allow
       # to set default value or templating.
-      author-run: str
+      source-extract: str
       run-date: datetime
    jobs:
       getting-api-data:
@@ -119,17 +128,56 @@ run_py_local:
               id: retrieve-api
               uses: tasks/get-api-with-oauth-to-s3@requests
               with:
-                 url: https://open-data/
-                 auth: ${API_ACCESS_REFRESH_TOKEN}
-                 aws_s3_path: my-data/open-data/
-
-                 # This Authentication code should implement with your custom hook function.
-                 # The template allow you to use environment variable.
+                 # Arguments of source data that want to retrieve.
+                 method: post
+                 url: https://finances/open-data/currency-pairs/
+                 body:
+                    resource: ${{ params.source-extract }}
+
+                    # You can able to use filtering like Jinja template but this
+                    # package does not use it.
+                    filter: ${{ params.run-date | fmt(fmt='%Y%m%d') }}
+                 auth:
+                    type: bearer
+                    keys: ${API_ACCESS_REFRESH_TOKEN}
+
+                 # Arguments of target data that want to landing.
+                 writing_mode: flatten
+                 aws_s3_path: my-data/open-data/${{ params.source-extract }}
+
+                 # This Authentication code should implement with your custom hook
+                 # function. The template allow you to use environment variable.
                  aws_access_client_id: ${AWS_ACCESS_CLIENT_ID}
                  aws_access_client_secret: ${AWS_ACCESS_CLIENT_SECRET}
 ```
 
-## Configuration
+The above workflow template is main executor pipeline that you want to do. If you
+want to schedule this workflow, you want to dynamic its parameters change base on
+execution time such as `run-date` should change base on that workflow running date.
+
+So, this package provide the `Schedule` template for this action.
+
+```yaml
+schedule-run-local-wf:
+
+   # Validate model that use to parsing exists for template file
+   type: ddeutil.workflow.scheduler.Schedule
+   workflows:
+
+      # Map existing workflow that want to deploy with scheduler application.
+      # It allow you to passing release parameter that dynamic change depend the
+      # current context of this scheduler application releasing that time.
+      - name: run-py-local
+        params:
+          source-extract: "USD-THB"
+          asat-dt: "${{ release.logical_date }}"
+```
+
+## :cookie: Configuration
+
+The main configuration that use to dynamic changing with your propose of this
+application. If any configuration values do not set yet, it will use default value
+and do not raise any error to you.
 
 | Environment                         | Component | Default                          | Description                                                                |
 |-------------------------------------|-----------|----------------------------------|----------------------------------------------------------------------------|
@@ -155,7 +203,7 @@ run_py_local:
 | `WORKFLOW_API_ENABLE_ROUTE_WORKFLOW` | API       | true    | A flag that enable workflow route to manage execute manually and workflow logging |
 | `WORKFLOW_API_ENABLE_ROUTE_SCHEDULE` | API       | true    | A flag that enable run scheduler                                                  |
 
-## Deployment
+## :rocket: Deployment
 
 This package able to run as a application service for receive manual trigger
 from the master node via RestAPI or use to be Scheduler background service

ddeutil_workflow-0.0.14.dist-info/RECORD

@@ -0,0 +1,22 @@
+ddeutil/workflow/__about__.py,sha256=Xas_M3BaGwGfZOJTvMWUlmlW09aGC-Apst-NtkPddY4,28
+ddeutil/workflow/__init__.py,sha256=-DIy8SGFsD7_wqp-V-K8v8jTxacmqrcyj_SFx1WS6qg,687
+ddeutil/workflow/__types.py,sha256=aBbytylSPIe_cip2KIyqLN2eUloMOJdkayqKWCBrwhk,2353
+ddeutil/workflow/api.py,sha256=cwju_qhY6m0kLtaoa77QLglC9tl7RjjZ4UnJYV3SlQQ,4810
+ddeutil/workflow/cli.py,sha256=Ikcq526WeIl-737-v55T0PwAZ2pNiZFxlN0Y-DjhDbQ,3374
+ddeutil/workflow/conf.py,sha256=j19G7rDxQRGgSRQW3pxIYtK5lB3fZv0eG_CAoqoIhPw,1140
+ddeutil/workflow/cron.py,sha256=naWefHc3EnVo41Yf1zQeXOzF27YlTlnfj0XnQ6_HO-U,25514
+ddeutil/workflow/exceptions.py,sha256=Uf1-Tn8rAzj0aiVHSqo4fBqO80W0za7UFZgKv24E-tg,706
+ddeutil/workflow/job.py,sha256=zEefiEAxyC34NvbNVpKexTVU1E_031446308zGMdcmE,21488
+ddeutil/workflow/log.py,sha256=Ev-Szi0KC_MmbFY4g4BWv6tUSmcLKWKZ03ZInmYPmgU,6490
+ddeutil/workflow/on.py,sha256=vsZG19mNoztDSB_ObD_4ZWPKgHYpBDJMWw97ZiTavNE,7237
+ddeutil/workflow/repeat.py,sha256=e3dekPTlMlxCCizfBYsZ8dD8Juy4rtfqDZJU3Iky2oA,5011
+ddeutil/workflow/route.py,sha256=ABEk-WlVo9XGFc7zCPbckX33URCNH7woQFU1keX_8PQ,6970
+ddeutil/workflow/scheduler.py,sha256=CcUFichnvPbQzSEk_ikNgFwZimTObGHfXxHChuysAo4,45706
+ddeutil/workflow/stage.py,sha256=Tt5QQrO_dN8MO9gPtiziOqVrd64UTJZwbgifWeXBCIA,23574
+ddeutil/workflow/utils.py,sha256=epJMTsA4BPQa0gECgcWJ38IENlejpnF3OTBNc0eaqYE,30715
+ddeutil_workflow-0.0.14.dist-info/LICENSE,sha256=nGFZ1QEhhhWeMHf9n99_fdt4vQaXS29xWKxt-OcLywk,1085
+ddeutil_workflow-0.0.14.dist-info/METADATA,sha256=sf9kiPoGazaRxqymTjaxZyH47yNyGj0RElc-NLmUa4w,11653
+ddeutil_workflow-0.0.14.dist-info/WHEEL,sha256=GV9aMThwP_4oNCtvEC2ec3qUYutgWeAzklro_0m4WJQ,91
+ddeutil_workflow-0.0.14.dist-info/entry_points.txt,sha256=0BVOgO3LdUdXVZ-CiHHDKxzEk2c8J30jEwHeKn2YCWI,62
+ddeutil_workflow-0.0.14.dist-info/top_level.txt,sha256=m9M6XeSWDwt_yMsmH6gcOjHZVK5O0-vgtNBuncHjzW4,8
+ddeutil_workflow-0.0.14.dist-info/RECORD,,

{ddeutil_workflow-0.0.12.dist-info → ddeutil_workflow-0.0.14.dist-info}/WHEEL

@@ -1,5 +1,5 @@
 Wheel-Version: 1.0
-Generator: setuptools (74.0.0)
+Generator: setuptools (75.1.0)
 Root-Is-Purelib: true
 Tag: py3-none-any
 

ddeutil_workflow-0.0.12.dist-info/RECORD

@@ -1,21 +0,0 @@
-ddeutil/workflow/__about__.py,sha256=dDoptHBL0iJdvMM__PxrYhbuaSInQh_epfEsH-vdqVw,28
-ddeutil/workflow/__init__.py,sha256=aEQiEWwTPGhfwpzzdb99xXaHchi5ABWUHl2iLIyT18E,664
-ddeutil/workflow/__types.py,sha256=SYMoxbENQX8uPsiCZkjtpHAqqHOh8rUrarAFicAJd0E,1773
-ddeutil/workflow/api.py,sha256=xVP8eGu1nnR8HM0ULTwxs9TV9tsxCOjZ68cAffw2f3o,4802
-ddeutil/workflow/cli.py,sha256=Ikcq526WeIl-737-v55T0PwAZ2pNiZFxlN0Y-DjhDbQ,3374
-ddeutil/workflow/cron.py,sha256=uhp3E5pl_tX_H88bsDujcwdhZmOE53csyV-ouPpPdK8,25321
-ddeutil/workflow/exceptions.py,sha256=Uf1-Tn8rAzj0aiVHSqo4fBqO80W0za7UFZgKv24E-tg,706
-ddeutil/workflow/job.py,sha256=eESvmIbIyYtiKgVLfILtb0lTz-shMhy4Bi7kHrtZSk0,19663
-ddeutil/workflow/log.py,sha256=bZyyqf3oNBB8oRf8RI0YvII7wHHoj4wC-nmW_pQjQ1c,6036
-ddeutil/workflow/on.py,sha256=Sxwnu0vPbIrMR_WWvH3_rOvD0tbiJntcB5378WoV19M,7163
-ddeutil/workflow/repeat.py,sha256=e3dekPTlMlxCCizfBYsZ8dD8Juy4rtfqDZJU3Iky2oA,5011
-ddeutil/workflow/route.py,sha256=ABEk-WlVo9XGFc7zCPbckX33URCNH7woQFU1keX_8PQ,6970
-ddeutil/workflow/scheduler.py,sha256=ISiVoKM0puh3XaXZ9NTi-J-vREGxdEa-lk1jR6a4OXk,41639
-ddeutil/workflow/stage.py,sha256=nYsKKT5ZKelEsFnDScBwaitXOv_aUEFkubDPQVK5isM,20644
-ddeutil/workflow/utils.py,sha256=TbqgPkDDYBpqCZ7HV2TU3AH1_Mv-zfrJdwVL-l2SPUo,28559
-ddeutil_workflow-0.0.12.dist-info/LICENSE,sha256=nGFZ1QEhhhWeMHf9n99_fdt4vQaXS29xWKxt-OcLywk,1085
-ddeutil_workflow-0.0.12.dist-info/METADATA,sha256=U95bFDPtWdPuA7KcIq8kDYNor6CBztiXjdjhrAPvMDY,9395
-ddeutil_workflow-0.0.12.dist-info/WHEEL,sha256=UvcQYKBHoFqaQd6LKyqHw9fxEolWLQnlzP0h_LgJAfI,91
-ddeutil_workflow-0.0.12.dist-info/entry_points.txt,sha256=0BVOgO3LdUdXVZ-CiHHDKxzEk2c8J30jEwHeKn2YCWI,62
-ddeutil_workflow-0.0.12.dist-info/top_level.txt,sha256=m9M6XeSWDwt_yMsmH6gcOjHZVK5O0-vgtNBuncHjzW4,8
-ddeutil_workflow-0.0.12.dist-info/RECORD,,