ddeutil-workflow 0.0.5__tar.gz → 0.0.7__tar.gz

{ddeutil_workflow-0.0.5/src/ddeutil_workflow.egg-info → ddeutil_workflow-0.0.7}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: ddeutil-workflow
-Version: 0.0.5
+Version: 0.0.7
 Summary: Data Developer & Engineer Workflow Utility Objects
 Author-email: ddeutils <korawich.anu@gmail.com>
 License: MIT
@@ -24,36 +24,42 @@ License-File: LICENSE
 Requires-Dist: fmtutil
 Requires-Dist: ddeutil-io
 Requires-Dist: python-dotenv==1.0.1
-Requires-Dist: schedule==1.2.2
+Provides-Extra: app
+Requires-Dist: schedule<2.0.0,==1.2.2; extra == "app"
+Provides-Extra: api
+Requires-Dist: fastapi[standard]==0.112.0; extra == "api"
+Requires-Dist: apscheduler[sqlalchemy]<4.0.0,==3.10.4; extra == "api"
+Requires-Dist: croniter==3.0.3; extra == "api"
 
-# Data Utility: _Workflow_
+# Workflow
 
 [![test](https://github.com/ddeutils/ddeutil-workflow/actions/workflows/tests.yml/badge.svg?branch=main)](https://github.com/ddeutils/ddeutil-workflow/actions/workflows/tests.yml)
 [![python support version](https://img.shields.io/pypi/pyversions/ddeutil-workflow)](https://pypi.org/project/ddeutil-workflow/)
 [![size](https://img.shields.io/github/languages/code-size/ddeutils/ddeutil-workflow)](https://github.com/ddeutils/ddeutil-workflow)
 [![gh license](https://img.shields.io/github/license/ddeutils/ddeutil-workflow)](https://github.com/ddeutils/ddeutil-workflow/blob/main/LICENSE)
 
-
 **Table of Contents**:
 
 - [Installation](#installation)
 - [Getting Started](#getting-started)
-  - [Connection](#connection)
-  - [Dataset](#dataset)
-  - [Schedule](#schedule)
-- [Pipeline Examples](#examples)
-  - [Python & Shell](#python--shell)
-  - [Tasks (EL)](#tasks-extract--load)
-  - [Hooks (T)](#tasks-transform)
+  - [On](#on)
+  - [Pipeline](#pipeline)
+- [Usage](#usage)
+  - [Python & Bash](#python--bash)
+  - [Hook (EL)](#hook-extract--load)
+  - [Hook (T)](#hook-transform)
 - [Configuration](#configuration)
+- [Deployment](#deployment)
 
-This **Utility Workflow** objects was created for easy to make a simple metadata
-driven pipeline that able to **ETL, T, EL, or ELT** by `.yaml` file.
+This **Workflow** objects was created for easy to make a simple metadata
+driven for data pipeline orchestration that able to use for **ETL, T, EL, or
+ELT** by a `.yaml` file template.
 
-I think we should not create the multiple pipeline per use-case if we able to
-write some dynamic pipeline that just change the input parameters per use-case
-instead. This way we can handle a lot of pipelines in our orgs with metadata only.
-It called **Metadata Driven**.
+In my opinion, I think it should not create duplicate pipeline codes if I can
+write with dynamic input parameters on the one template pipeline that just change
+the input parameters per use-case instead.
+This way I can handle a lot of logical pipelines in our orgs with only metadata
+configuration. It called **Metadata Driven Data Pipeline**.
 
 Next, we should get some monitoring tools for manage logging that return from
 pipeline running. Because it not show us what is a use-case that running data
@@ -70,7 +76,16 @@ pipeline.
 pip install ddeutil-workflow
 ```
 
-This project need `ddeutil-io`, `ddeutil-model` extension namespace packages.
+This project need `ddeutil-io` extension namespace packages. If you want to install
+this package with application add-ons, you should add `app` in installation;
+
+```shell
+pip install ddeutil-workflow[app]
+```
+
+```shell
+pip install ddeutil-workflow[api]
+```
 
 ## Getting Started
 
@@ -87,38 +102,42 @@ will passing parameters and catching the output for re-use it to next step.
 > dynamic registries instead of main features because it have a lot of maintain
 > vendor codes and deps. (I do not have time to handle this features)
 
----
+### On
 
-### Schedule
+The **On** is schedule object.
 
 ```yaml
-schd_for_node:
-  type: schedule.Schedule
+on_every_5_min:
+  type: on.On
   cron: "*/5 * * * *"
 ```
 
 ```python
-from ddeutil.workflow.on import Schedule
+from ddeutil.workflow.on import On
 
-scdl = Schedule.from_loader(name='schd_for_node', externals={})
-assert '*/5 * * * *' == str(scdl.cronjob)
+schedule = On.from_loader(name='on_every_5_min', externals={})
+assert '*/5 * * * *' == str(schedule.cronjob)
 
-cron_iterate = scdl.generate('2022-01-01 00:00:00')
-assert '2022-01-01 00:05:00' f"{cron_iterate.next:%Y-%m-%d %H:%M:%S}"
-assert '2022-01-01 00:10:00' f"{cron_iterate.next:%Y-%m-%d %H:%M:%S}"
-assert '2022-01-01 00:15:00' f"{cron_iterate.next:%Y-%m-%d %H:%M:%S}"
-assert '2022-01-01 00:20:00' f"{cron_iterate.next:%Y-%m-%d %H:%M:%S}"
-assert '2022-01-01 00:25:00' f"{cron_iterate.next:%Y-%m-%d %H:%M:%S}"
+cron_iter = schedule.generate('2022-01-01 00:00:00')
+assert '2022-01-01 00:05:00' f"{cron_iter.next:%Y-%m-%d %H:%M:%S}"
+assert '2022-01-01 00:10:00' f"{cron_iter.next:%Y-%m-%d %H:%M:%S}"
+assert '2022-01-01 00:15:00' f"{cron_iter.next:%Y-%m-%d %H:%M:%S}"
+assert '2022-01-01 00:20:00' f"{cron_iter.next:%Y-%m-%d %H:%M:%S}"
 ```
 
----
-
 ### Pipeline
 
+The **Pipeline** object that is the core feature of this project.
+
 ```yaml
 run_py_local:
   type: ddeutil.workflow.pipeline.Pipeline
-  ...
+  on: 'on_every_5_min'
+  params:
+    author-run:
+      type: str
+    run-date:
+      type: datetime
 ```
 
 ```python
@@ -128,27 +147,39 @@ pipe = Pipeline.from_loader(name='run_py_local', externals={})
 pipe.execute(params={'author-run': 'Local Workflow', 'run-date': '2024-01-01'})
 ```
 
-## Examples
+> [!NOTE]
+> The above parameter use short declarative statement. You can pass a parameter
+> type to the key of a parameter name.
+> ```yaml
+> params:
+>   author-run: str
+>   run-date: datetime
+> ```
+>
+> And for the type, you can remove `ddeutil.workflow` prefix because we can find
+> it by looping search from `WORKFLOW_CORE_REGISTRY` value.
+
+## Usage
 
 This is examples that use workflow file for running common Data Engineering
 use-case.
 
-### Python & Shell
+> [!IMPORTANT]
+> I recommend you to use `task` stage for all actions that you want to do with
+> pipeline object.
 
-The state of doing lists that worker should to do. It be collection of the stage.
+### Python & Bash
 
 ```yaml
 run_py_local:
-  type: ddeutil.workflow.pipeline.Pipeline
+  type: pipeline.Pipeline
   params:
-    author-run:
-      type: str
-    run-date:
-      type: datetime
+    author-run: str
+    run-date: datetime
   jobs:
     first-job:
       stages:
-        - name: Printing Information
+        - name: "Printing Information"
           id: define-func
           run: |
             x = '${{ params.author-run }}'
@@ -157,7 +188,7 @@ run_py_local:
             def echo(name: str):
               print(f'Hello {name}')
 
-        - name: Run Sequence and use var from Above
+        - name: "Run Sequence and use var from Above"
           vars:
             x: ${{ params.author-run }}
           run: |
@@ -165,16 +196,16 @@ run_py_local:
             # Change x value
             x: int = 1
 
-        - name: Call Function
+        - name: "Call Function"
           vars:
             echo: ${{ stages.define-func.outputs.echo }}
           run: |
             echo('Caller')
     second-job:
       stages:
-        - name: Echo Shell Script
+        - name: "Echo Bash Script"
           id: shell-echo
-          shell: |
+          bash: |
             echo "Hello World from Shell"
 ```
 
@@ -192,24 +223,20 @@ pipe.execute(params={'author-run': 'Local Workflow', 'run-date': '2024-01-01'})
 > Hello World from Shell
 ```
 
----
-
-### Tasks (Extract & Load)
+### Hook (Extract & Load)
 
 ```yaml
 pipe_el_pg_to_lake:
-  type: ddeutil.workflow.pipeline.Pipeline
+  type: pipeline.Pipeline
   params:
-    run-date:
-      type: datetime
-    author-email:
-      type: str
+    run-date: datetime
+    author-email: str
   jobs:
     extract-load:
       stages:
         - name: "Extract Load from Postgres to Lake"
           id: extract-load
-          task: tasks/postgres-to-delta@polars
+          uses: tasks/postgres-to-delta@polars
           with:
             source:
               conn: conn_postgres_url
@@ -221,15 +248,23 @@ pipe_el_pg_to_lake:
               endpoint: "/${{ params.name }}"
 ```
 
----
+Implement hook:
 
-### Tasks (Transform)
+```python
+from ddeutil.workflow.utils import tag
 
-> I recommend you to use task for all actions that you want to do.
+@tag('polars', alias='postgres-to-delta')
+def postgres_to_delta(source, sink):
+    return {
+        "source": source, "sink": sink
+    }
+```
+
+### Hook (Transform)
 
 ```yaml
-pipe_hook_mssql_proc:
-  type: ddeutil.workflow.pipeline.Pipeline
+pipeline_hook_mssql_proc:
+  type: pipeline.Pipeline
   params:
     run_date: datetime
     sp_name: str
@@ -240,7 +275,7 @@ pipe_hook_mssql_proc:
       stages:
         - name: "Transform Data in MS SQL Server"
           id: transform
-          task: tasks/mssql-proc@odbc
+          uses: tasks/mssql-proc@odbc
           with:
             exec: ${{ params.sp_name }}
             params:
@@ -250,16 +285,57 @@ pipe_hook_mssql_proc:
               target: ${{ params.target_name }}
 ```
 
-> [!NOTE]
-> The above parameter use short declarative statement. You can pass a parameter
-> type to the key of a parameter name.
+Implement hook:
+
+```python
+from ddeutil.workflow.utils import tag
+
+@tag('odbc', alias='mssql-proc')
+def odbc_mssql_procedure(_exec: str, params: dict):
+    return {
+        "exec": _exec, "params": params
+    }
+```
 
 ## Configuration
 
-```text
+```bash
+export WORKFLOW_ROOT_PATH=.
+export WORKFLOW_CORE_REGISTRY=ddeutil.workflow,tests.utils
+export WORKFLOW_CORE_REGISTRY_FILTER=ddeutil.workflow.utils
+export WORKFLOW_CORE_PATH_CONF=conf
+export WORKFLOW_CORE_TIMEZONE=Asia/Bangkok
+export WORKFLOW_CORE_DEFAULT_STAGE_ID=true
 
+export WORKFLOW_CORE_MAX_PIPELINE_POKING=4
+export WORKFLOW_CORE_MAX_JOB_PARALLEL=2
 ```
 
-## License
+Application config:
 
-This project was licensed under the terms of the [MIT license](LICENSE).
+```bash
+export WORKFLOW_APP_DB_URL=postgresql+asyncpg://user:pass@localhost:5432/schedule
+export WORKFLOW_APP_INTERVAL=10
+```
+
+## Deployment
+
+This package able to run as a application service for receive manual trigger
+from the master node via RestAPI or use to be Scheduler background service
+like crontab job but via Python API.
+
+### Schedule Service
+
+```shell
+(venv) $ python src.ddeutil.workflow.app
+```
+
+### API Server
+
+```shell
+(venv) $ uvicorn src.ddeutil.workflow.api:app --host 0.0.0.0 --port 80 --reload
+```
+
+> [!NOTE]
+> If this package already deploy, it able to use
+> `uvicorn ddeutil.workflow.api:app --host 0.0.0.0 --port 80`

{ddeutil_workflow-0.0.5 → ddeutil_workflow-0.0.7}/README.md

@@ -1,31 +1,32 @@
-# Data Utility: _Workflow_
+# Workflow
 
 [![test](https://github.com/ddeutils/ddeutil-workflow/actions/workflows/tests.yml/badge.svg?branch=main)](https://github.com/ddeutils/ddeutil-workflow/actions/workflows/tests.yml)
 [![python support version](https://img.shields.io/pypi/pyversions/ddeutil-workflow)](https://pypi.org/project/ddeutil-workflow/)
 [![size](https://img.shields.io/github/languages/code-size/ddeutils/ddeutil-workflow)](https://github.com/ddeutils/ddeutil-workflow)
 [![gh license](https://img.shields.io/github/license/ddeutils/ddeutil-workflow)](https://github.com/ddeutils/ddeutil-workflow/blob/main/LICENSE)
 
-
 **Table of Contents**:
 
 - [Installation](#installation)
 - [Getting Started](#getting-started)
-  - [Connection](#connection)
-  - [Dataset](#dataset)
-  - [Schedule](#schedule)
-- [Pipeline Examples](#examples)
-  - [Python & Shell](#python--shell)
-  - [Tasks (EL)](#tasks-extract--load)
-  - [Hooks (T)](#tasks-transform)
+  - [On](#on)
+  - [Pipeline](#pipeline)
+- [Usage](#usage)
+  - [Python & Bash](#python--bash)
+  - [Hook (EL)](#hook-extract--load)
+  - [Hook (T)](#hook-transform)
 - [Configuration](#configuration)
+- [Deployment](#deployment)
 
-This **Utility Workflow** objects was created for easy to make a simple metadata
-driven pipeline that able to **ETL, T, EL, or ELT** by `.yaml` file.
+This **Workflow** objects was created for easy to make a simple metadata
+driven for data pipeline orchestration that able to use for **ETL, T, EL, or
+ELT** by a `.yaml` file template.
 
-I think we should not create the multiple pipeline per use-case if we able to
-write some dynamic pipeline that just change the input parameters per use-case
-instead. This way we can handle a lot of pipelines in our orgs with metadata only.
-It called **Metadata Driven**.
+In my opinion, I think it should not create duplicate pipeline codes if I can
+write with dynamic input parameters on the one template pipeline that just change
+the input parameters per use-case instead.
+This way I can handle a lot of logical pipelines in our orgs with only metadata
+configuration. It called **Metadata Driven Data Pipeline**.
 
 Next, we should get some monitoring tools for manage logging that return from
 pipeline running. Because it not show us what is a use-case that running data
@@ -42,7 +43,16 @@ pipeline.
 pip install ddeutil-workflow
 ```
 
-This project need `ddeutil-io`, `ddeutil-model` extension namespace packages.
+This project need `ddeutil-io` extension namespace packages. If you want to install
+this package with application add-ons, you should add `app` in installation;
+
+```shell
+pip install ddeutil-workflow[app]
+```
+
+```shell
+pip install ddeutil-workflow[api]
+```
 
 ## Getting Started
 
@@ -59,38 +69,42 @@ will passing parameters and catching the output for re-use it to next step.
 > dynamic registries instead of main features because it have a lot of maintain
 > vendor codes and deps. (I do not have time to handle this features)
 
----
+### On
 
-### Schedule
+The **On** is schedule object.
 
 ```yaml
-schd_for_node:
-  type: schedule.Schedule
+on_every_5_min:
+  type: on.On
   cron: "*/5 * * * *"
 ```
 
 ```python
-from ddeutil.workflow.on import Schedule
+from ddeutil.workflow.on import On
 
-scdl = Schedule.from_loader(name='schd_for_node', externals={})
-assert '*/5 * * * *' == str(scdl.cronjob)
+schedule = On.from_loader(name='on_every_5_min', externals={})
+assert '*/5 * * * *' == str(schedule.cronjob)
 
-cron_iterate = scdl.generate('2022-01-01 00:00:00')
-assert '2022-01-01 00:05:00' f"{cron_iterate.next:%Y-%m-%d %H:%M:%S}"
-assert '2022-01-01 00:10:00' f"{cron_iterate.next:%Y-%m-%d %H:%M:%S}"
-assert '2022-01-01 00:15:00' f"{cron_iterate.next:%Y-%m-%d %H:%M:%S}"
-assert '2022-01-01 00:20:00' f"{cron_iterate.next:%Y-%m-%d %H:%M:%S}"
-assert '2022-01-01 00:25:00' f"{cron_iterate.next:%Y-%m-%d %H:%M:%S}"
+cron_iter = schedule.generate('2022-01-01 00:00:00')
+assert '2022-01-01 00:05:00' f"{cron_iter.next:%Y-%m-%d %H:%M:%S}"
+assert '2022-01-01 00:10:00' f"{cron_iter.next:%Y-%m-%d %H:%M:%S}"
+assert '2022-01-01 00:15:00' f"{cron_iter.next:%Y-%m-%d %H:%M:%S}"
+assert '2022-01-01 00:20:00' f"{cron_iter.next:%Y-%m-%d %H:%M:%S}"
 ```
 
----
-
 ### Pipeline
 
+The **Pipeline** object that is the core feature of this project.
+
 ```yaml
 run_py_local:
   type: ddeutil.workflow.pipeline.Pipeline
-  ...
+  on: 'on_every_5_min'
+  params:
+    author-run:
+      type: str
+    run-date:
+      type: datetime
 ```
 
 ```python
@@ -100,27 +114,39 @@ pipe = Pipeline.from_loader(name='run_py_local', externals={})
 pipe.execute(params={'author-run': 'Local Workflow', 'run-date': '2024-01-01'})
 ```
 
-## Examples
+> [!NOTE]
+> The above parameter use short declarative statement. You can pass a parameter
+> type to the key of a parameter name.
+> ```yaml
+> params:
+>   author-run: str
+>   run-date: datetime
+> ```
+>
+> And for the type, you can remove `ddeutil.workflow` prefix because we can find
+> it by looping search from `WORKFLOW_CORE_REGISTRY` value.
+
+## Usage
 
 This is examples that use workflow file for running common Data Engineering
 use-case.
 
-### Python & Shell
+> [!IMPORTANT]
+> I recommend you to use `task` stage for all actions that you want to do with
+> pipeline object.
 
-The state of doing lists that worker should to do. It be collection of the stage.
+### Python & Bash
 
 ```yaml
 run_py_local:
-  type: ddeutil.workflow.pipeline.Pipeline
+  type: pipeline.Pipeline
   params:
-    author-run:
-      type: str
-    run-date:
-      type: datetime
+    author-run: str
+    run-date: datetime
   jobs:
     first-job:
       stages:
-        - name: Printing Information
+        - name: "Printing Information"
           id: define-func
           run: |
             x = '${{ params.author-run }}'
@@ -129,7 +155,7 @@ run_py_local:
             def echo(name: str):
               print(f'Hello {name}')
 
-        - name: Run Sequence and use var from Above
+        - name: "Run Sequence and use var from Above"
           vars:
             x: ${{ params.author-run }}
           run: |
@@ -137,16 +163,16 @@ run_py_local:
             # Change x value
             x: int = 1
 
-        - name: Call Function
+        - name: "Call Function"
           vars:
             echo: ${{ stages.define-func.outputs.echo }}
           run: |
             echo('Caller')
     second-job:
       stages:
-        - name: Echo Shell Script
+        - name: "Echo Bash Script"
           id: shell-echo
-          shell: |
+          bash: |
             echo "Hello World from Shell"
 ```
 
@@ -164,24 +190,20 @@ pipe.execute(params={'author-run': 'Local Workflow', 'run-date': '2024-01-01'})
 > Hello World from Shell
 ```
 
----
-
-### Tasks (Extract & Load)
+### Hook (Extract & Load)
 
 ```yaml
 pipe_el_pg_to_lake:
-  type: ddeutil.workflow.pipeline.Pipeline
+  type: pipeline.Pipeline
   params:
-    run-date:
-      type: datetime
-    author-email:
-      type: str
+    run-date: datetime
+    author-email: str
   jobs:
     extract-load:
       stages:
         - name: "Extract Load from Postgres to Lake"
           id: extract-load
-          task: tasks/postgres-to-delta@polars
+          uses: tasks/postgres-to-delta@polars
           with:
             source:
               conn: conn_postgres_url
@@ -193,15 +215,23 @@ pipe_el_pg_to_lake:
               endpoint: "/${{ params.name }}"
 ```
 
----
+Implement hook:
 
-### Tasks (Transform)
+```python
+from ddeutil.workflow.utils import tag
 
-> I recommend you to use task for all actions that you want to do.
+@tag('polars', alias='postgres-to-delta')
+def postgres_to_delta(source, sink):
+    return {
+        "source": source, "sink": sink
+    }
+```
+
+### Hook (Transform)
 
 ```yaml
-pipe_hook_mssql_proc:
-  type: ddeutil.workflow.pipeline.Pipeline
+pipeline_hook_mssql_proc:
+  type: pipeline.Pipeline
   params:
     run_date: datetime
     sp_name: str
@@ -212,7 +242,7 @@ pipe_hook_mssql_proc:
       stages:
         - name: "Transform Data in MS SQL Server"
           id: transform
-          task: tasks/mssql-proc@odbc
+          uses: tasks/mssql-proc@odbc
           with:
             exec: ${{ params.sp_name }}
             params:
@@ -222,16 +252,57 @@ pipe_hook_mssql_proc:
               target: ${{ params.target_name }}
 ```
 
-> [!NOTE]
-> The above parameter use short declarative statement. You can pass a parameter
-> type to the key of a parameter name.
+Implement hook:
+
+```python
+from ddeutil.workflow.utils import tag
+
+@tag('odbc', alias='mssql-proc')
+def odbc_mssql_procedure(_exec: str, params: dict):
+    return {
+        "exec": _exec, "params": params
+    }
+```
 
 ## Configuration
 
-```text
+```bash
+export WORKFLOW_ROOT_PATH=.
+export WORKFLOW_CORE_REGISTRY=ddeutil.workflow,tests.utils
+export WORKFLOW_CORE_REGISTRY_FILTER=ddeutil.workflow.utils
+export WORKFLOW_CORE_PATH_CONF=conf
+export WORKFLOW_CORE_TIMEZONE=Asia/Bangkok
+export WORKFLOW_CORE_DEFAULT_STAGE_ID=true
 
+export WORKFLOW_CORE_MAX_PIPELINE_POKING=4
+export WORKFLOW_CORE_MAX_JOB_PARALLEL=2
 ```
 
-## License
+Application config:
 
-This project was licensed under the terms of the [MIT license](LICENSE).
+```bash
+export WORKFLOW_APP_DB_URL=postgresql+asyncpg://user:pass@localhost:5432/schedule
+export WORKFLOW_APP_INTERVAL=10
+```
+
+## Deployment
+
+This package able to run as a application service for receive manual trigger
+from the master node via RestAPI or use to be Scheduler background service
+like crontab job but via Python API.
+
+### Schedule Service
+
+```shell
+(venv) $ python src.ddeutil.workflow.app
+```
+
+### API Server
+
+```shell
+(venv) $ uvicorn src.ddeutil.workflow.api:app --host 0.0.0.0 --port 80 --reload
+```
+
+> [!NOTE]
+> If this package already deploy, it able to use
+> `uvicorn ddeutil.workflow.api:app --host 0.0.0.0 --port 80`