ddeutil-workflow 0.0.6__py3-none-any.whl → 0.0.8__py3-none-any.whl

{ddeutil_workflow-0.0.6.dist-info → ddeutil_workflow-0.0.8.dist-info}/METADATA

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: ddeutil-workflow
-Version: 0.0.6
+Version: 0.0.8
 Summary: Data Developer & Engineer Workflow Utility Objects
 Author-email: ddeutils <korawich.anu@gmail.com>
 License: MIT
@@ -24,9 +24,12 @@ License-File: LICENSE
 Requires-Dist: fmtutil
 Requires-Dist: ddeutil-io
 Requires-Dist: python-dotenv ==1.0.1
+Provides-Extra: api
+Requires-Dist: fastapi[standard] ==0.112.0 ; extra == 'api'
+Requires-Dist: apscheduler[sqlalchemy] <4.0.0,==3.10.4 ; extra == 'api'
+Requires-Dist: croniter ==3.0.3 ; extra == 'api'
 Provides-Extra: app
-Requires-Dist: fastapi ==0.112.0 ; extra == 'app'
-Requires-Dist: apscheduler[sqlalchemy] ==3.10.4 ; extra == 'app'
+Requires-Dist: schedule <2.0.0,==1.2.2 ; extra == 'app'
 
 # Workflow
 
@@ -39,7 +42,6 @@ Requires-Dist: apscheduler[sqlalchemy] ==3.10.4 ; extra == 'app'
 
 - [Installation](#installation)
 - [Getting Started](#getting-started)
-- [Core Features](#core-features)
   - [On](#on)
   - [Pipeline](#pipeline)
 - [Usage](#usage)
@@ -50,12 +52,14 @@ Requires-Dist: apscheduler[sqlalchemy] ==3.10.4 ; extra == 'app'
 - [Deployment](#deployment)
 
 This **Workflow** objects was created for easy to make a simple metadata
-driven pipeline that able to **ETL, T, EL, or ELT** by `.yaml` file.
+driven for data pipeline orchestration that able to use for **ETL, T, EL, or
+ELT** by a `.yaml` file template.
 
-I think we should not create the multiple pipeline per use-case if we able to
-write some dynamic pipeline that just change the input parameters per use-case
-instead. This way we can handle a lot of pipelines in our orgs with metadata only.
-It called **Metadata Driven**.
+In my opinion, I think it should not create duplicate pipeline codes if I can
+write with dynamic input parameters on the one template pipeline that just change
+the input parameters per use-case instead.
+This way I can handle a lot of logical pipelines in our orgs with only metadata
+configuration. It called **Metadata Driven Data Pipeline**.
 
 Next, we should get some monitoring tools for manage logging that return from
 pipeline running. Because it not show us what is a use-case that running data
@@ -75,9 +79,10 @@ pip install ddeutil-workflow
 This project need `ddeutil-io` extension namespace packages. If you want to install
 this package with application add-ons, you should add `app` in installation;
 
-```shell
-pip install ddeutil-workflow[app]
-```
+| Usecase            | Install Optional          |
+|--------------------|---------------------------|
+| Scheduler Service  | `ddeutil-workflow[app]`   |
+| FastAPI Server     | `ddeutil-workflow[api]`   |
 
 ## Getting Started
 
@@ -160,11 +165,12 @@ use-case.
 > I recommend you to use `task` stage for all actions that you want to do with
 > pipeline object.
 
-### Python & Bash
-
 ```yaml
 run_py_local:
   type: pipeline.Pipeline
+  on:
+    - cronjob: '* * * * *'
+      timezone: "Asia/Bangkok"
   params:
     author-run: str
     run-date: datetime
@@ -174,12 +180,11 @@ run_py_local:
         - name: "Printing Information"
           id: define-func
           run: |
-            x = '${{ params.author-run }}'
-            print(f'Hello {x}')
+            x = '${{ params.run-date | fmt("%Y%m%d") }}'
+            print(f'Hello at {x}')
 
             def echo(name: str):
               print(f'Hello {name}')
-
         - name: "Run Sequence and use var from Above"
           vars:
             x: ${{ params.author-run }}
@@ -187,7 +192,6 @@ run_py_local:
             print(f'Receive x from above with {x}')
             # Change x value
             x: int = 1
-
         - name: "Call Function"
           vars:
             echo: ${{ stages.define-func.outputs.echo }}
@@ -202,75 +206,34 @@ run_py_local:
 ```
 
 ```python
+from datetime import datetime
 from ddeutil.workflow.pipeline import Pipeline
 
-pipe = Pipeline.from_loader(name='run_py_local', externals={})
-pipe.execute(params={'author-run': 'Local Workflow', 'run-date': '2024-01-01'})
+pipe: Pipeline = Pipeline.from_loader(name='run_py_local', externals={})
+pipe.execute(params={
+    'author-run': 'Local Workflow',
+    'run-date': datetime(2024, 1, 1),
+})
 ```
 
 ```shell
-> Hello Local Workflow
+> Hello at 20240101
 > Receive x from above with Local Workflow
 > Hello Caller
 > Hello World from Shell
 ```
 
-### Hook (Extract & Load)
-
-```yaml
-pipe_el_pg_to_lake:
-  type: pipeline.Pipeline
-  params:
-    run-date: datetime
-    author-email: str
-  jobs:
-    extract-load:
-      stages:
-        - name: "Extract Load from Postgres to Lake"
-          id: extract-load
-          uses: tasks/postgres-to-delta@polars
-          with:
-            source:
-              conn: conn_postgres_url
-              query: |
-                select * from ${{ params.name }}
-                where update_date = '${{ params.datetime }}'
-            sink:
-              conn: conn_az_lake
-              endpoint: "/${{ params.name }}"
-```
-
-### Hook (Transform)
-
-```yaml
-pipeline_hook_mssql_proc:
-  type: pipeline.Pipeline
-  params:
-    run_date: datetime
-    sp_name: str
-    source_name: str
-    target_name: str
-  jobs:
-    transform:
-      stages:
-        - name: "Transform Data in MS SQL Server"
-          id: transform
-          uses: tasks/mssql-proc@odbc
-          with:
-            exec: ${{ params.sp_name }}
-            params:
-              run_mode: "T"
-              run_date: ${{ params.run_date }}
-              source: ${{ params.source_name }}
-              target: ${{ params.target_name }}
-```
-
 ## Configuration
 
 ```bash
 export WORKFLOW_ROOT_PATH=.
 export WORKFLOW_CORE_REGISTRY=ddeutil.workflow,tests.utils
+export WORKFLOW_CORE_REGISTRY_FILTER=ddeutil.workflow.utils
 export WORKFLOW_CORE_PATH_CONF=conf
+export WORKFLOW_CORE_TIMEZONE=Asia/Bangkok
+export WORKFLOW_CORE_DEFAULT_STAGE_ID=true
+export WORKFLOW_CORE_MAX_PIPELINE_POKING=4
+export WORKFLOW_CORE_MAX_JOB_PARALLEL=2
 ```
 
 Application config:
@@ -283,12 +246,21 @@ export WORKFLOW_APP_INTERVAL=10
 ## Deployment
 
 This package able to run as a application service for receive manual trigger
-from the master node via RestAPI.
+from the master node via RestAPI or use to be Scheduler background service
+like crontab job but via Python API.
 
-> [!WARNING]
-> This feature do not start yet because I still research and find the best tool
-> to use it provision an app service, like `starlette`, `fastapi`, `apscheduler`.
+### Schedule Service
 
 ```shell
-(venv) $ workflow start -p 7070
+(venv) $ python src.ddeutil.workflow.app
 ```
+
+### API Server
+
+```shell
+(venv) $ uvicorn src.ddeutil.workflow.api:app --host 0.0.0.0 --port 80 --reload
+```
+
+> [!NOTE]
+> If this package already deploy, it able to use
+> `uvicorn ddeutil.workflow.api:app --host 0.0.0.0 --port 80`

ddeutil_workflow-0.0.8.dist-info/RECORD

@@ -0,0 +1,20 @@
+ddeutil/workflow/__about__.py,sha256=FA15NQYpQvn7SrHupxQQQ9Ad5ZzEXOvwDS5UyB1h1bo,27
+ddeutil/workflow/__init__.py,sha256=4PEL3RdHmUowK0Dz-tK7fO0wvFX4u9CLd0Up7b3lrAQ,760
+ddeutil/workflow/__types.py,sha256=SYMoxbENQX8uPsiCZkjtpHAqqHOh8rUrarAFicAJd0E,1773
+ddeutil/workflow/api.py,sha256=d2Mmv9jTtN3FITIy-2mivyAKdBOGZxtkNWRMPbCLlFI,3341
+ddeutil/workflow/app.py,sha256=BuYhOoSJCHiSoj3xb2I5QoxaHrD3bKdmoJua3bKBetc,1165
+ddeutil/workflow/exceptions.py,sha256=zuCcsfJ1hFivubXz6lXCpGYXk07d_PkRaUD5ew3_LC0,632
+ddeutil/workflow/loader.py,sha256=_ZD-XP5P7VbUeqItrUVPaKIZu6dMUZ2aywbCbReW1hQ,2778
+ddeutil/workflow/log.py,sha256=N2TyjcuAoH0YTvzJCHTO037IHgVkLA986Xhtz1LSgE4,1742
+ddeutil/workflow/on.py,sha256=YoEqDbzJUwqOA3JRltbvlYr0rNTtxdmb7cWMxl8U19k,6717
+ddeutil/workflow/pipeline.py,sha256=VC6VDxycUdGKn13V42RZxAlCFySYb2HIZGq_ku5Kp5k,30844
+ddeutil/workflow/repeat.py,sha256=sNoRfbOR4cYm_edrSvlVy9N8Dk_osLIq9FC5GMZz32M,4621
+ddeutil/workflow/route.py,sha256=Ck_O1xJwI-vKkMJr37El0-1PGKlwKF8__DDNWVQrf0A,2079
+ddeutil/workflow/scheduler.py,sha256=FqmkvWCqwJ4eRf8aDn5Ce4FcNWqmcvu2aTTfL34lfgs,22184
+ddeutil/workflow/stage.py,sha256=tbxENx_-2BQ6peXKM_s6RQ1oGzTlXcZ4yDpP1Hufkdk,18095
+ddeutil/workflow/utils.py,sha256=seyU81JXfb2zz6QbJvVEb2Wn4qt8f-FBA6QFC97xY5k,21240
+ddeutil_workflow-0.0.8.dist-info/LICENSE,sha256=nGFZ1QEhhhWeMHf9n99_fdt4vQaXS29xWKxt-OcLywk,1085
+ddeutil_workflow-0.0.8.dist-info/METADATA,sha256=9i7Jk3CZlBpNkmFFjD247opgYA6Mc8AT6CtZjcvamYI,8314
+ddeutil_workflow-0.0.8.dist-info/WHEEL,sha256=HiCZjzuy6Dw0hdX5R3LCFPDmFS4BWl8H-8W39XfmgX4,91
+ddeutil_workflow-0.0.8.dist-info/top_level.txt,sha256=m9M6XeSWDwt_yMsmH6gcOjHZVK5O0-vgtNBuncHjzW4,8
+ddeutil_workflow-0.0.8.dist-info/RECORD,,

{ddeutil_workflow-0.0.6.dist-info → ddeutil_workflow-0.0.8.dist-info}/WHEEL

@@ -1,5 +1,5 @@
 Wheel-Version: 1.0
-Generator: setuptools (72.1.0)
+Generator: setuptools (72.2.0)
 Root-Is-Purelib: true
 Tag: py3-none-any
 

ddeutil_workflow-0.0.6.dist-info/RECORD

@@ -1,15 +0,0 @@
-ddeutil/workflow/__about__.py,sha256=VEYa91VchyTUnF57lvvquHvitTViBxxLXuhcEnr4TAY,27
-ddeutil/workflow/__init__.py,sha256=Y5wLiJ0zS1CfoSOZ0oo7OL3LNMKvPmpUO4fVHuAOv8E,429
-ddeutil/workflow/__types.py,sha256=PfwDZBnTwe2JImD7UFS0J6Nq-1TcjBGebOVzJZoSuTQ,1354
-ddeutil/workflow/exceptions.py,sha256=9O12c4aNLi0dyjVBgCLveV3HN9PXcZfwFfLXdgm3Ffs,626
-ddeutil/workflow/loader.py,sha256=_ZD-XP5P7VbUeqItrUVPaKIZu6dMUZ2aywbCbReW1hQ,2778
-ddeutil/workflow/on.py,sha256=YoEqDbzJUwqOA3JRltbvlYr0rNTtxdmb7cWMxl8U19k,6717
-ddeutil/workflow/pipeline.py,sha256=8mIvY34_fsiqscBa9JB94MgN3Km5fkuD2iaNZSAQVuM,17843
-ddeutil/workflow/scheduler.py,sha256=FqmkvWCqwJ4eRf8aDn5Ce4FcNWqmcvu2aTTfL34lfgs,22184
-ddeutil/workflow/stage.py,sha256=bDJiGS21gYlYbFDnLTKH9aIbXfej9fT-V1ADoPX7w4s,13829
-ddeutil/workflow/utils.py,sha256=xapKxxnqIzlbKA45GaRcWn-VL30AhE7M8f46ynr-vbI,11173
-ddeutil_workflow-0.0.6.dist-info/LICENSE,sha256=nGFZ1QEhhhWeMHf9n99_fdt4vQaXS29xWKxt-OcLywk,1085
-ddeutil_workflow-0.0.6.dist-info/METADATA,sha256=5X6ewXGn96MR9rDhVmmaoTuIwuRmfi72t7ezM8wxbvw,8612
-ddeutil_workflow-0.0.6.dist-info/WHEEL,sha256=R0nc6qTxuoLk7ShA2_Y-UWkN8ZdfDBG2B6Eqpz2WXbs,91
-ddeutil_workflow-0.0.6.dist-info/top_level.txt,sha256=m9M6XeSWDwt_yMsmH6gcOjHZVK5O0-vgtNBuncHjzW4,8
-ddeutil_workflow-0.0.6.dist-info/RECORD,,