ddeutil-workflow 0.0.6__tar.gz → 0.0.8__tar.gz

{ddeutil_workflow-0.0.6/src/ddeutil_workflow.egg-info → ddeutil_workflow-0.0.8}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: ddeutil-workflow
-Version: 0.0.6
+Version: 0.0.8
 Summary: Data Developer & Engineer Workflow Utility Objects
 Author-email: ddeutils <korawich.anu@gmail.com>
 License: MIT
@@ -25,8 +25,11 @@ Requires-Dist: fmtutil
 Requires-Dist: ddeutil-io
 Requires-Dist: python-dotenv==1.0.1
 Provides-Extra: app
-Requires-Dist: fastapi==0.112.0; extra == "app"
-Requires-Dist: apscheduler[sqlalchemy]==3.10.4; extra == "app"
+Requires-Dist: schedule<2.0.0,==1.2.2; extra == "app"
+Provides-Extra: api
+Requires-Dist: fastapi[standard]==0.112.0; extra == "api"
+Requires-Dist: apscheduler[sqlalchemy]<4.0.0,==3.10.4; extra == "api"
+Requires-Dist: croniter==3.0.3; extra == "api"
 
 # Workflow
 
@@ -39,7 +42,6 @@ Requires-Dist: apscheduler[sqlalchemy]==3.10.4; extra == "app"
 
 - [Installation](#installation)
 - [Getting Started](#getting-started)
-- [Core Features](#core-features)
   - [On](#on)
   - [Pipeline](#pipeline)
 - [Usage](#usage)
@@ -50,12 +52,14 @@ Requires-Dist: apscheduler[sqlalchemy]==3.10.4; extra == "app"
 - [Deployment](#deployment)
 
 This **Workflow** objects was created for easy to make a simple metadata
-driven pipeline that able to **ETL, T, EL, or ELT** by `.yaml` file.
+driven for data pipeline orchestration that able to use for **ETL, T, EL, or
+ELT** by a `.yaml` file template.
 
-I think we should not create the multiple pipeline per use-case if we able to
-write some dynamic pipeline that just change the input parameters per use-case
-instead. This way we can handle a lot of pipelines in our orgs with metadata only.
-It called **Metadata Driven**.
+In my opinion, I think it should not create duplicate pipeline codes if I can
+write with dynamic input parameters on the one template pipeline that just change
+the input parameters per use-case instead.
+This way I can handle a lot of logical pipelines in our orgs with only metadata
+configuration. It called **Metadata Driven Data Pipeline**.
 
 Next, we should get some monitoring tools for manage logging that return from
 pipeline running. Because it not show us what is a use-case that running data
@@ -75,9 +79,10 @@ pip install ddeutil-workflow
 This project need `ddeutil-io` extension namespace packages. If you want to install
 this package with application add-ons, you should add `app` in installation;
 
-```shell
-pip install ddeutil-workflow[app]
-```
+| Usecase            | Install Optional          |
+|--------------------|---------------------------|
+| Scheduler Service  | `ddeutil-workflow[app]`   |
+| FastAPI Server     | `ddeutil-workflow[api]`   |
 
 ## Getting Started
 
@@ -160,11 +165,12 @@ use-case.
 > I recommend you to use `task` stage for all actions that you want to do with
 > pipeline object.
 
-### Python & Bash
-
 ```yaml
 run_py_local:
   type: pipeline.Pipeline
+  on:
+    - cronjob: '* * * * *'
+      timezone: "Asia/Bangkok"
   params:
     author-run: str
     run-date: datetime
@@ -174,12 +180,11 @@ run_py_local:
         - name: "Printing Information"
           id: define-func
           run: |
-            x = '${{ params.author-run }}'
-            print(f'Hello {x}')
+            x = '${{ params.run-date | fmt("%Y%m%d") }}'
+            print(f'Hello at {x}')
 
             def echo(name: str):
               print(f'Hello {name}')
-
         - name: "Run Sequence and use var from Above"
           vars:
             x: ${{ params.author-run }}
@@ -187,7 +192,6 @@ run_py_local:
             print(f'Receive x from above with {x}')
             # Change x value
             x: int = 1
-
         - name: "Call Function"
           vars:
             echo: ${{ stages.define-func.outputs.echo }}
@@ -202,75 +206,34 @@ run_py_local:
 ```
 
 ```python
+from datetime import datetime
 from ddeutil.workflow.pipeline import Pipeline
 
-pipe = Pipeline.from_loader(name='run_py_local', externals={})
-pipe.execute(params={'author-run': 'Local Workflow', 'run-date': '2024-01-01'})
+pipe: Pipeline = Pipeline.from_loader(name='run_py_local', externals={})
+pipe.execute(params={
+    'author-run': 'Local Workflow',
+    'run-date': datetime(2024, 1, 1),
+})
 ```
 
 ```shell
-> Hello Local Workflow
+> Hello at 20240101
 > Receive x from above with Local Workflow
 > Hello Caller
 > Hello World from Shell
 ```
 
-### Hook (Extract & Load)
-
-```yaml
-pipe_el_pg_to_lake:
-  type: pipeline.Pipeline
-  params:
-    run-date: datetime
-    author-email: str
-  jobs:
-    extract-load:
-      stages:
-        - name: "Extract Load from Postgres to Lake"
-          id: extract-load
-          uses: tasks/postgres-to-delta@polars
-          with:
-            source:
-              conn: conn_postgres_url
-              query: |
-                select * from ${{ params.name }}
-                where update_date = '${{ params.datetime }}'
-            sink:
-              conn: conn_az_lake
-              endpoint: "/${{ params.name }}"
-```
-
-### Hook (Transform)
-
-```yaml
-pipeline_hook_mssql_proc:
-  type: pipeline.Pipeline
-  params:
-    run_date: datetime
-    sp_name: str
-    source_name: str
-    target_name: str
-  jobs:
-    transform:
-      stages:
-        - name: "Transform Data in MS SQL Server"
-          id: transform
-          uses: tasks/mssql-proc@odbc
-          with:
-            exec: ${{ params.sp_name }}
-            params:
-              run_mode: "T"
-              run_date: ${{ params.run_date }}
-              source: ${{ params.source_name }}
-              target: ${{ params.target_name }}
-```
-
 ## Configuration
 
 ```bash
 export WORKFLOW_ROOT_PATH=.
 export WORKFLOW_CORE_REGISTRY=ddeutil.workflow,tests.utils
+export WORKFLOW_CORE_REGISTRY_FILTER=ddeutil.workflow.utils
 export WORKFLOW_CORE_PATH_CONF=conf
+export WORKFLOW_CORE_TIMEZONE=Asia/Bangkok
+export WORKFLOW_CORE_DEFAULT_STAGE_ID=true
+export WORKFLOW_CORE_MAX_PIPELINE_POKING=4
+export WORKFLOW_CORE_MAX_JOB_PARALLEL=2
 ```
 
 Application config:
@@ -283,12 +246,21 @@ export WORKFLOW_APP_INTERVAL=10
 ## Deployment
 
 This package able to run as a application service for receive manual trigger
-from the master node via RestAPI.
+from the master node via RestAPI or use to be Scheduler background service
+like crontab job but via Python API.
 
-> [!WARNING]
-> This feature do not start yet because I still research and find the best tool
-> to use it provision an app service, like `starlette`, `fastapi`, `apscheduler`.
+### Schedule Service
 
 ```shell
-(venv) $ workflow start -p 7070
+(venv) $ python src.ddeutil.workflow.app
 ```
+
+### API Server
+
+```shell
+(venv) $ uvicorn src.ddeutil.workflow.api:app --host 0.0.0.0 --port 80 --reload
+```
+
+> [!NOTE]
+> If this package already deploy, it able to use
+> `uvicorn ddeutil.workflow.api:app --host 0.0.0.0 --port 80`

{ddeutil_workflow-0.0.6 → ddeutil_workflow-0.0.8}/README.md

@@ -9,7 +9,6 @@
 
 - [Installation](#installation)
 - [Getting Started](#getting-started)
-- [Core Features](#core-features)
   - [On](#on)
   - [Pipeline](#pipeline)
 - [Usage](#usage)
@@ -20,12 +19,14 @@
 - [Deployment](#deployment)
 
 This **Workflow** objects was created for easy to make a simple metadata
-driven pipeline that able to **ETL, T, EL, or ELT** by `.yaml` file.
+driven for data pipeline orchestration that able to use for **ETL, T, EL, or
+ELT** by a `.yaml` file template.
 
-I think we should not create the multiple pipeline per use-case if we able to
-write some dynamic pipeline that just change the input parameters per use-case
-instead. This way we can handle a lot of pipelines in our orgs with metadata only.
-It called **Metadata Driven**.
+In my opinion, I think it should not create duplicate pipeline codes if I can
+write with dynamic input parameters on the one template pipeline that just change
+the input parameters per use-case instead.
+This way I can handle a lot of logical pipelines in our orgs with only metadata
+configuration. It called **Metadata Driven Data Pipeline**.
 
 Next, we should get some monitoring tools for manage logging that return from
 pipeline running. Because it not show us what is a use-case that running data
@@ -45,9 +46,10 @@ pip install ddeutil-workflow
 This project need `ddeutil-io` extension namespace packages. If you want to install
 this package with application add-ons, you should add `app` in installation;
 
-```shell
-pip install ddeutil-workflow[app]
-```
+| Usecase            | Install Optional          |
+|--------------------|---------------------------|
+| Scheduler Service  | `ddeutil-workflow[app]`   |
+| FastAPI Server     | `ddeutil-workflow[api]`   |
 
 ## Getting Started
 
@@ -130,11 +132,12 @@ use-case.
 > I recommend you to use `task` stage for all actions that you want to do with
 > pipeline object.
 
-### Python & Bash
-
 ```yaml
 run_py_local:
   type: pipeline.Pipeline
+  on:
+    - cronjob: '* * * * *'
+      timezone: "Asia/Bangkok"
   params:
     author-run: str
     run-date: datetime
@@ -144,12 +147,11 @@ run_py_local:
         - name: "Printing Information"
           id: define-func
           run: |
-            x = '${{ params.author-run }}'
-            print(f'Hello {x}')
+            x = '${{ params.run-date | fmt("%Y%m%d") }}'
+            print(f'Hello at {x}')
 
             def echo(name: str):
               print(f'Hello {name}')
-
         - name: "Run Sequence and use var from Above"
           vars:
             x: ${{ params.author-run }}
@@ -157,7 +159,6 @@ run_py_local:
             print(f'Receive x from above with {x}')
             # Change x value
             x: int = 1
-
         - name: "Call Function"
           vars:
             echo: ${{ stages.define-func.outputs.echo }}
@@ -172,75 +173,34 @@ run_py_local:
 ```
 
 ```python
+from datetime import datetime
 from ddeutil.workflow.pipeline import Pipeline
 
-pipe = Pipeline.from_loader(name='run_py_local', externals={})
-pipe.execute(params={'author-run': 'Local Workflow', 'run-date': '2024-01-01'})
+pipe: Pipeline = Pipeline.from_loader(name='run_py_local', externals={})
+pipe.execute(params={
+    'author-run': 'Local Workflow',
+    'run-date': datetime(2024, 1, 1),
+})
 ```
 
 ```shell
-> Hello Local Workflow
+> Hello at 20240101
 > Receive x from above with Local Workflow
 > Hello Caller
 > Hello World from Shell
 ```
 
-### Hook (Extract & Load)
-
-```yaml
-pipe_el_pg_to_lake:
-  type: pipeline.Pipeline
-  params:
-    run-date: datetime
-    author-email: str
-  jobs:
-    extract-load:
-      stages:
-        - name: "Extract Load from Postgres to Lake"
-          id: extract-load
-          uses: tasks/postgres-to-delta@polars
-          with:
-            source:
-              conn: conn_postgres_url
-              query: |
-                select * from ${{ params.name }}
-                where update_date = '${{ params.datetime }}'
-            sink:
-              conn: conn_az_lake
-              endpoint: "/${{ params.name }}"
-```
-
-### Hook (Transform)
-
-```yaml
-pipeline_hook_mssql_proc:
-  type: pipeline.Pipeline
-  params:
-    run_date: datetime
-    sp_name: str
-    source_name: str
-    target_name: str
-  jobs:
-    transform:
-      stages:
-        - name: "Transform Data in MS SQL Server"
-          id: transform
-          uses: tasks/mssql-proc@odbc
-          with:
-            exec: ${{ params.sp_name }}
-            params:
-              run_mode: "T"
-              run_date: ${{ params.run_date }}
-              source: ${{ params.source_name }}
-              target: ${{ params.target_name }}
-```
-
 ## Configuration
 
 ```bash
 export WORKFLOW_ROOT_PATH=.
 export WORKFLOW_CORE_REGISTRY=ddeutil.workflow,tests.utils
+export WORKFLOW_CORE_REGISTRY_FILTER=ddeutil.workflow.utils
 export WORKFLOW_CORE_PATH_CONF=conf
+export WORKFLOW_CORE_TIMEZONE=Asia/Bangkok
+export WORKFLOW_CORE_DEFAULT_STAGE_ID=true
+export WORKFLOW_CORE_MAX_PIPELINE_POKING=4
+export WORKFLOW_CORE_MAX_JOB_PARALLEL=2
 ```
 
 Application config:
@@ -253,12 +213,21 @@ export WORKFLOW_APP_INTERVAL=10
 ## Deployment
 
 This package able to run as a application service for receive manual trigger
-from the master node via RestAPI.
+from the master node via RestAPI or use to be Scheduler background service
+like crontab job but via Python API.
 
-> [!WARNING]
-> This feature do not start yet because I still research and find the best tool
-> to use it provision an app service, like `starlette`, `fastapi`, `apscheduler`.
+### Schedule Service
 
 ```shell
-(venv) $ workflow start -p 7070
+(venv) $ python src.ddeutil.workflow.app
 ```
+
+### API Server
+
+```shell
+(venv) $ uvicorn src.ddeutil.workflow.api:app --host 0.0.0.0 --port 80 --reload
+```
+
+> [!NOTE]
+> If this package already deploy, it able to use
+> `uvicorn ddeutil.workflow.api:app --host 0.0.0.0 --port 80`

{ddeutil_workflow-0.0.6 → ddeutil_workflow-0.0.8}/pyproject.toml

@@ -33,8 +33,12 @@ dynamic = ["version"]
 
 [project.optional-dependencies]
 app = [
-    "fastapi==0.112.0",
-    "apscheduler[sqlalchemy]==3.10.4",
+    "schedule==1.2.2,<2.0.0",
+]
+api = [
+    "fastapi[standard]==0.112.0",
+    "apscheduler[sqlalchemy]==3.10.4,<4.0.0",
+    "croniter==3.0.3",
 ]
 
 [project.urls]
@@ -55,8 +59,16 @@ changelog = "CHANGELOG.md"
 branch = true
 relative_files = true
 concurrency = ["thread", "multiprocessing"]
-source = ["ddeutil", "tests"]
-omit = ["scripts/"]
+source = ["ddeutil.workflow", "tests"]
+omit = [
+    "scripts/",
+    "src/ddeutil/workflow/api.py",
+    "src/ddeutil/workflow/app.py",
+    "src/ddeutil/workflow/repeat.py",
+    "src/ddeutil/workflow/route.py",
+    "tests/utils.py",
+    "tests/tasks/dummy.py",
+]
 
 [tool.coverage.report]
 exclude_lines = ["raise NotImplementedError"]
@@ -71,8 +83,8 @@ addopts = [
 filterwarnings = ["error"]
 log_cli = true
 log_cli_level = "DEBUG"
-log_cli_format = "%(asctime)s [%(levelname)7s] %(message)s (%(filename)s:%(lineno)s)"
-log_cli_date_format = "%Y-%m-%d %H:%M:%S"
+log_cli_format = "%(asctime)s [%(levelname)-7s] %(message)-100s (%(filename)s:%(lineno)s)"
+log_cli_date_format = "%Y%m%d %H:%M:%S"
 
 [tool.black]
 line-length = 80

ddeutil_workflow-0.0.8/src/ddeutil/workflow/__about__.py

@@ -0,0 +1 @@
+__version__: str = "0.0.8"

ddeutil_workflow-0.0.8/src/ddeutil/workflow/__init__.py

@@ -0,0 +1,31 @@
+# ------------------------------------------------------------------------------
+# Copyright (c) 2022 Korawich Anuttra. All rights reserved.
+# Licensed under the MIT License. See LICENSE in the project root for
+# license information.
+# ------------------------------------------------------------------------------
+from .exceptions import (
+    JobException,
+    ParamValueException,
+    PipelineException,
+    StageException,
+    UtilException,
+)
+from .on import AwsOn, On
+from .pipeline import Job, Pipeline
+from .stage import (
+    BashStage,
+    EmptyStage,
+    HookStage,
+    PyStage,
+    Stage,
+    TriggerStage,
+)
+from .utils import (
+    ChoiceParam,
+    DatetimeParam,
+    IntParam,
+    Param,
+    StrParam,
+    dash2underscore,
+    param2template,
+)

{ddeutil_workflow-0.0.6 → ddeutil_workflow-0.0.8}/src/ddeutil/workflow/__types.py

@@ -27,12 +27,21 @@ class Re:
     """Regular expression config."""
 
     # NOTE: Search caller
+    # \${{\s*(?P<caller>[a-zA-Z0-9_.\s'\"\[\]\(\)\-\{}]+?)\s*(?P<post_filters>(?:\|\s*(?:[a-zA-Z0-9_]{3,}[a-zA-Z0-9_.,-\\%\s'\"[\]()\{}]+)\s*)*)}}
     __re_caller: str = r"""
         \$
         {{
-            \s*(?P<caller>
+            \s*
+            (?P<caller>
                 [a-zA-Z0-9_.\s'\"\[\]\(\)\-\{}]+?
             )\s*
+            (?P<post_filters>
+                (?:
+                    \|\s*
+                    (?:[a-zA-Z0-9_]{3,}[a-zA-Z0-9_.,-\\%\s'\"[\]()\{}]*)
+                    \s*
+                )*
+            )
         }}
     """
     RE_CALLER: Pattern = re.compile(
@@ -40,6 +49,7 @@ class Re:
     )
 
     # NOTE: Search task
+    # ^(?P<path>[^/@]+)/(?P<func>[^@]+)@(?P<tag>.+)$
     __re_task_fmt: str = r"""
         ^
             (?P<path>[^/@]+)

ddeutil_workflow-0.0.8/src/ddeutil/workflow/api.py

@@ -0,0 +1,120 @@
+# ------------------------------------------------------------------------------
+# Copyright (c) 2022 Korawich Anuttra. All rights reserved.
+# Licensed under the MIT License. See LICENSE in the project root for
+# license information.
+# ------------------------------------------------------------------------------
+from __future__ import annotations
+
+import asyncio
+import queue
+import time
+import uuid
+from contextlib import asynccontextmanager
+from datetime import datetime
+
+from apscheduler.executors.pool import ProcessPoolExecutor
+from apscheduler.jobstores.memory import MemoryJobStore
+from apscheduler.jobstores.sqlalchemy import SQLAlchemyJobStore
+from apscheduler.schedulers.asyncio import AsyncIOScheduler
+from fastapi import BackgroundTasks, FastAPI
+from fastapi.middleware.gzip import GZipMiddleware
+from fastapi.responses import UJSONResponse
+from pydantic import BaseModel
+
+from .log import get_logger
+from .repeat import repeat_every
+from .route import schedule_route, workflow_route
+
+logger = get_logger(__name__)
+
+
+def broker_upper_messages():
+    for _ in range(app.queue_limit):
+        try:
+            obj = app.queue.get_nowait()
+            app.output_dict[obj["request_id"]] = obj["text"].upper()
+            logger.info(f"Upper message: {app.output_dict}")
+        except queue.Empty:
+            pass
+
+
+jobstores = {
+    "default": MemoryJobStore(),
+    "sqlite": SQLAlchemyJobStore(url="sqlite:///jobs-store.sqlite"),
+}
+executors = {
+    "default": {"type": "threadpool", "max_workers": 5},
+    "processpool": ProcessPoolExecutor(max_workers=5),
+}
+scheduler = AsyncIOScheduler(
+    jobstores=jobstores,
+    executors=executors,
+    timezone="Asia/Bangkok",
+)
+
+
+@asynccontextmanager
+async def lifespan(_: FastAPI):
+    scheduler.start()
+    yield
+    scheduler.shutdown(wait=False)
+
+
+app = FastAPI(lifespan=lifespan)
+app.add_middleware(GZipMiddleware, minimum_size=1000)
+app.include_router(schedule_route)
+app.include_router(workflow_route)
+
+app.scheduler = scheduler
+app.scheduler.add_job(
+    broker_upper_messages,
+    "interval",
+    seconds=10,
+)
+app.queue = queue.Queue()
+app.output_dict = {}
+app.queue_limit = 2
+
+
+def write_pipeline(task_id: str, message=""):
+    logger.info(f"{task_id} : {message}")
+    time.sleep(5)
+    logger.info(f"{task_id} : run task successfully!!!")
+
+
+@app.post("/schedule/{name}", response_class=UJSONResponse)
+async def send_schedule(name: str, background_tasks: BackgroundTasks):
+    background_tasks.add_task(
+        write_pipeline,
+        name,
+        message=f"some message for {name}",
+    )
+    await fetch_current_time()
+    return {"message": f"Schedule sent {name!r} in the background"}
+
+
+@repeat_every(seconds=2, max_repetitions=3)
+async def fetch_current_time():
+    logger.info(f"Fetch: {datetime.now()}")
+
+
+class Payload(BaseModel):
+    text: str
+
+
+async def get_result(request_id):
+    while 1:
+        if request_id in app.output_dict:
+            result = app.output_dict[request_id]
+            del app.output_dict[request_id]
+            return {"message": result}
+        await asyncio.sleep(0.001)
+
+
+@app.post("/upper", response_class=UJSONResponse)
+async def message_upper(payload: Payload):
+    request_id: str = str(uuid.uuid4())
+    app.queue.put(
+        {"text": payload.text, "request_id": request_id},
+    )
+    return await get_result(request_id)

ddeutil_workflow-0.0.8/src/ddeutil/workflow/app.py

@@ -0,0 +1,45 @@
+# ------------------------------------------------------------------------------
+# Copyright (c) 2022 Korawich Anuttra. All rights reserved.
+# Licensed under the MIT License. See LICENSE in the project root for
+# license information.
+# ------------------------------------------------------------------------------
+from __future__ import annotations
+
+import functools
+import time
+
+import schedule
+
+
+def catch_exceptions(cancel_on_failure=False):
+    """Catch exception error from scheduler job."""
+
+    def catch_exceptions_decorator(job_func):
+        @functools.wraps(job_func)
+        def wrapper(*args, **kwargs):
+            try:
+                return job_func(*args, **kwargs)
+            except Exception as err:
+                print(err)
+
+                if cancel_on_failure:
+                    return schedule.CancelJob
+
+        return wrapper
+
+    return catch_exceptions_decorator
+
+
+@catch_exceptions(cancel_on_failure=True)
+def bad_task():
+    return 1 / 0
+
+
+schedule.every(5).seconds.do(bad_task)
+
+if __name__ == "__main__":
+    while True:
+        schedule.run_pending()
+        time.sleep(1)
+        if not schedule.get_jobs():
+            break