ddeutil-workflow 0.0.34__tar.gz → 0.0.36__tar.gz

{ddeutil_workflow-0.0.34 → ddeutil_workflow-0.0.36}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.2
 Name: ddeutil-workflow
-Version: 0.0.34
+Version: 0.0.36
 Summary: Lightweight workflow orchestration
 Author-email: ddeutils <korawich.anu@gmail.com>
 License: MIT
@@ -29,6 +29,8 @@ Requires-Dist: python-dotenv==1.0.1
 Requires-Dist: schedule<2.0.0,==1.2.2
 Provides-Extra: api
 Requires-Dist: fastapi<1.0.0,>=0.115.0; extra == "api"
+Requires-Dist: httpx; extra == "api"
+Requires-Dist: ujson; extra == "api"
 
 # Workflow Orchestration
 
@@ -62,7 +64,7 @@ configuration. It called **Metadata Driven Data Workflow**.
 1. The Minimum frequency unit of scheduling is **1 minute** :warning:
 2. Can not re-run only failed stage and its pending downstream :rotating_light:
 3. All parallel tasks inside workflow engine use Multi-Threading
-   (Python 3.13 unlock GIL :unlock:)
+   (🐍 Python 3.13 unlock GIL :unlock:)
 
 ---
 
@@ -78,45 +80,48 @@ flowchart LR
 
     subgraph Docker Container
         direction TB
-        G@{ shape: rounded, label: "Observe<br>Application" }
+        G@{ shape: rounded, label: "📡Observe<br>Application" }
     end
 
     subgraph Docker Container
         direction TB
-        B@{ shape: rounded, label: "Workflow<br>Application" }
+        B@{ shape: rounded, label: "🏃Workflow<br>Application" }
     end
 
-    A <--->|action &<br>response| B
-    B -....-> |response| G
-    G -....-> |request| B
+    A <-->|action &<br>response| B
+    B -...-> |response| G
+    G -...-> |request| B
 
     subgraph Data Context
-    D@{ shape: processes, label: "Logs" }
-    E@{ shape: lin-cyl, label: "Audit<br>Logs" }
+        D@{ shape: processes, label: "Logs" }
+        E@{ shape: lin-cyl, label: "Audit<br>Logs" }
     end
 
-    subgraph Git Context
-    F@{ shape: tag-rect, label: "YAML<br>files" }
+    subgraph Config Context
+        F@{ shape: tag-rect, label: "YAML<br>files" }
     end
 
-    B --->|disable| F
-    F --->|read| B
+    A ---> |push| H(Repo)
+    H -.-> |pull| F
 
-    B --->|write| E
-    E --->|read| B
-    B --->|write| D
+    B <-->|disable &<br>read| F
+
+    B <-->|read &<br>write| E
+
+    B -->|write| D
 
     D -.->|read| G
     E -.->|read| G
 ```
 
 > [!WARNING]
-> _Disclaimer_: I inspire the dynamic statement from the [**GitHub Action**](https://github.com/features/actions)
-> with `.yml` files and all configs file from several data orchestration framework
-> tools from my experience on Data Engineer. :grimacing:
->
-> Other workflow tools that I interest on them and pick some interested feature
-> implement to this package:
+> _**Disclaimer**_: I inspire the dynamic YAML statement from the [**GitHub Action**](https://github.com/features/actions),
+> and all configs pattern from several data orchestration framework tools from
+> my data engineering experience. :grimacing:
+
+> [!NOTE]
+> Other workflow orchestration tools that I interest and pick them to be inspiration
+> some for this package:
 >
 > - [Google **Workflows**](https://cloud.google.com/workflows)
 > - [AWS **Step Functions**](https://aws.amazon.com/step-functions/)
@@ -127,10 +132,10 @@ This project need `ddeutil` and `ddeutil-io` extension namespace packages.
 If you want to install this package with application add-ons, you should add
 `app` in installation;
 
-| Use-case       | Install Optional         | Support            |
-|----------------|--------------------------|--------------------|
-| Python         | `ddeutil-workflow`       | :heavy_check_mark: |
-| FastAPI Server | `ddeutil-workflow[api]`  | :heavy_check_mark: |
+| Use-case       | Install Optional         |       Support       |
+|----------------|--------------------------|:-------------------:|
+| Python         | `ddeutil-workflow`       | :heavy_check_mark:  |
+| FastAPI Server | `ddeutil-workflow[api]`  | :heavy_check_mark:  |
 
 ## :beers: Usage
 
@@ -292,7 +297,7 @@ like crontab job but via Python API.
 ### API Server
 
 ```shell
-(venv) $ uvicorn src.ddeutil.workflow.api:app \
+(venv) $ uvicorn ddeutil.workflow.api:app \
   --host 127.0.0.1 \
   --port 80 \
   --no-access-log

{ddeutil_workflow-0.0.34 → ddeutil_workflow-0.0.36}/README.md

@@ -30,7 +30,7 @@ configuration. It called **Metadata Driven Data Workflow**.
 1. The Minimum frequency unit of scheduling is **1 minute** :warning:
 2. Can not re-run only failed stage and its pending downstream :rotating_light:
 3. All parallel tasks inside workflow engine use Multi-Threading
-   (Python 3.13 unlock GIL :unlock:)
+   (🐍 Python 3.13 unlock GIL :unlock:)
 
 ---
 
@@ -46,45 +46,48 @@ flowchart LR
 
     subgraph Docker Container
         direction TB
-        G@{ shape: rounded, label: "Observe<br>Application" }
+        G@{ shape: rounded, label: "📡Observe<br>Application" }
     end
 
     subgraph Docker Container
         direction TB
-        B@{ shape: rounded, label: "Workflow<br>Application" }
+        B@{ shape: rounded, label: "🏃Workflow<br>Application" }
     end
 
-    A <--->|action &<br>response| B
-    B -....-> |response| G
-    G -....-> |request| B
+    A <-->|action &<br>response| B
+    B -...-> |response| G
+    G -...-> |request| B
 
     subgraph Data Context
-    D@{ shape: processes, label: "Logs" }
-    E@{ shape: lin-cyl, label: "Audit<br>Logs" }
+        D@{ shape: processes, label: "Logs" }
+        E@{ shape: lin-cyl, label: "Audit<br>Logs" }
     end
 
-    subgraph Git Context
-    F@{ shape: tag-rect, label: "YAML<br>files" }
+    subgraph Config Context
+        F@{ shape: tag-rect, label: "YAML<br>files" }
     end
 
-    B --->|disable| F
-    F --->|read| B
+    A ---> |push| H(Repo)
+    H -.-> |pull| F
 
-    B --->|write| E
-    E --->|read| B
-    B --->|write| D
+    B <-->|disable &<br>read| F
+
+    B <-->|read &<br>write| E
+
+    B -->|write| D
 
     D -.->|read| G
     E -.->|read| G
 ```
 
 > [!WARNING]
-> _Disclaimer_: I inspire the dynamic statement from the [**GitHub Action**](https://github.com/features/actions)
-> with `.yml` files and all configs file from several data orchestration framework
-> tools from my experience on Data Engineer. :grimacing:
->
-> Other workflow tools that I interest on them and pick some interested feature
-> implement to this package:
+> _**Disclaimer**_: I inspire the dynamic YAML statement from the [**GitHub Action**](https://github.com/features/actions),
+> and all configs pattern from several data orchestration framework tools from
+> my data engineering experience. :grimacing:
+
+> [!NOTE]
+> Other workflow orchestration tools that I interest and pick them to be inspiration
+> some for this package:
 >
 > - [Google **Workflows**](https://cloud.google.com/workflows)
 > - [AWS **Step Functions**](https://aws.amazon.com/step-functions/)
@@ -95,10 +98,10 @@ This project need `ddeutil` and `ddeutil-io` extension namespace packages.
 If you want to install this package with application add-ons, you should add
 `app` in installation;
 
-| Use-case       | Install Optional         | Support            |
-|----------------|--------------------------|--------------------|
-| Python         | `ddeutil-workflow`       | :heavy_check_mark: |
-| FastAPI Server | `ddeutil-workflow[api]`  | :heavy_check_mark: |
+| Use-case       | Install Optional         |       Support       |
+|----------------|--------------------------|:-------------------:|
+| Python         | `ddeutil-workflow`       | :heavy_check_mark:  |
+| FastAPI Server | `ddeutil-workflow[api]`  | :heavy_check_mark:  |
 
 ## :beers: Usage
 
@@ -260,7 +263,7 @@ like crontab job but via Python API.
 ### API Server
 
 ```shell
-(venv) $ uvicorn src.ddeutil.workflow.api:app \
+(venv) $ uvicorn ddeutil.workflow.api:app \
   --host 127.0.0.1 \
   --port 80 \
   --no-access-log

{ddeutil_workflow-0.0.34 → ddeutil_workflow-0.0.36}/pyproject.toml

@@ -35,7 +35,11 @@ dependencies = [
 dynamic = ["version"]
 
 [project.optional-dependencies]
-api = [ "fastapi>=0.115.0,<1.0.0" ]
+api = [
+    "fastapi>=0.115.0,<1.0.0",
+    "httpx",
+    "ujson",
+]
 
 [project.urls]
 Homepage = "https://github.com/ddeutils/ddeutil-workflow/"
@@ -66,7 +70,11 @@ omit = [
     "src/ddeutil/workflow/api/__init__.py",
     "src/ddeutil/workflow/api/api.py",
     "src/ddeutil/workflow/api/repeat.py",
-    "src/ddeutil/workflow/api/route.py",
+    "src/ddeutil/workflow/api/routes/__init__.py",
+    "src/ddeutil/workflow/api/routes/job.py",
+    "src/ddeutil/workflow/api/routes/logs.py",
+    "src/ddeutil/workflow/api/routes/schedules.py",
+    "src/ddeutil/workflow/api/routes/workflows.py",
     "app.py",
 ]
 
@@ -89,7 +97,10 @@ addopts = [
     "--strict-config",
     "--strict-markers",
 ]
-filterwarnings = ["error"]
+filterwarnings = [
+    "error",
+    "ignore::DeprecationWarning",
+]
 log_cli = true
 log_cli_level = "DEBUG"
 log_cli_format = "%(asctime)s [%(levelname)-7s] %(message)-120s (%(filename)s:%(lineno)s)"

ddeutil_workflow-0.0.36/src/ddeutil/workflow/__about__.py

@@ -0,0 +1 @@
+__version__: str = "0.0.36"

{ddeutil_workflow-0.0.34 → ddeutil_workflow-0.0.36}/src/ddeutil/workflow/__init__.py

@@ -9,7 +9,7 @@ from .audit import (
     Audit,
     get_audit,
 )
-from .call import (
+from .caller import (
     ReturnTagFunc,
     TagFunc,
     extract_call,
@@ -37,8 +37,15 @@ from .exceptions import (
 )
 from .job import (
     Job,
+    RunsOn,
     Strategy,
 )
+from .logs import (
+    TraceData,
+    TraceLog,
+    get_dt_tznow,
+    get_trace,
+)
 from .params import (
     ChoiceParam,
     DatetimeParam,
@@ -49,9 +56,7 @@ from .params import (
 from .result import (
     Result,
     Status,
-    TraceLog,
     default_gen_id,
-    get_dt_tznow,
 )
 from .scheduler import (
     Schedule,

{ddeutil_workflow-0.0.34 → ddeutil_workflow-0.0.36}/src/ddeutil/workflow/api/api.py

@@ -11,7 +11,11 @@ from datetime import datetime, timedelta
 from typing import TypedDict
 
 from dotenv import load_dotenv
-from fastapi import FastAPI
+from fastapi import FastAPI, Request
+from fastapi import status as st
+from fastapi.encoders import jsonable_encoder
+from fastapi.exceptions import RequestValidationError
+from fastapi.middleware.cors import CORSMiddleware
 from fastapi.middleware.gzip import GZipMiddleware
 from fastapi.responses import UJSONResponse
 
@@ -20,6 +24,7 @@ from ..conf import config, get_logger
 from ..scheduler import ReleaseThread, ReleaseThreads
 from ..workflow import ReleaseQueue, WorkflowTask
 from .repeat import repeat_at
+from .routes import job, log
 
 load_dotenv()
 logger = get_logger("ddeutil.workflow")
@@ -60,39 +65,57 @@ async def lifespan(a: FastAPI) -> AsyncIterator[State]:
 
 
 app = FastAPI(
-    titile="Workflow API",
+    titile="Workflow",
     description=(
-        "This is workflow FastAPI web application that use to manage manual "
-        "execute or schedule workflow via RestAPI."
+        "This is a workflow FastAPI application that use to manage manual "
+        "execute, logging, and schedule workflow via RestAPI."
     ),
     version=__version__,
     lifespan=lifespan,
     default_response_class=UJSONResponse,
 )
 app.add_middleware(GZipMiddleware, minimum_size=1000)
+origins: list[str] = [
+    "http://localhost",
+    "http://localhost:88",
+    "http://localhost:80",
+]
+app.add_middleware(
+    CORSMiddleware,
+    allow_origins=origins,
+    allow_credentials=True,
+    allow_methods=["*"],
+    allow_headers=["*"],
+)
 
 
 @app.get("/")
 async def health():
-    return {"message": "Workflow API already start up"}
+    """Index view that not return any template without json status."""
+    return {"message": "Workflow already start up with healthy status."}
+
 
+# NOTE Add the jobs and logs routes by default.
+app.include_router(job, prefix=config.prefix_path)
+app.include_router(log, prefix=config.prefix_path)
 
-# NOTE: Enable the workflow route.
+
+# NOTE: Enable the workflows route.
 if config.enable_route_workflow:
-    from .route import workflow_route
+    from .routes import workflow
 
-    app.include_router(workflow_route, prefix=config.prefix_path)
+    app.include_router(workflow, prefix=config.prefix_path)
 
 
-# NOTE: Enable the schedule route.
+# NOTE: Enable the schedules route.
 if config.enable_route_schedule:
     from ..audit import get_audit
     from ..scheduler import schedule_task
-    from .route import schedule_route
+    from .routes import schedule
 
-    app.include_router(schedule_route, prefix=config.prefix_path)
+    app.include_router(schedule, prefix=config.prefix_path)
 
-    @schedule_route.on_event("startup")
+    @schedule.on_event("startup")
     @repeat_at(cron="* * * * *", delay=2)
     def scheduler_listener():
         """Schedule broker every minute at 02 second."""
@@ -106,12 +129,13 @@ if config.enable_route_schedule:
                 stop=datetime.now(config.tz) + timedelta(minutes=1),
                 queue=app.state.workflow_queue,
                 threads=app.state.workflow_threads,
-                log=get_audit(),
+                audit=get_audit(),
             )
 
-    @schedule_route.on_event("startup")
+    @schedule.on_event("startup")
     @repeat_at(cron="*/5 * * * *", delay=10)
     def monitoring():
+        """Monitoring workflow thread that running in the background."""
         logger.debug("[MONITOR]: Start monitoring threading.")
         snapshot_threads: list[str] = list(app.state.workflow_threads.keys())
         for t_name in snapshot_threads:
@@ -121,3 +145,23 @@ if config.enable_route_schedule:
             # NOTE: remove the thread that running success.
             if not thread_release["thread"].is_alive():
                 app.state.workflow_threads.pop(t_name)
+
+
+@app.exception_handler(RequestValidationError)
+async def validation_exception_handler(
+    request: Request, exc: RequestValidationError
+):
+    return UJSONResponse(
+        status_code=st.HTTP_422_UNPROCESSABLE_ENTITY,
+        content=jsonable_encoder({"detail": exc.errors(), "body": exc.body}),
+    )
+
+
+if __name__ == "__main__":
+    import uvicorn
+
+    uvicorn.run(
+        app,
+        host="0.0.0.0",
+        port=80,
+    )

{ddeutil_workflow-0.0.34 → ddeutil_workflow-0.0.36}/src/ddeutil/workflow/api/repeat.py

@@ -21,17 +21,26 @@ logger = get_logger("ddeutil.workflow")
 def get_cronjob_delta(cron: str) -> float:
     """This function returns the time delta between now and the next cron
     execution time.
+
+    :rtype: float
     """
     now: datetime = datetime.now(tz=config.tz)
     cron = CronJob(cron)
     return (cron.schedule(now).next - now).total_seconds()
 
 
-def cron_valid(cron: str):
+def cron_valid(cron: str, raise_error: bool = True) -> bool:
+    """Check this crontab string value is valid with its cron syntax.
+
+    :rtype: bool
+    """
     try:
         CronJob(cron)
+        return True
     except Exception as err:
-        raise ValueError(f"Crontab value does not valid, {cron}") from err
+        if raise_error:
+            raise ValueError(f"Crontab value does not valid, {cron}") from err
+        return False
 
 
 async def run_func(
@@ -41,6 +50,7 @@ async def run_func(
     raise_exceptions: bool = False,
     **kwargs,
 ):
+    """Run function inside the repeat decorator functions."""
     try:
         if is_coroutine:
             await func(*args, **kwargs)
@@ -62,11 +72,11 @@ def repeat_at(
     """This function returns a decorator that makes a function execute
     periodically as per the cron expression provided.
 
-    :param cron: str
-        Cron-style string for periodic execution, eg. '0 0 * * *' every midnight
-    :param delay:
-    :param raise_exceptions: bool (default False)
-        Whether to raise exceptions or log them
+    :param cron: (str) A Cron-style string for periodic execution, e.g.
+        '0 0 * * *' every midnight
+    :param delay: (float) A delay seconds value.
+    :param raise_exceptions: (bool) A raise exception flag. Whether to raise
+        exceptions or log them if raise was set be false.
     :param max_repetitions: int (default None)
         Maximum number of times to repeat the function. If None, repeat
         indefinitely.
@@ -81,12 +91,12 @@ def repeat_at(
 
         @wraps(func)
         def wrapper(*_args, **_kwargs):
-            repititions: int = 0
+            repetitions: int = 0
             cron_valid(cron)
 
             async def loop(*args, **kwargs):
-                nonlocal repititions
-                while max_repetitions is None or repititions < max_repetitions:
+                nonlocal repetitions
+                while max_repetitions is None or repetitions < max_repetitions:
                     sleep_time = get_cronjob_delta(cron) + delay
                     await asyncio.sleep(sleep_time)
                     await run_func(
@@ -96,7 +106,7 @@ def repeat_at(
                         raise_exceptions=raise_exceptions,
                         **kwargs,
                     )
-                    repititions += 1
+                    repetitions += 1
 
             ensure_future(loop(*_args, **_kwargs))
 

ddeutil_workflow-0.0.36/src/ddeutil/workflow/api/routes/__init__.py

@@ -0,0 +1,9 @@
+# ------------------------------------------------------------------------------
+# Copyright (c) 2022 Korawich Anuttra. All rights reserved.
+# Licensed under the MIT License. See LICENSE in the project root for
+# license information.
+# ------------------------------------------------------------------------------
+from .job import job_route as job
+from .logs import log_route as log
+from .schedules import schedule_route as schedule
+from .workflows import workflow_route as workflow

ddeutil_workflow-0.0.36/src/ddeutil/workflow/api/routes/job.py

@@ -0,0 +1,73 @@
+# ------------------------------------------------------------------------------
+# Copyright (c) 2022 Korawich Anuttra. All rights reserved.
+# Licensed under the MIT License. See LICENSE in the project root for
+# license information.
+# ------------------------------------------------------------------------------
+from __future__ import annotations
+
+from typing import Any, Optional
+
+from fastapi import APIRouter
+from fastapi.responses import UJSONResponse
+from pydantic import BaseModel
+
+from ...__types import DictData
+from ...conf import get_logger
+from ...exceptions import JobException
+from ...job import Job
+from ...result import Result
+
+logger = get_logger("ddeutil.workflow")
+
+
+job_route = APIRouter(
+    prefix="/job",
+    tags=["job"],
+    default_response_class=UJSONResponse,
+)
+
+
+class ResultPost(BaseModel):
+    context: DictData
+    run_id: str
+    parent_run_id: Optional[str] = None
+
+
+@job_route.post(path="/execute/")
+async def job_execute(
+    result: ResultPost,
+    job: Job,
+    params: dict[str, Any],
+):
+    """Execute job via API."""
+    rs: Result = Result(
+        context=result.context,
+        run_id=result.run_id,
+        parent_run_id=result.parent_run_id,
+    )
+    try:
+        job.set_outputs(
+            job.execute(
+                params=params,
+                run_id=rs.run_id,
+                parent_run_id=rs.parent_run_id,
+            ).context,
+            to=params,
+        )
+    except JobException as err:
+        rs.trace.error(f"[WORKFLOW]: {err.__class__.__name__}: {err}")
+
+    return {
+        "message": "Start execute job via API.",
+        "result": {
+            "run_id": rs.run_id,
+            "parent_run_id": rs.parent_run_id,
+        },
+        "job": job.model_dump(
+            by_alias=True,
+            exclude_none=True,
+            exclude_unset=True,
+            exclude_defaults=True,
+        ),
+        "params": params,
+    }

ddeutil_workflow-0.0.36/src/ddeutil/workflow/api/routes/logs.py

@@ -0,0 +1,64 @@
+# ------------------------------------------------------------------------------
+# Copyright (c) 2022 Korawich Anuttra. All rights reserved.
+# Licensed under the MIT License. See LICENSE in the project root for
+# license information.
+# ------------------------------------------------------------------------------
+"""This route include audit and trace log paths."""
+from __future__ import annotations
+
+from fastapi import APIRouter
+from fastapi.responses import UJSONResponse
+
+from ...audit import get_audit
+from ...logs import get_trace_obj
+
+log_route = APIRouter(
+    prefix="/logs",
+    tags=["logs", "trace", "audit"],
+    default_response_class=UJSONResponse,
+)
+
+
+@log_route.get(path="/trace/")
+async def get_traces():
+    """Get all trace logs."""
+    return {
+        "message": "Getting trace logs",
+        "traces": list(get_trace_obj().find_logs()),
+    }
+
+
+@log_route.get(path="/trace/{run_id}")
+async def get_trace_with_id(run_id: str):
+    """Get trace log with specific running ID."""
+    return get_trace_obj().find_log_with_id(run_id)
+
+
+@log_route.get(path="/audit/")
+async def get_audits():
+    """Get all audit logs."""
+    return {
+        "message": "Getting audit logs",
+        "audits": list(get_audit().find_audits(name="demo")),
+    }
+
+
+@log_route.get(path="/audit/{workflow}/")
+async def get_audit_with_workflow(workflow: str):
+    """Get all audit logs."""
+    return {
+        "message": f"Getting audit logs with workflow name {workflow}",
+        "audits": list(get_audit().find_audits(name="demo")),
+    }
+
+
+@log_route.get(path="/audit/{workflow}/{release}")
+async def get_audit_with_workflow_release(workflow: str, release: str):
+    """Get all audit logs."""
+    return {
+        "message": (
+            f"Getting audit logs with workflow name {workflow} and release "
+            f"{release}"
+        ),
+        "audits": list(get_audit().find_audits(name="demo")),
+    }