ddeutil-workflow 0.0.28__tar.gz → 0.0.30__tar.gz

{ddeutil_workflow-0.0.28 → ddeutil_workflow-0.0.30}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.2
 Name: ddeutil-workflow
-Version: 0.0.28
+Version: 0.0.30
 Summary: Lightweight workflow orchestration
 Author-email: ddeutils <korawich.anu@gmail.com>
 License: MIT
@@ -22,7 +22,7 @@ Classifier: Programming Language :: Python :: 3.13
 Requires-Python: >=3.9.13
 Description-Content-Type: text/markdown
 License-File: LICENSE
-Requires-Dist: ddeutil==0.4.6
+Requires-Dist: ddeutil>=0.4.6
 Requires-Dist: ddeutil-io[toml,yaml]>=0.2.3
 Requires-Dist: pydantic==2.10.6
 Requires-Dist: python-dotenv==1.0.1
@@ -55,6 +55,8 @@ the input parameters per use-case instead.
 This way I can handle a lot of logical workflows in our orgs with only metadata
 configuration. It called **Metadata Driven Data Workflow**.
 
+---
+
 **:pushpin: <u>Rules of This Workflow engine</u>**:
 
 1. The Minimum frequency unit of scheduling is **1 minute** :warning:
@@ -62,8 +64,14 @@ configuration. It called **Metadata Driven Data Workflow**.
 3. All parallel tasks inside workflow engine use Multi-Threading
    (Python 3.13 unlock GIL :unlock:)
 
+---
+
 **:memo: <u>Workflow Diagrams</u>**:
 
+This diagram show where is this application run on the production infrastructure.
+You will see that this application do only running code with stress-less which mean
+you should to set the data layer separate this core program before run this application.
+
 ```mermaid
 flowchart LR
     subgraph Interface
@@ -204,9 +212,9 @@ schedule-run-local-wf:
 
 ## :cookie: Configuration
 
-The main configuration that use to dynamic changing with your objective of this
-application. If any configuration values do not set yet, it will use default value
-and do not raise any error to you.
+The main configuration that use to dynamic changing this workflow engine for your
+objective use environment variable only. If any configuration values do not set yet,
+it will use default value and do not raise any error to you.
 
 > [!IMPORTANT]
 > The config value that you will set on the environment should combine with
@@ -215,8 +223,8 @@ and do not raise any error to you.
 | Name                         | Component | Default                           | Description                                                                                                        |
 |:-----------------------------|:---------:|:----------------------------------|:-------------------------------------------------------------------------------------------------------------------|
 | **ROOT_PATH**                |   Core    | `.`                               | The root path of the workflow application.                                                                         |
-| **REGISTRY**                 |   Core    | `src`                             | List of importable string for the hook stage.                                                                      |
-| **REGISTRY_FILTER**          |   Core    | `ddeutil.workflow.utils`          | List of importable string for the filter template.                                                                 |
+| **REGISTRY**                 |   Core    | `.`                               | List of importable string for the hook stage.                                                                      |
+| **REGISTRY_FILTER**          |   Core    | `ddeutil.workflow.templates`      | List of importable string for the filter template.                                                                 |
 | **CONF_PATH**                |   Core    | `conf`                            | The config path that keep all template `.yaml` files.                                                              |
 | **TIMEZONE**                 |   Core    | `Asia/Bangkok`                    | A Timezone string value that will pass to `ZoneInfo` object.                                                       |
 | **STAGE_DEFAULT_ID**         |   Core    | `true`                            | A flag that enable default stage ID that use for catch an execution output.                                        |
@@ -238,6 +246,9 @@ and do not raise any error to you.
 
 **API Application**:
 
+This config part use for the workflow application that build from the FastAPI
+only.
+
 | Environment                |  Component  | Default | Description                                                                        |
 |:---------------------------|:-----------:|---------|------------------------------------------------------------------------------------|
 | **ENABLE_ROUTE_WORKFLOW**  |     API     | `true`  | A flag that enable workflow route to manage execute manually and workflow logging. |
@@ -264,16 +275,9 @@ like crontab job but via Python API.
 
 ### Docker Container
 
-Create Docker image;
-
 ```shell
 $ docker build -t ddeutil-workflow:latest -f .container/Dockerfile .
-```
-
-Run the above Docker image;
-
-```shell
-$ docker run -i ddeutil-workflow:latest
+$ docker run -i ddeutil-workflow:latest ddeutil-workflow
 ```
 
 ## :speech_balloon: Contribute

{ddeutil_workflow-0.0.28 → ddeutil_workflow-0.0.30}/README.md

@@ -23,6 +23,8 @@ the input parameters per use-case instead.
 This way I can handle a lot of logical workflows in our orgs with only metadata
 configuration. It called **Metadata Driven Data Workflow**.
 
+---
+
 **:pushpin: <u>Rules of This Workflow engine</u>**:
 
 1. The Minimum frequency unit of scheduling is **1 minute** :warning:
@@ -30,8 +32,14 @@ configuration. It called **Metadata Driven Data Workflow**.
 3. All parallel tasks inside workflow engine use Multi-Threading
    (Python 3.13 unlock GIL :unlock:)
 
+---
+
 **:memo: <u>Workflow Diagrams</u>**:
 
+This diagram show where is this application run on the production infrastructure.
+You will see that this application do only running code with stress-less which mean
+you should to set the data layer separate this core program before run this application.
+
 ```mermaid
 flowchart LR
     subgraph Interface
@@ -172,9 +180,9 @@ schedule-run-local-wf:
 
 ## :cookie: Configuration
 
-The main configuration that use to dynamic changing with your objective of this
-application. If any configuration values do not set yet, it will use default value
-and do not raise any error to you.
+The main configuration that use to dynamic changing this workflow engine for your
+objective use environment variable only. If any configuration values do not set yet,
+it will use default value and do not raise any error to you.
 
 > [!IMPORTANT]
 > The config value that you will set on the environment should combine with
@@ -183,8 +191,8 @@ and do not raise any error to you.
 | Name                         | Component | Default                           | Description                                                                                                        |
 |:-----------------------------|:---------:|:----------------------------------|:-------------------------------------------------------------------------------------------------------------------|
 | **ROOT_PATH**                |   Core    | `.`                               | The root path of the workflow application.                                                                         |
-| **REGISTRY**                 |   Core    | `src`                             | List of importable string for the hook stage.                                                                      |
-| **REGISTRY_FILTER**          |   Core    | `ddeutil.workflow.utils`          | List of importable string for the filter template.                                                                 |
+| **REGISTRY**                 |   Core    | `.`                               | List of importable string for the hook stage.                                                                      |
+| **REGISTRY_FILTER**          |   Core    | `ddeutil.workflow.templates`      | List of importable string for the filter template.                                                                 |
 | **CONF_PATH**                |   Core    | `conf`                            | The config path that keep all template `.yaml` files.                                                              |
 | **TIMEZONE**                 |   Core    | `Asia/Bangkok`                    | A Timezone string value that will pass to `ZoneInfo` object.                                                       |
 | **STAGE_DEFAULT_ID**         |   Core    | `true`                            | A flag that enable default stage ID that use for catch an execution output.                                        |
@@ -206,6 +214,9 @@ and do not raise any error to you.
 
 **API Application**:
 
+This config part use for the workflow application that build from the FastAPI
+only.
+
 | Environment                |  Component  | Default | Description                                                                        |
 |:---------------------------|:-----------:|---------|------------------------------------------------------------------------------------|
 | **ENABLE_ROUTE_WORKFLOW**  |     API     | `true`  | A flag that enable workflow route to manage execute manually and workflow logging. |
@@ -232,16 +243,9 @@ like crontab job but via Python API.
 
 ### Docker Container
 
-Create Docker image;
-
 ```shell
 $ docker build -t ddeutil-workflow:latest -f .container/Dockerfile .
-```
-
-Run the above Docker image;
-
-```shell
-$ docker run -i ddeutil-workflow:latest
+$ docker run -i ddeutil-workflow:latest ddeutil-workflow
 ```
 
 ## :speech_balloon: Contribute

{ddeutil_workflow-0.0.28 → ddeutil_workflow-0.0.30}/pyproject.toml

@@ -26,7 +26,7 @@ classifiers = [
 ]
 requires-python = ">=3.9.13"
 dependencies = [
-    "ddeutil==0.4.6",
+    "ddeutil>=0.4.6",
     "ddeutil-io[yaml,toml]>=0.2.3",
     "pydantic==2.10.6",
     "python-dotenv==1.0.1",
@@ -77,6 +77,12 @@ exclude_lines = [
 
 [tool.pytest.ini_options]
 pythonpath = ["src"]
+# NOTE: You can deslect multiple markers by '-m "not (poke or api)"'
+markers = [
+    "poke: marks tests as slow by poking (deselect with '-m \"not poke\"')",
+    "schedule: marks tests as schedule (deselect with '-m \"not schedule\"')",
+    "api: marks tests as api (deselect with '-m \"not api\"')",
+]
 console_output_style = "count"
 addopts = [
     "--strict-config",

ddeutil_workflow-0.0.30/src/ddeutil/workflow/__about__.py

@@ -0,0 +1 @@
+__version__: str = "0.0.30"

{ddeutil_workflow-0.0.28 → ddeutil_workflow-0.0.30}/src/ddeutil/workflow/__init__.py

@@ -4,6 +4,7 @@
 # license information.
 # ------------------------------------------------------------------------------
 from .__cron import CronJob, CronRunner
+from .__types import Re
 from .conf import (
     Config,
     Loader,

{ddeutil_workflow-0.0.28 → ddeutil_workflow-0.0.30}/src/ddeutil/workflow/__types.py

@@ -27,6 +27,8 @@ Matrix = dict[str, Union[list[str], list[int]]]
 
 
 class Context(TypedDict):
+    """TypeDict support the Context."""
+
     params: dict[str, Any]
     jobs: dict[str, Any]
 
@@ -71,14 +73,14 @@ class Re:
     #       - ${{ params.source?.schema }}
     #
     __re_caller: str = r"""
-        \$
-        {{
-            \s*
+        \$                                                      # start with $
+        {{                                                      # value open with {{
+            \s*                                                 # whitespace or not
             (?P<caller>
                 (?P<caller_prefix>(?:[a-zA-Z_-]+\??\.)*)
                 (?P<caller_last>[a-zA-Z0-9_\-.'\"(\)[\]{}]+\??)
             )
-            \s*
+            \s*                                                 # whitespace or not
             (?P<post_filters>
                 (?:
                     \|\s*
@@ -88,7 +90,7 @@ class Re:
                     )\s*
                 )*
             )
-        }}
+        }}                                                      # value close with }}
     """
     RE_CALLER: Pattern = re.compile(
         __re_caller, MULTILINE | IGNORECASE | UNICODE | VERBOSE
@@ -103,13 +105,13 @@ class Re:
     #       - tasks/function@dummy
     #
     __re_task_fmt: str = r"""
-        ^
+        ^                               # start task format
             (?P<path>[^/@]+)
-            /
+            /                           # start get function with /
             (?P<func>[^@]+)
-            @
+            @                           # start tag with @
             (?P<tag>.+)
-        $
+        $                               # end task format
     """
     RE_TASK_FMT: Pattern = re.compile(
         __re_task_fmt, MULTILINE | IGNORECASE | UNICODE | VERBOSE

{ddeutil_workflow-0.0.28 → ddeutil_workflow-0.0.30}/src/ddeutil/workflow/api/api.py

@@ -18,7 +18,7 @@ from fastapi.responses import UJSONResponse
 from ..__about__ import __version__
 from ..conf import config, get_logger
 from ..scheduler import ReleaseThread, ReleaseThreads
-from ..workflow import WorkflowQueue, WorkflowTask
+from ..workflow import ReleaseQueue, WorkflowTask
 from .repeat import repeat_at
 
 load_dotenv()
@@ -31,7 +31,7 @@ class State(TypedDict):
     scheduler: list[str]
     workflow_threads: ReleaseThreads
     workflow_tasks: list[WorkflowTask]
-    workflow_queue: dict[str, WorkflowQueue]
+    workflow_queue: dict[str, ReleaseQueue]
 
 
 @contextlib.asynccontextmanager

{ddeutil_workflow-0.0.28 → ddeutil_workflow-0.0.30}/src/ddeutil/workflow/conf.py

@@ -13,7 +13,7 @@ from collections.abc import Iterator
 from datetime import datetime, timedelta
 from functools import cached_property, lru_cache
 from pathlib import Path
-from typing import ClassVar, Optional, Union
+from typing import ClassVar, Optional, TypeVar, Union
 from zoneinfo import ZoneInfo
 
 from ddeutil.core import str2bool
@@ -39,6 +39,7 @@ __all__: TupleStr = (
     "env",
     "get_logger",
     "get_log",
+    "C",
     "Config",
     "SimLoad",
     "Loader",
@@ -81,18 +82,42 @@ def get_logger(name: str):
     return lg
 
 
-class Config:  # pragma: no cov
+class BaseConfig:  # pragma: no cov
+    """BaseConfig object inheritable."""
+
+    __slots__ = ()
+
+    @property
+    def root_path(self) -> Path:
+        """Root path or the project path.
+
+        :rtype: Path
+        """
+        return Path(os.getenv("ROOT_PATH", "."))
+
+    @property
+    def conf_path(self) -> Path:
+        """Config path that use root_path class argument for this construction.
+
+        :rtype: Path
+        """
+        return self.root_path / os.getenv("CONF_PATH", "conf")
+
+
+class Config(BaseConfig):  # pragma: no cov
     """Config object for keeping core configurations on the current session
     without changing when if the application still running.
 
         The config value can change when you call that config property again.
     """
 
-    __slots__ = ()
-
     # NOTE: Core
     @property
     def root_path(self) -> Path:
+        """Root path or the project path.
+
+        :rtype: Path
+        """
         return Path(env("CORE_ROOT_PATH", "."))
 
     @property
@@ -114,7 +139,7 @@ class Config:  # pragma: no cov
     # NOTE: Register
     @property
     def regis_hook(self) -> list[str]:
-        regis_hook_str: str = env("CORE_REGISTRY", "src")
+        regis_hook_str: str = env("CORE_REGISTRY", ".")
         return [r.strip() for r in regis_hook_str.split(",")]
 
     @property
@@ -220,6 +245,9 @@ class Config:  # pragma: no cov
         return str2bool(env("API_ENABLE_ROUTE_SCHEDULE", "true"))
 
 
+C = TypeVar("C", bound=BaseConfig)
+
+
 class SimLoad:
     """Simple Load Object that will search config data by given some identity
     value like name of workflow or on.
@@ -243,7 +271,7 @@ class SimLoad:
     def __init__(
         self,
         name: str,
-        conf: Config,
+        conf: C,
         externals: DictData | None = None,
     ) -> None:
         self.data: DictData = {}
@@ -256,7 +284,7 @@ class SimLoad:
         if not self.data:
             raise ValueError(f"Config {name!r} does not found on conf path")
 
-        self.conf: Config = conf
+        self.conf: C = conf
         self.externals: DictData = externals or {}
         self.data.update(self.externals)
 
@@ -264,7 +292,7 @@ class SimLoad:
     def finds(
         cls,
         obj: object,
-        conf: Config,
+        conf: C,
         *,
         included: list[str] | None = None,
         excluded: list[str] | None = None,
@@ -273,7 +301,7 @@ class SimLoad:
         method can use include and exclude list of identity name for filter and
         adds-on.
 
-        :param obj: A object that want to validate matching before return.
+        :param obj: An object that want to validate matching before return.
         :param conf: A config object.
         :param included:
         :param excluded:
@@ -338,7 +366,7 @@ class Loader(SimLoad):
     ) -> Iterator[tuple[str, DictData]]:
         """Override the find class method from the Simple Loader object.
 
-        :param obj: A object that want to validate matching before return.
+        :param obj: An object that want to validate matching before return.
         :param included:
         :param excluded:
 

{ddeutil_workflow-0.0.28 → ddeutil_workflow-0.0.30}/src/ddeutil/workflow/hook.py

@@ -50,6 +50,7 @@ def tag(
     :param: name: A tag name for make different use-case of a function.
     :param: alias: A alias function name that keeping in registries. If this
         value does not supply, it will use original function name from __name__.
+
     :rtype: Callable[P, TagFunc]
     """
 
@@ -58,7 +59,7 @@ def tag(
         func.name = alias or func.__name__.replace("_", "-")
 
         @wraps(func)
-        def wrapped(*args, **kwargs):
+        def wrapped(*args: P.args, **kwargs: P.kwargs) -> TagFunc:
             # NOTE: Able to do anything before calling hook function.
             return func(*args, **kwargs)
 
@@ -74,10 +75,13 @@ def make_registry(submodule: str) -> dict[str, Registry]:
     """Return registries of all functions that able to called with task.
 
     :param submodule: A module prefix that want to import registry.
+
     :rtype: dict[str, Registry]
     """
     rs: dict[str, Registry] = {}
-    for module in config.regis_hook:
+    regis_hooks: list[str] = config.regis_hook
+    regis_hooks.extend(["ddeutil.vendors"])
+    for module in regis_hooks:
         # NOTE: try to sequential import task functions
         try:
             importer = import_module(f"{module}.{submodule}")

{ddeutil_workflow-0.0.28 → ddeutil_workflow-0.0.30}/src/ddeutil/workflow/scheduler.py

@@ -58,7 +58,7 @@ from .utils import (
     batch,
     delay,
 )
-from .workflow import Workflow, WorkflowQueue, WorkflowRelease, WorkflowTask
+from .workflow import Release, ReleaseQueue, Workflow, WorkflowTask
 
 P = ParamSpec("P")
 logger = get_logger("ddeutil.workflow")
@@ -170,7 +170,7 @@ class WorkflowSchedule(BaseModel):
     def tasks(
         self,
         start_date: datetime,
-        queue: dict[str, WorkflowQueue],
+        queue: dict[str, ReleaseQueue],
         *,
         externals: DictData | None = None,
     ) -> list[WorkflowTask]:
@@ -193,7 +193,7 @@ class WorkflowSchedule(BaseModel):
 
         # NOTE: Loading workflow model from the name of workflow.
         wf: Workflow = Workflow.from_loader(self.name, externals=extras)
-        wf_queue: WorkflowQueue = queue[self.alias]
+        wf_queue: ReleaseQueue = queue[self.alias]
 
         # IMPORTANT: Create the default 'on' value if it does not pass the `on`
         #   field to the Schedule object.
@@ -280,7 +280,7 @@ class Schedule(BaseModel):
     def tasks(
         self,
         start_date: datetime,
-        queue: dict[str, WorkflowQueue],
+        queue: dict[str, ReleaseQueue],
         *,
         externals: DictData | None = None,
     ) -> list[WorkflowTask]:
@@ -289,7 +289,7 @@ class Schedule(BaseModel):
 
         :param start_date: A start date that get from the workflow schedule.
         :param queue: A mapping of name and list of datetime for queue.
-        :type queue: dict[str, WorkflowQueue]
+        :type queue: dict[str, ReleaseQueue]
         :param externals: An external parameters that pass to the Loader object.
         :type externals: DictData | None
 
@@ -302,7 +302,7 @@ class Schedule(BaseModel):
         for workflow in self.workflows:
 
             if workflow.alias not in queue:
-                queue[workflow.alias] = WorkflowQueue()
+                queue[workflow.alias] = ReleaseQueue()
 
             workflow_tasks.extend(
                 workflow.tasks(start_date, queue=queue, externals=externals)
@@ -355,7 +355,7 @@ ReleaseThreads = dict[str, ReleaseThread]
 def schedule_task(
     tasks: list[WorkflowTask],
     stop: datetime,
-    queue: dict[str, WorkflowQueue],
+    queue: dict[str, ReleaseQueue],
     threads: ReleaseThreads,
     log: type[Log],
 ) -> CancelJob | None:
@@ -366,7 +366,7 @@ def schedule_task(
 
     :param tasks: A list of WorkflowTask object.
     :param stop: A stop datetime object that force stop running scheduler.
-    :param queue: A mapping of alias name and WorkflowQueue object.
+    :param queue: A mapping of alias name and ReleaseQueue object.
     :param threads: A mapping of alias name and Thread object.
     :param log: A log class that want to make log object.
 
@@ -390,7 +390,7 @@ def schedule_task(
     #
     for task in tasks:
 
-        q: WorkflowQueue = queue[task.alias]
+        q: ReleaseQueue = queue[task.alias]
 
         # NOTE: Start adding queue and move the runner date in the WorkflowTask.
         task.queue(stop, q, log=log)
@@ -418,7 +418,7 @@ def schedule_task(
             continue
 
         # NOTE: Pop the latest release and push it to running.
-        release: WorkflowRelease = heappop(q.queue)
+        release: Release = heappop(q.queue)
         heappush(q.running, release)
 
         logger.info(
@@ -499,7 +499,7 @@ def schedule_control(
     stop_date: datetime = stop or (start_date + config.stop_boundary_delta)
 
     # IMPORTANT: Create main mapping of queue and thread object.
-    queue: dict[str, WorkflowQueue] = {}
+    queue: dict[str, ReleaseQueue] = {}
     threads: ReleaseThreads = {}
 
     start_date_waiting: datetime = start_date.replace(

{ddeutil_workflow-0.0.28 → ddeutil_workflow-0.0.30}/src/ddeutil/workflow/templates.py

@@ -295,10 +295,7 @@ def str2template(
     return search_env_replace(value)
 
 
-def param2template(
-    value: Any,
-    params: DictData,
-) -> Any:
+def param2template(value: Any, params: DictData) -> Any:
     """Pass param to template string that can search by ``RE_CALLER`` regular
     expression.