ddeutil-workflow 0.0.28__tar.gz → 0.0.29__tar.gz

{ddeutil_workflow-0.0.28 → ddeutil_workflow-0.0.29}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.2
 Name: ddeutil-workflow
-Version: 0.0.28
+Version: 0.0.29
 Summary: Lightweight workflow orchestration
 Author-email: ddeutils <korawich.anu@gmail.com>
 License: MIT
@@ -22,7 +22,7 @@ Classifier: Programming Language :: Python :: 3.13
 Requires-Python: >=3.9.13
 Description-Content-Type: text/markdown
 License-File: LICENSE
-Requires-Dist: ddeutil==0.4.6
+Requires-Dist: ddeutil>=0.4.6
 Requires-Dist: ddeutil-io[toml,yaml]>=0.2.3
 Requires-Dist: pydantic==2.10.6
 Requires-Dist: python-dotenv==1.0.1
@@ -55,6 +55,8 @@ the input parameters per use-case instead.
 This way I can handle a lot of logical workflows in our orgs with only metadata
 configuration. It called **Metadata Driven Data Workflow**.
 
+---
+
 **:pushpin: <u>Rules of This Workflow engine</u>**:
 
 1. The Minimum frequency unit of scheduling is **1 minute** :warning:
@@ -62,8 +64,14 @@ configuration. It called **Metadata Driven Data Workflow**.
 3. All parallel tasks inside workflow engine use Multi-Threading
    (Python 3.13 unlock GIL :unlock:)
 
+---
+
 **:memo: <u>Workflow Diagrams</u>**:
 
+This diagram show where is this application run on the production infrastructure.
+You will see that this application do only running code with stress-less which mean
+you should to set the data layer separate this core program before run this application.
+
 ```mermaid
 flowchart LR
     subgraph Interface
@@ -215,8 +223,8 @@ and do not raise any error to you.
 | Name                         | Component | Default                           | Description                                                                                                        |
 |:-----------------------------|:---------:|:----------------------------------|:-------------------------------------------------------------------------------------------------------------------|
 | **ROOT_PATH**                |   Core    | `.`                               | The root path of the workflow application.                                                                         |
-| **REGISTRY**                 |   Core    | `src`                             | List of importable string for the hook stage.                                                                      |
-| **REGISTRY_FILTER**          |   Core    | `ddeutil.workflow.utils`          | List of importable string for the filter template.                                                                 |
+| **REGISTRY**                 |   Core    | `.`                               | List of importable string for the hook stage.                                                                      |
+| **REGISTRY_FILTER**          |   Core    | `ddeutil.workflow.templates`      | List of importable string for the filter template.                                                                 |
 | **CONF_PATH**                |   Core    | `conf`                            | The config path that keep all template `.yaml` files.                                                              |
 | **TIMEZONE**                 |   Core    | `Asia/Bangkok`                    | A Timezone string value that will pass to `ZoneInfo` object.                                                       |
 | **STAGE_DEFAULT_ID**         |   Core    | `true`                            | A flag that enable default stage ID that use for catch an execution output.                                        |

{ddeutil_workflow-0.0.28 → ddeutil_workflow-0.0.29}/README.md

@@ -23,6 +23,8 @@ the input parameters per use-case instead.
 This way I can handle a lot of logical workflows in our orgs with only metadata
 configuration. It called **Metadata Driven Data Workflow**.
 
+---
+
 **:pushpin: <u>Rules of This Workflow engine</u>**:
 
 1. The Minimum frequency unit of scheduling is **1 minute** :warning:
@@ -30,8 +32,14 @@ configuration. It called **Metadata Driven Data Workflow**.
 3. All parallel tasks inside workflow engine use Multi-Threading
    (Python 3.13 unlock GIL :unlock:)
 
+---
+
 **:memo: <u>Workflow Diagrams</u>**:
 
+This diagram show where is this application run on the production infrastructure.
+You will see that this application do only running code with stress-less which mean
+you should to set the data layer separate this core program before run this application.
+
 ```mermaid
 flowchart LR
     subgraph Interface
@@ -183,8 +191,8 @@ and do not raise any error to you.
 | Name                         | Component | Default                           | Description                                                                                                        |
 |:-----------------------------|:---------:|:----------------------------------|:-------------------------------------------------------------------------------------------------------------------|
 | **ROOT_PATH**                |   Core    | `.`                               | The root path of the workflow application.                                                                         |
-| **REGISTRY**                 |   Core    | `src`                             | List of importable string for the hook stage.                                                                      |
-| **REGISTRY_FILTER**          |   Core    | `ddeutil.workflow.utils`          | List of importable string for the filter template.                                                                 |
+| **REGISTRY**                 |   Core    | `.`                               | List of importable string for the hook stage.                                                                      |
+| **REGISTRY_FILTER**          |   Core    | `ddeutil.workflow.templates`      | List of importable string for the filter template.                                                                 |
 | **CONF_PATH**                |   Core    | `conf`                            | The config path that keep all template `.yaml` files.                                                              |
 | **TIMEZONE**                 |   Core    | `Asia/Bangkok`                    | A Timezone string value that will pass to `ZoneInfo` object.                                                       |
 | **STAGE_DEFAULT_ID**         |   Core    | `true`                            | A flag that enable default stage ID that use for catch an execution output.                                        |

{ddeutil_workflow-0.0.28 → ddeutil_workflow-0.0.29}/pyproject.toml

@@ -26,7 +26,7 @@ classifiers = [
 ]
 requires-python = ">=3.9.13"
 dependencies = [
-    "ddeutil==0.4.6",
+    "ddeutil>=0.4.6",
     "ddeutil-io[yaml,toml]>=0.2.3",
     "pydantic==2.10.6",
     "python-dotenv==1.0.1",
@@ -77,6 +77,12 @@ exclude_lines = [
 
 [tool.pytest.ini_options]
 pythonpath = ["src"]
+# NOTE: You can deslect multiple markers by '-m "not (poke or api)"'
+markers = [
+    "poke: marks tests as slow by poking (deselect with '-m \"not poke\"')",
+    "schedule: marks tests as schedule (deselect with '-m \"not schedule\"')",
+    "api: marks tests as api (deselect with '-m \"not api\"')",
+]
 console_output_style = "count"
 addopts = [
     "--strict-config",

ddeutil_workflow-0.0.29/src/ddeutil/workflow/__about__.py

@@ -0,0 +1 @@
+__version__: str = "0.0.29"

{ddeutil_workflow-0.0.28 → ddeutil_workflow-0.0.29}/src/ddeutil/workflow/__init__.py

@@ -4,6 +4,7 @@
 # license information.
 # ------------------------------------------------------------------------------
 from .__cron import CronJob, CronRunner
+from .__types import Re
 from .conf import (
     Config,
     Loader,

{ddeutil_workflow-0.0.28 → ddeutil_workflow-0.0.29}/src/ddeutil/workflow/__types.py

@@ -27,6 +27,8 @@ Matrix = dict[str, Union[list[str], list[int]]]
 
 
 class Context(TypedDict):
+    """TypeDict support the Context."""
+
     params: dict[str, Any]
     jobs: dict[str, Any]
 
@@ -71,14 +73,14 @@ class Re:
     #       - ${{ params.source?.schema }}
     #
     __re_caller: str = r"""
-        \$
-        {{
-            \s*
+        \$                                                      # start with $
+        {{                                                      # value open with {{
+            \s*                                                 # whitespace or not
             (?P<caller>
                 (?P<caller_prefix>(?:[a-zA-Z_-]+\??\.)*)
                 (?P<caller_last>[a-zA-Z0-9_\-.'\"(\)[\]{}]+\??)
             )
-            \s*
+            \s*                                                 # whitespace or not
             (?P<post_filters>
                 (?:
                     \|\s*
@@ -88,7 +90,7 @@ class Re:
                     )\s*
                 )*
             )
-        }}
+        }}                                                      # value close with }}
     """
     RE_CALLER: Pattern = re.compile(
         __re_caller, MULTILINE | IGNORECASE | UNICODE | VERBOSE
@@ -103,13 +105,13 @@ class Re:
     #       - tasks/function@dummy
     #
     __re_task_fmt: str = r"""
-        ^
+        ^                               # start task format
             (?P<path>[^/@]+)
-            /
+            /                           # start get function with /
             (?P<func>[^@]+)
-            @
+            @                           # start tag with @
             (?P<tag>.+)
-        $
+        $                               # end task format
     """
     RE_TASK_FMT: Pattern = re.compile(
         __re_task_fmt, MULTILINE | IGNORECASE | UNICODE | VERBOSE

{ddeutil_workflow-0.0.28 → ddeutil_workflow-0.0.29}/src/ddeutil/workflow/conf.py

@@ -13,7 +13,7 @@ from collections.abc import Iterator
 from datetime import datetime, timedelta
 from functools import cached_property, lru_cache
 from pathlib import Path
-from typing import ClassVar, Optional, Union
+from typing import ClassVar, Optional, TypeVar, Union
 from zoneinfo import ZoneInfo
 
 from ddeutil.core import str2bool
@@ -39,6 +39,7 @@ __all__: TupleStr = (
     "env",
     "get_logger",
     "get_log",
+    "C",
     "Config",
     "SimLoad",
     "Loader",
@@ -81,18 +82,42 @@ def get_logger(name: str):
     return lg
 
 
-class Config:  # pragma: no cov
+class BaseConfig:  # pragma: no cov
+    """BaseConfig object inheritable."""
+
+    __slots__ = ()
+
+    @property
+    def root_path(self) -> Path:
+        """Root path or the project path.
+
+        :rtype: Path
+        """
+        return Path(os.getenv("ROOT_PATH", "."))
+
+    @property
+    def conf_path(self) -> Path:
+        """Config path that use root_path class argument for this construction.
+
+        :rtype: Path
+        """
+        return self.root_path / os.getenv("CONF_PATH", "conf")
+
+
+class Config(BaseConfig):  # pragma: no cov
     """Config object for keeping core configurations on the current session
     without changing when if the application still running.
 
         The config value can change when you call that config property again.
     """
 
-    __slots__ = ()
-
     # NOTE: Core
     @property
     def root_path(self) -> Path:
+        """Root path or the project path.
+
+        :rtype: Path
+        """
         return Path(env("CORE_ROOT_PATH", "."))
 
     @property
@@ -114,7 +139,7 @@ class Config:  # pragma: no cov
     # NOTE: Register
     @property
     def regis_hook(self) -> list[str]:
-        regis_hook_str: str = env("CORE_REGISTRY", "src")
+        regis_hook_str: str = env("CORE_REGISTRY", ".")
         return [r.strip() for r in regis_hook_str.split(",")]
 
     @property
@@ -220,6 +245,9 @@ class Config:  # pragma: no cov
         return str2bool(env("API_ENABLE_ROUTE_SCHEDULE", "true"))
 
 
+C = TypeVar("C", bound=BaseConfig)
+
+
 class SimLoad:
     """Simple Load Object that will search config data by given some identity
     value like name of workflow or on.
@@ -243,7 +271,7 @@ class SimLoad:
     def __init__(
         self,
         name: str,
-        conf: Config,
+        conf: C,
         externals: DictData | None = None,
     ) -> None:
         self.data: DictData = {}
@@ -256,7 +284,7 @@ class SimLoad:
         if not self.data:
             raise ValueError(f"Config {name!r} does not found on conf path")
 
-        self.conf: Config = conf
+        self.conf: C = conf
         self.externals: DictData = externals or {}
         self.data.update(self.externals)
 
@@ -264,7 +292,7 @@ class SimLoad:
     def finds(
         cls,
         obj: object,
-        conf: Config,
+        conf: C,
         *,
         included: list[str] | None = None,
         excluded: list[str] | None = None,
@@ -273,7 +301,7 @@ class SimLoad:
         method can use include and exclude list of identity name for filter and
         adds-on.
 
-        :param obj: A object that want to validate matching before return.
+        :param obj: An object that want to validate matching before return.
         :param conf: A config object.
         :param included:
         :param excluded:
@@ -338,7 +366,7 @@ class Loader(SimLoad):
     ) -> Iterator[tuple[str, DictData]]:
         """Override the find class method from the Simple Loader object.
 
-        :param obj: A object that want to validate matching before return.
+        :param obj: An object that want to validate matching before return.
         :param included:
         :param excluded:
 

{ddeutil_workflow-0.0.28 → ddeutil_workflow-0.0.29}/src/ddeutil/workflow/hook.py

@@ -50,6 +50,7 @@ def tag(
     :param: name: A tag name for make different use-case of a function.
     :param: alias: A alias function name that keeping in registries. If this
         value does not supply, it will use original function name from __name__.
+
     :rtype: Callable[P, TagFunc]
     """
 
@@ -58,7 +59,7 @@ def tag(
         func.name = alias or func.__name__.replace("_", "-")
 
         @wraps(func)
-        def wrapped(*args, **kwargs):
+        def wrapped(*args: P.args, **kwargs: P.kwargs) -> TagFunc:
             # NOTE: Able to do anything before calling hook function.
             return func(*args, **kwargs)
 
@@ -74,10 +75,11 @@ def make_registry(submodule: str) -> dict[str, Registry]:
     """Return registries of all functions that able to called with task.
 
     :param submodule: A module prefix that want to import registry.
+
     :rtype: dict[str, Registry]
     """
     rs: dict[str, Registry] = {}
-    for module in config.regis_hook:
+    for module in config.regis_hook | ["ddeutil.vendors"]:
         # NOTE: try to sequential import task functions
         try:
             importer = import_module(f"{module}.{submodule}")

{ddeutil_workflow-0.0.28 → ddeutil_workflow-0.0.29}/src/ddeutil/workflow/templates.py

@@ -295,10 +295,7 @@ def str2template(
     return search_env_replace(value)
 
 
-def param2template(
-    value: Any,
-    params: DictData,
-) -> Any:
+def param2template(value: Any, params: DictData) -> Any:
     """Pass param to template string that can search by ``RE_CALLER`` regular
     expression.
 

{ddeutil_workflow-0.0.28 → ddeutil_workflow-0.0.29}/src/ddeutil/workflow/workflow.py

@@ -884,7 +884,9 @@ class Workflow(BaseModel):
         :param run_id: A workflow running ID for this job execution.
         :type run_id: str | None (default: None)
         :param timeout: A workflow execution time out in second unit that use
-            for limit time of execution and waiting job dependency.
+            for limit time of execution and waiting job dependency. This value
+            does not force stop the task that still running more than this limit
+            time.
         :type timeout: int (default: 0)
 
         :rtype: Result
@@ -1220,6 +1222,7 @@ class WorkflowTask:
         return queue
 
     def __repr__(self) -> str:
+        """Override ___repr__ method."""
         return (
             f"{self.__class__.__name__}(alias={self.alias!r}, "
             f"workflow={self.workflow.name!r}, runner={self.runner!r}, "

{ddeutil_workflow-0.0.28 → ddeutil_workflow-0.0.29}/src/ddeutil_workflow.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.2
 Name: ddeutil-workflow
-Version: 0.0.28
+Version: 0.0.29
 Summary: Lightweight workflow orchestration
 Author-email: ddeutils <korawich.anu@gmail.com>
 License: MIT
@@ -22,7 +22,7 @@ Classifier: Programming Language :: Python :: 3.13
 Requires-Python: >=3.9.13
 Description-Content-Type: text/markdown
 License-File: LICENSE
-Requires-Dist: ddeutil==0.4.6
+Requires-Dist: ddeutil>=0.4.6
 Requires-Dist: ddeutil-io[toml,yaml]>=0.2.3
 Requires-Dist: pydantic==2.10.6
 Requires-Dist: python-dotenv==1.0.1
@@ -55,6 +55,8 @@ the input parameters per use-case instead.
 This way I can handle a lot of logical workflows in our orgs with only metadata
 configuration. It called **Metadata Driven Data Workflow**.
 
+---
+
 **:pushpin: <u>Rules of This Workflow engine</u>**:
 
 1. The Minimum frequency unit of scheduling is **1 minute** :warning:
@@ -62,8 +64,14 @@ configuration. It called **Metadata Driven Data Workflow**.
 3. All parallel tasks inside workflow engine use Multi-Threading
    (Python 3.13 unlock GIL :unlock:)
 
+---
+
 **:memo: <u>Workflow Diagrams</u>**:
 
+This diagram show where is this application run on the production infrastructure.
+You will see that this application do only running code with stress-less which mean
+you should to set the data layer separate this core program before run this application.
+
 ```mermaid
 flowchart LR
     subgraph Interface
@@ -215,8 +223,8 @@ and do not raise any error to you.
 | Name                         | Component | Default                           | Description                                                                                                        |
 |:-----------------------------|:---------:|:----------------------------------|:-------------------------------------------------------------------------------------------------------------------|
 | **ROOT_PATH**                |   Core    | `.`                               | The root path of the workflow application.                                                                         |
-| **REGISTRY**                 |   Core    | `src`                             | List of importable string for the hook stage.                                                                      |
-| **REGISTRY_FILTER**          |   Core    | `ddeutil.workflow.utils`          | List of importable string for the filter template.                                                                 |
+| **REGISTRY**                 |   Core    | `.`                               | List of importable string for the hook stage.                                                                      |
+| **REGISTRY_FILTER**          |   Core    | `ddeutil.workflow.templates`      | List of importable string for the filter template.                                                                 |
 | **CONF_PATH**                |   Core    | `conf`                            | The config path that keep all template `.yaml` files.                                                              |
 | **TIMEZONE**                 |   Core    | `Asia/Bangkok`                    | A Timezone string value that will pass to `ZoneInfo` object.                                                       |
 | **STAGE_DEFAULT_ID**         |   Core    | `true`                            | A flag that enable default stage ID that use for catch an execution output.                                        |

{ddeutil_workflow-0.0.28 → ddeutil_workflow-0.0.29}/src/ddeutil_workflow.egg-info/SOURCES.txt

@@ -42,17 +42,13 @@ tests/test_schedule.py
 tests/test_schedule_control.py
 tests/test_schedule_tasks.py
 tests/test_stage.py
-tests/test_stage_exec_bash.py
-tests/test_stage_exec_hook.py
-tests/test_stage_exec_py.py
-tests/test_stage_exec_trigger.py
+tests/test_stage_handler_exec.py
 tests/test_templates.py
 tests/test_templates_filter.py
 tests/test_utils.py
 tests/test_workflow.py
 tests/test_workflow_exec.py
 tests/test_workflow_exec_hook.py
-tests/test_workflow_exec_needs.py
 tests/test_workflow_job_exec.py
 tests/test_workflow_poke.py
 tests/test_workflow_release.py

{ddeutil_workflow-0.0.28 → ddeutil_workflow-0.0.29}/src/ddeutil_workflow.egg-info/requires.txt

@@ -1,4 +1,4 @@
-ddeutil==0.4.6
+ddeutil>=0.4.6
 ddeutil-io[toml,yaml]>=0.2.3
 pydantic==2.10.6
 python-dotenv==1.0.1

{ddeutil_workflow-0.0.28 → ddeutil_workflow-0.0.29}/tests/test_conf.py

@@ -87,9 +87,11 @@ def test_simple_load_finds(target_path: Path):
                 "test_simple_load_config",
                 {"type": "Config"},
             )
-        ] == list(SimLoad.finds(Config, Config(), included="type"))
+        ] == list(SimLoad.finds(Config, Config(), included=["type"]))
         assert [] == list(
-            SimLoad.finds(Config, Config(), excluded="test_simple_load_config")
+            SimLoad.finds(
+                Config, Config(), excluded=["test_simple_load_config"]
+            )
         )
 
     dummy_file.unlink()

{ddeutil_workflow-0.0.28 → ddeutil_workflow-0.0.29}/tests/test_hook_tag.py

@@ -3,7 +3,7 @@ from pathlib import Path
 from textwrap import dedent
 
 import pytest
-from ddeutil.workflow.hook import make_registry
+from ddeutil.workflow.hook import Registry, make_registry
 
 
 @pytest.fixture(scope="module")
@@ -39,11 +39,12 @@ def hook_function(test_path: Path):
 
 
 def test_make_registry_not_found():
-    rs = make_registry("not_found")
+    rs: dict[str, Registry] = make_registry("not_found")
     assert rs == {}
 
 
 def test_make_registry_raise(hook_function):
 
+    # NOTE: Raise error duplicate tag name, polars-dir, that set in this module.
     with pytest.raises(ValueError):
         make_registry("new_tasks")

{ddeutil_workflow-0.0.28 → ddeutil_workflow-0.0.29}/tests/test_stage.py

@@ -14,6 +14,7 @@ def test_stage():
     )
     assert stage.iden == "Empty Stage"
     assert stage.name == "Empty Stage"
+    assert stage == EmptyStage(name="Empty Stage", echo="hello world")
 
     # NOTE: Copy the stage model with adding the id field.
     new_stage: Stage = stage.model_copy(update={"id": "stage-empty"})

ddeutil_workflow-0.0.29/tests/test_stage_handler_exec.py

@@ -0,0 +1,182 @@
+from datetime import datetime
+from inspect import isfunction
+from unittest import mock
+
+import pytest
+from ddeutil.core import getdot
+from ddeutil.workflow import Workflow
+from ddeutil.workflow.conf import Config
+from ddeutil.workflow.exceptions import StageException
+from ddeutil.workflow.result import Result
+from ddeutil.workflow.stage import Stage
+
+
+def test_stage_exec_bash():
+    workflow: Workflow = Workflow.from_loader(name="wf-run-common")
+    stage: Stage = workflow.job("bash-run").stage("echo")
+    rs: Result = stage.handler_execute({})
+    assert {
+        "return_code": 0,
+        "stdout": "Hello World\nVariable Foo",
+        "stderr": None,
+    } == rs.context
+
+
+def test_stage_exec_bash_env():
+    workflow: Workflow = Workflow.from_loader(name="wf-run-common")
+    stage: Stage = workflow.job("bash-run-env").stage("echo-env")
+    rs: Result = stage.handler_execute({})
+    assert {
+        "return_code": 0,
+        "stdout": "Hello World\nVariable Foo\nENV Bar",
+        "stderr": None,
+    } == rs.context
+
+
+def test_stage_exec_bash_env_raise():
+    workflow: Workflow = Workflow.from_loader(name="wf-run-common")
+    stage: Stage = workflow.job("bash-run-env").stage("raise-error")
+
+    # NOTE: Raise error from bash that force exit 1.
+    with pytest.raises(StageException):
+        stage.handler_execute({})
+
+
+def test_stage_exec_hook():
+    workflow: Workflow = Workflow.from_loader(name="wf-hook-return-type")
+    stage: Stage = workflow.job("second-job").stage("extract-load")
+    rs: Result = stage.handler_execute({})
+
+    assert 0 == rs.status
+    assert {"records": 1} == rs.context
+
+
+def test_stage_exec_hook_raise_return_type():
+    workflow: Workflow = Workflow.from_loader(name="wf-hook-return-type")
+    stage: Stage = workflow.job("first-job").stage("valid-type")
+
+    with pytest.raises(StageException):
+        stage.handler_execute({})
+
+
+def test_stage_exec_hook_raise_args():
+    workflow: Workflow = Workflow.from_loader(name="wf-hook-return-type")
+    stage: Stage = workflow.job("first-job").stage("args-necessary")
+
+    with pytest.raises(StageException):
+        stage.handler_execute({})
+
+
+def test_stage_exec_hook_not_valid():
+    workflow: Workflow = Workflow.from_loader(name="wf-hook-return-type")
+    stage: Stage = workflow.job("first-job").stage("hook-not-valid")
+
+    with pytest.raises(StageException):
+        stage.handler_execute({})
+
+
+def test_stage_exec_hook_not_register():
+    workflow: Workflow = Workflow.from_loader(name="wf-hook-return-type")
+    stage: Stage = workflow.job("first-job").stage("hook-not-register")
+
+    with pytest.raises(StageException):
+        stage.handler_execute({})
+
+
+def test_stage_exec_py_raise():
+    with mock.patch.object(Config, "stage_raise_error", True):
+        workflow: Workflow = Workflow.from_loader(name="wf-run-common")
+        stage: Stage = workflow.job("raise-run").stage(stage_id="raise-error")
+        with pytest.raises(StageException):
+            stage.handler_execute(params={"x": "Foo"})
+
+
+def test_stage_exec_py_not_raise():
+    with mock.patch.object(Config, "stage_raise_error", False):
+        workflow: Workflow = Workflow.from_loader(name="wf-run-common")
+        stage: Stage = workflow.job("raise-run").stage(stage_id="raise-error")
+
+        rs = stage.handler_execute(params={"x": "Foo"})
+
+        assert rs.status == 1
+
+        # NOTE:
+        #   Context that return from error will be:
+        #   {
+        #       'error': ValueError("Testing ... PyStage!!!"),
+        #       'error_message': "ValueError: Testing ... PyStage!!!",
+        #   }
+        assert isinstance(rs.context["error"], ValueError)
+        assert rs.context["error_message"] == (
+            "ValueError: Testing raise error inside PyStage!!!"
+        )
+
+        rs_out = stage.set_outputs(rs.context, {})
+        assert rs_out == {
+            "stages": {
+                "raise-error": {
+                    "outputs": {
+                        "error": getdot(
+                            "stages.raise-error.outputs.error", rs_out
+                        ),
+                        "error_message": (
+                            "ValueError: Testing raise error inside PyStage!!!"
+                        ),
+                    },
+                },
+            },
+        }
+
+
+def test_stage_exec_py_with_vars():
+    workflow: Workflow = Workflow.from_loader(name="wf-run-common")
+    stage: Stage = workflow.job("demo-run").stage(stage_id="run-var")
+    assert stage.id == "run-var"
+
+    params = {
+        "params": {"name": "Author"},
+        "stages": {"hello-world": {"outputs": {"x": "Foo"}}},
+    }
+    rs_out = stage.set_outputs(
+        stage.handler_execute(params=params).context, to=params
+    )
+    assert {
+        "params": {"name": "Author"},
+        "stages": {
+            "hello-world": {"outputs": {"x": "Foo"}},
+            "run-var": {"outputs": {"x": 1}},
+        },
+    } == rs_out
+
+
+def test_stage_exec_py_func():
+    workflow: Workflow = Workflow.from_loader(name="wf-run-python")
+    stage: Stage = workflow.job("second-job").stage(stage_id="create-func")
+    rs: Result = stage.handler_execute(params={})
+    rs_out = stage.set_outputs(rs.context, to={})
+    assert ("var_inside", "echo") == tuple(
+        rs_out["stages"]["create-func"]["outputs"].keys()
+    )
+    assert isfunction(rs_out["stages"]["create-func"]["outputs"]["echo"])
+
+
+def test_stage_exec_trigger():
+    workflow = Workflow.from_loader(name="wf-trigger", externals={})
+    stage: Stage = workflow.job("trigger-job").stage(stage_id="trigger-stage")
+    rs: Result = stage.handler_execute(params={})
+    assert all(k in ("params", "jobs") for k in rs.context.keys())
+    assert {
+        "author-run": "Trigger Runner",
+        "run-date": datetime(2024, 8, 1),
+    } == rs.context["params"]
+
+
+def test_stage_exec_trigger_from_workflow():
+    workflow = Workflow.from_loader(name="wf-trigger", externals={})
+    rs: Result = workflow.execute(params={})
+    assert {
+        "author-run": "Trigger Runner",
+        "run-date": datetime(2024, 8, 1),
+    } == getdot(
+        "jobs.trigger-job.stages.trigger-stage.outputs.params", rs.context
+    )

{ddeutil_workflow-0.0.28 → ddeutil_workflow-0.0.29}/tests/test_workflow_exec.py

@@ -15,11 +15,21 @@ def test_workflow_exec():
     workflow: Workflow = Workflow(
         name="demo-workflow", jobs={"sleep-run": job, "sleep-again-run": job}
     )
-    workflow.execute(params={})
+    rs: Result = workflow.execute(params={})
+    assert rs.status == 0
+    assert rs.context == {
+        "params": {},
+        "jobs": {
+            "sleep-again-run": {
+                "matrix": {},
+                "stages": {"7972360640": {"outputs": {}}},
+            },
+        },
+    }
 
 
 @mock.patch.object(Config, "max_job_parallel", 1)
-def test_workflow_exec_timeout():
+def test_workflow_exec_raise_timeout():
     job: Job = Job(
         stages=[{"name": "Sleep", "run": "import time\ntime.sleep(2)"}],
     )
@@ -27,6 +37,7 @@ def test_workflow_exec_timeout():
         name="demo-workflow", jobs={"sleep-run": job, "sleep-again-run": job}
     )
     rs: Result = workflow.execute(params={}, timeout=1)
+    assert rs.status == 1
     assert rs.context["error_message"] == (
         "WorkflowException: Execution: 'demo-workflow' was timeout."
     )
@@ -268,3 +279,72 @@ def test_workflow_exec_with_matrix():
             },
         },
     } == rs.context
+
+
+def test_workflow_exec_needs():
+    workflow = Workflow.from_loader(name="wf-run-depends", externals={})
+    rs: Result = workflow.execute(params={"name": "bar"})
+    assert {
+        "params": {"name": "bar"},
+        "jobs": {
+            "final-job": {
+                "matrix": {},
+                "stages": {
+                    "8797330324": {
+                        "outputs": {},
+                    },
+                },
+            },
+            "first-job": {
+                "matrix": {},
+                "stages": {
+                    "7824513474": {
+                        "outputs": {},
+                    },
+                },
+            },
+            "second-job": {
+                "matrix": {},
+                "stages": {
+                    "1772094681": {
+                        "outputs": {},
+                    },
+                },
+            },
+        },
+    } == rs.context
+
+
+def test_workflow_exec_needs_parallel():
+    with mock.patch.object(Config, "max_job_parallel", 3):
+        workflow = Workflow.from_loader(name="wf-run-depends", externals={})
+        rs: Result = workflow.execute(params={"name": "bar"})
+        assert {
+            "params": {"name": "bar"},
+            "jobs": {
+                "final-job": {
+                    "matrix": {},
+                    "stages": {
+                        "8797330324": {
+                            "outputs": {},
+                        },
+                    },
+                },
+                "first-job": {
+                    "matrix": {},
+                    "stages": {
+                        "7824513474": {
+                            "outputs": {},
+                        },
+                    },
+                },
+                "second-job": {
+                    "matrix": {},
+                    "stages": {
+                        "1772094681": {
+                            "outputs": {},
+                        },
+                    },
+                },
+            },
+        } == rs.context

{ddeutil_workflow-0.0.28 → ddeutil_workflow-0.0.29}/tests/test_workflow_poke.py

@@ -10,6 +10,7 @@ from ddeutil.workflow.result import Result
 from .utils import dump_yaml_context
 
 
+@pytest.mark.poke
 @mock.patch.object(Config, "enable_write_log", False)
 def test_workflow_poke(test_path):
     with dump_yaml_context(
@@ -68,6 +69,7 @@ def test_workflow_poke(test_path):
         assert results[0].run_id != results[0].parent_run_id
 
 
+@pytest.mark.poke
 @mock.patch.object(Config, "enable_write_log", False)
 def test_workflow_poke_no_queue(test_path):
     with dump_yaml_context(
@@ -93,6 +95,7 @@ def test_workflow_poke_no_queue(test_path):
         assert results == []
 
 
+@pytest.mark.poke
 def test_workflow_poke_raise():
     workflow = Workflow.from_loader(name="wf-scheduling-common")
 
@@ -101,6 +104,7 @@ def test_workflow_poke_raise():
         workflow.poke(periods=-1)
 
 
+@pytest.mark.poke
 @mock.patch.object(Config, "enable_write_log", False)
 def test_workflow_poke_with_start_date_and_period(test_path):
     with dump_yaml_context(
@@ -134,6 +138,7 @@ def test_workflow_poke_with_start_date_and_period(test_path):
         assert results[0].parent_run_id == results[1].parent_run_id
 
 
+@pytest.mark.poke
 @mock.patch.object(Config, "enable_write_log", False)
 def test_workflow_poke_no_on(test_path):
     with dump_yaml_context(

{ddeutil_workflow-0.0.28 → ddeutil_workflow-0.0.29}/tests/test_workflow_schedule.py

@@ -25,7 +25,7 @@ def test_workflow_schedule():
     wf_schedule = WorkflowSchedule(name="demo", on=[{"cronjob": "2 * * * *"}])
     assert len(wf_schedule.on) == 1
 
-    # NOTE: Raise if do not pass any data to WorkflowSchedule
+    # NOTE: Raise if it does not pass any data to WorkflowSchedule
     with pytest.raises(ValidationError):
         WorkflowSchedule.model_validate({})
 
@@ -118,9 +118,7 @@ def test_workflow_schedule_tasks(test_path):
         assert len(tasks) == 1
 
         task = tasks[0]
-
         task.release(queue=queue["tmp-wf-schedule-tasks"])
-
         task.release(queue=queue["tmp-wf-schedule-tasks"])
 
         assert task.runner.date == datetime(2024, 1, 1, 1, 4, tzinfo=tz)

ddeutil_workflow-0.0.28/src/ddeutil/workflow/__about__.py

@@ -1 +0,0 @@
-__version__: str = "0.0.28"

ddeutil_workflow-0.0.28/tests/test_stage_exec_bash.py

@@ -1,34 +0,0 @@
-import pytest
-from ddeutil.workflow import Workflow
-from ddeutil.workflow.exceptions import StageException
-from ddeutil.workflow.result import Result
-from ddeutil.workflow.stage import Stage
-
-
-def test_stage_exec_bash():
-    workflow: Workflow = Workflow.from_loader(name="wf-run-common")
-    stage: Stage = workflow.job("bash-run").stage("echo")
-    rs: Result = stage.handler_execute({})
-    assert {
-        "return_code": 0,
-        "stdout": "Hello World\nVariable Foo",
-        "stderr": None,
-    } == rs.context
-
-
-def test_stage_exec_bash_env():
-    workflow: Workflow = Workflow.from_loader(name="wf-run-common")
-    stage: Stage = workflow.job("bash-run-env").stage("echo-env")
-    rs: Result = stage.handler_execute({})
-    assert {
-        "return_code": 0,
-        "stdout": "Hello World\nVariable Foo\nENV Bar",
-        "stderr": None,
-    } == rs.context
-
-
-def test_stage_exec_bash_env_raise():
-    workflow: Workflow = Workflow.from_loader(name="wf-run-common")
-    stage: Stage = workflow.job("bash-run-env").stage("raise-error")
-    with pytest.raises(StageException):
-        stage.handler_execute({})

ddeutil_workflow-0.0.28/tests/test_stage_exec_hook.py

@@ -1,46 +0,0 @@
-import pytest
-from ddeutil.workflow import Workflow
-from ddeutil.workflow.exceptions import StageException
-from ddeutil.workflow.result import Result
-from ddeutil.workflow.stage import Stage
-
-
-def test_stage_exec_hook():
-    workflow: Workflow = Workflow.from_loader(name="wf-hook-return-type")
-    stage: Stage = workflow.job("second-job").stage("extract-load")
-    rs: Result = stage.handler_execute({})
-
-    assert 0 == rs.status
-    assert {"records": 1} == rs.context
-
-
-def test_stage_exec_hook_raise_return_type():
-    workflow: Workflow = Workflow.from_loader(name="wf-hook-return-type")
-    stage: Stage = workflow.job("first-job").stage("valid-type")
-
-    with pytest.raises(StageException):
-        stage.handler_execute({})
-
-
-def test_stage_exec_hook_raise_args():
-    workflow: Workflow = Workflow.from_loader(name="wf-hook-return-type")
-    stage: Stage = workflow.job("first-job").stage("args-necessary")
-
-    with pytest.raises(StageException):
-        stage.handler_execute({})
-
-
-def test_stage_exec_hook_not_valid():
-    workflow: Workflow = Workflow.from_loader(name="wf-hook-return-type")
-    stage: Stage = workflow.job("first-job").stage("hook-not-valid")
-
-    with pytest.raises(StageException):
-        stage.handler_execute({})
-
-
-def test_stage_exec_hook_not_register():
-    workflow: Workflow = Workflow.from_loader(name="wf-hook-return-type")
-    stage: Stage = workflow.job("first-job").stage("hook-not-register")
-
-    with pytest.raises(StageException):
-        stage.handler_execute({})

ddeutil_workflow-0.0.28/tests/test_stage_exec_py.py

@@ -1,87 +0,0 @@
-from inspect import isfunction
-from unittest import mock
-
-import pytest
-from ddeutil.core import getdot
-from ddeutil.workflow import Workflow
-from ddeutil.workflow.conf import Config
-from ddeutil.workflow.exceptions import StageException
-from ddeutil.workflow.result import Result
-from ddeutil.workflow.stage import Stage
-
-
-def test_stage_exec_py_raise():
-    with mock.patch.object(Config, "stage_raise_error", True):
-        workflow: Workflow = Workflow.from_loader(name="wf-run-common")
-        stage: Stage = workflow.job("raise-run").stage(stage_id="raise-error")
-        with pytest.raises(StageException):
-            stage.handler_execute(params={"x": "Foo"})
-
-
-def test_stage_exec_py_not_raise():
-    with mock.patch.object(Config, "stage_raise_error", False):
-        workflow: Workflow = Workflow.from_loader(name="wf-run-common")
-        stage: Stage = workflow.job("raise-run").stage(stage_id="raise-error")
-
-        rs = stage.handler_execute(params={"x": "Foo"})
-
-        assert rs.status == 1
-
-        # NOTE:
-        #   Context that return from error will be:
-        #   {
-        #       'error': ValueError("Testing ... PyStage!!!"),
-        #       'error_message': "ValueError: Testing ... PyStage!!!",
-        #   }
-        assert isinstance(rs.context["error"], ValueError)
-        assert rs.context["error_message"] == (
-            "ValueError: Testing raise error inside PyStage!!!"
-        )
-
-        rs_out = stage.set_outputs(rs.context, {})
-        assert rs_out == {
-            "stages": {
-                "raise-error": {
-                    "outputs": {
-                        "error": getdot(
-                            "stages.raise-error.outputs.error", rs_out
-                        ),
-                        "error_message": (
-                            "ValueError: Testing raise error inside PyStage!!!"
-                        ),
-                    },
-                },
-            },
-        }
-
-
-def test_stage_exec_py_with_vars():
-    workflow: Workflow = Workflow.from_loader(name="wf-run-common")
-    stage: Stage = workflow.job("demo-run").stage(stage_id="run-var")
-    assert stage.id == "run-var"
-
-    params = {
-        "params": {"name": "Author"},
-        "stages": {"hello-world": {"outputs": {"x": "Foo"}}},
-    }
-    rs_out = stage.set_outputs(
-        stage.handler_execute(params=params).context, to=params
-    )
-    assert {
-        "params": {"name": "Author"},
-        "stages": {
-            "hello-world": {"outputs": {"x": "Foo"}},
-            "run-var": {"outputs": {"x": 1}},
-        },
-    } == rs_out
-
-
-def test_stage_exec_py_func():
-    workflow: Workflow = Workflow.from_loader(name="wf-run-python")
-    stage: Stage = workflow.job("second-job").stage(stage_id="create-func")
-    rs: Result = stage.handler_execute(params={})
-    rs_out = stage.set_outputs(rs.context, to={})
-    assert ("var_inside", "echo") == tuple(
-        rs_out["stages"]["create-func"]["outputs"].keys()
-    )
-    assert isfunction(rs_out["stages"]["create-func"]["outputs"]["echo"])

ddeutil_workflow-0.0.28/tests/test_stage_exec_trigger.py

@@ -1,30 +0,0 @@
-from datetime import datetime
-
-import ddeutil.workflow as wf
-import ddeutil.workflow.stage as st
-from ddeutil.core import getdot
-from ddeutil.workflow.result import Result
-
-
-def test_stage_exec_trigger():
-    workflow = wf.Workflow.from_loader(name="wf-trigger", externals={})
-    stage: st.Stage = workflow.job("trigger-job").stage(
-        stage_id="trigger-stage"
-    )
-    rs: Result = stage.handler_execute(params={})
-    assert all(k in ("params", "jobs") for k in rs.context.keys())
-    assert {
-        "author-run": "Trigger Runner",
-        "run-date": datetime(2024, 8, 1),
-    } == rs.context["params"]
-
-
-def test_stage_exec_trigger_from_workflow():
-    workflow = wf.Workflow.from_loader(name="wf-trigger", externals={})
-    rs: Result = workflow.execute(params={})
-    assert {
-        "author-run": "Trigger Runner",
-        "run-date": datetime(2024, 8, 1),
-    } == getdot(
-        "jobs.trigger-job.stages.trigger-stage.outputs.params", rs.context
-    )

ddeutil_workflow-0.0.28/tests/test_workflow_exec_needs.py

@@ -1,74 +0,0 @@
-from unittest import mock
-
-from ddeutil.workflow import Workflow
-from ddeutil.workflow.conf import Config
-from ddeutil.workflow.result import Result
-
-
-def test_workflow_exec_needs():
-    workflow = Workflow.from_loader(name="wf-run-depends", externals={})
-    rs: Result = workflow.execute(params={"name": "bar"})
-    assert {
-        "params": {"name": "bar"},
-        "jobs": {
-            "final-job": {
-                "matrix": {},
-                "stages": {
-                    "8797330324": {
-                        "outputs": {},
-                    },
-                },
-            },
-            "first-job": {
-                "matrix": {},
-                "stages": {
-                    "7824513474": {
-                        "outputs": {},
-                    },
-                },
-            },
-            "second-job": {
-                "matrix": {},
-                "stages": {
-                    "1772094681": {
-                        "outputs": {},
-                    },
-                },
-            },
-        },
-    } == rs.context
-
-
-def test_workflow_exec_needs_parallel():
-    with mock.patch.object(Config, "max_job_parallel", 3):
-        workflow = Workflow.from_loader(name="wf-run-depends", externals={})
-        rs: Result = workflow.execute(params={"name": "bar"})
-        assert {
-            "params": {"name": "bar"},
-            "jobs": {
-                "final-job": {
-                    "matrix": {},
-                    "stages": {
-                        "8797330324": {
-                            "outputs": {},
-                        },
-                    },
-                },
-                "first-job": {
-                    "matrix": {},
-                    "stages": {
-                        "7824513474": {
-                            "outputs": {},
-                        },
-                    },
-                },
-                "second-job": {
-                    "matrix": {},
-                    "stages": {
-                        "1772094681": {
-                            "outputs": {},
-                        },
-                    },
-                },
-            },
-        } == rs.context