ddeutil-workflow 0.0.15__py3-none-any.whl → 0.0.17__py3-none-any.whl

{ddeutil_workflow-0.0.15.dist-info → ddeutil_workflow-0.0.17.dist-info}/METADATA

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: ddeutil-workflow
-Version: 0.0.15
+Version: 0.0.17
 Summary: Lightweight workflow orchestration with less dependencies
 Author-email: ddeutils <korawich.anu@gmail.com>
 License: MIT
@@ -22,8 +22,8 @@ Classifier: Programming Language :: Python :: 3.13
 Requires-Python: >=3.9.13
 Description-Content-Type: text/markdown
 License-File: LICENSE
-Requires-Dist: ddeutil >=0.4.0
-Requires-Dist: ddeutil-io >=0.1.13
+Requires-Dist: ddeutil >=0.4.3
+Requires-Dist: ddeutil-io[toml,yaml] >=0.2.3
 Requires-Dist: python-dotenv ==1.0.1
 Requires-Dist: typer <1.0.0,==0.12.5
 Requires-Dist: schedule <2.0.0,==1.2.2
@@ -33,6 +33,7 @@ Requires-Dist: fastapi <1.0.0,>=0.115.0 ; extra == 'api'
 # Workflow
 
 [![test](https://github.com/ddeutils/ddeutil-workflow/actions/workflows/tests.yml/badge.svg?branch=main)](https://github.com/ddeutils/ddeutil-workflow/actions/workflows/tests.yml)
+[![codecov](https://codecov.io/gh/ddeutils/ddeutil-workflow/graph/badge.svg?token=3NDPN2I0H9)](https://codecov.io/gh/ddeutils/ddeutil-workflow)
 [![pypi version](https://img.shields.io/pypi/v/ddeutil-workflow)](https://pypi.org/project/ddeutil-workflow/)
 [![python support version](https://img.shields.io/pypi/pyversions/ddeutil-workflow)](https://pypi.org/project/ddeutil-workflow/)
 [![size](https://img.shields.io/github/languages/code-size/ddeutils/ddeutil-workflow)](https://github.com/ddeutils/ddeutil-workflow)
@@ -74,8 +75,9 @@ configuration. It called **Metadata Driven Data Workflow**.
 
 ## :round_pushpin: Installation
 
-This project need `ddeutil-io` extension namespace packages. If you want to install
-this package with application add-ons, you should add `app` in installation;
+This project need `ddeutil` and `ddeutil-io` extension namespace packages.
+If you want to install this package with application add-ons, you should add
+`app` in installation;
 
 | Usecase           | Install Optional                         | Support            |
 |-------------------|------------------------------------------|--------------------|
@@ -179,29 +181,32 @@ The main configuration that use to dynamic changing with your propose of this
 application. If any configuration values do not set yet, it will use default value
 and do not raise any error to you.
 
-| Environment                         | Component | Default                          | Description                                                                |
-|-------------------------------------|-----------|----------------------------------|----------------------------------------------------------------------------|
-| `WORKFLOW_ROOT_PATH`                | Core      | .                                | The root path of the workflow application                                  |
-| `WORKFLOW_CORE_REGISTRY`            | Core      | src.ddeutil.workflow,tests.utils | List of importable string for the hook stage                               |
-| `WORKFLOW_CORE_REGISTRY_FILTER`     | Core      | ddeutil.workflow.utils           | List of importable string for the filter template                          |
-| `WORKFLOW_CORE_PATH_CONF`           | Core      | conf                             | The config path that keep all template `.yaml` files                       |
-| `WORKFLOW_CORE_TIMEZONE`            | Core      | Asia/Bangkok                     | A Timezone string value that will pass to `ZoneInfo` object                |
-| `WORKFLOW_CORE_STAGE_DEFAULT_ID`    | Core      | true                             | A flag that enable default stage ID that use for catch an execution output |
-| `WORKFLOW_CORE_STAGE_RAISE_ERROR`   | Core      | true                             | A flag that all stage raise StageException from stage execution            |
-| `WORKFLOW_CORE_MAX_NUM_POKING`      | Core      | 4                                |                                                                            |
-| `WORKFLOW_CORE_MAX_JOB_PARALLEL`    | Core      | 2                                | The maximum job number that able to run parallel in workflow executor      |
-| `WORKFLOW_LOG_DEBUG_MODE`           | Log       | true                             | A flag that enable logging with debug level mode                           |
-| `WORKFLOW_LOG_ENABLE_WRITE`         | Log       | true                             | A flag that enable logging object saving log to its destination            |
-| `WORKFLOW_APP_PROCESS_WORKER`       | Schedule  | 2                                | The maximum process worker number that run in scheduler app module         |
-| `WORKFLOW_APP_SCHEDULE_PER_PROCESS` | Schedule  | 100                              | A schedule per process that run parallel                                   |
-| `WORKFLOW_APP_STOP_BOUNDARY_DELTA`  | Schedule  | '{"minutes": 5, "seconds": 20}'  | A time delta value that use to stop scheduler app in json string format    |
+| Environment                             | Component | Default                          | Description                                                                                                        | Remark |
+|:----------------------------------------|-----------|----------------------------------|--------------------------------------------------------------------------------------------------------------------|--------|
+| `WORKFLOW_ROOT_PATH`                    | Core      | .                                | The root path of the workflow application.                                                                         |        |
+| `WORKFLOW_CORE_REGISTRY`                | Core      | src.ddeutil.workflow,tests.utils | List of importable string for the hook stage.                                                                      |        |
+| `WORKFLOW_CORE_REGISTRY_FILTER`         | Core      | ddeutil.workflow.utils           | List of importable string for the filter template.                                                                 |        |
+| `WORKFLOW_CORE_PATH_CONF`               | Core      | conf                             | The config path that keep all template `.yaml` files.                                                              |        |
+| `WORKFLOW_CORE_TIMEZONE`                | Core      | Asia/Bangkok                     | A Timezone string value that will pass to `ZoneInfo` object.                                                       |        |
+| `WORKFLOW_CORE_STAGE_DEFAULT_ID`        | Core      | true                             | A flag that enable default stage ID that use for catch an execution output.                                        |        |
+| `WORKFLOW_CORE_STAGE_RAISE_ERROR`       | Core      | false                            | A flag that all stage raise StageException from stage execution.                                                   |        |
+| `WORKFLOW_CORE_JOB_DEFAULT_ID`          | Core      | false                            | A flag that enable default job ID that use for catch an execution output. The ID that use will be sequence number. |        |
+| `WORKFLOW_CORE_JOB_RAISE_ERROR`         | Core      | true                             | A flag that all job raise JobException from job strategy execution.                                                |        |
+| `WORKFLOW_CORE_MAX_NUM_POKING`          | Core      | 4                                | .                                                                                                                  |        |
+| `WORKFLOW_CORE_MAX_JOB_PARALLEL`        | Core      | 2                                | The maximum job number that able to run parallel in workflow executor.                                             |        |
+| `WORKFLOW_CORE_WORKFLOW_ID_SIMPLE_MODE` | Core      | true                             | .                                                                                                                  |        |
+| `WORKFLOW_LOG_DEBUG_MODE`               | Log       | true                             | A flag that enable logging with debug level mode.                                                                  |        |
+| `WORKFLOW_LOG_ENABLE_WRITE`             | Log       | true                             | A flag that enable logging object saving log to its destination.                                                   |        |
+| `WORKFLOW_APP_MAX_PROCESS`              | Schedule  | 2                                | The maximum process worker number that run in scheduler app module.                                                |        |
+| `WORKFLOW_APP_MAX_SCHEDULE_PER_PROCESS` | Schedule  | 100                              | A schedule per process that run parallel.                                                                          |        |
+| `WORKFLOW_APP_STOP_BOUNDARY_DELTA`      | Schedule  | '{"minutes": 5, "seconds": 20}'  | A time delta value that use to stop scheduler app in json string format.                                           |        |
 
 **API Application**:
 
-| Environment                          | Component | Default | Description                                                                       |
-|--------------------------------------|-----------|---------|-----------------------------------------------------------------------------------|
-| `WORKFLOW_API_ENABLE_ROUTE_WORKFLOW` | API       | true    | A flag that enable workflow route to manage execute manually and workflow logging |
-| `WORKFLOW_API_ENABLE_ROUTE_SCHEDULE` | API       | true    | A flag that enable run scheduler                                                  |
+| Environment                           | Component | Default | Description                                                                        | Remark |
+|:--------------------------------------|-----------|---------|------------------------------------------------------------------------------------|--------|
+| `WORKFLOW_API_ENABLE_ROUTE_WORKFLOW`  | API       | true    | A flag that enable workflow route to manage execute manually and workflow logging. |        |
+| `WORKFLOW_API_ENABLE_ROUTE_SCHEDULE`  | API       | true    | A flag that enable run scheduler.                                                  |        |
 
 ## :rocket: Deployment
 
@@ -224,3 +229,17 @@ like crontab job but via Python API.
 > [!NOTE]
 > If this package already deploy, it able to use
 > `uvicorn ddeutil.workflow.api:app --host 127.0.0.1 --port 80 --workers 4`
+
+### Docker Container
+
+Create Docker image;
+
+```shell
+$ docker build -t ddeutil-workflow:latest -f .container/Dockerfile .
+```
+
+Run the above Docker image;
+
+```shell
+$ docker run -i ddeutil-workflow:latest
+```

ddeutil_workflow-0.0.17.dist-info/RECORD

@@ -0,0 +1,21 @@
+ddeutil/workflow/__about__.py,sha256=z3f1GAF3VbZK1m4FWAXXMsWplP_jSe-X-wVlshvlDWU,28
+ddeutil/workflow/__cron.py,sha256=ZiuV4ASkXvAyFJYxEb9PKiAFNYnUt4AJozu_kH3pI4U,25777
+ddeutil/workflow/__init__.py,sha256=RNKME4FPMAjqtrBR-IBwQVEKeoY5yBAiHYcZw0k9cI4,729
+ddeutil/workflow/__types.py,sha256=yizLXzjQpBt_WPaof2pIyncitJvYeksw4Q1zYJeuCLA,3707
+ddeutil/workflow/api.py,sha256=vUT2RVS9sF3hvY-IrzAEnahxwq4ZFYP0G3xfctHbNsw,4701
+ddeutil/workflow/cli.py,sha256=baHhvtI8snbHYHeThoX401Cd6SMB2boyyCbCtTrIl3E,3278
+ddeutil/workflow/conf.py,sha256=SV4GMtjUc-Bor9BPi0yOtTIsiZ0FImsoRbuJysUIE9w,15395
+ddeutil/workflow/exceptions.py,sha256=Uf1-Tn8rAzj0aiVHSqo4fBqO80W0za7UFZgKv24E-tg,706
+ddeutil/workflow/job.py,sha256=dW9NXR_bttDGLwelVi7qXXlLd96KX-TKG8xnHejA6u0,24041
+ddeutil/workflow/on.py,sha256=rneZB5HyFWTBWriGef999bovA3glQIK6LTgC996q9Gc,7334
+ddeutil/workflow/repeat.py,sha256=9uKku5uMcQgzY5fWyaJMwJ0wPFX0oTwmu7vXKdgB_ec,4923
+ddeutil/workflow/route.py,sha256=JALwOH6xKu5rnII7DgA1Lbp_E5ehCoBbOW_eKqB_Olk,6753
+ddeutil/workflow/scheduler.py,sha256=Oa6bZpphjlGp0mXdBuLMk1m6G-dezaBNQxQX-SB3WJ0,47032
+ddeutil/workflow/stage.py,sha256=fMv_oFkoqpfoewzPUMdl3-BQcrJ8SE53cF7es8yGxfs,25525
+ddeutil/workflow/utils.py,sha256=lpnqGGd_Rw7eZo2wDbZ-NZNItBooFooPjwM4_40Csh8,25152
+ddeutil_workflow-0.0.17.dist-info/LICENSE,sha256=nGFZ1QEhhhWeMHf9n99_fdt4vQaXS29xWKxt-OcLywk,1085
+ddeutil_workflow-0.0.17.dist-info/METADATA,sha256=btmCr-yjy4gzhnZppfXjANfPH-3tKUJFGon2aOMUK30,13574
+ddeutil_workflow-0.0.17.dist-info/WHEEL,sha256=OVMc5UfuAQiSplgO0_WdW7vXVGAt9Hdd6qtN4HotdyA,91
+ddeutil_workflow-0.0.17.dist-info/entry_points.txt,sha256=0BVOgO3LdUdXVZ-CiHHDKxzEk2c8J30jEwHeKn2YCWI,62
+ddeutil_workflow-0.0.17.dist-info/top_level.txt,sha256=m9M6XeSWDwt_yMsmH6gcOjHZVK5O0-vgtNBuncHjzW4,8
+ddeutil_workflow-0.0.17.dist-info/RECORD,,

{ddeutil_workflow-0.0.15.dist-info → ddeutil_workflow-0.0.17.dist-info}/WHEEL

@@ -1,5 +1,5 @@
 Wheel-Version: 1.0
-Generator: setuptools (75.1.0)
+Generator: setuptools (75.2.0)
 Root-Is-Purelib: true
 Tag: py3-none-any
 

ddeutil/workflow/log.py

@@ -1,198 +0,0 @@
-# ------------------------------------------------------------------------------
-# Copyright (c) 2022 Korawich Anuttra. All rights reserved.
-# Licensed under the MIT License. See LICENSE in the project root for
-# license information.
-# ------------------------------------------------------------------------------
-from __future__ import annotations
-
-import json
-import logging
-import os
-from abc import ABC, abstractmethod
-from datetime import datetime
-from functools import lru_cache
-from pathlib import Path
-from typing import ClassVar, Optional, Union
-
-from ddeutil.core import str2bool
-from pydantic import BaseModel, Field
-from pydantic.functional_validators import model_validator
-from typing_extensions import Self
-
-from .__types import DictData
-from .utils import load_config
-
-
-@lru_cache
-def get_logger(name: str):
-    """Return logger object with an input module name.
-
-    :param name: A module name that want to log.
-    """
-    logger = logging.getLogger(name)
-    formatter = logging.Formatter(
-        fmt=(
-            "%(asctime)s.%(msecs)03d (%(name)-10s, %(process)-5d, "
-            "%(thread)-5d) [%(levelname)-7s] %(message)-120s "
-            "(%(filename)s:%(lineno)s)"
-        ),
-        datefmt="%Y-%m-%d %H:%M:%S",
-    )
-    stream = logging.StreamHandler()
-    stream.setFormatter(formatter)
-    logger.addHandler(stream)
-
-    debug: bool = str2bool(os.getenv("WORKFLOW_LOG_DEBUG_MODE", "true"))
-    logger.setLevel(logging.DEBUG if debug else logging.INFO)
-    return logger
-
-
-class BaseLog(BaseModel, ABC):
-    """Base Log Pydantic Model with abstraction class property that implement
-    only model fields. This model should to use with inherit to logging
-    sub-class like file, sqlite, etc.
-    """
-
-    name: str = Field(description="A workflow name.")
-    on: str = Field(description="A cronjob string of this piepline schedule.")
-    release: datetime = Field(description="A release datetime.")
-    context: DictData = Field(
-        default_factory=dict,
-        description=(
-            "A context data that receive from a workflow execution result.",
-        ),
-    )
-    parent_run_id: Optional[str] = Field(default=None)
-    run_id: str
-    update: datetime = Field(default_factory=datetime.now)
-
-    @model_validator(mode="after")
-    def __model_action(self) -> Self:
-        """Do before the Log action with WORKFLOW_LOG_ENABLE_WRITE env variable.
-
-        :rtype: Self
-        """
-        if str2bool(os.getenv("WORKFLOW_LOG_ENABLE_WRITE", "false")):
-            self.do_before()
-        return self
-
-    def do_before(self) -> None:
-        """To something before end up of initial log model."""
-
-    @abstractmethod
-    def save(self, excluded: list[str] | None) -> None:
-        """Save this model logging to target logging store."""
-        raise NotImplementedError("Log should implement ``save`` method.")
-
-
-class FileLog(BaseLog):
-    """File Log Pydantic Model that use to saving log data from result of
-    workflow execution. It inherit from BaseLog model that implement the
-    ``self.save`` method for file.
-    """
-
-    filename: ClassVar[str] = (
-        "./logs/workflow={name}/release={release:%Y%m%d%H%M%S}"
-    )
-
-    def do_before(self) -> None:
-        """Create directory of release before saving log file."""
-        self.pointer().mkdir(parents=True, exist_ok=True)
-
-    @classmethod
-    def find_logs(cls, name: str):
-        pointer: Path = (
-            load_config().engine.paths.root / f"./logs/workflow={name}"
-        )
-        for file in pointer.glob("./release=*/*.log"):
-            with file.open(mode="r", encoding="utf-8") as f:
-                yield json.load(f)
-
-    @classmethod
-    def find_log(cls, name: str, release: datetime | None = None):
-        if release is not None:
-            pointer: Path = (
-                load_config().engine.paths.root
-                / f"./logs/workflow={name}/release={release:%Y%m%d%H%M%S}"
-            )
-            if not pointer.exists():
-                raise FileNotFoundError(
-                    f"Pointer: ./logs/workflow={name}/"
-                    f"release={release:%Y%m%d%H%M%S} does not found."
-                )
-            return cls.model_validate(
-                obj=json.loads(pointer.read_text(encoding="utf-8"))
-            )
-        raise NotImplementedError("Find latest log does not implement yet.")
-
-    @classmethod
-    def is_pointed(
-        cls,
-        name: str,
-        release: datetime,
-        *,
-        queue: list[datetime] | None = None,
-    ) -> bool:
-        """Check this log already point in the destination.
-
-        :param name: A workflow name.
-        :param release: A release datetime.
-        :param queue: A list of queue of datetime that already run in the
-            future.
-        """
-        # NOTE: Check environ variable was set for real writing.
-        if not str2bool(os.getenv("WORKFLOW_LOG_ENABLE_WRITE", "false")):
-            return False
-
-        # NOTE: create pointer path that use the same logic of pointer method.
-        pointer: Path = load_config().engine.paths.root / cls.filename.format(
-            name=name, release=release
-        )
-
-        if not queue:
-            return pointer.exists()
-        return pointer.exists() or (release in queue)
-
-    def pointer(self) -> Path:
-        """Return release directory path that was generated from model data.
-
-        :rtype: Path
-        """
-        return load_config().engine.paths.root / self.filename.format(
-            name=self.name, release=self.release
-        )
-
-    def save(self, excluded: list[str] | None) -> Self:
-        """Save logging data that receive a context data from a workflow
-        execution result.
-
-        :param excluded: An excluded list of key name that want to pass in the
-            model_dump method.
-        :rtype: Self
-        """
-        # NOTE: Check environ variable was set for real writing.
-        if not str2bool(os.getenv("WORKFLOW_LOG_ENABLE_WRITE", "false")):
-            return self
-
-        log_file: Path = self.pointer() / f"{self.run_id}.log"
-        log_file.write_text(
-            json.dumps(
-                self.model_dump(exclude=excluded),
-                default=str,
-                indent=2,
-            ),
-            encoding="utf-8",
-        )
-        return self
-
-
-class SQLiteLog(BaseLog):
-
-    def save(self, excluded: list[str] | None) -> None:
-        raise NotImplementedError("SQLiteLog does not implement yet.")
-
-
-Log = Union[
-    FileLog,
-    SQLiteLog,
-]

ddeutil_workflow-0.0.15.dist-info/RECORD

@@ -1,22 +0,0 @@
-ddeutil/workflow/__about__.py,sha256=w_vBOopUg1crMbDyfdE0LgsxsncnhGYp0D39LSSnSVI,28
-ddeutil/workflow/__init__.py,sha256=-DIy8SGFsD7_wqp-V-K8v8jTxacmqrcyj_SFx1WS6qg,687
-ddeutil/workflow/__types.py,sha256=WWugALcayRiP0IQO-eBWK767_XxK7KGlY7SuVgyaJnk,3196
-ddeutil/workflow/api.py,sha256=cwju_qhY6m0kLtaoa77QLglC9tl7RjjZ4UnJYV3SlQQ,4810
-ddeutil/workflow/cli.py,sha256=Ikcq526WeIl-737-v55T0PwAZ2pNiZFxlN0Y-DjhDbQ,3374
-ddeutil/workflow/conf.py,sha256=D0g7rHXilpGwOD36QwVd9I5kEwqsAUA0Z3tAINS2Pws,1287
-ddeutil/workflow/cron.py,sha256=naWefHc3EnVo41Yf1zQeXOzF27YlTlnfj0XnQ6_HO-U,25514
-ddeutil/workflow/exceptions.py,sha256=Uf1-Tn8rAzj0aiVHSqo4fBqO80W0za7UFZgKv24E-tg,706
-ddeutil/workflow/job.py,sha256=9H_2C0ikD5y6jLVdIBj8de4CdSpS632XOfqYVhM4bHI,21582
-ddeutil/workflow/log.py,sha256=Ev-Szi0KC_MmbFY4g4BWv6tUSmcLKWKZ03ZInmYPmgU,6490
-ddeutil/workflow/on.py,sha256=vsZG19mNoztDSB_ObD_4ZWPKgHYpBDJMWw97ZiTavNE,7237
-ddeutil/workflow/repeat.py,sha256=e3dekPTlMlxCCizfBYsZ8dD8Juy4rtfqDZJU3Iky2oA,5011
-ddeutil/workflow/route.py,sha256=ABEk-WlVo9XGFc7zCPbckX33URCNH7woQFU1keX_8PQ,6970
-ddeutil/workflow/scheduler.py,sha256=12Dd5CVphOVKjUwoiB8dCHt4WpYRPG3dSOt-pR6NNxc,46167
-ddeutil/workflow/stage.py,sha256=Avz1Mbb8WAP6kFn0bnN0p14-EnQ_AzdKr435JRxjkao,23844
-ddeutil/workflow/utils.py,sha256=XUD5hoygAyxi4xo1spTacoDNGKN2TlRob_o8qfCj4Pc,30993
-ddeutil_workflow-0.0.15.dist-info/LICENSE,sha256=nGFZ1QEhhhWeMHf9n99_fdt4vQaXS29xWKxt-OcLywk,1085
-ddeutil_workflow-0.0.15.dist-info/METADATA,sha256=6JvY9y-cT3WnirRva45NS582Iz7ZuXJZpsiCtN57OoA,11653
-ddeutil_workflow-0.0.15.dist-info/WHEEL,sha256=GV9aMThwP_4oNCtvEC2ec3qUYutgWeAzklro_0m4WJQ,91
-ddeutil_workflow-0.0.15.dist-info/entry_points.txt,sha256=0BVOgO3LdUdXVZ-CiHHDKxzEk2c8J30jEwHeKn2YCWI,62
-ddeutil_workflow-0.0.15.dist-info/top_level.txt,sha256=m9M6XeSWDwt_yMsmH6gcOjHZVK5O0-vgtNBuncHjzW4,8
-ddeutil_workflow-0.0.15.dist-info/RECORD,,