ddeutil-workflow 0.0.9__tar.gz → 0.0.10__tar.gz

{ddeutil_workflow-0.0.9/src/ddeutil_workflow.egg-info → ddeutil_workflow-0.0.10}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: ddeutil-workflow
-Version: 0.0.9
+Version: 0.0.10
 Summary: Lightweight workflow orchestration with less dependencies
 Author-email: ddeutils <korawich.anu@gmail.com>
 License: MIT
@@ -21,15 +21,13 @@ Classifier: Programming Language :: Python :: 3.12
 Requires-Python: >=3.9.13
 Description-Content-Type: text/markdown
 License-File: LICENSE
-Requires-Dist: fmtutil
 Requires-Dist: ddeutil-io
 Requires-Dist: python-dotenv==1.0.1
-Requires-Dist: typer==0.12.4
+Requires-Dist: typer<1.0.0,==0.12.5
 Provides-Extra: schedule
 Requires-Dist: schedule<2.0.0,==1.2.2; extra == "schedule"
 Provides-Extra: api
-Requires-Dist: fastapi[standard]==0.112.1; extra == "api"
-Requires-Dist: croniter==3.0.3; extra == "api"
+Requires-Dist: fastapi[standard]<1.0.0,==0.112.2; extra == "api"
 
 # Workflow
 
@@ -39,17 +37,6 @@ Requires-Dist: croniter==3.0.3; extra == "api"
 [![gh license](https://img.shields.io/github/license/ddeutils/ddeutil-workflow)](https://github.com/ddeutils/ddeutil-workflow/blob/main/LICENSE)
 [![code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
 
-**Table of Contents**:
-
-- [Installation](#installation)
-- [Getting Started](#getting-started)
-  - [On](#on)
-  - [Pipeline](#pipeline)
-- [Usage](#usage)
-- [Configuration](#configuration)
-- [Future](#future)
-- [Deployment](#deployment)
-
 The **Lightweight workflow orchestration** with less dependencies the was created
 for easy to make a simple metadata driven for data pipeline orchestration.
 It can to use for data operator by a `.yaml` template.
@@ -103,82 +90,6 @@ this package with application add-ons, you should add `app` in installation;
 > | ddeutil-workflow:python3.11 | `3.11`         | :x:     |
 > | ddeutil-workflow:python3.12 | `3.12`         | :x:     |
 
-## Getting Started
-
-The main feature of this project is the `Pipeline` object that can call any
-registries function. The pipeline can handle everything that you want to do, it
-will passing parameters and catching the output for re-use it to next step.
-
-### On
-
-The **On** is schedule object that receive crontab value and able to generate
-datetime value with next or previous with any start point of an input datetime.
-
-```yaml
-# This file should keep under this path: `./root-path/conf-path/*`
-on_every_5_min:
-  type: on.On
-  cron: "*/5 * * * *"
-```
-
-```python
-from ddeutil.workflow.on import On
-
-# NOTE: Start load the on data from `.yaml` template file with this key.
-schedule = On.from_loader(name='on_every_5_min', externals={})
-
-assert '*/5 * * * *' == str(schedule.cronjob)
-
-cron_iter = schedule.generate('2022-01-01 00:00:00')
-
-assert "2022-01-01 00:05:00" f"{cron_iter.next:%Y-%m-%d %H:%M:%S}"
-assert "2022-01-01 00:10:00" f"{cron_iter.next:%Y-%m-%d %H:%M:%S}"
-assert "2022-01-01 00:15:00" f"{cron_iter.next:%Y-%m-%d %H:%M:%S}"
-```
-
-### Pipeline
-
-The **Pipeline** object that is the core feature of this project.
-
-```yaml
-# This file should keep under this path: `./root-path/conf-path/*`
-pipeline-name:
-  type: ddeutil.workflow.pipeline.Pipeline
-  on: 'on_every_5_min'
-  params:
-    author-run:
-      type: str
-    run-date:
-      type: datetime
-  jobs:
-    first-job:
-      stages:
-        - name: "Empty stage do logging to console only!!"
-```
-
-```python
-from ddeutil.workflow.pipeline import Pipeline
-
-pipe = Pipeline.from_loader(name='pipeline-name', externals={})
-pipe.execute(params={'author-run': 'Local Workflow', 'run-date': '2024-01-01'})
-```
-
-> [!NOTE]
-> The above parameter can use short declarative statement. You can pass a parameter
-> type to the key of a parameter name but it does not handler default value if you
-> run this pipeline workflow with schedule.
->
-> ```yaml
-> ...
-> params:
->   author-run: str
->   run-date: datetime
-> ...
-> ```
->
-> And for the type, you can remove `ddeutil.workflow` prefix because we can find
-> it by looping search from `WORKFLOW_CORE_REGISTRY` value.
-
 ## Usage
 
 This is examples that use workflow file for running common Data Engineering
@@ -209,7 +120,9 @@ run_py_local:
             url: https://open-data/
             auth: ${API_ACCESS_REFRESH_TOKEN}
             aws_s3_path: my-data/open-data/
-            # This Authentication code should implement with your custom hook function
+
+            # This Authentication code should implement with your custom hook function.
+            # The template allow you to use environment variable.
             aws_access_client_id: ${AWS_ACCESS_CLIENT_ID}
             aws_access_client_secret: ${AWS_ACCESS_CLIENT_SECRET}
 ```
@@ -227,28 +140,24 @@ run_py_local:
 | `WORKFLOW_CORE_STAGE_RAISE_ERROR`   | Core      | true                         | A flag that all stage raise StageException from stage execution            |
 | `WORKFLOW_CORE_MAX_PIPELINE_POKING` | Core      | 4                            |                                                                            |
 | `WORKFLOW_CORE_MAX_JOB_PARALLEL`    | Core      | 2                            | The maximum job number that able to run parallel in pipeline executor      |
+| `WORKFLOW_LOG_DEBUG_MODE`           | Log       | true                         | A flag that enable logging with debug level mode                           |
 | `WORKFLOW_LOG_ENABLE_WRITE`         | Log       | true                         | A flag that enable logging object saving log to its destination            |
 
 
 **Application**:
 
-| Environment                         | Default | Description |
-|-------------------------------------|---------|-------------|
-| `WORKFLOW_APP_PROCESS_WORKER`       | 2       |             |
-| `WORKFLOW_APP_PIPELINE_PER_PROCESS` | 100     |             |
+| Environment                         | Default                          | Description                                                             |
+|-------------------------------------|----------------------------------|-------------------------------------------------------------------------|
+| `WORKFLOW_APP_PROCESS_WORKER`       | 2                                | The maximum process worker number that run in scheduler app module      |
+| `WORKFLOW_APP_SCHEDULE_PER_PROCESS` | 100                              | A schedule per process that run parallel                                |
+| `WORKFLOW_APP_STOP_BOUNDARY_DELTA`  | '{"minutes": 5, "seconds": 20}'  | A time delta value that use to stop scheduler app in json string format |
 
 **API server**:
 
-| Environment           | Default                                                | Description                                                        |
-|-----------------------|--------------------------------------------------------|--------------------------------------------------------------------|
-| `WORKFLOW_API_DB_URL` | postgresql+asyncpg://user:pass@localhost:5432/schedule | A Database URL that will pass to SQLAlchemy create_engine function |
-
-## Future
-
-The current milestone that will develop and necessary features that should to
-implement on this project.
-
-- ...
+| Environment                          | Default | Description                                                                       |
+|--------------------------------------|---------|-----------------------------------------------------------------------------------|
+| `WORKFLOW_API_ENABLE_ROUTE_WORKFLOW` | true    | A flag that enable workflow route to manage execute manually and workflow logging |
+| `WORKFLOW_API_ENABLE_ROUTE_SCHEDULE` | true    | A flag that enable run scheduler                                                  |
 
 ## Deployment
 
@@ -270,4 +179,4 @@ like crontab job but via Python API.
 
 > [!NOTE]
 > If this package already deploy, it able to use
-> `uvicorn ddeutil.workflow.api:app --host 0.0.0.0 --port 80`
+> `uvicorn ddeutil.workflow.api:app --host 0.0.0.0 --port 80 --workers 4`

{ddeutil_workflow-0.0.9 → ddeutil_workflow-0.0.10}/README.md

@@ -6,17 +6,6 @@
 [![gh license](https://img.shields.io/github/license/ddeutils/ddeutil-workflow)](https://github.com/ddeutils/ddeutil-workflow/blob/main/LICENSE)
 [![code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
 
-**Table of Contents**:
-
-- [Installation](#installation)
-- [Getting Started](#getting-started)
-  - [On](#on)
-  - [Pipeline](#pipeline)
-- [Usage](#usage)
-- [Configuration](#configuration)
-- [Future](#future)
-- [Deployment](#deployment)
-
 The **Lightweight workflow orchestration** with less dependencies the was created
 for easy to make a simple metadata driven for data pipeline orchestration.
 It can to use for data operator by a `.yaml` template.
@@ -70,82 +59,6 @@ this package with application add-ons, you should add `app` in installation;
 > | ddeutil-workflow:python3.11 | `3.11`         | :x:     |
 > | ddeutil-workflow:python3.12 | `3.12`         | :x:     |
 
-## Getting Started
-
-The main feature of this project is the `Pipeline` object that can call any
-registries function. The pipeline can handle everything that you want to do, it
-will passing parameters and catching the output for re-use it to next step.
-
-### On
-
-The **On** is schedule object that receive crontab value and able to generate
-datetime value with next or previous with any start point of an input datetime.
-
-```yaml
-# This file should keep under this path: `./root-path/conf-path/*`
-on_every_5_min:
-  type: on.On
-  cron: "*/5 * * * *"
-```
-
-```python
-from ddeutil.workflow.on import On
-
-# NOTE: Start load the on data from `.yaml` template file with this key.
-schedule = On.from_loader(name='on_every_5_min', externals={})
-
-assert '*/5 * * * *' == str(schedule.cronjob)
-
-cron_iter = schedule.generate('2022-01-01 00:00:00')
-
-assert "2022-01-01 00:05:00" f"{cron_iter.next:%Y-%m-%d %H:%M:%S}"
-assert "2022-01-01 00:10:00" f"{cron_iter.next:%Y-%m-%d %H:%M:%S}"
-assert "2022-01-01 00:15:00" f"{cron_iter.next:%Y-%m-%d %H:%M:%S}"
-```
-
-### Pipeline
-
-The **Pipeline** object that is the core feature of this project.
-
-```yaml
-# This file should keep under this path: `./root-path/conf-path/*`
-pipeline-name:
-  type: ddeutil.workflow.pipeline.Pipeline
-  on: 'on_every_5_min'
-  params:
-    author-run:
-      type: str
-    run-date:
-      type: datetime
-  jobs:
-    first-job:
-      stages:
-        - name: "Empty stage do logging to console only!!"
-```
-
-```python
-from ddeutil.workflow.pipeline import Pipeline
-
-pipe = Pipeline.from_loader(name='pipeline-name', externals={})
-pipe.execute(params={'author-run': 'Local Workflow', 'run-date': '2024-01-01'})
-```
-
-> [!NOTE]
-> The above parameter can use short declarative statement. You can pass a parameter
-> type to the key of a parameter name but it does not handler default value if you
-> run this pipeline workflow with schedule.
->
-> ```yaml
-> ...
-> params:
->   author-run: str
->   run-date: datetime
-> ...
-> ```
->
-> And for the type, you can remove `ddeutil.workflow` prefix because we can find
-> it by looping search from `WORKFLOW_CORE_REGISTRY` value.
-
 ## Usage
 
 This is examples that use workflow file for running common Data Engineering
@@ -176,7 +89,9 @@ run_py_local:
             url: https://open-data/
             auth: ${API_ACCESS_REFRESH_TOKEN}
             aws_s3_path: my-data/open-data/
-            # This Authentication code should implement with your custom hook function
+
+            # This Authentication code should implement with your custom hook function.
+            # The template allow you to use environment variable.
             aws_access_client_id: ${AWS_ACCESS_CLIENT_ID}
             aws_access_client_secret: ${AWS_ACCESS_CLIENT_SECRET}
 ```
@@ -194,28 +109,24 @@ run_py_local:
 | `WORKFLOW_CORE_STAGE_RAISE_ERROR`   | Core      | true                         | A flag that all stage raise StageException from stage execution            |
 | `WORKFLOW_CORE_MAX_PIPELINE_POKING` | Core      | 4                            |                                                                            |
 | `WORKFLOW_CORE_MAX_JOB_PARALLEL`    | Core      | 2                            | The maximum job number that able to run parallel in pipeline executor      |
+| `WORKFLOW_LOG_DEBUG_MODE`           | Log       | true                         | A flag that enable logging with debug level mode                           |
 | `WORKFLOW_LOG_ENABLE_WRITE`         | Log       | true                         | A flag that enable logging object saving log to its destination            |
 
 
 **Application**:
 
-| Environment                         | Default | Description |
-|-------------------------------------|---------|-------------|
-| `WORKFLOW_APP_PROCESS_WORKER`       | 2       |             |
-| `WORKFLOW_APP_PIPELINE_PER_PROCESS` | 100     |             |
+| Environment                         | Default                          | Description                                                             |
+|-------------------------------------|----------------------------------|-------------------------------------------------------------------------|
+| `WORKFLOW_APP_PROCESS_WORKER`       | 2                                | The maximum process worker number that run in scheduler app module      |
+| `WORKFLOW_APP_SCHEDULE_PER_PROCESS` | 100                              | A schedule per process that run parallel                                |
+| `WORKFLOW_APP_STOP_BOUNDARY_DELTA`  | '{"minutes": 5, "seconds": 20}'  | A time delta value that use to stop scheduler app in json string format |
 
 **API server**:
 
-| Environment           | Default                                                | Description                                                        |
-|-----------------------|--------------------------------------------------------|--------------------------------------------------------------------|
-| `WORKFLOW_API_DB_URL` | postgresql+asyncpg://user:pass@localhost:5432/schedule | A Database URL that will pass to SQLAlchemy create_engine function |
-
-## Future
-
-The current milestone that will develop and necessary features that should to
-implement on this project.
-
-- ...
+| Environment                          | Default | Description                                                                       |
+|--------------------------------------|---------|-----------------------------------------------------------------------------------|
+| `WORKFLOW_API_ENABLE_ROUTE_WORKFLOW` | true    | A flag that enable workflow route to manage execute manually and workflow logging |
+| `WORKFLOW_API_ENABLE_ROUTE_SCHEDULE` | true    | A flag that enable run scheduler                                                  |
 
 ## Deployment
 
@@ -237,4 +148,4 @@ like crontab job but via Python API.
 
 > [!NOTE]
 > If this package already deploy, it able to use
-> `uvicorn ddeutil.workflow.api:app --host 0.0.0.0 --port 80`
+> `uvicorn ddeutil.workflow.api:app --host 0.0.0.0 --port 80 --workers 4`

{ddeutil_workflow-0.0.9 → ddeutil_workflow-0.0.10}/pyproject.toml

@@ -25,10 +25,9 @@ classifiers = [
 ]
 requires-python = ">=3.9.13"
 dependencies = [
-    "fmtutil",
     "ddeutil-io",
     "python-dotenv==1.0.1",
-    "typer==0.12.4",
+    "typer==0.12.5,<1.0.0",
 ]
 dynamic = ["version"]
 
@@ -37,9 +36,7 @@ schedule = [
     "schedule==1.2.2,<2.0.0",
 ]
 api = [
-    "fastapi[standard]==0.112.1",
-    # TODO: This package can migrate to use /cron/
-    "croniter==3.0.3",
+    "fastapi[standard]==0.112.2,<1.0.0",
 ]
 
 [project.urls]
@@ -47,7 +44,7 @@ Homepage = "https://github.com/ddeutils/ddeutil-workflow/"
 "Source Code" = "https://github.com/ddeutils/ddeutil-workflow/"
 
 [project.scripts]
-workflow = "ddeutil.workflow.cli:app"
+ddeutil-workflow = "ddeutil.workflow.cli:cli"
 
 [tool.setuptools.dynamic]
 version = {attr = "ddeutil.workflow.__about__.__version__"}
@@ -86,7 +83,7 @@ addopts = [
 filterwarnings = ["error"]
 log_cli = true
 log_cli_level = "DEBUG"
-log_cli_format = "%(asctime)s [%(levelname)-7s] %(message)-100s (%(filename)s:%(lineno)s)"
+log_cli_format = "%(asctime)s [%(levelname)-7s] %(message)-120s (%(filename)s:%(lineno)s)"
 log_cli_date_format = "%Y%m%d %H:%M:%S"
 
 [tool.black]

ddeutil_workflow-0.0.10/src/ddeutil/workflow/__about__.py

@@ -0,0 +1 @@
+__version__: str = "0.0.10"

{ddeutil_workflow-0.0.9 → ddeutil_workflow-0.0.10}/src/ddeutil/workflow/api.py

@@ -6,7 +6,6 @@
 from __future__ import annotations
 
 import asyncio
-import logging
 import os
 import uuid
 from queue import Empty, Queue
@@ -18,33 +17,33 @@ from fastapi.middleware.gzip import GZipMiddleware
 from fastapi.responses import UJSONResponse
 from pydantic import BaseModel
 
+from .__about__ import __version__
+from .log import get_logger
 from .repeat import repeat_every
 
 load_dotenv()
-logger = logging.getLogger(__name__)
-logging.basicConfig(
-    level=logging.DEBUG,
-    format=(
-        "%(asctime)s.%(msecs)03d (%(name)-10s, %(process)-5d, %(thread)-5d) "
-        "[%(levelname)-7s] %(message)-120s (%(filename)s:%(lineno)s)"
-    ),
-    handlers=[logging.StreamHandler()],
-    datefmt="%Y-%m-%d %H:%M:%S",
-)
+logger = get_logger("ddeutil.workflow")
 
 
-app = FastAPI()
+app = FastAPI(
+    titile="Workflow API",
+    description=(
+        "This is workflow FastAPI web application that use to manage manual "
+        "execute or schedule workflow via RestAPI."
+    ),
+    version=__version__,
+)
 app.add_middleware(GZipMiddleware, minimum_size=1000)
 app.queue = Queue()
 app.output_dict = {}
-app.queue_limit = 2
+app.queue_limit = 5
 
 
 @app.on_event("startup")
-@repeat_every(seconds=10, logger=logger)
+@repeat_every(seconds=10)
 def broker_upper_messages():
     """Broker for receive message from the `/upper` path and change it to upper
-    case.
+    case. This broker use interval running in background every 10 seconds.
     """
     for _ in range(app.queue_limit):
         try:
@@ -66,11 +65,12 @@ async def get_result(request_id):
             result = app.output_dict[request_id]
             del app.output_dict[request_id]
             return {"message": result}
-        await asyncio.sleep(0.001)
+        await asyncio.sleep(0.0025)
 
 
 @app.post("/upper", response_class=UJSONResponse)
 async def message_upper(payload: Payload):
+    """Convert message from any case to the upper case."""
     request_id: str = str(uuid.uuid4())
     app.queue.put(
         {"text": payload.text, "request_id": request_id},

ddeutil_workflow-0.0.10/src/ddeutil/workflow/cli.py

@@ -0,0 +1,134 @@
+# ------------------------------------------------------------------------------
+# Copyright (c) 2022 Korawich Anuttra. All rights reserved.
+# Licensed under the MIT License. See LICENSE in the project root for
+# license information.
+# ------------------------------------------------------------------------------
+from __future__ import annotations
+
+import json
+import os
+from datetime import datetime
+from enum import Enum
+from typing import Annotated, Optional
+from zoneinfo import ZoneInfo
+
+from ddeutil.core import str2list
+from typer import Argument, Option, Typer
+
+from .log import get_logger
+
+logger = get_logger("ddeutil.workflow")
+cli: Typer = Typer()
+cli_log: Typer = Typer()
+cli.add_typer(
+    cli_log,
+    name="log",
+    help="Logging of workflow CLI",
+)
+
+
+@cli.command()
+def run(
+    pipeline: Annotated[
+        str,
+        Argument(help="A pipeline name that want to run manually"),
+    ],
+    params: Annotated[
+        str,
+        Argument(
+            help="A json string for parameters of this pipeline execution."
+        ),
+    ],
+):
+    """Run pipeline workflow manually with an input custom parameters that able
+    to receive with pipeline params config.
+    """
+    logger.info(f"Running pipeline name: {pipeline}")
+    logger.info(f"... with Parameters: {json.dumps(json.loads(params))}")
+
+
+@cli.command()
+def schedule(
+    stop: Annotated[
+        Optional[datetime],
+        Argument(
+            formats=["%Y-%m-%d", "%Y-%m-%d %H:%M:%S"],
+            help="A stopping datetime that want to stop on schedule app.",
+        ),
+    ] = None,
+    excluded: Annotated[
+        Optional[str],
+        Argument(help="A list of exclude workflow name in str."),
+    ] = None,
+    externals: Annotated[
+        Optional[str],
+        Argument(
+            help="A json string for parameters of this pipeline execution."
+        ),
+    ] = None,
+):
+    """Start workflow scheduler that will call workflow function from scheduler
+    module.
+    """
+    excluded: list[str] = str2list(excluded) if excluded else []
+    externals: str = externals or "{}"
+    if stop:
+        stop: datetime = stop.astimezone(
+            tz=ZoneInfo(os.getenv("WORKFLOW_CORE_TIMEZONE", "UTC"))
+        )
+
+    from .scheduler import workflow
+
+    # NOTE: Start running workflow scheduler application.
+    workflow_rs: list[str] = workflow(
+        stop=stop, excluded=excluded, externals=json.loads(externals)
+    )
+    logger.info(f"Application run success: {workflow_rs}")
+
+
+@cli_log.command("pipeline-get")
+def pipeline_log_get(
+    name: Annotated[
+        str,
+        Argument(help="A pipeline name that want to getting log"),
+    ],
+    limit: Annotated[
+        int,
+        Argument(help="A number of the limitation of logging"),
+    ] = 100,
+    desc: Annotated[
+        bool,
+        Option(
+            "--desc",
+            help="A descending flag that order by logging release datetime.",
+        ),
+    ] = True,
+):
+    logger.info(f"{name} : limit {limit} : desc: {desc}")
+    return [""]
+
+
+class LogMode(str, Enum):
+    get = "get"
+    delete = "delete"
+
+
+@cli_log.command("pipeline-delete")
+def pipeline_log_delete(
+    mode: Annotated[
+        LogMode,
+        Argument(case_sensitive=True),
+    ]
+):
+    logger.info(mode)
+
+
+@cli.callback()
+def main():
+    """
+    Manage workflow with CLI.
+    """
+
+
+if __name__ == "__main__":
+    cli()