ddeutil-workflow 0.0.27__py3-none-any.whl → 0.0.29__py3-none-any.whl

ddeutil_workflow-0.0.27.dist-info/METADATA

@@ -1,230 +0,0 @@
-Metadata-Version: 2.1
-Name: ddeutil-workflow
-Version: 0.0.27
-Summary: Lightweight workflow orchestration with less dependencies
-Author-email: ddeutils <korawich.anu@gmail.com>
-License: MIT
-Project-URL: Homepage, https://github.com/ddeutils/ddeutil-workflow/
-Project-URL: Source Code, https://github.com/ddeutils/ddeutil-workflow/
-Keywords: orchestration,workflow
-Classifier: Topic :: Utilities
-Classifier: Natural Language :: English
-Classifier: Development Status :: 4 - Beta
-Classifier: Intended Audience :: Developers
-Classifier: Operating System :: OS Independent
-Classifier: Programming Language :: Python
-Classifier: Programming Language :: Python :: 3 :: Only
-Classifier: Programming Language :: Python :: 3.9
-Classifier: Programming Language :: Python :: 3.10
-Classifier: Programming Language :: Python :: 3.11
-Classifier: Programming Language :: Python :: 3.12
-Classifier: Programming Language :: Python :: 3.13
-Requires-Python: >=3.9.13
-Description-Content-Type: text/markdown
-License-File: LICENSE
-Requires-Dist: ddeutil>=0.4.3
-Requires-Dist: ddeutil-io[toml,yaml]>=0.2.3
-Requires-Dist: pydantic==2.10.4
-Requires-Dist: python-dotenv==1.0.1
-Requires-Dist: schedule<2.0.0,==1.2.2
-Provides-Extra: api
-Requires-Dist: fastapi<1.0.0,>=0.115.0; extra == "api"
-
-# Workflow
-
-[![test](https://github.com/ddeutils/ddeutil-workflow/actions/workflows/tests.yml/badge.svg?branch=main)](https://github.com/ddeutils/ddeutil-workflow/actions/workflows/tests.yml)
-[![codecov](https://codecov.io/gh/ddeutils/ddeutil-workflow/graph/badge.svg?token=3NDPN2I0H9)](https://codecov.io/gh/ddeutils/ddeutil-workflow)
-[![pypi version](https://img.shields.io/pypi/v/ddeutil-workflow)](https://pypi.org/project/ddeutil-workflow/)
-[![python support version](https://img.shields.io/pypi/pyversions/ddeutil-workflow)](https://pypi.org/project/ddeutil-workflow/)
-[![size](https://img.shields.io/github/languages/code-size/ddeutils/ddeutil-workflow)](https://github.com/ddeutils/ddeutil-workflow)
-[![gh license](https://img.shields.io/github/license/ddeutils/ddeutil-workflow)](https://github.com/ddeutils/ddeutil-workflow/blob/main/LICENSE)
-[![code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
-
-The **Lightweight workflow orchestration** with less dependencies the was created
-for easy to make a simple metadata driven for data workflow orchestration.
-It can to use for data operator by a `.yaml` template.
-
-> [!WARNING]
-> This package provide only orchestration workload. That mean you should not
-> use the workflow stage to process any large volume data which use lot of compute
-> resource. :cold_sweat:
-
-In my opinion, I think it should not create duplicate workflow codes if I can
-write with dynamic input parameters on the one template workflow that just change
-the input parameters per use-case instead.
-This way I can handle a lot of logical workflows in our orgs with only metadata
-configuration. It called **Metadata Driven Data Workflow**.
-
-**:pushpin: <u>Rules of This Workflow engine</u>**:
-
-1. The Minimum frequency unit of scheduling is **1 minute** :warning:
-2. Can not re-run only failed stage and its pending downstream :rotating_light:
-3. All parallel tasks inside workflow engine use Multi-Threading
-   (Python 3.13 unlock GIL :unlock:)
-
-> [!NOTE]
-> _Disclaimer_: I inspire the dynamic statement from the [**GitHub Action**](https://github.com/features/actions)
-> with `.yml` files and all of config file from several data orchestration framework
-> tools from my experience on Data Engineer. :grimacing:
->
-> Other workflow tools that I interest on them and pick some interested feature
-> implement to this package:
->
-> - [Google **Workflows**](https://cloud.google.com/workflows)
-> - [AWS **Step Functions**](https://aws.amazon.com/step-functions/)
-
-## :round_pushpin: Installation
-
-This project need `ddeutil` and `ddeutil-io` extension namespace packages.
-If you want to install this package with application add-ons, you should add
-`app` in installation;
-
-| Usecase        | Install Optional                         | Support            |
-|----------------|------------------------------------------|--------------------|
-| Python         | `pip install ddeutil-workflow`           | :heavy_check_mark: |
-| FastAPI Server | `pip install ddeutil-workflow[api]`      | :heavy_check_mark: |
-
-## :beers: Usage
-
-This is examples that use workflow file for running common Data Engineering
-use-case.
-
-> [!IMPORTANT]
-> I recommend you to use the `hook` stage for all actions that you want to do
-> with workflow activity that you want to orchestrate. Because it able to dynamic
-> an input argument with the same hook function that make you use less time to
-> maintenance your data workflows.
-
-```yaml
-run-py-local:
-
-   # Validate model that use to parsing exists for template file
-   type: Workflow
-   on:
-      # If workflow deploy to schedule, it will running every 5 minutes
-      # with Asia/Bangkok timezone.
-      - cronjob: '*/5 * * * *'
-        timezone: "Asia/Bangkok"
-   params:
-      # Incoming execution parameters will validate with this type. It allow
-      # to set default value or templating.
-      source-extract: str
-      run-date: datetime
-   jobs:
-      getting-api-data:
-         stages:
-            - name: "Retrieve API Data"
-              id: retrieve-api
-              uses: tasks/get-api-with-oauth-to-s3@requests
-              with:
-                 # Arguments of source data that want to retrieve.
-                 method: post
-                 url: https://finances/open-data/currency-pairs/
-                 body:
-                    resource: ${{ params.source-extract }}
-
-                    # You can able to use filtering like Jinja template but this
-                    # package does not use it.
-                    filter: ${{ params.run-date | fmt(fmt='%Y%m%d') }}
-                 auth:
-                    type: bearer
-                    keys: ${API_ACCESS_REFRESH_TOKEN}
-
-                 # Arguments of target data that want to landing.
-                 writing_mode: flatten
-                 aws_s3_path: my-data/open-data/${{ params.source-extract }}
-
-                 # This Authentication code should implement with your custom hook
-                 # function. The template allow you to use environment variable.
-                 aws_access_client_id: ${AWS_ACCESS_CLIENT_ID}
-                 aws_access_client_secret: ${AWS_ACCESS_CLIENT_SECRET}
-```
-
-The above workflow template is main executor pipeline that you want to do. If you
-want to schedule this workflow, you want to dynamic its parameters change base on
-execution time such as `run-date` should change base on that workflow running date.
-
-So, this package provide the `Schedule` template for this action.
-
-```yaml
-schedule-run-local-wf:
-
-   # Validate model that use to parsing exists for template file
-   type: Schedule
-   workflows:
-
-      # Map existing workflow that want to deploy with scheduler application.
-      # It allow you to passing release parameter that dynamic change depend the
-      # current context of this scheduler application releasing that time.
-      - name: run-py-local
-        params:
-          source-extract: "USD-THB"
-          asat-dt: "${{ release.logical_date }}"
-```
-
-## :cookie: Configuration
-
-The main configuration that use to dynamic changing with your propose of this
-application. If any configuration values do not set yet, it will use default value
-and do not raise any error to you.
-
-| Environment                                | Component | Default                           | Description                                                                                                        | Remark |
-|:-------------------------------------------|:---------:|:----------------------------------|:-------------------------------------------------------------------------------------------------------------------|--------|
-| **WORKFLOW_ROOT_PATH**                     |   Core    | `.`                               | The root path of the workflow application.                                                                         |        |
-| **WORKFLOW_CORE_REGISTRY**                 |   Core    | `src`                             | List of importable string for the hook stage.                                                                      |        |
-| **WORKFLOW_CORE_REGISTRY_FILTER**          |   Core    | `ddeutil.workflow.utils`          | List of importable string for the filter template.                                                                 |        |
-| **WORKFLOW_CORE_PATH_CONF**                |   Core    | `conf`                            | The config path that keep all template `.yaml` files.                                                              |        |
-| **WORKFLOW_CORE_TIMEZONE**                 |   Core    | `Asia/Bangkok`                    | A Timezone string value that will pass to `ZoneInfo` object.                                                       |        |
-| **WORKFLOW_CORE_STAGE_DEFAULT_ID**         |   Core    | `true`                            | A flag that enable default stage ID that use for catch an execution output.                                        |        |
-| **WORKFLOW_CORE_STAGE_RAISE_ERROR**        |   Core    | `false`                           | A flag that all stage raise StageException from stage execution.                                                   |        |
-| **WORKFLOW_CORE_JOB_DEFAULT_ID**           |   Core    | `false`                           | A flag that enable default job ID that use for catch an execution output. The ID that use will be sequence number. |        |
-| **WORKFLOW_CORE_JOB_RAISE_ERROR**          |   Core    | `true`                            | A flag that all job raise JobException from job strategy execution.                                                |        |
-| **WORKFLOW_CORE_MAX_NUM_POKING**           |   Core    | `4`                               | .                                                                                                                  |        |
-| **WORKFLOW_CORE_MAX_JOB_PARALLEL**         |   Core    | `2`                               | The maximum job number that able to run parallel in workflow executor.                                             |        |
-| **WORKFLOW_CORE_MAX_JOB_EXEC_TIMEOUT**     |   Core    | `600`                             |                                                                                                                    |        |
-| **WORKFLOW_CORE_MAX_CRON_PER_WORKFLOW**    |   Core    | `5`                               |                                                                                                                    |        |
-| **WORKFLOW_CORE_MAX_QUEUE_COMPLETE_HIST**  |   Core    | `16`                              |                                                                                                                    |        |
-| **WORKFLOW_CORE_GENERATE_ID_SIMPLE_MODE**  |   Core    | `true`                            | A flog that enable generating ID with `md5` algorithm.                                                             |        |
-| **WORKFLOW_LOG_PATH**                      |    Log    | `./logs`                          | The log path of the workflow saving log.                                                                           |        |
-| **WORKFLOW_LOG_DEBUG_MODE**                |    Log    | `true`                            | A flag that enable logging with debug level mode.                                                                  |        |
-| **WORKFLOW_LOG_ENABLE_WRITE**              |    Log    | `true`                            | A flag that enable logging object saving log to its destination.                                                   |        |
-| **WORKFLOW_APP_MAX_PROCESS**               | Schedule  | `2`                               | The maximum process worker number that run in scheduler app module.                                                |        |
-| **WORKFLOW_APP_MAX_SCHEDULE_PER_PROCESS**  | Schedule  | `100`                             | A schedule per process that run parallel.                                                                          |        |
-| **WORKFLOW_APP_STOP_BOUNDARY_DELTA**       | Schedule  | `'{"minutes": 5, "seconds": 20}'` | A time delta value that use to stop scheduler app in json string format.                                           |        |
-
-**API Application**:
-
-| Environment                             |  Component  | Default | Description                                                                        | Remark |
-|:----------------------------------------|:-----------:|---------|------------------------------------------------------------------------------------|--------|
-| **WORKFLOW_API_ENABLE_ROUTE_WORKFLOW**  |     API     | `true`  | A flag that enable workflow route to manage execute manually and workflow logging. |        |
-| **WORKFLOW_API_ENABLE_ROUTE_SCHEDULE**  |     API     | `true`  | A flag that enable run scheduler.                                                  |        |
-
-## :rocket: Deployment
-
-This package able to run as a application service for receive manual trigger
-from the master node via RestAPI or use to be Scheduler background service
-like crontab job but via Python API.
-
-### API Server
-
-```shell
-(venv) $ uvicorn src.ddeutil.workflow.api:app --host 127.0.0.1 --port 80
-```
-
-> [!NOTE]
-> If this package already deploy, it able to use multiprocess;
-> `uvicorn ddeutil.workflow.api:app --host 127.0.0.1 --port 80 --workers 4`
-
-### Docker Container
-
-Create Docker image;
-
-```shell
-$ docker build -t ddeutil-workflow:latest -f .container/Dockerfile .
-```
-
-Run the above Docker image;
-
-```shell
-$ docker run -i ddeutil-workflow:latest
-```

ddeutil_workflow-0.0.27.dist-info/RECORD

@@ -1,25 +0,0 @@
-ddeutil/workflow/__about__.py,sha256=Tb1KYKrlWsfQNnf5zwGCHzEkcwD78iHOrO3PQIyUgTY,28
-ddeutil/workflow/__cron.py,sha256=uA8XcbY_GwA9rJSHaHUaXaJyGDObJN0ZeYlJSinL8y8,26880
-ddeutil/workflow/__init__.py,sha256=ATYXzGtLyq4LWCtJ-Odz36QSrLL7dKymVs8ziThOVOk,1582
-ddeutil/workflow/__types.py,sha256=Ia7f38kvL3NibwmRKi0wQ1ud_45Z-SojYGhNJwIqcu8,3713
-ddeutil/workflow/conf.py,sha256=jr7KPnt3vd7icuXTLGcJt_kT9tlmN1Cu5QBDMHUrm94,16819
-ddeutil/workflow/cron.py,sha256=75A0hqevvouziKoLALncLJspVAeki9qCH3zniAJaxzY,7513
-ddeutil/workflow/exceptions.py,sha256=NqnQJP52S59XIYMeXbTDbr4xH2UZ5EA3ejpU5Z4g6cQ,894
-ddeutil/workflow/hook.py,sha256=yXhpr9E6ZzPb9_9ed79rWiRWDLnwkbRRg3TPmLvqoEI,4899
-ddeutil/workflow/job.py,sha256=JJ4vSpuhQnY7LOMf9xq6N8pBZQ1oAxqYQFbKHn_HjdQ,24237
-ddeutil/workflow/params.py,sha256=uPGkZx18E-iZ8BteqQ2ONgg0frhF3ZmP5cOyfK2j59U,5280
-ddeutil/workflow/result.py,sha256=WIC8MsnfLiWNpZomT6jS4YCdYhlbIVVBjtGGe2dkoKk,3404
-ddeutil/workflow/scheduler.py,sha256=BbY_3Y3QOdNwDfdvnRa7grGC2_a0Hn1KJbZKAscchk8,20454
-ddeutil/workflow/stage.py,sha256=JJDuObNzpw803-bECmDzeGuaxQW2DnR0Ps8Tl0uJZnw,25033
-ddeutil/workflow/templates.py,sha256=X-s5IZjwYpSD7UY3jaQiqbQBBG_Z3cWJDkzEIpicldg,10797
-ddeutil/workflow/utils.py,sha256=jg0ZsbglrrF3bAakQ3rSna9KTMq2Qf_NPLnlORHf3J0,6039
-ddeutil/workflow/workflow.py,sha256=QoSljQakGcumrx8l-W9yWuQZUTCrhArwAYktsj_L_9s,42204
-ddeutil/workflow/api/__init__.py,sha256=F53NMBWtb9IKaDWkPU5KvybGGfKAcbehgn6TLBwHuuM,21
-ddeutil/workflow/api/api.py,sha256=Md1cz3Edc7_uz63s_L_i-R3IE4mkO3aTADrX8GOGU-Y,5644
-ddeutil/workflow/api/repeat.py,sha256=zyvsrXKk-3-_N8ZRZSki0Mueshugum2jtqctEOp9QSc,4927
-ddeutil/workflow/api/route.py,sha256=v96jNbgjM1cJ2MpVSRWs2kgRqF8DQElEBdRZrVFEpEw,8578
-ddeutil_workflow-0.0.27.dist-info/LICENSE,sha256=nGFZ1QEhhhWeMHf9n99_fdt4vQaXS29xWKxt-OcLywk,1085
-ddeutil_workflow-0.0.27.dist-info/METADATA,sha256=SC2dWZTZ0eruriXwbnhuCHhOg-0Z6wT1t026hJvsLnw,13921
-ddeutil_workflow-0.0.27.dist-info/WHEEL,sha256=A3WOREP4zgxI0fKrHUG8DC8013e3dK3n7a6HDbcEIwE,91
-ddeutil_workflow-0.0.27.dist-info/top_level.txt,sha256=m9M6XeSWDwt_yMsmH6gcOjHZVK5O0-vgtNBuncHjzW4,8
-ddeutil_workflow-0.0.27.dist-info/RECORD,,