ddeutil-workflow 0.0.4__py3-none-any.whl → 0.0.5__py3-none-any.whl

{ddeutil_workflow-0.0.4.dist-info → ddeutil_workflow-0.0.5.dist-info}/METADATA

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: ddeutil-workflow
-Version: 0.0.4
+Version: 0.0.5
 Summary: Data Developer & Engineer Workflow Utility Objects
 Author-email: ddeutils <korawich.anu@gmail.com>
 License: MIT
@@ -9,7 +9,7 @@ Project-URL: Source Code, https://github.com/ddeutils/ddeutil-workflow/
 Keywords: data,workflow,utility,pipeline
 Classifier: Topic :: Utilities
 Classifier: Natural Language :: English
-Classifier: Development Status :: 3 - Alpha
+Classifier: Development Status :: 4 - Beta
 Classifier: Intended Audience :: Developers
 Classifier: Operating System :: OS Independent
 Classifier: Programming Language :: Python
@@ -23,21 +23,16 @@ Description-Content-Type: text/markdown
 License-File: LICENSE
 Requires-Dist: fmtutil
 Requires-Dist: ddeutil-io
-Requires-Dist: python-dotenv
-Provides-Extra: test
-Requires-Dist: sqlalchemy ==2.0.30 ; extra == 'test'
-Requires-Dist: paramiko ==3.4.0 ; extra == 'test'
-Requires-Dist: sshtunnel ==0.4.0 ; extra == 'test'
-Requires-Dist: boto3 ==1.34.117 ; extra == 'test'
-Requires-Dist: fsspec ==2024.5.0 ; extra == 'test'
-Requires-Dist: polars ==0.20.31 ; extra == 'test'
-Requires-Dist: pyarrow ==16.1.0 ; extra == 'test'
+Requires-Dist: python-dotenv ==1.0.1
+Requires-Dist: schedule ==1.2.2
 
 # Data Utility: _Workflow_
 
 [![test](https://github.com/ddeutils/ddeutil-workflow/actions/workflows/tests.yml/badge.svg?branch=main)](https://github.com/ddeutils/ddeutil-workflow/actions/workflows/tests.yml)
 [![python support version](https://img.shields.io/pypi/pyversions/ddeutil-workflow)](https://pypi.org/project/ddeutil-workflow/)
 [![size](https://img.shields.io/github/languages/code-size/ddeutils/ddeutil-workflow)](https://github.com/ddeutils/ddeutil-workflow)
+[![gh license](https://img.shields.io/github/license/ddeutils/ddeutil-workflow)](https://github.com/ddeutils/ddeutil-workflow/blob/main/LICENSE)
+
 
 **Table of Contents**:
 
@@ -46,10 +41,11 @@ Requires-Dist: pyarrow ==16.1.0 ; extra == 'test'
   - [Connection](#connection)
   - [Dataset](#dataset)
   - [Schedule](#schedule)
-- [Examples](#examples)
-  - [Python](#python)
+- [Pipeline Examples](#examples)
+  - [Python & Shell](#python--shell)
   - [Tasks (EL)](#tasks-extract--load)
-  - [Hooks (T)](#hooks-transform)
+  - [Hooks (T)](#tasks-transform)
+- [Configuration](#configuration)
 
 This **Utility Workflow** objects was created for easy to make a simple metadata
 driven pipeline that able to **ETL, T, EL, or ELT** by `.yaml` file.
@@ -80,7 +76,7 @@ This project need `ddeutil-io`, `ddeutil-model` extension namespace packages.
 
 The first step, you should start create the connections and datasets for In and
 Out of you data that want to use in pipeline of workflow. Some of this component
-is similar component of the **Airflow** because I like it concepts.
+is similar component of the **Airflow** because I like it orchestration concepts.
 
 The main feature of this project is the `Pipeline` object that can call any
 registries function. The pipeline can handle everything that you want to do, it
@@ -91,44 +87,7 @@ will passing parameters and catching the output for re-use it to next step.
 > dynamic registries instead of main features because it have a lot of maintain
 > vendor codes and deps. (I do not have time to handle this features)
 
-### Connection
-
-The connection for worker able to do any thing.
-
-```yaml
-conn_postgres_data:
-  type: conn.Postgres
-  url: 'postgres//username:${ENV_PASS}@hostname:port/database?echo=True&time_out=10'
-```
-
-```python
-from ddeutil.workflow.conn import Conn
-
-conn = Conn.from_loader(name='conn_postgres_data', externals={})
-assert conn.ping()
-```
-
-### Dataset
-
-The dataset is define any objects on the connection. This feature was implemented
-on `/vendors` because it has a lot of tools that can interact with any data systems
-in the data tool stacks.
-
-```yaml
-ds_postgres_customer_tbl:
-  type: dataset.PostgresTbl
-  conn: 'conn_postgres_data'
-  features:
-    id: serial primary key
-    name: varchar( 100 ) not null
-```
-
-```python
-from ddeutil.workflow.vendors.pg import PostgresTbl
-
-dataset = PostgresTbl.from_loader(name='ds_postgres_customer_tbl', externals={})
-assert dataset.exists()
-```
+---
 
 ### Schedule
 
@@ -139,7 +98,7 @@ schd_for_node:
 ```
 
 ```python
-from ddeutil.workflow.schedule import Schedule
+from ddeutil.workflow.on import Schedule
 
 scdl = Schedule.from_loader(name='schd_for_node', externals={})
 assert '*/5 * * * *' == str(scdl.cronjob)
@@ -152,18 +111,35 @@ assert '2022-01-01 00:20:00' f"{cron_iterate.next:%Y-%m-%d %H:%M:%S}"
 assert '2022-01-01 00:25:00' f"{cron_iterate.next:%Y-%m-%d %H:%M:%S}"
 ```
 
+---
+
+### Pipeline
+
+```yaml
+run_py_local:
+  type: ddeutil.workflow.pipeline.Pipeline
+  ...
+```
+
+```python
+from ddeutil.workflow.pipeline import Pipeline
+
+pipe = Pipeline.from_loader(name='run_py_local', externals={})
+pipe.execute(params={'author-run': 'Local Workflow', 'run-date': '2024-01-01'})
+```
+
 ## Examples
 
 This is examples that use workflow file for running common Data Engineering
 use-case.
 
-### Python
+### Python & Shell
 
 The state of doing lists that worker should to do. It be collection of the stage.
 
 ```yaml
 run_py_local:
-  type: ddeutil.workflow.pipe.Pipeline
+  type: ddeutil.workflow.pipeline.Pipeline
   params:
     author-run:
       type: str
@@ -194,6 +170,12 @@ run_py_local:
             echo: ${{ stages.define-func.outputs.echo }}
           run: |
             echo('Caller')
+    second-job:
+      stages:
+        - name: Echo Shell Script
+          id: shell-echo
+          shell: |
+            echo "Hello World from Shell"
 ```
 
 ```python
@@ -207,13 +189,16 @@ pipe.execute(params={'author-run': 'Local Workflow', 'run-date': '2024-01-01'})
 > Hello Local Workflow
 > Receive x from above with Local Workflow
 > Hello Caller
+> Hello World from Shell
 ```
 
+---
+
 ### Tasks (Extract & Load)
 
 ```yaml
 pipe_el_pg_to_lake:
-  type: ddeutil.workflow.pipe.Pipeline
+  type: ddeutil.workflow.pipeline.Pipeline
   params:
     run-date:
       type: datetime
@@ -236,11 +221,15 @@ pipe_el_pg_to_lake:
               endpoint: "/${{ params.name }}"
 ```
 
+---
+
 ### Tasks (Transform)
 
+> I recommend you to use task for all actions that you want to do.
+
 ```yaml
 pipe_hook_mssql_proc:
-  type: ddeutil.workflow.pipe.Pipeline
+  type: ddeutil.workflow.pipeline.Pipeline
   params:
     run_date: datetime
     sp_name: str
@@ -261,6 +250,16 @@ pipe_hook_mssql_proc:
               target: ${{ params.target_name }}
 ```
 
+> [!NOTE]
+> The above parameter use short declarative statement. You can pass a parameter
+> type to the key of a parameter name.
+
+## Configuration
+
+```text
+
+```
+
 ## License
 
 This project was licensed under the terms of the [MIT license](LICENSE).

ddeutil_workflow-0.0.5.dist-info/RECORD

@@ -0,0 +1,17 @@
+ddeutil/workflow/__about__.py,sha256=jgkUUyo8sKJnE1-6McC_AbxbZqvAFoYRfSE3HCAexlk,27
+ddeutil/workflow/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
+ddeutil/workflow/__regex.py,sha256=bOngaQ0zJgy3vfNwF2MlI8XhLu_Ei1Vz8y50iLj8ao4,1061
+ddeutil/workflow/__scheduler.py,sha256=wSzv6EN2Nx99lLxzrA80qfqQ_AxOFJAOk_EZYgafVzk,20965
+ddeutil/workflow/__types.py,sha256=AkpQq6QlrclpurCZZVY9RMxoyS9z2WGzhaz_ikeTaCU,453
+ddeutil/workflow/exceptions.py,sha256=XAq82VHSMLNb4UjGatp7hYfjxFtMiKFtBqJyAhwTl-s,434
+ddeutil/workflow/loader.py,sha256=xiOtxluhLXfryMp3q1OIJggykr01WENKV7zBkcJ9-Yc,5763
+ddeutil/workflow/on.py,sha256=vYNcvh74MN2ttd9KOehngOhJDNqDheFbbsXeEhDSqyk,5280
+ddeutil/workflow/pipeline.py,sha256=tnzeAQrXb_-OIo53IGv6LxZoMiOJyMPWXAbisdBCXHI,19298
+ddeutil/workflow/utils.py,sha256=A-k3L4OUFFq6utlsbpEVMGyofiLbDrvThM8IqFDE9gE,6093
+ddeutil/workflow/tasks/__init__.py,sha256=HcQ7xNETFOKovMOs4lL2Pl8hXFZ515jU5Mc3LFZcSGE,336
+ddeutil/workflow/tasks/dummy.py,sha256=b_y6eHGxj4aQ-ZmvcNL7aBHu3eIzL6BeXgqj0MDqSPw,1460
+ddeutil_workflow-0.0.5.dist-info/LICENSE,sha256=nGFZ1QEhhhWeMHf9n99_fdt4vQaXS29xWKxt-OcLywk,1085
+ddeutil_workflow-0.0.5.dist-info/METADATA,sha256=dcZadiwnhPD6DnoBaugdPHlOw108lRCmhDD7KF2s-Dg,7717
+ddeutil_workflow-0.0.5.dist-info/WHEEL,sha256=R0nc6qTxuoLk7ShA2_Y-UWkN8ZdfDBG2B6Eqpz2WXbs,91
+ddeutil_workflow-0.0.5.dist-info/top_level.txt,sha256=m9M6XeSWDwt_yMsmH6gcOjHZVK5O0-vgtNBuncHjzW4,8
+ddeutil_workflow-0.0.5.dist-info/RECORD,,

{ddeutil_workflow-0.0.4.dist-info → ddeutil_workflow-0.0.5.dist-info}/WHEEL

@@ -1,5 +1,5 @@
 Wheel-Version: 1.0
-Generator: bdist_wheel (0.43.0)
+Generator: setuptools (72.1.0)
 Root-Is-Purelib: true
 Tag: py3-none-any
 

ddeutil/workflow/conn.py

@@ -1,240 +0,0 @@
-# ------------------------------------------------------------------------------
-# Copyright (c) 2022 Korawich Anuttra. All rights reserved.
-# Licensed under the MIT License. See LICENSE in the project root for
-# license information.
-# ------------------------------------------------------------------------------
-from __future__ import annotations
-
-import logging
-from collections.abc import Iterator
-from pathlib import Path
-from typing import Annotated, Any, Literal, Optional, TypeVar
-
-from ddeutil.io.models.conn import Conn as ConnModel
-from pydantic import BaseModel, ConfigDict, Field
-from pydantic.functional_validators import field_validator
-from pydantic.types import SecretStr
-from typing_extensions import Self
-
-from .__types import DictData, TupleStr
-from .loader import Loader
-
-EXCLUDED_EXTRAS: TupleStr = (
-    "type",
-    "url",
-)
-
-
-class BaseConn(BaseModel):
-    """Base Conn (Connection) Model"""
-
-    model_config = ConfigDict(arbitrary_types_allowed=True)
-
-    # NOTE: This is fields
-    dialect: str
-    host: Optional[str] = None
-    port: Optional[int] = None
-    user: Optional[str] = None
-    pwd: Optional[SecretStr] = None
-    endpoint: str
-    extras: Annotated[
-        DictData,
-        Field(default_factory=dict, description="Extras mapping of parameters"),
-    ]
-
-    @classmethod
-    def from_dict(cls, values: DictData) -> Self:
-        """Construct Connection Model from dict data. This construct is
-        different with ``.model_validate()`` because it will prepare the values
-        before using it if the data dose not have 'url'.
-
-        :param values: A dict data that use to construct this model.
-        """
-        # NOTE: filter out the fields of this model.
-        filter_data: DictData = {
-            k: values.pop(k)
-            for k in values.copy()
-            if k not in cls.model_fields and k not in EXCLUDED_EXTRAS
-        }
-        if "url" in values:
-            url: ConnModel = ConnModel.from_url(values.pop("url"))
-            return cls(
-                dialect=url.dialect,
-                host=url.host,
-                port=url.port,
-                user=url.user,
-                pwd=url.pwd,
-                # NOTE:
-                #   I will replace None endpoint with memory value for SQLite
-                #   connection string.
-                endpoint=(url.endpoint or "memory"),
-                # NOTE: This order will show that externals this the top level.
-                extras=(url.options | filter_data),
-            )
-        return cls.model_validate(
-            obj={
-                "extras": (values.pop("extras", {}) | filter_data),
-                **values,
-            }
-        )
-
-    @classmethod
-    def from_loader(cls, name: str, externals: DictData) -> Self:
-        """Construct Connection with Loader object with specific config name.
-
-        :param name: A config name.
-        :param externals: A external data that want to adding to extras.
-        """
-        loader: Loader = Loader(name, externals=externals)
-        # NOTE: Validate the config type match with current connection model
-        if loader.type != cls:
-            raise ValueError(f"Type {loader.type} does not match with {cls}")
-        return cls.from_dict(
-            {
-                "extras": (loader.data.pop("extras", {}) | externals),
-                **loader.data,
-            }
-        )
-
-    @field_validator("endpoint")
-    def __prepare_slash(cls, value: str) -> str:
-        """Prepare slash character that map double form URL model loading."""
-        if value.startswith("//"):
-            return value[1:]
-        return value
-
-
-class Conn(BaseConn):
-    """Conn (Connection) Model that implement any necessary methods. This object
-    should be the base for abstraction to any connection model object.
-    """
-
-    def get_spec(self) -> str:
-        """Return full connection url that construct from all fields."""
-        return (
-            f"{self.dialect}://{self.user or ''}"
-            f"{f':{self.pwd}' if self.pwd else ''}"
-            f"{self.host or ''}{f':{self.port}' if self.port else ''}"
-            f"/{self.endpoint}"
-        )
-
-    def ping(self) -> bool:
-        """Ping the connection that able to use with this field value."""
-        raise NotImplementedError("Ping does not implement")
-
-    def glob(self, pattern: str) -> Iterator[Any]:
-        """Return a list of object from the endpoint of this connection."""
-        raise NotImplementedError("Glob does not implement")
-
-    def find_object(self, _object: str):
-        raise NotImplementedError("Glob does not implement")
-
-
-class FlSys(Conn):
-    """File System Connection."""
-
-    dialect: Literal["local"] = "local"
-
-    def ping(self) -> bool:
-        return Path(self.endpoint).exists()
-
-    def glob(self, pattern: str) -> Iterator[Path]:
-        yield from Path(self.endpoint).rglob(pattern=pattern)
-
-    def find_object(self, _object: str) -> bool:
-        return (Path(self.endpoint) / _object).exists()
-
-
-class SFTP(Conn):
-    """SFTP Server Connection."""
-
-    dialect: Literal["sftp"] = "sftp"
-
-    def __client(self):
-        from .vendors.sftp import WrapSFTP
-
-        return WrapSFTP(
-            host=self.host,
-            port=self.port,
-            user=self.user,
-            pwd=self.pwd.get_secret_value(),
-        )
-
-    def ping(self) -> bool:
-        with self.__client().simple_client():
-            return True
-
-    def glob(self, pattern: str) -> Iterator[str]:
-        yield from self.__client().walk(pattern=pattern)
-
-
-class Db(Conn):
-    """RDBMS System Connection"""
-
-    def ping(self) -> bool:
-        from sqlalchemy import create_engine
-        from sqlalchemy.engine import URL, Engine
-        from sqlalchemy.exc import OperationalError
-
-        engine: Engine = create_engine(
-            url=URL.create(
-                self.dialect,
-                username=self.user,
-                password=self.pwd.get_secret_value() if self.pwd else None,
-                host=self.host,
-                port=self.port,
-                database=self.endpoint,
-                query={},
-            ),
-            execution_options={},
-        )
-        try:
-            return engine.connect()
-        except OperationalError as err:
-            logging.warning(str(err))
-            return False
-
-
-class SQLite(Db):
-    dialect: Literal["sqlite"]
-
-
-class ODBC(Conn): ...
-
-
-class Doc(Conn):
-    """No SQL System Connection"""
-
-
-class Mongo(Doc): ...
-
-
-class SSHCred(BaseModel):
-    ssh_host: str
-    ssh_user: str
-    ssh_password: Optional[SecretStr] = Field(default=None)
-    ssh_private_key: Optional[str] = Field(default=None)
-    ssh_private_key_pwd: Optional[SecretStr] = Field(default=None)
-    ssh_port: int = Field(default=22)
-
-
-class S3Cred(BaseModel):
-    aws_access_key: str
-    aws_secret_access_key: SecretStr
-    region: str = Field(default="ap-southeast-1")
-    role_arn: Optional[str] = Field(default=None)
-    role_name: Optional[str] = Field(default=None)
-    mfa_serial: Optional[str] = Field(default=None)
-
-
-class AZServPrinCred(BaseModel):
-    tenant: str
-    client_id: str
-    secret_id: SecretStr
-
-
-class GoogleCred(BaseModel):
-    google_json_path: str
-
-
-SubclassConn = TypeVar("SubclassConn", bound=Conn)

ddeutil/workflow/schedule.py

@@ -1,82 +0,0 @@
-# ------------------------------------------------------------------------------
-# Copyright (c) 2022 Korawich Anuttra. All rights reserved.
-# Licensed under the MIT License. See LICENSE in the project root for
-# license information.
-# ------------------------------------------------------------------------------
-from __future__ import annotations
-
-from datetime import datetime
-from typing import Annotated
-from zoneinfo import ZoneInfo, ZoneInfoNotFoundError
-
-from ddeutil.workflow.vendors.__schedule import CronJob, CronRunner
-from pydantic import BaseModel, ConfigDict, Field
-from pydantic.functional_validators import field_validator
-from typing_extensions import Self
-
-from .__types import DictData
-from .loader import Loader
-
-
-class BaseSchedule(BaseModel):
-    """Base Schedule (Schedule) Model"""
-
-    model_config = ConfigDict(arbitrary_types_allowed=True)
-
-    # NOTE: This is fields
-    cronjob: Annotated[CronJob, Field(description="Cron job of this schedule")]
-    tz: Annotated[str, Field(description="Timezone")] = "utc"
-    extras: Annotated[
-        DictData,
-        Field(default_factory=dict, description="Extras mapping of parameters"),
-    ]
-
-    @classmethod
-    def from_loader(
-        cls,
-        name: str,
-        externals: DictData,
-    ) -> Self:
-        loader: Loader = Loader(name, externals=externals)
-        if "cronjob" not in loader.data:
-            raise ValueError("Config does not set ``cronjob`` value")
-        return cls(cronjob=loader.data["cronjob"], extras=externals)
-
-    @field_validator("tz")
-    def __validate_tz(cls, value: str):
-        try:
-            _ = ZoneInfo(value)
-            return value
-        except ZoneInfoNotFoundError as err:
-            raise ValueError(f"Invalid timezone: {value}") from err
-
-    @field_validator("cronjob", mode="before")
-    def __prepare_cronjob(cls, value: str | CronJob) -> CronJob:
-        return CronJob(value) if isinstance(value, str) else value
-
-    def generate(self, start: str | datetime) -> CronRunner:
-        """Return Cron runner object."""
-        if not isinstance(start, datetime):
-            start: datetime = datetime.fromisoformat(start)
-        return self.cronjob.schedule(date=(start.astimezone(ZoneInfo(self.tz))))
-
-
-class Schedule(BaseSchedule):
-    """Schedule (Schedule) Model.
-
-    See Also:
-        * ``generate()`` is the main usecase of this schedule object.
-    """
-
-
-class ScheduleBkk(Schedule):
-    """Asia Bangkok Schedule (Schedule) timezone Model.
-
-    This model use for change timezone from utc to Asia/Bangkok
-    """
-
-    tz: Annotated[str, Field(description="Timezone")] = "Asia/Bangkok"
-
-
-class AwsSchedule(BaseSchedule):
-    """Implement Schedule for AWS Service."""

ddeutil/workflow/tasks/_pandas.py

@@ -1,54 +0,0 @@
-import logging
-import math
-
-try:
-    import pandas as pd
-
-    logging.debug(f"Pandas version: {pd.__version__}")
-except ImportError as err:
-    raise ImportError(
-        "``split_iterable`` function want to use pandas package that does"
-        "not install on your interpreter."
-    ) from err
-
-
-def split_iterable(iterable, chunk_size=None, generator_flag: bool = True):
-    """
-    Split an iterable into mini batch with batch length of batch_number
-    supports batch of a pandas dataframe
-    usage:
-        >> for i in split_iterable([1,2,3,4,5], chunk_size=2):
-        >>    print(i)
-        [1, 2]
-        [3, 4]
-        [5]
-
-        for idx, mini_data in split_iterable(batch(df, chunk_size=10)):
-            print(idx)
-            print(mini_data)
-    """
-
-    chunk_size: int = chunk_size or 25000
-    num_chunks = math.ceil(len(iterable) / chunk_size)
-    if generator_flag:
-        for _ in range(num_chunks):
-            if isinstance(iterable, pd.DataFrame):
-                yield iterable.iloc[_ * chunk_size : (_ + 1) * chunk_size]
-            else:
-                yield iterable[_ * chunk_size : (_ + 1) * chunk_size]
-    else:
-        _chunks: list = []
-        for _ in range(num_chunks):
-            if isinstance(iterable, pd.DataFrame):
-                _chunks.append(
-                    iterable.iloc[_ * chunk_size : (_ + 1) * chunk_size]
-                )
-            else:
-                _chunks.append(iterable[_ * chunk_size : (_ + 1) * chunk_size])
-        return _chunks
-
-
-def chunks(dataframe: pd.DataFrame, n: int):
-    """Yield successive n-sized chunks from dataframe."""
-    for i in range(0, len(dataframe), n):
-        yield dataframe.iloc[i : i + n]