highway-dsl 0.0.1__tar.gz

highway_dsl-0.0.1/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025 Farseed Ashouri
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

highway_dsl-0.0.1/PKG-INFO

@@ -0,0 +1,203 @@
+Metadata-Version: 2.4
+Name: highway_dsl
+Version: 0.0.1
+Summary: A domain specific language (DSL) for defining and managing data processing pipelines.
+Author-email: Farseed Ashouri <farseed.ashouri@gmail.com>
+License: MIT
+Project-URL: Homepage, https://github.com/rodmena-limited/highway_dsl
+Project-URL: Issues, https://github.com/rodmena-limited/highway_dsl/issues
+Classifier: Programming Language :: Python :: 3
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: pydantic>=2.12.3
+Requires-Dist: pyyaml>=6.0
+Provides-Extra: dev
+Requires-Dist: pytest>=7.0.0; extra == "dev"
+Requires-Dist: mypy>=1.0.0; extra == "dev"
+Requires-Dist: types-PyYAML>=6.0.0; extra == "dev"
+Requires-Dist: pytest-cov>=2.12.1; extra == "dev"
+Dynamic: license-file
+
+# Highway DSL
+
+Highway DSL is a Python-based Domain Specific Language (DSL) for defining and managing complex workflows. It allows users to declaratively specify tasks, dependencies, and execution parameters, supporting various control flow mechanisms like conditions, parallel execution, and retries.
+
+## Features
+
+*   **Declarative Workflow Definition:** Define workflows using a clear and concise Python API or through YAML/JSON configurations.
+*   **Pydantic Models:** Leverages Pydantic for robust data validation and serialization/deserialization of workflow definitions.
+*   **Rich Task Types:** Supports various operators including:
+    *   `TaskOperator`: Executes a Python function.
+    *   `ConditionOperator`: Enables conditional branching based on expressions.
+    *   `WaitOperator`: Pauses workflow execution for a specified duration or until a specific datetime.
+    *   `ParallelOperator`: Executes multiple branches of tasks concurrently.
+    *   `ForEachOperator`: Iterates over a collection, executing a chain of tasks for each item.
+*   **Retry and Timeout Policies:** Define retry strategies and timeout limits for individual tasks.
+*   **Serialization/Deserialization:** Seamless conversion of workflow definitions between Python objects, YAML, and JSON formats.
+*   **Workflow Builder:** A fluent API for constructing workflows programmatically.
+
+## Installation
+
+To install Highway DSL, you can use pip:
+
+```bash
+pip install highway-dsl
+```
+
+If you want to install it for development, including testing dependencies:
+
+```bash
+pip install "highway-dsl[dev]"
+```
+
+## Usage
+
+### Defining a Simple Workflow
+
+```python
+from datetime import timedelta
+from workflow_dsl import WorkflowBuilder
+
+def demonstrate_basic_workflow():
+    """Show a simple complete workflow using just the builder"""
+
+    workflow = (
+        WorkflowBuilder("simple_etl")
+        .task("extract", "etl.extract_data", result_key="raw_data")
+        .task(
+            "transform",
+            "etl.transform_data",
+            args=["{{raw_data}}"],
+            result_key="transformed_data",
+        )
+        .retry(max_retries=3, delay=timedelta(seconds=10))
+        .task("load", "etl.load_data", args=["{{transformed_data}}"])
+        .timeout(timeout=timedelta(minutes=30))
+        .wait("wait_next", timedelta(hours=24))
+        .task("cleanup", "etl.cleanup")
+        .build()
+    )
+
+    workflow.set_variables(
+        {"database_url": "postgresql://localhost/mydb", "chunk_size": 1000}
+    )
+
+    return workflow
+
+if __name__ == "__main__":
+    basic_workflow = demonstrate_basic_workflow()
+    print(basic_workflow.to_yaml())
+```
+
+### Defining a Complex Workflow
+
+Refer to `example_usage.py` for a more complex example demonstrating conditional logic, parallel execution, and iteration.
+
+### YAML Configuration
+
+You can also define workflows directly in YAML:
+
+```yaml
+name: simple_etl
+version: 1.0.0
+description: Simple ETL workflow with retry and timeout
+variables:
+  database_url: postgresql://localhost/mydb
+  chunk_size: 1000
+start_task: extract
+tasks:
+  extract:
+    task_id: extract
+    operator_type: task
+    function: etl.extract_data
+    result_key: raw_data
+    dependencies: []
+    metadata: {}
+    
+  transform:
+    task_id: transform
+    operator_type: task
+    function: etl.transform_data
+    args: ["{{raw_data}}"]
+    result_key: transformed_data
+    dependencies: ["extract"]
+    retry_policy:
+      max_retries: 3
+      delay: PT10S
+      backoff_factor: 2.0
+    metadata: {}
+    
+  load:
+    task_id: load
+    operator_type: task
+    function: etl.load_data
+    args: ["{{transformed_data}}"]
+    dependencies: ["transform"]
+    timeout_policy:
+      timeout: PT30M
+      kill_on_timeout: true
+    metadata: {}
+    
+  wait_next:
+    task_id: wait_next
+    operator_type: wait
+    wait_for: "P1D"
+    dependencies: ["load"]
+    metadata: {}
+    
+  cleanup:
+    task_id: cleanup
+    operator_type: task
+    function: etl.cleanup
+    dependencies: ["wait_next"]
+    metadata: {}
+```
+
+To load this YAML:
+
+```python
+from workflow_dsl import Workflow
+
+yaml_content = """
+# ... (yaml content from above)
+"""
+
+workflow = Workflow.from_yaml(yaml_content)
+print(workflow.name)
+```
+
+## Development
+
+### Running Tests
+
+To run the unit tests, navigate to the project root and execute:
+
+```bash
+pytest
+```
+
+### Type Checking
+
+To perform static type checking with MyPy:
+
+```bash
+mypy .
+```
+
+## Project Structure
+
+```
+.highway/
+├── workflow_dsl.py         # Core DSL definitions (Pydantic models)
+├── example_usage.py        # Examples of how to use the DSL
+├── tests/
+│   ├── __init__.py
+│   ├── conftest.py         # Pytest configuration
+│   └── test_workflow_dsl.py # Unit and integration tests
+├── pyproject.toml          # Project metadata and dependencies
+├── README.md               # This file
+└── SUMMARY.md              # Summary of changes and future instructions
+```

highway_dsl-0.0.1/README.md

@@ -0,0 +1,180 @@
+# Highway DSL
+
+Highway DSL is a Python-based Domain Specific Language (DSL) for defining and managing complex workflows. It allows users to declaratively specify tasks, dependencies, and execution parameters, supporting various control flow mechanisms like conditions, parallel execution, and retries.
+
+## Features
+
+*   **Declarative Workflow Definition:** Define workflows using a clear and concise Python API or through YAML/JSON configurations.
+*   **Pydantic Models:** Leverages Pydantic for robust data validation and serialization/deserialization of workflow definitions.
+*   **Rich Task Types:** Supports various operators including:
+    *   `TaskOperator`: Executes a Python function.
+    *   `ConditionOperator`: Enables conditional branching based on expressions.
+    *   `WaitOperator`: Pauses workflow execution for a specified duration or until a specific datetime.
+    *   `ParallelOperator`: Executes multiple branches of tasks concurrently.
+    *   `ForEachOperator`: Iterates over a collection, executing a chain of tasks for each item.
+*   **Retry and Timeout Policies:** Define retry strategies and timeout limits for individual tasks.
+*   **Serialization/Deserialization:** Seamless conversion of workflow definitions between Python objects, YAML, and JSON formats.
+*   **Workflow Builder:** A fluent API for constructing workflows programmatically.
+
+## Installation
+
+To install Highway DSL, you can use pip:
+
+```bash
+pip install highway-dsl
+```
+
+If you want to install it for development, including testing dependencies:
+
+```bash
+pip install "highway-dsl[dev]"
+```
+
+## Usage
+
+### Defining a Simple Workflow
+
+```python
+from datetime import timedelta
+from workflow_dsl import WorkflowBuilder
+
+def demonstrate_basic_workflow():
+    """Show a simple complete workflow using just the builder"""
+
+    workflow = (
+        WorkflowBuilder("simple_etl")
+        .task("extract", "etl.extract_data", result_key="raw_data")
+        .task(
+            "transform",
+            "etl.transform_data",
+            args=["{{raw_data}}"],
+            result_key="transformed_data",
+        )
+        .retry(max_retries=3, delay=timedelta(seconds=10))
+        .task("load", "etl.load_data", args=["{{transformed_data}}"])
+        .timeout(timeout=timedelta(minutes=30))
+        .wait("wait_next", timedelta(hours=24))
+        .task("cleanup", "etl.cleanup")
+        .build()
+    )
+
+    workflow.set_variables(
+        {"database_url": "postgresql://localhost/mydb", "chunk_size": 1000}
+    )
+
+    return workflow
+
+if __name__ == "__main__":
+    basic_workflow = demonstrate_basic_workflow()
+    print(basic_workflow.to_yaml())
+```
+
+### Defining a Complex Workflow
+
+Refer to `example_usage.py` for a more complex example demonstrating conditional logic, parallel execution, and iteration.
+
+### YAML Configuration
+
+You can also define workflows directly in YAML:
+
+```yaml
+name: simple_etl
+version: 1.0.0
+description: Simple ETL workflow with retry and timeout
+variables:
+  database_url: postgresql://localhost/mydb
+  chunk_size: 1000
+start_task: extract
+tasks:
+  extract:
+    task_id: extract
+    operator_type: task
+    function: etl.extract_data
+    result_key: raw_data
+    dependencies: []
+    metadata: {}
+    
+  transform:
+    task_id: transform
+    operator_type: task
+    function: etl.transform_data
+    args: ["{{raw_data}}"]
+    result_key: transformed_data
+    dependencies: ["extract"]
+    retry_policy:
+      max_retries: 3
+      delay: PT10S
+      backoff_factor: 2.0
+    metadata: {}
+    
+  load:
+    task_id: load
+    operator_type: task
+    function: etl.load_data
+    args: ["{{transformed_data}}"]
+    dependencies: ["transform"]
+    timeout_policy:
+      timeout: PT30M
+      kill_on_timeout: true
+    metadata: {}
+    
+  wait_next:
+    task_id: wait_next
+    operator_type: wait
+    wait_for: "P1D"
+    dependencies: ["load"]
+    metadata: {}
+    
+  cleanup:
+    task_id: cleanup
+    operator_type: task
+    function: etl.cleanup
+    dependencies: ["wait_next"]
+    metadata: {}
+```
+
+To load this YAML:
+
+```python
+from workflow_dsl import Workflow
+
+yaml_content = """
+# ... (yaml content from above)
+"""
+
+workflow = Workflow.from_yaml(yaml_content)
+print(workflow.name)
+```
+
+## Development
+
+### Running Tests
+
+To run the unit tests, navigate to the project root and execute:
+
+```bash
+pytest
+```
+
+### Type Checking
+
+To perform static type checking with MyPy:
+
+```bash
+mypy .
+```
+
+## Project Structure
+
+```
+.highway/
+├── workflow_dsl.py         # Core DSL definitions (Pydantic models)
+├── example_usage.py        # Examples of how to use the DSL
+├── tests/
+│   ├── __init__.py
+│   ├── conftest.py         # Pytest configuration
+│   └── test_workflow_dsl.py # Unit and integration tests
+├── pyproject.toml          # Project metadata and dependencies
+├── README.md               # This file
+└── SUMMARY.md              # Summary of changes and future instructions
+```

highway_dsl-0.0.1/highway_dsl/workflow_dsl.py

@@ -0,0 +1,252 @@
+# workflow_dsl.py
+from typing import Any, Dict, List, Optional, Union, Callable, Type
+from enum import Enum
+from datetime import datetime, timedelta
+import yaml
+import json
+from abc import ABC, abstractmethod
+from pydantic import BaseModel, Field, model_validator, ConfigDict
+
+
+class OperatorType(Enum):
+    TASK = "task"
+    CONDITION = "condition"
+    WAIT = "wait"
+    PARALLEL = "parallel"
+    FOREACH = "foreach"
+    SWITCH = "switch"
+    TRY_CATCH = "try_catch"
+
+
+class RetryPolicy(BaseModel):
+    max_retries: int = Field(3, description="Maximum number of retries")
+    delay: timedelta = Field(timedelta(seconds=5), description="Delay between retries")
+    backoff_factor: float = Field(2.0, description="Factor by which to increase delay")
+
+
+class TimeoutPolicy(BaseModel):
+    timeout: timedelta = Field(..., description="Timeout duration")
+    kill_on_timeout: bool = Field(True, description="Whether to kill the task on timeout")
+
+
+class BaseOperator(BaseModel, ABC):
+    task_id: str
+    operator_type: OperatorType
+    dependencies: List[str] = Field(default_factory=list)
+    retry_policy: Optional[RetryPolicy] = None
+    timeout_policy: Optional[TimeoutPolicy] = None
+    metadata: Dict[str, Any] = Field(default_factory=dict)
+
+    model_config = ConfigDict(use_enum_values=True, arbitrary_types_allowed=True)
+
+
+class TaskOperator(BaseOperator):
+    function: str
+    args: List[Any] = Field(default_factory=list)
+    kwargs: Dict[str, Any] = Field(default_factory=dict)
+    result_key: Optional[str] = None
+    operator_type: OperatorType = Field(OperatorType.TASK, frozen=True)
+
+
+class ConditionOperator(BaseOperator):
+    condition: str
+    if_true: str
+    if_false: str
+    operator_type: OperatorType = Field(OperatorType.CONDITION, frozen=True)
+
+
+class WaitOperator(BaseOperator):
+    wait_for: Union[timedelta, datetime, str]
+    operator_type: OperatorType = Field(OperatorType.WAIT, frozen=True)
+
+    @model_validator(mode='before')
+    @classmethod
+    def parse_wait_for(cls, data: Any) -> Any:
+        if isinstance(data, dict) and "wait_for" in data:
+            wait_for = data["wait_for"]
+            if isinstance(wait_for, str):
+                if wait_for.startswith("duration:"):
+                    data["wait_for"] = timedelta(seconds=float(wait_for.split(":")[1]))
+                elif wait_for.startswith("datetime:"):
+                    data["wait_for"] = datetime.fromisoformat(wait_for.split(":", 1)[1])
+        return data
+
+    def model_dump(self, **kwargs) -> Dict[str, Any]:
+        data = super().model_dump(**kwargs)
+        wait_for = data["wait_for"]
+        if isinstance(wait_for, timedelta):
+            data["wait_for"] = f"duration:{wait_for.total_seconds()}"
+        elif isinstance(wait_for, datetime):
+            data["wait_for"] = f"datetime:{wait_for.isoformat()}"
+        return data
+
+
+class ParallelOperator(BaseOperator):
+    branches: Dict[str, List[str]] = Field(default_factory=dict)
+    operator_type: OperatorType = Field(OperatorType.PARALLEL, frozen=True)
+
+
+class ForEachOperator(BaseOperator):
+    items: str
+    task_chain: List[str] = Field(default_factory=list)
+    operator_type: OperatorType = Field(OperatorType.FOREACH, frozen=True)
+
+
+class Workflow(BaseModel):
+    name: str
+    version: str = "1.0.0"
+    description: str = ""
+    tasks: Dict[str, Union[
+        TaskOperator,
+        ConditionOperator,
+        WaitOperator,
+        ParallelOperator,
+        ForEachOperator,
+    ]] = Field(default_factory=dict)
+    variables: Dict[str, Any] = Field(default_factory=dict)
+    start_task: Optional[str] = None
+
+    @model_validator(mode='before')
+    @classmethod
+    def validate_tasks(cls, data: Any) -> Any:
+        if isinstance(data, dict) and "tasks" in data:
+            validated_tasks = {}
+            operator_classes: Dict[str, Type[BaseOperator]] = {
+                OperatorType.TASK.value: TaskOperator,
+                OperatorType.CONDITION.value: ConditionOperator,
+                OperatorType.WAIT.value: WaitOperator,
+                OperatorType.PARALLEL.value: ParallelOperator,
+                OperatorType.FOREACH.value: ForEachOperator,
+            }
+            for task_id, task_data in data["tasks"].items():
+                operator_type = task_data.get("operator_type")
+                if operator_type and operator_type in operator_classes:
+                    operator_class = operator_classes[operator_type]
+                    validated_tasks[task_id] = operator_class.model_validate(task_data)
+                else:
+                    raise ValueError(f"Unknown operator type: {operator_type}")
+            data["tasks"] = validated_tasks
+        return data
+
+    def add_task(self, task: Union[
+        TaskOperator,
+        ConditionOperator,
+        WaitOperator,
+        ParallelOperator,
+        ForEachOperator,
+    ]) -> "Workflow":
+        self.tasks[task.task_id] = task
+        return self
+
+    def set_variables(self, variables: Dict[str, Any]) -> "Workflow":
+        self.variables.update(variables)
+        return self
+
+    def set_start_task(self, task_id: str) -> "Workflow":
+        self.start_task = task_id
+        return self
+
+    def to_yaml(self) -> str:
+        data = self.model_dump(mode='json', by_alias=True, exclude_none=True)
+        return yaml.dump(data, default_flow_style=False)
+
+    def to_json(self) -> str:
+        return self.model_dump_json(indent=2)
+
+    @classmethod
+    def from_yaml(cls, yaml_str: str) -> "Workflow":
+        data = yaml.safe_load(yaml_str)
+        return cls.model_validate(data)
+
+    @classmethod
+    def from_json(cls, json_str: str) -> "Workflow":
+        return cls.model_validate_json(json_str)
+
+
+class WorkflowBuilder:
+    def __init__(self, name: str, existing_workflow: Optional[Workflow] = None):
+        if existing_workflow:
+            self.workflow = existing_workflow
+        else:
+            self.workflow = Workflow(name=name)
+        self._current_task: Optional[str] = None
+
+    def task(self, task_id: str, function: str, **kwargs) -> "WorkflowBuilder":
+        task = TaskOperator(task_id=task_id, function=function, **kwargs)
+        if self._current_task:
+            task.dependencies.append(self._current_task)
+        self.workflow.add_task(task)
+        self._current_task = task_id
+        return self
+
+    def condition(
+        self, task_id: str, condition: str, if_true: str, if_false: str, **kwargs
+    ) -> "WorkflowBuilder":
+        task = ConditionOperator(
+            task_id=task_id, condition=condition, if_true=if_true, if_false=if_false, **kwargs
+        )
+        if self._current_task:
+            task.dependencies.append(self._current_task)
+        self.workflow.add_task(task)
+        self._current_task = task_id
+        return self
+
+    def wait(
+        self, task_id: str, wait_for: Union[timedelta, datetime, str], **kwargs
+    ) -> "WorkflowBuilder":
+        task = WaitOperator(task_id=task_id, wait_for=wait_for, **kwargs)
+        if self._current_task:
+            task.dependencies.append(self._current_task)
+        self.workflow.add_task(task)
+        self._current_task = task_id
+        return self
+
+    def parallel(
+        self, task_id: str, branches: Dict[str, List[str]], **kwargs
+    ) -> "WorkflowBuilder":
+        task = ParallelOperator(task_id=task_id, branches=branches, **kwargs)
+        if self._current_task:
+            task.dependencies.append(self._current_task)
+        self.workflow.add_task(task)
+        self._current_task = task_id
+        return self
+
+    def foreach(
+        self, task_id: str, items: str, task_chain: List[str], **kwargs
+    ) -> "WorkflowBuilder":
+        task = ForEachOperator(task_id=task_id, items=items, task_chain=task_chain, **kwargs)
+        if self._current_task:
+            task.dependencies.append(self._current_task)
+        self.workflow.add_task(task)
+        self._current_task = task_id
+        return self
+
+    def retry(
+        self,
+        max_retries: int = 3,
+        delay: timedelta = timedelta(seconds=5),
+        backoff_factor: float = 2.0,
+    ) -> "WorkflowBuilder":
+        if self._current_task and isinstance(
+            self.workflow.tasks[self._current_task], TaskOperator
+        ):
+            self.workflow.tasks[self._current_task].retry_policy = RetryPolicy(
+                max_retries=max_retries, delay=delay, backoff_factor=backoff_factor
+            )
+        return self
+
+    def timeout(
+        self, timeout: timedelta, kill_on_timeout: bool = True
+    ) -> "WorkflowBuilder":
+        if self._current_task and isinstance(
+            self.workflow.tasks[self._current_task], TaskOperator
+        ):
+            self.workflow.tasks[self._current_task].timeout_policy = TimeoutPolicy(
+                timeout=timeout, kill_on_timeout=kill_on_timeout
+            )
+        return self
+
+    def build(self) -> Workflow:
+        if not self.workflow.start_task and self.workflow.tasks:
+            self.workflow.start_task = next(iter(self.workflow.tasks.keys()))
+        return self.workflow

highway_dsl-0.0.1/highway_dsl.egg-info/PKG-INFO

@@ -0,0 +1,203 @@
+Metadata-Version: 2.4
+Name: highway_dsl
+Version: 0.0.1
+Summary: A domain specific language (DSL) for defining and managing data processing pipelines.
+Author-email: Farseed Ashouri <farseed.ashouri@gmail.com>
+License: MIT
+Project-URL: Homepage, https://github.com/rodmena-limited/highway_dsl
+Project-URL: Issues, https://github.com/rodmena-limited/highway_dsl/issues
+Classifier: Programming Language :: Python :: 3
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: pydantic>=2.12.3
+Requires-Dist: pyyaml>=6.0
+Provides-Extra: dev
+Requires-Dist: pytest>=7.0.0; extra == "dev"
+Requires-Dist: mypy>=1.0.0; extra == "dev"
+Requires-Dist: types-PyYAML>=6.0.0; extra == "dev"
+Requires-Dist: pytest-cov>=2.12.1; extra == "dev"
+Dynamic: license-file
+
+# Highway DSL
+
+Highway DSL is a Python-based Domain Specific Language (DSL) for defining and managing complex workflows. It allows users to declaratively specify tasks, dependencies, and execution parameters, supporting various control flow mechanisms like conditions, parallel execution, and retries.
+
+## Features
+
+*   **Declarative Workflow Definition:** Define workflows using a clear and concise Python API or through YAML/JSON configurations.
+*   **Pydantic Models:** Leverages Pydantic for robust data validation and serialization/deserialization of workflow definitions.
+*   **Rich Task Types:** Supports various operators including:
+    *   `TaskOperator`: Executes a Python function.
+    *   `ConditionOperator`: Enables conditional branching based on expressions.
+    *   `WaitOperator`: Pauses workflow execution for a specified duration or until a specific datetime.
+    *   `ParallelOperator`: Executes multiple branches of tasks concurrently.
+    *   `ForEachOperator`: Iterates over a collection, executing a chain of tasks for each item.
+*   **Retry and Timeout Policies:** Define retry strategies and timeout limits for individual tasks.
+*   **Serialization/Deserialization:** Seamless conversion of workflow definitions between Python objects, YAML, and JSON formats.
+*   **Workflow Builder:** A fluent API for constructing workflows programmatically.
+
+## Installation
+
+To install Highway DSL, you can use pip:
+
+```bash
+pip install highway-dsl
+```
+
+If you want to install it for development, including testing dependencies:
+
+```bash
+pip install "highway-dsl[dev]"
+```
+
+## Usage
+
+### Defining a Simple Workflow
+
+```python
+from datetime import timedelta
+from workflow_dsl import WorkflowBuilder
+
+def demonstrate_basic_workflow():
+    """Show a simple complete workflow using just the builder"""
+
+    workflow = (
+        WorkflowBuilder("simple_etl")
+        .task("extract", "etl.extract_data", result_key="raw_data")
+        .task(
+            "transform",
+            "etl.transform_data",
+            args=["{{raw_data}}"],
+            result_key="transformed_data",
+        )
+        .retry(max_retries=3, delay=timedelta(seconds=10))
+        .task("load", "etl.load_data", args=["{{transformed_data}}"])
+        .timeout(timeout=timedelta(minutes=30))
+        .wait("wait_next", timedelta(hours=24))
+        .task("cleanup", "etl.cleanup")
+        .build()
+    )
+
+    workflow.set_variables(
+        {"database_url": "postgresql://localhost/mydb", "chunk_size": 1000}
+    )
+
+    return workflow
+
+if __name__ == "__main__":
+    basic_workflow = demonstrate_basic_workflow()
+    print(basic_workflow.to_yaml())
+```
+
+### Defining a Complex Workflow
+
+Refer to `example_usage.py` for a more complex example demonstrating conditional logic, parallel execution, and iteration.
+
+### YAML Configuration
+
+You can also define workflows directly in YAML:
+
+```yaml
+name: simple_etl
+version: 1.0.0
+description: Simple ETL workflow with retry and timeout
+variables:
+  database_url: postgresql://localhost/mydb
+  chunk_size: 1000
+start_task: extract
+tasks:
+  extract:
+    task_id: extract
+    operator_type: task
+    function: etl.extract_data
+    result_key: raw_data
+    dependencies: []
+    metadata: {}
+    
+  transform:
+    task_id: transform
+    operator_type: task
+    function: etl.transform_data
+    args: ["{{raw_data}}"]
+    result_key: transformed_data
+    dependencies: ["extract"]
+    retry_policy:
+      max_retries: 3
+      delay: PT10S
+      backoff_factor: 2.0
+    metadata: {}
+    
+  load:
+    task_id: load
+    operator_type: task
+    function: etl.load_data
+    args: ["{{transformed_data}}"]
+    dependencies: ["transform"]
+    timeout_policy:
+      timeout: PT30M
+      kill_on_timeout: true
+    metadata: {}
+    
+  wait_next:
+    task_id: wait_next
+    operator_type: wait
+    wait_for: "P1D"
+    dependencies: ["load"]
+    metadata: {}
+    
+  cleanup:
+    task_id: cleanup
+    operator_type: task
+    function: etl.cleanup
+    dependencies: ["wait_next"]
+    metadata: {}
+```
+
+To load this YAML:
+
+```python
+from workflow_dsl import Workflow
+
+yaml_content = """
+# ... (yaml content from above)
+"""
+
+workflow = Workflow.from_yaml(yaml_content)
+print(workflow.name)
+```
+
+## Development
+
+### Running Tests
+
+To run the unit tests, navigate to the project root and execute:
+
+```bash
+pytest
+```
+
+### Type Checking
+
+To perform static type checking with MyPy:
+
+```bash
+mypy .
+```
+
+## Project Structure
+
+```
+.highway/
+├── workflow_dsl.py         # Core DSL definitions (Pydantic models)
+├── example_usage.py        # Examples of how to use the DSL
+├── tests/
+│   ├── __init__.py
+│   ├── conftest.py         # Pytest configuration
+│   └── test_workflow_dsl.py # Unit and integration tests
+├── pyproject.toml          # Project metadata and dependencies
+├── README.md               # This file
+└── SUMMARY.md              # Summary of changes and future instructions
+```

highway_dsl-0.0.1/highway_dsl.egg-info/SOURCES.txt

@@ -0,0 +1,11 @@
+LICENSE
+README.md
+pyproject.toml
+highway_dsl/__init__.py
+highway_dsl/workflow_dsl.py
+highway_dsl.egg-info/PKG-INFO
+highway_dsl.egg-info/SOURCES.txt
+highway_dsl.egg-info/dependency_links.txt
+highway_dsl.egg-info/requires.txt
+highway_dsl.egg-info/top_level.txt
+tests/test_workflow_dsl.py

highway_dsl-0.0.1/highway_dsl.egg-info/dependency_links.txt

@@ -0,0 +1 @@
+

highway_dsl-0.0.1/highway_dsl.egg-info/requires.txt

@@ -0,0 +1,8 @@
+pydantic>=2.12.3
+pyyaml>=6.0
+
+[dev]
+pytest>=7.0.0
+mypy>=1.0.0
+types-PyYAML>=6.0.0
+pytest-cov>=2.12.1

highway_dsl-0.0.1/highway_dsl.egg-info/top_level.txt

@@ -0,0 +1 @@
+highway_dsl

highway_dsl-0.0.1/pyproject.toml

@@ -0,0 +1,37 @@
+[build-system]
+requires = ["setuptools>=61.0"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "highway_dsl"
+version = "0.0.1"
+authors = [
+  { name = "Farseed Ashouri", email = "farseed.ashouri@gmail.com" },
+]
+description = "A domain specific language (DSL) for defining and managing data processing pipelines."
+readme = "README.md"
+requires-python = ">=3.9"
+license = { text = "MIT" }
+classifiers = [
+    "Programming Language :: Python :: 3",
+    "License :: OSI Approved :: MIT License",
+    "Operating System :: OS Independent",
+]
+
+# add dependencies here
+dependencies = [
+    "pydantic>=2.12.3",
+    "pyyaml>=6.0"
+    ]
+# dev dependencies
+[project.optional-dependencies]
+dev = [
+    "pytest>=7.0.0",
+    "mypy>=1.0.0",
+    "types-PyYAML>=6.0.0",
+    "pytest-cov>=2.12.1",
+]
+
+[project.urls]
+Homepage = "https://github.com/rodmena-limited/highway_dsl"
+Issues = "https://github.com/rodmena-limited/highway_dsl/issues"

highway_dsl-0.0.1/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

highway_dsl-0.0.1/tests/test_workflow_dsl.py

@@ -0,0 +1,352 @@
+import pytest
+import json
+from datetime import timedelta, datetime
+from highway_dsl.workflow_dsl import (
+    Workflow,
+    WorkflowBuilder,
+    TaskOperator,
+    ConditionOperator,
+    ParallelOperator,
+    WaitOperator,
+    ForEachOperator,
+    RetryPolicy,
+    TimeoutPolicy,
+    OperatorType,
+)
+
+def sort_dict_recursively(d):
+    if not isinstance(d, dict):
+        return d
+    return {k: sort_dict_recursively(v) for k, v in sorted(d.items())}
+
+def test_workflow_creation():
+    workflow = Workflow(name="test_workflow", version="1.0.0", description="A test workflow")
+    assert workflow.name == "test_workflow"
+    assert workflow.version == "1.0.0"
+    assert workflow.description == "A test workflow"
+    assert workflow.tasks == {}
+    assert workflow.variables == {}
+    assert workflow.start_task is None
+
+def test_add_task_to_workflow():
+    workflow = Workflow(name="test_workflow")
+    task = TaskOperator(task_id="task1", function="func1")
+    workflow.add_task(task)
+    assert "task1" in workflow.tasks
+    assert workflow.tasks["task1"] == task
+
+def test_set_variables():
+    workflow = Workflow(name="test_workflow")
+    workflow.set_variables({"key1": "value1"})
+    assert workflow.variables == {"key1": "value1"}
+    workflow.set_variables({"key2": "value2"})
+    assert workflow.variables == {"key1": "value1", "key2": "value2"}
+
+def test_set_start_task():
+    workflow = Workflow(name="test_workflow")
+    workflow.set_start_task("task1")
+    assert workflow.start_task == "task1"
+
+def test_retry_policy_model():
+    policy = RetryPolicy(max_retries=5, delay=timedelta(seconds=10), backoff_factor=2.5)
+    assert policy.max_retries == 5
+    assert policy.delay == timedelta(seconds=10)
+    assert policy.backoff_factor == 2.5
+
+def test_timeout_policy_model():
+    policy = TimeoutPolicy(timeout=timedelta(minutes=5), kill_on_timeout=False)
+    assert policy.timeout == timedelta(minutes=5)
+    assert policy.kill_on_timeout is False
+
+def test_task_operator_model():
+    task = TaskOperator(
+        task_id="task1",
+        function="func1",
+        args=["arg1"],
+        kwargs={"kwarg1": "value1"},
+        result_key="res1",
+        dependencies=["dep1"],
+        retry_policy=RetryPolicy(max_retries=1),
+        timeout_policy=TimeoutPolicy(timeout=timedelta(seconds=30)),
+        metadata={"meta1": "data1"},
+    )
+    assert task.task_id == "task1"
+    assert task.operator_type == OperatorType.TASK
+    assert task.function == "func1"
+    assert task.args == ["arg1"]
+    assert task.kwargs == {"kwarg1": "value1"}
+    assert task.result_key == "res1"
+    assert task.dependencies == ["dep1"]
+    assert task.retry_policy.max_retries == 1
+    assert task.timeout_policy.timeout == timedelta(seconds=30)
+    assert task.metadata == {"meta1": "data1"}
+
+def test_condition_operator_model():
+    condition = ConditionOperator(
+        task_id="cond1",
+        condition="x > 5",
+        if_true="task_true",
+        if_false="task_false",
+        dependencies=["prev_task"],
+    )
+    assert condition.task_id == "cond1"
+    assert condition.operator_type == OperatorType.CONDITION
+    assert condition.condition == "x > 5"
+    assert condition.if_true == "task_true"
+    assert condition.if_false == "task_false"
+    assert condition.dependencies == ["prev_task"]
+
+def test_wait_operator_model():
+    wait_duration = WaitOperator(task_id="wait1", wait_for=timedelta(hours=1))
+    assert wait_duration.wait_for == timedelta(hours=1)
+    assert wait_duration.operator_type == OperatorType.WAIT
+
+    now = datetime.now().replace(microsecond=0)
+    wait_datetime = WaitOperator(task_id="wait2", wait_for=now)
+    assert wait_datetime.wait_for == now
+
+    wait_string = WaitOperator(task_id="wait3", wait_for="event_name")
+    assert wait_string.wait_for == "event_name"
+
+def test_parallel_operator_model():
+    parallel = ParallelOperator(
+        task_id="parallel1",
+        branches={"branch_a": ["task_a1", "task_a2"], "branch_b": ["task_b1"]},
+    )
+    assert parallel.task_id == "parallel1"
+    assert parallel.operator_type == OperatorType.PARALLEL
+    assert parallel.branches == {"branch_a": ["task_a1", "task_a2"], "branch_b": ["task_b1"]}
+
+def test_foreach_operator_model():
+    foreach = ForEachOperator(
+        task_id="foreach1", items="data_list", task_chain=["process_item"]
+    )
+    assert foreach.task_id == "foreach1"
+    assert foreach.operator_type == OperatorType.FOREACH
+    assert foreach.items == "data_list"
+    assert foreach.task_chain == ["process_item"]
+
+def test_wait_operator_serialization():
+    # Test with timedelta
+    wait_duration = WaitOperator(task_id="wait1", wait_for=timedelta(hours=1))
+    dump = wait_duration.model_dump()
+    assert dump["wait_for"] == "duration:3600.0"
+
+    # Test with datetime
+    now = datetime.now().replace(microsecond=0)
+    wait_datetime = WaitOperator(task_id="wait2", wait_for=now)
+    dump = wait_datetime.model_dump()
+    assert dump["wait_for"] == f"datetime:{now.isoformat()}"
+
+    # Test with string (no conversion)
+    wait_string = WaitOperator(task_id="wait3", wait_for="event_name")
+    dump = wait_string.model_dump()
+    assert dump["wait_for"] == "event_name"
+
+    # Test parsing of different data types
+    assert WaitOperator.model_validate({"task_id": "t", "wait_for": "duration:60"}).wait_for == timedelta(seconds=60)
+    now_iso = now.isoformat()
+    assert WaitOperator.model_validate({"task_id": "t", "wait_for": f"datetime:{now_iso}"}).wait_for == now
+    assert WaitOperator.model_validate({"task_id": "t", "wait_for": "event"}).wait_for == "event"
+
+def test_workflow_builder_simple_chain():
+    workflow = (
+        WorkflowBuilder("simple_chain")
+        .task("start", "func_start", result_key="start_res")
+        .task("middle", "func_middle", args=["{{start_res}}"])
+        .build()
+    )
+    assert workflow.name == "simple_chain"
+    assert "start" in workflow.tasks
+    assert "middle" in workflow.tasks
+    assert workflow.tasks["middle"].dependencies == ["start"]
+    assert workflow.start_task == "start"
+
+def test_workflow_builder_with_retry_and_timeout():
+    workflow = (
+        WorkflowBuilder("retry_timeout_workflow")
+        .task("step1", "func1")
+        .retry(max_retries=5, delay=timedelta(seconds=15))
+        .timeout(timeout=timedelta(minutes=1))
+        .build()
+    )
+    assert workflow.tasks["step1"].retry_policy.max_retries == 5
+    assert workflow.tasks["step1"].retry_policy.delay == timedelta(seconds=15)
+    assert workflow.tasks["step1"].timeout_policy.timeout == timedelta(minutes=1)
+
+def test_workflow_builder_condition():
+    workflow = (
+        WorkflowBuilder("conditional_workflow")
+        .task("initial", "init_func")
+        .condition("check", "val > 10", "high", "low")
+        .build()
+    )
+    assert "check" in workflow.tasks
+    assert workflow.tasks["check"].dependencies == ["initial"]
+    assert workflow.tasks["check"].if_true == "high"
+
+def test_workflow_builder_parallel():
+    workflow = (
+        WorkflowBuilder("parallel_workflow")
+        .task("init", "init_func")
+        .parallel("parallel_step", {"b1": ["t1"], "b2": ["t2"]})
+        .build()
+    )
+    assert "parallel_step" in workflow.tasks
+    assert workflow.tasks["parallel_step"].dependencies == ["init"]
+
+def test_workflow_builder_foreach():
+    workflow = (
+        WorkflowBuilder("foreach_workflow")
+        .task("fetch_items", "fetch_func")
+        .foreach("loop_items", "items_list", ["process_item"])
+        .build()
+    )
+    assert "loop_items" in workflow.tasks
+    assert workflow.tasks["loop_items"].dependencies == ["fetch_items"]
+
+def test_workflow_yaml_round_trip():
+    original_workflow = (
+        WorkflowBuilder("yaml_test")
+        .task("start", "func_start", result_key="start_res")
+        .retry(max_retries=2, delay=timedelta(seconds=5))
+        .wait("wait_step", timedelta(minutes=1))
+        .task("end", "func_end", args=["{{start_res}}"])
+        .build()
+    )
+    original_workflow.set_variables({"env": "dev"})
+
+    yaml_output = original_workflow.to_yaml()
+    loaded_workflow = Workflow.from_yaml(yaml_output)
+    assert sort_dict_recursively(json.loads(original_workflow.model_dump_json())) == sort_dict_recursively(json.loads(loaded_workflow.model_dump_json()))
+
+def test_workflow_json_round_trip():
+    original_workflow = (
+        WorkflowBuilder("json_test")
+        .task("stepA", "funcA")
+        .timeout(timeout=timedelta(seconds=60), kill_on_timeout=False)
+        .condition("check_val", "val == 'ok'", "success", "fail")
+        .build()
+    )
+    original_workflow.set_variables({"user": "test"})
+
+    json_output = original_workflow.to_json()
+    loaded_workflow = Workflow.from_json(json_output)
+
+    assert sort_dict_recursively(json.loads(original_workflow.model_dump_json())) == sort_dict_recursively(json.loads(loaded_workflow.model_dump_json()))
+
+def test_complex_workflow_creation_and_serialization():
+    # This test re-uses the logic from example_usage.py's create_complex_workflow
+    # to ensure it works with the new Pydantic models and can be serialized/deserialized.
+    workflow = (
+        WorkflowBuilder("data_processing_pipeline")
+        .task("start", "workflows.tasks.initialize", result_key="init_data")
+        .task(
+            "validate",
+            "workflows.tasks.validate_data",
+            args=["{{init_data}}"],
+            result_key="validated_data",
+        )
+        .condition(
+            "check_quality",
+            condition="{{validated_data.quality_score}} > 0.8",
+            if_true="high_quality_processing",
+            if_false="standard_processing",
+        )
+        .build()
+    )
+
+    workflow.add_task(
+        TaskOperator(
+            task_id="high_quality_processing",
+            function="workflows.tasks.advanced_processing",
+            args=["{{validated_data}}"],
+            dependencies=["check_quality"],
+            retry_policy=RetryPolicy(max_retries=5, delay=timedelta(seconds=10)),
+        )
+    )
+
+    workflow.add_task(
+        TaskOperator(
+            task_id="standard_processing",
+            function="workflows.tasks.basic_processing",
+            args=["{{validated_data}}"],
+            dependencies=["check_quality"],
+        )
+    )
+
+    workflow.add_task(
+        ParallelOperator(
+            task_id="parallel_processing",
+            branches={
+                "branch_a": ["transform_a", "enrich_a"],
+                "branch_b": ["transform_b", "enrich_b"],
+            },
+            dependencies=["high_quality_processing", "standard_processing"],
+        )
+    )
+
+    for branch in ["a", "b"]:
+        workflow.add_task(
+            TaskOperator(
+                task_id=f"transform_{branch}",
+                function=f"workflows.tasks.transform_{branch}",
+                dependencies=["parallel_processing"],
+                result_key=f"transformed_{branch}",
+            )
+        )
+
+        workflow.add_task(
+            TaskOperator(
+                task_id=f"enrich_{branch}",
+                function="workflows.tasks.enrich_data",
+                args=[f"{{{{transformed_{branch}}}}}",],
+                dependencies=[f"transform_{branch}"],
+                result_key=f"enriched_{branch}",
+            )
+        )
+
+    builder = WorkflowBuilder(workflow.name, existing_workflow=workflow)
+    builder._current_task = "enrich_b" # Manually set current task for builder continuation
+
+    workflow = (
+        builder.task(
+            "aggregate",
+            "workflows.tasks.aggregate_results",
+            dependencies=[
+                "enrich_a",
+                "enrich_b",
+            ],
+            result_key="final_result",
+        )
+        .wait("wait_notification", timedelta(hours=1))
+        .task("notify", "workflows.tasks.send_notification", args=["{{final_result}}"])
+        .build()
+    )
+
+    workflow.set_variables(
+        {
+            "environment": "production",
+            "batch_size": 1000,
+            "notify_email": "team@company.com",
+        }
+    )
+
+    # Test serialization and deserialization
+    yaml_output = workflow.to_yaml()
+    loaded_workflow_from_yaml = Workflow.from_yaml(yaml_output)
+    assert sort_dict_recursively(json.loads(workflow.model_dump_json())) == sort_dict_recursively(json.loads(loaded_workflow_from_yaml.model_dump_json()))
+
+    json_output = workflow.to_json()
+    loaded_workflow_from_json = Workflow.from_json(json_output)
+    assert sort_dict_recursively(json.loads(workflow.model_dump_json())) == sort_dict_recursively(json.loads(loaded_workflow_from_json.model_dump_json()))
+
+def test_unknown_operator_type_raises_error():
+    yaml_content = """
+    name: test
+    tasks:
+      task1:
+        operator_type: unknown_operator
+    """
+    with pytest.raises(ValueError, match="Unknown operator type: unknown_operator"):
+        Workflow.from_yaml(yaml_content)