flowyml 1.1.0__tar.gz → 1.3.0__tar.gz

{flowyml-1.1.0 → flowyml-1.3.0}/LICENSE

@@ -2,7 +2,7 @@ Apache License
 Version 2.0, January 2004
 http://www.apache.org/licenses/
 
-Copyright 2024 Flowy Team
+Copyright 2024 UnicoLab.ai
 
 Licensed under the Apache License, Version 2.0 (the "License");
 you may not use this file except in compliance with the License.

{flowyml-1.1.0 → flowyml-1.3.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: flowyml
-Version: 1.1.0
+Version: 1.3.0
 Summary: Next-Generation ML Pipeline Framework
 License: Apache-2.0
 License-File: LICENSE
@@ -19,16 +19,20 @@ Classifier: Programming Language :: Python :: 3.13
 Classifier: Programming Language :: Python :: 3.14
 Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
 Provides-Extra: all
+Provides-Extra: aws
+Provides-Extra: azure
 Provides-Extra: gcp
 Provides-Extra: pytorch
 Provides-Extra: sklearn
 Provides-Extra: tensorflow
 Provides-Extra: ui
 Requires-Dist: click (>=8.0.0)
+Requires-Dist: cloudpickle (>=2.0.0)
 Requires-Dist: croniter (>=2.0.1,<3.0.0)
 Requires-Dist: fastapi (>=0.122.0,<0.123.0) ; extra == "ui"
 Requires-Dist: google-cloud-aiplatform (>=1.35.0) ; extra == "gcp" or extra == "all"
 Requires-Dist: google-cloud-storage (>=2.10.0) ; extra == "gcp" or extra == "all"
+Requires-Dist: httpx (>=0.24,<0.28)
 Requires-Dist: loguru (>=0.7.3,<0.8.0)
 Requires-Dist: numpy (>=1.20.0)
 Requires-Dist: pandas (>=1.3.0)
@@ -66,21 +70,25 @@ Description-Content-Type: text/markdown
 
 ---
 
-**flowyml** is the comprehensive ML pipeline framework that combines the **simplicity of a Python script** with the **power of an enterprise MLOps platform**.
+**FlowyML** is the comprehensive ML pipeline framework that combines the **simplicity of a Python script** with the **power of an enterprise MLOps platform**.
 
-## 🚀 Why flowyml?
+## 🚀 Why FlowyML?
 
-| Feature | flowyml | Traditional Orchestrators |
+| Feature | FlowyML | Traditional Orchestrators |
 |---------|---------|---------------------------|
 | **Developer Experience** | 🐍 **Native Python** - No DSLs, no YAML hell. | 📜 Complex YAML or rigid DSLs. |
 | **Context Awareness** | 🧠 **Auto-Injection** - Params are just function args. | 🔌 Manual wiring of every parameter. |
 | **Caching** | ⚡ **Multi-Level** - Smart content-hashing & memoization. | 🐢 Basic file-timestamp checking. |
 | **Asset Management** | 📦 **First-Class Assets** - Models & Datasets with lineage. | 📁 Generic file paths only. |
 | **Architecture** | 🏗️ **Modular Stacks** - Local, Cloud, Hybrid. | 🔒 Vendor lock-in or complex setup. |
+| **Deployment** | 🏢 **Local or Centralized** - Run locally or deploy as a company-wide hub. | 🧩 Fragmented or cloud-only. |
+| **Flexibility** | 🔌 **Extensive Plugin Ecosystem** | Fixed integration with specific orchestrators or custom tools to be developed. |
+| **Separation of Concerns** | 🪾 **Steps Grouping, branching and conditions** | Handling only orchestrator logic and task execution oriented. |
+| **Features Rich** | 🌟 **Built-in experiment tracking, model leaderboard, human-in-the-loop, notifications, scheduling** | Very limited or none extra features. |
 
 ## 🚀 Feature Showcase
 
-flowyml is a complete toolkit for building, debugging, and deploying ML applications.
+FlowyML is a complete toolkit for building, debugging, and deploying ML applications.
 
 ### 1. Zero-Boilerplate Orchestration
 Write pipelines as standard Python functions. No YAML, no DSLs.
@@ -110,18 +118,21 @@ def expensive_processing(data):
 ```
 
 ### 3. 🤖 LLM & GenAI Ready
-Trace token usage, latency, and costs automatically.
+Trace token usage, latency, and costs automatically with built-in observability.
 
 ```python
 @step
 @trace_llm(model="gpt-4", tags=["production"])
 def generate_summary(text: str):
-    # flowyml automatically tracks tokens, cost, and latency
+    # flowyml automatically tracks:
+    # - Token usage (prompt/completion)
+    # - Cost estimation
+    # - Latency & Success/Failure rates
     return openai.ChatCompletion.create(...)
 ```
 
-### 4. ⚡ Efficient Step Grouping
-Group related steps to run in the same container - reduce overhead, maintain clarity.
+### 4. ⚡ Efficient Step Grouping & Separation of Concerns
+Group related steps to run in the same container - reduce overhead, maintain clarity, and keep logic separate from configuration.
 
 ```python
 # Run preprocessing steps in same container (shares resources)
@@ -238,12 +249,13 @@ pipeline.run(debug=True)  # Pauses at breakpoint
 ```
 
 ### 13. 📦 First-Class Assets
-Specialized types for ML workflows.
+Assets are not just files; they are first-class citizens with lineage, metadata, and versioning.
 
 ```python
 from flowyml.core import Dataset, Model, Metrics, FeatureSet
 
-dataset = Dataset.create(data=df, name="training_data")
+# Assets track their producer, lineage, and metadata automatically
+dataset = Dataset.create(data=df, name="training_data", metadata={"source": "s3"})
 model = Model.create(artifact=trained_model, score=0.95)
 metrics = Metrics.create(values={"accuracy": 0.95})
 ```
@@ -285,11 +297,80 @@ if drift['drift_detected']:
 - **🔔 Notifications**: Slack/Email alerts.
 - **🛡️ Circuit Breakers**: Stop cascading failures.
 
-### 18. 🔌 Universal Integrations
+### 18. 🏢 Centralized Hub & Docker
+Ready for the enterprise. Run locally per project or deploy as a centralized entity for the company.
+- **Docker Ready**: Backend and Frontend are fully dockerized.
+- **Centralized Hub**: Share pipelines, artifacts, and experiments across the team.
+- **Remote Execution**: Configure local clients to execute on the remote hub.
+
+### 19. 🔌 Universal Integrations
 - **ML Frameworks**: PyTorch, TensorFlow, Keras, Scikit-learn, HuggingFace.
 - **Cloud Providers**: AWS, GCP, Azure (via plugins).
 - **Tools**: MLflow, Weights & Biases, Great Expectations.
 
+### 20. 📊 Automatic Training History Tracking
+FlowyML automatically captures and visualizes your model training progress with zero manual intervention.
+
+```python
+from flowyml.integrations.keras import FlowymlKerasCallback
+
+# Just add the callback - that's it!
+callback = FlowymlKerasCallback(
+    experiment_name="my-experiment",
+    project="my-project"
+)
+
+model.fit(
+    x_train, y_train,
+    validation_data=(x_val, y_val),
+    epochs=50,
+    callbacks=[callback]  # Auto-tracks all metrics!
+)
+```
+
+**What gets captured automatically:**
+- ✅ Loss (train & validation) per epoch
+- ✅ Accuracy (train & validation) per epoch
+- ✅ All custom metrics (F1, precision, recall, etc.)
+- ✅ Model architecture and parameters
+- ✅ Interactive charts in the UI
+
+**View beautiful training graphs in the UI:**
+1. Navigate to your project's Structure tab
+2. Click on any model artifact
+3. See interactive loss/accuracy charts over epochs!
+
+No external tools needed - all visualization built into FlowyML.
+
+### 21. 📂 Project-Based Organization
+Built-in multi-tenancy for managing multiple teams and initiatives.
+
+```python
+from flowyml import Project
+
+project = Project("recommendation_system")
+pipeline = project.create_pipeline("training")
+
+# All runs, artifacts, and metadata are automatically scoped to the project
+runs = project.list_runs()
+stats = project.get_stats()
+```
+
+### 22. 📝 Pipeline Templates
+Stop reinventing the wheel. Use pre-built templates for common ML patterns.
+
+```python
+from flowyml.core.templates import create_from_template
+
+# Create a standard training pipeline in one line
+pipeline = create_from_template(
+    "ml_training",
+    data_loader=my_loader,
+    trainer=my_trainer,
+    evaluator=my_evaluator
+)
+```
+
 ## 📦 Installation
 
 ```bash
@@ -339,6 +420,26 @@ print(f"Run ID: {result.run_id}")
 print(f"Model Score: {result.outputs['model'].score}")
 ```
 
+### 23. 🌐 Pipeline Versioning
+Git-like versioning for pipelines. Track changes, compare, rollback.
+
+```python
+from flowyml import VersionedPipeline
+
+pipeline = VersionedPipeline("training", version="v1.0.0")
+pipeline.add_step(load_data)
+pipeline.save_version()
+
+# ... make changes ...
+pipeline.version = "v1.1.0"
+pipeline.save_version()
+
+# Compare versions to see exactly what changed (steps, code, context)
+diff = pipeline.compare_with("v1.0.0")
+print(diff["modified_steps"])  # ['train_model']
+print(diff["context_changes"]) # {'learning_rate': {'old': 0.01, 'new': 0.001}}
+```
+
 ## 🖥️ The flowyml UI
 
 Visualize your workflows, inspect artifacts, and monitor runs in real-time.

{flowyml-1.1.0 → flowyml-1.3.0}/README.md

@@ -17,21 +17,25 @@
 
 ---
 
-**flowyml** is the comprehensive ML pipeline framework that combines the **simplicity of a Python script** with the **power of an enterprise MLOps platform**.
+**FlowyML** is the comprehensive ML pipeline framework that combines the **simplicity of a Python script** with the **power of an enterprise MLOps platform**.
 
-## 🚀 Why flowyml?
+## 🚀 Why FlowyML?
 
-| Feature | flowyml | Traditional Orchestrators |
+| Feature | FlowyML | Traditional Orchestrators |
 |---------|---------|---------------------------|
 | **Developer Experience** | 🐍 **Native Python** - No DSLs, no YAML hell. | 📜 Complex YAML or rigid DSLs. |
 | **Context Awareness** | 🧠 **Auto-Injection** - Params are just function args. | 🔌 Manual wiring of every parameter. |
 | **Caching** | ⚡ **Multi-Level** - Smart content-hashing & memoization. | 🐢 Basic file-timestamp checking. |
 | **Asset Management** | 📦 **First-Class Assets** - Models & Datasets with lineage. | 📁 Generic file paths only. |
 | **Architecture** | 🏗️ **Modular Stacks** - Local, Cloud, Hybrid. | 🔒 Vendor lock-in or complex setup. |
+| **Deployment** | 🏢 **Local or Centralized** - Run locally or deploy as a company-wide hub. | 🧩 Fragmented or cloud-only. |
+| **Flexibility** | 🔌 **Extensive Plugin Ecosystem** | Fixed integration with specific orchestrators or custom tools to be developed. |
+| **Separation of Concerns** | 🪾 **Steps Grouping, branching and conditions** | Handling only orchestrator logic and task execution oriented. |
+| **Features Rich** | 🌟 **Built-in experiment tracking, model leaderboard, human-in-the-loop, notifications, scheduling** | Very limited or none extra features. |
 
 ## 🚀 Feature Showcase
 
-flowyml is a complete toolkit for building, debugging, and deploying ML applications.
+FlowyML is a complete toolkit for building, debugging, and deploying ML applications.
 
 ### 1. Zero-Boilerplate Orchestration
 Write pipelines as standard Python functions. No YAML, no DSLs.
@@ -61,18 +65,21 @@ def expensive_processing(data):
 ```
 
 ### 3. 🤖 LLM & GenAI Ready
-Trace token usage, latency, and costs automatically.
+Trace token usage, latency, and costs automatically with built-in observability.
 
 ```python
 @step
 @trace_llm(model="gpt-4", tags=["production"])
 def generate_summary(text: str):
-    # flowyml automatically tracks tokens, cost, and latency
+    # flowyml automatically tracks:
+    # - Token usage (prompt/completion)
+    # - Cost estimation
+    # - Latency & Success/Failure rates
     return openai.ChatCompletion.create(...)
 ```
 
-### 4. ⚡ Efficient Step Grouping
-Group related steps to run in the same container - reduce overhead, maintain clarity.
+### 4. ⚡ Efficient Step Grouping & Separation of Concerns
+Group related steps to run in the same container - reduce overhead, maintain clarity, and keep logic separate from configuration.
 
 ```python
 # Run preprocessing steps in same container (shares resources)
@@ -189,12 +196,13 @@ pipeline.run(debug=True)  # Pauses at breakpoint
 ```
 
 ### 13. 📦 First-Class Assets
-Specialized types for ML workflows.
+Assets are not just files; they are first-class citizens with lineage, metadata, and versioning.
 
 ```python
 from flowyml.core import Dataset, Model, Metrics, FeatureSet
 
-dataset = Dataset.create(data=df, name="training_data")
+# Assets track their producer, lineage, and metadata automatically
+dataset = Dataset.create(data=df, name="training_data", metadata={"source": "s3"})
 model = Model.create(artifact=trained_model, score=0.95)
 metrics = Metrics.create(values={"accuracy": 0.95})
 ```
@@ -236,11 +244,80 @@ if drift['drift_detected']:
 - **🔔 Notifications**: Slack/Email alerts.
 - **🛡️ Circuit Breakers**: Stop cascading failures.
 
-### 18. 🔌 Universal Integrations
+### 18. 🏢 Centralized Hub & Docker
+Ready for the enterprise. Run locally per project or deploy as a centralized entity for the company.
+- **Docker Ready**: Backend and Frontend are fully dockerized.
+- **Centralized Hub**: Share pipelines, artifacts, and experiments across the team.
+- **Remote Execution**: Configure local clients to execute on the remote hub.
+
+### 19. 🔌 Universal Integrations
 - **ML Frameworks**: PyTorch, TensorFlow, Keras, Scikit-learn, HuggingFace.
 - **Cloud Providers**: AWS, GCP, Azure (via plugins).
 - **Tools**: MLflow, Weights & Biases, Great Expectations.
 
+### 20. 📊 Automatic Training History Tracking
+FlowyML automatically captures and visualizes your model training progress with zero manual intervention.
+
+```python
+from flowyml.integrations.keras import FlowymlKerasCallback
+
+# Just add the callback - that's it!
+callback = FlowymlKerasCallback(
+    experiment_name="my-experiment",
+    project="my-project"
+)
+
+model.fit(
+    x_train, y_train,
+    validation_data=(x_val, y_val),
+    epochs=50,
+    callbacks=[callback]  # Auto-tracks all metrics!
+)
+```
+
+**What gets captured automatically:**
+- ✅ Loss (train & validation) per epoch
+- ✅ Accuracy (train & validation) per epoch
+- ✅ All custom metrics (F1, precision, recall, etc.)
+- ✅ Model architecture and parameters
+- ✅ Interactive charts in the UI
+
+**View beautiful training graphs in the UI:**
+1. Navigate to your project's Structure tab
+2. Click on any model artifact
+3. See interactive loss/accuracy charts over epochs!
+
+No external tools needed - all visualization built into FlowyML.
+
+### 21. 📂 Project-Based Organization
+Built-in multi-tenancy for managing multiple teams and initiatives.
+
+```python
+from flowyml import Project
+
+project = Project("recommendation_system")
+pipeline = project.create_pipeline("training")
+
+# All runs, artifacts, and metadata are automatically scoped to the project
+runs = project.list_runs()
+stats = project.get_stats()
+```
+
+### 22. 📝 Pipeline Templates
+Stop reinventing the wheel. Use pre-built templates for common ML patterns.
+
+```python
+from flowyml.core.templates import create_from_template
+
+# Create a standard training pipeline in one line
+pipeline = create_from_template(
+    "ml_training",
+    data_loader=my_loader,
+    trainer=my_trainer,
+    evaluator=my_evaluator
+)
+```
+
 ## 📦 Installation
 
 ```bash
@@ -290,6 +367,26 @@ print(f"Run ID: {result.run_id}")
 print(f"Model Score: {result.outputs['model'].score}")
 ```
 
+### 23. 🌐 Pipeline Versioning
+Git-like versioning for pipelines. Track changes, compare, rollback.
+
+```python
+from flowyml import VersionedPipeline
+
+pipeline = VersionedPipeline("training", version="v1.0.0")
+pipeline.add_step(load_data)
+pipeline.save_version()
+
+# ... make changes ...
+pipeline.version = "v1.1.0"
+pipeline.save_version()
+
+# Compare versions to see exactly what changed (steps, code, context)
+diff = pipeline.compare_with("v1.0.0")
+print(diff["modified_steps"])  # ['train_model']
+print(diff["context_changes"]) # {'learning_rate': {'old': 0.01, 'new': 0.001}}
+```
+
 ## 🖥️ The flowyml UI
 
 Visualize your workflows, inspect artifacts, and monitor runs in real-time.

{flowyml-1.1.0 → flowyml-1.3.0}/flowyml/__init__.py

@@ -38,6 +38,7 @@ from flowyml.assets.registry import AssetRegistry
 # Stack imports
 from flowyml.stacks.base import Stack
 from flowyml.stacks.local import LocalStack
+from flowyml.stacks.components import ResourceConfig, DockerConfig
 
 # Tracking imports
 from flowyml.tracking.experiment import Experiment
@@ -140,6 +141,8 @@ __all__ = [
     # Stacks
     "Stack",
     "LocalStack",
+    "ResourceConfig",
+    "DockerConfig",
     # Tracking
     "Experiment",
     "Run",

{flowyml-1.1.0 → flowyml-1.3.0}/flowyml/assets/base.py

@@ -77,6 +77,16 @@ class Asset:
         if parent:
             parent.children.append(self)
 
+    @property
+    def properties(self) -> dict[str, Any]:
+        """Expose mutable properties stored in metadata."""
+        return self.metadata.properties
+
+    @property
+    def tags(self) -> dict[str, str]:
+        """Expose mutable tags stored in metadata."""
+        return self.metadata.tags
+
     @classmethod
     def create(
         cls,

{flowyml-1.1.0 → flowyml-1.3.0}/flowyml/assets/metrics.py

@@ -89,7 +89,10 @@ class Metrics(Asset):
     def create(
         cls,
         name: str | None = None,
+        version: str | None = None,
         parent: Asset | None = None,
+        tags: dict[str, str] | None = None,
+        properties: dict[str, Any] | None = None,
         **metrics,
     ) -> "Metrics":
         """Factory method to create metrics.
@@ -99,6 +102,9 @@ class Metrics(Asset):
         """
         return cls(
             name=name or "metrics",
+            version=version,
             data=metrics,
             parent=parent,
+            tags=tags,
+            properties=properties,
         )

{flowyml-1.1.0 → flowyml-1.3.0}/flowyml/cli/main.py

@@ -48,9 +48,11 @@ def init(name: str, template: str, directory: str) -> None:
 @click.option("--stack", default="local", help="Stack to use for execution")
 @click.option("--context", "-c", multiple=True, help="Context parameters (key=value)")
 @click.option("--debug", is_flag=True, help="Enable debug mode")
-def run(pipeline_name: str, stack: str, context: tuple, debug: bool) -> None:
+@click.option("--retry", type=int, help="Number of retries for the pipeline")
+def run(pipeline_name: str, stack: str, context: tuple, debug: bool, retry: int | None) -> None:
     """Run a pipeline."""
     from flowyml.cli.run import run_pipeline
+    from flowyml.core.retry_policy import OrchestratorRetryPolicy
 
     # Parse context parameters
     ctx_params = {}
@@ -60,8 +62,13 @@ def run(pipeline_name: str, stack: str, context: tuple, debug: bool) -> None:
 
     click.echo(f"Running pipeline '{pipeline_name}' on stack '{stack}'...")
 
+    kwargs = {}
+    if retry:
+        kwargs["retry_policy"] = OrchestratorRetryPolicy(max_attempts=retry)
+        click.echo(f"  Retry policy enabled: max_attempts={retry}")
+
     try:
-        result = run_pipeline(pipeline_name, stack, ctx_params, debug)
+        result = run_pipeline(pipeline_name, stack, ctx_params, debug, **kwargs)
         click.echo("✓ Pipeline completed successfully")
         click.echo(f"  Run ID: {result.get('run_id', 'N/A')}")
         click.echo(f"  Duration: {result.get('duration', 'N/A')}")
@@ -70,6 +77,105 @@ def run(pipeline_name: str, stack: str, context: tuple, debug: bool) -> None:
         raise click.Abort()
 
 
+@cli.group()
+def schedule() -> None:
+    """Schedule management commands."""
+    pass
+
+
+@schedule.command("create")
+@click.argument("pipeline_name")
+@click.argument("schedule_type", type=click.Choice(["cron", "interval", "daily", "hourly"]))
+@click.argument("value")
+@click.option("--stack", default="local", help="Stack to use for execution")
+def create_schedule(pipeline_name: str, schedule_type: str, value: str, stack: str) -> None:
+    """Create a new schedule for a pipeline.
+
+    VALUE format depends on SCHEDULE_TYPE:
+    - cron: "*/5 * * * *"
+    - interval: seconds (e.g. 60)
+    - daily: "HH:MM" (e.g. 14:30)
+    - hourly: minute (e.g. 30)
+    """
+    from flowyml.core.scheduler import PipelineScheduler
+    from flowyml.cli.run import run_pipeline
+
+    # We need a callable for the scheduler.
+    # Since CLI is stateless, we wrap the run_pipeline command.
+    # Note: In a real distributed system, this would submit to a scheduler service.
+    # Here we are just registering it in the local scheduler DB.
+
+    # For now, we'll just use the scheduler API to register the definition
+    scheduler = PipelineScheduler()
+
+    # Define a wrapper that runs the pipeline via CLI logic
+    def job_func():
+        run_pipeline(pipeline_name, stack, {}, False)
+
+    try:
+        if schedule_type == "cron":
+            scheduler.schedule_cron(pipeline_name, job_func, value)
+        elif schedule_type == "interval":
+            scheduler.schedule_interval(pipeline_name, job_func, seconds=int(value))
+        elif schedule_type == "daily":
+            if ":" in value:
+                h, m = map(int, value.split(":"))
+                scheduler.schedule_daily(pipeline_name, job_func, hour=h, minute=m)
+            else:
+                raise ValueError("Daily value must be HH:MM")
+        elif schedule_type == "hourly":
+            scheduler.schedule_hourly(pipeline_name, job_func, minute=int(value))
+
+        click.echo(f"✓ Schedule created for '{pipeline_name}' ({schedule_type}={value})")
+        click.echo("  Note: Ensure the scheduler service is running to execute this schedule.")
+    except Exception as e:
+        click.echo(f"✗ Error creating schedule: {e}", err=True)
+
+
+@schedule.command("list")
+def list_schedules() -> None:
+    """List all active schedules."""
+    from flowyml.core.scheduler import PipelineScheduler
+
+    scheduler = PipelineScheduler()
+    jobs = scheduler.get_jobs()
+
+    if not jobs:
+        click.echo("No active schedules found.")
+        return
+
+    click.echo(f"Found {len(jobs)} schedules:\n")
+    for job in jobs:
+        click.echo(f"  {job.id} - {job.name}")
+        click.echo(f"    Next run: {job.next_run_time}")
+        click.echo()
+
+
+@schedule.command("start")
+def start_scheduler() -> None:
+    """Start the scheduler service (blocking)."""
+    from flowyml.core.scheduler import PipelineScheduler
+    import time
+
+    click.echo("🚀 Starting Scheduler Service...")
+    scheduler = PipelineScheduler()
+
+    try:
+        # In a real app, this would load definitions from DB and register them
+        # For now, it just runs the scheduler loop for existing in-memory jobs
+        # (which might be empty if we restarted).
+        # To make this persistent, we'd need to serialize job definitions to DB.
+        # The current Scheduler implementation supports SQLite persistence for job state,
+        # but we need to re-register jobs on startup.
+
+        click.echo("  Scheduler running. Press Ctrl+C to stop.")
+        while True:
+            scheduler.run_pending()
+            time.sleep(1)
+    except KeyboardInterrupt:
+        click.echo("\n🛑 Scheduler stopped.")
+
+
 @cli.group()
 def ui() -> None:
     """UI server commands."""

{flowyml-1.1.0 → flowyml-1.3.0}/flowyml/cli/run.py

@@ -6,7 +6,13 @@ from pathlib import Path
 from typing import Any
 
 
-def run_pipeline(pipeline_name: str, stack: str, context_params: dict[str, Any], debug: bool) -> dict[str, Any]:
+def run_pipeline(
+    pipeline_name: str,
+    stack: str,
+    context_params: dict[str, Any],
+    debug: bool,
+    **kwargs,
+) -> dict[str, Any]:
     """Run a pipeline by name.
 
     Args:
@@ -14,6 +20,7 @@ def run_pipeline(pipeline_name: str, stack: str, context_params: dict[str, Any],
         stack: Stack to use for execution
         context_params: Context parameters to override
         debug: Enable debug mode
+        **kwargs: Additional arguments passed to pipeline.run
 
     Returns:
         Dictionary with run results
@@ -65,7 +72,7 @@ def run_pipeline(pipeline_name: str, stack: str, context_params: dict[str, Any],
         pipeline.set_stack(stack)
 
     # Run pipeline
-    result = pipeline.run(debug=debug)
+    result = pipeline.run(debug=debug, **kwargs)
 
     return {
         "run_id": result.run_id,

flowyml-1.3.0/flowyml/core/execution_status.py

@@ -0,0 +1,52 @@
+"""Execution status tracking for pipeline runs."""
+
+from enum import Enum
+
+
+class ExecutionStatus(str, Enum):
+    """Status of a pipeline or step execution."""
+
+    # Pre-execution states
+    INITIALIZING = "initializing"
+    PROVISIONING = "provisioning"
+
+    # Active execution states
+    RUNNING = "running"
+
+    # Terminal success states
+    COMPLETED = "completed"
+    CACHED = "cached"
+
+    # Terminal failure states
+    FAILED = "failed"
+    STOPPED = "stopped"
+    CANCELLED = "cancelled"
+
+    # Intermediate states
+    STOPPING = "stopping"
+    CANCELLING = "cancelling"
+
+    @property
+    def is_finished(self) -> bool:
+        """Check if execution is in a terminal state."""
+        return self in {
+            ExecutionStatus.COMPLETED,
+            ExecutionStatus.CACHED,
+            ExecutionStatus.FAILED,
+            ExecutionStatus.STOPPED,
+            ExecutionStatus.CANCELLED,
+        }
+
+    @property
+    def is_successful(self) -> bool:
+        """Check if execution completed successfully."""
+        return self in {ExecutionStatus.COMPLETED, ExecutionStatus.CACHED}
+
+    @property
+    def is_failed(self) -> bool:
+        """Check if execution failed."""
+        return self in {
+            ExecutionStatus.FAILED,
+            ExecutionStatus.STOPPED,
+            ExecutionStatus.CANCELLED,
+        }