marsh-lib 0.3.4__tar.gz → 0.3.6__tar.gz

{marsh_lib-0.3.4 → marsh_lib-0.3.6}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: marsh-lib
-Version: 0.3.4
+Version: 0.3.6
 Summary: Lightweight, extensible Python library for building, managing, and executing command workflows.
 Project-URL: Repository, https://github.com/CedricAnover/marsh
 Author-email: Cedric Anover <cedric.anover@hotmail.com>
@@ -236,18 +236,32 @@ Execute a validated workflow with:
 results = execute_workflow(workflow)
 ```
 
-The current canonical runtime uses a **sequential local execution model**.
+The canonical runtime supports deterministic sequential execution and bounded concurrent execution.
 
 Conceptually:
 
-```mermaid
-flowchart TD
-    A[Workflow] --> B[Validation]
-    B --> C[Execution Plan]
-    C --> D[Sequential Scheduler]
-    D --> E[Local Process Execution]
-    E --> F[Structured Results]
-```
+    Workflow
+        |
+        v
+    Validation / Planning
+        |
+        v
+    Scheduler State Machine
+        |
+        +--> Sequential Scheduler
+        +--> Async Scheduler
+        +--> Thread Scheduler
+        +--> Process Scheduler
+        |
+        v
+    Execution Mechanism
+        |
+        v
+    Structured Results
+
+Concurrent schedulers share the same dependency semantics and bounded ready-queue state machine. The scheduler controls readiness and dispatch; execution mechanisms remain responsible for running individual tasks.
+
+For async execution, use `execute_workflow_async()`. For bounded thread or process execution, pass `ThreadScheduler(max_concurrency=...)` or `ProcessScheduler(max_concurrency=...)` to `execute_workflow()`.
 
 The workflow model is intentionally separated from the runtime so additional scheduling and execution mechanisms can be introduced without creating another workflow engine.
 
@@ -282,10 +296,10 @@ The dependency graph is validated before execution.
 
 For the example above, the dependency relationship is:
 
-```mermaid
-flowchart LR
-    build --> test
-```
+    build
+      |
+      v
+    test
 
 The current canonical scheduler executes tasks sequentially in deterministic topological order.
 
@@ -398,13 +412,19 @@ def add_prefix(stdout, stderr):
 
 The evaluation order is:
 
-```mermaid
-flowchart TD
-    A[Pre-modifiers] --> B[Pre-processors]
-    B --> C[Command runner]
-    C --> D[Post-modifiers]
-    D --> E[Post-processors]
-```
+    Pre-modifiers
+         |
+         v
+    Pre-processors
+         |
+         v
+    Command runner
+         |
+         v
+    Post-modifiers
+         |
+         v
+    Post-processors
 
 This distinction is important:
 

{marsh_lib-0.3.4 → marsh_lib-0.3.6}/README.md

@@ -207,18 +207,32 @@ Execute a validated workflow with:
 results = execute_workflow(workflow)
 ```
 
-The current canonical runtime uses a **sequential local execution model**.
+The canonical runtime supports deterministic sequential execution and bounded concurrent execution.
 
 Conceptually:
 
-```mermaid
-flowchart TD
-    A[Workflow] --> B[Validation]
-    B --> C[Execution Plan]
-    C --> D[Sequential Scheduler]
-    D --> E[Local Process Execution]
-    E --> F[Structured Results]
-```
+    Workflow
+        |
+        v
+    Validation / Planning
+        |
+        v
+    Scheduler State Machine
+        |
+        +--> Sequential Scheduler
+        +--> Async Scheduler
+        +--> Thread Scheduler
+        +--> Process Scheduler
+        |
+        v
+    Execution Mechanism
+        |
+        v
+    Structured Results
+
+Concurrent schedulers share the same dependency semantics and bounded ready-queue state machine. The scheduler controls readiness and dispatch; execution mechanisms remain responsible for running individual tasks.
+
+For async execution, use `execute_workflow_async()`. For bounded thread or process execution, pass `ThreadScheduler(max_concurrency=...)` or `ProcessScheduler(max_concurrency=...)` to `execute_workflow()`.
 
 The workflow model is intentionally separated from the runtime so additional scheduling and execution mechanisms can be introduced without creating another workflow engine.
 
@@ -253,10 +267,10 @@ The dependency graph is validated before execution.
 
 For the example above, the dependency relationship is:
 
-```mermaid
-flowchart LR
-    build --> test
-```
+    build
+      |
+      v
+    test
 
 The current canonical scheduler executes tasks sequentially in deterministic topological order.
 
@@ -369,13 +383,19 @@ def add_prefix(stdout, stderr):
 
 The evaluation order is:
 
-```mermaid
-flowchart TD
-    A[Pre-modifiers] --> B[Pre-processors]
-    B --> C[Command runner]
-    C --> D[Post-modifiers]
-    D --> E[Post-processors]
-```
+    Pre-modifiers
+         |
+         v
+    Pre-processors
+         |
+         v
+    Command runner
+         |
+         v
+    Post-modifiers
+         |
+         v
+    Post-processors
 
 This distinction is important:
 

{marsh_lib-0.3.4 → marsh_lib-0.3.6}/docs/architecture.md

@@ -34,6 +34,59 @@ The existing command and DAG APIs remain useful low-level building blocks and co
 
 This document describes the architecture implemented by the current Alpha release. Future capabilities are described only as extension boundaries and are not presented as implemented features.
 
+### Scheduler state machine (v0.3.5)
+
+The canonical runtime now uses one shared scheduler state model across sequential and bounded-concurrency modes:
+
+```text
+READY -> RUNNING -> SUCCEEDED
+                 -> FAILED
+                 -> CANCELLED
+READY -> BLOCKED
+```
+
+Dependency readiness is defined by `all(dependency == SUCCEEDED)`. Failed, cancelled, or blocked dependencies prevent downstream dispatch. Concurrent modes use a deterministic ready ordering and a hard `max_concurrency` bound.
+
+The scheduler owns readiness and dispatch policy; Machine/Process/Provider components own execution. Process-mode execution crosses the process boundary only with serializable task state, dependency results, and a picklable execution callback.
+
+```text
+Workflow
+  |
+  v
+Validation / graphlib planning
+  |
+  v
+Scheduler state machine
+  |---- READY tasks ----> bounded dispatch ----> execution
+  |                                             |
+  +<------------- Result / state ---------------+
+```
+
+
+
+## Canonical Workflow IR (v0.3.6)
+
+The canonical workflow boundary is versioned independently from the package release:
+
+```text
+Authoring
+   |
+   v
+Configuration
+   |
+   v
+marsh.workflow/v1
+   |
+   +--> deterministic JSON
+   |
+   v
+Reconstruction
+   |
+   v
+Existing Runtime / Scheduler
+```
+
+The IR is provider-independent and does not introduce a second execution model. Workflow and task IDs are semantic identities; executable Python callables are represented by explicit `OperationRef` values rather than arbitrary live object state. Unsupported runtime values fail explicitly at the boundary.
 
 ## Provider boundary (v0.3.4)
 

marsh_lib-0.3.6/docs/releases/v0.3.6.md

@@ -0,0 +1,27 @@
+# v0.3.6 — Workflow IR & Configuration Boundary
+
+## Highlights
+
+- Established the versioned `marsh.workflow/v1` canonical Workflow IR.
+- Added explicit `OperationRef` values for portable top-level Python callables.
+- Added deterministic canonical JSON serialization and explicit rejection of unsupported runtime values.
+- Added provider-neutral Machine, Provider, Scheduler, and Policy configuration boundaries.
+- Preserved legacy authoring normalization when no IR schema envelope is present.
+- Added cross-process reconstruction coverage and configuration boundary tests.
+- Documented the IR contract, security boundary, compatibility behavior, and architecture.
+
+## Compatibility
+
+The existing runtime and scheduler model remains unchanged. The IR is a semantic representation and serialization boundary, not a second execution engine.
+
+Unsupported schema versions, non-portable callables, non-finite numbers, bytes, and arbitrary runtime objects fail explicitly.
+
+## Verification
+
+- CircleCI build: passed
+- CircleCI lint: passed
+- CircleCI Python 3.10 tests: passed
+- CircleCI Python 3.11 tests: passed
+- CircleCI Python 3.12 tests: passed
+- CircleCI integration suite: passed
+- Security review: no secrets, arbitrary `eval`/`exec`, pickle/dill usage, or new dependencies in the v0.3.6 change set.

marsh_lib-0.3.6/docs/v0.3.6-workflow-ir.md

@@ -0,0 +1,59 @@
+# v0.3.6 Workflow IR
+
+Marsh v0.3.6 establishes the versioned `marsh.workflow/v1` canonical Workflow IR.
+
+## Boundary
+
+```text
+Authoring -> Configuration -> Canonical Workflow IR -> deterministic JSON
+                                      |
+                                      v
+                              Reconstruction
+                                      |
+                                      v
+                              Existing Runtime
+```
+
+The package version and IR schema version are intentionally independent. The document envelope is:
+
+```text
+WorkflowDocument
++- schema
+|  +- name = marsh.workflow
+|  +- version = 1
++- workflow
+|  +- id
+|  +- tasks[]
+|  +- inputs
+|  +- outputs
+|  +- metadata
++- extensions
+```
+
+Workflow and task identifiers are semantic identities. They are never derived from object addresses, PIDs, thread IDs, memory addresses, local paths, or callable representations.
+
+## Operations
+
+Importable top-level Python callables are represented explicitly:
+
+```json
+{"kind":"python_callable","module":"package.tasks","qualname":"build"}
+```
+
+Lambdas, local functions, and `__main__` callables are rejected. Reconstruction uses `importlib.import_module()` and qualified attribute traversal; serialized Python source is never evaluated.
+
+Process operations use structured `ProcessSpec` data. The v1 value contract accepts null, booleans, integers, finite numbers, strings, arrays, and string-keyed objects. Bytes and arbitrary runtime objects are rejected rather than silently encoded.
+
+## Configuration
+
+Configuration remains separate from runtime domain objects. v0.3.6 adds provider-neutral Machine, Scheduler, Policy, and Provider configuration families.
+
+Pydantic 2.x was evaluated as a boundary-validation dependency but is not required for this release: the existing dependency-free dataclass boundary is sufficiently small, explicit, and testable, while adding Pydantic would increase the minimal core dependency surface without removing a current semantic responsibility.
+
+## Determinism and compatibility
+
+Canonical JSON uses stable task ordering, recursively normalized mappings, string object keys, compact separators, and `allow_nan=False`.
+
+The existing `workflow_to_*()` APIs now emit/read the canonical v1 envelope, while `workflow_from_dict()` and `workflow_from_json()` continue to accept legacy data-oriented authoring inputs when no schema envelope is present.
+
+Unknown schema versions fail explicitly. Reconstruction does not require the original live Python object graph for supported operations.

{marsh_lib-0.3.4 → marsh_lib-0.3.6}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "marsh-lib"
-version = "0.3.4"
+version = "0.3.6"
 description = "Lightweight, extensible Python library for building, managing, and executing command workflows."
 authors = [{ name = "Cedric Anover", email = "cedric.anover@hotmail.com" }]
 license = { text = "MIT" }

{marsh_lib-0.3.4 → marsh_lib-0.3.6}/src/marsh/core/__init__.py

@@ -13,11 +13,21 @@ from marsh.core.serialization import (
     validate_workflow,
     workflow_from_dict,
     workflow_from_json,
+    workflow_to_document,
     workflow_to_dict,
     workflow_to_json,
 )
+from marsh.core.ir import (
+    IR_VERSION,
+    OperationRef,
+    WorkflowDocument,
+    WorkflowIR,
+    OperationResolutionError,
+    SerializationError,
+    UnsupportedSchemaVersionError,
+)
 
-from marsh.core.runtime import ExecutionPlan, LocalMachine, LocalProcess, SequentialScheduler, execute_workflow, plan_workflow
+from marsh.core.runtime import ExecutionPlan, LocalMachine, LocalProcess, SequentialScheduler, execute_workflow, execute_workflow_async, plan_workflow
 from marsh.core.policies import ExecutionPolicy, FailurePolicy, ResourcePolicy, RetryPolicy, TimeoutPolicy
 from marsh.core.providers import (
     LocalProvider,
@@ -32,3 +42,4 @@ from marsh.core.providers import (
 )
 from marsh.core.cache import Cache, CachePolicy, MemoryCache, cache_key_for_task
 from marsh.core.observability import EventType, Observer, RuntimeEvent, emit_event
+from marsh.core.scheduler import AsyncScheduler, ConcurrentScheduler, ExecutorScheduler, ProcessScheduler, SchedulerState, TaskState, ThreadScheduler

{marsh_lib-0.3.4 → marsh_lib-0.3.6}/src/marsh/core/configuration.py

@@ -1,9 +1,6 @@
-"""Boundary configuration models for the canonical Marsh workflow IR.
+"""Boundary configuration models for the canonical Marsh workflow IR."""
 
-Configuration objects validate and normalize authoring data before runtime
-objects are materialized. They intentionally remain separate from runtime
-domain objects.
-"""
+from __future__ import annotations
 
 from dataclasses import dataclass, field
 from typing import Any, Mapping
@@ -49,7 +46,6 @@ class TaskConfig:
     def from_mapping(cls, value: Mapping[str, Any]) -> "TaskConfig":
         if not isinstance(value, Mapping):
             raise ValueError("task must be a mapping")
-
         allowed = {
             "id", "operation", "machine", "inputs", "outputs",
             "dependencies", "metadata",
@@ -58,20 +54,17 @@ class TaskConfig:
         if unknown:
             names = ", ".join(sorted(unknown))
             raise ValueError(f"task has unknown fields: {names}")
-
         if "id" not in value:
             raise ValueError("task id is required")
         task_id = _identifier(value["id"], "task id")
         if "operation" not in value:
             raise ValueError(f"task {task_id!r} operation is required")
-
         outputs = _sequence(value.get("outputs"), "task outputs")
         dependencies = _sequence(value.get("dependencies"), "task dependencies")
         if not all(isinstance(item, str) and item.strip() for item in outputs):
             raise ValueError("task outputs must contain non-empty strings")
         if not all(isinstance(item, str) and item.strip() for item in dependencies):
             raise ValueError("task dependencies must contain non-empty strings")
-
         return cls(
             id=task_id,
             operation=value["operation"],
@@ -94,6 +87,72 @@ class TaskConfig:
         )
 
 
+@dataclass(frozen=True)
+class MachineConfig:
+    """Provider-neutral machine configuration."""
+
+    name: str
+    provider: str = "local"
+    options: Mapping[str, Any] = field(default_factory=dict)
+
+    @classmethod
+    def from_mapping(cls, value: Mapping[str, Any]) -> "MachineConfig":
+        if not isinstance(value, Mapping):
+            raise ValueError("machine configuration must be a mapping")
+        return cls(
+            name=_identifier(value.get("name"), "machine name"),
+            provider=_identifier(value.get("provider", "local"), "machine provider"),
+            options=_mapping(value.get("options"), "machine options"),
+        )
+
+
+@dataclass(frozen=True)
+class SchedulerConfig:
+    """Provider-independent scheduler selection and concurrency configuration."""
+
+    strategy: str = "sequential"
+    max_concurrency: int = 1
+
+    @classmethod
+    def from_mapping(cls, value: Mapping[str, Any] | None) -> "SchedulerConfig":
+        value = value or {}
+        if not isinstance(value, Mapping):
+            raise ValueError("scheduler configuration must be a mapping")
+        strategy = _identifier(value.get("strategy", "sequential"), "scheduler strategy")
+        max_concurrency = value.get("max_concurrency", 1)
+        if not isinstance(max_concurrency, int) or isinstance(max_concurrency, bool) or max_concurrency < 1:
+            raise ValueError("scheduler max_concurrency must be a positive integer")
+        return cls(strategy=strategy, max_concurrency=max_concurrency)
+
+
+@dataclass(frozen=True)
+class PolicyConfig:
+    """Opaque, validated policy data kept independent of runtime policy classes."""
+
+    values: Mapping[str, Any] = field(default_factory=dict)
+
+    @classmethod
+    def from_mapping(cls, value: Mapping[str, Any] | None) -> "PolicyConfig":
+        return cls(values=_mapping(value, "policy configuration"))
+
+
+@dataclass(frozen=True)
+class ProviderConfig:
+    """Provider-neutral provider selection at the configuration boundary."""
+
+    name: str
+    options: Mapping[str, Any] = field(default_factory=dict)
+
+    @classmethod
+    def from_mapping(cls, value: Mapping[str, Any]) -> "ProviderConfig":
+        if not isinstance(value, Mapping):
+            raise ValueError("provider configuration must be a mapping")
+        return cls(
+            name=_identifier(value.get("name"), "provider name"),
+            options=_mapping(value.get("options"), "provider options"),
+        )
+
+
 @dataclass(frozen=True)
 class WorkflowConfig:
     """Validated, syntax-neutral configuration for a workflow."""
@@ -103,13 +162,19 @@ class WorkflowConfig:
     inputs: Mapping[str, Any] = field(default_factory=dict)
     outputs: Mapping[str, Any] = field(default_factory=dict)
     metadata: Mapping[str, Any] = field(default_factory=dict)
+    machines: tuple[MachineConfig, ...] = ()
+    providers: tuple[ProviderConfig, ...] = ()
+    scheduler: SchedulerConfig = field(default_factory=SchedulerConfig)
+    policy: PolicyConfig = field(default_factory=PolicyConfig)
 
     @classmethod
     def from_mapping(cls, value: Mapping[str, Any]) -> "WorkflowConfig":
         if not isinstance(value, Mapping):
             raise ValueError("workflow must be a mapping")
-
-        allowed = {"id", "tasks", "inputs", "outputs", "metadata"}
+        allowed = {
+            "id", "tasks", "inputs", "outputs", "metadata",
+            "machines", "providers", "scheduler", "policy",
+        }
         unknown = set(value) - allowed
         if unknown:
             names = ", ".join(sorted(unknown))
@@ -117,16 +182,28 @@ class WorkflowConfig:
         if "id" not in value:
             raise ValueError("workflow id is required")
         workflow_id = _identifier(value["id"], "workflow id")
-
-        raw_tasks = _sequence(value.get("tasks"), "workflow tasks")
-        tasks = tuple(TaskConfig.from_mapping(task) for task in raw_tasks)
-
+        tasks = tuple(
+            TaskConfig.from_mapping(task)
+            for task in _sequence(value.get("tasks"), "workflow tasks")
+        )
+        machines = tuple(
+            MachineConfig.from_mapping(item)
+            for item in _sequence(value.get("machines"), "workflow machines")
+        )
+        providers = tuple(
+            ProviderConfig.from_mapping(item)
+            for item in _sequence(value.get("providers"), "workflow providers")
+        )
         return cls(
             id=workflow_id,
             tasks=tasks,
             inputs=_mapping(value.get("inputs"), "workflow inputs"),
             outputs=_mapping(value.get("outputs"), "workflow outputs"),
             metadata=_mapping(value.get("metadata"), "workflow metadata"),
+            machines=machines,
+            providers=providers,
+            scheduler=SchedulerConfig.from_mapping(value.get("scheduler")),
+            policy=PolicyConfig.from_mapping(value.get("policy")),
         )
 
     def to_workflow(self) -> Workflow:
@@ -140,8 +217,7 @@ class WorkflowConfig:
 
 
 def normalize_workflow(value: Workflow | WorkflowConfig | Mapping[str, Any]) -> Workflow:
-    """Normalize Python or mapping authoring into the canonical Workflow IR."""
-
+    """Normalize Python or mapping authoring into the runtime Workflow contract."""
     if isinstance(value, Workflow):
         return value
     if isinstance(value, WorkflowConfig):

marsh_lib-0.3.6/src/marsh/core/ir/__init__.py

@@ -0,0 +1,33 @@
+"""Canonical Workflow IR for Marsh."""
+
+from marsh.core.ir.errors import (
+    OperationResolutionError,
+    SerializationError,
+    UnsupportedSchemaVersionError,
+)
+from marsh.core.ir.models import (
+    IR_VERSION,
+    OperationRef,
+    WorkflowDocument,
+    WorkflowIR,
+)
+from marsh.core.ir.operations import resolve_operation
+from marsh.core.ir.codecs import (
+    canonical_json,
+    document_from_workflow,
+    document_to_workflow,
+)
+
+__all__ = [
+    "IR_VERSION",
+    "OperationRef",
+    "WorkflowDocument",
+    "WorkflowIR",
+    "OperationResolutionError",
+    "SerializationError",
+    "UnsupportedSchemaVersionError",
+    "canonical_json",
+    "document_from_workflow",
+    "document_to_workflow",
+    "resolve_operation",
+]