stageflow-framework 0.1.0__tar.gz

stageflow_framework-0.1.0/PKG-INFO

@@ -0,0 +1,167 @@
+Metadata-Version: 2.4
+Name: stageflow-framework
+Version: 0.1.0
+Summary: StageFlow: Pipeline framework for stages
+Author: lethargy
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+Requires-Dist: pyyaml
+Requires-Dist: jsonschema>=4.0.0
+
+# StageFlow
+
+StageFlow is a lightweight framework for describing and running JSON pipelines with nodes, stages, and schema validation.
+
+## Features
+- Node types: `stage`, `condition`, `parallel`, `map`, `subpipeline`, `terminal`.
+- Custom stages (async `run`) with documented arguments/config/outputs.
+- Input waiting (`wait_input`), event history, session snapshots.
+- Built-in stages for dict/list/string/logic utilities.
+- Pipeline JSON Schema (stage enum injected on the fly) and HTML doc generator.
+
+## Installation
+```bash
+python -m venv .venv
+source .venv/bin/activate  # Windows: .venv\Scripts\activate
+pip install stageflow-framework
+# or for local dev:
+# pip install -e .
+```
+Requires Python 3.10+.
+
+## Quickstart
+1) Register your stage:
+```python
+from stageflow.core.stage import BaseStage, register_stage
+
+@register_stage("HelloStage")
+class HelloStage(BaseStage):
+    """
+    description: "Custom stage example"
+    arguments:
+      name: string
+    outputs:
+      greeting: string
+    """
+    async def run(self):
+        name = self.get_arguments().get("name", "world")
+        self.set_outputs({"greeting": f"Hello, {name}!"})
+```
+
+2) Describe a pipeline in JSON:
+```python
+pipeline_dict = {
+    "entry": "start",
+    "nodes": [
+        {
+            "id": "start",
+            "type": "stage",
+            "stage": "HelloStage",
+            "arguments": {"name": "user_name"},
+            "outputs": {"greeting": "greeting"},
+            "next": "finish",
+        },
+        {
+            "id": "finish",
+            "type": "terminal",
+            "result": {"status": "ok"},
+            "artifacts": ["greeting"],
+        },
+    ],
+}
+```
+
+3) Run:
+```python
+import asyncio
+from stageflow.core.pipeline import Pipeline
+from stageflow.core.session import Session
+from stageflow.core.context import Context
+
+async def main():
+    pipe = Pipeline.from_dict(pipeline_dict)
+    session = Session(id="demo", pipeline=pipe, context=Context(payload={"user_name": "Alice"}))
+    result = await session.run()
+    print(result.result)        # {'status': 'ok'}
+    print(result.artifacts)     # {'greeting': 'Hello, Alice!'}
+
+asyncio.run(main())
+```
+
+## Docs and schema generation
+- Programmatic:
+```python
+from stageflow.docs.html import generate_docs_assets
+
+html_page, pipeline_schema, stages_json = generate_docs_assets()
+# html_page — ready-to-serve HTML
+# pipeline_schema — dict with JSON Schema (stage enum injected)
+# stages_json — raw JSON with stage specs
+```
+- CLI (prints JSON payload with `html`, `pipeline_schema`, `stages_json` to stdout):
+```bash
+python scripts/generate_docs_html.py > docs_assets.json
+```
+If needed, persist the `html` from the JSON payload yourself.
+
+## Tests
+```bash
+python -m unittest
+```
+
+## Node types overview
+- `stage`: Executes a registered Stage. Fields: `stage` (name), `config`/`arguments`/`outputs` mappings, `next`, optional `fallback`.
+- `condition`: List of `{if, then}` branches (JsonLogic conditions), optional `else`.
+- `parallel`: Runs several children; `policy` = `all|any`, `cancel_on_error` flag, optional `next`.
+- `map`: Iterates over items path, runs body node for each item (sequential/parallel), collects mapped outputs.
+- `subpipeline`: Invokes nested pipeline by id, with `inputs` mapping, `artifact_outputs`, optional `result_output`, `next`.
+- `terminal`: Ends execution, returns `result` and selects `artifacts`.
+
+### Node types in detail
+- **Stage node**
+  - Required: `id`, `type="stage"`, `stage` (registered stage name).
+  - Data flow: `arguments` map context paths -> stage inputs; `outputs` map stage return keys -> context paths.
+  - Control flow: `next` points to following node; optional `fallback` node is used on stage exceptions.
+  - Stage class declares `skipable`, `allowed_events`, `allowed_inputs`, `timeout`, `retries`.
+- **Condition node**
+  - Required: `id`, `type="condition"`, non-empty `conditions` list.
+  - Each condition: `{ "if": <JsonLogic>, "then": "<node id>" }`; first matched branch wins.
+  - Optional `else` target if no branch matches.
+- **Parallel node**
+  - Required: `id`, `type="parallel"`, non-empty `children` (node ids).
+  - `policy`: `all` (default) waits all children; `any` completes on first success.
+  - `cancel_on_error`: whether to cancel siblings on failure.
+  - `next`: node id to continue after parallel finishes.
+- **Map node**
+  - Required: `id`, `type="map"`, `items` (context path with iterable), `body` (node id).
+  - `mode`: `sequential` (default) or `parallel` for body executions.
+  - `output_map`: map of body outputs -> target context paths for aggregated results.
+  - `item_path`/`index_path`: where to inject current item/index into context for body runs.
+  - `cancel_on_error`: abort remaining iterations on error.
+  - `next`: node id after all iterations finish.
+- **Subpipeline node**
+  - Required: `id`, `type="subpipeline"`, `subpipeline_id` (key in `subpipelines` dict of pipeline).
+  - `inputs`: context mapping passed into subpipeline root context.
+  - `artifact_outputs`: map subpipeline artifacts -> parent context paths.
+  - `result_output`: path in parent context to store subpipeline `result`.
+  - `next`: next node in parent pipeline.
+- **Terminal node**
+  - Required: `id`, `type="terminal"`.
+  - `result`: arbitrary object returned as final result.
+  - `artifacts`: list of context paths that will be returned as artifacts map.
+
+## Built-in stages
+- Vars: `SetValueStage`, `CopyValueStage`, `IncrementStage`, `MergeDictStage`.
+- Dicts: `PickKeysStage`, `DropKeysStage`.
+- Lists: `AppendListStage`, `ExtendListStage`, `FilterListStage`, `UniqueListStage`, `PopListStage`.
+- Strings: `ConcatStage`, `TemplateStage`.
+- Logic: `AssertStage`, `FailStage`, `LogStage`, `SleepStage`.
+
+Each stage docstring describes arguments/config/outputs; they are also available via `generate_docs_assets()` / `stages_json`.
+
+## Useful refs
+- `stageflow/core` — core (pipeline, session, nodes, context).
+- `stageflow/builtins` — built-in stages.
+- `stageflow/docs` — schema/HTML generation helpers.
+- `scripts/` — helpers (docs generator wrapper).
+- `tests/` — unit tests.

stageflow_framework-0.1.0/README.md

@@ -0,0 +1,157 @@
+# StageFlow
+
+StageFlow is a lightweight framework for describing and running JSON pipelines with nodes, stages, and schema validation.
+
+## Features
+- Node types: `stage`, `condition`, `parallel`, `map`, `subpipeline`, `terminal`.
+- Custom stages (async `run`) with documented arguments/config/outputs.
+- Input waiting (`wait_input`), event history, session snapshots.
+- Built-in stages for dict/list/string/logic utilities.
+- Pipeline JSON Schema (stage enum injected on the fly) and HTML doc generator.
+
+## Installation
+```bash
+python -m venv .venv
+source .venv/bin/activate  # Windows: .venv\Scripts\activate
+pip install stageflow-framework
+# or for local dev:
+# pip install -e .
+```
+Requires Python 3.10+.
+
+## Quickstart
+1) Register your stage:
+```python
+from stageflow.core.stage import BaseStage, register_stage
+
+@register_stage("HelloStage")
+class HelloStage(BaseStage):
+    """
+    description: "Custom stage example"
+    arguments:
+      name: string
+    outputs:
+      greeting: string
+    """
+    async def run(self):
+        name = self.get_arguments().get("name", "world")
+        self.set_outputs({"greeting": f"Hello, {name}!"})
+```
+
+2) Describe a pipeline in JSON:
+```python
+pipeline_dict = {
+    "entry": "start",
+    "nodes": [
+        {
+            "id": "start",
+            "type": "stage",
+            "stage": "HelloStage",
+            "arguments": {"name": "user_name"},
+            "outputs": {"greeting": "greeting"},
+            "next": "finish",
+        },
+        {
+            "id": "finish",
+            "type": "terminal",
+            "result": {"status": "ok"},
+            "artifacts": ["greeting"],
+        },
+    ],
+}
+```
+
+3) Run:
+```python
+import asyncio
+from stageflow.core.pipeline import Pipeline
+from stageflow.core.session import Session
+from stageflow.core.context import Context
+
+async def main():
+    pipe = Pipeline.from_dict(pipeline_dict)
+    session = Session(id="demo", pipeline=pipe, context=Context(payload={"user_name": "Alice"}))
+    result = await session.run()
+    print(result.result)        # {'status': 'ok'}
+    print(result.artifacts)     # {'greeting': 'Hello, Alice!'}
+
+asyncio.run(main())
+```
+
+## Docs and schema generation
+- Programmatic:
+```python
+from stageflow.docs.html import generate_docs_assets
+
+html_page, pipeline_schema, stages_json = generate_docs_assets()
+# html_page — ready-to-serve HTML
+# pipeline_schema — dict with JSON Schema (stage enum injected)
+# stages_json — raw JSON with stage specs
+```
+- CLI (prints JSON payload with `html`, `pipeline_schema`, `stages_json` to stdout):
+```bash
+python scripts/generate_docs_html.py > docs_assets.json
+```
+If needed, persist the `html` from the JSON payload yourself.
+
+## Tests
+```bash
+python -m unittest
+```
+
+## Node types overview
+- `stage`: Executes a registered Stage. Fields: `stage` (name), `config`/`arguments`/`outputs` mappings, `next`, optional `fallback`.
+- `condition`: List of `{if, then}` branches (JsonLogic conditions), optional `else`.
+- `parallel`: Runs several children; `policy` = `all|any`, `cancel_on_error` flag, optional `next`.
+- `map`: Iterates over items path, runs body node for each item (sequential/parallel), collects mapped outputs.
+- `subpipeline`: Invokes nested pipeline by id, with `inputs` mapping, `artifact_outputs`, optional `result_output`, `next`.
+- `terminal`: Ends execution, returns `result` and selects `artifacts`.
+
+### Node types in detail
+- **Stage node**
+  - Required: `id`, `type="stage"`, `stage` (registered stage name).
+  - Data flow: `arguments` map context paths -> stage inputs; `outputs` map stage return keys -> context paths.
+  - Control flow: `next` points to following node; optional `fallback` node is used on stage exceptions.
+  - Stage class declares `skipable`, `allowed_events`, `allowed_inputs`, `timeout`, `retries`.
+- **Condition node**
+  - Required: `id`, `type="condition"`, non-empty `conditions` list.
+  - Each condition: `{ "if": <JsonLogic>, "then": "<node id>" }`; first matched branch wins.
+  - Optional `else` target if no branch matches.
+- **Parallel node**
+  - Required: `id`, `type="parallel"`, non-empty `children` (node ids).
+  - `policy`: `all` (default) waits all children; `any` completes on first success.
+  - `cancel_on_error`: whether to cancel siblings on failure.
+  - `next`: node id to continue after parallel finishes.
+- **Map node**
+  - Required: `id`, `type="map"`, `items` (context path with iterable), `body` (node id).
+  - `mode`: `sequential` (default) or `parallel` for body executions.
+  - `output_map`: map of body outputs -> target context paths for aggregated results.
+  - `item_path`/`index_path`: where to inject current item/index into context for body runs.
+  - `cancel_on_error`: abort remaining iterations on error.
+  - `next`: node id after all iterations finish.
+- **Subpipeline node**
+  - Required: `id`, `type="subpipeline"`, `subpipeline_id` (key in `subpipelines` dict of pipeline).
+  - `inputs`: context mapping passed into subpipeline root context.
+  - `artifact_outputs`: map subpipeline artifacts -> parent context paths.
+  - `result_output`: path in parent context to store subpipeline `result`.
+  - `next`: next node in parent pipeline.
+- **Terminal node**
+  - Required: `id`, `type="terminal"`.
+  - `result`: arbitrary object returned as final result.
+  - `artifacts`: list of context paths that will be returned as artifacts map.
+
+## Built-in stages
+- Vars: `SetValueStage`, `CopyValueStage`, `IncrementStage`, `MergeDictStage`.
+- Dicts: `PickKeysStage`, `DropKeysStage`.
+- Lists: `AppendListStage`, `ExtendListStage`, `FilterListStage`, `UniqueListStage`, `PopListStage`.
+- Strings: `ConcatStage`, `TemplateStage`.
+- Logic: `AssertStage`, `FailStage`, `LogStage`, `SleepStage`.
+
+Each stage docstring describes arguments/config/outputs; they are also available via `generate_docs_assets()` / `stages_json`.
+
+## Useful refs
+- `stageflow/core` — core (pipeline, session, nodes, context).
+- `stageflow/builtins` — built-in stages.
+- `stageflow/docs` — schema/HTML generation helpers.
+- `scripts/` — helpers (docs generator wrapper).
+- `tests/` — unit tests.

stageflow_framework-0.1.0/pyproject.toml

@@ -0,0 +1,24 @@
+[build-system]
+requires = ["setuptools>=61.0", "wheel"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "stageflow-framework"
+version = "0.1.0"
+description = "StageFlow: Pipeline framework for stages"
+readme = "README.md"
+authors = [
+    { name="lethargy" }
+]
+dependencies = [
+    "pyyaml",
+    "jsonschema>=4.0.0"
+]
+requires-python = ">=3.10"
+
+[tool.setuptools.packages.find]
+where = ["."]
+include = ["stageflow*"]
+
+[tool.setuptools.package-data]
+"stageflow" = ["py.typed", "docs/schemas/pipeline.json"]

stageflow_framework-0.1.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

stageflow_framework-0.1.0/stageflow/__init__.py

@@ -0,0 +1,25 @@
+from .core import (
+    Event, EventSpec, InputSpec,
+    JsonLogic,
+    Node, ConditionNode, Condition, ParallelNode, TerminalNode, StageNode,
+    Session, SessionResult,
+    BaseStage, get_stage, register_stage, get_stages, get_stages_by_category,
+    Context, DotDict,
+    Pipeline,
+)
+
+from .docs import generate_stages_yaml, generate_stages_json
+from . import builtins
+
+
+__all__ = [
+    "Event", "EventSpec", "InputSpec",
+    "JsonLogic",
+    "Node", "ConditionNode", "Condition", "ParallelNode", "TerminalNode", "StageNode",
+    "Session", "SessionResult",
+    "BaseStage", "get_stage", "register_stage", "get_stages", "get_stages_by_category",
+    "Context", "DotDict",
+    "Pipeline",
+    "generate_stages_yaml", "generate_stages_json",
+    "builtins",
+]

stageflow_framework-0.1.0/stageflow/builtins/__init__.py

@@ -0,0 +1,34 @@
+from .vars import (
+    SetValueStage,
+    CopyValueStage,
+    IncrementStage,
+    MergeDictStage,
+)
+from .lists import (
+    AppendListStage,
+    ExtendListStage,
+)
+from .logic import AssertStage, FailStage, LogStage, SleepStage
+from .strings import ConcatStage, TemplateStage
+from .dicts import PickKeysStage, DropKeysStage
+from .lists_extra import FilterListStage, UniqueListStage, PopListStage
+
+__all__ = [
+    "SetValueStage",
+    "CopyValueStage",
+    "IncrementStage",
+    "MergeDictStage",
+    "AppendListStage",
+    "ExtendListStage",
+    "AssertStage",
+    "FailStage",
+    "LogStage",
+    "SleepStage",
+    "ConcatStage",
+    "TemplateStage",
+    "PickKeysStage",
+    "DropKeysStage",
+    "FilterListStage",
+    "UniqueListStage",
+    "PopListStage",
+]

stageflow_framework-0.1.0/stageflow/builtins/dicts.py

@@ -0,0 +1,59 @@
+from stageflow.core.stage import BaseStage, register_stage
+
+
+@register_stage("PickKeysStage")
+class PickKeysStage(BaseStage):
+    """
+    description: "Pick only specified keys from a dict and return new object"
+    arguments:
+      src:
+        type: object
+        description: "Source dict"
+    config:
+      keys:
+        type: list
+        description: "Keys to keep in the result"
+    outputs:
+      result:
+        type: object
+        description: "Dict containing only picked keys"
+    """
+    category = "builtin.dicts"
+
+    async def run(self):
+        args = self.get_arguments()
+        src_val = args.get("src") or {}
+        keys = self.config.get("keys", [])
+        if not isinstance(src_val, dict):
+            raise ValueError("PickKeysStage expects dict in arguments.src")
+        picked = {k: src_val[k] for k in keys if k in src_val}
+        self.set_outputs({"result": picked})
+
+
+@register_stage("DropKeysStage")
+class DropKeysStage(BaseStage):
+    """
+    description: "Remove specified keys from dict and return cleaned object"
+    arguments:
+      src:
+        type: object
+        description: "Source dict"
+    config:
+      keys:
+        type: list
+        description: "Keys to remove from the dict"
+    outputs:
+      result:
+        type: object
+        description: "Dict without removed keys"
+    """
+    category = "builtin.dicts"
+
+    async def run(self):
+        args = self.get_arguments()
+        keys = set(self.config.get("keys", []))
+        src_val = args.get("src") or {}
+        if not isinstance(src_val, dict):
+            raise ValueError("DropKeysStage expects dict in arguments.src")
+        cleaned = {k: v for k, v in src_val.items() if k not in keys}
+        self.set_outputs({"result": cleaned})

stageflow_framework-0.1.0/stageflow/builtins/lists.py

@@ -0,0 +1,71 @@
+from stageflow.core.stage import BaseStage, register_stage
+
+
+@register_stage("AppendListStage")
+class AppendListStage(BaseStage):
+    """
+    description: "Append value from args/config to list (creates list if missing)"
+    arguments:
+      list:
+        type: list
+        description: "List to append to (from context)"
+      value:
+        type: any
+        description: "Value to append (overrides config)"
+    config:
+      value:
+        type: any
+        description: "Fallback value when argument is missing"
+    outputs:
+      list:
+        type: list
+        description: "Resulting list after append"
+    """
+    category = "builtin.lists"
+
+    async def run(self):
+        args = self.get_arguments()
+        value = args.get("value", self.config.get("value"))
+        lst = args.get("list", None)
+        if lst is None:
+            lst = []
+        if not isinstance(lst, list):
+            raise ValueError("AppendListStage expects list")
+        lst.append(value)
+        self.set_outputs({"list": lst})
+
+
+@register_stage("ExtendListStage")
+class ExtendListStage(BaseStage):
+    """
+    description: "Extend list with items from arguments (or default list)"
+    arguments:
+      list:
+        type: list
+        description: "Base list to extend"
+      items:
+        type: list
+        description: "Items to extend the list with"
+    config:
+      default_list:
+        type: list
+        description: "Fallback list when argument is missing"
+    outputs:
+      list:
+        type: list
+        description: "Resulting list after extend"
+    """
+    category = "builtin.lists"
+
+    async def run(self):
+        args = self.get_arguments()
+        lst = args.get("list", self.config.get("default_list", []))
+        if lst is None:
+            lst = []
+        if not isinstance(lst, list):
+            raise ValueError("ExtendListStage expects list")
+        src_val = args.get("items", [])
+        if not isinstance(src_val, (list, tuple)):
+            raise ValueError("Source for extend is not a list/tuple")
+        lst.extend(src_val)
+        self.set_outputs({"list": lst})

stageflow_framework-0.1.0/stageflow/builtins/lists_extra.py

@@ -0,0 +1,110 @@
+from stageflow.core.stage import BaseStage, register_stage
+from stageflow.core.jsonlogic import JsonLogic
+from stageflow.core.context import Context
+
+
+@register_stage("FilterListStage")
+class FilterListStage(BaseStage):
+    """
+    description: "Filter list items by JsonLogic condition using item_path binding"
+    arguments:
+      items:
+        type: list
+        description: "List to filter"
+    config:
+      condition:
+        type: object
+        description: "JsonLogic condition evaluated for each item"
+      item_path:
+        type: string
+        description: "Context key to bind current item during evaluation"
+    outputs:
+      list:
+        type: list
+        description: "Filtered list"
+    """
+    category = "builtin.lists"
+
+    async def run(self):
+        args = self.get_arguments()
+        condition = self.config.get("condition")
+        item_path = self.config.get("item_path", "item")
+        if condition is None:
+            raise ValueError("FilterListStage requires config.condition")
+        items = args.get("items", [])
+        if not isinstance(items, list):
+            raise ValueError("FilterListStage expects list in arguments.items")
+        base_payload = self.session.context.payload
+        result = []
+        for item in items:
+            temp_ctx = Context(payload=dict(base_payload))
+            temp_ctx.set(item_path, item)
+            if JsonLogic(condition).evaluate(temp_ctx):
+                result.append(item)
+        self.set_outputs({"list": result})
+
+
+@register_stage("UniqueListStage")
+class UniqueListStage(BaseStage):
+    """
+    description: "Deduplicate list while preserving original order"
+    arguments:
+      items:
+        type: list
+        description: "List to deduplicate"
+    outputs:
+      list:
+        type: list
+        description: "List with unique items"
+    """
+    category = "builtin.lists"
+
+    async def run(self):
+        args = self.get_arguments()
+        items = args.get("items", [])
+        if not isinstance(items, list):
+            raise ValueError("UniqueListStage expects list in arguments.items")
+        seen = set()
+        out = []
+        for item in items:
+            if item not in seen:
+                seen.add(item)
+                out.append(item)
+        self.set_outputs({"list": out})
+
+
+@register_stage("PopListStage")
+class PopListStage(BaseStage):
+    """
+    description: "Pop element from list (default last) and return list+popped value"
+    arguments:
+      items:
+        type: list
+        description: "List to pop from"
+      index:
+        type: int
+        description: "Index to pop (overrides config)"
+    config:
+      index:
+        type: int
+        description: "Default index to pop, -1 means last element"
+    outputs:
+      list:
+        type: list
+        description: "List after pop"
+      popped:
+        type: any
+        description: "Popped value"
+    """
+    category = "builtin.lists"
+
+    async def run(self):
+        args = self.get_arguments()
+        index = args.get("index", self.config.get("index", -1))
+        lst = args.get("items", [])
+        if not isinstance(lst, list):
+            raise ValueError("PopListStage expects list in arguments.items")
+        if not lst:
+            return
+        value = lst.pop(index)
+        self.set_outputs({"list": lst, "popped": value})