pyworkflow-engine 0.2.7__tar.gz → 0.2.8__tar.gz

{pyworkflow_engine-0.2.7 → pyworkflow_engine-0.2.8}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: pyworkflow-engine
-Version: 0.2.7
+Version: 0.2.8
 Summary: A Python implementation of durable, event-sourced workflows inspired by Vercel Workflow
 Author: PyWorkflow Contributors
 License: MIT

pyworkflow_engine-0.2.8/docs/concepts/stream-steps.mdx

@@ -0,0 +1,264 @@
+---
+title: 'Stream Steps'
+description: 'Reactive, long-lived steps that subscribe to signals on a stream and resume on each arrival'
+---
+
+## What is a Stream Step?
+
+A stream step is a long-lived reactive unit that subscribes to a named stream and runs in response to **signals**. Unlike a regular `@step` — which runs once when called from a workflow — a `@stream_step` is materialized as a subscription row when its parent [stream workflow](/concepts/stream-workflows) starts, and is then re-invoked each time a matching signal arrives, until it explicitly terminates.
+
+```python
+from pyworkflow import stream_step
+from pyworkflow.streams import get_current_signal, set_result, terminate
+
+@stream_step(stream="agent_comms", signals=["task.assigned"])
+async def worker():
+    signal = await get_current_signal()
+    if signal is None:
+        # First start — no signal yet, just register and wait.
+        return
+
+    task_id = signal.payload["task_id"]
+    result = await do_work(task_id)
+
+    await set_result({"task_id": task_id, "result": result})
+    await terminate()
+```
+
+## Key Characteristics
+
+<CardGroup cols={2}>
+  <Card title="Signal-driven" icon="bolt">
+    The step body runs once on start and again on each matching signal arrival.
+  </Card>
+  <Card title="Stateful" icon="database">
+    Use `save_checkpoint()` / `get_checkpoint()` to carry state across resumes without rebuilding from scratch.
+  </Card>
+  <Card title="Suspendable" icon="pause">
+    `suspend()` parks the step and bubbles a `SuspensionSignal` up to the parent workflow — perfect for HITL.
+  </Card>
+  <Card title="Schema-aware" icon="shield-check">
+    Bind Pydantic models to signal types and the dispatcher validates payloads before delivery.
+  </Card>
+</CardGroup>
+
+## The Two Code Paths
+
+A stream step has **two** code paths the runtime invokes at different times:
+
+1. **`on_signal` callback** — runs on every matching signal arrival. It receives the raw `Signal` and a `StreamStepContext`. Its job is to decide whether to `resume()`, `cancel()`, `terminate()`, or `suspend()` — typically very lightweight.
+2. **The lifecycle function** (the decorated function itself) — runs on first start and again on each explicit `ctx.resume()`. This is where the actual work happens.
+
+If you don't pass an `on_signal`, the default callback (`_auto_resume_on_signal`) just calls `ctx.resume()` for every matching signal — which is the right thing 90% of the time.
+
+```python
+from pyworkflow import stream_step
+from pyworkflow.streams import get_current_signal, terminate
+
+# Custom on_signal: only resume on the high-priority variant.
+async def gate_priority(signal, ctx):
+    if signal.payload.get("priority") == "high":
+        await ctx.resume()
+    # Otherwise: do nothing — the signal is acked and the step stays waiting.
+
+@stream_step(
+    stream="agent_comms",
+    signals=["task.created"],
+    on_signal=gate_priority,
+)
+async def priority_worker():
+    signal = await get_current_signal()
+    if signal is None:
+        return
+    await handle_high_priority(signal.payload)
+    await terminate()
+```
+
+## Defining Signals
+
+The `signals=` argument accepts either a plain list of names or a dict mapping names to Pydantic schemas.
+
+<Tabs>
+  <Tab title="List of names">
+    ```python
+    @stream_step(
+        stream="agent_comms",
+        signals=["task.created", "task.updated", "task.cancelled"],
+    )
+    async def planner():
+        signal = await get_current_signal()
+        ...
+    ```
+  </Tab>
+  <Tab title="Pydantic schemas">
+    ```python
+    from pydantic import BaseModel
+
+    class TaskAssigned(BaseModel):
+        task_id: str
+        assignee: str
+        priority: int
+
+    @stream_step(
+        stream="agent_comms",
+        signals={"task.assigned": TaskAssigned},
+    )
+    async def worker():
+        signal = await get_current_signal()
+        if signal:
+            # signal.payload is a validated dict matching TaskAssigned
+            await assign(signal.payload["task_id"], signal.payload["assignee"])
+    ```
+  </Tab>
+</Tabs>
+
+<Note>
+  When a schema is bound to a signal type, payloads that fail validation are rejected by the dispatcher and the step is **not** invoked for that signal.
+</Note>
+
+## Lifecycle Primitives
+
+These helpers, imported from `pyworkflow.streams`, are how the lifecycle function communicates with the dispatcher. They all set state that the dispatcher reads **after** the function returns — they don't immediately mutate the subscription row.
+
+| Primitive | Purpose |
+|---|---|
+| `await get_current_signal()` | Returns the `Signal` that triggered this resume (or `None` on first start). |
+| `await set_result(value)` | Attach a result payload to the subscription. Surfaced via `StreamWorkflowResult.get_result(...)`. |
+| `await terminate()` | Mark the step `terminated` — it will not be invoked again. |
+| `await suspend(reason, resume_signals=None)` | Mark the step `suspended` — bubbles a `SuspensionSignal` up to the parent workflow (HITL). |
+| `await save_checkpoint(data)` | Persist a JSON-serializable dict for the next resume. |
+| `await get_checkpoint()` | Load the most recently saved checkpoint dict (or `None`). |
+
+```python
+from pyworkflow.streams import (
+    get_current_signal,
+    get_checkpoint,
+    save_checkpoint,
+    set_result,
+    suspend,
+    terminate,
+)
+
+@stream_step(stream="reviews", signals=["review.submitted", "human.approved"])
+async def review_loop():
+    signal = await get_current_signal()
+    state = (await get_checkpoint()) or {"submissions": 0}
+
+    if signal is None:
+        return  # initial start
+
+    if signal.signal_type == "review.submitted":
+        state["submissions"] += 1
+        await save_checkpoint(state)
+
+        if state["submissions"] >= 3:
+            # Park the step until a human reviews it.
+            await suspend(
+                reason="hitl:needs_review",
+                resume_signals=["human.approved"],
+            )
+            return
+
+    if signal.signal_type == "human.approved":
+        await set_result({"approved": True, "submissions": state["submissions"]})
+        await terminate()
+```
+
+## The `Signal` Object
+
+`get_current_signal()` returns a `Signal` dataclass:
+
+```python
+@dataclass
+class Signal:
+    signal_id: str
+    stream_id: str
+    signal_type: str
+    payload: Any                      # dict (validated against schema if configured)
+    published_at: datetime
+    sequence: int | None              # Per-stream ordering, assigned by storage
+    source_run_id: str | None         # run_id of the workflow that called emit()
+    stream_run_id: str | None         # Scoping key for this stream run
+    metadata: dict[str, Any]
+```
+
+The `sequence` field gives you a strict ordering across all signals on the stream — useful for deduping or detecting gaps. `source_run_id` is set automatically when `emit()` is called from inside a workflow context.
+
+## Emitting Signals from a Step
+
+Stream steps can emit signals back to the same or another stream — this is how multi-actor pipelines fan out work between participants.
+
+```python
+from pyworkflow.streams import emit, get_current_signal, terminate
+
+@stream_step(stream="orders", signals=["order.placed"])
+async def fulfillment():
+    signal = await get_current_signal()
+    if signal is None:
+        return
+
+    order_id = signal.payload["order_id"]
+    await ship(order_id)
+
+    # Notify the billing stream that this order is ready to invoice.
+    await emit("billing", "order.shipped", {"order_id": order_id})
+
+    await terminate()
+```
+
+`emit()` automatically picks up the current `stream_run_id` from context, so signals stay scoped to the same run unless you pass an explicit override.
+
+## The `StreamStepContext` (for `on_signal` callbacks)
+
+When you pass a custom `on_signal=...`, the callback receives `(signal, ctx)` where `ctx` is a `StreamStepContext`:
+
+| Method | Effect |
+|---|---|
+| `await ctx.resume()` | Trigger the lifecycle function with `signal` as the current signal. |
+| `await ctx.terminate()` | Mark the step terminated without invoking the lifecycle function. |
+| `await ctx.suspend(reason, resume_signals=None)` | Park the step; bubbles to the parent workflow as a `SuspensionSignal`. |
+| `await ctx.cancel(reason=None)` | Cancel the step (terminal, with a reason). |
+
+`ctx` also exposes `ctx.status`, `ctx.run_id`, and `ctx.stream_id` for inspection.
+
+```python
+async def cancel_on_kill(signal, ctx):
+    if signal.signal_type == "stream.kill":
+        await ctx.cancel(reason="received kill signal")
+        return
+    await ctx.resume()
+
+@stream_step(
+    stream="agent_comms",
+    signals=["task.created", "stream.kill"],
+    on_signal=cancel_on_kill,
+)
+async def killable_worker():
+    ...
+```
+
+## Terminal States
+
+Every stream step subscription is in one of these states. The aggregate of all step states determines whether the parent stream workflow is `running`, `completed`, or `suspended` — see [Stream Workflows: Aggregate Lifecycle](/concepts/stream-workflows#aggregate-lifecycle).
+
+| State | Meaning |
+|---|---|
+| `waiting` | Subscribed and parked, ready for the next matching signal. |
+| `running` | Lifecycle function is currently executing on a worker. |
+| `suspended` | `suspend()` was called — bubbles `SuspensionSignal` up to the parent. |
+| `terminated` | `terminate()` was called — the step is done and will never run again. |
+| `cancelled` | `cancel()` was called from an `on_signal` callback. |
+
+## Best Practices
+
+- **Keep `on_signal` callbacks lightweight.** They run for every signal, including ones you ignore — do filtering there, do work in the lifecycle function.
+- **Always handle the `signal is None` case.** The first invocation has no triggering signal; that's the registration phase. Use it to seed state, not to do work.
+- **Use `set_result()` instead of `save_checkpoint()` for output the parent needs.** `save_checkpoint` is for *internal* state across resumes; `set_result` is the parent-facing return value.
+- **Call `terminate()` or `suspend()` when you're done.** A step that just `return`s without setting a terminal state stays in `waiting` and keeps the parent stream workflow alive.
+- **Don't rely on globals.** Step lifecycles can run on different workers between resumes — persist anything you need via `save_checkpoint()`.
+
+## Related Concepts
+
+- [Stream Workflows](/concepts/stream-workflows) — the parent runtime that hosts and drives stream steps
+- [Steps](/concepts/steps) — the regular, run-once step model for sequential workflows
+- [Hooks](/concepts/hooks) — the suspension primitive that stream workflows use under the hood

pyworkflow_engine-0.2.8/docs/concepts/stream-workflows.mdx

@@ -0,0 +1,221 @@
+---
+title: 'Stream Workflows'
+description: 'Long-lived, reactive workflows built from stream steps that react to signals on a named channel'
+---
+
+## What is a Stream Workflow?
+
+A stream workflow is a **long-lived, reactive unit of orchestration** built on top of the signal pub/sub system. Where a `@workflow` runs top-to-bottom and completes, a `@stream_workflow` defines a named channel ("stream") that one or more `@stream_step`s subscribe to. Each step reacts to incoming signals, can suspend and be resumed by new signals, and can publish results back.
+
+```python
+from pyworkflow import stream_workflow, stream_step, emit, run_stream_workflow, workflow
+
+@stream_workflow(name="agent_comms")
+async def agent_comms():
+    """Defines the 'agent_comms' channel. Body runs once on start."""
+    await emit("agent_comms", "agent.started", {"ts": "now"})
+
+@stream_step(stream="agent_comms", signals=["task.assigned"])
+async def worker():
+    from pyworkflow.streams import get_current_signal, set_result, terminate
+    signal = await get_current_signal()
+    if signal is None:
+        return  # initial start, no signal yet
+    await set_result({"task_id": signal.payload["task_id"]})
+    await terminate()
+
+@workflow()
+async def run_agent(run_id: str):
+    # Drive the stream workflow from inside a durable @workflow.
+    result = await run_stream_workflow(agent_comms, stream_run_id=run_id)
+    return result.step_results
+```
+
+## Key Characteristics
+
+<CardGroup cols={2}>
+  <Card title="Reactive" icon="bolt">
+    Steps react to signals arriving on the stream instead of running top-to-bottom.
+  </Card>
+  <Card title="Long-lived" icon="infinity">
+    A stream workflow stays alive until every subscribed step reaches a terminal state.
+  </Card>
+  <Card title="Durable Suspension" icon="pause">
+    The parent `@workflow` is released via `hook()` while steps wait for signals — no worker is pinned.
+  </Card>
+  <Card title="Scoped" icon="layer-group">
+    Each `stream_run_id` isolates its own set of subscriptions, signals, and results.
+  </Card>
+</CardGroup>
+
+## When to Use
+
+Use a stream workflow when your process is **event-driven and multi-actor** — e.g. a planner step waiting for task results from N worker steps, a supervisor loop waiting on human-in-the-loop feedback, or long-running agents that react to external signals over hours or days.
+
+Use a regular `@workflow` when your process is a linear sequence of steps that run to completion.
+
+| Feature | `@workflow` | `@stream_workflow` |
+|---|---|---|
+| Execution model | Top-to-bottom | Signal-driven, reactive |
+| Step activation | Explicit `await step()` | Subscription to signal types |
+| Termination | Return from function | All steps reach terminal state |
+| Suspension | `sleep` / `hook` | `hook` + signal arrival |
+| Best for | Sequential business logic | Agents, multi-actor coordination, HITL loops |
+
+## Defining a Stream Workflow
+
+```python
+from pyworkflow import stream_workflow, emit
+
+@stream_workflow(name="order_pipeline")
+async def order_pipeline():
+    """
+    The body runs once at startup, after all @stream_step subscriptions
+    for this stream have been materialized. Use it to emit the initial
+    signal that kicks the pipeline off.
+    """
+    await emit("order_pipeline", "pipeline.started", {})
+```
+
+<Note>
+  The stream name defaults to the function name when `name=` is omitted. Every `@stream_step` with matching `stream="..."` subscribes to this channel.
+</Note>
+
+## Running from a Parent Workflow
+
+Stream workflows are driven from inside a regular `@workflow` via `run_stream_workflow()`. This is what gives them durability: the parent workflow records the stream run, suspends via `hook()` between signals, and the dispatcher resumes the parent once the stream reaches a terminal aggregate state.
+
+```python
+from pyworkflow import workflow, run_stream_workflow
+from pyworkflow.streams import StreamWorkflowResult
+
+@workflow()
+async def agent_run(job_id: str):
+    async def init():
+        # Optional: seed initial state before the stream body runs
+        await seed_job(job_id)
+
+    result: StreamWorkflowResult = await run_stream_workflow(
+        order_pipeline,
+        stream_run_id=f"stream_{job_id}",
+        init=init,
+    )
+
+    # Read step outputs published via set_result()
+    planner_output = result.get_result("planner")
+    return {"job_id": job_id, "planner": planner_output}
+```
+
+### Signature
+
+```python
+await run_stream_workflow(
+    stream_workflow_func,     # the @stream_workflow-decorated function
+    *,
+    stream_run_id: str,       # unique scope key for this run
+    init: Callable | None = None,   # optional async init callback
+    storage = None,           # defaults to context / global config
+) -> StreamWorkflowResult
+```
+
+## Aggregate Lifecycle
+
+A stream run is in one of three aggregate states derived from its subscription rows:
+
+| Aggregate | Condition | Parent workflow behavior |
+|---|---|---|
+| `running` | At least one step is `waiting` or `running` | Parent is suspended via `hook()`, worker released |
+| `completed` | Every step is `terminated` | `run_stream_workflow` returns `StreamWorkflowResult(status="completed")` |
+| `suspended` | At least one step `suspended`, none `waiting`/`running` | Parent re-raises `SuspensionSignal` (bubbles up HITL) |
+
+```
+run_stream_workflow(...)
+        │
+        ▼
+┌────────────────────────┐
+│ Ensure subscriptions   │
+│ exist for every step   │
+└───────┬────────────────┘
+        │
+        ▼
+┌────────────────────────┐
+│ Run init() + body()    │  (first run only)
+└───────┬────────────────┘
+        │
+        ▼
+┌────────────────────────┐
+│ Compute aggregate      │
+└───────┬────────────────┘
+        │
+        ├─ "completed" ──────► Return StreamWorkflowResult
+        │
+        ├─ "suspended" ──────► Raise SuspensionSignal (HITL)
+        │
+        └─ "running"
+                │
+                ▼
+        ┌────────────────────────┐
+        │ await hook(...)        │  Parent worker released
+        │ Record parent link     │
+        └───────┬────────────────┘
+                │  ... signals flow, steps execute ...
+                │
+                ▼
+        ┌──────────────────────────────┐
+        │ Dispatcher sees terminal agg │
+        │ Calls resume_hook(token)     │
+        └───────┬──────────────────────┘
+                │
+                ▼
+        Parent re-enters run_stream_workflow,
+        recomputes aggregate, returns or re-raises.
+```
+
+## StreamWorkflowResult
+
+```python
+@dataclass
+class StreamWorkflowResult:
+    status: str                              # "completed"
+    step_states: dict[str, str]              # step_run_id -> final status
+    step_results: dict[str, Any]             # step_run_id -> set_result(...) payload
+
+    def get_result(self, step_name: str) -> Any:
+        """Look up a result by step name (matches against step_run_id)."""
+```
+
+Any value a step published via `await set_result(...)` in its lifecycle body is attached to the subscription row and surfaced here — the parent workflow does not need to read checkpoints.
+
+## Backend Support
+
+Stream workflows are fully supported on:
+
+- **PostgreSQL** — the primary target (migrations v5/v6/v7 add the required columns)
+- **Citus** — inherits from Postgres, with reference-table distribution
+- **InMemory** — for unit tests
+
+Other SQL backends (SQLite, MySQL) and NoSQL backends (DynamoDB, Cassandra) accept the stream API for signature compatibility but do **not** scope subscriptions by `stream_run_id`. Running concurrent stream runs on those backends will cross-contaminate subscription state — stick to Postgres/Citus for production streams.
+
+## Scheduled Signals
+
+Use `schedule_signal()` to emit a signal after a delay. A Celery beat task (`pyworkflow.streams.drain_scheduled_signals`, runs every 2s) polls the `scheduled_signals` table and emits due rows.
+
+```python
+from pyworkflow.streams import schedule_signal
+
+await schedule_signal(
+    stream_id="agent_comms",
+    signal_type="supervisor.wakeup",
+    payload={"reason": "periodic_check"},
+    delay_seconds=300,  # 5 minutes
+    stream_run_id="stream_job_42",
+)
+```
+
+This replaces in-process polling loops inside stream workflows — the scheduler is durable across worker restarts.
+
+## Related Concepts
+
+- [Stream Steps](/concepts/stream-steps) — the reactive units that subscribe to streams
+- [Hooks](/concepts/hooks) — stream workflows use `hook()` internally for parent suspension
+- [Workflows](/concepts/workflows) — stream workflows are driven from inside regular `@workflow`s

{pyworkflow_engine-0.2.7 → pyworkflow_engine-0.2.8}/pyproject.toml

@@ -7,7 +7,7 @@ packages = [{include = "pyworkflow"}]
 
 [project]
 name = "pyworkflow-engine"
-version = "0.2.7"
+version = "0.2.8"
 description = "A Python implementation of durable, event-sourced workflows inspired by Vercel Workflow"
 readme = "README.md"
 requires-python = ">=3.11"

{pyworkflow_engine-0.2.7 → pyworkflow_engine-0.2.8}/pyworkflow/__init__.py

@@ -29,7 +29,7 @@ Quick Start:
     >>> run_id = await start(my_workflow, "Alice")
 """
 
-__version__ = "0.2.7"
+__version__ = "0.2.8"
 
 # Configuration
 from pyworkflow.config import (

{pyworkflow_engine-0.2.7 → pyworkflow_engine-0.2.8}/pyworkflow/storage/citus.py

@@ -116,22 +116,16 @@ class CitusMigrationRunner(PostgresMigrationRunner):
                 # so this is a no-op but we still record the version.
                 pass
             elif migration.version == 5:
-                # V5: Add stream_run_id to stream_subscriptions + scheduled_signals table
-                await conn.execute("""
-                    DO $$
-                    BEGIN
-                        IF EXISTS (
-                            SELECT FROM information_schema.tables
-                            WHERE table_name = 'stream_subscriptions'
-                        ) AND NOT EXISTS (
-                            SELECT 1 FROM information_schema.columns
-                            WHERE table_name = 'stream_subscriptions'
-                              AND column_name = 'stream_run_id'
-                        ) THEN
-                            ALTER TABLE stream_subscriptions ADD COLUMN stream_run_id TEXT NULL;
-                        END IF;
-                    END $$
-                """)
+                # V5: Add stream_run_id to stream_subscriptions + scheduled_signals table.
+                # IMPORTANT: Use top-level ALTER (NOT inside a DO $$ block) so Citus
+                # propagates the DDL to all worker shards via the ProcessUtility hook.
+                # DDL inside PL/pgSQL DO blocks is executed via SPI and bypasses the
+                # hook, which would leave worker shards missing the column while the
+                # coordinator catalog appears correct.
+                await conn.execute(
+                    "ALTER TABLE stream_subscriptions "
+                    "ADD COLUMN IF NOT EXISTS stream_run_id TEXT NULL"
+                )
                 await conn.execute(
                     "CREATE INDEX IF NOT EXISTS idx_subscriptions_stream_run "
                     "ON stream_subscriptions(stream_id, stream_run_id)"
@@ -164,41 +158,22 @@ class CitusMigrationRunner(PostgresMigrationRunner):
                 if not already:
                     await conn.execute("SELECT create_reference_table('scheduled_signals')")
             elif migration.version == 6:
-                # V6: Add parent_run_id + parent_hook_token to stream_subscriptions
-                await conn.execute("""
-                    DO $$
-                    BEGIN
-                        IF EXISTS (
-                            SELECT FROM information_schema.tables
-                            WHERE table_name = 'stream_subscriptions'
-                        ) AND NOT EXISTS (
-                            SELECT 1 FROM information_schema.columns
-                            WHERE table_name = 'stream_subscriptions'
-                              AND column_name = 'parent_run_id'
-                        ) THEN
-                            ALTER TABLE stream_subscriptions
-                                ADD COLUMN parent_run_id TEXT NULL,
-                                ADD COLUMN parent_hook_token TEXT NULL;
-                        END IF;
-                    END $$
-                """)
+                # V6: Add parent_run_id + parent_hook_token to stream_subscriptions.
+                # Top-level ALTERs so Citus propagates to all worker shards (see V5 note).
+                await conn.execute(
+                    "ALTER TABLE stream_subscriptions "
+                    "ADD COLUMN IF NOT EXISTS parent_run_id TEXT NULL"
+                )
+                await conn.execute(
+                    "ALTER TABLE stream_subscriptions "
+                    "ADD COLUMN IF NOT EXISTS parent_hook_token TEXT NULL"
+                )
             elif migration.version == 7:
-                await conn.execute("""
-                    DO $$
-                    BEGIN
-                        IF EXISTS (
-                            SELECT FROM information_schema.tables
-                            WHERE table_name = 'stream_subscriptions'
-                        ) AND NOT EXISTS (
-                            SELECT 1 FROM information_schema.columns
-                            WHERE table_name = 'stream_subscriptions'
-                              AND column_name = 'result'
-                        ) THEN
-                            ALTER TABLE stream_subscriptions
-                                ADD COLUMN result JSONB NULL;
-                        END IF;
-                    END $$
-                """)
+                # V7: Add result column. Top-level ALTER for Citus propagation (see V5 note).
+                await conn.execute(
+                    "ALTER TABLE stream_subscriptions "
+                    "ADD COLUMN IF NOT EXISTS result JSONB NULL"
+                )
             elif migration.up_func:
                 await migration.up_func(conn)
             elif migration.up_sql and migration.up_sql != "SELECT 1":