pyworkflow-engine 0.1.7__py3-none-any.whl → 0.1.9__py3-none-any.whl

docs/concepts/continue-as-new.mdx

@@ -1,434 +0,0 @@
----
-title: 'Continue-As-New'
-description: 'Reset event history by continuing workflows as new executions'
----
-
-## Overview
-
-Long-running workflows can accumulate large event histories that impact performance. `continue_as_new()` solves this by completing the current workflow and immediately starting a fresh execution with clean event history.
-
-<CardGroup cols={2}>
-  <Card title="Fresh Event History" icon="rotate">
-    Each continuation starts with a clean event log.
-  </Card>
-  <Card title="Chain Tracking" icon="link">
-    Workflow runs are linked via `continued_from_run_id` and `continued_to_run_id`.
-  </Card>
-  <Card title="State Preservation" icon="database">
-    Pass state to the new execution via arguments.
-  </Card>
-  <Card title="Unlimited Duration" icon="infinity">
-    Run workflows indefinitely without unbounded history growth.
-  </Card>
-</CardGroup>
-
-## When to Use Continue-As-New
-
-`continue_as_new()` is ideal for:
-
-| Use Case | Description |
-|----------|-------------|
-| **Polling Workflows** | Continuously poll for updates without accumulating events |
-| **Batch Processing** | Process large datasets in chunks, resetting history between batches |
-| **Recurring Tasks** | Daily/weekly reports or scheduled jobs that run indefinitely |
-| **Queue Consumers** | Process messages from a queue without history growth |
-| **Long-Running Sync** | Sync data between systems over extended periods |
-
-## Basic Usage
-
-Call `continue_as_new()` with the arguments for the new execution:
-
-```python
-from pyworkflow import workflow, step, continue_as_new
-
-@step()
-async def fetch_batch(offset: int, batch_size: int) -> list:
-    """Fetch a batch of items to process."""
-    items = await db.query(offset=offset, limit=batch_size)
-    return items
-
-@step()
-async def process_item(item: dict) -> dict:
-    """Process a single item."""
-    return await transform(item)
-
-@workflow()
-async def batch_processor(offset: int = 0, batch_size: int = 100) -> str:
-    """Process items in batches with fresh event history."""
-    items = await fetch_batch(offset, batch_size)
-
-    if not items:
-        # No more items - workflow complete
-        return f"Processed {offset} total items"
-
-    # Process this batch
-    for item in items:
-        await process_item(item)
-
-    # Continue with next batch (fresh event history)
-    continue_as_new(offset=offset + batch_size, batch_size=batch_size)
-```
-
-<Note>
-`continue_as_new()` never returns - it raises an internal signal that the executor catches. Any code after it will not execute.
-</Note>
-
-## How It Works
-
-When `continue_as_new()` is called:
-
-```
-┌─────────────────────────────────────────────────────────────────────┐
-│                     Continue-As-New Flow                             │
-│                                                                      │
-│   ┌──────────────┐         ┌──────────────┐         ┌──────────────┐│
-│   │   Run #1     │────────▶│   Run #2     │────────▶│   Run #3     ││
-│   │  offset=0    │         │  offset=100  │         │  offset=200  ││
-│   │  ───────     │         │  ───────     │         │  ───────     ││
-│   │  10 events   │         │  10 events   │         │  5 events    ││
-│   │  CONTINUED   │         │  CONTINUED   │         │  COMPLETED   ││
-│   └──────────────┘         └──────────────┘         └──────────────┘│
-│         │                        │                        │         │
-│         └────────────────────────┴────────────────────────┘         │
-│                           Workflow Chain                             │
-└─────────────────────────────────────────────────────────────────────┘
-```
-
-1. Current run is marked as `CONTINUED_AS_NEW`
-2. A `WORKFLOW_CONTINUED_AS_NEW` event is recorded
-3. A new run is created with `continued_from_run_id` set
-4. The new run starts executing with the provided arguments
-5. New run has fresh, empty event history
-
-## Patterns
-
-### Polling Workflow
-
-```python
-@workflow()
-async def polling_workflow(cursor: str | None = None, poll_count: int = 0):
-    """Poll for updates indefinitely."""
-    # Check for new data
-    new_cursor, updates = await check_for_updates(cursor)
-
-    if updates:
-        for update in updates:
-            await process_update(update)
-
-    if new_cursor is None:
-        return f"Polling complete after {poll_count + 1} polls"
-
-    # Continue polling with new cursor
-    continue_as_new(cursor=new_cursor, poll_count=poll_count + 1)
-```
-
-### Recurring Task with Sleep
-
-```python
-@workflow()
-async def daily_report(day: int = 1):
-    """Generate daily reports indefinitely."""
-    await generate_report(day)
-
-    # Wait until next day
-    await sleep("24h")
-
-    # Continue with next day (fresh history)
-    continue_as_new(day=day + 1)
-```
-
-### Bounded Iterations
-
-```python
-@workflow()
-async def bounded_workflow(iteration: int = 1, max_iterations: int = 10):
-    """Run for a fixed number of iterations."""
-    await do_work(iteration)
-
-    if iteration >= max_iterations:
-        return f"Completed {max_iterations} iterations"
-
-    continue_as_new(iteration=iteration + 1, max_iterations=max_iterations)
-```
-
-## Tracking Workflow Chains
-
-Use `get_workflow_chain()` to retrieve all runs in a continuation chain:
-
-<Tabs>
-  <Tab title="Python API">
-    ```python
-    from pyworkflow import get_workflow_chain
-
-    # Get the full chain from any run in it
-    chain = await get_workflow_chain("run_abc123")
-
-    for run in chain:
-        print(f"{run.run_id}: {run.status.value}")
-        print(f"  From: {run.continued_from_run_id}")
-        print(f"  To: {run.continued_to_run_id}")
-    ```
-  </Tab>
-  <Tab title="CLI">
-    ```bash
-    # View the continuation chain
-    pyworkflow runs chain run_abc123
-
-    # Output:
-    # Continue-As-New Chain
-    # ────────────────────────────────────────────────────────
-    # Chain length: 3 run(s)
-    #
-    # START
-    #    Run ID: run_abc123def456
-    #    Workflow: batch_processor
-    #    Status: continued_as_new
-    #    Duration: 2.3s
-    #
-    #    ↓ continued as new
-    #
-    # #2
-    #    Run ID: run_789xyz123abc
-    #    Workflow: batch_processor
-    #    Status: continued_as_new
-    #    Duration: 1.8s
-    #
-    #    ↓ continued as new
-    #
-    # CURRENT
-    #    Run ID: run_456def789xyz
-    #    Workflow: batch_processor
-    #    Status: completed
-    #    Duration: 0.5s
-    ```
-  </Tab>
-</Tabs>
-
-## Workflow Run Schema
-
-The `WorkflowRun` schema includes continuation tracking fields:
-
-| Field | Type | Description |
-|-------|------|-------------|
-| `continued_from_run_id` | `str \| None` | Run ID this execution continued from |
-| `continued_to_run_id` | `str \| None` | Run ID this execution continued to |
-
-## Important Behaviors
-
-### Arguments Are Required
-
-`continue_as_new()` requires at least one argument:
-
-```python
-# Valid - explicit arguments
-continue_as_new(offset=100)
-continue_as_new(cursor="abc123", count=5)
-
-# Invalid - will raise ValueError
-continue_as_new()  # No arguments provided
-```
-
-<Warning>
-Unlike some workflow systems, PyWorkflow does not automatically use the original arguments. You must explicitly pass all arguments needed for the next execution.
-</Warning>
-
-### Child Workflows Are Cancelled
-
-When a parent workflow continues as new, all running child workflows are cancelled:
-
-```python
-@workflow()
-async def parent_workflow(iteration: int = 1):
-    # Start child workflow
-    handle = await start_child_workflow(child_workflow, "data")
-
-    # When we continue as new, the child is cancelled
-    continue_as_new(iteration=iteration + 1)
-```
-
-### Cancellation Takes Precedence
-
-If a workflow is cancelled, `continue_as_new()` will raise `CancellationError` instead:
-
-```python
-@workflow()
-async def my_workflow(count: int):
-    # If cancellation was requested, this raises CancellationError
-    # not ContinueAsNewSignal
-    continue_as_new(count=count + 1)
-```
-
-### Status is Terminal
-
-`CONTINUED_AS_NEW` is a terminal status like `COMPLETED` or `FAILED`:
-
-```python
-# Cannot cancel a workflow that already continued
-result = await cancel_workflow("run_that_continued")
-# Returns False
-```
-
-## Events
-
-The continuation is recorded as a `WORKFLOW_CONTINUED_AS_NEW` event:
-
-```json
-{
-  "type": "workflow.continued_as_new",
-  "run_id": "run_abc123",
-  "timestamp": "2025-01-15T10:30:00Z",
-  "data": {
-    "new_run_id": "run_def456",
-    "args": "[100]",
-    "kwargs": "{\"batch_size\": 50}",
-    "reason": null
-  }
-}
-```
-
-View continuation events with the CLI:
-
-```bash
-pyworkflow runs logs run_abc123 --filter continued
-```
-
-## Best Practices
-
-<AccordionGroup>
-  <Accordion title="Use for unbounded workflows">
-    Any workflow that could run indefinitely (polling, queues, recurring tasks) should use `continue_as_new()` to prevent unbounded event history growth.
-
-    ```python
-    @workflow()
-    async def message_consumer():
-        while True:  # Don't do this!
-            msg = await get_next_message()
-            await process_message(msg)
-
-    # Better - use continue_as_new
-    @workflow()
-    async def message_consumer(messages_processed: int = 0):
-        msg = await get_next_message()
-        if msg:
-            await process_message(msg)
-            continue_as_new(messages_processed=messages_processed + 1)
-        return f"Processed {messages_processed} messages"
-    ```
-  </Accordion>
-
-  <Accordion title="Pass minimal state">
-    Only pass the state needed for the next execution. Large payloads increase storage and serialization costs.
-
-    ```python
-    # Good - minimal state
-    continue_as_new(cursor="abc123", processed_count=1000)
-
-    # Bad - passing large data
-    continue_as_new(all_results=huge_list_of_results)
-    ```
-  </Accordion>
-
-  <Accordion title="Include progress tracking">
-    Include counters or timestamps to track overall progress across the chain:
-
-    ```python
-    @workflow()
-    async def sync_workflow(
-        cursor: str | None = None,
-        total_synced: int = 0,
-        started_at: str | None = None
-    ):
-        if started_at is None:
-            started_at = datetime.now().isoformat()
-
-        items, new_cursor = await fetch_items(cursor)
-        await sync_items(items)
-
-        if new_cursor:
-            continue_as_new(
-                cursor=new_cursor,
-                total_synced=total_synced + len(items),
-                started_at=started_at
-            )
-
-        return {
-            "total_synced": total_synced + len(items),
-            "started_at": started_at,
-            "completed_at": datetime.now().isoformat()
-        }
-    ```
-  </Accordion>
-
-  <Accordion title="Handle the final iteration">
-    Always have a termination condition that returns normally:
-
-    ```python
-    @workflow()
-    async def batch_workflow(offset: int = 0):
-        items = await fetch_items(offset)
-
-        if not items:
-            # Terminal condition - return result
-            return {"processed": offset, "status": "complete"}
-
-        await process_items(items)
-        continue_as_new(offset=offset + len(items))
-    ```
-  </Accordion>
-</AccordionGroup>
-
-## API Reference
-
-### `continue_as_new()`
-
-```python
-def continue_as_new(*args: Any, **kwargs: Any) -> NoReturn
-```
-
-Complete the current workflow and start a new execution with fresh event history.
-
-| Parameter | Type | Description |
-|-----------|------|-------------|
-| `*args` | `Any` | Positional arguments for the new execution |
-| `**kwargs` | `Any` | Keyword arguments for the new execution |
-
-**Raises:**
-- `ContinueAsNewSignal` - Internal signal caught by the executor
-- `ValueError` - If no arguments are provided
-- `RuntimeError` - If called outside a workflow context
-- `CancellationError` - If workflow is being cancelled
-
-### `get_workflow_chain()`
-
-```python
-async def get_workflow_chain(
-    run_id: str,
-    storage: StorageBackend | None = None,
-) -> list[WorkflowRun]
-```
-
-Get all workflow runs in a continuation chain.
-
-| Parameter | Type | Default | Description |
-|-----------|------|---------|-------------|
-| `run_id` | `str` | required | Any run ID in the chain |
-| `storage` | `StorageBackend` | `None` | Storage backend (uses configured default) |
-
-**Returns:** List of `WorkflowRun` objects ordered from first to last in the chain.
-
-## Next Steps
-
-<CardGroup cols={2}>
-  <Card title="Sleep" icon="clock" href="/concepts/sleep">
-    Learn about durable sleep for delays.
-  </Card>
-  <Card title="Hooks" icon="webhook" href="/concepts/hooks">
-    Wait for external events in your workflows.
-  </Card>
-  <Card title="Fault Tolerance" icon="shield-check" href="/concepts/fault-tolerance">
-    Automatic recovery from worker crashes.
-  </Card>
-  <Card title="CLI Guide" icon="terminal" href="/guides/cli">
-    Manage workflows from the command line.
-  </Card>
-</CardGroup>

docs/concepts/events.mdx

@@ -1,266 +0,0 @@
----
-title: 'Events'
-description: 'Event sourcing provides durability, auditability, and deterministic replay'
----
-
-## What is Event Sourcing?
-
-PyWorkflow uses event sourcing to achieve durable, fault-tolerant execution. Instead of storing just the current state, every state change is recorded as an immutable event in an append-only log. This enables:
-
-- **Durability**: Workflows survive crashes and restarts
-- **Replay**: Workflows can resume from any point
-- **Auditability**: Complete history of everything that happened
-
-```
-Workflow Execution Timeline
-────────────────────────────────────────────────────────►
-
-Event 1          Event 2           Event 3           Event 4
-workflow_started → step_completed → sleep_started → ...
-
-       │              │                │
-       ▼              ▼                ▼
-┌──────────────────────────────────────────────────┐
-│              Event Log (Append-Only)              │
-└──────────────────────────────────────────────────┘
-```
-
-## How It Works
-
-### Recording Events
-
-As your workflow executes, PyWorkflow automatically records events:
-
-```python
-@workflow()
-async def order_workflow(order_id: str):
-    # Event: workflow_started
-
-    order = await validate_order(order_id)
-    # Event: step_completed (validate_order)
-
-    await sleep("1h")
-    # Event: sleep_started
-    # Workflow suspends here...
-
-    # ... 1 hour later ...
-    # Event: workflow_resumed
-
-    await send_confirmation(order)
-    # Event: step_completed (send_confirmation)
-
-    return order
-    # Event: workflow_completed
-```
-
-### Replaying Events
-
-When a workflow resumes after suspension, PyWorkflow replays all recorded events to restore the exact state:
-
-```python
-# Replay process:
-# 1. Load all events for this run_id
-# 2. For each step_completed event, cache the result
-# 3. Re-execute the workflow function
-# 4. When a step is called, return cached result instead of executing
-# 5. Continue from where we left off
-```
-
-<Note>
-  During replay, steps are not re-executed. Their cached results from the event log are returned immediately. This ensures deterministic execution.
-</Note>
-
-## Event Types
-
-PyWorkflow records 16 different event types:
-
-### Workflow Events
-
-| Event | Description |
-|-------|-------------|
-| `workflow_started` | Workflow execution began |
-| `workflow_completed` | Workflow finished successfully |
-| `workflow_failed` | Workflow terminated with an error |
-| `workflow_suspended` | Workflow paused (sleep, webhook) |
-| `workflow_resumed` | Workflow continued after suspension |
-
-### Step Events
-
-| Event | Description |
-|-------|-------------|
-| `step_started` | Step execution began |
-| `step_completed` | Step finished successfully (result cached) |
-| `step_failed` | Step failed (may retry) |
-| `step_retrying` | Step is being retried |
-
-### Sleep Events
-
-| Event | Description |
-|-------|-------------|
-| `sleep_started` | Sleep/delay began |
-| `sleep_completed` | Sleep finished, workflow resuming |
-
-### Log Events
-
-| Event | Description |
-|-------|-------------|
-| `log_info` | Info-level log message |
-| `log_warning` | Warning-level log message |
-| `log_error` | Error-level log message |
-| `log_debug` | Debug-level log message |
-
-## Event Structure
-
-Each event contains:
-
-```python
-{
-    "id": "evt_abc123",           # Unique event ID
-    "run_id": "run_xyz789",       # Workflow run ID
-    "type": "step_completed",     # Event type
-    "timestamp": "2025-01-15T10:30:45Z",
-    "sequence": 3,                # Order in the event log
-    "data": {                     # Event-specific data
-        "step_id": "validate_order",
-        "step_name": "validate_order",
-        "result": {"order_id": "ORD-123", "valid": True},
-        "duration_ms": 150
-    }
-}
-```
-
-## Inspecting Events
-
-### Via Storage Backend
-
-```python
-from pyworkflow.storage.file import FileStorageBackend
-
-storage = FileStorageBackend()
-
-# Get all events for a workflow run
-events = await storage.get_events("run_xyz789")
-
-for event in events:
-    print(f"{event.sequence}: {event.type}")
-    print(f"  Data: {event.data}")
-    print(f"  Time: {event.timestamp}")
-```
-
-### Example Event Log
-
-```
-Sequence | Type              | Data
----------|-------------------|----------------------------------
-1        | workflow_started  | {workflow: "order_processing"}
-2        | step_started      | {step_id: "validate_order"}
-3        | step_completed    | {step_id: "validate_order", result: {...}}
-4        | step_started      | {step_id: "process_payment"}
-5        | step_failed       | {step_id: "process_payment", error: "timeout"}
-6        | step_retrying     | {step_id: "process_payment", attempt: 2}
-7        | step_started      | {step_id: "process_payment"}
-8        | step_completed    | {step_id: "process_payment", result: {...}}
-9        | sleep_started     | {duration: "1h", wake_time: "..."}
-10       | workflow_suspended| {reason: "sleep"}
-```
-
-## Deterministic Replay
-
-For replay to work correctly, workflows must be deterministic:
-
-<Warning>
-  **Don't do this** - Non-deterministic operations break replay:
-
-  ```python
-  @workflow()
-  async def bad_workflow():
-      # BAD: Random values differ on replay
-      order_id = f"ORD-{random.randint(1000, 9999)}"
-
-      # BAD: Current time differs on replay
-      if datetime.now().hour < 12:
-          await morning_flow()
-      else:
-          await afternoon_flow()
-  ```
-</Warning>
-
-<Tip>
-  **Do this instead** - Use steps for non-deterministic operations:
-
-  ```python
-  @step()
-  async def generate_order_id():
-      # Results are cached, so same ID on replay
-      return f"ORD-{random.randint(1000, 9999)}"
-
-  @step()
-  async def get_current_time():
-      # Cached, so same time on replay
-      return datetime.now()
-
-  @workflow()
-  async def good_workflow():
-      order_id = await generate_order_id()
-      current_time = await get_current_time()
-
-      if current_time.hour < 12:
-          await morning_flow()
-      else:
-          await afternoon_flow()
-  ```
-</Tip>
-
-## Storage Backends
-
-Events are stored in a pluggable storage backend:
-
-| Backend | Status | Use Case |
-|---------|--------|----------|
-| **File** | Available | Development, single-machine |
-| **Redis** | Planned | Production, distributed |
-| **PostgreSQL** | Planned | Enterprise, complex queries |
-| **SQLite** | Planned | Embedded applications |
-
-### Configuring Storage
-
-```python
-from pyworkflow.storage.file import FileStorageBackend
-
-# File storage (default)
-storage = FileStorageBackend(
-    base_path="/var/lib/pyworkflow/events"
-)
-
-# Configure PyWorkflow to use this storage
-from pyworkflow import configure
-configure(storage=storage)
-```
-
-## Benefits of Event Sourcing
-
-<CardGroup cols={2}>
-  <Card title="Complete Audit Trail" icon="scroll">
-    Every action is recorded. Know exactly what happened and when.
-  </Card>
-  <Card title="Time Travel Debugging" icon="clock-rotate-left">
-    Replay workflows to debug issues. See the exact state at any point.
-  </Card>
-  <Card title="Failure Recovery" icon="shield">
-    Resume from the last successful point after a crash or restart.
-  </Card>
-  <Card title="Event-Driven Architecture" icon="bolt">
-    Events can trigger other systems, enabling loose coupling.
-  </Card>
-</CardGroup>
-
-## Next Steps
-
-<CardGroup cols={2}>
-  <Card title="Sleep" icon="clock" href="/concepts/sleep">
-    Learn how workflows suspend and resume with sleep.
-  </Card>
-  <Card title="Deployment" icon="rocket" href="/guides/deployment">
-    Configure storage backends for production.
-  </Card>
-</CardGroup>