pyworkflow-engine 0.1.7__py3-none-any.whl

docs/concepts/hooks.mdx

@@ -0,0 +1,552 @@
+---
+title: 'Hooks'
+description: 'Pause workflows and wait for external events like approvals, webhooks, or user actions'
+---
+
+## What are Hooks?
+
+Hooks allow workflows to **suspend execution and wait for external events**. Unlike `sleep()` which resumes after a time duration, hooks wait for an external system or user to provide data before continuing.
+
+```python
+from pydantic import BaseModel
+from pyworkflow import define_hook, workflow
+
+class ApprovalPayload(BaseModel):
+    approved: bool
+    reviewer: str
+    comments: str | None = None
+
+# Define a typed hook with Pydantic validation
+approval_hook = define_hook("manager_approval", ApprovalPayload)
+
+@workflow()
+async def approval_workflow(order_id: str):
+    order = await prepare_order(order_id)
+
+    # Workflow suspends here - waits for external approval
+    approval: ApprovalPayload = await approval_hook(timeout="7d")
+
+    if approval.approved:
+        return await fulfill_order(order)
+    else:
+        return await cancel_order(order, approval.comments)
+```
+
+## How It Works
+
+```
+Workflow Execution
+        │
+        ▼
+┌───────────────┐
+│ Execute Steps │
+└───────┬───────┘
+        │
+        ▼
+┌───────────────────┐
+│ await hook(...)   │
+└───────┬───────────┘
+        │
+        ├─── 1. Generate token (run_id:hook_id)
+        │
+        ├─── 2. Record hook_created event
+        │
+        ├─── 3. Store hook with schema (for CLI)
+        │
+        ├─── 4. Call on_created callback with token
+        │
+        ├─── 5. Raise SuspensionSignal
+        │
+        └─── 6. Worker is freed
+                  │
+                  │  ... waiting for external event ...
+                  │
+                  ▼
+        ┌─────────────────────┐
+        │ External system     │
+        │ calls resume_hook() │
+        │ with token + payload│
+        └────────┬────────────┘
+                 │
+                 ├─── 7. Validate payload (if typed)
+                 │
+                 ├─── 8. Record hook_received event
+                 │
+                 └─── 9. Schedule workflow resumption
+                          │
+                          ▼
+                 ┌───────────────┐
+                 │ Replay events │
+                 │ Resume work   │
+                 └───────────────┘
+```
+
+## Choosing a Hook Type
+
+PyWorkflow offers two ways to define hooks:
+
+| Feature | TypedHook (Recommended) | Simple Hook |
+|---------|-------------------------|-------------|
+| Type safety | Full Pydantic validation | Dict with any keys |
+| CLI prompts | Interactive field-by-field | JSON payload only |
+| IDE support | Autocomplete, type hints | None |
+| Schema stored | Yes (enables CLI features) | No |
+| Best for | Production workflows | Quick prototypes |
+
+<Note>
+  We recommend **TypedHook** for all production workflows. The Pydantic schema enables CLI interactive prompts, payload validation, and better IDE support.
+</Note>
+
+## TypedHook with Pydantic (Recommended)
+
+TypedHook combines hook suspension with Pydantic validation for type-safe payloads.
+
+### Defining a Typed Hook
+
+```python
+from pydantic import BaseModel
+from typing import Optional
+from pyworkflow import define_hook
+
+# 1. Define the payload schema
+class ApprovalPayload(BaseModel):
+    approved: bool
+    reviewer: str
+    comments: Optional[str] = None
+
+# 2. Create the typed hook
+approval_hook = define_hook("manager_approval", ApprovalPayload)
+```
+
+### Using in a Workflow
+
+```python
+from pyworkflow import workflow
+
+@workflow()
+async def order_approval(order_id: str):
+    order = await prepare_order(order_id)
+
+    async def on_hook_created(token: str):
+        print(f"Approval needed! Token: {token}")
+        # Send token to external system, email, Slack, etc.
+
+    # Wait for approval - returns validated ApprovalPayload
+    approval: ApprovalPayload = await approval_hook(
+        timeout="7d",
+        on_created=on_hook_created,
+    )
+
+    # Type-safe access to payload fields
+    if approval.approved:
+        print(f"Approved by {approval.reviewer}")
+        return await fulfill_order(order)
+    else:
+        return await cancel_order(order, approval.comments or "Rejected")
+```
+
+### CLI Interactive Resume
+
+When you run `pyworkflow hooks resume`, the CLI uses the stored Pydantic schema to prompt for each field:
+
+```bash
+$ pyworkflow hooks resume
+
+? Select a pending hook to resume:
+  > manager_approval (run_abc123:hook_manager_approval_1)
+
+? approved (bool): yes
+? reviewer (str): admin@example.com
+? comments (str, optional): Looks good!
+
+Hook resumed successfully.
+```
+
+## Simple Hook
+
+For quick prototyping or when you don't need typed payloads, use the `hook()` function directly.
+
+```python
+from pyworkflow import hook, workflow
+
+@workflow()
+async def simple_approval(order_id: str):
+    order = await prepare_order(order_id)
+
+    async def on_hook_created(token: str):
+        print(f"Resume with: pyworkflow hooks resume {token}")
+
+    # Wait for any payload (untyped dict)
+    approval = await hook(
+        "approval",
+        timeout="24h",
+        on_created=on_hook_created,
+    )
+
+    # Payload is a dict - no type safety
+    if approval.get("approved"):
+        return await fulfill_order(order)
+    else:
+        return await cancel_order(order, approval.get("reason", "Rejected"))
+```
+
+<Warning>
+  Simple hooks don't enable CLI interactive prompts. You must provide the full JSON payload when resuming.
+</Warning>
+
+## Resuming Hooks
+
+### Using the CLI
+
+```bash
+# List all pending hooks
+pyworkflow hooks list --status pending
+
+# View hook details
+pyworkflow hooks info <token>
+
+# Resume interactively (TypedHook only - prompts for fields)
+pyworkflow hooks resume
+
+# Resume with explicit payload
+pyworkflow hooks resume <token> --payload '{"approved": true, "reviewer": "admin"}'
+
+# Resume with payload from file
+pyworkflow hooks resume <token> --payload-file approval.json
+```
+
+### Programmatically
+
+```python
+from pyworkflow import resume_hook
+
+# Resume a hook with payload
+result = await resume_hook(
+    token="run_abc123:hook_manager_approval_1",
+    payload={"approved": True, "reviewer": "admin@example.com"},
+)
+
+if result.success:
+    print(f"Hook resumed, workflow continuing")
+else:
+    print(f"Failed: {result.error}")
+```
+
+### Token Format
+
+Tokens are composite identifiers in the format `run_id:hook_id`:
+
+```
+run_abc123:hook_manager_approval_1
+└────┬────┘ └─────────┬──────────┘
+  run_id         hook_id
+```
+
+This self-describing format allows the system to route the resumption to the correct workflow run.
+
+## Configuration Options
+
+| Option | Type | Description |
+|--------|------|-------------|
+| `timeout` | `str \| int` | Maximum wait time before hook expires. String format (`"24h"`, `"7d"`) or seconds. |
+| `on_created` | `Callable[[str], Awaitable[None]]` | Async callback invoked with token when hook is created. |
+| `payload_schema` | `Type[BaseModel]` | Pydantic model for payload validation (used by simple `hook()` only). |
+
+### Timeout Examples
+
+```python
+# String format (recommended)
+await approval_hook(timeout="24h")    # 24 hours
+await approval_hook(timeout="7d")     # 7 days
+await approval_hook(timeout="1h30m")  # 1 hour 30 minutes
+
+# Integer (seconds)
+await approval_hook(timeout=3600)     # 1 hour
+
+# No timeout (waits indefinitely)
+await approval_hook()
+```
+
+### on_created Callback
+
+The `on_created` callback is called with the hook token when the hook is created. Use it to notify external systems:
+
+```python
+async def notify_approver(token: str):
+    await send_slack_message(
+        channel="#approvals",
+        text=f"Approval needed! Resume: `pyworkflow hooks resume {token}`"
+    )
+    await send_email(
+        to="manager@example.com",
+        subject="Approval Required",
+        body=f"Click to approve: https://app.example.com/approve?token={token}"
+    )
+
+approval = await approval_hook(
+    timeout="7d",
+    on_created=notify_approver,
+)
+```
+
+## Use Cases
+
+### Human Approval Workflows
+
+```python
+@workflow()
+async def expense_approval(expense_id: str, amount: float):
+    expense = await prepare_expense(expense_id)
+
+    if amount > 1000:
+        # High-value expenses need manager approval
+        approval = await manager_approval_hook(timeout="48h")
+        if not approval.approved:
+            return await reject_expense(expense, approval.reason)
+
+    return await process_expense(expense)
+```
+
+### Multi-Level Approval
+
+```python
+@workflow()
+async def contract_approval(contract_id: str):
+    contract = await prepare_contract(contract_id)
+
+    # Level 1: Manager approval
+    manager = await manager_hook(timeout="24h")
+    if not manager.approved:
+        return await reject_contract(contract, "Manager rejected")
+
+    # Level 2: Legal review
+    legal = await legal_hook(timeout="72h")
+    if not legal.approved:
+        return await reject_contract(contract, "Legal rejected")
+
+    # Level 3: Executive sign-off
+    executive = await executive_hook(timeout="7d")
+    if not executive.approved:
+        return await reject_contract(contract, "Executive rejected")
+
+    return await execute_contract(contract)
+```
+
+### Webhook Integration
+
+```python
+@workflow()
+async def payment_processing(order_id: str):
+    order = await create_order(order_id)
+
+    async def setup_webhook(token: str):
+        # Register token with payment provider
+        await payment_provider.register_webhook(
+            callback_url=f"https://api.example.com/hooks/{token}",
+        )
+
+    # Wait for payment confirmation from external provider
+    payment = await payment_confirmation_hook(
+        timeout="1h",
+        on_created=setup_webhook,
+    )
+
+    if payment.status == "completed":
+        return await fulfill_order(order)
+    else:
+        return await cancel_order(order, payment.error)
+```
+
+### User Confirmation
+
+```python
+@workflow()
+async def account_deletion(user_id: str):
+    user = await get_user(user_id)
+
+    async def send_confirmation_email(token: str):
+        await send_email(
+            to=user.email,
+            subject="Confirm Account Deletion",
+            body=f"Click to confirm: https://app.example.com/confirm?token={token}"
+        )
+
+    # Wait for user to confirm deletion
+    confirmation = await confirmation_hook(
+        timeout="24h",
+        on_created=send_confirmation_email,
+    )
+
+    if confirmation.confirmed:
+        return await delete_user(user_id)
+    else:
+        return {"status": "cancelled", "user_id": user_id}
+```
+
+## Best Practices
+
+<AccordionGroup>
+  <Accordion title="Use TypedHook for production workflows">
+    TypedHook provides validation, IDE support, and CLI interactive prompts:
+
+    ```python
+    # Good: Typed hook with Pydantic
+    class ApprovalPayload(BaseModel):
+        approved: bool
+        reviewer: str
+
+    approval_hook = define_hook("approval", ApprovalPayload)
+
+    # Avoid in production: Untyped hook
+    approval = await hook("approval")
+    ```
+  </Accordion>
+
+  <Accordion title="Set appropriate timeouts">
+    Always set timeouts to prevent workflows from waiting indefinitely:
+
+    ```python
+    # Good: Set reasonable timeout
+    await approval_hook(timeout="7d")
+
+    # Avoid: No timeout (waits forever)
+    await approval_hook()
+    ```
+
+    Handle expiration gracefully in your workflow logic.
+  </Accordion>
+
+  <Accordion title="Use descriptive hook names">
+    Hook names should clearly indicate their purpose:
+
+    ```python
+    # Good: Clear purpose
+    manager_approval_hook = define_hook("manager_approval", ApprovalPayload)
+    payment_confirmation_hook = define_hook("payment_confirmation", PaymentPayload)
+
+    # Avoid: Vague names
+    hook1 = define_hook("hook1", Payload)
+    ```
+  </Accordion>
+
+  <Accordion title="Notify external systems via on_created">
+    Use the `on_created` callback to trigger notifications:
+
+    ```python
+    async def notify_systems(token: str):
+        await send_slack_notification(token)
+        await send_email_notification(token)
+        await register_webhook(token)
+
+    approval = await approval_hook(
+        timeout="24h",
+        on_created=notify_systems,
+    )
+    ```
+  </Accordion>
+
+  <Accordion title="Handle hook errors gracefully">
+    Catch and handle hook-related exceptions:
+
+    ```python
+    from pyworkflow import HookExpiredError, HookNotFoundError
+
+    try:
+        result = await resume_hook(token, payload)
+    except HookExpiredError:
+        print("Hook has expired")
+    except HookNotFoundError:
+        print("Hook not found")
+    ```
+  </Accordion>
+</AccordionGroup>
+
+## Testing Hooks
+
+Use `MockContext` to test workflows with hooks without actual suspension:
+
+```python
+import asyncio
+from pyworkflow import MockContext, set_context
+
+def test_approval_workflow():
+    # Create mock context with predefined hook responses
+    ctx = MockContext(
+        run_id="test_run",
+        workflow_name="approval_workflow",
+        mock_hooks={
+            "manager_approval": {
+                "approved": True,
+                "reviewer": "test@example.com",
+                "comments": "Approved in test",
+            }
+        }
+    )
+    set_context(ctx)
+
+    try:
+        # Run workflow - hook returns mock response immediately
+        result = asyncio.run(approval_workflow("order-123"))
+
+        # Verify workflow completed correctly
+        assert result["status"] == "fulfilled"
+        assert ctx.hook_count == 1
+    finally:
+        set_context(None)
+```
+
+### Testing Multiple Hooks
+
+```python
+def test_multi_approval_workflow():
+    ctx = MockContext(
+        run_id="test_run",
+        workflow_name="multi_approval",
+        mock_hooks={
+            "manager_approval": {"approved": True, "approver": "manager"},
+            "finance_approval": {"approved": True, "approver": "finance"},
+        }
+    )
+    set_context(ctx)
+
+    try:
+        result = asyncio.run(multi_approval_workflow("order-123"))
+        assert result["status"] == "fulfilled"
+        assert ctx.hook_count == 2
+    finally:
+        set_context(None)
+```
+
+## Hook Events
+
+Hooks generate events that are stored in the event log:
+
+| Event Type | When | Data |
+|------------|------|------|
+| `hook.created` | Hook is awaited | hook_id, token, name, expires_at, schema |
+| `hook.received` | Hook is resumed | hook_id, payload |
+| `hook.expired` | Timeout reached | hook_id |
+| `hook.disposed` | Hook cleaned up | hook_id |
+
+View hook events for a run:
+
+```bash
+pyworkflow runs logs <run_id> --type hook
+```
+
+## Next Steps
+
+<CardGroup cols={2}>
+  <Card title="Sleep" icon="clock" href="/concepts/sleep">
+    Pause workflows for time durations.
+  </Card>
+  <Card title="Workflows" icon="diagram-project" href="/concepts/workflows">
+    Learn about workflow orchestration.
+  </Card>
+  <Card title="CLI Guide" icon="terminal" href="/guides/cli">
+    Manage hooks with CLI commands.
+  </Card>
+  <Card title="Fault Tolerance" icon="shield" href="/concepts/fault-tolerance">
+    Auto-recovery from worker crashes.
+  </Card>
+</CardGroup>

docs/concepts/limitations.mdx

@@ -0,0 +1,167 @@
+---
+title: 'Limitations'
+description: 'Built-in safeguards and limits to ensure stable workflow execution'
+---
+
+## Overview
+
+PyWorkflow includes built-in safeguards to prevent runaway workflows from consuming excessive resources. These limits help ensure system stability and predictable behavior.
+
+## Event History Limits
+
+Since PyWorkflow uses event sourcing, every workflow action is recorded as an event. To prevent unbounded growth and memory issues, there are limits on the number of events a workflow can generate.
+
+### Soft Limit (Warning)
+
+| Setting | Default | Description |
+|---------|---------|-------------|
+| `event_soft_limit` | 10,000 | Logs a warning when reached |
+| `event_warning_interval` | 100 | Logs warning every N events after soft limit |
+
+When a workflow reaches 10,000 events, PyWorkflow logs a warning:
+
+```
+WARNING - Workflow approaching event limit: 10000/50000
+```
+
+After the soft limit, warnings continue every 100 events (10,100, 10,200, etc.) to alert you that the workflow is growing large.
+
+### Hard Limit (Failure)
+
+| Setting | Default | Description |
+|---------|---------|-------------|
+| `event_hard_limit` | 50,000 | Terminates workflow with failure |
+
+When a workflow reaches 50,000 events, it is terminated with an `EventLimitExceededError`:
+
+```python
+from pyworkflow.core.exceptions import EventLimitExceededError
+
+# This error is raised when hard limit is reached
+# EventLimitExceededError: Workflow run_abc123 exceeded maximum event limit: 50000 >= 50000
+```
+
+<Warning>
+  The hard limit is a safety mechanism. If your workflow is hitting this limit, it likely indicates a design issue such as an infinite loop or processing too many items in a single workflow run.
+</Warning>
+
+## Why These Limits Exist
+
+1. **Memory Protection**: Each event consumes memory. Unbounded event growth can exhaust system resources.
+
+2. **Replay Performance**: When workflows resume, all events are replayed. Large event logs slow down resumption.
+
+3. **Storage Costs**: Events are persisted to storage. Excessive events increase storage requirements.
+
+4. **Bug Detection**: Hitting limits often indicates bugs like infinite loops or improper workflow design.
+
+## Best Practices
+
+<Tip>
+  **Design for bounded event counts:**
+
+  ```python
+  # BAD: Processing millions of items in one workflow
+  @workflow()
+  async def process_all_orders():
+      orders = await get_all_orders()  # Could be millions!
+      for order in orders:
+          await process_order(order)  # Each creates events
+
+  # GOOD: Process in batches with separate workflow runs
+  @workflow()
+  async def process_order_batch(batch_ids: list[str]):
+      for order_id in batch_ids[:100]:  # Bounded batch size
+          await process_order(order_id)
+
+  # Orchestrate batches externally
+  for batch in chunk_list(all_order_ids, 100):
+      await start(process_order_batch, batch)
+  ```
+</Tip>
+
+## Configuring Limits
+
+<Warning>
+  Modifying event limits is **not recommended**. The defaults are carefully chosen to balance flexibility with safety. Only change these if you fully understand the implications.
+</Warning>
+
+If you must change the limits:
+
+```python
+import pyworkflow
+
+# This will emit a UserWarning - proceed with caution
+pyworkflow.configure(
+    event_soft_limit=20_000,      # Warning at 20K events
+    event_hard_limit=100_000,     # Fail at 100K events
+    event_warning_interval=200,   # Warn every 200 events after soft limit
+)
+```
+
+### Configuration Options
+
+| Option | Type | Default | Description |
+|--------|------|---------|-------------|
+| `event_soft_limit` | `int` | 10,000 | Event count to start logging warnings |
+| `event_hard_limit` | `int` | 50,000 | Event count to terminate workflow |
+| `event_warning_interval` | `int` | 100 | Events between warnings after soft limit |
+
+## Transient Mode
+
+Event limits only apply to **durable** workflows. Transient workflows (with `durable=False`) do not record events and are not subject to these limits.
+
+```python
+@workflow(durable=False)
+async def quick_task():
+    # No event limits - events aren't recorded
+    for i in range(100_000):
+        await some_step(i)
+```
+
+<Note>
+  Transient workflows sacrifice durability for performance. They cannot be resumed after crashes or restarts.
+</Note>
+
+## Monitoring Event Counts
+
+You can monitor event counts using the storage backend:
+
+```python
+from pyworkflow import get_storage
+
+storage = get_storage()
+events = await storage.get_events(run_id)
+
+print(f"Event count: {len(events)}")
+
+# Check if approaching limits
+if len(events) > 8000:
+    print("Warning: Workflow has many events, consider redesigning")
+```
+
+## Handling Limit Errors
+
+When the hard limit is reached, an `EventLimitExceededError` is raised. This error inherits from `FatalError`, meaning it will **not** be retried.
+
+```python
+from pyworkflow.core.exceptions import EventLimitExceededError, FatalError
+
+try:
+    await start(my_workflow, args)
+except EventLimitExceededError as e:
+    print(f"Workflow {e.run_id} exceeded {e.limit} events")
+    print(f"Actual count: {e.event_count}")
+    # Consider splitting the work into smaller workflows
+```
+
+## Next Steps
+
+<CardGroup cols={2}>
+  <Card title="Events" icon="scroll" href="/concepts/events">
+    Learn how event sourcing works in PyWorkflow.
+  </Card>
+  <Card title="Configuration" icon="gear" href="/guides/configuration">
+    See all available configuration options.
+  </Card>
+</CardGroup>