pyworkflow-engine 0.1.7__py3-none-any.whl

docs/concepts/sleep.mdx

@@ -0,0 +1,312 @@
+---
+title: 'Sleep'
+description: 'Pause workflows for any duration without consuming resources'
+---
+
+## What is Sleep?
+
+The `sleep()` primitive pauses a workflow for a specified duration. Unlike traditional sleep that blocks a thread, PyWorkflow's sleep **suspends** the workflow completely - no resources are consumed during the sleep period.
+
+```python
+from pyworkflow import workflow, sleep
+
+@workflow()
+async def reminder_sequence(user_id: str):
+    await send_reminder(user_id, "First reminder")
+
+    # Workflow suspends here - zero resources used
+    await sleep("1d")
+
+    # Resumes automatically after 1 day
+    await send_reminder(user_id, "Second reminder")
+
+    await sleep("7d")
+
+    await send_reminder(user_id, "Final reminder")
+```
+
+## How It Works
+
+```
+Workflow Execution
+        │
+        ▼
+┌───────────────┐
+│ Execute Steps │
+└───────┬───────┘
+        │
+        ▼
+┌───────────────┐
+│ sleep("1d")   │
+└───────┬───────┘
+        │
+        ├─── 1. Record sleep_started event
+        │
+        ├─── 2. Schedule Celery Beat task for wake time
+        │
+        ├─── 3. Raise SuspensionSignal
+        │
+        └─── 4. Worker is freed
+                  │
+                  │  ... 1 day passes ...
+                  │
+                  ▼
+        ┌─────────────────┐
+        │ Celery Beat     │
+        │ triggers resume │
+        └────────┬────────┘
+                 │
+                 ▼
+        ┌───────────────┐
+        │ Replay events │
+        │ Resume work   │
+        └───────────────┘
+```
+
+## Duration Formats
+
+### String Format (Recommended)
+
+```python
+await sleep("30s")   # 30 seconds
+await sleep("5m")    # 5 minutes
+await sleep("2h")    # 2 hours
+await sleep("1d")    # 1 day
+await sleep("1w")    # 1 week
+
+# Combined
+await sleep("1h30m") # 1 hour 30 minutes
+await sleep("2d12h") # 2 days 12 hours
+```
+
+### Timedelta
+
+```python
+from datetime import timedelta
+
+await sleep(timedelta(hours=2, minutes=30))
+await sleep(timedelta(days=7))
+```
+
+### Until Specific Time
+
+```python
+from datetime import datetime
+
+# Sleep until a specific datetime
+await sleep(datetime(2025, 12, 25, 9, 0, 0))
+
+# Sleep until next Monday at 9 AM
+next_monday = get_next_monday()
+await sleep(next_monday.replace(hour=9, minute=0))
+```
+
+### Integer (Seconds)
+
+```python
+await sleep(300)  # 300 seconds (5 minutes)
+```
+
+## Zero-Resource Suspension
+
+Traditional async sleep blocks a worker:
+
+```python
+# BAD: This holds a worker for 24 hours!
+import asyncio
+
+async def traditional_sleep():
+    await do_something()
+    await asyncio.sleep(86400)  # Blocks worker for 24h
+    await do_something_else()
+```
+
+PyWorkflow's sleep releases the worker:
+
+```python
+# GOOD: Worker is freed during sleep
+from pyworkflow import sleep
+
+@workflow()
+async def efficient_sleep():
+    await do_something()
+    await sleep("24h")  # Worker freed, resumes later
+    await do_something_else()
+```
+
+<Note>
+  With 100 workflows each sleeping for 1 day, traditional sleep would need 100 workers blocked for 24 hours. PyWorkflow needs 0 workers during the sleep period.
+</Note>
+
+## Use Cases
+
+### Scheduled Reminders
+
+```python
+@workflow()
+async def onboarding_drip(user_id: str):
+    await send_email(user_id, "Welcome!")
+
+    await sleep("1d")
+    await send_email(user_id, "Getting started tips")
+
+    await sleep("3d")
+    await send_email(user_id, "Advanced features")
+
+    await sleep("7d")
+    await send_email(user_id, "How's it going?")
+```
+
+### Delayed Processing
+
+```python
+@workflow()
+async def process_refund(order_id: str):
+    await validate_refund_request(order_id)
+
+    # Wait for potential fraud review
+    await sleep("24h")
+
+    # If not flagged, process refund
+    await execute_refund(order_id)
+    await notify_customer(order_id)
+```
+
+### Rate Limiting
+
+```python
+@workflow()
+async def batch_api_calls(items: list):
+    for i, item in enumerate(items):
+        await call_api(item)
+
+        # Rate limit: 10 calls per minute
+        if (i + 1) % 10 == 0:
+            await sleep("1m")
+```
+
+### Retry with Backoff
+
+```python
+@workflow()
+async def resilient_operation():
+    for attempt in range(5):
+        try:
+            result = await risky_step()
+            return result
+        except TemporaryError:
+            if attempt < 4:
+                # Exponential backoff: 1m, 2m, 4m, 8m
+                delay = f"{2 ** attempt}m"
+                await sleep(delay)
+
+    raise FatalError("All retries exhausted")
+```
+
+## Sleep vs Step Timeout
+
+Sleep and timeouts serve different purposes:
+
+| Feature | Sleep | Timeout |
+|---------|-------|---------|
+| Purpose | Intentional delay | Maximum execution time |
+| Resources | Zero during sleep | Worker is active |
+| Failure | Never fails | Fails if exceeded |
+
+```python
+@step(timeout="30s")  # Fails if step takes > 30s
+async def quick_step():
+    pass
+
+@workflow()
+async def my_workflow():
+    await quick_step()
+    await sleep("1h")  # Intentional 1-hour pause
+    await quick_step()
+```
+
+## Celery Beat Requirement
+
+Sleep resumption requires Celery Beat to be running:
+
+```bash
+# Start Celery Beat for scheduled task execution
+celery -A pyworkflow.celery.app beat --loglevel=info
+```
+
+<Warning>
+  Without Celery Beat, workflows will suspend but never resume automatically. Make sure Beat is running in production.
+</Warning>
+
+### Docker Compose Setup
+
+```yaml
+services:
+  worker:
+    command: celery -A pyworkflow.celery.app worker --loglevel=info
+
+  beat:
+    command: celery -A pyworkflow.celery.app beat --loglevel=info
+```
+
+## Best Practices
+
+<AccordionGroup>
+  <Accordion title="Use sleep for intentional delays only">
+    Don't use sleep as a retry mechanism. Use step retry configuration instead:
+
+    ```python
+    # Good: Use retry config
+    @step(max_retries=3, retry_delay="exponential")
+    async def my_step():
+        pass
+
+    # Avoid: Manual retry with sleep
+    @workflow()
+    async def my_workflow():
+        for i in range(3):
+            try:
+                await my_step()
+                break
+            except:
+                await sleep(f"{2**i}m")
+    ```
+  </Accordion>
+
+  <Accordion title="Consider timezone implications">
+    When sleeping until a specific time, be aware of timezones:
+
+    ```python
+    from datetime import datetime
+    import pytz
+
+    # Explicit timezone
+    eastern = pytz.timezone("US/Eastern")
+    wake_time = eastern.localize(datetime(2025, 1, 15, 9, 0))
+    await sleep(wake_time)
+    ```
+  </Accordion>
+
+  <Accordion title="Keep sleep durations reasonable">
+    Very long sleeps (months, years) work but consider if a different approach is better:
+
+    ```python
+    # Works, but consider alternatives
+    await sleep("365d")
+
+    # Alternative: Schedule a new workflow
+    schedule_workflow(annual_review, run_at="2026-01-01")
+    ```
+  </Accordion>
+</AccordionGroup>
+
+## Next Steps
+
+<CardGroup cols={2}>
+  <Card title="Workflows" icon="diagram-project" href="/concepts/workflows">
+    Learn about workflow orchestration.
+  </Card>
+  <Card title="Deployment" icon="rocket" href="/guides/deployment">
+    Set up Celery Beat in production.
+  </Card>
+</CardGroup>

docs/concepts/steps.mdx

@@ -0,0 +1,301 @@
+---
+title: 'Steps'
+description: 'Isolated, retryable units of work that form the building blocks of workflows'
+---
+
+## What is a Step?
+
+A step is an isolated, retryable unit of work within a workflow. Steps are where the actual business logic executes - calling APIs, processing data, sending emails, etc. Each step runs on a Celery worker and can be retried independently if it fails.
+
+```python
+from pyworkflow import step
+
+@step()
+async def send_email(to: str, subject: str, body: str):
+    """Send an email - retries automatically on failure."""
+    async with EmailClient() as client:
+        await client.send(to=to, subject=subject, body=body)
+    return {"sent": True, "to": to}
+```
+
+## Key Characteristics
+
+<CardGroup cols={2}>
+  <Card title="Isolated" icon="box">
+    Each step runs independently with its own retry policy and timeout.
+  </Card>
+  <Card title="Retryable" icon="rotate">
+    Failed steps automatically retry with configurable backoff strategies.
+  </Card>
+  <Card title="Cached" icon="database">
+    Completed step results are cached and replayed during workflow resumption.
+  </Card>
+  <Card title="Distributed" icon="server">
+    Steps execute on Celery workers, distributing load across your cluster.
+  </Card>
+</CardGroup>
+
+## Creating Steps
+
+<Tabs>
+  <Tab title="Decorator">
+    ```python
+    from pyworkflow import step
+
+    @step()
+    async def fetch_user(user_id: str):
+        async with httpx.AsyncClient() as client:
+            response = await client.get(f"/api/users/{user_id}")
+            return response.json()
+    ```
+  </Tab>
+  <Tab title="Class">
+    ```python
+    from pyworkflow import Step
+
+    class FetchUserStep(Step):
+        async def execute(self, user_id: str):
+            async with httpx.AsyncClient() as client:
+                response = await client.get(f"/api/users/{user_id}")
+                return response.json()
+
+    # Usage in workflow
+    user = await FetchUserStep()(user_id)
+    ```
+  </Tab>
+</Tabs>
+
+### Configuration Options
+
+<Tabs>
+  <Tab title="Decorator">
+    ```python
+    @step(
+        name="fetch_user_data",      # Custom step name
+        max_retries=5,               # Retry up to 5 times
+        retry_delay="exponential",   # Exponential backoff
+        timeout="30s"                # Step timeout
+    )
+    async def fetch_user(user_id: str):
+        pass
+    ```
+  </Tab>
+  <Tab title="Class">
+    ```python
+    class FetchUserStep(Step):
+        name = "fetch_user_data"
+        max_retries = 5
+        retry_delay = "exponential"
+        timeout = "30s"
+
+        async def execute(self, user_id: str):
+            pass
+    ```
+  </Tab>
+</Tabs>
+
+| Option | Type | Default | Description |
+|--------|------|---------|-------------|
+| `name` | `str` | Function/class name | Unique identifier for the step |
+| `max_retries` | `int` | `3` | Maximum retry attempts |
+| `retry_delay` | `str` | `"fixed"` | Retry strategy: `"fixed"`, `"exponential"`, or duration |
+| `timeout` | `str` | `"60s"` | Maximum step execution time |
+
+## Retry Strategies
+
+### Fixed Delay
+
+Retry with a constant delay between attempts:
+
+```python
+@step(max_retries=3, retry_delay="30s")
+async def call_api():
+    # Retries at: 30s, 60s, 90s
+    pass
+```
+
+### Exponential Backoff
+
+Retry with increasing delays (recommended for external APIs):
+
+```python
+@step(max_retries=5, retry_delay="exponential")
+async def call_external_api():
+    # Retries at: 1s, 2s, 4s, 8s, 16s
+    pass
+```
+
+### Custom Delay in Error
+
+Specify retry delay when raising an error:
+
+```python
+from pyworkflow import step, RetryableError
+
+@step(max_retries=3)
+async def rate_limited_api():
+    response = await call_api()
+
+    if response.status_code == 429:
+        retry_after = response.headers.get("Retry-After", "60")
+        raise RetryableError(
+            "Rate limited",
+            retry_after=f"{retry_after}s"
+        )
+
+    return response.json()
+```
+
+## Error Handling
+
+### RetryableError
+
+Use for transient failures that should be retried:
+
+```python
+from pyworkflow import step, RetryableError
+
+@step(max_retries=3)
+async def fetch_data():
+    try:
+        return await external_api.get_data()
+    except ConnectionError:
+        raise RetryableError("Connection failed")
+    except TimeoutError:
+        raise RetryableError("Request timed out", retry_after="10s")
+```
+
+### FatalError
+
+Use for permanent failures that should stop the workflow:
+
+```python
+from pyworkflow import step, FatalError
+
+@step()
+async def validate_input(data: dict):
+    if "email" not in data:
+        raise FatalError("Email is required")
+
+    if not is_valid_email(data["email"]):
+        raise FatalError(f"Invalid email: {data['email']}")
+
+    return data
+```
+
+### Error Flow
+
+```
+Step Execution
+      │
+      ├─── Success ───────────────────────> Return Result
+      │
+      └─── Exception
+              │
+              ├─── FatalError ────────────> Workflow Failed
+              │
+              └─── RetryableError / Other
+                      │
+                      ├─── Retries Left ──> Wait & Retry
+                      │
+                      └─── No Retries ────> Workflow Failed
+```
+
+## Step Results and Caching
+
+When a workflow resumes after suspension, completed steps are not re-executed. Instead, their cached results are returned:
+
+```python
+@workflow()
+async def my_workflow():
+    # First run: executes the step
+    # After resume: returns cached result
+    user = await fetch_user("user_123")
+
+    await sleep("1h")
+
+    # After 1 hour, workflow resumes
+    # fetch_user is NOT called again - cached result is used
+    await send_email(user["email"], "Hello!")
+```
+
+<Warning>
+  Step results must be serializable. PyWorkflow supports common Python types (dict, list, str, int, datetime, etc.) and uses cloudpickle for complex objects.
+</Warning>
+
+## Parallel Step Execution
+
+Execute multiple steps concurrently using `asyncio.gather()`:
+
+```python
+import asyncio
+from pyworkflow import workflow, step
+
+@step()
+async def fetch_user(user_id: str):
+    return await api.get_user(user_id)
+
+@step()
+async def fetch_orders(user_id: str):
+    return await api.get_orders(user_id)
+
+@step()
+async def fetch_preferences(user_id: str):
+    return await api.get_preferences(user_id)
+
+@workflow()
+async def load_dashboard(user_id: str):
+    # All three steps run in parallel
+    user, orders, preferences = await asyncio.gather(
+        fetch_user(user_id),
+        fetch_orders(user_id),
+        fetch_preferences(user_id)
+    )
+
+    return {
+        "user": user,
+        "orders": orders,
+        "preferences": preferences
+    }
+```
+
+## Best Practices
+
+<AccordionGroup>
+  <Accordion title="Keep steps small and focused">
+    Each step should do one thing well. This makes retries more efficient - if a step fails, only that specific operation is retried.
+  </Accordion>
+
+  <Accordion title="Make steps idempotent">
+    Steps may be retried, so ensure they can be safely re-executed. Use idempotency keys when calling external APIs.
+
+    ```python
+    @step()
+    async def charge_payment(order_id: str, amount: float):
+        # Use order_id as idempotency key
+        return await stripe.charges.create(
+            amount=amount,
+            idempotency_key=f"charge-{order_id}"
+        )
+    ```
+  </Accordion>
+
+  <Accordion title="Use appropriate timeouts">
+    Set timeouts based on expected execution time. External API calls should have shorter timeouts than data processing steps.
+  </Accordion>
+
+  <Accordion title="Handle errors explicitly">
+    Distinguish between retryable and fatal errors. Don't retry errors that will never succeed.
+  </Accordion>
+</AccordionGroup>
+
+## Next Steps
+
+<CardGroup cols={2}>
+  <Card title="Events" icon="timeline" href="/concepts/events">
+    Learn how event sourcing enables durability and replay.
+  </Card>
+  <Card title="Error Handling" icon="shield-check" href="/guides/error-handling">
+    Deep dive into retry strategies and error handling.
+  </Card>
+</CardGroup>