soe-ai 0.1.0__py3-none-any.whl → 0.1.2__py3-none-any.whl

soe/docs/guide_00_getting_started.md

@@ -0,0 +1,341 @@
+# SOE Guide: Getting Started
+
+## What is SOE?
+
+SOE (Signal-driven Orchestration Engine) is a Python library for building workflows through **signal-based orchestration**. Workflows are defined in YAML and executed by nodes that communicate through signals.
+
+**If you've already seen the [README](../README.md)**, this guide expands on the quick start with explanations and guidance to other guides.
+
+---
+
+## The 7 Primitives
+
+SOE is built on **7 core concepts**. Understanding these gives you complete control:
+
+| Primitive | What It Is | Quick Example |
+|-----------|------------|---------------|
+| **Signals** | Named events for node communication | `START`, `DONE`, `ERROR` |
+| **Context** | Shared state dictionary (with Jinja access) | `{{ context.user_id }}` |
+| **Workflows** | YAML definitions of nodes and their relationships | See below |
+| **Backends** | Pluggable storage implementations | PostgreSQL, DynamoDB, in-memory |
+| **Nodes** | Execution units (router, llm, tool, agent, child) | `node_type: llm` |
+| **Identities** | System prompts for LLM/Agent nodes | Optional role definitions |
+| **Context Schema** | Type validation for LLM outputs | Optional field definitions |
+
+**Key insight**: Context is accessible via Jinja2 in conditions and prompts. This makes workflows readable and debuggable.
+
+For detailed coverage, see [The 7 Primitives](primitives/primitives.md).
+
+---
+
+## Signal Types
+
+SOE supports different signal emission patterns:
+
+| Type | Description | Example |
+|------|-------------|---------|
+| **Conditional** | Emitted when Jinja condition is true | `condition: "{{ context.valid }}"` |
+| **Unconditional** | Always emitted (no condition) | `signal_name: DONE` |
+| **Semantic** | LLM selects signal (for testing/prototyping) | `signal_name: APPROVE` with LLM choice |
+
+---
+
+## Key Concept: Synchronous Execution
+
+SOE executes **synchronously** within a single Python process. This is intentional:
+
+```python
+# This is how SOE works - synchronous, blocking execution
+execution_id = orchestrate(
+    config=my_workflow,
+    initial_workflow_name="example_workflow",
+    initial_signals=["START"],
+    initial_context={"user": "alice"},
+    backends=backends,
+    broadcast_signals_caller=broadcast_signals_caller,
+)
+# When this returns, the entire workflow has completed
+```
+
+**Why synchronous?**
+
+1. **Simplicity** — No async/await complexity, no event loop management
+2. **Predictability** — Easy to debug, test, and reason about
+3. **Portability** — The same workflow runs locally, in Lambda, in Jenkins, anywhere
+
+**But what about distribution?**
+
+The synchronous nature is about *how a single workflow executes*. Distribution happens at the **caller level**:
+
+- Replace `broadcast_signals_caller` with an HTTP webhook → signals trigger remote services
+- Replace `broadcast_signals_caller` with an SQS publisher → signals become queue messages
+- Replace backends with PostgreSQL/DynamoDB → state becomes distributed
+
+Your workflow YAML stays exactly the same. See [Chapter 10: Custom Infrastructure](guide_10_infrastructure.md) for details.
+
+---
+
+## Installation
+
+```bash
+# With uv (recommended)
+uv add soe-ai
+
+# Or with pip
+pip install soe-ai
+```
+
+---
+
+## Quick Start: Copy-Paste Template
+
+Here's a minimal working example. Copy this, add your own workflow, and run:
+
+```python
+from soe import orchestrate, broadcast_signals
+from soe.local_backends import create_in_memory_backends
+from soe.nodes.router.factory import create_router_node_caller
+
+# 1. Define your workflow in YAML
+workflow_yaml = """
+example_workflow:
+  Start:
+    node_type: router
+    event_triggers: [START]
+    event_emissions:
+      - signal_name: VALID
+        condition: "{{ context.user_input is defined }}"
+      - signal_name: INVALID
+        condition: "{{ context.user_input is not defined }}"
+
+  # Routers can also act as signal converters (tunneling)
+  # This decouples downstream nodes from knowing validation logic
+  OnValid:
+    node_type: router
+    event_triggers: [VALID]
+    event_emissions:
+      - signal_name: DONE
+
+  OnInvalid:
+    node_type: router
+    event_triggers: [INVALID]
+    event_emissions:
+      - signal_name: DONE
+"""
+
+# 2. Create in-memory backends (simplest option)
+backends = create_in_memory_backends()
+
+# 3. Set up nodes and caller
+nodes = {}
+
+def broadcast_signals_caller(id: str, signals):
+    broadcast_signals(id, signals, nodes, backends)
+
+nodes["router"] = create_router_node_caller(backends, broadcast_signals_caller)
+
+# 4. Execute!
+execution_id = orchestrate(
+    config=workflow_yaml,
+    initial_workflow_name="example_workflow",
+    initial_signals=["START"],
+    initial_context={"user_input": "Hello, SOE!"},
+    backends=backends,
+    broadcast_signals_caller=broadcast_signals_caller,
+)
+
+# 5. Check the result
+context = backends.context.get_context(execution_id)
+print(f"Status: {context['status']}")  # Output: Status: success
+```
+
+---
+
+## Adding More Node Types
+
+The template above only uses `router` nodes. To add more capabilities:
+
+### Adding Tool Nodes
+
+```python
+from soe.nodes.tool.factory import create_tool_node_caller
+
+# Define your tools
+def greet(name: str) -> dict:
+    return {"greeting": f"Hello, {name}!"}
+
+tools_registry = {"greet": greet}
+
+# Add to nodes
+nodes["tool"] = create_tool_node_caller(backends, tools_registry, broadcast_signals_caller)
+```
+
+### Adding LLM Nodes
+
+SOE **does not provide** an LLM. Instead, you provide a `call_llm` function that wraps your chosen provider:
+
+```python
+from soe.nodes.llm.factory import create_llm_node_caller
+
+# Define your LLM caller (wrap your API)
+def call_llm(prompt: str, config: dict) -> str:
+    """
+    The LLM caller contract:
+    - prompt: The rendered prompt string (text in)
+    - config: The full node configuration from YAML
+    - returns: The LLM response (text out)
+    """
+    # Example: Using OpenAI
+    import openai
+    response = openai.chat.completions.create(
+        model=config.get("model", "gpt-4o"),  # Custom field from YAML
+        messages=[{"role": "user", "content": prompt}]
+    )
+    return response.choices[0].message.content
+
+# Add to nodes
+nodes["llm"] = create_llm_node_caller(backends, call_llm, broadcast_signals_caller)
+```
+
+The `config` parameter receives the **entire node configuration** from your YAML. This lets you add custom fields:
+
+```yaml
+MyLLM:
+  node_type: llm
+  prompt: "Summarize: {{ context.text }}"
+  model: gpt-4o-mini    # Custom field - your call_llm can read this
+  temperature: 0.7       # Another custom field
+  output_field: summary
+```
+
+See [Chapter 10: Infrastructure](guide_10_infrastructure.md#the-llm-caller) for detailed examples.
+
+### Adding Agent Nodes
+
+```python
+from soe.nodes.agent.factory import create_agent_node_caller
+
+# Agent needs both LLM and tools
+tools = [{"function": greet, "max_retries": 0}]
+nodes["agent"] = create_agent_node_caller(backends, tools, call_llm, broadcast_signals_caller)
+```
+
+### Adding Child Workflow Nodes
+
+```python
+import copy
+from soe import orchestrate, broadcast_signals
+from soe.nodes.child.factory import create_child_node_caller
+from soe.lib.yaml_parser import parse_yaml
+
+# Create orchestrate caller for child workflows
+def orchestrate_caller(config, initial_workflow_name, initial_signals, initial_context, backends):
+    if isinstance(config, str):
+        config = parse_yaml(config)
+    else:
+        config = copy.deepcopy(config)
+
+    def broadcast_signals_caller(execution_id, signals):
+        broadcast_signals(execution_id, signals, nodes, backends)
+
+    return orchestrate(
+        config=config,
+        initial_workflow_name=initial_workflow_name,
+        initial_signals=initial_signals,
+        initial_context=initial_context,
+        backends=backends,
+        broadcast_signals_caller=broadcast_signals_caller,
+    )
+
+nodes["child"] = create_child_node_caller(backends, orchestrate_caller)
+```
+
+---
+
+## Helper Function: All Nodes at Once
+
+For convenience, you can set up all node types in one call using the built-in `create_all_nodes` helper:
+
+```python
+from soe import create_all_nodes, orchestrate
+from soe.local_backends import create_in_memory_backends
+
+backends = create_in_memory_backends()
+
+# Set up all nodes (pass your call_llm and tools_registry)
+nodes, broadcast = create_all_nodes(
+    backends,
+    call_llm=my_call_llm,
+    tools_registry=my_tools
+)
+
+# Usage
+execution_id = orchestrate(
+    config=workflow_yaml,
+    initial_workflow_name="example_workflow",
+    initial_signals=["START"],
+    initial_context={"user_input": "Hello!"},
+    backends=backends,
+    broadcast_signals_caller=broadcast,
+)
+```
+
+---
+
+## Choosing Your Infrastructure
+
+SOE supports different infrastructure configurations depending on your needs:
+
+| Scenario | Backends | Caller | Use Case |
+|----------|----------|--------|----------|
+| **Development** | In-Memory | Local | Fast iteration, unit tests |
+| **Local Debugging** | Local Files | Local | Inspect state in JSON files |
+| **Distributed State** | PostgreSQL/DynamoDB | Local | Shared state, multiple workers |
+| **Fully Distributed** | PostgreSQL/DynamoDB | HTTP/SQS/Lambda | Scalable production |
+
+For custom backends and distributed callers, see [Chapter 10: Infrastructure](guide_10_infrastructure.md).
+
+---
+
+## Workflow Portability
+
+A critical design principle: **your workflow YAML works everywhere**.
+
+```yaml
+# This exact workflow runs:
+# - Locally with in-memory backends
+# - In production with PostgreSQL + Lambda callers
+# - In Jenkins with file backends + webhook callers
+example_workflow:
+  ValidateInput:
+    node_type: router
+    event_triggers: [START]
+    event_emissions:
+      - signal_name: VALID
+        condition: "{{ context.data is defined }}"
+```
+
+The workflow defines *what* happens. The infrastructure (backends + callers) defines *how* it executes. Change the infrastructure without touching workflow logic.
+
+---
+
+## Next Steps
+
+**Understand the Building Blocks:**
+- **[The 7 Primitives](primitives/primitives.md)** — Deep dive into signals, context, workflows, and backends
+
+**Core Guides:**
+1. **[Chapter 1: Tool Nodes](guide_01_tool.md)** — Execute Python functions—APIs, databases, real work
+2. **[Chapter 2: LLM Nodes](guide_02_llm.md)** — Add language model capabilities
+3. **[Chapter 3: Router Nodes](guide_03_router.md)** — Pure routing, branching, no execution
+4. **[Chapter 4: Patterns](guide_04_patterns.md)** — Combine primitives: ReAct, chain-of-thought
+5. **[Chapter 5: Agent Nodes](guide_05_agent.md)** — Built-in ReAct loop for tool-using agents
+6. **[Chapter 6: Schema](guide_06_schema.md)** — Validate LLM outputs with context schema
+7. **[Chapter 7: Identity](guide_07_identity.md)** — System prompts and conversation history
+8. **[Chapter 8: Child Workflows](guide_08_child.md)** — Nested workflows with signal communication
+9. **[Chapter 9: Ecosystem](guide_09_ecosystem.md)** — Multi-workflow registries and versioning
+10. **[Chapter 10: Infrastructure](guide_10_infrastructure.md)** — Custom backends and callers
+11. **[Chapter 11: Built-in Tools](guide_11_builtins.md)** — Self-evolution, documentation exploration, and runtime modification
+
+**Advanced:**
+- **[Self-Evolving Workflows](advanced_patterns/self_evolving_workflows.md)** — Workflows that modify themselves at runtime

soe/docs/guide_01_tool.md

@@ -0,0 +1,206 @@
+
+# SOE Guide: Chapter 1 - Tool Nodes
+
+## Introduction to Tool Nodes
+
+The **Tool Node** executes a Python function directly. It's the most concrete node type in SOE—it does real work: calling APIs, processing data, interacting with databases.
+
+We start with Tool nodes because they represent **actual execution**. While other nodes route signals or generate text, Tool nodes are where your workflow touches the real world.
+
+### When to Use Tool Nodes
+
+- **External Operations**: Sending emails, processing payments, database queries
+- **API Calls**: HTTP requests, third-party service integrations
+- **Data Transformations**: Calculations, file processing, format conversions
+- **Deterministic Logic**: When you need code, not AI
+
+## Your First Tool Node
+
+Let's send an email using a tool node.
+
+### The Workflow
+
+```yaml
+example_workflow:
+  SendEmail:
+    node_type: tool
+    event_triggers: [START]
+    tool_name: send_email
+    context_parameter_field: email_data
+    output_field: email_result
+    event_emissions:
+      - signal_name: EMAIL_SENT
+```
+
+### How It Works
+
+1.  **`tool_name`**: The key in your `tools_registry` dict.
+2.  **`context_parameter_field`**: The context field containing the tool's kwargs (a dict).
+3.  **`output_field`**: Where to store the result in context.
+4.  **`event_emissions`**: Signals to emit after execution (conditions evaluate `result`).
+
+### Understanding context_parameter_field
+
+The `context_parameter_field` specifies which context field contains the parameters to pass to your tool. This field must contain a dictionary that will be unpacked as keyword arguments.
+
+**Where does this data come from?**
+
+1. **Initial context**: You can pass it when starting the workflow:
+   ```python
+   orchestrate(
+       config=workflow,
+       initial_context={"email_data": {"to": "user@example.com", "subject": "Hello"}}
+   )
+   ```
+
+2. **LLM output**: An LLM node can generate structured parameters:
+   ```yaml
+   ExtractEmailParams:
+     node_type: llm
+     prompt: "Extract email parameters from: {{ context.user_request }}"
+     output_field: email_data  # LLM returns {"to": "...", "subject": "...", "body": "..."}
+
+   SendEmail:
+     node_type: tool
+     tool_name: send_email
+     context_parameter_field: email_data  # Uses LLM output as tool input
+   ```
+
+3. **Another tool's output**: Chain tools together:
+   ```yaml
+   PrepareData:
+     node_type: tool
+     tool_name: prepare_email_data
+     output_field: email_data
+
+   SendEmail:
+     node_type: tool
+     tool_name: send_email
+     context_parameter_field: email_data  # Uses previous tool's output
+   ```
+
+**Tip**: Use [Context Schema](guide_06_schema.md) to validate that LLM output has the correct structure before passing to tools.
+
+### The Tools Registry
+
+Tools are registered as plain Python functions:
+
+```python
+def send_email(to: str, subject: str, body: str) -> dict:
+    # Your email sending logic
+    return {"status": "sent", "to": to}
+
+tools_registry = {
+    "send_email": send_email,
+}
+```
+
+## Event Emissions with Conditions
+
+Tool nodes use `event_emissions` with optional Jinja conditions. Signals without conditions are always emitted on success. Signals with conditions can reference:
+
+- **`result`**: The tool's return value
+- **`context`**: The full workflow context
+
+### Conditional Signal Example
+
+```yaml
+example_workflow:
+  ProcessPayment:
+    node_type: tool
+    event_triggers: [START]
+    tool_name: process_payment
+    context_parameter_field: payment_data
+    output_field: payment_result
+    event_emissions:
+      - signal_name: PAYMENT_SUCCESS
+        condition: "{{ result.status == 'approved' }}"
+      - signal_name: PAYMENT_DECLINED
+        condition: "{{ result.status == 'declined' }}"
+      - signal_name: PAYMENT_PENDING
+        condition: "{{ result.status == 'pending' }}"
+```
+
+### How It Works
+
+1.  Tool executes and stores result in `output_field`.
+2.  Each `event_emissions` condition is evaluated against both `result` and `context`.
+3.  Signals with matching conditions (or no condition) are emitted.
+
+**Note:** `event_emissions` are only evaluated on successful execution. For failure handling, use `failure_signal` in the registry.
+
+### Combining Result and Context
+
+Conditions can use both the tool's output and existing context values:
+
+```yaml
+example_workflow:
+  ProcessPayment:
+    node_type: tool
+    event_triggers: [START]
+    tool_name: process_payment
+    context_parameter_field: payment_data
+    output_field: payment_result
+    event_emissions:
+      - signal_name: PAYMENT_SUCCESS
+        condition: "{{ result.status == 'approved' }}"
+      - signal_name: VIP_PAYMENT_SUCCESS
+        condition: "{{ result.status == 'approved' and context.customer.is_vip }}"
+      - signal_name: LARGE_PAYMENT_SUCCESS
+        condition: "{{ result.status == 'approved' and context.payment_data.amount > 1000 }}"
+```
+
+This is useful when you need to:
+- Combine tool output with customer data (VIP handling)
+- Check result against thresholds from context
+- Route based on both tool success and workflow state
+
+## Extended Tool Registry
+
+For more control over tool behavior, use the extended format:
+
+```python
+tools_registry = {
+    # Simple format (default: max_retries=1)
+    "send_email": send_email,
+
+    # Extended format with configuration
+    "process_payment": {
+        "function": process_payment,
+        "max_retries": 3,  # Retry up to 3 times on failure
+        "failure_signal": "PAYMENT_FAILED",  # Emit when all retries exhausted
+    },
+}
+```
+
+**Extended registry fields:**
+
+| Field | Type | Default | Description |
+|-------|------|---------|-------------|
+| `function` | Callable | (required) | The Python function to execute |
+| `max_retries` | int | 1 | Number of retries after initial failure |
+| `failure_signal` | str | None | Signal to emit when all retries exhausted |
+| `process_accumulated` | bool | False | Pass full history list (advanced) |
+
+> **Note:** The `process_accumulated` option is an advanced feature for aggregation patterns. It allows a tool to receive the entire history of a context field instead of just the last value. See [Fan-Out, Fan-In & Aggregations](advanced_patterns/guide_fanout_and_aggregations.md) for details.
+
+### Success vs Failure Flow
+
+```
+Tool executes
+    │
+    ├── Success → evaluate event_emissions → emit matching signals
+    │
+    └── Exception → retry up to max_retries
+                        │
+                        └── All retries failed → emit failure_signal (if defined)
+```
+
+**When to use extended format:**
+
+- **Flaky operations**: Set `max_retries` higher for network calls or external APIs
+- **Critical failures**: Use `failure_signal` to handle unrecoverable errors
+
+## Next Steps
+
+Now that you can execute real operations, let's add intelligence with [LLM Nodes](guide_02_llm.md) →

soe/docs/guide_02_llm.md

@@ -0,0 +1,143 @@
+
+# SOE Guide: Chapter 2 - LLM Nodes
+
+## Introduction to LLM Nodes
+
+The **LLM Node** is a simple way to call a language model directly. Unlike Agent nodes (which we'll cover later), LLM nodes make a single call to the model and store the response.
+
+Think of an LLM node as a specialist: it receives context, generates a response, and passes it along.
+
+## Your First LLM Node: Text Summarization
+
+Let's start with a common pattern: summarizing text.
+
+### The Workflow
+
+```yaml
+example_workflow:
+  SimpleLLM:
+    node_type: llm
+    event_triggers: [START]
+    prompt: "Summarize the following text in one sentence: {{ context.text }}"
+    output_field: summary
+    event_emissions:
+      - signal_name: SUMMARY_COMPLETE
+```
+
+### How It Works
+
+1. **Event Trigger**: The `START` signal triggers the `SimpleLLM` node.
+2. **Prompt Rendering**: The Jinja2 template `{{ context.text }}` is replaced with the actual text.
+3. **LLM Call**: The rendered prompt is sent to your LLM provider.
+4. **Output Storage**: The response is stored in `context.summary` (the `output_field`).
+5. **Signal Emission**: `SUMMARY_COMPLETE` is emitted to continue the workflow.
+
+### Key Concepts
+
+- **`prompt`**: A Jinja2 template. Use `{{ context.field }}` to inject context values.
+- **`output_field`**: Where the LLM response is stored in context.
+- **`event_emissions`**: Signals to emit after the LLM call completes.
+
+## Chaining LLM Nodes
+
+LLM nodes become powerful when chained together. Each node can use the output of the previous one.
+
+### The Workflow
+
+```yaml
+example_workflow:
+  Translator:
+    node_type: llm
+    event_triggers: [START]
+    prompt: "Translate the following to Spanish: {{ context.text }}"
+    output_field: spanish_text
+    event_emissions:
+      - signal_name: TRANSLATED
+
+  Summarizer:
+    node_type: llm
+    event_triggers: [TRANSLATED]
+    prompt: "Summarize this Spanish text: {{ context.spanish_text }}"
+    output_field: summary
+    event_emissions:
+      - signal_name: CHAIN_COMPLETE
+```
+
+### How It Works
+
+1. `START` triggers `Translator`
+2. Translator stores Spanish text in `context.spanish_text`
+3. Translator emits `TRANSLATED`
+4. `TRANSLATED` triggers `Summarizer`
+5. Summarizer reads `context.spanish_text` and stores summary
+6. Summarizer emits `CHAIN_COMPLETE`
+
+This pattern is incredibly useful for:
+- Multi-step processing pipelines
+- Translation → Analysis workflows
+- Extract → Transform → Load patterns
+
+## LLM Signal Selection (Resolution Step)
+
+When an LLM node has multiple signals with **conditions** (plain text, not Jinja), the LLM itself decides which signal to emit.
+
+### The Workflow
+
+```yaml
+example_workflow:
+  SentimentAnalyzer:
+    node_type: llm
+    event_triggers: [START]
+    prompt: "Analyze the sentiment of: {{ context.user_message }}"
+    output_field: analysis
+    event_emissions:
+      - signal_name: POSITIVE
+        condition: "The message expresses positive sentiment"
+      - signal_name: NEGATIVE
+        condition: "The message expresses negative sentiment"
+      - signal_name: NEUTRAL
+        condition: "The message is neutral or factual"
+```
+
+### How It Works
+
+1. The LLM analyzes the sentiment
+2. SOE sees multiple signals with plain-text conditions (no `{{ }}`)
+3. SOE asks the LLM: "Based on your analysis, which signal should be emitted?" using the conditions as descriptions
+4. The LLM returns the appropriate signal (POSITIVE, NEGATIVE, or NEUTRAL)
+
+This is called the **resolution step** - it lets the LLM make routing decisions based on its understanding.
+
+### Signal Emission Rules
+
+The `condition` field controls how signals are emitted:
+
+| Condition | Behavior |
+|-----------|----------|
+| **No condition** | Signal is always emitted |
+| **Plain text** | Semantic—LLM selects which signal to emit based on the description |
+| **Jinja template (`{{ }}`)** | Programmatic—evaluated against `context`, emits if truthy |
+
+**How SOE decides:**
+
+1. **No conditions**: All signals emit unconditionally after node execution
+2. **Has conditions**: SOE checks if they contain `{{ }}`:
+   - **Yes (Jinja)**: Evaluate expression against `context`—emit if result is truthy
+   - **No (plain text)**: Ask LLM to choose which signal best matches its output
+
+## Testing LLM Nodes
+
+In tests, we inject a stub function instead of calling a real LLM:
+
+```python
+def stub_llm(prompt: str, config: dict) -> str:
+    return "This is a predictable response."
+
+nodes["llm"] = create_llm_node_caller(backends, stub_llm, broadcast_signals_caller)
+```
+
+The stub knows the *contract* (prompt → response), not the implementation. This keeps tests fast and deterministic.
+
+## Next Steps
+
+Now that you understand LLM nodes, let's see how [Router Nodes](guide_03_router.md) connect your workflows together →