soe-ai 0.1.1__py3-none-any.whl → 0.1.3__py3-none-any.whl

soe/docs/primitives/backends.md

@@ -0,0 +1,281 @@
+
+# Backends: Pluggable Storage
+
+Backends are **dumb data retrievers**—they store and retrieve data, nothing more. All business logic lives in the orchestration engine, not in backends.
+
+---
+
+## The Principle: Dumb Data Retrievers
+
+Backends follow a simple principle:
+
+1. **Store data** when told to
+2. **Retrieve data** when asked
+3. **No business logic** — no validation, no transformation, no side effects
+
+This makes backends trivially swappable. Whether you use in-memory, local files, PostgreSQL, or DynamoDB, the orchestration engine works exactly the same.
+
+---
+
+## The 6 Backend Types
+
+| Backend | Protocol | Required | Purpose |
+|---------|----------|----------|---------|
+| `context` | `ContextBackend` | ✅ Yes | Execution state dictionary |
+| `workflow` | `WorkflowBackend` | ✅ Yes | Workflow definitions and current workflow name |
+| `telemetry` | `TelemetryBackend` | ❌ Optional | Logs and events for debugging |
+| `conversation_history` | `ConversationHistoryBackend` | ❌ Optional | LLM chat history per identity |
+| `context_schema` | `ContextSchemaBackend` | ❌ Optional | Type validation for outputs |
+| `identity` | `IdentityBackend` | ❌ Optional | System prompts for LLM identities |
+
+---
+
+## Quick Start: Built-in Backends
+
+SOE provides two factory functions for common use cases:
+
+```python
+from soe.local_backends import create_in_memory_backends, create_local_backends
+
+# For testing: everything in memory (lost on restart)
+backends = create_in_memory_backends()
+
+# For development: persisted to local files
+backends = create_local_backends(
+    context_storage_dir="./data/contexts",
+    workflow_storage_dir="./data/workflows",
+    telemetry_storage_dir="./data/telemetry",
+    conversation_history_storage_dir="./data/conversations",
+    context_schema_storage_dir="./data/schemas",
+    identity_storage_dir="./data/identities",
+)
+```
+
+---
+
+## Implementing Custom Backends
+
+Each backend is a Python Protocol (structural typing). Implement the required methods and you're done.
+
+### ContextBackend (Required)
+
+Stores execution context—the shared state dictionary.
+
+```python
+from typing import Dict, Any
+
+class MyContextBackend:
+    """Store contexts in your database."""
+
+    def save_context(self, id: str, context: Dict[str, Any]) -> None:
+        """Save context for an execution ID."""
+        # Your database write here
+        pass
+
+    def get_context(self, id: str) -> Dict[str, Any]:
+        """Get context for an execution ID."""
+        # Your database read here
+        return {}
+```
+
+### WorkflowBackend (Required)
+
+Stores workflow definitions and tracks current workflow.
+
+```python
+from typing import Dict, Any
+
+class MyWorkflowBackend:
+    """Store workflows in your database."""
+
+    def save_workflows_registry(self, id: str, workflows: Dict[str, Any]) -> None:
+        """Save all workflows for an execution."""
+        pass
+
+    def get_workflows_registry(self, id: str) -> Dict[str, Any]:
+        """Get all workflows for an execution."""
+        return {}
+
+    def save_current_workflow_name(self, id: str, name: str) -> None:
+        """Save current workflow name (for child transitions)."""
+        pass
+
+    def get_current_workflow_name(self, id: str) -> str:
+        """Get current workflow name."""
+        return ""
+```
+
+### TelemetryBackend (Optional)
+
+Logs events for debugging and observability.
+
+```python
+class MyTelemetryBackend:
+    """Log events to your observability stack."""
+
+    def log_event(self, execution_id: str, event_type: str, **event_data) -> None:
+        """Log an event. event_data contains timestamp and event-specific fields."""
+        pass
+```
+
+### ConversationHistoryBackend (Optional)
+
+Persists LLM conversation history per identity.
+
+```python
+from typing import List, Dict, Any
+
+class MyConversationHistoryBackend:
+    """Store chat history in your database."""
+
+    def get_conversation_history(self, identity: str) -> List[Dict[str, Any]]:
+        """Get conversation history. Returns list of {role, content} dicts."""
+        return []
+
+    def append_to_conversation_history(self, identity: str, entry: Dict[str, Any]) -> None:
+        """Append a single message to history."""
+        pass
+
+    def save_conversation_history(self, identity: str, history: List[Dict[str, Any]]) -> None:
+        """Replace entire history."""
+        pass
+
+    def delete_conversation_history(self, identity: str) -> None:
+        """Delete all history for an identity."""
+        pass
+```
+
+### ContextSchemaBackend (Optional)
+
+Stores output field type schemas for validation.
+
+```python
+from typing import Dict, Any, Optional
+
+class MyContextSchemaBackend:
+    """Store context schemas for output validation."""
+
+    def save_context_schema(self, execution_id: str, schema: Dict[str, Any]) -> None:
+        """Save context schema for an execution."""
+        pass
+
+    def get_context_schema(self, execution_id: str) -> Optional[Dict[str, Any]]:
+        """Get context schema. Returns None if not found."""
+        return None
+
+    def delete_context_schema(self, execution_id: str) -> bool:
+        """Delete context schema. Returns True if deleted."""
+        return False
+```
+
+### IdentityBackend (Optional)
+
+Stores LLM identity system prompts.
+
+```python
+from typing import Dict, Optional
+
+class MyIdentityBackend:
+    """Store identity definitions."""
+
+    def save_identities(self, execution_id: str, identities: Dict[str, str]) -> None:
+        """Save identity definitions. Format: {identity_name: system_prompt}"""
+        pass
+
+    def get_identities(self, execution_id: str) -> Optional[Dict[str, str]]:
+        """Get all identities for an execution."""
+        return None
+
+    def get_identity(self, execution_id: str, identity_name: str) -> Optional[str]:
+        """Get a specific identity's system prompt."""
+        return None
+
+    def delete_identities(self, execution_id: str) -> bool:
+        """Delete all identities for an execution."""
+        return False
+```
+
+---
+
+## Combining Into a Backends Container
+
+Pass your backends to orchestrate via a container object:
+
+```python
+class MyBackends:
+    def __init__(self):
+        self.context = MyContextBackend()
+        self.workflow = MyWorkflowBackend()
+        self.telemetry = MyTelemetryBackend()  # or None
+        self.conversation_history = MyConversationHistoryBackend()  # or None
+        self.context_schema = MyContextSchemaBackend()  # or None
+        self.identity = MyIdentityBackend()  # or None
+
+backends = MyBackends()
+```
+
+---
+
+## Database Recommendations
+
+### Single Database, Multiple Tables
+
+We recommend using **one database** with **separate tables** for each backend:
+
+```
+your_database/
+├── contexts          # ContextBackend
+├── workflows         # WorkflowBackend
+├── telemetry         # TelemetryBackend
+├── conversations     # ConversationHistoryBackend
+├── context_schemas   # ContextSchemaBackend
+└── identities        # IdentityBackend
+```
+
+This gives you:
+- **Transactional consistency** across backends
+- **Simpler deployment** (one connection string)
+- **Easier backups** (one database to backup)
+
+### Example: PostgreSQL Tables
+
+```sql
+CREATE TABLE contexts (
+    execution_id VARCHAR PRIMARY KEY,
+    context JSONB NOT NULL,
+    created_at TIMESTAMP DEFAULT NOW()
+);
+
+CREATE TABLE workflows (
+    execution_id VARCHAR PRIMARY KEY,
+    registry JSONB NOT NULL,
+    current_workflow VARCHAR NOT NULL
+);
+
+CREATE TABLE telemetry (
+    id SERIAL PRIMARY KEY,
+    execution_id VARCHAR NOT NULL,
+    event_type VARCHAR NOT NULL,
+    event_data JSONB,
+    timestamp TIMESTAMP DEFAULT NOW()
+);
+```
+
+---
+
+## Key Insight: Execution ID as Primary Key
+
+Most backends key data by `execution_id`:
+
+- **Context**: One context dict per execution
+- **Workflows**: Workflow registry per execution
+- **Telemetry**: Events indexed by execution
+
+For **conversation history**, **context schema**, and **identities**, data is keyed by `main_execution_id`—this allows child workflows to access parent's definitions.
+
+---
+
+## See Also
+
+- [Chapter 10: Infrastructure](../guide_10_infrastructure.md) — Full implementation examples
+- [Primitives Overview](primitives.md) — All 7 SOE primitives

soe/docs/primitives/context.md

@@ -0,0 +1,256 @@
+
+# Context: State Management with History
+
+Context is the shared state dictionary for an execution. Unlike simple key-value stores, SOE context maintains **history** for each field—every update is appended, not replaced.
+
+---
+
+## Context as History
+
+Each context field is stored as a **list** of values:
+
+```python
+# After three tool calls that update "result":
+context = {
+    "result": ["first", "second", "third"],  # History preserved
+    "user_id": ["alice"],  # Single value, still a list
+    "__operational__": {...}  # Internal fields not wrapped
+}
+```
+
+When you read a field:
+- **Reading normally** returns the **latest** value (last item)
+- **Reading with `| accumulated`** returns the **full history** (all items)
+
+---
+
+## Reading Context in Jinja
+
+### Get Latest Value (Default)
+
+
+```yaml
+prompt: |
+  User {{ context.user_id }} asked: {{ context.user_request }}
+
+  Previous result: {{ context.result }}
+```
+
+
+This returns the **most recent** value for each field.
+
+### Get Full History with `| accumulated`
+
+
+```yaml
+prompt: |
+  All results so far:
+  {% for r in context.result | accumulated %}
+  - {{ r }}
+  {% endfor %}
+
+condition: "{{ (context.result | accumulated) | length >= 3 }}"
+```
+
+
+Use `| accumulated` when you need:
+- All previous LLM responses
+- Aggregating results from multiple tool calls
+- Checking how many updates have occurred
+
+---
+
+## Context in Conditions
+
+Context is available in all condition expressions:
+
+
+```yaml
+event_emissions:
+  # Check latest value
+  - signal_name: HAS_RESULT
+    condition: "{{ context.result is defined }}"
+
+  # Check history length
+  - signal_name: ENOUGH_DATA
+    condition: "{{ (context.data | accumulated) | length >= 5 }}"
+
+  # Complex conditions
+  - signal_name: VALIDATED
+    condition: "{{ context.user_id and context.user_profile.verified }}"
+```
+
+
+---
+
+## Context in Prompts
+
+Full context is available in LLM/Agent prompts:
+
+
+```yaml
+prompt: |
+  You are helping user {{ context.user_id }}.
+
+  Their profile: {{ context.user_profile | tojson }}
+
+  Conversation so far:
+  {% for msg in context.messages | accumulated %}
+  {{ msg.role }}: {{ msg.content }}
+  {% endfor %}
+
+  Current request: {{ context.user_request }}
+```
+
+
+---
+
+## How Context is Updated
+
+### Tool Nodes
+
+Tool output is saved via `output_field`:
+
+```yaml
+ProcessData:
+  node_type: tool
+  tool_name: process
+  input_fields: [raw_data]
+  output_field: processed  # Tool return value saved here
+```
+
+Each execution **appends** to the field's history.
+
+### LLM/Agent Nodes
+
+LLM response is saved via `output_field`:
+
+
+```yaml
+Summarize:
+  node_type: llm
+  prompt: "Summarize: {{ context.data }}"
+  output_field: summary  # LLM response saved here
+```
+
+
+### Initial Context
+
+When starting orchestration:
+
+```python
+orchestrate(
+    config=workflow,
+    initial_workflow_name="my_workflow",
+    initial_signals=["START"],
+    initial_context={
+        "user_id": "alice",
+        "user_request": "Help me with X"
+    },
+    backends=backends,
+    broadcast_signals_caller=broadcast,
+)
+```
+
+Initial context fields are wrapped in lists automatically:
+```python
+# Internal representation:
+{"user_id": ["alice"], "user_request": ["Help me with X"]}
+```
+
+---
+
+## The `__operational__` Field
+
+Context includes a special `__operational__` field that tracks execution metadata:
+
+```python
+context["__operational__"] = {
+    "signals": ["START", "VALIDATED"],  # Signals emitted
+    "nodes": {"ValidateInput": 1},      # Node execution counts
+    "llm_calls": 3,                     # Total LLM calls
+    "tool_calls": 5,                    # Total tool calls
+    "errors": 0,                        # Error count
+    "main_execution_id": "abc-123"      # Root execution ID
+}
+```
+
+**Important**: Fields starting with `__` are **not** wrapped in lists—they store direct values.
+
+---
+
+## Practical Examples
+
+### Aggregating Multiple Results
+
+
+```yaml
+# Fan-out pattern: multiple child workflows write to same field
+AggregateResults:
+  node_type: router
+  event_triggers: [CHILD_COMPLETE]
+  event_emissions:
+    - signal_name: ALL_DONE
+      condition: "{{ (context.child_results | accumulated) | length >= 3 }}"
+```
+
+
+### Retry with History
+
+
+```yaml
+RetryLogic:
+  node_type: router
+  event_triggers: [FAILED]
+  event_emissions:
+    - signal_name: RETRY
+      condition: "{{ (context.attempts | accumulated) | length < 3 }}"
+    - signal_name: GIVE_UP
+      condition: "{{ (context.attempts | accumulated) | length >= 3 }}"
+```
+
+
+### Building Context Over Time
+
+
+```yaml
+# Each step adds to enrichment
+EnrichStep1:
+  node_type: tool
+  tool_name: get_user_profile
+  input_fields: [user_id]
+  output_field: enrichment
+
+EnrichStep2:
+  node_type: tool
+  tool_name: get_purchase_history
+  input_fields: [user_id]
+  output_field: enrichment  # Same field, appends
+
+# Later: context.enrichment | accumulated = [profile, history]
+```
+
+
+---
+
+## Context Backend
+
+Context is persisted via `ContextBackend`:
+
+```python
+# Save
+backends.context.save_context(execution_id, context)
+
+# Retrieve
+context = backends.context.get_context(execution_id)
+```
+
+See [Backends](backends.md) for implementation details.
+
+---
+
+## See Also
+
+- [Backends](backends.md) — How context is persisted
+- [Signals](signals.md) — How signals trigger nodes based on context
+- [Primitives Overview](primitives.md) — All 7 SOE primitives