soe-ai 0.1.1__py3-none-any.whl → 0.1.3__py3-none-any.whl

soe/docs/advanced_patterns/guide_inheritance.md

@@ -0,0 +1,435 @@
+
+# Advanced Pattern: Configuration and Context Inheritance
+
+This guide covers inheritance patterns for workflow chaining and multi-phase execution using SOE's `inherit_config_from_id` and `inherit_context_from_id` parameters.
+
+## The Hidden Power: Execution Chaining
+
+Until now, each `orchestrate()` call required a full config. However, SOE supports **inheritance** between executions, enabling:
+
+- **Workflow Chaining**: Run Phase 2 using Phase 1's config (workflows, identities, schema)
+- **Context Sharing**: Continue with existing context data from a previous execution
+- **Operational Reset**: Always start fresh operational counters even when inheriting context
+
+This architecture enables powerful multi-phase patterns without re-sending configuration.
+
+---
+
+## The Two Inheritance Parameters
+
+| Parameter | Inherits | Behavior |
+|-----------|----------|----------|
+| `inherit_config_from_id` | Workflows, identities, context_schema | Config becomes optional |
+| `inherit_context_from_id` | Context fields (excludes `__operational__`) | **Always** resets operational state |
+
+**Key Rule**: At least one of `config` or `inherit_config_from_id` must be provided.
+
+---
+
+## Pattern 1: Config Inheritance
+
+Inherit workflows, identities, and context_schema from an existing execution.
+
+### The Workflow
+
+```yaml
+workflows:
+  main_workflow:
+    ProcessData:
+      node_type: tool
+      event_triggers: [START]
+      tool_name: process
+      context_parameter_field: input_data
+      output_field: result
+      event_emissions:
+        - signal_name: PROCESSED
+
+    Complete:
+      node_type: router
+      event_triggers: [PROCESSED]
+      event_emissions:
+        - signal_name: COMPLETE
+
+identities:
+  assistant: "You are a helpful assistant."
+  analyst: "You are a data analyst."
+
+context_schema:
+  input_data:
+    type: object
+    description: "Input data to process"
+  result:
+    type: object
+    description: "Processing result"
+```
+
+### Usage: First Execution
+
+```python
+# First execution - establishes config
+first_id = orchestrate(
+    config=workflow_config,
+    initial_workflow_name="main_workflow",
+    initial_signals=["START"],
+    initial_context={"input_data": {"value": "test1"}},
+    backends=backends,
+    broadcast_signals_caller=broadcast,
+)
+```
+
+### Usage: Inherited Execution
+
+```python
+# Second execution - inherits config, no need to resend
+second_id = orchestrate(
+    config=None,  # Not needed!
+    initial_workflow_name="main_workflow",
+    initial_signals=["START"],
+    initial_context={"input_data": {"value": "test2"}},
+    backends=backends,
+    broadcast_signals_caller=broadcast,
+    inherit_config_from_id=first_id,  # <-- Inherit from first
+)
+```
+
+**What gets inherited:**
+- Workflows registry (all workflow definitions)
+- Identities (LLM system prompts)
+- Context schema (field definitions)
+
+---
+
+## Pattern 2: Context Inheritance
+
+Inherit context from an existing execution, but **always** reset operational state.
+
+### The Workflow
+
+```yaml
+workflows:
+  main_workflow:
+    Step1:
+      node_type: tool
+      event_triggers: [START]
+      tool_name: step1_tool
+      output_field: step1_result
+      event_emissions:
+        - signal_name: STEP1_DONE
+
+    Step2:
+      node_type: tool
+      event_triggers: [STEP1_DONE]
+      tool_name: step2_tool
+      context_parameter_field: step1_result
+      output_field: step2_result
+      event_emissions:
+        - signal_name: COMPLETE
+
+  retry_workflow:
+    RetryStep2:
+      node_type: tool
+      event_triggers: [START]
+      tool_name: step2_tool_v2
+      context_parameter_field: step1_result
+      output_field: step2_result
+      event_emissions:
+        - signal_name: RETRY_COMPLETE
+```
+
+### Usage: Retry with Existing Context
+
+```python
+# First execution - builds up context
+first_id = orchestrate(
+    config=workflow_config,
+    initial_workflow_name="main_workflow",
+    initial_signals=["START"],
+    initial_context={},
+    backends=backends,
+    broadcast_signals_caller=broadcast,
+)
+
+# Second execution - inherits context, runs retry workflow
+second_id = orchestrate(
+    config=None,
+    initial_workflow_name="retry_workflow",
+    initial_signals=["START"],
+    initial_context={},
+    backends=backends,
+    broadcast_signals_caller=broadcast,
+    inherit_config_from_id=first_id,
+    inherit_context_from_id=first_id,  # <-- Inherit context too
+)
+```
+
+**Critical**: `inherit_context_from_id` **always** resets `__operational__`:
+- Fresh signal list
+- Fresh counters (llm_calls, tool_calls, errors)
+- New main_execution_id
+
+This ensures each execution has its own operational tracking.
+
+---
+
+## Pattern 3: Multi-Phase Execution
+
+Combine both inheritance types for complex, multi-phase workflows.
+
+### The Workflow
+
+```yaml
+workflows:
+  phase1_workflow:
+    Analyze:
+      node_type: llm
+      event_triggers: [START]
+      identity: analyst
+      prompt: "Analyze this data: &#123;&#123; context.raw_data &#125;&#125;"
+      output_field: analysis
+      event_emissions:
+        - signal_name: ANALYSIS_DONE
+
+    Validate:
+      node_type: tool
+      event_triggers: [ANALYSIS_DONE]
+      tool_name: validate_analysis
+      context_parameter_field: analysis
+      output_field: validated_analysis
+      event_emissions:
+        - signal_name: PHASE1_COMPLETE
+
+  phase2_workflow:
+    Generate:
+      node_type: llm
+      event_triggers: [START]
+      identity: writer
+      prompt: "Based on analysis: &#123;&#123; context.validated_analysis &#125;&#125;, generate report"
+      output_field: report
+      event_emissions:
+        - signal_name: REPORT_DONE
+
+    Finalize:
+      node_type: router
+      event_triggers: [REPORT_DONE]
+      event_emissions:
+        - signal_name: PHASE2_COMPLETE
+
+identities:
+  analyst: "You are a data analyst. Focus on insights and patterns."
+  writer: "You are a technical writer. Be clear and professional."
+
+context_schema:
+  raw_data:
+    type: string
+    description: "Raw data to analyze"
+  analysis:
+    type: object
+    description: "Analysis results"
+  validated_analysis:
+    type: object
+    description: "Validated analysis"
+  report:
+    type: string
+    description: "Generated report"
+```
+
+### Usage: Phase 1 → Phase 2
+
+```python
+# Phase 1 - Analysis
+phase1_id = orchestrate(
+    config=multi_phase_config,
+    initial_workflow_name="phase1_workflow",
+    initial_signals=["START"],
+    initial_context={"raw_data": "sample data"},
+    backends=backends,
+    broadcast_signals_caller=broadcast,
+)
+
+# Phase 2 - Generation (inherits everything)
+phase2_id = orchestrate(
+    config=None,
+    initial_workflow_name="phase2_workflow",
+    initial_signals=["START"],
+    initial_context={},  # Uses Phase 1's context
+    backends=backends,
+    broadcast_signals_caller=broadcast,
+    inherit_config_from_id=phase1_id,
+    inherit_context_from_id=phase1_id,
+)
+```
+
+**Benefits:**
+- Phase 2 has access to Phase 1's `validated_analysis`
+- Phase 2 uses same identities (`analyst`, `writer`)
+- Phase 2 has fresh operational state for clean tracking
+
+---
+
+## Pattern 4: Identity Sharing Across Conversations
+
+Inherit config to share identities across multiple conversation executions.
+
+### The Workflow
+
+```yaml
+workflows:
+  conversation_workflow:
+    FirstMessage:
+      node_type: llm
+      event_triggers: [START]
+      identity: assistant
+      prompt: "User says: &#123;&#123; context.user_message &#125;&#125;"
+      output_field: assistant_response
+      event_emissions:
+        - signal_name: RESPONDED
+
+    Complete:
+      node_type: router
+      event_triggers: [RESPONDED]
+      event_emissions:
+        - signal_name: CONVERSATION_DONE
+
+  followup_workflow:
+    FollowUp:
+      node_type: llm
+      event_triggers: [START]
+      identity: assistant
+      prompt: "User follows up: &#123;&#123; context.followup_message &#125;&#125;"
+      output_field: followup_response
+      event_emissions:
+        - signal_name: FOLLOWUP_DONE
+
+identities:
+  assistant: "You are a helpful assistant. Be concise and friendly."
+```
+
+### Usage: Continued Conversation
+
+```python
+# First conversation
+first_id = orchestrate(
+    config=conversation_config,
+    initial_workflow_name="conversation_workflow",
+    initial_signals=["START"],
+    initial_context={"user_message": "Hello"},
+    backends=backends,
+    broadcast_signals_caller=broadcast,
+)
+
+# Follow-up conversation (same identity definitions)
+second_id = orchestrate(
+    config=None,
+    initial_workflow_name="followup_workflow",
+    initial_signals=["START"],
+    initial_context={"followup_message": "Tell me more"},
+    backends=backends,
+    broadcast_signals_caller=broadcast,
+    inherit_config_from_id=first_id,
+)
+```
+
+---
+
+## Override Behavior
+
+### Config Override
+
+If both `config` and `inherit_config_from_id` are provided, `config` takes precedence:
+
+```python
+# Inherits, then overrides with new config
+orchestrate(
+    config=new_config,  # <-- This wins
+    initial_workflow_name="workflow",
+    initial_signals=["START"],
+    initial_context={},
+    backends=backends,
+    broadcast_signals_caller=broadcast,
+    inherit_config_from_id=old_id,  # <-- Loaded first, then overridden
+)
+```
+
+### Context Append (History Preservation)
+
+`initial_context` fields are **appended** to inherited context history using `set_field()`. This preserves the full history chain:
+
+```python
+# Inherits context, appends new values to history
+orchestrate(
+    config=None,
+    initial_workflow_name="retry_workflow",
+    initial_signals=["START"],
+    initial_context={"user_input": "new message"},  # <-- Appends to history
+    backends=backends,
+    broadcast_signals_caller=broadcast,
+    inherit_config_from_id=first_id,
+    inherit_context_from_id=first_id,
+)
+
+# Result: context["user_input"] = ["old message", "new message"]
+# Reading with get_field() returns "new message" (latest)
+# Reading with get_accumulated() returns ["old message", "new message"] (full history)
+```
+
+---
+
+## Validation Rules
+
+| Scenario | Result |
+|----------|--------|
+| `config=None`, `inherit_config_from_id=None` | **Error**: Must provide one |
+| `inherit_config_from_id="nonexistent"` | **Error**: Execution not found |
+| `inherit_context_from_id="nonexistent"` | **Error**: Context not found |
+| `inherit_context_from_id` without `inherit_config_from_id` | Valid if `config` provided |
+
+---
+
+## When to Use Each Pattern
+
+| Use Case | Pattern |
+|----------|---------|
+| Run same workflow with different inputs | Config inheritance only |
+| Retry failed step with existing context | Config + context inheritance |
+| Multi-phase pipelines | Config + context inheritance |
+| Share identities across API calls | Config inheritance only |
+| Fresh start with same config | Config inheritance only (no context) |
+| Continue with old version of workflow | Config inheritance from old execution |
+| Continue with evolved workflow | Config inheritance from evolved execution |
+
+### Version Management with Inheritance
+
+Inheritance enables interesting version management patterns:
+
+1. **Continue with Old Version**: If you have a running process and update your workflow definition, existing executions continue with their captured config. New executions use the new definition.
+
+2. **Continue with Evolved Version**: If a workflow self-evolves (using `soe_inject_workflow` or `inject_node`), you can start new executions that inherit the evolved config:
+
+```python
+# Original execution - workflow may have self-evolved
+evolved_id = orchestrate(
+    config=original_config,
+    initial_workflow_name="self_evolving_workflow",
+    initial_signals=["START"],
+    ...
+)
+
+# New execution inherits the evolved config (including injected nodes)
+new_execution = orchestrate(
+    config=None,
+    initial_workflow_name="self_evolving_workflow",
+    initial_signals=["START"],
+    inherit_config_from_id=evolved_id,  # Gets evolved workflow
+    ...
+)
+```
+
+---
+
+## Best Practices
+
+1. **Always reset operational for retries**: Use `inherit_context_from_id` to get fresh counters
+2. **Store execution IDs**: Keep track of execution IDs for chaining
+3. **Explicit is better**: When in doubt, provide full config
+4. **Test inheritance chains**: Multi-phase tests catch subtle bugs
+5. **Don't assume context shape**: Validate inherited context before use

soe/docs/advanced_patterns/hybrid_intelligence.md

@@ -0,0 +1,237 @@
+# SOE Advanced Patterns: Hybrid Intelligence
+
+## Introduction
+
+The power of SOE comes from mixing **deterministic code** with **probabilistic AI**. Routers are 100% reliable. LLMs are creative. Combine them for systems that are both flexible and safe.
+
+This is the "Centaur" model — human-like intelligence bounded by machine precision.
+
+### Why Hybrid Intelligence Matters
+
+Hybrid patterns are one of SOE's biggest selling points:
+
+1. **Low-Latency User Endpoints**: Fast, deterministic routers handle user requests immediately
+2. **Async AI Processing**: LLM-driven workflows run in the background for complex analysis
+3. **Self-Evolving Context**: Deterministic workflows can be enhanced by AI that modifies context
+
+This enables architectures where:
+- Users get instant responses from deterministic logic
+- AI enriches/improves the system asynchronously
+- Complex decisions combine both in a single workflow
+
+---
+
+## Pattern 1: Safety Rails
+
+Validate inputs and outputs around AI processing.
+
+### The Workflow
+
+```yaml
+hybrid_workflow:
+  ValidateInput:
+    node_type: router
+    event_triggers: [START]
+    event_emissions:
+      - signal_name: INPUT_VALID
+        condition: "{{ context.user_input is defined and context.user_input | length > 0 }}"
+      - signal_name: INPUT_INVALID
+        condition: "{{ context.user_input is not defined or context.user_input | length == 0 }}"
+
+  ProcessInput:
+    node_type: tool
+    event_triggers: [INPUT_VALID]
+    tool_name: process_input
+    context_parameter_field: process_params
+    output_field: processed_result
+    event_emissions:
+      - signal_name: PROCESSED
+
+  ValidateOutput:
+    node_type: router
+    event_triggers: [PROCESSED]
+    event_emissions:
+      - signal_name: OUTPUT_VALID
+        condition: "{{ context.processed_result is defined and context.processed_result.valid == true }}"
+      - signal_name: OUTPUT_INVALID
+        condition: "{{ context.processed_result is not defined or context.processed_result.valid != true }}"
+
+  Complete:
+    node_type: router
+    event_triggers: [OUTPUT_VALID]
+    event_emissions:
+      - signal_name: DONE
+
+  HandleError:
+    node_type: router
+    event_triggers: [INPUT_INVALID, OUTPUT_INVALID, PROCESS_FAILED]
+    event_emissions:
+      - signal_name: ERROR
+```
+
+### How It Works
+
+1. **ValidateInput** (Router) — Checks input meets requirements
+2. **ProcessInput** (Tool/LLM) — Does the actual work
+3. **ValidateOutput** (Router) — Ensures output meets requirements
+4. **HandleError** (Router) — Catches any failures
+
+**The AI never runs with bad input. The AI output never escapes validation.**
+
+### The Flow
+
+```
+START
+  ├─ INPUT_VALID ──→ ProcessInput ──→ PROCESSED
+  │                                      ├─ OUTPUT_VALID ──→ DONE
+  │                                      └─ OUTPUT_INVALID ──→ ERROR
+  └─ INPUT_INVALID ──→ ERROR
+```
+
+---
+
+## Pattern 2: LLM with Safety Rails
+
+Same pattern but with an LLM doing the processing.
+
+### The Workflow
+
+```yaml
+safety_workflow:
+  ValidateInput:
+    node_type: router
+    event_triggers: [START]
+    event_emissions:
+      - signal_name: INPUT_VALID
+        condition: "{{ context.amount is defined and context.amount > 0 }}"
+      - signal_name: INPUT_INVALID
+        condition: "&#123;&#123; context.amount is not defined or context.amount <= 0 &#125;&#125;"
+
+  GenerateResponse:
+    node_type: llm
+    event_triggers: [INPUT_VALID]
+    prompt: |
+      Generate a professional message about a transaction of $&#123;&#123; context.amount &#125;&#125;.
+      Keep it brief and professional.
+    output_field: generated_message
+    event_emissions:
+      - signal_name: MESSAGE_GENERATED
+
+  ValidateOutput:
+    node_type: router
+    event_triggers: [MESSAGE_GENERATED]
+    event_emissions:
+      - signal_name: OUTPUT_SAFE
+        condition: "&#123;&#123; context.generated_message is defined and context.generated_message | length < 500 &#125;&#125;"
+      - signal_name: OUTPUT_UNSAFE
+        condition: "&#123;&#123; context.generated_message is not defined or context.generated_message | length >= 500 &#125;&#125;"
+
+  HandleInputError:
+    node_type: router
+    event_triggers: [INPUT_INVALID]
+    event_emissions:
+      - signal_name: ERROR
+
+  Complete:
+    node_type: router
+    event_triggers: [OUTPUT_SAFE]
+    event_emissions:
+      - signal_name: DONE
+
+  HandleOutputError:
+    node_type: router
+    event_triggers: [OUTPUT_UNSAFE]
+    event_emissions:
+      - signal_name: ERROR
+```
+
+### How It Works
+
+1. Router validates `amount > 0` (deterministic)
+2. LLM generates professional message (creative)
+3. Router validates output length < 500 (deterministic)
+
+**Enterprise use case**: Financial messages that must be professional but also compliant.
+
+---
+
+## Pattern 3: Retry with Validation
+
+LLM generates output, router validates, retry if invalid.
+
+### The Workflow
+
+```yaml
+retry_workflow:
+  Start:
+    node_type: router
+    event_triggers: [START]
+    event_emissions:
+      - signal_name: ATTEMPT
+
+  Generate:
+    node_type: llm
+    event_triggers: [ATTEMPT]
+    prompt: "Generate a valid JSON object with fields: name, age"
+    output_field: llm_output
+    event_emissions:
+      - signal_name: GENERATED
+
+  Validate:
+    node_type: router
+    event_triggers: [GENERATED]
+    event_emissions:
+      - signal_name: VALID_JSON
+        condition: "&#123;&#123; context.llm_output is mapping &#125;&#125;"
+      - signal_name: INVALID_JSON
+        condition: "&#123;&#123; context.llm_output is not mapping &#125;&#125;"
+
+  IncrementRetry:
+    node_type: router
+    event_triggers: [INVALID_JSON]
+    event_emissions:
+      - signal_name: ATTEMPT
+        condition: "&#123;&#123; context.retry_count | default(0) < 3 &#125;&#125;"
+      - signal_name: MAX_RETRIES
+        condition: "&#123;&#123; context.retry_count | default(0) >= 3 &#125;&#125;"
+
+  Complete:
+    node_type: router
+    event_triggers: [VALID_JSON]
+    event_emissions:
+      - signal_name: DONE
+```
+
+### How It Works
+
+1. LLM generates JSON
+2. Router checks if output is valid mapping
+3. If invalid, increment retry counter and loop back
+4. If max retries exceeded, emit `MAX_RETRIES`
+
+**Key Insight**: The retry logic is deterministic. Only the generation is probabilistic. You control the boundaries.
+
+---
+
+## Why Hybrid Matters
+
+| Pure LLM | Pure Code | Hybrid (SOE) |
+|----------|-----------|--------------|
+| Creative but unreliable | Reliable but rigid | Creative AND reliable |
+| "Hope it follows rules" | "Can't adapt" | "Enforce rules, allow creativity" |
+
+---
+
+## Best Practices
+
+1. **Validate early** — Check inputs before any AI processing
+2. **Validate late** — Check outputs before any downstream effects
+3. **Bound retries** — Never let loops run forever
+4. **Log everything** — Use telemetry backend to track decisions
+
+---
+
+## Related Patterns
+
+- [Swarm Intelligence](swarm_intelligence.md) — Multiple agents coordinating
+- [Self-Evolving Workflows](self_evolving_workflows.md) — Agents that modify their own workflows