soe-ai 0.1.1__py3-none-any.whl → 0.1.2__py3-none-any.whl

soe/docs/guide_10_infrastructure.md

@@ -0,0 +1,427 @@
+# SOE Guide: Chapter 10 - Custom Infrastructure
+
+## Introduction
+
+SOE is **infrastructure agnostic** by design. The core orchestration engine doesn't care *where* your data lives or *how* signals are broadcast—it only requires that you provide implementations that follow the defined protocols.
+
+This chapter covers two distinct infrastructure concerns:
+
+1. **Persistent Infrastructure (Backends)** — Where data is stored
+2. **Communication Infrastructure (Callers)** — How signals are broadcast
+
+---
+
+## Part 1: Persistent Infrastructure (Backends)
+
+### What Are Backends?
+
+Backends are the **persistence layer** of SOE. They store:
+
+| Backend | Purpose | Example Use Cases |
+|---------|---------|-------------------|
+| `context` | Workflow execution state | DynamoDB, Redis, PostgreSQL |
+| `workflow` | Workflow definitions | S3, local files, database |
+| `telemetry` | Logs and events | Datadog, CloudWatch, Elasticsearch |
+| `conversation_history` | LLM chat history | MongoDB, PostgreSQL, text files |
+| `schema` | Context validation schemas | Database, config files |
+
+### Built-in Backends
+
+SOE ships with two backend implementations for immediate use:
+
+#### In-Memory Backends
+```python
+from soe.local_backends import create_in_memory_backends
+
+backends = create_in_memory_backends()
+```
+- **Use case**: Unit tests, prototyping
+- **Pros**: Fast, no setup required
+- **Cons**: Data lost on process exit, not distributable
+
+#### Local File Backends
+```python
+from soe.local_backends import create_local_backends
+
+backends = create_local_backends(
+    context_storage_dir="./orchestration_data/contexts",
+    workflow_storage_dir="./orchestration_data/workflows",
+    telemetry_storage_dir="./orchestration_data/telemetry",  # Optional
+    conversation_history_storage_dir="./orchestration_data/conversations",  # Optional
+    schema_storage_dir="./orchestration_data/schemas",  # Optional
+)
+```
+- **Use case**: Local development, debugging, simple deployments
+- **Pros**: Persistent, human-readable (JSON files)
+- **Cons**: Not distributable, no concurrent access
+
+### The Backend Protocols
+
+Each backend type follows a Python `Protocol` (structural typing). Here are the required interfaces:
+
+#### ContextBackend Protocol
+```python
+class ContextBackend(Protocol):
+    def save_context(self, id: str, context: dict) -> None: ...
+    def get_context(self, id: str) -> dict: ...
+```
+
+#### WorkflowBackend Protocol
+```python
+class WorkflowBackend(Protocol):
+    def save_workflows_registry(self, id: str, workflows: Any) -> None: ...
+    def get_workflows_registry(self, id: str) -> Any: ...
+    def save_current_workflow_name(self, id: str, name: str) -> None: ...
+    def get_current_workflow_name(self, id: str) -> str: ...
+```
+
+#### TelemetryBackend Protocol (Optional)
+```python
+class TelemetryBackend(Protocol):
+    def log_event(self, execution_id: str, event_type: str, **event_data) -> None: ...
+```
+
+#### ConversationHistoryBackend Protocol (Optional)
+```python
+class ConversationHistoryBackend(Protocol):
+    def get_conversation_history(self, identity: str) -> List[Dict[str, Any]]: ...
+    def append_to_conversation_history(self, identity: str, entry: Dict[str, Any]) -> None: ...
+    def save_conversation_history(self, identity: str, history: List[Dict[str, Any]]) -> None: ...
+    def delete_conversation_history(self, identity: str) -> None: ...
+```
+
+#### ContextSchemaBackend Protocol (Optional)
+```python
+class ContextSchemaBackend(Protocol):
+    def save_context_schema(self, execution_id: str, schema: Dict[str, Any]) -> None: ...
+    def get_context_schema(self, execution_id: str) -> Optional[Dict[str, Any]]: ...
+    def get_field_schema(self, execution_id: str, field_name: str) -> Optional[Dict[str, Any]]: ...
+    def delete_context_schema(self, execution_id: str) -> bool: ...
+```
+
+**Note**: Context schemas are keyed by `execution_id` (specifically `main_execution_id`), not workflow name. This allows child workflows to access the parent's schema definitions.
+
+#### IdentityBackend Protocol (Optional)
+```python
+class IdentityBackend(Protocol):
+    def save_identities(self, execution_id: str, identities: Dict[str, str]) -> None: ...
+    def get_identities(self, execution_id: str) -> Optional[Dict[str, str]]: ...
+    def get_identity(self, execution_id: str, identity_name: str) -> Optional[str]: ...
+    def delete_identities(self, execution_id: str) -> bool: ...
+```
+
+**Note**: Identities are simple mappings of `identity_name` → `system_prompt` (string). Like context schemas, they're keyed by `main_execution_id` for child workflow access.
+
+### Database Recommendations
+
+For production deployments, we recommend using **the same database** for all config-related backends:
+
+| Backend | Recommended Storage | Why |
+|---------|---------------------|-----|
+| `context` | Same DB, separate table | Simplifies connection management |
+| `workflow` | Same DB, separate table | Allows atomic config updates |
+| `context_schema` | Same DB, separate table | Related to workflows |
+| `identity` | Same DB, separate table | Related to workflows |
+| `conversation_history` | Same DB, separate table | Per-execution keying |
+| `telemetry` | Can be separate | High volume, different access patterns |
+
+The backend methods handle table creation. Using one database (with separate tables) is simpler to manage than multiple databases.
+
+### Creating Custom Backends
+
+#### Example: PostgreSQL Context Backend
+
+```python
+import psycopg2
+import json
+from typing import Dict, Any
+
+class PostgresContextBackend:
+    """PostgreSQL-based context storage"""
+
+    def __init__(self, connection_string: str):
+        self.conn = psycopg2.connect(connection_string)
+        self._ensure_table()
+
+    def _ensure_table(self):
+        with self.conn.cursor() as cur:
+            cur.execute("""
+                CREATE TABLE IF NOT EXISTS soe_context (
+                    execution_id VARCHAR(255) PRIMARY KEY,
+                    context JSONB NOT NULL,
+                    updated_at TIMESTAMP DEFAULT NOW()
+                )
+            """)
+        self.conn.commit()
+
+    def save_context(self, id: str, context: Dict[str, Any]) -> None:
+        with self.conn.cursor() as cur:
+            cur.execute("""
+                INSERT INTO soe_context (execution_id, context)
+                VALUES (%s, %s)
+                ON CONFLICT (execution_id)
+                DO UPDATE SET context = %s, updated_at = NOW()
+            """, (id, json.dumps(context), json.dumps(context)))
+        self.conn.commit()
+
+    def get_context(self, id: str) -> Dict[str, Any]:
+        with self.conn.cursor() as cur:
+            cur.execute(
+                "SELECT context FROM soe_context WHERE execution_id = %s",
+                (id,)
+            )
+            row = cur.fetchone()
+            return row[0] if row else {}
+
+    def cleanup_all(self) -> None:
+        with self.conn.cursor() as cur:
+            cur.execute("DELETE FROM soe_context")
+        self.conn.commit()
+```
+
+#### Example: DynamoDB Context Backend
+
+```python
+import boto3
+import json
+from typing import Dict, Any
+
+class DynamoDBContextBackend:
+    """AWS DynamoDB-based context storage"""
+
+    def __init__(self, table_name: str, region: str = "us-east-1"):
+        self.dynamodb = boto3.resource('dynamodb', region_name=region)
+        self.table = self.dynamodb.Table(table_name)
+
+    def save_context(self, id: str, context: Dict[str, Any]) -> None:
+        self.table.put_item(Item={
+            'execution_id': id,
+            'context': json.dumps(context)
+        })
+
+    def get_context(self, id: str) -> Dict[str, Any]:
+        response = self.table.get_item(Key={'execution_id': id})
+        if 'Item' in response:
+            return json.loads(response['Item']['context'])
+        return {}
+
+    def cleanup_all(self) -> None:
+        # Scan and delete all items (use with caution in production)
+        scan = self.table.scan()
+        with self.table.batch_writer() as batch:
+            for item in scan['Items']:
+                batch.delete_item(Key={'execution_id': item['execution_id']})
+```
+
+#### Example: Datadog Telemetry Backend
+
+```python
+from datadog import initialize, statsd
+from typing import Any
+
+class DatadogTelemetryBackend:
+    """Datadog-based telemetry backend"""
+
+    def __init__(self, api_key: str, app_key: str):
+        initialize(api_key=api_key, app_key=app_key)
+
+    def log_event(self, execution_id: str, event_type: str, **event_data) -> None:
+        # Send as a custom metric
+        statsd.event(
+            title=f"SOE: {event_type}",
+            text=f"Execution: {execution_id}",
+            tags=[
+                f"execution_id:{execution_id}",
+                f"event_type:{event_type}",
+                *[f"{k}:{v}" for k, v in event_data.items()]
+            ]
+        )
+
+    def cleanup_all(self) -> None:
+        # Datadog events are immutable; no cleanup needed
+        pass
+```
+
+### Assembling Custom Backends
+
+Create a `Backends` container with your custom implementations:
+
+```python
+from soe.local_backends.factory import LocalBackends
+
+# Mix and match backends based on your infrastructure
+backends = LocalBackends(
+    context_backend=PostgresContextBackend("postgresql://..."),
+    workflow_backend=S3WorkflowBackend("my-bucket"),
+    telemetry_backend=DatadogTelemetryBackend(api_key="...", app_key="..."),
+    conversation_history_backend=MongoConversationBackend("mongodb://..."),
+    schema_backend=None,  # Optional
+)
+```
+
+---
+
+## Part 2: Communication Infrastructure (Callers)
+
+### What Are Callers?
+
+Callers are the **communication layer** of SOE. The most important is `broadcast_signals_caller`, which controls how signals propagate through the system.
+
+### The Default Pattern (In-Process)
+
+```python
+from soe import broadcast_signals
+
+def broadcast_signals_caller(id: str, signals: List[str]) -> None:
+    broadcast_signals(id, signals, nodes, backends)
+```
+
+This is synchronous and in-process—perfect for testing and simple deployments.
+
+### Distributed Callers
+
+For production systems, you might want signals to trigger external services.
+
+#### Example: AWS Lambda Caller
+
+```python
+import boto3
+import json
+
+def create_lambda_caller(function_name: str, region: str = "us-east-1"):
+    """Create a caller that invokes AWS Lambda for signal broadcasting"""
+    lambda_client = boto3.client('lambda', region_name=region)
+
+    def broadcast_signals_caller(id: str, signals: List[str]) -> None:
+        payload = {
+            'execution_id': id,
+            'signals': signals
+        }
+        lambda_client.invoke(
+            FunctionName=function_name,
+            InvocationType='Event',  # Async invocation
+            Payload=json.dumps(payload)
+        )
+
+    return broadcast_signals_caller
+```
+
+#### Example: HTTP Webhook Caller
+
+```python
+import requests
+
+def create_http_caller(webhook_url: str, auth_token: str = None):
+    """Create a caller that sends signals via HTTP POST"""
+    headers = {'Content-Type': 'application/json'}
+    if auth_token:
+        headers['Authorization'] = f'Bearer {auth_token}'
+
+    def broadcast_signals_caller(id: str, signals: List[str]) -> None:
+        payload = {
+            'execution_id': id,
+            'signals': signals
+        }
+        response = requests.post(webhook_url, json=payload, headers=headers)
+        response.raise_for_status()
+
+    return broadcast_signals_caller
+```
+
+#### Example: Message Queue Caller (SQS)
+
+```python
+import boto3
+import json
+
+def create_sqs_caller(queue_url: str, region: str = "us-east-1"):
+    """Create a caller that sends signals to an SQS queue"""
+    sqs = boto3.client('sqs', region_name=region)
+
+    def broadcast_signals_caller(id: str, signals: List[str]) -> None:
+        message = {
+            'execution_id': id,
+            'signals': signals
+        }
+        sqs.send_message(
+            QueueUrl=queue_url,
+            MessageBody=json.dumps(message)
+        )
+
+    return broadcast_signals_caller
+```
+
+---
+
+## Part 3: The LLM Caller
+
+### SOE Doesn't Provide an LLM
+
+Unlike some frameworks that bundle specific LLM providers, SOE is **LLM agnostic**. You provide a `call_llm` function that wraps your chosen provider.
+
+### The CallLlm Protocol
+
+```python
+class CallLlm(Protocol):
+    def __call__(self, prompt: str, config: Dict[str, Any]) -> str:
+        ...
+```
+
+### Example: OpenAI Caller
+
+```python
+import openai
+
+def create_openai_caller(api_key: str, default_model: str = "gpt-4o"):
+    """Create an LLM caller for OpenAI API"""
+    client = openai.OpenAI(api_key=api_key)
+
+    def call_llm(prompt: str, config: dict) -> str:
+        model = config.get("model", default_model)
+        temperature = config.get("temperature", 1.0)
+
+        response = client.chat.completions.create(
+            model=model,
+            temperature=temperature,
+            messages=[{"role": "user", "content": prompt}]
+        )
+        return response.choices[0].message.content
+
+    return call_llm
+```
+
+### Example: Test Stub Caller
+
+```python
+def create_stub_caller(responses: dict = None):
+    """Create a stub caller for testing"""
+    responses = responses or {}
+
+    def call_llm(prompt: str, config: dict) -> str:
+        for pattern, response in responses.items():
+            if pattern in prompt:
+                return response
+        return '{"response": "stub response"}'
+
+    return call_llm
+```
+
+---
+
+## Summary
+
+| Concern | Component | Built-in Options | Custom Examples |
+|---------|-----------|------------------|-----------------|
+| **Persistence** | Backends | In-Memory, Local Files | PostgreSQL, DynamoDB, MongoDB |
+| **Telemetry** | TelemetryBackend | In-Memory, Local Files | Datadog, CloudWatch |
+| **Communication** | Callers | In-process | Lambda, HTTP, SQS |
+| **LLM** | CallLlm | None (you provide) | OpenAI, Anthropic, Ollama |
+
+The key principle: **SOE defines the protocols, you provide the implementations**.
+
+---
+
+## Next Steps
+
+Now that you've mastered the infrastructure, explore the capabilities built directly into the engine:
+- **[Chapter 11: Built-in Tools](guide_11_builtins.md)** — Self-evolution, documentation exploration, and runtime modification

soe/docs/guide_11_builtins.md

@@ -0,0 +1,118 @@
+
+# SOE Guide: Chapter 11 - Built-in Tools
+
+## The Power of Self-Evolution
+
+SOE includes a set of **built-in tools** that are always available to your workflows. These aren't ordinary utilities—they're the foundation for **self-evolving systems**.
+
+Built-in tools enable workflows to:
+
+- **Explore their own documentation** — Understand what they can do
+- **Modify workflows at runtime** — Inject, remove, or reconfigure nodes
+- **Manage execution context** — Read, update, and copy context data
+- **Query available capabilities** — Discover registered tools dynamically
+
+---
+
+## Naming Convention
+
+All built-in tools use the `soe_` prefix to clearly distinguish them from user-defined tools. This prevents naming conflicts and makes it clear these are system-provided utilities.
+
+---
+
+## Principles
+
+### Always Available
+
+Built-in tools require no registration. Simply reference them by name in any tool node:
+
+```yaml
+example_workflow:
+  ExploreDocs:
+    node_type: tool
+    event_triggers: [START]
+    tool_name: soe_explore_docs
+    context_parameter_field: explore_params
+    output_field: docs_list
+    event_emissions:
+      - signal_name: DOCS_LISTED
+```
+
+### Essential for Self-Evolution
+
+These tools are the building blocks for autonomous systems. An LLM can:
+
+1. **Read documentation** to understand capabilities
+2. **Query workflows** to see current structure
+3. **Inject new nodes** to extend behavior
+4. **Manage context** to persist learned patterns
+
+### No Registration Required
+
+Unlike custom tools that need a `tools_registry`, built-ins work immediately. SOE provides them automatically during orchestration.
+
+---
+
+## Available Built-ins
+
+| Built-in | Purpose | Documentation |
+|----------|---------|---------------|
+| `soe_explore_docs` | Make SOE self-aware by exploring its documentation | [explore_docs](builtins/explore_docs.md) |
+| `soe_get_workflows` | Query registered workflow definitions | [workflows](builtins/workflows.md) |
+| `soe_inject_workflow` | Add new workflows at runtime | [workflows](builtins/workflows.md) |
+| `soe_inject_node` | Add or modify nodes in workflows | [workflows](builtins/workflows.md) |
+| `soe_remove_workflow` | Remove workflows from registry | [workflows](builtins/workflows.md) |
+| `soe_remove_node` | Remove nodes from workflows | [workflows](builtins/workflows.md) |
+| `soe_get_context` | Read execution context | [context](builtins/context.md) |
+| `soe_update_context` | Modify execution context | [context](builtins/context.md) |
+| `soe_copy_context` | Clone context for parallel execution | [context](builtins/context.md) |
+| `soe_list_contexts` | Discover available contexts | [context](builtins/context.md) |
+| `soe_get_available_tools` | List all registered tools | [tools](builtins/tools.md) |
+| `soe_call_tool` | Invoke a tool by name dynamically | [tools](builtins/tools.md) |
+| `soe_add_signal` | Emit signals programmatically | [workflows](builtins/workflows.md) |
+
+---
+
+## Self-Awareness in Practice
+
+Here's a workflow that becomes self-aware by exploring docs and querying its own state:
+
+```yaml
+self_aware_workflow:
+  ExploreCapabilities:
+    node_type: tool
+    event_triggers: [START]
+    tool_name: soe_explore_docs
+    context_parameter_field: explore_params
+    output_field: capabilities_tree
+    event_emissions:
+      - signal_name: CAPABILITIES_KNOWN
+
+  QueryCurrentState:
+    node_type: tool
+    event_triggers: [CAPABILITIES_KNOWN]
+    tool_name: soe_get_workflows
+    output_field: current_workflows
+    event_emissions:
+      - signal_name: STATE_KNOWN
+```
+
+This pattern is the foundation for metacognitive workflows that can reason about their own capabilities.
+
+---
+
+## Next Steps
+
+Explore each built-in category in detail:
+
+- **[explore_docs](builtins/explore_docs.md)** — Self-awareness through documentation
+- **[workflows](builtins/workflows.md)** — Runtime workflow modification
+- **[context](builtins/context.md)** — Execution state management
+- **[tools](builtins/tools.md)** — Dynamic tool discovery and invocation
+
+---
+
+## Related
+
+- [Advanced Patterns](advanced_patterns/index.md) — Complex workflow patterns
+- [Self-Evolving Workflows](advanced_patterns/self_evolving_workflows.md) — Full self-evolution guide

soe/docs/index.md

@@ -0,0 +1,104 @@
+# SOE User Guide
+
+Welcome to the **Signal-driven Orchestration Engine** user guide. This is your starting point for learning how to build workflows with SOE.
+
+---
+
+## Quick Start
+
+New to SOE? Start here:
+
+1. **[Getting Started](guide_00_getting_started.md)** — Installation, core concepts, and your first workflow
+2. **[Tool Nodes](guide_01_tool.md)** — Execute Python functions in your workflows
+3. **[LLM Nodes](guide_02_llm.md)** — Integrate language models into your workflows
+4. **[Router Nodes](guide_03_router.md)** — Pure routing and branching (recommended)
+
+---
+
+## The Guide
+
+### The Three Core Node Types
+
+SOE has three core node types that can build **any** workflow pattern:
+
+| Chapter | Topic | What You'll Learn |
+|---------|-------|-------------------|
+| [00 - Getting Started](guide_00_getting_started.md) | Installation & First Run | How SOE works, synchronous execution, copy-paste template |
+| [01 - Tool](guide_01_tool.md) | Tool Nodes | Execute Python functions—APIs, databases, real work |
+| [02 - LLM](guide_02_llm.md) | LLM Nodes | Call language models, prompt templating, signal selection |
+| [03 - Router](guide_03_router.md) | Router Nodes | Pure routing—entry points, branching, no execution |
+| [04 - Patterns](guide_04_patterns.md) | Building Custom Workflows | Combine core nodes: ReAct, chain-of-thought, metacognition |
+
+With just **Tool**, **LLM**, and **Router**, you can build remarkably sophisticated workflows—including custom agent loops, reasoning chains, and multi-step AI systems.
+
+### Convenience & Advanced Features
+
+These chapters cover higher-level abstractions and advanced capabilities:
+
+| Chapter | Topic | What You'll Learn |
+|---------|-------|-------------------|
+| [05 - Agent](guide_05_agent.md) | Agent Nodes | Built-in ReAct loop—convenience wrapper for tool-using agents |
+| [06 - Schema](guide_06_schema.md) | Context Schema | Type validation for structured LLM output |
+| [07 - Identity](guide_07_identity.md) | Identity & History | System prompts and conversation history across nodes |
+| [08 - Child Workflows](guide_08_child.md) | Sub-orchestration | Nested workflows with signal communication |
+| [09 - Ecosystem](guide_09_ecosystem.md) | The Workflows Ecosystem | Multi-workflow registries, parallel execution, versioning |
+| [10 - Infrastructure](guide_10_infrastructure.md) | Custom Backends & Callers | PostgreSQL, DynamoDB, Lambda, HTTP webhooks |
+| [11 - Built-in Tools](guide_11_builtins.md) | Self-Evolution | Always-available tools for introspection and evolution |
+
+---
+
+## Primitives
+
+Understand the core building blocks:
+
+| Concept | Description |
+|---------|-------------|
+| [The 7 Primitives](primitives/primitives.md) | Signals, Context, Workflows, Backends, Nodes, Identities, Context Schema |
+| [Signals Reference](primitives/signals.md) | Signal emission, conditions, and propagation |
+| [Context Deep Dive](primitives/context.md) | History, accumulated filter, Jinja access |
+| [Backends Reference](primitives/backends.md) | Setup, protocols, custom implementations |
+| [Node Reference](primitives/node_reference.md) | Complete node configuration reference |
+
+---
+
+## Advanced Patterns
+
+For researchers and advanced users exploring SOE's full potential:
+
+| Pattern | Description |
+|---------|-------------|
+| [Advanced Patterns Index](advanced_patterns/index.md) | Overview of all advanced patterns |
+| [Fan-Out & Aggregations](advanced_patterns/guide_fanout_and_aggregations.md) | Parallel processing and accumulated context |
+| [Self-Evolving Workflows](advanced_patterns/self_evolving_workflows.md) | LLM-driven workflow generation and injection |
+| [Swarm Intelligence](advanced_patterns/swarm_intelligence.md) | Multi-agent voting and consensus |
+| [Hybrid Intelligence](advanced_patterns/hybrid_intelligence.md) | Mixing deterministic logic with AI |
+| [Inheritance](advanced_patterns/guide_inheritance.md) | Config and context inheritance between executions |
+| [Operational Features](advanced_patterns/operational.md) | Runtime control, limits, and guardrails |
+
+### Built-in Tools for Self-Evolution
+
+Built-in tools enable granular runtime modifications:
+
+| Built-in | Description |
+|----------|-------------|
+| [explore_docs](builtins/explore_docs.md) | Make SOE self-aware by exploring its documentation |
+| [workflows](builtins/workflows.md) | Query, inject, and modify workflows at runtime |
+| [context](builtins/context.md) | Read, update, and copy execution context |
+| [tools](builtins/tools.md) | Discover and dynamically call registered tools |
+
+---
+
+## Philosophy
+
+SOE is designed around three core principles:
+
+1. **Signal-Driven** — Nodes communicate through signals, not direct calls
+2. **Infrastructure-Agnostic** — Plug in your own databases, LLMs, and distribution mechanisms
+3. **Data-Driven** — Everything flows through context; master the primitives to build anything
+
+---
+
+## Getting Help
+
+- **Source Code**: [github.com/pgarcia14180/soe](https://github.com/pgarcia14180/soe)
+- **PyPI Package**: [pypi.org/project/soe](https://pypi.org/project/soe/)