smart_message 0.0.10 → 0.0.12

data/mkdocs.yml

@@ -0,0 +1,184 @@
+site_name: SmartMessage Documentation
+site_description: "Intelligent messages that know what to do and how to do it"
+site_author: "madbomber"
+site_url: "https://madbomber.github.io/smart_message/"
+
+repo_name: "madbomber/smart_message"
+repo_url: "https://github.com/madbomber/smart_message"
+edit_uri: "edit/main/docs/"
+
+theme:
+  name: material
+  language: en
+
+  # Logo and favicon
+  logo: assets/smart_message.jpg
+  favicon: assets/smart_message.jpg
+
+  # Color palette
+  palette:
+    # Light mode
+    - media: "(prefers-color-scheme: light)"
+      scheme: default
+      primary: brown
+      accent: amber
+      toggle:
+        icon: material/brightness-7
+        name: Switch to dark mode
+
+    # Dark mode
+    - media: "(prefers-color-scheme: dark)"
+      scheme: slate
+      primary: brown
+      accent: amber
+      toggle:
+        icon: material/brightness-4
+        name: Switch to light mode
+
+  # Font configuration
+  font:
+    text: Roboto
+    code: Roboto Mono
+
+  # Theme features
+  features:
+    # Navigation
+    - navigation.instant
+    - navigation.tracking
+    - navigation.tabs
+    - navigation.tabs.sticky
+    - navigation.sections
+    - navigation.path
+    - navigation.indexes
+    - navigation.top
+
+    # Table of contents
+    - toc.follow
+
+    # Search
+    - search.suggest
+    - search.highlight
+    - search.share
+
+    # Header
+    - header.autohide
+
+    # Content
+    - content.code.copy
+    - content.code.annotate
+    - content.tabs.link
+    - content.tooltips
+    - content.action.edit
+    - content.action.view
+
+# Plugins
+plugins:
+  - search:
+      separator: '[\s\-,:!=\[\]()"`/]+|\.(?!\d)|&[lg]t;|(?!\b)(?=[A-Z][a-z])'
+  - macros
+
+# Extensions
+markdown_extensions:
+  # Python Markdown
+  - abbr
+  - admonition
+  - attr_list
+  - def_list
+  - footnotes
+  - md_in_html
+  - toc:
+      permalink: true
+      title: On this page
+
+  # Python Markdown Extensions
+  - pymdownx.arithmatex:
+      generic: true
+  - pymdownx.betterem:
+      smart_enable: all
+  - pymdownx.caret
+  - pymdownx.details
+  - pymdownx.emoji:
+      emoji_index: !!python/name:material.extensions.emoji.twemoji
+      emoji_generator: !!python/name:material.extensions.emoji.to_svg
+  - pymdownx.highlight:
+      anchor_linenums: true
+      line_spans: __span
+      pygments_lang_class: true
+  - pymdownx.inlinehilite
+  - pymdownx.keys
+  - pymdownx.magiclink:
+      repo_url_shorthand: true
+      user: madbomber
+      repo: smart_message
+  - pymdownx.mark
+  - pymdownx.smartsymbols
+  - pymdownx.superfences:
+      custom_fences:
+        - name: mermaid
+          class: mermaid
+          format: !!python/name:pymdownx.superfences.fence_code_format
+  - pymdownx.tabbed:
+      alternate_style: true
+  - pymdownx.tasklist:
+      custom_checkbox: true
+  - pymdownx.tilde
+
+# Additional CSS and JavaScript
+extra_css:
+  - assets/stylesheets/extra.css
+
+extra_javascript:
+  - assets/javascripts/mathjax.js
+  - https://polyfill.io/v3/polyfill.min.js?features=es6
+  - https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-mml-chtml.js
+
+# Extra content
+extra:
+  # Version string
+  version: 0.1.11
+
+  # Social media links
+  social:
+    - icon: fontawesome/brands/github
+      link: https://github.com/madbomber/smart_message
+      name: GitHub Repository
+
+  # Generator notice
+  generator: false
+
+# Navigation with icons
+nav:
+  - Home: index.md
+  - 🚀 Getting Started:
+      - Quick Start: getting-started/quick-start.md
+      - Examples: getting-started/examples.md
+  - 📖 Core Concepts:
+      - Architecture: core-concepts/architecture.md
+      - Properties: core-concepts/properties.md
+      - Addressing: core-concepts/addressing.md
+      - Dispatcher: core-concepts/dispatcher.md
+      - Message Filtering: core-concepts/message-filtering.md
+      - Message Processing: core-concepts/message-processing.md
+  - 🔌 Reference:
+      - Transports: reference/transports.md
+      - Serializers: reference/serializers.md
+      - Logging: reference/logging.md
+      - Dead Letter Queue: reference/dead-letter-queue.md
+      - Message Deduplication: reference/message-deduplication.md
+      - Proc Handlers: reference/proc-handlers.md
+  - 🚚 Transports:
+      - Memory Transport: transports/memory-transport.md
+      - Redis Transport: transports/redis-transport.md
+      - Redis Enhanced Transport: transports/redis-enhanced-transport.md
+      - Redis Queue Transport: transports/redis-queue-transport.md
+      - Redis Transport Comparison: transports/redis-transport-comparison.md
+  - 📚 Guides:
+      - Redis Queue Getting Started: guides/redis-queue-getting-started.md
+      - Redis Queue Patterns: guides/redis-queue-patterns.md
+      - Redis Queue Production: guides/redis-queue-production.md
+  - ⚙️ Development:
+      - Troubleshooting: development/troubleshooting.md
+      - Ideas & Roadmap: development/ideas.md
+
+# Copyright
+copyright: Copyright © 2025 madbomber

data/p2p_plan.md

@@ -0,0 +1,326 @@
+# SmartMessage P2P Communication Plan
+
+## SmartMessage in Multi-Gem Integration Strategy
+
+### Enhanced BunnyFarm + SmartMessage Extension Architecture
+
+**Strategic Insight**: Instead of integrating four separate gems, enhance BunnyFarm with SmartMessage capabilities first, creating a powerful messaging foundation for Agent99. Use extension gems for transport modularity.
+
+**Revised Architecture:**
+
+The integrated architecture provides clean separation of concerns with enhanced BunnyFarm as the unified messaging layer, supported by modular transport extensions.
+
+## BunnyFarm + SmartMessage Integration Analysis
+
+### SmartMessage Enhancement Strategy
+
+**Replace `BunnyFarm::Message` with `SmartMessage::Base`:**
+```ruby
+# Enhanced BunnyFarm using SmartMessage
+class CapabilityRequest < SmartMessage::Base
+  config do
+    transport BunnyFarm::Transport.new  # Retains workflow capabilities
+    serializer SmartMessage::Serializer::JSON.new
+  end
+
+  # BunnyFarm's workflow methods
+  def process
+    # Message processing with transport abstraction
+  end
+
+  def success
+    # Success handling across any transport
+  end
+
+  def failure
+    # Failure handling with transport flexibility
+  end
+end
+```
+
+### Enhanced BunnyFarm Benefits
+1. **Unified Messaging System**: SmartMessage transport abstraction + BunnyFarm workflows
+2. **Multi-Transport Workflows**: Run BunnyFarm patterns across rabb, Lanet, NATS
+3. **Stronger Foundation**: Single enhanced gem vs multiple integrations
+4. **Reusable Components**: Enhanced BunnyFarm benefits other projects
+5. **Proven Patterns**: Combines best of both architectural approaches
+
+## Lanet Gem Integration Analysis
+
+### Integrated Three-Gem Strategy
+
+**Recommended Approach: SmartMessage + Lanet + Existing Brokers**
+- **SmartMessage provides unified messaging API** across all transports
+- **Lanet handles LAN P2P communication** (high-performance local scenarios)
+- **Existing NATS/rabbitmq brokers handle WAN** communication and initial discovery
+- **Registry remains** for global agent discovery and coordination
+
+**Implementation Benefits:**
+- **Unified API**: SmartMessage provides consistent interface across all transports
+- **Transport Abstraction**: Easy to switch or combine Lanet, NATS, rabbitmq
+- **Proven Components**: Leverages existing gems rather than custom implementation
+- **Built-in Security**: Lanet's encryption + SmartMessage's serialization
+- **Flexible Routing**: SmartMessage can intelligently choose transport based on recipient
+
+**Integration Points:**
+```ruby
+# SmartMessage transport plugins for Agent99
+class Agent99::LanetTransport < SmartMessage::Transport::Base
+  def initialize(options = {})
+    @lanet_sender = Lanet::Sender.new
+    @lanet_receiver = Lanet::Receiver.new
+  end
+
+  def publish(message, options = {})
+    target_ip = resolve_agent_ip(message.to)
+    @lanet_sender.send_to(target_ip, message.to_json, encrypt: true)
+  end
+end
+
+# Agent message classes using SmartMessage
+class Agent99::RequestMessage < SmartMessage::Base
+  config do
+    # Automatically choose best transport based on target
+    transport Agent99::SmartTransport.new
+    serializer SmartMessage::Serializer::JSON.new
+  end
+
+  # Agent99 can now use clean messaging API
+  def send_capability_request(target_agent, capability)
+    publish :capability_request,
+            to: target_agent,
+            capability: capability
+  end
+end
+```
+
+**Smart Transport Selection Strategy:**
+
+**Agent99 Multi-Process Transport Selection:**
+1. **Same Process**: Memory transport (instant delivery)
+2. **Same Machine (Direct)**: Named Pipes transport (OS-level IPC)
+3. **Same Machine (Pub/Sub)**: Redis transport (local pub/sub)
+4. **Same LAN**: Lanet transport (P2P encrypted)
+5. **Reliable Required**: rabbitmq transport (guaranteed delivery)
+6. **High Performance**: NATS transport (distributed coordination)
+
+**Transport Performance Characteristics:**
+- **Memory**: ~1μs latency (in-process)
+- **Named Pipes**: ~25μs latency (kernel-level IPC)
+- **Redis**: ~100μs latency (local network stack)
+- **Lanet**: ~1ms latency (LAN P2P encrypted)
+- **NATS**: ~2ms latency (high-performance distributed)
+- **rabbitmq**: ~5ms latency (reliable enterprise messaging)
+
+## SmartMessage Transport Extension System
+
+### Extension Gem Architecture
+
+**Modular Transport Design:**
+- **Core SmartMessage**: Lightweight with Memory + Redis transports
+- **Extension Gems**: Optional transport implementations
+- **Plugin System**: Auto-registration when gems are loaded
+- **Unified API**: Same interface across all transports
+
+### Extension Gem Structure
+
+**Recommended Extension Gems:**
+```ruby
+# Core lightweight gem
+gem 'smart_message'                         # Memory + Redis
+
+# Transport extensions (install as needed)
+gem 'smart_message-transport-named_pipes'   # OS-level IPC for same-machine
+gem 'smart_message-transport-rabbitmq'          # Enterprise reliability
+gem 'smart_message-transport-lanet'         # LAN P2P optimization
+gem 'smart_message-transport-nats'          # High-performance distributed
+```
+
+## Named Pipes Transport Design
+
+### Naming Convention & Configuration
+
+**Standard Naming Pattern:**
+```
+/tmp/agent99/pipes/{namespace}/{agent_id}.{direction}.pipe
+```
+
+**Components:**
+- **Base Path**: `/tmp/agent99/pipes/` (configurable via ENV)
+- **Namespace**: Group agents by application/environment
+- **Agent ID**: Unique agent identifier (UUID or name)
+- **Direction**: `in` (receive) or `out` (send) for unidirectional pipes
+- **Extension**: `.pipe` for clarity
+
+**Configuration Options:**
+```ruby
+class SmartMessage::Transport::NamedPipes
+  DEFAULT_CONFIG = {
+    base_path: ENV['AGENT99_PIPE_BASE'] || '/tmp/agent99/pipes',
+    namespace: ENV['AGENT99_NAMESPACE'] || 'default',
+    mode: :unidirectional,      # Recommended for avoiding deadlocks
+    permissions: 0600,           # Owner read/write only
+    cleanup: true,              # Delete pipes on shutdown
+    buffer_size: 65536          # 64KB default buffer
+  }
+end
+```
+
+### Named Pipes vs Redis Comparison
+
+**Named Pipes Advantages:**
+- **Performance**: 4x faster than Redis (~25μs vs ~100μs)
+- **Zero Dependencies**: No Redis server required
+- **Lower Resources**: Direct kernel communication
+- **Native OS Support**: Built into all *nix systems
+- **File System Security**: OS-level permission control
+
+**Redis Advantages:**
+- **Persistence**: Survives process restarts
+- **Pub/Sub**: Built-in fan-out capabilities
+- **Network Ready**: Can scale across machines
+- **Mature Tooling**: Extensive debugging tools
+
+**Selection Strategy:**
+```ruby
+def select_same_machine_transport(message_type)
+  if persistence_required?(message_type) || fan_out_required?(message_type)
+    :redis  # Complex scenarios requiring pub/sub or persistence
+  else
+    :named_pipes  # Default for direct agent-to-agent communication
+  end
+end
+```
+
+### Transport Extension Benefits
+
+**Modular Architecture Advantages:**
+1. **Lightweight Core**: SmartMessage stays minimal with essential transports
+2. **Optional Dependencies**: Users install only needed transport gems
+3. **Independent Evolution**: Each transport can develop at its own pace
+4. **Community Growth**: Plugin ecosystem encourages transport contributions
+5. **Flexible Deployment**: Choose transports based on infrastructure needs
+6. **Performance Optimization**: Named pipes for local, Redis for pub/sub
+
+### Technical Considerations
+
+**Message Format Adaptation:**
+- SmartMessage handles JSON serialization uniformly across all transports
+- Agent99 message headers map to SmartMessage entity addressing (FROM/TO/REPLY_TO)
+- Enhanced BunnyFarm workflow methods (process/success/failure) work across transports
+- Maintain compatibility with existing Agent99 message structure
+
+**Transport Selection Logic:**
+- Intelligent routing based on target agent location and message requirements
+- Automatic fallback mechanisms when preferred transport unavailable
+- Performance optimization through transport-specific configurations
+- Health monitoring and connection management per transport
+
+**Security Integration:**
+- Transport-specific security implementations (Lanet encryption, rabbitmq SSL, etc.)
+- SmartMessage can layer additional security through serialization
+- Enhanced BunnyFarm maintains message workflow integrity across transports
+- Centralized key management through Agent99 registry integration
+
+## Revised Implementation Roadmap
+
+### Phase 1: Enhanced BunnyFarm Foundation (Weeks 1-4)
+1. **BunnyFarm + SmartMessage Integration**:
+   - Replace `BunnyFarm::Message` with `SmartMessage::Base`
+   - Migrate BunnyFarm's workflow capabilities to SmartMessage pattern
+   - Maintain automatic routing and configuration flexibility
+2. **Multi-Transport Support**:
+   - Add transport abstraction while preserving BunnyFarm workflows
+   - Implement rabbitmq transport plugin using existing BunnyFarm patterns
+   - Design plugin architecture for future transports
+3. **Enhanced Workflow System**:
+   - Extend BunnyFarm's process/success/failure pattern across transports
+   - Add SmartMessage entity addressing (FROM/TO/REPLY_TO)
+   - Maintain BunnyFarm's K.I.S.S. design philosophy
+
+### Phase 2: Agent99 Integration (Weeks 5-6)
+1. **Replace Agent99's rabbitmq Client**:
+   - Substitute basic rabbitmq client with enhanced BunnyFarm
+   - Map Agent99 message patterns to enhanced BunnyFarm workflows
+   - Maintain existing Agent99 API compatibility
+2. **Workflow Integration**:
+   - Leverage enhanced BunnyFarm's workflow capabilities for agent processing
+   - Add success/failure handling to Agent99 message types
+   - Implement automatic routing for agent-to-agent communication
+
+### Phase 3: Lanet P2P Integration (Weeks 7-8)
+1. **Lanet Transport Plugin**:
+   - Add Lanet transport to enhanced BunnyFarm system
+   - Implement BunnyFarm workflow patterns over Lanet P2P
+   - Network discovery integration (registry + Lanet scanning)
+2. **Intelligent Routing**:
+   - Smart transport selection (LAN via Lanet, WAN via rabbitmq)
+   - Fallback mechanisms and connection health monitoring
+   - Complete hybrid P2P system with workflow support
+
+### Phase 4: Production Readiness (Weeks 9-10)
+1. **System Integration**:
+   - End-to-end testing of Agent99 + Enhanced BunnyFarm + Lanet
+   - Performance optimization and monitoring
+   - Documentation and migration guides
+2. **Advanced Features**:
+   - Load balancing and auto-scaling capabilities
+   - Advanced security and authentication integration
+   - Backward compatibility validation
+
+## Open Questions
+
+### Enhanced BunnyFarm + SmartMessage Questions
+1. How do we migrate BunnyFarm's message workflows to SmartMessage without losing functionality?
+2. Can BunnyFarm's automatic routing (`ClassName.action`) work with SmartMessage transport abstraction?
+3. How do we maintain BunnyFarm's configuration flexibility while adding transport plugins?
+4. What's the performance impact of adding transport abstraction to BunnyFarm workflows?
+
+### Transport Extension Questions
+5. Should transport extensions be auto-loaded or explicitly required?
+6. How do we handle version compatibility between core SmartMessage and transport extensions?
+7. What's the plugin registration mechanism for transport discovery?
+8. How do we manage transport-specific configuration and connection pooling?
+
+### Lanet Integration Questions
+9. How does Lanet handle enhanced BunnyFarm workflow messages?
+10. Can Lanet's network discovery integrate with Agent99's registry system?
+11. What are Lanet's performance characteristics compared to other transports?
+12. How should we handle key management for Lanet's encryption across multiple agents?
+
+### NATS Integration Questions
+13. How does NATS subject-based routing map to Agent99's capability-based routing?
+14. Can NATS handle enhanced BunnyFarm workflow patterns effectively?
+15. What's the optimal NATS clustering strategy for Agent99 multi-process coordination?
+16. How do we integrate NATS monitoring with Agent99's health check system?
+
+## Summary
+
+This plan proposes enhancing BunnyFarm with SmartMessage capabilities first, creating a powerful unified messaging foundation for Agent99's P2P evolution:
+
+**Revised Strategy Benefits:**
+- **Enhanced BunnyFarm Foundation**: Single powerful messaging gem instead of multiple integrations
+- **Workflow-Enabled Multi-Transport**: BunnyFarm's process/success/failure patterns across all transports
+- **Cleaner Architecture**: Agent99 builds on enhanced BunnyFarm rather than managing multiple gems
+- **Stronger Foundation**: Enhanced BunnyFarm benefits other projects beyond Agent99
+- **Proven Patterns**: Combines SmartMessage transport abstraction with BunnyFarm workflow design
+
+**Key Advantages:**
+- **Unified Messaging System**: Enhanced BunnyFarm becomes the messaging layer for Agent99
+- **Automatic Optimization**: Smart routing (LAN via Lanet, WAN via rabbitmq) with workflow support
+- **Built-in Security**: Lanet encryption + SmartMessage abstraction + BunnyFarm reliability
+- **Extensibility**: Plugin architecture in enhanced BunnyFarm supports future transports
+- **Reusability**: Enhanced BunnyFarm becomes valuable for broader Ruby ecosystem
+
+**Strategic Impact:**
+- Creates a more cohesive and maintainable architecture
+- Reduces integration complexity while increasing capabilities
+- Positions both BunnyFarm and Agent99 as leading Ruby messaging solutions
+- Provides foundation for advanced AI agent communication patterns
+
+This approach transforms both BunnyFarm and Agent99 into industry-leading tools while maintaining their core design philosophies.
+
+---
+
+*Last Updated: 2025-01-03*
+*Status: Planning Complete - Ready for Implementation*