smart_message 0.0.10 → 0.0.12

data/docs/{architecture.md → core-concepts/architecture.md}

@@ -16,54 +16,7 @@ SmartMessage is designed around the principle that **messages should be independ
 
 ## Architecture Overview
 
-```mermaid
-graph TB
-    subgraph "SmartMessage Core"
-        Base[SmartMessage::Base]
-        Header[Message Header<br/>• UUID<br/>• Timestamps<br/>• Addressing]
-        Props[Message Properties<br/>• Business Data<br/>• Validation<br/>• Versioning]
-    end
-    
-    subgraph "Plugin System"
-        Transport[Transport Plugin<br/>• publish()<br/>• subscribe()<br/>• Memory/Redis/STDOUT]
-        Serializer[Serializer Plugin<br/>• encode()<br/>• decode()<br/>• JSON/Custom]
-        Logger[Logger Plugin<br/>• Structured logging<br/>• Multiple outputs<br/>• Colorization]
-    end
-    
-    subgraph "Message Processing"
-        Dispatcher[Dispatcher<br/>• Route messages<br/>• Thread pool<br/>• Subscriptions<br/>• DDQ management]
-        DDQ[Deduplication Queue<br/>• Handler-scoped<br/>• Memory/Redis storage<br/>• O(1) performance<br/>• Circular buffer]
-        Handlers[Message Handlers<br/>• Default handler<br/>• Block handlers<br/>• Proc handlers<br/>• Method handlers]
-    end
-    
-    subgraph "Reliability Layer"
-        CircuitBreaker[Circuit Breaker<br/>• Failure thresholds<br/>• Automatic fallback<br/>• Recovery detection]
-        DLQ[Dead Letter Queue<br/>• Failed messages<br/>• Replay mechanism<br/>• JSON Lines format]
-    end
-    
-    subgraph "Monitoring"
-        Stats[Statistics<br/>• Message counts<br/>• Processing metrics<br/>• Thread pool status]
-        Filters[Message Filtering<br/>• Entity-aware routing<br/>• Regex patterns<br/>• Broadcast handling]
-    end
-    
-    Base --> Header
-    Base --> Props
-    Base --> Transport
-    Base --> Serializer
-    Base --> Logger
-    
-    Transport --> Dispatcher
-    Dispatcher --> DDQ
-    Dispatcher --> Handlers
-    Dispatcher --> Stats
-    Dispatcher --> Filters
-    
-    Transport --> CircuitBreaker
-    CircuitBreaker --> DLQ
-    
-    DDQ -.-> Stats
-    Handlers -.-> Stats
-```
+![SmartMessage Architecture Overview](../assets/images/smartmessage_architecture_overview.svg)
 
 ## Core Components
 
@@ -77,7 +30,7 @@ The foundation class that all messages inherit from, built on `Hashie::Dash`.
 - Message lifecycle management
 - Header generation and management
 
-**Location:** `lib/smart_message/base.rb:11-278`
+**Location:** `lib/smart_message/base.rb:17-199`
 
 ```ruby
 class MyMessage < SmartMessage::Base
@@ -107,12 +60,13 @@ Handles message delivery and routing between systems.
 ```ruby
 # Transport interface
 class CustomTransport < SmartMessage::Transport::Base
-  def publish(message_header, message_payload)
+  def do_publish(message_class, serialized_message)
     # Send message via your transport
   end
   
-  def subscribe(message_class, process_method)
-    # Set up subscription
+  def subscribe(message_class, process_method, filter_options = {})
+    # Set up subscription via dispatcher
+    @dispatcher.add(message_class, process_method, filter_options)
   end
 end
 ```
@@ -130,12 +84,14 @@ Handles encoding and decoding of message content.
 
 ```ruby
 class CustomSerializer < SmartMessage::Serializer::Base
-  def encode(message_instance)
-    # Convert to wire format
+  def do_encode(message_instance)
+    # Convert message instance to wire format
+    # Default implementation uses message_instance.to_h
   end
   
-  def decode(payload)
-    # Convert from wire format
+  def do_decode(payload)
+    # Convert from wire format back to hash
+    # Must return hash compatible with message initialization
   end
 end
 ```
@@ -156,7 +112,7 @@ Routes incoming messages to appropriate handlers using concurrent processing wit
 ```ruby
 dispatcher = SmartMessage::Dispatcher.new
 dispatcher.add("MyMessage", "MyMessage.process")
-dispatcher.route(header, payload)
+dispatcher.route(decoded_message)
 
 # DDQ integration is automatic when enabled
 MyMessage.enable_deduplication!
@@ -173,33 +129,18 @@ Handler-scoped message deduplication system preventing duplicate processing.
 - O(1) performance with hybrid Array + Set data structure
 
 **Architecture:**
-```mermaid
-graph LR
-    subgraph "Handler A DDQ"
-        A1[Circular Array]
-        A2[Lookup Set]
-        A3[Mutex Lock]
-    end
-    
-    subgraph "Handler B DDQ"
-        B1[Circular Array]
-        B2[Lookup Set] 
-        B3[Mutex Lock]
-    end
-    
-    Message[Incoming Message<br/>UUID: abc-123] --> Dispatcher
-    Dispatcher --> |Check Handler A| A2
-    Dispatcher --> |Check Handler B| B2
-    
-    A2 --> |Not Found| ProcessA[Process with Handler A]
-    B2 --> |Found| SkipB[Skip Handler B - Duplicate]
-    
-    ProcessA --> |Add UUID| A1
-    ProcessA --> |Add UUID| A2
-```
+
+![DDQ Architecture](../assets/images/ddq_architecture.svg)
 
 **Location:** `lib/smart_message/deduplication.rb`, `lib/smart_message/ddq/`
 
+**Key Features:**
+- Handler-scoped deduplication (each handler gets its own DDQ)
+- UUID-based duplicate detection
+- Multiple storage backends (Memory, Redis)
+- O(1) performance with hybrid Array + Set data structure
+- Thread-safe operations with mutex locks
+
 ### 6. Message Headers
 
 Standard metadata attached to every message with entity addressing support.
@@ -217,9 +158,12 @@ header = message._sm_header
 puts header.uuid          # "550e8400-e29b-41d4-a716-446655440000"
 puts header.message_class # "MyMessage"
 puts header.published_at  # 2025-08-17 10:30:00 UTC
+puts header.publisher_pid # 12345
 puts header.from          # "payment-service"
 puts header.to            # "order-service"
+puts header.reply_to      # "payment-service" (defaults to from)
 puts header.version       # 1
+puts header.serializer    # "SmartMessage::Serializer::JSON"
 ```
 
 ## Message Lifecycle
@@ -232,7 +176,7 @@ class OrderMessage < SmartMessage::Base
   
   config do
     transport SmartMessage::Transport.create(:memory)
-    serializer SmartMessage::Serializer::JSON.new
+    serializer SmartMessage::Serializer::Json.new
   end
 end
 ```
@@ -264,13 +208,14 @@ order.publish
 
 ### 4. Receiving Phase
 ```ruby
-# Transport receives message
-transport.receive(header, payload)
-# 1. Routes to dispatcher
-# 2. Dispatcher checks DDQ for duplicates per handler
-# 3. Applies message filters (from/to/broadcast)
-# 4. Spawns thread for processing matching handlers
-# 5. Marks UUID as processed in handler's DDQ
+# Transport receives serialized message
+transport.receive(message_class, serialized_message)
+# 1. Decodes message using class's configured serializer
+# 2. Routes decoded message to dispatcher
+# 3. Dispatcher checks DDQ for duplicates per handler
+# 4. Applies message filters (from/to/broadcast)
+# 5. Spawns thread for processing matching handlers
+# 6. Marks UUID as processed in handler's DDQ after successful processing
 ```
 
 ### 5. Message Handler Processing
@@ -278,43 +223,42 @@ transport.receive(header, payload)
 SmartMessage supports multiple handler types, routed through the dispatcher:
 
 ```ruby
-# Default handler (self.process method)
-def self.process(message_header, message_payload)
-  data = JSON.parse(message_payload)
-  order = new(data)
+# Default handler (self.process method) - receives decoded message instance
+def self.process(decoded_message)
+  order = decoded_message  # Already a fully decoded OrderMessage instance
   fulfill_order(order)
 end
 
-# Block handler (inline processing)
-OrderMessage.subscribe do |header, payload|
-  data = JSON.parse(payload)
-  quick_processing(data)
+# Block handler (inline processing) - receives decoded message instance
+OrderMessage.subscribe do |decoded_message|
+  quick_processing(decoded_message)
 end
 
-# Proc handler (reusable across message types)
-audit_proc = proc do |header, payload|
-  AuditService.log_message(header.message_class, payload)
+# Proc handler (reusable across message types) - receives decoded message instance
+audit_proc = proc do |decoded_message|
+  AuditService.log_message(decoded_message.class.name, decoded_message)
 end
 OrderMessage.subscribe(audit_proc)
 
-# Method handler (service class processing)
+# Method handler (service class processing) - receives decoded message instance
 class OrderService
-  def self.process_order(header, payload)
-    data = JSON.parse(payload)
-    complex_business_logic(data)
+  def self.process_order(decoded_message)
+    complex_business_logic(decoded_message)
   end
 end
 OrderMessage.subscribe("OrderService.process_order")
 ```
 
 **Handler Routing Process:**
-1. Dispatcher receives message and header
+1. Dispatcher receives decoded message instance
 2. Looks up all registered handlers for message class
-3. For each handler:
+3. For each handler that matches filters:
+   - Checks DDQ for duplicates (handler-scoped)
    - **String handlers**: Resolves to class method via constantize
    - **Proc handlers**: Calls proc directly from registry
-4. Executes handlers in parallel threads
-5. Collects statistics and handles errors
+4. Executes handlers in parallel threads with circuit breaker protection
+5. Marks UUID as processed in handler's DDQ after successful completion
+6. Collects statistics and handles errors
 
 ## Plugin System Architecture
 
@@ -364,10 +308,15 @@ end
 The dispatcher uses `Concurrent::CachedThreadPool` for processing:
 
 ```ruby
-# Each message processing happens in its own thread
+# Each message processing happens in its own thread with circuit breaker protection
 @router_pool.post do
-  # Message processing happens here
-  target_class.constantize.process(header, payload)
+  circuit_result = circuit(:message_processor).wrap do
+    if proc_handler?(message_processor)
+      SmartMessage::Base.call_proc_handler(message_processor, decoded_message)
+    else
+      target_class.constantize.method(class_method).call(decoded_message)
+    end
+  end
 end
 ```
 
@@ -396,11 +345,13 @@ puts "Completed tasks: #{status[:completed_task_count]}"
 Processing exceptions are isolated to prevent cascade failures:
 
 ```ruby
-begin
-  target_class.constantize.process(header, payload)
-rescue Exception => e
-  # Log error but don't crash the dispatcher
-  # TODO: Add proper exception logging
+circuit_result = circuit(:message_processor).wrap do
+  # Handler execution with circuit breaker protection
+end
+
+# Handle circuit breaker fallback responses
+if circuit_result.is_a?(Hash) && circuit_result[:circuit_breaker]
+  handle_circuit_breaker_fallback(circuit_result, decoded_message, message_processor)
 end
 ```
 
@@ -415,6 +366,7 @@ module SmartMessage::Errors
   class NotImplemented < RuntimeError; end
   class ReceivedMessageNotSubscribed < RuntimeError; end
   class UnknownMessageClass < RuntimeError; end
+  class ValidationError < RuntimeError; end
 end
 ```
 
@@ -425,13 +377,12 @@ end
 SmartMessage integrates BreakerMachines for production-grade reliability:
 
 ```ruby
-# Transport operations protected by circuit breakers
+# Circuit breakers are automatically configured for all transports
 class MyTransport < SmartMessage::Transport::Base
-  circuit :transport_publish do
-    threshold failures: 5, within: 30.seconds
-    reset_after 15.seconds
-    fallback SmartMessage::CircuitBreaker::Fallbacks.dead_letter_queue
-  end
+  # Inherits circuit breaker configuration:
+  # - :transport_publish for publishing operations
+  # - :transport_subscribe for subscription operations
+  # - Automatic DLQ fallback for failed publishes
 end
 ```
 
@@ -450,25 +401,12 @@ message.publish  # If transport fails, goes to DLQ
 
 # Manual capture for business logic failures
 dlq = SmartMessage::DeadLetterQueue.default
-dlq.enqueue(header, payload, error: "Validation failed")
+dlq.enqueue(decoded_message, error: "Validation failed", transport: "manual")
 ```
 
 **DLQ Architecture:**
 
-```mermaid
-graph TB
-    Publish[Message Publishing]
-    CB[Circuit Breaker<br/>Monitoring]
-    Transport[Transport<br/>Success]
-    DLQ[Dead Letter Queue<br/>Failure Storage]
-    Replay[Replay Mechanism<br/>Manual/Automated]
-    
-    Publish --> CB
-    CB --> |Success| Transport
-    CB --> |Failure| DLQ
-    DLQ --> Replay
-    Replay --> |Retry| Publish
-```
+![Dead Letter Queue Architecture](../assets/images/dlq_architecture.svg)
 
 **DLQ Features:**
 - JSON Lines format for efficient append operations
@@ -536,7 +474,9 @@ When a message needs a plugin:
 
 ```ruby
 def transport
-  @transport || @@transport || raise(Errors::TransportNotConfigured)
+  @transport || self.class.class_variable_get(:@@transport) || raise(Errors::TransportNotConfigured)
+rescue NameError
+  raise(Errors::TransportNotConfigured)
 end
 ```
 

data/docs/{dispatcher.md → core-concepts/dispatcher.md}

@@ -187,10 +187,10 @@ puts dispatcher.subscribers["MyMessage"]
 When a transport receives a message, it calls the dispatcher:
 
 ```ruby
-# Transport receives message and routes it
-transport.receive(message_header, message_payload)
-# This internally calls:
-dispatcher.route(message_header, message_payload)
+# Transport receives serialized message and routes it
+transport.receive(message_class, serialized_message)
+# This internally decodes the message and calls:
+dispatcher.route(decoded_message)
 ```
 
 ### 2. Subscription Lookup
@@ -198,12 +198,12 @@ dispatcher.route(message_header, message_payload)
 The dispatcher finds all registered handlers:
 
 ```ruby
-def route(message_header, message_payload)
-  message_klass = message_header.message_class
+def route(decoded_message)
+  message_klass = decoded_message._sm_header.message_class
   return nil if @subscribers[message_klass].empty?
   
-  @subscribers[message_klass].each do |message_processor|
-    # Process each handler
+  @subscribers[message_klass].each do |subscription|
+    # Process each handler with filters
   end
 end
 ```
@@ -213,16 +213,17 @@ end
 Each handler is processed in its own thread, with support for both method and proc handlers:
 
 ```ruby
-@subscribers[message_klass].each do |message_processor|
+@subscribers[message_klass].each do |subscription|
+  message_processor = subscription[:process_method]
   SS.add(message_klass, message_processor, 'routed')
   
   @router_pool.post do
-    # This runs in a separate thread
-    begin
+    # This runs in a separate thread with circuit breaker protection
+    circuit_result = circuit(:message_processor).wrap do
       # Check if this is a proc handler or a regular method call
       if proc_handler?(message_processor)
         # Call the proc handler via SmartMessage::Base
-        SmartMessage::Base.call_proc_handler(message_processor, message_header, message_payload)
+        SmartMessage::Base.call_proc_handler(message_processor, decoded_message)
       else
         # Original method call logic
         parts = message_processor.split('.')
@@ -231,11 +232,13 @@ Each handler is processed in its own thread, with support for both method and pr
         
         target_klass.constantize
                     .method(class_method)
-                    .call(message_header, message_payload)
+                    .call(decoded_message)
       end
-    rescue Exception => e
-      # Error handling - doesn't crash the dispatcher
-      puts "Error processing message: #{e.message}" if $DEBUG
+    end
+    
+    # Handle circuit breaker fallback if triggered
+    if circuit_result.is_a?(Hash) && circuit_result[:circuit_breaker]
+      handle_circuit_breaker_fallback(circuit_result, decoded_message, message_processor)
     end
   end
 end
@@ -691,7 +694,7 @@ RSpec.describe "Message Processing" do
   before do
     TestMessage.config do
       transport transport
-      serializer SmartMessage::Serializer::JSON.new
+      serializer SmartMessage::Serializer::Json.new
     end
     
     TestMessage.subscribe
@@ -710,7 +713,4 @@ end
 
 ## Next Steps
 
-- [Thread Safety](thread-safety.md) - Understanding concurrent processing
-- [Statistics & Monitoring](monitoring.md) - Detailed monitoring guide
-- [Custom Transports](custom-transports.md) - How transports interact with the dispatcher
-- [Troubleshooting](troubleshooting.md) - Common dispatcher issues
+- [Troubleshooting](../development/troubleshooting.md) - Common dispatcher issues

data/docs/{message_filtering.md → core-concepts/message-filtering.md}

@@ -320,7 +320,7 @@ class FilterTest < Minitest::Test
     @transport = SmartMessage::Transport.create(:memory, auto_process: true)
     TestMessage.config do
       transport @transport
-      serializer SmartMessage::Serializer::JSON.new
+      serializer SmartMessage::Serializer::Json.new
     end
     TestMessage.unsubscribe!
   end
@@ -447,5 +447,4 @@ end
 
 - [Dispatcher Documentation](dispatcher.md) - How filtering integrates with message routing
 - [Entity Addressing](addressing.md) - Understanding `from`, `to`, and `reply_to` fields  
-- [Examples](examples.md) - Complete working examples with filtering
-- [Testing Guide](testing.md) - Best practices for testing filtered subscriptions
+- [Examples](../getting-started/examples.md) - Complete working examples with filtering

data/docs/{message_processing.md → core-concepts/message-processing.md}

@@ -36,9 +36,9 @@ SmartMessage supports multiple ways to handle incoming messages:
 ### 1. Default Handler Pattern (using `self.process`)
 ```ruby
 class SensorDataMessage < SmartMessage::Base
-  def self.process(message_header, message_payload)
+  def self.process(decoded_message)
     # This gets called when a SensorDataMessage is received
-    data = JSON.parse(message_payload)
+    # decoded_message is already a message instance
     puts "Sensor reading: #{data['value']}"
   end
 end
@@ -51,7 +51,7 @@ SensorDataMessage.subscribe  # Uses "SensorDataMessage.process"
 class ThermostatService
   def self.handle_sensor_data(message_header, message_payload)
     # Custom processing logic
-    data = JSON.parse(message_payload)
+    # decoded_message is already a message instance
     adjust_temperature(data)
   end
 end
@@ -99,8 +99,8 @@ Looking at the smart home IoT example:
 
 ```ruby
 class SensorDataMessage < SmartMessage::Base
-  def self.process(message_header, message_payload)
-    sensor_data = JSON.parse(message_payload)
+  def self.process(decoded_message)
+    sensor_# decoded_message is already a message instance
     icon = case sensor_data['device_type']
            when 'thermostat' then '🌡️'
            when 'security_camera' then '📹'
@@ -145,8 +145,8 @@ A single message type can have multiple subscribers with different handlers usin
 ```ruby
 # Default handler for logging
 class SensorDataMessage < SmartMessage::Base
-  def self.process(message_header, message_payload)
-    data = JSON.parse(message_payload)
+  def self.process(decoded_message)
+    # decoded_message is already a message instance
     puts "📊 Sensor data logged: #{data['device_id']}"
   end
 end
@@ -154,7 +154,7 @@ end
 # Custom method handler for specific services
 class ThermostatService
   def self.handle_sensor_data(message_header, message_payload)
-    data = JSON.parse(message_payload)
+    # decoded_message is already a message instance
     return unless data['device_type'] == 'thermostat'
     adjust_temperature(data['value'])
   end
@@ -197,10 +197,10 @@ SensorDataMessage.subscribe(database_logger)  # Proc handler
 
 ```ruby
 class SensorDataMessage < SmartMessage::Base
-  def self.process(message_header, message_payload)
+  def self.process(decoded_message)
     # This runs in its own thread
     # Be careful with shared state
-    data = JSON.parse(message_payload)
+    # decoded_message is already a message instance
     
     # Thread-safe operations
     update_local_cache(data)
@@ -216,9 +216,9 @@ Handlers should include proper error handling:
 
 ```ruby
 class SensorDataMessage < SmartMessage::Base
-  def self.process(message_header, message_payload)
+  def self.process(decoded_message)
     begin
-      data = JSON.parse(message_payload)
+      # decoded_message is already a message instance
       
       # Validate required fields
       raise "Missing device_id" unless data['device_id']
@@ -310,9 +310,9 @@ PaymentEventMessage.subscribe(audit_logger)
 
 ### 1. Keep Handlers Fast
 ```ruby
-def self.process(message_header, message_payload)
+def self.process(decoded_message)
   # Quick validation
-  data = JSON.parse(message_payload)
+  # decoded_message is already a message instance
   return unless valid_message?(data)
   
   # Delegate heavy work to background jobs
@@ -348,7 +348,7 @@ SensorDataMessage.subscribe do |h, p|; process_stuff(p); end
 ### 3. Filter Messages Early
 ```ruby
 def self.handle_thermostat_data(message_header, message_payload)
-  data = JSON.parse(message_payload)
+  # decoded_message is already a message instance
   
   # Filter early to avoid unnecessary processing
   return unless data['device_type'] == 'thermostat'
@@ -361,11 +361,11 @@ end
 
 ### 4. Include Logging and Monitoring
 ```ruby
-def self.process(message_header, message_payload)
+def self.process(decoded_message)
   start_time = Time.now
   
   begin
-    data = JSON.parse(message_payload)
+    # decoded_message is already a message instance
     logger.info "Processing sensor data from #{data['device_id']}"
     
     # Business logic here

data/docs/{troubleshooting.md → development/troubleshooting.md}

@@ -39,7 +39,7 @@ class MyMessage < SmartMessage::Base
   
   config do
     transport SmartMessage::Transport.create(:stdout)  # No loopback
-    serializer SmartMessage::Serializer::JSON.new
+    serializer SmartMessage::Serializer::Json.new
   end
 end
 
@@ -47,7 +47,7 @@ end
 class MyMessage < SmartMessage::Base
   config do
     transport SmartMessage::Transport.create(:stdout, loopback: true)
-    serializer SmartMessage::Serializer::JSON.new
+    serializer SmartMessage::Serializer::Json.new
   end
 end
 ```
@@ -102,7 +102,7 @@ MyMessage.new(data: "test").publish
 class MyMessage < SmartMessage::Base
   config do
     transport SmartMessage::Transport.create(:memory)
-    serializer SmartMessage::Serializer::JSON.new
+    serializer SmartMessage::Serializer::Json.new
   end
 end
 ```
@@ -114,7 +114,7 @@ class ProblematicMessage < SmartMessage::Base
   property :data
   
   config do
-    serializer SmartMessage::Serializer::JSON.new
+    serializer SmartMessage::Serializer::Json.new
   end
 end
 
@@ -154,7 +154,7 @@ class MyMessage < SmartMessage::Base
   property :data
   
   config do
-    serializer SmartMessage::Serializer::JSON.new
+    serializer SmartMessage::Serializer::Json.new
     # Missing transport!
   end
 end
@@ -165,7 +165,7 @@ MyMessage.new(data: "test").publish  # Throws TransportNotConfigured
 class MyMessage < SmartMessage::Base
   config do
     transport SmartMessage::Transport.create(:memory)
-    serializer SmartMessage::Serializer::JSON.new
+    serializer SmartMessage::Serializer::Json.new
   end
 end
 ```
@@ -471,7 +471,7 @@ class MyMessage < SmartMessage::Base
   # Reconfigure
   config do
     transport SmartMessage::Transport.create(:memory)
-    serializer SmartMessage::Serializer::JSON.new
+    serializer SmartMessage::Serializer::Json.new
   end
 end
 ```