smart_message 0.0.9 → 0.0.12

data/docs/{architecture.md → core-concepts/architecture.md}

@@ -16,38 +16,7 @@ SmartMessage is designed around the principle that **messages should be independ
 
 ## Architecture Overview
 
-```
-┌─────────────────────────────────────────────────────────────┐
-│                    SmartMessage::Base                       │
-│  ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐│
-│  │    Message      │ │   Transport     │ │   Serializer    ││
-│  │   Properties    │ │     Plugin      │ │     Plugin      ││
-│  │                 │ │                 │ │                 ││
-│  │ • user_id       │ │ • publish()     │ │ • encode()      ││
-│  │ • action        │ │ • subscribe()   │ │ • decode()      ││
-│  │ • timestamp     │ │ • receive()     │ │                 ││
-│  └─────────────────┘ └─────────────────┘ └─────────────────┘│
-└─────────────────────────────────────────────────────────────┘
-                                │
-                                ▼
-                    ┌─────────────────────┐
-                    │     Dispatcher      │
-                    │                     │
-                    │ • Route messages    │
-                    │ • Thread pool       │
-                    │ • Subscriptions     │
-                    └─────────────────────┘
-                                │
-                                ▼
-                    ┌─────────────────────┐
-                    │  Message Handlers   │
-                    │                     │
-                    │ • Default handler   │
-                    │ • Block handlers    │
-                    │ • Proc handlers     │
-                    │ • Method handlers   │
-                    └─────────────────────┘
-```
+![SmartMessage Architecture Overview](../assets/images/smartmessage_architecture_overview.svg)
 
 ## Core Components
 
@@ -61,7 +30,7 @@ The foundation class that all messages inherit from, built on `Hashie::Dash`.
 - Message lifecycle management
 - Header generation and management
 
-**Location:** `lib/smart_message/base.rb:11-278`
+**Location:** `lib/smart_message/base.rb:17-199`
 
 ```ruby
 class MyMessage < SmartMessage::Base
@@ -91,12 +60,13 @@ Handles message delivery and routing between systems.
 ```ruby
 # Transport interface
 class CustomTransport < SmartMessage::Transport::Base
-  def publish(message_header, message_payload)
+  def do_publish(message_class, serialized_message)
     # Send message via your transport
   end
   
-  def subscribe(message_class, process_method)
-    # Set up subscription
+  def subscribe(message_class, process_method, filter_options = {})
+    # Set up subscription via dispatcher
+    @dispatcher.add(message_class, process_method, filter_options)
   end
 end
 ```
@@ -114,50 +84,86 @@ Handles encoding and decoding of message content.
 
 ```ruby
 class CustomSerializer < SmartMessage::Serializer::Base
-  def encode(message_instance)
-    # Convert to wire format
+  def do_encode(message_instance)
+    # Convert message instance to wire format
+    # Default implementation uses message_instance.to_h
   end
   
-  def decode(payload)
-    # Convert from wire format
+  def do_decode(payload)
+    # Convert from wire format back to hash
+    # Must return hash compatible with message initialization
   end
 end
 ```
 
 ### 4. Dispatcher
 
-Routes incoming messages to appropriate handlers using concurrent processing.
+Routes incoming messages to appropriate handlers using concurrent processing with integrated deduplication.
 
 **Key Responsibilities:**
 - Message routing based on class
-- Thread pool management
+- Thread pool management  
 - Subscription catalog management
-- Statistics collection
+- Handler-scoped DDQ management
+- Message filtering and statistics collection
 
-**Location:** `lib/smart_message/dispatcher.rb:11-147`
+**Location:** `lib/smart_message/dispatcher.rb`
 
 ```ruby
 dispatcher = SmartMessage::Dispatcher.new
 dispatcher.add("MyMessage", "MyMessage.process")
-dispatcher.route(header, payload)
+dispatcher.route(decoded_message)
+
+# DDQ integration is automatic when enabled
+MyMessage.enable_deduplication!
 ```
 
-### 5. Message Headers
+### 5. Deduplication Queue (DDQ)
+
+Handler-scoped message deduplication system preventing duplicate processing.
+
+**Key Responsibilities:**
+- UUID-based duplicate detection
+- Handler isolation (each handler gets own DDQ)
+- Memory and Redis storage backends
+- O(1) performance with hybrid Array + Set data structure
+
+**Architecture:**
+
+![DDQ Architecture](../assets/images/ddq_architecture.svg)
+
+**Location:** `lib/smart_message/deduplication.rb`, `lib/smart_message/ddq/`
+
+**Key Features:**
+- Handler-scoped deduplication (each handler gets its own DDQ)
+- UUID-based duplicate detection
+- Multiple storage backends (Memory, Redis)
+- O(1) performance with hybrid Array + Set data structure
+- Thread-safe operations with mutex locks
 
-Standard metadata attached to every message.
+### 6. Message Headers
+
+Standard metadata attached to every message with entity addressing support.
 
 **Key Responsibilities:**
 - Message identification (UUID)
-- Routing information (message class)
+- Routing information (message class, version)
 - Tracking data (timestamps, process IDs)
+- Entity addressing (from, to, reply_to)
 
-**Location:** `lib/smart_message/header.rb:9-20`
+**Location:** `lib/smart_message/header.rb`
 
 ```ruby
 header = message._sm_header
 puts header.uuid          # "550e8400-e29b-41d4-a716-446655440000"
 puts header.message_class # "MyMessage"
 puts header.published_at  # 2025-08-17 10:30:00 UTC
+puts header.publisher_pid # 12345
+puts header.from          # "payment-service"
+puts header.to            # "order-service"
+puts header.reply_to      # "payment-service" (defaults to from)
+puts header.version       # 1
+puts header.serializer    # "SmartMessage::Serializer::JSON"
 ```
 
 ## Message Lifecycle
@@ -170,34 +176,46 @@ class OrderMessage < SmartMessage::Base
   
   config do
     transport SmartMessage::Transport.create(:memory)
-    serializer SmartMessage::Serializer::JSON.new
+    serializer SmartMessage::Serializer::Json.new
   end
 end
 ```
 
 ### 2. Subscription Phase
 ```ruby
+# Basic subscription
 OrderMessage.subscribe
-# Registers "OrderMessage.process" with dispatcher
+
+# Subscription with filtering
+OrderMessage.subscribe(from: /^payment-.*/, to: 'order-service')
+OrderMessage.subscribe('PaymentService.process', broadcast: true)
+
+# Each subscription gets its own DDQ automatically
+# DDQ Key: "OrderMessage:OrderMessage.process"
+# DDQ Key: "OrderMessage:PaymentService.process"
 ```
 
 ### 3. Publishing Phase
 ```ruby
 order = OrderMessage.new(order_id: "123", amount: 99.99)
+order.from("order-service").to("payment-service")
 order.publish
-# 1. Creates header with UUID, timestamp, etc.
-# 2. Encodes message via serializer
+# 1. Creates header with UUID, timestamp, addressing
+# 2. Encodes message via serializer  
 # 3. Sends via transport
+# 4. Circuit breaker monitors for failures
 ```
 
 ### 4. Receiving Phase
 ```ruby
-# Transport receives message
-transport.receive(header, payload)
-# 1. Routes to dispatcher
-# 2. Dispatcher finds subscribers
-# 3. Spawns thread for processing
-# 4. Calls registered message handlers
+# Transport receives serialized message
+transport.receive(message_class, serialized_message)
+# 1. Decodes message using class's configured serializer
+# 2. Routes decoded message to dispatcher
+# 3. Dispatcher checks DDQ for duplicates per handler
+# 4. Applies message filters (from/to/broadcast)
+# 5. Spawns thread for processing matching handlers
+# 6. Marks UUID as processed in handler's DDQ after successful processing
 ```
 
 ### 5. Message Handler Processing
@@ -205,43 +223,42 @@ transport.receive(header, payload)
 SmartMessage supports multiple handler types, routed through the dispatcher:
 
 ```ruby
-# Default handler (self.process method)
-def self.process(message_header, message_payload)
-  data = JSON.parse(message_payload)
-  order = new(data)
+# Default handler (self.process method) - receives decoded message instance
+def self.process(decoded_message)
+  order = decoded_message  # Already a fully decoded OrderMessage instance
   fulfill_order(order)
 end
 
-# Block handler (inline processing)
-OrderMessage.subscribe do |header, payload|
-  data = JSON.parse(payload)
-  quick_processing(data)
+# Block handler (inline processing) - receives decoded message instance
+OrderMessage.subscribe do |decoded_message|
+  quick_processing(decoded_message)
 end
 
-# Proc handler (reusable across message types)
-audit_proc = proc do |header, payload|
-  AuditService.log_message(header.message_class, payload)
+# Proc handler (reusable across message types) - receives decoded message instance
+audit_proc = proc do |decoded_message|
+  AuditService.log_message(decoded_message.class.name, decoded_message)
 end
 OrderMessage.subscribe(audit_proc)
 
-# Method handler (service class processing)
+# Method handler (service class processing) - receives decoded message instance
 class OrderService
-  def self.process_order(header, payload)
-    data = JSON.parse(payload)
-    complex_business_logic(data)
+  def self.process_order(decoded_message)
+    complex_business_logic(decoded_message)
   end
 end
 OrderMessage.subscribe("OrderService.process_order")
 ```
 
 **Handler Routing Process:**
-1. Dispatcher receives message and header
+1. Dispatcher receives decoded message instance
 2. Looks up all registered handlers for message class
-3. For each handler:
+3. For each handler that matches filters:
+   - Checks DDQ for duplicates (handler-scoped)
    - **String handlers**: Resolves to class method via constantize
    - **Proc handlers**: Calls proc directly from registry
-4. Executes handlers in parallel threads
-5. Collects statistics and handles errors
+4. Executes handlers in parallel threads with circuit breaker protection
+5. Marks UUID as processed in handler's DDQ after successful completion
+6. Collects statistics and handles errors
 
 ## Plugin System Architecture
 
@@ -291,10 +308,15 @@ end
 The dispatcher uses `Concurrent::CachedThreadPool` for processing:
 
 ```ruby
-# Each message processing happens in its own thread
+# Each message processing happens in its own thread with circuit breaker protection
 @router_pool.post do
-  # Message processing happens here
-  target_class.constantize.process(header, payload)
+  circuit_result = circuit(:message_processor).wrap do
+    if proc_handler?(message_processor)
+      SmartMessage::Base.call_proc_handler(message_processor, decoded_message)
+    else
+      target_class.constantize.method(class_method).call(decoded_message)
+    end
+  end
 end
 ```
 
@@ -323,11 +345,13 @@ puts "Completed tasks: #{status[:completed_task_count]}"
 Processing exceptions are isolated to prevent cascade failures:
 
 ```ruby
-begin
-  target_class.constantize.process(header, payload)
-rescue Exception => e
-  # Log error but don't crash the dispatcher
-  # TODO: Add proper exception logging
+circuit_result = circuit(:message_processor).wrap do
+  # Handler execution with circuit breaker protection
+end
+
+# Handle circuit breaker fallback responses
+if circuit_result.is_a?(Hash) && circuit_result[:circuit_breaker]
+  handle_circuit_breaker_fallback(circuit_result, decoded_message, message_processor)
 end
 ```
 
@@ -342,6 +366,7 @@ module SmartMessage::Errors
   class NotImplemented < RuntimeError; end
   class ReceivedMessageNotSubscribed < RuntimeError; end
   class UnknownMessageClass < RuntimeError; end
+  class ValidationError < RuntimeError; end
 end
 ```
 
@@ -352,13 +377,12 @@ end
 SmartMessage integrates BreakerMachines for production-grade reliability:
 
 ```ruby
-# Transport operations protected by circuit breakers
+# Circuit breakers are automatically configured for all transports
 class MyTransport < SmartMessage::Transport::Base
-  circuit :transport_publish do
-    threshold failures: 5, within: 30.seconds
-    reset_after 15.seconds
-    fallback SmartMessage::CircuitBreaker::Fallbacks.dead_letter_queue
-  end
+  # Inherits circuit breaker configuration:
+  # - :transport_publish for publishing operations
+  # - :transport_subscribe for subscription operations
+  # - Automatic DLQ fallback for failed publishes
 end
 ```
 
@@ -377,35 +401,12 @@ message.publish  # If transport fails, goes to DLQ
 
 # Manual capture for business logic failures
 dlq = SmartMessage::DeadLetterQueue.default
-dlq.enqueue(header, payload, error: "Validation failed")
+dlq.enqueue(decoded_message, error: "Validation failed", transport: "manual")
 ```
 
 **DLQ Architecture:**
 
-```
-┌─────────────────────────────────────────────────────────────┐
-│                    Message Publishing                       │
-│                           │                                 │
-│                           ▼                                 │
-│                  ┌─────────────────┐                        │
-│                  │ Circuit Breaker │                        │
-│                  │   (Monitoring)   │                        │
-│                  └─────────────────┘                        │
-│                     │           │                           │
-│              Success │           │ Failure                  │
-│                     ▼           ▼                           │
-│              ┌──────────┐ ┌─────────────┐                  │
-│              │Transport │ │Dead Letter  │                  │
-│              │          │ │   Queue     │                  │
-│              └──────────┘ └─────────────┘                  │
-│                                │                            │
-│                                ▼                            │
-│                         ┌─────────────┐                     │
-│                         │   Replay    │                     │
-│                         │ Mechanism   │                     │
-│                         └─────────────┘                     │
-└─────────────────────────────────────────────────────────────┘
-```
+![Dead Letter Queue Architecture](../assets/images/dlq_architecture.svg)
 
 **DLQ Features:**
 - JSON Lines format for efficient append operations
@@ -417,7 +418,7 @@ dlq.enqueue(header, payload, error: "Validation failed")
 
 ### Built-in Statistics
 
-SmartMessage automatically collects processing statistics:
+SmartMessage automatically collects processing statistics including DDQ metrics:
 
 ```ruby
 # Statistics are collected for:
@@ -427,6 +428,11 @@ SS.add(message_class, process_method, 'routed')
 # Access statistics
 puts SS.stat
 puts SS.get("MyMessage", "publish")
+
+# DDQ-specific statistics
+stats = OrderMessage.ddq_stats
+puts "DDQ utilization: #{stats[:utilization]}%"
+puts "Current count: #{stats[:current_count]}"
 ```
 
 ### Monitoring Points
@@ -435,6 +441,8 @@ puts SS.get("MyMessage", "publish")
 2. **Message Routing**: Count of routed messages per processor
 3. **Thread Pool**: Queue length, completed tasks, running status
 4. **Transport Status**: Connection status, message counts
+5. **DDQ Metrics**: Utilization, duplicate detection rates, memory usage
+6. **Message Filtering**: Filter match rates, entity-aware routing statistics
 
 ## Configuration Architecture
 
@@ -466,7 +474,9 @@ When a message needs a plugin:
 
 ```ruby
 def transport
-  @transport || @@transport || raise(Errors::TransportNotConfigured)
+  @transport || self.class.class_variable_get(:@@transport) || raise(Errors::TransportNotConfigured)
+rescue NameError
+  raise(Errors::TransportNotConfigured)
 end
 ```
 

data/docs/{dispatcher.md → core-concepts/dispatcher.md}

@@ -187,10 +187,10 @@ puts dispatcher.subscribers["MyMessage"]
 When a transport receives a message, it calls the dispatcher:
 
 ```ruby
-# Transport receives message and routes it
-transport.receive(message_header, message_payload)
-# This internally calls:
-dispatcher.route(message_header, message_payload)
+# Transport receives serialized message and routes it
+transport.receive(message_class, serialized_message)
+# This internally decodes the message and calls:
+dispatcher.route(decoded_message)
 ```
 
 ### 2. Subscription Lookup
@@ -198,12 +198,12 @@ dispatcher.route(message_header, message_payload)
 The dispatcher finds all registered handlers:
 
 ```ruby
-def route(message_header, message_payload)
-  message_klass = message_header.message_class
+def route(decoded_message)
+  message_klass = decoded_message._sm_header.message_class
   return nil if @subscribers[message_klass].empty?
   
-  @subscribers[message_klass].each do |message_processor|
-    # Process each handler
+  @subscribers[message_klass].each do |subscription|
+    # Process each handler with filters
   end
 end
 ```
@@ -213,16 +213,17 @@ end
 Each handler is processed in its own thread, with support for both method and proc handlers:
 
 ```ruby
-@subscribers[message_klass].each do |message_processor|
+@subscribers[message_klass].each do |subscription|
+  message_processor = subscription[:process_method]
   SS.add(message_klass, message_processor, 'routed')
   
   @router_pool.post do
-    # This runs in a separate thread
-    begin
+    # This runs in a separate thread with circuit breaker protection
+    circuit_result = circuit(:message_processor).wrap do
       # Check if this is a proc handler or a regular method call
       if proc_handler?(message_processor)
         # Call the proc handler via SmartMessage::Base
-        SmartMessage::Base.call_proc_handler(message_processor, message_header, message_payload)
+        SmartMessage::Base.call_proc_handler(message_processor, decoded_message)
       else
         # Original method call logic
         parts = message_processor.split('.')
@@ -231,11 +232,13 @@ Each handler is processed in its own thread, with support for both method and pr
         
         target_klass.constantize
                     .method(class_method)
-                    .call(message_header, message_payload)
+                    .call(decoded_message)
       end
-    rescue Exception => e
-      # Error handling - doesn't crash the dispatcher
-      puts "Error processing message: #{e.message}" if $DEBUG
+    end
+    
+    # Handle circuit breaker fallback if triggered
+    if circuit_result.is_a?(Hash) && circuit_result[:circuit_breaker]
+      handle_circuit_breaker_fallback(circuit_result, decoded_message, message_processor)
     end
   end
 end
@@ -691,7 +694,7 @@ RSpec.describe "Message Processing" do
   before do
     TestMessage.config do
       transport transport
-      serializer SmartMessage::Serializer::JSON.new
+      serializer SmartMessage::Serializer::Json.new
     end
     
     TestMessage.subscribe
@@ -710,7 +713,4 @@ end
 
 ## Next Steps
 
-- [Thread Safety](thread-safety.md) - Understanding concurrent processing
-- [Statistics & Monitoring](monitoring.md) - Detailed monitoring guide
-- [Custom Transports](custom-transports.md) - How transports interact with the dispatcher
-- [Troubleshooting](troubleshooting.md) - Common dispatcher issues
+- [Troubleshooting](../development/troubleshooting.md) - Common dispatcher issues

data/docs/{message_filtering.md → core-concepts/message-filtering.md}

@@ -320,7 +320,7 @@ class FilterTest < Minitest::Test
     @transport = SmartMessage::Transport.create(:memory, auto_process: true)
     TestMessage.config do
       transport @transport
-      serializer SmartMessage::Serializer::JSON.new
+      serializer SmartMessage::Serializer::Json.new
     end
     TestMessage.unsubscribe!
   end
@@ -447,5 +447,4 @@ end
 
 - [Dispatcher Documentation](dispatcher.md) - How filtering integrates with message routing
 - [Entity Addressing](addressing.md) - Understanding `from`, `to`, and `reply_to` fields  
-- [Examples](examples.md) - Complete working examples with filtering
-- [Testing Guide](testing.md) - Best practices for testing filtered subscriptions
+- [Examples](../getting-started/examples.md) - Complete working examples with filtering

data/docs/{message_processing.md → core-concepts/message-processing.md}

@@ -36,9 +36,9 @@ SmartMessage supports multiple ways to handle incoming messages:
 ### 1. Default Handler Pattern (using `self.process`)
 ```ruby
 class SensorDataMessage < SmartMessage::Base
-  def self.process(message_header, message_payload)
+  def self.process(decoded_message)
     # This gets called when a SensorDataMessage is received
-    data = JSON.parse(message_payload)
+    # decoded_message is already a message instance
     puts "Sensor reading: #{data['value']}"
   end
 end
@@ -51,7 +51,7 @@ SensorDataMessage.subscribe  # Uses "SensorDataMessage.process"
 class ThermostatService
   def self.handle_sensor_data(message_header, message_payload)
     # Custom processing logic
-    data = JSON.parse(message_payload)
+    # decoded_message is already a message instance
     adjust_temperature(data)
   end
 end
@@ -99,8 +99,8 @@ Looking at the smart home IoT example:
 
 ```ruby
 class SensorDataMessage < SmartMessage::Base
-  def self.process(message_header, message_payload)
-    sensor_data = JSON.parse(message_payload)
+  def self.process(decoded_message)
+    sensor_# decoded_message is already a message instance
     icon = case sensor_data['device_type']
            when 'thermostat' then '🌡️'
            when 'security_camera' then '📹'
@@ -145,8 +145,8 @@ A single message type can have multiple subscribers with different handlers usin
 ```ruby
 # Default handler for logging
 class SensorDataMessage < SmartMessage::Base
-  def self.process(message_header, message_payload)
-    data = JSON.parse(message_payload)
+  def self.process(decoded_message)
+    # decoded_message is already a message instance
     puts "📊 Sensor data logged: #{data['device_id']}"
   end
 end
@@ -154,7 +154,7 @@ end
 # Custom method handler for specific services
 class ThermostatService
   def self.handle_sensor_data(message_header, message_payload)
-    data = JSON.parse(message_payload)
+    # decoded_message is already a message instance
     return unless data['device_type'] == 'thermostat'
     adjust_temperature(data['value'])
   end
@@ -197,10 +197,10 @@ SensorDataMessage.subscribe(database_logger)  # Proc handler
 
 ```ruby
 class SensorDataMessage < SmartMessage::Base
-  def self.process(message_header, message_payload)
+  def self.process(decoded_message)
     # This runs in its own thread
     # Be careful with shared state
-    data = JSON.parse(message_payload)
+    # decoded_message is already a message instance
     
     # Thread-safe operations
     update_local_cache(data)
@@ -216,9 +216,9 @@ Handlers should include proper error handling:
 
 ```ruby
 class SensorDataMessage < SmartMessage::Base
-  def self.process(message_header, message_payload)
+  def self.process(decoded_message)
     begin
-      data = JSON.parse(message_payload)
+      # decoded_message is already a message instance
       
       # Validate required fields
       raise "Missing device_id" unless data['device_id']
@@ -310,9 +310,9 @@ PaymentEventMessage.subscribe(audit_logger)
 
 ### 1. Keep Handlers Fast
 ```ruby
-def self.process(message_header, message_payload)
+def self.process(decoded_message)
   # Quick validation
-  data = JSON.parse(message_payload)
+  # decoded_message is already a message instance
   return unless valid_message?(data)
   
   # Delegate heavy work to background jobs
@@ -348,7 +348,7 @@ SensorDataMessage.subscribe do |h, p|; process_stuff(p); end
 ### 3. Filter Messages Early
 ```ruby
 def self.handle_thermostat_data(message_header, message_payload)
-  data = JSON.parse(message_payload)
+  # decoded_message is already a message instance
   
   # Filter early to avoid unnecessary processing
   return unless data['device_type'] == 'thermostat'
@@ -361,11 +361,11 @@ end
 
 ### 4. Include Logging and Monitoring
 ```ruby
-def self.process(message_header, message_payload)
+def self.process(decoded_message)
   start_time = Time.now
   
   begin
-    data = JSON.parse(message_payload)
+    # decoded_message is already a message instance
     logger.info "Processing sensor data from #{data['device_id']}"
     
     # Business logic here