smart_message 0.0.8 → 0.0.10

data/docs/README.md

@@ -17,11 +17,13 @@ Welcome to the comprehensive documentation for SmartMessage, a Ruby gem that abs
 - [SmartMessage::Base](base.md)
 - [Property System](properties.md)
 - [Entity Addressing](addressing.md)
+- [Message Filtering](message_filtering.md)
 - [Transport Layer](transports.md)
 - [Serializers](serializers.md)
 - [Logging System](logging.md)
 - [Dispatcher & Routing](dispatcher.md)
 - [Message Headers](headers.md)
+- [Dead Letter Queue](dead_letter_queue.md)
 
 ### Advanced Topics
 - [Custom Transports](custom-transports.md)
@@ -50,6 +52,6 @@ Welcome to the comprehensive documentation for SmartMessage, a Ruby gem that abs
 
 ## Version
 
-This documentation is for SmartMessage v0.0.2.
+This documentation is for SmartMessage v0.0.8.
 
 For older versions, please check the git tags and corresponding documentation.

data/docs/addressing.md

@@ -63,13 +63,14 @@ end
 
 ### Class-Level Configuration
 
-Set default addressing for all instances of a message class:
+Set default addressing for all instances of a message class using three different approaches:
 
+#### Method 1: Direct Class Methods
 ```ruby
 class PaymentMessage < SmartMessage::Base
   version 1
   
-  # Class-level addressing
+  # Class-level addressing using direct methods
   from 'payment-service'
   to 'bank-gateway'
   reply_to 'payment-service'
@@ -77,6 +78,42 @@ class PaymentMessage < SmartMessage::Base
   property :amount, required: true
   property :account_id, required: true
 end
+```
+
+#### Method 2: Header Block DSL
+```ruby
+class PaymentMessage < SmartMessage::Base
+  version 1
+  
+  # Class-level addressing using header block
+  header do
+    from 'payment-service'
+    to 'bank-gateway'
+    reply_to 'payment-service'
+  end
+  
+  property :amount, required: true
+  property :account_id, required: true
+end
+```
+
+#### Method 3: Mixed Approach
+```ruby
+class PaymentMessage < SmartMessage::Base
+  version 1
+  
+  # You can mix approaches if needed
+  header do
+    from 'payment-service'
+    to 'bank-gateway'
+  end
+  
+  # Additional addressing outside the block
+  reply_to 'payment-service'
+  
+  property :amount, required: true
+  property :account_id, required: true
+end
 
 # All instances inherit class addressing
 payment = PaymentMessage.new(amount: 100.00, account_id: "ACCT-123")
@@ -87,27 +124,54 @@ puts payment._sm_header.reply_to # => 'payment-service'
 
 ### Instance-Level Overrides
 
-Override addressing for specific message instances:
+Override addressing for specific message instances using multiple approaches:
 
+#### Method Chaining
 ```ruby
 class FlexibleMessage < SmartMessage::Base
-  from 'service-a'
-  to 'service-b'
+  header do
+    from 'service-a'
+    to 'service-b'
+  end
   
   property :data
 end
 
-# Override addressing for this instance
+# Override addressing using method chaining
 message = FlexibleMessage.new(data: "test")
 message.from('different-sender')
-message.to('different-recipient')
-message.reply_to('different-reply-service')
+       .to('different-recipient')
+       .reply_to('different-reply-service')
 
 puts message.from     # => 'different-sender'
 puts message.to       # => 'different-recipient' 
 puts message.reply_to # => 'different-reply-service'
+```
+
+#### Setter Methods
+```ruby
+# Override addressing using setter syntax
+message = FlexibleMessage.new(data: "test")
+message.from = 'different-sender'
+message.to = 'different-recipient'
+message.reply_to = 'different-reply-service'
+```
 
-# Class defaults remain unchanged
+#### Accessing Addressing Values
+```ruby
+# Three ways to access addressing values
+
+# 1. Direct shortcut methods
+puts message.from      # => 'different-sender'
+puts message.to        # => 'different-recipient'
+puts message.reply_to  # => 'different-reply-service'
+
+# 2. Via header object
+puts message._sm_header.from      # => 'different-sender'
+puts message._sm_header.to        # => 'different-recipient'
+puts message._sm_header.reply_to  # => 'different-reply-service'
+
+# 3. Class defaults remain unchanged
 puts FlexibleMessage.from # => 'service-a'
 puts FlexibleMessage.to   # => 'service-b'
 ```
@@ -257,27 +321,69 @@ message.publish  # Raises SmartMessage::Errors::ValidationError
 # => "The property 'from' From entity ID is required for message routing and replies"
 ```
 
+### Checking Address Configuration
+```ruby
+class ValidatedMessage < SmartMessage::Base
+  header do
+    from 'my-service'
+    to 'target-service'
+  end
+  
+  property :data
+end
+
+# Class-level checks
+puts ValidatedMessage.from_configured?     # => true
+puts ValidatedMessage.to_configured?       # => true
+puts ValidatedMessage.reply_to_configured? # => false
+puts ValidatedMessage.reply_to_missing?    # => true
+
+# Instance-level checks
+message = ValidatedMessage.new(data: "test")
+puts message.from_configured?     # => true
+puts message.to_missing?          # => false
+
+# Reset addressing
+message.reset_from
+puts message.from_configured?     # => false
+puts message.from                 # => nil
+```
+
 ## Header Access
 
 Access addressing information from message headers:
 
 ```ruby
 class SampleMessage < SmartMessage::Base
-  from 'sample-service'
-  to 'target-service'
-  reply_to 'callback-service'
+  # Using header block for configuration
+  header do
+    from 'sample-service'
+    to 'target-service'
+    reply_to 'callback-service'
+  end
   
   property :content
 end
 
 message = SampleMessage.new(content: "Hello")
-header = message._sm_header
 
+# Access via shortcut methods (recommended)
+puts message.from      # => 'sample-service'
+puts message.to        # => 'target-service'  
+puts message.reply_to  # => 'callback-service'
+
+# Access via header object
+header = message._sm_header
 puts header.from      # => 'sample-service'
 puts header.to        # => 'target-service'  
 puts header.reply_to  # => 'callback-service'
 puts header.uuid      # => Generated UUID
 puts header.message_class  # => 'SampleMessage'
+
+# Headers automatically sync with instance changes
+message.to = 'new-target'
+puts message.to           # => 'new-target'
+puts header.to            # => 'new-target' (automatically updated)
 ```
 
 ## Integration with Dispatcher

data/docs/architecture.md

@@ -16,37 +16,53 @@ SmartMessage is designed around the principle that **messages should be independ
 
 ## Architecture Overview
 
-```
-┌─────────────────────────────────────────────────────────────┐
-│                    SmartMessage::Base                       │
-│  ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐│
-│  │    Message      │ │   Transport     │ │   Serializer    ││
-│  │   Properties    │ │     Plugin      │ │     Plugin      ││
-│  │                 │ │                 │ │                 ││
-│  │ • user_id       │ │ • publish()     │ │ • encode()      ││
-│  │ • action        │ │ • subscribe()   │ │ • decode()      ││
-│  │ • timestamp     │ │ • receive()     │ │                 ││
-│  └─────────────────┘ └─────────────────┘ └─────────────────┘│
-└─────────────────────────────────────────────────────────────┘
-                                │
-                                ▼
-                    ┌─────────────────────┐
-                    │     Dispatcher      │
-                    │                     │
-                    │ • Route messages    │
-                    │ • Thread pool       │
-                    │ • Subscriptions     │
-                    └─────────────────────┘
-                                │
-                                ▼
-                    ┌─────────────────────┐
-                    │  Message Handlers   │
-                    │                     │
-                    │ • Default handler   │
-                    │ • Block handlers    │
-                    │ • Proc handlers     │
-                    │ • Method handlers   │
-                    └─────────────────────┘
+```mermaid
+graph TB
+    subgraph "SmartMessage Core"
+        Base[SmartMessage::Base]
+        Header[Message Header<br/>• UUID<br/>• Timestamps<br/>• Addressing]
+        Props[Message Properties<br/>• Business Data<br/>• Validation<br/>• Versioning]
+    end
+    
+    subgraph "Plugin System"
+        Transport[Transport Plugin<br/>• publish()<br/>• subscribe()<br/>• Memory/Redis/STDOUT]
+        Serializer[Serializer Plugin<br/>• encode()<br/>• decode()<br/>• JSON/Custom]
+        Logger[Logger Plugin<br/>• Structured logging<br/>• Multiple outputs<br/>• Colorization]
+    end
+    
+    subgraph "Message Processing"
+        Dispatcher[Dispatcher<br/>• Route messages<br/>• Thread pool<br/>• Subscriptions<br/>• DDQ management]
+        DDQ[Deduplication Queue<br/>• Handler-scoped<br/>• Memory/Redis storage<br/>• O(1) performance<br/>• Circular buffer]
+        Handlers[Message Handlers<br/>• Default handler<br/>• Block handlers<br/>• Proc handlers<br/>• Method handlers]
+    end
+    
+    subgraph "Reliability Layer"
+        CircuitBreaker[Circuit Breaker<br/>• Failure thresholds<br/>• Automatic fallback<br/>• Recovery detection]
+        DLQ[Dead Letter Queue<br/>• Failed messages<br/>• Replay mechanism<br/>• JSON Lines format]
+    end
+    
+    subgraph "Monitoring"
+        Stats[Statistics<br/>• Message counts<br/>• Processing metrics<br/>• Thread pool status]
+        Filters[Message Filtering<br/>• Entity-aware routing<br/>• Regex patterns<br/>• Broadcast handling]
+    end
+    
+    Base --> Header
+    Base --> Props
+    Base --> Transport
+    Base --> Serializer
+    Base --> Logger
+    
+    Transport --> Dispatcher
+    Dispatcher --> DDQ
+    Dispatcher --> Handlers
+    Dispatcher --> Stats
+    Dispatcher --> Filters
+    
+    Transport --> CircuitBreaker
+    CircuitBreaker --> DLQ
+    
+    DDQ -.-> Stats
+    Handlers -.-> Stats
 ```
 
 ## Core Components
@@ -126,38 +142,84 @@ end
 
 ### 4. Dispatcher
 
-Routes incoming messages to appropriate handlers using concurrent processing.
+Routes incoming messages to appropriate handlers using concurrent processing with integrated deduplication.
 
 **Key Responsibilities:**
 - Message routing based on class
-- Thread pool management
+- Thread pool management  
 - Subscription catalog management
-- Statistics collection
+- Handler-scoped DDQ management
+- Message filtering and statistics collection
 
-**Location:** `lib/smart_message/dispatcher.rb:11-147`
+**Location:** `lib/smart_message/dispatcher.rb`
 
 ```ruby
 dispatcher = SmartMessage::Dispatcher.new
 dispatcher.add("MyMessage", "MyMessage.process")
 dispatcher.route(header, payload)
+
+# DDQ integration is automatic when enabled
+MyMessage.enable_deduplication!
+```
+
+### 5. Deduplication Queue (DDQ)
+
+Handler-scoped message deduplication system preventing duplicate processing.
+
+**Key Responsibilities:**
+- UUID-based duplicate detection
+- Handler isolation (each handler gets own DDQ)
+- Memory and Redis storage backends
+- O(1) performance with hybrid Array + Set data structure
+
+**Architecture:**
+```mermaid
+graph LR
+    subgraph "Handler A DDQ"
+        A1[Circular Array]
+        A2[Lookup Set]
+        A3[Mutex Lock]
+    end
+    
+    subgraph "Handler B DDQ"
+        B1[Circular Array]
+        B2[Lookup Set] 
+        B3[Mutex Lock]
+    end
+    
+    Message[Incoming Message<br/>UUID: abc-123] --> Dispatcher
+    Dispatcher --> |Check Handler A| A2
+    Dispatcher --> |Check Handler B| B2
+    
+    A2 --> |Not Found| ProcessA[Process with Handler A]
+    B2 --> |Found| SkipB[Skip Handler B - Duplicate]
+    
+    ProcessA --> |Add UUID| A1
+    ProcessA --> |Add UUID| A2
 ```
 
-### 5. Message Headers
+**Location:** `lib/smart_message/deduplication.rb`, `lib/smart_message/ddq/`
+
+### 6. Message Headers
 
-Standard metadata attached to every message.
+Standard metadata attached to every message with entity addressing support.
 
 **Key Responsibilities:**
 - Message identification (UUID)
-- Routing information (message class)
+- Routing information (message class, version)
 - Tracking data (timestamps, process IDs)
+- Entity addressing (from, to, reply_to)
 
-**Location:** `lib/smart_message/header.rb:9-20`
+**Location:** `lib/smart_message/header.rb`
 
 ```ruby
 header = message._sm_header
 puts header.uuid          # "550e8400-e29b-41d4-a716-446655440000"
 puts header.message_class # "MyMessage"
 puts header.published_at  # 2025-08-17 10:30:00 UTC
+puts header.from          # "payment-service"
+puts header.to            # "order-service"
+puts header.version       # 1
 ```
 
 ## Message Lifecycle
@@ -177,17 +239,27 @@ end
 
 ### 2. Subscription Phase
 ```ruby
+# Basic subscription
 OrderMessage.subscribe
-# Registers "OrderMessage.process" with dispatcher
+
+# Subscription with filtering
+OrderMessage.subscribe(from: /^payment-.*/, to: 'order-service')
+OrderMessage.subscribe('PaymentService.process', broadcast: true)
+
+# Each subscription gets its own DDQ automatically
+# DDQ Key: "OrderMessage:OrderMessage.process"
+# DDQ Key: "OrderMessage:PaymentService.process"
 ```
 
 ### 3. Publishing Phase
 ```ruby
 order = OrderMessage.new(order_id: "123", amount: 99.99)
+order.from("order-service").to("payment-service")
 order.publish
-# 1. Creates header with UUID, timestamp, etc.
-# 2. Encodes message via serializer
+# 1. Creates header with UUID, timestamp, addressing
+# 2. Encodes message via serializer  
 # 3. Sends via transport
+# 4. Circuit breaker monitors for failures
 ```
 
 ### 4. Receiving Phase
@@ -195,9 +267,10 @@ order.publish
 # Transport receives message
 transport.receive(header, payload)
 # 1. Routes to dispatcher
-# 2. Dispatcher finds subscribers
-# 3. Spawns thread for processing
-# 4. Calls registered message handlers
+# 2. Dispatcher checks DDQ for duplicates per handler
+# 3. Applies message filters (from/to/broadcast)
+# 4. Spawns thread for processing matching handlers
+# 5. Marks UUID as processed in handler's DDQ
 ```
 
 ### 5. Message Handler Processing
@@ -345,11 +418,69 @@ module SmartMessage::Errors
 end
 ```
 
+## Reliability & Fault Tolerance
+
+### Circuit Breaker Integration
+
+SmartMessage integrates BreakerMachines for production-grade reliability:
+
+```ruby
+# Transport operations protected by circuit breakers
+class MyTransport < SmartMessage::Transport::Base
+  circuit :transport_publish do
+    threshold failures: 5, within: 30.seconds
+    reset_after 15.seconds
+    fallback SmartMessage::CircuitBreaker::Fallbacks.dead_letter_queue
+  end
+end
+```
+
+**Circuit Breaker States:**
+- **Closed**: Normal operation, requests pass through
+- **Open**: Threshold exceeded, requests fail fast
+- **Half-Open**: Testing if service recovered
+
+### Dead Letter Queue
+
+Failed messages are automatically captured in the Dead Letter Queue:
+
+```ruby
+# Automatic capture when circuit breaker trips
+message.publish  # If transport fails, goes to DLQ
+
+# Manual capture for business logic failures
+dlq = SmartMessage::DeadLetterQueue.default
+dlq.enqueue(header, payload, error: "Validation failed")
+```
+
+**DLQ Architecture:**
+
+```mermaid
+graph TB
+    Publish[Message Publishing]
+    CB[Circuit Breaker<br/>Monitoring]
+    Transport[Transport<br/>Success]
+    DLQ[Dead Letter Queue<br/>Failure Storage]
+    Replay[Replay Mechanism<br/>Manual/Automated]
+    
+    Publish --> CB
+    CB --> |Success| Transport
+    CB --> |Failure| DLQ
+    DLQ --> Replay
+    Replay --> |Retry| Publish
+```
+
+**DLQ Features:**
+- JSON Lines format for efficient append operations
+- FIFO queue operations with thread safety
+- Replay capabilities with transport override
+- Administrative tools for filtering and analysis
+
 ## Statistics & Monitoring
 
 ### Built-in Statistics
 
-SmartMessage automatically collects processing statistics:
+SmartMessage automatically collects processing statistics including DDQ metrics:
 
 ```ruby
 # Statistics are collected for:
@@ -359,6 +490,11 @@ SS.add(message_class, process_method, 'routed')
 # Access statistics
 puts SS.stat
 puts SS.get("MyMessage", "publish")
+
+# DDQ-specific statistics
+stats = OrderMessage.ddq_stats
+puts "DDQ utilization: #{stats[:utilization]}%"
+puts "Current count: #{stats[:current_count]}"
 ```
 
 ### Monitoring Points
@@ -367,6 +503,8 @@ puts SS.get("MyMessage", "publish")
 2. **Message Routing**: Count of routed messages per processor
 3. **Thread Pool**: Queue length, completed tasks, running status
 4. **Transport Status**: Connection status, message counts
+5. **DDQ Metrics**: Utilization, duplicate detection rates, memory usage
+6. **Message Filtering**: Filter match rates, entity-aware routing statistics
 
 ## Configuration Architecture