smart_message 0.0.9 → 0.0.10

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 5f02a7e60919734663addbef7ae2b5582bc142993654ef33fe17c90e1a70d138
-  data.tar.gz: 50e7250d3e217faba23eb2a1d995b6edff0d9e0d7935bf2ea31602aff483a092
+  metadata.gz: 8485ad851b6868c57b813977a4b2be3b1e06fe2843510a5b64c79468da3a65cf
+  data.tar.gz: 1c2345c52d1dd57202a83a49d3d4e7769f14b2855d0a08745d3ed4c970ef0f0e
 SHA512:
-  metadata.gz: 7c157a60a127d4bd7a86742ccee8dff5385f11110d1449bc859742c497cc79d9a0e528dd39645d66a8e2b413602c31c4c79f73c9c560b5aacb37acc9610d3183
-  data.tar.gz: 31fc0e365c549326e9a9314f435d50715c4c980e7380311e19a7428638198d2ce499f8acd104381a631b2a40cd66d2deb8bb7fa3e16c59a61a443cc980698b56
+  metadata.gz: 26cfdd1030e880074e557f7bc315220995b3267ec0a11fd4d0f621933e2101f40f35da425649388b83bf8055246934e6c9f8e60a281246e04e5cd89763c33e07
+  data.tar.gz: 25d66186e9ebe3f3fcf595e7093ec08c50db83d84205fcb4567b0c69804e44b80496693c40e36a0ea30b9f315a36d2070e701977bec6a0660f3c3bda327f1d88

data/CHANGELOG.md

@@ -7,6 +7,29 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [0.0.10] 2025-08-20
+### Added
+- **Handler-Scoped Message Deduplication**: Advanced DDQ (Deduplication Queue) system with automatic handler isolation
+  - Handler-only DDQ scoping using `"MessageClass:HandlerMethod"` key format for natural isolation
+  - Automatic subscriber identification eliminates need for explicit `me:` parameter in API
+  - Cross-process deduplication support with independent DDQ instances per handler
+  - Memory and Redis storage backends with O(1) performance using hybrid Array + Set data structure
+  - Seamless integration with dispatcher - no changes required to existing subscription code
+  - Per-handler statistics and monitoring capabilities with DDQ utilization tracking
+  - Thread-safe circular buffer implementation with automatic eviction of oldest entries
+  - Configuration DSL: `ddq_size`, `ddq_storage`, `enable_deduplication!`, `disable_deduplication!`
+  - Comprehensive documentation and examples demonstrating distributed message processing scenarios
+
+### Fixed
+- **Message Filtering Logic**: Fixed critical bug in subscription filtering where nil broadcast filters caused incorrect behavior
+  - **Root Cause**: `filters.key?(:broadcast)` returned true even when broadcast value was nil
+  - **Solution**: Changed condition to check for non-nil values: `filters[:to] || filters[:broadcast]`
+  - **Impact**: Subscription filtering now works correctly with nil filter values and maintains backward compatibility
+- **Circuit Breaker Test Compatibility**: Added error handling for non-existent message classes in DDQ initialization
+  - Added rescue blocks around `constantize` calls to gracefully handle fake test classes
+  - DDQ initialization now skips gracefully for test classes that cannot be constantized
+  - Maintains proper DDQ functionality for real message classes while supporting test scenarios
+
 ## [0.0.9] 2025-08-20
 ### Added
 - **Advanced Logging Configuration System**: Comprehensive logger configuration with multiple output formats and options

data/Gemfile.lock

@@ -1,7 +1,7 @@
 PATH
   remote: .
   specs:
-    smart_message (0.0.9)
+    smart_message (0.0.10)
       activesupport
       breaker_machines
       colorize

data/README.md

@@ -19,6 +19,7 @@ SmartMessage is a message abstraction framework that decouples business logic fr
 - **Concurrent Processing**: Thread-safe message routing using `Concurrent::CachedThreadPool`
 - **Advanced Logging System**: Comprehensive logging with colorized console output, JSON structured logging, and file rolling
 - **Built-in Statistics**: Message processing metrics and monitoring
+- **Message Deduplication**: Handler-scoped deduplication queues (DDQ) with memory or Redis storage for preventing duplicate message processing
 - **Development Tools**: STDOUT and in-memory transports for testing
 - **Production Ready**: Redis transport with automatic reconnection and error handling
 - **Dead Letter Queue**: File-based DLQ with JSON Lines format for failed message capture and replay
@@ -201,7 +202,103 @@ DevService.subscribe(to: /^(dev|staging)-.*/)
 ProdService.subscribe(to: /^prod-.*/)
 ```
 
-### 5. Entity Addressing
+### 5. Message Deduplication
+
+SmartMessage provides handler-scoped message deduplication to prevent duplicate processing of messages with the same UUID. Each handler gets its own Deduplication Queue (DDQ) that tracks recently processed message UUIDs.
+
+#### Basic Deduplication Setup
+
+```ruby
+class OrderMessage < SmartMessage::Base
+  version 1
+  property :order_id, required: true
+  property :amount, required: true
+  
+  from "order-service"
+  
+  # Configure deduplication
+  ddq_size 100              # Track last 100 message UUIDs
+  ddq_storage :memory       # Use memory storage (or :redis for distributed)
+  enable_deduplication!     # Enable deduplication for this message class
+  
+  def self.process(message)
+    puts "Processing order: #{message.order_id}"
+    # Business logic here
+  end
+end
+```
+
+#### Handler-Scoped Isolation
+
+Each handler gets its own DDQ scope, preventing cross-contamination between different subscribers:
+
+```ruby
+# Each handler gets separate deduplication tracking
+OrderMessage.subscribe('PaymentService.process')     # DDQ: "OrderMessage:PaymentService.process"
+OrderMessage.subscribe('FulfillmentService.handle')  # DDQ: "OrderMessage:FulfillmentService.handle"
+OrderMessage.subscribe('AuditService.log_order')     # DDQ: "OrderMessage:AuditService.log_order"
+
+# Same handler across message classes = separate DDQs
+PaymentMessage.subscribe('PaymentService.process')   # DDQ: "PaymentMessage:PaymentService.process"
+InvoiceMessage.subscribe('PaymentService.process')   # DDQ: "InvoiceMessage:PaymentService.process"
+```
+
+#### Storage Options
+
+```ruby
+# Memory-based DDQ (single process)
+class LocalMessage < SmartMessage::Base
+  ddq_size 50
+  ddq_storage :memory
+  enable_deduplication!
+end
+
+# Redis-based DDQ (distributed/multi-process)
+class DistributedMessage < SmartMessage::Base
+  ddq_size 1000
+  ddq_storage :redis, redis_url: 'redis://localhost:6379', redis_db: 1
+  enable_deduplication!
+end
+```
+
+#### DDQ Statistics and Management
+
+```ruby
+# Check deduplication configuration
+config = OrderMessage.ddq_config
+puts "Enabled: #{config[:enabled]}"
+puts "Size: #{config[:size]}"
+puts "Storage: #{config[:storage]}"
+
+# Get DDQ statistics
+stats = OrderMessage.ddq_stats
+puts "Current count: #{stats[:current_count]}"
+puts "Utilization: #{stats[:utilization]}%"
+
+# Clear DDQ if needed
+OrderMessage.clear_ddq!
+
+# Check if specific UUID is duplicate
+OrderMessage.duplicate_uuid?("some-uuid-123")
+```
+
+#### How Deduplication Works
+
+1. **Message Receipt**: When a message arrives, the dispatcher checks the handler's DDQ for the message UUID
+2. **Duplicate Detection**: If UUID exists in DDQ, the message is ignored (logged but not processed)
+3. **Processing**: If UUID is new, the message is processed by the handler
+4. **UUID Storage**: After successful processing, the UUID is added to the handler's DDQ
+5. **Circular Buffer**: When DDQ reaches capacity, oldest UUIDs are evicted to make room for new ones
+
+#### Benefits
+
+- **Handler Isolation**: Each handler maintains independent deduplication state
+- **Cross-Process Support**: Redis DDQ enables deduplication across multiple processes
+- **Memory Efficient**: Circular buffer with configurable size limits memory usage
+- **High Performance**: O(1) UUID lookup using hybrid array + set data structure
+- **Automatic Integration**: Seamlessly works with existing subscription patterns
+
+### 6. Entity Addressing
 
 SmartMessage supports entity-to-entity addressing with FROM/TO/REPLY_TO fields for advanced message routing. You can configure addressing using three different approaches:
 
@@ -412,9 +509,10 @@ Pluggable message encoding/decoding:
 #### Dispatcher
 Concurrent message routing engine that:
 - Uses thread pools for async processing
-- Routes messages to subscribed handlers
-- Provides processing statistics
+- Routes messages to subscribed handlers with handler-scoped deduplication
+- Provides processing statistics and DDQ management
 - Handles graceful shutdown
+- Maintains separate DDQ instances per handler for isolated deduplication tracking
 
 ### Plugin Architecture
 

data/docs/architecture.md

@@ -16,37 +16,53 @@ SmartMessage is designed around the principle that **messages should be independ
 
 ## Architecture Overview
 
-```
-┌─────────────────────────────────────────────────────────────┐
-│                    SmartMessage::Base                       │
-│  ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐│
-│  │    Message      │ │   Transport     │ │   Serializer    ││
-│  │   Properties    │ │     Plugin      │ │     Plugin      ││
-│  │                 │ │                 │ │                 ││
-│  │ • user_id       │ │ • publish()     │ │ • encode()      ││
-│  │ • action        │ │ • subscribe()   │ │ • decode()      ││
-│  │ • timestamp     │ │ • receive()     │ │                 ││
-│  └─────────────────┘ └─────────────────┘ └─────────────────┘│
-└─────────────────────────────────────────────────────────────┘
-                                │
-                                ▼
-                    ┌─────────────────────┐
-                    │     Dispatcher      │
-                    │                     │
-                    │ • Route messages    │
-                    │ • Thread pool       │
-                    │ • Subscriptions     │
-                    └─────────────────────┘
-                                │
-                                ▼
-                    ┌─────────────────────┐
-                    │  Message Handlers   │
-                    │                     │
-                    │ • Default handler   │
-                    │ • Block handlers    │
-                    │ • Proc handlers     │
-                    │ • Method handlers   │
-                    └─────────────────────┘
+```mermaid
+graph TB
+    subgraph "SmartMessage Core"
+        Base[SmartMessage::Base]
+        Header[Message Header<br/>• UUID<br/>• Timestamps<br/>• Addressing]
+        Props[Message Properties<br/>• Business Data<br/>• Validation<br/>• Versioning]
+    end
+    
+    subgraph "Plugin System"
+        Transport[Transport Plugin<br/>• publish()<br/>• subscribe()<br/>• Memory/Redis/STDOUT]
+        Serializer[Serializer Plugin<br/>• encode()<br/>• decode()<br/>• JSON/Custom]
+        Logger[Logger Plugin<br/>• Structured logging<br/>• Multiple outputs<br/>• Colorization]
+    end
+    
+    subgraph "Message Processing"
+        Dispatcher[Dispatcher<br/>• Route messages<br/>• Thread pool<br/>• Subscriptions<br/>• DDQ management]
+        DDQ[Deduplication Queue<br/>• Handler-scoped<br/>• Memory/Redis storage<br/>• O(1) performance<br/>• Circular buffer]
+        Handlers[Message Handlers<br/>• Default handler<br/>• Block handlers<br/>• Proc handlers<br/>• Method handlers]
+    end
+    
+    subgraph "Reliability Layer"
+        CircuitBreaker[Circuit Breaker<br/>• Failure thresholds<br/>• Automatic fallback<br/>• Recovery detection]
+        DLQ[Dead Letter Queue<br/>• Failed messages<br/>• Replay mechanism<br/>• JSON Lines format]
+    end
+    
+    subgraph "Monitoring"
+        Stats[Statistics<br/>• Message counts<br/>• Processing metrics<br/>• Thread pool status]
+        Filters[Message Filtering<br/>• Entity-aware routing<br/>• Regex patterns<br/>• Broadcast handling]
+    end
+    
+    Base --> Header
+    Base --> Props
+    Base --> Transport
+    Base --> Serializer
+    Base --> Logger
+    
+    Transport --> Dispatcher
+    Dispatcher --> DDQ
+    Dispatcher --> Handlers
+    Dispatcher --> Stats
+    Dispatcher --> Filters
+    
+    Transport --> CircuitBreaker
+    CircuitBreaker --> DLQ
+    
+    DDQ -.-> Stats
+    Handlers -.-> Stats
 ```
 
 ## Core Components
@@ -126,38 +142,84 @@ end
 
 ### 4. Dispatcher
 
-Routes incoming messages to appropriate handlers using concurrent processing.
+Routes incoming messages to appropriate handlers using concurrent processing with integrated deduplication.
 
 **Key Responsibilities:**
 - Message routing based on class
-- Thread pool management
+- Thread pool management  
 - Subscription catalog management
-- Statistics collection
+- Handler-scoped DDQ management
+- Message filtering and statistics collection
 
-**Location:** `lib/smart_message/dispatcher.rb:11-147`
+**Location:** `lib/smart_message/dispatcher.rb`
 
 ```ruby
 dispatcher = SmartMessage::Dispatcher.new
 dispatcher.add("MyMessage", "MyMessage.process")
 dispatcher.route(header, payload)
+
+# DDQ integration is automatic when enabled
+MyMessage.enable_deduplication!
+```
+
+### 5. Deduplication Queue (DDQ)
+
+Handler-scoped message deduplication system preventing duplicate processing.
+
+**Key Responsibilities:**
+- UUID-based duplicate detection
+- Handler isolation (each handler gets own DDQ)
+- Memory and Redis storage backends
+- O(1) performance with hybrid Array + Set data structure
+
+**Architecture:**
+```mermaid
+graph LR
+    subgraph "Handler A DDQ"
+        A1[Circular Array]
+        A2[Lookup Set]
+        A3[Mutex Lock]
+    end
+    
+    subgraph "Handler B DDQ"
+        B1[Circular Array]
+        B2[Lookup Set] 
+        B3[Mutex Lock]
+    end
+    
+    Message[Incoming Message<br/>UUID: abc-123] --> Dispatcher
+    Dispatcher --> |Check Handler A| A2
+    Dispatcher --> |Check Handler B| B2
+    
+    A2 --> |Not Found| ProcessA[Process with Handler A]
+    B2 --> |Found| SkipB[Skip Handler B - Duplicate]
+    
+    ProcessA --> |Add UUID| A1
+    ProcessA --> |Add UUID| A2
 ```
 
-### 5. Message Headers
+**Location:** `lib/smart_message/deduplication.rb`, `lib/smart_message/ddq/`
+
+### 6. Message Headers
 
-Standard metadata attached to every message.
+Standard metadata attached to every message with entity addressing support.
 
 **Key Responsibilities:**
 - Message identification (UUID)
-- Routing information (message class)
+- Routing information (message class, version)
 - Tracking data (timestamps, process IDs)
+- Entity addressing (from, to, reply_to)
 
-**Location:** `lib/smart_message/header.rb:9-20`
+**Location:** `lib/smart_message/header.rb`
 
 ```ruby
 header = message._sm_header
 puts header.uuid          # "550e8400-e29b-41d4-a716-446655440000"
 puts header.message_class # "MyMessage"
 puts header.published_at  # 2025-08-17 10:30:00 UTC
+puts header.from          # "payment-service"
+puts header.to            # "order-service"
+puts header.version       # 1
 ```
 
 ## Message Lifecycle
@@ -177,17 +239,27 @@ end
 
 ### 2. Subscription Phase
 ```ruby
+# Basic subscription
 OrderMessage.subscribe
-# Registers "OrderMessage.process" with dispatcher
+
+# Subscription with filtering
+OrderMessage.subscribe(from: /^payment-.*/, to: 'order-service')
+OrderMessage.subscribe('PaymentService.process', broadcast: true)
+
+# Each subscription gets its own DDQ automatically
+# DDQ Key: "OrderMessage:OrderMessage.process"
+# DDQ Key: "OrderMessage:PaymentService.process"
 ```
 
 ### 3. Publishing Phase
 ```ruby
 order = OrderMessage.new(order_id: "123", amount: 99.99)
+order.from("order-service").to("payment-service")
 order.publish
-# 1. Creates header with UUID, timestamp, etc.
-# 2. Encodes message via serializer
+# 1. Creates header with UUID, timestamp, addressing
+# 2. Encodes message via serializer  
 # 3. Sends via transport
+# 4. Circuit breaker monitors for failures
 ```
 
 ### 4. Receiving Phase
@@ -195,9 +267,10 @@ order.publish
 # Transport receives message
 transport.receive(header, payload)
 # 1. Routes to dispatcher
-# 2. Dispatcher finds subscribers
-# 3. Spawns thread for processing
-# 4. Calls registered message handlers
+# 2. Dispatcher checks DDQ for duplicates per handler
+# 3. Applies message filters (from/to/broadcast)
+# 4. Spawns thread for processing matching handlers
+# 5. Marks UUID as processed in handler's DDQ
 ```
 
 ### 5. Message Handler Processing
@@ -382,29 +455,19 @@ dlq.enqueue(header, payload, error: "Validation failed")
 
 **DLQ Architecture:**
 
-```
-┌─────────────────────────────────────────────────────────────┐
-│                    Message Publishing                       │
-│                           │                                 │
-│                           ▼                                 │
-│                  ┌─────────────────┐                        │
-│                  │ Circuit Breaker │                        │
-│                  │   (Monitoring)   │                        │
-│                  └─────────────────┘                        │
-│                     │           │                           │
-│              Success │           │ Failure                  │
-│                     ▼           ▼                           │
-│              ┌──────────┐ ┌─────────────┐                  │
-│              │Transport │ │Dead Letter  │                  │
-│              │          │ │   Queue     │                  │
-│              └──────────┘ └─────────────┘                  │
-│                                │                            │
-│                                ▼                            │
-│                         ┌─────────────┐                     │
-│                         │   Replay    │                     │
-│                         │ Mechanism   │                     │
-│                         └─────────────┘                     │
-└─────────────────────────────────────────────────────────────┘
+```mermaid
+graph TB
+    Publish[Message Publishing]
+    CB[Circuit Breaker<br/>Monitoring]
+    Transport[Transport<br/>Success]
+    DLQ[Dead Letter Queue<br/>Failure Storage]
+    Replay[Replay Mechanism<br/>Manual/Automated]
+    
+    Publish --> CB
+    CB --> |Success| Transport
+    CB --> |Failure| DLQ
+    DLQ --> Replay
+    Replay --> |Retry| Publish
 ```
 
 **DLQ Features:**
@@ -417,7 +480,7 @@ dlq.enqueue(header, payload, error: "Validation failed")
 
 ### Built-in Statistics
 
-SmartMessage automatically collects processing statistics:
+SmartMessage automatically collects processing statistics including DDQ metrics:
 
 ```ruby
 # Statistics are collected for:
@@ -427,6 +490,11 @@ SS.add(message_class, process_method, 'routed')
 # Access statistics
 puts SS.stat
 puts SS.get("MyMessage", "publish")
+
+# DDQ-specific statistics
+stats = OrderMessage.ddq_stats
+puts "DDQ utilization: #{stats[:utilization]}%"
+puts "Current count: #{stats[:current_count]}"
 ```
 
 ### Monitoring Points
@@ -435,6 +503,8 @@ puts SS.get("MyMessage", "publish")
 2. **Message Routing**: Count of routed messages per processor
 3. **Thread Pool**: Queue length, completed tasks, running status
 4. **Transport Status**: Connection status, message counts
+5. **DDQ Metrics**: Utilization, duplicate detection rates, memory usage
+6. **Message Filtering**: Filter match rates, entity-aware routing statistics
 
 ## Configuration Architecture