smart_message 0.0.9 → 0.0.12

data/README.md

@@ -1,13 +1,29 @@
+<table border="0">
+<tr>
+<td width="30%" valign="top">
+
+<img src="docs/assets/images/smart_message.jpg" alt="SmartMessage Logo" />
+<br/>
+See <a href="https://madbomber.github.io/smart_message/">Documentation Websit</a>
+</td>
+<td width="70%" valign="top">
+
 # SmartMessage
 
-[![Gem Version](https://badge.fury.io/rb/smart_message.svg)](https://badge.fury.io/rb/smart_message)
-[![Ruby](https://img.shields.io/badge/ruby-%3E%3D%203.0.0-ruby.svg)](https://www.ruby-lang.org/en/)
+Can Walk, Talk, and Think at the Same Time
+
+**SmartMessage** is a powerful Ruby framework that transforms ordinary messages into intelligent, self-aware entities capable of routing themselves, validating their contents, and executing business logic. By abstracting away the complexities of transport mechanisms (Redis, RabbitMQ, Kafka) and serialization formats (JSON, MessagePack), SmartMessage lets you focus on what matters: your business logic.
+
+Think of SmartMessage as ActiveRecord for messaging - just as ActiveRecord frees you from database-specific SQL, SmartMessage liberates your messages from transport-specific implementations. Each message knows how to validate itself, where it came from, where it's going, and what to do when it arrives. With built-in support for filtering, versioning, deduplication, and concurrent processing, SmartMessage provides enterprise-grade messaging capabilities with the simplicity Ruby developers love.
 
-SmartMessage is a message abstraction framework that decouples business logic from message transports and serialization formats. Much like ActiveRecord abstracts models from database implementations, SmartMessage abstracts messages from their backend transport and serialization mechanisms.
+</td>
+</tr>
+</table>
 
 ## Features
 
 - **Transport Abstraction**: Plugin architecture supporting multiple message transports (Redis, RabbitMQ, Kafka, etc.)
+- **🌟 Redis Queue Transport**: Advanced transport with RabbitMQ-style routing patterns, persistent FIFO queues, load balancing, and 10x faster performance than traditional message brokers. Built on Ruby's Async framework for fiber-based concurrency supporting thousands of concurrent subscriptions - [see full documentation](docs/transports/redis-queue.md)
 - **Serialization Flexibility**: Pluggable serialization formats (JSON, MessagePack, etc.)
 - **Entity-to-Entity Addressing**: Built-in FROM/TO/REPLY_TO addressing for point-to-point and broadcast messaging patterns
 - **Advanced Message Filtering**: Filter subscriptions using exact strings, regular expressions, or mixed arrays for precise message routing
@@ -16,9 +32,10 @@ SmartMessage is a message abstraction framework that decouples business logic fr
 - **Message Documentation**: Built-in documentation support for message classes and properties with automatic defaults
 - **Flexible Message Handlers**: Multiple subscription patterns - default methods, custom methods, blocks, procs, and lambdas
 - **Dual-Level Configuration**: Class and instance-level plugin overrides for gateway patterns
-- **Concurrent Processing**: Thread-safe message routing using `Concurrent::CachedThreadPool`
+- **Concurrent Processing**: Thread-safe message routing using `Concurrent::CachedThreadPool` with Async/Fiber-based Redis Queue Transport for massive scalability
 - **Advanced Logging System**: Comprehensive logging with colorized console output, JSON structured logging, and file rolling
 - **Built-in Statistics**: Message processing metrics and monitoring
+- **Message Deduplication**: Handler-scoped deduplication queues (DDQ) with memory or Redis storage for preventing duplicate message processing
 - **Development Tools**: STDOUT and in-memory transports for testing
 - **Production Ready**: Redis transport with automatic reconnection and error handling
 - **Dead Letter Queue**: File-based DLQ with JSON Lines format for failed message capture and replay
@@ -34,78 +51,106 @@ gem 'smart_message'
 
 And then execute:
 
-    $ bundle install
+    bundle install
 
 Or install it yourself as:
 
-    $ gem install smart_message
+    gem install smart_message
+
+### Redis Transport Setup
+
+To use the built-in Redis transport, you'll need to have Redis server installed:
+
+**macOS:**
+```bash
+brew install redis
+brew services start redis  # To start Redis as a service
+```
+
+**Ubuntu/Debian:**
+```bash
+sudo apt-get update
+sudo apt-get install redis-server
+```
+
+**CentOS/RHEL/Fedora:**
+```bash
+sudo yum install redis
+sudo systemctl start redis
+```
 
 ## Quick Start
 
 ### 1. Define a Message Class
 
 ```ruby
+require 'smart_message'
+
 class OrderMessage < SmartMessage::Base
   # Declare schema version for compatibility tracking
   version 2
-  
+
   # Add a description for the message class
   description "Represents customer order data for processing and fulfillment"
-  
+
   # Configure entity addressing (Method 1: Direct methods)
   from 'order-service'
   to 'fulfillment-service'  # Point-to-point message
   reply_to 'order-service'  # Responses come back here
-  
+
   # Alternative Method 2: Using header block
   # header do
   #   from 'order-service'
   #   to 'fulfillment-service'
   #   reply_to 'order-service'
   # end
-  
+
   # Required properties with validation
-  property :order_id, 
+  property :order_id,
     required: true,
     message: "Order ID is required",
     validate: ->(v) { v.is_a?(String) && v.length > 0 },
     validation_message: "Order ID must be a non-empty string",
     description: "Unique order identifier"
-    
-  property :customer_id, 
+
+  property :customer_id,
     required: true,
     message: "Customer ID is required",
     description: "Customer's unique ID"
-    
-  property :amount, 
+
+  property :amount,
     required: true,
     message: "Amount is required",
     validate: ->(v) { v.is_a?(Numeric) && v > 0 },
     validation_message: "Amount must be a positive number",
     description: "Total order amount in dollars"
-    
-  property :items, 
+
+  property :items,
     default: [],
     description: "Array of ordered items"
 
   # Configure transport and serializer at class level
   config do
+    # Option 1: Simple STDOUT for development
     transport SmartMessage::Transport.create(:stdout, loopback: true)
+
+    # Option 2: Redis Queue for production (10x faster than RabbitMQ!)
+    # transport SmartMessage::Transport.create(:redis_queue,
+    #   url: 'redis://localhost:6379',
+    #   queue_prefix: 'myapp'
+    # )
+
     serializer SmartMessage::Serializer::JSON.new
   end
 
   # Business logic for processing received messages
-  def self.process(message_header, message_payload)
-    # Decode the message
-    order_data = JSON.parse(message_payload)
-    order = new(order_data)
-    
-    # Process the order
-    puts "Processing order #{order.order_id} for customer #{order.customer_id}"
-    puts "Amount: $#{order.amount}"
-    
+  def self.process(message_instance)
+    # Message instance is already decoded and validated
+    puts "Processing order #{message_instance.order_id} for customer #{message_instance.customer_id}"
+    puts "Amount: $#{message_instance.amount}"
+
     # Your business logic here
-    process_order(order)
+    process_order(message_instance)
   end
 
   private
@@ -122,7 +167,7 @@ end
 # Create and publish a message (automatically validated before publishing)
 order = OrderMessage.new(
   order_id: "ORD-123",
-  customer_id: "CUST-456", 
+  customer_id: "CUST-456",
   amount: 99.99,
   items: ["Widget A", "Widget B"]
 )
@@ -151,20 +196,23 @@ OrderMessage.subscribe
 OrderMessage.subscribe("PaymentService.process_order")
 
 # 3. Block handler (NEW!)
-OrderMessage.subscribe do |header, payload|
+OrderMessage.subscribe do |wrapper|
+  header, payload = wrapper.split
   order_data = JSON.parse(payload)
   puts "Quick processing: Order #{order_data['order_id']}"
 end
 
 # 4. Proc handler (NEW!)
-order_processor = proc do |header, payload|
+order_processor = proc do |wrapper|
+  header, payload = wrapper.split
   order_data = JSON.parse(payload)
   EmailService.send_confirmation(order_data['customer_id'])
 end
 OrderMessage.subscribe(order_processor)
 
 # 5. Lambda handler (NEW!)
-audit_handler = lambda do |header, payload|
+audit_handler = lambda do |wrapper|
+  header, payload = wrapper.split
   AuditLog.record("Order processed at #{header.published_at}")
 end
 OrderMessage.subscribe(audit_handler)
@@ -192,7 +240,7 @@ OrderMessage.subscribe(to: /^(dev|staging)-.*/)
 
 # Combined filtering
 OrderMessage.subscribe(
-  from: /^admin-.*/, 
+  from: /^admin-.*/,
   to: ['order-service', /^fulfillment-.*/]
 )
 
@@ -201,7 +249,103 @@ DevService.subscribe(to: /^(dev|staging)-.*/)
 ProdService.subscribe(to: /^prod-.*/)
 ```
 
-### 5. Entity Addressing
+### 5. Message Deduplication
+
+SmartMessage provides handler-scoped message deduplication to prevent duplicate processing of messages with the same UUID. Each handler gets its own Deduplication Queue (DDQ) that tracks recently processed message UUIDs.
+
+#### Basic Deduplication Setup
+
+```ruby
+class OrderMessage < SmartMessage::Base
+  version 1
+  property :order_id, required: true
+  property :amount, required: true
+
+  from "order-service"
+
+  # Configure deduplication
+  ddq_size 100              # Track last 100 message UUIDs
+  ddq_storage :memory       # Use memory storage (or :redis for distributed)
+  enable_deduplication!     # Enable deduplication for this message class
+
+  def self.process(message_instance)
+    puts "Processing order: #{message_instance.order_id}"
+    # Business logic here
+  end
+end
+```
+
+#### Handler-Scoped Isolation
+
+Each handler gets its own DDQ scope, preventing cross-contamination between different subscribers:
+
+```ruby
+# Each handler gets separate deduplication tracking
+OrderMessage.subscribe('PaymentService.process')     # DDQ: "OrderMessage:PaymentService.process"
+OrderMessage.subscribe('FulfillmentService.handle')  # DDQ: "OrderMessage:FulfillmentService.handle"
+OrderMessage.subscribe('AuditService.log_order')     # DDQ: "OrderMessage:AuditService.log_order"
+
+# Same handler across message classes = separate DDQs
+PaymentMessage.subscribe('PaymentService.process')   # DDQ: "PaymentMessage:PaymentService.process"
+InvoiceMessage.subscribe('PaymentService.process')   # DDQ: "InvoiceMessage:PaymentService.process"
+```
+
+#### Storage Options
+
+```ruby
+# Memory-based DDQ (single process)
+class LocalMessage < SmartMessage::Base
+  ddq_size 50
+  ddq_storage :memory
+  enable_deduplication!
+end
+
+# Redis-based DDQ (distributed/multi-process)
+class DistributedMessage < SmartMessage::Base
+  ddq_size 1000
+  ddq_storage :redis, redis_url: 'redis://localhost:6379', redis_db: 1
+  enable_deduplication!
+end
+```
+
+#### DDQ Statistics and Management
+
+```ruby
+# Check deduplication configuration
+config = OrderMessage.ddq_config
+puts "Enabled: #{config[:enabled]}"
+puts "Size: #{config[:size]}"
+puts "Storage: #{config[:storage]}"
+
+# Get DDQ statistics
+stats = OrderMessage.ddq_stats
+puts "Current count: #{stats[:current_count]}"
+puts "Utilization: #{stats[:utilization]}%"
+
+# Clear DDQ if needed
+OrderMessage.clear_ddq!
+
+# Check if specific UUID is duplicate
+OrderMessage.duplicate_uuid?("some-uuid-123")
+```
+
+#### How Deduplication Works
+
+1. **Message Receipt**: When a message arrives, the dispatcher checks the handler's DDQ for the message UUID
+2. **Duplicate Detection**: If UUID exists in DDQ, the message is ignored (logged but not processed)
+3. **Processing**: If UUID is new, the message is processed by the handler
+4. **UUID Storage**: After successful processing, the UUID is added to the handler's DDQ
+5. **Circular Buffer**: When DDQ reaches capacity, oldest UUIDs are evicted to make room for new ones
+
+#### Benefits
+
+- **Handler Isolation**: Each handler maintains independent deduplication state
+- **Cross-Process Support**: Redis DDQ enables deduplication across multiple processes
+- **Memory Efficient**: Circular buffer with configurable size limits memory usage
+- **High Performance**: O(1) UUID lookup using hybrid array + set data structure
+- **Automatic Integration**: Seamlessly works with existing subscription patterns
+
+### 6. Entity Addressing
 
 SmartMessage supports entity-to-entity addressing with FROM/TO/REPLY_TO fields for advanced message routing. You can configure addressing using three different approaches:
 
@@ -212,7 +356,7 @@ class PaymentMessage < SmartMessage::Base
   from 'payment-service'     # Required: sender identity
   to 'bank-gateway'          # Optional: specific recipient
   reply_to 'payment-service' # Optional: where responses go
-  
+
   property :amount, required: true
   property :account_id, required: true
 end
@@ -222,14 +366,14 @@ end
 ```ruby
 class PaymentMessage < SmartMessage::Base
   version 1
-  
+
   # Configure all addressing in a single block
   header do
     from 'payment-service'
     to 'bank-gateway'
     reply_to 'payment-service'
   end
-  
+
   property :amount, required: true
   property :account_id, required: true
 end
@@ -267,13 +411,13 @@ payment.publish
 ```ruby
 class SystemAnnouncementMessage < SmartMessage::Base
   version 1
-  
+
   # Using header block for broadcast configuration
   header do
     from 'admin-service'  # Required: sender identity
     # No 'to' field = broadcast to all subscribers
   end
-  
+
   property :message, required: true
   property :priority, default: 'normal'
 end
@@ -282,7 +426,7 @@ end
 #### Messaging Patterns Supported
 
 - **Point-to-Point**: Set `to` field for direct entity targeting
-- **Broadcast**: Omit `to` field (nil) for message broadcast to all subscribers  
+- **Broadcast**: Omit `to` field (nil) for message broadcast to all subscribers
 - **Request-Reply**: Use `reply_to` field to specify response routing
 - **Gateway Patterns**: Override addressing at instance level for message forwarding
 
@@ -336,9 +480,9 @@ SmartMessage.configure do |config|
 end
 
 logger = SmartMessage.configuration.default_logger
-logger.info("User action", 
-            user_id: 12345, 
-            action: "login", 
+logger.info("User action",
+            user_id: 12345,
+            action: "login",
             ip_address: "192.168.1.1")
 # Output: {"timestamp":"2025-01-15T10:30:45.123Z","level":"INFO","message":"User action","user_id":12345,"action":"login","ip_address":"192.168.1.1","source":"app.rb:42:in `authenticate`"}
 ```
@@ -352,7 +496,7 @@ SmartMessage.configure do |config|
     roll_by_size: true,
     max_file_size: 10 * 1024 * 1024,  # 10 MB
     keep_files: 5,                     # Keep 5 old files
-    
+
     # Date-based rolling (alternative to size-based)
     roll_by_date: false,               # Set to true for date-based
     date_pattern: '%Y-%m-%d'           # Daily rolling pattern
@@ -368,11 +512,11 @@ SmartMessage classes automatically use the configured logger:
 class OrderMessage < SmartMessage::Base
   property :order_id, required: true
   property :amount, required: true
-  
+
   def process
     # Logger is automatically available
-    logger.info("Processing order", 
-                order_id: order_id, 
+    logger.info("Processing order",
+                order_id: order_id,
                 amount: amount,
                 header: _sm_header.to_h,
                 payload: _sm_payload)
@@ -412,9 +556,10 @@ Pluggable message encoding/decoding:
 #### Dispatcher
 Concurrent message routing engine that:
 - Uses thread pools for async processing
-- Routes messages to subscribed handlers
-- Provides processing statistics
+- Routes messages to subscribed handlers with handler-scoped deduplication
+- Provides processing statistics and DDQ management
 - Handles graceful shutdown
+- Maintains separate DDQ instances per handler for isolated deduplication tracking
 
 ### Plugin Architecture
 
@@ -441,6 +586,57 @@ This enables gateway patterns where messages can be received from one transport/
 
 ## Transport Implementations
 
+### Redis Queue Transport (Featured) 🌟
+
+The Redis Queue Transport provides enterprise-grade message routing with exceptional performance:
+
+```ruby
+# Configure with RabbitMQ-style routing
+transport = SmartMessage::Transport.create(:redis_queue,
+  url: 'redis://localhost:6379',
+  queue_prefix: 'myapp',
+  consumer_group: 'workers'
+)
+
+# Pattern-based subscriptions (RabbitMQ compatible)
+transport.subscribe_pattern("#.*.payment_service")  # All messages TO payment_service
+transport.subscribe_pattern("#.api_gateway.*")      # All messages FROM api_gateway
+transport.subscribe_pattern("order.#.*.*")          # All order messages
+
+# Fluent API for complex routing
+transport.where
+  .from('web_app')
+  .to('analytics')
+  .consumer_group('analytics_workers')
+  .subscribe
+
+# Configure message class
+class OrderMessage < SmartMessage::Base
+  transport :redis_queue
+
+  property :order_id, required: true
+  property :amount, required: true
+end
+
+# Publish with enhanced routing
+OrderMessage.new(
+  order_id: 'ORD-001',
+  amount: 99.99,
+  _sm_header: { from: 'api_gateway', to: 'payment_service' }
+).publish
+```
+
+**Key Features:**
+- 10x faster than RabbitMQ (0.5ms vs 5ms latency)
+- Pattern routing with `#` and `*` wildcards
+- Persistent FIFO queues using Redis Lists
+- Load balancing via consumer groups
+- Enhanced routing keys: `namespace.type.from.to`
+- Queue monitoring and management
+- Production-ready with circuit breakers and dead letter queues
+
+📚 **Full Documentation:** [Redis Queue Transport Guide](docs/transports/redis-queue.md) | [Getting Started](docs/guides/redis-queue-getting-started.md) | [Examples](examples/redis_queue/)
+
 ### STDOUT Transport (Development)
 
 ```ruby
@@ -473,7 +669,7 @@ transport.process_all  # Process all pending messages
 
 ```ruby
 # Basic Redis configuration
-transport = SmartMessage::Transport.create(:redis, 
+transport = SmartMessage::Transport.create(:redis,
   url: 'redis://localhost:6379',
   db: 0
 )
@@ -508,7 +704,7 @@ The Redis transport uses the message class name as the Redis channel name, enabl
 ```ruby
 class WebhookTransport < SmartMessage::Transport::Base
   def default_options
-    { 
+    {
       webhook_url: "https://api.example.com/webhooks",
       timeout: 30,
       retries: 3
@@ -523,19 +719,19 @@ class WebhookTransport < SmartMessage::Transport::Base
   def publish(message_header, message_payload)
     http = Net::HTTP.new(@uri.host, @uri.port)
     http.use_ssl = @uri.scheme == 'https'
-    
+
     request = Net::HTTP::Post.new(@uri)
     request['Content-Type'] = 'application/json'
     request['X-Message-Class'] = message_header.message_class
     request.body = message_payload
-    
+
     response = http.request(request)
     raise "Webhook failed: #{response.code}" unless response.code.to_i < 400
   end
 
   def subscribe(message_class, process_method)
     super
-    # For webhooks, subscription would typically be configured 
+    # For webhooks, subscription would typically be configured
     # externally on the webhook provider's side
   end
 end
@@ -545,7 +741,7 @@ SmartMessage::Transport.register(:webhook, WebhookTransport)
 
 # Use the transport
 MyMessage.config do
-  transport SmartMessage::Transport.create(:webhook, 
+  transport SmartMessage::Transport.create(:webhook,
     webhook_url: "https://api.myservice.com/messages"
   )
 end
@@ -576,7 +772,7 @@ Declare your message schema version using the `version` class method:
 ```ruby
 class OrderMessage < SmartMessage::Base
   version 2  # Schema version 2
-  
+
   property :order_id, required: true
   property :customer_email  # Added in version 2
 end
@@ -589,22 +785,22 @@ Properties support multiple validation types with custom error messages:
 ```ruby
 class UserMessage < SmartMessage::Base
   version 1
-  
+
   # Required field validation (Hashie built-in)
-  property :user_id, 
+  property :user_id,
     required: true,
     message: "User ID is required and cannot be blank"
-  
+
   # Custom validation with lambda
   property :age,
     validate: ->(v) { v.is_a?(Integer) && v.between?(1, 120) },
     validation_message: "Age must be an integer between 1 and 120"
-  
+
   # Email validation with regex
   property :email,
     validate: /\A[\w+\-.]+@[a-z\d\-]+(\.[a-z\d\-]+)*\.[a-z]+\z/i,
     validation_message: "Must be a valid email address"
-  
+
   # Inclusion validation with array
   property :status,
     validate: ['active', 'inactive', 'pending'],
@@ -682,20 +878,20 @@ Use the `description` DSL method to document what your message class represents:
 ```ruby
 class OrderMessage < SmartMessage::Base
   description "Represents customer order data for processing and fulfillment"
-  
+
   property :order_id, required: true
   property :amount, required: true
 end
 
-class UserMessage < SmartMessage::Base  
+class UserMessage < SmartMessage::Base
   description "Handles user management operations including registration and updates"
-  
+
   property :user_id, required: true
   property :email, required: true
 end
 
 # Access descriptions
-puts OrderMessage.description  
+puts OrderMessage.description
 # => "Represents customer order data for processing and fulfillment"
 
 puts UserMessage.description
@@ -716,7 +912,7 @@ class MyMessage < SmartMessage::Base
   property :data
 end
 
-puts MyMessage.description  
+puts MyMessage.description
 # => "MyMessage is a SmartMessage"
 ```
 
@@ -727,10 +923,10 @@ Combine class descriptions with property descriptions for comprehensive document
 ```ruby
 class FullyDocumented < SmartMessage::Base
   description "A fully documented message class for demonstration purposes"
-  
-  property :id, 
+
+  property :id,
     description: "Unique identifier for the record"
-  property :name, 
+  property :name,
     description: "Display name for the entity"
   property :status,
     description: "Current processing status",
@@ -945,4 +1141,4 @@ Bug reports and pull requests are welcome on GitHub at https://github.com/MadBom
 
 ## License
 
-The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).
+The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).

data/Rakefile

@@ -1,10 +1,35 @@
 require "bundler/gem_tasks"
 require "rake/testtask"
 
-Rake::TestTask.new(:test) do |t|
-  t.libs << "test"
-  t.libs << "lib"
-  t.test_files = FileList["test/**/*_test.rb"]
+# Main test task - runs all tests with proper environment setup
+task :test do
+  ENV['SM_LOGGER_TEST'] = 'true'
+  
+  puts "Running SmartMessage test suite..."
+  puts "=" * 40
+  
+  # Check for Redis availability for informational purposes
+  begin
+    require 'redis'
+    redis = Redis.new(url: 'redis://localhost:6379')
+    redis.ping
+    puts "✅ Redis available - all tests including Redis Queue Transport will run"
+    redis.quit
+  rescue => e
+    puts "⚠️  Redis not available: #{e.message}"
+    puts "   Redis Queue Transport tests will be skipped automatically"
+  end
+  
+  puts ""
+  
+  # Run the actual tests
+  Rake::TestTask.new(:run_tests) do |t|
+    t.libs << "test"
+    t.libs << "lib"
+    t.test_files = FileList["test/**/*_test.rb"]
+  end
+  
+  Rake::Task[:run_tests].invoke
 end
 
 task :default => :test