smart_message 0.0.10 → 0.0.13

data/docs/reference/serializers.md

@@ -0,0 +1,245 @@
+# Transport-Based Serialization
+
+In SmartMessage's architecture, serialization is handled at the transport level rather than being configured for individual messages. Each transport manages its own optimal serialization format, eliminating the need for separate serializer configuration.
+
+## Overview
+
+Transport-based serialization provides:
+- **Automatic Format Selection**: Each transport chooses its optimal serialization format
+- **Simplified Configuration**: No need to configure serializers separately
+- **Format Optimization**: Transports can choose the best format for their medium
+- **Consistent Behavior**: All messages using a transport share the same serialization format
+
+## Transport Serialization Formats
+
+### Memory Transport
+- **Format**: No serialization (objects passed directly)
+- **Use case**: Testing and development where no network transmission occurs
+- **Performance**: Fastest possible (no encoding/decoding overhead)
+
+```ruby
+# Memory transport - no serialization needed
+transport = SmartMessage::Transport::MemoryTransport.new
+```
+
+### STDOUT Transport
+- **Format**: JSON (human-readable)
+- **Use case**: Debugging, development logging, message inspection
+- **Features**: Pretty-printed output for easy reading
+
+```ruby
+# STDOUT transport - uses JSON for readability
+transport = SmartMessage::Transport::StdoutTransport.new(
+  format: :pretty  # or :json for compact format
+)
+```
+
+### Redis Transport
+- **Format**: MessagePack (primary), JSON (fallback)
+- **Use case**: Production messaging where efficiency matters
+- **Benefits**: Compact binary format reduces network overhead
+
+```ruby
+# Redis transport - automatically uses MessagePack if available
+transport = SmartMessage::Transport::RedisTransport.new(
+  url: 'redis://localhost:6379'
+)
+```
+
+## How It Works
+
+### Transport Serialization Process
+
+1. **Message Publishing**: 
+   ```ruby
+   message = OrderMessage.new(order_id: "123", amount: 99.99)
+   message.publish  # Transport handles serialization automatically
+   ```
+
+2. **Automatic Encoding**: Transport calls its serializer internally
+   ```ruby
+   # Inside transport.publish(message):
+   serialized = transport.serializer.encode(message.to_hash)
+   ```
+
+3. **Message Receiving**: Transport deserializes automatically
+   ```ruby
+   # Inside transport.receive(serialized_data):
+   data = transport.serializer.decode(serialized_data)
+   message = MessageClass.new(data)
+   ```
+
+### Message Structure
+
+All messages are serialized as flat hashes with the `_sm_header` property containing routing metadata:
+
+```ruby
+{
+  _sm_header: {
+    uuid: "...",
+    message_class: "OrderMessage", 
+    published_at: "2025-01-09T...",
+    from: "order-service",
+    to: "fulfillment-service",
+    serializer: "SmartMessage::Serializer::Json"
+  },
+  order_id: "123",
+  amount: 99.99,
+  items: ["Widget A", "Widget B"]
+}
+```
+
+## Custom Transport Serializers
+
+You can specify a custom serializer when creating a transport:
+
+```ruby
+# Custom serializer for a transport
+class MyCustomSerializer
+  def encode(data_hash)
+    # Your encoding logic here
+    # Must return a string
+  end
+
+  def decode(serialized_string)
+    # Your decoding logic here
+    # Must return a hash
+  end
+end
+
+# Use custom serializer with transport
+transport = SmartMessage::Transport::RedisTransport.new(
+  serializer: MyCustomSerializer.new,
+  url: 'redis://localhost:6379'
+)
+```
+
+## Built-in Serializer Classes
+
+SmartMessage includes these serializer implementations that transports use internally:
+
+### JSON Serializer
+```ruby
+SmartMessage::Serializer::Json.new
+```
+- Human-readable format
+- Wide compatibility
+- Used by STDOUT transport and as fallback
+
+### MessagePack Serializer
+```ruby
+SmartMessage::Serializer::MessagePack.new
+```
+- Binary format for efficiency
+- Smaller payload size
+- Used by Redis transport when available
+
+## Migration from Message-Level Serializers
+
+If you were previously configuring serializers at the message level, here's how to migrate:
+
+### Before (Message-Level Configuration)
+```ruby
+class OrderMessage < SmartMessage::Base
+  property :order_id
+  property :amount
+  
+  config do
+    transport SmartMessage::Transport::RedisTransport.new
+    serializer SmartMessage::Serializer::Json.new  # ❌ No longer needed
+  end
+end
+```
+
+### After (Transport-Level Serialization)
+```ruby
+class OrderMessage < SmartMessage::Base
+  property :order_id
+  property :amount
+  
+  config do
+    # Transport automatically handles serialization
+    transport SmartMessage::Transport::RedisTransport.new
+  end
+end
+
+# Or specify custom serializer for transport
+class OrderMessage < SmartMessage::Base
+  property :order_id
+  property :amount
+  
+  config do
+    transport SmartMessage::Transport::RedisTransport.new(
+      serializer: MyCustomSerializer.new
+    )
+  end
+end
+```
+
+## Serialization Best Practices
+
+### 1. Let Transports Choose
+Let each transport use its optimal format:
+- Memory: No serialization
+- STDOUT: JSON for readability  
+- Redis: MessagePack for efficiency
+
+### 2. Custom Serializers
+Only use custom serializers when you have specific requirements:
+- Special data formats (XML, Protocol Buffers)
+- Encryption/compression needs
+- Legacy system compatibility
+
+### 3. Testing
+Test with actual transports to ensure serialization works correctly:
+
+```ruby
+RSpec.describe OrderMessage do
+  it "serializes correctly with Redis transport" do
+    transport = SmartMessage::Transport::RedisTransport.new
+    message = OrderMessage.new(order_id: "123", amount: 99.99)
+    
+    # Test roundtrip serialization
+    serialized = transport.encode_message(message)
+    deserialized = transport.decode_message(serialized)
+    
+    expect(deserialized[:order_id]).to eq("123")
+    expect(deserialized[:amount]).to eq(99.99)
+  end
+end
+```
+
+### 4. Error Handling
+Transports handle serialization errors internally, but you can still catch them:
+
+```ruby
+begin
+  message.publish
+rescue SmartMessage::Errors::SerializationError => e
+  logger.error "Failed to serialize message: #{e.message}"
+end
+```
+
+## Performance Considerations
+
+### Format Efficiency
+- **MessagePack**: 20-30% more compact than JSON
+- **JSON**: Human-readable but larger payload
+- **Memory**: No serialization overhead
+
+### Network Optimization
+- Redis transport automatically uses MessagePack when available
+- Falls back to JSON if MessagePack gem is not installed
+- STDOUT uses JSON for debugging clarity
+
+### Monitoring
+Each transport logs its serializer choice:
+```
+[SmartMessage::Transport::RedisTransport] Using serializer: SmartMessage::Serializer::MessagePack
+```
+
+## Next Steps
+
+- [Transports](transports.md) - Available transport implementations
+- [Configuration](../getting-started/quick-start.md) - Setting up transports
+- [Examples](../getting-started/examples.md) - Real-world usage patterns

data/docs/{transports.md → reference/transports.md}

@@ -137,6 +137,8 @@ Production-ready Redis pub/sub transport for distributed messaging.
 - Configurable connection parameters
 - Background message subscription threads
 
+> **💡 Redis Transport:** SmartMessage provides a Redis-based transport for production messaging using pub/sub channels. See the [Redis Transport documentation](../transports/redis-transport.md) for detailed usage guidance.
+
 **Usage:**
 
 ```ruby
@@ -166,12 +168,11 @@ class OrderMessage < SmartMessage::Base
       url: 'redis://localhost:6379',
       db: 1
     )
-    serializer SmartMessage::Serializer::JSON.new
   end
   
-  def self.process(message_header, message_payload)
-    order_data = JSON.parse(message_payload)
-    order = new(order_data)
+  def self.process(decoded_message)
+    # decoded_message is already a message instance
+    order = decoded_message
     puts "Processing order #{order.order_id} for $#{order.amount}"
     # Your business logic here
   end
@@ -243,7 +244,6 @@ redis_transport = SmartMessage::Transport.create(:redis,
 [OrderMessage, PaymentMessage, ShippingMessage].each do |msg_class|
   msg_class.config do
     transport redis_transport
-    serializer SmartMessage::Serializer::JSON.new
   end
   
   # Subscribe to each message type (creates separate Redis subscriptions)
@@ -285,7 +285,6 @@ class ProductionMessage < SmartMessage::Base
       reconnect_attempts: 10,
       reconnect_delay: 5
     )
-    serializer SmartMessage::Serializer::JSON.new
     logger Logger.new(STDOUT)
   end
 end
@@ -302,7 +301,6 @@ class TestMessage < SmartMessage::Base
       db: 15,  # Use separate database for tests
       auto_subscribe: true
     )
-    serializer SmartMessage::Serializer::JSON.new
   end
 end
 
@@ -313,6 +311,7 @@ def setup
 end
 ```
 
+
 ## Transport Interface
 
 All transports must implement the `SmartMessage::Transport::Base` interface:
@@ -411,7 +410,6 @@ class OrderMessage < SmartMessage::Base
   # All instances use this transport by default
   config do
     transport SmartMessage::Transport.create(:memory, auto_process: true)
-    serializer SmartMessage::Serializer::JSON.new
   end
 end
 ```
@@ -689,7 +687,7 @@ end
 
 ## Next Steps
 
-- [Custom Transports](custom-transports.md) - Build your own transport
+- [Redis Transport Comparison](../transports/redis-transport-comparison.md) - Detailed comparison of Redis, Enhanced, and Queue transports
 - [Serializers](serializers.md) - Understanding message serialization
-- [Dispatcher](dispatcher.md) - Message routing and processing
-- [Examples](examples.md) - Real-world transport usage patterns
+- [Dispatcher](../core-concepts/dispatcher.md) - Message routing and processing  
+- [Examples](../getting-started/examples.md) - Real-world transport usage patterns

data/docs/transports/memory-transport.md

@@ -0,0 +1,374 @@
+# Memory Transport
+
+The **Memory Transport** is an in-memory transport implementation designed for testing, development, and rapid prototyping. It stores messages in memory and provides synchronous processing capabilities.
+
+## Overview
+
+The Memory Transport is perfect for:
+- **Unit testing** - No external dependencies required
+- **Local development** - Fast, lightweight message processing
+- **Rapid prototyping** - Quick setup without infrastructure
+- **Debug and inspection** - Full visibility into message flow
+
+## Key Features
+
+- 🧠 **In-Memory Storage** - Messages stored in process memory
+- ⚡ **Synchronous Processing** - Immediate message processing
+- 🔍 **Message Inspection** - View and count stored messages
+- 🔄 **Auto-Processing** - Optional automatic message processing
+- 🛡️ **Memory Protection** - Configurable message limits to prevent overflow
+- 🧵 **Thread-Safe** - Mutex-protected operations
+
+## Configuration
+
+### Basic Setup
+
+```ruby
+# Minimal configuration
+transport = SmartMessage::Transport::MemoryTransport.new
+
+# With options
+transport = SmartMessage::Transport::MemoryTransport.new(
+  auto_process: true,     # Process messages immediately (default: true)
+  max_messages: 1000      # Maximum messages to store (default: 1000)
+)
+```
+
+### Using with SmartMessage
+
+```ruby
+# Configure as default transport
+SmartMessage.configure do |config|
+  config.default_transport = SmartMessage::Transport::MemoryTransport.new
+end
+
+# Use in message class
+class TestMessage < SmartMessage::Base
+  property :content, required: true
+  
+  transport :memory
+  
+  def process
+    puts "Processing: #{content}"
+  end
+end
+```
+
+## Configuration Options
+
+| Option | Type | Default | Description |
+|--------|------|---------|-------------|
+| `auto_process` | Boolean | `true` | Automatically process messages when published |
+| `max_messages` | Integer | `1000` | Maximum messages to store (prevents memory overflow) |
+
+## Usage Examples
+
+### Basic Message Processing
+
+```ruby
+# Create transport
+transport = SmartMessage::Transport::MemoryTransport.new
+
+# Define message
+class AlertMessage < SmartMessage::Base
+  property :message, required: true
+  property :severity, default: 'info'
+  
+  transport transport
+  
+  def process
+    puts "[#{severity.upcase}] #{message}"
+  end
+end
+
+# Publish message
+AlertMessage.new(
+  message: "System startup complete",
+  severity: "info"
+).publish
+
+# Output: [INFO] System startup complete
+```
+
+### Manual Processing Control
+
+```ruby
+# Disable auto-processing for batch operations
+transport = SmartMessage::Transport::MemoryTransport.new(auto_process: false)
+
+class DataMessage < SmartMessage::Base
+  property :data
+  transport transport
+  
+  def process
+    puts "Processing: #{data}"
+  end
+end
+
+# Publish multiple messages
+DataMessage.new(data: "batch 1").publish
+DataMessage.new(data: "batch 2").publish
+DataMessage.new(data: "batch 3").publish
+
+puts "Messages stored: #{transport.message_count}"
+# Output: Messages stored: 3
+
+# Process all at once
+transport.process_all
+# Output: 
+# Processing: batch 1
+# Processing: batch 2
+# Processing: batch 3
+```
+
+### Message Inspection
+
+```ruby
+transport = SmartMessage::Transport::MemoryTransport.new(auto_process: false)
+
+class OrderMessage < SmartMessage::Base
+  property :order_id, required: true
+  property :amount, required: true
+  transport transport
+end
+
+# Publish test messages
+OrderMessage.new(order_id: "ORD-001", amount: 99.99).publish
+OrderMessage.new(order_id: "ORD-002", amount: 149.50).publish
+
+# Inspect stored messages
+puts "Total messages: #{transport.message_count}"
+transport.all_messages.each_with_index do |msg, index|
+  puts "Message #{index + 1}: #{msg[:message_class]} at #{msg[:published_at]}"
+end
+
+# Clear messages when done
+transport.clear_messages
+puts "Messages after clear: #{transport.message_count}"
+```
+
+## API Reference
+
+### Instance Methods
+
+#### `#message_count`
+Returns the number of messages currently stored.
+
+```ruby
+count = transport.message_count
+puts "Stored messages: #{count}"
+```
+
+#### `#all_messages`
+Returns a copy of all stored messages with metadata.
+
+```ruby
+messages = transport.all_messages
+messages.each do |msg|
+  puts "Class: #{msg[:message_class]}"
+  puts "Time: #{msg[:published_at]}"
+  puts "Data: #{msg[:serialized_message]}"
+end
+```
+
+#### `#clear_messages`
+Removes all stored messages from memory.
+
+```ruby
+transport.clear_messages
+```
+
+#### `#process_all`
+Manually processes all stored messages (useful when `auto_process: false`).
+
+```ruby
+# Publish messages without auto-processing
+transport = SmartMessage::Transport::MemoryTransport.new(auto_process: false)
+# ... publish messages ...
+
+# Process them all at once
+transport.process_all
+```
+
+#### `#connected?`
+Always returns `true` since memory transport is always available.
+
+```ruby
+puts transport.connected?  # => true
+```
+
+## Use Cases
+
+### Unit Testing
+
+```ruby
+RSpec.describe "Message Processing" do
+  let(:transport) { SmartMessage::Transport::MemoryTransport.new }
+  
+  before do
+    MyMessage.transport = transport
+    transport.clear_messages
+  end
+  
+  it "processes messages correctly" do
+    MyMessage.new(data: "test").publish
+    expect(transport.message_count).to eq(1)
+  end
+  
+  it "respects message limits" do
+    transport = SmartMessage::Transport::MemoryTransport.new(max_messages: 2)
+    
+    3.times { |i| MyMessage.new(data: i).publish }
+    expect(transport.message_count).to eq(2)  # Oldest message dropped
+  end
+end
+```
+
+### Development Environment
+
+```ruby
+# config/environments/development.rb
+SmartMessage.configure do |config|
+  config.default_transport = SmartMessage::Transport::MemoryTransport.new(
+    auto_process: true,
+    max_messages: 500
+  )
+  config.logger.level = Logger::DEBUG  # See all message activity
+end
+```
+
+### Batch Processing
+
+```ruby
+# Collect messages for batch processing
+transport = SmartMessage::Transport::MemoryTransport.new(auto_process: false)
+
+# Publish work items
+work_items.each do |item|
+  WorkMessage.new(item: item).publish
+end
+
+# Process batch when ready
+puts "Processing #{transport.message_count} work items..."
+start_time = Time.now
+transport.process_all
+puts "Completed in #{Time.now - start_time} seconds"
+```
+
+## Performance Characteristics
+
+- **Latency**: ~0.01ms (memory access)
+- **Throughput**: 100K+ messages/second
+- **Memory Usage**: ~1KB per stored message
+- **Concurrency**: Thread-safe with mutex protection
+- **Persistence**: None (messages lost when process ends)
+
+## Best Practices
+
+### Testing
+- Use `clear_messages` in test setup/teardown
+- Set reasonable `max_messages` limits for long-running tests
+- Disable `auto_process` for message inspection tests
+
+### Development
+- Enable debug logging to see message flow
+- Use message inspection methods for debugging
+- Consider memory limits in long-running development processes
+
+### Production
+⚠️ **Not recommended for production use**
+- Messages are lost when process restarts
+- No persistence or durability guarantees
+- Limited by process memory
+
+## Thread Safety
+
+The Memory Transport is fully thread-safe:
+- All operations use mutex synchronization
+- Messages can be published from multiple threads
+- Inspection methods return safe copies
+
+```ruby
+# Thread-safe concurrent publishing
+threads = []
+10.times do |i|
+  threads << Thread.new do
+    100.times { |j| TestMessage.new(data: "#{i}-#{j}").publish }
+  end
+end
+threads.each(&:join)
+
+puts "Total messages: #{transport.message_count}"  # Always accurate
+```
+
+## Migration from Memory Transport
+
+When moving from Memory Transport to production transports:
+
+```ruby
+# Development (Memory Transport)
+SmartMessage.configure do |config|
+  config.default_transport = SmartMessage::Transport::MemoryTransport.new
+end
+
+# Production (Redis Transport)
+SmartMessage.configure do |config|
+  config.default_transport = SmartMessage::Transport::RedisTransport.new(
+    url: ENV['REDIS_URL']
+  )
+end
+```
+
+Messages and processing logic remain identical - only the transport configuration changes.
+
+## Examples
+
+The `examples/memory/` directory contains comprehensive, runnable examples demonstrating Memory Transport capabilities:
+
+### Core Messaging Examples
+- **[03_point_to_point_orders.rb](https://github.com/MadBomber/smart_message/blob/main/examples/memory/03_point_to_point_orders.rb)** - Point-to-point order processing with payment integration
+- **[04_publish_subscribe_events.rb](https://github.com/MadBomber/smart_message/blob/main/examples/memory/04_publish_subscribe_events.rb)** - Event broadcasting to multiple services (email, SMS, audit)
+- **[05_many_to_many_chat.rb](https://github.com/MadBomber/smart_message/blob/main/examples/memory/05_many_to_many_chat.rb)** - Interactive chat system with rooms, bots, and human agents
+
+### Advanced Features
+- **[01_message_deduplication_demo.rb](https://github.com/MadBomber/smart_message/blob/main/examples/memory/01_message_deduplication_demo.rb)** - Message deduplication patterns and strategies
+- **[02_dead_letter_queue_demo.rb](https://github.com/MadBomber/smart_message/blob/main/examples/memory/02_dead_letter_queue_demo.rb)** - Complete Dead Letter Queue system with circuit breakers
+- **[07_proc_handlers_demo.rb](https://github.com/MadBomber/smart_message/blob/main/examples/memory/07_proc_handlers_demo.rb)** - Flexible message handlers (blocks, procs, lambdas, methods)
+
+### Configuration & Monitoring
+- **[08_custom_logger_demo.rb](https://github.com/MadBomber/smart_message/blob/main/examples/memory/08_custom_logger_demo.rb)** - Advanced logging with SmartMessage::Logger::Default
+- **[09_error_handling_demo.rb](https://github.com/MadBomber/smart_message/blob/main/examples/memory/09_error_handling_demo.rb)** - Comprehensive validation, version mismatch, and error handling
+- **[13_header_block_configuration.rb](https://github.com/MadBomber/smart_message/blob/main/examples/memory/13_header_block_configuration.rb)** - Header and block configuration examples
+- **[14_global_configuration_demo.rb](https://github.com/MadBomber/smart_message/blob/main/examples/memory/14_global_configuration_demo.rb)** - Global configuration management
+- **[15_logger_demo.rb](https://github.com/MadBomber/smart_message/blob/main/examples/memory/15_logger_demo.rb)** - Advanced logging demonstrations
+
+### Entity Addressing & Filtering
+- **[10_entity_addressing_basic.rb](https://github.com/MadBomber/smart_message/blob/main/examples/memory/10_entity_addressing_basic.rb)** - Basic FROM/TO/REPLY_TO message addressing
+- **[11_entity_addressing_with_filtering.rb](https://github.com/MadBomber/smart_message/blob/main/examples/memory/11_entity_addressing_with_filtering.rb)** - Advanced entity-aware message filtering
+- **[12_regex_filtering_microservices.rb](https://github.com/MadBomber/smart_message/blob/main/examples/memory/12_regex_filtering_microservices.rb)** - Advanced regex filtering for microservices
+
+### Visual Demonstrations
+- **[06_pretty_print_demo.rb](https://github.com/MadBomber/smart_message/blob/main/examples/memory/06_pretty_print_demo.rb)** - Message inspection and pretty-printing capabilities
+
+### Running Examples
+
+```bash
+# Navigate to the SmartMessage directory
+cd smart_message
+
+# Run any Memory Transport example
+ruby examples/memory/03_point_to_point_orders.rb
+ruby examples/memory/05_many_to_many_chat.rb
+ruby examples/memory/02_dead_letter_queue_demo.rb
+
+# Or explore the entire directory
+ls examples/memory/
+```
+
+Each example is self-contained and demonstrates specific Memory Transport features with clear educational comments and real-world scenarios.
+
+## Related Documentation
+
+- [Transport Overview](../reference/transports.md) - All available transports
+- [Redis Transport](redis-transport.md) - Production-ready Redis transport
+- [Testing Guide](../development/troubleshooting.md) - Testing strategies with SmartMessage