smart_message 0.0.12 → 0.0.16

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (117) hide show
  1. checksums.yaml +4 -4
  2. data/.gitignore +1 -0
  3. data/CHANGELOG.md +155 -1
  4. data/Gemfile.lock +6 -6
  5. data/README.md +71 -25
  6. data/docs/core-concepts/architecture.md +5 -10
  7. data/docs/getting-started/examples.md +0 -12
  8. data/docs/getting-started/quick-start.md +4 -9
  9. data/docs/index.md +6 -4
  10. data/docs/reference/serializers.md +160 -488
  11. data/docs/reference/transports.md +47 -146
  12. data/docs/transports/memory-transport.md +2 -1
  13. data/docs/transports/multi-transport.md +484 -0
  14. data/docs/transports/redis-transport-comparison.md +215 -350
  15. data/docs/transports/redis-transport.md +3 -22
  16. data/examples/README.md +6 -9
  17. data/examples/city_scenario/README.md +1 -1
  18. data/examples/city_scenario/messages/emergency_911_message.rb +0 -1
  19. data/examples/city_scenario/messages/emergency_resolved_message.rb +0 -1
  20. data/examples/city_scenario/messages/fire_dispatch_message.rb +0 -1
  21. data/examples/city_scenario/messages/fire_emergency_message.rb +0 -1
  22. data/examples/city_scenario/messages/health_check_message.rb +0 -1
  23. data/examples/city_scenario/messages/health_status_message.rb +0 -1
  24. data/examples/city_scenario/messages/police_dispatch_message.rb +0 -1
  25. data/examples/city_scenario/messages/silent_alarm_message.rb +0 -1
  26. data/examples/file/00_run_all_file_demos.rb +260 -0
  27. data/examples/file/01_basic_file_transport_demo.rb +237 -0
  28. data/examples/file/02_fifo_transport_demo.rb +289 -0
  29. data/examples/file/03_file_watching_demo.rb +332 -0
  30. data/examples/file/04_multi_transport_file_demo.rb +432 -0
  31. data/examples/file/README.md +257 -0
  32. data/examples/memory/00_run_all_demos.rb +317 -0
  33. data/examples/memory/01_message_deduplication_demo.rb +18 -32
  34. data/examples/memory/02_dead_letter_queue_demo.rb +9 -12
  35. data/examples/memory/03_point_to_point_orders.rb +3 -5
  36. data/examples/memory/04_publish_subscribe_events.rb +15 -16
  37. data/examples/memory/05_many_to_many_chat.rb +19 -22
  38. data/examples/memory/06_stdout_publish_only.rb +145 -0
  39. data/examples/memory/07_proc_handlers_demo.rb +13 -14
  40. data/examples/memory/08_custom_logger_demo.rb +136 -140
  41. data/examples/memory/09_error_handling_demo.rb +7 -10
  42. data/examples/memory/10_entity_addressing_basic.rb +25 -31
  43. data/examples/memory/11_entity_addressing_with_filtering.rb +32 -36
  44. data/examples/memory/12_regex_filtering_microservices.rb +10 -11
  45. data/examples/memory/13_header_block_configuration.rb +0 -5
  46. data/examples/memory/14_global_configuration_demo.rb +12 -14
  47. data/examples/memory/15_logger_demo.rb +0 -1
  48. data/examples/memory/README.md +37 -20
  49. data/examples/memory/log/demo_app.log.1 +100 -0
  50. data/examples/memory/log/demo_app.log.2 +100 -0
  51. data/examples/multi_transport_example.rb +114 -0
  52. data/examples/redis/01_smart_home_iot_demo.rb +20 -24
  53. data/examples/redis/README.md +0 -2
  54. data/examples/utilities/box_it.rb +12 -0
  55. data/examples/utilities/doing.rb +19 -0
  56. data/examples/utilities/temp.md +28 -0
  57. data/lib/smart_message/base.rb +24 -17
  58. data/lib/smart_message/configuration.rb +2 -23
  59. data/lib/smart_message/dead_letter_queue.rb +1 -1
  60. data/lib/smart_message/errors.rb +3 -0
  61. data/lib/smart_message/header.rb +1 -1
  62. data/lib/smart_message/logger/default.rb +1 -1
  63. data/lib/smart_message/messaging.rb +37 -66
  64. data/lib/smart_message/plugins.rb +42 -41
  65. data/lib/smart_message/serializer/base.rb +1 -1
  66. data/lib/smart_message/serializer.rb +3 -2
  67. data/lib/smart_message/subscription.rb +18 -20
  68. data/lib/smart_message/transport/async_publish_queue.rb +284 -0
  69. data/lib/smart_message/transport/base.rb +42 -8
  70. data/lib/smart_message/transport/fifo_operations.rb +264 -0
  71. data/lib/smart_message/transport/file_operations.rb +200 -0
  72. data/lib/smart_message/transport/file_transport.rb +149 -0
  73. data/lib/smart_message/transport/file_watching.rb +72 -0
  74. data/lib/smart_message/transport/memory_transport.rb +23 -4
  75. data/lib/smart_message/transport/partitioned_files.rb +46 -0
  76. data/lib/smart_message/transport/redis_transport.rb +11 -0
  77. data/lib/smart_message/transport/registry.rb +0 -1
  78. data/lib/smart_message/transport/stdout_transport.rb +73 -41
  79. data/lib/smart_message/transport/stdout_transport.rb.backup +88 -0
  80. data/lib/smart_message/transport.rb +0 -1
  81. data/lib/smart_message/version.rb +1 -1
  82. metadata +25 -37
  83. data/docs/guides/redis-queue-getting-started.md +0 -697
  84. data/docs/guides/redis-queue-patterns.md +0 -889
  85. data/docs/guides/redis-queue-production.md +0 -1091
  86. data/docs/transports/redis-enhanced-transport.md +0 -524
  87. data/docs/transports/redis-queue-transport.md +0 -1304
  88. data/examples/redis_enhanced/README.md +0 -319
  89. data/examples/redis_enhanced/enhanced_01_basic_patterns.rb +0 -233
  90. data/examples/redis_enhanced/enhanced_02_fluent_api.rb +0 -331
  91. data/examples/redis_enhanced/enhanced_03_dual_publishing.rb +0 -281
  92. data/examples/redis_enhanced/enhanced_04_advanced_routing.rb +0 -419
  93. data/examples/redis_queue/01_basic_messaging.rb +0 -221
  94. data/examples/redis_queue/01_comprehensive_examples.rb +0 -508
  95. data/examples/redis_queue/02_pattern_routing.rb +0 -405
  96. data/examples/redis_queue/03_fluent_api.rb +0 -422
  97. data/examples/redis_queue/04_load_balancing.rb +0 -486
  98. data/examples/redis_queue/05_microservices.rb +0 -735
  99. data/examples/redis_queue/06_emergency_alerts.rb +0 -777
  100. data/examples/redis_queue/07_queue_management.rb +0 -587
  101. data/examples/redis_queue/README.md +0 -366
  102. data/examples/redis_queue/enhanced_01_basic_patterns.rb +0 -233
  103. data/examples/redis_queue/enhanced_02_fluent_api.rb +0 -331
  104. data/examples/redis_queue/enhanced_03_dual_publishing.rb +0 -281
  105. data/examples/redis_queue/enhanced_04_advanced_routing.rb +0 -419
  106. data/examples/redis_queue/redis_queue_architecture.svg +0 -148
  107. data/ideas/README.md +0 -41
  108. data/ideas/agents.md +0 -1001
  109. data/ideas/database_transport.md +0 -980
  110. data/ideas/improvement.md +0 -359
  111. data/ideas/meshage.md +0 -1788
  112. data/ideas/message_discovery.md +0 -178
  113. data/ideas/message_schema.md +0 -1381
  114. data/lib/smart_message/transport/redis_enhanced_transport.rb +0 -399
  115. data/lib/smart_message/transport/redis_queue_transport.rb +0 -555
  116. data/lib/smart_message/wrapper.rb.bak +0 -132
  117. /data/examples/memory/{06_pretty_print_demo.rb → 16_pretty_print_demo.rb} +0 -0
@@ -5,34 +5,57 @@ The transport layer is responsible for moving messages between systems. SmartMes
5
5
  ## Overview
6
6
 
7
7
  Transports handle:
8
- - **Publishing**: Sending messages to a destination
8
+ - **Publishing**: Sending messages to a destination (single or multiple transports)
9
9
  - **Subscribing**: Registering interest in message types
10
10
  - **Routing**: Directing incoming messages to the dispatcher
11
11
  - **Connection Management**: Handling connections to external systems
12
12
 
13
+ ## Multi-Transport Publishing
14
+
15
+ SmartMessage supports publishing to **multiple transports simultaneously** for redundancy, integration, and migration scenarios. Configure an array of transports to send messages to multiple destinations with a single `publish()` call.
16
+
17
+ ```ruby
18
+ class CriticalMessage < SmartMessage::Base
19
+ transport [
20
+ SmartMessage::Transport.create(:redis_queue, url: 'redis://primary:6379'),
21
+ SmartMessage::Transport.create(:redis, url: 'redis://backup:6379'),
22
+ SmartMessage::Transport::StdoutTransport.new(format: :json)
23
+ ]
24
+ end
25
+
26
+ message = CriticalMessage.new(data: "important")
27
+ message.publish # ✅ Publishes to all three transports
28
+ ```
29
+
30
+ **📚 See [Multi-Transport Documentation](../transports/multi-transport.md) for comprehensive examples and best practices.**
31
+
13
32
  ## Built-in Transports
14
33
 
15
34
  ### STDOUT Transport
16
35
 
17
- Perfect for development, debugging, and logging scenarios.
36
+ **Publish-only transport** perfect for debugging, logging, and integration with external systems.
18
37
 
19
38
  **Features:**
20
- - Outputs messages to console or file
21
- - Optional loopback for testing subscriptions
22
- - Human-readable message formatting
39
+ - **Publish-only**: No message processing or loopback capability
40
+ - Outputs messages to console or file in pretty-print or JSON formats
41
+ - Subscription attempts are ignored with warning logs
42
+ - Perfect for piping to external tools and log aggregators
23
43
  - No external dependencies
24
44
 
25
45
  **Usage:**
26
46
 
27
47
  ```ruby
28
- # Basic STDOUT output
29
- transport = SmartMessage::Transport.create(:stdout)
48
+ # Basic STDOUT output (publish-only)
49
+ transport = SmartMessage::Transport::StdoutTransport.new
50
+
51
+ # Pretty-printed format for human reading (default)
52
+ transport = SmartMessage::Transport::StdoutTransport.new(format: :pretty)
30
53
 
31
- # With loopback enabled (messages get processed locally)
32
- transport = SmartMessage::Transport.create(:stdout, loopback: true)
54
+ # JSON format for machine processing
55
+ transport = SmartMessage::Transport::StdoutTransport.new(format: :json)
33
56
 
34
57
  # Output to file instead of console
35
- transport = SmartMessage::Transport.create(:stdout, output: "messages.log")
58
+ transport = SmartMessage::Transport::StdoutTransport.new(output: "messages.log")
36
59
 
37
60
  # Configure in message class
38
61
  class LogMessage < SmartMessage::Base
@@ -40,17 +63,19 @@ class LogMessage < SmartMessage::Base
40
63
  property :message
41
64
 
42
65
  config do
43
- transport SmartMessage::Transport.create(:stdout,
44
- output: "app.log",
45
- loopback: false
66
+ transport SmartMessage::Transport::StdoutTransport.new(
67
+ format: :json,
68
+ output: "app.log"
46
69
  )
47
70
  end
48
71
  end
49
72
  ```
50
73
 
51
74
  **Options:**
52
- - `loopback` (Boolean): Whether to process published messages locally (default: false)
53
- - `output` (String|IO): Output destination - filename string or IO object (default: $stdout)
75
+ - `format` (Symbol): Output format - `:pretty` for debugging, `:json` for integration (default: `:pretty`)
76
+ - `output` (String|IO): Output destination - filename string or IO object (default: `$stdout`)
77
+
78
+ **Important:** For local message processing during development, use **MemoryTransport** instead.
54
79
 
55
80
  **Example Output:**
56
81
  ```
@@ -137,12 +162,7 @@ Production-ready Redis pub/sub transport for distributed messaging.
137
162
  - Configurable connection parameters
138
163
  - Background message subscription threads
139
164
 
140
- > **💡 Redis Transport Options:** SmartMessage provides three Redis-based transports:
141
- > - **Redis (Basic)** - Simple pub/sub for basic scenarios
142
- > - **Redis Enhanced** - Pattern-based routing with backwards compatibility
143
- > - **Redis Queue** - Persistent queues with load balancing
144
- >
145
- > See the [Redis Transport Comparison](../transports/redis-transport-comparison.md) for detailed differences and usage guidance.
165
+ > **💡 Redis Transport:** SmartMessage provides a Redis-based transport for production messaging using pub/sub channels. See the [Redis Transport documentation](../transports/redis-transport.md) for detailed usage guidance.
146
166
 
147
167
  **Usage:**
148
168
 
@@ -173,7 +193,6 @@ class OrderMessage < SmartMessage::Base
173
193
  url: 'redis://localhost:6379',
174
194
  db: 1
175
195
  )
176
- serializer SmartMessage::Serializer::Json.new
177
196
  end
178
197
 
179
198
  def self.process(decoded_message)
@@ -250,7 +269,6 @@ redis_transport = SmartMessage::Transport.create(:redis,
250
269
  [OrderMessage, PaymentMessage, ShippingMessage].each do |msg_class|
251
270
  msg_class.config do
252
271
  transport redis_transport
253
- serializer SmartMessage::Serializer::Json.new
254
272
  end
255
273
 
256
274
  # Subscribe to each message type (creates separate Redis subscriptions)
@@ -292,7 +310,6 @@ class ProductionMessage < SmartMessage::Base
292
310
  reconnect_attempts: 10,
293
311
  reconnect_delay: 5
294
312
  )
295
- serializer SmartMessage::Serializer::Json.new
296
313
  logger Logger.new(STDOUT)
297
314
  end
298
315
  end
@@ -309,7 +326,6 @@ class TestMessage < SmartMessage::Base
309
326
  db: 15, # Use separate database for tests
310
327
  auto_subscribe: true
311
328
  )
312
- serializer SmartMessage::Serializer::Json.new
313
329
  end
314
330
  end
315
331
 
@@ -320,120 +336,6 @@ def setup
320
336
  end
321
337
  ```
322
338
 
323
- ### Redis Enhanced Transport
324
-
325
- Advanced Redis transport with intelligent pattern-based routing and RabbitMQ-style capabilities.
326
-
327
- **Features:**
328
- - Pattern-based subscriptions with wildcard support (`*` matching)
329
- - Dual channel publishing for backwards compatibility
330
- - Fluent API for building complex subscription patterns
331
- - Enhanced routing with 3-part channel names: `message_type.from.to`
332
- - Convenience methods for common routing patterns
333
- - Service-oriented messaging patterns
334
-
335
- ![Redis Enhanced Architecture](../assets/images/redis-enhanced-architecture.svg)
336
-
337
- **Usage:**
338
-
339
- ```ruby
340
- # Create enhanced transport
341
- transport = SmartMessage::Transport::RedisEnhancedTransport.new(
342
- url: 'redis://localhost:6379',
343
- db: 0,
344
- auto_subscribe: true
345
- )
346
-
347
- # Configure message with enhanced transport
348
- class OrderMessage < SmartMessage::Base
349
- from 'e-commerce-api'
350
- to 'order-processor'
351
-
352
- transport transport
353
- serializer SmartMessage::Serializer::Json.new
354
-
355
- property :order_id, required: true
356
- property :amount, required: true
357
- end
358
-
359
- # Pattern-based subscriptions
360
- transport.subscribe_pattern("ordermessage.*.*") # All order messages
361
- transport.subscribe_pattern("*.payment_gateway.*") # All from payment gateway
362
- transport.subscribe_to_recipient('order-processor') # Convenience method
363
- transport.subscribe_to_alerts # Emergency patterns
364
-
365
- # Fluent API subscriptions
366
- transport.where.from('web-app').to('user-service').subscribe
367
- transport.where.type('OrderMessage').from('api').subscribe
368
- transport.where.from('monitoring').subscribe
369
-
370
- # Publishing (goes to BOTH channels for compatibility)
371
- order = OrderMessage.new(order_id: 'ORD-123', amount: 99.99)
372
- order.publish
373
- # Publishes to:
374
- # - "OrderMessage" (original format)
375
- # - "ordermessage.e_commerce_api.order_processor" (enhanced format)
376
- ```
377
-
378
- **Pattern Wildcards:**
379
- - `*` - Matches exactly one segment
380
- - `ordermessage.*.*` - All order messages regardless of from/to
381
- - `*.payment_service.*` - All messages from payment service
382
- - `*.*.order_processor` - All messages to order processor
383
-
384
- **Convenience Methods:**
385
- ```ruby
386
- transport.subscribe_to_recipient('service-name') # *.*.service-name
387
- transport.subscribe_from_sender('api-gateway') # *.api-gateway.*
388
- transport.subscribe_to_type('OrderMessage') # ordermessage.*.*
389
- transport.subscribe_to_alerts # Multiple alert patterns
390
- transport.subscribe_to_broadcasts # *.*.broadcast
391
- ```
392
-
393
- **Fluent API:**
394
- ```ruby
395
- # Single conditions
396
- transport.where.from('api').subscribe # *.api.*
397
- transport.where.to('database').subscribe # *.*.database
398
- transport.where.type('LogMessage').subscribe # logmessage.*.*
399
-
400
- # Combined conditions
401
- transport.where.from('web').to('api').subscribe # *.web.api
402
- transport.where.type('Order').from('shop').subscribe # order.shop.*
403
- ```
404
-
405
- **Options:**
406
- - `url` (String): Redis connection URL (default: 'redis://localhost:6379')
407
- - `db` (Integer): Redis database number (default: 0)
408
- - `auto_subscribe` (Boolean): Automatically start subscriber threads (default: true)
409
- - `reconnect_attempts` (Integer): Connection retry attempts (default: 5)
410
- - `reconnect_delay` (Integer): Delay between retries in seconds (default: 1)
411
-
412
- **Backwards Compatibility:**
413
-
414
- The Enhanced Transport maintains full backwards compatibility with the basic Redis transport by publishing to both channel formats:
415
-
416
- ```ruby
417
- # Enhanced transport publishes to BOTH:
418
- # 1. "OrderMessage" (basic format - for backwards compatibility)
419
- # 2. "ordermessage.from_service.to_service" (enhanced format - for pattern matching)
420
-
421
- # Basic Redis transport can still subscribe to "OrderMessage"
422
- # Enhanced transport can use patterns on the enhanced format
423
- ```
424
-
425
- **Use Cases:**
426
- - Microservices architectures requiring sophisticated routing
427
- - Migration from basic Redis transport (maintains compatibility)
428
- - Development environments needing flexible message routing
429
- - Applications requiring RabbitMQ-style patterns without RabbitMQ setup
430
- - Service-to-service communication with intelligent filtering
431
-
432
- **Performance:**
433
- - Minimal overhead over basic Redis transport
434
- - Client-side pattern matching (slightly higher CPU usage)
435
- - Dual publishing increases Redis network traffic by ~2x
436
- - Excellent performance for most production scenarios
437
339
 
438
340
  ## Transport Interface
439
341
 
@@ -533,7 +435,6 @@ class OrderMessage < SmartMessage::Base
533
435
  # All instances use this transport by default
534
436
  config do
535
437
  transport SmartMessage::Transport.create(:memory, auto_process: true)
536
- serializer SmartMessage::Serializer::Json.new
537
438
  end
538
439
  end
539
440
  ```
@@ -545,11 +446,11 @@ end
545
446
  order = OrderMessage.new(order_id: "123", amount: 99.99)
546
447
 
547
448
  order.config do
548
- # This instance will use STDOUT instead of memory
549
- transport SmartMessage::Transport.create(:stdout, loopback: true)
449
+ # This instance will use STDOUT instead of memory (publish-only)
450
+ transport SmartMessage::Transport::StdoutTransport.new(format: :json)
550
451
  end
551
452
 
552
- order.publish # Uses STDOUT transport
453
+ order.publish # Uses STDOUT transport (publish-only)
553
454
  ```
554
455
 
555
456
  ### Runtime Transport Switching
@@ -614,9 +515,9 @@ transport = SmartMessage::Transport.create(:custom,
614
515
  Each transport may have specific options:
615
516
 
616
517
  ```ruby
617
- # STDOUT specific
618
- SmartMessage::Transport.create(:stdout,
619
- loopback: true,
518
+ # STDOUT specific (publish-only)
519
+ SmartMessage::Transport::StdoutTransport.new(
520
+ format: :json,
620
521
  output: "/var/log/messages.log"
621
522
  )
622
523
 
@@ -348,7 +348,8 @@ The `examples/memory/` directory contains comprehensive, runnable examples demon
348
348
  - **[12_regex_filtering_microservices.rb](https://github.com/MadBomber/smart_message/blob/main/examples/memory/12_regex_filtering_microservices.rb)** - Advanced regex filtering for microservices
349
349
 
350
350
  ### Visual Demonstrations
351
- - **[06_pretty_print_demo.rb](https://github.com/MadBomber/smart_message/blob/main/examples/memory/06_pretty_print_demo.rb)** - Message inspection and pretty-printing capabilities
351
+ - **[06_stdout_publish_only.rb](https://github.com/MadBomber/smart_message/blob/main/examples/memory/06_stdout_publish_only.rb)** - STDOUT transport publish-only demonstration with logging and metrics examples
352
+ - **[16_pretty_print_demo.rb](https://github.com/MadBomber/smart_message/blob/main/examples/memory/16_pretty_print_demo.rb)** - Message inspection and pretty-printing capabilities
352
353
 
353
354
  ### Running Examples
354
355