faye-redis-ng 1.0.5 → 1.0.7

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 02a75e3e557cd2916224537b1d688333c2661e1006a836ca92afde12f59c04bb
-  data.tar.gz: 33a2a61187282c3801de1ac648f4d55fa0ad2a46d5ef1c1f964f1e5db5df7ed7
+  metadata.gz: 4a2f33a6f83547306e5a0e52a70e2e8e06a236703c5a56bffc7cf85a829d0a54
+  data.tar.gz: 0e62be0064f4307be4a87d94ddce9e424d7ee4dbad9a85fdf41c03c7ed5db854
 SHA512:
-  metadata.gz: 79f1cdaeb24197a454fbcf9ffa2a2a0678ebe2e09c0ab82a694e9e6b643fba4a43fa14d836c1cb3eeca91c9f4091d63faec489d60c7237e8652b42679c8a811f
-  data.tar.gz: d96e691b5bb66c86c64852474b4bbb9cb1797420adfed71210f64c27f0e695b5489dec71c18d6636daa3ecdcea6e6a1580f90cba850f140e1c7ebc5734d615b8
+  metadata.gz: f67fd292dd0bf0b9fb90a3af34fc7b8ae15627569090c303a06b58f89702124319a8ad96f75603cc38570c5881aa0f4ecaf0e0df39a43ad96ddb9483c3bac51e
+  data.tar.gz: 697a5b1bbd62ebd8f87da936ffc0aa3dc4cad84c9a010f4537dfd5e21c1ea1847ce0af757085bb3a4c7dc6f3a2952cf4c53e01d4a6f33e2fa4e714faffec61be

data/CHANGELOG.md

@@ -7,6 +7,79 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [1.0.7] - 2025-10-30
+
+### Fixed
+- **Critical: Publish Race Condition**: Fixed race condition in `publish` method where callback could be called multiple times
+  - Added `callback_called` flag to prevent duplicate callback invocations
+  - Properly track completion of all async operations before calling final callback
+  - Ensures `success` status is correctly aggregated from all operations
+  - **Impact**: Eliminates unreliable message delivery status in high-concurrency scenarios
+
+- **Critical: Thread Safety Issue**: Fixed thread safety issue in PubSubCoordinator message handling
+  - Changed `EventMachine.next_tick` to `EventMachine.schedule` for cross-thread safety
+  - Added reactor running check before scheduling
+  - Added error handling for subscriber callbacks
+  - **Impact**: Prevents undefined behavior when messages arrive from Redis pub/sub thread
+
+- **Message Deduplication**: Fixed duplicate message enqueue issue
+  - Local published messages were being enqueued twice (local + pub/sub echo)
+  - Added message ID tracking to filter out locally published messages from pub/sub
+  - Messages now include unique IDs for deduplication
+  - **Impact**: Eliminates duplicate messages in single-server deployments
+
+- **Batch Enqueue Logic**: Fixed `enqueue_messages_batch` to handle nil callbacks correctly
+  - Separated empty client list check from callback check
+  - Allows batch enqueue without callback (used by setup_message_routing)
+  - **Impact**: Fixes NoMethodError when enqueue is called without callback
+
+### Added
+- **Concurrency Test Suite**: Added comprehensive concurrency tests (spec/faye/redis_concurrency_spec.rb)
+  - Tests for callback guarantee (single invocation)
+  - Tests for concurrent publish operations
+  - Tests for multi-channel publishing
+  - Tests for error handling
+  - Stress test with 50 rapid publishes
+  - Thread safety tests
+
+### Technical Details
+**Publish Race Condition Fix**:
+- Before: Multiple async callbacks could decrement counter and call callback multiple times
+- After: Track completion with callback_called flag, ensure atomic callback invocation
+
+**Thread Safety Fix**:
+- Before: `EventMachine.next_tick` called from Redis subscriber thread (unsafe)
+- After: `EventMachine.schedule` safely queues work from any thread to EM reactor
+
+**Message Deduplication**:
+- Before: Message published locally → enqueued → published to Redis → received back → enqueued again
+- After: Track local message IDs, filter out self-published messages from pub/sub
+
+## [1.0.6] - 2025-10-30
+
+### Added
+- **Automatic Garbage Collection**: Implemented automatic GC timer that runs periodically to clean up expired clients and orphaned data
+  - New `gc_interval` configuration option (default: 60 seconds)
+  - Automatically starts when EventMachine is running
+  - Can be disabled by setting `gc_interval` to 0 or false
+  - Lazy initialization ensures timer starts even if engine is created before EventMachine starts
+  - Timer is properly stopped on disconnect to prevent resource leaks
+
+### Changed
+- **Improved User Experience**: No longer requires manual setup of periodic cleanup
+  - Memory leak prevention is now automatic by default
+  - Matches behavior of original faye-redis-ruby project
+  - Users can still manually call `cleanup_expired` if needed
+  - Custom GC schedules possible by disabling automatic GC
+
+### Technical Details
+The automatic GC timer:
+- Runs `cleanup_expired` every 60 seconds by default
+- Only starts when EventMachine reactor is running
+- Supports lazy initialization for engines created outside EM context
+- Properly handles cleanup on disconnect
+- Can be customized or disabled via `gc_interval` option
+
 ## [1.0.5] - 2025-10-30
 
 ### Fixed
@@ -126,7 +199,9 @@ For 100 subscribers receiving one message:
 ### Security
 - Client and message IDs now use `SecureRandom.uuid` instead of predictable time-based generation
 
-[Unreleased]: https://github.com/7a6163/faye-redis-ng/compare/v1.0.5...HEAD
+[Unreleased]: https://github.com/7a6163/faye-redis-ng/compare/v1.0.7...HEAD
+[1.0.7]: https://github.com/7a6163/faye-redis-ng/compare/v1.0.6...v1.0.7
+[1.0.6]: https://github.com/7a6163/faye-redis-ng/compare/v1.0.5...v1.0.6
 [1.0.5]: https://github.com/7a6163/faye-redis-ng/compare/v1.0.4...v1.0.5
 [1.0.4]: https://github.com/7a6163/faye-redis-ng/compare/v1.0.3...v1.0.4
 [1.0.3]: https://github.com/7a6163/faye-redis-ng/compare/v1.0.2...v1.0.3

data/README.md

@@ -81,6 +81,9 @@ bayeux = Faye::RackAdapter.new(app, {
   client_timeout: 60,         # Client session timeout (seconds)
   message_ttl: 3600,          # Message TTL (seconds)
 
+  # Garbage collection
+  gc_interval: 60,            # Automatic GC interval (seconds), set to 0 or false to disable
+
   # Logging
   log_level: :info,           # Log level (:silent, :info, :debug)
 
@@ -236,11 +239,45 @@ The CI/CD pipeline will automatically:
 
 ## Memory Management
 
-### Cleaning Up Expired Clients
+### Automatic Garbage Collection
+
+**New in v1.0.6**: faye-redis-ng now includes automatic garbage collection that runs every 60 seconds by default. This automatically cleans up expired clients and orphaned subscription keys, preventing memory leaks without any manual intervention.
+
+```ruby
+bayeux = Faye::RackAdapter.new(app, {
+  mount: '/faye',
+  timeout: 25,
+  engine: {
+    type: Faye::Redis,
+    host: 'localhost',
+    port: 6379,
+    gc_interval: 60  # Run GC every 60 seconds (default)
+  }
+})
+```
+
+To customize the GC interval or disable it:
 
-To prevent memory leaks from orphaned subscription keys, you should periodically clean up expired clients:
+```ruby
+engine: {
+  type: Faye::Redis,
+  host: 'localhost',
+  port: 6379,
+  gc_interval: 300  # Run GC every 5 minutes
+}
 
-#### Manual Cleanup
+# Or disable automatic GC
+engine: {
+  type: Faye::Redis,
+  host: 'localhost',
+  port: 6379,
+  gc_interval: 0  # Disabled - you'll need to call cleanup_expired manually
+}
+```
+
+### Manual Cleanup
+
+If you've disabled automatic GC, you can manually clean up expired clients:
 
 ```ruby
 # Get the engine instance
@@ -252,9 +289,9 @@ engine.cleanup_expired do |expired_count|
 end
 ```
 
-#### Automatic Periodic Cleanup (Recommended)
+#### Custom GC Schedule (Optional)
 
-Add this to your Faye server setup:
+If you need more control, you can disable automatic GC and implement your own schedule:
 
 ```ruby
 require 'eventmachine'
@@ -268,11 +305,12 @@ bayeux = Faye::RackAdapter.new(app, {
     type: Faye::Redis,
     host: 'localhost',
     port: 6379,
-    namespace: 'my-app'
+    namespace: 'my-app',
+    gc_interval: 0  # Disable automatic GC
   }
 })
 
-# Schedule automatic cleanup every 5 minutes
+# Custom cleanup schedule - every 5 minutes
 EM.add_periodic_timer(300) do
   bayeux.get_engine.cleanup_expired do |count|
     puts "[#{Time.now}] Cleaned up #{count} expired clients" if count > 0
@@ -330,12 +368,15 @@ The `cleanup_expired` method removes:
 
 ### Memory Leak Prevention
 
-Without periodic cleanup, abnormal client disconnections (crashes, network failures, etc.) can cause orphaned keys to accumulate:
+**v1.0.6+**: Automatic garbage collection is now enabled by default, preventing memory leaks from orphaned keys without any configuration needed.
+
+Without GC, abnormal client disconnections (crashes, network failures, etc.) can cause orphaned keys to accumulate:
 
-- **Before fix**: 10,000 orphaned clients × 5 channels = 50,000+ keys = 100-500 MB leaked
-- **After fix**: All orphaned keys are cleaned up automatically
+- **Before v1.0.5**: 10,000 orphaned clients × 5 channels = 50,000+ keys = 100-500 MB leaked
+- **v1.0.5**: Manual cleanup required via `cleanup_expired` method
+- **v1.0.6+**: Automatic GC runs every 60 seconds by default - no manual intervention needed
 
-**Recommendation**: Schedule cleanup every 5-10 minutes in production environments.
+The automatic GC ensures memory usage remains stable even with frequent client disconnections.
 
 ## Troubleshooting
 

data/lib/faye/redis/pubsub_coordinator.rb

@@ -166,11 +166,21 @@ module Faye
         begin
           message = JSON.parse(message_json)
 
-          # Notify all subscribers (use dup to avoid concurrent modification)
-          EventMachine.next_tick do
-            @subscribers.dup.each do |subscriber|
-              subscriber.call(channel, message)
+          # Notify all subscribers
+          # Use EventMachine.schedule to safely call from non-EM thread
+          # (handle_message is called from subscriber_thread, not EM reactor thread)
+          if EventMachine.reactor_running?
+            EventMachine.schedule do
+              @subscribers.dup.each do |subscriber|
+                begin
+                  subscriber.call(channel, message)
+                rescue => e
+                  log_error("Subscriber callback error for #{channel}: #{e.message}")
+                end
+              end
             end
+          else
+            log_error("Cannot handle message: EventMachine reactor not running")
           end
         rescue JSON::ParserError => e
           log_error("Failed to parse message from #{channel}: #{e.message}")

data/lib/faye/redis/version.rb

@@ -1,5 +1,5 @@
 module Faye
   class Redis
-    VERSION = '1.0.5'
+    VERSION = '1.0.7'
   end
 end

data/lib/faye/redis.rb

@@ -25,7 +25,8 @@ module Faye
       retry_delay: 1,
       client_timeout: 60,
       message_ttl: 3600,
-      namespace: 'faye'
+      namespace: 'faye',
+      gc_interval: 60  # Automatic garbage collection interval (seconds), set to 0 or false to disable
     }.freeze
 
     attr_reader :server, :options, :connection, :client_registry,
@@ -50,10 +51,16 @@ module Faye
 
       # Set up message routing
       setup_message_routing
+
+      # Start automatic garbage collection timer
+      start_gc_timer
     end
 
     # Create a new client
     def create_client(&callback)
+      # Ensure GC timer is started (lazy initialization)
+      ensure_gc_timer_started
+
       client_id = generate_client_id
       @client_registry.create(client_id) do |success|
         if success
@@ -98,34 +105,68 @@ module Faye
       channels = [channels] unless channels.is_a?(Array)
 
       begin
-        remaining_operations = channels.size
-        success = true
+        # Ensure message has an ID for deduplication
+        message = message.dup unless message.frozen?
+        message['id'] ||= generate_message_id
+
+        # Track this message as locally published
+        if @local_message_ids
+          if @local_message_ids_mutex
+            @local_message_ids_mutex.synchronize { @local_message_ids.add(message['id']) }
+          else
+            @local_message_ids.add(message['id'])
+          end
+        end
+
+        total_channels = channels.size
+        completed_channels = 0
+        callback_called = false
+        all_success = true
 
         channels.each do |channel|
           # Get subscribers and process in parallel
           @subscription_manager.get_subscribers(channel) do |client_ids|
-            # Immediately publish to pub/sub (don't wait for enqueue)
+            # Track operations for this channel
+            pending_ops = 2  # pubsub + enqueue
+            channel_success = true
+            ops_completed = 0
+
+            complete_channel = lambda do
+              ops_completed += 1
+              if ops_completed == pending_ops
+                # This channel is complete
+                all_success &&= channel_success
+                completed_channels += 1
+
+                # Call final callback when all channels are done
+                if completed_channels == total_channels && !callback_called && callback
+                  callback_called = true
+                  EventMachine.next_tick { callback.call(all_success) }
+                end
+              end
+            end
+
+            # Publish to pub/sub
             @pubsub_coordinator.publish(channel, message) do |published|
-              success &&= published
+              channel_success &&= published
+              complete_channel.call
             end
 
-            # Enqueue for all subscribed clients in parallel (batch operation)
+            # Enqueue for all subscribed clients
             if client_ids.any?
               enqueue_messages_batch(client_ids, message) do |enqueued|
-                success &&= enqueued
+                channel_success &&= enqueued
+                complete_channel.call
               end
-            end
-
-            # Track completion
-            remaining_operations -= 1
-            if remaining_operations == 0 && callback
-              EventMachine.next_tick { callback.call(success) }
+            else
+              # No clients, but still need to complete
+              complete_channel.call
             end
           end
         end
       rescue => e
         log_error("Failed to publish message to channels #{channels}: #{e.message}")
-        EventMachine.next_tick { callback.call(false) } if callback
+        EventMachine.next_tick { callback.call(false) } if callback && !callback_called
       end
     end
 
@@ -136,6 +177,9 @@ module Faye
 
     # Disconnect the engine
     def disconnect
+      # Stop GC timer if running
+      stop_gc_timer
+
       @pubsub_coordinator.disconnect
       @connection.disconnect
     end
@@ -159,9 +203,20 @@ module Faye
       SecureRandom.uuid
     end
 
+    def generate_message_id
+      SecureRandom.uuid
+    end
+
     # Batch enqueue messages to multiple clients using a single Redis pipeline
     def enqueue_messages_batch(client_ids, message, &callback)
-      return EventMachine.next_tick { callback.call(true) } if client_ids.empty? || !callback
+      # Handle empty client list
+      if client_ids.empty?
+        EventMachine.next_tick { callback.call(true) } if callback
+        return
+      end
+
+      # No callback provided, but still need to enqueue
+      # (setup_message_routing calls this without callback)
 
       message_json = message.to_json
       message_ttl = @options[:message_ttl] || 3600
@@ -242,10 +297,31 @@ module Faye
     end
 
     def setup_message_routing
+      # Track locally published message IDs to avoid duplicate enqueue
+      @local_message_ids = Set.new
+      @local_message_ids_mutex = Mutex.new if defined?(Mutex)
+
       # Subscribe to message events from other servers
       @pubsub_coordinator.on_message do |channel, message|
+        # Skip if this is a message we just published locally
+        # (Redis pub/sub echoes back messages to the publisher)
+        message_id = message['id']
+        is_local = false
+
+        if message_id
+          if @local_message_ids_mutex
+            @local_message_ids_mutex.synchronize do
+              is_local = @local_message_ids.delete(message_id)
+            end
+          else
+            is_local = @local_message_ids.delete(message_id)
+          end
+        end
+
+        next if is_local
+
+        # Enqueue for remote servers' messages only
         @subscription_manager.get_subscribers(channel) do |client_ids|
-          # Use batch enqueue for better performance
           enqueue_messages_batch(client_ids, message) if client_ids.any?
         end
       end
@@ -254,5 +330,42 @@ module Faye
     def log_error(message)
       @logger.error(message)
     end
+
+    # Start automatic garbage collection timer
+    def start_gc_timer
+      gc_interval = @options[:gc_interval]
+
+      # Skip if GC is disabled (0, false, or nil)
+      return if !gc_interval || gc_interval == 0
+
+      # Only start timer if EventMachine is running
+      return unless EventMachine.reactor_running?
+
+      @logger.info("Starting automatic GC timer with interval: #{gc_interval} seconds")
+
+      @gc_timer = EventMachine.add_periodic_timer(gc_interval) do
+        @logger.debug("Running automatic garbage collection")
+        cleanup_expired do |count|
+          @logger.debug("GC completed: #{count} expired clients cleaned") if count > 0
+        end
+      end
+    end
+
+    # Ensure GC timer is started (called lazily on first operation)
+    def ensure_gc_timer_started
+      return if @gc_timer  # Already started
+      return if !@options[:gc_interval] || @options[:gc_interval] == 0  # Disabled
+
+      start_gc_timer
+    end
+
+    # Stop automatic garbage collection timer
+    def stop_gc_timer
+      if @gc_timer
+        EventMachine.cancel_timer(@gc_timer)
+        @gc_timer = nil
+        @logger.info("Stopped automatic GC timer")
+      end
+    end
   end
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: faye-redis-ng
 version: !ruby/object:Gem::Version
-  version: 1.0.5
+  version: 1.0.7
 platform: ruby
 authors:
 - Zac