faye-redis-ng 1.0.4 → 1.0.6

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: fde6220c9baee883a47eced86a4cd7245ecca48a05a55c906660a5286260c401
-  data.tar.gz: dd3e0d6190cc019070da513d57ac076537b7d8aa1626e29b17c1f7fb91910b79
+  metadata.gz: f87f5800dede3c5720fc5ab77944c3322f735cd808c1b02edd218fc6d97bca8b
+  data.tar.gz: 13d579f33d6620ab7ecdc3567b45dd7c4a58264ed0115b25ddc39454662b2e0c
 SHA512:
-  metadata.gz: 968761695fa0cc17df7c810a345068bf0de18437fd909eba1ed803f784daff107734f846106f67154cfe1c5a3295905c61f1863de40a59a5304199147144fd80
-  data.tar.gz: 8137b865b357b20024edbbb870ff51b76b44779eebe3db18e77167ef326a64322adae9beec35adad04c54028236f6b30c87736fa67642b2684ac9ef73a951c13
+  metadata.gz: fbd69bad2590d4e55b141941952c95e316c3d8dd2a1e37f4ebbe3129ffe00b48957e8ba86c02732ef67aad97200c47b27dc6c663c5c41640cbaa0dd6787e8514
+  data.tar.gz: 8c08572629040c40e130f03c0881e864f02a1dbfb89b5db9c0509c255d4cf986a9f38bf5267c4834216dcae0ca26ef6c115742b42c4f61095c9e4cb14264824a

data/CHANGELOG.md

@@ -7,6 +7,67 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [1.0.6] - 2025-10-30
+
+### Added
+- **Automatic Garbage Collection**: Implemented automatic GC timer that runs periodically to clean up expired clients and orphaned data
+  - New `gc_interval` configuration option (default: 60 seconds)
+  - Automatically starts when EventMachine is running
+  - Can be disabled by setting `gc_interval` to 0 or false
+  - Lazy initialization ensures timer starts even if engine is created before EventMachine starts
+  - Timer is properly stopped on disconnect to prevent resource leaks
+
+### Changed
+- **Improved User Experience**: No longer requires manual setup of periodic cleanup
+  - Memory leak prevention is now automatic by default
+  - Matches behavior of original faye-redis-ruby project
+  - Users can still manually call `cleanup_expired` if needed
+  - Custom GC schedules possible by disabling automatic GC
+
+### Technical Details
+The automatic GC timer:
+- Runs `cleanup_expired` every 60 seconds by default
+- Only starts when EventMachine reactor is running
+- Supports lazy initialization for engines created outside EM context
+- Properly handles cleanup on disconnect
+- Can be customized or disabled via `gc_interval` option
+
+## [1.0.5] - 2025-10-30
+
+### Fixed
+- **Memory Leak**: Fixed critical memory leak where subscription keys were never cleaned up after client disconnection
+  - Orphaned `subscriptions:{client_id}` keys remained permanently in Redis
+  - Orphaned `subscription:{client_id}:{channel}` hash keys accumulated over time
+  - Orphaned client IDs remained in `channels:{channel}` sets
+  - Message queues for disconnected clients were not cleaned up
+  - Could result in hundreds of MB memory leak in production environments
+
+### Added
+- **`cleanup_expired` Method**: New public method to clean up expired clients and orphaned data
+  - Automatically detects and removes orphaned subscription keys
+  - Cleans up message queues for disconnected clients
+  - Removes stale client IDs from channel subscriber lists
+  - Uses Redis SCAN to avoid blocking operations
+  - Batch deletion using pipelining for efficiency
+  - Can be called manually or scheduled as periodic task
+
+### Changed
+- **Improved Cleanup Strategy**: Enhanced cleanup process now handles orphaned data
+  - `cleanup_expired` now cleans both expired clients AND orphaned subscriptions
+  - Works even when no expired clients are found
+  - Prevents memory leaks from abnormal client disconnections
+
+### Technical Details
+Memory leak scenario (before fix):
+- 10,000 abnormally disconnected clients × 5 channels each = 50,000+ orphaned keys
+- Estimated memory waste: 100-500 MB
+- Keys remained permanently without TTL
+
+After fix:
+- All orphaned keys cleaned up automatically
+- Memory usage remains stable
+- Production environments can schedule periodic cleanup
+
 ## [1.0.4] - 2025-10-15
 
 ### Performance
@@ -90,7 +151,9 @@ For 100 subscribers receiving one message:
 ### Security
 - Client and message IDs now use `SecureRandom.uuid` instead of predictable time-based generation
 
-[Unreleased]: https://github.com/7a6163/faye-redis-ng/compare/v1.0.4...HEAD
+[Unreleased]: https://github.com/7a6163/faye-redis-ng/compare/v1.0.6...HEAD
+[1.0.6]: https://github.com/7a6163/faye-redis-ng/compare/v1.0.5...v1.0.6
+[1.0.5]: https://github.com/7a6163/faye-redis-ng/compare/v1.0.4...v1.0.5
 [1.0.4]: https://github.com/7a6163/faye-redis-ng/compare/v1.0.3...v1.0.4
 [1.0.3]: https://github.com/7a6163/faye-redis-ng/compare/v1.0.2...v1.0.3
 [1.0.2]: https://github.com/7a6163/faye-redis-ng/compare/v1.0.1...v1.0.2

data/README.md

@@ -81,6 +81,9 @@ bayeux = Faye::RackAdapter.new(app, {
   client_timeout: 60,         # Client session timeout (seconds)
   message_ttl: 3600,          # Message TTL (seconds)
 
+  # Garbage collection
+  gc_interval: 60,            # Automatic GC interval (seconds), set to 0 or false to disable
+
   # Logging
   log_level: :info,           # Log level (:silent, :info, :debug)
 
@@ -234,6 +237,147 @@ The CI/CD pipeline will automatically:
 - Add `RUBYGEMS_API_KEY` to GitHub repository secrets
 - The tag must start with 'v' (e.g., v0.1.0, v1.2.3)
 
+## Memory Management
+
+### Automatic Garbage Collection
+
+**New in v1.0.6**: faye-redis-ng now includes automatic garbage collection that runs every 60 seconds by default. This automatically cleans up expired clients and orphaned subscription keys, preventing memory leaks without any manual intervention.
+
+```ruby
+bayeux = Faye::RackAdapter.new(app, {
+  mount: '/faye',
+  timeout: 25,
+  engine: {
+    type: Faye::Redis,
+    host: 'localhost',
+    port: 6379,
+    gc_interval: 60  # Run GC every 60 seconds (default)
+  }
+})
+```
+
+To customize the GC interval or disable it:
+
+```ruby
+engine: {
+  type: Faye::Redis,
+  host: 'localhost',
+  port: 6379,
+  gc_interval: 300  # Run GC every 5 minutes
+}
+
+# Or disable automatic GC
+engine: {
+  type: Faye::Redis,
+  host: 'localhost',
+  port: 6379,
+  gc_interval: 0  # Disabled - you'll need to call cleanup_expired manually
+}
+```
+
+### Manual Cleanup
+
+If you've disabled automatic GC, you can manually clean up expired clients:
+
+```ruby
+# Get the engine instance
+engine = bayeux.get_engine
+
+# Clean up expired clients and orphaned data
+engine.cleanup_expired do |expired_count|
+  puts "Cleaned up #{expired_count} expired clients"
+end
+```
+
+#### Custom GC Schedule (Optional)
+
+If you need more control, you can disable automatic GC and implement your own schedule:
+
+```ruby
+require 'eventmachine'
+require 'faye'
+require 'faye-redis-ng'
+
+bayeux = Faye::RackAdapter.new(app, {
+  mount: '/faye',
+  timeout: 25,
+  engine: {
+    type: Faye::Redis,
+    host: 'localhost',
+    port: 6379,
+    namespace: 'my-app',
+    gc_interval: 0  # Disable automatic GC
+  }
+})
+
+# Custom cleanup schedule - every 5 minutes
+EM.add_periodic_timer(300) do
+  bayeux.get_engine.cleanup_expired do |count|
+    puts "[#{Time.now}] Cleaned up #{count} expired clients" if count > 0
+  end
+end
+
+run bayeux
+```
+
+#### Using Rake Task
+
+Create a Rake task for manual or scheduled cleanup:
+
+```ruby
+# lib/tasks/faye_cleanup.rake
+namespace :faye do
+  desc "Clean up expired Faye clients and orphaned subscriptions"
+  task cleanup: :environment do
+    require 'eventmachine'
+
+    EM.run do
+      engine = Faye::Redis.new(
+        nil,
+        host: ENV['REDIS_HOST'] || 'localhost',
+        port: ENV['REDIS_PORT']&.to_i || 6379,
+        namespace: 'my-app'
+      )
+
+      engine.cleanup_expired do |count|
+        puts "✅ Cleaned up #{count} expired clients"
+        engine.disconnect
+        EM.stop
+      end
+    end
+  end
+end
+```
+
+Then schedule it with cron:
+
+```bash
+# Run cleanup every hour
+0 * * * * cd /path/to/app && bundle exec rake faye:cleanup
+```
+
+### What Gets Cleaned Up
+
+The `cleanup_expired` method removes:
+
+1. **Expired client keys** (`clients:{client_id}`)
+2. **Orphaned subscription lists** (`subscriptions:{client_id}`)
+3. **Orphaned subscription metadata** (`subscription:{client_id}:{channel}`)
+4. **Stale client IDs from channel subscribers** (`channels:{channel}`)
+5. **Orphaned message queues** (`messages:{client_id}`)
+
+### Memory Leak Prevention
+
+**v1.0.6+**: Automatic garbage collection is now enabled by default, preventing memory leaks from orphaned keys without any configuration needed.
+
+Without GC, abnormal client disconnections (crashes, network failures, etc.) can cause orphaned keys to accumulate:
+
+- **Before v1.0.5**: 10,000 orphaned clients × 5 channels = 50,000+ keys = 100-500 MB leaked
+- **v1.0.5**: Manual cleanup required via `cleanup_expired` method
+- **v1.0.6+**: Automatic GC runs every 60 seconds by default - no manual intervention needed
+
+The automatic GC ensures memory usage remains stable even with frequent client disconnections.
+
 ## Troubleshooting
 
 ### Connection Issues

data/lib/faye/redis/version.rb

@@ -1,5 +1,5 @@
 module Faye
   class Redis
-    VERSION = '1.0.4'
+    VERSION = '1.0.6'
   end
 end

data/lib/faye/redis.rb

@@ -1,4 +1,5 @@
 require 'securerandom'
+require 'set'
 require_relative 'redis/version'
 require_relative 'redis/logger'
 require_relative 'redis/connection'
@@ -24,7 +25,8 @@ module Faye
       retry_delay: 1,
       client_timeout: 60,
       message_ttl: 3600,
-      namespace: 'faye'
+      namespace: 'faye',
+      gc_interval: 60  # Automatic garbage collection interval (seconds), set to 0 or false to disable
     }.freeze
 
     attr_reader :server, :options, :connection, :client_registry,
@@ -49,10 +51,16 @@ module Faye
 
       # Set up message routing
       setup_message_routing
+
+      # Start automatic garbage collection timer
+      start_gc_timer
     end
 
     # Create a new client
     def create_client(&callback)
+      # Ensure GC timer is started (lazy initialization)
+      ensure_gc_timer_started
+
       client_id = generate_client_id
       @client_registry.create(client_id) do |success|
         if success
@@ -135,10 +143,26 @@ module Faye
 
     # Disconnect the engine
     def disconnect
+      # Stop GC timer if running
+      stop_gc_timer
+
       @pubsub_coordinator.disconnect
       @connection.disconnect
     end
 
+    # Clean up expired clients and their associated data
+    def cleanup_expired(&callback)
+      @client_registry.cleanup_expired do |expired_count|
+        @logger.info("Cleaned up #{expired_count} expired clients") if expired_count > 0
+
+        # Always clean up orphaned subscription keys (even if no expired clients)
+        # This handles cases where subscriptions were orphaned due to crashes
+        cleanup_orphaned_subscriptions do
+          callback.call(expired_count) if callback
+        end
+      end
+    end
+
     private
 
     def generate_client_id
@@ -171,6 +195,62 @@ module Faye
       end
     end
 
+    def cleanup_orphaned_subscriptions(&callback)
+      # Get all active client IDs
+      @client_registry.all do |active_clients|
+        active_set = active_clients.to_set
+        namespace = @options[:namespace] || 'faye'
+
+        # Scan for subscription keys and clean up orphaned ones
+        @connection.with_redis do |redis|
+          cursor = "0"
+          orphaned_keys = []
+
+          loop do
+            cursor, keys = redis.scan(cursor, match: "#{namespace}:subscriptions:*", count: 100)
+
+            keys.each do |key|
+              # Extract client_id from key (format: namespace:subscriptions:client_id)
+              client_id = key.split(':').last
+              orphaned_keys << client_id unless active_set.include?(client_id)
+            end
+
+            break if cursor == "0"
+          end
+
+          # Clean up orphaned subscription data
+          if orphaned_keys.any?
+            @logger.info("Cleaning up #{orphaned_keys.size} orphaned subscription sets")
+
+            orphaned_keys.each do |client_id|
+              # Get channels for this orphaned client
+              channels = redis.smembers("#{namespace}:subscriptions:#{client_id}")
+
+              # Remove in batch
+              redis.pipelined do |pipeline|
+                # Delete client's subscription list
+                pipeline.del("#{namespace}:subscriptions:#{client_id}")
+
+                # Delete each subscription metadata and remove from channel subscribers
+                channels.each do |channel|
+                  pipeline.del("#{namespace}:subscription:#{client_id}:#{channel}")
+                  pipeline.srem("#{namespace}:channels:#{channel}", client_id)
+                end
+
+                # Delete message queue if exists
+                pipeline.del("#{namespace}:messages:#{client_id}")
+              end
+            end
+          end
+        end
+
+        EventMachine.next_tick { callback.call } if callback
+      end
+    rescue => e
+      log_error("Failed to cleanup orphaned subscriptions: #{e.message}")
+      EventMachine.next_tick { callback.call } if callback
+    end
+
     def setup_message_routing
       # Subscribe to message events from other servers
       @pubsub_coordinator.on_message do |channel, message|
@@ -184,5 +264,42 @@ module Faye
     def log_error(message)
       @logger.error(message)
     end
+
+    # Start automatic garbage collection timer
+    def start_gc_timer
+      gc_interval = @options[:gc_interval]
+
+      # Skip if GC is disabled (0, false, or nil)
+      return if !gc_interval || gc_interval == 0
+
+      # Only start timer if EventMachine is running
+      return unless EventMachine.reactor_running?
+
+      @logger.info("Starting automatic GC timer with interval: #{gc_interval} seconds")
+
+      @gc_timer = EventMachine.add_periodic_timer(gc_interval) do
+        @logger.debug("Running automatic garbage collection")
+        cleanup_expired do |count|
+          @logger.debug("GC completed: #{count} expired clients cleaned") if count > 0
+        end
+      end
+    end
+
+    # Ensure GC timer is started (called lazily on first operation)
+    def ensure_gc_timer_started
+      return if @gc_timer  # Already started
+      return if !@options[:gc_interval] || @options[:gc_interval] == 0  # Disabled
+
+      start_gc_timer
+    end
+
+    # Stop automatic garbage collection timer
+    def stop_gc_timer
+      if @gc_timer
+        EventMachine.cancel_timer(@gc_timer)
+        @gc_timer = nil
+        @logger.info("Stopped automatic GC timer")
+      end
+    end
   end
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: faye-redis-ng
 version: !ruby/object:Gem::Version
-  version: 1.0.4
+  version: 1.0.6
 platform: ruby
 authors:
 - Zac
 autorequire:
 bindir: bin
 cert_chain: []
-date: 2025-10-15 00:00:00.000000000 Z
+date: 2025-10-30 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: redis