faye-redis-ng 1.0.4 → 1.0.5

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: fde6220c9baee883a47eced86a4cd7245ecca48a05a55c906660a5286260c401
-  data.tar.gz: dd3e0d6190cc019070da513d57ac076537b7d8aa1626e29b17c1f7fb91910b79
+  metadata.gz: 02a75e3e557cd2916224537b1d688333c2661e1006a836ca92afde12f59c04bb
+  data.tar.gz: 33a2a61187282c3801de1ac648f4d55fa0ad2a46d5ef1c1f964f1e5db5df7ed7
 SHA512:
-  metadata.gz: 968761695fa0cc17df7c810a345068bf0de18437fd909eba1ed803f784daff107734f846106f67154cfe1c5a3295905c61f1863de40a59a5304199147144fd80
-  data.tar.gz: 8137b865b357b20024edbbb870ff51b76b44779eebe3db18e77167ef326a64322adae9beec35adad04c54028236f6b30c87736fa67642b2684ac9ef73a951c13
+  metadata.gz: 79f1cdaeb24197a454fbcf9ffa2a2a0678ebe2e09c0ab82a694e9e6b643fba4a43fa14d836c1cb3eeca91c9f4091d63faec489d60c7237e8652b42679c8a811f
+  data.tar.gz: d96e691b5bb66c86c64852474b4bbb9cb1797420adfed71210f64c27f0e695b5489dec71c18d6636daa3ecdcea6e6a1580f90cba850f140e1c7ebc5734d615b8

data/CHANGELOG.md

@@ -7,6 +7,42 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [1.0.5] - 2025-10-30
+
+### Fixed
+- **Memory Leak**: Fixed critical memory leak where subscription keys were never cleaned up after client disconnection
+  - Orphaned `subscriptions:{client_id}` keys remained permanently in Redis
+  - Orphaned `subscription:{client_id}:{channel}` hash keys accumulated over time
+  - Orphaned client IDs remained in `channels:{channel}` sets
+  - Message queues for disconnected clients were not cleaned up
+  - Could result in hundreds of MB memory leak in production environments
+
+### Added
+- **`cleanup_expired` Method**: New public method to clean up expired clients and orphaned data
+  - Automatically detects and removes orphaned subscription keys
+  - Cleans up message queues for disconnected clients
+  - Removes stale client IDs from channel subscriber lists
+  - Uses Redis SCAN to avoid blocking operations
+  - Batch deletion using pipelining for efficiency
+  - Can be called manually or scheduled as periodic task
+
+### Changed
+- **Improved Cleanup Strategy**: Enhanced cleanup process now handles orphaned data
+  - `cleanup_expired` now cleans both expired clients AND orphaned subscriptions
+  - Works even when no expired clients are found
+  - Prevents memory leaks from abnormal client disconnections
+
+### Technical Details
+Memory leak scenario (before fix):
+- 10,000 abnormally disconnected clients × 5 channels each = 50,000+ orphaned keys
+- Estimated memory waste: 100-500 MB
+- Keys remained permanently without TTL
+
+After fix:
+- All orphaned keys cleaned up automatically
+- Memory usage remains stable
+- Production environments can schedule periodic cleanup
+
 ## [1.0.4] - 2025-10-15
 
 ### Performance
@@ -90,7 +126,8 @@ For 100 subscribers receiving one message:
 ### Security
 - Client and message IDs now use `SecureRandom.uuid` instead of predictable time-based generation
 
-[Unreleased]: https://github.com/7a6163/faye-redis-ng/compare/v1.0.4...HEAD
+[Unreleased]: https://github.com/7a6163/faye-redis-ng/compare/v1.0.5...HEAD
+[1.0.5]: https://github.com/7a6163/faye-redis-ng/compare/v1.0.4...v1.0.5
 [1.0.4]: https://github.com/7a6163/faye-redis-ng/compare/v1.0.3...v1.0.4
 [1.0.3]: https://github.com/7a6163/faye-redis-ng/compare/v1.0.2...v1.0.3
 [1.0.2]: https://github.com/7a6163/faye-redis-ng/compare/v1.0.1...v1.0.2

data/README.md

@@ -234,6 +234,109 @@ The CI/CD pipeline will automatically:
 - Add `RUBYGEMS_API_KEY` to GitHub repository secrets
 - The tag must start with 'v' (e.g., v0.1.0, v1.2.3)
 
+## Memory Management
+
+### Cleaning Up Expired Clients
+
+To prevent memory leaks from orphaned subscription keys, you should periodically clean up expired clients:
+
+#### Manual Cleanup
+
+```ruby
+# Get the engine instance
+engine = bayeux.get_engine
+
+# Clean up expired clients and orphaned data
+engine.cleanup_expired do |expired_count|
+  puts "Cleaned up #{expired_count} expired clients"
+end
+```
+
+#### Automatic Periodic Cleanup (Recommended)
+
+Add this to your Faye server setup:
+
+```ruby
+require 'eventmachine'
+require 'faye'
+require 'faye-redis-ng'
+
+bayeux = Faye::RackAdapter.new(app, {
+  mount: '/faye',
+  timeout: 25,
+  engine: {
+    type: Faye::Redis,
+    host: 'localhost',
+    port: 6379,
+    namespace: 'my-app'
+  }
+})
+
+# Schedule automatic cleanup every 5 minutes
+EM.add_periodic_timer(300) do
+  bayeux.get_engine.cleanup_expired do |count|
+    puts "[#{Time.now}] Cleaned up #{count} expired clients" if count > 0
+  end
+end
+
+run bayeux
+```
+
+#### Using Rake Task
+
+Create a Rake task for manual or scheduled cleanup:
+
+```ruby
+# lib/tasks/faye_cleanup.rake
+namespace :faye do
+  desc "Clean up expired Faye clients and orphaned subscriptions"
+  task cleanup: :environment do
+    require 'eventmachine'
+
+    EM.run do
+      engine = Faye::Redis.new(
+        nil,
+        host: ENV['REDIS_HOST'] || 'localhost',
+        port: ENV['REDIS_PORT']&.to_i || 6379,
+        namespace: 'my-app'
+      )
+
+      engine.cleanup_expired do |count|
+        puts "✅ Cleaned up #{count} expired clients"
+        engine.disconnect
+        EM.stop
+      end
+    end
+  end
+end
+```
+
+Then schedule it with cron:
+
+```bash
+# Run cleanup every hour
+0 * * * * cd /path/to/app && bundle exec rake faye:cleanup
+```
+
+### What Gets Cleaned Up
+
+The `cleanup_expired` method removes:
+
+1. **Expired client keys** (`clients:{client_id}`)
+2. **Orphaned subscription lists** (`subscriptions:{client_id}`)
+3. **Orphaned subscription metadata** (`subscription:{client_id}:{channel}`)
+4. **Stale client IDs from channel subscribers** (`channels:{channel}`)
+5. **Orphaned message queues** (`messages:{client_id}`)
+
+### Memory Leak Prevention
+
+Without periodic cleanup, abnormal client disconnections (crashes, network failures, etc.) can cause orphaned keys to accumulate:
+
+- **Before fix**: 10,000 orphaned clients × 5 channels = 50,000+ keys = 100-500 MB leaked
+- **After fix**: All orphaned keys are cleaned up automatically
+
+**Recommendation**: Schedule cleanup every 5-10 minutes in production environments.
+
 ## Troubleshooting
 
 ### Connection Issues

data/lib/faye/redis/version.rb

@@ -1,5 +1,5 @@
 module Faye
   class Redis
-    VERSION = '1.0.4'
+    VERSION = '1.0.5'
   end
 end

data/lib/faye/redis.rb

@@ -1,4 +1,5 @@
 require 'securerandom'
+require 'set'
 require_relative 'redis/version'
 require_relative 'redis/logger'
 require_relative 'redis/connection'
@@ -139,6 +140,19 @@ module Faye
       @connection.disconnect
     end
 
+    # Clean up expired clients and their associated data
+    def cleanup_expired(&callback)
+      @client_registry.cleanup_expired do |expired_count|
+        @logger.info("Cleaned up #{expired_count} expired clients") if expired_count > 0
+
+        # Always clean up orphaned subscription keys (even if no expired clients)
+        # This handles cases where subscriptions were orphaned due to crashes
+        cleanup_orphaned_subscriptions do
+          callback.call(expired_count) if callback
+        end
+      end
+    end
+
     private
 
     def generate_client_id
@@ -171,6 +185,62 @@ module Faye
       end
     end
 
+    def cleanup_orphaned_subscriptions(&callback)
+      # Get all active client IDs
+      @client_registry.all do |active_clients|
+        active_set = active_clients.to_set
+        namespace = @options[:namespace] || 'faye'
+
+        # Scan for subscription keys and clean up orphaned ones
+        @connection.with_redis do |redis|
+          cursor = "0"
+          orphaned_keys = []
+
+          loop do
+            cursor, keys = redis.scan(cursor, match: "#{namespace}:subscriptions:*", count: 100)
+
+            keys.each do |key|
+              # Extract client_id from key (format: namespace:subscriptions:client_id)
+              client_id = key.split(':').last
+              orphaned_keys << client_id unless active_set.include?(client_id)
+            end
+
+            break if cursor == "0"
+          end
+
+          # Clean up orphaned subscription data
+          if orphaned_keys.any?
+            @logger.info("Cleaning up #{orphaned_keys.size} orphaned subscription sets")
+
+            orphaned_keys.each do |client_id|
+              # Get channels for this orphaned client
+              channels = redis.smembers("#{namespace}:subscriptions:#{client_id}")
+
+              # Remove in batch
+              redis.pipelined do |pipeline|
+                # Delete client's subscription list
+                pipeline.del("#{namespace}:subscriptions:#{client_id}")
+
+                # Delete each subscription metadata and remove from channel subscribers
+                channels.each do |channel|
+                  pipeline.del("#{namespace}:subscription:#{client_id}:#{channel}")
+                  pipeline.srem("#{namespace}:channels:#{channel}", client_id)
+                end
+
+                # Delete message queue if exists
+                pipeline.del("#{namespace}:messages:#{client_id}")
+              end
+            end
+          end
+        end
+
+        EventMachine.next_tick { callback.call } if callback
+      end
+    rescue => e
+      log_error("Failed to cleanup orphaned subscriptions: #{e.message}")
+      EventMachine.next_tick { callback.call } if callback
+    end
+
     def setup_message_routing
       # Subscribe to message events from other servers
       @pubsub_coordinator.on_message do |channel, message|

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: faye-redis-ng
 version: !ruby/object:Gem::Version
-  version: 1.0.4
+  version: 1.0.5
 platform: ruby
 authors:
 - Zac
 autorequire:
 bindir: bin
 cert_chain: []
-date: 2025-10-15 00:00:00.000000000 Z
+date: 2025-10-30 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: redis