fast_exists 1.0.0

data/docs/SCALING.md

@@ -0,0 +1,74 @@
+# 🏢 Enterprise Adaptive Multi-Tenant Bloom Filter Management
+
+SaaS applications often serve thousands of tenants with dramatically different data sizes (e.g. 10 Enterprise customers vs 4,500 SMB customers).
+
+- Allocating a dedicated Bloom filter per tenant wastes memory (4,500 Redis keys).
+- Using a single global Bloom filter increases false positives and reduces operational flexibility.
+
+`FastExists` provides **Adaptive Multi-Tenant Bloom Filter Management** (`:adaptive`), automatically classifying tenants into optimal pools based on record volume and lookup traffic.
+
+---
+
+## ⚙️ Configuration
+
+```ruby
+# config/initializers/fast_exists.rb
+FastExists.configure do |config|
+  config.backend = :redis
+  config.multi_tenant = true
+  config.tenant_strategy = :adaptive # :global, :per_tenant, :shared, :adaptive
+
+  # Configurable record count classification thresholds
+  config.tenant_thresholds = {
+    tiny: 10_000,
+    small: 100_000,
+    medium: 1_000_000
+  }
+end
+```
+
+---
+
+## 🎯 Supported Strategies
+
+1. **`:global` (`GlobalStrategy`)**: Shared Bloom filter (`fast_exists:global`). Hashing `tenant_id:attribute:value`.
+2. **`:per_tenant` (`PerTenantStrategy`)**: Dedicated Bloom filter per tenant (`fast_exists:tenant:<id>`). Best for large Enterprise tenants (>1,000,000 records).
+3. **`:shared` (`SharedStrategy`)**: Configurable shared pools (`small`, `medium`, `large`).
+4. **`:adaptive` (`AdaptiveStrategy`)**: Recommended default. Automatically classifies tenants:
+   - `Tiny` (< 10,000 records) -> Shared `tiny` pool (`fast_exists:pool:tiny`)
+   - `Small` (10,000–100,000 records) -> Shared `small` pool (`fast_exists:pool:small`)
+   - `Medium` (100,000–1,000,000 records) -> Shared `medium` pool (`fast_exists:pool:medium`)
+   - `Large` (> 1,000,000 records) -> Dedicated filter (`fast_exists:tenant:<id>`)
+
+---
+
+## 🔑 Redis Key Layout
+
+```text
+fast_exists:global
+fast_exists:pool:tiny
+fast_exists:pool:small
+fast_exists:pool:medium
+fast_exists:tenant:42
+fast_exists:tenant:83
+```
+
+---
+
+## 📈 Promotion & Demotion Recommendations
+
+As tenants grow or shrink, `FastExists.doctor!` and `FastExists.analyze!` recommend promotion or demotion actions.
+
+> [!IMPORTANT]
+> **Zero Automatic Data Migration**: Recommendations are strictly read-only. Automatic rebalancing or migration never occurs without explicit developer action.
+
+---
+
+## 📊 Analytics & Doctor Output Example
+
+For an application with 4,287 tenants (3,901 Tiny, 312 Small, 64 Medium, 10 Large):
+
+- **Per-Tenant Strategy**: 4,287 Redis keys (~220 MB RAM)
+- **Adaptive Strategy**: 13 Redis keys (~48 MB RAM)
+- **Memory Reduction**: **78%**
+- **Redis Key Reduction**: **99.7%**

data/docs/TROUBLESHOOTING.md

@@ -0,0 +1,9 @@
+# Troubleshooting Guide
+
+### Issue: High False Positive Rate
+- **Symptom**: `FastExists.stats[:false_positive_rate]` is > 1%.
+- **Solution**: Increase `expected_elements` in your initializer or run `FastExists::Optimizer::AiAdvisor.analyze(User)` to get recommended parameters.
+
+### Issue: Redis Connection Error
+- **Symptom**: `FastExists::BackendError: Redis client not configured`.
+- **Solution**: Set `config.redis = Redis.new(...)` in `config/initializers/fast_exists.rb`.

data/lib/fast_exists/active_record/extension.rb

@@ -0,0 +1,36 @@
+# frozen_string_literal: true
+
+module FastExists
+  module ActiveRecord
+    module Extension
+      def self.included(base)
+        base.extend(ClassMethods)
+      end
+
+      module ClassMethods
+        def fast_exists(*attributes)
+          options = attributes.extract_options!
+
+          class_attribute :fast_exists_attributes, default: {} unless respond_to?(:fast_exists_attributes)
+
+          attributes.map(&:to_sym).each do |attr|
+            self.fast_exists_attributes = fast_exists_attributes.merge(attr => options)
+
+            # Generate attribute_exists?(val)
+            singleton_class.define_method("#{attr}_exists?") do |value|
+              fast_exists?(attr, value)
+            end
+
+            # Generate attribute_available?(val)
+            singleton_class.define_method("#{attr}_available?") do |value|
+              fast_available?(attr, value)
+            end
+          end
+
+          extend FastExists::ActiveRecord::ModelMethods
+          include FastExists::ActiveRecord::Hooks
+        end
+      end
+    end
+  end
+end

data/lib/fast_exists/active_record/hooks.rb

@@ -0,0 +1,25 @@
+# frozen_string_literal: true
+
+module FastExists
+  module ActiveRecord
+    module Hooks
+      def self.included(base)
+        base.after_commit :fast_exists_sync_on_commit, on: [:create, :update]
+      end
+
+      private
+
+      def fast_exists_sync_on_commit
+        return unless FastExists.configuration.auto_sync
+
+        self.class.fast_exists_attributes.each do |attr, options|
+          val = send(attr)
+          next if val.nil? || val.to_s.strip.empty?
+
+          backend = self.class.fast_exists_backend_for(attr)
+          backend.add(val.to_s)
+        end
+      end
+    end
+  end
+end

data/lib/fast_exists/active_record/model_methods.rb

@@ -0,0 +1,109 @@
+# frozen_string_literal: true
+
+module FastExists
+  module ActiveRecord
+    module ModelMethods
+      def fast_exists?(*args)
+        attribute, value = args.size == 2 ? args : [fast_exists_attributes.keys.first, args.first]
+        attribute = attribute.to_sym
+        raise InvalidArgumentError, "Attribute #{attribute} is not configured with fast_exists" unless fast_exists_attributes.key?(attribute)
+
+        backend = fast_exists_backend_for(attribute)
+        str_val = value.to_s
+
+        FastExists::Instrumentation::EventSubscriber.instrument(:lookup, model: name, attribute: attribute, value: str_val)
+
+        # Check probabilistic filter
+        if backend.contains?(str_val)
+          FastExists.stats_tracker.record_hit
+          FastExists.stats_tracker.record_db_lookup
+
+          # DB check
+          exists_in_db = where(attribute => value).exists?
+
+          if exists_in_db
+            FastExists::Instrumentation::EventSubscriber.instrument(:hit, model: name, attribute: attribute, value: str_val)
+            true
+          else
+            FastExists.stats_tracker.record_false_positive
+            FastExists::Instrumentation::EventSubscriber.instrument(:false_positive, model: name, attribute: attribute, value: str_val)
+            false
+          end
+        else
+          FastExists.stats_tracker.record_avoided
+          FastExists::Instrumentation::EventSubscriber.instrument(:miss, model: name, attribute: attribute, value: str_val)
+          false
+        end
+      end
+
+      def fast_available?(attribute, value = nil)
+        if value.nil?
+          value = attribute
+          attribute = fast_exists_attributes.keys.first
+        end
+        !fast_exists?(attribute, value)
+      end
+
+      def rebuild_fast_exists!(attribute = nil)
+        attributes_to_rebuild = attribute ? [attribute.to_sym] : fast_exists_attributes.keys
+
+        attributes_to_rebuild.each do |attr|
+          backend = fast_exists_backend_for(attr)
+          backend.clear
+
+          if respond_to?(:find_in_batches)
+            find_in_batches(batch_size: 5000) do |batch|
+              batch.each do |record|
+                val = record.send(attr)
+                backend.add(val.to_s) if val.present?
+              end
+            end
+          else
+            pluck(attr).compact.each do |val|
+              backend.add(val.to_s)
+            end
+          end
+          backend.save if backend.respond_to?(:save)
+        end
+        true
+      end
+
+      def rebuild_fast_exists_async!(attribute = nil)
+        if defined?(Sidekiq)
+          # Asynchronous sidekiq dispatch
+          FastExists::Jobs::RebuildJob.perform_async(name, attribute&.to_s)
+        else
+          rebuild_fast_exists!(attribute)
+        end
+      end
+
+      def fast_exists_stats(attribute = nil)
+        attributes = attribute ? [attribute.to_sym] : fast_exists_attributes.keys
+        stats_hash = {}
+
+        attributes.each do |attr|
+          backend = fast_exists_backend_for(attr)
+          stats_hash[attr] = backend.stats.merge(global_stats: FastExists.stats)
+        end
+
+        attribute ? stats_hash[attribute.to_sym] : stats_hash
+      end
+
+      def fast_exists_backend_for(attribute)
+        @fast_exists_backends ||= {}
+        @fast_exists_backends[attribute.to_sym] ||= begin
+          options = fast_exists_attributes[attribute.to_sym] || {}
+          backend_name = options[:backend] || FastExists.configuration.backend
+          namespace = options[:namespace] || "#{name.underscore}:#{attribute}"
+
+          resolved_ns = FastExists::MultiTenancy::Resolver.resolve(namespace)
+
+          FastExists.backends.fetch(
+            backend_name,
+            options.merge(namespace: resolved_ns)
+          )
+        end
+      end
+    end
+  end
+end

data/lib/fast_exists/backends/base.rb

@@ -0,0 +1,60 @@
+# frozen_string_literal: true
+
+module FastExists
+  module Backends
+    class Base
+      attr_reader :options
+
+      def initialize(options = {})
+        @options = options
+        @expected_elements = options[:expected_elements] || FastExists.configuration.expected_elements
+        @false_positive_rate = options[:false_positive_rate] || FastExists.configuration.false_positive_rate
+      end
+
+      def add(key)
+        raise NotImplementedError, "#{self.class.name}#add is not implemented"
+      end
+
+      def contains?(key)
+        raise NotImplementedError, "#{self.class.name}#contains? is not implemented"
+      end
+
+      def clear
+        raise NotImplementedError, "#{self.class.name}#clear is not implemented"
+      end
+
+      def save
+        true
+      end
+
+      def load
+        true
+      end
+
+      def count
+        0
+      end
+
+      def capacity
+        @expected_elements
+      end
+
+      def stats
+        {
+          backend: self.class.name,
+          capacity: capacity,
+          count: count
+        }
+      end
+
+      def rebuild(enumerable)
+        clear
+        enumerable.each do |item|
+          add(item)
+        end
+        save
+        true
+      end
+    end
+  end
+end

data/lib/fast_exists/backends/file.rb

@@ -0,0 +1,70 @@
+# frozen_string_literal: true
+
+require "fileutils"
+require "json"
+
+module FastExists
+  module Backends
+    class File < Base
+      attr_reader :file_path, :filter
+
+      def initialize(options = {})
+        super
+        @file_path = options[:file_path] || FastExists.configuration.file_path || "tmp/fast_exists_#{@options[:namespace] || 'default'}.bloom"
+        @filter = FastExists::Bloom::Filter.new(
+          expected_elements: @expected_elements,
+          false_positive_rate: @false_positive_rate
+        )
+        load if ::File.exist?(@file_path)
+      end
+
+      def add(key)
+        res = @filter.add(key)
+        save
+        res
+      end
+
+      def contains?(key)
+        @filter.contains?(key)
+      end
+
+      def clear
+        res = @filter.clear
+        ::File.delete(@file_path) if ::File.exist?(@file_path)
+        res
+      end
+
+      def count
+        @filter.count
+      end
+
+      def capacity
+        @filter.capacity
+      end
+
+      def save
+        FileUtils.mkdir_p(::File.dirname(@file_path))
+        dump = @filter.dump
+        temp_file = "#{@file_path}.tmp"
+        ::File.binwrite(temp_file, JSON.generate(dump))
+        ::File.rename(temp_file, @file_path)
+        true
+      rescue => e
+        raise BackendError, "Failed to save file backend: #{e.message}"
+      end
+
+      def load
+        return false unless ::File.exist?(@file_path)
+        data = JSON.parse(::File.binread(@file_path), symbolize_names: true)
+        @filter = FastExists::Bloom::Filter.load(data)
+        true
+      rescue => e
+        raise BackendError, "Failed to load file backend: #{e.message}"
+      end
+
+      def stats
+        @filter.stats.merge(backend: :file, file_path: @file_path)
+      end
+    end
+  end
+end

data/lib/fast_exists/backends/memory.rb

@@ -0,0 +1,41 @@
+# frozen_string_literal: true
+
+module FastExists
+  module Backends
+    class Memory < Base
+      attr_reader :filter
+
+      def initialize(options = {})
+        super
+        @filter = FastExists::Bloom::Filter.new(
+          expected_elements: @expected_elements,
+          false_positive_rate: @false_positive_rate
+        )
+      end
+
+      def add(key)
+        @filter.add(key)
+      end
+
+      def contains?(key)
+        @filter.contains?(key)
+      end
+
+      def clear
+        @filter.clear
+      end
+
+      def count
+        @filter.count
+      end
+
+      def capacity
+        @filter.capacity
+      end
+
+      def stats
+        @filter.stats.merge(backend: :memory)
+      end
+    end
+  end
+end

data/lib/fast_exists/backends/null.rb

@@ -0,0 +1,31 @@
+# frozen_string_literal: true
+
+module FastExists
+  module Backends
+    class Null < Base
+      def add(_key)
+        true
+      end
+
+      def contains?(_key)
+        true # Always maybe present -> forces DB check (safe fallback)
+      end
+
+      def clear
+        true
+      end
+
+      def count
+        0
+      end
+
+      def capacity
+        @expected_elements
+      end
+
+      def stats
+        { backend: :null, capacity: capacity, count: 0, status: :disabled }
+      end
+    end
+  end
+end

data/lib/fast_exists/backends/redis.rb

@@ -0,0 +1,98 @@
+# frozen_string_literal: true
+
+require "zlib"
+require "digest"
+
+module FastExists
+  module Backends
+    class Redis < Base
+      attr_reader :key_prefix, :bit_size, :hash_count
+
+      def initialize(options = {})
+        super
+        @redis = options[:redis] || FastExists.configuration.redis
+        @key_prefix = options[:namespace] ? "fast_exists:#{options[:namespace]}" : "fast_exists:default"
+
+        # Bloom filter parameters calculation
+        n = @expected_elements
+        p = @false_positive_rate
+        @bit_size = (-(n * Math.log(p)) / (Math.log(2)**2)).ceil
+        @hash_count = [((@bit_size.to_f / n) * Math.log(2)).round, 1].max
+        @count_key = "#{@key_prefix}:count"
+        @bit_key = "#{@key_prefix}:bits"
+      end
+
+      def add(element)
+        indexes = hash_indexes(element.to_s)
+        execute_redis do |client|
+          indexes.each do |idx|
+            client.setbit(@bit_key, idx, 1)
+          end
+          client.incr(@count_key)
+        end
+        true
+      end
+
+      def contains?(element)
+        indexes = hash_indexes(element.to_s)
+        execute_redis do |client|
+          indexes.all? do |idx|
+            client.getbit(@bit_key, idx) == 1
+          end
+        end
+      end
+
+      def clear
+        execute_redis do |client|
+          client.del(@bit_key, @count_key)
+        end
+        true
+      end
+
+      def count
+        execute_redis do |client|
+          (client.get(@count_key) || 0).to_i
+        end
+      end
+
+      def capacity
+        @expected_elements
+      end
+
+      def stats
+        {
+          backend: :redis,
+          bit_key: @bit_key,
+          expected_elements: @expected_elements,
+          false_positive_rate: @false_positive_rate,
+          inserted_items: count,
+          bit_size: @bit_size,
+          hash_count: @hash_count
+        }
+      end
+
+      private
+
+      def execute_redis
+        if @redis.respond_to?(:with)
+          @redis.with { |conn| yield conn }
+        elsif @redis
+          yield @redis
+        elsif defined?(::Redis) && ::Redis.respond_to?(:current)
+          yield ::Redis.current
+        else
+          raise BackendError, "Redis client not configured. Set config.redis = Redis.new"
+        end
+      end
+
+      def hash_indexes(key)
+        h1 = Zlib.crc32(key)
+        h2 = Digest::SHA256.hexdigest(key)[0..7].hex
+
+        Array.new(@hash_count) do |i|
+          (h1 + i * h2) % @bit_size
+        end
+      end
+    end
+  end
+end

data/lib/fast_exists/backends/redis_bloom.rb

@@ -0,0 +1,83 @@
+# frozen_string_literal: true
+
+module FastExists
+  module Backends
+    class RedisBloom < Base
+      attr_reader :filter_key
+
+      def initialize(options = {})
+        super
+        @redis = options[:redis] || FastExists.configuration.redis
+        @filter_key = options[:namespace] ? "fast_exists:bf:#{options[:namespace]}" : "fast_exists:bf:default"
+        ensure_filter_reserved
+      end
+
+      def add(element)
+        execute_redis do |client|
+          client.call(["BF.ADD", @filter_key, element.to_s]) == 1
+        end
+      end
+
+      def contains?(element)
+        execute_redis do |client|
+          client.call(["BF.EXISTS", @filter_key, element.to_s]) == 1
+        end
+      end
+
+      def clear
+        execute_redis do |client|
+          client.del(@filter_key)
+        end
+        ensure_filter_reserved
+        true
+      end
+
+      def count
+        info = stats
+        info[:inserted_items] || 0
+      end
+
+      def capacity
+        @expected_elements
+      end
+
+      def stats
+        execute_redis do |client|
+          raw_info = client.call(["BF.INFO", @filter_key]) rescue []
+          info_hash = Hash[*raw_info] rescue {}
+          {
+            backend: :redis_bloom,
+            filter_key: @filter_key,
+            expected_elements: @expected_elements,
+            false_positive_rate: @false_positive_rate,
+            inserted_items: info_hash["Number of items inserted"] || 0,
+            capacity: info_hash["Capacity"] || @expected_elements,
+            bytes: info_hash["Size"] || 0
+          }
+        end
+      end
+
+      private
+
+      def ensure_filter_reserved
+        execute_redis do |client|
+          client.call(["BF.RESERVE", @filter_key, @false_positive_rate, @expected_elements])
+        end
+      rescue => e
+        # Ignore if filter already exists
+      end
+
+      def execute_redis
+        if @redis.respond_to?(:with)
+          @redis.with { |conn| yield conn }
+        elsif @redis
+          yield @redis
+        elsif defined?(::Redis) && ::Redis.respond_to?(:current)
+          yield ::Redis.current
+        else
+          raise BackendError, "Redis client not configured. Set config.redis = Redis.new"
+        end
+      end
+    end
+  end
+end

data/lib/fast_exists/backends/registry.rb

@@ -0,0 +1,38 @@
+# frozen_string_literal: true
+
+module FastExists
+  module Backends
+    class Registry
+      def initialize
+        @backends = {}
+        register_default_backends
+      end
+
+      def register(name, klass)
+        @backends[name.to_sym] = klass
+      end
+
+      def fetch(name, options = {})
+        backend_name = name.to_sym
+        klass = @backends[backend_name]
+        raise UnsupportedBackendError, "Backend '#{name}' is not registered" unless klass
+
+        klass.new(options)
+      end
+
+      def registered?(name)
+        @backends.key?(name.to_sym)
+      end
+
+      private
+
+      def register_default_backends
+        register(:memory, FastExists::Backends::Memory)
+        register(:file, FastExists::Backends::File)
+        register(:null, FastExists::Backends::Null)
+        register(:redis, FastExists::Backends::Redis)
+        register(:redis_bloom, FastExists::Backends::RedisBloom)
+      end
+    end
+  end
+end