ros-apartment 4.0.0.alpha2 → 4.0.0.alpha3

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: efc3d3a91e17846826ac5d5ca707cd17bdc5a5923b18fb4c0162ac87f24014d0
-  data.tar.gz: 5c1787f4e9cee805842e6035fcc2769415dfccdf90c3c2b354315271315638a2
+  metadata.gz: 0b2254c1f3f95622ba9fa45147fcc2f5b2036db4c5c75a2d68c3718772df6149
+  data.tar.gz: efcc9149b0aaaa6c7b614cc981848f02e999259eaa0826120b4df16b555f6caf
 SHA512:
-  metadata.gz: 486bb0cd94225925ada99a497c89e314536300a261b898c3575a5008211e1e8f1632a9755dadae096f5814eaee00920e9652f70f8feb8676ff58897fc4262f2c
-  data.tar.gz: e9b88d3c86be4e0891ff584e85c822979c6146ece01ce2bab17f442603e8add4aef3e826157157df3325722e39209a70bf94ffdd218504abdbf6be7dd4af29cf
+  metadata.gz: 35b3bdeb392ed750be9a20868082f1708e2ae3c4c8449cef5bda163d3e2e911409274abf9a1f8161811678192c8aaaa21c2f4d6a177a5932a9567942172bc234
+  data.tar.gz: 3d0909cc593dcbd014f21626a2565e0e92d373cb74aa768f308b1033f689fbdea1aa48a2f886294e581d4d3954b1d719b15ab3277990a8fde216449eed40485f

data/README.md

@@ -99,11 +99,15 @@ All options are set in `config/initializers/apartment.rb` inside an `Apartment.c
 
 ### Pool Settings
 
-`tenant_pool_size`: connections per tenant pool (default: 5).
+`tenant_pool_size`: max connections per tenant pool. Default `nil` — each tenant pool inherits the app's base pool size (`DB_POOL_SIZE` / `pool:` in `database.yml`). Set it to size tenant pools independently of the app pool (e.g. to bound total connections across many schema-per-tenant pools).
 
-`pool_idle_timeout`: seconds before an idle tenant pool is eligible for reaping (default: 300).
+`pool_idle_timeout`: seconds an idle tenant pool must exceed before it is eligible for reaping (default: 300).
 
-`max_total_connections`: hard cap across all tenant pools; nil for unlimited (default: nil).
+`reaper_interval`: seconds between background reap passes. Default `nil` — derives from `pool_idle_timeout`. Set it lower to reap more often without shrinking the idle window.
+
+`max_total_connections`: ceiling on the number of live tenant pools; `nil` for unlimited (default: `nil`). Enforced synchronously at pool-creation time (see `pool_overflow_policy`) and trimmed continuously by the background reaper. Total backend connections ≈ `max_total_connections × tenant_pool_size`.
+
+`pool_overflow_policy`: behavior when a new pool would breach `max_total_connections` and every existing pool is pinned or in use (no idle pool to evict). `:evict_idle` (default) — allow the new pool, emit a `cap_unmet` notification (soft cap, prioritizes availability). `:raise` — raise `Apartment::PoolCapacityReached` (hard cap, sheds load). When an idle pool *is* available it is always evicted inline regardless of policy. See `docs/designs/pool-admission-control.md`.
 
 ### Elevator (Request Tenant Detection)
 

data/lib/apartment/CLAUDE.md

@@ -59,11 +59,11 @@ lib/apartment/
 
 ### pool_manager.rb — Pool Cache
 
-`Concurrent::Map` storing connection pools by tenant key. Monotonic clock timestamps for idle/LRU tracking. `stats_for` returns `{ seconds_idle: N }`. `clear` disconnects all pools before clearing.
+`Concurrent::Map` storing connection pools by tenant key. Monotonic clock timestamps for idle/LRU tracking. `stats_for` returns `{ seconds_idle: N }`. `clear` disconnects all pools before clearing. When `max_total_connections` is set, `Apartment.configure` wires an `admission_controller` (the reaper) so cold creates route through a serialized, capacity-bounded path; otherwise the lock-free `compute_if_absent` fast path is used. See `docs/designs/pool-admission-control.md`.
 
-### pool_reaper.rb — Pool Eviction
+### pool_reaper.rb — Pool Eviction + Admission
 
-Background `Concurrent::TimerTask` instance that evicts idle and excess tenant pools. Created by `Apartment.configure`, stored as `Apartment.pool_reaper`. Deregisters evicted pools from AR's ConnectionHandler. Default tenant is never evicted.
+Background `Concurrent::TimerTask` that evicts idle and excess tenant pools, and also serves as the pool manager's synchronous admission controller (`admit!`) when a cap is configured — evicting the LRU idle pool inline before a new one is established, applying `pool_overflow_policy` (`:evict_idle` / `:raise`) when no idle pool can be freed. A single `evict_tenant` primitive backs the timer (idle/LRU) and admission paths. Reap cadence (`interval`/`reaper_interval`) is decoupled from the idle window (`idle_timeout`/`pool_idle_timeout`). Created by `Apartment.configure`, stored as `Apartment.pool_reaper`. Deregisters evicted pools from AR's ConnectionHandler. Default tenant is never evicted.
 
 ### adapters/abstract_adapter.rb — Base Adapter
 

data/lib/apartment/adapters/abstract_adapter.rb

@@ -30,7 +30,8 @@ module Apartment
           strategy: Apartment.config.tenant_strategy,
           adapter_name: effective_base['adapter']
         )
-        resolve_connection_config(tenant, base_config: effective_base)
+        config = resolve_connection_config(tenant, base_config: effective_base)
+        apply_tenant_pool_size(config)
       end
 
       # Resolve a tenant-specific connection config hash.
@@ -259,6 +260,19 @@ module Apartment
         connection_config.transform_keys(&:to_s)
       end
 
+      # Cap the tenant pool's max checkout size to Apartment.config.tenant_pool_size
+      # when configured. Defaults to nil — the tenant pool inherits the base/default
+      # pool's `pool:` (the app's DB_POOL_SIZE), preserving pre-4.0.0.alpha3 behavior.
+      # Set tenant_pool_size to size each per-tenant pool independently of the app pool
+      # (e.g. to bound total connections across many schema-per-tenant pools). The config
+      # is string-keyed here; HashConfig symbolizes it and reads `:pool` for the size.
+      def apply_tenant_pool_size(config)
+        size = Apartment.config.tenant_pool_size
+        return config unless size
+
+        config.merge('pool' => size)
+      end
+
       # Connection config for pinned models on the separate-pool path.
       # For schema strategy, pins schema_search_path to the default tenant
       # (plus persistent schemas) so the connection resolves tables and FK

data/lib/apartment/config.rb

@@ -16,7 +16,8 @@ module Apartment
 
     attr_accessor :tenants_provider, :default_tenant,
                   :default_tenant_switch_allowed,
-                  :tenant_pool_size, :pool_idle_timeout, :max_total_connections,
+                  :tenant_pool_size, :pool_idle_timeout, :reaper_interval,
+                  :max_total_connections, :pool_overflow_policy,
                   :seed_after_create, :seed_data_file,
                   :schema_load_strategy, :schema_file,
                   :parallel_migration_threads,
@@ -33,9 +34,11 @@ module Apartment
       @default_tenant = nil
       @default_tenant_switch_allowed = true
       @excluded_models = []
-      @tenant_pool_size = 5
+      @tenant_pool_size = nil
       @pool_idle_timeout = 300
+      @reaper_interval = nil
       @max_total_connections = nil
+      @pool_overflow_policy = :evict_idle
       @seed_after_create = false
       @seed_data_file = nil
       @schema_load_strategy = nil
@@ -117,6 +120,9 @@ module Apartment
     def apply_defaults!
       # PostgreSQL's default schema is 'public'; avoid forcing every user to set it.
       @default_tenant ||= 'public' if @tenant_strategy == :schema
+      # Reap on the idle-timeout cadence unless an explicit interval decouples
+      # the two (reap more often without shrinking the idle window).
+      @reaper_interval = @pool_idle_timeout if @reaper_interval.nil?
     end
 
     # Validate configuration completeness and consistency.
@@ -144,19 +150,32 @@ module Apartment
 
       @postgres_config&.validate!
 
-      unless @tenant_pool_size.is_a?(Integer) && @tenant_pool_size.positive?
-        raise(ConfigurationError, "tenant_pool_size must be a positive integer, got: #{@tenant_pool_size.inspect}")
+      if @tenant_pool_size && (!@tenant_pool_size.is_a?(Integer) || @tenant_pool_size < 1)
+        raise(ConfigurationError,
+              "tenant_pool_size must be a positive integer or nil, got: #{@tenant_pool_size.inspect}")
       end
 
       unless @pool_idle_timeout.is_a?(Numeric) && @pool_idle_timeout.positive?
         raise(ConfigurationError, "pool_idle_timeout must be a positive number, got: #{@pool_idle_timeout.inspect}")
       end
 
+      # nil is valid pre-apply_defaults! (it derives from pool_idle_timeout); a
+      # set value must be a positive number.
+      if @reaper_interval && (!@reaper_interval.is_a?(Numeric) || !@reaper_interval.positive?)
+        raise(ConfigurationError, "reaper_interval must be a positive number or nil, got: #{@reaper_interval.inspect}")
+      end
+
       if @max_total_connections && (!@max_total_connections.is_a?(Integer) || @max_total_connections < 1)
         raise(ConfigurationError,
               "max_total_connections must be a positive integer or nil, got: #{@max_total_connections.inspect}")
       end
 
+      unless %i[evict_idle raise].include?(@pool_overflow_policy)
+        raise(ConfigurationError,
+              'pool_overflow_policy must be :evict_idle or :raise, ' \
+              "got: #{@pool_overflow_policy.inspect}")
+      end
+
       unless [nil, :schema_rb, :sql].include?(@schema_load_strategy)
         raise(ConfigurationError, "Invalid schema_load_strategy: #{@schema_load_strategy.inspect}. " \
                                   'Must be nil, :schema_rb, or :sql')

data/lib/apartment/errors.rb

@@ -33,6 +33,26 @@ module Apartment
   # Raised when the tenant connection pool is exhausted.
   class PoolExhausted < ApartmentError; end
 
+  # Raised when admitting a new tenant pool would exceed max_total_connections
+  # and no idle pool can be evicted to make room, under the :raise overflow
+  # policy. Distinct from PoolExhausted (a single pool's connections) — this is
+  # the process-wide pool-count ceiling. See docs/designs/pool-admission-control.md.
+  class PoolCapacityReached < ApartmentError
+    attr_reader :max_total, :current
+
+    def initialize(max_total: nil, current: nil)
+      @max_total = max_total
+      @current = current
+      super(
+        "Tenant pool capacity reached: #{current.inspect} pools open, " \
+        "max_total_connections is #{max_total.inspect}, and no idle pool could " \
+        'be evicted to admit another (all pinned or in use). Raise ' \
+        'max_total_connections, reduce concurrent tenants, or set ' \
+        'pool_overflow_policy to :evict_idle to allow soft overflow.'
+      )
+    end
+  end
+
   # Raised when schema loading fails during tenant creation.
   class SchemaLoadError < ApartmentError; end
 

data/lib/apartment/patches/connection_handling.rb

@@ -32,6 +32,13 @@ module Apartment
         pool_key = "#{tenant}:#{role}"
 
         Apartment.pool_manager.fetch_or_create(pool_key) do
+          # RE-ENTRANCY: when max_total_connections is set, this block runs under
+          # PoolManager's @create_mutex (non-reentrant). Nothing here may resolve
+          # ActiveRecord::Base.connection_pool for the current tenant — it would
+          # re-enter fetch_or_create and self-deadlock. `super` resolves the
+          # default pool (bypasses the patch), and check_pending_migrations? /
+          # schema-cache load operate on the explicit `pool`, so all are safe.
+          # Keep it that way if you add work to this block.
           # Resolve base config from the current role's default pool when available.
           # Falls back to nil (adapter uses its own base_config) when the default pool
           # is not accessible — e.g., in worker threads during parallel migration where
@@ -63,9 +70,20 @@ module Apartment
             shard: shard_key
           )
 
-          raise(Apartment::PendingMigrationError, tenant) if check_pending_migrations?(pool)
-
-          load_tenant_schema_cache(tenant, pool) if cfg.schema_cache_per_tenant
+          # establish_connection has registered the shard in AR's ConnectionHandler.
+          # If a post-establish check raises, the pool is returned to neither the
+          # caller nor PoolManager — it would be orphaned: live in AR but invisible
+          # to the reaper and to max_total accounting (a connection leak that also
+          # undercounts the cap). Deregister it before re-raising so AR and the
+          # manager stay consistent. The next request re-establishes cleanly.
+          begin
+            raise(Apartment::PendingMigrationError, tenant) if check_pending_migrations?(pool)
+
+            load_tenant_schema_cache(tenant, pool) if cfg.schema_cache_per_tenant
+          rescue StandardError
+            Apartment.deregister_shard(pool_key)
+            raise
+          end
 
           pool
         end
@@ -80,7 +98,7 @@ module Apartment
 
       def check_pending_migrations?(pool)
         return false unless Apartment.config.check_pending_migrations
-        return false unless defined?(Rails) && Rails.env.local? # rubocop:disable Rails/UnknownEnv
+        return false unless defined?(Rails) && Rails.env.local?
         return false if Apartment::Current.migrating
 
         pool.migration_context.needs_migration?

data/lib/apartment/pool_manager.rb

@@ -4,17 +4,25 @@ require 'concurrent'
 
 module Apartment
   class PoolManager
+    # Set by Apartment.configure to the PoolReaper when max_total_connections is
+    # configured. nil (no cap) keeps the lock-free compute_if_absent fast path.
+    attr_accessor :admission_controller
+
     def initialize
       @pools = Concurrent::Map.new
       @timestamps = Concurrent::Map.new
+      @create_mutex = Mutex.new
+      @admission_controller = nil
     end
 
     # Fetch an existing pool or create one via the block.
     # Timestamp is updated after pool creation to avoid orphaned timestamps if the block raises.
+    # When an admission controller is wired (a cap is configured), cold creates
+    # go through the bounded path so the pool count cannot exceed max_total.
     def fetch_or_create(tenant_key, &)
-      pool = @pools.compute_if_absent(tenant_key, &)
-      touch(tenant_key)
-      pool
+      return fetch_or_admit(tenant_key, &) if @admission_controller
+
+      touch_and_return(tenant_key, @pools.compute_if_absent(tenant_key, &))
     end
 
     def get(tenant_key)
@@ -119,6 +127,32 @@ module Apartment
 
     private
 
+    # Capacity-bounded creation path. Serializes cold creates so the admission
+    # controller's capacity check + eviction + insert is atomic across creators;
+    # the new pool is only inserted after admit! confirms (or makes) room. The
+    # hot path (existing pool) stays lock-free in fetch_or_create. Establishing
+    # the connection under the lock is deliberate: it serializes only cold
+    # creates (once per tenant per worker) — the price of a hard count bound.
+    def fetch_or_admit(tenant_key)
+      existing = @pools[tenant_key]
+      return touch_and_return(tenant_key, existing) if existing
+
+      @create_mutex.synchronize do
+        cached = @pools[tenant_key]
+        return touch_and_return(tenant_key, cached) if cached
+
+        @admission_controller.admit!(tenant_key)
+        pool = yield
+        @pools[tenant_key] = pool
+        touch_and_return(tenant_key, pool)
+      end
+    end
+
+    def touch_and_return(tenant_key, pool)
+      touch(tenant_key)
+      pool
+    end
+
     def touch(tenant_key)
       @timestamps[tenant_key] = monotonic_now
     end

data/lib/apartment/pool_reaper.rb

@@ -7,7 +7,13 @@ module Apartment
   # Evicts idle and excess tenant pools on a background timer.
   # Complementary to ActiveRecord's ConnectionPool::Reaper which handles
   # intra-pool connection reaping — this handles inter-pool (tenant) eviction.
-  class PoolReaper
+  class PoolReaper # rubocop:disable Metrics/ClassLength
+    # Reap cadence (seconds) and the idle window (seconds) a pool must exceed
+    # before it is eligible for idle eviction. Decoupled so a deployment can
+    # reap frequently without shrinking the idle window. Exposed for
+    # introspection and wiring assertions.
+    attr_reader :interval, :idle_timeout
+
     # True when Rails' transactional-fixture machinery has pinned the pool
     # (ConnectionPool#pin_connection!, Rails 7.1+). Evicting or discarding a
     # pinned pool strands the fixture transaction; teardown then errors or
@@ -24,7 +30,8 @@ module Apartment
     end
 
     def initialize(pool_manager:, interval:, idle_timeout:, max_total: nil,
-                   default_tenant: nil, shard_key_prefix: nil, on_evict: nil)
+                   default_tenant: nil, shard_key_prefix: nil, on_evict: nil,
+                   overflow_policy: :evict_idle)
       raise(ArgumentError, 'interval must be a positive number') unless interval.is_a?(Numeric) && interval.positive?
       unless idle_timeout.is_a?(Numeric) && idle_timeout.positive?
         raise(ArgumentError, 'idle_timeout must be a positive number')
@@ -40,10 +47,29 @@ module Apartment
       @default_tenant = default_tenant
       @shard_key_prefix = shard_key_prefix
       @on_evict = on_evict
+      @overflow_policy = overflow_policy
       @mutex = Mutex.new
       @timer = nil
     end
 
+    # Synchronously enforce max_total before a new tenant pool is admitted.
+    # Called by {PoolManager#fetch_or_create} under its creation lock (so the
+    # capacity check, eviction, and insert are atomic w.r.t. other creators).
+    # Evicts LRU idle (non-protected, non-default) pools until there is room for
+    # one more; if none can be freed, applies the overflow policy. A no-op when
+    # no cap is configured. See docs/designs/pool-admission-control.md.
+    def admit!(incoming_tenant_key)
+      return unless @max_total
+
+      loop do
+        break if @pool_manager.stats[:total_pools] < @max_total
+        break unless evict_one_for_admission(incoming_tenant_key)
+      end
+      return if @pool_manager.stats[:total_pools] < @max_total
+
+      apply_overflow_policy
+    end
+
     def start
       @mutex.synchronize do
         stop_internal
@@ -98,17 +124,61 @@ module Apartment
         next if default_tenant_pool?(tenant)
         next if protected_pool?(tenant, eviction_reason: :idle)
 
-        pool = @pool_manager.remove(tenant)
-        deregister_from_ar_handler(tenant)
-        Instrumentation.instrument(:evict, tenant: tenant, reason: :idle)
-        @on_evict&.call(tenant, pool)
-        count += 1
+        count += 1 if evict_tenant(tenant, reason: :idle)
       rescue StandardError => e
         warn "[Apartment::PoolReaper] Failed to evict tenant #{tenant}: #{e.class}: #{e.message}"
       end
       count
     end
 
+    # Single eviction primitive shared by the timer paths (idle/LRU) and the
+    # synchronous admission path: drop from the manager, deregister from AR's
+    # ConnectionHandler (which disconnects the pool), instrument, and fire the
+    # on_evict hook. The +reason+ flows into the :evict event payload.
+    def evict_tenant(tenant, reason:)
+      # The timer (no lock) and admission (under @create_mutex) use different
+      # locks, so both can target the same idle tenant. The loser's remove
+      # returns nil — bail before firing a duplicate :evict event or calling
+      # on_evict with a nil pool.
+      return nil unless (pool = @pool_manager.remove(tenant))
+
+      deregister_from_ar_handler(tenant)
+      Instrumentation.instrument(:evict, tenant: tenant, reason: reason)
+      @on_evict&.call(tenant, pool)
+      pool
+    end
+
+    # Evict the single LRU evictable pool to make room for an incoming one,
+    # skipping the incoming key, the default tenant, and pinned/in-use pools.
+    # Returns the evicted tenant key, or nil if nothing is evictable.
+    def evict_one_for_admission(incoming_tenant_key)
+      @pool_manager.lru_tenants(count: @pool_manager.stats[:total_pools]).each do |tenant|
+        next if tenant == incoming_tenant_key
+        next if default_tenant_pool?(tenant)
+        next if protected_pool?(tenant, eviction_reason: :admission)
+
+        evict_tenant(tenant, reason: :admission)
+        return tenant
+      rescue StandardError => e
+        warn "[Apartment::PoolReaper] Failed to evict tenant #{tenant} for admission: #{e.class}: #{e.message}"
+      end
+      nil
+    end
+
+    # Applied when the cap can't be met by eviction (every other pool is pinned
+    # or in use). :evict_idle degrades to a soft cap — allow the new pool, surface
+    # the breach via :cap_unmet. :raise fails the admission so the caller sheds
+    # load. See docs/designs/pool-admission-control.md.
+    def apply_overflow_policy
+      current = @pool_manager.stats[:total_pools]
+      case @overflow_policy
+      when :raise
+        raise(Apartment::PoolCapacityReached.new(max_total: @max_total, current: current))
+      else
+        Instrumentation.instrument(:cap_unmet, max_total: @max_total, current: current, unevicted: 1)
+      end
+    end
+
     def evict_lru
       total = @pool_manager.stats[:total_pools]
       excess = total - @max_total
@@ -135,11 +205,7 @@ module Apartment
         next if default_tenant_pool?(tenant)
         next if protected_pool?(tenant, eviction_reason: :lru)
 
-        pool = @pool_manager.remove(tenant)
-        deregister_from_ar_handler(tenant)
-        Instrumentation.instrument(:evict, tenant: tenant, reason: :lru)
-        @on_evict&.call(tenant, pool)
-        evicted += 1
+        evicted += 1 if evict_tenant(tenant, reason: :lru)
       rescue StandardError => e
         warn "[Apartment::PoolReaper] Failed to evict tenant #{tenant}: #{e.class}: #{e.message}"
       end

data/lib/apartment/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module Apartment
-  VERSION = '4.0.0.alpha2'
+  VERSION = '4.0.0.alpha3'
 end

data/lib/apartment.rb

@@ -158,16 +158,7 @@ module Apartment # rubocop:disable Metrics/ModuleLength
       @built_in_tenant_validator&.shutdown
       @built_in_tenant_validator = nil
       @config = new_config
-      @pool_manager = PoolManager.new
-      @pool_reaper = PoolReaper.new(
-        pool_manager: @pool_manager,
-        interval: new_config.pool_idle_timeout,
-        idle_timeout: new_config.pool_idle_timeout,
-        max_total: new_config.max_total_connections,
-        default_tenant: new_config.default_tenant,
-        shard_key_prefix: new_config.shard_key_prefix
-      )
-      @pool_reaper.start
+      setup_pools!(new_config)
       @config
     end
 
@@ -263,6 +254,25 @@ module Apartment # rubocop:disable Metrics/ModuleLength
         BUILT_IN_VALIDATOR_MUTEX.synchronize { @built_in_tenant_validator ||= TenantValidator.new }
     end
 
+    # Build the pool manager + reaper for a freshly-validated config and start
+    # the background reaper. When max_total_connections is set, wire the reaper
+    # as the pool manager's admission controller so cold creates are bounded
+    # synchronously; otherwise the manager keeps its lock-free create path.
+    def setup_pools!(new_config)
+      @pool_manager = PoolManager.new
+      @pool_reaper = PoolReaper.new(
+        pool_manager: @pool_manager,
+        interval: new_config.reaper_interval,
+        idle_timeout: new_config.pool_idle_timeout,
+        max_total: new_config.max_total_connections,
+        default_tenant: new_config.default_tenant,
+        shard_key_prefix: new_config.shard_key_prefix,
+        overflow_policy: new_config.pool_overflow_policy
+      )
+      @pool_manager.admission_controller = @pool_reaper if new_config.max_total_connections
+      @pool_reaper.start
+    end
+
     # Safely tear down old state. Stops the reaper first (so it doesn't
     # evict mid-cleanup), then deregisters tenant pools from AR's
     # ConnectionHandler, then clears the pool manager.

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: ros-apartment
 version: !ruby/object:Gem::Version
-  version: 4.0.0.alpha2
+  version: 4.0.0.alpha3
 platform: ruby
 authors:
 - Ryan Brunner