connection_pool 2.5.5 → 3.0.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 24c74a1caa5e04827c47155b75e19805bcda4c329e28793309dfa95b5881c4bd
-  data.tar.gz: 23aadf2da494be8c3314039700606cd4e01b58d6cae1f45e3b7cdef57a98e7bd
+  metadata.gz: 60f53f4a83f6a13be41059753aeff5fad1518a45946aa13ab29111f1612c5119
+  data.tar.gz: 7edefa375ea7f3852eee9d7644f2aae955119e0cd96049697b7f38bfc54e5c87
 SHA512:
-  metadata.gz: 0f2385ddf4619ebc1bf9e2f202024b330c170ae2e4e7e63480be0f556242140af75eb3d272f6d262b2fc859625f0861c560af6a58c20bea3d69b005e2d248472
-  data.tar.gz: 0ac5103c69aa2e8226d2fa872714f9bd611b24ea171e91ce3d7a3c9be62b9887e24916cf4b6b1e5bc8c407349f87c3ea855c4659dbf5f3f137cd45738301d1a6
+  metadata.gz: 3007eb8a2cf804da6b3850162098b221a5a613244ba20506fed109e3a47e0046099ba56c48ccd8a260a3afc13362552ddd9040a512ca1b512cfae1f18937c32b
+  data.tar.gz: 4cdb681918cefe624b3c00e6d7580330f120c6c53cc414e694d6568eee7f24ed109d69ec150c7a7ccbf5dec17ffb1cfdc8422e78bcb9d1b436240bb81c5464a6

data/Changes.md

@@ -1,5 +1,19 @@
 # connection_pool Changelog
 
+3.0.0
+------
+
+- **BREAKING CHANGES** `ConnectionPool` and `ConnectionPool::TimedStack` now
+  use keyword arguments rather than positional arguments everywhere. Expected impact is minimal as most people use the `with` API, which is unchanged.
+```ruby
+pool = ConnectionPool.new(size: 5, timeout: 5)
+pool.checkout(1) # 2.x
+pool.reap(30)    # 2.x
+pool.checkout(timeout: 1) # 3.x
+pool.reap(idle_seconds: 30) # 3.x
+```
+- Dropped support for Ruby <3.2.0
+
 2.5.5
 ------
 

data/README.md

@@ -38,7 +38,7 @@ connection pool and a raw client.
 $redis.then { |r| r.set 'foo' 'bar' }
 ```
 
-Optionally, you can specify a timeout override using the with-block semantics:
+Optionally, you can specify a timeout override:
 
 ``` ruby
 $memcached.with(timeout: 2.0) do |conn|
@@ -46,11 +46,9 @@ $memcached.with(timeout: 2.0) do |conn|
 end
 ```
 
-This will only modify the resource-get timeout for this particular
-invocation.
+This will only modify the timeout for this particular invocation.
 This is useful if you want to fail-fast on certain non-critical
 sections when a resource is not available, or conversely if you are comfortable blocking longer on a particular resource.
-This is not implemented in the `ConnectionPool::Wrapper` class.
 
 ## Migrating to a Connection Pool
 
@@ -72,7 +70,7 @@ $redis.with do |conn|
 end
 ```
 
-Once you've ported your entire system to use `with`, you can simply remove `Wrapper` and use the simpler and faster `ConnectionPool`.
+Once you've ported your entire system to use `with`, you can remove `::Wrapper` and use `ConnectionPool` directly.
 
 
 ## Shutdown
@@ -90,64 +88,52 @@ Shutting down a connection pool will block until all connections are checked in
 
 ## Reload
 
-You can reload a ConnectionPool instance in the case it is desired to close all connections to the pool and, unlike `shutdown`, afterwards recreate connections so the pool may continue to be used.
-Reloading may be useful after forking the process.
+You can reload a ConnectionPool instance if it is necessary to close all existing connections and continue to use the pool.
+ConnectionPool will automatically reload if the process is forked.
+Use `auto_reload_after_fork: false` if you don't want this behavior.
 
 ```ruby
-cp = ConnectionPool.new { Redis.new }
-cp.reload { |conn| conn.quit }
+cp = ConnectionPool.new(auto_reload_after_fork: false) { Redis.new }
+cp.reload { |conn| conn.quit } # reload manually
 cp.with { |conn| conn.get('some-count') }
 ```
 
-Like `shutdown`, this will block until all connections are checked in and closed.
+Like `shutdown`, `reload` will block until all connections are checked in and closed.
 
 ## Reap
 
-You can reap idle connections in the ConnectionPool instance to close connections that were created but have not been used for a certain amount of time. This can be useful to run periodically in a separate thread especially if keeping the connection open is resource intensive.
+You can call `reap` periodically on the ConnectionPool instance to close connections that were created but have not been used for a certain amount of time. This can be useful in environments where connections are expensive.
 
-You can specify how many seconds the connections have to be idle for them to be reaped.
-Defaults to 60 seconds.
+You can specify how many seconds the connections have to be idle for them to be reaped, defaulting to 60 seconds.
 
 ```ruby
 cp = ConnectionPool.new { Redis.new }
-cp.reap(300) { |conn| conn.close } # Reaps connections that have been idle for 300 seconds (5 minutes).
-```
-
-### Reaper Thread
 
-You can start your own reaper thread to reap idle connections in the ConnectionPool instance on a regular interval.
-
-```ruby
-cp = ConnectionPool.new { Redis.new }
-
-# Start a reaper thread to reap connections that have been idle for 300 seconds (5 minutes).
+# Start a reaper thread to reap connections that have been
+# idle more than 300 seconds (5 minutes)
 Thread.new do
   loop do
-    cp.reap(300) { |conn| conn.close }
-    sleep 300
+    cp.reap(idle_seconds: 300, &:close)
+    sleep 30
   end
 end
 ```
 
 ## Discarding Connections
 
-You can discard connections in the ConnectionPool instance to remove connections that are broken and can't be restarted. 
-
-NOTE: the connection is not closed. It will just be removed from the pool so it won't be selected again.
-
-It can only be done inside the block passed to `with` or `with_timeout`.
-
+You can discard connections in the ConnectionPool instance to remove connections that are broken and can't be repaired. 
+It can only be done inside the block passed to `with`.
 Takes an optional block that will be executed with the connection.
 
 ```ruby
-   pool.with do |conn|
-     begin
-       conn.execute("SELECT 1")
-     rescue SomeConnectionError
-       pool.discard_current_connection  # remove the connection from the pool
-       raise
-     end
-   end
+pool.with do |conn|
+  begin
+    conn.execute("SELECT 1")
+  rescue SomeConnectionError
+    pool.discard_current_connection(&:close)  # remove the connection from the pool
+    raise
+  end
+end
 ```
 
 ## Current State
@@ -169,20 +155,28 @@ end
 cp.idle # => 1
 ```
 
-Notes
------
+## Upgrading from ConnectionPool 2
+
+* Support for Ruby <3.2 has been removed.
+* ConnectionPool's APIs now consistently use keyword arguments everywhere.
+Positional arguments must be converted to keywords:
+```ruby
+pool = ConnectionPool.new(size: 5, timeout: 5)
+pool.checkout(1) # 2.x
+pool.reap(30)    # 2.x
+pool.checkout(timeout: 1) # 3.x
+pool.reap(idle_seconds: 30) # 3.x
+```
+
+## Notes
 
 - Connections are lazily created as needed.
-- There is no provision for repairing or checking the health of a connection;
-  connections should be self-repairing. This is true of the Dalli and Redis
-  clients.
-- **WARNING**: Don't ever use `Timeout.timeout` in your Ruby code or you will see
+- **WARNING**: Avoid `Timeout.timeout` in your Ruby code or you can see
   occasional silent corruption and mysterious errors. The Timeout API is unsafe
-  and cannot be used correctly, ever. Use proper socket timeout options as
-  exposed by Net::HTTP, Redis, Dalli, etc.
+  and dangerous to use. Use proper socket timeout options as exposed by
+  Net::HTTP, Redis, Dalli, etc.
 
 
-Author
-------
+## Author
 
-Mike Perham, [@getajobmike](https://twitter.com/getajobmike), <https://www.mikeperham.com>
+Mike Perham, [@getajobmike](https://ruby.social/@getajobmike), <https://www.mikeperham.com>

data/connection_pool.gemspec

@@ -15,10 +15,18 @@ Gem::Specification.new do |s|
   s.executables = []
   s.require_paths = ["lib"]
   s.license = "MIT"
+
+  s.required_ruby_version = ">= 3.2.0"
   s.add_development_dependency "bundler"
-  s.add_development_dependency "minitest", ">= 5.0.0"
+  s.add_development_dependency "maxitest"
   s.add_development_dependency "rake"
-  s.required_ruby_version = ">= 2.5.0"
 
-  s.metadata = {"changelog_uri" => "https://github.com/mperham/connection_pool/blob/main/Changes.md", "rubygems_mfa_required" => "true"}
+  s.metadata = {
+    "bug_tracker_uri" => "https://github.com/mperham/connection_pool/issues",
+    "documentation_uri" => "https://github.com/mperham/connection_pool/wiki",
+    "changelog_uri" => "https://github.com/mperham/connection_pool/blob/main/Changes.md",
+    "source_code_uri" => "https://github.com/mperham/connection_pool",
+    "homepage_uri" => "https://github.com/mperham/connection_pool",
+    "rubygems_mfa_required" => "true"
+  }
 end

data/lib/connection_pool/timed_stack.rb

@@ -5,7 +5,7 @@
 #
 # Examples:
 #
-#    ts = TimedStack.new(1) { MyConnection.new }
+#    ts = TimedStack.new(size: 1) { MyConnection.new }
 #
 #    # fetch a connection
 #    conn = ts.pop
@@ -22,7 +22,7 @@ class ConnectionPool::TimedStack
   ##
   # Creates a new pool with +size+ connections that are created from the given
   # +block+.
-  def initialize(size = 0, &block)
+  def initialize(size: 0, &block)
     @create_block = block
     @created = 0
     @que = []
@@ -33,15 +33,15 @@ class ConnectionPool::TimedStack
   end
 
   ##
-  # Returns +obj+ to the stack. +options+ is ignored in TimedStack but may be
+  # Returns +obj+ to the stack. Additional kwargs are ignored in TimedStack but may be
   # used by subclasses that extend TimedStack.
-  def push(obj, options = {})
+  def push(obj, **)
     @mutex.synchronize do
       if @shutdown_block
         @created -= 1 unless @created == 0
         @shutdown_block.call(obj)
       else
-        store_connection obj, options
+        store_connection obj, **
       end
 
       @resource.broadcast
@@ -58,28 +58,23 @@ class ConnectionPool::TimedStack
   # @option options [Class] :exception (ConnectionPool::TimeoutError) Exception class to raise
   #   if an entry was not available within the timeout period. Use `exception: false` to return nil.
   #
-  # The +timeout+ argument will be removed in 3.0.
   # Other options may be used by subclasses that extend TimedStack.
-  def pop(timeout = 0.5, options = {})
-    options, timeout = timeout, 0.5 if Hash === timeout
-    timeout = options.fetch :timeout, timeout
-
+  def pop(timeout: 0.5, exception: ConnectionPool::TimeoutError, **)
     deadline = current_time + timeout
     @mutex.synchronize do
       loop do
         raise ConnectionPool::PoolShuttingDownError if @shutdown_block
-        if (conn = try_fetch_connection(options))
+        if (conn = try_fetch_connection(**))
           return conn
         end
 
-        connection = try_create(options)
+        connection = try_create(**)
         return connection if connection
 
         to_wait = deadline - current_time
         if to_wait <= 0
-          exc = options.fetch(:exception, ConnectionPool::TimeoutError)
-          if exc
-            raise ConnectionPool::TimeoutError, "Waited #{timeout} sec, #{length}/#{@max} available"
+          if exception
+            raise exception, "Waited #{timeout} sec, #{length}/#{@max} available"
           else
             return nil
           end
@@ -108,21 +103,20 @@ class ConnectionPool::TimedStack
 
   ##
   # Reaps connections that were checked in more than +idle_seconds+ ago.
-  def reap(idle_seconds, &block)
-    raise ArgumentError, "reap must receive a block" unless block
+  def reap(idle_seconds:)
+    raise ArgumentError, "reap must receive a block" unless block_given?
     raise ArgumentError, "idle_seconds must be a number" unless idle_seconds.is_a?(Numeric)
     raise ConnectionPool::PoolShuttingDownError if @shutdown_block
 
-    idle.times do
-      conn =
-        @mutex.synchronize do
-          raise ConnectionPool::PoolShuttingDownError if @shutdown_block
-
-          reserve_idle_connection(idle_seconds)
-        end
+    count = idle
+    count.times do
+      conn = @mutex.synchronize do
+        raise ConnectionPool::PoolShuttingDownError if @shutdown_block
+        reserve_idle_connection(idle_seconds)
+      end
       break unless conn
 
-      block.call(conn)
+      yield conn
     end
   end
 
@@ -162,15 +156,15 @@ class ConnectionPool::TimedStack
   # This method must returns a connection from the stack if one exists. Allows
   # subclasses with expensive match/search algorithms to avoid double-handling
   # their stack.
-  def try_fetch_connection(options = nil)
-    connection_stored?(options) && fetch_connection(options)
+  def try_fetch_connection(**)
+    connection_stored?(**) && fetch_connection(**)
   end
 
   ##
   # This is an extension point for TimedStack and is called with a mutex.
   #
   # This method must returns true if a connection is available on the stack.
-  def connection_stored?(options = nil)
+  def connection_stored?(**)
     !@que.empty?
   end
 
@@ -178,7 +172,7 @@ class ConnectionPool::TimedStack
   # This is an extension point for TimedStack and is called with a mutex.
   #
   # This method must return a connection from the stack.
-  def fetch_connection(options = nil)
+  def fetch_connection(**)
     @que.pop&.first
   end
 
@@ -186,8 +180,8 @@ class ConnectionPool::TimedStack
   # This is an extension point for TimedStack and is called with a mutex.
   #
   # This method must shut down all connections on the stack.
-  def shutdown_connections(options = nil)
-    while (conn = try_fetch_connection(options))
+  def shutdown_connections(**)
+    while (conn = try_fetch_connection(**))
       @created -= 1 unless @created == 0
       @shutdown_block.call(conn)
     end
@@ -203,6 +197,8 @@ class ConnectionPool::TimedStack
 
     @created -= 1 unless @created == 0
 
+    # Most active elements are at the tail of the array.
+    # Most idle will be at the head so `shift` rather than `pop`.
     @que.shift.first
   end
 
@@ -211,14 +207,17 @@ class ConnectionPool::TimedStack
   #
   # Returns true if the first connection in the stack has been idle for more than idle_seconds
   def idle_connections?(idle_seconds)
-    connection_stored? && (current_time - @que.first.last > idle_seconds)
+    return unless connection_stored?
+    # Most idle will be at the head so `first`
+    age = (current_time - @que.first.last)
+    age > idle_seconds
   end
 
   ##
   # This is an extension point for TimedStack and is called with a mutex.
   #
   # This method must return +obj+ to the stack.
-  def store_connection(obj, options = nil)
+  def store_connection(obj, **)
     @que.push [obj, current_time]
   end
 
@@ -227,7 +226,7 @@ class ConnectionPool::TimedStack
   #
   # This method must create a connection if and only if the total number of
   # connections allowed has not been met.
-  def try_create(options = nil)
+  def try_create(**)
     unless @created == @max
       object = @create_block.call
       @created += 1

data/lib/connection_pool/version.rb

@@ -1,3 +1,3 @@
 class ConnectionPool
-  VERSION = "2.5.5"
+  VERSION = "3.0.0"
 end

data/lib/connection_pool/wrapper.rb

@@ -2,20 +2,20 @@ class ConnectionPool
   class Wrapper < ::BasicObject
     METHODS = [:with, :pool_shutdown, :wrapped_pool]
 
-    def initialize(options = {}, &block)
-      @pool = options.fetch(:pool) { ::ConnectionPool.new(options, &block) }
+    def initialize(**options, &)
+      @pool = options.fetch(:pool) { ::ConnectionPool.new(**options, &) }
     end
 
     def wrapped_pool
       @pool
     end
 
-    def with(&block)
-      @pool.with(&block)
+    def with(**, &)
+      @pool.with(**, &)
     end
 
-    def pool_shutdown(&block)
-      @pool.shutdown(&block)
+    def pool_shutdown(&)
+      @pool.shutdown(&)
     end
 
     def pool_size
@@ -26,31 +26,18 @@ class ConnectionPool
       @pool.available
     end
 
-    def respond_to?(id, *args)
-      METHODS.include?(id) || with { |c| c.respond_to?(id, *args) }
+    def respond_to?(id, *, **)
+      METHODS.include?(id) || with { |c| c.respond_to?(id, *, **) }
     end
 
-    # rubocop:disable Style/MissingRespondToMissing
-    if ::RUBY_VERSION >= "3.0.0"
-      def method_missing(name, *args, **kwargs, &block)
-        with do |connection|
-          connection.send(name, *args, **kwargs, &block)
-        end
-      end
-    elsif ::RUBY_VERSION >= "2.7.0"
-      ruby2_keywords def method_missing(name, *args, &block)
-        with do |connection|
-          connection.send(name, *args, &block)
-        end
-      end
-    else
-      def method_missing(name, *args, &block)
-        with do |connection|
-          connection.send(name, *args, &block)
-        end
+    def respond_to_missing?(id, *, **)
+      with { |c| c.respond_to?(id, *, **) }
+    end
+
+    def method_missing(name, *, **, &)
+      with do |connection|
+        connection.send(name, *, **, &)
       end
     end
-    # rubocop:enable Style/MethodMissingSuper
-    # rubocop:enable Style/MissingRespondToMissing
   end
 end

data/lib/connection_pool.rb

@@ -39,76 +39,30 @@ end
 # - :auto_reload_after_fork - automatically drop all connections after fork, defaults to true
 #
 class ConnectionPool
-  DEFAULTS = {size: 5, timeout: 5, auto_reload_after_fork: true}.freeze
-
-  def self.wrap(options, &block)
-    Wrapper.new(options, &block)
+  def self.wrap(**, &)
+    Wrapper.new(**, &)
   end
 
-  if Process.respond_to?(:fork)
-    INSTANCES = ObjectSpace::WeakMap.new
-    private_constant :INSTANCES
-
-    def self.after_fork
-      INSTANCES.values.each do |pool|
-        next unless pool.auto_reload_after_fork
-
-        # We're on after fork, so we know all other threads are dead.
-        # All we need to do is to ensure the main thread doesn't have a
-        # checked out connection
-        pool.checkin(force: true)
-        pool.reload do |connection|
-          # Unfortunately we don't know what method to call to close the connection,
-          # so we try the most common one.
-          connection.close if connection.respond_to?(:close)
-        end
-      end
-      nil
-    end
-
-    if ::Process.respond_to?(:_fork) # MRI 3.1+
-      module ForkTracker
-        def _fork
-          pid = super
-          if pid == 0
-            ConnectionPool.after_fork
-          end
-          pid
-        end
-      end
-      Process.singleton_class.prepend(ForkTracker)
-    end
-  else
-    INSTANCES = nil
-    private_constant :INSTANCES
-
-    def self.after_fork
-      # noop
-    end
-  end
-
-  def initialize(options = {}, &block)
-    raise ArgumentError, "Connection pool requires a block" unless block
-
-    options = DEFAULTS.merge(options)
+  attr_reader :size
 
-    @size = Integer(options.fetch(:size))
-    @timeout = options.fetch(:timeout)
-    @auto_reload_after_fork = options.fetch(:auto_reload_after_fork)
+  def initialize(timeout: 5, size: 5, auto_reload_after_fork: true, &)
+    raise ArgumentError, "Connection pool requires a block" unless block_given?
 
-    @available = TimedStack.new(@size, &block)
+    @size = Integer(size)
+    @timeout = Float(timeout)
+    @available = TimedStack.new(size: @size, &)
     @key = :"pool-#{@available.object_id}"
     @key_count = :"pool-#{@available.object_id}-count"
     @discard_key = :"pool-#{@available.object_id}-discard"
-    INSTANCES[self] = self if @auto_reload_after_fork && INSTANCES
+    INSTANCES[self] = self if auto_reload_after_fork && INSTANCES
   end
 
-  def with(options = {})
+  def with(**)
     # We need to manage exception handling manually here in order
     # to work correctly with `Timeout.timeout` and `Thread#raise`.
     # Otherwise an interrupted Thread can leak connections.
     Thread.handle_interrupt(Exception => :never) do
-      conn = checkout(options)
+      conn = checkout(**)
       begin
         Thread.handle_interrupt(Exception => :immediate) do
           yield conn
@@ -154,13 +108,15 @@ class ConnectionPool
     ::Thread.current[@discard_key] = block || proc { |conn| conn }
   end
 
-  def checkout(options = {})
+  def checkout(timeout: @timeout, **)
     if ::Thread.current[@key]
       ::Thread.current[@key_count] += 1
       ::Thread.current[@key]
     else
+      conn = @available.pop(timeout:, **)
+      ::Thread.current[@key] = conn
       ::Thread.current[@key_count] = 1
-      ::Thread.current[@key] = @available.pop(options[:timeout] || @timeout, options)
+      conn
     end
   end
 
@@ -195,29 +151,24 @@ class ConnectionPool
   # Shuts down the ConnectionPool by passing each connection to +block+ and
   # then removing it from the pool. Attempting to checkout a connection after
   # shutdown will raise +ConnectionPool::PoolShuttingDownError+.
-  def shutdown(&block)
-    @available.shutdown(&block)
+  def shutdown(&)
+    @available.shutdown(&)
   end
 
   ##
   # Reloads the ConnectionPool by passing each connection to +block+ and then
   # removing it the pool. Subsequent checkouts will create new connections as
   # needed.
-  def reload(&block)
-    @available.shutdown(reload: true, &block)
+  def reload(&)
+    @available.shutdown(reload: true, &)
   end
 
   ## Reaps idle connections that have been idle for over +idle_seconds+.
   # +idle_seconds+ defaults to 60.
-  def reap(idle_seconds = 60, &block)
-    @available.reap(idle_seconds, &block)
+  def reap(idle_seconds: 60, &)
+    @available.reap(idle_seconds:, &)
   end
 
-  # Size of this connection pool
-  attr_reader :size
-  # Automatically drop all connections after fork
-  attr_reader :auto_reload_after_fork
-
   # Number of pool entries available for checkout at this instant.
   def available
     @available.length
@@ -231,3 +182,4 @@ end
 
 require_relative "connection_pool/timed_stack"
 require_relative "connection_pool/wrapper"
+require_relative "connection_pool/fork"

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: connection_pool
 version: !ruby/object:Gem::Version
-  version: 2.5.5
+  version: 3.0.0
 platform: ruby
 authors:
 - Mike Perham
@@ -25,19 +25,19 @@ dependencies:
       - !ruby/object:Gem::Version
         version: '0'
 - !ruby/object:Gem::Dependency
-  name: minitest
+  name: maxitest
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - ">="
       - !ruby/object:Gem::Version
-        version: 5.0.0
+        version: '0'
   type: :development
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - ">="
       - !ruby/object:Gem::Version
-        version: 5.0.0
+        version: '0'
 - !ruby/object:Gem::Dependency
   name: rake
   requirement: !ruby/object:Gem::Requirement
@@ -72,7 +72,11 @@ homepage: https://github.com/mperham/connection_pool
 licenses:
 - MIT
 metadata:
+  bug_tracker_uri: https://github.com/mperham/connection_pool/issues
+  documentation_uri: https://github.com/mperham/connection_pool/wiki
   changelog_uri: https://github.com/mperham/connection_pool/blob/main/Changes.md
+  source_code_uri: https://github.com/mperham/connection_pool
+  homepage_uri: https://github.com/mperham/connection_pool
   rubygems_mfa_required: 'true'
 rdoc_options: []
 require_paths:
@@ -81,7 +85,7 @@ required_ruby_version: !ruby/object:Gem::Requirement
   requirements:
   - - ">="
     - !ruby/object:Gem::Version
-      version: 2.5.0
+      version: 3.2.0
 required_rubygems_version: !ruby/object:Gem::Requirement
   requirements:
   - - ">="