connection_pool 2.1.3 → 2.2.4

checksums.yaml

@@ -1,7 +1,7 @@
 ---
-SHA1:
-  metadata.gz: ec91685df50fe18b79ef2cd2ed2fb038e322ff81
-  data.tar.gz: 99b88e1aa01811e18c2737e8e062c4726ce54666
+SHA256:
+  metadata.gz: 0af12e8d2f551b932ee863a1385ddc9a3f3140fff886cacaca25298a4c21d218
+  data.tar.gz: 16bd7b78d7b35337d35befe063ceeb2cb749dfc9e68cd373e7f76bd856f6cba3
 SHA512:
-  metadata.gz: 53d34f8ca6a7d51ce535a83e2821dd33eb15a623df3022b89775c232876c8fec28a2b33adda24248136014d99f0ebe9f09c1f49e1f0ced76435db26538e907a7
-  data.tar.gz: 660f22d0f828c789942da1da3d7c4131e008dd383a83c74e59ea03f306a59b966e879d42f9704ffa3b7de6accb6c114cfff982d3343bbe34c5e60ecc8f705ac5
+  metadata.gz: edb345021997307fe736408ad3e3cbebd2ce86fcbdc636fe2d4f03d61258ecea22285a4f91e854d43631533513d6500c9f6cf2094f52d6756a2d61ee2daba256
+  data.tar.gz: d58d7519b9d4be9bcf7b0628e2aa9d937df2d61d0d1a9423b43618685adf242810256b9decc22bca90cb3ee3eec88e8eabffb409aa9e49f28d56815f26056b81

data/.github/workflows/ci.yml

@@ -0,0 +1,21 @@
+name: CI
+
+on: [push, pull_request]
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    strategy:
+      fail-fast: false
+      matrix:
+        ruby: ["2.4", "2.5", "2.6", "2.7", "3.0", "jruby", "truffleruby"]
+    steps:
+    - uses: actions/checkout@v2
+    - uses: ruby/setup-ruby@v1
+      with:
+        ruby-version: ${{matrix.ruby}}
+        bundler-cache: true
+
+    - name: Run tests
+      timeout-minutes: 5
+      run: ${{matrix.env}} bundle exec rake

data/Changes.md

@@ -1,3 +1,38 @@
+# connection_pool Changelog
+
+2.2.4
+------
+
+- Add `reload` to close all connections, recreating them afterwards [Andrew Marshall, #140]
+- Add `then` as a way to use a pool or a bare connection with the same code path [#138]
+
+2.2.3
+------
+
+- Pool now throws `ConnectionPool::TimeoutError` on timeout. [#130]
+- Use monotonic clock present in all modern Rubies [Tero Tasanen, #109]
+- Remove code hacks necessary for JRuby 1.7
+- Expose wrapped pool from ConnectionPool::Wrapper [Thomas Lecavelier, #113]
+
+2.2.2
+------
+
+- Add pool `size` and `available` accessors for metrics and monitoring
+  purposes [#97, robholland]
+
+2.2.1
+------
+
+- Allow CP::Wrapper to use an existing pool [#87, etiennebarrie]
+- Use monotonic time for more accurate timeouts [#84, jdantonio]
+
+2.2.0
+------
+
+- Rollback `Timeout` handling introduced in 2.1.1 and 2.1.2.  It seems
+  impossible to safely work around the issue. Please never, ever use
+  `Timeout.timeout` in your code or you will see rare but mysterious bugs. [#75]
+
 2.1.3
 ------
 

data/README.md

@@ -31,6 +31,14 @@ If all the objects in the connection pool are in use, `with` will block
 until one becomes available.  If no object is available within `:timeout` seconds,
 `with` will raise a `Timeout::Error`.
 
+You can also use `ConnectionPool#then` to support _both_ a
+connection pool and a raw client (requires Ruby 2.5+).
+
+```ruby
+# Compatible with a raw Redis::Client, and ConnectionPool Redis
+$redis.then { |r| r.set 'foo' 'bar' }
+```
+
 Optionally, you can specify a timeout override using the with-block semantics:
 
 ``` ruby
@@ -45,11 +53,13 @@ sections when a resource is not available, or conversely if you are comfortable
 blocking longer on a particular resource. This is not implemented in the below
 `ConnectionPool::Wrapper` class.
 
+## Migrating to a Connection Pool
+
 You can use `ConnectionPool::Wrapper` to wrap a single global connection,
-making it easier to port your connection code over time:
+making it easier to migrate existing connection code over time:
 
 ``` ruby
-$redis = ConnectionPool::Wrapper.new(size: 5, timeout: 3) { Redis.connect }
+$redis = ConnectionPool::Wrapper.new(size: 5, timeout: 3) { Redis.new }
 $redis.sadd('foo', 1)
 $redis.smembers('foo')
 ```
@@ -69,38 +79,67 @@ end
 Once you've ported your entire system to use `with`, you can simply remove
 `Wrapper` and use the simpler and faster `ConnectionPool`.
 
+
+## Shutdown
+
 You can shut down a ConnectionPool instance once it should no longer be used.
 Further checkout attempts will immediately raise an error but existing checkouts
 will work.
 
 ```ruby
 cp = ConnectionPool.new { Redis.new }
-cp.shutdown { |conn| conn.close }
+cp.shutdown { |conn| conn.quit }
 ```
 
 Shutting down a connection pool will block until all connections are checked in and closed.
-Note that shutting down is completely optional; Ruby's garbage collector will reclaim
+**Note that shutting down is completely optional**; Ruby's garbage collector will reclaim
 unreferenced pools under normal circumstances.
 
+## Reload
+
+You can reload a ConnectionPool instance in the case it is desired to close all
+connections to the pool and, unlike `shutdown`, afterwards recreate connections
+so the pool may continue to be used. Reloading may be useful after forking the
+process.
+
+```ruby
+cp = ConnectionPool.new { Redis.new }
+cp.reload { |conn| conn.quit }
+cp.with { |conn| conn.get('some-count') }
+```
+
+Like `shutdown`, this will block until all connections are checked in and
+closed.
+
+## Current State
+
+There are several methods that return information about a pool.
+
+```ruby
+cp = ConnectionPool.new(size: 10) { Redis.new }
+cp.size # => 10
+cp.available # => 10
+
+cp.with do |conn|
+  cp.size # => 10
+  cp.available # => 9
+end
+```
 
 Notes
 -----
 
 - Connections are lazily created as needed.
 - There is no provision for repairing or checking the health of a connection;
-  connections should be self-repairing.  This is true of the Dalli and Redis
+  connections should be self-repairing. This is true of the Dalli and Redis
   clients.
-
-
-Install
--------
-
-```
-$ gem install connection_pool
-```
+- **WARNING**: Don't ever use `Timeout.timeout` in your Ruby code or you will see
+  occasional silent corruption and mysterious errors. The Timeout API is unsafe
+  and cannot be used correctly, ever. Use proper socket timeout options as
+  exposed by Net::HTTP, Redis, Dalli, etc.
 
 
 Author
 ------
 
-Mike Perham, [@mperham](https://twitter.com/mperham), <http://mikeperham.com>
+Mike Perham, [@getajobmike](https://twitter.com/getajobmike), <https://www.mikeperham.com>

data/Rakefile

@@ -1,6 +1,6 @@
-require 'bundler/gem_tasks'
+require "bundler/gem_tasks"
 
-require 'rake/testtask'
+require "rake/testtask"
 Rake::TestTask.new
 
-task :default => :test
+task default: :test

data/connection_pool.gemspec

@@ -1,21 +1,21 @@
-# -*- encoding: utf-8 -*-
 require "./lib/connection_pool/version"
 
 Gem::Specification.new do |s|
-  s.name        = "connection_pool"
-  s.version     = ConnectionPool::VERSION
-  s.platform    = Gem::Platform::RUBY
-  s.authors     = ["Mike Perham", "Damian Janowski"]
-  s.email       = ["mperham@gmail.com", "damian@educabilia.com"]
-  s.homepage    = "https://github.com/mperham/connection_pool"
-  s.description = s.summary = %q{Generic connection pool for Ruby}
+  s.name = "connection_pool"
+  s.version = ConnectionPool::VERSION
+  s.platform = Gem::Platform::RUBY
+  s.authors = ["Mike Perham", "Damian Janowski"]
+  s.email = ["mperham@gmail.com", "damian@educabilia.com"]
+  s.homepage = "https://github.com/mperham/connection_pool"
+  s.description = s.summary = "Generic connection pool for Ruby"
 
-  s.files         = `git ls-files`.split("\n")
-  s.test_files    = `git ls-files -- {test,spec,features}/*`.split("\n")
-  s.executables   = `git ls-files -- bin/*`.split("\n").map{ |f| File.basename(f) }
+  s.files = `git ls-files`.split("\n")
+  s.test_files = `git ls-files -- {test,spec,features}/*`.split("\n")
+  s.executables = `git ls-files -- bin/*`.split("\n").map { |f| File.basename(f) }
   s.require_paths = ["lib"]
   s.license = "MIT"
-  s.add_development_dependency 'bundler'
-  s.add_development_dependency 'minitest', '>= 5.0.0'
-  s.add_development_dependency 'rake'
+  s.add_development_dependency "bundler"
+  s.add_development_dependency "minitest", ">= 5.0.0"
+  s.add_development_dependency "rake"
+  s.required_ruby_version = ">= 2.2.0"
 end

data/lib/connection_pool.rb

@@ -1,20 +1,25 @@
-require_relative 'connection_pool/version'
-require_relative 'connection_pool/timed_stack'
+require "timeout"
+require "connection_pool/version"
 
-# Generic connection pool class for e.g. sharing a limited number of network connections
-# among many threads.  Note: Connections are lazily created.
+class ConnectionPool
+  class Error < ::RuntimeError; end
+  class PoolShuttingDownError < ::ConnectionPool::Error; end
+  class TimeoutError < ::Timeout::Error; end
+end
+
+# Generic connection pool class for sharing a limited number of objects or network connections
+# among many threads.  Note: pool elements are lazily created.
 #
 # Example usage with block (faster):
 #
 #    @pool = ConnectionPool.new { Redis.new }
-#
 #    @pool.with do |redis|
 #      redis.lpop('my-list') if redis.llen('my-list') > 0
 #    end
 #
 # Using optional timeout override (for that single invocation)
 #
-#    @pool.with(:timeout => 2.0) do |redis|
+#    @pool.with(timeout: 2.0) do |redis|
 #      redis.lpop('my-list') if redis.llen('my-list') > 0
 #    end
 #
@@ -33,105 +38,89 @@ require_relative 'connection_pool/timed_stack'
 class ConnectionPool
   DEFAULTS = {size: 5, timeout: 5}
 
-  class Error < RuntimeError
-  end
-
   def self.wrap(options, &block)
     Wrapper.new(options, &block)
   end
 
   def initialize(options = {}, &block)
-    raise ArgumentError, 'Connection pool requires a block' unless block
+    raise ArgumentError, "Connection pool requires a block" unless block
 
     options = DEFAULTS.merge(options)
 
-    @size = options.fetch(:size)
+    @size = Integer(options.fetch(:size))
     @timeout = options.fetch(:timeout)
 
     @available = TimedStack.new(@size, &block)
-    @key = :"current-#{@available.object_id}"
+    @key = :"pool-#{@available.object_id}"
+    @key_count = :"pool-#{@available.object_id}-count"
   end
 
   def with(options = {})
-    # Connections can become corrupted via Timeout::Error.  Discard
-    # any connection whose usage after checkout does not finish as expected.
-    # See #67
-    success = false
-    conn = checkout(options)
-    begin
-      result = yield conn
-      success = true # means the connection wasn't interrupted
-      result
-    ensure
-      if success
-        # everything is roses, we can safely check the connection back in
+    Thread.handle_interrupt(Exception => :never) do
+      conn = checkout(options)
+      begin
+        Thread.handle_interrupt(Exception => :immediate) do
+          yield conn
+        end
+      ensure
         checkin
-      else
-        @available.discard!(pop_connection)
       end
     end
   end
+  alias then with
 
   def checkout(options = {})
-    conn = if stack.empty?
-      timeout = options[:timeout] || @timeout
-      @available.pop(timeout: timeout)
+    if ::Thread.current[@key]
+      ::Thread.current[@key_count] += 1
+      ::Thread.current[@key]
     else
-      stack.last
+      ::Thread.current[@key_count] = 1
+      ::Thread.current[@key] = @available.pop(options[:timeout] || @timeout)
     end
-
-    stack.push conn
-    conn
   end
 
   def checkin
-    conn = pop_connection # mutates stack, must be on its own line
-    @available.push(conn) if stack.empty?
+    if ::Thread.current[@key]
+      if ::Thread.current[@key_count] == 1
+        @available.push(::Thread.current[@key])
+        ::Thread.current[@key] = nil
+        ::Thread.current[@key_count] = nil
+      else
+        ::Thread.current[@key_count] -= 1
+      end
+    else
+      raise ConnectionPool::Error, "no connections are checked out"
+    end
 
     nil
   end
 
+  ##
+  # Shuts down the ConnectionPool by passing each connection to +block+ and
+  # then removing it from the pool. Attempting to checkout a connection after
+  # shutdown will raise +ConnectionPool::PoolShuttingDownError+.
+
   def shutdown(&block)
     @available.shutdown(&block)
   end
 
-  private
-
-  def pop_connection
-    if stack.empty?
-      raise ConnectionPool::Error, 'no connections are checked out'
-    else
-      stack.pop
-    end
-  end
+  ##
+  # Reloads the ConnectionPool by passing each connection to +block+ and then
+  # removing it the pool. Subsequent checkouts will create new connections as
+  # needed.
 
-  def stack
-    ::Thread.current[@key] ||= []
+  def reload(&block)
+    @available.shutdown(reload: true, &block)
   end
 
-  class Wrapper < ::BasicObject
-    METHODS = [:with, :pool_shutdown]
-
-    def initialize(options = {}, &block)
-      @pool = ::ConnectionPool.new(options, &block)
-    end
-
-    def with(&block)
-      @pool.with(&block)
-    end
-
-    def pool_shutdown(&block)
-      @pool.shutdown(&block)
-    end
-
-    def respond_to?(id, *args)
-      METHODS.include?(id) || with { |c| c.respond_to?(id, *args) }
-    end
+  # Size of this connection pool
+  attr_reader :size
 
-    def method_missing(name, *args, &block)
-      with do |connection|
-        connection.send(name, *args, &block)
-      end
-    end
+  # Number of pool entries available for checkout at this instant.
+  def available
+    @available.length
   end
 end
+
+require "connection_pool/timed_stack"
+require "connection_pool/wrapper"

data/lib/connection_pool/timed_stack.rb

@@ -1,12 +1,3 @@
-require 'thread'
-require 'timeout'
-
-##
-# Raised when you attempt to retrieve a connection from a pool that has been
-# shut down.
-
-class ConnectionPool::PoolShuttingDownError < RuntimeError; end
-
 ##
 # The TimedStack manages a pool of homogeneous connections (or any resource
 # you wish to manage).  Connections are created lazily up to a given maximum
@@ -24,9 +15,10 @@ class ConnectionPool::PoolShuttingDownError < RuntimeError; end
 #
 #    conn = ts.pop
 #    ts.pop timeout: 5
-#    #=> raises Timeout::Error after 5 seconds
+#    #=> raises ConnectionPool::TimeoutError after 5 seconds
 
 class ConnectionPool::TimedStack
+  attr_reader :max
 
   ##
   # Creates a new pool with +size+ connections that are created from the given
@@ -57,12 +49,12 @@ class ConnectionPool::TimedStack
       @resource.broadcast
     end
   end
-  alias_method :<<, :push
+  alias << push
 
   ##
   # Retrieves a connection from the stack.  If a connection is available it is
   # immediately returned.  If no connection is available within the given
-  # timeout a Timeout::Error is raised.
+  # timeout a ConnectionPool::TimeoutError is raised.
   #
   # +:timeout+ is the only checked entry in +options+ and is preferred over
   # the +timeout+ argument (which will be removed in a future release).  Other
@@ -72,7 +64,7 @@ class ConnectionPool::TimedStack
     options, timeout = timeout, 0.5 if Hash === timeout
     timeout = options.fetch :timeout, timeout
 
-    deadline = Time.now + timeout
+    deadline = current_time + timeout
     @mutex.synchronize do
       loop do
         raise ConnectionPool::PoolShuttingDownError if @shutdown_block
@@ -81,18 +73,20 @@ class ConnectionPool::TimedStack
         connection = try_create(options)
         return connection if connection
 
-        to_wait = deadline - Time.now
-        raise Timeout::Error, "Waited #{timeout} sec" if to_wait <= 0
+        to_wait = deadline - current_time
+        raise ConnectionPool::TimeoutError, "Waited #{timeout} sec" if to_wait <= 0
         @resource.wait(@mutex, to_wait)
       end
     end
   end
 
   ##
-  # Shuts down the TimedStack which prevents connections from being checked
-  # out.  The +block+ is called once for each connection on the stack.
+  # Shuts down the TimedStack by passing each connection to +block+ and then
+  # removing it from the pool. Attempting to checkout a connection after
+  # shutdown will raise +ConnectionPool::PoolShuttingDownError+ unless
+  # +:reload+ is +true+.
 
-  def shutdown(&block)
+  def shutdown(reload: false, &block)
     raise ArgumentError, "shutdown must receive a block" unless block_given?
 
     @mutex.synchronize do
@@ -100,6 +94,7 @@ class ConnectionPool::TimedStack
       @resource.broadcast
 
       shutdown_connections
+      @shutdown_block = nil if reload
     end
   end
 
@@ -117,30 +112,12 @@ class ConnectionPool::TimedStack
     @max - @created + @que.length
   end
 
-  ##
-  # Indicates that a connection isn't coming back, allowing a new one to be
-  # created to replace it.
-
-  def discard!(obj)
-    @mutex.synchronize do
-      if @shutdown_block
-        @shutdown_block.call(obj)
-      else
-        # try to shut down the connection before throwing it away
-        if obj.respond_to?(:close) # Dalli::Client
-          obj.close rescue nil
-        elsif obj.respond_to?(:disconnect!) # Redis
-          obj.disconnect! rescue nil
-        end
-        @created -= 1
-      end
+  private
 
-      @resource.broadcast
-    end
+  def current_time
+    Process.clock_gettime(Process::CLOCK_MONOTONIC)
   end
 
-  private
-
   ##
   # This is an extension point for TimedStack and is called with a mutex.
   #
@@ -169,6 +146,7 @@ class ConnectionPool::TimedStack
       conn = fetch_connection(options)
       @shutdown_block.call(conn)
     end
+    @created = 0
   end
 
   ##