connection_pool 2.2.2
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +7 -0
- data/.gitignore +4 -0
- data/.travis.yml +10 -0
- data/Changes.md +123 -0
- data/Gemfile +3 -0
- data/LICENSE +20 -0
- data/README.md +107 -0
- data/Rakefile +6 -0
- data/connection_pool.gemspec +21 -0
- data/lib/connection_pool.rb +161 -0
- data/lib/connection_pool/monotonic_time.rb +66 -0
- data/lib/connection_pool/timed_stack.rb +176 -0
- data/lib/connection_pool/version.rb +3 -0
- data/test/helper.rb +8 -0
- data/test/test_connection_pool.rb +516 -0
- data/test/test_connection_pool_timed_stack.rb +149 -0
- metadata +106 -0
checksums.yaml
ADDED
@@ -0,0 +1,7 @@
|
|
1
|
+
---
|
2
|
+
SHA1:
|
3
|
+
metadata.gz: cf8177bf07f6c8e68b13f28416f6ccc0c64803af
|
4
|
+
data.tar.gz: ff9c836ce5576d4d0cb1edf8d858a761f982e437
|
5
|
+
SHA512:
|
6
|
+
metadata.gz: 73bda4d8d0b3cc9daef5c93bb98b7649d811bd5c8f467e25c3aaf7d3353541225e7b30e1efd481a14e45f5c56b5fc846d49dabdd9a167702a71c4a9631a9b6a2
|
7
|
+
data.tar.gz: a331275f67b3635e09792e646113e32bdc74254a2cdff032e972c6f2ebedd6391a1e508c36be7c4e33b40e2a07efe029a2ad0267c3ba4dc20b22813df86427bb
|
data/.gitignore
ADDED
data/.travis.yml
ADDED
data/Changes.md
ADDED
@@ -0,0 +1,123 @@
|
|
1
|
+
connection\_pool changelog
|
2
|
+
---------------------------
|
3
|
+
|
4
|
+
2.2.2
|
5
|
+
------
|
6
|
+
|
7
|
+
- Add pool `size` and `available` accessors for metrics and monitoring
|
8
|
+
purposes [#97, robholland]
|
9
|
+
|
10
|
+
2.2.1
|
11
|
+
------
|
12
|
+
|
13
|
+
- Allow CP::Wrapper to use an existing pool [#87, etiennebarrie]
|
14
|
+
- Use monotonic time for more accurate timeouts [#84, jdantonio]
|
15
|
+
|
16
|
+
2.2.0
|
17
|
+
------
|
18
|
+
|
19
|
+
- Rollback `Timeout` handling introduced in 2.1.1 and 2.1.2. It seems
|
20
|
+
impossible to safely work around the issue. Please never, ever use
|
21
|
+
`Timeout.timeout` in your code or you will see rare but mysterious bugs. [#75]
|
22
|
+
|
23
|
+
2.1.3
|
24
|
+
------
|
25
|
+
|
26
|
+
- Don't increment created count until connection is successfully
|
27
|
+
created. [mylesmegyesi, #73]
|
28
|
+
|
29
|
+
2.1.2
|
30
|
+
------
|
31
|
+
|
32
|
+
- The connection\_pool will now close any connections which respond to
|
33
|
+
`close` (Dalli) or `disconnect!` (Redis). This ensures discarded connections
|
34
|
+
from the fix in 2.1.1 are torn down ASAP and don't linger open.
|
35
|
+
|
36
|
+
|
37
|
+
2.1.1
|
38
|
+
------
|
39
|
+
|
40
|
+
- Work around a subtle race condition with code which uses `Timeout.timeout` and
|
41
|
+
checks out a connection within the timeout block. This might cause
|
42
|
+
connections to get into a bad state and raise very odd errors. [tamird, #67]
|
43
|
+
|
44
|
+
|
45
|
+
2.1.0
|
46
|
+
------
|
47
|
+
|
48
|
+
- Refactoring to better support connection pool subclasses [drbrain,
|
49
|
+
#55]
|
50
|
+
- `with` should return value of the last expression [#59]
|
51
|
+
|
52
|
+
|
53
|
+
2.0.0
|
54
|
+
-----
|
55
|
+
|
56
|
+
- The connection pool is now lazy. Connections are created as needed
|
57
|
+
and retained until the pool is shut down. [drbrain, #52]
|
58
|
+
|
59
|
+
1.2.0
|
60
|
+
-----
|
61
|
+
|
62
|
+
- Add `with(options)` and `checkout(options)`. [mattcamuto]
|
63
|
+
Allows the caller to override the pool timeout.
|
64
|
+
```ruby
|
65
|
+
@pool.with(:timeout => 2) do |conn|
|
66
|
+
end
|
67
|
+
```
|
68
|
+
|
69
|
+
1.1.0
|
70
|
+
-----
|
71
|
+
|
72
|
+
- New `#shutdown` method (simao)
|
73
|
+
|
74
|
+
This method accepts a block and calls the block for each
|
75
|
+
connection in the pool. After calling this method, trying to get a
|
76
|
+
connection from the pool raises `PoolShuttingDownError`.
|
77
|
+
|
78
|
+
1.0.0
|
79
|
+
-----
|
80
|
+
|
81
|
+
- `#with_connection` is now gone in favor of `#with`.
|
82
|
+
|
83
|
+
- We no longer pollute the top level namespace with our internal
|
84
|
+
`TimedStack` class.
|
85
|
+
|
86
|
+
0.9.3
|
87
|
+
--------
|
88
|
+
|
89
|
+
- `#with_connection` is now deprecated in favor of `#with`.
|
90
|
+
|
91
|
+
A warning will be issued in the 0.9 series and the method will be
|
92
|
+
removed in 1.0.
|
93
|
+
|
94
|
+
- We now reuse objects when possible.
|
95
|
+
|
96
|
+
This means that under no contention, the same object will be checked
|
97
|
+
out from the pool after subsequent calls to `ConnectionPool#with`.
|
98
|
+
|
99
|
+
This change should have no impact on end user performance. If
|
100
|
+
anything, it should be an improvement, depending on what objects you
|
101
|
+
are pooling.
|
102
|
+
|
103
|
+
0.9.2
|
104
|
+
--------
|
105
|
+
|
106
|
+
- Fix reentrant checkout leading to early checkin.
|
107
|
+
|
108
|
+
0.9.1
|
109
|
+
--------
|
110
|
+
|
111
|
+
- Fix invalid superclass in version.rb
|
112
|
+
|
113
|
+
0.9.0
|
114
|
+
--------
|
115
|
+
|
116
|
+
- Move method\_missing magic into ConnectionPool::Wrapper (djanowski)
|
117
|
+
- Remove BasicObject superclass (djanowski)
|
118
|
+
|
119
|
+
0.1.0
|
120
|
+
--------
|
121
|
+
|
122
|
+
- More precise timeouts and better error message
|
123
|
+
- ConnectionPool now subclasses BasicObject so `method_missing` is more effective.
|
data/Gemfile
ADDED
data/LICENSE
ADDED
@@ -0,0 +1,20 @@
|
|
1
|
+
Copyright (c) 2011 Mike Perham
|
2
|
+
|
3
|
+
Permission is hereby granted, free of charge, to any person obtaining
|
4
|
+
a copy of this software and associated documentation files (the
|
5
|
+
"Software"), to deal in the Software without restriction, including
|
6
|
+
without limitation the rights to use, copy, modify, merge, publish,
|
7
|
+
distribute, sublicense, and/or sell copies of the Software, and to
|
8
|
+
permit persons to whom the Software is furnished to do so, subject to
|
9
|
+
the following conditions:
|
10
|
+
|
11
|
+
The above copyright notice and this permission notice shall be
|
12
|
+
included in all copies or substantial portions of the Software.
|
13
|
+
|
14
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
|
15
|
+
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
|
16
|
+
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
|
17
|
+
NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
|
18
|
+
LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
|
19
|
+
OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
|
20
|
+
WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
|
data/README.md
ADDED
@@ -0,0 +1,107 @@
|
|
1
|
+
connection\_pool
|
2
|
+
=================
|
3
|
+
[![Build Status](https://travis-ci.org/mperham/connection_pool.svg)](https://travis-ci.org/mperham/connection_pool)
|
4
|
+
|
5
|
+
Generic connection pooling for Ruby.
|
6
|
+
|
7
|
+
MongoDB has its own connection pool. ActiveRecord has its own connection pool.
|
8
|
+
This is a generic connection pool that can be used with anything, e.g. Redis,
|
9
|
+
Dalli and other Ruby network clients.
|
10
|
+
|
11
|
+
|
12
|
+
Usage
|
13
|
+
-----
|
14
|
+
|
15
|
+
Create a pool of objects to share amongst the fibers or threads in your Ruby
|
16
|
+
application:
|
17
|
+
|
18
|
+
``` ruby
|
19
|
+
$memcached = ConnectionPool.new(size: 5, timeout: 5) { Dalli::Client.new }
|
20
|
+
```
|
21
|
+
|
22
|
+
Then use the pool in your application:
|
23
|
+
|
24
|
+
``` ruby
|
25
|
+
$memcached.with do |conn|
|
26
|
+
conn.get('some-count')
|
27
|
+
end
|
28
|
+
```
|
29
|
+
|
30
|
+
If all the objects in the connection pool are in use, `with` will block
|
31
|
+
until one becomes available. If no object is available within `:timeout` seconds,
|
32
|
+
`with` will raise a `Timeout::Error`.
|
33
|
+
|
34
|
+
Optionally, you can specify a timeout override using the with-block semantics:
|
35
|
+
|
36
|
+
``` ruby
|
37
|
+
$memcached.with(timeout: 2.0) do |conn|
|
38
|
+
conn.get('some-count')
|
39
|
+
end
|
40
|
+
```
|
41
|
+
|
42
|
+
This will only modify the resource-get timeout for this particular
|
43
|
+
invocation. This is useful if you want to fail-fast on certain non critical
|
44
|
+
sections when a resource is not available, or conversely if you are comfortable
|
45
|
+
blocking longer on a particular resource. This is not implemented in the below
|
46
|
+
`ConnectionPool::Wrapper` class.
|
47
|
+
|
48
|
+
## Migrating to a Connection Pool
|
49
|
+
|
50
|
+
You can use `ConnectionPool::Wrapper` to wrap a single global connection,
|
51
|
+
making it easier to migrate existing connection code over time:
|
52
|
+
|
53
|
+
``` ruby
|
54
|
+
$redis = ConnectionPool::Wrapper.new(size: 5, timeout: 3) { Redis.connect }
|
55
|
+
$redis.sadd('foo', 1)
|
56
|
+
$redis.smembers('foo')
|
57
|
+
```
|
58
|
+
|
59
|
+
The wrapper uses `method_missing` to checkout a connection, run the requested
|
60
|
+
method and then immediately check the connection back into the pool. It's
|
61
|
+
**not** high-performance so you'll want to port your performance sensitive code
|
62
|
+
to use `with` as soon as possible.
|
63
|
+
|
64
|
+
``` ruby
|
65
|
+
$redis.with do |conn|
|
66
|
+
conn.sadd('foo', 1)
|
67
|
+
conn.smembers('foo')
|
68
|
+
end
|
69
|
+
```
|
70
|
+
|
71
|
+
Once you've ported your entire system to use `with`, you can simply remove
|
72
|
+
`Wrapper` and use the simpler and faster `ConnectionPool`.
|
73
|
+
|
74
|
+
|
75
|
+
## Shutdown
|
76
|
+
|
77
|
+
You can shut down a ConnectionPool instance once it should no longer be used.
|
78
|
+
Further checkout attempts will immediately raise an error but existing checkouts
|
79
|
+
will work.
|
80
|
+
|
81
|
+
```ruby
|
82
|
+
cp = ConnectionPool.new { Redis.new }
|
83
|
+
cp.shutdown { |conn| conn.quit }
|
84
|
+
```
|
85
|
+
|
86
|
+
Shutting down a connection pool will block until all connections are checked in and closed.
|
87
|
+
**Note that shutting down is completely optional**; Ruby's garbage collector will reclaim
|
88
|
+
unreferenced pools under normal circumstances.
|
89
|
+
|
90
|
+
|
91
|
+
Notes
|
92
|
+
-----
|
93
|
+
|
94
|
+
- Connections are lazily created as needed.
|
95
|
+
- There is no provision for repairing or checking the health of a connection;
|
96
|
+
connections should be self-repairing. This is true of the Dalli and Redis
|
97
|
+
clients.
|
98
|
+
- **WARNING**: Don't ever use `Timeout.timeout` in your Ruby code or you will see
|
99
|
+
occasional silent corruption and mysterious errors. The Timeout API is unsafe
|
100
|
+
and cannot be used correctly, ever. Use proper socket timeout options as
|
101
|
+
exposed by Net::HTTP, Redis, Dalli, etc.
|
102
|
+
|
103
|
+
|
104
|
+
Author
|
105
|
+
------
|
106
|
+
|
107
|
+
Mike Perham, [@mperham](https://twitter.com/mperham), <http://mikeperham.com>
|
data/Rakefile
ADDED
@@ -0,0 +1,21 @@
|
|
1
|
+
# -*- encoding: utf-8 -*-
|
2
|
+
require "./lib/connection_pool/version"
|
3
|
+
|
4
|
+
Gem::Specification.new do |s|
|
5
|
+
s.name = "connection_pool"
|
6
|
+
s.version = ConnectionPool::VERSION
|
7
|
+
s.platform = Gem::Platform::RUBY
|
8
|
+
s.authors = ["Mike Perham", "Damian Janowski"]
|
9
|
+
s.email = ["mperham@gmail.com", "damian@educabilia.com"]
|
10
|
+
s.homepage = "https://github.com/mperham/connection_pool"
|
11
|
+
s.description = s.summary = %q{Generic connection pool for Ruby}
|
12
|
+
|
13
|
+
s.files = `git ls-files`.split("\n")
|
14
|
+
s.test_files = `git ls-files -- {test,spec,features}/*`.split("\n")
|
15
|
+
s.executables = `git ls-files -- bin/*`.split("\n").map{ |f| File.basename(f) }
|
16
|
+
s.require_paths = ["lib"]
|
17
|
+
s.license = "MIT"
|
18
|
+
s.add_development_dependency 'bundler'
|
19
|
+
s.add_development_dependency 'minitest', '>= 5.0.0'
|
20
|
+
s.add_development_dependency 'rake'
|
21
|
+
end
|
@@ -0,0 +1,161 @@
|
|
1
|
+
require_relative 'connection_pool/version'
|
2
|
+
require_relative 'connection_pool/timed_stack'
|
3
|
+
|
4
|
+
|
5
|
+
# Generic connection pool class for e.g. sharing a limited number of network connections
|
6
|
+
# among many threads. Note: Connections are lazily created.
|
7
|
+
#
|
8
|
+
# Example usage with block (faster):
|
9
|
+
#
|
10
|
+
# @pool = ConnectionPool.new { Redis.new }
|
11
|
+
#
|
12
|
+
# @pool.with do |redis|
|
13
|
+
# redis.lpop('my-list') if redis.llen('my-list') > 0
|
14
|
+
# end
|
15
|
+
#
|
16
|
+
# Using optional timeout override (for that single invocation)
|
17
|
+
#
|
18
|
+
# @pool.with(timeout: 2.0) do |redis|
|
19
|
+
# redis.lpop('my-list') if redis.llen('my-list') > 0
|
20
|
+
# end
|
21
|
+
#
|
22
|
+
# Example usage replacing an existing connection (slower):
|
23
|
+
#
|
24
|
+
# $redis = ConnectionPool.wrap { Redis.new }
|
25
|
+
#
|
26
|
+
# def do_work
|
27
|
+
# $redis.lpop('my-list') if $redis.llen('my-list') > 0
|
28
|
+
# end
|
29
|
+
#
|
30
|
+
# Accepts the following options:
|
31
|
+
# - :size - number of connections to pool, defaults to 5
|
32
|
+
# - :timeout - amount of time to wait for a connection if none currently available, defaults to 5 seconds
|
33
|
+
#
|
34
|
+
class ConnectionPool
|
35
|
+
DEFAULTS = {size: 5, timeout: 5}
|
36
|
+
|
37
|
+
class Error < RuntimeError
|
38
|
+
end
|
39
|
+
|
40
|
+
def self.wrap(options, &block)
|
41
|
+
Wrapper.new(options, &block)
|
42
|
+
end
|
43
|
+
|
44
|
+
def initialize(options = {}, &block)
|
45
|
+
raise ArgumentError, 'Connection pool requires a block' unless block
|
46
|
+
|
47
|
+
options = DEFAULTS.merge(options)
|
48
|
+
|
49
|
+
@size = options.fetch(:size)
|
50
|
+
@timeout = options.fetch(:timeout)
|
51
|
+
|
52
|
+
@available = TimedStack.new(@size, &block)
|
53
|
+
@key = :"current-#{@available.object_id}"
|
54
|
+
@key_count = :"current-#{@available.object_id}-count"
|
55
|
+
end
|
56
|
+
|
57
|
+
if Thread.respond_to?(:handle_interrupt)
|
58
|
+
|
59
|
+
# MRI
|
60
|
+
def with(options = {})
|
61
|
+
Thread.handle_interrupt(Exception => :never) do
|
62
|
+
conn = checkout(options)
|
63
|
+
begin
|
64
|
+
Thread.handle_interrupt(Exception => :immediate) do
|
65
|
+
yield conn
|
66
|
+
end
|
67
|
+
ensure
|
68
|
+
checkin
|
69
|
+
end
|
70
|
+
end
|
71
|
+
end
|
72
|
+
|
73
|
+
else
|
74
|
+
|
75
|
+
# jruby 1.7.x
|
76
|
+
def with(options = {})
|
77
|
+
conn = checkout(options)
|
78
|
+
begin
|
79
|
+
yield conn
|
80
|
+
ensure
|
81
|
+
checkin
|
82
|
+
end
|
83
|
+
end
|
84
|
+
|
85
|
+
end
|
86
|
+
|
87
|
+
def checkout(options = {})
|
88
|
+
if ::Thread.current[@key]
|
89
|
+
::Thread.current[@key_count]+= 1
|
90
|
+
::Thread.current[@key]
|
91
|
+
else
|
92
|
+
::Thread.current[@key_count]= 1
|
93
|
+
::Thread.current[@key]= @available.pop(options[:timeout] || @timeout)
|
94
|
+
end
|
95
|
+
end
|
96
|
+
|
97
|
+
def checkin
|
98
|
+
if ::Thread.current[@key]
|
99
|
+
if ::Thread.current[@key_count] == 1
|
100
|
+
@available.push(::Thread.current[@key])
|
101
|
+
::Thread.current[@key]= nil
|
102
|
+
else
|
103
|
+
::Thread.current[@key_count]-= 1
|
104
|
+
end
|
105
|
+
else
|
106
|
+
raise ConnectionPool::Error, 'no connections are checked out'
|
107
|
+
end
|
108
|
+
|
109
|
+
nil
|
110
|
+
end
|
111
|
+
|
112
|
+
def shutdown(&block)
|
113
|
+
@available.shutdown(&block)
|
114
|
+
end
|
115
|
+
|
116
|
+
# Size of this connection pool
|
117
|
+
def size
|
118
|
+
@size
|
119
|
+
end
|
120
|
+
|
121
|
+
# Number of pool entries available for checkout at this instant.
|
122
|
+
def available
|
123
|
+
@available.length
|
124
|
+
end
|
125
|
+
|
126
|
+
private
|
127
|
+
|
128
|
+
class Wrapper < ::BasicObject
|
129
|
+
METHODS = [:with, :pool_shutdown]
|
130
|
+
|
131
|
+
def initialize(options = {}, &block)
|
132
|
+
@pool = options.fetch(:pool) { ::ConnectionPool.new(options, &block) }
|
133
|
+
end
|
134
|
+
|
135
|
+
def with(&block)
|
136
|
+
@pool.with(&block)
|
137
|
+
end
|
138
|
+
|
139
|
+
def pool_shutdown(&block)
|
140
|
+
@pool.shutdown(&block)
|
141
|
+
end
|
142
|
+
|
143
|
+
def pool_size
|
144
|
+
@pool.size
|
145
|
+
end
|
146
|
+
|
147
|
+
def pool_available
|
148
|
+
@pool.available
|
149
|
+
end
|
150
|
+
|
151
|
+
def respond_to?(id, *args)
|
152
|
+
METHODS.include?(id) || with { |c| c.respond_to?(id, *args) }
|
153
|
+
end
|
154
|
+
|
155
|
+
def method_missing(name, *args, &block)
|
156
|
+
with do |connection|
|
157
|
+
connection.send(name, *args, &block)
|
158
|
+
end
|
159
|
+
end
|
160
|
+
end
|
161
|
+
end
|