fiber_connection_pool 0.1.2 → 0.2.0
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/.travis.yml +0 -6
- data/README.md +102 -28
- data/fiber_connection_pool.gemspec +2 -3
- data/lib/fiber_connection_pool/exceptions.rb +6 -0
- data/lib/fiber_connection_pool.rb +144 -44
- data/test/fiber_connection_pool_test.rb +183 -21
- data/test/helper.rb +40 -6
- metadata +8 -5
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA1:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 3805c4817fd7ddab4eaafcdf30bacdd99dd79646
|
4
|
+
data.tar.gz: 2af6d4c85fd8d672b7d8c0b9eaea6a02a76e3658
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 65a29c9021d57a6f7de8390aa28cf56a5d330ed7b1259dd760b6b4dd5e5be67d8a7e5b1d0c127fb62d7b7c0d252746866269e4fd2a134cf2ab92e8dcba5a9b8c
|
7
|
+
data.tar.gz: ae6b1518828313527b91be4005e6cdc510f364158ed7626df6dff083feee43166006267baba0aae70f519df54bc20ee93bdf0b94e0fd97072bacf3fd6b4e823b
|
data/.travis.yml
CHANGED
data/README.md
CHANGED
@@ -47,25 +47,31 @@ It just keeps an array (the internal pool) holding the result of running
|
|
47
47
|
the given block _size_ times. Inside the reactor loop (either EventMachine's or Celluloid's),
|
48
48
|
each request is wrapped on a Fiber, and then `pool` plays its magic.
|
49
49
|
|
50
|
-
|
51
|
-
|
50
|
+
``` ruby
|
51
|
+
results = pool.query_me(sql)
|
52
|
+
```
|
52
53
|
|
53
|
-
|
54
|
-
|
55
|
-
|
56
|
-
|
57
|
-
and
|
58
|
-
|
54
|
+
When a method `query_me` is called on `pool` it:
|
55
|
+
|
56
|
+
1. Reserves one connection from the internal pool and associates it __with the current fiber__.
|
57
|
+
2. If no connection is available, then that fiber stays on a _pending_ queue,
|
58
|
+
and __is yielded__ until another connection is released.
|
59
|
+
3. When a connection is available, then the pool calls `query_me` on that `MyFancyConnection` instance.
|
60
|
+
4. When `query_me` returns, the reserved instance is released again,
|
61
|
+
and the next fiber on the _pending_ queue __is resumed__.
|
62
|
+
5. The return value is sent back to the caller.
|
59
63
|
|
60
64
|
Methods from `MyFancyConnection` instance should yield the fiber before
|
61
65
|
perform any blocking IO. That returns control to te underlying reactor,
|
62
66
|
that spawns another fiber to process the next request, while the previous
|
63
67
|
one is still waiting for the IO response. That new fiber will get its own
|
64
68
|
connection from the pool, or else it will yield until there
|
65
|
-
is one available.
|
69
|
+
is one available. That behaviour is implemented on `Mysql2::EM::Client`
|
70
|
+
from [em-synchrony](https://github.com/igrigorik/em-synchrony),
|
71
|
+
and on a patched version of [ruby-mysql](https://github.com/rubencaro/ruby-mysql), for example.
|
66
72
|
|
67
|
-
The whole process looks synchronous from the
|
68
|
-
The
|
73
|
+
The whole process looks synchronous from the fiber perspective, _because it is_ indeed.
|
74
|
+
The fiber will really block ( or _yield_ ) until it gets the result.
|
69
75
|
|
70
76
|
``` ruby
|
71
77
|
results = pool.query_me(sql)
|
@@ -77,21 +83,21 @@ The magic resides on the fact that other fibers are being processed while this o
|
|
77
83
|
Not thread-safe
|
78
84
|
------------------
|
79
85
|
|
80
|
-
`FiberConnectionPool` is not thread-safe
|
86
|
+
`FiberConnectionPool` is not thread-safe. You will not be able to use it
|
81
87
|
from different threads, as eventually it will try to resume a Fiber that resides
|
82
88
|
on a different Thread. That will raise a FiberError( _"calling a fiber across threads"_ ).
|
83
|
-
Maybe one day we add that feature too.
|
89
|
+
Maybe one day we add that feature too. Or maybe it's not worth the added code complexity.
|
84
90
|
|
85
|
-
We
|
86
|
-
having one pool on each Actor thread. Take a look at the `examples` folder for details.
|
91
|
+
We use it with no need to be thread-safe on Goliath servers having one pool on each server instance,
|
92
|
+
and on Reel servers having one pool on each Actor thread. Take a look at the `examples` folder for details.
|
87
93
|
|
88
|
-
|
94
|
+
Generic
|
89
95
|
------------------
|
90
96
|
|
91
|
-
|
92
|
-
|
97
|
+
We use it extensively with MySQL connections with Goliath servers by using `Mysql2::EM::Client`
|
98
|
+
from [em-synchrony](https://github.com/igrigorik/em-synchrony).
|
93
99
|
And for Celluloid by using a patched version of [ruby-mysql](https://github.com/rubencaro/ruby-mysql).
|
94
|
-
|
100
|
+
By >=0.2 there is no MySQL-specific code, so it can be used with any kind of connection that can be fibered.
|
95
101
|
|
96
102
|
Reacting to connection failure
|
97
103
|
------------------
|
@@ -102,8 +108,81 @@ react as you would do normally.
|
|
102
108
|
|
103
109
|
You have to be aware that the connection instance will remain in the pool, and other fibers
|
104
110
|
will surely use it. If the Exception you rescued indicates that the connection should be
|
105
|
-
recreated
|
106
|
-
|
111
|
+
recreated or treated somehow, there's a way to access that particular connection:
|
112
|
+
|
113
|
+
``` ruby
|
114
|
+
begin
|
115
|
+
|
116
|
+
pool.bad_query('will make me worse')
|
117
|
+
|
118
|
+
rescue BadQueryMadeMeWorse
|
119
|
+
|
120
|
+
pool.with_failed_connection do |connection|
|
121
|
+
puts "Replacing #{connection.inspect} with a new one!"
|
122
|
+
MyFancyConnection.new
|
123
|
+
end
|
124
|
+
|
125
|
+
end
|
126
|
+
```
|
127
|
+
|
128
|
+
The pool saves the connection when it raises an exception on a fiber, and with `with_failed_connection` lets
|
129
|
+
you execute a block of code over it. It must return a connection instance, and it will be put inside the pool
|
130
|
+
in place of the failed one. It can be the same instance after being fixed, or maybe a new one.
|
131
|
+
The call to `with_failed_connection` must be made from the very same
|
132
|
+
fiber that raised the exception.
|
133
|
+
|
134
|
+
Also the reference to the failed connection will be lost after any method execution from that
|
135
|
+
fiber. So you must call `with_failed_connection` before any other method that may acquire a new instance from the pool.
|
136
|
+
|
137
|
+
Any reference to a failed connection is released when the fiber is dead, but as you must access it from the fiber itself, worry should not.
|
138
|
+
|
139
|
+
Save data
|
140
|
+
-------------------
|
141
|
+
|
142
|
+
Sometimes we need to get something more than de return value from the `query_me` call, but that _something_ is related to _that_ call on _that_ connection.
|
143
|
+
For example, maybe you need to call `affected_rows` right after the query was made on that particular connection.
|
144
|
+
If you make that extra calls on the `pool` object, it will acquire a new connection from the pool an run on it. So it's useless.
|
145
|
+
There is a way to gather all that data from the connection so we can work on it, but also release the connection for other fiber to use it.
|
146
|
+
|
147
|
+
``` ruby
|
148
|
+
# define the pool
|
149
|
+
pool = FiberConnectionPool.new(:size => 5){ MyFancyConnection.new }
|
150
|
+
|
151
|
+
# add a request to save data for each successful call on a connection
|
152
|
+
# will save the return value inside a hash on the key ':affected_rows'
|
153
|
+
# and make it available for the fiber that made the call
|
154
|
+
pool.save_data(:affected_rows) do |connection|
|
155
|
+
connection.affected_rows
|
156
|
+
end
|
157
|
+
```
|
158
|
+
|
159
|
+
Then from our fiber:
|
160
|
+
|
161
|
+
``` ruby
|
162
|
+
pool.query_me('affecting 5 rows right now')
|
163
|
+
|
164
|
+
# recover gathered data for this fiber
|
165
|
+
puts pool.gathered_data
|
166
|
+
=> { :affected_rows => 5 }
|
167
|
+
```
|
168
|
+
|
169
|
+
You must access the gathered data from the same fiber that triggered its gathering.
|
170
|
+
Also any new call to `query_me` or any other method from the connection would execute the block again,
|
171
|
+
overwriting that position on the hash (unless you code to prevent it, of course). Usually you would use the gathered data
|
172
|
+
right after you made the query that generated it. But you could:
|
173
|
+
|
174
|
+
``` ruby
|
175
|
+
# save only the first run
|
176
|
+
pool.save_data(:affected_rows) do |connection|
|
177
|
+
pool.gathered_data[:affected_rows] || connection.affected_rows
|
178
|
+
end
|
179
|
+
```
|
180
|
+
|
181
|
+
You can define as much `save_data` blocks as you want, and run any wonder ruby lets you. But great power comes with great responsability.
|
182
|
+
You must consider that any requests for saving data are executed for _every call_ on the pool from that fiber.
|
183
|
+
So keep it stupid simple, and blindly fast. At least as much as you can. That would affect performance otherwise.
|
184
|
+
|
185
|
+
Any gathered_data is released when the fiber is dead, but as you must access it from the fiber itself, worry should not.
|
107
186
|
|
108
187
|
Supported Platforms
|
109
188
|
-------------------
|
@@ -111,11 +190,6 @@ Supported Platforms
|
|
111
190
|
Used in production environments on Ruby 1.9.3 and 2.0.0.
|
112
191
|
Tested against Ruby 1.9.3, 2.0.0, and rbx-19mode ([See details..](http://travis-ci.org/rubencaro/fiber_connection_pool)).
|
113
192
|
|
114
|
-
|
193
|
+
More to come !
|
115
194
|
-------------------
|
116
|
-
|
117
|
-
* no MySQL-specific code
|
118
|
-
* better testing
|
119
|
-
* improve reaction to failure
|
120
|
-
* better in-code docs
|
121
|
-
* make thread-safe
|
195
|
+
See [issues](https://github.com/rubencaro/fiber_connection_pool/issues?direction=desc&sort=updated&state=open)
|
@@ -6,7 +6,7 @@ Gem::Specification.new do |s|
|
|
6
6
|
s.version = FiberConnectionPool::VERSION
|
7
7
|
s.platform = Gem::Platform::RUBY
|
8
8
|
s.authors = ["Ruben Caro", "Oriol Francès"]
|
9
|
-
s.email = ["ruben@lanuez.org"]
|
9
|
+
s.email = ["ruben.caro@lanuez.org"]
|
10
10
|
s.homepage = "https://github.com/rubencaro/fiber_connection_pool"
|
11
11
|
s.summary = "Fiber-based generic connection pool for Ruby"
|
12
12
|
s.description = "Fiber-based generic connection pool for Ruby, allowing
|
@@ -14,8 +14,7 @@ Gem::Specification.new do |s|
|
|
14
14
|
as provided by EventMachine or Celluloid."
|
15
15
|
|
16
16
|
s.files = `git ls-files`.split("\n")
|
17
|
-
s.test_files = `git ls-files --
|
18
|
-
s.executables = `git ls-files -- bin/*`.split("\n").map{ |f| File.basename(f) }
|
17
|
+
s.test_files = `git ls-files -- test/*`.split("\n")
|
19
18
|
s.require_paths = ["lib"]
|
20
19
|
s.license = "GPLv3"
|
21
20
|
|
@@ -1,9 +1,13 @@
|
|
1
1
|
require 'fiber'
|
2
|
+
require_relative 'fiber_connection_pool/exceptions'
|
2
3
|
|
3
4
|
class FiberConnectionPool
|
4
|
-
VERSION = '0.
|
5
|
+
VERSION = '0.2.0'
|
5
6
|
|
6
|
-
|
7
|
+
RESERVED_BACKUP_TTL_SECS = 30 # reserved backup cleanup trigger
|
8
|
+
SAVED_DATA_TTL_SECS = 30 # saved_data cleanup trigger
|
9
|
+
|
10
|
+
attr_accessor :saved_data, :reserved_backup
|
7
11
|
|
8
12
|
# Initializes the pool with 'size' instances
|
9
13
|
# running the given block to get each one. Ex:
|
@@ -16,33 +20,113 @@ class FiberConnectionPool
|
|
16
20
|
@saved_data = {} # placeholder for requested save data
|
17
21
|
@reserved = {} # map of in-progress connections
|
18
22
|
@reserved_backup = {} # backup map of in-progress connections, to catch failures
|
23
|
+
@last_backup_cleanup = Time.now # reserved backup cleanup trigger
|
19
24
|
@available = [] # pool of free connections
|
20
25
|
@pending = [] # pending reservations (FIFO)
|
26
|
+
@save_data_requests = {} # blocks to be yielded to save data
|
27
|
+
@last_data_cleanup = Time.now # saved_data cleanup trigger
|
21
28
|
|
22
29
|
@available = Array.new(opts[:size].to_i) { yield }
|
23
30
|
end
|
24
31
|
|
32
|
+
# DEPRECATED: use save_data
|
25
33
|
def save_data_for_fiber
|
26
|
-
|
34
|
+
nil
|
27
35
|
end
|
28
36
|
|
37
|
+
# DEPRECATED: use release_data
|
29
38
|
def stop_saving_data_for_fiber
|
30
|
-
@saved_data.delete Fiber.current
|
39
|
+
@saved_data.delete Fiber.current
|
31
40
|
end
|
32
41
|
|
33
|
-
|
34
|
-
#
|
42
|
+
# Add a save_data request to the pool.
|
43
|
+
# The given block will be executed after each successful
|
44
|
+
# call to -any- method on the connection.
|
45
|
+
# The connection and the method name are passed to the block.
|
46
|
+
#
|
47
|
+
# The returned value will be saved in pool.saved_data[Fiber.current][key],
|
48
|
+
# and will be kept as long as the fiber stays alive.
|
49
|
+
#
|
50
|
+
# Ex:
|
51
|
+
#
|
52
|
+
# # (...right after pool's creation...)
|
53
|
+
# pool.save_data(:hey_or_hoo) do |conn, method|
|
54
|
+
# return 'hey' if method == 'query'
|
55
|
+
# 'hoo'
|
56
|
+
# end
|
57
|
+
#
|
58
|
+
# # (...from a reactor fiber...)
|
59
|
+
# myfiber = Fiber.current
|
60
|
+
# pool.query('select anything from anywhere')
|
61
|
+
# puts pool.saved_data[myfiber][:hey_or_hoo]
|
62
|
+
# => 'hey'
|
63
|
+
#
|
64
|
+
# # (...eventually fiber dies...)
|
65
|
+
# puts pool.saved_data[myfiber].inspect
|
66
|
+
# => nil
|
67
|
+
#
|
68
|
+
def save_data(key, &block)
|
69
|
+
@save_data_requests[key] = block
|
70
|
+
end
|
71
|
+
|
72
|
+
# Return the gathered data for this fiber
|
73
|
+
#
|
74
|
+
def gathered_data
|
75
|
+
@saved_data[Fiber.current]
|
76
|
+
end
|
77
|
+
|
78
|
+
# Clear any save_data requests in the pool.
|
79
|
+
# No data will be saved after this, unless new requests are added with #save_data.
|
80
|
+
#
|
81
|
+
def clear_save_data_requests
|
82
|
+
@save_data_requests = {}
|
83
|
+
end
|
84
|
+
|
85
|
+
# Delete any saved_data for given fiber
|
86
|
+
#
|
87
|
+
def release_data(fiber)
|
88
|
+
@saved_data.delete(fiber)
|
89
|
+
end
|
90
|
+
|
91
|
+
# Delete any saved_data held for dead fibers
|
92
|
+
#
|
93
|
+
def save_data_cleanup
|
94
|
+
@saved_data.dup.each do |k,v|
|
95
|
+
@saved_data.delete(k) if not k.alive?
|
96
|
+
end
|
97
|
+
@last_data_cleanup = Time.now
|
98
|
+
end
|
99
|
+
|
100
|
+
# Avoid method_missing stack for 'query'
|
35
101
|
#
|
36
102
|
def query(sql)
|
37
|
-
execute(
|
103
|
+
execute('query') do |conn|
|
38
104
|
conn.query sql
|
39
105
|
end
|
40
106
|
end
|
41
107
|
|
108
|
+
# True if the given connection is anywhere inside the pool
|
109
|
+
#
|
110
|
+
def has_connection?(conn)
|
111
|
+
(@available + @reserved.values).include?(conn)
|
112
|
+
end
|
42
113
|
|
114
|
+
# DEPRECATED: use with_failed_connection
|
43
115
|
def recreate_connection(new_conn)
|
44
|
-
|
45
|
-
|
116
|
+
with_failed_connection { new_conn }
|
117
|
+
end
|
118
|
+
|
119
|
+
# Identify the connection that just failed for current fiber.
|
120
|
+
# Pass it to the given block, which must return a valid instance of connection.
|
121
|
+
# After that, put the new connection into the pool in failed connection's place.
|
122
|
+
# Raises NoBackupConnection if cannot find the failed connection instance.
|
123
|
+
#
|
124
|
+
def with_failed_connection
|
125
|
+
f = Fiber.current
|
126
|
+
bad_conn = @reserved_backup[f]
|
127
|
+
raise NoBackupConnection.new if bad_conn.nil?
|
128
|
+
new_conn = yield bad_conn
|
129
|
+
release_backup f
|
46
130
|
@available.reject!{ |v| v == bad_conn }
|
47
131
|
@reserved.reject!{ |k,v| v == bad_conn }
|
48
132
|
@available.push new_conn
|
@@ -53,35 +137,62 @@ class FiberConnectionPool
|
|
53
137
|
end
|
54
138
|
end
|
55
139
|
|
140
|
+
# Delete any backups held for dead fibers
|
141
|
+
#
|
142
|
+
def backup_cleanup
|
143
|
+
@reserved_backup.dup.each do |k,v|
|
144
|
+
@reserved_backup.delete(k) if not k.alive?
|
145
|
+
end
|
146
|
+
@last_backup_cleanup = Time.now
|
147
|
+
end
|
148
|
+
|
56
149
|
private
|
57
150
|
|
58
151
|
# Choose first available connection and pass it to the supplied
|
59
|
-
# block. This will block indefinitely until there is an available
|
152
|
+
# block. This will block (yield) indefinitely until there is an available
|
60
153
|
# connection to service the request.
|
61
|
-
|
154
|
+
#
|
155
|
+
# After running the block, save requested data and release the connection.
|
156
|
+
#
|
157
|
+
def execute(method)
|
62
158
|
f = Fiber.current
|
63
|
-
|
64
159
|
begin
|
160
|
+
# get a connection and use it
|
65
161
|
conn = acquire(f)
|
66
162
|
retval = yield conn
|
67
|
-
|
68
|
-
|
69
|
-
|
70
|
-
|
163
|
+
|
164
|
+
# save anything requested
|
165
|
+
process_save_data(f, conn, method)
|
166
|
+
|
167
|
+
# successful run, release_backup
|
168
|
+
release_backup(f)
|
169
|
+
|
71
170
|
retval
|
72
171
|
ensure
|
73
|
-
release(f)
|
172
|
+
release(f)
|
74
173
|
end
|
75
174
|
end
|
76
175
|
|
77
|
-
#
|
78
|
-
#
|
79
|
-
#
|
80
|
-
|
176
|
+
# Run each save_data_block over the given connection
|
177
|
+
# and save the data for the given fiber.
|
178
|
+
# Also perform cleanup if TTL is past
|
179
|
+
#
|
180
|
+
def process_save_data(fiber, conn, method)
|
181
|
+
@save_data_requests.each do |key,block|
|
182
|
+
@saved_data[fiber] ||= {}
|
183
|
+
@saved_data[fiber][key] = block.call(conn, method)
|
184
|
+
end
|
185
|
+
# try cleanup
|
186
|
+
save_data_cleanup if (Time.now - @last_data_cleanup) >= SAVED_DATA_TTL_SECS
|
187
|
+
end
|
81
188
|
|
189
|
+
# Acquire a lock on a connection and assign it to given fiber
|
190
|
+
# If no connection is available, yield the given fiber on the pending array
|
191
|
+
#
|
192
|
+
def acquire(fiber)
|
82
193
|
if conn = @available.pop
|
83
194
|
@reserved[fiber.object_id] = conn
|
84
|
-
@reserved_backup[fiber
|
195
|
+
@reserved_backup[fiber] = conn
|
85
196
|
conn
|
86
197
|
else
|
87
198
|
Fiber.yield @pending.push fiber
|
@@ -90,13 +201,18 @@ class FiberConnectionPool
|
|
90
201
|
end
|
91
202
|
|
92
203
|
# Release connection from the backup hash
|
204
|
+
# Also perform cleanup if TTL is past
|
205
|
+
#
|
93
206
|
def release_backup(fiber)
|
94
|
-
@reserved_backup.delete(fiber
|
207
|
+
@reserved_backup.delete(fiber)
|
208
|
+
# try cleanup
|
209
|
+
backup_cleanup if (Time.now - @last_backup_cleanup) >= RESERVED_BACKUP_TTL_SECS
|
95
210
|
end
|
96
211
|
|
97
212
|
# Release connection assigned to the supplied fiber and
|
98
213
|
# resume any other pending connections (which will
|
99
214
|
# immediately try to run acquire on the pool)
|
215
|
+
#
|
100
216
|
def release(fiber)
|
101
217
|
@available.push(@reserved.delete(fiber.object_id)).compact!
|
102
218
|
|
@@ -107,29 +223,13 @@ class FiberConnectionPool
|
|
107
223
|
|
108
224
|
# Allow the pool to behave as the underlying connection
|
109
225
|
#
|
110
|
-
#
|
111
|
-
#
|
112
|
-
#
|
113
|
-
# yield the connection within execute method and release
|
114
|
-
# once it is complete (assumption: fiber will yield until
|
115
|
-
# data is available, or request is complete)
|
226
|
+
# Yield the connection within execute method and release
|
227
|
+
# once it is complete (assumption: fiber will yield while
|
228
|
+
# waiting for IO, allowing the reactor run other fibers)
|
116
229
|
#
|
117
230
|
def method_missing(method, *args, &blk)
|
118
|
-
|
119
|
-
|
120
|
-
execute(async,method) do |conn|
|
121
|
-
df = conn.send(method, *args, &blk)
|
122
|
-
|
123
|
-
if async
|
124
|
-
fiber = Fiber.current
|
125
|
-
df.callback do
|
126
|
-
release(fiber)
|
127
|
-
release_backup(fiber)
|
128
|
-
end
|
129
|
-
df.errback { release(fiber) }
|
130
|
-
end
|
131
|
-
|
132
|
-
df
|
231
|
+
execute(method) do |conn|
|
232
|
+
conn.send(method, *args, &blk)
|
133
233
|
end
|
134
234
|
end
|
135
235
|
end
|
@@ -1,4 +1,3 @@
|
|
1
|
-
Thread.abort_on_exception = true
|
2
1
|
require 'helper'
|
3
2
|
|
4
3
|
class TestFiberConnectionPool < Minitest::Test
|
@@ -6,11 +5,12 @@ class TestFiberConnectionPool < Minitest::Test
|
|
6
5
|
def test_blocking_behaviour
|
7
6
|
# get pool and fibers
|
8
7
|
pool = FiberConnectionPool.new(:size => 5) { ::BlockingConnection.new(:delay => 0.05) }
|
8
|
+
info = { :threads => [], :fibers => [], :instances => []}
|
9
9
|
|
10
|
-
fibers = Array.new(15){ Fiber.new { pool.do_something } }
|
10
|
+
fibers = Array.new(15){ Fiber.new { pool.do_something(info) } }
|
11
11
|
|
12
12
|
a = Time.now
|
13
|
-
|
13
|
+
fibers.each{ |f| f.resume }
|
14
14
|
b = Time.now
|
15
15
|
|
16
16
|
# 15 fibers on a size 5 pool, but -blocking- connections
|
@@ -20,36 +20,28 @@ class TestFiberConnectionPool < Minitest::Test
|
|
20
20
|
# Also we only use the first connection from the pool,
|
21
21
|
# because as we are -blocking- it's always available
|
22
22
|
# again for the next request
|
23
|
-
|
23
|
+
# we should have visited 1 thread, 15 fibers and 1 instances
|
24
|
+
info.dup.each{ |k,v| info[k] = v.uniq }
|
25
|
+
assert_equal 1, info[:threads].count
|
26
|
+
assert_equal 15, info[:fibers].count
|
27
|
+
assert_equal 1, info[:instances].count
|
24
28
|
end
|
25
29
|
|
26
30
|
def test_em_synchrony_behaviour
|
27
|
-
require 'em-synchrony'
|
28
|
-
|
29
|
-
a = b = nil
|
30
31
|
info = { :threads => [], :fibers => [], :instances => []}
|
31
32
|
|
32
|
-
|
33
|
-
|
34
|
-
pool = FiberConnectionPool.new(:size => 5) { ::EMSynchronyConnection.new(:delay => 0.05) }
|
33
|
+
# get pool and fibers
|
34
|
+
pool = FiberConnectionPool.new(:size => 5) { ::EMSynchronyConnection.new(:delay => 0.05) }
|
35
35
|
|
36
|
-
|
36
|
+
fibers = Array.new(15){ Fiber.new { pool.do_something(info) } }
|
37
37
|
|
38
|
-
|
39
|
-
fibers.each{ |f| f.resume }
|
40
|
-
# wait all fibers to end
|
41
|
-
while fibers.any?{ |f| f.alive? } do
|
42
|
-
EM::Synchrony.sleep 0.01
|
43
|
-
end
|
44
|
-
b = Time.now
|
45
|
-
EM.stop
|
46
|
-
end
|
38
|
+
lapse = run_em_reactor fibers
|
47
39
|
|
48
40
|
# 15 fibers on a size 5 pool, and -non-blocking- connections
|
49
41
|
# with a 0.05 delay we expect to spend at least: 0.05*15/5 = 0.15
|
50
42
|
# plus some breeze lost on precision on the wait loop
|
51
43
|
# then we should be under 0.20 for sure
|
52
|
-
assert_operator(
|
44
|
+
assert_operator(lapse, :<, 0.20)
|
53
45
|
|
54
46
|
# we should have visited 1 thread, 15 fibers and 5 instances
|
55
47
|
info.dup.each{ |k,v| info[k] = v.uniq }
|
@@ -58,6 +50,11 @@ class TestFiberConnectionPool < Minitest::Test
|
|
58
50
|
assert_equal 5, info[:instances].count
|
59
51
|
end
|
60
52
|
|
53
|
+
def test_celluloid_behaviour
|
54
|
+
skip 'Could not test celluloid 0.15.0pre, as it would not start reactor on test environment.
|
55
|
+
See the examples folder for a working celluloid (reel) server.'
|
56
|
+
end
|
57
|
+
|
61
58
|
def test_size_is_mandatory
|
62
59
|
assert_raises ArgumentError do
|
63
60
|
FiberConnectionPool.new { ::BlockingConnection.new }
|
@@ -70,5 +67,170 @@ class TestFiberConnectionPool < Minitest::Test
|
|
70
67
|
end
|
71
68
|
end
|
72
69
|
|
70
|
+
def test_failure_reaction
|
71
|
+
info = { :instances => [] }
|
72
|
+
|
73
|
+
# get pool and fibers
|
74
|
+
pool = FiberConnectionPool.new(:size => 5) { ::EMSynchronyConnection.new(:delay => 0.05) }
|
75
|
+
|
76
|
+
fibers = Array.new(14){ Fiber.new { pool.do_something(info) } }
|
77
|
+
|
78
|
+
failing_fiber = Fiber.new do
|
79
|
+
begin
|
80
|
+
pool.fail(info)
|
81
|
+
rescue
|
82
|
+
pool.with_failed_connection do |connection|
|
83
|
+
info[:repaired_connection] = connection
|
84
|
+
# replace it in the pool
|
85
|
+
::EMSynchronyConnection.new(:delay => 0.05)
|
86
|
+
end
|
87
|
+
end
|
88
|
+
end
|
89
|
+
# put it among others, not the first or the last
|
90
|
+
# so we see it does not mistake the failing connection
|
91
|
+
fibers.insert 7,failing_fiber
|
92
|
+
|
93
|
+
run_em_reactor fibers
|
94
|
+
|
95
|
+
# we should have visited 1 thread, 15 fibers and 6 instances (including failed)
|
96
|
+
info.dup.each{ |k,v| info[k] = v.uniq if v.is_a?(Array) }
|
97
|
+
assert_equal 6, info[:instances].count
|
98
|
+
|
99
|
+
# assert we do not lose track of failing connection
|
100
|
+
assert_equal info[:repaired_connection], info[:failing_connection]
|
101
|
+
|
102
|
+
# assert we replaced it
|
103
|
+
refute pool.has_connection?(info[:failing_connection])
|
104
|
+
|
105
|
+
# nothing left
|
106
|
+
assert_equal(0, pool.reserved_backup.count)
|
107
|
+
|
108
|
+
# if dealing with failed connection where you shouldn't...
|
109
|
+
assert_raises NoBackupConnection do
|
110
|
+
pool.with_failed_connection{ |c| 'boo' }
|
111
|
+
end
|
112
|
+
end
|
113
|
+
|
114
|
+
def test_reserved_backups
|
115
|
+
# create pool, run fibers and gather info
|
116
|
+
pool, info = run_reserved_backups
|
117
|
+
|
118
|
+
# one left
|
119
|
+
assert_equal(1, pool.reserved_backup.count)
|
120
|
+
|
121
|
+
# fire cleanup
|
122
|
+
pool.backup_cleanup
|
123
|
+
|
124
|
+
# nothing left
|
125
|
+
assert_equal(0, pool.reserved_backup.count)
|
126
|
+
|
127
|
+
# assert we did not replace it
|
128
|
+
assert pool.has_connection?(info[:failing_connection])
|
129
|
+
end
|
130
|
+
|
131
|
+
def test_auto_cleanup_reserved_backups
|
132
|
+
# lower ttl to force auto cleanup
|
133
|
+
prev_ttl = force_constant FiberConnectionPool, :RESERVED_BACKUP_TTL_SECS, 0
|
134
|
+
|
135
|
+
# create pool, run fibers and gather info
|
136
|
+
pool, info = run_reserved_backups
|
137
|
+
|
138
|
+
# nothing left, because failing fiber was not the last to run
|
139
|
+
# the following fiber made the cleanup
|
140
|
+
assert_equal(0, pool.reserved_backup.count)
|
141
|
+
|
142
|
+
# assert we did not replace it
|
143
|
+
assert pool.has_connection?(info[:failing_connection])
|
144
|
+
ensure
|
145
|
+
# restore
|
146
|
+
force_constant FiberConnectionPool, :RESERVED_BACKUP_TTL_SECS, prev_ttl
|
147
|
+
end
|
148
|
+
|
149
|
+
def test_save_data
|
150
|
+
# create pool, run fibers and gather info
|
151
|
+
pool, fibers, info = run_saved_data
|
152
|
+
|
153
|
+
# gathered data for all 4 fibers
|
154
|
+
assert fibers.all?{ |f| not pool.saved_data[f].nil? },
|
155
|
+
"fibers: #{fibers}, saved_data: #{pool.saved_data}"
|
156
|
+
|
157
|
+
# gathered 2 times each connection
|
158
|
+
connection_ids = pool.saved_data.values.map{ |v| v[:connection_id] }
|
159
|
+
assert info[:instances].all?{ |i| connection_ids.count(i) == 2 },
|
160
|
+
"info: #{info}, saved_data: #{pool.saved_data}"
|
161
|
+
|
162
|
+
# fire cleanup
|
163
|
+
pool.save_data_cleanup
|
164
|
+
|
165
|
+
# nothing left
|
166
|
+
assert_equal(0, pool.saved_data.count)
|
167
|
+
end
|
168
|
+
|
169
|
+
def test_auto_cleanup_saved_data
|
170
|
+
# lower ttl to force auto cleanup
|
171
|
+
prev_ttl = force_constant FiberConnectionPool, :SAVED_DATA_TTL_SECS, 0
|
172
|
+
|
173
|
+
# create pool, run fibers and gather info
|
174
|
+
pool, _, _ = run_saved_data
|
175
|
+
|
176
|
+
# only the last run left
|
177
|
+
# that fiber was the one making the cleanup, so it was still alive
|
178
|
+
assert_equal(1, pool.saved_data.count)
|
179
|
+
ensure
|
180
|
+
# restore
|
181
|
+
force_constant FiberConnectionPool, :SAVED_DATA_TTL_SECS, prev_ttl
|
182
|
+
end
|
183
|
+
|
184
|
+
private
|
185
|
+
|
186
|
+
def run_reserved_backups
|
187
|
+
info = { :instances => [] }
|
188
|
+
|
189
|
+
# get pool and fibers
|
190
|
+
pool = FiberConnectionPool.new(:size => 2) { ::EMSynchronyConnection.new(:delay => 0.05) }
|
191
|
+
|
192
|
+
fibers = Array.new(4){ Fiber.new { pool.do_something(info) } }
|
193
|
+
|
194
|
+
# we do not repair it, backup associated with this Fiber stays in the pool
|
195
|
+
failing_fiber = Fiber.new { pool.fail(info) rescue nil }
|
196
|
+
|
197
|
+
# put it among others, not the first or the last
|
198
|
+
# so we see it does not mistake the failing connection
|
199
|
+
fibers.insert 2,failing_fiber
|
200
|
+
|
201
|
+
run_em_reactor fibers
|
202
|
+
|
203
|
+
# we should have visited only 2 instances (no instance added by repairing broken one)
|
204
|
+
info.dup.each{ |k,v| info[k] = v.uniq if v.is_a?(Array) }
|
205
|
+
assert_equal 2, info[:instances].count
|
206
|
+
|
207
|
+
[ pool, info ]
|
208
|
+
end
|
209
|
+
|
210
|
+
def run_saved_data
|
211
|
+
info = { :instances => [] }
|
212
|
+
|
213
|
+
# get pool and fibers
|
214
|
+
pool = FiberConnectionPool.new(:size => 2) { ::EMSynchronyConnection.new(:delay => 0.05) }
|
215
|
+
|
216
|
+
# ask to save some data
|
217
|
+
pool.save_data(:connection_id) { |conn| conn.object_id }
|
218
|
+
pool.save_data(:fiber_id) { |conn| Fiber.current.object_id }
|
219
|
+
|
220
|
+
fibers = Array.new(4) do
|
221
|
+
Fiber.new do
|
222
|
+
pool.do_something(info)
|
223
|
+
assert_equal Fiber.current.object_id, pool.gathered_data[:fiber_id]
|
224
|
+
end
|
225
|
+
end
|
226
|
+
|
227
|
+
run_em_reactor fibers
|
228
|
+
|
229
|
+
# we should have visited 2 instances
|
230
|
+
info.dup.each{ |k,v| info[k] = v.uniq if v.is_a?(Array) }
|
231
|
+
assert_equal 2, info[:instances].count
|
232
|
+
|
233
|
+
[ pool, fibers, info ]
|
234
|
+
end
|
73
235
|
|
74
236
|
end
|
data/test/helper.rb
CHANGED
@@ -1,5 +1,6 @@
|
|
1
1
|
require 'minitest/pride'
|
2
2
|
require 'minitest/autorun'
|
3
|
+
require 'em-synchrony'
|
3
4
|
|
4
5
|
require_relative '../lib/fiber_connection_pool'
|
5
6
|
|
@@ -8,17 +9,50 @@ class BlockingConnection
|
|
8
9
|
@delay = opts[:delay] || 0.05
|
9
10
|
end
|
10
11
|
|
11
|
-
def do_something
|
12
|
+
def do_something(info = {})
|
13
|
+
fill_info info
|
12
14
|
sleep @delay
|
13
|
-
|
15
|
+
end
|
16
|
+
|
17
|
+
def fill_info(info = {})
|
18
|
+
info[:threads] << Thread.current.object_id if info[:threads]
|
19
|
+
info[:fibers] << Fiber.current.object_id if info[:fibers]
|
20
|
+
info[:instances] << self.object_id if info[:instances]
|
14
21
|
end
|
15
22
|
end
|
16
23
|
|
17
24
|
class EMSynchronyConnection < BlockingConnection
|
18
|
-
def do_something(info)
|
19
|
-
info
|
20
|
-
info[:fibers] << Fiber.current.object_id
|
21
|
-
info[:instances] << self.object_id
|
25
|
+
def do_something(info = {})
|
26
|
+
fill_info info
|
22
27
|
EM::Synchrony.sleep @delay
|
23
28
|
end
|
29
|
+
|
30
|
+
def fail(info)
|
31
|
+
fill_info info
|
32
|
+
info[:failing_connection] = self
|
33
|
+
raise "Sadly failing here..."
|
34
|
+
end
|
35
|
+
end
|
36
|
+
|
37
|
+
# start an EM reactor and run given fibers
|
38
|
+
# return time spent
|
39
|
+
def run_em_reactor(fibers)
|
40
|
+
a = b = nil
|
41
|
+
EM.synchrony do
|
42
|
+
a = Time.now
|
43
|
+
fibers.each{ |f| f.resume }
|
44
|
+
# wait all fibers to end
|
45
|
+
while fibers.any?{ |f| f.alive? } do
|
46
|
+
EM::Synchrony.sleep 0.01
|
47
|
+
end
|
48
|
+
b = Time.now
|
49
|
+
EM.stop
|
50
|
+
end
|
51
|
+
b-a
|
52
|
+
end
|
53
|
+
|
54
|
+
def force_constant(klass, name, value)
|
55
|
+
previous_value = klass.send(:remove_const, name)
|
56
|
+
klass.const_set name.to_s, value
|
57
|
+
previous_value
|
24
58
|
end
|
metadata
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: fiber_connection_pool
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 0.
|
4
|
+
version: 0.2.0
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Ruben Caro
|
@@ -9,7 +9,7 @@ authors:
|
|
9
9
|
autorequire:
|
10
10
|
bindir: bin
|
11
11
|
cert_chain: []
|
12
|
-
date: 2013-08-
|
12
|
+
date: 2013-08-20 00:00:00.000000000 Z
|
13
13
|
dependencies:
|
14
14
|
- !ruby/object:Gem::Dependency
|
15
15
|
name: minitest
|
@@ -44,7 +44,7 @@ description: |-
|
|
44
44
|
non-blocking IO behaviour on the same thread
|
45
45
|
as provided by EventMachine or Celluloid.
|
46
46
|
email:
|
47
|
-
- ruben@lanuez.org
|
47
|
+
- ruben.caro@lanuez.org
|
48
48
|
executables: []
|
49
49
|
extensions: []
|
50
50
|
extra_rdoc_files: []
|
@@ -63,6 +63,7 @@ files:
|
|
63
63
|
- examples/reel_server/main.rb
|
64
64
|
- fiber_connection_pool.gemspec
|
65
65
|
- lib/fiber_connection_pool.rb
|
66
|
+
- lib/fiber_connection_pool/exceptions.rb
|
66
67
|
- test/fiber_connection_pool_test.rb
|
67
68
|
- test/helper.rb
|
68
69
|
homepage: https://github.com/rubencaro/fiber_connection_pool
|
@@ -85,9 +86,11 @@ required_rubygems_version: !ruby/object:Gem::Requirement
|
|
85
86
|
version: '0'
|
86
87
|
requirements: []
|
87
88
|
rubyforge_project:
|
88
|
-
rubygems_version: 2.0.
|
89
|
+
rubygems_version: 2.0.6
|
89
90
|
signing_key:
|
90
91
|
specification_version: 4
|
91
92
|
summary: Fiber-based generic connection pool for Ruby
|
92
|
-
test_files:
|
93
|
+
test_files:
|
94
|
+
- test/fiber_connection_pool_test.rb
|
95
|
+
- test/helper.rb
|
93
96
|
has_rdoc:
|