fiber_connection_pool 0.1.1 → 0.1.2
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/README.md +83 -4
- data/examples/goliath_server/Gemfile +5 -0
- data/examples/goliath_server/config/main.rb +13 -0
- data/examples/goliath_server/main.rb +13 -0
- data/lib/fiber_connection_pool.rb +67 -64
- metadata +5 -2
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA1:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 654c0a80fecfc77daa838a3a0611ad534e4413fc
|
4
|
+
data.tar.gz: cbb3bd2ccb7f50fc534d98f976cfe2ec42c87302
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 4f01cfc453ecfc286069f18ac34545c079a64214b2e99dee2a2ef3ef6c1414e72c2da586d653d23e61a35d363c62f28bcd45f160bab9d5d93323cf92e2d9765a
|
7
|
+
data.tar.gz: c0995f388b1552a246cc9f68ab1ebcc929cac66a088b0aa1a4d0743bae84b5f34f853e34b0164d1c323e458e81be4a249c0d0d52fe7fc9347b8e610ac76af2a2
|
data/README.md
CHANGED
@@ -6,17 +6,20 @@ fiber_connection_pool
|
|
6
6
|
|
7
7
|
Fiber-based generic connection pool
|
8
8
|
|
9
|
+
A connection pool meant to be used inside a Fiber-based _reactor_,
|
10
|
+
such as any [EventMachine](https://github.com/eventmachine/eventmachine)
|
11
|
+
or [Celluloid](http://celluloid.io/) server.
|
12
|
+
|
9
13
|
Widely based on `ConnectionPool`
|
10
14
|
from [em-synchrony](https://github.com/igrigorik/em-synchrony) gem, and
|
11
15
|
some things borrowed also from
|
12
16
|
threaded [connection_pool](https://github.com/mperham/connection_pool) gem.
|
13
|
-
|
14
17
|
Used in production environments
|
15
18
|
with [Goliath](https://github.com/postrank-labs/goliath)
|
16
|
-
(
|
19
|
+
(EventMachine based) servers,
|
17
20
|
and in promising experiments with
|
18
21
|
[Reel](https://github.com/celluloid/reel)
|
19
|
-
(
|
22
|
+
(Celluloid based) servers.
|
20
23
|
|
21
24
|
Install
|
22
25
|
----------------
|
@@ -33,10 +36,86 @@ Inside of your Ruby program, require FiberConnectionPool with:
|
|
33
36
|
|
34
37
|
require 'fiber_connection_pool'
|
35
38
|
|
39
|
+
How It Works
|
40
|
+
-------------------
|
41
|
+
|
42
|
+
``` ruby
|
43
|
+
pool = FiberConnectionPool.new(:size => 5){ MyFancyConnection.new }
|
44
|
+
```
|
45
|
+
|
46
|
+
It just keeps an array (the internal pool) holding the result of running
|
47
|
+
the given block _size_ times. Inside the reactor loop (either EventMachine's or Celluloid's),
|
48
|
+
each request is wrapped on a Fiber, and then `pool` plays its magic.
|
49
|
+
|
50
|
+
When a method `query_me` is called on `pool` and it's not one of its own methods,
|
51
|
+
then it:
|
52
|
+
|
53
|
+
1. reserves one connection from the internal pool and associates it __with the current Fiber__
|
54
|
+
2. if no connection is available, then that Fiber stays on a _pending_ queue, and __is yielded__
|
55
|
+
3. when a connection is available, then the pool calls `query_me` on that `MyFancyConnection` instance
|
56
|
+
4. when `query_me` returns, the reserved instance is released again,
|
57
|
+
and the next Fiber on the _pending_ queue __is resumed__
|
58
|
+
5. the return value is sent back to the caller
|
59
|
+
|
60
|
+
Methods from `MyFancyConnection` instance should yield the fiber before
|
61
|
+
perform any blocking IO. That returns control to te underlying reactor,
|
62
|
+
that spawns another fiber to process the next request, while the previous
|
63
|
+
one is still waiting for the IO response. That new fiber will get its own
|
64
|
+
connection from the pool, or else it will yield until there
|
65
|
+
is one available.
|
66
|
+
|
67
|
+
The whole process looks synchronous from the Fiber perspective, _because it is_.
|
68
|
+
The Fiber will really block ( _yield_ ) until it gets the result.
|
69
|
+
|
70
|
+
``` ruby
|
71
|
+
results = pool.query_me(sql)
|
72
|
+
puts "I waited for this: #{results}"
|
73
|
+
```
|
74
|
+
|
75
|
+
The magic resides on the fact that other fibers are being processed while this one is waiting.
|
76
|
+
|
77
|
+
Not thread-safe
|
78
|
+
------------------
|
79
|
+
|
80
|
+
`FiberConnectionPool` is not thread-safe right now. You will not be able to use it
|
81
|
+
from different threads, as eventually it will try to resume a Fiber that resides
|
82
|
+
on a different Thread. That will raise a FiberError( _"calling a fiber across threads"_ ).
|
83
|
+
Maybe one day we add that feature too.
|
84
|
+
|
85
|
+
We have tested it on Goliath servers having one pool on each server instance, and on Reel servers
|
86
|
+
having one pool on each Actor thread. Take a look at the `examples` folder for details.
|
87
|
+
|
88
|
+
MySQL specific
|
89
|
+
------------------
|
90
|
+
|
91
|
+
By now we have only thought and tested it to be used with MySQL connections.
|
92
|
+
For EventMachine by using `Mysql2::EM::Client` from [em-synchrony](https://github.com/igrigorik/em-synchrony).
|
93
|
+
And for Celluloid by using a patched version of [ruby-mysql](https://github.com/rubencaro/ruby-mysql).
|
94
|
+
We plan on removing any MySQL specific code, so it becomes completely generic. Does not seem so hard to achieve.
|
95
|
+
|
96
|
+
Reacting to connection failure
|
97
|
+
------------------
|
98
|
+
|
99
|
+
When the call to a method raises an Exception it will raise as if there was no pool between
|
100
|
+
your code and the connetion itself. You can rescue the Exception as usual and
|
101
|
+
react as you would do normally.
|
102
|
+
|
103
|
+
You have to be aware that the connection instance will remain in the pool, and other fibers
|
104
|
+
will surely use it. If the Exception you rescued indicates that the connection should be
|
105
|
+
recreated, you can call `recreate_connection` passing it a new instance. The instance that
|
106
|
+
just failed will be replaced inside the pool by the brand new connection.
|
107
|
+
|
36
108
|
Supported Platforms
|
37
109
|
-------------------
|
38
110
|
|
39
111
|
Used in production environments on Ruby 1.9.3 and 2.0.0.
|
40
112
|
Tested against Ruby 1.9.3, 2.0.0, and rbx-19mode ([See details..](http://travis-ci.org/rubencaro/fiber_connection_pool)).
|
41
113
|
|
42
|
-
|
114
|
+
TODOS
|
115
|
+
-------------------
|
116
|
+
|
117
|
+
* no MySQL-specific code
|
118
|
+
* better testing
|
119
|
+
* improve reaction to failure
|
120
|
+
* better in-code docs
|
121
|
+
* make thread-safe
|
@@ -0,0 +1,13 @@
|
|
1
|
+
require 'fiber_connection_pool'
|
2
|
+
require 'mysql2'
|
3
|
+
require 'em-synchrony'
|
4
|
+
require 'em-synchrony/mysql2'
|
5
|
+
|
6
|
+
config['db'] = FiberConnectionPool.new(:size => 5) do
|
7
|
+
Mysql2::EM::Client.new({
|
8
|
+
host: 'localhost',
|
9
|
+
username: 'user',
|
10
|
+
password: 'pass',
|
11
|
+
database: 'bogusdb'
|
12
|
+
})
|
13
|
+
end
|
@@ -0,0 +1,13 @@
|
|
1
|
+
require 'goliath'
|
2
|
+
require 'fiber'
|
3
|
+
|
4
|
+
class Main < Goliath::API
|
5
|
+
|
6
|
+
def response(env)
|
7
|
+
print '.'
|
8
|
+
db.query 'select sleep(2);'
|
9
|
+
puts "Done #{Thread.current.to_s}, #{Fiber.current.to_s}"
|
10
|
+
[200,{"Content-Type" => "text/html"},"hello, world! #{Time.now.strftime('%T')}"]
|
11
|
+
end
|
12
|
+
|
13
|
+
end
|
@@ -1,7 +1,7 @@
|
|
1
1
|
require 'fiber'
|
2
2
|
|
3
3
|
class FiberConnectionPool
|
4
|
-
VERSION = '0.1.
|
4
|
+
VERSION = '0.1.2'
|
5
5
|
|
6
6
|
attr_accessor :saved_data
|
7
7
|
|
@@ -30,22 +30,6 @@ class FiberConnectionPool
|
|
30
30
|
@saved_data.delete Fiber.current.object_id
|
31
31
|
end
|
32
32
|
|
33
|
-
# Choose first available connection and pass it to the supplied
|
34
|
-
# block. This will block indefinitely until there is an available
|
35
|
-
# connection to service the request.
|
36
|
-
def execute(async,method)
|
37
|
-
f = Fiber.current
|
38
|
-
|
39
|
-
begin
|
40
|
-
conn = acquire(f)
|
41
|
-
retval = yield conn
|
42
|
-
release_backup(f) if !async and method == 'query'
|
43
|
-
retval
|
44
|
-
ensure
|
45
|
-
release(f) if not async
|
46
|
-
end
|
47
|
-
end
|
48
|
-
|
49
33
|
##
|
50
34
|
# avoid method_missing for most common methods
|
51
35
|
#
|
@@ -71,62 +55,81 @@ class FiberConnectionPool
|
|
71
55
|
|
72
56
|
private
|
73
57
|
|
74
|
-
|
75
|
-
|
76
|
-
|
77
|
-
|
78
|
-
|
79
|
-
|
80
|
-
|
81
|
-
|
82
|
-
|
83
|
-
|
84
|
-
Fiber.
|
85
|
-
acquire(fiber)
|
58
|
+
# Choose first available connection and pass it to the supplied
|
59
|
+
# block. This will block indefinitely until there is an available
|
60
|
+
# connection to service the request.
|
61
|
+
def execute(async,method)
|
62
|
+
f = Fiber.current
|
63
|
+
|
64
|
+
begin
|
65
|
+
conn = acquire(f)
|
66
|
+
retval = yield conn
|
67
|
+
if !@saved_data[Fiber.current.object_id].nil?
|
68
|
+
@saved_data[Fiber.current.object_id]['affected_rows'] = conn.affected_rows
|
86
69
|
end
|
70
|
+
release_backup(f) if !async and method == 'query'
|
71
|
+
retval
|
72
|
+
ensure
|
73
|
+
release(f) if not async
|
87
74
|
end
|
75
|
+
end
|
88
76
|
|
89
|
-
|
90
|
-
|
91
|
-
|
77
|
+
# Acquire a lock on a connection and assign it to executing fiber
|
78
|
+
# - if connection is available, pass it back to the calling block
|
79
|
+
# - if pool is full, yield the current fiber until connection is available
|
80
|
+
def acquire(fiber)
|
81
|
+
|
82
|
+
if conn = @available.pop
|
83
|
+
@reserved[fiber.object_id] = conn
|
84
|
+
@reserved_backup[fiber.object_id] = conn
|
85
|
+
conn
|
86
|
+
else
|
87
|
+
Fiber.yield @pending.push fiber
|
88
|
+
acquire(fiber)
|
92
89
|
end
|
90
|
+
end
|
93
91
|
|
94
|
-
|
95
|
-
|
96
|
-
|
97
|
-
|
98
|
-
@available.push(@reserved.delete(fiber.object_id)).compact!
|
92
|
+
# Release connection from the backup hash
|
93
|
+
def release_backup(fiber)
|
94
|
+
@reserved_backup.delete(fiber.object_id)
|
95
|
+
end
|
99
96
|
|
100
|
-
|
101
|
-
|
102
|
-
|
97
|
+
# Release connection assigned to the supplied fiber and
|
98
|
+
# resume any other pending connections (which will
|
99
|
+
# immediately try to run acquire on the pool)
|
100
|
+
def release(fiber)
|
101
|
+
@available.push(@reserved.delete(fiber.object_id)).compact!
|
102
|
+
|
103
|
+
if pending = @pending.shift
|
104
|
+
pending.resume
|
103
105
|
end
|
106
|
+
end
|
104
107
|
|
105
|
-
|
106
|
-
|
107
|
-
|
108
|
-
|
109
|
-
|
110
|
-
|
111
|
-
|
112
|
-
|
113
|
-
|
114
|
-
|
115
|
-
|
116
|
-
|
117
|
-
|
118
|
-
|
119
|
-
|
120
|
-
if async
|
121
|
-
fiber = Fiber.current
|
122
|
-
df.callback do
|
123
|
-
release(fiber)
|
124
|
-
release_backup(fiber)
|
125
|
-
end
|
126
|
-
df.errback { release(fiber) }
|
127
|
-
end
|
108
|
+
# Allow the pool to behave as the underlying connection
|
109
|
+
#
|
110
|
+
# If the requesting method begins with "a" prefix, then
|
111
|
+
# hijack the callbacks and errbacks to fire a connection
|
112
|
+
# pool release whenever the request is complete. Otherwise
|
113
|
+
# yield the connection within execute method and release
|
114
|
+
# once it is complete (assumption: fiber will yield until
|
115
|
+
# data is available, or request is complete)
|
116
|
+
#
|
117
|
+
def method_missing(method, *args, &blk)
|
118
|
+
async = (method[0,1] == "a")
|
119
|
+
|
120
|
+
execute(async,method) do |conn|
|
121
|
+
df = conn.send(method, *args, &blk)
|
128
122
|
|
129
|
-
|
123
|
+
if async
|
124
|
+
fiber = Fiber.current
|
125
|
+
df.callback do
|
126
|
+
release(fiber)
|
127
|
+
release_backup(fiber)
|
128
|
+
end
|
129
|
+
df.errback { release(fiber) }
|
130
130
|
end
|
131
|
+
|
132
|
+
df
|
131
133
|
end
|
134
|
+
end
|
132
135
|
end
|
metadata
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: fiber_connection_pool
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 0.1.
|
4
|
+
version: 0.1.2
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Ruben Caro
|
@@ -9,7 +9,7 @@ authors:
|
|
9
9
|
autorequire:
|
10
10
|
bindir: bin
|
11
11
|
cert_chain: []
|
12
|
-
date: 2013-08-
|
12
|
+
date: 2013-08-16 00:00:00.000000000 Z
|
13
13
|
dependencies:
|
14
14
|
- !ruby/object:Gem::Dependency
|
15
15
|
name: minitest
|
@@ -56,6 +56,9 @@ files:
|
|
56
56
|
- README.md
|
57
57
|
- Rakefile
|
58
58
|
- examples/.placeholder
|
59
|
+
- examples/goliath_server/Gemfile
|
60
|
+
- examples/goliath_server/config/main.rb
|
61
|
+
- examples/goliath_server/main.rb
|
59
62
|
- examples/reel_server/Gemfile
|
60
63
|
- examples/reel_server/main.rb
|
61
64
|
- fiber_connection_pool.gemspec
|