fiber_connection_pool 0.1.1 → 0.1.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/README.md +83 -4
- data/examples/goliath_server/Gemfile +5 -0
- data/examples/goliath_server/config/main.rb +13 -0
- data/examples/goliath_server/main.rb +13 -0
- data/lib/fiber_connection_pool.rb +67 -64
- metadata +5 -2
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA1:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 654c0a80fecfc77daa838a3a0611ad534e4413fc
|
4
|
+
data.tar.gz: cbb3bd2ccb7f50fc534d98f976cfe2ec42c87302
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 4f01cfc453ecfc286069f18ac34545c079a64214b2e99dee2a2ef3ef6c1414e72c2da586d653d23e61a35d363c62f28bcd45f160bab9d5d93323cf92e2d9765a
|
7
|
+
data.tar.gz: c0995f388b1552a246cc9f68ab1ebcc929cac66a088b0aa1a4d0743bae84b5f34f853e34b0164d1c323e458e81be4a249c0d0d52fe7fc9347b8e610ac76af2a2
|
data/README.md
CHANGED
@@ -6,17 +6,20 @@ fiber_connection_pool
|
|
6
6
|
|
7
7
|
Fiber-based generic connection pool
|
8
8
|
|
9
|
+
A connection pool meant to be used inside a Fiber-based _reactor_,
|
10
|
+
such as any [EventMachine](https://github.com/eventmachine/eventmachine)
|
11
|
+
or [Celluloid](http://celluloid.io/) server.
|
12
|
+
|
9
13
|
Widely based on `ConnectionPool`
|
10
14
|
from [em-synchrony](https://github.com/igrigorik/em-synchrony) gem, and
|
11
15
|
some things borrowed also from
|
12
16
|
threaded [connection_pool](https://github.com/mperham/connection_pool) gem.
|
13
|
-
|
14
17
|
Used in production environments
|
15
18
|
with [Goliath](https://github.com/postrank-labs/goliath)
|
16
|
-
(
|
19
|
+
(EventMachine based) servers,
|
17
20
|
and in promising experiments with
|
18
21
|
[Reel](https://github.com/celluloid/reel)
|
19
|
-
(
|
22
|
+
(Celluloid based) servers.
|
20
23
|
|
21
24
|
Install
|
22
25
|
----------------
|
@@ -33,10 +36,86 @@ Inside of your Ruby program, require FiberConnectionPool with:
|
|
33
36
|
|
34
37
|
require 'fiber_connection_pool'
|
35
38
|
|
39
|
+
How It Works
|
40
|
+
-------------------
|
41
|
+
|
42
|
+
``` ruby
|
43
|
+
pool = FiberConnectionPool.new(:size => 5){ MyFancyConnection.new }
|
44
|
+
```
|
45
|
+
|
46
|
+
It just keeps an array (the internal pool) holding the result of running
|
47
|
+
the given block _size_ times. Inside the reactor loop (either EventMachine's or Celluloid's),
|
48
|
+
each request is wrapped on a Fiber, and then `pool` plays its magic.
|
49
|
+
|
50
|
+
When a method `query_me` is called on `pool` and it's not one of its own methods,
|
51
|
+
then it:
|
52
|
+
|
53
|
+
1. reserves one connection from the internal pool and associates it __with the current Fiber__
|
54
|
+
2. if no connection is available, then that Fiber stays on a _pending_ queue, and __is yielded__
|
55
|
+
3. when a connection is available, then the pool calls `query_me` on that `MyFancyConnection` instance
|
56
|
+
4. when `query_me` returns, the reserved instance is released again,
|
57
|
+
and the next Fiber on the _pending_ queue __is resumed__
|
58
|
+
5. the return value is sent back to the caller
|
59
|
+
|
60
|
+
Methods from `MyFancyConnection` instance should yield the fiber before
|
61
|
+
perform any blocking IO. That returns control to te underlying reactor,
|
62
|
+
that spawns another fiber to process the next request, while the previous
|
63
|
+
one is still waiting for the IO response. That new fiber will get its own
|
64
|
+
connection from the pool, or else it will yield until there
|
65
|
+
is one available.
|
66
|
+
|
67
|
+
The whole process looks synchronous from the Fiber perspective, _because it is_.
|
68
|
+
The Fiber will really block ( _yield_ ) until it gets the result.
|
69
|
+
|
70
|
+
``` ruby
|
71
|
+
results = pool.query_me(sql)
|
72
|
+
puts "I waited for this: #{results}"
|
73
|
+
```
|
74
|
+
|
75
|
+
The magic resides on the fact that other fibers are being processed while this one is waiting.
|
76
|
+
|
77
|
+
Not thread-safe
|
78
|
+
------------------
|
79
|
+
|
80
|
+
`FiberConnectionPool` is not thread-safe right now. You will not be able to use it
|
81
|
+
from different threads, as eventually it will try to resume a Fiber that resides
|
82
|
+
on a different Thread. That will raise a FiberError( _"calling a fiber across threads"_ ).
|
83
|
+
Maybe one day we add that feature too.
|
84
|
+
|
85
|
+
We have tested it on Goliath servers having one pool on each server instance, and on Reel servers
|
86
|
+
having one pool on each Actor thread. Take a look at the `examples` folder for details.
|
87
|
+
|
88
|
+
MySQL specific
|
89
|
+
------------------
|
90
|
+
|
91
|
+
By now we have only thought and tested it to be used with MySQL connections.
|
92
|
+
For EventMachine by using `Mysql2::EM::Client` from [em-synchrony](https://github.com/igrigorik/em-synchrony).
|
93
|
+
And for Celluloid by using a patched version of [ruby-mysql](https://github.com/rubencaro/ruby-mysql).
|
94
|
+
We plan on removing any MySQL specific code, so it becomes completely generic. Does not seem so hard to achieve.
|
95
|
+
|
96
|
+
Reacting to connection failure
|
97
|
+
------------------
|
98
|
+
|
99
|
+
When the call to a method raises an Exception it will raise as if there was no pool between
|
100
|
+
your code and the connetion itself. You can rescue the Exception as usual and
|
101
|
+
react as you would do normally.
|
102
|
+
|
103
|
+
You have to be aware that the connection instance will remain in the pool, and other fibers
|
104
|
+
will surely use it. If the Exception you rescued indicates that the connection should be
|
105
|
+
recreated, you can call `recreate_connection` passing it a new instance. The instance that
|
106
|
+
just failed will be replaced inside the pool by the brand new connection.
|
107
|
+
|
36
108
|
Supported Platforms
|
37
109
|
-------------------
|
38
110
|
|
39
111
|
Used in production environments on Ruby 1.9.3 and 2.0.0.
|
40
112
|
Tested against Ruby 1.9.3, 2.0.0, and rbx-19mode ([See details..](http://travis-ci.org/rubencaro/fiber_connection_pool)).
|
41
113
|
|
42
|
-
|
114
|
+
TODOS
|
115
|
+
-------------------
|
116
|
+
|
117
|
+
* no MySQL-specific code
|
118
|
+
* better testing
|
119
|
+
* improve reaction to failure
|
120
|
+
* better in-code docs
|
121
|
+
* make thread-safe
|
@@ -0,0 +1,13 @@
|
|
1
|
+
require 'fiber_connection_pool'
|
2
|
+
require 'mysql2'
|
3
|
+
require 'em-synchrony'
|
4
|
+
require 'em-synchrony/mysql2'
|
5
|
+
|
6
|
+
config['db'] = FiberConnectionPool.new(:size => 5) do
|
7
|
+
Mysql2::EM::Client.new({
|
8
|
+
host: 'localhost',
|
9
|
+
username: 'user',
|
10
|
+
password: 'pass',
|
11
|
+
database: 'bogusdb'
|
12
|
+
})
|
13
|
+
end
|
@@ -0,0 +1,13 @@
|
|
1
|
+
require 'goliath'
|
2
|
+
require 'fiber'
|
3
|
+
|
4
|
+
class Main < Goliath::API
|
5
|
+
|
6
|
+
def response(env)
|
7
|
+
print '.'
|
8
|
+
db.query 'select sleep(2);'
|
9
|
+
puts "Done #{Thread.current.to_s}, #{Fiber.current.to_s}"
|
10
|
+
[200,{"Content-Type" => "text/html"},"hello, world! #{Time.now.strftime('%T')}"]
|
11
|
+
end
|
12
|
+
|
13
|
+
end
|
@@ -1,7 +1,7 @@
|
|
1
1
|
require 'fiber'
|
2
2
|
|
3
3
|
class FiberConnectionPool
|
4
|
-
VERSION = '0.1.
|
4
|
+
VERSION = '0.1.2'
|
5
5
|
|
6
6
|
attr_accessor :saved_data
|
7
7
|
|
@@ -30,22 +30,6 @@ class FiberConnectionPool
|
|
30
30
|
@saved_data.delete Fiber.current.object_id
|
31
31
|
end
|
32
32
|
|
33
|
-
# Choose first available connection and pass it to the supplied
|
34
|
-
# block. This will block indefinitely until there is an available
|
35
|
-
# connection to service the request.
|
36
|
-
def execute(async,method)
|
37
|
-
f = Fiber.current
|
38
|
-
|
39
|
-
begin
|
40
|
-
conn = acquire(f)
|
41
|
-
retval = yield conn
|
42
|
-
release_backup(f) if !async and method == 'query'
|
43
|
-
retval
|
44
|
-
ensure
|
45
|
-
release(f) if not async
|
46
|
-
end
|
47
|
-
end
|
48
|
-
|
49
33
|
##
|
50
34
|
# avoid method_missing for most common methods
|
51
35
|
#
|
@@ -71,62 +55,81 @@ class FiberConnectionPool
|
|
71
55
|
|
72
56
|
private
|
73
57
|
|
74
|
-
|
75
|
-
|
76
|
-
|
77
|
-
|
78
|
-
|
79
|
-
|
80
|
-
|
81
|
-
|
82
|
-
|
83
|
-
|
84
|
-
Fiber.
|
85
|
-
acquire(fiber)
|
58
|
+
# Choose first available connection and pass it to the supplied
|
59
|
+
# block. This will block indefinitely until there is an available
|
60
|
+
# connection to service the request.
|
61
|
+
def execute(async,method)
|
62
|
+
f = Fiber.current
|
63
|
+
|
64
|
+
begin
|
65
|
+
conn = acquire(f)
|
66
|
+
retval = yield conn
|
67
|
+
if !@saved_data[Fiber.current.object_id].nil?
|
68
|
+
@saved_data[Fiber.current.object_id]['affected_rows'] = conn.affected_rows
|
86
69
|
end
|
70
|
+
release_backup(f) if !async and method == 'query'
|
71
|
+
retval
|
72
|
+
ensure
|
73
|
+
release(f) if not async
|
87
74
|
end
|
75
|
+
end
|
88
76
|
|
89
|
-
|
90
|
-
|
91
|
-
|
77
|
+
# Acquire a lock on a connection and assign it to executing fiber
|
78
|
+
# - if connection is available, pass it back to the calling block
|
79
|
+
# - if pool is full, yield the current fiber until connection is available
|
80
|
+
def acquire(fiber)
|
81
|
+
|
82
|
+
if conn = @available.pop
|
83
|
+
@reserved[fiber.object_id] = conn
|
84
|
+
@reserved_backup[fiber.object_id] = conn
|
85
|
+
conn
|
86
|
+
else
|
87
|
+
Fiber.yield @pending.push fiber
|
88
|
+
acquire(fiber)
|
92
89
|
end
|
90
|
+
end
|
93
91
|
|
94
|
-
|
95
|
-
|
96
|
-
|
97
|
-
|
98
|
-
@available.push(@reserved.delete(fiber.object_id)).compact!
|
92
|
+
# Release connection from the backup hash
|
93
|
+
def release_backup(fiber)
|
94
|
+
@reserved_backup.delete(fiber.object_id)
|
95
|
+
end
|
99
96
|
|
100
|
-
|
101
|
-
|
102
|
-
|
97
|
+
# Release connection assigned to the supplied fiber and
|
98
|
+
# resume any other pending connections (which will
|
99
|
+
# immediately try to run acquire on the pool)
|
100
|
+
def release(fiber)
|
101
|
+
@available.push(@reserved.delete(fiber.object_id)).compact!
|
102
|
+
|
103
|
+
if pending = @pending.shift
|
104
|
+
pending.resume
|
103
105
|
end
|
106
|
+
end
|
104
107
|
|
105
|
-
|
106
|
-
|
107
|
-
|
108
|
-
|
109
|
-
|
110
|
-
|
111
|
-
|
112
|
-
|
113
|
-
|
114
|
-
|
115
|
-
|
116
|
-
|
117
|
-
|
118
|
-
|
119
|
-
|
120
|
-
if async
|
121
|
-
fiber = Fiber.current
|
122
|
-
df.callback do
|
123
|
-
release(fiber)
|
124
|
-
release_backup(fiber)
|
125
|
-
end
|
126
|
-
df.errback { release(fiber) }
|
127
|
-
end
|
108
|
+
# Allow the pool to behave as the underlying connection
|
109
|
+
#
|
110
|
+
# If the requesting method begins with "a" prefix, then
|
111
|
+
# hijack the callbacks and errbacks to fire a connection
|
112
|
+
# pool release whenever the request is complete. Otherwise
|
113
|
+
# yield the connection within execute method and release
|
114
|
+
# once it is complete (assumption: fiber will yield until
|
115
|
+
# data is available, or request is complete)
|
116
|
+
#
|
117
|
+
def method_missing(method, *args, &blk)
|
118
|
+
async = (method[0,1] == "a")
|
119
|
+
|
120
|
+
execute(async,method) do |conn|
|
121
|
+
df = conn.send(method, *args, &blk)
|
128
122
|
|
129
|
-
|
123
|
+
if async
|
124
|
+
fiber = Fiber.current
|
125
|
+
df.callback do
|
126
|
+
release(fiber)
|
127
|
+
release_backup(fiber)
|
128
|
+
end
|
129
|
+
df.errback { release(fiber) }
|
130
130
|
end
|
131
|
+
|
132
|
+
df
|
131
133
|
end
|
134
|
+
end
|
132
135
|
end
|
metadata
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: fiber_connection_pool
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 0.1.
|
4
|
+
version: 0.1.2
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Ruben Caro
|
@@ -9,7 +9,7 @@ authors:
|
|
9
9
|
autorequire:
|
10
10
|
bindir: bin
|
11
11
|
cert_chain: []
|
12
|
-
date: 2013-08-
|
12
|
+
date: 2013-08-16 00:00:00.000000000 Z
|
13
13
|
dependencies:
|
14
14
|
- !ruby/object:Gem::Dependency
|
15
15
|
name: minitest
|
@@ -56,6 +56,9 @@ files:
|
|
56
56
|
- README.md
|
57
57
|
- Rakefile
|
58
58
|
- examples/.placeholder
|
59
|
+
- examples/goliath_server/Gemfile
|
60
|
+
- examples/goliath_server/config/main.rb
|
61
|
+
- examples/goliath_server/main.rb
|
59
62
|
- examples/reel_server/Gemfile
|
60
63
|
- examples/reel_server/main.rb
|
61
64
|
- fiber_connection_pool.gemspec
|