em-pg-client 0.1.0
Sign up to get free protection for your applications and to get access to all the features.
- data/BENCHMARKS.rdoc +43 -0
- data/README.rdoc +203 -0
- data/Rakefile +46 -0
- data/benchmarks/em_pg.rb +93 -0
- data/em-pg-client.gemspec +25 -0
- data/lib/em-synchrony/pg.rb +45 -0
- data/lib/pg/em.rb +124 -0
- data/spec/em_client.rb +57 -0
- data/spec/em_synchrony_client.rb +54 -0
- metadata +126 -0
data/BENCHMARKS.rdoc
ADDED
@@ -0,0 +1,43 @@
|
|
1
|
+
== Benchmarks
|
2
|
+
|
3
|
+
I've done some benchmark tests[link:benchmarks/em_pg.rb] to compare fully async and blocking em-pg drivers.
|
4
|
+
|
5
|
+
The goal of the test is simply to retrieve (~80000) rows from table with a lot of text data, in chunks, using parallel connections.
|
6
|
+
The parallel method uses synchrony for simplicity.
|
7
|
+
|
8
|
+
* +single+ is (eventmachine-less) job for retrieving a whole data table in
|
9
|
+
one query simple query "select * from resources"
|
10
|
+
* +parallel+ chunk_row_count / concurrency] uses em-pg-client for retrieving
|
11
|
+
result in chunks by +chunk_row_count+ rows and using +concurrency+ parallel
|
12
|
+
connections
|
13
|
+
* +blocking+ chunk_row_count / concurrency is similiar to +parallel+ except
|
14
|
+
that it uses special patched version of library that uses blocking
|
15
|
+
PGConnection methods
|
16
|
+
|
17
|
+
== Environment
|
18
|
+
|
19
|
+
The machine used for test is Linux CentOS 2.6.18-194.32.1.el5xen #1 SMP with Quad Core Xeon X3360 @ 2.83GHz, 4GB RAM.
|
20
|
+
Postgres version used: 9.0.3.
|
21
|
+
|
22
|
+
== The results:
|
23
|
+
|
24
|
+
>> benchmark 1000
|
25
|
+
user system total real
|
26
|
+
single: 80.970000 0.350000 81.320000 (205.592592)
|
27
|
+
|
28
|
+
parallel 90000/1: 87.380000 0.710000 88.090000 (208.171564)
|
29
|
+
parallel 5000/5: 84.250000 3.760000 88.010000 (141.031289)
|
30
|
+
parallel 2000/10: 90.190000 4.970000 95.160000 (152.844950)
|
31
|
+
parallel 1000/20: 97.070000 5.390000 102.460000 (212.358631)
|
32
|
+
|
33
|
+
blocking 90000/1: 93.590000 0.610000 94.200000 (230.190776)
|
34
|
+
blocking 5000/5: 79.930000 1.810000 81.740000 (223.342432)
|
35
|
+
blocking 2000/10: 76.990000 2.820000 79.810000 (225.347169)
|
36
|
+
blocking 1000/20: 78.790000 3.230000 82.020000 (225.949107)
|
37
|
+
|
38
|
+
As we can see the gain from using asynchronous pg client while
|
39
|
+
using +parallel+ queries is noticeable (up to ~30%).
|
40
|
+
|
41
|
+
The +blocking+ client however doesn't gain much from parallel execution.
|
42
|
+
This was expected because it freezes eventmachine until the whole
|
43
|
+
dataset is consumed by the client.
|
data/README.rdoc
ADDED
@@ -0,0 +1,203 @@
|
|
1
|
+
= em-pg-client
|
2
|
+
|
3
|
+
Author:: Rafał Michalski (mailto:royaltm75@gmail.com)
|
4
|
+
|
5
|
+
* http://github.com/royaltm/ruby-em-pg-client
|
6
|
+
|
7
|
+
== DESCRIPTION
|
8
|
+
|
9
|
+
*em-pg-client* is a PostgreSQL EventMachine client wrapper for Ruby
|
10
|
+
based on (ruby-pg)[https://bitbucket.org/ged/ruby-pg]
|
11
|
+
|
12
|
+
== FEATURES
|
13
|
+
|
14
|
+
* minimal changes to PG::Conncet API
|
15
|
+
* auto reconnects on socket connection loss (like server restarts)
|
16
|
+
* true non-blocking asynchronous processing
|
17
|
+
* EM-Synchrony[https://github.com/igrigorik/em-synchrony] support
|
18
|
+
|
19
|
+
== BUGS/LIMITATIONS
|
20
|
+
|
21
|
+
* actually no ActiveRecord or Sequel support (you are welcome to contribute).
|
22
|
+
* connecting/reconnecting operation is blocking EM
|
23
|
+
|
24
|
+
== TODO:
|
25
|
+
|
26
|
+
* em-synchrony ORM (ActiveRecord, Sequel and maybe Datamapper) support
|
27
|
+
* full async connection process (low priority)
|
28
|
+
|
29
|
+
== REQUIREMENTS
|
30
|
+
|
31
|
+
* ruby >= 1.9
|
32
|
+
* https://bitbucket.org/ged/ruby-pg (>= 0.13)
|
33
|
+
* http://rubyeventmachine.com
|
34
|
+
* (optional) EM-Synchrony[https://github.com/igrigorik/em-synchrony]
|
35
|
+
|
36
|
+
== INSTALL
|
37
|
+
|
38
|
+
$ sudo gem install em-pg-client
|
39
|
+
|
40
|
+
== WHY?
|
41
|
+
|
42
|
+
Because until now nobody did it to fit my needs.
|
43
|
+
I've found at least 3 other implementations of EM postgres client:
|
44
|
+
|
45
|
+
* https://github.com/jzimmek/em-postgresql-sequel
|
46
|
+
* https://github.com/leftbee/em-postgresql-adapter
|
47
|
+
* https://github.com/jtoy/em-postgres
|
48
|
+
|
49
|
+
and (except the bundled one which uses no longer maintained postgres-pr library)
|
50
|
+
all of them have similiar flaws:
|
51
|
+
|
52
|
+
* 2 of them are designed to support some ORM only (ActiveRecord or Sequel)
|
53
|
+
so they are EM-Synchrony only,
|
54
|
+
* non-standard API method names,
|
55
|
+
* no (nonexistent or non-working) autoreconnect implementation,
|
56
|
+
* not fully supporting asynchronous PG::Connection API.
|
57
|
+
|
58
|
+
The last one is worth some comment:
|
59
|
+
|
60
|
+
They all use blocking methods to retrieve whole result from server
|
61
|
+
(PGConn#block() or PGConn#get_result() which also
|
62
|
+
blocks when there is not enough buffered data on socket).
|
63
|
+
|
64
|
+
This implementation makes use of non-blocking: PGConn#is_busy and PGConn#consume_input methods.
|
65
|
+
Depending on size of result sets and concurrency level the gain in overall speed and responsiveness of your
|
66
|
+
application might be actually quite huge. I already have done some tests[link:BENCHMARKS.rdoc].
|
67
|
+
|
68
|
+
== Thanks
|
69
|
+
|
70
|
+
The greetz go to:
|
71
|
+
- Authors[https://bitbucket.org/ged/ruby-pg/wiki/Home#!copying] of +pg+ driver (especially for its async-api)
|
72
|
+
- Francis Cianfrocca for great reactor framework (EventMachine[https://github.com/eventmachine/eventmachine])
|
73
|
+
- Ilya Grigorik (igrigorik[https://github.com/igrigorik]) for (untangling)[http://www.igvita.com/2010/03/22/untangling-evented-code-with-ruby-fibers/] EM with Fibers
|
74
|
+
|
75
|
+
== USAGE
|
76
|
+
|
77
|
+
=== BASIC
|
78
|
+
|
79
|
+
require 'pg/em'
|
80
|
+
|
81
|
+
# no async
|
82
|
+
pg = PG::EM::Client.new dbname: 'test'
|
83
|
+
pq.query('select * from foo') do |result|
|
84
|
+
puts result
|
85
|
+
end
|
86
|
+
|
87
|
+
# asynchronous
|
88
|
+
EM.run do
|
89
|
+
pq.query('select * from foo') do |result|
|
90
|
+
raise result if result.is_a? ::Exception
|
91
|
+
puts result
|
92
|
+
EM.stop
|
93
|
+
end
|
94
|
+
puts "sent"
|
95
|
+
end
|
96
|
+
|
97
|
+
=== AUTORECONNECTING IN ASYNC MODE
|
98
|
+
|
99
|
+
EM.run do
|
100
|
+
pg = PG::EM::Client.new dbname: 'test'
|
101
|
+
|
102
|
+
try_query = lambda |&blk| do
|
103
|
+
pq.query('select * from foo') do |result|
|
104
|
+
raise result if result.is_a? ::Exception
|
105
|
+
puts result
|
106
|
+
blk.call
|
107
|
+
end
|
108
|
+
end
|
109
|
+
try_query.call {
|
110
|
+
system 'pg_ctl stop -m fast'
|
111
|
+
system 'pg_ctl start -w'
|
112
|
+
EM.add_timer(1) do
|
113
|
+
try_query.call { EM.stop }
|
114
|
+
end
|
115
|
+
}
|
116
|
+
end
|
117
|
+
|
118
|
+
to disable this feature call:
|
119
|
+
|
120
|
+
pg.async_autoreconnect = false
|
121
|
+
|
122
|
+
or
|
123
|
+
|
124
|
+
pg = PG::EM::Client.new dbname: 'test',
|
125
|
+
async_autoreconnect: false
|
126
|
+
|
127
|
+
=== TRUE ASYNC
|
128
|
+
|
129
|
+
EM.run do
|
130
|
+
pool = (1..10).map { PG::EM::Client.new dbname: 'alpha' }
|
131
|
+
|
132
|
+
togo = pool.length
|
133
|
+
|
134
|
+
pool.each_with_index do |pg, i|
|
135
|
+
pg.query("select * from foo") do |result|
|
136
|
+
puts "recv: #{i}"
|
137
|
+
EM.stop if (togo-=1).zero?
|
138
|
+
end
|
139
|
+
puts "sent: #{i}"
|
140
|
+
end
|
141
|
+
end
|
142
|
+
|
143
|
+
=== EM-Synchrony
|
144
|
+
|
145
|
+
require 'em-synchrony/pg'
|
146
|
+
|
147
|
+
EM.synchrony do
|
148
|
+
pg = PG::EM::Client.new dbname: 'test'
|
149
|
+
pg.query('select * from foo') do |result|
|
150
|
+
puts result
|
151
|
+
end
|
152
|
+
EM.stop
|
153
|
+
end
|
154
|
+
|
155
|
+
==== Handling errors
|
156
|
+
|
157
|
+
EM.synchrony do
|
158
|
+
begin
|
159
|
+
pg.query('select * from foo') do |result|
|
160
|
+
puts result
|
161
|
+
end
|
162
|
+
rescue PG::Error => e
|
163
|
+
puts "PSQL error: #{e.inspect}"
|
164
|
+
end
|
165
|
+
EM.stop
|
166
|
+
end
|
167
|
+
|
168
|
+
==== Parallel async queries
|
169
|
+
|
170
|
+
EM.synchrony do
|
171
|
+
pg = EM::Synchrony::ConnectionPool.new(size: 2) do
|
172
|
+
PG::EM::Client.new :dbname => 'alpha'
|
173
|
+
end
|
174
|
+
multi = EventMachine::Synchrony::Multi.new
|
175
|
+
multi.add :foo, pg.aquery('select * from foo') # or #async_query()
|
176
|
+
multi.add :bar, pg.aquery('select * from bar') # #aquery() is just an alias
|
177
|
+
res = multi.perform
|
178
|
+
p res
|
179
|
+
EM.stop
|
180
|
+
end
|
181
|
+
|
182
|
+
==== Fiber Concurrency
|
183
|
+
|
184
|
+
EM.synchrony do
|
185
|
+
# use ConnectionPool when more Fibers will be querying at the same time!
|
186
|
+
pg = EM::Synchrony::ConnectionPool.new(size: 5) do
|
187
|
+
PG::EM::Client.new :dbname => 'alpha'
|
188
|
+
end
|
189
|
+
counter = 0
|
190
|
+
EM::Synchrony::FiberIterator.new(['select * from foo']*10, 5) do |query|
|
191
|
+
i = counter
|
192
|
+
pg.query(query) do |result|
|
193
|
+
puts "recv: #{i}"
|
194
|
+
end
|
195
|
+
puts "sent: #{i}"
|
196
|
+
counter += 1
|
197
|
+
end
|
198
|
+
EM.stop
|
199
|
+
end
|
200
|
+
|
201
|
+
== LICENCE
|
202
|
+
|
203
|
+
The MIT License - Copyright (c) 2012 Rafał Michalski
|
data/Rakefile
ADDED
@@ -0,0 +1,46 @@
|
|
1
|
+
$:.unshift "lib"
|
2
|
+
|
3
|
+
task :default => [:test]
|
4
|
+
|
5
|
+
$gem_name = "em-pg-client"
|
6
|
+
|
7
|
+
desc "Run tests"
|
8
|
+
task :test do
|
9
|
+
puts "WARNING: The test needs to be run with an available local PostgreSQL server"
|
10
|
+
sh "rspec spec/em_client.rb"
|
11
|
+
sh "rspec spec/em_synchrony_client.rb"
|
12
|
+
end
|
13
|
+
|
14
|
+
desc "Build the gem"
|
15
|
+
task :gem do
|
16
|
+
sh "gem build #$gem_name.gemspec"
|
17
|
+
end
|
18
|
+
|
19
|
+
desc "Install the library at local machnie"
|
20
|
+
task :install => :gem do
|
21
|
+
sh "gem install #$gem_name -l"
|
22
|
+
end
|
23
|
+
|
24
|
+
desc "Uninstall the library from local machnie"
|
25
|
+
task :uninstall do
|
26
|
+
sh "gem uninstall #$gem_name"
|
27
|
+
end
|
28
|
+
|
29
|
+
desc "Clean"
|
30
|
+
task :clean do
|
31
|
+
sh "rm #$gem_name*.gem"
|
32
|
+
end
|
33
|
+
|
34
|
+
desc "Documentation"
|
35
|
+
task :doc do
|
36
|
+
sh "rdoc --encoding=UTF-8 --title=em-pg-client --main=README.rdoc README.rdoc BENCHMARKS.rdoc lib/*/*.rb"
|
37
|
+
end
|
38
|
+
|
39
|
+
desc "Benchmark"
|
40
|
+
task :benchmark do
|
41
|
+
require "./benchmarks/em_pg.rb"
|
42
|
+
[10, 100, 1000].each do |i|
|
43
|
+
puts "Repeat: #{i}"
|
44
|
+
benchmark(i)
|
45
|
+
end
|
46
|
+
end
|
data/benchmarks/em_pg.rb
ADDED
@@ -0,0 +1,93 @@
|
|
1
|
+
$:.unshift('./lib')
|
2
|
+
require 'eventmachine'
|
3
|
+
require 'em-synchrony'
|
4
|
+
require 'em-synchrony/pg'
|
5
|
+
require "em-synchrony/fiber_iterator"
|
6
|
+
require 'pp'
|
7
|
+
require 'benchmark'
|
8
|
+
|
9
|
+
$dbname = 'alpha'
|
10
|
+
|
11
|
+
def benchmark(repeat=100)
|
12
|
+
Benchmark.bm(20) do |b|
|
13
|
+
b.report('single:') { single(repeat) }
|
14
|
+
puts
|
15
|
+
b.report('parallel 90000/1:') { parallel(repeat, 90000, 1) }
|
16
|
+
b.report('parallel 5000/5:') { parallel(repeat, 5000, 5) }
|
17
|
+
b.report('parallel 2000/10:') { parallel(repeat, 2000, 10) }
|
18
|
+
b.report('parallel 1000/20:') { parallel(repeat, 1000, 20) }
|
19
|
+
puts
|
20
|
+
patch_blocking
|
21
|
+
b.report('blocking 90000/1:') { parallel(repeat, 90000, 1) }
|
22
|
+
b.report('blocking 5000/5:') { parallel(repeat, 5000, 5) }
|
23
|
+
b.report('blocking 2000/10:') { parallel(repeat, 2000, 10) }
|
24
|
+
b.report('blocking 1000/20:') { parallel(repeat, 1000, 20) }
|
25
|
+
patch_remove_blocking
|
26
|
+
end
|
27
|
+
end
|
28
|
+
|
29
|
+
def patch_remove_blocking
|
30
|
+
PG::EM::Client::Watcher.module_eval <<-EOE
|
31
|
+
alias_method :notify_readable, :original_notify_readable
|
32
|
+
undef :original_notify_readable
|
33
|
+
EOE
|
34
|
+
end
|
35
|
+
|
36
|
+
def patch_blocking
|
37
|
+
PG::EM::Client::Watcher.module_eval <<-EOE
|
38
|
+
alias_method :original_notify_readable, :notify_readable
|
39
|
+
def notify_readable
|
40
|
+
detach
|
41
|
+
begin
|
42
|
+
result = @client.get_last_result
|
43
|
+
rescue Exception => e
|
44
|
+
@deferrable.fail(e)
|
45
|
+
else
|
46
|
+
@deferrable.succeed(result)
|
47
|
+
end
|
48
|
+
end
|
49
|
+
EOE
|
50
|
+
end
|
51
|
+
|
52
|
+
# retrieve resources using single select query
|
53
|
+
def single(repeat=1)
|
54
|
+
rowcount = 0
|
55
|
+
p = PGconn.new :dbname => $dbname
|
56
|
+
p.query('select count(*) from resources') do |result|
|
57
|
+
rowcount = result.getvalue(0,0).to_i
|
58
|
+
end
|
59
|
+
repeat.times do
|
60
|
+
p.query('select * from resources order by cdate') do |result|
|
61
|
+
$resources = result.values
|
62
|
+
end
|
63
|
+
end
|
64
|
+
# raise "invalid count #{$resources.length} != #{rowcount}" if $resources.length != rowcount
|
65
|
+
end
|
66
|
+
|
67
|
+
# retrieve resources using parallel queries
|
68
|
+
def parallel(repeat=1, chunk_size=2000, concurrency=10)
|
69
|
+
resources = []
|
70
|
+
rowcount = 0
|
71
|
+
EM.synchrony do
|
72
|
+
p = EM::Synchrony::ConnectionPool.new(size: concurrency) { PG::EM::Client.new :dbname => $dbname }
|
73
|
+
p.query('select count(*) from resources') do |result|
|
74
|
+
rowcount = result.getvalue(0,0).to_i
|
75
|
+
end
|
76
|
+
offsets = (rowcount / chunk_size.to_f).ceil.times.map {|n| n*chunk_size }
|
77
|
+
repeat.times do
|
78
|
+
EM::Synchrony::FiberIterator.new(offsets, concurrency).each do |offset|
|
79
|
+
p.query('select * from resources order by cdate limit $1 offset $2', [chunk_size, offset]) do |result|
|
80
|
+
resources[offset, chunk_size] = result.values
|
81
|
+
end
|
82
|
+
end
|
83
|
+
end
|
84
|
+
EM.stop
|
85
|
+
end
|
86
|
+
# raise "invalid count #{resources.length} != #{rowcount}" if resources.length != rowcount
|
87
|
+
# raise "resources != $resources" if resources != $resources
|
88
|
+
resources
|
89
|
+
end
|
90
|
+
|
91
|
+
if $0 == __FILE__
|
92
|
+
benchmark (ARGV.first || 10).to_i
|
93
|
+
end
|
@@ -0,0 +1,25 @@
|
|
1
|
+
$:.unshift "lib"
|
2
|
+
|
3
|
+
Gem::Specification.new do |s|
|
4
|
+
s.name = "em-pg-client"
|
5
|
+
s.version = "0.1.0"
|
6
|
+
s.required_ruby_version = ">= 1.9.1"
|
7
|
+
s.date = "#{Time.now.strftime("%Y-%m-%d")}"
|
8
|
+
s.summary = "EventMachine PostgreSQL client"
|
9
|
+
s.email = "rafal@yeondir.com"
|
10
|
+
s.homepage = "http://github.com/royaltm/ruby-em-pg-client"
|
11
|
+
s.require_path = "lib"
|
12
|
+
s.description = "PostgreSQL asynchronous EventMachine client (ruby-pg) wrapper"
|
13
|
+
s.authors = ["Rafal Michalski"]
|
14
|
+
s.files = `git ls-files`.split("\n") - ['.gitignore']
|
15
|
+
s.test_files = Dir.glob("spec/**/*")
|
16
|
+
s.rdoc_options << "--title" << "em-pg-client" <<
|
17
|
+
"--main" << "README.rdoc"
|
18
|
+
s.has_rdoc = true
|
19
|
+
s.extra_rdoc_files = ["README.rdoc", "BENCHMARKS.rdoc"]
|
20
|
+
s.requirements << "PostgreSQL server"
|
21
|
+
s.add_runtime_dependency "pg", ">= 0.13.2"
|
22
|
+
s.add_runtime_dependency "eventmachine", ">= 0.12.10"
|
23
|
+
s.add_development_dependency "rspec", "~> 2.8.0"
|
24
|
+
s.add_development_dependency "em-synchrony", "~> 1.0.0"
|
25
|
+
end
|
@@ -0,0 +1,45 @@
|
|
1
|
+
require 'pg/em'
|
2
|
+
module PG
|
3
|
+
module EM
|
4
|
+
class Client
|
5
|
+
# Author:: Rafal Michalski (mailto:royaltm75@gmail.com)
|
6
|
+
# Licence:: MIT License
|
7
|
+
#
|
8
|
+
# =PostgreSQL Client for EM-Synchrony
|
9
|
+
#
|
10
|
+
|
11
|
+
# conform to *standard*
|
12
|
+
alias_method :aquery, :async_query
|
13
|
+
|
14
|
+
# fiber untangled version of theese methods:
|
15
|
+
# - exec (aliased as query)
|
16
|
+
# - exec_prepared
|
17
|
+
# - prepare
|
18
|
+
%w(exec exec_prepared prepare).each do |name|
|
19
|
+
class_eval <<-EOD
|
20
|
+
def #{name}(*args, &blk)
|
21
|
+
if ::EM.reactor_running?
|
22
|
+
df = async_#{name}(*args)
|
23
|
+
f = Fiber.current
|
24
|
+
df.callback { |res| f.resume(res) }
|
25
|
+
df.errback { |err| f.resume(err) }
|
26
|
+
|
27
|
+
result = Fiber.yield
|
28
|
+
raise result if result.is_a?(::Exception)
|
29
|
+
if block_given?
|
30
|
+
yield result
|
31
|
+
else
|
32
|
+
result
|
33
|
+
end
|
34
|
+
else
|
35
|
+
super(*args, &blk)
|
36
|
+
end
|
37
|
+
end
|
38
|
+
EOD
|
39
|
+
|
40
|
+
end
|
41
|
+
|
42
|
+
alias_method :query, :exec
|
43
|
+
end
|
44
|
+
end
|
45
|
+
end
|
data/lib/pg/em.rb
ADDED
@@ -0,0 +1,124 @@
|
|
1
|
+
require 'pg'
|
2
|
+
module PG
|
3
|
+
module EM
|
4
|
+
class Client < PG::Connection
|
5
|
+
# == PostgreSQL EventMachine client
|
6
|
+
#
|
7
|
+
# Author:: Rafal Michalski (mailto:royaltm75@gmail.com)
|
8
|
+
# Licence:: MIT License
|
9
|
+
#
|
10
|
+
#
|
11
|
+
# PG::EM::Client is a wrapper for PG::Connection which (re)defines methods:
|
12
|
+
#
|
13
|
+
# - +async_exec+ (alias: +async_query+)
|
14
|
+
# - +async_prepare+
|
15
|
+
# - +async_exec_prepared+
|
16
|
+
#
|
17
|
+
# and following:
|
18
|
+
#
|
19
|
+
# - +exec+ (alias: +query+)
|
20
|
+
# - +exec_prepared+
|
21
|
+
# - +prepare+
|
22
|
+
#
|
23
|
+
# which autodetects if EventMachine is running and uses appropriate
|
24
|
+
# (async or sync) method version.
|
25
|
+
#
|
26
|
+
# Async methods might try to reset connection on connection error,
|
27
|
+
# you won't even notice that (except for warning message from PG).
|
28
|
+
#
|
29
|
+
# To disable such behavior set:
|
30
|
+
# client.async_autoreconnect = false
|
31
|
+
#
|
32
|
+
# or pass as new() hash argument:
|
33
|
+
# PG::EM::Client.new database: 'bar', async_autoreconnect: false
|
34
|
+
#
|
35
|
+
# Otherwise nothing changes in PG::Connect API.
|
36
|
+
# See PG::Connect docs for arguments to above methods.
|
37
|
+
#
|
38
|
+
# *Warning:*
|
39
|
+
#
|
40
|
+
# +async_exec_prepared+ after +async_prepare+ should only be invoked on
|
41
|
+
# the *same* connection.
|
42
|
+
# If you are using connection pool, make sure to acquire single connection first.
|
43
|
+
#
|
44
|
+
|
45
|
+
attr_accessor :async_autoreconnect
|
46
|
+
|
47
|
+
module Watcher
|
48
|
+
def initialize(client, deferrable)
|
49
|
+
@client = client
|
50
|
+
@deferrable = deferrable
|
51
|
+
end
|
52
|
+
|
53
|
+
def notify_readable
|
54
|
+
@client.consume_input
|
55
|
+
return if @client.is_busy
|
56
|
+
detach
|
57
|
+
begin
|
58
|
+
result = @client.get_last_result
|
59
|
+
rescue Exception => e
|
60
|
+
@deferrable.fail(e)
|
61
|
+
else
|
62
|
+
@deferrable.succeed(result)
|
63
|
+
end
|
64
|
+
end
|
65
|
+
end
|
66
|
+
|
67
|
+
def initialize(*args)
|
68
|
+
@async_autoreconnect = true
|
69
|
+
if args.last.is_a? Hash
|
70
|
+
args.last.reject! do |key, value|
|
71
|
+
if key.to_s == 'async_autoreconnect'
|
72
|
+
@async_autoreconnect = !!value
|
73
|
+
true
|
74
|
+
end
|
75
|
+
end
|
76
|
+
end
|
77
|
+
super(*args)
|
78
|
+
end
|
79
|
+
|
80
|
+
%w(
|
81
|
+
exec send_query
|
82
|
+
prepare send_prepare
|
83
|
+
exec_prepared send_query_prepared
|
84
|
+
).each_slice(2) do |name, send_name|
|
85
|
+
|
86
|
+
class_eval <<-EOD
|
87
|
+
def async_#{name}(*args, &blk)
|
88
|
+
begin
|
89
|
+
#{send_name}(*args)
|
90
|
+
rescue PG::Error => e
|
91
|
+
if self.status != PG::CONNECTION_OK && async_autoreconnect
|
92
|
+
reset
|
93
|
+
#{send_name}(*args)
|
94
|
+
else
|
95
|
+
raise e
|
96
|
+
end
|
97
|
+
end
|
98
|
+
df = ::EM::DefaultDeferrable.new
|
99
|
+
::EM.watch(self.socket, Watcher, self, df).notify_readable = true
|
100
|
+
if block_given?
|
101
|
+
df.callback(&blk)
|
102
|
+
df.errback(&blk)
|
103
|
+
end
|
104
|
+
df
|
105
|
+
end
|
106
|
+
EOD
|
107
|
+
|
108
|
+
class_eval <<-EOD
|
109
|
+
def #{name}(*args, &blk)
|
110
|
+
if ::EM.reactor_running?
|
111
|
+
async_#{name}(*args, &blk)
|
112
|
+
else
|
113
|
+
super(*args, &blk)
|
114
|
+
end
|
115
|
+
end
|
116
|
+
EOD
|
117
|
+
|
118
|
+
end
|
119
|
+
|
120
|
+
alias_method :query, :exec
|
121
|
+
alias_method :async_query, :async_exec
|
122
|
+
end
|
123
|
+
end
|
124
|
+
end
|
data/spec/em_client.rb
ADDED
@@ -0,0 +1,57 @@
|
|
1
|
+
$:.unshift "lib"
|
2
|
+
require 'date'
|
3
|
+
require 'eventmachine'
|
4
|
+
require 'pg/em'
|
5
|
+
|
6
|
+
describe PG::EM::Client do
|
7
|
+
|
8
|
+
it "should create simple table `foo`" do
|
9
|
+
@client.query('DROP TABLE IF EXISTS foo') do
|
10
|
+
@client.query('CREATE TABLE foo (id integer,cdate timestamp with time zone,data varchar)') do
|
11
|
+
EM.stop
|
12
|
+
end
|
13
|
+
end
|
14
|
+
end
|
15
|
+
|
16
|
+
it "should populate foo with some data " do
|
17
|
+
EM::Iterator.new(@values).map(proc{ |(data, id), iter|
|
18
|
+
@client.query('INSERT INTO foo (id,cdate,data) VALUES($1,$2,$3) returning cdate', [id, DateTime.now, data]) do |result|
|
19
|
+
iter.return(DateTime.parse(result[0]['cdate']))
|
20
|
+
end
|
21
|
+
}, proc{ |results|
|
22
|
+
@cdates.replace results
|
23
|
+
results.length.should == @values.length
|
24
|
+
results.each {|r| r.class.should == DateTime }
|
25
|
+
EM.stop
|
26
|
+
})
|
27
|
+
end
|
28
|
+
|
29
|
+
it "should read foo table with prepared statement" do
|
30
|
+
@client.prepare('get_foo', 'SELECT * FROM foo order by id') do
|
31
|
+
@client.exec_prepared('get_foo') do |result|
|
32
|
+
result.each_with_index do |row, i|
|
33
|
+
row['id'].to_i.should == i
|
34
|
+
DateTime.parse(row['cdate']).should == @cdates[i]
|
35
|
+
row['data'].should == @values[i][0]
|
36
|
+
end
|
37
|
+
EM.stop
|
38
|
+
end
|
39
|
+
end
|
40
|
+
end
|
41
|
+
|
42
|
+
around(:each) do |testcase|
|
43
|
+
EM.run &testcase
|
44
|
+
end
|
45
|
+
|
46
|
+
before(:all) do
|
47
|
+
@cdates = []
|
48
|
+
@values = Array(('AA'..'ZZ').each_with_index)
|
49
|
+
@client = PG::EM::Client.new(dbname: 'test')
|
50
|
+
@client.query 'BEGIN TRANSACTION'
|
51
|
+
end
|
52
|
+
|
53
|
+
after(:all) do
|
54
|
+
@client.query 'ROLLBACK TRANSACTION'
|
55
|
+
@client.close
|
56
|
+
end
|
57
|
+
end
|
@@ -0,0 +1,54 @@
|
|
1
|
+
$:.unshift "lib"
|
2
|
+
require 'date'
|
3
|
+
require 'em-synchrony'
|
4
|
+
require 'em-synchrony/pg'
|
5
|
+
|
6
|
+
describe PG::EM::Client do
|
7
|
+
|
8
|
+
it "should create simple table `foo`" do
|
9
|
+
@client.query('DROP TABLE IF EXISTS foo')
|
10
|
+
@client.query('CREATE TABLE foo (id integer,cdate timestamp with time zone,data varchar)')
|
11
|
+
EM.stop
|
12
|
+
end
|
13
|
+
|
14
|
+
it "should populate foo with some data " do
|
15
|
+
results = @values.map do |(data, id)|
|
16
|
+
@client.query('INSERT INTO foo (id,cdate,data) VALUES($1,$2,$3) returning cdate', [id, DateTime.now, data]) do |result|
|
17
|
+
DateTime.parse(result[0]['cdate'])
|
18
|
+
end
|
19
|
+
end
|
20
|
+
@cdates.replace results
|
21
|
+
results.length.should == @values.length
|
22
|
+
results.each {|r| r.class.should == DateTime }
|
23
|
+
EM.stop
|
24
|
+
end
|
25
|
+
|
26
|
+
it "should read foo table with prepared statement" do
|
27
|
+
@client.prepare('get_foo', 'SELECT * FROM foo order by id')
|
28
|
+
@client.exec_prepared('get_foo') do |result|
|
29
|
+
result.each_with_index do |row, i|
|
30
|
+
row['id'].to_i.should == i
|
31
|
+
DateTime.parse(row['cdate']).should == @cdates[i]
|
32
|
+
row['data'].should == @values[i][0]
|
33
|
+
end
|
34
|
+
end
|
35
|
+
EM.stop
|
36
|
+
end
|
37
|
+
|
38
|
+
around(:each) do |testcase|
|
39
|
+
EM.synchrony &testcase
|
40
|
+
end
|
41
|
+
|
42
|
+
before(:all) do
|
43
|
+
@cdates = []
|
44
|
+
@values = Array(('AA'..'ZZ').each_with_index)
|
45
|
+
@client = PG::EM::Client.new(dbname: 'test')
|
46
|
+
@client.query 'BEGIN TRANSACTION'
|
47
|
+
end
|
48
|
+
|
49
|
+
after(:all) do
|
50
|
+
@client.query 'ROLLBACK TRANSACTION'
|
51
|
+
@client.close
|
52
|
+
end
|
53
|
+
|
54
|
+
end
|
metadata
ADDED
@@ -0,0 +1,126 @@
|
|
1
|
+
--- !ruby/object:Gem::Specification
|
2
|
+
name: em-pg-client
|
3
|
+
version: !ruby/object:Gem::Version
|
4
|
+
version: 0.1.0
|
5
|
+
prerelease:
|
6
|
+
platform: ruby
|
7
|
+
authors:
|
8
|
+
- Rafal Michalski
|
9
|
+
autorequire:
|
10
|
+
bindir: bin
|
11
|
+
cert_chain: []
|
12
|
+
date: 2012-03-29 00:00:00.000000000 Z
|
13
|
+
dependencies:
|
14
|
+
- !ruby/object:Gem::Dependency
|
15
|
+
name: pg
|
16
|
+
requirement: !ruby/object:Gem::Requirement
|
17
|
+
none: false
|
18
|
+
requirements:
|
19
|
+
- - ! '>='
|
20
|
+
- !ruby/object:Gem::Version
|
21
|
+
version: 0.13.2
|
22
|
+
type: :runtime
|
23
|
+
prerelease: false
|
24
|
+
version_requirements: !ruby/object:Gem::Requirement
|
25
|
+
none: false
|
26
|
+
requirements:
|
27
|
+
- - ! '>='
|
28
|
+
- !ruby/object:Gem::Version
|
29
|
+
version: 0.13.2
|
30
|
+
- !ruby/object:Gem::Dependency
|
31
|
+
name: eventmachine
|
32
|
+
requirement: !ruby/object:Gem::Requirement
|
33
|
+
none: false
|
34
|
+
requirements:
|
35
|
+
- - ! '>='
|
36
|
+
- !ruby/object:Gem::Version
|
37
|
+
version: 0.12.10
|
38
|
+
type: :runtime
|
39
|
+
prerelease: false
|
40
|
+
version_requirements: !ruby/object:Gem::Requirement
|
41
|
+
none: false
|
42
|
+
requirements:
|
43
|
+
- - ! '>='
|
44
|
+
- !ruby/object:Gem::Version
|
45
|
+
version: 0.12.10
|
46
|
+
- !ruby/object:Gem::Dependency
|
47
|
+
name: rspec
|
48
|
+
requirement: !ruby/object:Gem::Requirement
|
49
|
+
none: false
|
50
|
+
requirements:
|
51
|
+
- - ~>
|
52
|
+
- !ruby/object:Gem::Version
|
53
|
+
version: 2.8.0
|
54
|
+
type: :development
|
55
|
+
prerelease: false
|
56
|
+
version_requirements: !ruby/object:Gem::Requirement
|
57
|
+
none: false
|
58
|
+
requirements:
|
59
|
+
- - ~>
|
60
|
+
- !ruby/object:Gem::Version
|
61
|
+
version: 2.8.0
|
62
|
+
- !ruby/object:Gem::Dependency
|
63
|
+
name: em-synchrony
|
64
|
+
requirement: !ruby/object:Gem::Requirement
|
65
|
+
none: false
|
66
|
+
requirements:
|
67
|
+
- - ~>
|
68
|
+
- !ruby/object:Gem::Version
|
69
|
+
version: 1.0.0
|
70
|
+
type: :development
|
71
|
+
prerelease: false
|
72
|
+
version_requirements: !ruby/object:Gem::Requirement
|
73
|
+
none: false
|
74
|
+
requirements:
|
75
|
+
- - ~>
|
76
|
+
- !ruby/object:Gem::Version
|
77
|
+
version: 1.0.0
|
78
|
+
description: PostgreSQL asynchronous EventMachine client (ruby-pg) wrapper
|
79
|
+
email: rafal@yeondir.com
|
80
|
+
executables: []
|
81
|
+
extensions: []
|
82
|
+
extra_rdoc_files:
|
83
|
+
- README.rdoc
|
84
|
+
- BENCHMARKS.rdoc
|
85
|
+
files:
|
86
|
+
- BENCHMARKS.rdoc
|
87
|
+
- README.rdoc
|
88
|
+
- Rakefile
|
89
|
+
- benchmarks/em_pg.rb
|
90
|
+
- em-pg-client.gemspec
|
91
|
+
- lib/em-synchrony/pg.rb
|
92
|
+
- lib/pg/em.rb
|
93
|
+
- spec/em_client.rb
|
94
|
+
- spec/em_synchrony_client.rb
|
95
|
+
homepage: http://github.com/royaltm/ruby-em-pg-client
|
96
|
+
licenses: []
|
97
|
+
post_install_message:
|
98
|
+
rdoc_options:
|
99
|
+
- --title
|
100
|
+
- em-pg-client
|
101
|
+
- --main
|
102
|
+
- README.rdoc
|
103
|
+
require_paths:
|
104
|
+
- lib
|
105
|
+
required_ruby_version: !ruby/object:Gem::Requirement
|
106
|
+
none: false
|
107
|
+
requirements:
|
108
|
+
- - ! '>='
|
109
|
+
- !ruby/object:Gem::Version
|
110
|
+
version: 1.9.1
|
111
|
+
required_rubygems_version: !ruby/object:Gem::Requirement
|
112
|
+
none: false
|
113
|
+
requirements:
|
114
|
+
- - ! '>='
|
115
|
+
- !ruby/object:Gem::Version
|
116
|
+
version: '0'
|
117
|
+
requirements:
|
118
|
+
- PostgreSQL server
|
119
|
+
rubyforge_project:
|
120
|
+
rubygems_version: 1.8.21
|
121
|
+
signing_key:
|
122
|
+
specification_version: 3
|
123
|
+
summary: EventMachine PostgreSQL client
|
124
|
+
test_files:
|
125
|
+
- spec/em_client.rb
|
126
|
+
- spec/em_synchrony_client.rb
|