em-pg-client 0.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- data/BENCHMARKS.rdoc +43 -0
- data/README.rdoc +203 -0
- data/Rakefile +46 -0
- data/benchmarks/em_pg.rb +93 -0
- data/em-pg-client.gemspec +25 -0
- data/lib/em-synchrony/pg.rb +45 -0
- data/lib/pg/em.rb +124 -0
- data/spec/em_client.rb +57 -0
- data/spec/em_synchrony_client.rb +54 -0
- metadata +126 -0
data/BENCHMARKS.rdoc
ADDED
@@ -0,0 +1,43 @@
|
|
1
|
+
== Benchmarks
|
2
|
+
|
3
|
+
I've done some benchmark tests[link:benchmarks/em_pg.rb] to compare fully async and blocking em-pg drivers.
|
4
|
+
|
5
|
+
The goal of the test is simply to retrieve (~80000) rows from table with a lot of text data, in chunks, using parallel connections.
|
6
|
+
The parallel method uses synchrony for simplicity.
|
7
|
+
|
8
|
+
* +single+ is (eventmachine-less) job for retrieving a whole data table in
|
9
|
+
one query simple query "select * from resources"
|
10
|
+
* +parallel+ chunk_row_count / concurrency] uses em-pg-client for retrieving
|
11
|
+
result in chunks by +chunk_row_count+ rows and using +concurrency+ parallel
|
12
|
+
connections
|
13
|
+
* +blocking+ chunk_row_count / concurrency is similiar to +parallel+ except
|
14
|
+
that it uses special patched version of library that uses blocking
|
15
|
+
PGConnection methods
|
16
|
+
|
17
|
+
== Environment
|
18
|
+
|
19
|
+
The machine used for test is Linux CentOS 2.6.18-194.32.1.el5xen #1 SMP with Quad Core Xeon X3360 @ 2.83GHz, 4GB RAM.
|
20
|
+
Postgres version used: 9.0.3.
|
21
|
+
|
22
|
+
== The results:
|
23
|
+
|
24
|
+
>> benchmark 1000
|
25
|
+
user system total real
|
26
|
+
single: 80.970000 0.350000 81.320000 (205.592592)
|
27
|
+
|
28
|
+
parallel 90000/1: 87.380000 0.710000 88.090000 (208.171564)
|
29
|
+
parallel 5000/5: 84.250000 3.760000 88.010000 (141.031289)
|
30
|
+
parallel 2000/10: 90.190000 4.970000 95.160000 (152.844950)
|
31
|
+
parallel 1000/20: 97.070000 5.390000 102.460000 (212.358631)
|
32
|
+
|
33
|
+
blocking 90000/1: 93.590000 0.610000 94.200000 (230.190776)
|
34
|
+
blocking 5000/5: 79.930000 1.810000 81.740000 (223.342432)
|
35
|
+
blocking 2000/10: 76.990000 2.820000 79.810000 (225.347169)
|
36
|
+
blocking 1000/20: 78.790000 3.230000 82.020000 (225.949107)
|
37
|
+
|
38
|
+
As we can see the gain from using asynchronous pg client while
|
39
|
+
using +parallel+ queries is noticeable (up to ~30%).
|
40
|
+
|
41
|
+
The +blocking+ client however doesn't gain much from parallel execution.
|
42
|
+
This was expected because it freezes eventmachine until the whole
|
43
|
+
dataset is consumed by the client.
|
data/README.rdoc
ADDED
@@ -0,0 +1,203 @@
|
|
1
|
+
= em-pg-client
|
2
|
+
|
3
|
+
Author:: Rafał Michalski (mailto:royaltm75@gmail.com)
|
4
|
+
|
5
|
+
* http://github.com/royaltm/ruby-em-pg-client
|
6
|
+
|
7
|
+
== DESCRIPTION
|
8
|
+
|
9
|
+
*em-pg-client* is a PostgreSQL EventMachine client wrapper for Ruby
|
10
|
+
based on (ruby-pg)[https://bitbucket.org/ged/ruby-pg]
|
11
|
+
|
12
|
+
== FEATURES
|
13
|
+
|
14
|
+
* minimal changes to PG::Conncet API
|
15
|
+
* auto reconnects on socket connection loss (like server restarts)
|
16
|
+
* true non-blocking asynchronous processing
|
17
|
+
* EM-Synchrony[https://github.com/igrigorik/em-synchrony] support
|
18
|
+
|
19
|
+
== BUGS/LIMITATIONS
|
20
|
+
|
21
|
+
* actually no ActiveRecord or Sequel support (you are welcome to contribute).
|
22
|
+
* connecting/reconnecting operation is blocking EM
|
23
|
+
|
24
|
+
== TODO:
|
25
|
+
|
26
|
+
* em-synchrony ORM (ActiveRecord, Sequel and maybe Datamapper) support
|
27
|
+
* full async connection process (low priority)
|
28
|
+
|
29
|
+
== REQUIREMENTS
|
30
|
+
|
31
|
+
* ruby >= 1.9
|
32
|
+
* https://bitbucket.org/ged/ruby-pg (>= 0.13)
|
33
|
+
* http://rubyeventmachine.com
|
34
|
+
* (optional) EM-Synchrony[https://github.com/igrigorik/em-synchrony]
|
35
|
+
|
36
|
+
== INSTALL
|
37
|
+
|
38
|
+
$ sudo gem install em-pg-client
|
39
|
+
|
40
|
+
== WHY?
|
41
|
+
|
42
|
+
Because until now nobody did it to fit my needs.
|
43
|
+
I've found at least 3 other implementations of EM postgres client:
|
44
|
+
|
45
|
+
* https://github.com/jzimmek/em-postgresql-sequel
|
46
|
+
* https://github.com/leftbee/em-postgresql-adapter
|
47
|
+
* https://github.com/jtoy/em-postgres
|
48
|
+
|
49
|
+
and (except the bundled one which uses no longer maintained postgres-pr library)
|
50
|
+
all of them have similiar flaws:
|
51
|
+
|
52
|
+
* 2 of them are designed to support some ORM only (ActiveRecord or Sequel)
|
53
|
+
so they are EM-Synchrony only,
|
54
|
+
* non-standard API method names,
|
55
|
+
* no (nonexistent or non-working) autoreconnect implementation,
|
56
|
+
* not fully supporting asynchronous PG::Connection API.
|
57
|
+
|
58
|
+
The last one is worth some comment:
|
59
|
+
|
60
|
+
They all use blocking methods to retrieve whole result from server
|
61
|
+
(PGConn#block() or PGConn#get_result() which also
|
62
|
+
blocks when there is not enough buffered data on socket).
|
63
|
+
|
64
|
+
This implementation makes use of non-blocking: PGConn#is_busy and PGConn#consume_input methods.
|
65
|
+
Depending on size of result sets and concurrency level the gain in overall speed and responsiveness of your
|
66
|
+
application might be actually quite huge. I already have done some tests[link:BENCHMARKS.rdoc].
|
67
|
+
|
68
|
+
== Thanks
|
69
|
+
|
70
|
+
The greetz go to:
|
71
|
+
- Authors[https://bitbucket.org/ged/ruby-pg/wiki/Home#!copying] of +pg+ driver (especially for its async-api)
|
72
|
+
- Francis Cianfrocca for great reactor framework (EventMachine[https://github.com/eventmachine/eventmachine])
|
73
|
+
- Ilya Grigorik (igrigorik[https://github.com/igrigorik]) for (untangling)[http://www.igvita.com/2010/03/22/untangling-evented-code-with-ruby-fibers/] EM with Fibers
|
74
|
+
|
75
|
+
== USAGE
|
76
|
+
|
77
|
+
=== BASIC
|
78
|
+
|
79
|
+
require 'pg/em'
|
80
|
+
|
81
|
+
# no async
|
82
|
+
pg = PG::EM::Client.new dbname: 'test'
|
83
|
+
pq.query('select * from foo') do |result|
|
84
|
+
puts result
|
85
|
+
end
|
86
|
+
|
87
|
+
# asynchronous
|
88
|
+
EM.run do
|
89
|
+
pq.query('select * from foo') do |result|
|
90
|
+
raise result if result.is_a? ::Exception
|
91
|
+
puts result
|
92
|
+
EM.stop
|
93
|
+
end
|
94
|
+
puts "sent"
|
95
|
+
end
|
96
|
+
|
97
|
+
=== AUTORECONNECTING IN ASYNC MODE
|
98
|
+
|
99
|
+
EM.run do
|
100
|
+
pg = PG::EM::Client.new dbname: 'test'
|
101
|
+
|
102
|
+
try_query = lambda |&blk| do
|
103
|
+
pq.query('select * from foo') do |result|
|
104
|
+
raise result if result.is_a? ::Exception
|
105
|
+
puts result
|
106
|
+
blk.call
|
107
|
+
end
|
108
|
+
end
|
109
|
+
try_query.call {
|
110
|
+
system 'pg_ctl stop -m fast'
|
111
|
+
system 'pg_ctl start -w'
|
112
|
+
EM.add_timer(1) do
|
113
|
+
try_query.call { EM.stop }
|
114
|
+
end
|
115
|
+
}
|
116
|
+
end
|
117
|
+
|
118
|
+
to disable this feature call:
|
119
|
+
|
120
|
+
pg.async_autoreconnect = false
|
121
|
+
|
122
|
+
or
|
123
|
+
|
124
|
+
pg = PG::EM::Client.new dbname: 'test',
|
125
|
+
async_autoreconnect: false
|
126
|
+
|
127
|
+
=== TRUE ASYNC
|
128
|
+
|
129
|
+
EM.run do
|
130
|
+
pool = (1..10).map { PG::EM::Client.new dbname: 'alpha' }
|
131
|
+
|
132
|
+
togo = pool.length
|
133
|
+
|
134
|
+
pool.each_with_index do |pg, i|
|
135
|
+
pg.query("select * from foo") do |result|
|
136
|
+
puts "recv: #{i}"
|
137
|
+
EM.stop if (togo-=1).zero?
|
138
|
+
end
|
139
|
+
puts "sent: #{i}"
|
140
|
+
end
|
141
|
+
end
|
142
|
+
|
143
|
+
=== EM-Synchrony
|
144
|
+
|
145
|
+
require 'em-synchrony/pg'
|
146
|
+
|
147
|
+
EM.synchrony do
|
148
|
+
pg = PG::EM::Client.new dbname: 'test'
|
149
|
+
pg.query('select * from foo') do |result|
|
150
|
+
puts result
|
151
|
+
end
|
152
|
+
EM.stop
|
153
|
+
end
|
154
|
+
|
155
|
+
==== Handling errors
|
156
|
+
|
157
|
+
EM.synchrony do
|
158
|
+
begin
|
159
|
+
pg.query('select * from foo') do |result|
|
160
|
+
puts result
|
161
|
+
end
|
162
|
+
rescue PG::Error => e
|
163
|
+
puts "PSQL error: #{e.inspect}"
|
164
|
+
end
|
165
|
+
EM.stop
|
166
|
+
end
|
167
|
+
|
168
|
+
==== Parallel async queries
|
169
|
+
|
170
|
+
EM.synchrony do
|
171
|
+
pg = EM::Synchrony::ConnectionPool.new(size: 2) do
|
172
|
+
PG::EM::Client.new :dbname => 'alpha'
|
173
|
+
end
|
174
|
+
multi = EventMachine::Synchrony::Multi.new
|
175
|
+
multi.add :foo, pg.aquery('select * from foo') # or #async_query()
|
176
|
+
multi.add :bar, pg.aquery('select * from bar') # #aquery() is just an alias
|
177
|
+
res = multi.perform
|
178
|
+
p res
|
179
|
+
EM.stop
|
180
|
+
end
|
181
|
+
|
182
|
+
==== Fiber Concurrency
|
183
|
+
|
184
|
+
EM.synchrony do
|
185
|
+
# use ConnectionPool when more Fibers will be querying at the same time!
|
186
|
+
pg = EM::Synchrony::ConnectionPool.new(size: 5) do
|
187
|
+
PG::EM::Client.new :dbname => 'alpha'
|
188
|
+
end
|
189
|
+
counter = 0
|
190
|
+
EM::Synchrony::FiberIterator.new(['select * from foo']*10, 5) do |query|
|
191
|
+
i = counter
|
192
|
+
pg.query(query) do |result|
|
193
|
+
puts "recv: #{i}"
|
194
|
+
end
|
195
|
+
puts "sent: #{i}"
|
196
|
+
counter += 1
|
197
|
+
end
|
198
|
+
EM.stop
|
199
|
+
end
|
200
|
+
|
201
|
+
== LICENCE
|
202
|
+
|
203
|
+
The MIT License - Copyright (c) 2012 Rafał Michalski
|
data/Rakefile
ADDED
@@ -0,0 +1,46 @@
|
|
1
|
+
$:.unshift "lib"
|
2
|
+
|
3
|
+
task :default => [:test]
|
4
|
+
|
5
|
+
$gem_name = "em-pg-client"
|
6
|
+
|
7
|
+
desc "Run tests"
|
8
|
+
task :test do
|
9
|
+
puts "WARNING: The test needs to be run with an available local PostgreSQL server"
|
10
|
+
sh "rspec spec/em_client.rb"
|
11
|
+
sh "rspec spec/em_synchrony_client.rb"
|
12
|
+
end
|
13
|
+
|
14
|
+
desc "Build the gem"
|
15
|
+
task :gem do
|
16
|
+
sh "gem build #$gem_name.gemspec"
|
17
|
+
end
|
18
|
+
|
19
|
+
desc "Install the library at local machnie"
|
20
|
+
task :install => :gem do
|
21
|
+
sh "gem install #$gem_name -l"
|
22
|
+
end
|
23
|
+
|
24
|
+
desc "Uninstall the library from local machnie"
|
25
|
+
task :uninstall do
|
26
|
+
sh "gem uninstall #$gem_name"
|
27
|
+
end
|
28
|
+
|
29
|
+
desc "Clean"
|
30
|
+
task :clean do
|
31
|
+
sh "rm #$gem_name*.gem"
|
32
|
+
end
|
33
|
+
|
34
|
+
desc "Documentation"
|
35
|
+
task :doc do
|
36
|
+
sh "rdoc --encoding=UTF-8 --title=em-pg-client --main=README.rdoc README.rdoc BENCHMARKS.rdoc lib/*/*.rb"
|
37
|
+
end
|
38
|
+
|
39
|
+
desc "Benchmark"
|
40
|
+
task :benchmark do
|
41
|
+
require "./benchmarks/em_pg.rb"
|
42
|
+
[10, 100, 1000].each do |i|
|
43
|
+
puts "Repeat: #{i}"
|
44
|
+
benchmark(i)
|
45
|
+
end
|
46
|
+
end
|
data/benchmarks/em_pg.rb
ADDED
@@ -0,0 +1,93 @@
|
|
1
|
+
$:.unshift('./lib')
|
2
|
+
require 'eventmachine'
|
3
|
+
require 'em-synchrony'
|
4
|
+
require 'em-synchrony/pg'
|
5
|
+
require "em-synchrony/fiber_iterator"
|
6
|
+
require 'pp'
|
7
|
+
require 'benchmark'
|
8
|
+
|
9
|
+
$dbname = 'alpha'
|
10
|
+
|
11
|
+
def benchmark(repeat=100)
|
12
|
+
Benchmark.bm(20) do |b|
|
13
|
+
b.report('single:') { single(repeat) }
|
14
|
+
puts
|
15
|
+
b.report('parallel 90000/1:') { parallel(repeat, 90000, 1) }
|
16
|
+
b.report('parallel 5000/5:') { parallel(repeat, 5000, 5) }
|
17
|
+
b.report('parallel 2000/10:') { parallel(repeat, 2000, 10) }
|
18
|
+
b.report('parallel 1000/20:') { parallel(repeat, 1000, 20) }
|
19
|
+
puts
|
20
|
+
patch_blocking
|
21
|
+
b.report('blocking 90000/1:') { parallel(repeat, 90000, 1) }
|
22
|
+
b.report('blocking 5000/5:') { parallel(repeat, 5000, 5) }
|
23
|
+
b.report('blocking 2000/10:') { parallel(repeat, 2000, 10) }
|
24
|
+
b.report('blocking 1000/20:') { parallel(repeat, 1000, 20) }
|
25
|
+
patch_remove_blocking
|
26
|
+
end
|
27
|
+
end
|
28
|
+
|
29
|
+
def patch_remove_blocking
|
30
|
+
PG::EM::Client::Watcher.module_eval <<-EOE
|
31
|
+
alias_method :notify_readable, :original_notify_readable
|
32
|
+
undef :original_notify_readable
|
33
|
+
EOE
|
34
|
+
end
|
35
|
+
|
36
|
+
def patch_blocking
|
37
|
+
PG::EM::Client::Watcher.module_eval <<-EOE
|
38
|
+
alias_method :original_notify_readable, :notify_readable
|
39
|
+
def notify_readable
|
40
|
+
detach
|
41
|
+
begin
|
42
|
+
result = @client.get_last_result
|
43
|
+
rescue Exception => e
|
44
|
+
@deferrable.fail(e)
|
45
|
+
else
|
46
|
+
@deferrable.succeed(result)
|
47
|
+
end
|
48
|
+
end
|
49
|
+
EOE
|
50
|
+
end
|
51
|
+
|
52
|
+
# retrieve resources using single select query
|
53
|
+
def single(repeat=1)
|
54
|
+
rowcount = 0
|
55
|
+
p = PGconn.new :dbname => $dbname
|
56
|
+
p.query('select count(*) from resources') do |result|
|
57
|
+
rowcount = result.getvalue(0,0).to_i
|
58
|
+
end
|
59
|
+
repeat.times do
|
60
|
+
p.query('select * from resources order by cdate') do |result|
|
61
|
+
$resources = result.values
|
62
|
+
end
|
63
|
+
end
|
64
|
+
# raise "invalid count #{$resources.length} != #{rowcount}" if $resources.length != rowcount
|
65
|
+
end
|
66
|
+
|
67
|
+
# retrieve resources using parallel queries
|
68
|
+
def parallel(repeat=1, chunk_size=2000, concurrency=10)
|
69
|
+
resources = []
|
70
|
+
rowcount = 0
|
71
|
+
EM.synchrony do
|
72
|
+
p = EM::Synchrony::ConnectionPool.new(size: concurrency) { PG::EM::Client.new :dbname => $dbname }
|
73
|
+
p.query('select count(*) from resources') do |result|
|
74
|
+
rowcount = result.getvalue(0,0).to_i
|
75
|
+
end
|
76
|
+
offsets = (rowcount / chunk_size.to_f).ceil.times.map {|n| n*chunk_size }
|
77
|
+
repeat.times do
|
78
|
+
EM::Synchrony::FiberIterator.new(offsets, concurrency).each do |offset|
|
79
|
+
p.query('select * from resources order by cdate limit $1 offset $2', [chunk_size, offset]) do |result|
|
80
|
+
resources[offset, chunk_size] = result.values
|
81
|
+
end
|
82
|
+
end
|
83
|
+
end
|
84
|
+
EM.stop
|
85
|
+
end
|
86
|
+
# raise "invalid count #{resources.length} != #{rowcount}" if resources.length != rowcount
|
87
|
+
# raise "resources != $resources" if resources != $resources
|
88
|
+
resources
|
89
|
+
end
|
90
|
+
|
91
|
+
if $0 == __FILE__
|
92
|
+
benchmark (ARGV.first || 10).to_i
|
93
|
+
end
|
@@ -0,0 +1,25 @@
|
|
1
|
+
$:.unshift "lib"
|
2
|
+
|
3
|
+
Gem::Specification.new do |s|
|
4
|
+
s.name = "em-pg-client"
|
5
|
+
s.version = "0.1.0"
|
6
|
+
s.required_ruby_version = ">= 1.9.1"
|
7
|
+
s.date = "#{Time.now.strftime("%Y-%m-%d")}"
|
8
|
+
s.summary = "EventMachine PostgreSQL client"
|
9
|
+
s.email = "rafal@yeondir.com"
|
10
|
+
s.homepage = "http://github.com/royaltm/ruby-em-pg-client"
|
11
|
+
s.require_path = "lib"
|
12
|
+
s.description = "PostgreSQL asynchronous EventMachine client (ruby-pg) wrapper"
|
13
|
+
s.authors = ["Rafal Michalski"]
|
14
|
+
s.files = `git ls-files`.split("\n") - ['.gitignore']
|
15
|
+
s.test_files = Dir.glob("spec/**/*")
|
16
|
+
s.rdoc_options << "--title" << "em-pg-client" <<
|
17
|
+
"--main" << "README.rdoc"
|
18
|
+
s.has_rdoc = true
|
19
|
+
s.extra_rdoc_files = ["README.rdoc", "BENCHMARKS.rdoc"]
|
20
|
+
s.requirements << "PostgreSQL server"
|
21
|
+
s.add_runtime_dependency "pg", ">= 0.13.2"
|
22
|
+
s.add_runtime_dependency "eventmachine", ">= 0.12.10"
|
23
|
+
s.add_development_dependency "rspec", "~> 2.8.0"
|
24
|
+
s.add_development_dependency "em-synchrony", "~> 1.0.0"
|
25
|
+
end
|
@@ -0,0 +1,45 @@
|
|
1
|
+
require 'pg/em'
|
2
|
+
module PG
|
3
|
+
module EM
|
4
|
+
class Client
|
5
|
+
# Author:: Rafal Michalski (mailto:royaltm75@gmail.com)
|
6
|
+
# Licence:: MIT License
|
7
|
+
#
|
8
|
+
# =PostgreSQL Client for EM-Synchrony
|
9
|
+
#
|
10
|
+
|
11
|
+
# conform to *standard*
|
12
|
+
alias_method :aquery, :async_query
|
13
|
+
|
14
|
+
# fiber untangled version of theese methods:
|
15
|
+
# - exec (aliased as query)
|
16
|
+
# - exec_prepared
|
17
|
+
# - prepare
|
18
|
+
%w(exec exec_prepared prepare).each do |name|
|
19
|
+
class_eval <<-EOD
|
20
|
+
def #{name}(*args, &blk)
|
21
|
+
if ::EM.reactor_running?
|
22
|
+
df = async_#{name}(*args)
|
23
|
+
f = Fiber.current
|
24
|
+
df.callback { |res| f.resume(res) }
|
25
|
+
df.errback { |err| f.resume(err) }
|
26
|
+
|
27
|
+
result = Fiber.yield
|
28
|
+
raise result if result.is_a?(::Exception)
|
29
|
+
if block_given?
|
30
|
+
yield result
|
31
|
+
else
|
32
|
+
result
|
33
|
+
end
|
34
|
+
else
|
35
|
+
super(*args, &blk)
|
36
|
+
end
|
37
|
+
end
|
38
|
+
EOD
|
39
|
+
|
40
|
+
end
|
41
|
+
|
42
|
+
alias_method :query, :exec
|
43
|
+
end
|
44
|
+
end
|
45
|
+
end
|
data/lib/pg/em.rb
ADDED
@@ -0,0 +1,124 @@
|
|
1
|
+
require 'pg'
|
2
|
+
module PG
|
3
|
+
module EM
|
4
|
+
class Client < PG::Connection
|
5
|
+
# == PostgreSQL EventMachine client
|
6
|
+
#
|
7
|
+
# Author:: Rafal Michalski (mailto:royaltm75@gmail.com)
|
8
|
+
# Licence:: MIT License
|
9
|
+
#
|
10
|
+
#
|
11
|
+
# PG::EM::Client is a wrapper for PG::Connection which (re)defines methods:
|
12
|
+
#
|
13
|
+
# - +async_exec+ (alias: +async_query+)
|
14
|
+
# - +async_prepare+
|
15
|
+
# - +async_exec_prepared+
|
16
|
+
#
|
17
|
+
# and following:
|
18
|
+
#
|
19
|
+
# - +exec+ (alias: +query+)
|
20
|
+
# - +exec_prepared+
|
21
|
+
# - +prepare+
|
22
|
+
#
|
23
|
+
# which autodetects if EventMachine is running and uses appropriate
|
24
|
+
# (async or sync) method version.
|
25
|
+
#
|
26
|
+
# Async methods might try to reset connection on connection error,
|
27
|
+
# you won't even notice that (except for warning message from PG).
|
28
|
+
#
|
29
|
+
# To disable such behavior set:
|
30
|
+
# client.async_autoreconnect = false
|
31
|
+
#
|
32
|
+
# or pass as new() hash argument:
|
33
|
+
# PG::EM::Client.new database: 'bar', async_autoreconnect: false
|
34
|
+
#
|
35
|
+
# Otherwise nothing changes in PG::Connect API.
|
36
|
+
# See PG::Connect docs for arguments to above methods.
|
37
|
+
#
|
38
|
+
# *Warning:*
|
39
|
+
#
|
40
|
+
# +async_exec_prepared+ after +async_prepare+ should only be invoked on
|
41
|
+
# the *same* connection.
|
42
|
+
# If you are using connection pool, make sure to acquire single connection first.
|
43
|
+
#
|
44
|
+
|
45
|
+
attr_accessor :async_autoreconnect
|
46
|
+
|
47
|
+
module Watcher
|
48
|
+
def initialize(client, deferrable)
|
49
|
+
@client = client
|
50
|
+
@deferrable = deferrable
|
51
|
+
end
|
52
|
+
|
53
|
+
def notify_readable
|
54
|
+
@client.consume_input
|
55
|
+
return if @client.is_busy
|
56
|
+
detach
|
57
|
+
begin
|
58
|
+
result = @client.get_last_result
|
59
|
+
rescue Exception => e
|
60
|
+
@deferrable.fail(e)
|
61
|
+
else
|
62
|
+
@deferrable.succeed(result)
|
63
|
+
end
|
64
|
+
end
|
65
|
+
end
|
66
|
+
|
67
|
+
def initialize(*args)
|
68
|
+
@async_autoreconnect = true
|
69
|
+
if args.last.is_a? Hash
|
70
|
+
args.last.reject! do |key, value|
|
71
|
+
if key.to_s == 'async_autoreconnect'
|
72
|
+
@async_autoreconnect = !!value
|
73
|
+
true
|
74
|
+
end
|
75
|
+
end
|
76
|
+
end
|
77
|
+
super(*args)
|
78
|
+
end
|
79
|
+
|
80
|
+
%w(
|
81
|
+
exec send_query
|
82
|
+
prepare send_prepare
|
83
|
+
exec_prepared send_query_prepared
|
84
|
+
).each_slice(2) do |name, send_name|
|
85
|
+
|
86
|
+
class_eval <<-EOD
|
87
|
+
def async_#{name}(*args, &blk)
|
88
|
+
begin
|
89
|
+
#{send_name}(*args)
|
90
|
+
rescue PG::Error => e
|
91
|
+
if self.status != PG::CONNECTION_OK && async_autoreconnect
|
92
|
+
reset
|
93
|
+
#{send_name}(*args)
|
94
|
+
else
|
95
|
+
raise e
|
96
|
+
end
|
97
|
+
end
|
98
|
+
df = ::EM::DefaultDeferrable.new
|
99
|
+
::EM.watch(self.socket, Watcher, self, df).notify_readable = true
|
100
|
+
if block_given?
|
101
|
+
df.callback(&blk)
|
102
|
+
df.errback(&blk)
|
103
|
+
end
|
104
|
+
df
|
105
|
+
end
|
106
|
+
EOD
|
107
|
+
|
108
|
+
class_eval <<-EOD
|
109
|
+
def #{name}(*args, &blk)
|
110
|
+
if ::EM.reactor_running?
|
111
|
+
async_#{name}(*args, &blk)
|
112
|
+
else
|
113
|
+
super(*args, &blk)
|
114
|
+
end
|
115
|
+
end
|
116
|
+
EOD
|
117
|
+
|
118
|
+
end
|
119
|
+
|
120
|
+
alias_method :query, :exec
|
121
|
+
alias_method :async_query, :async_exec
|
122
|
+
end
|
123
|
+
end
|
124
|
+
end
|
data/spec/em_client.rb
ADDED
@@ -0,0 +1,57 @@
|
|
1
|
+
$:.unshift "lib"
|
2
|
+
require 'date'
|
3
|
+
require 'eventmachine'
|
4
|
+
require 'pg/em'
|
5
|
+
|
6
|
+
describe PG::EM::Client do
|
7
|
+
|
8
|
+
it "should create simple table `foo`" do
|
9
|
+
@client.query('DROP TABLE IF EXISTS foo') do
|
10
|
+
@client.query('CREATE TABLE foo (id integer,cdate timestamp with time zone,data varchar)') do
|
11
|
+
EM.stop
|
12
|
+
end
|
13
|
+
end
|
14
|
+
end
|
15
|
+
|
16
|
+
it "should populate foo with some data " do
|
17
|
+
EM::Iterator.new(@values).map(proc{ |(data, id), iter|
|
18
|
+
@client.query('INSERT INTO foo (id,cdate,data) VALUES($1,$2,$3) returning cdate', [id, DateTime.now, data]) do |result|
|
19
|
+
iter.return(DateTime.parse(result[0]['cdate']))
|
20
|
+
end
|
21
|
+
}, proc{ |results|
|
22
|
+
@cdates.replace results
|
23
|
+
results.length.should == @values.length
|
24
|
+
results.each {|r| r.class.should == DateTime }
|
25
|
+
EM.stop
|
26
|
+
})
|
27
|
+
end
|
28
|
+
|
29
|
+
it "should read foo table with prepared statement" do
|
30
|
+
@client.prepare('get_foo', 'SELECT * FROM foo order by id') do
|
31
|
+
@client.exec_prepared('get_foo') do |result|
|
32
|
+
result.each_with_index do |row, i|
|
33
|
+
row['id'].to_i.should == i
|
34
|
+
DateTime.parse(row['cdate']).should == @cdates[i]
|
35
|
+
row['data'].should == @values[i][0]
|
36
|
+
end
|
37
|
+
EM.stop
|
38
|
+
end
|
39
|
+
end
|
40
|
+
end
|
41
|
+
|
42
|
+
around(:each) do |testcase|
|
43
|
+
EM.run &testcase
|
44
|
+
end
|
45
|
+
|
46
|
+
before(:all) do
|
47
|
+
@cdates = []
|
48
|
+
@values = Array(('AA'..'ZZ').each_with_index)
|
49
|
+
@client = PG::EM::Client.new(dbname: 'test')
|
50
|
+
@client.query 'BEGIN TRANSACTION'
|
51
|
+
end
|
52
|
+
|
53
|
+
after(:all) do
|
54
|
+
@client.query 'ROLLBACK TRANSACTION'
|
55
|
+
@client.close
|
56
|
+
end
|
57
|
+
end
|
@@ -0,0 +1,54 @@
|
|
1
|
+
$:.unshift "lib"
|
2
|
+
require 'date'
|
3
|
+
require 'em-synchrony'
|
4
|
+
require 'em-synchrony/pg'
|
5
|
+
|
6
|
+
describe PG::EM::Client do
|
7
|
+
|
8
|
+
it "should create simple table `foo`" do
|
9
|
+
@client.query('DROP TABLE IF EXISTS foo')
|
10
|
+
@client.query('CREATE TABLE foo (id integer,cdate timestamp with time zone,data varchar)')
|
11
|
+
EM.stop
|
12
|
+
end
|
13
|
+
|
14
|
+
it "should populate foo with some data " do
|
15
|
+
results = @values.map do |(data, id)|
|
16
|
+
@client.query('INSERT INTO foo (id,cdate,data) VALUES($1,$2,$3) returning cdate', [id, DateTime.now, data]) do |result|
|
17
|
+
DateTime.parse(result[0]['cdate'])
|
18
|
+
end
|
19
|
+
end
|
20
|
+
@cdates.replace results
|
21
|
+
results.length.should == @values.length
|
22
|
+
results.each {|r| r.class.should == DateTime }
|
23
|
+
EM.stop
|
24
|
+
end
|
25
|
+
|
26
|
+
it "should read foo table with prepared statement" do
|
27
|
+
@client.prepare('get_foo', 'SELECT * FROM foo order by id')
|
28
|
+
@client.exec_prepared('get_foo') do |result|
|
29
|
+
result.each_with_index do |row, i|
|
30
|
+
row['id'].to_i.should == i
|
31
|
+
DateTime.parse(row['cdate']).should == @cdates[i]
|
32
|
+
row['data'].should == @values[i][0]
|
33
|
+
end
|
34
|
+
end
|
35
|
+
EM.stop
|
36
|
+
end
|
37
|
+
|
38
|
+
around(:each) do |testcase|
|
39
|
+
EM.synchrony &testcase
|
40
|
+
end
|
41
|
+
|
42
|
+
before(:all) do
|
43
|
+
@cdates = []
|
44
|
+
@values = Array(('AA'..'ZZ').each_with_index)
|
45
|
+
@client = PG::EM::Client.new(dbname: 'test')
|
46
|
+
@client.query 'BEGIN TRANSACTION'
|
47
|
+
end
|
48
|
+
|
49
|
+
after(:all) do
|
50
|
+
@client.query 'ROLLBACK TRANSACTION'
|
51
|
+
@client.close
|
52
|
+
end
|
53
|
+
|
54
|
+
end
|
metadata
ADDED
@@ -0,0 +1,126 @@
|
|
1
|
+
--- !ruby/object:Gem::Specification
|
2
|
+
name: em-pg-client
|
3
|
+
version: !ruby/object:Gem::Version
|
4
|
+
version: 0.1.0
|
5
|
+
prerelease:
|
6
|
+
platform: ruby
|
7
|
+
authors:
|
8
|
+
- Rafal Michalski
|
9
|
+
autorequire:
|
10
|
+
bindir: bin
|
11
|
+
cert_chain: []
|
12
|
+
date: 2012-03-29 00:00:00.000000000 Z
|
13
|
+
dependencies:
|
14
|
+
- !ruby/object:Gem::Dependency
|
15
|
+
name: pg
|
16
|
+
requirement: !ruby/object:Gem::Requirement
|
17
|
+
none: false
|
18
|
+
requirements:
|
19
|
+
- - ! '>='
|
20
|
+
- !ruby/object:Gem::Version
|
21
|
+
version: 0.13.2
|
22
|
+
type: :runtime
|
23
|
+
prerelease: false
|
24
|
+
version_requirements: !ruby/object:Gem::Requirement
|
25
|
+
none: false
|
26
|
+
requirements:
|
27
|
+
- - ! '>='
|
28
|
+
- !ruby/object:Gem::Version
|
29
|
+
version: 0.13.2
|
30
|
+
- !ruby/object:Gem::Dependency
|
31
|
+
name: eventmachine
|
32
|
+
requirement: !ruby/object:Gem::Requirement
|
33
|
+
none: false
|
34
|
+
requirements:
|
35
|
+
- - ! '>='
|
36
|
+
- !ruby/object:Gem::Version
|
37
|
+
version: 0.12.10
|
38
|
+
type: :runtime
|
39
|
+
prerelease: false
|
40
|
+
version_requirements: !ruby/object:Gem::Requirement
|
41
|
+
none: false
|
42
|
+
requirements:
|
43
|
+
- - ! '>='
|
44
|
+
- !ruby/object:Gem::Version
|
45
|
+
version: 0.12.10
|
46
|
+
- !ruby/object:Gem::Dependency
|
47
|
+
name: rspec
|
48
|
+
requirement: !ruby/object:Gem::Requirement
|
49
|
+
none: false
|
50
|
+
requirements:
|
51
|
+
- - ~>
|
52
|
+
- !ruby/object:Gem::Version
|
53
|
+
version: 2.8.0
|
54
|
+
type: :development
|
55
|
+
prerelease: false
|
56
|
+
version_requirements: !ruby/object:Gem::Requirement
|
57
|
+
none: false
|
58
|
+
requirements:
|
59
|
+
- - ~>
|
60
|
+
- !ruby/object:Gem::Version
|
61
|
+
version: 2.8.0
|
62
|
+
- !ruby/object:Gem::Dependency
|
63
|
+
name: em-synchrony
|
64
|
+
requirement: !ruby/object:Gem::Requirement
|
65
|
+
none: false
|
66
|
+
requirements:
|
67
|
+
- - ~>
|
68
|
+
- !ruby/object:Gem::Version
|
69
|
+
version: 1.0.0
|
70
|
+
type: :development
|
71
|
+
prerelease: false
|
72
|
+
version_requirements: !ruby/object:Gem::Requirement
|
73
|
+
none: false
|
74
|
+
requirements:
|
75
|
+
- - ~>
|
76
|
+
- !ruby/object:Gem::Version
|
77
|
+
version: 1.0.0
|
78
|
+
description: PostgreSQL asynchronous EventMachine client (ruby-pg) wrapper
|
79
|
+
email: rafal@yeondir.com
|
80
|
+
executables: []
|
81
|
+
extensions: []
|
82
|
+
extra_rdoc_files:
|
83
|
+
- README.rdoc
|
84
|
+
- BENCHMARKS.rdoc
|
85
|
+
files:
|
86
|
+
- BENCHMARKS.rdoc
|
87
|
+
- README.rdoc
|
88
|
+
- Rakefile
|
89
|
+
- benchmarks/em_pg.rb
|
90
|
+
- em-pg-client.gemspec
|
91
|
+
- lib/em-synchrony/pg.rb
|
92
|
+
- lib/pg/em.rb
|
93
|
+
- spec/em_client.rb
|
94
|
+
- spec/em_synchrony_client.rb
|
95
|
+
homepage: http://github.com/royaltm/ruby-em-pg-client
|
96
|
+
licenses: []
|
97
|
+
post_install_message:
|
98
|
+
rdoc_options:
|
99
|
+
- --title
|
100
|
+
- em-pg-client
|
101
|
+
- --main
|
102
|
+
- README.rdoc
|
103
|
+
require_paths:
|
104
|
+
- lib
|
105
|
+
required_ruby_version: !ruby/object:Gem::Requirement
|
106
|
+
none: false
|
107
|
+
requirements:
|
108
|
+
- - ! '>='
|
109
|
+
- !ruby/object:Gem::Version
|
110
|
+
version: 1.9.1
|
111
|
+
required_rubygems_version: !ruby/object:Gem::Requirement
|
112
|
+
none: false
|
113
|
+
requirements:
|
114
|
+
- - ! '>='
|
115
|
+
- !ruby/object:Gem::Version
|
116
|
+
version: '0'
|
117
|
+
requirements:
|
118
|
+
- PostgreSQL server
|
119
|
+
rubyforge_project:
|
120
|
+
rubygems_version: 1.8.21
|
121
|
+
signing_key:
|
122
|
+
specification_version: 3
|
123
|
+
summary: EventMachine PostgreSQL client
|
124
|
+
test_files:
|
125
|
+
- spec/em_client.rb
|
126
|
+
- spec/em_synchrony_client.rb
|