em-pg-client 0.3.1 → 0.3.2
Sign up to get free protection for your applications and to get access to all the features.
- data/.yardopts +1 -1
- data/BENCHMARKS.md +56 -13
- data/HISTORY.md +7 -0
- data/README.md +5 -5
- data/benchmarks/em_pg.rb +1 -1
- data/benchmarks/single_row_mode.rb +88 -0
- data/em-pg-client.gemspec +7 -2
- data/examples/single_row_mode.rb +1 -11
- data/lib/pg/em-version.rb +1 -1
- data/lib/pg/em.rb +67 -41
- data/lib/pg/em/client/watcher.rb +33 -31
- data/spec/em_client_autoreconnect.rb +26 -16
- data/spec/em_client_common.rb +40 -0
- data/spec/em_synchrony_client.rb +5 -0
- data/spec/em_synchrony_client_autoreconnect.rb +13 -12
- metadata +5 -2
data/.yardopts
CHANGED
@@ -1 +1 @@
|
|
1
|
-
lib/pg/em/client/*.rb lib/pg/em/*.rb lib/pg/*.rb - BENCHMARKS.md LICENSE HISTORY.md
|
1
|
+
lib/pg/em/client/*.rb lib/pg/em/*.rb lib/pg/*.rb - benchmarks/*.rb BENCHMARKS.md LICENSE HISTORY.md
|
data/BENCHMARKS.md
CHANGED
@@ -1,9 +1,24 @@
|
|
1
|
-
|
2
|
-
|
1
|
+
Benchmarking
|
2
|
+
============
|
3
3
|
|
4
|
-
|
4
|
+
Environment
|
5
|
+
-----------
|
6
|
+
|
7
|
+
The machine running tests is Linux CentOS 2.6.18-194.32.1.el5xen #1 SMP with
|
8
|
+
Quad Core Xeon X3360 @ 2.83GHz, 4GB RAM.
|
9
|
+
|
10
|
+
Postgres server version: 9.0.3
|
11
|
+
Postgres pqlib version: 9.3
|
12
|
+
|
13
|
+
Fully Asynchronous vs. poor man's async
|
14
|
+
---------------------------------------
|
15
|
+
|
16
|
+
The following {file:benchmarks/em_pg.rb benchmark} compares fully
|
17
|
+
asynchronous implementation (`em-pg-client`) versus blocking em-pg drivers.
|
18
|
+
|
19
|
+
The goal of the test is to retrieve (~80000) rows from the same table with
|
20
|
+
a lot of text data, in chunks, using parallel connections.
|
5
21
|
|
6
|
-
The goal of the test is simply to retrieve (~80000) rows from table with a lot of text data, in chunks, using parallel connections.
|
7
22
|
The parallel method uses synchrony for simplicity.
|
8
23
|
|
9
24
|
* `single` is (eventmachine-less) job for retrieving a whole data table in
|
@@ -15,14 +30,6 @@ The parallel method uses synchrony for simplicity.
|
|
15
30
|
that it uses special patched version of library that uses blocking
|
16
31
|
PGConnection methods
|
17
32
|
|
18
|
-
Environment
|
19
|
-
-----------
|
20
|
-
|
21
|
-
The machine used for test is Linux CentOS 2.6.18-194.32.1.el5xen #1 SMP with Quad Core Xeon X3360 @ 2.83GHz, 4GB RAM.
|
22
|
-
Postgres version used: 9.0.3.
|
23
|
-
|
24
|
-
The results:
|
25
|
-
------------
|
26
33
|
|
27
34
|
```
|
28
35
|
>> benchmark 1000
|
@@ -40,9 +47,45 @@ The results:
|
|
40
47
|
blocking 1000/20: 78.790000 3.230000 82.020000 (225.949107)
|
41
48
|
```
|
42
49
|
|
43
|
-
As we can see the gain from using asynchronous pg
|
50
|
+
As we can see the gain from using asynchronous em-pg-client while
|
44
51
|
using `parallel` queries is noticeable (up to ~30%).
|
45
52
|
|
46
53
|
The `blocking` client however doesn't gain much from parallel execution.
|
47
54
|
This was expected because it freezes eventmachine until the whole
|
48
55
|
dataset is consumed by the client.
|
56
|
+
|
57
|
+
|
58
|
+
Threads vs. Fibers Streaming Benchmark
|
59
|
+
--------------------------------------
|
60
|
+
|
61
|
+
The following {file:benchmarks/single_row_mode.rb benchmark} compares
|
62
|
+
performance of parallel running threads using vanilla PG::Connection driver
|
63
|
+
versus EventMachine driven parallel Fibers using PG::EM::Client v0.3.2.
|
64
|
+
|
65
|
+
Each thread/fiber retrieves first 5000 rows from the same table with
|
66
|
+
a lot of text data in a `single_row_mode`. After 5000 rows is retrieved
|
67
|
+
the connection is being reset. The process is repeated after all parallel
|
68
|
+
running threads/fibers finish their task.
|
69
|
+
|
70
|
+
Both Thread and Fiber versions use the same chunk of code to retrieve rows.
|
71
|
+
|
72
|
+
```
|
73
|
+
>> benchmark 400
|
74
|
+
user system total real
|
75
|
+
threads 400x1: 24.970000 1.090000 26.060000 ( 30.683818)
|
76
|
+
threads 80x5: 24.730000 7.020000 31.750000 ( 51.402710)
|
77
|
+
threads 40x10: 22.880000 7.460000 30.340000 ( 52.548910)
|
78
|
+
threads 20x20: 22.220000 7.130000 29.350000 ( 53.911111)
|
79
|
+
threads 10x40: 22.570000 7.620000 30.190000 ( 54.111841)
|
80
|
+
|
81
|
+
fibers 400x1: 26.040000 1.060000 27.100000 ( 31.619598)
|
82
|
+
fibers 80x5: 28.690000 1.140000 29.830000 ( 33.025573)
|
83
|
+
fibers 40x10: 28.790000 1.280000 30.070000 ( 33.498418)
|
84
|
+
fibers 20x20: 29.100000 1.210000 30.310000 ( 33.289344)
|
85
|
+
fibers 10x40: 29.220000 1.340000 30.560000 ( 33.691188)
|
86
|
+
```
|
87
|
+
|
88
|
+
```
|
89
|
+
AxB - repeat A times running B parallel threads/fibers.
|
90
|
+
```
|
91
|
+
|
data/HISTORY.md
CHANGED
@@ -1,3 +1,10 @@
|
|
1
|
+
0.3.2
|
2
|
+
|
3
|
+
- fix: asynchronous get_result performance
|
4
|
+
- fix+specs: query_timeout timer is canceled on connection breakdown
|
5
|
+
- comply with pg+specs: asynchronous get_result and get_last_result return nil
|
6
|
+
when connection status is not ok
|
7
|
+
|
1
8
|
0.3.1
|
2
9
|
|
3
10
|
- support for asynchronous data streaming in single row mode -
|
data/README.md
CHANGED
@@ -1,4 +1,5 @@
|
|
1
|
-
|
1
|
+
em-pg-client
|
2
|
+
============
|
2
3
|
|
3
4
|
The Ruby EventMachine driver interface to the PostgreSQL RDBMS. It is based on
|
4
5
|
[ruby-pg](https://bitbucket.org/ged/ruby-pg).
|
@@ -72,7 +73,7 @@ Features
|
|
72
73
|
* [Sequel Adapter](https://github.com/fl00r/em-pg-sequel) by Peter Yanovich.
|
73
74
|
* Works on windows (requires ruby 2.0) ([issue #7][Issue 7]).
|
74
75
|
* __New__ - supports asynchronous query data processing in single row mode
|
75
|
-
([issue #12][Issue 12]).
|
76
|
+
([issue #12][Issue 12]). See {file:BENCHMARKS.md BENCHMARKING}.
|
76
77
|
|
77
78
|
Requirements
|
78
79
|
------------
|
@@ -95,7 +96,7 @@ Install
|
|
95
96
|
#### Gemfile
|
96
97
|
|
97
98
|
```ruby
|
98
|
-
gem "em-pg-client", "~> 0.3.
|
99
|
+
gem "em-pg-client", "~> 0.3.2"
|
99
100
|
```
|
100
101
|
|
101
102
|
#### Github
|
@@ -365,11 +366,10 @@ Bugs/Limitations
|
|
365
366
|
TODO:
|
366
367
|
-----
|
367
368
|
|
368
|
-
*
|
369
|
+
* more convenient streaming API
|
369
370
|
* implement EM adapted version of `get_copy_data`, `put_copy_data`,
|
370
371
|
`wait_for_notify` and `transaction`
|
371
372
|
* ORM (ActiveRecord and maybe Datamapper) support as separate projects
|
372
|
-
* present more benchmarks
|
373
373
|
|
374
374
|
More Info
|
375
375
|
---------
|
data/benchmarks/em_pg.rb
CHANGED
@@ -0,0 +1,88 @@
|
|
1
|
+
$:.unshift('./lib')
|
2
|
+
require 'pg/em/connection_pool'
|
3
|
+
require 'em-synchrony'
|
4
|
+
require 'em-synchrony/fiber_iterator'
|
5
|
+
require 'pp'
|
6
|
+
require 'benchmark'
|
7
|
+
|
8
|
+
TABLE_NAME = 'resources'
|
9
|
+
LIMIT_ROWS = 5000
|
10
|
+
|
11
|
+
include EM::Synchrony
|
12
|
+
|
13
|
+
unless PG::EM::Client.single_row_mode?
|
14
|
+
raise 'compile pg against pqlib >= 9.2 to support single row mode'
|
15
|
+
end
|
16
|
+
|
17
|
+
def benchmark(repeat=40)
|
18
|
+
Benchmark.bm(20) do |b|
|
19
|
+
puts
|
20
|
+
b.report("threads #{repeat/1}x1:") { threads(repeat, 1) }
|
21
|
+
b.report("threads #{repeat/5}x5:") { threads(repeat, 5) }
|
22
|
+
b.report("threads #{repeat/10}x10:") { threads(repeat, 10) }
|
23
|
+
b.report("threads #{repeat/20}x20:") { threads(repeat, 20) }
|
24
|
+
b.report("threads #{repeat/40}x40:") { threads(repeat, 40) }
|
25
|
+
puts
|
26
|
+
b.report("fibers #{repeat/1}x1:") { fibers(repeat, 1) }
|
27
|
+
b.report("fibers #{repeat/5}x5:") { fibers(repeat, 5) }
|
28
|
+
b.report("fibers #{repeat/10}x10:") { fibers(repeat, 10) }
|
29
|
+
b.report("fibers #{repeat/20}x20:") { fibers(repeat, 20) }
|
30
|
+
b.report("fibers #{repeat/40}x40:") { fibers(repeat, 40) }
|
31
|
+
end
|
32
|
+
end
|
33
|
+
|
34
|
+
def threads(repeat, concurrency)
|
35
|
+
db = Hash.new { |pool, id| pool[id] = PG::Connection.new }
|
36
|
+
(repeat/concurrency).times do
|
37
|
+
(0...concurrency).map do |i|
|
38
|
+
Thread.new do
|
39
|
+
stream_results(db[i])
|
40
|
+
end
|
41
|
+
end.each(&:join)
|
42
|
+
end
|
43
|
+
db.each_value(&:finish).clear
|
44
|
+
end
|
45
|
+
|
46
|
+
def fibers(repeat, concurrency)
|
47
|
+
EM.synchrony do
|
48
|
+
db = PG::EM::ConnectionPool.new size: concurrency, lazy: true
|
49
|
+
(repeat/concurrency).times do
|
50
|
+
FiberIterator.new((0...concurrency), concurrency).each do
|
51
|
+
db.hold do |pg|
|
52
|
+
stream_results(pg)
|
53
|
+
end
|
54
|
+
end
|
55
|
+
end
|
56
|
+
db.finish
|
57
|
+
EM.stop
|
58
|
+
end
|
59
|
+
end
|
60
|
+
|
61
|
+
def stream_results(pg)
|
62
|
+
pg.send_query("select * from #{TABLE_NAME}")
|
63
|
+
pg.set_single_row_mode
|
64
|
+
rows = 0
|
65
|
+
last_time = Time.now
|
66
|
+
while result = pg.get_result
|
67
|
+
begin
|
68
|
+
result.check
|
69
|
+
result.each do |tuple|
|
70
|
+
rows += 1
|
71
|
+
if rows >= LIMIT_ROWS
|
72
|
+
pg.reset
|
73
|
+
break
|
74
|
+
end
|
75
|
+
end
|
76
|
+
rescue PG::Error => e
|
77
|
+
pg.get_last_result
|
78
|
+
raise e
|
79
|
+
ensure
|
80
|
+
result.clear
|
81
|
+
end
|
82
|
+
end
|
83
|
+
end
|
84
|
+
|
85
|
+
if $0 == __FILE__
|
86
|
+
benchmark ARGV[0].to_i.nonzero? || 40
|
87
|
+
end
|
88
|
+
|
data/em-pg-client.gemspec
CHANGED
@@ -1,6 +1,8 @@
|
|
1
1
|
$:.unshift "lib"
|
2
2
|
require 'pg/em-version'
|
3
3
|
|
4
|
+
files = `git ls-files`.split("\n")
|
5
|
+
|
4
6
|
Gem::Specification.new do |s|
|
5
7
|
s.name = "em-pg-client"
|
6
8
|
s.version = PG::EM::VERSION
|
@@ -13,12 +15,15 @@ Gem::Specification.new do |s|
|
|
13
15
|
s.require_path = "lib"
|
14
16
|
s.description = "PostgreSQL asynchronous EventMachine client, based on pg interface (PG::Connection)"
|
15
17
|
s.authors = ["Rafal Michalski"]
|
16
|
-
s.files =
|
18
|
+
s.files = files - ['.gitignore']
|
17
19
|
s.test_files = Dir.glob("spec/**/*")
|
18
20
|
s.rdoc_options << "--title" << "em-pg-client" <<
|
19
21
|
"--main" << "README.md"
|
20
22
|
s.has_rdoc = true
|
21
|
-
s.extra_rdoc_files = [
|
23
|
+
s.extra_rdoc_files = [
|
24
|
+
files.grep(/^benchmarks\/.*\.rb$/),
|
25
|
+
"README.md", "BENCHMARKS.md", "LICENSE", "HISTORY.md"
|
26
|
+
].flatten
|
22
27
|
s.requirements << "PostgreSQL server"
|
23
28
|
s.add_runtime_dependency "pg", ">= 0.17.0"
|
24
29
|
s.add_runtime_dependency "eventmachine", "~> 1.0.0"
|
data/examples/single_row_mode.rb
CHANGED
@@ -4,7 +4,7 @@
|
|
4
4
|
#
|
5
5
|
# upside: it's the same way you would work with PG::Connection
|
6
6
|
# downside: it's a little verbose and doesn't support automatic re-connects
|
7
|
-
gem 'em-pg-client', '>= 0.3.
|
7
|
+
gem 'em-pg-client', '>= 0.3.2'
|
8
8
|
require 'pg/em/connection_pool'
|
9
9
|
require 'em-synchrony'
|
10
10
|
require 'em-synchrony/fiber_iterator'
|
@@ -15,12 +15,6 @@ unless PG::EM::Client.single_row_mode?
|
|
15
15
|
raise 'compile pg against pqlib >= 9.2 to support single row mode'
|
16
16
|
end
|
17
17
|
|
18
|
-
def tick_sleep
|
19
|
-
f = Fiber.current
|
20
|
-
EM.next_tick { f.resume }
|
21
|
-
Fiber.yield
|
22
|
-
end
|
23
|
-
|
24
18
|
EM.synchrony do
|
25
19
|
EM.add_periodic_timer(0.01) { print ' ' }
|
26
20
|
|
@@ -41,10 +35,6 @@ EM.synchrony do
|
|
41
35
|
rows += 1
|
42
36
|
# process tuple
|
43
37
|
print mark
|
44
|
-
if (rows % 10).zero?
|
45
|
-
# let reactor do some work if data is coming too fast
|
46
|
-
tick_sleep
|
47
|
-
end
|
48
38
|
# break stream cleanly
|
49
39
|
pg.reset if rows > 1000
|
50
40
|
end
|
data/lib/pg/em-version.rb
CHANGED
data/lib/pg/em.rb
CHANGED
@@ -34,6 +34,8 @@ module PG
|
|
34
34
|
# - {#exec_prepared_defer}
|
35
35
|
# - {#describe_prepared_defer}
|
36
36
|
# - {#describe_portal_defer}
|
37
|
+
# - {#get_result_defer}
|
38
|
+
# - {#get_last_result_defer}
|
37
39
|
#
|
38
40
|
# are added to execute queries asynchronously,
|
39
41
|
# returning +Deferrable+ object.
|
@@ -47,6 +49,8 @@ module PG
|
|
47
49
|
# - {#exec_prepared}
|
48
50
|
# - {#describe_prepared}
|
49
51
|
# - {#describe_portal}
|
52
|
+
# - {#get_result}
|
53
|
+
# - {#get_last_result}
|
50
54
|
#
|
51
55
|
# and are now auto-detecting if EventMachine is running and
|
52
56
|
# performing commands asynchronously (blocking only current fiber) or
|
@@ -577,8 +581,6 @@ module PG
|
|
577
581
|
alias_method :async_exec_defer, :exec_defer
|
578
582
|
alias_method :exec_params_defer, :exec_defer
|
579
583
|
|
580
|
-
# @!endgroup
|
581
|
-
|
582
584
|
# Asynchronously retrieves the next result from a call to
|
583
585
|
# #send_query (or another asynchronous command) and immediately
|
584
586
|
# returns with a Deferrable.
|
@@ -595,8 +597,16 @@ module PG
|
|
595
597
|
def get_result_defer(&blk)
|
596
598
|
begin
|
597
599
|
df = FeaturedDeferrable.new(&blk)
|
598
|
-
|
599
|
-
|
600
|
+
if status == CONNECTION_OK
|
601
|
+
if is_busy
|
602
|
+
check_async_command_aborted!
|
603
|
+
setup_emio_watcher.watch_results(df, nil, true)
|
604
|
+
else
|
605
|
+
df.succeed blocking_get_result
|
606
|
+
end
|
607
|
+
else
|
608
|
+
df.succeed
|
609
|
+
end
|
600
610
|
rescue Error => e
|
601
611
|
::EM.next_tick { async_autoreconnect!(df, e) }
|
602
612
|
rescue Exception => e
|
@@ -605,8 +615,6 @@ module PG
|
|
605
615
|
df
|
606
616
|
end
|
607
617
|
|
608
|
-
alias_method :blocking_get_result, :get_result
|
609
|
-
|
610
618
|
# Asynchronously retrieves all available results on the current
|
611
619
|
# connection (from previously issued asynchronous commands like
|
612
620
|
# +send_query()+) and immediately returns with a Deferrable.
|
@@ -623,8 +631,12 @@ module PG
|
|
623
631
|
def get_last_result_defer(&blk)
|
624
632
|
begin
|
625
633
|
df = FeaturedDeferrable.new(&blk)
|
626
|
-
|
627
|
-
|
634
|
+
if status == CONNECTION_OK
|
635
|
+
check_async_command_aborted!
|
636
|
+
setup_emio_watcher.watch_results(df)
|
637
|
+
else
|
638
|
+
df.succeed
|
639
|
+
end
|
628
640
|
rescue Error => e
|
629
641
|
::EM.next_tick { async_autoreconnect!(df, e) }
|
630
642
|
rescue Exception => e
|
@@ -633,6 +645,10 @@ module PG
|
|
633
645
|
df
|
634
646
|
end
|
635
647
|
|
648
|
+
# @!endgroup
|
649
|
+
|
650
|
+
alias_method :blocking_get_result, :get_result
|
651
|
+
|
636
652
|
def raise_error(klass=Error, message=error_message)
|
637
653
|
error = klass.new(message)
|
638
654
|
error.instance_variable_set(:@connection, self)
|
@@ -641,6 +657,15 @@ module PG
|
|
641
657
|
|
642
658
|
private
|
643
659
|
|
660
|
+
def fiber_sync(df, fiber)
|
661
|
+
f = nil
|
662
|
+
df.completion do |res|
|
663
|
+
if f then f.resume res else return res end
|
664
|
+
end
|
665
|
+
f = fiber
|
666
|
+
Fiber.yield
|
667
|
+
end
|
668
|
+
|
644
669
|
def check_async_command_aborted!
|
645
670
|
if @async_command_aborted
|
646
671
|
raise_error ConnectionBad, "previous query expired, need connection reset"
|
@@ -648,17 +673,10 @@ module PG
|
|
648
673
|
end
|
649
674
|
|
650
675
|
def setup_emio_watcher
|
651
|
-
|
652
|
-
|
653
|
-
raise_error ConnectionBad
|
654
|
-
when CONNECTION_OK
|
655
|
-
if @watcher && @watcher.watching?
|
656
|
-
@watcher
|
657
|
-
else
|
658
|
-
@watcher = ::EM.watch(self.socket_io, Watcher, self)
|
659
|
-
end
|
676
|
+
if @watcher && @watcher.watching?
|
677
|
+
@watcher
|
660
678
|
else
|
661
|
-
|
679
|
+
@watcher = ::EM.watch(self.socket_io, Watcher, self)
|
662
680
|
end
|
663
681
|
end
|
664
682
|
|
@@ -680,7 +698,7 @@ module PG
|
|
680
698
|
# @return [PG::Result] if block wasn't given
|
681
699
|
# @return [Object] result of the given block
|
682
700
|
|
683
|
-
# @!group Auto-sensing
|
701
|
+
# @!group Auto-sensing fiber-synchronized command methods
|
684
702
|
|
685
703
|
# @!method exec(sql, &blk)
|
686
704
|
# Sends SQL query request specified by +sql+ to PostgreSQL.
|
@@ -730,17 +748,6 @@ module PG
|
|
730
748
|
# @see PG::EM::Client#describe_portal_defer
|
731
749
|
# @see http://deveiate.org/code/pg/PG/Connection.html#method-i-describe_portal PG::Connection#describe_portal
|
732
750
|
#
|
733
|
-
# @!method get_result(&blk)
|
734
|
-
# Retrieves the next result from a call to #send_query (or another
|
735
|
-
# asynchronous command). If no more results are available returns
|
736
|
-
# +nil+ and the block (if given) is never called.
|
737
|
-
#
|
738
|
-
# @macro auto_synchrony_api
|
739
|
-
# @return [nil] if no more results
|
740
|
-
#
|
741
|
-
# @see #get_result_defer
|
742
|
-
# @see http://deveiate.org/code/pg/PG/Connection.html#method-i-get_result PG::Connection#get_result
|
743
|
-
#
|
744
751
|
# @!method get_last_result
|
745
752
|
# Retrieves all available results on the current connection
|
746
753
|
# (from previously issued asynchronous commands like +send_query()+)
|
@@ -759,24 +766,15 @@ module PG
|
|
759
766
|
prepare prepare_defer
|
760
767
|
describe_prepared describe_prepared_defer
|
761
768
|
describe_portal describe_portal_defer
|
762
|
-
get_result get_result_defer
|
763
769
|
get_last_result get_last_result_defer
|
764
770
|
).each_slice(2) do |name, defer_name|
|
765
771
|
|
766
772
|
class_eval <<-EOD, __FILE__, __LINE__
|
767
773
|
def #{name}(*args, &blk)
|
768
774
|
if ::EM.reactor_running? && !(f = Fiber.current).equal?(ROOT_FIBER)
|
769
|
-
result =
|
770
|
-
|
771
|
-
f = nil
|
772
|
-
if fiber
|
773
|
-
fiber.resume(res)
|
774
|
-
else
|
775
|
-
result = res
|
776
|
-
end
|
775
|
+
if (result = fiber_sync #{defer_name}(*args), f).is_a?(::Exception)
|
776
|
+
raise result
|
777
777
|
end
|
778
|
-
result = Fiber.yield if (fiber = f)
|
779
|
-
raise result if result.is_a?(::Exception)
|
780
778
|
if block_given? && result
|
781
779
|
begin
|
782
780
|
yield result
|
@@ -797,6 +795,34 @@ module PG
|
|
797
795
|
alias_method :async_query, :exec
|
798
796
|
alias_method :async_exec, :exec
|
799
797
|
|
798
|
+
# Retrieves the next result from a call to #send_query (or another
|
799
|
+
# asynchronous command). If no more results are available returns
|
800
|
+
# +nil+ and the block (if given) is never called.
|
801
|
+
#
|
802
|
+
# @macro auto_synchrony_api
|
803
|
+
# @return [nil] if no more results
|
804
|
+
#
|
805
|
+
# @see #get_result_defer
|
806
|
+
# @see http://deveiate.org/code/pg/PG/Connection.html#method-i-get_result PG::Connection#get_result
|
807
|
+
def get_result
|
808
|
+
if is_busy && ::EM.reactor_running? && !(f = Fiber.current).equal?(ROOT_FIBER)
|
809
|
+
if (result = fiber_sync get_result_defer, f).is_a?(::Exception)
|
810
|
+
raise result
|
811
|
+
end
|
812
|
+
if block_given? && result
|
813
|
+
begin
|
814
|
+
yield result
|
815
|
+
ensure
|
816
|
+
result.clear
|
817
|
+
end
|
818
|
+
else
|
819
|
+
result
|
820
|
+
end
|
821
|
+
else
|
822
|
+
super
|
823
|
+
end
|
824
|
+
end
|
825
|
+
|
800
826
|
# @!endgroup
|
801
827
|
|
802
828
|
|
data/lib/pg/em/client/watcher.rb
CHANGED
@@ -27,23 +27,12 @@ module PG
|
|
27
27
|
@deferrable = deferrable
|
28
28
|
@send_proc = send_proc
|
29
29
|
cancel_timer
|
30
|
-
|
31
|
-
|
32
|
-
|
33
|
-
|
34
|
-
@notify_timestamp = Time.now
|
35
|
-
setup_timer timeout
|
36
|
-
end
|
37
|
-
else
|
38
|
-
@deferrable.protect do
|
39
|
-
@client.raise_error ConnectionBad
|
40
|
-
end
|
41
|
-
end
|
42
|
-
else
|
43
|
-
self.notify_readable = true
|
44
|
-
fetch_results
|
30
|
+
self.notify_readable = true
|
31
|
+
if (timeout = @client.query_timeout) > 0
|
32
|
+
@notify_timestamp = Time.now
|
33
|
+
setup_timer timeout
|
45
34
|
end
|
46
|
-
|
35
|
+
fetch_results
|
47
36
|
end
|
48
37
|
|
49
38
|
def setup_timer(timeout, adjustment = 0)
|
@@ -68,11 +57,18 @@ module PG
|
|
68
57
|
end
|
69
58
|
end
|
70
59
|
|
60
|
+
def notify_readable
|
61
|
+
@client.consume_input
|
62
|
+
rescue Exception => e
|
63
|
+
handle_error(e)
|
64
|
+
else
|
65
|
+
fetch_results
|
66
|
+
end
|
67
|
+
|
71
68
|
# Carefully extract results without
|
72
69
|
# blocking the EventMachine reactor.
|
73
70
|
def fetch_results
|
74
71
|
result = false
|
75
|
-
@client.consume_input
|
76
72
|
until @client.is_busy
|
77
73
|
single_result = @client.blocking_get_result
|
78
74
|
if one_result_mode?
|
@@ -88,18 +84,7 @@ module PG
|
|
88
84
|
@last_result = single_result
|
89
85
|
end
|
90
86
|
rescue Exception => e
|
91
|
-
|
92
|
-
cancel_timer
|
93
|
-
send_proc = @send_proc
|
94
|
-
@send_proc = nil
|
95
|
-
df = @deferrable
|
96
|
-
# prevent unbind error on auto re-connect
|
97
|
-
@deferrable = false
|
98
|
-
if e.is_a?(PG::Error)
|
99
|
-
@client.async_autoreconnect!(df, e, &send_proc)
|
100
|
-
else
|
101
|
-
df.fail(e)
|
102
|
-
end
|
87
|
+
handle_error(e)
|
103
88
|
else
|
104
89
|
if result == false
|
105
90
|
@notify_timestamp = Time.now if @timer
|
@@ -111,14 +96,31 @@ module PG
|
|
111
96
|
end
|
112
97
|
end
|
113
98
|
|
114
|
-
alias_method :notify_readable, :fetch_results
|
115
|
-
|
116
99
|
def unbind
|
117
100
|
@is_connected = false
|
118
101
|
@deferrable.protect do
|
102
|
+
cancel_timer
|
119
103
|
@client.raise_error ConnectionBad, "connection reset"
|
120
104
|
end if @deferrable
|
121
105
|
end
|
106
|
+
|
107
|
+
private
|
108
|
+
|
109
|
+
def handle_error(e)
|
110
|
+
self.notify_readable = false
|
111
|
+
cancel_timer
|
112
|
+
send_proc = @send_proc
|
113
|
+
@send_proc = nil
|
114
|
+
df = @deferrable
|
115
|
+
# prevent unbind error on auto re-connect
|
116
|
+
@deferrable = false
|
117
|
+
if e.is_a?(PG::Error)
|
118
|
+
@client.async_autoreconnect!(df, e, &send_proc)
|
119
|
+
else
|
120
|
+
df.fail(e)
|
121
|
+
end
|
122
|
+
end
|
123
|
+
|
122
124
|
end
|
123
125
|
|
124
126
|
end
|
@@ -332,30 +332,40 @@ describe 'pg-em with autoreconnect disabled' do
|
|
332
332
|
end
|
333
333
|
|
334
334
|
it "should fail to get last result asynchronously after server restart" do
|
335
|
+
check_get_last_result = proc do
|
336
|
+
@client.get_last_result_defer do |result|
|
337
|
+
result.should be_nil
|
338
|
+
@client.reset_defer do |conn|
|
339
|
+
conn.should be @client
|
340
|
+
@client.status.should be PG::CONNECTION_OK
|
341
|
+
EM.stop
|
342
|
+
end.should be_a_kind_of ::EM::DefaultDeferrable
|
343
|
+
end
|
344
|
+
end
|
335
345
|
system($pgserver_cmd_stop).should be_true
|
336
346
|
system($pgserver_cmd_start).should be_true
|
337
347
|
begin
|
338
348
|
@client.send_query('SELECT pg_sleep(5); SELECT pg_database_size(current_database());')
|
339
349
|
rescue PG::UnableToSend
|
340
|
-
end
|
341
|
-
@client.get_last_result_defer do |ex|
|
342
|
-
ex.should be_an_instance_of PG::ConnectionBad
|
343
350
|
@client.status.should be PG::CONNECTION_BAD
|
351
|
+
@client.get_last_result_defer do |ex|
|
352
|
+
ex.should be_nil
|
353
|
+
@client.status.should be PG::CONNECTION_BAD
|
354
|
+
check_get_last_result.call
|
355
|
+
end.should be_a_kind_of EM::DefaultDeferrable
|
356
|
+
else
|
344
357
|
@client.get_last_result_defer do |ex|
|
345
358
|
ex.should be_an_instance_of PG::ConnectionBad
|
346
|
-
@client.
|
347
|
-
|
348
|
-
|
349
|
-
|
350
|
-
end.should be_a_kind_of ::EM::DefaultDeferrable
|
351
|
-
end
|
352
|
-
end.should be_a_kind_of EM::DefaultDeferrable
|
359
|
+
@client.status.should be PG::CONNECTION_BAD
|
360
|
+
check_get_last_result.call
|
361
|
+
end.should be_a_kind_of EM::DefaultDeferrable
|
362
|
+
end
|
353
363
|
end
|
354
364
|
|
355
365
|
it "should fail to get each result asynchronously after server restart" do
|
356
|
-
check_get_result = proc do
|
357
|
-
@client.get_result_defer do |
|
358
|
-
|
366
|
+
check_get_result = proc do |expected_class|
|
367
|
+
@client.get_result_defer do |result|
|
368
|
+
result.should be_an_instance_of expected_class
|
359
369
|
@client.status.should be PG::CONNECTION_BAD
|
360
370
|
@client.reset_defer do |conn|
|
361
371
|
conn.should be @client
|
@@ -370,9 +380,9 @@ describe 'pg-em with autoreconnect disabled' do
|
|
370
380
|
@client.send_query('SELECT pg_sleep(5); SELECT pg_database_size(current_database());')
|
371
381
|
rescue PG::UnableToSend
|
372
382
|
@client.get_result_defer do |result|
|
373
|
-
result.should
|
383
|
+
result.should be_nil
|
374
384
|
@client.status.should be PG::CONNECTION_BAD
|
375
|
-
check_get_result.call
|
385
|
+
check_get_result.call NilClass
|
376
386
|
end
|
377
387
|
else
|
378
388
|
@client.get_result_defer do |result|
|
@@ -381,7 +391,7 @@ describe 'pg-em with autoreconnect disabled' do
|
|
381
391
|
result.check
|
382
392
|
end.to raise_error PG::Error
|
383
393
|
@client.status.should be PG::CONNECTION_OK
|
384
|
-
check_get_result.call
|
394
|
+
check_get_result.call PG::ConnectionBad
|
385
395
|
end
|
386
396
|
end
|
387
397
|
end
|
data/spec/em_client_common.rb
CHANGED
@@ -359,4 +359,44 @@ shared_context 'em-pg common after' do
|
|
359
359
|
end.should be_a_kind_of ::EM::DefaultDeferrable
|
360
360
|
end
|
361
361
|
|
362
|
+
it "should raise connection reset on connection breakdown" do
|
363
|
+
client = described_class.new query_timeout: 0.1
|
364
|
+
client.send_query('SELECT pg_sleep(10)')
|
365
|
+
client.get_last_result_defer do |ex|
|
366
|
+
ex.should be_an_instance_of PG::ConnectionBad
|
367
|
+
ex.message.should match /connection reset/
|
368
|
+
EM.add_timer(0.2) do
|
369
|
+
client.async_command_aborted.should be_false
|
370
|
+
EM.stop
|
371
|
+
end
|
372
|
+
end
|
373
|
+
# simulate connection breakdown
|
374
|
+
client.finish
|
375
|
+
end
|
376
|
+
|
377
|
+
it "should return nil result after async reset began" do
|
378
|
+
checkpoint = 0
|
379
|
+
@client.send_query('SELECT 1')
|
380
|
+
@client.reset_defer do |conn|
|
381
|
+
conn.should be @client
|
382
|
+
EM.next_tick do
|
383
|
+
checkpoint.should eq 2
|
384
|
+
@client.send_query('SELECT 1')
|
385
|
+
@client.get_last_result_defer do |result|
|
386
|
+
result.should be_an_instance_of PG::Result
|
387
|
+
result.getvalue(0,0).should eq '1'
|
388
|
+
EM.stop
|
389
|
+
end
|
390
|
+
end
|
391
|
+
end
|
392
|
+
@client.get_result_defer do |result|
|
393
|
+
result.should be_nil
|
394
|
+
checkpoint += 1
|
395
|
+
end
|
396
|
+
@client.get_last_result_defer do |result|
|
397
|
+
result.should be_nil
|
398
|
+
checkpoint += 1
|
399
|
+
end
|
400
|
+
end
|
401
|
+
|
362
402
|
end
|
data/spec/em_synchrony_client.rb
CHANGED
@@ -338,6 +338,11 @@ describe PG::EM::Client do
|
|
338
338
|
|
339
339
|
describe 'PG::EM::Client#transaction' do
|
340
340
|
|
341
|
+
before(:all) do
|
342
|
+
@client.query_timeout = 0
|
343
|
+
@client.query_timeout.should eq 0
|
344
|
+
end
|
345
|
+
|
341
346
|
it "should raise ArgumentError when there is no block" do
|
342
347
|
expect do
|
343
348
|
@client.transaction
|
@@ -213,14 +213,15 @@ describe 'em-synchrony-pg with autoreconnect disabled' do
|
|
213
213
|
begin
|
214
214
|
@client.send_query('SELECT pg_sleep(5); SELECT pg_database_size(current_database());')
|
215
215
|
rescue PG::UnableToSend
|
216
|
+
@client.status.should be PG::CONNECTION_BAD
|
217
|
+
@client.get_last_result.should be_nil
|
218
|
+
else
|
219
|
+
expect do
|
220
|
+
@client.get_last_result
|
221
|
+
end.to raise_error PG::ConnectionBad
|
216
222
|
end
|
217
|
-
expect do
|
218
|
-
@client.get_last_result
|
219
|
-
end.to raise_error PG::ConnectionBad
|
220
223
|
@client.status.should be PG::CONNECTION_BAD
|
221
|
-
|
222
|
-
@client.get_last_result
|
223
|
-
end.to raise_error PG::ConnectionBad
|
224
|
+
@client.get_last_result.should be_nil
|
224
225
|
@client.reset
|
225
226
|
@client.status.should be PG::CONNECTION_OK
|
226
227
|
@client.get_last_result.should be_nil
|
@@ -233,10 +234,8 @@ describe 'em-synchrony-pg with autoreconnect disabled' do
|
|
233
234
|
begin
|
234
235
|
@client.send_query('SELECT pg_sleep(5); SELECT pg_database_size(current_database());')
|
235
236
|
rescue PG::UnableToSend
|
236
|
-
expect do
|
237
|
-
@client.get_result
|
238
|
-
end.to raise_error PG::ConnectionBad
|
239
237
|
@client.status.should be PG::CONNECTION_BAD
|
238
|
+
@client.get_result.should be_nil
|
240
239
|
else
|
241
240
|
result = @client.get_result
|
242
241
|
result.should be_an_instance_of PG::Result
|
@@ -244,10 +243,12 @@ describe 'em-synchrony-pg with autoreconnect disabled' do
|
|
244
243
|
result.check
|
245
244
|
end.to raise_error PG::Error
|
246
245
|
@client.status.should be PG::CONNECTION_OK
|
246
|
+
expect do
|
247
|
+
@client.get_result
|
248
|
+
end.to raise_error PG::ConnectionBad
|
247
249
|
end
|
248
|
-
|
249
|
-
|
250
|
-
end.to raise_error PG::ConnectionBad
|
250
|
+
@client.status.should be PG::CONNECTION_BAD
|
251
|
+
@client.get_result.should be_nil
|
251
252
|
@client.status.should be PG::CONNECTION_BAD
|
252
253
|
@client.reset
|
253
254
|
@client.status.should be PG::CONNECTION_OK
|
metadata
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: em-pg-client
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 0.3.
|
4
|
+
version: 0.3.2
|
5
5
|
prerelease:
|
6
6
|
platform: ruby
|
7
7
|
authors:
|
@@ -9,7 +9,7 @@ authors:
|
|
9
9
|
autorequire:
|
10
10
|
bindir: bin
|
11
11
|
cert_chain: []
|
12
|
-
date: 2014-01-
|
12
|
+
date: 2014-01-07 00:00:00.000000000 Z
|
13
13
|
dependencies:
|
14
14
|
- !ruby/object:Gem::Dependency
|
15
15
|
name: pg
|
@@ -112,6 +112,8 @@ email: rafal@yeondir.com
|
|
112
112
|
executables: []
|
113
113
|
extensions: []
|
114
114
|
extra_rdoc_files:
|
115
|
+
- benchmarks/em_pg.rb
|
116
|
+
- benchmarks/single_row_mode.rb
|
115
117
|
- README.md
|
116
118
|
- BENCHMARKS.md
|
117
119
|
- LICENSE
|
@@ -127,6 +129,7 @@ files:
|
|
127
129
|
- README.md
|
128
130
|
- Rakefile
|
129
131
|
- benchmarks/em_pg.rb
|
132
|
+
- benchmarks/single_row_mode.rb
|
130
133
|
- em-pg-client.gemspec
|
131
134
|
- examples/single_row_mode.rb
|
132
135
|
- lib/em-pg-client.rb
|