em-pg-client 0.3.1 → 0.3.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- data/.yardopts +1 -1
- data/BENCHMARKS.md +56 -13
- data/HISTORY.md +7 -0
- data/README.md +5 -5
- data/benchmarks/em_pg.rb +1 -1
- data/benchmarks/single_row_mode.rb +88 -0
- data/em-pg-client.gemspec +7 -2
- data/examples/single_row_mode.rb +1 -11
- data/lib/pg/em-version.rb +1 -1
- data/lib/pg/em.rb +67 -41
- data/lib/pg/em/client/watcher.rb +33 -31
- data/spec/em_client_autoreconnect.rb +26 -16
- data/spec/em_client_common.rb +40 -0
- data/spec/em_synchrony_client.rb +5 -0
- data/spec/em_synchrony_client_autoreconnect.rb +13 -12
- metadata +5 -2
data/.yardopts
CHANGED
@@ -1 +1 @@
|
|
1
|
-
lib/pg/em/client/*.rb lib/pg/em/*.rb lib/pg/*.rb - BENCHMARKS.md LICENSE HISTORY.md
|
1
|
+
lib/pg/em/client/*.rb lib/pg/em/*.rb lib/pg/*.rb - benchmarks/*.rb BENCHMARKS.md LICENSE HISTORY.md
|
data/BENCHMARKS.md
CHANGED
@@ -1,9 +1,24 @@
|
|
1
|
-
|
2
|
-
|
1
|
+
Benchmarking
|
2
|
+
============
|
3
3
|
|
4
|
-
|
4
|
+
Environment
|
5
|
+
-----------
|
6
|
+
|
7
|
+
The machine running tests is Linux CentOS 2.6.18-194.32.1.el5xen #1 SMP with
|
8
|
+
Quad Core Xeon X3360 @ 2.83GHz, 4GB RAM.
|
9
|
+
|
10
|
+
Postgres server version: 9.0.3
|
11
|
+
Postgres pqlib version: 9.3
|
12
|
+
|
13
|
+
Fully Asynchronous vs. poor man's async
|
14
|
+
---------------------------------------
|
15
|
+
|
16
|
+
The following {file:benchmarks/em_pg.rb benchmark} compares fully
|
17
|
+
asynchronous implementation (`em-pg-client`) versus blocking em-pg drivers.
|
18
|
+
|
19
|
+
The goal of the test is to retrieve (~80000) rows from the same table with
|
20
|
+
a lot of text data, in chunks, using parallel connections.
|
5
21
|
|
6
|
-
The goal of the test is simply to retrieve (~80000) rows from table with a lot of text data, in chunks, using parallel connections.
|
7
22
|
The parallel method uses synchrony for simplicity.
|
8
23
|
|
9
24
|
* `single` is (eventmachine-less) job for retrieving a whole data table in
|
@@ -15,14 +30,6 @@ The parallel method uses synchrony for simplicity.
|
|
15
30
|
that it uses special patched version of library that uses blocking
|
16
31
|
PGConnection methods
|
17
32
|
|
18
|
-
Environment
|
19
|
-
-----------
|
20
|
-
|
21
|
-
The machine used for test is Linux CentOS 2.6.18-194.32.1.el5xen #1 SMP with Quad Core Xeon X3360 @ 2.83GHz, 4GB RAM.
|
22
|
-
Postgres version used: 9.0.3.
|
23
|
-
|
24
|
-
The results:
|
25
|
-
------------
|
26
33
|
|
27
34
|
```
|
28
35
|
>> benchmark 1000
|
@@ -40,9 +47,45 @@ The results:
|
|
40
47
|
blocking 1000/20: 78.790000 3.230000 82.020000 (225.949107)
|
41
48
|
```
|
42
49
|
|
43
|
-
As we can see the gain from using asynchronous pg
|
50
|
+
As we can see the gain from using asynchronous em-pg-client while
|
44
51
|
using `parallel` queries is noticeable (up to ~30%).
|
45
52
|
|
46
53
|
The `blocking` client however doesn't gain much from parallel execution.
|
47
54
|
This was expected because it freezes eventmachine until the whole
|
48
55
|
dataset is consumed by the client.
|
56
|
+
|
57
|
+
|
58
|
+
Threads vs. Fibers Streaming Benchmark
|
59
|
+
--------------------------------------
|
60
|
+
|
61
|
+
The following {file:benchmarks/single_row_mode.rb benchmark} compares
|
62
|
+
performance of parallel running threads using vanilla PG::Connection driver
|
63
|
+
versus EventMachine driven parallel Fibers using PG::EM::Client v0.3.2.
|
64
|
+
|
65
|
+
Each thread/fiber retrieves first 5000 rows from the same table with
|
66
|
+
a lot of text data in a `single_row_mode`. After 5000 rows is retrieved
|
67
|
+
the connection is being reset. The process is repeated after all parallel
|
68
|
+
running threads/fibers finish their task.
|
69
|
+
|
70
|
+
Both Thread and Fiber versions use the same chunk of code to retrieve rows.
|
71
|
+
|
72
|
+
```
|
73
|
+
>> benchmark 400
|
74
|
+
user system total real
|
75
|
+
threads 400x1: 24.970000 1.090000 26.060000 ( 30.683818)
|
76
|
+
threads 80x5: 24.730000 7.020000 31.750000 ( 51.402710)
|
77
|
+
threads 40x10: 22.880000 7.460000 30.340000 ( 52.548910)
|
78
|
+
threads 20x20: 22.220000 7.130000 29.350000 ( 53.911111)
|
79
|
+
threads 10x40: 22.570000 7.620000 30.190000 ( 54.111841)
|
80
|
+
|
81
|
+
fibers 400x1: 26.040000 1.060000 27.100000 ( 31.619598)
|
82
|
+
fibers 80x5: 28.690000 1.140000 29.830000 ( 33.025573)
|
83
|
+
fibers 40x10: 28.790000 1.280000 30.070000 ( 33.498418)
|
84
|
+
fibers 20x20: 29.100000 1.210000 30.310000 ( 33.289344)
|
85
|
+
fibers 10x40: 29.220000 1.340000 30.560000 ( 33.691188)
|
86
|
+
```
|
87
|
+
|
88
|
+
```
|
89
|
+
AxB - repeat A times running B parallel threads/fibers.
|
90
|
+
```
|
91
|
+
|
data/HISTORY.md
CHANGED
@@ -1,3 +1,10 @@
|
|
1
|
+
0.3.2
|
2
|
+
|
3
|
+
- fix: asynchronous get_result performance
|
4
|
+
- fix+specs: query_timeout timer is canceled on connection breakdown
|
5
|
+
- comply with pg+specs: asynchronous get_result and get_last_result return nil
|
6
|
+
when connection status is not ok
|
7
|
+
|
1
8
|
0.3.1
|
2
9
|
|
3
10
|
- support for asynchronous data streaming in single row mode -
|
data/README.md
CHANGED
@@ -1,4 +1,5 @@
|
|
1
|
-
|
1
|
+
em-pg-client
|
2
|
+
============
|
2
3
|
|
3
4
|
The Ruby EventMachine driver interface to the PostgreSQL RDBMS. It is based on
|
4
5
|
[ruby-pg](https://bitbucket.org/ged/ruby-pg).
|
@@ -72,7 +73,7 @@ Features
|
|
72
73
|
* [Sequel Adapter](https://github.com/fl00r/em-pg-sequel) by Peter Yanovich.
|
73
74
|
* Works on windows (requires ruby 2.0) ([issue #7][Issue 7]).
|
74
75
|
* __New__ - supports asynchronous query data processing in single row mode
|
75
|
-
([issue #12][Issue 12]).
|
76
|
+
([issue #12][Issue 12]). See {file:BENCHMARKS.md BENCHMARKING}.
|
76
77
|
|
77
78
|
Requirements
|
78
79
|
------------
|
@@ -95,7 +96,7 @@ Install
|
|
95
96
|
#### Gemfile
|
96
97
|
|
97
98
|
```ruby
|
98
|
-
gem "em-pg-client", "~> 0.3.
|
99
|
+
gem "em-pg-client", "~> 0.3.2"
|
99
100
|
```
|
100
101
|
|
101
102
|
#### Github
|
@@ -365,11 +366,10 @@ Bugs/Limitations
|
|
365
366
|
TODO:
|
366
367
|
-----
|
367
368
|
|
368
|
-
*
|
369
|
+
* more convenient streaming API
|
369
370
|
* implement EM adapted version of `get_copy_data`, `put_copy_data`,
|
370
371
|
`wait_for_notify` and `transaction`
|
371
372
|
* ORM (ActiveRecord and maybe Datamapper) support as separate projects
|
372
|
-
* present more benchmarks
|
373
373
|
|
374
374
|
More Info
|
375
375
|
---------
|
data/benchmarks/em_pg.rb
CHANGED
@@ -0,0 +1,88 @@
|
|
1
|
+
$:.unshift('./lib')
|
2
|
+
require 'pg/em/connection_pool'
|
3
|
+
require 'em-synchrony'
|
4
|
+
require 'em-synchrony/fiber_iterator'
|
5
|
+
require 'pp'
|
6
|
+
require 'benchmark'
|
7
|
+
|
8
|
+
TABLE_NAME = 'resources'
|
9
|
+
LIMIT_ROWS = 5000
|
10
|
+
|
11
|
+
include EM::Synchrony
|
12
|
+
|
13
|
+
unless PG::EM::Client.single_row_mode?
|
14
|
+
raise 'compile pg against pqlib >= 9.2 to support single row mode'
|
15
|
+
end
|
16
|
+
|
17
|
+
def benchmark(repeat=40)
|
18
|
+
Benchmark.bm(20) do |b|
|
19
|
+
puts
|
20
|
+
b.report("threads #{repeat/1}x1:") { threads(repeat, 1) }
|
21
|
+
b.report("threads #{repeat/5}x5:") { threads(repeat, 5) }
|
22
|
+
b.report("threads #{repeat/10}x10:") { threads(repeat, 10) }
|
23
|
+
b.report("threads #{repeat/20}x20:") { threads(repeat, 20) }
|
24
|
+
b.report("threads #{repeat/40}x40:") { threads(repeat, 40) }
|
25
|
+
puts
|
26
|
+
b.report("fibers #{repeat/1}x1:") { fibers(repeat, 1) }
|
27
|
+
b.report("fibers #{repeat/5}x5:") { fibers(repeat, 5) }
|
28
|
+
b.report("fibers #{repeat/10}x10:") { fibers(repeat, 10) }
|
29
|
+
b.report("fibers #{repeat/20}x20:") { fibers(repeat, 20) }
|
30
|
+
b.report("fibers #{repeat/40}x40:") { fibers(repeat, 40) }
|
31
|
+
end
|
32
|
+
end
|
33
|
+
|
34
|
+
def threads(repeat, concurrency)
|
35
|
+
db = Hash.new { |pool, id| pool[id] = PG::Connection.new }
|
36
|
+
(repeat/concurrency).times do
|
37
|
+
(0...concurrency).map do |i|
|
38
|
+
Thread.new do
|
39
|
+
stream_results(db[i])
|
40
|
+
end
|
41
|
+
end.each(&:join)
|
42
|
+
end
|
43
|
+
db.each_value(&:finish).clear
|
44
|
+
end
|
45
|
+
|
46
|
+
def fibers(repeat, concurrency)
|
47
|
+
EM.synchrony do
|
48
|
+
db = PG::EM::ConnectionPool.new size: concurrency, lazy: true
|
49
|
+
(repeat/concurrency).times do
|
50
|
+
FiberIterator.new((0...concurrency), concurrency).each do
|
51
|
+
db.hold do |pg|
|
52
|
+
stream_results(pg)
|
53
|
+
end
|
54
|
+
end
|
55
|
+
end
|
56
|
+
db.finish
|
57
|
+
EM.stop
|
58
|
+
end
|
59
|
+
end
|
60
|
+
|
61
|
+
def stream_results(pg)
|
62
|
+
pg.send_query("select * from #{TABLE_NAME}")
|
63
|
+
pg.set_single_row_mode
|
64
|
+
rows = 0
|
65
|
+
last_time = Time.now
|
66
|
+
while result = pg.get_result
|
67
|
+
begin
|
68
|
+
result.check
|
69
|
+
result.each do |tuple|
|
70
|
+
rows += 1
|
71
|
+
if rows >= LIMIT_ROWS
|
72
|
+
pg.reset
|
73
|
+
break
|
74
|
+
end
|
75
|
+
end
|
76
|
+
rescue PG::Error => e
|
77
|
+
pg.get_last_result
|
78
|
+
raise e
|
79
|
+
ensure
|
80
|
+
result.clear
|
81
|
+
end
|
82
|
+
end
|
83
|
+
end
|
84
|
+
|
85
|
+
if $0 == __FILE__
|
86
|
+
benchmark ARGV[0].to_i.nonzero? || 40
|
87
|
+
end
|
88
|
+
|
data/em-pg-client.gemspec
CHANGED
@@ -1,6 +1,8 @@
|
|
1
1
|
$:.unshift "lib"
|
2
2
|
require 'pg/em-version'
|
3
3
|
|
4
|
+
files = `git ls-files`.split("\n")
|
5
|
+
|
4
6
|
Gem::Specification.new do |s|
|
5
7
|
s.name = "em-pg-client"
|
6
8
|
s.version = PG::EM::VERSION
|
@@ -13,12 +15,15 @@ Gem::Specification.new do |s|
|
|
13
15
|
s.require_path = "lib"
|
14
16
|
s.description = "PostgreSQL asynchronous EventMachine client, based on pg interface (PG::Connection)"
|
15
17
|
s.authors = ["Rafal Michalski"]
|
16
|
-
s.files =
|
18
|
+
s.files = files - ['.gitignore']
|
17
19
|
s.test_files = Dir.glob("spec/**/*")
|
18
20
|
s.rdoc_options << "--title" << "em-pg-client" <<
|
19
21
|
"--main" << "README.md"
|
20
22
|
s.has_rdoc = true
|
21
|
-
s.extra_rdoc_files = [
|
23
|
+
s.extra_rdoc_files = [
|
24
|
+
files.grep(/^benchmarks\/.*\.rb$/),
|
25
|
+
"README.md", "BENCHMARKS.md", "LICENSE", "HISTORY.md"
|
26
|
+
].flatten
|
22
27
|
s.requirements << "PostgreSQL server"
|
23
28
|
s.add_runtime_dependency "pg", ">= 0.17.0"
|
24
29
|
s.add_runtime_dependency "eventmachine", "~> 1.0.0"
|
data/examples/single_row_mode.rb
CHANGED
@@ -4,7 +4,7 @@
|
|
4
4
|
#
|
5
5
|
# upside: it's the same way you would work with PG::Connection
|
6
6
|
# downside: it's a little verbose and doesn't support automatic re-connects
|
7
|
-
gem 'em-pg-client', '>= 0.3.
|
7
|
+
gem 'em-pg-client', '>= 0.3.2'
|
8
8
|
require 'pg/em/connection_pool'
|
9
9
|
require 'em-synchrony'
|
10
10
|
require 'em-synchrony/fiber_iterator'
|
@@ -15,12 +15,6 @@ unless PG::EM::Client.single_row_mode?
|
|
15
15
|
raise 'compile pg against pqlib >= 9.2 to support single row mode'
|
16
16
|
end
|
17
17
|
|
18
|
-
def tick_sleep
|
19
|
-
f = Fiber.current
|
20
|
-
EM.next_tick { f.resume }
|
21
|
-
Fiber.yield
|
22
|
-
end
|
23
|
-
|
24
18
|
EM.synchrony do
|
25
19
|
EM.add_periodic_timer(0.01) { print ' ' }
|
26
20
|
|
@@ -41,10 +35,6 @@ EM.synchrony do
|
|
41
35
|
rows += 1
|
42
36
|
# process tuple
|
43
37
|
print mark
|
44
|
-
if (rows % 10).zero?
|
45
|
-
# let reactor do some work if data is coming too fast
|
46
|
-
tick_sleep
|
47
|
-
end
|
48
38
|
# break stream cleanly
|
49
39
|
pg.reset if rows > 1000
|
50
40
|
end
|
data/lib/pg/em-version.rb
CHANGED
data/lib/pg/em.rb
CHANGED
@@ -34,6 +34,8 @@ module PG
|
|
34
34
|
# - {#exec_prepared_defer}
|
35
35
|
# - {#describe_prepared_defer}
|
36
36
|
# - {#describe_portal_defer}
|
37
|
+
# - {#get_result_defer}
|
38
|
+
# - {#get_last_result_defer}
|
37
39
|
#
|
38
40
|
# are added to execute queries asynchronously,
|
39
41
|
# returning +Deferrable+ object.
|
@@ -47,6 +49,8 @@ module PG
|
|
47
49
|
# - {#exec_prepared}
|
48
50
|
# - {#describe_prepared}
|
49
51
|
# - {#describe_portal}
|
52
|
+
# - {#get_result}
|
53
|
+
# - {#get_last_result}
|
50
54
|
#
|
51
55
|
# and are now auto-detecting if EventMachine is running and
|
52
56
|
# performing commands asynchronously (blocking only current fiber) or
|
@@ -577,8 +581,6 @@ module PG
|
|
577
581
|
alias_method :async_exec_defer, :exec_defer
|
578
582
|
alias_method :exec_params_defer, :exec_defer
|
579
583
|
|
580
|
-
# @!endgroup
|
581
|
-
|
582
584
|
# Asynchronously retrieves the next result from a call to
|
583
585
|
# #send_query (or another asynchronous command) and immediately
|
584
586
|
# returns with a Deferrable.
|
@@ -595,8 +597,16 @@ module PG
|
|
595
597
|
def get_result_defer(&blk)
|
596
598
|
begin
|
597
599
|
df = FeaturedDeferrable.new(&blk)
|
598
|
-
|
599
|
-
|
600
|
+
if status == CONNECTION_OK
|
601
|
+
if is_busy
|
602
|
+
check_async_command_aborted!
|
603
|
+
setup_emio_watcher.watch_results(df, nil, true)
|
604
|
+
else
|
605
|
+
df.succeed blocking_get_result
|
606
|
+
end
|
607
|
+
else
|
608
|
+
df.succeed
|
609
|
+
end
|
600
610
|
rescue Error => e
|
601
611
|
::EM.next_tick { async_autoreconnect!(df, e) }
|
602
612
|
rescue Exception => e
|
@@ -605,8 +615,6 @@ module PG
|
|
605
615
|
df
|
606
616
|
end
|
607
617
|
|
608
|
-
alias_method :blocking_get_result, :get_result
|
609
|
-
|
610
618
|
# Asynchronously retrieves all available results on the current
|
611
619
|
# connection (from previously issued asynchronous commands like
|
612
620
|
# +send_query()+) and immediately returns with a Deferrable.
|
@@ -623,8 +631,12 @@ module PG
|
|
623
631
|
def get_last_result_defer(&blk)
|
624
632
|
begin
|
625
633
|
df = FeaturedDeferrable.new(&blk)
|
626
|
-
|
627
|
-
|
634
|
+
if status == CONNECTION_OK
|
635
|
+
check_async_command_aborted!
|
636
|
+
setup_emio_watcher.watch_results(df)
|
637
|
+
else
|
638
|
+
df.succeed
|
639
|
+
end
|
628
640
|
rescue Error => e
|
629
641
|
::EM.next_tick { async_autoreconnect!(df, e) }
|
630
642
|
rescue Exception => e
|
@@ -633,6 +645,10 @@ module PG
|
|
633
645
|
df
|
634
646
|
end
|
635
647
|
|
648
|
+
# @!endgroup
|
649
|
+
|
650
|
+
alias_method :blocking_get_result, :get_result
|
651
|
+
|
636
652
|
def raise_error(klass=Error, message=error_message)
|
637
653
|
error = klass.new(message)
|
638
654
|
error.instance_variable_set(:@connection, self)
|
@@ -641,6 +657,15 @@ module PG
|
|
641
657
|
|
642
658
|
private
|
643
659
|
|
660
|
+
def fiber_sync(df, fiber)
|
661
|
+
f = nil
|
662
|
+
df.completion do |res|
|
663
|
+
if f then f.resume res else return res end
|
664
|
+
end
|
665
|
+
f = fiber
|
666
|
+
Fiber.yield
|
667
|
+
end
|
668
|
+
|
644
669
|
def check_async_command_aborted!
|
645
670
|
if @async_command_aborted
|
646
671
|
raise_error ConnectionBad, "previous query expired, need connection reset"
|
@@ -648,17 +673,10 @@ module PG
|
|
648
673
|
end
|
649
674
|
|
650
675
|
def setup_emio_watcher
|
651
|
-
|
652
|
-
|
653
|
-
raise_error ConnectionBad
|
654
|
-
when CONNECTION_OK
|
655
|
-
if @watcher && @watcher.watching?
|
656
|
-
@watcher
|
657
|
-
else
|
658
|
-
@watcher = ::EM.watch(self.socket_io, Watcher, self)
|
659
|
-
end
|
676
|
+
if @watcher && @watcher.watching?
|
677
|
+
@watcher
|
660
678
|
else
|
661
|
-
|
679
|
+
@watcher = ::EM.watch(self.socket_io, Watcher, self)
|
662
680
|
end
|
663
681
|
end
|
664
682
|
|
@@ -680,7 +698,7 @@ module PG
|
|
680
698
|
# @return [PG::Result] if block wasn't given
|
681
699
|
# @return [Object] result of the given block
|
682
700
|
|
683
|
-
# @!group Auto-sensing
|
701
|
+
# @!group Auto-sensing fiber-synchronized command methods
|
684
702
|
|
685
703
|
# @!method exec(sql, &blk)
|
686
704
|
# Sends SQL query request specified by +sql+ to PostgreSQL.
|
@@ -730,17 +748,6 @@ module PG
|
|
730
748
|
# @see PG::EM::Client#describe_portal_defer
|
731
749
|
# @see http://deveiate.org/code/pg/PG/Connection.html#method-i-describe_portal PG::Connection#describe_portal
|
732
750
|
#
|
733
|
-
# @!method get_result(&blk)
|
734
|
-
# Retrieves the next result from a call to #send_query (or another
|
735
|
-
# asynchronous command). If no more results are available returns
|
736
|
-
# +nil+ and the block (if given) is never called.
|
737
|
-
#
|
738
|
-
# @macro auto_synchrony_api
|
739
|
-
# @return [nil] if no more results
|
740
|
-
#
|
741
|
-
# @see #get_result_defer
|
742
|
-
# @see http://deveiate.org/code/pg/PG/Connection.html#method-i-get_result PG::Connection#get_result
|
743
|
-
#
|
744
751
|
# @!method get_last_result
|
745
752
|
# Retrieves all available results on the current connection
|
746
753
|
# (from previously issued asynchronous commands like +send_query()+)
|
@@ -759,24 +766,15 @@ module PG
|
|
759
766
|
prepare prepare_defer
|
760
767
|
describe_prepared describe_prepared_defer
|
761
768
|
describe_portal describe_portal_defer
|
762
|
-
get_result get_result_defer
|
763
769
|
get_last_result get_last_result_defer
|
764
770
|
).each_slice(2) do |name, defer_name|
|
765
771
|
|
766
772
|
class_eval <<-EOD, __FILE__, __LINE__
|
767
773
|
def #{name}(*args, &blk)
|
768
774
|
if ::EM.reactor_running? && !(f = Fiber.current).equal?(ROOT_FIBER)
|
769
|
-
result =
|
770
|
-
|
771
|
-
f = nil
|
772
|
-
if fiber
|
773
|
-
fiber.resume(res)
|
774
|
-
else
|
775
|
-
result = res
|
776
|
-
end
|
775
|
+
if (result = fiber_sync #{defer_name}(*args), f).is_a?(::Exception)
|
776
|
+
raise result
|
777
777
|
end
|
778
|
-
result = Fiber.yield if (fiber = f)
|
779
|
-
raise result if result.is_a?(::Exception)
|
780
778
|
if block_given? && result
|
781
779
|
begin
|
782
780
|
yield result
|
@@ -797,6 +795,34 @@ module PG
|
|
797
795
|
alias_method :async_query, :exec
|
798
796
|
alias_method :async_exec, :exec
|
799
797
|
|
798
|
+
# Retrieves the next result from a call to #send_query (or another
|
799
|
+
# asynchronous command). If no more results are available returns
|
800
|
+
# +nil+ and the block (if given) is never called.
|
801
|
+
#
|
802
|
+
# @macro auto_synchrony_api
|
803
|
+
# @return [nil] if no more results
|
804
|
+
#
|
805
|
+
# @see #get_result_defer
|
806
|
+
# @see http://deveiate.org/code/pg/PG/Connection.html#method-i-get_result PG::Connection#get_result
|
807
|
+
def get_result
|
808
|
+
if is_busy && ::EM.reactor_running? && !(f = Fiber.current).equal?(ROOT_FIBER)
|
809
|
+
if (result = fiber_sync get_result_defer, f).is_a?(::Exception)
|
810
|
+
raise result
|
811
|
+
end
|
812
|
+
if block_given? && result
|
813
|
+
begin
|
814
|
+
yield result
|
815
|
+
ensure
|
816
|
+
result.clear
|
817
|
+
end
|
818
|
+
else
|
819
|
+
result
|
820
|
+
end
|
821
|
+
else
|
822
|
+
super
|
823
|
+
end
|
824
|
+
end
|
825
|
+
|
800
826
|
# @!endgroup
|
801
827
|
|
802
828
|
|
data/lib/pg/em/client/watcher.rb
CHANGED
@@ -27,23 +27,12 @@ module PG
|
|
27
27
|
@deferrable = deferrable
|
28
28
|
@send_proc = send_proc
|
29
29
|
cancel_timer
|
30
|
-
|
31
|
-
|
32
|
-
|
33
|
-
|
34
|
-
@notify_timestamp = Time.now
|
35
|
-
setup_timer timeout
|
36
|
-
end
|
37
|
-
else
|
38
|
-
@deferrable.protect do
|
39
|
-
@client.raise_error ConnectionBad
|
40
|
-
end
|
41
|
-
end
|
42
|
-
else
|
43
|
-
self.notify_readable = true
|
44
|
-
fetch_results
|
30
|
+
self.notify_readable = true
|
31
|
+
if (timeout = @client.query_timeout) > 0
|
32
|
+
@notify_timestamp = Time.now
|
33
|
+
setup_timer timeout
|
45
34
|
end
|
46
|
-
|
35
|
+
fetch_results
|
47
36
|
end
|
48
37
|
|
49
38
|
def setup_timer(timeout, adjustment = 0)
|
@@ -68,11 +57,18 @@ module PG
|
|
68
57
|
end
|
69
58
|
end
|
70
59
|
|
60
|
+
def notify_readable
|
61
|
+
@client.consume_input
|
62
|
+
rescue Exception => e
|
63
|
+
handle_error(e)
|
64
|
+
else
|
65
|
+
fetch_results
|
66
|
+
end
|
67
|
+
|
71
68
|
# Carefully extract results without
|
72
69
|
# blocking the EventMachine reactor.
|
73
70
|
def fetch_results
|
74
71
|
result = false
|
75
|
-
@client.consume_input
|
76
72
|
until @client.is_busy
|
77
73
|
single_result = @client.blocking_get_result
|
78
74
|
if one_result_mode?
|
@@ -88,18 +84,7 @@ module PG
|
|
88
84
|
@last_result = single_result
|
89
85
|
end
|
90
86
|
rescue Exception => e
|
91
|
-
|
92
|
-
cancel_timer
|
93
|
-
send_proc = @send_proc
|
94
|
-
@send_proc = nil
|
95
|
-
df = @deferrable
|
96
|
-
# prevent unbind error on auto re-connect
|
97
|
-
@deferrable = false
|
98
|
-
if e.is_a?(PG::Error)
|
99
|
-
@client.async_autoreconnect!(df, e, &send_proc)
|
100
|
-
else
|
101
|
-
df.fail(e)
|
102
|
-
end
|
87
|
+
handle_error(e)
|
103
88
|
else
|
104
89
|
if result == false
|
105
90
|
@notify_timestamp = Time.now if @timer
|
@@ -111,14 +96,31 @@ module PG
|
|
111
96
|
end
|
112
97
|
end
|
113
98
|
|
114
|
-
alias_method :notify_readable, :fetch_results
|
115
|
-
|
116
99
|
def unbind
|
117
100
|
@is_connected = false
|
118
101
|
@deferrable.protect do
|
102
|
+
cancel_timer
|
119
103
|
@client.raise_error ConnectionBad, "connection reset"
|
120
104
|
end if @deferrable
|
121
105
|
end
|
106
|
+
|
107
|
+
private
|
108
|
+
|
109
|
+
def handle_error(e)
|
110
|
+
self.notify_readable = false
|
111
|
+
cancel_timer
|
112
|
+
send_proc = @send_proc
|
113
|
+
@send_proc = nil
|
114
|
+
df = @deferrable
|
115
|
+
# prevent unbind error on auto re-connect
|
116
|
+
@deferrable = false
|
117
|
+
if e.is_a?(PG::Error)
|
118
|
+
@client.async_autoreconnect!(df, e, &send_proc)
|
119
|
+
else
|
120
|
+
df.fail(e)
|
121
|
+
end
|
122
|
+
end
|
123
|
+
|
122
124
|
end
|
123
125
|
|
124
126
|
end
|
@@ -332,30 +332,40 @@ describe 'pg-em with autoreconnect disabled' do
|
|
332
332
|
end
|
333
333
|
|
334
334
|
it "should fail to get last result asynchronously after server restart" do
|
335
|
+
check_get_last_result = proc do
|
336
|
+
@client.get_last_result_defer do |result|
|
337
|
+
result.should be_nil
|
338
|
+
@client.reset_defer do |conn|
|
339
|
+
conn.should be @client
|
340
|
+
@client.status.should be PG::CONNECTION_OK
|
341
|
+
EM.stop
|
342
|
+
end.should be_a_kind_of ::EM::DefaultDeferrable
|
343
|
+
end
|
344
|
+
end
|
335
345
|
system($pgserver_cmd_stop).should be_true
|
336
346
|
system($pgserver_cmd_start).should be_true
|
337
347
|
begin
|
338
348
|
@client.send_query('SELECT pg_sleep(5); SELECT pg_database_size(current_database());')
|
339
349
|
rescue PG::UnableToSend
|
340
|
-
end
|
341
|
-
@client.get_last_result_defer do |ex|
|
342
|
-
ex.should be_an_instance_of PG::ConnectionBad
|
343
350
|
@client.status.should be PG::CONNECTION_BAD
|
351
|
+
@client.get_last_result_defer do |ex|
|
352
|
+
ex.should be_nil
|
353
|
+
@client.status.should be PG::CONNECTION_BAD
|
354
|
+
check_get_last_result.call
|
355
|
+
end.should be_a_kind_of EM::DefaultDeferrable
|
356
|
+
else
|
344
357
|
@client.get_last_result_defer do |ex|
|
345
358
|
ex.should be_an_instance_of PG::ConnectionBad
|
346
|
-
@client.
|
347
|
-
|
348
|
-
|
349
|
-
|
350
|
-
end.should be_a_kind_of ::EM::DefaultDeferrable
|
351
|
-
end
|
352
|
-
end.should be_a_kind_of EM::DefaultDeferrable
|
359
|
+
@client.status.should be PG::CONNECTION_BAD
|
360
|
+
check_get_last_result.call
|
361
|
+
end.should be_a_kind_of EM::DefaultDeferrable
|
362
|
+
end
|
353
363
|
end
|
354
364
|
|
355
365
|
it "should fail to get each result asynchronously after server restart" do
|
356
|
-
check_get_result = proc do
|
357
|
-
@client.get_result_defer do |
|
358
|
-
|
366
|
+
check_get_result = proc do |expected_class|
|
367
|
+
@client.get_result_defer do |result|
|
368
|
+
result.should be_an_instance_of expected_class
|
359
369
|
@client.status.should be PG::CONNECTION_BAD
|
360
370
|
@client.reset_defer do |conn|
|
361
371
|
conn.should be @client
|
@@ -370,9 +380,9 @@ describe 'pg-em with autoreconnect disabled' do
|
|
370
380
|
@client.send_query('SELECT pg_sleep(5); SELECT pg_database_size(current_database());')
|
371
381
|
rescue PG::UnableToSend
|
372
382
|
@client.get_result_defer do |result|
|
373
|
-
result.should
|
383
|
+
result.should be_nil
|
374
384
|
@client.status.should be PG::CONNECTION_BAD
|
375
|
-
check_get_result.call
|
385
|
+
check_get_result.call NilClass
|
376
386
|
end
|
377
387
|
else
|
378
388
|
@client.get_result_defer do |result|
|
@@ -381,7 +391,7 @@ describe 'pg-em with autoreconnect disabled' do
|
|
381
391
|
result.check
|
382
392
|
end.to raise_error PG::Error
|
383
393
|
@client.status.should be PG::CONNECTION_OK
|
384
|
-
check_get_result.call
|
394
|
+
check_get_result.call PG::ConnectionBad
|
385
395
|
end
|
386
396
|
end
|
387
397
|
end
|
data/spec/em_client_common.rb
CHANGED
@@ -359,4 +359,44 @@ shared_context 'em-pg common after' do
|
|
359
359
|
end.should be_a_kind_of ::EM::DefaultDeferrable
|
360
360
|
end
|
361
361
|
|
362
|
+
it "should raise connection reset on connection breakdown" do
|
363
|
+
client = described_class.new query_timeout: 0.1
|
364
|
+
client.send_query('SELECT pg_sleep(10)')
|
365
|
+
client.get_last_result_defer do |ex|
|
366
|
+
ex.should be_an_instance_of PG::ConnectionBad
|
367
|
+
ex.message.should match /connection reset/
|
368
|
+
EM.add_timer(0.2) do
|
369
|
+
client.async_command_aborted.should be_false
|
370
|
+
EM.stop
|
371
|
+
end
|
372
|
+
end
|
373
|
+
# simulate connection breakdown
|
374
|
+
client.finish
|
375
|
+
end
|
376
|
+
|
377
|
+
it "should return nil result after async reset began" do
|
378
|
+
checkpoint = 0
|
379
|
+
@client.send_query('SELECT 1')
|
380
|
+
@client.reset_defer do |conn|
|
381
|
+
conn.should be @client
|
382
|
+
EM.next_tick do
|
383
|
+
checkpoint.should eq 2
|
384
|
+
@client.send_query('SELECT 1')
|
385
|
+
@client.get_last_result_defer do |result|
|
386
|
+
result.should be_an_instance_of PG::Result
|
387
|
+
result.getvalue(0,0).should eq '1'
|
388
|
+
EM.stop
|
389
|
+
end
|
390
|
+
end
|
391
|
+
end
|
392
|
+
@client.get_result_defer do |result|
|
393
|
+
result.should be_nil
|
394
|
+
checkpoint += 1
|
395
|
+
end
|
396
|
+
@client.get_last_result_defer do |result|
|
397
|
+
result.should be_nil
|
398
|
+
checkpoint += 1
|
399
|
+
end
|
400
|
+
end
|
401
|
+
|
362
402
|
end
|
data/spec/em_synchrony_client.rb
CHANGED
@@ -338,6 +338,11 @@ describe PG::EM::Client do
|
|
338
338
|
|
339
339
|
describe 'PG::EM::Client#transaction' do
|
340
340
|
|
341
|
+
before(:all) do
|
342
|
+
@client.query_timeout = 0
|
343
|
+
@client.query_timeout.should eq 0
|
344
|
+
end
|
345
|
+
|
341
346
|
it "should raise ArgumentError when there is no block" do
|
342
347
|
expect do
|
343
348
|
@client.transaction
|
@@ -213,14 +213,15 @@ describe 'em-synchrony-pg with autoreconnect disabled' do
|
|
213
213
|
begin
|
214
214
|
@client.send_query('SELECT pg_sleep(5); SELECT pg_database_size(current_database());')
|
215
215
|
rescue PG::UnableToSend
|
216
|
+
@client.status.should be PG::CONNECTION_BAD
|
217
|
+
@client.get_last_result.should be_nil
|
218
|
+
else
|
219
|
+
expect do
|
220
|
+
@client.get_last_result
|
221
|
+
end.to raise_error PG::ConnectionBad
|
216
222
|
end
|
217
|
-
expect do
|
218
|
-
@client.get_last_result
|
219
|
-
end.to raise_error PG::ConnectionBad
|
220
223
|
@client.status.should be PG::CONNECTION_BAD
|
221
|
-
|
222
|
-
@client.get_last_result
|
223
|
-
end.to raise_error PG::ConnectionBad
|
224
|
+
@client.get_last_result.should be_nil
|
224
225
|
@client.reset
|
225
226
|
@client.status.should be PG::CONNECTION_OK
|
226
227
|
@client.get_last_result.should be_nil
|
@@ -233,10 +234,8 @@ describe 'em-synchrony-pg with autoreconnect disabled' do
|
|
233
234
|
begin
|
234
235
|
@client.send_query('SELECT pg_sleep(5); SELECT pg_database_size(current_database());')
|
235
236
|
rescue PG::UnableToSend
|
236
|
-
expect do
|
237
|
-
@client.get_result
|
238
|
-
end.to raise_error PG::ConnectionBad
|
239
237
|
@client.status.should be PG::CONNECTION_BAD
|
238
|
+
@client.get_result.should be_nil
|
240
239
|
else
|
241
240
|
result = @client.get_result
|
242
241
|
result.should be_an_instance_of PG::Result
|
@@ -244,10 +243,12 @@ describe 'em-synchrony-pg with autoreconnect disabled' do
|
|
244
243
|
result.check
|
245
244
|
end.to raise_error PG::Error
|
246
245
|
@client.status.should be PG::CONNECTION_OK
|
246
|
+
expect do
|
247
|
+
@client.get_result
|
248
|
+
end.to raise_error PG::ConnectionBad
|
247
249
|
end
|
248
|
-
|
249
|
-
|
250
|
-
end.to raise_error PG::ConnectionBad
|
250
|
+
@client.status.should be PG::CONNECTION_BAD
|
251
|
+
@client.get_result.should be_nil
|
251
252
|
@client.status.should be PG::CONNECTION_BAD
|
252
253
|
@client.reset
|
253
254
|
@client.status.should be PG::CONNECTION_OK
|
metadata
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: em-pg-client
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 0.3.
|
4
|
+
version: 0.3.2
|
5
5
|
prerelease:
|
6
6
|
platform: ruby
|
7
7
|
authors:
|
@@ -9,7 +9,7 @@ authors:
|
|
9
9
|
autorequire:
|
10
10
|
bindir: bin
|
11
11
|
cert_chain: []
|
12
|
-
date: 2014-01-
|
12
|
+
date: 2014-01-07 00:00:00.000000000 Z
|
13
13
|
dependencies:
|
14
14
|
- !ruby/object:Gem::Dependency
|
15
15
|
name: pg
|
@@ -112,6 +112,8 @@ email: rafal@yeondir.com
|
|
112
112
|
executables: []
|
113
113
|
extensions: []
|
114
114
|
extra_rdoc_files:
|
115
|
+
- benchmarks/em_pg.rb
|
116
|
+
- benchmarks/single_row_mode.rb
|
115
117
|
- README.md
|
116
118
|
- BENCHMARKS.md
|
117
119
|
- LICENSE
|
@@ -127,6 +129,7 @@ files:
|
|
127
129
|
- README.md
|
128
130
|
- Rakefile
|
129
131
|
- benchmarks/em_pg.rb
|
132
|
+
- benchmarks/single_row_mode.rb
|
130
133
|
- em-pg-client.gemspec
|
131
134
|
- examples/single_row_mode.rb
|
132
135
|
- lib/em-pg-client.rb
|