em-pg-client 0.1.1 → 0.2.0.pre.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
data/HISTORY.rdoc CHANGED
@@ -1,9 +1,16 @@
1
- 0.1.1
2
- - added on_reconnect callback
3
- - docs updated
4
- - added development dependency for eventmachine >= 1.0.0.beta.1
5
- - added separate client specs for eventmachine = 0.12.10
6
- - added error checking to eventmachine specs
7
-
8
- 0.1.0
9
- - first release
1
+ next release
2
+ - added connect_timeout property with for async connect/reset
3
+ - fix: async_autoreconnect for tcp/ip connections
4
+ - fix: async_* does not raise errors; errors handled by deferrable
5
+ - rework async_autoreconnect in fully async manner
6
+ - added async_connect() and #async_reset()
7
+ - API change: on_reconnect -> on_autoreconnect
8
+ 0.1.1
9
+ - added on_reconnect callback
10
+ - docs updated
11
+ - added development dependency for eventmachine >= 1.0.0.beta.1
12
+ - added separate client specs for eventmachine = 0.12.10
13
+ - added error checking to eventmachine specs
14
+
15
+ 0.1.0
16
+ - first release
data/README.rdoc CHANGED
@@ -1,258 +1,294 @@
1
- = em-pg-client
2
-
3
- Author:: Rafał Michalski (mailto:rafal@yeondir.com)
4
-
5
- * http://github.com/royaltm/ruby-em-pg-client
6
-
7
- == DESCRIPTION
8
-
9
- *em-pg-client* is a PostgreSQL EventMachine client wrapper for Ruby
10
- based on ruby-pg[https://bitbucket.org/ged/ruby-pg]
11
-
12
- == FEATURES
13
-
14
- * minimal changes to PG::Connection API
15
- * auto reconnects on socket connection loss (like server restarts)
16
- * true non-blocking asynchronous processing
17
- * EM-Synchrony[https://github.com/igrigorik/em-synchrony] support
18
-
19
- == BUGS/LIMITATIONS
20
-
21
- * actually no ActiveRecord or Sequel support (you are welcome to contribute).
22
- * connecting/reconnecting operation is blocking EM
23
-
24
- == TODO:
25
-
26
- * em-synchrony ORM (ActiveRecord, Sequel and maybe Datamapper) support
27
- * full async connection process (low priority)
28
-
29
- == REQUIREMENTS
30
-
31
- * ruby >= 1.9
32
- * https://bitbucket.org/ged/ruby-pg (>= 0.13)
33
- * http://rubyeventmachine.com
34
- * (optional) EM-Synchrony[https://github.com/igrigorik/em-synchrony]
35
-
36
- == INSTALL
37
-
38
- $ sudo gem install em-pg-client
39
-
40
- == WHY?
41
-
42
- Because until now nobody did it to fit my needs.
43
- I've found at least 3 other implementations of EM postgres client:
44
-
45
- * https://github.com/jzimmek/em-postgresql-sequel
46
- * https://github.com/leftbee/em-postgresql-adapter
47
- * https://github.com/jtoy/em-postgres
48
-
49
- and (except the bundled one which uses no longer maintained postgres-pr library)
50
- all of them have similiar flaws:
51
-
52
- * 2 of them are designed to support some ORM (ActiveRecord or Sequel)
53
- so they are EM-Synchrony only,
54
- * non-standard API method names,
55
- * no (nonexistent or non-working) autoreconnect implementation,
56
- * not fully supporting asynchronous PG::Connection API.
57
-
58
- The last one is worth some comment:
59
-
60
- They all use blocking methods to retrieve whole result from server
61
- (PGConn#block() or PGConn#get_result() which also
62
- blocks when there is not enough buffered data on socket).
63
-
64
- This implementation makes use of non-blocking: PGConn#is_busy and PGConn#consume_input methods.
65
- Depending on size of result sets and concurrency level the gain in overall speed and responsiveness of your
66
- application might be actually quite huge. I already have done some tests[link:BENCHMARKS.rdoc].
67
-
68
- == Thanks
69
-
70
- The greetz go to:
71
- - Authors[https://bitbucket.org/ged/ruby-pg/wiki/Home#!copying] of +pg+ driver (especially for its async-api)
72
- - Francis Cianfrocca for great reactor framework (EventMachine[https://github.com/eventmachine/eventmachine])
73
- - Ilya Grigorik (igrigorik[https://github.com/igrigorik]) for (untangling)[http://www.igvita.com/2010/03/22/untangling-evented-code-with-ruby-fibers/] EM with Fibers
74
-
75
- == USAGE
76
-
77
- === BASIC
78
-
79
- require 'pg/em'
80
-
81
- # no async
82
- pg = PG::EM::Client.new dbname: 'test'
83
- pq.query('select * from foo') do |result|
84
- puts Array(result).inspect
85
- end
86
-
87
- # asynchronous
88
- EM.run do
89
- pq.query('select * from foo') do |result|
90
- raise result if result.is_a? ::Exception
91
- puts Array(result).inspect
92
- EM.stop
93
- end
94
- puts "sent"
95
- end
96
-
97
- === AUTORECONNECTING IN ASYNC MODE
98
-
99
- EM.run do
100
- pg = PG::EM::Client.new dbname: 'test'
101
- try_query = lambda do |&blk|
102
- pg.query('select * from foo') do |result|
103
- raise result if result.is_a? ::Exception
104
- puts Array(result).inspect
105
- blk.call
106
- end
107
- end
108
- try_query.call {
109
- system 'pg_ctl stop -m fast'
110
- system 'pg_ctl start -w'
111
- EM.add_timer(1) do
112
- try_query.call { EM.stop }
113
- end
114
- }
115
- end
116
-
117
- to disable this feature call:
118
-
119
- pg.async_autoreconnect = false
120
-
121
- or
122
-
123
- pg = PG::EM::Client.new dbname: 'test',
124
- async_autoreconnect: false
125
-
126
- It's also possible to define +on_reconnect+ callback to be invoked
127
- while the connection has been reset. It's called just before the send query
128
- command is executed:
129
-
130
- EM.run do
131
- pg = PG::EM::Client.new dbname: 'test'
132
- pg.prepare('bar', 'select * from foo order by cdate desc') do
133
- pg.on_reconnect = proc { |c, e|
134
- c.prepare('bar', 'select * from foo order by cdate desc')
135
- }
136
- try_query = lambda do |&blk|
137
- pg.exec_prepared('bar') do |result|
138
- raise result if result.is_a? ::Exception
139
- puts Array(result).inspect
140
- blk.call
141
- end
142
- end
143
- try_query.call {
144
- system 'pg_ctl stop -m fast'
145
- system 'pg_ctl start -w'
146
- EM.add_timer(1) do
147
- try_query.call { EM.stop }
148
- end
149
- }
150
- end
151
- end
152
-
153
- As you can see it's possible to send async query from inside on_reconnect
154
- proc. However you have to pass +Deferrable+ from the async callback to the
155
- caller. See +#on_reconnect+ docs for details.
156
-
157
- === TRUE ASYNC
158
-
159
- EM.run do
160
- pool = (1..10).map { PG::EM::Client.new dbname: 'alpha' }
161
-
162
- togo = pool.length
163
-
164
- pool.each_with_index do |pg, i|
165
- pg.query("select * from foo") do |result|
166
- puts "recv: #{i}"
167
- EM.stop if (togo-=1).zero?
168
- end
169
- puts "sent: #{i}"
170
- end
171
- end
172
-
173
- === EM-Synchrony
174
-
175
- require 'em-synchrony/pg'
176
-
177
- EM.synchrony do
178
- pg = PG::EM::Client.new dbname: 'test'
179
- pg.query('select * from foo') do |result|
180
- puts Array(result).inspect
181
- end
182
- EM.stop
183
- end
184
-
185
- ==== Handling errors
186
-
187
- EM.synchrony do
188
- begin
189
- pg.query('select * from foo') do |result|
190
- puts result
191
- end
192
- rescue PG::Error => e
193
- puts "PSQL error: #{e.inspect}"
194
- end
195
- EM.stop
196
- end
197
-
198
- ==== Parallel async queries
199
-
200
- EM.synchrony do
201
- pg = EM::Synchrony::ConnectionPool.new(size: 2) do
202
- PG::EM::Client.new :dbname => 'alpha'
203
- end
204
- multi = EventMachine::Synchrony::Multi.new
205
- multi.add :foo, pg.aquery('select * from foo') # or #async_query()
206
- multi.add :bar, pg.aquery('select * from bar') # #aquery() is just an alias
207
- res = multi.perform
208
- p res
209
- EM.stop
210
- end
211
-
212
- ==== Fiber Concurrency
213
-
214
- EM.synchrony do
215
- # use ConnectionPool when more Fibers will be querying at the same time!
216
- pg = EM::Synchrony::ConnectionPool.new(size: 5) do
217
- PG::EM::Client.new :dbname => 'alpha'
218
- end
219
- counter = 0
220
- EM::Synchrony::FiberIterator.new(['select * from foo']*10, 5) do |query|
221
- i = counter
222
- pg.query(query) do |result|
223
- puts "recv: #{i}"
224
- end
225
- puts "sent: #{i}"
226
- counter += 1
227
- end
228
- EM.stop
229
- end
230
-
231
- ==== Async reconnect with on_reconnect callback
232
-
233
- EM.synchrony do
234
- pg = EM::Synchrony::ConnectionPool.new(size: 5) do
235
- p = PG::EM::Client.new dbname: 'test'
236
- p.on_reconnect = proc do |c, e|
237
- c.prepare('bar', 'select * from foo order by cdate desc')
238
- end
239
- p.on_reconnect.call p
240
- p
241
- end
242
- try_query = lambda do
243
- pg.exec_prepared('bar') do |result|
244
- raise result if result.is_a? ::Exception
245
- puts Array(result).inspect
246
- end
247
- end
248
- try_query.call
249
- system 'pg_ctl stop -m fast'
250
- system 'pg_ctl start -w'
251
- EM::Synchrony.sleep(1)
252
- try_query.call
253
- EM.stop
254
- end
255
-
256
- == LICENCE
257
-
258
- The MIT License - Copyright (c) 2012 Rafał Michalski
1
+ = em-pg-client
2
+
3
+ Author:: Rafał Michalski (mailto:rafal@yeondir.com)
4
+
5
+ * http://github.com/royaltm/ruby-em-pg-client
6
+
7
+ == DESCRIPTION
8
+
9
+ *em-pg-client* is a PostgreSQL EventMachine client wrapper for Ruby
10
+ based on ruby-pg[https://bitbucket.org/ged/ruby-pg]
11
+
12
+ == FEATURES
13
+
14
+ * minimal changes to PG::Connection API
15
+ * fully async auto reconnects on socket connection loss (like server restarts)
16
+ * true non-blocking asynchronous processing
17
+ * EM-Synchrony[https://github.com/igrigorik/em-synchrony] support
18
+
19
+ == BUGS/LIMITATIONS
20
+
21
+ * actually no ActiveRecord nor Sequel support (you are welcome to contribute).
22
+
23
+ == API Changes
24
+
25
+ * +on_reconnect+ renamed to more accurate +on_autoreconnect+
26
+ (well, it's not used by +#reset+ call)
27
+
28
+ == TODO:
29
+
30
+ * em-synchrony ORM (ActiveRecord, Sequel and maybe Datamapper) support
31
+ as separate projects
32
+
33
+ == REQUIREMENTS
34
+
35
+ * ruby >= 1.9
36
+ * https://bitbucket.org/ged/ruby-pg (>= 0.13)
37
+ * http://rubyeventmachine.com
38
+ * (optional) EM-Synchrony[https://github.com/igrigorik/em-synchrony]
39
+
40
+ == INSTALL
41
+
42
+ === Legacy
43
+
44
+ $ [sudo] gem install em-pg-client
45
+
46
+ ==== Gemfile
47
+
48
+ # eventmachine
49
+ gem "em-pg-client", "~> 0.1.1", :require => 'pg/em'
50
+ # em-synchrony
51
+ gem "em-pg-client", "~> 0.1.1", :require => ['pg/em', 'em-synchrony/pg']
52
+
53
+ === Latest branch (fully-async)
54
+
55
+ $ [sudo] gem install em-pg-client --prerelease
56
+
57
+ ==== Gemfile
58
+
59
+ # eventmachine
60
+ gem "em-pg-client", "~> 0.2.0.pre", :require => 'pg/em'
61
+ # em-synchrony
62
+ gem "em-pg-client", "~> 0.2.0.pre", :require => ['pg/em', 'em-synchrony/pg']
63
+
64
+ ==== Github
65
+
66
+ git clone git://github.com/royaltm/ruby-em-pg-client.git
67
+ git checkout fully-async
68
+
69
+ == WHY?
70
+
71
+ Because until now nobody did it to fit my needs.
72
+ I've found at least 3 other implementations of EM postgres client:
73
+
74
+ * https://github.com/jzimmek/em-postgresql-sequel
75
+ * https://github.com/leftbee/em-postgresql-adapter
76
+ * https://github.com/jtoy/em-postgres
77
+
78
+ and (except the bundled one which uses no longer maintained postgres-pr library)
79
+ all of them have similiar flaws:
80
+
81
+ * 2 of them are designed to support some ORM (ActiveRecord or Sequel)
82
+ so they are EM-Synchrony only,
83
+ * non-standard API method names,
84
+ * no (nonexistent or non-working) autoreconnect implementation,
85
+ * not fully supporting asynchronous PG::Connection API.
86
+
87
+ The last one is worth some comment:
88
+
89
+ They all use blocking methods to retrieve whole result from server
90
+ (PGConn#block() or PGConn#get_result() which also
91
+ blocks when there is not enough buffered data on socket).
92
+
93
+ This implementation makes use of non-blocking: PGConn#is_busy and PGConn#consume_input methods.
94
+ Depending on size of result sets and concurrency level the gain in overall speed and responsiveness of your
95
+ application might be actually quite huge. I've done some tests[link:BENCHMARKS.rdoc] already.
96
+
97
+ == Thanks
98
+
99
+ The greetz go to:
100
+ - Authors[https://bitbucket.org/ged/ruby-pg/wiki/Home#!copying] of +pg+ driver (especially for its async-api)
101
+ - Francis Cianfrocca for great reactor framework (EventMachine[https://github.com/eventmachine/eventmachine])
102
+ - Ilya Grigorik (igrigorik[https://github.com/igrigorik]) for (untangling)[http://www.igvita.com/2010/03/22/untangling-evented-code-with-ruby-fibers/] EM with Fibers
103
+
104
+ == USAGE
105
+
106
+ === BASIC
107
+
108
+ require 'pg/em'
109
+
110
+ # no async
111
+ pg = PG::EM::Client.new dbname: 'test'
112
+ pq.query('select * from foo') do |result|
113
+ puts Array(result).inspect
114
+ end
115
+
116
+ # asynchronous
117
+ EM.run do
118
+ pq.query('select * from foo') do |result|
119
+ raise result if result.is_a? ::Exception
120
+ puts Array(result).inspect
121
+ EM.stop
122
+ end
123
+ puts "sent"
124
+ end
125
+
126
+ === AUTORECONNECTING IN ASYNC MODE
127
+ Autoreconnecting is done in non-blocking manner using #async_reset internally.
128
+
129
+ EM.run do
130
+ pg = PG::EM::Client.new dbname: 'test',
131
+ connect_timeout: 5, query_timeout: 50
132
+ try_query = lambda do |&blk|
133
+ pg.query('select * from foo') do |result|
134
+ raise result if result.is_a? ::Exception
135
+ puts Array(result).inspect
136
+ blk.call
137
+ end
138
+ end
139
+ try_query.call {
140
+ system 'pg_ctl stop -m fast'
141
+ system 'pg_ctl start -w'
142
+ try_query.call { EM.stop }
143
+ }
144
+ end
145
+
146
+ to disable this feature call:
147
+
148
+ pg.async_autoreconnect = false
149
+
150
+ or
151
+
152
+ pg = PG::EM::Client.new dbname: 'test',
153
+ async_autoreconnect: false
154
+
155
+ It's also possible to define +on_autoreconnect+ callback to be invoked
156
+ while the connection has been reset. It's called just before the send query
157
+ command is executed:
158
+
159
+ EM.run do
160
+ pg = PG::EM::Client.new dbname: 'test'
161
+ pg.prepare('bar', 'select * from foo order by cdate desc') do
162
+ pg.on_autoreconnect = proc { |c, e|
163
+ c.prepare('bar', 'select * from foo order by cdate desc')
164
+ }
165
+ try_query = lambda do |&blk|
166
+ pg.exec_prepared('bar') do |result|
167
+ raise result if result.is_a? ::Exception
168
+ puts Array(result).inspect
169
+ blk.call
170
+ end
171
+ end
172
+ try_query.call {
173
+ system 'pg_ctl stop -m fast'
174
+ system 'pg_ctl start -w'
175
+ try_query.call { EM.stop }
176
+ }
177
+ end
178
+ end
179
+
180
+ As you can see it's possible to send async query from inside on_autoreconnect
181
+ proc. However you have to pass +Deferrable+ from the async callback to the
182
+ caller. See +#on_autoreconnect+ docs for details.
183
+
184
+ === TRUE ASYNC
185
+ For non-blocking connect use PG::EM::Client.async_connect and #async_reset for
186
+ asynchronous connection reset. Like other async methods they return deferrable object.
187
+ Use #callback to obtain already connected Client.
188
+
189
+ EM.run do
190
+ pool = (1..10).map {
191
+ PG::EM::Client.async_connect dbname: 'test',
192
+ connect_timeout: 5, query_timeout: 50 }
193
+
194
+ togo = pool.length
195
+
196
+ pool.each_with_index do |df, i|
197
+ df.callback do |pg|
198
+ pg.query("select * from foo") do |result|
199
+ puts "recv: #{i}"
200
+ EM.stop if (togo-=1).zero?
201
+ end
202
+ puts "sent: #{i}"
203
+ end
204
+ df.errback { |ex| raise ex }
205
+ end
206
+ end
207
+
208
+ === EM-Synchrony
209
+ Under +em-synchrony+ PG::EM::Client.new is fully asynchronous and blocks only current fiber.
210
+ This also applies to #reset.
211
+
212
+ require 'em-synchrony/pg'
213
+
214
+ EM.synchrony do
215
+ pg = PG::EM::Client.new dbname: 'test'
216
+ pg.query('select * from foo') do |result|
217
+ puts Array(result).inspect
218
+ end
219
+ EM.stop
220
+ end
221
+
222
+ ==== Handling errors
223
+
224
+ EM.synchrony do
225
+ begin
226
+ pg.query('select * from foo') do |result|
227
+ puts result
228
+ end
229
+ rescue PG::Error => e
230
+ puts "PSQL error: #{e.inspect}"
231
+ end
232
+ EM.stop
233
+ end
234
+
235
+ ==== Parallel async queries
236
+
237
+ EM.synchrony do
238
+ pg = EM::Synchrony::ConnectionPool.new(size: 2) do
239
+ PG::EM::Client.new :dbname => 'test'
240
+ end
241
+ multi = EventMachine::Synchrony::Multi.new
242
+ multi.add :foo, pg.aquery('select * from foo') # or #async_query()
243
+ multi.add :bar, pg.aquery('select * from bar') # #aquery() is just an alias
244
+ res = multi.perform
245
+ p res
246
+ EM.stop
247
+ end
248
+
249
+ ==== Fiber Concurrency
250
+
251
+ EM.synchrony do
252
+ # use ConnectionPool when more Fibers will be querying at the same time!
253
+ pg = EM::Synchrony::ConnectionPool.new(size: 5) do
254
+ PG::EM::Client.new :dbname => 'test'
255
+ end
256
+ counter = 0
257
+ EM::Synchrony::FiberIterator.new(['select * from foo']*10, 5) do |query|
258
+ i = counter
259
+ pg.query(query) do |result|
260
+ puts "recv: #{i}"
261
+ end
262
+ puts "sent: #{i}"
263
+ counter += 1
264
+ end
265
+ EM.stop
266
+ end
267
+
268
+ ==== Async reconnect with on_autoreconnect callback
269
+
270
+ EM.synchrony do
271
+ on_autoreconnect = proc do |c, e|
272
+ c.prepare('bar', 'select * from foo order by cdate desc')
273
+ end
274
+ pg = EM::Synchrony::ConnectionPool.new(size: 5) do
275
+ p = PG::EM::Client.new dbname: 'test', on_autoreconnect: on_autoreconnect
276
+ on_autoreconnect.call p
277
+ p
278
+ end
279
+ try_query = lambda do
280
+ pg.exec_prepared('bar') do |result|
281
+ raise result if result.is_a? ::Exception
282
+ puts Array(result).inspect
283
+ end
284
+ end
285
+ try_query.call
286
+ system 'pg_ctl stop -m fast'
287
+ system 'pg_ctl start -w'
288
+ try_query.call
289
+ EM.stop
290
+ end
291
+
292
+ == LICENCE
293
+
294
+ The MIT License - Copyright (c) 2012 Rafał Michalski