em-pg-client 0.2.1 → 0.3.0

data/.yardopts

@@ -0,0 +1 @@
+lib/pg/em/client/*.rb  lib/pg/em/*.rb lib/pg/*.rb - BENCHMARKS.md LICENSE HISTORY.md

data/{BENCHMARKS.rdoc → BENCHMARKS.md}

@@ -1,26 +1,30 @@
-== Benchmarks
+Benchmarks
+----------
 
-I've done some benchmark tests[link:benchmarks/em_pg.rb] to compare fully async and blocking em-pg drivers.
+I've done some benchmark {file:benchmarks/em_pg.rb tests} to compare fully async and blocking em-pg drivers.
 
 The goal of the test is simply to retrieve (~80000) rows from table with a lot of text data, in chunks, using parallel connections.
 The parallel method uses synchrony for simplicity.
 
-* +single+ is (eventmachine-less) job for retrieving a whole data table in
+* `single` is (eventmachine-less) job for retrieving a whole data table in
   one simple query "select * from resources"
-* +parallel+ chunk_row_count / concurrency] uses em-pg-client for retrieving
-  result in chunks by +chunk_row_count+ rows and using +concurrency+ parallel
+* `parallel` chunk_row_count / concurrency] uses em-pg-client for retrieving
+  result in chunks by `chunk_row_count` rows and using `concurrency` parallel
   connections
-* +blocking+ chunk_row_count / concurrency is similiar to +parallel+ except
+* `blocking` chunk_row_count / concurrency is similiar to `parallel` except
   that it uses special patched version of library that uses blocking
   PGConnection methods
 
-== Environment 
+Environment
+-----------
 
 The machine used for test is Linux CentOS 2.6.18-194.32.1.el5xen #1 SMP with Quad Core Xeon X3360 @ 2.83GHz, 4GB RAM.
 Postgres version used: 9.0.3.
 
-== The results:
+The results:
+------------
 
+```
     >> benchmark 1000
                               user     system      total        real
     single:              80.970000   0.350000  81.320000 (205.592592)
@@ -34,10 +38,11 @@ Postgres version used: 9.0.3.
     blocking 5000/5:     79.930000   1.810000  81.740000 (223.342432)
     blocking 2000/10:    76.990000   2.820000  79.810000 (225.347169)
     blocking 1000/20:    78.790000   3.230000  82.020000 (225.949107)
+```
 
 As we can see the gain from using asynchronous pg client while
-using +parallel+ queries is noticeable (up to ~30%).
+using `parallel` queries is noticeable (up to ~30%).
 
-The +blocking+ client however doesn't gain much from parallel execution.
+The `blocking` client however doesn't gain much from parallel execution.
 This was expected because it freezes eventmachine until the whole
 dataset is consumed by the client.

data/{HISTORY.rdoc → HISTORY.md}

@@ -1,20 +1,43 @@
+0.3.0
+
+- dedicated asynchronous connection pool
+- works on windows (with ruby 2.0+): uses PGConn#socket_io object instead of
+  #socket file descriptor
+- socket watch handler is not being detached between command calls
+- no more separate em and em-synchrony client
+- api changes: async_exec and async_query command are now fiber-synchronized
+- api changes: other async_* methods are removed or deprecated
+- api changes: asynchronous methods renamed to *_defer
+- transaction() helper method that can be called recursively
+- requirements updated: eventmachine >~ 1.0.0, pg >= 0.17.0, ruby >= 1.9.2
+- spec: more auto re-connect test cases
+- spec: more tests for connection establishing
+- comply with pg: do not close the client on connection failure
+- comply with pg: asynchronous connect_timeout fallbacks to environment variable
+- fix: auto re-connect raises an error if the failed connection had unfinished
+  transaction state
+- yardoc docs
+
 0.2.1
+
 - support for pg >= 0.14 native PG::Result#check
 - support for pg >= 0.14 native PG::Connection#set_default_encoding
 - fix: connection option Hash argument was modified by Client.new and Client.async_connect
 
 0.2.0
+
 - disabled async_autoreconnect by default unless on_autoreconnect is set
 - async_connect sets #internal_encoding to Encoding.default_internal
 - fix: finish connection on async connect_timeout
 - nice errors generated on missing dependencies
 - blocking #reset() should clear async_command_aborted flag
 - less calls to #is_busy in Watcher#notify_readable
-- #async_describe_portal + specs
-- #async_describe_prepared + specs
+- async_describe_portal() + specs
+- async_describe_prepared() + specs
 
 0.2.0.pre.3
-- #status() returns CONNECTION_BAD for connections with expired query
+
+- status() returns CONNECTION_BAD for connections with expired query
 - spec: query timeout expiration
 - non-blocking result processing for multiple data query statements sent at once
 - refine code in em-synchrony/pg
@@ -23,20 +46,23 @@
 - spec: autoreconnect
 
 0.2.0.pre.2
+
 - errors from consume_input fails deferrable
-- query_timeout now measures only network response timeout, 
+- query_timeout now measures only network response timeout,
   so it's not fired for large datasets
 
 0.2.0.pre.1
+
 - added query_timeout feature for async query commands
 - added connect_timeout property for async connect/reset
 - fix: async_autoreconnect for tcp/ip connections
 - fix: async_* does not raise errors; errors handled by deferrable
 - rework async_autoreconnect in fully async manner
-- added async_connect() and #async_reset()
+- added async_connect() and async_reset()
 - API change: on_reconnect -> on_autoreconnect
 
 0.1.1
+
 - added on_reconnect callback
 - docs updated
 - added development dependency for eventmachine >= 1.0.0.beta.1
@@ -44,4 +70,5 @@
 - added error checking to eventmachine specs
 
 0.1.0
+
 - first release

data/LICENSE

@@ -0,0 +1,21 @@
+Copyright (c) 2013 Rafal Michalski (rafal at yeondir dot com)
+
+MIT License
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

data/README.md

@@ -0,0 +1,392 @@
+em-pg-client
+============
+
+Author: Rafał Michalski  (rafal at yeondir dot com)
+
+* http://github.com/royaltm/ruby-em-pg-client
+
+Description
+-----------
+
+__em-pg-client__ is the Ruby EventMachine driver interface to the
+PostgreSQL RDBMS. It is based on [ruby-pg](https://bitbucket.org/ged/ruby-pg).
+
+__em-pg-client__ provides {PG::EM::Client} class which inherits
+[PG::Connection](http://deveiate.org/code/pg/PG/Connection.html).
+You can work with {PG::EM::Client} almost the same way you would work
+with PG::Connection.
+
+The real difference begins when you turn the EventMachine reactor on.
+
+```ruby
+  require 'pg/em'
+  
+  pg = PG::EM::Client.new dbname: 'test'
+
+  # no async
+  pg.query('select * from foo') do |result|
+    puts Array(result).inspect
+  end
+
+  # asynchronous
+  EM.run do
+    Fiber.new do
+      pg.query('select * from foo') do |result|
+        puts Array(result).inspect
+      end
+      EM.stop
+    end.resume
+  end
+
+  # asynchronous + deferrable
+  EM.run do
+    df = pg.query_defer('select * from foo')
+    df.callback { |result|
+      puts Array(result).inspect
+      EM.stop
+    }
+    df.errback {|ex|
+      raise ex
+    }
+    puts "sent"
+  end
+```
+
+Features
+--------
+
+* Non-blocking / fully asynchronous processing with EventMachine.
+* Event reactor auto-detecting, asynchronous fiber-synchronized command methods
+  (the same code can be used regardless of the EventMachine reactor state)
+* Asynchronous EM-style (deferrable returning) command methods.
+* Fully asynchronous automatic re-connects on connection failures
+  (e.g.: RDBMS restarts, network failures).
+* Minimal changes to [PG::Connection](http://deveiate.org/code/pg/PG/Connection.html) API.
+* Configurable timeouts (connect or execute) of asynchronous processing.
+* Dedicated connection pool with dynamic size, supporting asynchronous
+  processing and transactions.
+* [Sequel Adapter](https://github.com/fl00r/em-pg-sequel) by Peter Yanovich.
+* Works on windows (requires ruby 2.0) (issue #7).
+
+Requirements
+------------
+
+* ruby >= 1.9.2 (tested: 2.1.0, 2.0.0-p353, 1.9.3-p374, 1.9.2-p320)
+* https://bitbucket.org/ged/ruby-pg >= 0.17.0
+* [PostgreSQL](http://www.postgresql.org/ftp/source/) RDBMS >= 8.3
+* http://rubyeventmachine.com >= 1.0.0
+*  [EM-Synchrony](https://github.com/igrigorik/em-synchrony)
+  (optional - not needed for any of the client functionality,
+  just wrap your code in a fiber)
+
+Install
+-------
+
+```
+  $ [sudo] gem install em-pg-client
+```
+
+#### Gemfile
+
+```ruby
+  gem "em-pg-client", "~> 0.3.0"
+```
+
+#### Github
+
+```
+  git clone git://github.com/royaltm/ruby-em-pg-client.git
+```
+
+Usage
+-----
+
+### PG::Connection commands adapted to the EventMachine
+
+#### Asynchronous, the EventMachine style:
+
+* `Client.connect_defer` (singleton method)
+* `reset_defer`
+* `exec_defer` (alias: `query_defer`)
+* `prepare_defer`
+* `exec_prepared_defer`
+* `describe_prepared_defer`
+* `describe_portal_defer`
+
+For arguments of these methods consult their original (without the `_defer`
+suffix) counterparts in the
+[PG::Connection](http://deveiate.org/code/pg/PG/Connection.html) manual.
+
+Use `callback` with a block on the returned deferrable object to receive the
+result. In case of `connect_defer` and `reset_defer` the result is an instance
+of the {PG::EM::Client}. The received client is in connected state and ready
+for the queries. Otherwise an instance of the
+[PG::Result](http://deveiate.org/code/pg/PG/Result.html) is received. You may
+`clear` the obtained result object or leave it to `gc`.
+
+To detect an error in the executed command call `errback` on the deferrable
+with a block. You should expect an instance of the raised `Exception`
+(usually PG::Error) as the block argument.
+
+#### Reactor sensing methods, EM-Synchrony style:
+
+* `Client.new` (singleton, alias: `connect`, `open`, `setdb`, `setdblogin`)
+* `reset`
+* `exec` (alias: `query`, `async_exec`, `async_query`)
+* `prepare`
+* `exec_prepared`
+* `describe_prepared`
+* `describe_portal`
+
+The above methods call `*_defer` counterparts of themselves and `yield`
+from the current fiber awaiting for the result. The PG::Result instance
+(or PG::EM::Client for `new`) is then returned to the caller.
+If a code block is given, it will be passed the result as an argument.
+In that case the value of the block is returned instead and the result is
+being cleared (or in case of `new` - client is being closed) after block
+terminates.
+
+These methods check if EventMachine's reactor is running and the current fiber
+is not a root fiber. Otherwise the parent (thread-blocking) PG::Connection
+methods are being called.
+
+You can call asynchronous, fiber aware and blocking methods without finishing
+the connection. You only need to start/stop EventMachine in between the
+asynchronous calls.
+
+Although the [em-synchrony](https://github.com/igrigorik/em-synchrony/)
+provides very nice set of tools for the untangled EventMachine, you don't
+really require it to fully benefit from the PG::EM::Client. Just wrap your
+asynchronous code in a fiber:
+
+    Fiber.new { ... }.resume
+
+#### Special options
+
+There are four special connection options and one of them is a standard `pg`
+option used by the async methods. You may pass them as one of the __hash__
+options to {PG::EM::Client.new} or {PG::EM::Client.connect_defer} or simply
+use the accessor methods to change them on the fly.
+
+The options are:
+
+* `connect_timeout`
+* `query_timeout`
+* `async_autoreconnect`
+* `on_autoreconnect`
+
+Only `connect_timeout` is a standard `libpq` option, although changing it with
+the accessor method affects asynchronous functions only.
+See {PG::EM::Client} for more details.
+
+#### Handling errors
+
+Exactly like in `pg`:
+
+```ruby
+  EM.synchrony do
+    begin
+      pg.query('smellect 1')
+    rescue => e
+      puts "error: #{e.inspect}"
+    end
+    EM.stop
+  end
+```
+
+with *_defer methods:
+
+```ruby
+  EM.run do
+    pg.query_defer('smellect 1') do |ret|
+      if ret.is_a?(Exception)
+        puts "PSQL error: #{ret.inspect}"
+      end
+    end
+  end
+```
+
+or
+
+```ruby
+  EM.run do
+    pg.query_defer('smellect 1').callback do |ret|
+      puts "do something with #{ret}"
+    end.errback do |err|
+      puts "PSQL error: #{err.inspect}"
+    end
+  end
+```
+
+### Auto re-connecting in asynchronous mode
+
+Connection reset is done in a non-blocking manner using `reset_defer` internally.
+
+```ruby
+  EM.run do
+    Fiber.new do
+      pg = PG::EM::Client.new async_autoreconnect: true
+
+      try_query = lambda do
+        pg.query('select * from foo') do |result|
+          puts Array(result).inspect
+        end
+      end
+
+      try_query.call
+      system 'pg_ctl stop -m fast'
+      system 'pg_ctl start -w'
+      try_query.call
+
+      EM.stop
+    end.resume
+  end
+```
+
+to enable this feature call:
+
+```ruby
+  pg.async_autoreconnect = true
+```
+
+Additionally the `on_autoreconnect` callback may be set on the connection.
+It's being invoked after successfull connection restart, just before the
+pending command is sent again to the server.
+
+### Connection Pool
+
+Forever alone? Not anymore! There is a dedicated {PG::EM::ConnectionPool}
+class with dynamic pool for both types of asynchronous commands (deferral
+and fiber-synchronized).
+
+It also provides a #transaction method which locks the in-transaction
+connection to the calling fiber and allows to execute commands
+on the same connection within a transaction block. The transactions may
+be nested. See also docs for the {PG::EM::Client#transaction} method.
+
+#### Parallel async queries
+
+```ruby
+  require 'pg/em/connection_pool'
+  require 'em-synchrony'
+
+  EM.synchrony do
+    pg = PG::EM::ConnectionPool.new(size: 2, dbname: 'test')
+
+    multi = EM::Synchrony::Multi.new
+    multi.add :foo, pg.query_defer('select pg_sleep(1)')
+    multi.add :bar, pg.query_defer('select pg_sleep(1)')
+
+    start = Time.now
+    res = multi.perform
+    # around 1 sec.
+    puts Time.now - start
+
+    EM.stop
+  end
+```
+
+#### Fiber Concurrency
+
+```ruby
+  require 'pg/em/connection_pool'
+  require 'em-synchrony'
+  require "em-synchrony/fiber_iterator"
+
+  EM.synchrony do
+    concurrency = 5
+    queries = (1..10).map {|i| "select pg_sleep(1); select #{i}" }
+
+    pg = PG::EM::ConnectionPool.new(size: concurrency, dbname: 'test')
+
+    start = Time.now
+    EM::Synchrony::FiberIterator.new(queries, concurrency).each do |query|
+      pg.query(query) do |result|
+        puts "recv: #{result.getvalue(0,0)}"
+      end
+    end
+    # around 2 secs.
+    puts Time.now - start
+
+    EM.stop
+  end
+```
+
+API Changes
+-----------
+
+### 0.2.x -> 0.3.x
+
+There is a substantial difference in the API between this and the previous
+releases. The idea behind it was to make this implementation as much
+compatible as possible with the threaded `pg` interface.
+E.g. the `#async_exec` is now an alias to `#exec`.
+
+The other reason was to get rid of the ugly em / em-synchrony duality.
+
+* There is no separate em-synchrony client version anymore.
+* The methods returning Deferrable have now the `*_defer` suffix.
+* The `#async_exec` and `#async_query` (in <= 0.2 they were deferrable methods)
+  are now aliases to `#exec`.
+* The command methods `#exec`, `#query`, `#exec_*`, `#describe_*` are now
+  em-synchrony style methods (fiber-synchronized).
+* The following methods were removed:
+
+    - `#async_prepare`,
+    - `#async_exec_prepared`,
+    - `#async_describe_prepared`,
+    - `#async_describe_portal`
+
+  as their names were confusing due to the unfortunate `#async_exec`.
+
+* The `async_connect` and `#async_reset` are renamed to `connect_defer` and `#reset_defer`
+  respectively.
+
+### 0.1.x -> 0.2.x
+
+* `on_reconnect` renamed to more accurate `on_autoreconnect`
+  (well, it's not used by PG::EM::Client#reset call).
+* `async_autoreconnect` is `false` by default if `on_autoreconnect`
+  is __not__ specified as initialization option.
+
+Bugs/Limitations
+----------------
+
+* no async support for: COPY commands (`get_copy_data`,  `put_copy_data`),
+  `wait_for_notify`
+* actually no ActiveRecord support (you are welcome to contribute).
+
+TODO:
+-----
+
+* implement streaming results (Postgres >= 9.2)
+* implement EM adapted version of `get_copy_data`, `put_copy_data`,
+  `wait_for_notify` and `transaction`
+* ORM (ActiveRecord and maybe Datamapper) support as separate projects
+* present more benchmarks
+
+More Info
+---------
+
+This implementation makes use of non-blocking:
+[PGConn#is_busy](http://deveiate.org/code/pg/PG/Connection.html#method-i-is_busy) and
+[PGConn#consume_input](http://deveiate.org/code/pg/PG/Connection.html#method-i-consume_input) methods.
+Depending on the size of queried results and the concurrency level, the gain
+in overall speed and responsiveness of your application might be actually quite huge.
+See {file:BENCHMARKS.md BENCHMARKING}.
+
+Thanks
+------
+
+The greetz go to:
+
+* [Authors](https://bitbucket.org/ged/ruby-pg/wiki/Home#!copying) of __pg__
+  driver (especially for its async-api)
+* Francis Cianfrocca for great reactor framework
+  [EventMachine](https://github.com/eventmachine/eventmachine)
+* Ilya Grigorik [igrigorik](https://github.com/igrigorik) for
+  [untangling EM with Fibers](http://www.igvita.com/2010/03/22/untangling-evented-code-with-ruby-fibers/)
+* Peter Yanovich [fl00r](https://github.com/fl00r) for the
+  [em-pg-sequel](https://github.com/fl00r/em-pg-sequel)
+* Andrew Rudenko [prepor](https://github.com/prepor) for the implicit idea
+  of the re-usable watcher from his [em-pg](https://github.com/prepor/em-pg).