em-pg-client 0.2.0.pre.3 → 0.2.0

data/BENCHMARKS.rdoc

@@ -6,7 +6,7 @@ The goal of the test is simply to retrieve (~80000) rows from table with a lot o
 The parallel method uses synchrony for simplicity.
 
 * +single+ is (eventmachine-less) job for retrieving a whole data table in
-  one query simple query "select * from resources"
+  one simple query "select * from resources"
 * +parallel+ chunk_row_count / concurrency] uses em-pg-client for retrieving
   result in chunks by +chunk_row_count+ rows and using +concurrency+ parallel
   connections

data/HISTORY.rdoc

@@ -1,3 +1,13 @@
+0.2.0
+- disabled async_autoreconnect by default unless on_autoreconnect is set
+- async_connect sets #internal_encoding to Encoding.default_internal
+- fix: finish connection on async connect_timeout
+- nice errors generated on missing dependencies
+- blocking #reset() should clear async_command_aborted flag
+- less calls to #is_busy in Watcher#notify_readable
+- #async_describe_portal + specs
+- #async_describe_prepared + specs
+
 0.2.0.pre.3
 - #status() returns CONNECTION_BAD for connections with expired query
 - spec: query timeout expiration

data/README.rdoc

@@ -6,102 +6,111 @@ Author::    Rafał Michalski  (mailto:rafal@yeondir.com)
 
 == DESCRIPTION
 
-*em-pg-client* is a PostgreSQL EventMachine client wrapper for Ruby
-based on ruby-pg[https://bitbucket.org/ged/ruby-pg]
+*em-pg-client* is the Ruby and EventMachine driver interface to the PostgreSQL
+RDBMS. It is based on ruby-pg[https://bitbucket.org/ged/ruby-pg].
 
 == FEATURES
 
-* minimal changes to PG::Connection API
-* fully async auto reconnects on socket connection loss (like server restarts)
-* true non-blocking asynchronous processing
-* EM-Synchrony[https://github.com/igrigorik/em-synchrony] support
+* Non-blocking / asynchronous processing with EventMachine,
+* fully async auto re-connects on connection losses (e.g.: RDBMS restarts),
+* minimal changes to PG::Connection[http://deveiate.org/code/pg/PG/Connection.html] API,
+* configurable timeouts (connect or execute) of asynchronous processing,
+* additional Fiber-aware version supporting EM-Synchrony[https://github.com/igrigorik/em-synchrony].
 
 == BUGS/LIMITATIONS
 
+* no async support for: COPY commands (+get_copy_data+,  +put_copy_data+),
+  +wait_for_notify+ and +transaction+
 * actually no ActiveRecord nor Sequel support (you are welcome to contribute).
+* doesn't work on Windows (issue #7)
 
-== API Changes
+== API Changes between versions
 
+=== 0.1.x -> 0.2.x
 * +on_reconnect+ renamed to more accurate +on_autoreconnect+
-  (well, it's not used by +#reset+ call)
+  (well, it's not used by PG::EM::Client#reset call).
+* +async_autoreconnect+ is +false+ by default if +on_autoreconnect+
+  is *not* specified as initialization option.
 
 == TODO:
 
+* implement EM adapted version of +get_copy_data+, +put_copy_data+,
+  +wait_for_notify+ and +transaction+
+* add some fd/socket hackery to get it working on Windows (issue #7)
 * em-synchrony ORM (ActiveRecord, Sequel and maybe Datamapper) support
   as separate projects
+* present more benchmarks
 
 == REQUIREMENTS
 
-* ruby >= 1.9
-* https://bitbucket.org/ged/ruby-pg (>= 0.13)
-* http://rubyeventmachine.com
+* ruby >= 1.9 (tested: 1.9.3-p194, 1.9.2-p320, 1.9.2-p280, 1.9.1-p378)
+* https://bitbucket.org/ged/ruby-pg >= 0.13
+* PostgreSQL[http://www.postgresql.org/ftp/source/] RDBMS >= 8.3
+* http://rubyeventmachine.com >= 0.12.10
 * (optional) EM-Synchrony[https://github.com/igrigorik/em-synchrony]
 
 == INSTALL
 
-=== Legacy
-
   $ [sudo] gem install em-pg-client
-  
-==== Gemfile
-
-  # eventmachine
-  gem "em-pg-client", "~> 0.1.1", :require => 'pg/em'
-  # em-synchrony
-  gem "em-pg-client", "~> 0.1.1", :require => ['pg/em', 'em-synchrony/pg']
-
-=== Latest branch (fully-async)
-
-  $ [sudo] gem install em-pg-client --pre
 
 ==== Gemfile
 
   # eventmachine
-  gem "em-pg-client", "~> 0.2.0.pre", :require => 'pg/em'
+  gem "em-pg-client", "~> 0.2.0", :require => 'pg/em'
   # em-synchrony
-  gem "em-pg-client", "~> 0.2.0.pre", :require => ['pg/em', 'em-synchrony/pg']
+  gem "em-pg-client", "~> 0.2.0", :require => ['pg/em', 'em-synchrony/pg']
 
 ==== Github
 
   git clone git://github.com/royaltm/ruby-em-pg-client.git
-  git checkout fully-async
 
 == WHY?
 
-Because until now nobody did it to fit my needs.
+Because I didn't find any ruby-pg's EM implementation to fit my needs.
 I've found at least 3 other implementations of EM postgres client:
 
 * https://github.com/jzimmek/em-postgresql-sequel
 * https://github.com/leftbee/em-postgresql-adapter
 * https://github.com/jtoy/em-postgres
 
-and (except the bundled one which uses no longer maintained postgres-pr library)
+and (except the EM-bundled one which uses no longer maintained postgres-pr library)
 all of them have similiar flaws:
 
-* 2 of them are designed to support some ORM (ActiveRecord or Sequel)
+* 2 of them are designed to support some ORM (ActiveRecord or Sequel),
   so they are EM-Synchrony only,
 * non-standard API method names,
 * no (nonexistent or non-working) autoreconnect implementation,
+* poor error handling,
 * not fully supporting asynchronous PG::Connection API.
 
 The last one is worth some comment:
 
 They all use blocking methods to retrieve whole result from server
-(PGConn#block() or PGConn#get_result() which also
+(PGConn#block[http://deveiate.org/code/pg/PG/Connection.html#method-i-block] or
+PGConn#get_result[http://deveiate.org/code/pg/PG/Connection.html#method-i-get_result] which also
 blocks when there is not enough buffered data on socket).
 
-This implementation makes use of non-blocking: PGConn#is_busy and PGConn#consume_input methods.
-Depending on size of result sets and concurrency level the gain in overall speed and responsiveness of your
-application might be actually quite huge. I've done some tests[link:BENCHMARKS.rdoc] already.
+This implementation makes use of non-blocking:
+PGConn#is_busy[http://deveiate.org/code/pg/PG/Connection.html#method-i-is_busy] and
+PGConn#consume_input[http://deveiate.org/code/pg/PG/Connection.html#method-i-consume_input] methods.
+Depending on the size of queries results and the level of concurrency, the gain in overall speed and
+responsiveness of your application might be actually quite huge. I've done some
+tests[link:BENCHMARKS.rdoc] already.
 
 == Thanks
 
 The greetz go to:
 - Authors[https://bitbucket.org/ged/ruby-pg/wiki/Home#!copying] of +pg+ driver (especially for its async-api)
 - Francis Cianfrocca for great reactor framework (EventMachine[https://github.com/eventmachine/eventmachine])
-- Ilya Grigorik (igrigorik[https://github.com/igrigorik]) for (untangling)[http://www.igvita.com/2010/03/22/untangling-evented-code-with-ruby-fibers/] EM with Fibers
+- Ilya Grigorik (igrigorik[https://github.com/igrigorik]) for (untangling[http://www.igvita.com/2010/03/22/untangling-evented-code-with-ruby-fibers/]) EM with Fibers
 
 == USAGE
++em-pg-client+ provides PG::EM::Client class which inherits
+PG::Connection[http://deveiate.org/code/pg/PG/Connection.html].
+You can work with PG::EM::Client almost the same way you would with
+PG::Connection.
+
+The real difference begins when you turn EventMachine reactor on.
 
 === BASIC
 
@@ -109,13 +118,26 @@ The greetz go to:
   
   # no async
   pg = PG::EM::Client.new dbname: 'test'
-  pq.query('select * from foo') do |result|
+  pg.query('select * from foo') do |result|
     puts Array(result).inspect
   end
-  
+
   # asynchronous
   EM.run do
-    pq.query('select * from foo') do |result|
+    df = pg.query('select * from foo')
+    df.callback { |result|
+      puts Array(result).inspect
+      EM.stop
+    }
+    df.errback {|ex|
+      raise ex
+    }
+    puts "sent"
+  end
+
+  # alternatively
+  EM.run do
+    pg.query('select * from foo') do |result|
       raise result if result.is_a? ::Exception
       puts Array(result).inspect
       EM.stop
@@ -123,12 +145,79 @@ The greetz go to:
     puts "sent"
   end
 
+=== PG::Connection methods adapted to EventMachine
+The list of PG::EM::Client async methods for processing with EventMachine.
+
+==== 1. Async methods (always returning +Deferrable+ object):
+
+* +Client.async_connect+ (singleton)
+* +async_reset+
+* +async_exec+ (alias: +async_query+)
+* +async_prepare+
+* +async_exec_prepared+
+* +async_describe_prepared+
+* +async_describe_portal+
+
+For arguments of theese methods consult their original blocking (without +async_+ prefix)
+counterparts in PG::Connection[http://deveiate.org/code/pg/PG/Connection.html] manual.
+
+Use +callback+ on the returned +Deferrable+ to receive result. 
+The result you receive is PG::EM::Client for PG::EM::Client.async_connect
+and +async_reset+, and PG::Result[http://deveiate.org/code/pg/PG/Result.html] for the rest
+of the methods. The received PG::EM::Client is in a connected state and ready for queries.
+You need to +clear+ obtained PG::Result object yourself or leave it to +gc+.
+
+To detect a failure in an executed method use +errback+ on returned +Deferrable+.
+You should expect an instance of +Exception+ (usually PG::Error) as +errback+
+argument. You may check its +backtrace+ to find origin of the error.
+
+==== 2. Async / blocking methods (returning +Deferrable+ only when EM is running):
+
+* +exec+ (alias: +query+)
+* +prepare+
+* +exec_prepared+
+* +describe_prepared+
+* +describe_portal+
+
+Outside EventMachine's event loop these methods are regular, blocking PG::Connection
+methods.
+
+All the methods (1 & 2) accept block argument which they attach to +callback+ and +errback+
+hooks of returned +Deferrable+.
+
+You may also mix async and blocking methods without closing the connection.
+You only need to start/stop EventMachine in between async calls.
+
+==== Special options
+There are 3 additional connection options and 1 standard +pg+ option used by
+async methods. You may add them as one of the *hash* options to
+PG::EM::Client.new or PG::EM::Client.async_connect or simply use accessor
+methods to change them on the fly. The additional options are not passed to
++libpq+.
+
+The options are:
+
+- +async_autoreconnect+ (+true+ / +false+ with default +false+ unless
+  +on_autoreconnect+ is specified)
+  allows automatic re-connection when there was a problem with connection
+  to the server,
+- +on_autoreconnect+ (+nil+ / +Proc+ with default +nil+)
+  a hook which is called after auto-reconnecting,
+- +query_timeout+ (+Float+ / +Fixnum+ with default +0+)
+  allows to set timeout for query execution,
+- +connect_timeout+ (+Float+ / +Fixnum+ with default +0+)
+  connection establishing and resetting timeout.
+
+Only +connect_timeout+ is a standard +libpq+ option, although changing it by
+accessor method only affects asynchronous functions.
+
 === AUTORECONNECTING IN ASYNC MODE
-Autoreconnecting is done in non-blocking manner using #async_reset internally.
+Autoreconnecting is done in non-blocking manner using +async_reset+ internally.
 
   EM.run do
     pg = PG::EM::Client.new dbname: 'test',
-          connect_timeout: 5, query_timeout: 50
+          connect_timeout: 5, query_timeout: 50,
+          async_autoreconnect: true
     try_query = lambda do |&blk|
       pg.query('select * from foo') do |result|
         raise result if result.is_a? ::Exception
@@ -143,21 +232,22 @@ Autoreconnecting is done in non-blocking manner using #async_reset internally.
     }
   end
 
-to disable this feature call:
+to enable this feature call:
 
-  pg.async_autoreconnect = false
+  pg.async_autoreconnect = true
 
 or
 
   pg = PG::EM::Client.new dbname: 'test',
-    async_autoreconnect: false
+    async_autoreconnect: true
 
 It's also possible to define +on_autoreconnect+ callback to be invoked
 while the connection has been reset. It's called just before the send query
 command is executed:
 
   EM.run do
-    pg = PG::EM::Client.new dbname: 'test'
+    pg = PG::EM::Client.new dbname: 'test',
+          async_autoreconnect: true
     pg.prepare('bar', 'select * from foo order by cdate desc') do
       pg.on_autoreconnect = proc { |c, e|
         c.prepare('bar', 'select * from foo order by cdate desc')
@@ -177,14 +267,15 @@ command is executed:
     end
   end
 
-As you can see it's possible to send async query from inside on_autoreconnect
+As you can see it's possible to send async query from inside +on_autoreconnect+
 proc. However you have to pass +Deferrable+ from the async callback to the
-caller. See +#on_autoreconnect+ docs for details.
+caller. See PG::EM::Client#on_autoreconnect docs for details.
 
 === TRUE ASYNC
-For non-blocking connect use PG::EM::Client.async_connect and #async_reset for 
-asynchronous connection reset. Like other async methods they return deferrable object.
-Use #callback to obtain already connected Client.
+For non-blocking connect use PG::EM::Client.async_connect and
+PG::EM::Client#async_reset for asynchronous re-connect. Like other async
+methods they return +Deferrable+ object.
+Use Deferrable's #callback to obtain already connected PG::EM::Client.
 
   EM.run do
     pool = (1..10).map {
@@ -205,10 +296,16 @@ Use #callback to obtain already connected Client.
     end
   end
 
-=== EM-Synchrony
-Under +em-synchrony+ PG::EM::Client.new is fully asynchronous and blocks only current fiber.
-This also applies to #reset.
+=== Fibers / EM-Synchrony
+There is a special version of PG::EM::Client library with fiber aware methods
+for EM-Synchrony or other implementations of Fiber untangled EventMachine.
+
+The +require+ string is "em-synchrony/pg" instead of "pg/em".
+
++em-synchrony/pg+ version of PG::EM::Client.new is fully asynchronous and
+blocks only current fiber. This also applies to PG::EM::Client#reset.
 
+  require 'em-synchrony'
   require 'em-synchrony/pg'
 
   EM.synchrony do
@@ -219,6 +316,43 @@ This also applies to #reset.
     EM.stop
   end
 
+Although em-synchrony[https://github.com/igrigorik/em-synchrony/] provides
+very nice set of tools for untangled EventMachine, you don't really require
+it to fully benefit from this version of PG::EM::Client.
+
+==== PG::Connection methods adapted to EM-Synchrony
+The list of PG::EM::Client fiber aware methods for processing with
+EM-Synchrony / EventMachine.
+
+All +async_*+ methods are exactly the same as in pure EventMachine version
+of PG::EM::Client.
+
+The fiber aware methods are:
+
+* +Client.connect+ (singleton, alias: +new+, +open+, +setdb+, +setdblogin+)
+* +reset+
+* +exec+ (alias: +query+)
+* +prepare+
+* +exec_prepared+
+* +describe_prepared+
+* +describe_portal+
+
+Under the hood, these methods call async counterparts of themselves and +yield+ from current
+fiber awaiting for the result. The PG::Result (or PG::EM::Client for +connect+
+and +reset+) is then returned to the caller. If code block was given, it is
+executed with result as the argument. In that case the value of the block is
+returned instead and PG::Result is cleared (or in case of +connect+ or +reset+
+PG::EM::Client is being closed) after executing block. From single fiber point
+of view, they behave like regular blocking PG::Connection methods.
+
+Each of them is also automatic, detecting if EventMachine is running.
+If called outside EM event loop they are exactly the original methods of
+PG::Connection.
+
+Like in pure EventMachine version you can mix async, fiber aware and
+blocking methods without finishing the connection. You only need to
+start/stop EventMachine in between async calls.
+
 ==== Handling errors
 
   EM.synchrony do
@@ -289,6 +423,9 @@ This also applies to #reset.
     EM.stop
   end
 
+Specifying +on_autoreconnect+ as PG::EM::Client.new initialization option,
+implicitly enables +async_autoreconnect+.
+
 == LICENCE
 
 The MIT License - Copyright (c) 2012 Rafał Michalski

data/em-pg-client.gemspec

@@ -2,14 +2,14 @@ $:.unshift "lib"
 
 Gem::Specification.new do |s|
   s.name = "em-pg-client"
-  s.version = "0.2.0.pre.3"
+  s.version = "0.2.0"
   s.required_ruby_version = ">= 1.9.1"
   s.date = "#{Time.now.strftime("%Y-%m-%d")}"
   s.summary = "EventMachine PostgreSQL client"
   s.email = "rafal@yeondir.com"
   s.homepage = "http://github.com/royaltm/ruby-em-pg-client"
   s.require_path = "lib"
-  s.description = "PostgreSQL asynchronous EventMachine client (ruby-pg) wrapper"
+  s.description = "PostgreSQL asynchronous EventMachine client, based on pg interface (PG::Connection)"
   s.authors = ["Rafal Michalski"]
   s.files = `git ls-files`.split("\n") - ['.gitignore']
   s.test_files = Dir.glob("spec/**/*")

data/lib/em-synchrony/pg.rb

@@ -5,18 +5,41 @@ module PG
       # Author:: Rafal Michalski (mailto:royaltm75@gmail.com)
       # Licence:: MIT License
       #
-      # =PostgreSQL Client for EM-Synchrony
+      # =PostgreSQL Client for EM-Synchrony/Fibered EventMachine
       #
 
       # conform to *standard*
       alias_method :aquery, :async_query
 
-      # fiber untangled version of theese methods:
+      # fiber aware methods:
       # - exec (aliased as query)
       # - exec_prepared
       # - prepare
-      %w(exec exec_prepared prepare reset self.connect).each do |name|
+      # - describe_prepared
+      # - describe_portal
+      # - reset
+      # - Client.connect
+      %w(exec
+         exec_prepared
+         prepare
+         describe_prepared
+         describe_portal
+         reset
+         self.connect).each do |name|
         async_name = "async_#{name.split('.').last}"
+        blocking_call = case name
+        when 'reset'
+          '@async_command_aborted = false
+            super(*args, &blk)'
+        else
+          'super(*args, &blk)'
+        end
+        clear_method = case name
+        when 'reset', 'self.connect'
+          'finish'
+        else
+          'clear'
+        end
         class_eval <<-EOD
           def #{name}(*args, &blk)
             if ::EM.reactor_running?
@@ -28,12 +51,16 @@ module PG
               result = Fiber.yield
               raise result if result.is_a?(::Exception)
               if block_given?
-                yield result 
+                begin
+                  yield result
+                ensure
+                  result.#{clear_method}
+                end
               else
                 result
               end
             else
-              super(*args, &blk)
+              #{blocking_call}
             end
           end
         EOD