postgresql_cursor 0.6.1 → 0.6.2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
-SHA1:
-  metadata.gz: df0425fd30e37c9d61b270416c9892375b0abe69
-  data.tar.gz: f59a3b1291c6476a16b08b084118be22b76020f5
+SHA256:
+  metadata.gz: 8f60d787898c546a7c79c8ff6ca3aee03a54793e9a4e3c0798fc39ad6d6ae7a0
+  data.tar.gz: c175f5708bd1f0b1ba06c005254e7fb513d5ea72f9a7b373f16a42d969cb45eb
 SHA512:
-  metadata.gz: d5decdc3d118f14eac9d93d46c34bb3b11f51f784bf5ea5a6825aaffe9f85ae23d9094992ef938f363d003435014ef7d771d5d5db835cac22e811248cc881a38
-  data.tar.gz: 59f0e52d29bccd10d3b9e693d4c40e2920672eefcd3f103309b4f8a4c96d8aa055c9326c03dba11a21b2bb29a05896454d6c37730054c5f85f17c24c1bff23cf
+  metadata.gz: 547822d06073ffc68611179e93332d4840f250fe130d4448a0563986e6e41f9e30f26e3894ba326c4981ad51a44bff77bb326f4e4700f3267f80e6fc4e0008d5
+  data.tar.gz: 2479ab683e1a598ad18ef5a287106dda78113ab6e745c9ff3bd4eed982c5af9f0a2ad7d1b933b73328229bf39208aa0d25ed513607b06b417742091e7a4c7df6

data/README.md

@@ -1,4 +1,4 @@
-#PostgreSQLCursor for handling large Result Sets
+# PostgreSQLCursor for handling large Result Sets
 
 [![Gem Version](https://badge.fury.io/rb/postgresql_cursor.svg)](http://badge.fury.io/rb/postgresql_cursor)
 
@@ -13,16 +13,12 @@ set is exhausted. By fetching a smaller chunk of data, this reduces the
 amount of memory your application uses and prevents the potential crash
 of running out of memory.
 
-This extension is not intended to support the "FOR UPDATE / WHERE
-CURRENT OF" syntax to process and update each row in place. The primary
-goal is to read a large number of rows using buffering.
-
 Supports Rails/ActiveRecord v3.1 (v3.2 recommended) higher (including
 v5.0) and Ruby 1.9 and higher. Not all features work in ActiveRecord v3.1.
 Support for this gem will only be for officially supported versions of
 ActiveRecord and Ruby; others can try older versions of the gem.
 
-##Using Cursors
+## Using Cursors
 
 PostgreSQLCursor was developed to take advantage of PostgreSQL's cursors. Cursors allow the program
 to declare a cursor to run a given query returning "chunks" of rows to the application program while
@@ -79,7 +75,7 @@ Notes:
 * Aliases each_hash and each_hash_by_sql are provided for each_row and each_row_by_sql
   if you prefer to express what types are being returned.
 
-###PostgreSQLCursor is an Enumerable
+### PostgreSQLCursor is an Enumerable
 
 If you do not pass in a block, the cursor is returned, which mixes in the Enumerable
 libary. With that, you can pass it around, or chain in the awesome enumerable things
@@ -91,7 +87,7 @@ Product.each_row.map {|r| r["id"].to_i } #=> [1, 2, 3, ...]
 Product.each_instance.map {|r| r.id }.each {|id| p id } #=> [1, 2, 3, ...]
 Product.each_instance.lazy.inject(0) {|sum,r| sum +  r.quantity } #=> 499500
 ```
-###Hashes vs. Instances
+### Hashes vs. Instances
 
 The each_row method returns the Hash of strings for speed (as this allows you to process a lot of rows).
 Hashes are returned with String values, and you must take care of any type conversion.
@@ -103,7 +99,7 @@ If you find you need the types cast for your attributes, consider using each_ins
 insead. ActiveRecord's read casting algorithm will only cast the values you need and
 has become more efficient over time.
 
-###Select and Pluck
+### Select and Pluck
 
 To limit the columns returned to just those you need, use `.select(:id, :name)`
 query method.
@@ -128,7 +124,7 @@ Product.pluck_rows(:id) #=> ["1", "2", ...]
 Product.pluck_instances(:id, :quantity) #=> [[1, 503], [2, 932], ...]
 ```
 
-###Associations and Eager Loading
+### Associations and Eager Loading
 
 ActiveRecord performs some magic when eager-loading associated row. It
 will usually not join the tables, and prefers to load the data in
@@ -138,7 +134,33 @@ This library hooks onto the `to_sql` feature of the query builder. As a
 result, it can't do the join if ActiveRecord decided not to join, nor
 can it construct the association objects eagerly.
 
-##Background: Why PostgreSQL Cursors?
+## Locking and Updating Each Row (FOR UPDATE Queries)
+
+When you use the AREL `lock` method, a "FOR UPDATE" clause is added to
+the query. This causes the block of rows returned from each FETCH
+operation (see the `block_size` option) to be locked for you to update.
+The lock is released on those rows once the block is exhausted and the
+next FETCH or CLOSE statement is executed.
+
+This example will run through a large table and potentially update each
+row, locking only a set of rows at a time to allow concurrent use.
+
+```ruby
+Product.lock.each_instance(block_size:100) do |p|
+  p.update(price: p.price * 1.05)
+end
+```
+
+Also, pay attention to the `block_size` you request. Locking large
+blocks of rows for an extended time can cause deadlocks or other
+performance issues in your application. On a busy table, or if the
+processing of each row consumes a lot of time or resources, try a
+`block_size` <= 10.
+
+See the [PostgreSQL Select Documentation](https://www.postgresql.org/docs/current/static/sql-select.html)
+for more information and limitations when using "FOR UPDATE" locking.
+
+## Background: Why PostgreSQL Cursors?
 
 ActiveRecord is designed and optimized for web performance. In a web transaction, only a "page" of
 around 20 rows is returned to the user. When you do this
@@ -155,7 +177,7 @@ When there is a very large number of rows, this requires a lot more memory to ho
 does not return that memory after processing the array, and the causes your process to "bloat". If you
 don't have enough memory, it will cause an exception.
 
-###ActiveRecord.find_each and find_in_batches
+### ActiveRecord.find_each and find_in_batches
 
 To solve this problem, ActiveRecord gives us two alternative methods that work in "chunks" of your data:
 
@@ -179,7 +201,7 @@ There are drawbacks with these methods:
 ### How it works
 
 Under the covers, the library calls the PostgreSQL cursor operations
-with the psuedo-code:
+with the pseudo-code:
 
     SET cursor_tuple_fraction TO 1.0;
     DECLARE cursor_1 CURSOR WITH HOLD FOR select * from widgets;
@@ -189,17 +211,11 @@ with the psuedo-code:
     until rows.size < 100;
     CLOSE cursor_1;
 
-##Meta
-###Author
-Allen Fair, [@allenfair](https://twitter.com/allenfair), http://github.com/afair
-
-Thanks to:
-
-* Iulian Dogariu, http://github.com/iulianu (Fixes)
-* Julian Mehnle, julian@mehnle.net (Suggestions)
-* ...And all the other contributers!
+## Meta
+### Author
+Allen Fair, [@allenfair](https://twitter.com/allenfair), [github://afair](https://github.com/afair)
 
-###Note on Patches/Pull Requests
+### Note on Patches/Pull Requests
 
 * Fork the project.
 * Make your feature addition or bug fix.
@@ -209,11 +225,11 @@ Thanks to:
   (if you want to have your own version, that is fine but bump version in a commit by itself I can ignore when I pull)
 * Send me a pull request. Bonus points for topic branches.
 
-###Code of Conduct
+### Code of Conduct
 
 This project adheres to the [Open Code of Conduct](http://todogroup.org/opencodeofconduct/#postgresql_cursor/2016@allenfair.com).
 By participating, you are expected to honor this code.
 
-###Copyright
+### Copyright
 
-Copyright (c) 2010-2014 Allen Fair. See (MIT) LICENSE for details.
+Copyright (c) 2010-2017 Allen Fair. See (MIT) LICENSE for details.

data/lib/postgresql_cursor/active_record/relation/cursor_iterators.rb

@@ -42,6 +42,47 @@ module PostgreSQLCursor
           cursor.iterate_type(self)
         end
 
+        # Public: Executes the query, yielding each batch of up to block_size
+        # rows where each row is a hash to the given block.
+        #
+        # Parameters: same as each_row
+        #
+        # Example:
+        #   Post.where(user_id:123).each_row_batch do |batch|
+        #     Post.process_batch(batch)
+        #   end
+        #   Post.each_row_batch.map { |batch| Post.transform_batch(batch) }
+        #
+        # Returns the number of rows yielded to the block
+        def each_row_batch(options={}, &block)
+          options = {:connection => self.connection}.merge(options)
+          cursor  = PostgreSQLCursor::Cursor.new(to_unprepared_sql, options)
+          return cursor.each_row_batch(&block) if block_given?
+          cursor.iterate_batched
+        end
+        alias :each_hash_batch :each_row_batch
+
+        # Public: Like each_row, but yields an array of instantiated model
+        # objects to the block
+        #
+        # Parameters: same as each_row
+        #
+        # Example:
+        #   Post.where(user_id:123).each_instance_batch do |batch|
+        #     Post.process_batch(batch)
+        #   end
+        #   Post.where(user_id:123).each_instance_batch.map do |batch|
+        #     Post.transform_batch(batch)
+        #   end
+        #
+        # Returns the number of rows yielded to the block
+        def each_instance_batch(options={}, &block)
+          options = {:connection => self.connection}.merge(options)
+          cursor = PostgreSQLCursor::Cursor.new(to_unprepared_sql, options)
+          return cursor.each_instance_batch(self, &block) if block_given?
+          cursor.iterate_type(self).iterate_batched
+        end
+
         # Plucks the column names from the rows, and return them in an array
         def pluck_rows(*cols)
           options = cols.last.is_a?(Hash) ? cols.pop : {}

data/lib/postgresql_cursor/active_record/sql_cursor.rb

@@ -74,7 +74,81 @@ module PostgreSQLCursor
         cursor.iterate_type(self)
       end
 
-      # Returns and array of the given column names. Use if you need cursors and don't expect
+      # Public: Executes the query, yielding an array of up to block_size rows
+      # where each row is a hash to the given block.
+      #
+      # Parameters: same as each_row
+      #
+      # Example:
+      #   Post.each_row_batch { |batch| Post.process_batch(batch) }
+      #
+      # Returns the number of rows yielded to the block
+      def each_row_batch(options={}, &block)
+        options = {:connection => self.connection}.merge(options)
+        all.each_row_batch(options, &block)
+      end
+      alias :each_hash_batch :each_row_batch
+
+      # Public: Like each_row_batch, but yields an array of instantiated model
+      # objects to the block
+      #
+      # Parameters: same as each_row
+      #
+      # Example:
+      #   Post.each_instance_batch { |batch| Post.process_batch(batch) }
+      #
+      # Returns the number of rows yielded to the block
+      def each_instance_batch(options={}, &block)
+        options = {:connection => self.connection}.merge(options)
+        all.each_instance_batch(options, &block)
+      end
+
+      # Public: Yields each batch of up to block_size rows as an array of rows
+      # where each row as a hash to the given block
+      #
+      # Parameters: see each_row_by_sql
+      #
+      # Example:
+      #   Post.each_row_batch_by_sql("select * from posts") do |batch|
+      #     Post.process_batch(batch)
+      #   end
+      #   Post.each_row_batch_by_sql("select * from posts").map do |batch|
+      #     Post.transform_batch(batch)
+      #   end
+      #
+      # Returns the number of rows yielded to the block
+      def each_row_batch_by_sql(sql, options={}, &block)
+        options = {:connection => self.connection}.merge(options)
+        cursor  = PostgreSQLCursor::Cursor.new(sql, options)
+        return cursor.each_row_batch(&block) if block_given?
+        cursor.iterate_batched
+      end
+      alias :each_hash_batch_by_sql :each_row_batch_by_sql
+
+      # Public: Yields each batch up to block_size of rows as model instances
+      # to the given block
+      #
+      # As this instantiates a model object, it is slower than each_row_batch_by_sql
+      #
+      # Paramaters: see each_row_by_sql
+      #
+      # Example:
+      #   Post.each_instance_batch_by_sql("select * from posts") do |batch|
+      #     Post.process_batch(batch)
+      #   end
+      #   Post.each_instance_batch_by_sql("select * from posts").map do |batch|
+      #     Post.transform_batch(batch)
+      #   end
+      #
+      # Returns the number of rows yielded to the block
+      def each_instance_batch_by_sql(sql, options={}, &block)
+        options = {:connection => self.connection}.merge(options)
+        cursor  = PostgreSQLCursor::Cursor.new(sql, options)
+        return cursor.each_instance_batch(self, &block) if block_given?
+        cursor.iterate_type(self).iterate_batched
+      end
+
+      # Returns an array of the given column names. Use if you need cursors and don't expect
       # this to comsume too much memory. Values are strings. Like ActiveRecord's pluck.
       def pluck_rows(*cols)
         options = cols.last.is_a?(Hash) ? cols.pop : {}
@@ -82,7 +156,7 @@ module PostgreSQLCursor
       end
       alias :pluck_row :pluck_rows
 
-      # Returns and array of the given column names. Use if you need cursors and don't expect
+      # Returns an array of the given column names. Use if you need cursors and don't expect
       # this to comsume too much memory. Values are instance types. Like ActiveRecord's pluck.
       def pluck_instances(*cols)
         options = cols.last.is_a?(Hash) ? cols.pop : {}

data/lib/postgresql_cursor/cursor.rb

@@ -1,3 +1,5 @@
+require 'active_record/connection_adapters/postgresql/oid'
+
 ################################################################################
 # PostgreSQLCursor: library class provides postgresql cursor for large result
 # set processing. Requires ActiveRecord, but can be adapted to other DBI/ORM libraries.
@@ -17,6 +19,14 @@
 #   ActiveRecordModel.each_row_by_sql("select ...") { |hash| ... }
 #   ActiveRecordModel.each_instance_by_sql("select ...") { |model| ... }
 #
+
+
+if ::ActiveRecord::VERSION::MAJOR <= 4
+  OID = ActiveRecord::ConnectionAdapters::PostgreSQLAdapter::OID
+else
+  OID = ActiveRecord::ConnectionAdapters::PostgreSQL::OID
+end
+
 module PostgreSQLCursor
   class Cursor
     include Enumerable
@@ -43,6 +53,7 @@ module PostgreSQLCursor
       @connection = @options.fetch(:connection) { ::ActiveRecord::Base.connection }
       @count      = 0
       @iterate    = options[:instances] ? :each_instance : :each_row
+      @batched    = false
     end
 
     # Specify the type to instantiate, or reset to return a Hash
@@ -58,6 +69,11 @@ module PostgreSQLCursor
       self
     end
 
+    def iterate_batched(batched=true)
+      @batched = batched
+      self
+    end
+
     # Public: Yields each row of the result set to the passed block
     #
     # Yields the row to the block. The row is a hash with symbolized keys.
@@ -66,11 +82,11 @@ module PostgreSQLCursor
     # Returns the count of rows processed
     def each(&block)
       if @iterate == :each_row
-        self.each_row(&block)
+        @batched ? self.each_row_batch(&block) : self.each_row(&block)
       elsif @iterate == :each_array
-        self.each_array(&block)
+        @batched ? self.each_array_batch(&block) : self.each_array(&block)
       else
-        self.each_instance(@type, &block)
+        @batched ? self.each_instance_batch(@type, &block) : self.each_instance(@type, &block)
       end
     end
 
@@ -107,6 +123,41 @@ module PostgreSQLCursor
       end
     end
 
+    def each_row_batch(&block)
+      self.each_batch do |batch|
+        batch.map!(&:symbolize_keys) if @options[:symbolize_keys]
+        block.call(batch)
+      end
+    end
+
+    def each_array_batch(&block)
+      old_iterate = @iterate
+      @iterate = :each_array
+      begin
+        rv = self.each_batch do |batch|
+          block.call(batch)
+        end
+      ensure
+        @iterate = old_iterate
+      end
+      rv
+    end
+
+    def each_instance_batch(klass=nil, &block)
+      klass ||= @type
+      self.each_batch do |batch|
+        models = batch.map do |row|
+          if ::ActiveRecord::VERSION::MAJOR < 4
+            model = klass.send(:instantiate, row)
+          else
+            @column_types ||= column_types
+            model = klass.send(:instantiate, row, @column_types)
+          end
+        end
+        block.call(models)
+      end
+    end
+
     # Returns an array of columns plucked from the result rows.
     # Experimental function, as this could still use too much memory
     # and negate the purpose of this libarary.
@@ -152,6 +203,28 @@ module PostgreSQLCursor
       @count
     end
 
+    def each_batch(&block) #:nodoc:
+      has_do_until = @options.key?(:until)
+      has_do_while = @options.key?(:while)
+      @count = 0
+      @column_types = nil
+      with_optional_transaction do
+        begin
+          open
+          while (batch = fetch_block)
+            break if batch.empty?
+            @count += 1
+            rc = block.call(batch)
+            break if has_do_until && rc == @options[:until]
+            break if has_do_while && rc != @options[:while]
+          end
+        ensure
+          close if @block
+        end
+      end
+      @count
+    end
+
     def cast_types(row)
       row
     end
@@ -177,10 +250,9 @@ module PostgreSQLCursor
     # Public: Opens (actually, "declares") the cursor. Call this before fetching
     def open
       set_cursor_tuple_fraction
-      #@cursor = Digest::MD5.hexdigest(@sql)
       @cursor = SecureRandom.uuid.gsub("-","")
       hold = @options[:with_hold] ? 'with hold ' : ''
-      @result = @connection.execute("declare cursor_#{@cursor} cursor #{hold}for #{@sql}")
+      @result = @connection.execute("declare cursor_#{@cursor} no scroll cursor #{hold}for #{@sql}")
       @block = []
     end
 

data/lib/postgresql_cursor/version.rb

@@ -1,3 +1,3 @@
 module PostgresqlCursor
-  VERSION = "0.6.1"
+  VERSION = "0.6.2"
 end

data/test-app/Gemfile

@@ -1,7 +1,7 @@
 # A sample Gemfile
 source "https://rubygems.org"
 
-gem 'activerecord', '~> 3.1.0'
+gem 'activerecord', '~> 5.2.0'
 #gem 'activerecord', '~> 3.2.0'
 #gem 'activerecord', '~> 4.0.0'
 #gem 'activerecord', '~> 4.1.0'
@@ -12,4 +12,4 @@ gem 'activerecord', '~> 3.1.0'
 #gem 'arel', github: 'rails/arel', branch: 'master'
 
 gem 'pg'
-gem 'postgresql_cursor', path:"#{ENV['HOME']}/src/postgresql_cursor"
+gem 'postgresql_cursor', path:"../"

data/test/test_postgresql_cursor.rb

@@ -16,6 +16,13 @@ class TestPostgresqlCursor < Minitest::Test
     assert_equal nn, n
   end
 
+  def test_each_batch
+    c = PostgreSQLCursor::Cursor.new("select * from products order by 1")
+    nn = 0
+    n = c.each_batch { |b| nn += 1 }
+    assert_equal nn, n
+  end
+
   def test_enumerables
     assert_equal true, PostgreSQLCursor::Cursor.new("select * from products order by 1").any?
     assert_equal false, PostgreSQLCursor::Cursor.new("select * from products where id<0").any?
@@ -23,12 +30,22 @@ class TestPostgresqlCursor < Minitest::Test
 
   def test_each_while_until
     c = PostgreSQLCursor::Cursor.new("select * from products order by 1", until:true)
-    n = c.each { |r| r[:id].to_i > 100 }
-    assert_equal 1000, n
+    n = c.each { |r| r['id'].to_i > 100 }
+    assert_equal 101, n
 
     c = PostgreSQLCursor::Cursor.new("select * from products order by 1", while:true)
-    n = c.each { |r| r[:id].to_i < 100 }
-    assert_equal 1000, n
+    n = c.each { |r| r['id'].to_i < 100 }
+    assert_equal 100, n
+  end
+
+  def test_each_batch_while_until
+    c = PostgreSQLCursor::Cursor.new("select * from products order by id asc", until: true, block_size: 50)
+    n = c.each_batch { |b| b.last['id'].to_i > 100 }
+    assert_equal 3, n
+
+    c = PostgreSQLCursor::Cursor.new("select * from products order by id asc", while: true, block_size: 50)
+    n = c.each_batch { |b| b.last['id'].to_i < 100 }
+    assert_equal 2, n
   end
 
   def test_each_array
@@ -39,12 +56,36 @@ class TestPostgresqlCursor < Minitest::Test
     end
   end
 
+  def test_each_array_batch
+    c = PostgreSQLCursor::Cursor.new("select * from products where id = 1")
+    c.each_array_batch do |b|
+      assert_equal 1, b.size
+      ary = b.first
+      assert_equal Array, ary.class
+      assert_equal 1, ary[0].to_i
+    end
+  end
+
   def test_relation
     nn = 0
     Product.where("id>0").each_row {|r| nn += 1 }
     assert_equal 1000, nn
   end
 
+  def test_relation_batch
+    nn = 0
+    row = nil
+    Product.where("id>0").each_row_batch(block_size: 100) { |b| row = b.last; nn += 1 }
+    assert_equal 10, nn
+    assert_equal Hash, row.class
+
+    nn = 0
+    row = nil
+    Product.where("id>0").each_instance_batch(block_size: 100) { |b| row = b.last; nn += 1 }
+    assert_equal 10, nn
+    assert_equal Product, row.class
+  end
+
   def test_activerecord
     nn = 0
     row = nil
@@ -58,6 +99,19 @@ class TestPostgresqlCursor < Minitest::Test
     assert_equal Product, row.class
   end
 
+  def test_activerecord_batch
+    nn = 0
+    row = nil
+    Product.each_row_batch_by_sql("select * from products", block_size: 100) { |b| row = b.last; nn += 1 }
+    assert_equal 10, nn
+    assert_equal Hash, row.class
+
+    nn = 0
+    Product.each_instance_batch_by_sql("select * from products", block_size: 100) { |b| row = b.last; nn += 1 }
+    assert_equal 10, nn
+    assert_equal Product, row.class
+  end
+
   def test_exception
     begin
       Product.each_row_by_sql("select * from products") do |r|
@@ -68,6 +122,14 @@ class TestPostgresqlCursor < Minitest::Test
     end
   end
 
+  def test_batch_exception
+    Product.each_row_batch_by_sql("select * from products") do |r|
+      raise 'Oops'
+    end
+  rescue => e
+    assert_equal e.message, 'Oops'
+  end
+
   def test_cursor
     cursor = Product.all.each_row
     assert cursor.respond_to?(:each)
@@ -79,12 +141,23 @@ class TestPostgresqlCursor < Minitest::Test
     assert_equal 1000, r.size
   end
 
+  def test_batched_cursor
+    cursor = Product.all.each_row_batch(block_size: 100)
+    assert cursor.respond_to?(:each)
+    b = cursor.map { |batch| batch.map { |r| r['id'] } }
+    assert_equal 10, b.size
+    cursor = Product.each_row_batch_by_sql("select * from products", block_size: 100)
+    assert cursor.respond_to?(:each)
+    b = cursor.map { |batch| batch.map { |r| r['id'] } }
+    assert_equal 10, b.size
+  end
+
   def test_pluck
     r = Product.pluck_rows(:id)
     assert_equal 1000, r.size
     r = Product.all.pluck_instances(:id)
     assert_equal 1000, r.size
-    assert_equal Fixnum, r.first.class
+    assert_equal Integer, r.first.class
   end
 
   def test_with_hold

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: postgresql_cursor
 version: !ruby/object:Gem::Version
-  version: 0.6.1
+  version: 0.6.2
 platform: ruby
 authors:
 - Allen Fair
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2016-09-02 00:00:00.000000000 Z
+date: 2018-09-08 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: activerecord
@@ -116,7 +116,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
       version: '0'
 requirements: []
 rubyforge_project: 
-rubygems_version: 2.5.1
+rubygems_version: 2.7.7
 signing_key: 
 specification_version: 4
 summary: ActiveRecord PostgreSQL Adapter extension for using a cursor to return a