occams-record 1.2.1 → 1.4.0.pre.beta1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: a1e01657287159a2aa1a27ff66c1932ae83235ad0140dbaafd58103862ddaa8f
-  data.tar.gz: ec9530c3844cd79739d1a2387031654e85f4a2a127d948df0c537431aa235ed5
+  metadata.gz: 662bd65e909a1d957ebdbbd2d3bec3fe96fb45aac2117a786c9a3ad02b03259b
+  data.tar.gz: abeb69ff706dbb11f3fae3672ae60c407080f418fce7330779a7b550915aa886
 SHA512:
-  metadata.gz: a0303147fa894e42d0642020a4a207c67b3a79fa3fc2dfa1b9be819224c8eee6bb261bfa2079a073cd1894f744b4995f307afff594f458c5b7ae7574b26b0d32
-  data.tar.gz: bf81dc3eeb78cbf9ed99740a1c4c8e1c71287f4af278c2a623d9220384460f952ccc50fa835393ef3fd416fa0019fccc6a6e41d0c8dbd77c637f08689ca5c585
+  metadata.gz: 8981d9c820b64ebeca28dcbe3ab7ac44e35d8ec7e14df201e890ccf6cf1cf1888d6f78ba6cac001f1a9c458f604189b1aafaf7ff644288499faf5d5afeb896c3
+  data.tar.gz: cdd61c50313a964db4ebdb271dbb875fc4ea888dd0b57df78be78d043e655e04e821e0cef8d9b60034e90866289a0d764f72a7728a18f46c778e511101b72f4a

data/README.md

@@ -17,17 +17,17 @@ Continue using ActiveRecord's query builder, but let Occams take over running th
 **Customize the SQL used to eager load associations**
 
 ```ruby
-OccamsRecord.
-  query(User.active).
-  eager_load(:orders, ->(q) { q.where("created_at >= ?", date).order("created_at DESC") })
+OccamsRecord
+  .query(User.active)
+  .eager_load(:orders, ->(q) { q.where("created_at >= ?", date).order("created_at DESC") })
 ```
 
 **Use `ORDER BY` with `find_each`/`find_in_batches`**
 
 ```ruby
-OccamsRecord.
-  query(Order.order("created_at DESC")).
-  find_each { |order|
+OccamsRecord
+  .query(Order.order("created_at DESC"))
+  .find_each { |order|
     ...
   }
 ```
@@ -35,15 +35,15 @@ OccamsRecord.
 **Use `find_each`/`find_in_batches` with raw SQL**
 
 ```ruby
-OccamsRecord.
-  sql("
+OccamsRecord
+  .sql("
     SELECT * FROM orders
     WHERE created_at >= %{date}
     LIMIT %{batch_limit}
     OFFSET %{batch_offset}",
     {date: 10.years.ago}
-  ).
-  find_each { |order|
+  )
+  .find_each { |order|
     ...
   }
 ```
@@ -51,13 +51,13 @@ OccamsRecord.
 **Eager load associations when you're writing raw SQL**
 
 ```ruby
-OccamsRecord.
-  sql("
+OccamsRecord
+  .sql("
     SELECT * FROM users
     LEFT OUTER JOIN ...
-  ").
-  model(User).
-  eager_load(:orders)
+  ")
+  .model(User)
+  .eager_load(:orders)
 ```
 
 **Eager load "ad hoc associations" using raw SQL**
@@ -66,9 +66,9 @@ Relationships are complicated, and sometimes they can't be expressed in ActiveRe
 (Don't worry, there's a full explanation later on.)
 
 ```ruby
-OccamsRecord.
-  query(User.all).
-  eager_load_many(:orders, {:id => :user_id}, "
+OccamsRecord
+  .query(User.all)
+  .eager_load_many(:orders, {:id => :user_id}, "
     SELECT user_id, orders.*
     FROM orders INNER JOIN ...
     WHERE user_id IN (%{ids})
@@ -102,14 +102,14 @@ Code lives at at [github.com/jhollinger/occams-record](https://github.com/jholli
 Build your queries like normal, using ActiveRecord's excellent query builder. Then pass them off to Occams Record.
 
 ```ruby
-q = Order.
-  completed.
-  where("order_date > ?", 30.days.ago).
-  order("order_date DESC")
-
-orders = OccamsRecord.
-  query(q).
-  run
+q = Order
+  .completed
+  .where("order_date > ?", 30.days.ago)
+  .order("order_date DESC")
+
+orders = OccamsRecord
+  .query(q)
+  .run
 ````
 
 `each`, `map`, `reduce`, and other Enumerable methods may be used instead of *run*. `find_each` and `find_in_batches` are also supported, and unlike in ActiveRecord, `ORDER BY` works as you'd expect.
@@ -121,14 +121,14 @@ Occams Record has great support for raw SQL queries too, but we'll get to those
 Eager loading is similiar to ActiveRecord's `preload`: each association is loaded in a separate query. Unlike ActiveRecord, nested associations use blocks instead of Hashes. More importantly, if you try to use an association you didn't eager load *an exception will be raised*. In other words, the N+1 query problem simply doesn't exist.
 
 ```ruby
-OccamsRecord.
-  query(q).
-  eager_load(:customer).
-  eager_load(:line_items) {
+OccamsRecord
+  .query(q)
+  .eager_load(:customer)
+  .eager_load(:line_items) {
     eager_load(:product)
     eager_load(:something_else)
-  }.
-  find_each { |order|
+  }
+  .find_each { |order|
     puts order.customer.name
     order.line_items.each { |line_item|
       puts line_item.product.name
@@ -143,17 +143,17 @@ OccamsRecord.
 Occams Record allows you to tweak the SQL of any eager load. Pull back only the columns you need, change the order, add a `WHERE` clause, etc.
 
 ```ruby
-orders = OccamsRecord.
-  query(q).
+orders = OccamsRecord
+  .query(q)
   # Only SELECT the columns you need. Your DBA will thank you.
-  eager_load(:customer, select: "id, name").
+  .eager_load(:customer, select: "id, name")
   
   # A Proc can use ActiveRecord's query builder
-  eager_load(:line_items, ->(q) { q.active.order("created_at") }) {
+  .eager_load(:line_items, ->(q) { q.active.order("created_at") }) {
     eager_load(:product)
     eager_load(:something_else)
-  }.
-  run
+  }
+  .run
 ```
 
 Occams Record also supports loading ad hoc associations using raw SQL. We'll get to that in the next section.
@@ -167,8 +167,8 @@ ActiveRecord has raw SQL escape hatches like `find_by_sql` and `exec_query`, but
 To use `find_each`/`find_in_batches` you must provide the limit and offset statements yourself; Occams will provide the values. Also, notice that the binding syntax is a bit different (it uses Ruby's built-in named string substitution).
 
 ```ruby
-OccamsRecord.
-  sql("
+OccamsRecord
+  .sql("
     SELECT * FROM orders
     WHERE order_date > %{date}
     ORDER BY order_date DESC, id
@@ -176,8 +176,8 @@ OccamsRecord.
     OFFSET %{batch_offset}
   ", {
     date: 10.years.ago
-  }).
-  find_each(batch_size: 1000) do |order|
+  })
+  .find_each(batch_size: 1000) do |order|
     ...
   end
 ```
@@ -187,17 +187,17 @@ OccamsRecord.
 To use `eager_load` with a raw SQL query you must tell Occams what the base model is. (That doesn't apply if you're loading an ad hoc, raw SQL association. We'll get to those next.)
 
 ```ruby
-orders = OccamsRecord.
-  sql("
+orders = OccamsRecord
+  .sql("
     SELECT * FROM orders
     WHERE order_date > %{date}
     ORDER BY order_date DESC, id
   ", {
     date: 30.days.ago
-  }).
-  model(Order).
-  eager_load(:customer).
-  run
+  })
+  .model(Order)
+  .eager_load(:customer)
+  .run
 ```
 
 ## Raw SQL eager loading
@@ -205,14 +205,14 @@ orders = OccamsRecord.
 Let's say we want to load each product with an array of all customers who've ordered it. We *could* do that by loading various nested associations:
 
 ```ruby
-products_with_orders = OccamsRecord.
-  query(Product.all).
-  eager_load(:line_items) {
+products_with_orders = OccamsRecord
+  .query(Product.all)
+  .eager_load(:line_items) {
     eager_load(:order) {
       eager_load(:customer)
     }
-  }.
-  map { |product|
+  }
+  .map { |product|
     customers = product.line_items.map(&:order).map(&:customer).uniq
     [product, customers]
   }
@@ -221,9 +221,9 @@ products_with_orders = OccamsRecord.
 But that's very wasteful. Occams gives us better options: `eager_load_many` and `eager_load_one`.
 
 ```ruby
-products = OccamsRecord.
-  query(Product.all).
-  eager_load_many(:customers, {:id => :product_id}, "
+products = OccamsRecord
+  .query(Product.all)
+  .eager_load_many(:customers, {:id => :product_id}, "
     SELECT DISTINCT product_id, customers.*
     FROM line_items
       INNER JOIN orders ON line_items.order_id = orders.id
@@ -231,8 +231,8 @@ products = OccamsRecord.
     WHERE line_items.product_id IN (%{ids})
   ", binds: {
     # additional bind values (ids will be passed in for you)
-  }).
-  run
+  })
+  .run
 ```
 
 `eager_load_many` is declaring an ad hoc *has_many* association called *customers*. The `{:id => :product_id}` Hash defines the mapping: *id* in the parent record maps to *product_id* in the child records.
@@ -258,12 +258,12 @@ module MyProductMethods
   end
 end
 
-orders = OccamsRecord.
-  query(Order.all, use: MyOrderMethods).
-  eager_load(:line_items) {
+orders = OccamsRecord
+  .query(Order.all, use: MyOrderMethods)
+  .eager_load(:line_items) {
     eager_load(:product, use: [MyProductMethods, OtherMethods])
-  }.
-  run
+  }
+  .run
 ```
 
 ---
@@ -300,6 +300,9 @@ bundle exec rake test
 
 # test against Postgres
 TEST_DATABASE_URL=postgres://postgres@localhost:5432/occams_record bundle exec rake test
+
+# test against MySQL
+TEST_DATABASE_URL=mysql2://root:@127.0.0.1:3306/occams_record bundle exec rake test
 ```
 
 **Test against all supported ActiveRecord versions**
@@ -307,11 +310,14 @@ TEST_DATABASE_URL=postgres://postgres@localhost:5432/occams_record bundle exec r
 ```bash
 bundle exec appraisal install
 
-# test against SQLite
+# test against all supported AR versions (defaults to SQLite)
 bundle exec appraisal rake test
 
+# test against a specific AR version
+bundle exec appraisal ar-7.0 rake test
+
 # test against Postgres
-TEST_DATABASE_URL=postgres://postgres@localhost:5432/occams_record bundle exec appraisal rake test
+TEST_DATABASE_URL=postgresql://postgres@localhost:5432/occams_record bundle exec appraisal rake test
 ```
 
 # License

data/lib/occams-record/batches/cursor_helpers.rb

@@ -0,0 +1,59 @@
+module OccamsRecord
+  module Batches
+    module CursorHelpers
+      #
+      # Loads records in batches of N and yields each record to a block (if given). If no block is given,
+      # returns an Enumerator.
+      #
+      # NOTE Unlike find_each, batches are loaded using a cursor, which offers better performance.
+      # Postgres only. See the docs for OccamsRecord::Cursor for more details.
+      #
+      # @param batch_size [Integer] fetch this many rows at once
+      # @param use_transaction [Boolean] Ensure it runs inside of a database transaction
+      # @yield [OccamsRecord::Results::Row]
+      # @return [Enumerator] will yield each record
+      #
+      def find_each_with_cursor(batch_size: 1000, use_transaction: true)
+        enum = Enumerator.new { |y|
+          cursor.open(use_transaction: use_transaction) { |c|
+            c.each(batch_size: batch_size) { |record|
+              y.yield record
+            }
+          }
+        }
+        if block_given?
+          enum.each { |record| yield record }
+        else
+          enum
+        end
+      end
+
+      #
+      # Loads records in batches of N and yields each batch to a block (if given). If no block is given,
+      # returns an Enumerator.
+      #
+      # NOTE Unlike find_in_batches, batches are loaded using a cursor, which offers better performance.
+      # Postgres only. See the docs for OccamsRecord::Cursor for more details.
+      #
+      # @param batch_size [Integer] fetch this many rows at once
+      # @param use_transaction [Boolean] Ensure it runs inside of a database transaction
+      # @yield [OccamsRecord::Results::Row]
+      # @return [Enumerator] will yield each record
+      #
+      def find_in_batches_with_cursor(batch_size: 1000, use_transaction: true)
+        enum = Enumerator.new { |y|
+          cursor.open(use_transaction: use_transaction) { |c|
+            c.each_batch(batch_size: batch_size) { |batch|
+              y.yield batch
+            }
+          }
+        } 
+        if block_given?
+          enum.each { |batch| yield batch }
+        else
+          enum
+        end
+      end
+    end
+  end
+end

data/lib/occams-record/batches/offset_limit/raw_query.rb

@@ -0,0 +1,53 @@
+module OccamsRecord
+  module Batches
+    module OffsetLimit
+      class RawQuery
+        def initialize(conn, sql, binds, use: nil, query_logger: nil, eager_loaders: nil)
+          @conn, @sql, @binds = conn, sql, binds
+          @use, @query_logger, @eager_loaders = use, query_logger, eager_loaders
+
+          unless @sql =~ /LIMIT\s+%\{batch_limit\}/i and @sql =~ /OFFSET\s+%\{batch_offset\}/i
+            raise ArgumentError, "When using find_each/find_in_batches you must specify 'LIMIT %{batch_limit} OFFSET %{batch_offset}'. SQL statement: #{@sql}"
+          end
+        end
+
+        #
+        # Returns an Enumerator that yields batches of records, of size "of".
+        # The SQL string must include 'LIMIT %{batch_limit} OFFSET %{batch_offset}'.
+        # The bind values will be provided by OccamsRecord.
+        #
+        # @param batch_size [Integer] batch size
+        # @param use_transaction [Boolean] Ensure it runs inside of a database transaction
+        # @return [Enumerator] yields batches
+        #
+        def enum(batch_size:, use_transaction: true)
+          Enumerator.new do |y|
+            if use_transaction and @conn.open_transactions == 0
+              @conn.transaction {
+                run_batches y, batch_size
+              }
+            else
+              run_batches y, batch_size
+            end
+          end
+        end
+
+        private
+
+        def run_batches(y, of)
+          offset = 0
+          loop do
+            results = ::OccamsRecord::RawQuery.new(@sql, @binds.merge({
+              batch_limit: of,
+              batch_offset: offset,
+            }), use: @use, query_logger: @query_logger, eager_loaders: @eager_loaders).run
+
+            y.yield results if results.any?
+            break if results.size < of
+            offset += results.size
+          end
+        end
+      end
+    end
+  end
+end

data/lib/occams-record/batches/offset_limit/scoped.rb

@@ -0,0 +1,72 @@
+module OccamsRecord
+  #
+  # Methods for building batch finding methods. It expects "model" and "scope" methods to be present.
+  #
+  module Batches
+    module OffsetLimit
+      class Scoped
+        def initialize(model, scope, use: nil, query_logger: nil, eager_loaders: nil)
+          @model, @scope = model, scope
+          @use, @query_logger, @eager_loaders = use, query_logger, eager_loaders
+        end
+
+        #
+        # Returns an Enumerator that yields batches of records, of size "of".
+        # NOTE ActiveRecord 5+ provides the 'in_batches' method to do something
+        # similiar, although 4.2 does not. Also it does not respect ORDER BY,
+        # whereas this does.
+        #
+        # @param batch_size [Integer] batch size
+        # @param use_transaction [Boolean] Ensure it runs inside of a database transaction
+        # @param append_order_by [String] Append this column to ORDER BY to ensure consistent results. Defaults to the primary key. Pass false to disable.
+        # @return [Enumerator] yields batches
+        #
+        def enum(batch_size:, use_transaction: true, append_order_by: nil)
+          append_order =
+            case append_order_by
+            when false then nil
+            when nil then @model.primary_key
+            else append_order_by
+            end
+
+          Enumerator.new do |y|
+            if use_transaction and @model.connection.open_transactions == 0
+              @model.connection.transaction {
+                run_batches y, batch_size, append_order
+              }
+            else
+              run_batches y, batch_size, append_order
+            end
+          end
+        end
+
+        private
+
+        def run_batches(y, of, append_order_by = nil)
+          limit = @scope.limit_value
+          batch_size = limit && limit < of ? limit : of
+
+          offset = @scope.offset_value || 0
+          out_of_records, count = false, 0
+          order_by =
+            if append_order_by
+              append_order_by.to_s == @model.primary_key.to_s ? append_order_by.to_sym : append_order_by
+            end
+
+          until out_of_records
+            l = limit && batch_size > limit - count ? limit - count : batch_size
+            q = @scope
+            q = q.order(order_by) if order_by
+            q = q.offset(offset).limit(l)
+            results = ::OccamsRecord::Query.new(q, use: @use, query_logger: @query_logger, eager_loaders: @eager_loaders).run
+
+            y.yield results if results.any?
+            count += results.size
+            offset += results.size
+            out_of_records = results.size < batch_size || (limit && count >= limit)
+          end
+        end
+      end
+    end
+  end
+end

data/lib/occams-record/cursor.rb

@@ -0,0 +1,237 @@
+require 'securerandom'
+
+module OccamsRecord
+#
+# An interface to database cursors. Supported databases:
+#   * PostgreSQL
+#
+  class Cursor
+    # @private
+    SCROLL = {
+      true => "SCROLL",
+      false => "NO SCROLL",
+      nil => "",
+    }.freeze
+
+    # @private
+    HOLD = {
+      true => "WITH HOLD",
+      false => "WITHOUT HOLD",
+      nil => "",
+    }.freeze
+
+    # @private
+    DIRECTIONS = {
+      next: "NEXT",
+      prior: "PRIOR",
+      first: "FIRST",
+      last: "LAST",
+      absolute: "ABSOLUTE",
+      relative: "RELATIVE",
+      forward: "FORWARD",
+      backward: "BACKWARD",
+    }.freeze
+
+    # @return [ActiveRecord::Connection]
+    attr_reader :conn
+
+    # Name of the cursor
+    # @return [String]
+    attr_reader :name
+
+    # Name of the cursor (safely SQL-escaped)
+    # @return [String]
+    attr_reader :quoted_name
+
+    #
+    # Initializes a new Cursor. NOTE all operations must be performed within a block passed to #open.
+    #
+    # While you CAN manually initialize a cursor, it's more common to get one via OccamsRecord::Query#cursor
+    # or OccamsRecord::RawQuery#cursor.
+    #
+    # @param conn [ActiveRecord::Connection]
+    # @param sql [String] The query to run
+    # @param name [String] Specify a name for the cursor (defaults to a random name)
+    # @param scroll [Boolean] true = SCROLL, false = NO SCROLL, nil = default behavior of DB
+    # @param hold [Boolean] true = WITH HOLD, false = WITHOUT HOLD, nil = default behavior of DB
+    # @param use [Array<Module>] optional Module to include in the result class (single or array)
+    # @param query_logger [Array] (optional) an array into which all queries will be inserted for logging/debug purposes
+    # @param eager_loaders [OccamsRecord::EagerLoaders::Context]
+    #
+    def initialize(conn, sql, name: nil, scroll: nil, hold: nil, use: nil, query_logger: nil, eager_loaders: nil)
+      @conn, @sql = conn, sql
+      @scroll = SCROLL.fetch(scroll)
+      @hold = HOLD.fetch(hold)
+      @use, @query_logger, @eager_loaders = use, query_logger, eager_loaders
+      @name = name || "occams_cursor_#{SecureRandom.hex 4}"
+      @quoted_name = conn.quote_table_name(@name)
+    end
+
+    #
+    # Declares and opens the cursor, runs the given block (yielding self), and closes the cursor.
+    #
+    #   cursor.open do |c|
+    #     c.fetch :forward, 100
+    #   end
+    #
+    # @param use_transaction [Boolean] When true, ensures it's wrapped in a transaction
+    # @yield [self]
+    # @return the value returned by the block
+    #
+    def open(use_transaction: true)
+      raise ArgumentError, "A block is required" unless block_given?
+      if use_transaction and conn.open_transactions == 0
+        conn.transaction {
+          perform { yield self }
+        }
+      else
+        perform { yield self }
+      end
+    end
+
+    #
+    # Loads records in batches of N and yields each record to a block (if given). If no block is given,
+    # returns an Enumerator.
+    #
+    #   cursor.open do |c|
+    #     c.each do |record|
+    #       ...
+    #     end
+    #   end
+    #
+    # @param batch_size [Integer] fetch this many rows at once
+    #
+    def each(batch_size: 1000)
+      enum = Enumerator.new { |y|
+        each_batch(batch_size: batch_size).each { |batch|
+          batch.each { |record| y.yield record }
+        }
+      }
+      if block_given?
+        enum.each { |record| yield record }
+      else
+        enum
+      end
+    end
+
+    #
+    # Loads records in batches of N and yields each batch to a block (if given). If no block is given,
+    # returns an Enumerator.
+    #
+    #   cursor.open do |c|
+    #     c.each_batch do |batch|
+    #       ...
+    #     end
+    #   end
+    #
+    # @param batch_size [Integer] fetch this many rows at once
+    #
+    def each_batch(batch_size: 1000)
+      enum = Enumerator.new { |y|
+        out_of_records = false
+        until out_of_records
+          results = fetch :forward, batch_size
+          y.yield results if results.any?
+          out_of_records = results.size < batch_size
+        end
+      }
+      if block_given?
+        enum.each { |batch| yield batch }
+      else
+        enum
+      end
+    end
+
+    #
+    # Fetch records in the given direction.
+    #
+    #   cursor.open do |c|
+    #     c.fetch :forward, 100
+    #     ...
+    #   end
+    #
+    # @param direction [Symbol] :next, :prior, :first, :last, :absolute, :relative, :forward or :backward
+    # @param num [Integer] number of rows to fetch (optional for some directions)
+    # @return [OccamsRecord::Results::Row]
+    #
+    def fetch(direction, num = nil)
+      query "FETCH %{dir} %{num} FROM %{name}".freeze % {
+        dir: DIRECTIONS.fetch(direction),
+        num: num&.to_i,
+        name: @quoted_name,
+      }
+    end
+
+    #
+    # Move the cursor the given direction.
+    #
+    #   cursor.open do |c|
+    #     ...
+    #     c.move :backward, 100
+    #     ...
+    #   end
+    #
+    # @param direction [Symbol] :next, :prior, :first, :last, :absolute, :relative, :forward or :backward
+    # @param num [Integer] number of rows to move (optional for some directions)
+    #
+    def move(direction, num = nil)
+      query "MOVE %{dir} %{num} FROM %{name}".freeze % {
+        dir: DIRECTIONS.fetch(direction),
+        num: num&.to_i,
+        name: @quoted_name,
+      }
+    end
+
+    #
+    # Run an arbitrary query on the cursor. Use 'binds' to escape inputs.
+    #
+    #   cursor.open do |c|
+    #     c.query("FETCH FORWARD %{num} FOR #{c.quoted_name}", {num: 100})
+    #     ...
+    #   end
+    #
+    def query(sql, binds = {})
+      ::OccamsRecord::RawQuery.new(sql, binds, use: @use, query_logger: @query_logger, eager_loaders: @eager_loaders, connection: conn).run
+    end
+
+    #
+    # Run an arbitrary command on the cursor. Use 'binds' to escape inputs.
+    #
+    #   cursor.open do |c|
+    #     c.execute("MOVE FORWARD %{num} FOR #{c.quoted_name}", {num: 100})
+    #     ...
+    #   end
+    #
+    def execute(sql, binds = {})
+      conn.execute(sql % binds.reduce({}) { |acc, (key, val)|
+        acc[key] = conn.quote(val)
+        acc
+      })
+    end
+
+    private
+
+    def perform
+      ex = nil
+      conn.execute "DECLARE %{name} %{scroll} CURSOR %{hold} FOR %{query}".freeze % {
+        name: @quoted_name,
+        scroll: @scroll,
+        hold: @hold,
+        query: @sql,
+      }
+      yield
+    rescue => e
+      ex = e
+      raise ex
+    ensure
+      begin
+        conn.execute "CLOSE %{name}".freeze % {name: @quoted_name}
+      rescue => e
+        # Don't let an error from CLOSE (like a dead transaction) hide what lead to the error with CLOSE (like bad SQL that raised an error and aborted the transaction)
+        raise ex || e
+      else
+        raise ex if ex
+      end
+    end
+  end
+end

data/lib/occams-record/eager_loaders/builder.rb

@@ -56,14 +56,14 @@ module OccamsRecord
       #
       # The %{category_ids} bind param will be provided for you, and in this case will be all the category_id values from the Widget query.
       #
-      #   res = OccamsRecord.
-      #     query(Widget.order("name")).
-      #     eager_load_one(:category, {:category_id => :id}, "
+      #   res = OccamsRecord
+      #     .query(Widget.order("name"))
+      #     .eager_load_one(:category, {:category_id => :id}, "
       #       SELECT * FROM categories WHERE id IN (%{category_ids}) AND name != %{bad_name}
       #     ", binds: {
       #       bad_name: "Bad Category"
-      #     }).
-      #     run
+      #     })
+      #     .run
       #
       # @param name [Symbol] name of attribute to load records into
       # @param mapping [Hash] a Hash that defines the key mapping of the parent (widgets.category_id) to the child (categories.id).
@@ -89,14 +89,14 @@ module OccamsRecord
       # The %{ids} bind param will be provided for you, and in this case will be all the id values from the Widget
       # query.
       #
-      #   res = OccamsRecord.
-      #     query(Widget.order("name")).
-      #     eager_load_many(:parts, {:id => :widget_id}, "
+      #   res = OccamsRecord
+      #     .query(Widget.order("name"))
+      #     .eager_load_many(:parts, {:id => :widget_id}, "
       #       SELECT * FROM parts WHERE widget_id IN (%{ids}) AND sku NOT IN (%{bad_skus})
       #     ", binds: {
       #       bad_skus: ["G90023ASDf0"]
-      #     }).
-      #     run
+      #     })
+      #     .run
       #
       # @param name [Symbol] name of attribute to load records into
       # @param mapping [Hash] a Hash that defines the key mapping of the parent (widgets.id) to the children (parts.widget_id).

data/lib/occams-record/eager_loaders/through.rb

@@ -9,7 +9,7 @@ module OccamsRecord
       #
       # See documentation for OccamsRecord::EagerLoaders::Base.
       #
-      def initialize(*args)
+      def initialize(ref, scope = nil, use: nil, as: nil, optimizer: :select, &builder)
         super
 
         unless @ref.macro == :has_one or @ref.macro == :has_many

data/lib/occams-record/measureable.rb

@@ -5,18 +5,18 @@ module OccamsRecord
   #
   # Measure the time each query takes. Useful for figuring out which query is the slow one when you're doing a bunch of eager loads.
   #
-  # orders = OccamsRecord.
-  #   query(Order.all).
-  #   eager_load(:customer).
+  # orders = OccamsRecord
+  #   .query(Order.all)
+  #   .eager_load(:customer)
   #   ...
-  #   measure { |x|
+  #   .measure { |x|
   #     puts "Total time: #{x.total_time} sec"
   #     x.queries.each { |q|
   #       puts "Table: #{q.table_name} (#{q.time} sec)"
   #       puts q.sql
   #     }
-  #   }.
-  #   run
+  #   }
+  #   .run
   #
   module Measureable
     #

data/lib/occams-record/query.rb

@@ -1,5 +1,3 @@
-require 'occams-record/batches'
-
 module OccamsRecord
   #
   # Starts building a OccamsRecord::Query. Pass it a scope from any of ActiveRecord's query builder
@@ -7,15 +5,15 @@ module OccamsRecord
   # Instead use OccamsRecord::Query#eager_load. Finally, call `run` (or any Enumerable method) to run
   # the query and get back an array of objects.
   #
-  #  results = OccamsRecord.
-  #    query(Widget.order("name")).
-  #    eager_load(:category).
-  #    eager_load(:order_items, ->(q) { q.select("widget_id, order_id") }) {
+  #  results = OccamsRecord
+  #    .query(Widget.order("name"))
+  #    .eager_load(:category)
+  #    .eager_load(:order_items, ->(q) { q.select("widget_id, order_id") }) {
   #      eager_load(:orders) {
   #        eager_load(:customer, ->(q) { q.select("name") })
   #      }
-  #    }.
-  #    run
+  #    }
+  #    .run
   #
   # @param scope [ActiveRecord::Relation]
   # @param use [Module] optional Module to include in the result class
@@ -36,7 +34,7 @@ module OccamsRecord
     # @return [ActiveRecord::Relation] scope for building the main SQL query
     attr_reader :scope
 
-    include Batches
+    include OccamsRecord::Batches::CursorHelpers
     include EagerLoaders::Builder
     include Enumerable
     include Measureable
@@ -154,5 +152,75 @@ module OccamsRecord
         to_a.each
       end
     end
+
+    #
+    # Load records in batches of N and yield each record to a block if given. If no block is given,
+    # returns an Enumerator.
+    #
+    # NOTE Unlike ActiveRecord's find_each, ORDER BY is respected. The primary key will be appended
+    # to the ORDER BY clause to help ensure consistent batches. Additionally, it will be run inside
+    # of a transaction.
+    #
+    # @param batch_size [Integer]
+    # @param use_transaction [Boolean] Ensure it runs inside of a database transaction
+    # @param append_order_by [String] Append this column to ORDER BY to ensure consistent results. Defaults to the primary key. Pass false to disable.
+    # @yield [OccamsRecord::Results::Row]
+    # @return [Enumerator] will yield each record
+    #
+    def find_each(batch_size: 1000, use_transaction: true, append_order_by: nil)
+      enum = Enumerator.new { |y|
+        find_in_batches(batch_size: 1000, use_transaction: use_transaction, append_order_by: append_order_by).each { |batch|
+          batch.each { |record| y.yield record }
+        }
+      }
+      if block_given?
+        enum.each { |record| yield record }
+      else
+        enum
+      end
+    end
+
+    #
+    # Load records in batches of N and yield each batch to a block if given.
+    # If no block is given, returns an Enumerator.
+    #
+    # NOTE Unlike ActiveRecord's find_each, ORDER BY is respected. The primary key will be appended
+    # to the ORDER BY clause to help ensure consistent batches. Additionally, it will be run inside
+    # of a transaction.
+    #
+    # @param batch_size [Integer]
+    # @param use_transaction [Boolean] Ensure it runs inside of a database transaction
+    # @param append_order_by [String] Append this column to ORDER BY to ensure consistent results. Defaults to the primary key. Pass false to disable.
+    # @yield [OccamsRecord::Results::Row]
+    # @return [Enumerator] will yield each batch
+    #
+    def find_in_batches(batch_size: 1000, use_transaction: true, append_order_by: nil)
+      enum = Batches::OffsetLimit::Scoped
+        .new(model, scope, use: @use, query_logger: @query_logger, eager_loaders: @eager_loaders)
+        .enum(batch_size: batch_size, use_transaction: use_transaction, append_order_by: append_order_by)
+      if block_given?
+        enum.each { |batch| yield batch }
+      else
+        enum
+      end
+    end
+
+    #
+    # Returns a cursor you can open and perform operations on. A lower-level alternative to 
+    # find_each_with_cursor and find_in_batches_with_cursor.
+    #
+    # NOTE Postgres only. See the docs for OccamsRecord::Cursor for more details.
+    #
+    # @param name [String] Specify a name for the cursor (defaults to a random name)
+    # @param scroll [Boolean] true = SCROLL, false = NO SCROLL, nil = default behavior of DB
+    # @param hold [Boolean] true = WITH HOLD, false = WITHOUT HOLD, nil = default behavior of DB
+    # @return [OccamsRecord::Cursor]
+    #
+    def cursor(name: nil, scroll: nil, hold: nil)
+      Cursor.new(model.connection, scope.to_sql,
+        name: name, scroll: scroll, hold: hold,
+        use: @use, query_logger: @query_logger, eager_loaders: @eager_loaders,
+      )
+    end
   end
 end

data/lib/occams-record/raw_query.rb

@@ -14,16 +14,16 @@ module OccamsRecord
   # If you want to do eager loading, you must first the define a model to pull the associations from (unless
   # you're using the raw SQL eager loaders `eager_load_one` or `eager_load_many`).
   #
-  #   results = OccamsRecord.
-  #     sql("
+  #   results = OccamsRecord
+  #     .sql("
   #       SELECT * FROM widgets
   #       WHERE category_id IN (%{cat_ids})
   #     ", {
   #       cat_ids: [5, 10]
-  #     }).
-  #     model(Widget).
-  #     eager_load(:category).
-  #     run
+  #     })
+  #     .model(Widget)
+  #     .eager_load(:category)
+  #     .run
   #
   # NOTE To use find_each/find_in_batches, your SQL string must include 'LIMIT %{batch_limit} OFFSET %{batch_offset}',
   # and an ORDER BY is strongly recomended. OccamsRecord will provide the bind values for you.
@@ -61,7 +61,7 @@ module OccamsRecord
     # @return [Hash]
     attr_reader :binds
 
-    include Batches
+    include OccamsRecord::Batches::CursorHelpers
     include EagerLoaders::Builder
     include Enumerable
     include Measureable
@@ -75,13 +75,15 @@ module OccamsRecord
     # @param eager_loaders [OccamsRecord::EagerLoaders::Context]
     # @param query_logger [Array] (optional) an array into which all queries will be inserted for logging/debug purposes
     # @param measurements [Array]
+    # @param connection
     #
-    def initialize(sql, binds, use: nil, eager_loaders: nil, query_logger: nil, measurements: nil)
+    def initialize(sql, binds, use: nil, eager_loaders: nil, query_logger: nil, measurements: nil, connection: nil)
       @sql = sql
       @binds = binds
       @use = use
       @eager_loaders = eager_loaders || EagerLoaders::Context.new
       @query_logger, @measurements = query_logger, measurements
+      @conn = connection
     end
 
     #
@@ -139,6 +141,74 @@ module OccamsRecord
       end
     end
 
+    #
+    # Load records in batches of N and yield each record to a block if given. If no block is given,
+    # returns an Enumerator.
+    #
+    # NOTE Unlike ActiveRecord's find_each, ORDER BY is respected. The primary key will be appended
+    # to the ORDER BY clause to help ensure consistent batches. Additionally, it will be run inside
+    # of a transaction.
+    #
+    # @param batch_size [Integer]
+    # @param use_transaction [Boolean] Ensure it runs inside of a database transaction
+    # @yield [OccamsRecord::Results::Row]
+    # @return [Enumerator] will yield each record
+    #
+    def find_each(batch_size: 1000, use_transaction: true)
+      enum = Enumerator.new { |y|
+        find_in_batches(batch_size: batch_size, use_transaction: use_transaction).each { |batch|
+          batch.each { |record| y.yield record }
+        }
+      }
+      if block_given?
+        enum.each { |record| yield record }
+      else
+        enum
+      end
+    end
+
+    #
+    # Load records in batches of N and yield each batch to a block if given.
+    # If no block is given, returns an Enumerator.
+    #
+    # NOTE Unlike ActiveRecord's find_each, ORDER BY is respected. The primary key will be appended
+    # to the ORDER BY clause to help ensure consistent batches. Additionally, it will be run inside
+    # of a transaction.
+    #
+    # @param batch_size [Integer]
+    # @param use_transaction [Boolean] Ensure it runs inside of a database transaction
+    # @yield [OccamsRecord::Results::Row]
+    # @return [Enumerator] will yield each batch
+    #
+    def find_in_batches(batch_size: 1000, use_transaction: true)
+      enum = Batches::OffsetLimit::RawQuery
+        .new(conn, @sql, @binds, use: @use, query_logger: @query_logger, eager_loaders: @eager_loaders)
+        .enum(batch_size: batch_size, use_transaction: use_transaction)
+      if block_given?
+        enum.each { |batch| yield batch }
+      else
+        enum
+      end
+    end
+
+    #
+    # Returns a cursor you can open and perform operations on. A lower-level alternative to 
+    # find_each_with_cursor and find_in_batches_with_cursor.
+    #
+    # NOTE Postgres only. See the docs for OccamsRecord::Cursor for more details.
+    #
+    # @param name [String] Specify a name for the cursor (defaults to a random name)
+    # @param scroll [Boolean] true = SCROLL, false = NO SCROLL, nil = default behavior of DB
+    # @param hold [Boolean] true = WITH HOLD, false = WITHOUT HOLD, nil = default behavior of DB
+    # @return [OccamsRecord::Cursor]
+    #
+    def cursor(name: nil, scroll: nil, hold: nil)
+      Cursor.new(conn, @sql,
+        name: name, scroll: scroll, hold: hold,
+        use: @use, query_logger: @query_logger, eager_loaders: @eager_loaders,
+      )
+    end
+
     private
 
     # Returns the SQL as a String with all variables escaped
@@ -158,45 +228,6 @@ module OccamsRecord
       @sql.match(/\s+FROM\s+"?(\w+)"?/i)&.captures&.first
     end
 
-    #
-    # Returns an Enumerator that yields batches of records, of size "of".
-    # The SQL string must include 'LIMIT %{batch_limit} OFFSET %{batch_offset}'.
-    # The bind values will be provided by OccamsRecord.
-    #
-    # @param of [Integer] batch size
-    # @param use_transaction [Boolean] Ensure it runs inside of a database transaction
-    # @return [Enumerator] yields batches
-    #
-    def batches(of:, use_transaction: true, append_order_by: nil)
-      unless @sql =~ /LIMIT\s+%\{batch_limit\}/i and @sql =~ /OFFSET\s+%\{batch_offset\}/i
-        raise ArgumentError, "When using find_each/find_in_batches you must specify 'LIMIT %{batch_limit} OFFSET %{batch_offset}'. SQL statement: #{@sql}"
-      end
-
-      Enumerator.new do |y|
-        if use_transaction and conn.open_transactions == 0
-          conn.transaction {
-            run_batches y, of
-          }
-        else
-          run_batches y, of
-        end
-      end
-    end
-
-    def run_batches(y, of)
-      offset = 0
-      loop do
-        results = RawQuery.new(@sql, @binds.merge({
-          batch_limit: of,
-          batch_offset: offset,
-        }), use: @use, query_logger: @query_logger, eager_loaders: @eager_loaders).run
-
-        y.yield results if results.any?
-        break if results.size < of
-        offset += results.size
-      end
-    end
-
     def conn
       @conn ||= @eager_loaders.model&.connection || ActiveRecord::Base.connection
     end

data/lib/occams-record/results/results.rb

@@ -4,7 +4,7 @@ module OccamsRecord
     # ActiveRecord's internal type casting API changes from version to version.
     CASTER = case ActiveRecord::VERSION::MAJOR
              when 4 then :type_cast_from_database
-             when 5, 6 then :deserialize
+             when 5, 6, 7 then :deserialize
              else raise "OccamsRecord::Results::CASTER does yet support this version of ActiveRecord"
              end
 

data/lib/occams-record/ugly.rb

@@ -8,10 +8,10 @@ module OccamsRecord
     # The ONLY reason to use this is if you absolutely need ActiveRecord objects but still want to use Occams's
     # more advanced eager loading or find_each/find_in_batches features.
     #
-    #   OccamsRecord.
-    #     query(Order.order("created_at DESC")).
-    #     eager_load(:line_items, ->(q) { q.order("price") }).
-    #     find_each do |o|
+    #   OccamsRecord
+    #     .query(Order.order("created_at DESC"))
+    #     .eager_load(:line_items, ->(q) { q.order("price") })
+    #     .find_each do |o|
     #       order = OccamsRecord::Ugly.active_record(o)
     #       ...
     #     end

data/lib/occams-record/version.rb

@@ -2,6 +2,6 @@
 # Main entry point for using OccamsRecord.
 #
 module OccamsRecord
-  # Library version
-  VERSION = "1.2.1".freeze
+  # @private
+  VERSION = "1.4.0-beta1".freeze
 end

data/lib/occams-record.rb

@@ -5,9 +5,15 @@ require 'occams-record/measureable'
 require 'occams-record/eager_loaders/eager_loaders'
 require 'occams-record/results/results'
 require 'occams-record/results/row'
+require 'occams-record/cursor'
+require 'occams-record/errors'
+
+require 'occams-record/batches/offset_limit/scoped'
+require 'occams-record/batches/offset_limit/raw_query'
+require 'occams-record/batches/cursor_helpers'
+
 require 'occams-record/query'
 require 'occams-record/raw_query'
-require 'occams-record/errors'
 
 module OccamsRecord
   autoload :Ugly, 'occams-record/ugly'

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: occams-record
 version: !ruby/object:Gem::Version
-  version: 1.2.1
+  version: 1.4.0.pre.beta1
 platform: ruby
 authors:
 - Jordan Hollinger
-autorequire: 
+autorequire:
 bindir: bin
 cert_chain: []
-date: 2021-07-04 00:00:00.000000000 Z
+date: 2022-05-22 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: activerecord
@@ -19,7 +19,7 @@ dependencies:
         version: '4.2'
     - - "<"
       - !ruby/object:Gem::Version
-        version: '6.2'
+        version: '7.1'
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
@@ -29,7 +29,7 @@ dependencies:
         version: '4.2'
     - - "<"
       - !ruby/object:Gem::Version
-        version: '6.2'
+        version: '7.1'
 - !ruby/object:Gem::Dependency
   name: appraisal
   requirement: !ruby/object:Gem::Requirement
@@ -44,8 +44,8 @@ dependencies:
     - - ">="
       - !ruby/object:Gem::Version
         version: '0'
-description: A faster, lower-memory querying API for ActiveRecord that returns results
-  as unadorned, read-only objects.
+description: A faster, lower-memory, fuller-featured querying API for ActiveRecord
+  that returns results as unadorned, read-only objects.
 email: jordan.hollinger@gmail.com
 executables: []
 extensions: []
@@ -53,8 +53,10 @@ extra_rdoc_files: []
 files:
 - README.md
 - lib/occams-record.rb
-- lib/occams-record/batches.rb
-- lib/occams-record/connection.rb
+- lib/occams-record/batches/cursor_helpers.rb
+- lib/occams-record/batches/offset_limit/raw_query.rb
+- lib/occams-record/batches/offset_limit/scoped.rb
+- lib/occams-record/cursor.rb
 - lib/occams-record/eager_loaders/ad_hoc_base.rb
 - lib/occams-record/eager_loaders/ad_hoc_many.rb
 - lib/occams-record/eager_loaders/ad_hoc_one.rb
@@ -81,7 +83,7 @@ homepage: https://jhollinger.github.io/occams-record/
 licenses:
 - MIT
 metadata: {}
-post_install_message: 
+post_install_message:
 rdoc_options: []
 require_paths:
 - lib
@@ -92,12 +94,12 @@ required_ruby_version: !ruby/object:Gem::Requirement
       version: 2.3.0
 required_rubygems_version: !ruby/object:Gem::Requirement
   requirements:
-  - - ">="
+  - - ">"
     - !ruby/object:Gem::Version
-      version: '0'
+      version: 1.3.1
 requirements: []
-rubygems_version: 3.0.3
-signing_key: 
+rubygems_version: 3.1.6
+signing_key:
 specification_version: 4
 summary: The missing high-efficiency query API for ActiveRecord
 test_files: []

data/lib/occams-record/batches.rb

@@ -1,113 +0,0 @@
-module OccamsRecord
-  #
-  # Methods for building batch finding methods. It expects "model" and "scope" methods to be present.
-  #
-  module Batches
-    #
-    # Load records in batches of N and yield each record to a block if given. If no block is given,
-    # returns an Enumerator.
-    #
-    # NOTE Unlike ActiveRecord's find_each, ORDER BY is respected. The primary key will be appended
-    # to the ORDER BY clause to help ensure consistent batches. Additionally, it will be run inside
-    # of a transaction.
-    #
-    # @param batch_size [Integer]
-    # @param use_transaction [Boolean] Ensure it runs inside of a database transaction
-    # @param append_order_by [String] Append this column to ORDER BY to ensure consistent results. Defaults to the primary key. Pass false to disable.
-    # @yield [OccamsRecord::Results::Row]
-    # @return [Enumerator] will yield each record
-    #
-    def find_each(batch_size: 1000, use_transaction: true, append_order_by: nil)
-      enum = Enumerator.new { |y|
-        batches(of: batch_size, use_transaction: use_transaction, append_order_by: append_order_by).each { |batch|
-          batch.each { |record| y.yield record }
-        }
-      }
-      if block_given?
-        enum.each { |record| yield record }
-      else
-        enum
-      end
-    end
-
-    #
-    # Load records in batches of N and yield each batch to a block if given.
-    # If no block is given, returns an Enumerator.
-    #
-    # NOTE Unlike ActiveRecord's find_each, ORDER BY is respected. The primary key will be appended
-    # to the ORDER BY clause to help ensure consistent batches. Additionally, it will be run inside
-    # of a transaction.
-    #
-    # @param batch_size [Integer]
-    # @param use_transaction [Boolean] Ensure it runs inside of a database transaction
-    # @param append_order_by [String] Append this column to ORDER BY to ensure consistent results. Defaults to the primary key. Pass false to disable.
-    # @yield [OccamsRecord::Results::Row]
-    # @return [Enumerator] will yield each batch
-    #
-    def find_in_batches(batch_size: 1000, use_transaction: true, append_order_by: nil)
-      enum = batches(of: batch_size, use_transaction: use_transaction, append_order_by: append_order_by)
-      if block_given?
-        enum.each { |batch| yield batch }
-      else
-        enum
-      end
-    end
-
-    private
-
-    #
-    # Returns an Enumerator that yields batches of records, of size "of".
-    # NOTE ActiveRecord 5+ provides the 'in_batches' method to do something
-    # similiar, although 4.2 does not. Also it does not respect ORDER BY,
-    # whereas this does.
-    #
-    # @param of [Integer] batch size
-    # @param use_transaction [Boolean] Ensure it runs inside of a database transaction
-    # @param append_order_by [String] Append this column to ORDER BY to ensure consistent results. Defaults to the primary key. Pass false to disable.
-    # @return [Enumerator] yields batches
-    #
-    def batches(of:, use_transaction: true, append_order_by: nil)
-      append_order =
-        case append_order_by
-        when false then nil
-        when nil then model.primary_key
-        else append_order_by
-        end
-
-      Enumerator.new do |y|
-        if use_transaction and model.connection.open_transactions == 0
-          model.connection.transaction {
-            run_batches y, of, append_order
-          }
-        else
-          run_batches y, of, append_order
-        end
-      end
-    end
-
-    def run_batches(y, of, append_order_by = nil)
-      limit = scope.limit_value
-      batch_size = limit && limit < of ? limit : of
-
-      offset = scope.offset_value || 0
-      out_of_records, count = false, 0
-      order_by =
-        if append_order_by
-          append_order_by.to_s == model.primary_key.to_s ? append_order_by.to_sym : append_order_by
-        end
-
-      until out_of_records
-        l = limit && batch_size > limit - count ? limit - count : batch_size
-        q = scope
-        q = q.order(order_by) if order_by
-        q = q.offset(offset).limit(l)
-        results = Query.new(q, use: @use, query_logger: @query_logger, eager_loaders: @eager_loaders).run
-
-        y.yield results if results.any?
-        count += results.size
-        offset += results.size
-        out_of_records = results.size < batch_size || (limit && count >= limit)
-      end
-    end
-  end
-end

data/lib/occams-record/connection.rb

@@ -1,7 +0,0 @@
-module OccamsRecord
-  class Connection
-    def initialize(model, role = nil)
-      @model, @role = model, role
-    end
-  end
-end