simple_query 0.3.2 → 0.4.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 6aeb71d493d14743e43b8c9a37eed697f131db105ffc4dc0fa35dd0dd189255b
-  data.tar.gz: 82b1815e569f24a4c4da9e4cd727d95ad6c938403bd2b781eccfc39398cb264b
+  metadata.gz: 8a62c7c2fe2d6c43a27b9cc2dd22e7ddcf14a01f9ca335144b472db1559f4b0a
+  data.tar.gz: 172eddfa79634dcf78d3547031af80a9f1e008dc9e0414c0f5671c66b4264617
 SHA512:
-  metadata.gz: d422a6a3484d4242b6666bb6536f4828b666c0f16dd6a0c42104b12e3f077bd30e580dc9cf5eb4292ecbb1cd0eda218aabe29af6e221a86c3291fc116b62b856
-  data.tar.gz: cf6e7c8a7e198d1b5376053237d7bdfac59819e07571adb5656e859de9a3bb840dace69e2e600703b14a5ff01bfedbf448d1aadb471a3875874c7d390599878a
+  metadata.gz: abbd8b5034e7b3a9a72b5de79f0d522a1ba2ac2b066bcc78387fb068ad9647b685c006318f6efafd31b5b616b48bd18ea677a5c74e7c25647a38c062a934109b
+  data.tar.gz: e499597efd33f703d136b0e053006ef109dea762e19148acd6bb5dcf6717f14f8b63845d61a3ea7b770353dd37e17178ca8b51ef7785e0714828ee78bda1fb8e

data/README.md

@@ -176,32 +176,70 @@ Each scope block (e.g. by_name) is evaluated in the context of the SimpleQuery b
 Parameterized scopes accept arguments — passed directly to the block (e.g. |name| above).
 Scopes return self, so you can chain multiple scopes or mix them with standard query methods.
 
-## Features
-
-- Efficient query building
-- Support for complex joins
-- Lazy execution
-- DISTINCT queries
-- Aggregations
-- LIMIT and OFFSET
-- ORDER BY clause
-- Having and Grouping
-- Subqueries
-- Custom Read models
-- Named Scopes
+## Streaming Large Datasets
+
+For massive queries (millions of rows), **SimpleQuery** offers a `.stream_each` method to avoid loading the entire result set into memory. It **automatically** picks a streaming approach depending on your database adapter:
+
+- **PostgreSQL**: Uses a **server-side cursor** via `DECLARE ... FETCH`.
+- **MySQL**: Uses `mysql2` gem’s **streaming** (`stream: true, cache_rows: false, as: :hash`).
+
+```ruby
+# Example usage:
+User.simple_query
+    .where(active: true)
+    .stream_each(batch_size: 10_000) do |row|
+  # row is a struct or read-model instance
+  puts row.name
+end
+```
 
 ## Performance
 
-SimpleQuery is designed to potentially outperform standard ActiveRecord queries on large datasets. In our benchmarks with 100,000 records, SimpleQuery showed improved performance compared to equivalent ActiveRecord queries.
+SimpleQuery aims to outperform standard ActiveRecord queries at scale. We’ve benchmarked **1,000,000** records on **both PostgreSQL** and **MySQL**, with the following results:
+
+### PostgreSQL (1,000,000 records)
+```
+🚀 Performance Results (1000,000 records):
+ActiveRecord Query:                  10.36932 seconds
+SimpleQuery Execution (Struct):      3.46136 seconds
+SimpleQuery Execution (Read model):  2.20905 seconds
+
+----------------------------------------------------
+ActiveRecord find_each:              6.10077 seconds
+SimpleQuery stream_each:             2.75639 seconds
+
+--- AR find_each Memory Report ---
+Total allocated: 1.98 GB (16,001,659 objects)
+Retained:        ~2 KB
+
+--- SimpleQuery stream_each Memory Report ---
+Total allocated: 1.38 GB (8,000,211 objects)
+Retained:        ~3 KB
+```
+- **Struct-based** approach remains the fastest, skipping model overhead.
+- **Read model** approach is still significantly faster than standard ActiveRecord while allowing domain-specific logic.
 
+### MySQL (1,000,000 records)
 ```
-🚀 Performance Results (100,000 records):
-ActiveRecord Query:                  0.47441 seconds
-SimpleQuery Execution (Struct):      0.05346 seconds
-SimpleQuery Execution (Read model):  0.14408 seconds
+🚀 Performance Results (1000,000 records):
+ActiveRecord Query:                  10.45833 seconds
+SimpleQuery Execution (Struct):      3.04655 seconds
+SimpleQuery Execution (Read model):  3.69052 seconds
+
+----------------------------------------------------
+ActiveRecord find_each:              5.04671 seconds
+SimpleQuery stream_each:             2.96602 seconds
+
+--- AR find_each Memory Report ---
+Total allocated: 1.32 GB (11,001,445 objects)
+Retained:        ~2.7 KB
+
+--- SimpleQuery stream_each Memory Report ---
+Total allocated: 1.22 GB (8,000,068 objects)
+Retained:        ~3.9 KB
 ```
-- The **Struct-based** approach is the fastest. 
-- The **Read model** approach is still significantly faster than ActiveRecord, while letting you define custom logic or domain-specific attributes.
+- Even in MySQL, **Struct** was roughly **three times faster** than ActiveRecord’s overhead.
+- Read models still outperform AR, though by a narrower margin in this scenario.
 
 ## Development
 

data/lib/simple_query/builder.rb

@@ -2,6 +2,9 @@
 
 module SimpleQuery
   class Builder
+    include SimpleQuery::Stream::PostgresStream
+    include SimpleQuery::Stream::MysqlStream
+
     attr_reader :model, :arel_table
 
     def initialize(source)
@@ -94,6 +97,30 @@ module SimpleQuery
       self
     end
 
+    def bulk_update(set:)
+      table_name = @arel_table.name
+      set_sql = SetClause.new(set).to_sql
+
+      raise ArgumentError, "No columns to update" if set_sql.empty?
+
+      where_sql = build_where_sql
+      sql = "UPDATE #{table_name} SET #{set_sql}"
+      sql += " WHERE #{where_sql}" unless where_sql.nil? || where_sql.empty?
+
+      ActiveRecord::Base.connection.execute(sql)
+    end
+
+    def stream_each(batch_size: 1000, &block)
+      adapter = ActiveRecord::Base.connection.adapter_name.downcase
+      if adapter.include?("postgres")
+        stream_each_postgres(batch_size, &block)
+      elsif adapter.include?("mysql")
+        stream_each_mysql(&block)
+      else
+        raise "stream_each is only implemented for Postgres and MySQL."
+      end
+    end
+
     def execute
       records = ActiveRecord::Base.connection.select_all(cached_sql)
       build_result_objects_from_rows(records)
@@ -133,6 +160,13 @@ module SimpleQuery
 
     private
 
+    def build_where_sql
+      condition = @wheres.to_arel
+      return "" unless condition
+
+      condition.to_sql
+    end
+
     def reset_query
       @query_built = false
       @query_cache.clear

data/lib/simple_query/clauses/set_clause.rb

@@ -0,0 +1,25 @@
+# frozen_string_literal: true
+
+module SimpleQuery
+  class SetClause
+    def initialize(set_hash)
+      @set_hash = set_hash
+    end
+
+    def to_sql
+      @set_hash.map do |col, val|
+        "#{quote_column(col)} = #{quote_value(val)}"
+      end.join(", ")
+    end
+
+    private
+
+    def quote_column(col)
+      ActiveRecord::Base.connection.quote_column_name(col)
+    end
+
+    def quote_value(val)
+      ActiveRecord::Base.connection.quote(val)
+    end
+  end
+end

data/lib/simple_query/stream/mysql_stream.rb

@@ -0,0 +1,36 @@
+# frozen_string_literal: true
+
+module SimpleQuery
+  module Stream
+    module MysqlStream
+      def stream_each_mysql(&block)
+        select_sql = cached_sql
+
+        raw_conn = ActiveRecord::Base.connection.raw_connection
+
+        result = raw_conn.query(select_sql, stream: true, cache_rows: false, as: :hash)
+        result.each do |mysql_row|
+          record = build_row_object_mysql(mysql_row)
+          block.call(record)
+        end
+      end
+
+      private
+
+      def build_row_object_mysql(mysql_row)
+        if @read_model_class
+          obj = @read_model_class.allocate
+          @read_model_class.attributes.each do |attr_name, col_name|
+            obj.instance_variable_set(:"@#{attr_name}", mysql_row[col_name])
+          end
+          obj
+        else
+          columns = mysql_row.keys
+          values = columns.map { |k| mysql_row[k] }
+          struct = result_struct(columns)
+          struct.new(*values)
+        end
+      end
+    end
+  end
+end

data/lib/simple_query/stream/postgres_stream.rb

@@ -0,0 +1,59 @@
+# frozen_string_literal: true
+
+module SimpleQuery
+  module Stream
+    module PostgresStream
+      # rubocop:disable Metrics/MethodLength
+      def stream_each_postgres(batch_size, &block)
+        select_sql = cached_sql
+
+        conn = ActiveRecord::Base.connection.raw_connection
+        cursor_name = "simple_query_cursor_#{object_id}"
+
+        begin
+          conn.exec("BEGIN")
+          declare_sql = "DECLARE #{cursor_name} NO SCROLL CURSOR FOR #{select_sql}"
+          conn.exec(declare_sql)
+
+          loop do
+            res = conn.exec("FETCH #{batch_size} FROM #{cursor_name}")
+            break if res.ntuples.zero?
+
+            res.each do |pg_row|
+              record = build_row_object(pg_row)
+              block.call(record)
+            end
+          end
+
+          conn.exec("CLOSE #{cursor_name}")
+          conn.exec("COMMIT")
+        rescue StandardError => e
+          begin
+            conn.exec("ROLLBACK")
+          rescue StandardError
+            nil
+          end
+          raise e
+        end
+      end
+      # rubocop:enable Metrics/MethodLength
+
+      private
+
+      def build_row_object(pg_row)
+        if @read_model_class
+          obj = @read_model_class.allocate
+          @read_model_class.attributes.each do |attr_name, col_name|
+            obj.instance_variable_set(:"@#{attr_name}", pg_row[col_name])
+          end
+          obj
+        else
+          columns = pg_row.keys
+          values = columns.map { |k| pg_row[k] }
+          struct = result_struct(columns)
+          struct.new(*values)
+        end
+      end
+    end
+  end
+end

data/lib/simple_query/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module SimpleQuery
-  VERSION = "0.3.2"
+  VERSION = "0.4.0"
 end

data/lib/simple_query.rb

@@ -3,6 +3,8 @@
 require "active_support/concern"
 require "active_record"
 
+require_relative "simple_query/stream/mysql_stream"
+require_relative "simple_query/stream/postgres_stream"
 require_relative "simple_query/builder"
 require_relative "simple_query/read_model"
 require_relative "simple_query/clauses/where_clause"
@@ -11,6 +13,7 @@ require_relative "simple_query/clauses/order_clause"
 require_relative "simple_query/clauses/distinct_clause"
 require_relative "simple_query/clauses/limit_offset_clause"
 require_relative "simple_query/clauses/group_having_clause"
+require_relative "simple_query/clauses/set_clause"
 
 module SimpleQuery
   extend ActiveSupport::Concern

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: simple_query
 version: !ruby/object:Gem::Version
-  version: 0.3.2
+  version: 0.4.0
 platform: ruby
 authors:
 - Alex Kholodniak
 autorequire:
 bindir: bin
 cert_chain: []
-date: 2025-03-01 00:00:00.000000000 Z
+date: 2025-03-15 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: activerecord
@@ -30,62 +30,6 @@ dependencies:
     - - "<="
       - !ruby/object:Gem::Version
         version: '8.0'
-- !ruby/object:Gem::Dependency
-  name: rake
-  requirement: !ruby/object:Gem::Requirement
-    requirements:
-    - - "~>"
-      - !ruby/object:Gem::Version
-        version: '13.0'
-  type: :development
-  prerelease: false
-  version_requirements: !ruby/object:Gem::Requirement
-    requirements:
-    - - "~>"
-      - !ruby/object:Gem::Version
-        version: '13.0'
-- !ruby/object:Gem::Dependency
-  name: rspec
-  requirement: !ruby/object:Gem::Requirement
-    requirements:
-    - - "~>"
-      - !ruby/object:Gem::Version
-        version: '3.0'
-  type: :development
-  prerelease: false
-  version_requirements: !ruby/object:Gem::Requirement
-    requirements:
-    - - "~>"
-      - !ruby/object:Gem::Version
-        version: '3.0'
-- !ruby/object:Gem::Dependency
-  name: rubocop
-  requirement: !ruby/object:Gem::Requirement
-    requirements:
-    - - "~>"
-      - !ruby/object:Gem::Version
-        version: '1.21'
-  type: :development
-  prerelease: false
-  version_requirements: !ruby/object:Gem::Requirement
-    requirements:
-    - - "~>"
-      - !ruby/object:Gem::Version
-        version: '1.21'
-- !ruby/object:Gem::Dependency
-  name: sqlite3
-  requirement: !ruby/object:Gem::Requirement
-    requirements:
-    - - "~>"
-      - !ruby/object:Gem::Version
-        version: 1.5.0
-  type: :development
-  prerelease: false
-  version_requirements: !ruby/object:Gem::Requirement
-    requirements:
-    - - "~>"
-      - !ruby/object:Gem::Version
-        version: 1.5.0
 description: SimpleQuery provides a flexible and performant way to construct complex
   database queries in Ruby on Rails applications. It offers an intuitive interface
   for building queries with joins, conditions, and aggregations, while potentially
@@ -105,8 +49,11 @@ files:
 - lib/simple_query/clauses/join_clause.rb
 - lib/simple_query/clauses/limit_offset_clause.rb
 - lib/simple_query/clauses/order_clause.rb
+- lib/simple_query/clauses/set_clause.rb
 - lib/simple_query/clauses/where_clause.rb
 - lib/simple_query/read_model.rb
+- lib/simple_query/stream/mysql_stream.rb
+- lib/simple_query/stream/postgres_stream.rb
 - lib/simple_query/version.rb
 homepage: https://github.com/kholdrex/simple_query
 licenses:
@@ -133,5 +80,6 @@ requirements: []
 rubygems_version: 3.5.9
 signing_key:
 specification_version: 4
-summary: A lightweight and efficient query builder for ActiveRecord.
+summary: A lightweight, multi-DB-friendly, and high-performance query builder for
+  ActiveRecord, featuring streaming, bulk updates, and read-model support.
 test_files: []