simple_query 0.3.1 → 0.4.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: fa6c4c630a5cd486970ce4ca43b444f082050924305cb20e835a861b32676275
-  data.tar.gz: de8f1cc3758f4a65edbc4a1d4bbe1c5edde866d6016332d5b415c35d97423e7e
+  metadata.gz: 8a62c7c2fe2d6c43a27b9cc2dd22e7ddcf14a01f9ca335144b472db1559f4b0a
+  data.tar.gz: 172eddfa79634dcf78d3547031af80a9f1e008dc9e0414c0f5671c66b4264617
 SHA512:
-  metadata.gz: c9fb69f0d5505926239e5dcc396dedb098f974b38a0a1d135ddeb50927de9b88b8dc0f0fb850a2ab2fe6b9e7ba9a4955738c36d4a1c0f974e507740fa00edb15
-  data.tar.gz: e8acc149f25a8b898a04ec7ca5203f98898d88bff334700ff5e4b6e0bb538b56fb47edd0b92cb66fbd86ac89c3f98d7c48930f25e9870bb34208b0bbd2735c4f
+  metadata.gz: abbd8b5034e7b3a9a72b5de79f0d522a1ba2ac2b066bcc78387fb068ad9647b685c006318f6efafd31b5b616b48bd18ea677a5c74e7c25647a38c062a934109b
+  data.tar.gz: e499597efd33f703d136b0e053006ef109dea762e19148acd6bb5dcf6717f14f8b63845d61a3ea7b770353dd37e17178ca8b51ef7785e0714828ee78bda1fb8e

data/README.md

@@ -54,11 +54,12 @@ User.simple_query.select(:name, :email).where(active: true).execute
 ```
 
 Query with join
+
+SimpleQuery now supports **all major SQL join types** — including LEFT, RIGHT, and FULL — through the following DSL methods:
 ```ruby
 User.simple_query
-    .select(:name, :email)
-    .join(:users, :companies, foreign_key: :user_id, primary_key: :id)
-    .where(Company.arel_table[:name].eq("TechCorp"))
+    .left_join(:users, :companies, foreign_key: :user_id, primary_key: :id)
+    .select("users.name", "companies.name")
     .execute
 ```
 
@@ -143,9 +144,13 @@ class User < ActiveRecord::Base
     where(admin: true)
   end
 
+  # Block-based scope with parameter
   simple_scope :by_name do |name|
     where(name: name)
   end
+
+  # Lambda-based scope with parameter
+  simple_scope :by_name, ->(name) { where(name: name) }
 end
 ```
 You can then chain these scopes seamlessly with the normal SimpleQuery DSL:
@@ -171,32 +176,70 @@ Each scope block (e.g. by_name) is evaluated in the context of the SimpleQuery b
 Parameterized scopes accept arguments — passed directly to the block (e.g. |name| above).
 Scopes return self, so you can chain multiple scopes or mix them with standard query methods.
 
-## Features
+## Streaming Large Datasets
+
+For massive queries (millions of rows), **SimpleQuery** offers a `.stream_each` method to avoid loading the entire result set into memory. It **automatically** picks a streaming approach depending on your database adapter:
 
-- Efficient query building
-- Support for complex joins
-- Lazy execution
-- DISTINCT queries
-- Aggregations
-- LIMIT and OFFSET
-- ORDER BY clause
-- Having and Grouping
-- Subqueries
-- Custom Read models
-- Named Scopes
+- **PostgreSQL**: Uses a **server-side cursor** via `DECLARE ... FETCH`.
+- **MySQL**: Uses `mysql2` gem’s **streaming** (`stream: true, cache_rows: false, as: :hash`).
+
+```ruby
+# Example usage:
+User.simple_query
+    .where(active: true)
+    .stream_each(batch_size: 10_000) do |row|
+  # row is a struct or read-model instance
+  puts row.name
+end
+```
 
 ## Performance
 
-SimpleQuery is designed to potentially outperform standard ActiveRecord queries on large datasets. In our benchmarks with 100,000 records, SimpleQuery showed improved performance compared to equivalent ActiveRecord queries.
+SimpleQuery aims to outperform standard ActiveRecord queries at scale. We’ve benchmarked **1,000,000** records on **both PostgreSQL** and **MySQL**, with the following results:
+
+### PostgreSQL (1,000,000 records)
+```
+🚀 Performance Results (1000,000 records):
+ActiveRecord Query:                  10.36932 seconds
+SimpleQuery Execution (Struct):      3.46136 seconds
+SimpleQuery Execution (Read model):  2.20905 seconds
+
+----------------------------------------------------
+ActiveRecord find_each:              6.10077 seconds
+SimpleQuery stream_each:             2.75639 seconds
+
+--- AR find_each Memory Report ---
+Total allocated: 1.98 GB (16,001,659 objects)
+Retained:        ~2 KB
+
+--- SimpleQuery stream_each Memory Report ---
+Total allocated: 1.38 GB (8,000,211 objects)
+Retained:        ~3 KB
+```
+- **Struct-based** approach remains the fastest, skipping model overhead.
+- **Read model** approach is still significantly faster than standard ActiveRecord while allowing domain-specific logic.
 
+### MySQL (1,000,000 records)
 ```
-🚀 Performance Results (100,000 records):
-ActiveRecord Query:                  0.47441 seconds
-SimpleQuery Execution (Struct):      0.05346 seconds
-SimpleQuery Execution (Read model):  0.14408 seconds
+🚀 Performance Results (1000,000 records):
+ActiveRecord Query:                  10.45833 seconds
+SimpleQuery Execution (Struct):      3.04655 seconds
+SimpleQuery Execution (Read model):  3.69052 seconds
+
+----------------------------------------------------
+ActiveRecord find_each:              5.04671 seconds
+SimpleQuery stream_each:             2.96602 seconds
+
+--- AR find_each Memory Report ---
+Total allocated: 1.32 GB (11,001,445 objects)
+Retained:        ~2.7 KB
+
+--- SimpleQuery stream_each Memory Report ---
+Total allocated: 1.22 GB (8,000,068 objects)
+Retained:        ~3.9 KB
 ```
-- The **Struct-based** approach is the fastest. 
-- The **Read model** approach is still significantly faster than ActiveRecord, while letting you define custom logic or domain-specific attributes.
+- Even in MySQL, **Struct** was roughly **three times faster** than ActiveRecord’s overhead.
+- Read models still outperform AR, though by a narrower margin in this scenario.
 
 ## Development
 

data/lib/simple_query/builder.rb

@@ -2,6 +2,9 @@
 
 module SimpleQuery
   class Builder
+    include SimpleQuery::Stream::PostgresStream
+    include SimpleQuery::Stream::MysqlStream
+
     attr_reader :model, :arel_table
 
     def initialize(source)
@@ -34,12 +37,24 @@ module SimpleQuery
       self
     end
 
-    def join(table1, table2, foreign_key:, primary_key:)
-      @joins.add(table1, table2, foreign_key: foreign_key, primary_key: primary_key)
+    def join(table1, table2, foreign_key:, primary_key:, type: :inner)
+      @joins.add(table1, table2, foreign_key: foreign_key, primary_key: primary_key, join_type: type)
       reset_query
       self
     end
 
+    def left_join(table1, table2, foreign_key:, primary_key:)
+      join(table1, table2, foreign_key: foreign_key, primary_key: primary_key, type: :left)
+    end
+
+    def right_join(table1, table2, foreign_key:, primary_key:)
+      join(table1, table2, foreign_key: foreign_key, primary_key: primary_key, type: :right)
+    end
+
+    def full_join(table1, table2, foreign_key:, primary_key:)
+      join(table1, table2, foreign_key: foreign_key, primary_key: primary_key, type: :full)
+    end
+
     def order(order_conditions)
       @orders.add(order_conditions)
       reset_query
@@ -82,6 +97,30 @@ module SimpleQuery
       self
     end
 
+    def bulk_update(set:)
+      table_name = @arel_table.name
+      set_sql = SetClause.new(set).to_sql
+
+      raise ArgumentError, "No columns to update" if set_sql.empty?
+
+      where_sql = build_where_sql
+      sql = "UPDATE #{table_name} SET #{set_sql}"
+      sql += " WHERE #{where_sql}" unless where_sql.nil? || where_sql.empty?
+
+      ActiveRecord::Base.connection.execute(sql)
+    end
+
+    def stream_each(batch_size: 1000, &block)
+      adapter = ActiveRecord::Base.connection.adapter_name.downcase
+      if adapter.include?("postgres")
+        stream_each_postgres(batch_size, &block)
+      elsif adapter.include?("mysql")
+        stream_each_mysql(&block)
+      else
+        raise "stream_each is only implemented for Postgres and MySQL."
+      end
+    end
+
     def execute
       records = ActiveRecord::Base.connection.select_all(cached_sql)
       build_result_objects_from_rows(records)
@@ -121,6 +160,13 @@ module SimpleQuery
 
     private
 
+    def build_where_sql
+      condition = @wheres.to_arel
+      return "" unless condition
+
+      condition.to_sql
+    end
+
     def reset_query
       @query_built = false
       @query_cache.clear

data/lib/simple_query/clauses/join_clause.rb

@@ -8,20 +8,37 @@ module SimpleQuery
       @joins = []
     end
 
-    def add(table1, table2, foreign_key:, primary_key:)
+    def add(table1, table2, foreign_key:, primary_key:, join_type: :inner)
       @joins << {
         table1: to_arel_table(table1),
         table2: to_arel_table(table2),
         foreign_key: foreign_key,
-        primary_key: primary_key
+        primary_key: primary_key,
+        type: join_type
       }
     end
 
     def apply_to(query)
-      @joins.each do |join|
-        query.join(join[:table2])
-             .on(join[:table2][join[:foreign_key]]
-                   .eq(join[:table1][join[:primary_key]]))
+      @joins.each do |join_def|
+        table1 = join_def[:table1]
+        table2 = join_def[:table2]
+        fk = join_def[:foreign_key]
+        pk = join_def[:primary_key]
+        type = join_def[:type]
+
+        join_class = case type
+                     when :left
+                       Arel::Nodes::OuterJoin
+                     when :right
+                       Arel::Nodes::RightOuterJoin
+                     when :full
+                       Arel::Nodes::FullOuterJoin
+                     else
+                       Arel::Nodes::InnerJoin
+                     end
+
+        condition = table2[fk].eq(table1[pk])
+        query.join(table2, join_class).on(condition)
       end
       query
     end

data/lib/simple_query/clauses/set_clause.rb

@@ -0,0 +1,25 @@
+# frozen_string_literal: true
+
+module SimpleQuery
+  class SetClause
+    def initialize(set_hash)
+      @set_hash = set_hash
+    end
+
+    def to_sql
+      @set_hash.map do |col, val|
+        "#{quote_column(col)} = #{quote_value(val)}"
+      end.join(", ")
+    end
+
+    private
+
+    def quote_column(col)
+      ActiveRecord::Base.connection.quote_column_name(col)
+    end
+
+    def quote_value(val)
+      ActiveRecord::Base.connection.quote(val)
+    end
+  end
+end

data/lib/simple_query/stream/mysql_stream.rb

@@ -0,0 +1,36 @@
+# frozen_string_literal: true
+
+module SimpleQuery
+  module Stream
+    module MysqlStream
+      def stream_each_mysql(&block)
+        select_sql = cached_sql
+
+        raw_conn = ActiveRecord::Base.connection.raw_connection
+
+        result = raw_conn.query(select_sql, stream: true, cache_rows: false, as: :hash)
+        result.each do |mysql_row|
+          record = build_row_object_mysql(mysql_row)
+          block.call(record)
+        end
+      end
+
+      private
+
+      def build_row_object_mysql(mysql_row)
+        if @read_model_class
+          obj = @read_model_class.allocate
+          @read_model_class.attributes.each do |attr_name, col_name|
+            obj.instance_variable_set(:"@#{attr_name}", mysql_row[col_name])
+          end
+          obj
+        else
+          columns = mysql_row.keys
+          values = columns.map { |k| mysql_row[k] }
+          struct = result_struct(columns)
+          struct.new(*values)
+        end
+      end
+    end
+  end
+end

data/lib/simple_query/stream/postgres_stream.rb

@@ -0,0 +1,59 @@
+# frozen_string_literal: true
+
+module SimpleQuery
+  module Stream
+    module PostgresStream
+      # rubocop:disable Metrics/MethodLength
+      def stream_each_postgres(batch_size, &block)
+        select_sql = cached_sql
+
+        conn = ActiveRecord::Base.connection.raw_connection
+        cursor_name = "simple_query_cursor_#{object_id}"
+
+        begin
+          conn.exec("BEGIN")
+          declare_sql = "DECLARE #{cursor_name} NO SCROLL CURSOR FOR #{select_sql}"
+          conn.exec(declare_sql)
+
+          loop do
+            res = conn.exec("FETCH #{batch_size} FROM #{cursor_name}")
+            break if res.ntuples.zero?
+
+            res.each do |pg_row|
+              record = build_row_object(pg_row)
+              block.call(record)
+            end
+          end
+
+          conn.exec("CLOSE #{cursor_name}")
+          conn.exec("COMMIT")
+        rescue StandardError => e
+          begin
+            conn.exec("ROLLBACK")
+          rescue StandardError
+            nil
+          end
+          raise e
+        end
+      end
+      # rubocop:enable Metrics/MethodLength
+
+      private
+
+      def build_row_object(pg_row)
+        if @read_model_class
+          obj = @read_model_class.allocate
+          @read_model_class.attributes.each do |attr_name, col_name|
+            obj.instance_variable_set(:"@#{attr_name}", pg_row[col_name])
+          end
+          obj
+        else
+          columns = pg_row.keys
+          values = columns.map { |k| pg_row[k] }
+          struct = result_struct(columns)
+          struct.new(*values)
+        end
+      end
+    end
+  end
+end

data/lib/simple_query/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module SimpleQuery
-  VERSION = "0.3.1"
+  VERSION = "0.4.0"
 end

data/lib/simple_query.rb

@@ -3,6 +3,8 @@
 require "active_support/concern"
 require "active_record"
 
+require_relative "simple_query/stream/mysql_stream"
+require_relative "simple_query/stream/postgres_stream"
 require_relative "simple_query/builder"
 require_relative "simple_query/read_model"
 require_relative "simple_query/clauses/where_clause"
@@ -11,6 +13,7 @@ require_relative "simple_query/clauses/order_clause"
 require_relative "simple_query/clauses/distinct_clause"
 require_relative "simple_query/clauses/limit_offset_clause"
 require_relative "simple_query/clauses/group_having_clause"
+require_relative "simple_query/clauses/set_clause"
 
 module SimpleQuery
   extend ActiveSupport::Concern
@@ -41,19 +44,13 @@ module SimpleQuery
       @_simple_scopes ||= {}
     end
 
-    # A reusable scope that can be applied to a SimpleQuery::Builder instance
-    # Example:
-    #   simple_scope :active do
-    #     where(active: true)
-    #   end
-    #
-    # Parameterized scope:
-    #   simple_scope :by_name do |name|
-    #     where(name: name)
-    #   end
-    #
-    def simple_scope(name, &block)
-      _simple_scopes[name.to_sym] = block
+    def simple_scope(name, body = nil, &block)
+      raise ArgumentError, "Pass either a proc/lambda or a block, not both" if body && block_given?
+
+      scope_body = body || block
+      raise ArgumentError, "You must provide a block or a proc" unless scope_body
+
+      _simple_scopes[name.to_sym] = scope_body
     end
   end
 

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: simple_query
 version: !ruby/object:Gem::Version
-  version: 0.3.1
+  version: 0.4.0
 platform: ruby
 authors:
 - Alex Kholodniak
 autorequire:
 bindir: bin
 cert_chain: []
-date: 2025-02-27 00:00:00.000000000 Z
+date: 2025-03-15 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: activerecord
@@ -30,62 +30,6 @@ dependencies:
     - - "<="
       - !ruby/object:Gem::Version
         version: '8.0'
-- !ruby/object:Gem::Dependency
-  name: rake
-  requirement: !ruby/object:Gem::Requirement
-    requirements:
-    - - "~>"
-      - !ruby/object:Gem::Version
-        version: '13.0'
-  type: :development
-  prerelease: false
-  version_requirements: !ruby/object:Gem::Requirement
-    requirements:
-    - - "~>"
-      - !ruby/object:Gem::Version
-        version: '13.0'
-- !ruby/object:Gem::Dependency
-  name: rspec
-  requirement: !ruby/object:Gem::Requirement
-    requirements:
-    - - "~>"
-      - !ruby/object:Gem::Version
-        version: '3.0'
-  type: :development
-  prerelease: false
-  version_requirements: !ruby/object:Gem::Requirement
-    requirements:
-    - - "~>"
-      - !ruby/object:Gem::Version
-        version: '3.0'
-- !ruby/object:Gem::Dependency
-  name: rubocop
-  requirement: !ruby/object:Gem::Requirement
-    requirements:
-    - - "~>"
-      - !ruby/object:Gem::Version
-        version: '1.21'
-  type: :development
-  prerelease: false
-  version_requirements: !ruby/object:Gem::Requirement
-    requirements:
-    - - "~>"
-      - !ruby/object:Gem::Version
-        version: '1.21'
-- !ruby/object:Gem::Dependency
-  name: sqlite3
-  requirement: !ruby/object:Gem::Requirement
-    requirements:
-    - - "~>"
-      - !ruby/object:Gem::Version
-        version: 1.5.0
-  type: :development
-  prerelease: false
-  version_requirements: !ruby/object:Gem::Requirement
-    requirements:
-    - - "~>"
-      - !ruby/object:Gem::Version
-        version: 1.5.0
 description: SimpleQuery provides a flexible and performant way to construct complex
   database queries in Ruby on Rails applications. It offers an intuitive interface
   for building queries with joins, conditions, and aggregations, while potentially
@@ -105,8 +49,11 @@ files:
 - lib/simple_query/clauses/join_clause.rb
 - lib/simple_query/clauses/limit_offset_clause.rb
 - lib/simple_query/clauses/order_clause.rb
+- lib/simple_query/clauses/set_clause.rb
 - lib/simple_query/clauses/where_clause.rb
 - lib/simple_query/read_model.rb
+- lib/simple_query/stream/mysql_stream.rb
+- lib/simple_query/stream/postgres_stream.rb
 - lib/simple_query/version.rb
 homepage: https://github.com/kholdrex/simple_query
 licenses:
@@ -133,5 +80,6 @@ requirements: []
 rubygems_version: 3.5.9
 signing_key:
 specification_version: 4
-summary: A lightweight and efficient query builder for ActiveRecord.
+summary: A lightweight, multi-DB-friendly, and high-performance query builder for
+  ActiveRecord, featuring streaming, bulk updates, and read-model support.
 test_files: []