sequel 5.71.0 → 5.73.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 17fbd14a63974634d39c289194210ea773d5e2017ad24ac3f6a5c89cc6eb481d
-  data.tar.gz: 390a9c664bf0a7710bb341ef8699e53b0ed2ea4d4fd2c221f7e140393d52047e
+  metadata.gz: bf3b8749cd91ac285e66359ae16adf9605c13e673a0cf3f7251527573d71fed6
+  data.tar.gz: 817f6980be08b29368eb0f08631d64ae48a4281aa1b2974b589441fea50e2dac
 SHA512:
-  metadata.gz: 0af7a0afdad27b270d69eb8248e734408c0c132d209c111f03957a14d1fc2ef44fc4a6da66eaa4986dadf1565f428a140dc9b807bb0215117b015b69057445a1
-  data.tar.gz: a2aade8559d69fe43306b6834069e976f57809170e1a9102501e4031d899dd046119de95f485712024e68aa5b077cf63cccc74b7a77b244e821a2c42b87a09a3
+  metadata.gz: 6da7c5e7a7c74c4767c55d126adc3743875a5dd0822ee780b22b1ca3c3791cc10d08cf74c984776dbd9fbbcf61d9246bcef4c81e64204e5d72aa59545ef9a2d9
+  data.tar.gz: 7ac12f6689d846345fed77e23665679ae4886d60e4d25d4e201a3cfdac4f231882af0aaf3e615bd9981d09c2e9537c1e287fad614f81760f2d897b87402b38b1

data/CHANGELOG

@@ -1,3 +1,31 @@
+=== 5.73.0 (2023-10-01)
+
+* Handle disconnect errors in ibmdb and jdbc/db2 adapters (jeremyevans) (#2083)
+
+* Support skipping transactions in Dataset#{import,paged_each} using :skip_transaction option (jeremyevans)
+
+* Add Database#transaction :skip_transaction option to skip creating a transaction or savepoint (jeremyevans)
+
+* Stop using a transaction for a single query if calling Dataset#import with a dataset (jeremyevans)
+
+* Add paged_operations plugin for paged deletes and updates and other custom operations (jeremyevans) (#2080)
+
+* Support to_tsquery: :websearch option to Dataset#full_text_search on PostgreSQL 11+ (jeremyevans) (#2075)
+
+* Add MassAssignmentRestriction#model and #column for getting the model instance and related column for mass assignment errors (artofhuman, jeremyevans) (#2079)
+
+* Stop using base64 library in column_encryption plugin (jeremyevans)
+
+=== 5.72.0 (2023-09-01)
+
+* Sort caches before marshalling when using schema_caching, index_caching, static_cache_cache, and pg_auto_constraint_validations (jeremyevans)
+
+* Change the defaults_setter plugin do a deep-copy of database default hash/array values and delegates (jeremyevans) (#2069)
+
+* Add pg_auto_parameterize_in_array extension, for converting IN/NOT IN to = ANY or != ALL for more types (jeremyevans)
+
+* Fix literalization of infinite and NaN float values in PostgreSQL array bound variables (jeremyevans)
+
 === 5.71.0 (2023-08-01)
 
 * Support ILIKE ANY on PostgreSQL by not forcing the use of ESCAPE for ILIKE (gilesbowkett) (#2066)

data/README.rdoc

@@ -825,7 +825,7 @@ You can dynamically customize eager loads for both +eager+ and +eager_graph+ whi
 
 === Joining with Associations
 
-You can use the +association_join+ method to add a join to the model's dataset based on the assocation:
+You can use the +association_join+ method to add a join to the model's dataset based on the association:
 
   Post.association_join(:author)
   # SELECT * FROM posts
@@ -927,7 +927,7 @@ Sequel fully supports the currently supported versions of Ruby (MRI) and JRuby.
 support unsupported versions of Ruby or JRuby, but such support may be dropped in any
 minor version if keeping it becomes a support issue.  The minimum Ruby version
 required to run the current version of Sequel is 1.9.2, and the minimum JRuby version is
-9.0.0.0.
+9.2.0.0 (due to the bigdecimal dependency).
 
 == Maintainer
 

data/doc/mass_assignment.rdoc

@@ -48,7 +48,7 @@ If you want to change mass assignment so it ignores attempts to access restricte
 Since mass assignment by default allows modification of all column values except for primary key columns, it can be a security risk in some cases.
 If you are dealing with untrusted input, you are generally going to want to restrict what should be updated.
 
-Sequel has <tt>Model#set_fields</tt> and <tt>Model#update_fields</tt> methods, which are designed to be used with untrused input.
+Sequel has <tt>Model#set_fields</tt> and <tt>Model#update_fields</tt> methods, which are designed to be used with untrusted input.
 These methods take two arguments, the untrusted hash as the first argument, and a trusted array of field names as the second argument:
 
   post.set_fields({title: 'T', body: 'B'}, [:title, :body])

data/doc/release_notes/5.72.0.txt

@@ -0,0 +1,33 @@
+= New Features
+
+* A pg_auto_parameterize_in_array extension has been added, which
+  handles conversion of IN/NOT IN to = ANY or != ALL for more types.
+  The pg_auto_parameterize extension only handles integer types by
+  default, because other types require the pg_array extension. This
+  new extension adds handling for Float, BigDecimal, Date, Time,
+  DateTime, Sequel::SQLTime, and Sequel::SQL::Blob types.  It can
+  also handle String types if the :treat_string_list_as_text_array
+  Database option is present, using the text type for that. Handling
+  String values as text is not the default because that may cause
+  issues for some queries.
+
+= Other Improvements
+
+* The defaults_setter plugin now does a deep copy of database
+  default values that are hash/array or delegates to hash/array.
+  This fixes cases where the database default values are mutated.
+
+* Sequel now correctly handles infinite and NaN float values used
+  inside PostgreSQL array bound variables.
+
+* The data in the cache files used by the schema_caching and
+  index_caching extensions and static_cache_cache and
+  pg_auto_constraint_validations plugins are now sorted before the
+  cache file is saved, increasing consistency between runs.
+
+* bigdecimal has been added as a dependency.  bigdecimal is currently
+  a default gem in Ruby from 1.9 to 3.2, but it will move to a
+  bundled gem in Ruby 3.4, and there will be warnings in Ruby 3.3
+  for cases that will break in Ruby 3.4.  Adding bigdecimal as a
+  dependency should avoid warnings when using bundler in Ruby 3.3,
+  and should avoid errors in Ruby 3.4.

data/doc/release_notes/5.73.0.txt

@@ -0,0 +1,66 @@
+= New Features
+
+* A paged_operations plugin has been added, which adds support for
+  paged_datasets, paged_update, and paged_delete dataset methods.
+  This methods are designed to be used on large datasets, to split 
+  a large query into separate smaller queries, to avoid locking the
+  related database table for a long period of time.
+  paged_update and paged_delete operate the same as update and delete,
+  returning the number of rows updated or deleted. paged_datasets yields
+  one or more datasets representing subsets of the receiver, with the
+  union of all of those datasets comprising all records in the receiver:
+
+    Album.plugin :paged_operations
+
+    Album.where{name > 'M'}.paged_datasets{|ds| puts ds.sql}
+    # Runs: SELECT id FROM albums WHERE (name <= 'M') ORDER BY id LIMIT 1 OFFSET 1000
+    # Prints: SELECT * FROM albums WHERE ((name <= 'M') AND ("id" < 1002))
+    # Runs: SELECT id FROM albums WHERE ((name <= 'M') AND (id >= 1002)) ORDER BY id LIMIT 1 OFFSET 1000
+    # Prints: SELECT * FROM albums WHERE ((name <= 'M') AND ("id" < 2002) AND (id >= 1002))
+    # ...
+    # Runs: SELECT id FROM albums WHERE ((name <= 'M') AND (id >= 10002)) ORDER BY id LIMIT 1 OFFSET 1000
+    # Prints: SELECT * FROM albums WHERE ((name <= 'M') AND (id >= 10002))
+
+    Album.where{name <= 'M'}.paged_update(:updated_at=>Sequel::CURRENT_TIMESTAMP)
+    # SELECT id FROM albums WHERE (name <= 'M') ORDER BY id LIMIT 1 OFFSET 1000
+    # UPDATE albums SET updated_at = CURRENT_TIMESTAMP WHERE ((name <= 'M') AND ("id" < 1002))
+    # SELECT id FROM albums WHERE ((name <= 'M') AND (id >= 1002)) ORDER BY id LIMIT 1 OFFSET 1000
+    # UPDATE albums SET updated_at = CURRENT_TIMESTAMP WHERE ((name <= 'M') AND ("id" < 2002) AND (id >= 1002))
+    # ...
+    # SELECT id FROM albums WHERE ((name <= 'M') AND (id >= 10002)) ORDER BY id LIMIT 1 OFFSET 1000
+    # UPDATE albums SET updated_at = CURRENT_TIMESTAMP WHERE ((name <= 'M') AND (id >= 10002))
+
+    Album.where{name > 'M'}.paged_delete
+    # SELECT id FROM albums WHERE (name > 'M') ORDER BY id LIMIT 1 OFFSET 1000
+    # DELETE FROM albums WHERE ((name > 'M') AND (id < 1002))
+    # SELECT id FROM albums WHERE (name > 'M') ORDER BY id LIMIT 1 OFFSET 1000
+    # DELETE FROM albums WHERE ((name > 'M') AND (id < 2002))
+    # ...
+    # SELECT id FROM albums WHERE (name > 'M') ORDER BY id LIMIT 1 OFFSET 1000
+    # DELETE FROM albums WHERE (name > 'M')
+
+* A Dataset#transaction :skip_transaction option is now support to
+  checkout a connection from the pool without opening a transaction.  This
+  makes it easier to handle cases where a transaction may or not be used
+  based on configuration/options.  Dataset#import and Dataset#paged_each
+  now both support the :skip_transaction option to skip transactions.
+
+* Dataset#full_text_search now supports the to_tsquery: :websearch option
+  on PostgreSQL 11+, to use the websearch_to_tsquery database function.
+
+* The Sequel::MassAssignmentRestriction exception now supports model
+  and column methods to get provide additional information about the
+  exception.  Additionally, the exception message now includes information
+  about the model class.
+
+= Other Improvements
+
+* The ibmdb and jdbc/db2 adapter now both handle disconnect errors
+  correctly, removing the related connection from the pool.
+
+* Dataset#import no longer uses an explicit transaction if given a dataset
+  value, as in that case, only a single query is used.
+
+* The column_encryption plugin no longer uses the base64 library.  The
+  base64 library is moving from the standard library to a bundled gem
+  in Ruby 3.4, and this avoids having a dependency on it.

data/doc/testing.rdoc

@@ -176,7 +176,7 @@ SEQUEL_MODEL_PREPARED_STATEMENTS :: Use the prepared_statements plugin when runn
 SEQUEL_MODEL_THROW_FAILURES :: Use the throw_failures plugin when running the specs
 SEQUEL_NO_CACHE_ASSOCIATIONS :: Don't cache association metadata when running the specs
 SEQUEL_NO_PENDING :: Don't skip any specs, try running all specs (note, can cause lockups for some adapters)
-SEQUEL_PG_AUTO_PARAMETERIZE :: Use the pg_auto_parameterize extension when running the postgres specs
+SEQUEL_PG_AUTO_PARAMETERIZE :: Use the pg_auto_parameterize extension when running the postgres specs.  Value can be +in_array+ to test the pg_auto_parameterize_in_array extension, and +in_array_string+ to test the pg_auto_parameterize_in_array extension with the +:treat_in_string_list_as_text_array+ Database option set.
 SEQUEL_PG_TIMESTAMPTZ :: Use the pg_timestamptz extension when running the postgres specs
 SEQUEL_PRIMARY_KEY_LOOKUP_CHECK_VALUES :: Use the primary_key_lookup_check_values extension when running the adapter or integration specs
 SEQUEL_QUERY_PER_ASSOCIATION_DB_0_URL :: Run query-per-association integration tests with multiple databases (all 4 must be set to run)

data/lib/sequel/adapters/ibmdb.rb

@@ -301,7 +301,7 @@ module Sequel
       end
 
       def database_exception_sqlstate(exception, opts)
-        exception.sqlstate
+        exception.sqlstate if exception.respond_to?(:sqlstate)
       end
 
       def dataset_class_default

data/lib/sequel/adapters/jdbc/db2.rb

@@ -36,6 +36,10 @@ module Sequel
 
         private
 
+        def database_exception_sqlstate(exception, opts)
+          exception.sql_state if exception.respond_to?(:sql_state)
+        end
+
         def set_ps_arg(cps, arg, i)
           case arg
           when Sequel::SQL::Blob

data/lib/sequel/adapters/shared/db2.rb

@@ -215,6 +215,18 @@ module Sequel
         DATABASE_ERROR_REGEXPS
       end
 
+      DISCONNECT_SQL_STATES = %w'40003 08001 08003'.freeze
+      def disconnect_error?(exception, opts)
+        sqlstate = database_exception_sqlstate(exception, opts)
+
+        case sqlstate
+        when *DISCONNECT_SQL_STATES
+          true
+        else
+          super
+        end
+      end
+
       # DB2 has issues with quoted identifiers, so
       # turn off database quoting by default.
       def quote_identifiers_default

data/lib/sequel/adapters/shared/postgres.rb

@@ -1798,7 +1798,7 @@ module Sequel
       # :phrase :: Similar to :plain, but also adding an ILIKE filter to ensure that
       #            returned rows also include the exact phrase used.
       # :rank :: Set to true to order by the rank, so that closer matches are returned first.
-      # :to_tsquery :: Can be set to :plain or :phrase to specify the function to use to
+      # :to_tsquery :: Can be set to :plain, :phrase, or :websearch to specify the function to use to
       #                convert the terms to a ts_query.
       # :tsquery :: Specifies the terms argument is already a valid SQL expression returning a
       #             tsquery, and can be used directly in the query.
@@ -1818,6 +1818,8 @@ module Sequel
           query_func = case to_tsquery = opts[:to_tsquery]
           when :phrase, :plain
             :"#{to_tsquery}to_tsquery"
+          when :websearch
+            :"websearch_to_tsquery"
           else
             (opts[:phrase] || opts[:plain]) ? :plainto_tsquery : :to_tsquery
           end

data/lib/sequel/database/schema_methods.rb

@@ -712,8 +712,9 @@ module Sequel
       e = options[:ignore_index_errors] || options[:if_not_exists]
       generator.indexes.each do |index|
         begin
-          pr = proc{index_sql_list(name, [index]).each{|sql| execute_ddl(sql)}}
-          supports_transactional_ddl? ? transaction(:savepoint=>:only, &pr) : pr.call
+          transaction(:savepoint=>:only, :skip_transaction=>supports_transactional_ddl? == false) do
+            index_sql_list(name, [index]).each{|sql| execute_ddl(sql)}
+          end
         rescue Error
           raise unless e
         end

data/lib/sequel/database/transactions.rb

@@ -166,6 +166,8 @@ module Sequel
     #               uses :auto_savepoint, you can set this to false to not use a savepoint.
     #               If the value given for this option is :only, it will only create a
     #               savepoint if it is inside a transaction.
+    # :skip_transaction :: If set, do not actually open a transaction or savepoint,
+    #                      just checkout a connection and yield it.
     #
     # PostgreSQL specific options:
     #
@@ -193,6 +195,10 @@ module Sequel
         end
       else
         synchronize(opts[:server]) do |conn|
+          if opts[:skip_transaction]
+            return yield(conn)
+          end
+
           if opts[:savepoint] == :only
             if supports_savepoints?
               if _trans(conn)

data/lib/sequel/dataset/actions.rb

@@ -356,9 +356,11 @@ module Sequel
     #            This does not have an effect if +values+ is a Dataset.
     # :server :: Set the server/shard to use for the transaction and insert
     #            queries.
+    # :skip_transaction :: Do not use a transaction even when using multiple
+    #                      INSERT queries.
     # :slice :: Same as :commit_every, :commit_every takes precedence.
     def import(columns, values, opts=OPTS)
-      return @db.transaction{insert(columns, values)} if values.is_a?(Dataset)
+      return insert(columns, values) if values.is_a?(Dataset)
 
       return if values.empty?
       raise(Error, 'Using Sequel::Dataset#import with an empty column array is not allowed') if columns.empty?
@@ -588,6 +590,8 @@ module Sequel
     #                   if your ORDER BY expressions are not simple columns, if they contain
     #                   qualified identifiers that would be ambiguous unqualified, if they contain
     #                   any identifiers that are aliased in SELECT, and potentially other cases.
+    # :skip_transaction :: Do not use a transaction. This can be useful if you want to prevent
+    #                      a lock on the database table, at the expense of consistency.
     #
     # Examples:
     #
@@ -1111,11 +1115,9 @@ module Sequel
     # are provided. When only a single value or statement is provided, then yield
     # without using a transaction.
     def _import_transaction(values, trans_opts, &block)
-      if values.length > 1
-        @db.transaction(trans_opts, &block)
-      else
-        yield
-      end
+      # OK to mutate trans_opts as it is generated by _import
+      trans_opts[:skip_transaction] = true if values.length <= 1
+      @db.transaction(trans_opts, &block)
     end
   
     # Internals of +select_hash+ and +select_hash_groups+

data/lib/sequel/extensions/index_caching.rb

@@ -56,7 +56,11 @@ module Sequel
     
     # Dump the index cache to the filename given in Marshal format.
     def dump_index_cache(file)
-      File.open(file, 'wb'){|f| f.write(Marshal.dump(@indexes))}
+      indexes = {}
+      @indexes.sort.each do |k, v|
+        indexes[k] = v
+      end
+      File.open(file, 'wb'){|f| f.write(Marshal.dump(indexes))}
       nil
     end
 

data/lib/sequel/extensions/migration.rb

@@ -482,11 +482,7 @@ module Sequel
         @use_transactions
       end
 
-      if use_trans
-        db.transaction(&block)
-      else
-        yield
-      end
+      db.transaction(:skip_transaction=>use_trans == false, &block)
     end
 
     # Load the migration file, raising an exception if the file does not define

data/lib/sequel/extensions/pg_array.rb

@@ -233,6 +233,14 @@ module Sequel
             a
           when String
             bound_variable_array_string(a)
+          when Float
+            if a.infinite?
+              a > 0 ? '"Infinity"' : '"-Infinity"'
+            elsif a.nan?
+              '"NaN"'
+            else
+              literal(a)
+            end
           else
             if (s = bound_variable_arg(a, nil)).is_a?(String)
               bound_variable_array_string(s)

data/lib/sequel/extensions/pg_auto_parameterize_in_array.rb

@@ -0,0 +1,110 @@
+# frozen-string-literal: true
+#
+# The pg_auto_parameterize_in_array extension builds on the pg_auto_parameterize
+# extension, adding support for handling additional types when converting from
+# IN to = ANY and NOT IN to != ALL:
+#
+#   DB[:table].where(column: [1.0, 2.0, ...])
+#   # Without extension: column IN ($1::numeric, $2:numeric, ...) # bound variables: 1.0, 2.0, ...
+#   # With extension:    column = ANY($1::numeric[]) # bound variables: [1.0, 2.0, ...]
+#
+# This prevents the use of an unbounded number of bound variables based on the
+# size of the array, as well as using different SQL for different array sizes.
+#
+# The following types are supported when doing the conversions, with the database
+# type used:
+#
+# Float :: if any are infinite or NaN, double precision, otherwise numeric
+# BigDecimal :: numeric
+# Date :: date
+# Time :: timestamp (or timestamptz if pg_timestamptz extension is used)
+# DateTime :: timestamp (or timestamptz if pg_timestamptz extension is used)
+# Sequel::SQLTime :: time
+# Sequel::SQL::Blob :: bytea
+# 
+# String values are also supported using the +text+ type, but only if the 
+# +:treat_string_list_as_text_array+ Database option is used. This is because
+# treating strings as text can break programs, since the type for
+# literal strings in PostgreSQL is +unknown+, not +text+.
+#
+# The conversion is only done for single dimensional arrays that have more
+# than two elements, where all elements are of the same class (other than
+# nil values).
+#
+# Related module: Sequel::Postgres::AutoParameterizeInArray
+
+module Sequel
+  module Postgres
+    # Enable automatically parameterizing queries.
+    module AutoParameterizeInArray
+      # Transform column IN (...) expressions into column = ANY($)
+      # and column NOT IN (...) expressions into column != ALL($)
+      # using an array bound variable for the ANY/ALL argument,
+      # if all values inside the predicate are of the same type and
+      # the type is handled by the extension.
+      # This is the same optimization PostgreSQL performs internally,
+      # but this reduces the number of bound variables.
+      def complex_expression_sql_append(sql, op, args)
+        case op
+        when :IN, :"NOT IN"
+          l, r = args
+          if auto_param?(sql) && (type = _bound_variable_type_for_array(r))
+            if op == :IN 
+              op = :"="
+              func = :ANY
+            else
+              op = :!=
+              func = :ALL
+            end
+            args = [l, Sequel.function(func, Sequel.pg_array(r, type))]
+          end
+        end
+
+        super
+      end
+
+      private
+
+      # The bound variable type string to use for the bound variable array.
+      # Returns nil if a bound variable should not be used for the array.
+      def _bound_variable_type_for_array(r)
+        return unless Array === r && r.size > 1
+        classes = r.map(&:class)
+        classes.uniq!
+        classes.delete(NilClass)
+        return unless classes.size == 1
+
+        klass = classes[0]
+        if klass == Integer
+          # This branch is not taken on Ruby <2.4, because of the Fixnum/Bignum split.
+          # However, that causes no problems as pg_auto_parameterize handles integer
+          # arrays natively (though the SQL used is different)
+          "int8"
+        elsif klass == String
+          "text" if db.typecast_value(:boolean, db.opts[:treat_string_list_as_text_array])
+        elsif klass == BigDecimal
+          "numeric"
+        elsif klass == Date
+          "date"
+        elsif klass == Time
+          @db.cast_type_literal(Time)
+        elsif klass == Float
+          # PostgreSQL treats literal floats as numeric, not double precision
+          # But older versions of PostgreSQL don't handle Infinity/NaN in numeric
+          r.all?{|v| v.nil? || v.finite?} ? "numeric" : "double precision"
+        elsif klass == Sequel::SQLTime
+          "time"
+        elsif klass == DateTime
+          @db.cast_type_literal(DateTime)
+        elsif klass == Sequel::SQL::Blob
+          "bytea"
+        end
+      end
+    end
+  end
+
+  Database.register_extension(:pg_auto_parameterize_in_array) do |db|
+    db.extension(:pg_array, :pg_auto_parameterize)
+    db.extend_datasets(Postgres::AutoParameterizeInArray)
+  end
+end

data/lib/sequel/extensions/schema_caching.rb

@@ -52,7 +52,7 @@ module Sequel
     # Dump the cached schema to the filename given in Marshal format.
     def dump_schema_cache(file)
       sch = {}
-      @schemas.each do |k,v|
+      @schemas.sort.each do |k,v|
         sch[k] = v.map do |c, h|
           h = Hash[h]
           h.delete(:callable_default)

data/lib/sequel/model/base.rb

@@ -1945,8 +1945,10 @@ module Sequel
       end
       
       # If transactions should be used, wrap the yield in a transaction block.
-      def checked_transaction(opts=OPTS)
-        use_transaction?(opts) ? db.transaction({:server=>this_server}.merge!(opts)){yield} : yield
+      def checked_transaction(opts=OPTS, &block)
+        h = {:server=>this_server}.merge!(opts)
+        h[:skip_transaction] = true unless use_transaction?(opts)
+        db.transaction(h, &block)
       end
 
       # Change the value of the column to given value, recording the change.
@@ -2031,19 +2033,20 @@ module Sequel
         meths = setter_methods(type)
         strict = strict_param_setting
         hash.each do |k,v|
+          k = k.to_s
           m = "#{k}="
           if meths.include?(m)
             set_column_value(m, v)
           elsif strict
             # Avoid using respond_to? or creating symbols from user input
             if public_methods.map(&:to_s).include?(m)
-              if Array(model.primary_key).map(&:to_s).member?(k.to_s) && model.restrict_primary_key?
-                raise MassAssignmentRestriction, "#{k} is a restricted primary key"
+              if Array(model.primary_key).map(&:to_s).member?(k) && model.restrict_primary_key?
+                raise MassAssignmentRestriction.create("#{k} is a restricted primary key", self, k)
               else
-                raise MassAssignmentRestriction, "#{k} is a restricted column"
+                raise MassAssignmentRestriction.create("#{k} is a restricted column", self, k)
               end
             else
-              raise MassAssignmentRestriction, "method #{m} doesn't exist"
+              raise MassAssignmentRestriction.create("method #{m} doesn't exist", self, k)
             end
           end
         end
@@ -2147,8 +2150,9 @@ module Sequel
       #   # DELETE FROM artists WHERE (id = 2)
       #   # ...
       def destroy
-        pr = proc{all(&:destroy).length}
-        model.use_transactions ? @db.transaction(:server=>opts[:server], &pr) : pr.call
+        @db.transaction(:server=>opts[:server], :skip_transaction=>model.use_transactions == false) do
+          all(&:destroy).length
+        end
       end
 
       # If there is no order already defined on this dataset, order it by
@@ -2228,11 +2232,17 @@ module Sequel
 
       private
 
+      # Return the dataset ordered by the model's primary key.  This should not
+      # be used if the model does not have a primary key.
+      def _force_primary_key_order
+        cached_dataset(:_pk_order_ds){order(*model.primary_key)}
+      end
+
       # If the dataset is not already ordered, and the model has a primary key,
       # return a clone ordered by the primary key.
       def _primary_key_order
-        if @opts[:order].nil? && model && (pk = model.primary_key)
-          cached_dataset(:_pk_order_ds){order(*pk)}
+        if @opts[:order].nil? && model && model.primary_key
+          _force_primary_key_order
         end
       end
 

data/lib/sequel/model/exceptions.rb

@@ -24,11 +24,23 @@ module Sequel
   UndefinedAssociation = Class.new(Error)
   ).name
 
-  (
   # Raised when a mass assignment method is called in strict mode with either a restricted column
   # or a column without a setter method.
-  MassAssignmentRestriction = Class.new(Error)
-  ).name
+  class MassAssignmentRestriction < Error
+    # The Sequel::Model object related to this exception.
+    attr_reader :model
+
+    # The column related to this exception, as a string.
+    attr_reader :column
+
+    # Create an instance of this class with the model and column set.
+    def self.create(msg, model, column)
+      e = new("#{msg} for class #{model.class.inspect}")
+      e.instance_variable_set(:@model, model)
+      e.instance_variable_set(:@column, column)
+      e
+    end
+  end
   
   # Exception class raised when +raise_on_save_failure+ is set and validation fails
   class ValidationFailed < Error

data/lib/sequel/plugins/column_encryption.rb

@@ -31,7 +31,6 @@ rescue RuntimeError, OpenSSL::Cipher::CipherError
   # :nocov:
 end
 
-require 'base64'
 require 'securerandom'
 
 module Sequel
@@ -375,7 +374,7 @@ module Sequel
         # Decrypt using any supported format and any available key.
         def decrypt(data)
           begin
-            data = Base64.urlsafe_decode64(data)
+            data = urlsafe_decode64(data)
           rescue ArgumentError
             raise Error, "Unable to decode encrypted column: invalid base64"
           end
@@ -448,7 +447,7 @@ module Sequel
         # The prefix string of columns for the given search type and the first configured encryption key.
         # Used to find values that do not use this prefix in order to perform reencryption.
         def current_key_prefix(search_type)
-          Base64.urlsafe_encode64("#{search_type.chr}\0#{@key_id.chr}")
+          urlsafe_encode64("#{search_type.chr}\0#{@key_id.chr}")
         end
 
         # The prefix values to search for the given data (an array of strings), assuming the column uses
@@ -472,11 +471,33 @@ module Sequel
 
         private
 
+        if RUBY_VERSION >= '2.4'
+          def decode64(str)
+            str.unpack1("m0")
+          end
+        # :nocov:
+        else
+          def decode64(str)
+            str.unpack("m0")[0]
+          end
+        # :nocov:
+        end
+
+        def urlsafe_encode64(bin)
+          str = [bin].pack("m0")
+          str.tr!("+/", "-_")
+          str
+        end
+
+        def urlsafe_decode64(str)
+          decode64(str.tr("-_", "+/"))
+        end
+
         # An array of strings, one for each configured encryption key, to find encypted values matching
         # the given data and search format.
         def _search_prefixes(data, search_type)
           @key_map.map do |key_id, (key, _)|
-            Base64.urlsafe_encode64(_search_prefix(data, search_type, key_id, key))
+            urlsafe_encode64(_search_prefix(data, search_type, key_id, key))
           end
         end
 
@@ -509,7 +530,7 @@ module Sequel
           cipher_text << cipher.update(data) if data_size > 0
           cipher_text << cipher.final
 
-          Base64.urlsafe_encode64("#{prefix}#{random_data}#{cipher_iv}#{cipher.auth_tag}#{cipher_text}")
+          urlsafe_encode64("#{prefix}#{random_data}#{cipher_iv}#{cipher.auth_tag}#{cipher_text}")
         end
       end
 

data/lib/sequel/plugins/defaults_setter.rb

@@ -1,5 +1,7 @@
 # frozen-string-literal: true
 
+require 'delegate'
+
 module Sequel
   module Plugins
     # The defaults_setter plugin makes the column getter methods return the default
@@ -106,6 +108,20 @@ module Sequel
             lambda{Date.today}
           when Sequel::CURRENT_TIMESTAMP
             lambda{dataset.current_datetime}
+          when Hash, Array
+            v = Marshal.dump(v).freeze
+            lambda{Marshal.load(v)}
+          when Delegator
+            # DelegateClass returns an anonymous case, which cannot be marshalled, so marshal the
+            # underlying object and create a new instance of the class with the unmarshalled object.
+            klass = v.class
+            case o = v.__getobj__
+            when Hash, Array
+              v = Marshal.dump(o).freeze
+              lambda{klass.new(Marshal.load(v))}
+            else
+              v
+            end
           else
             v
           end

data/lib/sequel/plugins/paged_operations.rb

@@ -0,0 +1,181 @@
+# frozen-string-literal: true
+
+module Sequel
+  module Plugins
+    # The paged_operations plugin adds +paged_update+ and
+    # +paged_delete+ dataset methods.  These behave similarly to
+    # the default +update+ and +delete+ dataset methods, except
+    # that the update or deletion is done in potentially multiple
+    # queries (by default, affecting 1000 rows per query).
+    # For a large table, this prevents the change from
+    # locking the table for a long period of time.
+    #
+    # Because the point of this is to prevent locking tables for
+    # long periods of time, the separate queries are not contained
+    # in a transaction, which means if a later query fails,
+    # earlier queries will still be committed.  You could prevent
+    # this by using a transaction manually, but that defeats the
+    # purpose of using these methods.
+    #
+    # Examples:
+    #
+    #   Album.where{name <= 'M'}.paged_update(updated_at: Sequel::CURRENT_TIMESTAMP)
+    #   # SELECT id FROM albums WHERE (name <= 'M') ORDER BY id LIMIT 1 OFFSET 1000
+    #   # UPDATE albums SET updated_at = CURRENT_TIMESTAMP WHERE ((name <= 'M') AND ("id" < 1002))
+    #   # SELECT id FROM albums WHERE ((name <= 'M') AND (id >= 1002)) ORDER BY id LIMIT 1 OFFSET 1000
+    #   # UPDATE albums SET updated_at = CURRENT_TIMESTAMP WHERE ((name <= 'M') AND ("id" < 2002) AND (id >= 1002))
+    #   # ...
+    #   # SELECT id FROM albums WHERE ((name <= 'M') AND (id >= 10002)) ORDER BY id LIMIT 1 OFFSET 1000
+    #   # UPDATE albums SET updated_at = CURRENT_TIMESTAMP WHERE ((name <= 'M') AND (id >= 10002))
+    #
+    #   Album.where{name > 'M'}.paged_delete
+    #   # SELECT id FROM albums WHERE (name > 'M') ORDER BY id LIMIT 1 OFFSET 1000
+    #   # DELETE FROM albums WHERE ((name > 'M') AND (id < 1002))
+    #   # SELECT id FROM albums WHERE (name > 'M') ORDER BY id LIMIT 1 OFFSET 1000
+    #   # DELETE FROM albums WHERE ((name > 'M') AND (id < 2002))
+    #   # ...
+    #   # SELECT id FROM albums WHERE (name > 'M') ORDER BY id LIMIT 1 OFFSET 1000
+    #   # DELETE FROM albums WHERE (name > 'M')
+    #
+    # The plugin also adds a +paged_datasets+ method that will yield
+    # separate datasets limited in size that in total handle all
+    # rows in the receiver:
+    #
+    #   Album.where{name > 'M'}.paged_datasets{|ds| puts ds.sql}
+    #   # Runs: SELECT id FROM albums WHERE (name <= 'M') ORDER BY id LIMIT 1 OFFSET 1000
+    #   # Prints: SELECT * FROM albums WHERE ((name <= 'M') AND ("id" < 1002))
+    #   # Runs: SELECT id FROM albums WHERE ((name <= 'M') AND (id >= 1002)) ORDER BY id LIMIT 1 OFFSET 1000
+    #   # Prints: SELECT * FROM albums WHERE ((name <= 'M') AND ("id" < 2002) AND (id >= 1002))
+    #   # ...
+    #   # Runs: SELECT id FROM albums WHERE ((name <= 'M') AND (id >= 10002)) ORDER BY id LIMIT 1 OFFSET 1000
+    #   # Prints: SELECT * FROM albums WHERE ((name <= 'M') AND (id >= 10002))
+    #
+    # To set the number of rows per page, pass a :rows_per_page option:
+    #
+    #   Album.where{name <= 'M'}.paged_update({x: Sequel[:x] + 1}, rows_per_page: 4)
+    #   # SELECT id FROM albums WHERE (name <= 'M') ORDER BY id LIMIT 1 OFFSET 4
+    #   # UPDATE albums SET x = x + 1 WHERE ((name <= 'M') AND ("id" < 5))
+    #   # SELECT id FROM albums WHERE ((name <= 'M') AND (id >= 5)) ORDER BY id LIMIT 1 OFFSET 4
+    #   # UPDATE albums SET x = x + 1 WHERE ((name <= 'M') AND ("id" < 9) AND (id >= 5))
+    #   # ...
+    #   # SELECT id FROM albums WHERE ((name <= 'M') AND (id >= 12345)) ORDER BY id LIMIT 1 OFFSET 4
+    #   # UPDATE albums SET x = x + 1 WHERE ((name <= 'M') AND (id >= 12345))
+    # 
+    # You should avoid using +paged_update+ or +paged_datasets+
+    # with updates that modify the primary key, as such usage is
+    # not supported by this plugin.
+    #
+    # This plugin only supports models with scalar primary keys.
+    #
+    # Usage:
+    #
+    #   # Make all model subclasses support paged update/delete/datasets
+    #   # (called before loading subclasses)
+    #   Sequel::Model.plugin :paged_operations
+    #
+    #   # Make the Album class support paged update/delete/datasets
+    #   Album.plugin :paged_operations
+    module PagedOperations
+      module ClassMethods
+        Plugins.def_dataset_methods(self, [:paged_datasets, :paged_delete, :paged_update])
+      end
+
+      module DatasetMethods
+        # Yield datasets for subsets of the receiver that are limited
+        # to no more than 1000 rows (you can configure the number of
+        # rows using +:rows_per_page+).
+        #
+        # Options:
+        # :rows_per_page :: The maximum number of rows in each yielded dataset
+        #                   (unless concurrent modifications are made to the table).
+        def paged_datasets(opts=OPTS)
+          unless defined?(yield)
+            return enum_for(:paged_datasets, opts)
+          end
+
+          pk = _paged_operations_pk(:paged_update)
+          base_offset_ds = offset_ds = _paged_operations_offset_ds(opts)
+          first = nil
+
+          while last = offset_ds.get(pk)
+            ds = where(pk < last)
+            ds = ds.where(pk >= first) if first
+            yield ds
+            first = last
+            offset_ds = base_offset_ds.where(pk >= first)
+          end
+
+          ds = self
+          ds = ds.where(pk >= first) if first
+          yield ds
+          nil
+        end
+
+        # Delete all rows of the dataset using using multiple queries so that
+        # no more than 1000 rows are deleted at a time (you can configure the
+        # number of rows using +:rows_per_page+).
+        #
+        # Options:
+        # :rows_per_page :: The maximum number of rows affected by each DELETE query
+        #                   (unless concurrent modifications are made to the table).
+        def paged_delete(opts=OPTS)
+          if (db.database_type == :oracle && !supports_fetch_next_rows?) || (db.database_type == :mssql && !is_2012_or_later?)
+            raise Error, "paged_delete is not supported on MSSQL/Oracle when using emulated offsets"
+          end
+          pk = _paged_operations_pk(:paged_delete)
+          rows_deleted = 0
+          offset_ds = _paged_operations_offset_ds(opts)
+          while last = offset_ds.get(pk)
+            rows_deleted += where(pk < last).delete
+          end
+          rows_deleted + delete
+        end
+
+        # Update all rows of the dataset using using multiple queries so that
+        # no more than 1000 rows are updated at a time (you can configure the
+        # number of rows using +:rows_per_page+). All arguments are
+        # passed to Dataset#update.
+        #
+        # Options:
+        # :rows_per_page :: The maximum number of rows affected by each UPDATE query
+        #                   (unless concurrent modifications are made to the table).
+        def paged_update(values, opts=OPTS)
+          rows_updated = 0
+          paged_datasets(opts) do |ds|
+            rows_updated += ds.update(values)
+          end
+          rows_updated
+        end
+
+        private
+
+        # Run some basic checks common to paged_{datasets,delete,update}
+        # and return the primary key to operate on as a Sequel::Identifier.
+        def _paged_operations_pk(meth)
+          raise Error, "cannot use #{meth} if dataset has a limit or offset" if @opts[:limit] || @opts[:offset]
+          if db.database_type == :db2 && db.offset_strategy == :emulate
+            raise Error, "the paged_operations plugin is not supported on DB2 when using emulated offsets, set the :offset_strategy Database option to 'limit_offset' or 'offset_fetch'"
+          end
+
+          case pk = model.primary_key
+          when Symbol
+            Sequel.identifier(pk)
+          when Array
+            raise Error, "cannot use #{meth} on a model with a composite primary key"
+          else
+            raise Error, "cannot use #{meth} on a model without a primary key"
+          end
+        end
+
+        # The dataset that will be used by paged_{datasets,delete,update}
+        # to get the upper limit for the next query.
+        def _paged_operations_offset_ds(opts)
+          if rows_per_page = opts[:rows_per_page]
+            raise Error, ":rows_per_page option must be at least 1" unless rows_per_page >= 1
+          end
+          _force_primary_key_order.offset(rows_per_page || 1000)
+        end
+      end
+    end
+  end
+end

data/lib/sequel/plugins/pg_auto_constraint_validations.rb

@@ -133,7 +133,11 @@ module Sequel
         # Dump the in-memory cached metadata to the cache file.
         def dump_pg_auto_constraint_validations_cache
           raise Error, "No pg_auto_constraint_validations setup" unless file = @pg_auto_constraint_validations_cache_file
-          File.open(file, 'wb'){|f| f.write(Marshal.dump(@pg_auto_constraint_validations_cache))}
+          pg_auto_constraint_validations_cache = {}
+          @pg_auto_constraint_validations_cache.sort.each do |k, v|
+            pg_auto_constraint_validations_cache[k] = v
+          end
+          File.open(file, 'wb'){|f| f.write(Marshal.dump(pg_auto_constraint_validations_cache))}
           nil
         end
 

data/lib/sequel/plugins/static_cache_cache.rb

@@ -26,7 +26,11 @@ module Sequel
       module ClassMethods
         # Dump the in-memory cached rows to the cache file.
         def dump_static_cache_cache
-          File.open(@static_cache_cache_file, 'wb'){|f| f.write(Marshal.dump(@static_cache_cache))}
+          static_cache_cache = {}
+          @static_cache_cache.sort.each do |k, v|
+            static_cache_cache[k] = v
+          end
+          File.open(@static_cache_cache_file, 'wb'){|f| f.write(Marshal.dump(static_cache_cache))}
           nil
         end
 

data/lib/sequel/version.rb

@@ -6,7 +6,7 @@ module Sequel
 
   # The minor version of Sequel.  Bumped for every non-patch level
   # release, generally around once a month.
-  MINOR = 71
+  MINOR = 73
 
   # The tiny version of Sequel.  Usually 0, only bumped for bugfix
   # releases that fix regressions from previous versions.

metadata

@@ -1,15 +1,29 @@
 --- !ruby/object:Gem::Specification
 name: sequel
 version: !ruby/object:Gem::Version
-  version: 5.71.0
+  version: 5.73.0
 platform: ruby
 authors:
 - Jeremy Evans
 autorequire:
 bindir: bin
 cert_chain: []
-date: 2023-08-01 00:00:00.000000000 Z
+date: 2023-10-01 00:00:00.000000000 Z
 dependencies:
+- !ruby/object:Gem::Dependency
+  name: bigdecimal
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
 - !ruby/object:Gem::Dependency
   name: minitest
   requirement: !ruby/object:Gem::Requirement
@@ -204,6 +218,8 @@ extra_rdoc_files:
 - doc/release_notes/5.7.0.txt
 - doc/release_notes/5.70.0.txt
 - doc/release_notes/5.71.0.txt
+- doc/release_notes/5.72.0.txt
+- doc/release_notes/5.73.0.txt
 - doc/release_notes/5.8.0.txt
 - doc/release_notes/5.9.0.txt
 files:
@@ -303,6 +319,8 @@ files:
 - doc/release_notes/5.7.0.txt
 - doc/release_notes/5.70.0.txt
 - doc/release_notes/5.71.0.txt
+- doc/release_notes/5.72.0.txt
+- doc/release_notes/5.73.0.txt
 - doc/release_notes/5.8.0.txt
 - doc/release_notes/5.9.0.txt
 - doc/schema_modification.rdoc
@@ -445,6 +463,7 @@ files:
 - lib/sequel/extensions/pg_array.rb
 - lib/sequel/extensions/pg_array_ops.rb
 - lib/sequel/extensions/pg_auto_parameterize.rb
+- lib/sequel/extensions/pg_auto_parameterize_in_array.rb
 - lib/sequel/extensions/pg_enum.rb
 - lib/sequel/extensions/pg_extended_date_support.rb
 - lib/sequel/extensions/pg_extended_integer_support.rb
@@ -555,6 +574,7 @@ files:
 - lib/sequel/plugins/nested_attributes.rb
 - lib/sequel/plugins/optimistic_locking.rb
 - lib/sequel/plugins/optimistic_locking_base.rb
+- lib/sequel/plugins/paged_operations.rb
 - lib/sequel/plugins/pg_array_associations.rb
 - lib/sequel/plugins/pg_auto_constraint_validations.rb
 - lib/sequel/plugins/pg_row.rb