sequel 5.30.0 → 5.35.0

data/doc/release_notes/5.32.0.txt

@@ -0,0 +1,46 @@
+= New Features
+
+* A fiber_concurrency extension has been added, for using
+  Fiber.current instead of Thread.current when checking out a
+  connection.  This allows separate fibers of the same thread
+  to use separate connections.  In addition to allowing direct use
+  of fibers, this also allows concurrent use of multiple enumerators
+  that use database connections in the same thread.
+
+  When using this extension, you must be careful and ensure that you
+  are not using more concurrent fibers than your connection pool size.
+  Otherwise, all fibers will block while one fiber waits until a
+  connection is available.  It is possible this issue will be
+  addressed when Ruby implements a fiber scheduler (currently
+  being discussed for inclusion in Ruby 3).
+
+* A run_transaction_hooks Database extension has been added,
+  allowing for running the transaction hooks before commit/rollback,
+  which can be helpful for testing the hooks when using transactional
+  testing.
+
+= Other Improvements
+
+* Database#create_table? now works correctly with the :partition_of
+  option on PostgreSQL.
+
+* The timestamp(N) with time zone type is now recognized by the
+  schema parser.
+
+* Singleton methods of the Sequel module have now been moved into a
+  Sequel::SequelMethods module.  This allows you to extend Sequel
+  with a module that overrides the methods and call super to get
+  the default behavior.
+
+* The pg_inet extension no longer defines inet/cidr conversion procs
+  if sequel_pg 1.13.0+ is in use.  This is because sequel_pg 1.13.0+
+  will respect the conversion procs and defining them makes things
+  slower.  sequel_pg 1.13.0+ handles the same conversion by default
+  without needing a conversion proc.
+
+* Method visibility issues in the model, plugin, extension, and adapter
+  code have been fixed.  Most cases fixed were private methods being
+  accidentally made public when they were overridden.
+
+  During this change, Model#_insert_values was changed from public to
+  private, since it was originally intended to be private.

data/doc/release_notes/5.33.0.txt

@@ -0,0 +1,24 @@
+= New Features
+
+* Custom join types are now supported on a per-association basis when
+  using eager_graph/association_join.  This builds on the previous
+  support for custom aliases, using Sequel::SQL::AliasedExpression:
+
+    class Artist < Sequel::Model; end
+    class Album < Sequel::Model; end
+    class Track < Sequel::Model; end
+    Artist.one_to_many :albums
+    Album.one_to_many :tracks
+    Artist.eager_graph(
+      Sequel[:albums].as(:a, join_type: :inner) =>
+        Sequel[:tracks].as(:t, join_type: :left)
+    )
+
+* A Database#current_timestamp_utc accessor has been added on SQLite.
+  Setting this to true will keep CURRENT_TIMESTAMP, CURRENT_TIME, and
+  CURRENT_DATE in UTC instead of converting them to localtime.
+
+= Other Improvements
+
+* The smallserial PostgreSQL type is now recognized and Sequel will
+  not try to mark smallserial columns as identity columns.

data/doc/release_notes/5.34.0.txt

@@ -0,0 +1,40 @@
+= New Features
+
+* The association_pks plugin now creates *_pks_dataset methods for
+  each association.  These are similar to the existing *_pks getter
+  methods, but they return a dataset of the keys instead of the keys
+  themselves.
+
+* The association_pks plugin now supports a :cache_pks association
+  option, which will cache calls to the *_pks getter method. The
+  default behavior remains that the *_pks getter method only returns
+  cached values if the *_pks= setter method has been used to set the
+  values.
+
+* The *_pks getter methods supported by the association_pks plugin
+  now support a :refresh option to ignore any cached values, similar
+  to how the association getter methods work.
+
+= Other Improvements
+
+* If trying to disconnect a server that doesn't exist when using a
+  sharded connection pool, a Sequel::Error is now raised.  Previously,
+  the sharded threaded pool raised a NoMethodError and the sharded
+  single connection pool did not raise an error.
+
+* If using the :savepoint option when savepoints are not supported,
+  a Sequel::InvalidOperation exception is now raised, instead of a
+  NoMethodError.
+
+* Calling Dataset#eager_graph with no arguments now returns the
+  dataset.
+
+* If not connected to the database, the single connection pool will
+  not yield any connections to Database#pool.all_connections.
+
+* Forcing a :ruby eager limit strategy for an association without a
+  limit or offset now works correctly.
+
+* Multiple unnecessary conditionals have been removed.
+
+* Sequel core and model code now have 100% branch coverage.

data/doc/release_notes/5.35.0.txt

@@ -0,0 +1,56 @@
+= New Features
+
+* An instance_specific_default plugin has been added for setting the
+  default for the :instance_specific association option, or
+  warning/raises in cases where it is not specified.  This allows
+  you to easily find associations that would be considering instance
+  specific by default, and mark them as not instance specific for
+  better performance.
+
+= Other Improvements
+
+* Setting the :instance_specific association option to false now
+  works correctly if the association uses a block.  Associations
+  that set the :dataset option are now always considered instance
+  specific, even if the :instance_specific option is explicitly
+  passed.
+
+* The validation_class_methods plugin now considers all :if,
+  :allow_missing, :allow_nil, and :allow_blank options.  Previously,
+  it only considered the first of those options that was set.
+
+* Model.finalize_associations no longer breaks if you have
+  instance-specific associations.
+
+* Model.plugin now warns if you load the plugin with arguments or a
+  block if the plugin does not accept arguments or block.  This is
+  because a future change to Sequel could break the call.
+
+* When emulating unsupported alter table operations on SQLite, Sequel
+  now copies composite unique constraints unless the alter table
+  operation is the dropping of a unique constraint.
+
+* Sequel now recognizes an additional disconnect error in the oracle
+  adapter.
+
+* In the run_transaction_hooks extension, calling
+  run_after_{commit,rollback}_hooks now raises the correct exception
+  class.
+
+* In the pg_range extension, conversion procs for the tsrange[] and
+  tstzrange[] types are not added unless the Database uses the
+  pg_array extension.
+
+* Multiple unnecessary conditionals in plugins and extensions have
+  been removed.
+
+* Sequel plugin and extension code now have 100% branch coverage.
+
+* Sequel now avoids a statement not reached verbose warning in
+  Dataset#clone.
+
+= Backwards Compatibility
+
+* The output of Dataset#to_dot in the to_dot extension has changed
+  slightly, including hash entries with nil keys.  These entries
+  were previously ignored.

data/doc/testing.rdoc

@@ -162,13 +162,13 @@ SEQUEL_CONNECTION_VALIDATOR :: Use the connection validator extension when runni
 SEQUEL_DUPLICATE_COLUMNS_HANDLER :: Use the duplicate columns handler extension with value given when running the specs
 SEQUEL_ERROR_SQL :: Use the error_sql extension when running the specs
 SEQUEL_INDEX_CACHING :: Use the index_caching extension when running the specs
+SEQUEL_FIBER_CONCURRENCY :: Use the fiber_concurrency extension when running the adapter and integration specs
 SEQUEL_FREEZE_DATABASE :: Freeze the database before running the integration specs
 SEQUEL_IDENTIFIER_MANGLING :: Use the identifier_mangling extension when running the specs
 SEQUEL_INTEGER64 :: Use the integer64 extension when running the adapter or integration specs
 SEQUEL_MODEL_PREPARED_STATEMENTS :: Use the prepared_statements plugin when running the specs
 SEQUEL_MODEL_THROW_FAILURES :: Use the throw_failures plugin when running the specs
 SEQUEL_NO_CACHE_ASSOCIATIONS :: Don't cache association metadata when running the specs
-SEQUEL_NO_CHECK_SQLS :: Don't check for specific SQL syntax when running the specs
 SEQUEL_CHECK_PENDING :: Try running all specs (note, can cause lockups for some adapters), and raise errors for skipped specs that don't fail
 SEQUEL_NO_PENDING :: Don't skip any specs, try running all specs (note, can cause lockups for some adapters)
 SEQUEL_PG_TIMESTAMPTZ :: Use the pg_timestamptz extension when running the postgres specs

data/lib/sequel/adapters/oracle.rb

@@ -14,7 +14,8 @@ module Sequel
       # ORA-02396: exceeded maximum idle time, please connect again
       # ORA-03113: end-of-file on communication channel
       # ORA-03114: not connected to ORACLE
-      CONNECTION_ERROR_CODES = [ 28, 1012, 2396, 3113, 3114 ].freeze
+      # ORA-03135: connection lost contact
+      CONNECTION_ERROR_CODES = [ 28, 1012, 2396, 3113, 3114, 3135 ].freeze
       
       ORACLE_TYPES = {
         :blob=>lambda{|b| Sequel::SQL::Blob.new(b.read)},

data/lib/sequel/adapters/shared/access.rb

@@ -171,6 +171,12 @@ module Sequel
         clone(:into => table)
       end
 
+      # Access uses [] for quoting identifiers, and can't handle
+      # ] inside identifiers.
+      def quoted_identifier_append(sql, v)
+        sql << '[' << v.to_s << ']'
+      end
+
       # Access does not support derived column lists.
       def supports_derived_column_lists?
         false
@@ -279,12 +285,6 @@ module Sequel
           literal_append(sql, l)
         end
       end
-
-      # Access uses [] for quoting identifiers, and can't handle
-      # ] inside identifiers.
-      def quoted_identifier_append(sql, v)
-        sql << '[' << v.to_s << ']'
-      end
     end
   end
 end

data/lib/sequel/adapters/shared/mssql.rb

@@ -775,11 +775,6 @@ module Sequel
         end
       end
 
-      # MSSQL does not allow ordering in sub-clauses unless TOP (limit) is specified
-      def aggregate_dataset
-        (options_overlap(Sequel::Dataset::COUNT_FROM_SELF_OPTS) && !options_overlap([:limit])) ? unordered.from_self : super
-      end
-
       # If the dataset using a order without a limit or offset or custom SQL, 
       # remove the order.  Compounds on Microsoft SQL Server have undefined
       # order unless the result is specifically ordered.  Applying the current
@@ -799,6 +794,11 @@ module Sequel
 
       private
 
+      # MSSQL does not allow ordering in sub-clauses unless TOP (limit) is specified
+      def aggregate_dataset
+        (options_overlap(Sequel::Dataset::COUNT_FROM_SELF_OPTS) && !options_overlap([:limit])) ? unordered.from_self : super
+      end
+
       # Allow update and delete for unordered, limited datasets only.
       def check_not_limited!(type)
         return if @opts[:skip_limit_check] && type != :truncate

data/lib/sequel/adapters/shared/mysql.rb

@@ -717,15 +717,6 @@ module Sequel
         SQL::PlaceholderLiteralString.new((opts[:boolean] ? MATCH_AGAINST_BOOLEAN : MATCH_AGAINST), [Array(cols), terms])
       end
 
-      # Transforms :straight to STRAIGHT_JOIN.
-      def join_type_sql(join_type)
-        if join_type == :straight
-          'STRAIGHT_JOIN'
-        else
-          super
-        end
-      end
-      
       # Sets up the insert methods to use INSERT IGNORE.
       # Useful if you have a unique key and want to just skip
       # inserting rows that violate the unique key restriction.
@@ -956,6 +947,15 @@ module Sequel
         end
       end
 
+      # Transforms :straight to STRAIGHT_JOIN.
+      def join_type_sql(join_type)
+        if join_type == :straight
+          'STRAIGHT_JOIN'
+        else
+          super
+        end
+      end
+      
       # MySQL allows a LIMIT in DELETE and UPDATE statements.
       def limit_sql(sql)
         if l = @opts[:limit]

data/lib/sequel/adapters/shared/oracle.rb

@@ -422,22 +422,6 @@ module Sequel
         end
       end
 
-      def select_limit_sql(sql)
-        return unless supports_fetch_next_rows?
-
-        if offset = @opts[:offset]
-          sql << " OFFSET "
-          literal_append(sql, offset)
-          sql << " ROWS"
-        end
-
-        if limit = @opts[:limit]
-          sql << " FETCH NEXT "
-          literal_append(sql, limit)
-          sql << " ROWS ONLY"
-        end
-      end
-
       # Oracle requires recursive CTEs to have column aliases.
       def recursive_cte_requires_column_aliases?
         true
@@ -624,6 +608,22 @@ module Sequel
         :union
       end
 
+      def select_limit_sql(sql)
+        return unless supports_fetch_next_rows?
+
+        if offset = @opts[:offset]
+          sql << " OFFSET "
+          literal_append(sql, offset)
+          sql << " ROWS"
+        end
+
+        if limit = @opts[:limit]
+          sql << " FETCH NEXT "
+          literal_append(sql, limit)
+          sql << " ROWS ONLY"
+        end
+      end
+
       # Use SKIP LOCKED if skipping locked rows.
       def select_lock_sql(sql)
         super

data/lib/sequel/adapters/shared/postgres.rb

@@ -134,6 +134,96 @@ module Sequel
       end
     end
 
+    # Generator used for creating tables that are partitions of other tables.
+    class CreatePartitionOfTableGenerator
+      MINVALUE = Sequel.lit('MINVALUE').freeze
+      MAXVALUE = Sequel.lit('MAXVALUE').freeze
+
+      def initialize(&block)
+        instance_exec(&block)
+      end
+
+      # The minimum value of the data type used in range partitions, useful
+      # as an argument to #from.
+      def minvalue
+        MINVALUE
+      end
+
+      # The minimum value of the data type used in range partitions, useful
+      # as an argument to #to.
+      def maxvalue
+        MAXVALUE
+      end
+
+      # Assumes range partitioning, sets the inclusive minimum value of the range for
+      # this partition.
+      def from(*v)
+        @from = v
+      end
+
+      # Assumes range partitioning, sets the exclusive maximum value of the range for
+      # this partition.
+      def to(*v)
+        @to = v
+      end
+
+      # Assumes list partitioning, sets the values to be included in this partition.
+      def values_in(*v)
+        @in = v
+      end
+
+      # Assumes hash partitioning, sets the modulus for this parition.
+      def modulus(v)
+        @modulus = v
+      end
+
+      # Assumes hash partitioning, sets the remainder for this parition.
+      def remainder(v)
+        @remainder = v
+      end
+
+      # Sets that this is a default partition, where values not in other partitions
+      # are stored.
+      def default
+        @default = true
+      end
+
+      # The from and to values of this partition for a range partition.
+      def range
+        [@from, @to]
+      end
+
+      # The values to include in this partition for a list partition.
+      def list
+        @in
+      end
+
+      # The modulus and remainder to use for this partition for a hash partition.
+      def hash_values
+        [@modulus, @remainder]
+      end
+
+      # Determine the appropriate partition type for this partition by which methods
+      # were called on it.
+      def partition_type
+        raise Error, "Unable to determine partition type, multiple different partitioning methods called" if [@from || @to, @list, @modulus || @remainder, @default].compact.length > 1
+
+        if @from || @to
+          raise Error, "must call both from and to when creating a partition of a table if calling either" unless @from && @to
+          :range
+        elsif @in
+          :list
+        elsif @modulus || @remainder
+          raise Error, "must call both modulus and remainder when creating a partition of a table if calling either" unless @modulus && @remainder
+          :hash
+        elsif @default
+          :default
+        else
+          raise Error, "unable to determine partition type, no partitioning methods called"
+        end
+      end
+    end
+
     # Error raised when Sequel determines a PostgreSQL exclusion constraint has been violated.
     class ExclusionConstraintViolation < Sequel::ConstraintViolation; end
 
@@ -359,6 +449,26 @@ module Sequel
         self << create_schema_sql(name, opts)
       end
 
+      # Support partitions of tables using the :partition_of option.
+      def create_table(name, options=OPTS, &block)
+        if options[:partition_of]
+          create_partition_of_table_from_generator(name, CreatePartitionOfTableGenerator.new(&block), options)
+          return
+        end
+
+        super
+      end
+
+      # Support partitions of tables using the :partition_of option.
+      def create_table?(name, options=OPTS, &block)
+        if options[:partition_of]
+          create_table(name, options.merge!(:if_not_exists=>true), &block)
+          return
+        end
+
+        super
+      end
+
       # Create a trigger in the database.  Arguments:
       # table :: the table on which this trigger operates
       # name :: the name of this trigger
@@ -840,7 +950,7 @@ module Sequel
       # default value is given.
       def column_definition_default_sql(sql, column)
         super
-        if !column[:serial] && !['serial', 'bigserial'].include?(column[:type].to_s) && !column[:default]
+        if !column[:serial] && !['smallserial', 'serial', 'bigserial'].include?(column[:type].to_s) && !column[:default]
           if (identity = column[:identity])
             sql << " GENERATED "
             sql << (identity == :always ? "ALWAYS" : "BY DEFAULT")
@@ -1018,6 +1128,36 @@ module Sequel
         "CREATE#{' OR REPLACE' if opts[:replace] && server_version >= 90000}#{' TRUSTED' if opts[:trusted]} LANGUAGE #{name}#{" HANDLER #{opts[:handler]}" if opts[:handler]}#{" VALIDATOR #{opts[:validator]}" if opts[:validator]}"
       end
 
+      # Create a partition of another table, used when the create_table with
+      # the :partition_of option is given.
+      def create_partition_of_table_from_generator(name, generator, options)
+        execute_ddl(create_partition_of_table_sql(name, generator, options))
+      end
+
+      # SQL for creating a partition of another table.
+      def create_partition_of_table_sql(name, generator, options)
+        sql = create_table_prefix_sql(name, options).dup
+
+        sql << " PARTITION OF #{quote_schema_table(options[:partition_of])}"
+
+        case generator.partition_type
+        when :range
+          from, to = generator.range
+          sql << " FOR VALUES FROM #{literal(from)} TO #{literal(to)}"
+        when :list
+          sql << " FOR VALUES IN #{literal(generator.list)}"
+        when :hash
+          mod, remainder = generator.hash_values
+          sql << " FOR VALUES WITH (MODULUS #{literal(mod)}, REMAINDER #{literal(remainder)})"
+        when :default
+          sql << " DEFAULT"
+        end
+
+        sql << create_table_suffix_sql(name, options)
+
+        sql
+      end
+
       # SQL for creating a schema.
       def create_schema_sql(name, opts=OPTS)
         "CREATE SCHEMA #{'IF NOT EXISTS ' if opts[:if_not_exists]}#{quote_identifier(name)}#{" AUTHORIZATION #{literal(opts[:owner])}" if opts[:owner]}"
@@ -1039,25 +1179,36 @@ module Sequel
         "CREATE #{prefix_sql}TABLE#{' IF NOT EXISTS' if options[:if_not_exists]} #{options[:temp] ? quote_identifier(name) : quote_schema_table(name)}"
       end
 
+      # SQL for creating a table with PostgreSQL specific options
       def create_table_sql(name, generator, options)
-        sql = super
+        "#{super}#{create_table_suffix_sql(name, options)}"
+      end
+
+      # Handle various PostgreSQl specific table extensions such as inheritance,
+      # partitioning, tablespaces, and foreign tables.
+      def create_table_suffix_sql(name, options)
+        sql = String.new
 
         if inherits = options[:inherits]
-          sql += " INHERITS (#{Array(inherits).map{|t| quote_schema_table(t)}.join(', ')})"
+          sql << " INHERITS (#{Array(inherits).map{|t| quote_schema_table(t)}.join(', ')})"
+        end
+
+        if partition_by = options[:partition_by]
+          sql << " PARTITION BY #{options[:partition_type]||'RANGE'} #{literal(Array(partition_by))}"
         end
 
         if on_commit = options[:on_commit]
           raise(Error, "can't provide :on_commit without :temp to create_table") unless options[:temp]
           raise(Error, "unsupported on_commit option: #{on_commit.inspect}") unless ON_COMMIT.has_key?(on_commit)
-          sql += " ON COMMIT #{ON_COMMIT[on_commit]}"
+          sql << " ON COMMIT #{ON_COMMIT[on_commit]}"
         end
 
         if tablespace = options[:tablespace]
-          sql += " TABLESPACE #{quote_identifier(tablespace)}"
+          sql << " TABLESPACE #{quote_identifier(tablespace)}"
         end
 
         if server = options[:foreign]
-          sql += " SERVER #{quote_identifier(server)}"
+          sql << " SERVER #{quote_identifier(server)}"
           if foreign_opts = options[:options]
             sql << " OPTIONS (#{foreign_opts.map{|k, v| "#{k} #{literal(v.to_s)}"}.join(', ')})"
           end
@@ -1273,6 +1424,10 @@ module Sequel
 
         if server_version > 100000
           ds = ds.select_append{pg_attribute[:attidentity]}
+
+          if server_version > 120000
+            ds = ds.select_append{Sequel.~(pg_attribute[:attgenerated]=>'').as(:generated)}
+          end
         end
 
         ds.map do |row|
@@ -1751,6 +1906,14 @@ module Sequel
         end
       end
 
+      def to_prepared_statement(type, *a)
+        if type == :insert && !@opts.has_key?(:returning)
+          returning(insert_pk).send(:to_prepared_statement, :insert_pk, *a)
+        else
+          super
+        end
+      end
+
       private
 
       # Format TRUNCATE statement with PostgreSQL specific options.
@@ -1965,14 +2128,6 @@ module Sequel
         true
       end
 
-      def to_prepared_statement(type, *a)
-        if type == :insert && !@opts.has_key?(:returning)
-          returning(insert_pk).send(:to_prepared_statement, :insert_pk, *a)
-        else
-          super
-        end
-      end
-
       # Concatenate the expressions with a space in between
       def full_text_string_join(cols)
         cols = Array(cols).map{|x| SQL::Function.new(:COALESCE, x, '')}