sequel 5.39.0 → 5.66.0

data/doc/querying.rdoc

@@ -357,7 +357,9 @@ For ranges, Sequel uses a pair of inequality statements:
   # SELECT * FROM artists WHERE ((id >= 1) AND (id < 5))
 
 Finally, for regexps, Sequel uses an SQL regular expression.  Note that this
-is probably only supported on PostgreSQL and MySQL.
+is only supported by default on PostgreSQL and MySQL.  It can also be supported
+on SQLite when using the sqlite adapter with the :setup_regexp_function
+Database option.
 
   Artist.where(name: /JM$/)
   # SELECT * FROM artists WHERE (name ~ 'JM$')
@@ -498,7 +500,7 @@ filters:
   # SELECT * FROM artists WHERE (id != 5)
   
   Artist.where(id: 5).exclude{name > 'A'}
-  # SELECT * FROM artists WHERE ((id = 5) OR (name <= 'A')
+  # SELECT * FROM artists WHERE ((id = 5) AND (name <= 'A')
 
 So to do a NOT IN with an array:
 
@@ -713,7 +715,7 @@ aggregation:
 
   Album.select_group(:artist_id).select_append{sum(num_tracks).as(tracks)}
   # SELECT artist_id, sum(num_tracks) AS tracks FROM albums GROUP BY artist_id
-
+  
 == Having
 
 The SQL HAVING clause is similar to the WHERE clause, except that

data/doc/release_notes/5.40.0.txt

@@ -0,0 +1,40 @@
+= New Features
+
+* On SQLite 3.33.0+, the UPDATE FROM syntax is now supported. This
+  allows you to update one table based on a join to another table.
+  The SQLite syntax is based on the PostgreSQL syntax, and the
+  Sequel API is the same for both.  You need to pass multiple tables
+  to Dataset#from.  The first table is the table to update, and the
+  remaining tables are used to construct the UPDATE FROM clause:
+
+    DB[:a, :b].where{{a[:c]=>b[:d]}}.update(:e=>'f')
+    # UPDATE a SET e = 'f' FROM b WHERE (a.c = b.d)
+
+  Unlike PostgreSQL, SQLite does not support the deletion of joined
+  datasets.  Related to this, the following methods for testing
+  database support for modifying joined datasets have been added:
+
+  * supports_updating_joins?
+  * supports_deleting_joins?
+
+= Other Improvements
+
+* The pg_interval and date_arithmetic extensions now support
+  ActiveSupport 6.1.
+
+* Sequel no longer issues method redefinition warnings in verbose
+  mode.  As Ruby 3 has dropped uninitialized instance variable
+  warnings, Sequel is now verbose warning free on Ruby 3.
+
+= Backwards Compatibility
+
+* Trying to truncate or insert into a joined dataset now correctly
+  raises an exception even if the joined dataset supports updates.
+
+* The private Dataset#check_modification_allowed! method is now
+  deprecated, and users (custom adapters) should now switch to one
+  of the more specific methods introduced in this version:
+
+  * check_insert_allowed!
+  * check_update_allowed!
+  * check_delete_allowed!

data/doc/release_notes/5.41.0.txt

@@ -0,0 +1,25 @@
+= New Features
+
+* The validation methods added by the validation_helpers plugin now
+  support the :skip_invalid option, which will not add a validation
+  error on a column if it already has a validation error.  This can
+  be useful if you want to avoid having duplicate errors.
+
+* The auto_validations plugin now supports a :skip_invalid plugin
+  option, which will pass the :skip_invalid option when calling
+  validation methods.
+
+= Other Improvements
+
+* The :adder, :remover, and :clearer association options now
+  support keyword arguments in Ruby 2.7+.
+
+* In the pg_interval extension, Sequel now uses the same number of
+  seconds per month and seconds per year as active_support.  It
+  originally used the same number, but active_support changed the
+  values in active_support 5.1.  Sequel now uses the active_support
+  values if they are available.
+
+* When adding a String column on PostgreSQL, an explicit text: true
+  option now takes precedence over an explicit :size option, as it
+  does in Sequel's default behavior.

data/doc/release_notes/5.42.0.txt

@@ -0,0 +1,136 @@
+= New Features
+
+* An async_thread_pool Database extension has been added, which
+  executes queries and processes results using a separate thread
+  pool.  This allows you do do things like:
+
+    foos = DB[:foos].async.all
+    bars = DB[:bars].async.select_map(:name)
+    foo_bars = DB[:foo_bars].async.each{|x| p x}
+
+  and have the three method calls (all, select_map, and each)
+  execute concurrently.  On Ruby implementations without a global
+  VM lock, such as JRuby, it will allow for parallel execution of
+  the method calls.  On CRuby, the main benefit will be for cases
+  where query execution takes a long time or there is significant
+  latency between the application and the database.
+  
+  When you call a method on foos, bars, or foo_bars, if the thread
+  pool hasn't finished processing the method, the calling code will
+  block until the method call has finished.
+
+  By default, for consistency, calling code will not preempt the
+  async thread pool.  For example, if you do:
+
+    DB[:foos].async.all.size
+
+  The calling code will always wait for the async thread pool to
+  run the all method, and then the calling code will call size on
+  the result.  This ensures that async queries will not use the
+  same connection as the the calling thread, even if calling thread
+  has a connection checked out.
+
+  In some cases, such as when the async thread pool is very busy,
+  preemption is desired for performance reasons.  If you set the
+  :preempt_async_thread Database option before loading the
+  async_thread_pool extension, preemption will be allowed.  With
+  preemption allowed, if the async thread pool has not started the
+  processing of the method at the time the calling code needs the
+  results of the method, the calling code will preempt the async
+  thread pool, and run the method on the current thread.
+
+  By default, the async thread pool uses the same number of threads as
+  the Database objects :max_connections attribute (the default for
+  that is 4).  You can modify the number of async threads by setting
+  the :num_async_threads Database option before loading the Database
+  async_thread_pool extension.
+
+  Most Dataset methods that execute queries on the database and return
+  results will operate asynchronously if the the dataset is set to be
+  asynchronous via the Dataset#async method.  This includes most
+  methods available due to the inclusion in Enumerable, even if not
+  defined by Dataset itself.
+
+  There are multiple caveats when using the async_thread_pool
+  extension:
+  
+  * Asynchronous behavior is harder to understand and harder to
+    debug.  It would be wise to only use this support in cases where
+    it provides is significant performance benefit.
+
+  * Dataset methods executed asynchronously will use a separate
+    database connection than the calling thread, so they will not
+    respect transactions in the calling thread, or other cases where
+    the calling thread checks out a connection directly using
+    Database#synchronize.  They will also not respect the use of
+    Database#with_server (from the server_block extension) in the
+    calling thread.
+
+  * Dataset methods executed asynchronously should never ignore their
+    return value.  Code such as:
+
+      DB[:table].async.insert(1)
+
+    is probablematic because without storing the return value, you
+    have no way to block until the insert has been completed.
+
+  * The returned object for Dataset methods executed asynchronously is
+    a proxy object (promise).  So you should never do:
+
+      row = DB[:table].async.first
+      # ...
+      if row
+      end
+
+      # or:
+
+      bool = DB[:table].async.get(:boolean_column)
+      # ...
+      if bool
+      end
+
+    because the if branches will always be taken as row and bool will
+    never be nil or false.  If you want to get the underlying value,
+    call itself on the proxy object (or __value if using Ruby <2.2).
+
+    For the same reason, you should not use the proxy objects directly
+    in case expressions or as arguments to Class#===.  Use itself or
+    __value in those cases.
+
+  * Dataset methods executed asynchronously that include blocks have the
+    block executed asynchronously as well, assuming that the method
+    calls the block.  Because these blocks are executed in a separate
+    thread, you cannot use control flow modifiers such as break or
+    return in them.
+
+* An async_thread_pool model plugin has been added.  This requires the
+  async_thread_pool extension has been loaded into the model's Database
+  object, and allows you to call Model.async instead of
+  Model.dataset.async.  It also adds async support to the destroy,
+  with_pk, and with_pk! model dataset methods.
+
+* Model#to_json_data has been added to the json_serializer plugin, for
+  returning a hash of data that can be converted to JSON, instead of
+  a JSON string.
+
+* A :reject_nil option has been added to the nested_attributes method
+  in the nested_attributes plugin.  This will ignore calls to the
+  nested attributes setter method where nil is passed as the setter
+  method argument.
+
+= Other Improvements
+
+* Model#freeze now works in case where model validation modifies the
+  object beyond adding errors.
+
+* Model#freeze in the composition, serialization, and
+  serialization_modification_detection plugins now works in cases
+  where validation would end up loading the composed or
+  serialized values.
+
+* Database#extension now avoids a possible thread safety issue that
+  could result in the extension being loaded into the Database twice.
+
+* The ado adapter now supports overriding the timestamp conversion
+  proc.  Previously, unlike other conversion procs, the timestamp
+  conversion proc was hard coded and could not be overridden.

data/doc/release_notes/5.43.0.txt

@@ -0,0 +1,98 @@
+= New Features
+
+* A column_encryption plugin has been added to support encrypting the
+  content of individual columns in a table.
+
+  Column values are encrypted with AES-256-GCM using a per-value
+  cipher key derived from a key provided in the configuration using
+  HMAC-SHA256.
+
+  If you would like to support encryption of columns in more than one
+  model, you should probably load the plugin into the parent class of
+  your models and specify the keys:
+ 
+    Sequel::Model.plugin :column_encryption do |enc|
+      enc.key 0, ENV["SEQUEL_COLUMN_ENCRYPTION_KEY"]
+    end
+ 
+  This specifies a single master encryption key.  Unless you are
+  actively rotating keys, it is best to use a single master key.
+ 
+  In the above call, 0 is the id of the key, and
+  ENV["SEQUEL_COLUMN_ENCRYPTION_KEY"] is the content of the key, which
+  must be a string with exactly 32 bytes. As indicated, this key
+  should not be hardcoded or otherwise committed to the source control
+  repository.
+ 
+  For models that need encrypted columns, you load the plugin again,
+  but specify the columns to encrypt:
+ 
+    ConfidentialModel.plugin :column_encryption do |enc|
+      enc.column :encrypted_column_name
+      enc.column :searchable_column_name, searchable: true
+      enc.column :ci_searchable_column_name, searchable: :case_insensitive
+    end
+ 
+  With this, all three specified columns (encrypted_column_name, 
+  searchable_column_name, and ci_searchable_column_name) will be
+  marked as encrypted columns.  When you run the following code:
+ 
+    ConfidentialModel.create(
+      encrypted_column_name: 'These',
+      searchable_column_name: 'will be',
+      ci_searchable_column_name: 'Encrypted'
+    )
+ 
+  It will save encrypted versions to the database.
+  encrypted_column_name will not be searchable, searchable_column_name
+  will be searchable with an exact match, and
+  ci_searchable_column_name will be searchable with a case insensitive
+  match.
+ 
+  To search searchable encrypted columns, use with_encrypted_value.
+  This example code will return the model instance created in the code
+  example in the previous section:
+ 
+    ConfidentialModel.
+      with_encrypted_value(:searchable_column_name, "will be")
+      with_encrypted_value(:ci_searchable_column_name, "encrypted").
+      first
+
+  To rotate encryption keys, add a new key above the existing key,
+  with a new key ID:
+ 
+    Sequel::Model.plugin :column_encryption do |enc|
+      enc.key 1, ENV["SEQUEL_COLUMN_ENCRYPTION_KEY"]
+      enc.key 0, ENV["SEQUEL_OLD_COLUMN_ENCRYPTION_KEY"]
+    end
+ 
+  Newly encrypted data will then use the new key.  Records encrypted
+  with the older key will still be decrypted correctly.
+ 
+  To force reencryption for existing records that are using the older
+  key, you can use the needing_reencryption dataset method and the
+  reencrypt instance method. For a small number of records, you can
+  probably do:
+ 
+    ConfidentialModel.needing_reencryption.all(&:reencrypt)
+ 
+  With more than a small number of records, you'll want to do this in
+  batches.  It's possible you could use an approach such as:
+ 
+    ds = ConfidentialModel.needing_reencryption.limit(100)
+    true until ds.all(&:reencrypt).empty?
+ 
+  After all values have been reencrypted for all models, and no models
+  use the older encryption key, you can remove it from the
+  configuration:
+ 
+    Sequel::Model.plugin :column_encryption do |enc|
+      enc.key 1, ENV["SEQUEL_COLUMN_ENCRYPTION_KEY"]
+    end
+
+  The column_encryption plugin supports encrypting serialized data,
+  as well as enforcing uniquenss of searchable encrypted columns
+  (in the absence of key rotation).  By design, it does not support
+  compression, mixing encrypted and unencrypted data in the same
+  column, or support arbitrary encryption ciphers.  See the plugin
+  documentation for more details.

data/doc/release_notes/5.44.0.txt

@@ -0,0 +1,32 @@
+= New Features
+
+* A concurrent_eager_loading plugin has been added.  This plugin
+  builds on top of the async_thread_pool Database extension and
+  allows eager loading multiple associations concurrently in
+  separate threads.  With this plugin, you can mark datasets for
+  concurrent eager loading using eager_load_concurrently:
+
+    Album.eager_load_concurrently.eager(:artist, :genre, :tracks).all
+
+  Datasets that are marked for concurrent eager loading will use
+  concurrent eager loading if they are eager loading more than one
+  association.  If you would like to make concurrent eager loading
+  the default, you can load the plugin with the :always option.
+
+  All of the association types that ship with Sequel now support
+  concurrent eager loading when using this plugin. For custom eager
+  loaders using the :eager_loader association option, please see the
+  documentation for the plugin for how to enable custom eager loading
+  for them.
+
+= Other Improvements
+
+* The date_arithmetic extension now handles ActiveSupport::Duration
+  values with weeks, as well as :weeks as a key in a hash value. Weeks
+  are converted into 7 days internally.
+
+* The shared SQLite adapter now emulates the dropping of non-composite
+  unique constraints.  Non-composite unique constraints are now
+  treated similarly to composite unique constraints, in that dropping
+  any unique constraints on a table will drop all unique constraints
+  on that table.

data/doc/release_notes/5.45.0.txt

@@ -0,0 +1,34 @@
+= New Features
+
+* A auto_validations_constraint_validations_presence_message plugin
+  has been added that provides integration for the auto_validations
+  and constraint_validations plugin in the following conditions:
+  
+  * The column has a NOT NULL constraint
+  * The column has a presence constraint validation with both
+    the :message and :allow_nil options used.
+
+  In this case, when saving a nil value in the column, the plugin
+  will make it so the more specific message from the presence
+  constraint validation is used, instead of the generic message
+  from auto_validations.
+
+= Other Improvements
+
+* On SQLite 3.35.0+, Sequel now uses ALTER TABLE DROP COLUMN for
+  dropping columns, instead of emulating the dropped column by
+  recreating the table.
+
+* The Dataset#with :materialized option is now supported on SQLite
+  3.35.0+ for specifying whether common table expressions should be
+  materialized.
+
+* The odbc adapter now correct handles boolean columns with NULL
+  values.  Previously, such values were returned as false instead
+  of nil.
+
+= Backwards Compatibility
+
+* The change to use ALTER TABLE DROP COLUMN on SQLite 3.35.0+ can
+  cause backwards compatibility issues if SQLite 3.35.0+ does
+  not allow dropping the column.

data/doc/release_notes/5.46.0.txt

@@ -0,0 +1,87 @@
+= New Features
+
+* An unused_associations plugin has been added, which allows you to
+  determine which associations and association methods are not used.
+  You can use this to avoid defining the unused associations and
+  association methods, which can save memory.
+
+  This plugin is supported on Ruby 2.5+, and uses method coverage to
+  determine if the plugin's methods are called.  Because Sequel::Model
+  adds association methods to an anonymous module included in the
+  class, directly using the method coverage data to determine which
+  associations are used is challenging.
+
+  This plugin is mostly designed for reporting.  You can have a
+  test suite that runs with method coverage enabled, and use the
+  coverage information to get data on unused associations:
+
+    # Calls Coverage.result
+    cov_data = Sequel::Model.update_associations_coverage
+    unused_associations_data = Sequel::Model.update_unused_associations_data(coverage_data: cov_data)
+    Sequel::Model.unused_associations(unused_associations_data: unused_associations_data)
+    # => [["Class1", "assoc1"], ...]
+
+  unused_associations returns an array of two element arrays, where
+  the first element is the class name and the second element is the
+  association name.  The returned values will be associations where
+  all of the association methods are not used.
+
+  In addition to determining which associations are not used, you can
+  also use this to determine if you are defining association methods
+  that are not used:
+
+    Sequel::Model.unused_association_options(unused_associations_data: unused_associations_data)
+    # => [["Class2", "assoc2", {:read_only=>true}], ...]
+
+  unused_association_options is similar to unused_associations, but
+  returns an array of three element arrays, where the third element
+  is a hash of association options that should be used to avoid
+  defining the unused association methods.  It's common in Sequel to
+  define associations and only use them for reading data and not for
+  modifications, and you can use this to easily see which associations
+  are only used for reading data.
+
+  As the determination of whether associations are used is based on
+  method coverage, this will report as unused any associations that are
+  used but where the association methods are not called.  These cases
+  are rare, but can happen if you have libraries that use the
+  association reflection metadata without calling the association
+  methods, or use the association only in combination with another
+  plugin such as dataset_associations.  You can set the :is_used
+  association option to explicitly mark an association as used, and
+  have this plugin avoid reporting it as unused.
+
+  In addition to just reporting on unused associations, you can also
+  directly use the unused associations metadata to automatically avoid
+  defining unused associations or unused associations methods.  You
+  can set a :file option when loading the plugin:
+
+    Sequel::Model.plugin :unused_associations, file: 'unused_associations.json'
+
+  Then run the method coverage testing.  This will save the unused
+  associations metadata to the file.  Then you can use this metadata
+  automatically by also setting the :modify_associations option:
+
+    Sequel::Model.plugin :unused_associations, file: 'unused_associations.json',
+      modify_associations: true
+
+  With the :modify_associations option, unused associations are
+  skipped instead of being defined, and the options returned by
+  unused_association_options are automatically used.  Note that using
+  the :modify_associations option is risky unless you have complete 
+  coverage and do not have cases where the associations are used
+  without calling methods.
+
+  It is common to have multiple test suites where you need to combine
+  coverage.  The plugin supports this by using a :coverage_file option:
+
+    Sequel::Model.plugin :unused_associations, coverage_file: 'unused_associations_coverage.json'
+
+  In this case, you would run update_associations_coverage after each
+  test suite, and update_unused_associations_data only after all test
+  suites have been run.
+
+* Passing nil as the value of the :setter, :adder, :remover, or
+  :clearer association options will cause the related method to not be
+  defined, instead of using the default value.  This allows you to
+  only define the methods you will actually be using.

data/doc/release_notes/5.47.0.txt

@@ -0,0 +1,59 @@
+= New Features
+
+* Sequel now supports using separate queries for each table for both
+  lazy and eager loading of the following associations:
+
+  * many_to_many
+  * one_through_one
+  * many_through_many # many_through_many plugin
+  * one_through_many # many_through_many plugin
+
+  For many_to_many/one_through_one, you specify the :join_table_db
+  association option, which should be a Sequel::Database instance
+  containing the join table.  It is possible for the current table,
+  join table, and associated table all to be in separate databases:
+
+    JOIN_TABLE_DB = Sequel.connect('...')
+    Album.many_to_many :artists, join_table_db: JOIN_TABLE_DB
+
+  For many_through_many/one_through_many, you can use the :db option
+  in each join table specification.  All join tables can be in
+  separate databases:
+
+    JTDB1 = Sequel.connect('...')
+    JTDB2 = Sequel.connect('...')
+    # Tracks on all albums this artist appears on
+    Artist.many_through_many :album_tracks, [
+        {table: :albums_artists, left: :artist_id, right: :album_id, db: JTDB1},
+        {table: :artists, left: :id, right: :id, db: JTDB2}
+      ],
+      class: :Track, right_primary_key: :album_id
+
+* The :allow_eager_graph association option has been added. Setting
+  this option to false will disallow eager loading via #eager_graph.
+  This is useful if you can eager load the association via #eager,
+  but not with #eager_graph.
+
+* The :allow_filtering_by association option has been added. Setting
+  this option to false will disallow the use of filtering by
+  associations for the association.
+
+* Dataset#returning is now supported on SQLite 3.35.0+. To work around
+  bugs in the SQLite implementation, identifiers used in the RETURNING
+  clause are automatically aliased.  Additionally, prepared statements
+  that use the RETURNING clause on SQLite seem to have issues, so the
+  prepared_statements plugin does not automatically use prepared
+  statements on SQLite for queries that use the RETURNING clause.
+
+* Database#rename_tables has been added on MySQL to support renaming
+  multiple tables in the same query.
+
+= Other Improvements
+
+* The unused_associations plugin now tracks access to the association
+  reflection for associations, so it will no longer show an
+  association as completely unused if something is accessing the
+  association reflection for it.  This eliminates most of the false
+  positives, where the plugin would show an association as unused
+  even though something was using it without calling the association
+  methods.

data/doc/release_notes/5.48.0.txt

@@ -0,0 +1,14 @@
+= New Features
+
+* A Sequel::Database#like_without_collate accessor has been added on
+  Microsoft SQL Server, which avoids using the COLLATE clause for
+  LIKE expressions.  This can speed up query performance significantly.
+
+* A private Sequel::Model::Errors#full_message method has been added
+  to make it easier to support internationalization for Sequel::Model
+  error messages.
+
+= Other Improvements
+
+* The association reflection tracking in the unused_associations
+  plugin now works correctly when combining coverage runs.

data/doc/release_notes/5.49.0.txt

@@ -0,0 +1,59 @@
+= New Features
+
+* Model#validates_no_null_byte has been added to the
+  validation_helpers.  It checks that the value being validated does
+  not contain an ASCII NUL ('\0') byte.  Some databases will return an
+  error if a string contains a NUL byte.
+
+  The auto_validations plugin will now automatically add no_null_byte
+  validations for all string columns in the model's table.  This will
+  change exceptions raised by NUL bytes from database errors to
+  validation failures.
+  
+  If you are using auto_validations and would like to have a table
+  accept NUL bytes in string columns, use the following code inside
+  the model:
+
+    skip_auto_validations(:no_null_byte)
+
+* JSONB subscripts are now supported on PostgreSQL 14+ when using the
+  pg_json_ops extension.  You can use JSONB subscripts to more easily
+  update part of a JSONB column:
+
+    DB[:table].update(Sequel.pg_jsonb_op(:column)['key'] => 'value')
+    UPDATE "table" SET "column"['key'] = 'value'
+
+* hstore subscripts are now supported on PostgreSQL 14+ when using the
+  pg_hstore_ops extension.  You can use hstore subscripts to more
+  easily update part of an hstore column:
+
+    DB[:table].update(Sequel.hstore_op(:column)['key'] => 'value')
+    UPDATE "table" SET "column"['key'] = 'value'
+
+* Sequel now supports table aliases for JOIN USING columns on
+  PostgreSQL 14+.  These allow you to reference the USING columns in
+  the query using a qualified identifier. To use this support, pass an
+  SQL::AliasedExpression as the expression to join on:
+
+    DB[:t1].join(:t2, Sequel.as([:c1, :c2], :alias))
+    # SELECT * FROM "t1" INNER JOIN "t2" USING ("c1", "c2") AS "alias"
+
+* Database#create_trigger on PostgreSQL now supports a :replace option
+  for CREATE OR REPLACE TRIGGER (supported in PostgreSQL 14+).
+
+* SQL::Expression#sequel_ast_transform has been added to support
+  AST transforms of custom expression classes.
+
+= Other Improvements
+
+* Sequel now supports calling PostgreSQL procedures without arguments
+  when using Database#call_procedure.  Previously, attempts to call
+  procuredures without arguments would call the procedure with a
+  single NULL argument.
+
+* Sequel now uses defined?(yield) instead of block_given? internally
+  for better performance on CRuby.  defined?(yield) is faster as it is
+  built into the VM, while block_given? is a regular method and has
+  the overhead of calling a regular method.  Note that defined?(yield)
+  is not implemented correctly on JRuby before 9.0.0.0, so this
+  release of Sequel drops support for JRuby versions before 9.0.0.0.