sequel 5.20.0 → 5.49.0

data/doc/release_notes/5.43.0.txt

@@ -0,0 +1,98 @@
+= New Features
+
+* A column_encryption plugin has been added to support encrypting the
+  content of individual columns in a table.
+
+  Column values are encrypted with AES-256-GCM using a per-value
+  cipher key derived from a key provided in the configuration using
+  HMAC-SHA256.
+
+  If you would like to support encryption of columns in more than one
+  model, you should probably load the plugin into the parent class of
+  your models and specify the keys:
+ 
+    Sequel::Model.plugin :column_encryption do |enc|
+      enc.key 0, ENV["SEQUEL_COLUMN_ENCRYPTION_KEY"]
+    end
+ 
+  This specifies a single master encryption key.  Unless you are
+  actively rotating keys, it is best to use a single master key.
+ 
+  In the above call, 0 is the id of the key, and
+  ENV["SEQUEL_COLUMN_ENCRYPTION_KEY"] is the content of the key, which
+  must be a string with exactly 32 bytes. As indicated, this key
+  should not be hardcoded or otherwise committed to the source control
+  repository.
+ 
+  For models that need encrypted columns, you load the plugin again,
+  but specify the columns to encrypt:
+ 
+    ConfidentialModel.plugin :column_encryption do |enc|
+      enc.column :encrypted_column_name
+      enc.column :searchable_column_name, searchable: true
+      enc.column :ci_searchable_column_name, searchable: :case_insensitive
+    end
+ 
+  With this, all three specified columns (encrypted_column_name, 
+  searchable_column_name, and ci_searchable_column_name) will be
+  marked as encrypted columns.  When you run the following code:
+ 
+    ConfidentialModel.create(
+      encrypted_column_name: 'These',
+      searchable_column_name: 'will be',
+      ci_searchable_column_name: 'Encrypted'
+    )
+ 
+  It will save encrypted versions to the database.
+  encrypted_column_name will not be searchable, searchable_column_name
+  will be searchable with an exact match, and
+  ci_searchable_column_name will be searchable with a case insensitive
+  match.
+ 
+  To search searchable encrypted columns, use with_encrypted_value.
+  This example code will return the model instance created in the code
+  example in the previous section:
+ 
+    ConfidentialModel.
+      with_encrypted_value(:searchable_column_name, "will be")
+      with_encrypted_value(:ci_searchable_column_name, "encrypted").
+      first
+
+  To rotate encryption keys, add a new key above the existing key,
+  with a new key ID:
+ 
+    Sequel::Model.plugin :column_encryption do |enc|
+      enc.key 1, ENV["SEQUEL_COLUMN_ENCRYPTION_KEY"]
+      enc.key 0, ENV["SEQUEL_OLD_COLUMN_ENCRYPTION_KEY"]
+    end
+ 
+  Newly encrypted data will then use the new key.  Records encrypted
+  with the older key will still be decrypted correctly.
+ 
+  To force reencryption for existing records that are using the older
+  key, you can use the needing_reencryption dataset method and the
+  reencrypt instance method. For a small number of records, you can
+  probably do:
+ 
+    ConfidentialModel.needing_reencryption.all(&:reencrypt)
+ 
+  With more than a small number of records, you'll want to do this in
+  batches.  It's possible you could use an approach such as:
+ 
+    ds = ConfidentialModel.needing_reencryption.limit(100)
+    true until ds.all(&:reencrypt).empty?
+ 
+  After all values have been reencrypted for all models, and no models
+  use the older encryption key, you can remove it from the
+  configuration:
+ 
+    Sequel::Model.plugin :column_encryption do |enc|
+      enc.key 1, ENV["SEQUEL_COLUMN_ENCRYPTION_KEY"]
+    end
+
+  The column_encryption plugin supports encrypting serialized data,
+  as well as enforcing uniquenss of searchable encrypted columns
+  (in the absence of key rotation).  By design, it does not support
+  compression, mixing encrypted and unencrypted data in the same
+  column, or support arbitrary encryption ciphers.  See the plugin
+  documentation for more details.

data/doc/release_notes/5.44.0.txt

@@ -0,0 +1,32 @@
+= New Features
+
+* A concurrent_eager_loading plugin has been added.  This plugin
+  builds on top of the async_thread_pool Database extension and
+  allows eager loading multiple associations concurrently in
+  separate threads.  With this plugin, you can mark datasets for
+  concurrent eager loading using eager_load_concurrently:
+
+    Album.eager_load_concurrently.eager(:artist, :genre, :tracks).all
+
+  Datasets that are marked for concurrent eager loading will use
+  concurrent eager loading if they are eager loading more than one
+  association.  If you would like to make concurrent eager loading
+  the default, you can load the plugin with the :always option.
+
+  All of the association types that ship with Sequel now support
+  concurrent eager loading when using this plugin. For custom eager
+  loaders using the :eager_loader association option, please see the
+  documentation for the plugin for how to enable custom eager loading
+  for them.
+
+= Other Improvements
+
+* The date_arithmetic extension now handles ActiveSupport::Duration
+  values with weeks, as well as :weeks as a key in a hash value. Weeks
+  are converted into 7 days internally.
+
+* The shared SQLite adapter now emulates the dropping of non-composite
+  unique constraints.  Non-composite unique constraints are now
+  treated similarly to composite unique constraints, in that dropping
+  any unique constraints on a table will drop all unique constraints
+  on that table.

data/doc/release_notes/5.45.0.txt

@@ -0,0 +1,34 @@
+= New Features
+
+* A auto_validations_constraint_validations_presence_message plugin
+  has been added that provides integration for the auto_validations
+  and constraint_validations plugin in the following conditions:
+  
+  * The column has a NOT NULL constraint
+  * The column has a presence constraint validation with both
+    the :message and :allow_nil options used.
+
+  In this case, when saving a nil value in the column, the plugin
+  will make it so the more specific message from the presence
+  constraint validation is used, instead of the generic message
+  from auto_validations.
+
+= Other Improvements
+
+* On SQLite 3.35.0+, Sequel now uses ALTER TABLE DROP COLUMN for
+  dropping columns, instead of emulating the dropped column by
+  recreating the table.
+
+* The Dataset#with :materialized option is now supported on SQLite
+  3.35.0+ for specifying whether common table expressions should be
+  materialized.
+
+* The odbc adapter now correct handles boolean columns with NULL
+  values.  Previously, such values were returned as false instead
+  of nil.
+
+= Backwards Compatibility
+
+* The change to use ALTER TABLE DROP COLUMN on SQLite 3.35.0+ can
+  cause backwards compatibility issues if SQLite 3.35.0+ does
+  not allow dropping the column.

data/doc/release_notes/5.46.0.txt

@@ -0,0 +1,87 @@
+= New Features
+
+* An unused_associations plugin has been added, which allows you to
+  determine which associations and association methods are not used.
+  You can use this to avoid defining the unused associations and
+  association methods, which can save memory.
+
+  This plugin is supported on Ruby 2.5+, and uses method coverage to
+  determine if the plugin's methods are called.  Because Sequel::Model
+  adds association methods to an anonymous module included in the
+  class, directly using the method coverage data to determine which
+  associations are used is challenging.
+
+  This plugin is mostly designed for reporting.  You can have a
+  test suite that runs with method coverage enabled, and use the
+  coverage information to get data on unused associations:
+
+    # Calls Coverage.result
+    cov_data = Sequel::Model.update_associations_coverage
+    unused_associations_data = Sequel::Model.update_unused_associations_data(coverage_data: cov_data)
+    Sequel::Model.unused_associations(unused_associations_data: unused_associations_data)
+    # => [["Class1", "assoc1"], ...]
+
+  unused_associations returns an array of two element arrays, where
+  the first element is the class name and the second element is the
+  association name.  The returned values will be associations where
+  all of the association methods are not used.
+
+  In addition to determining which associations are not used, you can
+  also use this to determine if you are defining association methods
+  that are not used:
+
+    Sequel::Model.unused_association_options(unused_associations_data: unused_associations_data)
+    # => [["Class2", "assoc2", {:read_only=>true}], ...]
+
+  unused_association_options is similar to unused_associations, but
+  returns an array of three element arrays, where the third element
+  is a hash of association options that should be used to avoid
+  defining the unused association methods.  It's common in Sequel to
+  define associations and only use them for reading data and not for
+  modifications, and you can use this to easily see which associations
+  are only used for reading data.
+
+  As the determination of whether associations are used is based on
+  method coverage, this will report as unused any associations that are
+  used but where the association methods are not called.  These cases
+  are rare, but can happen if you have libraries that use the
+  association reflection metadata without calling the association
+  methods, or use the association only in combination with another
+  plugin such as dataset_associations.  You can set the :is_used
+  association option to explicitly mark an association as used, and
+  have this plugin avoid reporting it as unused.
+
+  In addition to just reporting on unused associations, you can also
+  directly use the unused associations metadata to automatically avoid
+  defining unused associations or unused associations methods.  You
+  can set a :file option when loading the plugin:
+
+    Sequel::Model.plugin :unused_associations, file: 'unused_associations.json'
+
+  Then run the method coverage testing.  This will save the unused
+  associations metadata to the file.  Then you can use this metadata
+  automatically by also setting the :modify_associations option:
+
+    Sequel::Model.plugin :unused_associations, file: 'unused_associations.json',
+      modify_associations: true
+
+  With the :modify_associations option, unused associations are
+  skipped instead of being defined, and the options returned by
+  unused_association_options are automatically used.  Note that using
+  the :modify_associations option is risky unless you have complete 
+  coverage and do not have cases where the associations are used
+  without calling methods.
+
+  It is common to have multiple test suites where you need to combine
+  coverage.  The plugin supports this by using a :coverage_file option:
+
+    Sequel::Model.plugin :unused_associations, coverage_file: 'unused_associations_coverage.json'
+
+  In this case, you would run update_associations_coverage after each
+  test suite, and update_unused_associations_data only after all test
+  suites have been run.
+
+* Passing nil as the value of the :setter, :adder, :remover, or
+  :clearer association options will cause the related method to not be
+  defined, instead of using the default value.  This allows you to
+  only define the methods you will actually be using.

data/doc/release_notes/5.47.0.txt

@@ -0,0 +1,59 @@
+= New Features
+
+* Sequel now supports using separate queries for each table for both
+  lazy and eager loading of the following associations:
+
+  * many_to_many
+  * one_through_one
+  * many_through_many # many_through_many plugin
+  * one_through_many # many_through_many plugin
+
+  For many_to_many/one_through_one, you specify the :join_table_db
+  association option, which should be a Sequel::Database instance
+  containing the join table.  It is possible for the current table,
+  join table, and associated table all to be in separate databases:
+
+    JOIN_TABLE_DB = Sequel.connect('...')
+    Album.many_to_many :artists, join_table_db: JOIN_TABLE_DB
+
+  For many_through_many/one_through_many, you can use the :db option
+  in each join table specification.  All join tables can be in
+  separate databases:
+
+    JTDB1 = Sequel.connect('...')
+    JTDB2 = Sequel.connect('...')
+    # Tracks on all albums this artist appears on
+    Artist.many_through_many :album_tracks, [
+        {table: :albums_artists, left: :artist_id, right: :album_id, db: JTDB1},
+        {table: :artists, left: :id, right: :id, db: JTDB2}
+      ],
+      class: :Track, right_primary_key: :album_id
+
+* The :allow_eager_graph association option has been added. Setting
+  this option to false will disallow eager loading via #eager_graph.
+  This is useful if you can eager load the association via #eager,
+  but not with #eager_graph.
+
+* The :allow_filtering_by association option has been added. Setting
+  this option to false will disallow the use of filtering by
+  associations for the association.
+
+* Dataset#returning is now supported on SQLite 3.35.0+. To work around
+  bugs in the SQLite implementation, identifiers used in the RETURNING
+  clause are automatically aliased.  Additionally, prepared statements
+  that use the RETURNING clause on SQLite seem to have issues, so the
+  prepared_statements plugin does not automatically use prepared
+  statements on SQLite for queries that use the RETURNING clause.
+
+* Database#rename_tables has been added on MySQL to support renaming
+  multiple tables in the same query.
+
+= Other Improvements
+
+* The unused_associations plugin now tracks access to the association
+  reflection for associations, so it will no longer show an
+  association as completely unused if something is accessing the
+  association reflection for it.  This eliminates most of the false
+  positives, where the plugin would show an association as unused
+  even though something was using it without calling the association
+  methods.

data/doc/release_notes/5.48.0.txt

@@ -0,0 +1,14 @@
+= New Features
+
+* A Sequel::Database#like_without_collate accessor has been added on
+  Microsoft SQL Server, which avoids using the COLLATE clause for
+  LIKE expressions.  This can speed up query performance significantly.
+
+* A private Sequel::Model::Errors#full_message method has been added
+  to make it easier to support internationalization for Sequel::Model
+  error messages.
+
+= Other Improvements
+
+* The association reflection tracking in the unused_associations
+  plugin now works correctly when combining coverage runs.

data/doc/release_notes/5.49.0.txt

@@ -0,0 +1,59 @@
+= New Features
+
+* Model#validates_no_null_byte has been added to the
+  validation_helpers.  It checks that the value being validated does
+  not contain an ASCII NUL ('\0') byte.  Some databases will return an
+  error if a string contains a NUL byte.
+
+  The auto_validations plugin will now automatically add no_null_byte
+  validations for all string columns in the model's table.  This will
+  change exceptions raised by NUL bytes from database errors to
+  validation failures.
+  
+  If you are using auto_validations and would like to have a table
+  accept NUL bytes in string columns, use the following code inside
+  the model:
+
+    skip_auto_validations(:no_null_byte)
+
+* JSONB subscripts are now supported on PostgreSQL 14+ when using the
+  pg_json_ops extension.  You can use JSONB subscripts to more easily
+  update part of a JSONB column:
+
+    DB[:table].update(Sequel.pg_jsonb_op(:column)['key'] => 'value')
+    UPDATE "table" SET "column"['key'] = 'value'
+
+* hstore subscripts are now supported on PostgreSQL 14+ when using the
+  pg_hstore_ops extension.  You can use hstore subscripts to more
+  easily update part of an hstore column:
+
+    DB[:table].update(Sequel.hstore_op(:column)['key'] => 'value')
+    UPDATE "table" SET "column"['key'] = 'value'
+
+* Sequel now supports table aliases for JOIN USING columns on
+  PostgreSQL 14+.  These allow you to reference the USING columns in
+  the query using a qualified identifier. To use this support, pass an
+  SQL::AliasedExpression as the expression to join on:
+
+    DB[:t1].join(:t2, Sequel.as([:c1, :c2], :alias))
+    # SELECT * FROM "t1" INNER JOIN "t2" USING ("c1", "c2") AS "alias"
+
+* Database#create_trigger on PostgreSQL now supports a :replace option
+  for CREATE OR REPLACE TRIGGER (supported in PostgreSQL 14+).
+
+* SQL::Expression#sequel_ast_transform has been added to support
+  AST transforms of custom expression classes.
+
+= Other Improvements
+
+* Sequel now supports calling PostgreSQL procedures without arguments
+  when using Database#call_procedure.  Previously, attempts to call
+  procuredures without arguments would call the procedure with a
+  single NULL argument.
+
+* Sequel now uses defined?(yield) instead of block_given? internally
+  for better performance on CRuby.  defined?(yield) is faster as it is
+  built into the VM, while block_given? is a regular method and has
+  the overhead of calling a regular method.  Note that defined?(yield)
+  is not implemented correctly on JRuby before 9.0.0.0, so this
+  release of Sequel drops support for JRuby versions before 9.0.0.0.

data/doc/sharding.rdoc

@@ -140,6 +140,8 @@ the shard to use.  This is fairly easy using a Sequel::Model:
   
   Rainbow.plaintext_for_hash("e580726d31f6e1ad216ffd87279e536d1f74e606")
 
+=== :servers_hash Option
+
 The connection pool can be further controlled to change how it handles attempts
 to access shards that haven't been configured.  The default is
 to assume the :default shard.  However, you can specify a

data/doc/sql.rdoc

@@ -428,6 +428,18 @@ As you can see, these literalize with ANDs by default.  You can use the <tt>Sequ
 
   Sequel.or(column1: 1, column2: 2) # (("column1" = 1) OR ("column2" = 2))
 
+As you can see in the above examples, <tt>Sequel.|</tt> and <tt>Sequel.or</tt> work differently.
+<tt>Sequel.|</tt> is for combining an arbitrary number of expressions using OR. If you pass a single
+argument, <tt>Sequel.|</tt> will just convert it to a Sequel expression, similar to <tt>Sequel.expr</tt>.
+<tt>Sequel.or</tt> is for taking a single hash or array of two element arrays and combining the
+elements of that single argument using OR instead of AND:
+
+  Sequel.|(column1: 1, column2: 2) # (("column1" = 1) AND ("column2" = 2))
+  Sequel.or(column1: 1, column2: 2) # (("column1" = 1) OR ("column2" = 2))
+
+  Sequel.|({column1: 1}, {column2: 2}) # (("column1" = 1) OR ("column2" = 2))
+  Sequel.or({column1: 1}, {column2: 2}) # ArgumentError
+
 You've already seen the <tt>Sequel.negate</tt> method, which will use ANDs if multiple entries are used:
 
   Sequel.negate(column1: 1, column2: 2) # (("column1" != 1) AND ("column2" != 2))
@@ -544,7 +556,7 @@ On some databases, you can specify null ordering:
 
 === All Columns (.*)
 
-To select all columns in a table, Sequel supports the * method on identifiers and qualified without an argument:
+To select all columns in a table, Sequel supports the * method on identifiers and qualified identifiers without an argument:
 
   Sequel[:table].*          # "table".*
   Sequel[:schema][:table].* # "schema"."table".*

data/doc/testing.rdoc

@@ -8,17 +8,20 @@ These run each test in its own transaction, the recommended way to test.
 
 === minitest/spec
 
-==== with minitest-hooks
-
+==== with minitest-hooks    
   require 'minitest/hooks/default'
+  
+  DB = Sequel.postgres # change if using sqlite etc
+  
   class Minitest::HooksSpec
     def around
       DB.transaction(:rollback=>:always, :auto_savepoint=>true){super}
     end
   end
 
-==== without minitest-hooks
-
+==== without minitest-hooks  
+  DB = Sequel.postgres # change if using sqlite etc
+  
   class Minitest::Spec
     def run(*args, &block)
       DB.transaction(:rollback=>:always, :auto_savepoint=>true){super}
@@ -26,7 +29,8 @@ These run each test in its own transaction, the recommended way to test.
   end
 
 === minitest/test
-
+  DB = Sequel.postgres # change if using sqlite etc
+  
   # Use this class as the base class for your tests
   class SequelTestCase < Minitest::Test
     def run(*args, &block)
@@ -35,7 +39,8 @@ These run each test in its own transaction, the recommended way to test.
   end
 
 === rspec >= 2.8
-
+  DB = Sequel.postgres # change the database if you are using sqlite etc.
+  
   RSpec.configure do |c|
     c.around(:each) do |example|
       DB.transaction(:rollback=>:always, :auto_savepoint=>true){example.run}
@@ -152,20 +157,28 @@ The SEQUEL_INTEGRATION_URL environment variable specifies the Database connectio
 
 === Other
 
+SEQUEL_ASYNC_THREAD_POOL :: Use the async_thread_pool extension when running the specs
+SEQUEL_ASYNC_THREAD_POOL_PREEMPT :: Use the async_thread_pool extension when running the specs, with the :preempt_async_thread option
 SEQUEL_COLUMNS_INTROSPECTION :: Use the columns_introspection extension when running the specs
 SEQUEL_CONNECTION_VALIDATOR :: Use the connection validator extension when running the specs
 SEQUEL_DUPLICATE_COLUMNS_HANDLER :: Use the duplicate columns handler extension with value given when running the specs
+SEQUEL_CONCURRENT_EAGER_LOADING :: Use the async_thread_pool extension and concurrent_eager_loading plugin when running the specs
 SEQUEL_ERROR_SQL :: Use the error_sql extension when running the specs
 SEQUEL_INDEX_CACHING :: Use the index_caching extension when running the specs
+SEQUEL_FIBER_CONCURRENCY :: Use the fiber_concurrency extension when running the adapter and integration specs
 SEQUEL_FREEZE_DATABASE :: Freeze the database before running the integration specs
 SEQUEL_IDENTIFIER_MANGLING :: Use the identifier_mangling extension when running the specs
 SEQUEL_INTEGER64 :: Use the integer64 extension when running the adapter or integration specs
 SEQUEL_MODEL_PREPARED_STATEMENTS :: Use the prepared_statements plugin when running the specs
 SEQUEL_MODEL_THROW_FAILURES :: Use the throw_failures plugin when running the specs
 SEQUEL_NO_CACHE_ASSOCIATIONS :: Don't cache association metadata when running the specs
-SEQUEL_NO_CHECK_SQLS :: Don't check for specific SQL syntax when running the specs
 SEQUEL_CHECK_PENDING :: Try running all specs (note, can cause lockups for some adapters), and raise errors for skipped specs that don't fail
 SEQUEL_NO_PENDING :: Don't skip any specs, try running all specs (note, can cause lockups for some adapters)
 SEQUEL_PG_TIMESTAMPTZ :: Use the pg_timestamptz extension when running the postgres specs
+SEQUEL_QUERY_PER_ASSOCIATION_DB_0_URL :: Run query-per-association integration tests with multiple databases (all 4 must be set to run)
+SEQUEL_QUERY_PER_ASSOCIATION_DB_1_URL :: Run query-per-association integration tests with multiple databases (all 4 must be set to run)
+SEQUEL_QUERY_PER_ASSOCIATION_DB_2_URL :: Run query-per-association integration tests with multiple databases (all 4 must be set to run)
+SEQUEL_QUERY_PER_ASSOCIATION_DB_3_URL :: Run query-per-association integration tests with multiple databases (all 4 must be set to run)
 SEQUEL_SPLIT_SYMBOLS :: Turn on symbol splitting when running the adapter and integration specs
 SEQUEL_SYNCHRONIZE_SQL :: Use the synchronize_sql extension when running the specs
+SEQUEL_TZINFO_VERSION :: Force the given tzinfo version when running the specs (e.g. '>=2')

data/doc/transactions.rdoc

@@ -143,14 +143,6 @@ If you want the current savepoint to be rolled back when the savepoint block exi
     end # RELEASE SAVEPOINT
   end # COMMIT
 
-If you want the current savepoint to be rolled back when the savepoint block exits (even if an exception is not raised), use <tt>Database#rollback_on_exit(:savepoint=>true)</tt>
-
-  DB.transaction do # BEGIN
-    DB.transaction(savepoint: true) do # SAVEPOINT
-      DB.rollback_on_exit(:savepoint=>true)
-    end # ROLLBACK TO SAVEPOINT
-  end # COMMIT
-
 If you want the current savepoint and potentially enclosing savepoints to be rolled back when the savepoint blocks exit  (even if an exception is not raised), use <tt>Database#rollback_on_exit(:savepoint=>integer)</tt>
 
   DB.transaction do # BEGIN

data/doc/validations.rdoc

@@ -232,7 +232,7 @@ These methods check that the specified attributes can be valid integers or valid
 
 === +validates_operator+
 
-+validates_operator+ checks that a given +operator+ method returns a truthy value when called on attribute with a specified value for comparison.  Generally, this is used for inequality checks (>, >=, etc) but any method that can be called on the attribute that accepts an argument and returns a truthy value may be used.
++validates_operator+ checks that a given +operator+ method returns a truthy value when called on attribute with a specified value for comparison.  Generally, this is used for inequality checks (>, >=, etc.) but any method that can be called on the attribute that accepts an argument and returns a truthy value may be used.
 
   class Album < Sequel::Model
     def validate

data/doc/virtual_rows.rdoc

@@ -54,7 +54,7 @@ methods in the surrounding scope. For example:
   
   # Regular block
   ds.where{|o| o.c > a - b + @d}
-  # WHERE (c > 100)
+  # WHERE (c > 110)
   
   # Instance-evaled block
   ds.where{c > a - b + @d}

data/lib/sequel/adapters/ado/access.rb

@@ -318,7 +318,7 @@ module Sequel
                   conn.OpenSchema(ado_schema.type, ado_schema.criteria) 
                 end
               }
-              yield(r) if block_given?
+              yield(r) if defined?(yield)
             rescue ::WIN32OLERuntimeError => e
               raise_error(e)
             end