sequel 4.43.0 → 4.44.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: f5f04eb2ea922da521a4e67f1cf8fd4346d034df
-  data.tar.gz: 39ea6ce91593a98832c7e53c03c4ec1ea5607fb8
+  metadata.gz: f527b0eacda71627a6bfed400fce411509a1ce77
+  data.tar.gz: ca7bf3f62b76c4618d59b825d270cca436d54d54
 SHA512:
-  metadata.gz: e449fabbb5ec97c98df590e9afcd6378e392513509dfcb95fbc503cf668029bf31a3e120adf906af3040c695e7998f8ab24d7883409965379ef242498b988b3f
-  data.tar.gz: 2829985707e14c79044a296d34fa62300053e1a279f232da1aa08cd93217eb001ccb78117c049a287b6c370091b8dfb1b5e3a7424d18fdff5ff289081fa2ec58
+  metadata.gz: fb226f39b19f50d82d6eb712c685bb614f77e68e266c898c049bbb1c6345b6347cd661b06a1986be7f1c22add658036dc4f3960966bc2cea30e26e57e0554a36
+  data.tar.gz: 4cb5d97a25593e77119726bba294145c8b5f3ee0503f12e52389c4fbe6f5a4cd4f2b6d7c2f5d592f5a58d3cc827e58824e4f03c282dff0146c2760142401163b

data/CHANGELOG

@@ -1,3 +1,43 @@
+=== 4.44.0 (2017-03-01)
+
+* Add where_all, where_each, where_single_value model dataset methods, optimized for frozen datasets (jeremyevans)
+
+* Add eager method to dataset_module (jeremyevans)
+
+* Add implicit_subquery extension, for implicitly using a subquery for datasets using raw SQL when calling dataset methods that modify SQL (jeremyevans)
+
+* Make Dataset#from_self keep the columns from the current dataset if present (jeremyevans)
+
+* Add implicit_subquery extension, implicitly using subqueries for dataset methods if the current dataset uses raw SQL (jeremyevans)
+
+* Make SQL::ValueList#inspect show that it is a value list (jeremyevans)
+
+* Make LiteralString#inspect show that it is a literal string (jeremyevans)
+
+* Make Model::Associations::AssociationReflection#inspect show reflection class and guess at association definition line (jeremyevans)
+
+* Make SQLTime#inspect show it is an SQLTime instance, and only the time component (jeremyevans)
+
+* Make SQL::Blob#inspect show that it is a blob, the number of bytes, and some or all of the content (jeremyevans)
+
+* Make plugins not modify the constant namespace for the model class that uses them (jeremyevans)
+
+* Do not modify encoding of SQL::Blob instances in force_encoding plugin (jeremyevans)
+
+* Add Model.freeze_descendents to subclasses plugin, for easier finalizing associations/freezing of descendent classes (jeremyevans)
+
+* Add Model.finalize_associations method for finalizing associations, speeding up some association reflections methods almost 10x (jeremyevans)
+
+* Implement Model.freeze such that it can be used in production (jeremyevans)
+
+* Recognize another disconnect error in the jdbc/as400 adapter (perlun) (#1300)
+
+* Correctly handle conversion of false values when typecasting PostgreSQL arrays (mistoo) (#1299)
+
+* Raise error if the postgres adapter attempts to load an incompatible version of sequel_pg (mahlonsmith) (#1298)
+
+* Fix jdbc adapter so basic_type_convertor_map is not shared between instances, work with Database#freeze (jeremyevans)
+
 === 4.43.0 (2017-02-01)
 
 * Make jdbc/postgresql adapter work if pg_hstore extension is loaded first (jeremyevans) (#1296)

data/doc/active_record.rdoc

@@ -297,7 +297,7 @@ Sequel doesn't have a <tt>has_many :through</tt> association, instead you can us
 
 Sequel doesn't come with support for polymorphic associations.  Using polymorphic associations is generally bad from a database design perspective, as it violates referential integrity.  If you have an old database and must have polymorphic associations, there is an external +sequel_polymorphic+ plugin that can handle them, just by using the standard association options provided by Sequel.
 
-Sequel doesn't directly support creating a bunch of associated objects and delaying saving them to the database until the main object is saved, like you can with the <tt>association.build</tt> methods in ActiveRecord.  You can use +before_save or +after_save+ hooks, or the +nested_attributes+ or +instance_hooks+ plugins to get similar support.
+Sequel doesn't directly support creating a bunch of associated objects and delaying saving them to the database until the main object is saved, like you can with the <tt>association.build</tt> methods in ActiveRecord.  You can use +before_save+ or +after_save+ hooks, or the +nested_attributes+ or +instance_hooks+ plugins to get similar support.
 
 Sequel supports the same basic association hooks/callbacks as ActiveRecord.  It also supports <tt>:after_load</tt>, which is run after the associated objects are loaded.  For <tt>*_to_one</tt> associations, it supports +before_set+ and +after_set+ hooks, since a setter method is used instead of an add/remove method pair.
 
@@ -316,7 +316,7 @@ Association datasets are just like any other Sequel dataset, in that you can fil
 
   gold_albums = artist.albums_dataset.where{copies_sold > 500000}.order(:name).all  
 
-Sequel caches associated objects similarly to ActiveRecord, and you can skip the cache by passing +true+ to the association method, just like ActiveRecord.
+Sequel caches associated objects similarly to ActiveRecord, and you can skip the cache by passing +:reload=>true+ to the association method.
 
 === Eager Loading
 

data/doc/code_order.rdoc

@@ -83,6 +83,21 @@ unless the model classes are very simple.  Example:
 
   Dir['./models/*.rb'].each{|f| require f}
 
+== Finalize Associations and Freeze Model Classes and Database
+
+After all the models have been setup, you can finalize the associations.
+This can speed up association reflection methods by doing a lookup in
+advance to find the associated class, and cache related association
+information in the association itself.
+
+Additionally, in production and testing, you should freeze the
+model classes and Database instance, so that you can detect
+unsafe runtime modification of the configuration:
+
+  model_classes.each(&:finalize_associations)
+  model_classes.each(&:freeze)
+  DB.freeze
+
 == Disconnect If Using Forking Webserver with Code Preloading
 
 If you are using a forking webserver such as unicorn or passenger, with

data/doc/dataset_filtering.rdoc

@@ -1,6 +1,6 @@
 = Dataset Filtering
 
-Sequel is very flexibile when it comes to filtering records. You can specify your conditions as a custom string, as a string with parameters, as a hash of values to compare against, or as ruby code that Sequel translates into SQL expressions.
+Sequel is very flexible when it comes to filtering records. You can specify your conditions as a custom string, as a string with parameters, as a hash of values to compare against, or as ruby code that Sequel translates into SQL expressions.
 
 == Filtering using a custom filter string
 

data/doc/model_dataset_method_design.rdoc

@@ -0,0 +1,132 @@
+= Model Dataset Method Design Guide
+
+How you design your model dataset methods can significantly affect the flexibility of your API for your model classes, as well as the performance.  The goal of this guide is to provide an example of how to design your model dataset methods for maximum flexibility and performance.
+
+== Flexibility: Use Single Method Per Task
+
+In general, it is recommended that you have a single method per task for maximum flexibilty.  For example, let's say you need to retrieve all albums released in a given year, ordered by number of units sold descending, and only care about the id, name and number of units sold.  One way to do this is in your application code (outside the model), you can
+call the dataset methods directly:
+
+  Album.
+    select(:id, :name, :copies_sold).
+    where(:release_year=>params[:year].to_i).
+    order(Sequel.desc(:copies_sold)).
+    all
+
+One issue with this design is that it ties you to your current database schema, and will make it necessary to change your application code if your schema changes.  In general, it is better to encapsulate your code into a dataset method (or a class method, but a dataset method is more flexible):
+
+  class Album < Sequel::Model
+    dataset_module do
+      def all_albums_released_in_year(year)
+        select(:id, :name, :copies_sold).
+        where(:release_year=>year).
+        order(Sequel.desc(:copies_sold)).
+        all
+      end
+    end
+  end
+
+Then your application code just needs to call your dataset method:
+
+  Album.all_albums_released_in_year(params[:year].to_i)
+
+The advantage of this approach is that you can change your schema at any point in the future, and you should only need to change your model code, you should never need to change your application code.
+
+== Performance
+
+After designing your dataset methods for flexibility, stop.  Don't worry about performance until you need to worry about performance.  However, assuming you have profiled your application and profiling shows you can benefit from optimizing the above method, you can then consider the performance impact of future design choices.
+
+First, considering that the root cause of the performance issue may not be at the Sequel level, it may be at the database itself.  Use +EXPLAIN+ or the equivalent to analyze the query plan for the query in use, and see if there is something you can do to optimize it, such as adding an appropriate index.
+
+Second, assuming the performance issue is at the Sequel level, you need to understand that one of the best ways to improve performance in most ruby code is to reduce the number of objects allocated.  Here is the above code with comments showing datasets allocated:
+
+  def all_albums_released_in_year(year)
+    select(:id, :name, :copies_sold). # new dataset allocated
+    where(:release_year=>year).       # new dataset allocated
+    order(Sequel.desc(:copies_sold)). # new dataset allocated
+    all
+  end
+
+Third, you need to understand that Sequel has optimizations specifically designed to reduce the number of objects allocated, by caching intermediate data structures.  However, these optimizations apply only to frozen datasets, and can only provide a benefit if related methods are called multiple times on the same dataset.  One easy way to do this is to use frozen model datasets is to use the +freeze_datasets+ Database extension before creating your model class:
+
+  DB.extension :freeze_datasets
+
+Note that freezing datasets is not enough to enable caching in this case.  The reason for this is that +select+, +where+, and +order+ can potentially receive arbitrary arguments, and enabling caching for them could easily lead to unbounded cache size (denial of service due to memory exhaustion).  You need to signal to Sequel that particular arguments to these methods should be cached, and you can do that by calling methods inside +dataset_module+ blocks such as +select+ and +order+.  These methods will add dataset methods to the model that can use caching to optimize performance.  Here is an example using these methods:
+
+  class Album < Sequel::Model
+    dataset_module do
+      select :with_name_and_units, :id, :name, :copies_sold
+      order :by_units_sold, Sequel.desc(:copies_sold)
+
+      def all_albums_released_in_year(year)
+        with_name_and_units.
+        by_units_sold.
+        where(:release_year=>year).
+        all
+      end
+    end
+  end
+
+Performance aside, this does provide a slightly nicer and more readable internal API, though naming such methods can be problematic.
+
+By calling +select+ and +order+ here, Sequel expects that the created dataset methods may be called more than once on the same dataset, and it knows that the arguments to the underlying +select+ and +order+ methods are fixed, so it can cache the resulting datasets.  Let's comment the above example with dataset allocations:
+
+  def all_albums_released_in_year(year)
+    with_name_and_units.        # cached dataset returned 
+    by_units_sold.              # cached dataset returned
+    where(:release_year=>year). # new dataset allocated
+    all
+  end
+
+Note that the order of methods here is important.  If you instead change the method chain to filter the dataset first, then no caching happens:
+
+  def all_albums_released_in_year(year)
+    where(:release_year=>year). # new dataset allocated
+    with_name_and_units.        # new dataset allocated
+    by_units_sold.              # new dataset allocated
+    all
+  end
+
+This is because any time a new, uncached dataset is returned by a dataset method, all subsequent methods in the method chain cannot benefit from caching.
+
+Usually, when you are designing methods to process data based on user input, the user input affects the rows selected, and not the columns selected or the order in which the rows are returned.  Sequel is aware of this and has model dataset methods that specifically take user input (arguments), interpret them as a filter condition and either:
+
+* Return all matching rows in an array (+where_all+)
+* Iterate over all matching rows (+where_each+)
+* Return first matching row (+first+)
+* Return first column in first matching row, assumes only a single column is selected (+where_single_value+)
+
+After calling these methods on a cached dataset a number of times (currently 3), Sequel will automatically build an optimized loader, cache it, and use it for future loads.  So the above example changes to:
+
+  def all_albums_released_in_year(year)
+    with_name_and_units.           # cached dataset returned
+    by_units_sold.                 # cached dataset returned
+    where_all(:release_year=>year) # cached loader used
+  end
+
+This can significantly improve performance, up to 3x for complex method chains that only return a few rows.
+
+So the general advice on designing datasets for performance is:
+
+* Use frozen datasets
+* Use +dataset_module+ methods to create named dataset methods that return cached datasets
+* If any filtering is to be done, have it done last using +where_all+, +where_each+, +first+, or +where_single_value+.
+
+By following this advice, you can significantly increase the performance of your model dataset code.
+
+=== Further Increasing Performance
+
+The best way to further increase performance at the Sequel level is to switch to using prepared statements.  This does require more significant changes to the API.  Here's an example using prepared statements:
+
+  class Album < Sequel::Model
+    ALBUMS_RELEASED_IN_YEAR = select(:id, :name, :copies_sold).
+      where(:release_year=>:$year).
+      order(Sequel.desc(:copies_sold)).
+      prepare(:all, :all_albums_released_in_year)
+
+    def self.all_albums_released_in_year(year)
+      ALBUMS_RELEASED_IN_YEAR.call(:year=>year)
+    end
+  end
+
+Note that when using prepared statements, you need to use a class method instead of a dataset method, as the SQL for the prepared statement must be fixed for the class.  This limits the flexibility of the method, since you can no longer call it on arbitrary datasets on the class.

data/doc/opening_databases.rdoc

@@ -195,8 +195,8 @@ Houses Sequel's JDBC support when running on JRuby.
 Support for individual database types is done using subadapters.
 There are currently subadapters for PostgreSQL, MySQL, SQLite, H2, HSQLDB, Derby,
 Oracle, MSSQL, JTDS, AS400, Progress, Firebird, Informix, and DB2.
-For PostgreSQL, MySQL, SQLite, H2, HSQLDB, Derby, and JTDS,
-this can use the jdbc-* gem, for the others you need to have the .jar in your CLASSPATH
+For PostgreSQL, MySQL, SQLite, H2, HSQLDB, Derby, AS400 and JTDS,
+this can use the `jdbc-*` gem, for the others you need to have the `.jar` in your CLASSPATH
 or load the Java class manually before calling Sequel.connect.
 
 You just use the JDBC connection string directly, which can be specified

data/doc/release_notes/4.44.0.txt

@@ -0,0 +1,125 @@
+= New Features
+
+* Model.freeze is now supported and recommended in production and
+  during testing.  It freezes all class-level metadata, preventing
+  possible thread-safety issues at runtime.
+
+* Model.finalize_associations has been added, speeding up some
+  association reflection methods by about 10x.  This method
+  should be called after all associated models have been loaded.
+  This can speed up the retrieval of associated objects for small
+  datasets by 5-10%.
+
+  One advantage of using this is it will raise an exception if it
+  recognizes that any of your associations are not defined
+  correctly, such as referencing an associated class that doesn't
+  exist.
+
+* Model.freeze_descendents has been added to the subclasses plugin.
+  This method finalizes associations for all descendent classes,
+  then freezes the descendent class.  It's designed to make it
+  easy to freeze all model classes in use:
+
+    Sequel::Model.plugin :subclasses
+    Dir['./models/*.rb'].each{|f| require f}
+    Sequel::Model.freeze_descendents
+
+* An implicit_subquery dataset extension has been added, which
+  implicitly uses a subquery if you have a dataset with raw SQL and
+  you call a method that would modify the SQL used:
+
+    DB['SELECT * FROM foo'].where(:bar=>1)
+    # SELECT * FROM foo
+
+    DB.extension :implicit_subquery
+    DB['SELECT * FROM foo'].where(:bar=>1)
+    # SELECT * FROM (SELECT * FROM foo) AS t1 WHERE (bar = 1)
+
+* Model datasets now have where_all, where_each, and
+  where_single_value methods for returning data:
+
+    class Album < Sequel::Model; end
+    Album.where_all(:id=>[1,2,3])
+    # => [Album[1], Album[3], Album[2]]
+
+    Album.where_each(:id=>[1,2,3]) do |album|
+      # ...
+    end
+
+    Album.select(:name).where_single_value(:id=>1)
+    # "Album's Name"
+
+  These methods are designed for use by other dataset methods you
+  define, and are optimized for frozen datasets if the methods will
+  be called multiple times on the same dataset.  where_all and
+  where_each can increase performance by up to 40% for small datasets
+  compared to where.all and where.each.  where_single_value can be up
+  to twice as fast as where.single_value.
+
+* Model.dataset_module now supports an eager method for eager loading:
+
+    class Album < Sequel::Model
+      many_to_one :artist
+
+      dataset_module do
+        eager :with_artist, :artist
+      end
+    end
+
+    Album.with_artist.all # eagerly loads artist association
+
+= Other Improvements
+
+* The jdbc adapter now supports Database#freeze.  Possible
+  thread-safety issues when initializing multiple jdbc Database
+  instances in separate threads at the same time have been fixed.
+
+* The postgres adapter now raises an exception if it recognizes that
+  the loaded version of sequel_pg is incompatible.
+
+* Sequel classes that are subclasses of core classes now define
+  custom #inspect methods so instances can easily be differentiated from
+  core class instances.  For example:
+
+    Sequel::SQL::Blob.new('a')
+    # => #<Sequel::SQL::Blob:0xa6f3a3c3710 bytes=1 content="a">
+    Sequel::SQLTime.now
+    # => #<Sequel::SQLTime 10:03:06>
+    Sequel::LiteralString.new("foo")
+    # => #<Sequel::LiteralString "foo">
+    class Album < Sequel::Model; end
+    Album.many_to_one :artist
+    # => #<Sequel::Model::Associations::ManyToOneAssociationReflection Album.many_to_one :artist>
+    Sequel::SQL::ValueList.new([[1,2]])
+    # => #<Sequel::SQL::ValueList [[1, 2]]>
+
+* Dataset#from_self now copies the columns from the current dataset
+  if they are present, since wrapping a dataset in a subquery should
+  not change the columns returned.
+
+* On PostgreSQL, array type conversion now correctly handles false
+  values.
+
+* Another disconnect error is now recognized by the jdbc/as400
+  adapter.
+
+* Modifications to Sequel::Model::Associations::ASSOCIATION_TYPES
+  are now thread safe, fixing issues if separate threads attempt
+  to load separate model plugins that modify this hash.
+
+* The force_encoding plugin no longer modifies the encoding of
+  Sequel::SQL::Blob instances.
+
+* Many plugins were updated so they no longer add constants to the
+  namespace of the model that loads them.
+
+= Backwards Compatibility
+
+* Maintainers of external model plugins should update their
+  code to support Model.freeze.
+
+= Upcoming Deprecation
+
+* Starting in Sequel 4.45.0, Sequel will be adding deprecation
+  warnings for features that will be removed or where behavior will
+  change in Sequel 5.

data/lib/sequel/adapters/jdbc.rb

@@ -137,6 +137,10 @@ module Sequel
         BASIC_MAP[types.const_get(type)] = o.method(meth) 
         MAP[types.const_get(type)] = o.method(:"Ruby#{meth}") 
       end
+
+      MAP.freeze
+      BASIC_MAP.freeze
+      INSTANCE.freeze
     end
 
     # JDBC Databases offer a fairly uniform interface that does not change
@@ -672,7 +676,7 @@ module Sequel
       # that runs queries works correctly.  This cannot be overriding in subadapters,
       def setup_type_convertor_map_early
         @type_convertor_map = TypeConvertor::MAP.merge(Java::JavaSQL::Types::TIMESTAMP=>timestamp_convertor)
-        @basic_type_convertor_map = TypeConvertor::BASIC_MAP
+        @basic_type_convertor_map = TypeConvertor::BASIC_MAP.dup
       end
 
       # Yield a new statement object, and ensure that it is closed before returning.

data/lib/sequel/adapters/jdbc/as400.rb

@@ -42,7 +42,7 @@ module Sequel
         private
 
         def disconnect_error?(exception, opts)
-          super || exception.message =~ /\AThe connection does not exist\./
+          super || exception.message =~ /\A(The connection does not exist|Communication link failure)\./
         end
 
         # Use JDBC connection's setAutoCommit to false to start transactions

data/lib/sequel/adapters/postgres.rb

@@ -8,9 +8,9 @@ begin
   # Work around postgres-pr 0.7.0+ which ships with a pg.rb file
   raise LoadError unless defined?(PGconn::CONNECTION_OK)
 
-  SEQUEL_POSTGRES_USES_PG = true
+  Sequel::Postgres::USES_PG = true
 rescue LoadError => e 
-  SEQUEL_POSTGRES_USES_PG = false
+  Sequel::Postgres::USES_PG = false
   begin
     require 'postgres'
     # Attempt to get uniform behavior for the PGconn object no matter
@@ -101,6 +101,13 @@ module Sequel
       def bytea(s) ::Sequel::SQL::Blob.new(Adapter.unescape_bytea(s)) end
     end.new.method(:bytea)
 
+    if Sequel::Postgres::USES_PG
+      # Whether the given sequel_pg version integer is supported.
+      def self.sequel_pg_version_supported?(version)
+        version >= 10617
+      end
+    end
+
     @use_iso_date_format = true
 
     class << self
@@ -139,7 +146,7 @@ module Sequel
       # Hash of prepared statements for this connection.  Keys are
       # string names of the server side prepared statement, and values
       # are SQL strings.
-      attr_reader(:prepared_statements) if SEQUEL_POSTGRES_USES_PG
+      attr_reader(:prepared_statements) if USES_PG
       
       # Raise a Sequel::DatabaseDisconnectError if a one of the disconnect
       # error classes is raised, or a PGError is raised and the connection
@@ -230,7 +237,7 @@ module Sequel
       # are only supported if the pg driver is used.
       def connect(server)
         opts = server_opts(server)
-        if SEQUEL_POSTGRES_USES_PG
+        if USES_PG
           connection_params = {
             :host => opts[:host],
             :port => opts[:port] || 5432,
@@ -312,7 +319,7 @@ module Sequel
         nil
       end
 
-      if SEQUEL_POSTGRES_USES_PG && Object.const_defined?(:PG) && ::PG.const_defined?(:Constants) && ::PG::Constants.const_defined?(:PG_DIAG_SCHEMA_NAME)
+      if USES_PG && Object.const_defined?(:PG) && ::PG.const_defined?(:Constants) && ::PG::Constants.const_defined?(:PG_DIAG_SCHEMA_NAME)
         # Return a hash of information about the related PGError (or Sequel::DatabaseError that
         # wraps a PGError), with the following entries:
         #
@@ -342,7 +349,7 @@ module Sequel
         synchronize(opts[:server]){|conn| check_database_errors{_execute(conn, sql, opts, &block)}}
       end
 
-      if SEQUEL_POSTGRES_USES_PG
+      if USES_PG
         # +copy_table+ uses PostgreSQL's +COPY TO STDOUT+ SQL statement to return formatted
         # results directly to the caller.  This method is only supported if pg is the
         # underlying ruby driver.  This method should only be called if you want
@@ -719,7 +726,7 @@ module Sequel
         clone(:where=>Sequel.lit(['CURRENT OF '], Sequel.identifier(cursor_name)))
       end
 
-      if SEQUEL_POSTGRES_USES_PG
+      if USES_PG
         
         PREPARED_ARG_PLACEHOLDER = LiteralString.new('$').freeze
         
@@ -854,15 +861,17 @@ module Sequel
   end
 end
 
-if SEQUEL_POSTGRES_USES_PG && !ENV['NO_SEQUEL_PG']
+if Sequel::Postgres::USES_PG && !ENV['NO_SEQUEL_PG']
   begin
     require 'sequel_pg'
-  rescue LoadError
-    if RUBY_PLATFORM =~ /mingw|mswin/
-      begin
-        require "#{RUBY_VERSION[0...3]}/sequel_pg"
-      rescue LoadError
-      end
+    if defined?(Gem) &&
+       (sequel_pg_spec = Gem.loaded_specs['sequel_pg'] rescue nil) &&
+       (sequel_pg_spec.version < Gem::Version.new('1.6.17'))
+        raise Sequel::Error, "the installed sequel_pg is too old, please update to at least sequel_pg-1.6.17"
     end
+  rescue LoadError
   end
 end
+
+# SEQUEL5: Remove
+SEQUEL_POSTGRES_USES_PG = Sequel::Postgres::USES_PG