sequel 3.43.0 → 3.44.0

data/CHANGELOG

@@ -1,3 +1,35 @@
+=== 3.44.0 (2013-02-04)
+
+* Speedup mysql2 adapter with identifier output method fetch speed by up to 50% (jeremyevans)
+
+* Speedup tinytds adapter fetch speed by up to 60% (jeremyevans)
+
+* Expand columns_introspection extension to consider cached schema values in the database (jeremyevans)
+
+* Expand columns_introspection extension to handle subselects (jeremyevans)
+
+* Have #last and #paged_each for model datasets order by the model's primary key by default (jeremyevans)
+
+* Improve emulated offset support to handle subqueries (jeremyevans)
+
+* Remove use of Object#extend from the eager_each plugin (jeremyevans)
+
+* Add support for temporary views on SQLite and PostgreSQL via the :temp option to create_view (chanks, jeremyevans)
+
+* Emulate Database#create_or_replace_view if not supported directly (jeremyevans)
+
+* Add Dataset#paged_each, for processing entire datasets without keeping all rows in memory (jeremyevans)
+
+* Add Sequel::ConstraintViolation exception class and subclasses for easier exception handling (jeremyevans)
+
+* Fix use of identity_map plugin with many_to_many associations with right composite keys (chanks) (#603)
+
+* Increase virtual row performance by using a shared VirtualRow instance (jeremyevans)
+
+* Allow the :dataset association option to accept the association reflection as an argument (jeremyevans)
+
+* Improve association method performance by caching intermediate dataset (jeremyevans)
+
 === 3.43.0 (2013-01-08)
 
 * Move the #meta_def support for Database, Dataset, and Model to the meta_def extension (jeremyevans)

data/doc/association_basics.rdoc

@@ -1007,14 +1007,21 @@ in additon to the :clone option.
 ==== :dataset
 
 This is generally only specified for custom associations that aren't based on
-primary/foreign key relationships.  It should be a proc that is instance evaled
+primary/foreign key relationships.  It should be a proc that is instance_execed
 to get the base dataset to use before the other options are applied.
 
+If the proc accepts an argument, it is passed the related association reflection.
+For best performance, it's recommended that custom associations call the
++associated_dataset+ method on the association reflection as the starting point
+for the dataset to return.  The +associated_dataset+ method will return a
+dataset based on the associated class with most of the association options
+already applied, and the proc should return a modified copy of this dataset.
+
 Here's an example of an association of songs to artists through lyrics, where
 the artist can perform any one of four tasks for the lyric:
 
-  Album.one_to_many :songs, :dataset=>(proc do
-    Song.select_all(:songs).
+  Album.one_to_many :songs, :dataset=>(proc do |r|
+    r.associated_dataset.select_all(:songs).
      join(Lyric, :id=>:lyricid,
       id=>[:composer_id, :arranger_id, :vocalist_id, :lyricist_id])
   end)

data/doc/release_notes/3.37.0.txt

@@ -162,7 +162,7 @@
 
     DB.create_table(:t) do
       ...
-      exclusion_constraint([[:col1, '&&'], [:col2, '=']])
+      exclude([[:col1, '&&'], [:col2, '=']])
       # EXCLUDE USING gist (col1 WITH &&, col2 WITH =)
     end
 

data/doc/release_notes/3.44.0.txt

@@ -0,0 +1,152 @@
+= New Features
+
+* Dataset#paged_each has been added, for processing entire datasets
+  without keeping all rows in memory, even if the underlying driver
+  keeps all query results in memory.  This is implemented using
+  limits and offsets, and requires an order (model datasets use a
+  default order by primary key).  It defaults to fetching 1000
+  rows at a time, but that can be changed via the :rows_per_fetch
+  option.
+
+  This method is drop-in compatible for each.  Previously, the
+  pagination extension's each_page method could be used for a
+  similar purpose, but users of each_page are now encouraged to
+  switch to paged_each.
+
+* Sequel now recognizes constraint violation exceptions on most
+  databases, and will raise specific exceptions for different
+  types of constraint violations, instead of the generic
+  Sequel::DatabaseError:
+
+  * Sequel::ConstraintViolation (generic superclass)
+  * Sequel::CheckConstraintViolation
+  * Sequel::NotNullConstraintViolation
+  * Sequel::ForeignKeyConstraintViolation
+  * Sequel::UniqueConstraintViolation
+  * Sequel::Postgres::ExclusionConstraintViolation
+
+* The :dataset association option can now take accept an optional
+  association reflection option.  Instead of doing:
+
+    Album.one_to_many :artists,
+      :dataset=>{Artist...}
+
+  you can now do:
+
+    Album.one_to_many :artists,
+      :dataset=>{|r| r.associated_dataset...}
+
+  This second form will preform better.
+
+* Temporary views are now supported on PostgreSQL and SQLite using
+  the :temp option to create_view.
+
+= Other Improvements
+
+* Row fetching speed in the tinytds adapter has been increased by
+  up to 60%.
+
+* Row fetching speed in the mysql2 adapter when using an identifier
+  output method has been increased by up to 50%.
+
+* On databases where offsets are emulated via the ROW_NUMBER window
+  function (Oracle, DB2, Microsoft SQL Server), using an offset in
+  a subselect is now supported.  For example, the following code
+  previously didn't work correctly with emulated offsets:
+
+    # Second 5 rows ordered by column2 of the second 10 rows ordered
+    # by column 1.
+    DB[:table].order(:column1).limit(10, 10).
+      from_self.order(:column2).limit(5, 5)
+
+  Row processing speed has been increased slightly for all adapters
+  that supported databases where offsets are emulated.
+
+* Association method performance has improved by caching an
+  intermediate dataset.  This can close to triple the performance
+  of the association_dataset method, and increase the performance
+  of the association method by close to 30%.
+
+* Virtual Row performance has increased about 30% in the typical
+  case by using a shared VirtualRow instance.
+
+* Database#create_or_replace_view is now emulated on databases that
+  don't support it directly by dropping the view before attempting
+  to create it.
+
+* The columns_introspection extension can now introspect for simple
+  select * queries from subselects, and it can now use the cached
+  schema information in the database for simple select * queries
+  from tables.
+
+* The identity_map plugin now works correctly with many-to-many
+  right-side composite keys.
+
+* Dataset#last for Model datasets now works even if you don't specify
+  an order explicitly, giving the last entry by primary key.  Note
+  that Dataset#first for model datasets still does not order by
+  default.
+
+* The eager_each plugin no longer uses Object#extend at runtime.
+
+* Database#remove_cached_schema is now thread-safe on non-GVL ruby
+  implementations.
+
+* Connection errors in the jdbc adapter now provide slightly more
+  helpful messages.
+
+* Sequel now uses the standard offset emulation code in the
+  jdbc/as400 adapter, instead of custom offset emulation code
+  specific to that adapter.
+
+* Database#create_view with a dataset now works correctly when using
+  the pg_auto_parameterize extension.
+
+* Database#columns no longer calls the row_proc.
+
+* Dataset#schema_and_table no longer turns a literal string into a
+  non-literal string.
+
+* The oracle adapter now works with a :prefetch_rows=>nil option,
+  which explicitly disables prefetching.
+
+* The mock mssql adapter now sets a server_version so that more
+  parts of it work.
+
+= Backwards Compatibility
+
+* Offset emulation via ROW_NUMBER works by moving the query to a
+  subselect that also selects from the ROW_NUMBER window function,
+  and filtering on the ROW_NUMBER in the main query.  Previously, the
+  ROW_NUMBER was also present in the output columns, and some
+  adapter code was needed to hide that fact.  Now, the outer select
+  selects all of the inner columns in the subselect except for the
+  ROW_NUMBER, reducing the adapter code needed. This has the side
+  effect of potentially requiring a query (or multiple queries for
+  multiple subselects) to determine the columns to use.  The
+  columns_introspection extension may reduce the number of queries
+  needed.
+
+* The correlated_subquery eager limit strategy is no longer supported
+  on Microsoft SQL Server for many_*_many associations.  As the
+  window_function eager limit strategy is supported there, there is
+  no reason to use the correlated_subquery strategy.
+
+* The public AssociationReflection#_dataset_method method has been
+  removed.
+
+* The private _*_dataset methods for associations (e.g.
+  _albums_dataset) have been removed.
+
+* The private Dataset#offset_returns_row_number_column? method has
+  been removed.
+
+* :conditions options for associations are now added to the
+  association dataset before the foreign key filters, instead of
+  after.  This should have no effect unless you were introspecting
+  the dataset's opts or sql and acting on it.
+
+* The added abilities in the columns_introspection plugin to use
+  cached schema for introspection can now cause it to return
+  incorrect results if the table's schema has changed since it was
+  cached by Sequel.

data/lib/sequel/adapters/ado.rb

@@ -132,16 +132,10 @@ module Sequel
       def fetch_rows(sql)
         execute(sql) do |s|
           columns = cols = s.Fields.extend(Enumerable).map{|column| output_identifier(column.Name)}
-          if opts[:offset] && offset_returns_row_number_column?
-            rn = row_number_column
-            columns = columns.dup
-            columns.delete(rn)
-          end
           @columns = columns
           s.getRows.transpose.each do |r|
             row = {}
             cols.each{|c| row[c] = r.shift}
-            row.delete(rn) if rn
             yield row
           end unless s.eof
         end

data/lib/sequel/adapters/db2.rb

@@ -181,12 +181,10 @@ module Sequel
 
       def fetch_rows(sql)
         execute(sql) do |sth|
-          offset = @opts[:offset]
           db = @db
           i = 1
           column_info = get_column_info(sth)
           cols = column_info.map{|c| c.at(1)}
-          cols.delete(row_number_column) if offset
           @columns = cols
           errors = [DB2CLI::SQL_NO_DATA_FOUND, DB2CLI::SQL_ERROR]
           until errors.include?(rc = DB2CLI.SQLFetch(sth))
@@ -202,7 +200,6 @@ module Sequel
                 v
               end
             end
-            row.delete(row_number_column) if offset
             yield row
           end
         end

data/lib/sequel/adapters/dbi.rb

@@ -88,16 +88,10 @@ module Sequel
         execute(sql) do |s|
           begin
             columns = cols = s.column_names.map{|c| output_identifier(c)}
-            if opts[:offset] && offset_returns_row_number_column?
-              rn = row_number_column
-              columns = columns.dup
-              columns.delete(rn)
-            end
             @columns = columns
             s.fetch do |r|
               row = {}
               cols.each{|c| row[c] = r.shift}
-              row.delete(rn) if rn
               yield row
             end
           ensure

data/lib/sequel/adapters/ibmdb.rb

@@ -408,7 +408,6 @@ module Sequel
       # Fetch the rows from the database and yield plain hashes.
       def fetch_rows(sql)
         execute(sql) do |stmt|
-          offset = @opts[:offset]
           columns = []
           convert = convert_smallint_to_bool
           cps = db.conversion_procs
@@ -421,7 +420,6 @@ module Sequel
             columns << [key, cps[type]]
           end
           cols = columns.map{|c| c.at(0)}
-          cols.delete(row_number_column) if offset
           @columns = cols
 
           while res = stmt.fetch_array
@@ -429,7 +427,6 @@ module Sequel
             res.zip(columns).each do |v, (k, pr)|
               row[k] = ((pr ? pr.call(v) : v) if v)
             end
-            row.delete(row_number_column) if offset
             yield row
           end
         end

data/lib/sequel/adapters/jdbc.rb

@@ -237,8 +237,10 @@ module Sequel
               raise(Sequel::DatabaseError, 'driver.new.connect returned nil: probably bad JDBC connection string') unless c
               c
             rescue JavaSQL::SQLException, NativeException, StandardError => e2
-              e.message << "\n#{e2.class.name}: #{e2.message}"
-              raise e
+              unless e2.message == e.message
+                e2.message << "\n#{e.class.name}: #{e.message}"
+              end
+              raise e2
             end
           end
         end
@@ -742,17 +744,13 @@ module Sequel
         i = 0
         meta.getColumnCount.times{cols << [output_identifier(meta.getColumnLabel(i+=1)), i]}
         columns = cols.map{|c| c.at(0)}
-        if opts[:offset] && offset_returns_row_number_column?
-          rn = row_number_column
-          columns.delete(rn)
-        end
         @columns = columns
         ct = @convert_types
         if (ct.nil? ? db.convert_types : ct)
           cols.each{|c| c << nil}
-          process_result_set_convert(cols, result, rn, &block)
+          process_result_set_convert(cols, result, &block)
         else
-          process_result_set_no_convert(cols, result, rn, &block)
+          process_result_set_no_convert(cols, result, &block)
         end
       ensure
         result.close
@@ -776,7 +774,7 @@ module Sequel
       #   the result of as the column's conversion proc to speed up
       #   later processing.  If the conversion proc exists, call it
       #   and return the result, otherwise, return the object.
-      def process_result_set_convert(cols, result, rn)
+      def process_result_set_convert(cols, result)
         while result.next
           row = {}
           cols.each do |n, i, p|
@@ -798,18 +796,16 @@ module Sequel
               v
             end
           end
-          row.delete(rn) if rn
           yield row
         end
       end
 
       # Yield rows without calling any conversion procs.  This
       # may yield Java values and not ruby values.
-      def process_result_set_no_convert(cols, result, rn)
+      def process_result_set_no_convert(cols, result)
         while result.next
           row = {}
           cols.each{|n, i| row[n] = result.getObject(i)}
-          row.delete(rn) if rn
           yield row
         end
       end

data/lib/sequel/adapters/jdbc/as400.rb

@@ -38,29 +38,13 @@ module Sequel
       
       # Dataset class for AS400 datasets accessed via JDBC.
       class Dataset < JDBC::Dataset
+        include EmulateOffsetWithRowNumber
+
         WILDCARD = Sequel::LiteralString.new('*').freeze
         FETCH_FIRST_ROW_ONLY = " FETCH FIRST ROW ONLY".freeze
         FETCH_FIRST = " FETCH FIRST ".freeze
         ROWS_ONLY = " ROWS ONLY".freeze
         
-        # AS400 needs to use a couple of subselects for queries with offsets.
-        def select_sql
-          return super unless o = @opts[:offset]
-          l = @opts[:limit]
-          order = @opts[:order]
-          dsa1 = dataset_alias(1)
-          dsa2 = dataset_alias(2)
-          rn = row_number_column
-          irn = Sequel::SQL::Identifier.new(rn).qualify(dsa2)
-          subselect_sql(unlimited.
-              from_self(:alias=>dsa1).
-              select_more(Sequel::SQL::QualifiedIdentifier.new(dsa1, WILDCARD),
-              Sequel::SQL::WindowFunction.new(SQL::Function.new(:ROW_NUMBER), Sequel::SQL::Window.new(:order=>order)).as(rn)).
-              from_self(:alias=>dsa2).
-              select(Sequel::SQL::QualifiedIdentifier.new(dsa2, WILDCARD)).
-              where(l ? ((irn > o) & (irn <= l + o)) : (irn > o))) # Leave off limit in case of limit(nil, offset)
-        end
-
         # Modify the sql to limit the number of rows returned
         def select_limit_sql(sql)
           if l = @opts[:limit]

data/lib/sequel/adapters/jdbc/derby.rb

@@ -104,6 +104,16 @@ module Sequel
           end
         end
 
+        DATABASE_ERROR_REGEXPS = {
+          /The statement was aborted because it would have caused a duplicate key value in a unique or primary key constraint or unique index/ => UniqueConstraintViolation,
+          /violation of foreign key constraint/ => ForeignKeyConstraintViolation,
+          /The check constraint .+ was violated/ => CheckConstraintViolation,
+          /cannot accept a NULL value/ => NotNullConstraintViolation,
+        }.freeze
+        def database_error_regexps
+          DATABASE_ERROR_REGEXPS
+        end
+
         # Use IDENTITY_VAL_LOCAL() to get the last inserted id.
         def last_insert_id(conn, opts={})
           statement(conn) do |stmt|

data/lib/sequel/adapters/jdbc/h2.rb

@@ -96,6 +96,16 @@ module Sequel
           uri == 'jdbc:h2:mem:' ? o.merge(:max_connections=>1) : o
         end
       
+        DATABASE_ERROR_REGEXPS = {
+          /Unique index or primary key violation/ => UniqueConstraintViolation,
+          /Referential integrity constraint violation/ => ForeignKeyConstraintViolation,
+          /Check constraint violation/ => CheckConstraintViolation,
+          /NULL not allowed for column/ => NotNullConstraintViolation,
+        }.freeze
+        def database_error_regexps
+          DATABASE_ERROR_REGEXPS
+        end
+
         # Use IDENTITY() to get the last inserted id.
         def last_insert_id(conn, opts={})
           statement(conn) do |stmt|

data/lib/sequel/adapters/jdbc/hsqldb.rb

@@ -52,6 +52,16 @@ module Sequel
           "#{create_table_prefix_sql(name, options)} AS (#{sql}) WITH DATA"
         end
 
+        DATABASE_ERROR_REGEXPS = {
+          /integrity constraint violation: unique constraint or index violation/ => UniqueConstraintViolation,
+          /integrity constraint violation: foreign key/ => ForeignKeyConstraintViolation,
+          /integrity constraint violation: check constraint/ => CheckConstraintViolation,
+          /integrity constraint violation: NOT NULL check constraint/ => NotNullConstraintViolation,
+        }.freeze
+        def database_error_regexps
+          DATABASE_ERROR_REGEXPS
+        end
+
         # Use IDENTITY() to get the last inserted id.
         def last_insert_id(conn, opts={})
           statement(conn) do |stmt|