sequel 3.29.0 → 3.30.0

data/CHANGELOG

@@ -1,3 +1,35 @@
+=== 3.30.0 (2011-12-01)
+
+* Handle usage of on_duplicate_key_update in MySQL prepared statements (jeremyevans) (#404)
+
+* Make after_commit and after_rollback respect :server option (jeremyevans) (#401)
+
+* Respect :connect_timeout option in the postgres adapter when using pg (glebpom, jeremyevans) (#402)
+
+* Make Dataset#destroy for model datasets respect dataset shard when using a transaction (jeremyevans)
+
+* Make :server option to Model#save set the shard to use (jeremyevans)
+
+* Move Model#set_server from the sharding plugin to the base plugin (jeremyevans)
+
+* Add :graph_alias_base association option for setting base name to use for table aliases when eager graphing (jeremyevans)
+
+* Make ILIKE work correctly on Microsoft SQL Server if database/column collation is case sensitive (jfirebaugh) (#398)
+
+* When starting a new dataset graph, assume existing selection is the columns to select from the current table (jeremyevans)
+
+* Allow specifying nanoseconds and offsets when converting a hash or array to a timestamp (jeremyevans, jfirebaugh) (#395)
+
+* Improve performance when converting Java types to ruby types in the jdbc adapter (jeremyevans, jfirebaugh) (#395)
+
+* Fix tinytds adapter if DB.identifier_output_method = nil (jeremyevans)
+
+* Explicitly order by the row number column when emulating offsets (jfirebaugh) (#393)
+
+* Fix Dataset#graph and #eager_graph modifying the receiver if the receiver is already graphed (jeremyevans) (#392)
+
+* Change dataset literalization to an append-only-all-the-way-down design (jeremyevans)
+
 === 3.29.0 (2011-11-01)
 
 * Allow Model.dataset_module to take a Module instance (jeremyevans)
@@ -64,7 +96,7 @@
 
 * Add after_commit, after_rollback, after_destroy_commit, and after_destroy_rollback hooks to Model objects (jeremyevans)
 
-* Add after_commit and after_rollback hooks to Database objects (jeremyevans)
+* Add after_commit and after_rollback hooks to Database objects (jeremyevans) (#383)
 
 * Support savepoints inside prepared transactions on MySQL (jeremyevans)
 
@@ -184,9 +216,9 @@
 
 * Fix the db2 adapter so it actually works (jeremyevans)
 
-* Add ibmdb adapter for accessing DB2 (roylez, jeremyevans)
+* Add ibmdb adapter for accessing DB2 (roylez, jeremyevans) (#376)
 
-* Add much better support for DB2 databases (roylez, jeremyevans)
+* Add much better support for DB2 databases (roylez, jeremyevans) (#376)
 
 * Handle SQL::AliasedExpressions and SQL::JoinClauses in Dataset#select_all (jeremyevans)
 

data/Rakefile

@@ -12,7 +12,8 @@ CLEAN.include ["**/.*.sw?", "sequel-*.gem", ".config", "rdoc", "coverage", "www/
 
 desc "Packages sequel"
 task :package=>[:clean] do |p|
-  sh %{gem build sequel.gemspec}
+  load './sequel.gemspec'
+  Gem::Builder.new(SEQUEL_GEMSPEC).build
 end
 
 desc "Install sequel gem"

data/doc/association_basics.rdoc

@@ -1329,6 +1329,17 @@ a JOIN USING or NATURAL JOIN for the graph:
   Artist.one_to_many :albums, :key=>:artist_name,
    :graph_only_conditions=>nil, :graph_join_type=>:natural
 
+==== :graph_alias_base
+
+The base name to use for the table alias when eager graphing.  Defaults to the name
+of the association.  If the alias name has already been used in the query, Sequel will create
+a unique alias by appending a numeric suffix (e.g. alias_0, alias_1, ...) until the alias is
+unique.
+
+This is mostly useful if you have associations with the same name in many models, and you want
+to be able to easily tell which table alias corresponds to which association when eagerly
+graphing multiple associations with the same name.
+
 ==== :eager_grapher
 
 Sets up a custom grapher to use when eager loading the objects via eager_graph.

data/doc/opening_databases.rdoc

@@ -342,6 +342,8 @@ The following additional options are supported:
 
 :charset :: Same as :encoding, :encoding takes precedence
 :encoding :: Set the client_encoding to the given string
+:connect_timeout :: Set the number of seconds to wait for a connection (default 20, only respected
+                    if using the pg library).
 
 === sqlite
 

data/doc/release_notes/3.30.0.txt

@@ -0,0 +1,135 @@
+= Dataset Literalization Refactoring
+
+* As warned about in the 3.29.0 release notes, dataset literalization
+  has been completely refactored.  It now uses an append-only design
+  which is faster in all cases, about twice as fast for large objects
+  and deeply nested structures, and over two orders of magnitude
+  faster in some pathological cases.
+
+  This change should not affect applications, but may affect custom
+  extensions or adapters that dealt with literalization of objects.
+  Most literalization methods now have a method with an _append
+  suffix that does the actual literalization, which takes the sql
+  string to append to as the first argument.  If you were overriding
+  a literalization method, you now probably need to override the
+  _append version instead.  If you have this literalization method:
+
+    def foo_sql(bar)
+      "BAR #{literal(bar.baz)}"
+    end
+
+   You need to change the code to:
+
+    def foo_sql_append(sql, bar)
+      sql << "BAR "
+      literal_append(sql, bar.baz)
+    end
+
+    def foo_sql(bar)
+      sql = ""
+      foo_sql_append(sql, bar)
+      sql
+    end
+
+  If you have questions about modifying your custom adapter or
+  extension, please ask on the Google Group or the IRC channel.
+
+= New Features
+
+* Model#set_server has been added to the base support (it was
+  previously only in the sharding plugin), which allows you to
+  set the shard on which to save/delete the model instance:
+
+    foo1.set_server(:server_a).save
+    foo2.set_server(:server_a).destroy
+
+* Model#save now accepts a :server option that uses set_server
+  to set the shard to use.  Unlike most other #save options, this
+  option persists past the end of the save.  Previously, the
+  :server option only affected the transaction code, it now
+  affects the INSERT/UPDATE statement as well.
+
+* When initiating a new dataset graph, any existing selected
+  columns is assumed to be the columns to select for the graph from
+  the current/master table.  Before, there was not a way to specify
+  the columns to select from the current/master table.
+
+* A :graph_alias_base association option has been added, which is
+  used to set the base alias name to use when eager graphing.  This
+  is mostly useful when cascading eager graphs to dependent
+  associations, where multiple associations with the same name in
+  different models are being graphed simultaneously.
+
+* You can now specify nanoseconds and a timezone offset
+  when converting a hash or array to a timestamp.  The nanoseconds
+  and offset are the 7th and 8th entries in the array, and the :nanos
+  and :offset entry in the hash.
+
+* The postgres adapter now respects a :connect_timeout option if you
+  are using the pg driver.
+
+= Other Improvements
+
+* Type conversion of Java to Ruby types in the JDBC adapter has been
+  made much faster, as conversion method lookup is now
+  O(number of columns) instead of
+  O(number of columns*number of rows).
+
+* Sequel::SQL::Blob literalization is now much faster on adapters that
+  use hex encoding, by switching to String#unpack('H*').
+
+* Database#after_commit and after_rollback now respect the :server
+  option to set the server/shard to use.
+
+* Symbol splitting (e.g. for table__column) is now slightly faster.
+
+* All adapters now pass the dataset :limit/:offset value through
+  Dataset#literal instead of using it verbatim.  Note that
+  Dataset#limit already called to_i on input strings, so this isn't
+  a security issue.  However, the previous code broke if you
+  provided a Sequel-specific object (e.g. Sequel::SQL::Function) as
+  the :limit/:offset value.
+
+* Calling graph and eager_graph on an already graphed dataset no
+  longer modifies the receiver.
+
+* Model#set_server now correctly handles the case where @this is
+  already loaded.
+
+* Dataset#destroy for model datasets now uses the dataset's shard
+  for transactions.
+
+* When emulating offset support using ROW_NUMBER (on Microsoft SQL
+  Server, DB2, and Oracle), explicitly order by the ROW_NUMBER
+  result, as otherwise the results are not guaranteed to be ordered.
+
+* Explicitly force a case insensitive collation when emulating ILIKE
+  on Microsoft SQL Server.  Previously, ILIKE could be case sensitive
+  on Microsoft SQL Server if case sensitive collation was the
+  database default.
+
+* Using on_duplicate_key_update with prepared statements on MySQL now
+  works correctly.
+
+* The tinytds adapter now works correctly if the
+  identifier_output_method is nil.
+
+* The plugin/extension specs were cleaned up using the mock adapter.
+
+= Backwards Compatibility
+
+* In addition to the previously mentioned dataset literalization
+  changes, any custom adapters that overrode *_clause_methods
+  methods need to be modified to add a method that adds the
+  SELECT/UPDATE/INSERT/DELETE.  Previously, this was done by default,
+  but due to common table expressions and the dataset literalization
+  changes, a separate method is now needed.
+
+* Dataset#on_duplicate_key_update_sql has been removed from the shared
+  mysql adapter.
+
+* The :columns dataset option used when inserting is no longer
+  literalized in advance.
+
+* Dataset#as_sql no longer takes an expression, it just takes the
+  alias, and only adds the alias part.

data/doc/testing.rdoc

@@ -20,9 +20,14 @@ Make sure you are using Sequel 3.29.0 or above when using these examples, as old
 
 === RSpec 2
 
-  class Spec::Example::ExampleGroup
-    around do |example|
-      Sequel::Model.db.transaction(:rollback=>:always){example.call}
+  class RSpec::Core::ExampleGroup
+    # Setting an around filter globally doesn't appear to work in 2.7 (and maybe other versions),
+    # so set one up for each subclass.
+    def self.inherited(subclass)
+      super
+      subclass.around do |example|
+        Sequel::Model.db.transaction(:rollback=>:always){example.call}
+      end
     end
   end
 
@@ -35,6 +40,15 @@ Make sure you are using Sequel 3.29.0 or above when using these examples, as old
     end
   end
 
+=== MiniTest::Unit
+
+  # Must use this class as the base class for your tests
+  class SequelTestCase < MiniTest::Unit::TestCase
+    def run(*args, &block)
+      Sequel::Model.db.transaction(:rollback=>:always){super}
+    end
+  end
+
 == Transactional testing with multiple databases
 
 You can use the Sequel.transaction method to run a transaction on multiple databases, rolling all of them back.  Instead of:

data/lib/sequel/adapters/amalgalite.rb

@@ -176,8 +176,8 @@ module Sequel
       private
       
       # Quote the string using the adapter instance method.
-      def literal_string(v)
-        db.synchronize{|c| c.quote(v)}
+      def literal_string_append(sql, v)
+        db.synchronize{|c| sql << c.quote(v)}
       end
     end
   end

data/lib/sequel/adapters/do/mysql.rb

@@ -26,6 +26,9 @@ module Sequel
       # Dataset class for MySQL datasets accessed via DataObjects.
       class Dataset < DataObjects::Dataset
         include Sequel::MySQL::DatasetMethods
+        APOS = Dataset::APOS
+        APOS_RE = Dataset::APOS_RE
+        DOUBLE_APOS = Dataset::DOUBLE_APOS
         
         # Use execute_insert to execute the replace_sql.
         def replace(*args)
@@ -35,8 +38,8 @@ module Sequel
         private
         
         # do_mysql sets NO_BACKSLASH_ESCAPES, so use standard SQL string escaping
-        def literal_string(s)
-          "'#{s.gsub("'", "''")}'"
+        def literal_string_append(sql, s)
+          sql << APOS << s.gsub(APOS_RE, DOUBLE_APOS) << APOS
         end
       end
     end

data/lib/sequel/adapters/ibmdb.rb

@@ -354,8 +354,8 @@ module Sequel
       module CallableStatementMethods
         # Extend given dataset with this module so subselects inside subselects in
         # prepared statements work.
-        def subselect_sql(ds)
-          ps = ds.to_prepared_statement(:select)
+        def subselect_sql_append(sql, ds)
+          ps = ds.to_prepared_statement(:select).clone(:append_sql=>sql)
           ps.extend(CallableStatementMethods)
           ps = ps.bind(@opts[:bind_vars]) if @opts[:bind_vars]
           ps.prepared_args = prepared_args

data/lib/sequel/adapters/jdbc.rb

@@ -588,12 +588,6 @@ module Sequel
       # double performance when fetching rows.
       attr_accessor :convert_types
 
-      # Use the convert_types default setting from the database.
-      def initialize(db, opts={})
-        @convert_types = db.convert_types
-        super
-      end
-      
       # Correctly return rows from the database and return them as hashes.
       def fetch_rows(sql, &block)
         execute(sql){|result| process_result_set(result, &block)}
@@ -613,30 +607,77 @@ module Sequel
       end
       
       private
-      
-      # Convert the type.  Used for converting Java types to ruby types.
-      def convert_type(v)
+
+      # Cache Java class constants to speed up lookups
+      JAVA_SQL_TIMESTAMP    = Java::JavaSQL::Timestamp
+      JAVA_SQL_TIME         = Java::JavaSQL::Time
+      JAVA_SQL_DATE         = Java::JavaSQL::Date
+      JAVA_SQL_BLOB         = Java::JavaSQL::Blob
+      JAVA_SQL_CLOB         = Java::JavaSQL::Clob
+      JAVA_BUFFERED_READER  = Java::JavaIo::BufferedReader
+      JAVA_BIG_DECIMAL      = Java::JavaMath::BigDecimal
+      JAVA_BYTE_ARRAY       = Java::byte[]
+
+      # Handle type conversions for common Java types.
+      class TYPE_TRANSLATOR
+        LF = "\n".freeze
+        def time(v) Sequel.string_to_time(v.to_string) end
+        def date(v) Date.civil(v.getYear + 1900, v.getMonth + 1, v.getDate) end
+        def decimal(v) BigDecimal.new(v.to_string) end
+        def byte_array(v) Sequel::SQL::Blob.new(String.from_java_bytes(v)) end
+        def blob(v) Sequel::SQL::Blob.new(String.from_java_bytes(v.getBytes(1, v.length))) end
+        def clob(v) Sequel::SQL::Blob.new(v.getSubString(1, v.length)) end
+        def buffered_reader(v)
+          lines = ""
+          c = false
+          while(line = v.read_line) do
+            lines << LF if c
+            lines << line
+            c ||= true
+          end
+          lines
+        end
+      end
+      TYPE_TRANSLATOR_INSTANCE = tt = TYPE_TRANSLATOR.new
+
+      # Cache type translator methods so that duplicate Method
+      # objects are not created.
+      DECIMAL_METHOD = tt.method(:decimal)
+      TIME_METHOD = tt.method(:time)
+      DATE_METHOD = tt.method(:date)
+      BUFFERED_READER_METHOD = tt.method(:buffered_reader)
+      BYTE_ARRAY_METHOD = tt.method(:byte_array)
+      BLOB_METHOD = tt.method(:blob)
+      CLOB_METHOD = tt.method(:clob)
+
+      # Convert the given Java timestamp to an instance of Sequel.datetime_class.
+      def convert_type_timestamp(v)
+        db.to_application_timestamp([v.getYear + 1900, v.getMonth + 1, v.getDate, v.getHours, v.getMinutes, v.getSeconds, v.getNanos])
+      end
+
+      # Return a callable object that will convert any value of <tt>v</tt>'s
+      # class to a ruby object.  If no callable object can handle <tt>v</tt>'s
+      # class, return false so that the negative lookup is cached.
+      def convert_type_proc(v)
         case v
-        when Java::JavaSQL::Timestamp
-          db.to_application_timestamp(v.to_string)
-        when Java::JavaSQL::Time
-          Sequel.string_to_time(v.to_string)
-        when Java::JavaSQL::Date
-          Sequel.string_to_date(v.to_string)
-        when Java::JavaIo::BufferedReader
-          lines = []
-          while(line = v.read_line) do lines << line end
-          lines.join("\n")
-        when Java::JavaMath::BigDecimal
-          BigDecimal.new(v.to_string)
-        when Java::byte[]
-          Sequel::SQL::Blob.new(String.from_java_bytes(v))
-        when Java::JavaSQL::Blob
-          convert_type(v.getBytes(1, v.length))
-        when Java::JavaSQL::Clob
-          Sequel::SQL::Blob.new(v.getSubString(1, v.length))
+        when JAVA_BIG_DECIMAL
+          DECIMAL_METHOD
+        when JAVA_SQL_TIMESTAMP
+          method(:convert_type_timestamp)
+        when JAVA_SQL_TIME
+          TIME_METHOD
+        when JAVA_SQL_DATE
+          DATE_METHOD
+        when JAVA_BUFFERED_READER
+          BUFFERED_READER_METHOD
+        when JAVA_BYTE_ARRAY
+          BYTE_ARRAY_METHOD
+        when JAVA_SQL_BLOB
+          BLOB_METHOD
+        when JAVA_SQL_CLOB
+          CLOB_METHOD
         else
-          v
+          false
         end
       end
       
@@ -647,7 +688,7 @@ module Sequel
       
       # Split out from fetch rows to allow processing of JDBC result sets
       # that don't come from issuing an SQL string.
-      def process_result_set(result)
+      def process_result_set(result, &block)
         # get column names
         meta = result.getMetaData
         cols = []
@@ -659,28 +700,70 @@ module Sequel
           columns.delete(rn)
         end
         @columns = columns
-        blk = result_set_object_getter
-        # get rows
+        ct = @convert_types
+        if (ct.nil? ? db.convert_types : ct)
+          cols.each{|c| c << nil}
+          process_result_set_convert(cols, result, rn, &block)
+        else
+          process_result_set_no_convert(cols, result, rn, &block)
+        end
+      ensure
+        result.close
+      end
+
+      # Use conversion procs to convert data retrieved
+      # from the database.  This has been optimized, the algorithm it uses
+      # is roughly, for each column value in each row:
+      # * check if the value is truthy (not false/nil)
+      # * if not truthy, return object
+      # * otherwise, see if a conversion method exists for
+      #   the column.  All columns start with a nil conversion proc,
+      #   since unlike other adapters, Sequel doesn't get the type of
+      #   the column when parsing the column metadata.
+      # * if a conversion proc is not false/nil, call it with the object
+      #   and return the result.
+      # * if a conversion proc has already been looked up and doesn't
+      #   exist (false value), return object.  
+      # * if a conversion proc hasn't been looked up yet (nil value),
+      #   call convert_type_proc to get the conversion method.  Cache
+      #   the result of as the column's conversion proc to speed up
+      #   later processing.  If the conversion proc exists, call it
+      #   and return the result, otherwise, return the object.
+      def process_result_set_convert(cols, result, rn)
         while result.next
           row = {}
-          cols.each do |n, i|
-            row[n] = blk.call(result, n, i)
+          cols.each do |n, i, p|
+            v = result.getObject(i)
+            row[n] = if v
+              if p
+                p.call(v)
+              elsif p.nil?
+                cols[i-1][2] = p = convert_type_proc(v)
+                if p
+                  p.call(v)
+                else
+                  v
+                end
+              else
+                v
+              end
+            else
+              v
+            end
           end
           row.delete(rn) if rn
           yield row
         end
-      ensure
-        result.close
       end
 
-      # Proc that takes the ResultSet and an integer field number and returns
-      # the object for that field.  Respects the @convert_types setting to
-      # convert Java types to ruby types.
-      def result_set_object_getter
-        if @convert_types
-          lambda {|result, n, i| convert_type(result.getObject(i))}
-        else
-          lambda {|result, n, i| result.getObject(i)}
+      # Yield rows without calling any conversion procs.  This
+      # may yield Java values and not ruby values.
+      def process_result_set_no_convert(cols, result, rn)
+        while result.next
+          row = {}
+          cols.each{|n, i| row[n] = result.getObject(i)}
+          row.delete(rn) if rn
+          yield row
         end
       end
     end