sequel 3.17.0 → 3.18.0

data/CHANGELOG

@@ -1,3 +1,25 @@
+=== 3.18.0 (2010-12-01)
+
+* Allow the user to control how the connection pool deals with attempts to access shards that aren't configured (jeremyevans)
+
+* Typecast columns when creating model objects from JSON in the json_serializer plugin (jeremyevans)
+
+* When parsing the schema for a model that uses an aliased table, use the unaliased table name (jeremyevans)
+
+* When emulating schema methods such as drop_column on SQLite, recreate applicable indexes on the recreated table (jeremyevans)
+
+* Only remove hook pairs that have been run successfully in the instance_hooks plugin (jeremyevans)
+
+* Add reversible migration support to the migration extension (jeremyevans)
+
+* Add to_dot extension, for producing visualizations of Dataset abstract syntax trees with Graphviz (jeremyevans)
+
+* Switch to using manual type translation in the SQLite adapter (jeremyevans)
+
+* Support :read_timeout option in the native mysql adapter (tmm1)
+
+* Support :connect_timeout option in the native mysql and mysql2 adapters (tmm1)
+
 === 3.17.0 (2010-11-05)
 
 * Ensure that the optimistic locking plugin increments the lock column when using Model#modified! (jfirebaugh)

data/doc/migration.rdoc

@@ -52,6 +52,40 @@ the change made by up.  However, if you never need to be able to migrate down
 the +down+ block.  In this case, the +down+ block just reverses the changes made by up,
 dropping the table.
 
+You can simplify the migration given above by using a reversible migration with a +change+
+block:
+
+  Sequel.migration do
+    change do
+      create_table(:artists) do
+        primary_key :id
+        String :name, :null=>false
+      end
+    end
+  end
+
+The +change+ block acts exactly like an +up+ block.  The only difference is that
+it will attempt to create a +down+ block for you, assuming that it knows how to
+reverse the given migration.  The +change+ block can usually correctly reverse
+the following methods:
+
+* +create_table+
+* +add_column+
+* +add_index+
+* +rename_column+
+* +rename_table+
+* +alter_table+ (supporting the following methods in the +alter_table+ block):
+  * +add_column+
+  * +add_constraint+
+  * +add_foreign_key+ (with a symbol, not an array)
+  * +add_primary_key+ (with a symbol, not an array)
+  * +add_index+
+  * +add_full_text_index+
+  * +add_spatial_index+
+  * +rename_column+
+
+If you use any other methods, you should create your own +down+ block.
+
 In normal usage, when Sequel's migrator runs, it runs the +up+ blocks for all
 migrations that have not yet been applied.  However, you can use the <tt>-M</tt>
 switch to specify the version to which to migrate, and if it is lower than the

data/doc/release_notes/3.18.0.txt

@@ -0,0 +1,121 @@
+= New Features
+
+* Reversible migration support has been added:
+
+    Sequel.migration do
+      change do
+        create_table(:artists) do
+          primary_key :id
+          String :name, :null=>false
+        end
+      end
+    end
+
+  The change block acts the same way as an up block, except that
+  it automatically creates a down block that reverses the changes.
+  So the above is equivalent to:
+
+    Sequel.migration do
+      up do
+        create_table(:artists) do
+          primary_key :id
+          String :name, :null=>false
+        end
+      end
+      down do
+        drop_table :artists
+      end
+    end
+
+  The following methods are supported in a change block:
+ 
+  * create_table
+  * add_column
+  * add_index
+  * rename_column
+  * rename_table
+  * alter_table (supporting the following methods):
+    * add_column
+    * add_constraint
+    * add_foreign_key (with a symbol, not an array)
+    * add_primary_key (with a symbol, not an array)
+    * add_index
+    * add_full_text_index
+    * add_spatial_index
+    * rename_column
+
+  Use of an other method in a change block will result in the
+  creation of a down block that raises an exception.
+
+* A to_dot extension has been added that adds a Dataset#to_dot
+  method, which returns a string that can be used as input to
+  the graphviz dot program in order to create visualizations
+  of the dataset's abstract syntax tree.  Examples:
+
+  * http://sequel.heroku.com/images/to_dot_simple.gif
+  * http://sequel.heroku.com/images/to_dot_complex.gif
+  * http://imgpaste.com/i/lxngy.gif
+
+  Both the to_dot extension and reversible migrations support
+  were inspired by Aaron Patterson's recent work on ActiveRecord
+  and ARel.
+
+* The user can now control how the connection pool handles attempts
+  to access shards that haven't been configured.  The default is
+  still to assume the :default shard.  However, you can specify a
+  different shard using the :servers_hash option when connecting
+  to the database:
+
+    DB = Sequel.connect(..., :servers_hash=>Hash.new(:some_shard))
+
+  You can also use this feature to raise an exception if an
+  unconfigured shard is used:
+
+    DB = Sequel.connect(..., :servers_hash=>Hash.new{raise ...})
+
+* The mysql and mysql2 adapters now both support the :read_timeout
+  and :connect_timeout options.  read_timeout is the timeout in
+  seconds for reading back results of a query, and connect_timeout
+  is the timeout in seconds before a connection attempt is abandoned.
+
+= Other Improvements
+
+* The json_serializer plugin will now typecast column values for
+  columns with unrestricted setter methods when parsing JSON into
+  model objects.  It now also calls the getter method when creating
+  the JSON, instead of directly taking the values from the underlying
+  hash.
+
+* When parsing the schema for a model with an aliased table name,
+  the unaliased table name is now used.
+
+* The SQLite adapter has been updated to not rely on the native
+  type_translation support, since that will be removed in the next
+  major version of sqlite3-ruby.  Sequel now implements it's own
+  type translation in the sqlite adapter, similarly to how the mysql
+  and postgres adapters handle type translation.
+
+* On SQLite, when emulating natively unsupported schema methods such
+  as drop_column, Sequel will now attempt to recreate applicable
+  indexes on the underlying table.
+
+* A more informative error message is now used when connecting fails
+  when using the jdbc adapter.
+
+* method_missing is no longer removed from Sequel::BasicObject on
+  ruby 1.8.  This should improve compatibility in some cases with
+  Rubinius.
+
+= Backwards Compatibility
+
+* On SQLite, Sequel no longer assumes that a plain integer in a
+  datetime or timestamp field represents a unix epoch time.
+
+* Previously, saving a model object that used the instance_hooks
+  plugin removed all instance hooks.  Now, only the applicable hooks
+  are removed.  So if you save a new object, the update instance
+  hooks won't be removed.  And if you save an existing object, delete
+  instance hooks won't be removed.
+
+* The private Dataset#identifier_list method has been moved into the
+  SQLite adapter, since that is the only place it was used.

data/lib/sequel/adapters/jdbc.rb

@@ -168,7 +168,12 @@ module Sequel
               props.setProperty("user", opts[:user])
               props.setProperty("password", opts[:password])
             end
-            driver.new.connect(args[0], props) rescue (raise e)
+            begin
+              driver.new.connect(args[0], props)
+            rescue => e2
+              e.message << "\n#{e2.class.name}: #{e2.message}"
+              raise e
+            end
           end
         end
         setup_connection(conn)

data/lib/sequel/adapters/mysql.rb

@@ -83,11 +83,15 @@ module Sequel
       #   the MySQL config file.
       # * :config_local_infile - If provided, sets the Mysql::OPT_LOCAL_INFILE
       #   option on the connection with the given value.
+      # * :connect_timeout - Set the timeout in seconds before a connection
+      #   attempt is abandoned.
       # * :encoding - Set all the related character sets for this
       #   connection (connection, client, database, server, and results).
+      # * :read_timeout - Set the timeout in seconds for reading back results
+      #   to a query.
       # * :socket - Use a unix socket file instead of connecting via TCP/IP.
       # * :timeout - Set the timeout in seconds before the server will
-      #   disconnect this connection.
+      #   disconnect this connection (a.k.a @@wait_timeout).
       def connect(server)
         opts = server_opts(server)
         conn = Mysql.init
@@ -99,6 +103,12 @@ module Sequel
           # encoding we want to use, but this can be overridden by READ_DEFAULT_GROUP.
           conn.options(Mysql::SET_CHARSET_NAME, encoding)
         end
+        if read_timeout = opts[:read_timeout] and Mysql::const_defined(Mysql::OPT_READ_TIMEOUT)
+          conn.options(Mysql::OPT_READ_TIMEOUT, read_timeout)
+        end
+        if connect_timeout = opts[:connect_timeout] and Mysql::const_defined(Mysql::OPT_CONNECT_TIMEOUT)
+          conn.options(Mysql::OPT_CONNECT_TIMEOUT, connect_timeout)
+        end
         conn.real_connect(
           opts[:host] || 'localhost',
           opts[:user],

data/lib/sequel/adapters/mysql2.rb

@@ -26,11 +26,13 @@ module Sequel
       #   the MySQL config file.
       # * :config_local_infile - If provided, sets the Mysql::OPT_LOCAL_INFILE
       #   option on the connection with the given value.
+      # * :connect_timeout - Set the timeout in seconds before a connection
+      #   attempt is abandoned.
       # * :encoding - Set all the related character sets for this
       #   connection (connection, client, database, server, and results).
       # * :socket - Use a unix socket file instead of connecting via TCP/IP.
       # * :timeout - Set the timeout in seconds before the server will
-      #   disconnect this connection.
+      #   disconnect this connection (a.k.a. @@wait_timeout).
       def connect(server)
         opts = server_opts(server)
         opts[:host] ||= 'localhost'

data/lib/sequel/adapters/postgres.rb

@@ -93,7 +93,7 @@ module Sequel
 
     # Use a single proc for each type to conserve memory
     PG_TYPE_PROCS  = {
-      [16] =>  lambda{|s| s == 't'}, # boolean
+      [16] => lambda{|s| s == 't'}, # boolean
       [17] => lambda{|s| ::Sequel::SQL::Blob.new(Adapter.unescape_bytea(s))}, # bytea
       [20, 21, 22, 23, 26] => lambda{|s| s.to_i}, # integer
       [700, 701] => lambda{|s| s.to_f}, # float

data/lib/sequel/adapters/shared/sqlite.rb

@@ -237,12 +237,18 @@ module Sequel
 
         qt = quote_schema_table(table)
         bt = quote_identifier(backup_table_name(qt))
-        [
+        a = [
            "CREATE TABLE #{bt}(#{def_columns_str})",
            "INSERT INTO #{bt}(#{dataset.send(:identifier_list, new_columns)}) SELECT #{dataset.send(:identifier_list, old_columns)} FROM #{qt}",
            "DROP TABLE #{qt}",
            "ALTER TABLE #{bt} RENAME TO #{qt}"
         ]
+        indexes(table).each do |name, h|
+          if (h[:columns].map{|x| x.to_s} - new_columns).empty?
+            a << alter_table_sql(table, h.merge(:op=>:add_index, :name=>name))
+          end
+        end
+        a
       end
 
       # SQLite folds unquoted identifiers to lowercase, so it shouldn't need to upcase identifiers on input.
@@ -312,6 +318,7 @@ module Sequel
     # Instance methods for datasets that connect to an SQLite database
     module DatasetMethods
       SELECT_CLAUSE_METHODS = Dataset.clause_methods(:select, %w'distinct columns from join where group having compounds order limit')
+      COMMA_SEPARATOR = ', '.freeze
       CONSTANT_MAP = {:CURRENT_DATE=>"date(CURRENT_TIMESTAMP, 'localtime')".freeze, :CURRENT_TIMESTAMP=>"datetime(CURRENT_TIMESTAMP, 'localtime')".freeze, :CURRENT_TIME=>"time(CURRENT_TIMESTAMP, 'localtime')".freeze}
     
       # SQLite does not support pattern matching via regular expressions.
@@ -388,6 +395,11 @@ module Sequel
         "#{expression} AS #{literal(aliaz.to_s)}"
       end
       
+      # SQL fragment specifying a list of identifiers
+      def identifier_list(columns)
+        columns.map{|i| quote_identifier(i)}.join(COMMA_SEPARATOR)
+      end
+    
       # SQLite uses a preceding X for hex escaping strings
       def literal_blob(v)
         blob = ''

data/lib/sequel/adapters/sqlite.rb

@@ -10,6 +10,23 @@ module Sequel
   # Top level module for holding all SQLite-related modules and classes
   # for Sequel.
   module SQLite
+    UNIX_EPOCH_TIME_FORMAT = /\A\d+\z/.freeze
+    SQLITE_TYPES = {}
+    FALSE_VALUES = %w'0 false f no n'.freeze
+    SQLITE_TYPE_PROCS = {
+      %w'timestamp datetime' => proc{|v| Sequel.database_to_application_timestamp(v)},
+      %w'date' => proc{|v| Sequel.string_to_date(v)},
+      %w'time' => proc{|v| Sequel.string_to_time(v)},
+      %w'bit bool boolean' => proc{|v| !FALSE_VALUES.include?(v.downcase)},
+      %w'integer smallint mediumint int bigint' => proc{|v| Integer(v) rescue v},
+      %w'numeric decimal money' => proc{|v| BigDecimal.new(v) rescue v},
+      %w'float double real dec fixed' + ['double precision'] => proc{|v| Float(v) rescue v},
+      %w'blob' => proc{|v| ::Sequel::SQL::Blob.new(v)}
+    }
+    SQLITE_TYPE_PROCS.each do |k,v|
+      k.each{|n| SQLITE_TYPES[n] = v}
+    end
+    
     # Database class for SQLite databases used with Sequel and the
     # ruby-sqlite3 driver.
     class Database < Sequel::Database
@@ -35,33 +52,9 @@ module Sequel
         opts[:database] = ':memory:' if blank_object?(opts[:database])
         db = ::SQLite3::Database.new(opts[:database])
         db.busy_timeout(opts.fetch(:timeout, 5000))
-        db.type_translation = true
         
         connection_pragmas.each{|s| log_yield(s){db.execute_batch(s)}}
         
-        # Handle datetimes with Sequel.datetime_class
-        prok = proc do |t,v|
-          v = Time.at(v.to_i).iso8601 if UNIX_EPOCH_TIME_FORMAT.match(v)
-          Sequel.database_to_application_timestamp(v)
-        end
-        db.translator.add_translator("timestamp", &prok)
-        db.translator.add_translator("datetime", &prok)
-        
-        # Handle numeric values with BigDecimal
-        prok = proc{|t,v| BigDecimal.new(v) rescue v}
-        db.translator.add_translator("numeric", &prok)
-        db.translator.add_translator("decimal", &prok)
-        db.translator.add_translator("money", &prok)
-        
-        # Handle floating point values with Float
-        prok = proc{|t,v| Float(v) rescue v}
-        db.translator.add_translator("float", &prok)
-        db.translator.add_translator("real", &prok)
-        db.translator.add_translator("double precision", &prok)
-        
-        # Handle blob values with Sequel::SQL::Blob
-        db.translator.add_translator("blob"){|t,v| ::Sequel::SQL::Blob.new(v)}
-
         class << db
           attr_reader :prepared_statements
         end
@@ -271,11 +264,18 @@ module Sequel
       def fetch_rows(sql)
         execute(sql) do |result|
           i = -1
-          cols = result.columns.map{|c| [output_identifier(c), i+=1]}
+          type_procs = result.types.map{|t| SQLITE_TYPES[base_type_name(t)]}
+          cols = result.columns.map{|c| i+=1; [output_identifier(c), i, type_procs[i]]}
           @columns = cols.map{|c| c.first}
           result.each do |values|
             row = {}
-            cols.each{|n,i| row[n] = values[i]}
+            cols.each do |name,i,type_proc|
+              v = values[i]
+              if type_proc && v.is_a?(String)
+                v = type_proc.call(v)
+              end
+              row[name] = v
+            end
             yield row
           end
         end
@@ -297,6 +297,11 @@ module Sequel
       
       private
       
+      # The base type name for a given type, without any parenthetical part.
+      def base_type_name(t)
+        (t =~ /^(.*?)\(/ ? $1 : t).downcase if t
+      end
+
       # Quote the string using the adapter class method.
       def literal_string(v)
         "'#{::SQLite3::Database.quote(v)}'"

data/lib/sequel/connection_pool/sharded_single.rb

@@ -8,10 +8,14 @@ class Sequel::ShardedSingleConnectionPool < Sequel::ConnectionPool
   # * :servers - A hash of servers to use.  Keys should be symbols.  If not
   #   present, will use a single :default server.  The server name symbol will
   #   be passed to the connection_proc.
+  # * :servers_hash - The base hash to use for the servers.  By default,
+  #   Sequel uses Hash.new(:default).  You can use a hash with a default proc
+  #   that raises an error if you want to catch all cases where a nonexistent
+  #   server is used.
   def initialize(opts={}, &block)
     super
     @conns = {}
-    @servers = Hash.new(:default)
+    @servers = opts.fetch(:servers_hash, Hash.new(:default))
     add_servers([:default])
     add_servers(opts[:servers].keys) if opts[:servers]
   end

data/lib/sequel/connection_pool/sharded_threaded.rb

@@ -10,11 +10,15 @@ class Sequel::ShardedThreadedConnectionPool < Sequel::ThreadedConnectionPool
   # * :servers - A hash of servers to use.  Keys should be symbols.  If not
   #   present, will use a single :default server.  The server name symbol will
   #   be passed to the connection_proc.
+  # * :servers_hash - The base hash to use for the servers.  By default,
+  #   Sequel uses Hash.new(:default).  You can use a hash with a default proc
+  #   that raises an error if you want to catch all cases where a nonexistent
+  #   server is used.
   def initialize(opts = {}, &block)
     super
     @available_connections = {}
     @connections_to_remove = []
-    @servers = Hash.new(:default)
+    @servers = opts.fetch(:servers_hash, Hash.new(:default))
     add_servers([:default])
     add_servers(opts[:servers].keys) if opts[:servers]
   end

data/lib/sequel/dataset/sql.rb

@@ -600,11 +600,6 @@ module Sequel
       sprintf(".%06d", usec)
     end
 
-    # SQL fragment specifying a list of identifiers
-    def identifier_list(columns)
-      columns.map{|i| quote_identifier(i)}.join(COMMA_SEPARATOR)
-    end
-    
     # Modify the identifier returned from the database based on the
     # identifier_output_method.
     def input_identifier(v)

data/lib/sequel/extensions/migration.rb

@@ -80,7 +80,7 @@ module Sequel
 
   # Internal class used by the Sequel.migration DSL, part of the +migration+ extension.
   class MigrationDSL < BasicObject
-    # The underlying Migration class.
+    # The underlying Migration instance
     attr_reader :migration
 
     def self.create(&block)
@@ -103,6 +103,122 @@ module Sequel
     def up(&block)
       migration.up = block
     end
+
+    # Creates a reversible migration.  This is the same as creating
+    # the same block with +up+, but it also calls the block and attempts
+    # to create a +down+ block that will reverse the changes made by
+    # the block.
+    #
+    # There are no guarantees that this will work perfectly
+    # in all cases, but it should work for most common cases.
+    def change(&block)
+      migration.up = block
+      migration.down = MigrationReverser.new.reverse(&block)
+    end
+  end
+
+  # Handles the reversing of reversible migrations.  Basically records
+  # supported methods calls, translates them to reversed calls, and
+  # returns them in reverse order.
+  class MigrationReverser < Sequel::BasicObject
+    def initialize
+      @actions = []
+    end
+
+    # Reverse the actions for the given block.  Takes the block given
+    # and returns a new block that reverses the actions taken by
+    # the given block.
+    def reverse(&block)
+      begin
+        instance_eval(&block)
+      rescue
+        just_raise = true
+      end
+      if just_raise
+        Proc.new{raise Sequel::Error, 'irreversible migration method used, you may need to write your own down method'}
+      else
+        actions = @actions.reverse
+        Proc.new do
+          actions.each do |a|
+            if a.last.is_a?(Proc)
+              pr = a.pop
+              send(*a, &pr)
+            else
+              send(*a)
+            end
+          end
+        end
+      end
+    end
+
+    private
+
+    def add_column(*args)
+      @actions << [:drop_column, args[0], args[1]]
+    end
+
+    def add_index(*args)
+      @actions << [:drop_index, *args]
+    end
+
+    def alter_table(table, &block)
+      @actions << [:alter_table, table, MigrationAlterTableReverser.new.reverse(&block)]
+    end
+
+    def create_table(*args)
+      @actions << [:drop_table, args.first]
+    end
+
+    def create_view(*args)
+      @actions << [:drop_view, args.first]
+    end
+
+    def rename_column(table, name, new_name)
+      @actions << [:rename_column, table, new_name, name]
+    end
+
+    def rename_table(table, new_name)
+      @actions << [:rename_table, new_name, table]
+    end
+  end
+
+  # Handles reversing an alter_table block in a reversible migration.
+  class MigrationAlterTableReverser < Sequel::BasicObject
+    def initialize
+      @actions = []
+    end
+
+    def reverse(&block)
+      instance_eval(&block)
+      actions = @actions.reverse
+      Proc.new{actions.each{|a| send(*a)}}
+    end
+
+    private
+
+    def add_column(*args)
+      @actions << [:drop_column, args.first]
+    end
+
+    def add_constraint(*args)
+      @actions << [:drop_constraint, args.first]
+    end
+
+    def add_foreign_key(*args)
+      raise if args.first.is_a?(Array)
+      @actions << [:drop_column, args.first]
+    end
+    alias add_primary_key add_foreign_key
+
+    def add_index(*args)
+      @actions << [:drop_index, *args]
+    end
+    alias add_full_text_index add_index
+    alias add_spatial_index add_index
+
+    def rename_column(name, new_name)
+      @actions << [:rename_column, new_name, name]
+    end
   end
 
   # The preferred method for writing Sequel migrations, using a DSL: