sequel 3.7.0 → 3.8.0

data/CHANGELOG

@@ -1,3 +1,53 @@
+=== 3.8.0 (2010-01-04)
+
+* Catch cases in the postgres adapter where exceptions weren't converted or raised appropriately (jeremyevans)
+
+* Don't double escape backslashes in string literals in the mssql shared adapter (john_firebaugh)
+
+* Fix order of ORDER and HAVING clauses in the mssql shared adapter (mluu)
+
+* Add validates_type to the validation_helpers plugin (mluu)
+
+* Attempt to detect database disconnects in the JDBC adapter (john_firebaugh)
+
+* Add Sequel::SQL::Expression#==, so arbtirary expressions can be compared by value (dlee)
+
+* Respect the :size option for the generic File type on MySQL to create tinyblob, mediumblob, and longblob (ibc)
+
+* Don't use the OUTPUT clause on SQL Server versions that don't support it (pre-2005) (jeremyevans) (#281)
+
+* Raise DatabaseConnectionErrors in the single-threaded connection pool if unable to connect (jeremyevans)
+
+* Fix handling of non-existent server in single-threaded connection pool (jeremyevans)
+
+* Default to using mysqlplus driver in the native mysql adapter, fall back to mysql driver (ibc, jeremyevans)
+
+* Handle 64-bit integers in JDBC prepared statements (paulfras)
+
+* Improve blob support when using the H2 JDBC subadapter (nullstyle, jeremyevans, paulfras)
+
+* Add Database#each_server, which yields a new Database object for each server in the connection pool which is connected to only that server (jeremyevans)
+
+* Add Dataset#each_server, which yields a dataset for each server in the connection pool which is will execute on that server (jeremyevans)
+
+* Remove meta_eval and metaclass private methods from Sequel::Metaprogramming (jeremyevans)
+
+* Merge Dataset::FROM_SELF_KEEP_OPTS into Dataset::NON_SQL_OPTIONS (jeremyevans)
+
+* Add Database#remove_servers for removing servers from the pool on the fly (jeremyevans)
+
+* When disconnecting servers, if there are any connections to the server currently in use, schedule them to be disconnected (jeremyevans)
+
+* Allow disconnecting specific server(s)/shard(s) in Database#disconnect via a :servers option (jeremyevans)
+
+* Handle multiple statements in a single query in the native MySQL adapter in all cases, not just when selecting via Dataset#each (jeremyevans)
+
+* In the boolean_readers plugin, don't raise an error if the model's columns can't be determined (jeremyevans)
+
+* In the identity_map plugin, remove instances from the cache if they are deleted/destroyed (jeremyevans)
+
+* Add Database#add_servers, for adding new servers/shards on the fly (chuckremes, jeremyevans)
+
 === 3.7.0 (2009-12-01)
 
 * Add Dataset#sequence to the shared Oracle Adapter, for returning autogenerated primary key values on insert (jeremyevans) (#280)

data/doc/advanced_associations.rdoc

@@ -476,8 +476,8 @@ Let's say you want to store a tree relationship in your database, it's pretty
 simple:
 
   class Node < Sequel::Model
-    many_to_one :parent
-    one_to_many :children, :key=>:parent_id
+    many_to_one :parent, :class=>self
+    one_to_many :children, :key=>:parent_id, :class=>self
   end
   
 You can easily get a node's parent with node.parent, and a node's children with
@@ -492,7 +492,8 @@ What if you want to get all ancestors up to the root node, or all descendents,
 without knowing the depth of the tree?
 
   class Node < Sequel::Model
-    many_to_one :ancestors, :eager_loader=>(proc do |key_hash, nodes, associations|
+    many_to_one :ancestors, :class=>self,
+     :eager_loader=>(proc do |key_hash, nodes, associations|
       # Handle cases where the root node has the same parent_id as primary_key
       # and also when it is NULL
       non_root_nodes = nodes.reject do |n| 

data/doc/release_notes/3.7.0.txt

@@ -97,8 +97,8 @@ New Features
   database could be changed between retrieving the model object and
   updating it.
   
-* The Dataest #union, #intersect, and #except methods now accept an
-  :alias option that it used as the alias for the returned dataset.
+* The Dataset #union, #intersect, and #except methods now accept an
+  :alias option which is used as the alias for the returned dataset.
   
     DB[:table].union(DB[:old_table], :alias=>:table)
 

data/doc/release_notes/3.8.0.txt

@@ -0,0 +1,151 @@
+New Features
+------------
+
+* Dataset#each_server was added, allowing you to run the same query
+  (most likely insert/update/delete) on all shards.  This is useful
+  if you have a sharded database but have lookup tables that should
+  be identical on all shards.  It works by yielding copies of the
+  current dataset that are tied to each server/shard:
+  
+    DB[:table].filter(:id=>1).each_server do |ds|
+      ds.update(:name=>'foo')
+    end
+
+* Database#each_server was added, allowing you to run schema
+  modification methods on all shards.  It works by yielding a
+  new Sequel::Database object for each shard, that will connect to
+  only that shard:
+  
+    DB.each_server do |db|
+      db.create_table(:t){Integer :num}
+    end
+    
+* You can now add and remove servers/shards from the connection
+  pool while Sequel is running:
+  
+    DB.add_servers(:shard1=>{:host=>'s1'}, :shard2=>{:host=>'s2'})
+    DB.remove_servers(:shard1, :shard2)
+    
+* When you attempt to disconnect from a server that has connections
+  currently in use, Sequel will now schedule those connections to
+  be disconnected when they are returned to the pool.  Previously,
+  Sequel disconnected available connections, but ignored connections
+  currently in use, so it wasn't possible to guarantee complete
+  disconnection from the server.  Even with this new feature, you can
+  only guarantee eventual disconnection, since disconnection of
+  connections in use happens asynchronously.
+  
+* Database#disconnect now accepts a :servers option specifying the
+  server(s) from which to disconnect.  This should be a symbol or
+  array of symbols representing servers/shards.  Only those specified
+  will be disconnected:
+  
+    DB.disconnect(:servers=>[:shard1, :shard2])
+
+* A validates_type validation was added to the validation_helpers
+  plugin.  It allows you to check that a given column contains
+  the correct type.  I can be helpful if you are also using the
+  serialization plugin to store serialized ruby objects, by making
+  sure that the objects are of the correct type (e.g. Hash):
+  
+    def validate
+      validates_type(Hash, :options)
+    end
+
+* Sequel::SQL::Expression#== is now supported for all expressions:
+
+    :column.qualify(:table).cast(:type) == \
+      :column.qualify(:table).cast(:type)
+    # => true
+    :column.qualify(:table).cast(:type) == \
+      :other_column.qualify(:table).cast(:type)
+    # => false
+
+* When using the generic File type to create blob columns on
+  MySQL, you can specify the specific database type by using the
+  :size option (with :tiny, :medium, and :long values recognized):
+  
+    DB.create_table(:docs){File :body, :size=>:long} # longblob
+
+* The mysql adapter will now default to using mysqlplus, falling
+  back to use mysql.  mysqlplus is significantly better for threaded
+  code because queries do not block the entire interpreter.
+
+* The JDBC adapter is now able to detect certain types of disconnect
+  errors.
+  
+* ConnectionPool.servers and Database.servers were added, which
+  return an array of symbols specifying the servers/shards in use.
+
+Other Improvements
+------------------
+
+* The single-threaded connection pool now raises
+  DatabaseConnectionErrors if unable to connect, so it now operates
+  more similarly to the default connection pool.
+
+* The single-threaded connection pool now operates more similar
+  to the default connection pool when given a nonexistent server.
+
+* PGErrors are now correctly converted to DatabaseErrors in the
+  postgres adapter when preparing statements or executing prepared
+  statements.
+  
+* DatabaseDisconnectErrors are now raised correctly in the postgres
+  adapter if the connection status is not OK after a query raises an
+  error.
+
+* In the mysql adapter, multiple statements in a single query should
+  now be handled correctly in the all cases, not just when using
+  Dataset#each.  So you can now submit multiple queries in a single
+  string to Database#run.
+
+* Model object creation on Microsoft SQL Server 2000 once again
+  works correctly.  Previously, an optimization was used that was
+  only supported on 2005+.
+
+* Backslashes are no longer doubled inside string literals when
+  connecting to Microsoft SQL Server.
+  
+* The ORDER clause now correctly comes after the HAVING clause on
+  Microsoft SQL Server.
+  
+* Sequel now checks that there is an active transaction before
+  rolling back transactions on Microsoft SQL Server, since
+  there are cases where Microsoft SQL Server will roll back
+  transactions implicitly.
+  
+* Blobs are now handled correctly when connecting to H2.
+
+* 64-bit integers are now handled correctly in JDBC prepared
+  statements.
+
+* In the boolean_readers plugin, correctly handle columns not in
+  the db_schema, and don't raise an error if the model's columns
+  can't be determined.
+  
+* In the identity_map plugin, remove instances from the cache if they
+  are deleted or destroyed.
+
+Backwards Compatibility
+-----------------------
+
+* Dataset::FROM_SELF_KEEP_OPTS was merged into
+  Dataset::NON_SQL_OPTIONS.  While used in different places, they
+  were used for the same purpose, and entries missing from one should
+  have been included in the other.
+
+* The connection pool internals changed substantially.  Now,
+  ConnectionPool #allocated and #available_connections will return
+  nil instead of an array or hash if they are called with a
+  nonexistent server.  These are generally only used internally,
+  though they are part of the public API.  #created_count and #size
+  still return the size of the :default server when called with a
+  nonexistent server, though.
+
+* The meta_eval and metaclass private methods were removed from
+  Sequel::MetaProgramming (only the meta_def public method remains).
+  If you want these methods, use the metaid gem.
+
+* The irregular ox->oxen pluralization rule was removed from the
+  default inflections, as it screws up the more common box->boxes.

data/lib/sequel/adapters/jdbc.rb

@@ -322,7 +322,13 @@ module Sequel
       def metadata(*args, &block)
         synchronize{|c| metadata_dataset.send(:process_result_set, c.getMetaData.send(*args), &block)}
       end
-      
+
+      # Treat SQLExceptions with a "Connection Error" SQLState as disconnects
+      def raise_error(exception, opts={})
+        cause = exception.respond_to?(:cause) ? exception.cause : exception
+        super(exception, {:disconnect => cause.respond_to?(:getSQLState) && cause.getSQLState =~ /^08/}.merge(opts))
+      end
+
       # Close the given statement when removing the transaction
       def remove_transaction(stmt)
         stmt.close if stmt && !supports_savepoints?
@@ -336,7 +342,7 @@ module Sequel
       def set_ps_arg(cps, arg, i)
         case arg
         when Integer
-          cps.setInt(i, arg)
+          cps.setLong(i, arg)
         when Sequel::SQL::Blob
           cps.setBytes(i, arg.to_java_bytes)
         when String
@@ -492,6 +498,8 @@ module Sequel
           BigDecimal.new(v.to_string)
         when Java::byte[]
           Sequel::SQL::Blob.new(String.from_java_bytes(v))
+        when Java::JavaSQL::Blob
+          convert_type(v.getBytes(0, v.length))
         else
           v
         end

data/lib/sequel/adapters/jdbc/h2.rb

@@ -105,6 +105,20 @@ module Sequel
 
         private
       
+        # H2 expects hexadecimal strings for blob values
+        def literal_blob(v)
+          literal_string v.unpack("H*").first
+        end
+        
+        def convert_type(v)
+          case v
+          when Java::OrgH2Jdbc::JdbcClob
+            convert_type(v.getSubString(1, v.length))
+          else
+            super(v)
+          end
+        end
+        
         def select_clause_methods
           SELECT_CLAUSE_METHODS
         end

data/lib/sequel/adapters/mysql.rb

@@ -1,5 +1,10 @@
-require 'mysql'
+begin
+  require "mysqlplus"
+rescue LoadError
+  require 'mysql'
+end
 raise(LoadError, "require 'mysql' did not define Mysql::CLIENT_MULTI_RESULTS!\n  You are probably using the pure ruby mysql.rb driver,\n  which Sequel does not support. You need to install\n  the C based adapter, and make sure that the mysql.so\n  file is loaded instead of the mysql.rb file.\n") unless defined?(Mysql::CLIENT_MULTI_RESULTS)
+
 Sequel.require %w'shared/mysql utils/stored_procedures', 'adapters'
 
 module Sequel
@@ -160,35 +165,40 @@ module Sequel
           r = conn.query(sql)
           if opts[:type] == :select
             yield r if r
-            if conn.respond_to?(:next_result) && conn.next_result
-              loop do
-                if r
-                  r.free
-                  r = nil
-                end
-                begin
-                  r = conn.use_result
-                rescue Mysql::Error
-                  break
-                end
-                yield r
-                break unless conn.next_result
+          elsif block_given?
+            yield conn
+          end
+          if conn.respond_to?(:more_results?)
+            while conn.more_results? do
+              if r
+                r.free
+                r = nil
               end
+              begin
+                conn.next_result
+                r = conn.use_result
+              rescue Mysql::Error => e
+                raise_error(e, :disconnect=>true) if MYSQL_DATABASE_DISCONNECT_ERRORS.match(e.message)
+                break
+              end
+              yield r if opts[:type] == :select
             end
-          else
-            yield conn if block_given?
           end
         rescue Mysql::Error => e
           raise_error(e, :disconnect=>MYSQL_DATABASE_DISCONNECT_ERRORS.match(e.message))
         ensure
-          if r
-            r.free 
-            # Use up all results to avoid a commands out of sync message.
-            if conn.respond_to?(:next_result)
-              while conn.next_result
+          r.free if r
+          # Use up all results to avoid a commands out of sync message.
+          if conn.respond_to?(:more_results?)
+            while conn.more_results? do
+              begin
+                conn.next_result
                 r = conn.use_result
-                r.free if r
+              rescue Mysql::Error => e
+                raise_error(e, :disconnect=>true) if MYSQL_DATABASE_DISCONNECT_ERRORS.match(e.message)
+                break
               end
+              r.free if r
             end
           end
         end
@@ -314,7 +324,7 @@ module Sequel
       
       # Delete rows matching this dataset
       def delete
-        execute_dui(delete_sql){|c| c.affected_rows}
+        execute_dui(delete_sql){|c| return c.affected_rows}
       end
       
       # Yield all rows matching this dataset.  If the dataset is set to
@@ -344,7 +354,7 @@ module Sequel
       
       # Insert a new value into this dataset
       def insert(*values)
-        execute_dui(insert_sql(*values)){|c| c.insert_id}
+        execute_dui(insert_sql(*values)){|c| return c.insert_id}
       end
       
       # Store the given type of prepared statement in the associated database
@@ -361,7 +371,7 @@ module Sequel
       
       # Replace (update or insert) the matching row.
       def replace(*args)
-        execute_dui(replace_sql(*args)){|c| c.insert_id}
+        execute_dui(replace_sql(*args)){|c| return c.insert_id}
       end
       
       # Makes each yield arrays of rows, with each array containing the rows
@@ -382,7 +392,7 @@ module Sequel
       
       # Update the matching rows.
       def update(values={})
-        execute_dui(update_sql(values)){|c| c.affected_rows}
+        execute_dui(update_sql(values)){|c| return c.affected_rows}
       end
       
       private

data/lib/sequel/adapters/postgres.rb

@@ -138,24 +138,29 @@ module Sequel
         end
         @prepared_statements = {} if SEQUEL_POSTGRES_USES_PG
       end
-
-      # Execute the given SQL with this connection.  If a block is given,
-      # yield the results, otherwise, return the number of changed rows.
-      def execute(sql, args=nil)
-        q = nil
+      
+      # Raise a Sequel::DatabaseDisconnectError if a PGError is raised and
+      # the connection status cannot be determined or it is not OK.
+      def check_disconnect_errors
         begin
-          q = args ? async_exec(sql, args) : async_exec(sql)
-        rescue PGError => e
+          yield
+        rescue PGError =>e
           begin
             s = status
           rescue PGError
             raise Sequel.convert_exception_class(e, Sequel::DatabaseDisconnectError)
           end
           status_ok = (s == Adapter::CONNECTION_OK)
-          status_ok ? raise : Sequel.convert_exception_class(e, Sequel::DatabaseDisconnectError)
+          status_ok ? raise : raise(Sequel.convert_exception_class(e, Sequel::DatabaseDisconnectError))
         ensure
           block if status_ok
         end
+      end
+
+      # Execute the given SQL with this connection.  If a block is given,
+      # yield the results, otherwise, return the number of changed rows.
+      def execute(sql, args=nil)
+        q = check_disconnect_errors{args ? async_exec(sql, args) : async_exec(sql)}
         begin
           block_given? ? yield(q) : q.cmd_tuples
         ensure
@@ -218,13 +223,10 @@ module Sequel
       
       # Execute the given SQL with the given args on an available connection.
       def execute(sql, opts={}, &block)
-        return execute_prepared_statement(sql, opts, &block) if Symbol === sql
-        begin
+        check_database_errors do
+          return execute_prepared_statement(sql, opts, &block) if Symbol === sql
           log_info(sql, opts[:arguments])
           synchronize(opts[:server]){|conn| conn.execute(sql, opts[:arguments], &block)}
-        rescue => e
-          log_info(e.message)
-          raise_error(e, :classes=>CONVERTED_EXCEPTIONS)
         end
       end
       
@@ -232,19 +234,25 @@ module Sequel
       # automatically generated).
       def execute_insert(sql, opts={})
         return execute(sql, opts) if Symbol === sql
-        begin 
+        check_database_errors do
           log_info(sql, opts[:arguments])
           synchronize(opts[:server]) do |conn|
             conn.execute(sql, opts[:arguments])
             insert_result(conn, opts[:table], opts[:values])
           end
-        rescue => e
-          log_info(e.message)
-          raise_error(e, :classes=>CONVERTED_EXCEPTIONS)
         end
       end
       
       private
+      
+      # Convert exceptions raised from the block into DatabaseErrors.
+      def check_database_errors
+        begin
+          yield
+        rescue => e
+          raise_error(e, :classes=>CONVERTED_EXCEPTIONS)
+        end
+      end
 
       # PostgreSQL doesn't need the connection pool to convert exceptions.
       def connection_pool_default_options
@@ -281,10 +289,10 @@ module Sequel
             end
             conn.prepared_statements[ps_name] = sql
             log_info("PREPARE #{ps_name} AS #{sql}")
-            conn.prepare(ps_name, sql)
+            conn.check_disconnect_errors{conn.prepare(ps_name, sql)}
           end
           log_info("EXECUTE #{ps_name}", args)
-          q = conn.exec_prepared(ps_name, args)
+          q = conn.check_disconnect_errors{conn.exec_prepared(ps_name, args)}
           if opts[:table] && opts[:values]
             insert_result(conn, opts[:table], opts[:values])
           else