sequel 3.21.0 → 3.36.0

data/doc/testing.rdoc

@@ -0,0 +1,129 @@
+= Testing with Sequel
+
+Whether or not you use Sequel in your application, you are usually going to want to have tests that ensure that your code works.  When you are using Sequel, it's helpful to integrate it into your testing framework, and it's generally best to run each test in its own transaction if possible.  That keeps all tests isolated from each other, and it's simple as it handles all of the cleanup for you.  Sequel doesn't ship with helpers for common libraries, as the exact code you need is often application-specific, but this page offers some examples that you can either use directly or build on.
+
+== Transactional tests
+
+These run each test in its own transaction, the recommended way to test.
+
+Make sure you are using Sequel 3.29.0 or above when using these examples, as older versions don't support the <tt>:rollback=>:always</tt> option.
+
+=== RSpec 1
+
+  class Spec::Example::ExampleGroup
+    def execute(*args, &block)
+      x = nil
+      Sequel::Model.db.transaction(:rollback=>:always){x = super(*args, &block)}
+      x
+    end
+  end
+
+=== RSpec 2
+
+  class RSpec::Core::ExampleGroup
+    # Setting an around filter globally doesn't appear to work in 2.7 (and maybe other versions),
+    # so set one up for each subclass.
+    def self.inherited(subclass)
+      super
+      subclass.around do |example|
+        Sequel::Model.db.transaction(:rollback=>:always){example.call}
+      end
+    end
+  end
+
+=== Test::Unit
+
+  # Must use this class as the base class for your tests
+  class SequelTestCase < Test::Unit::TestCase
+    def run(*args, &block)
+      Sequel::Model.db.transaction(:rollback=>:always){super}
+    end
+  end
+
+=== MiniTest::Unit
+
+  # Must use this class as the base class for your tests
+  class SequelTestCase < MiniTest::Unit::TestCase
+    def run(*args, &block)
+      Sequel::Model.db.transaction(:rollback=>:always){super}
+    end
+  end
+
+== Transactional testing with multiple databases
+
+You can use the Sequel.transaction method to run a transaction on multiple databases, rolling all of them back.  Instead of:
+
+  Sequel::Model.db.transaction(:rollback=>:always)
+
+Use Sequel.transaction with an array of databases:
+
+  Sequel.transaction([DB1, DB2, DB3], :rollback=>:always)
+
+== Nontransactional tests
+
+In some cases, it is not possible to use transactions.  For example, if you are testing a web application that is running in a separate process, you don't have access to that process's database connections, so you can't run your examples in transactions.  In that case, the best way to handle things is to cleanup after each test by deleting or truncating the database tables used in the test.
+
+The order in which you delete/truncate the tables is important if you are using referential integrity in your database (which you probably should be doing).  If you are using referential integrity, you need to make sure to delete in tables referencing other tables before the tables that are being referenced.  For example, if you have an +albums+ table with an +artist_id+ field referencing the +artists+ table, you want to delete/truncate the +albums+ table before the +artists+ table.  Note that if you have cyclic references in your database, you will probably need to write your own custom cleaning code.
+
+=== RSpec
+
+  class Spec::Example::ExampleGroup
+    after do
+      [:table1, :table2].each{|x| Sequel::Model.db.from(x).truncate}
+      # or
+      [:table1, :table2].each{|x| Sequel::Model.db.from(x).delete}
+    end
+  end
+
+=== Test::Unit
+
+  # Must use this class as the base class for your tests
+  class SequelTestCase < Test::Unit::TestCase
+    def teardown
+      [:table1, :table2].each{|x| Sequel::Model.db.from(x).truncate}
+      # or
+      [:table1, :table2].each{|x| Sequel::Model.db.from(x).delete}
+    end
+  end
+
+= Testing Sequel Itself
+
+Sequel has multiple separate test suites.  All test suites run under either RSpec 1 or RSpec 2.
+
+== rake spec
+
+The +spec+ rake task (which is also the default rake task) runs Sequel's core and model specs.  These specs use a mocked database connection, and test for specific SQL used and for generally correct behavior.
+
+== rake spec_plugin
+
+The +spec_plugin+ rake task runs the specs for the plugins and extensions that ship with Sequel.  These also use a mocked database connection, and operate very similarly to the general Sequel core and model specs.
+
+== rake spec_<i>adapter</i> (e.g. rake spec_postgres)
+
+The <tt>spec_<i>adapter</i></tt> specs run against a real database connection with nothing mocked, and test for correct results.  They are slower than the standard specs, but they will catch errors that are mocked out by the default specs, as well as show issues that only occur on a certain database, adapter, or a combination of the two.
+
+These specs are broken down into two parts.  For each database, there are specific specs that only apply to that database, and these are called the adapter specs.  There are also shared specs that apply to all (or almost all) databases, these are called the integration specs.
+
+== Environment variables
+
+Sequel often uses environment variables when testing to specify either the database to be tested or specify how testing should be done.  You can also specify the databases to test by copying spec/spec_config.rb.example to spec/spec_config.rb and modifying it.  See that file for details.  It may be necessary to use spec_config.rb as opposed to an environment variable if your database connection cannot be specified by a connection string.
+
+=== Connection Strings
+
+The following environment variables specify Database connection URL strings:
+
+* SEQUEL_INTEGRATION_URL: integration specs
+* SEQUEL_FB_SPEC_DB: firebird adapter specs
+* SEQUEL_MSSQL_SPEC_DB: mssql adapter specs
+* SEQUEL_PG_SPEC_DB: postgres adapter specs
+* SEQUEL_SQLITE_SPEC_DB: sqlite adapter specs
+* SEQUEL_MY_SPEC_DB: mysql adapter specs
+* SEQUEL_DB2_SPEC_DB: db2 adapter specs
+
+=== Other
+
+* SEQUEL_MSSQL_SPEC_REQUIRE: Separate file to require when running mssql adapter specs
+* SEQUEL_DB2_SPEC_REQUIRE: Separate file to require when running db2 adapter specs
+* SEQUEL_COLUMNS_INTROSPECTION: Whether to run the specs with the columns_introspection extension loaded by default
+* SEQUEL_NO_PENDING: Don't mark any specs as pending, try running all specs
+* SKIPPED_TEST_WARN: Warn when skipping any tests because libraries aren't available

data/doc/thread_safety.rdoc

@@ -0,0 +1,17 @@
+= Thread Safety
+
+Most Sequel usage (and all common Sequel usage) is thread safe by default.  Specifically, multiple threads can operate on Database instances, Dataset instances, and Model classes concurrently without problems.  In general, Database instance and Model classes are not modified after application startup, and modifying Dataset instances returns modified copies of the dataset instead of mutating it.
+
+== Connection Pool
+
+In order to allow multiple threads to operate on the same database at the same time, Sequel uses a connection pool.  The connection pool is designed so that a thread uses a connection for the minimum amount of time, returning the connection to the pool as soon as it is done using the connection.  If a thread requests a connection and the pool is empty, the thread will block (actually busy-wait) until a connection is available or the the connection pool timeout has elapsed (in which case a PoolTimeout error will be raised).
+
+== Exceptions
+
+This is a small list of things that are specifically non thread-safe.  This is not an exhaustive list, there may be cases not mentioned here.
+
+1) Model instances: Model instances are not thread-safe unless they are frozen first.  Multiple threads should not operate on an unfrozen model instance concurrently.
+
+2) Model class modifications: Model class modifications, such as adding associations and loading plugins, are not designed to be thread safe.  You should not modify a class in one thread if any other thread can concurrently access it.  Model subclassing is designed to be thread-safe, so you create a model subclass in a thread and modify it safely.
+
+3) Dataset mutation methods: Dataset mutation methods are not thread safe, you should not call them on datasets that could be accessed by other threads.  It is safe to clone the dataset first inside a thread and call mutation methods on the cloned dataset.

data/doc/transactions.rdoc

@@ -0,0 +1,137 @@
+= Database Transactions 
+
+Sequel uses autocommit mode by default for all of its database adapters, so in general in Sequel if you want to use database transactions, you need to be explicit about it.  There are a few cases where transactions are used implicitly by default:
+
+* Dataset#import to insert many records at once
+* Model#save
+* Model#destroy
+* Migrations
+* A few model plugins
+
+Everywhere else, it is up to use to use a database transaction if you want to.
+
+== Basic Transaction Usage
+
+In Sequel, the <tt>Database#transaction</tt> method should be called if you want to use a database transaction.  This method must be called with a block.  If the block does not raise an exception, the transaction is committed:
+
+  DB.transaction do # BEGIN
+    DB[:foo].insert(1) # INSERT
+  end # COMMIT
+   
+If the block raises a Sequel::Rollback exception, the transaction is rolled back, but no exception is raised outside the block:
+
+  DB.transaction do # BEGIN
+    raise Sequel::Rollback
+  end # ROLLBACK
+  # no exception raised
+
+If any other exception is raised, the transaction is rolled back, and the exception is raised outside the block:
+
+  # ArgumentError raised
+  DB.transaction do # BEGIN
+    raise ArgumentError
+  end # ROLLBACK
+  # ArgumentError raised
+
+If you want Sequel::Rollback exceptions to be reraised, use the <tt>:rollback => :reraise</tt> option:
+
+  DB.transaction(:rollback => :reraise) do # BEGIN
+    raise Sequel::Rollback
+  end # ROLLBACK
+  # Sequel::Rollback raised
+
+If you always want to rollback (useful for testing), use the <tt>:rollback => :always</tt> option:
+
+  DB.transaction(:rollback => :always) do # BEGIN
+    DB[:foo].insert(1) # INSERT
+  end # ROLLBACK
+
+If you want to check whether you are currently in a transaction, use the Database#in_transaction? method:
+
+  DB.in_transaction? # false
+  DB.transaction do
+    DB.in_transaction? # true
+  end
+
+== Transaction Hooks
+
+You can add hooks to an in progress transaction that are called after the transaction commits or rolls back:
+
+  x = nil
+  DB.transaction do
+    DB.after_commit{x = 1}
+    DB.after_rollback{x = 2}
+    x # nil
+  end
+  x # 1
+  
+  x = nil
+  DB.transaction do
+    DB.after_commit{x = 1}
+    DB.after_rollback{x = 2}
+    raise Sequel::Rollback
+  end
+  x # 2
+
+== Nested Transaction Calls / Savepoints
+
+You can nest calls to transaction, which by default just reuses the existing transaction:
+
+  DB.transaction do # BEGIN
+    DB.transaction do
+      DB[:foo].insert(1) # INSERT
+    end
+  end # COMMIT
+
+You can use the <tt>:savepoint => true</tt> option in the inner transaction to explicitly use a savepoint (if the database supports it):
+
+  DB.transaction do # BEGIN
+    DB.transaction(:savepoint => true) do # SAVEPOINT
+      DB[:foo].insert(1) # INSERT
+    end # RELEASE SAVEPOINT
+  end # COMMIT
+
+If a Sequel::Rollback exception is raised inside the savepoint block, it will only rollback to the savepoint:
+
+  DB.transaction do # BEGIN
+    DB.transaction(:savepoint => true) do # SAVEPOINT
+      raise Sequel::Rollback
+    end # ROLLBACK TO SAVEPOINT
+    # no exception raised
+  end # COMMIT
+
+Other exceptions, unless rescued inside the outer transaction block, will rollback the savepoint and the outer transactions, since they are reraised by the transaction code:
+
+  DB.transaction do # BEGIN
+    DB.transaction(:savepoint => true) do # SAVEPOINT
+      raise ArgumentError
+    end # ROLLBACK TO SAVEPOINT
+  end # ROLLBACK
+  # ArgumentError raised
+
+== Prepared Transactions / Two-Phase Commit
+
+Sequel supports database prepared transactions on PostgreSQL, MySQL, and H2.  With prepared transactions, at the end of the transaction, the transaction is not immediately committed (it acts like a rollback).  Later, you can call +commit_prepared_transaction+ to commit the transaction or +rollback_prepared_transaction+ to roll the transaction back.  Prepared transactions are usually used with distributed databases to make sure all databases commit the same transaction or none of them do.
+
+To use prepared transactions in Sequel, you provide a string as the value of the :prepare option:
+
+  DB.transaction(:prepare => 'foo') do # BEGIN
+    DB[:foo].insert(1) # INSERT
+  end # PREPARE TRANSACTION 'foo'
+   
+Later, you can commit the prepared transaction:
+
+  DB.commit_prepared_transaction('foo')
+
+or roll the prepared transaction back:
+
+  DB.rollback_prepared_transaction('foo')
+
+== Transaction Isolation Levels
+
+The SQL standard supports 4 isolation levels: READ UNCOMMITTED, READ COMMITTED, REPEATABLE READ, and SERIALIZABLE.  Not all databases implement the levels as specified in the standard (or implement the levels at all), but on PostgreSQL, MySQL, and Microsoft SQL Server, you can specify which transaction isolation level you want to use via the :isolation option to <tt>Database#transaction</tt>.  The isolation level is specified as one of the following symbols: :uncommitted, :committed, :repeatable, and :serializable.  Using this option make Sequel use the correct transaction isolation syntax for your database:
+
+  DB.transaction(:isolation => :serializable) do # BEGIN
+    # SET TRANSACTION ISOLATION LEVEL SERIALIZABLE
+    DB[:foo].insert(1) # INSERT
+  end # COMMIT

data/doc/virtual_rows.rdoc

@@ -40,7 +40,7 @@ available.  For example, you are no longer able to do:
   # WHERE a AND (b OR NOT c)
   
 Because Symbol#&, Symbol#| and Symbol#~ are not defined when the core
-extensions are turned off.  However, with virtual rows allow almost the same
+extensions are turned off.  However, virtual rows allow almost the same
 syntax even without the core extensions:
 
   dataset.filter{a & (b | ~c)}
@@ -87,7 +87,7 @@ and second is the difference between the the use of "a".  In the regular proc,
 you couldn't call c without an explicit receiver in the proc, unless the self of the
 surrounding scope responded to it.  For a, note how ruby calls the method on
 the receiver of the surrounding scope in the regular proc, which returns an integer,
-and does the substraction before Sequel gets access to it.  In the instance evaled
+and does the substitution before Sequel gets access to it.  In the instance evaled
 proc, calling a without a receiver calls the a method on the VirtualRow instance.
 For b, note that it operates the same in both cases, as it is a local variable.
 
@@ -205,6 +205,55 @@ call, with an optional hash as the second argument: Here are some examples of us
   ds.select{sum(:over, :args=>col1, :partition=>col2, :order=>col3){}}
   # SELECT sum(col1) OVER (PARTITION BY col2 ORDER BY col3)
 
+== Operators
+
+VirtualRows use method_missing to handle almost all method calls.  However, they
+have special handling of some operator methods to make certain things easier. The
+operators all use a prefix form.
+
+=== Math Operators
+
+The standard +, -, *, and / mathematical operators are defined:
+
+  ds.select{|o| o.-(1, o.a).as(b)}
+  ds.select{self.-(1, a).as(b)}
+  # SELECT (1 - a) AS b
+
+=== Boolean Operators
+
+The & and | methods are defined to use AND and OR:
+
+  ds.where{|o| o.&({:a=>:b}, :c)}
+  ds.where{self.&({:a=>:b}, :c)}
+  # WHERE ((a = b) AND c)
+
+The ~ method is defined to do inversion:
+
+  ds.where{|o| o.~({:a=>1, :b=>2})}
+  ds.where{self.~({:a=>1, :b=>2})}
+  # WHERE ((a != 1) OR (b != 2))
+
+=== Inequality Operators
+
+The standard >, <, >=, and <= inequality operators are defined:
+
+  ds.where{|o| o.>(1, :c)}
+  ds.where{self.>(1, :c)}
+  # WHERE (1 > c)
+
+== Literal Strings
+
+The backtick operator can be used inside an instance-evaled
+virtual row block to create a literal string:
+
+  ds.where{a > `some SQL`}
+  # WHERE (a > some SQL)
+
+You can use this on a regular virtual row block too, but it
+doesn't look as nice:
+
+  ds.where{|o| o.>(:a, o.`('some SQL')}
+
 == Returning multiple values
 
 It's common when using select and order virtual row blocks to want to

data/lib/sequel/adapters/ado/mssql.rb

@@ -11,11 +11,6 @@ module Sequel
         # delete query.
         ROWS_AFFECTED = "SELECT @@ROWCOUNT AS AffectedRows"
         
-        # Return instance of Sequel::ADO::MSSQL::Dataset with the given opts.
-        def dataset(opts=nil)
-          Sequel::ADO::MSSQL::Dataset.new(self, opts)
-        end
-        
         # Just execute so it doesn't attempt to return the number of rows modified.
         def execute_ddl(sql, opts={})
           execute(sql, opts)
@@ -37,11 +32,28 @@ module Sequel
             end
           end
         end
+
+        private
+
+        # The ADO adapter's default provider doesn't support transactions, since it 
+        # creates a new native connection for each query.  So Sequel only attempts
+        # to use transactions if an explicit :provider is given.
+        def begin_transaction(conn, opts={})
+          super if @opts[:provider]
+        end
+
+        def commit_transaction(conn, opts={})
+          super if @opts[:provider]
+        end
+
+        def rollback_transaction(conn, opts={})
+          super if @opts[:provider]
+        end
       end
       
       class Dataset < ADO::Dataset
         include Sequel::MSSQL::DatasetMethods
-        
+
         # Use a nasty hack of multiple SQL statements in the same call and
         # having the last one return the most recently inserted id.  This
         # is necessary as ADO's default :provider uses a separate native

data/lib/sequel/adapters/ado.rb

@@ -4,6 +4,8 @@ module Sequel
   # The ADO adapter provides connectivity to ADO databases in Windows.
   module ADO
     class Database < Sequel::Database
+      DISCONNECT_ERROR_RE = /Communication link failure/
+
       set_adapter_scheme :ado
 
       def initialize(opts)
@@ -12,12 +14,15 @@ module Sequel
         when /Microsoft\.(Jet|ACE)\.OLEDB/io
           Sequel.ts_require 'adapters/shared/access'
           extend Sequel::Access::DatabaseMethods
-        when nil
+          extend_datasets(Sequel::Access::DatasetMethods)
+        else
           @opts[:driver] ||= 'SQL Server'
           case @opts[:driver]
           when 'SQL Server'
             Sequel.ts_require 'adapters/ado/mssql'
             extend Sequel::ADO::MSSQL::DatabaseMethods
+            @dataset_class = ADO::MSSQL::Dataset
+            set_mssql_unicode_strings
           end
         end
       end
@@ -52,10 +57,6 @@ module Sequel
         handle
       end
       
-      def dataset(opts = nil)
-        ADO::Dataset.new(self, opts)
-      end
-    
       def execute(sql, opts={})
         synchronize(opts[:server]) do |conn|
           begin
@@ -74,28 +75,49 @@ module Sequel
       # The ADO adapter's default provider doesn't support transactions, since it 
       # creates a new native connection for each query.  So Sequel only attempts
       # to use transactions if an explicit :provider is given.
-      def _transaction(conn, o={})
-        return super if opts[:provider]
-        th = Thread.current
-        begin
-          @transactions << th
-          yield conn
-        rescue Sequel::Rollback
-        ensure
-          @transactions.delete(th)
-        end
+      def begin_transaction(conn, opts={})
+        super if @opts[:provider]
       end
-      
+
+      def commit_transaction(conn, opts={})
+        super if @opts[:provider]
+      end
+
+      def database_error_classes
+        [::WIN32OLERuntimeError]
+      end
+
       def disconnect_connection(conn)
         conn.Close
       end
+
+      def disconnect_error?(e, opts)
+        super || (e.is_a?(::WIN32OLERuntimeError) && e.message =~ DISCONNECT_ERROR_RE)
+      end
+
+      def rollback_transaction(conn, opts={})
+        super if @opts[:provider]
+      end
     end
     
     class Dataset < Sequel::Dataset
+      Database::DatasetClass = self
+
       def fetch_rows(sql)
         execute(sql) do |s|
-          @columns = cols = s.Fields.extend(Enumerable).map{|column| output_identifier(column.Name)}
-          s.getRows.transpose.each{|r| yield cols.inject({}){|m,c| m[c] = r.shift; m}} unless s.eof
+          columns = cols = s.Fields.extend(Enumerable).map{|column| output_identifier(column.Name)}
+          if opts[:offset] && offset_returns_row_number_column?
+            rn = row_number_column
+            columns = columns.dup
+            columns.delete(rn)
+          end
+          @columns = columns
+          s.getRows.transpose.each do |r|
+            row = {}
+            cols.each{|c| row[c] = r.shift}
+            row.delete(rn) if rn
+            yield row
+          end unless s.eof
         end
       end
       

data/lib/sequel/adapters/amalgalite.rb

@@ -14,6 +14,12 @@ module Sequel
         'float' => ['float', 'double', 'real', 'double precision'],
         'decimal' => %w'numeric decimal money'
       )
+
+      # Store the related database object, in order to be able to correctly
+      # handle the database timezone.
+      def initialize(db)
+        @db = db
+      end
       
       # Return blobs as instances of Sequel::SQL::Blob instead of
       # Amalgamite::Blob
@@ -29,7 +35,11 @@ module Sequel
       
       # Return datetime types as instances of Sequel.datetime_class
       def datetime(s)
-        Sequel.database_to_application_timestamp(s)
+        @db.to_application_timestamp(s)
+      end
+
+      def time(s)
+        Sequel.string_to_time(s)
       end
       
       # Don't raise an error if the value is a string and the declared
@@ -68,7 +78,8 @@ module Sequel
         opts[:database] = ':memory:' if blank_object?(opts[:database])
         db = ::Amalgalite::Database.new(opts[:database])
         db.busy_handler(::Amalgalite::BusyTimeout.new(opts.fetch(:timeout, 5000)/50, 50))
-        db.type_map = SequelTypeMap.new
+        db.type_map = SequelTypeMap.new(self)
+        connection_pragmas.each{|s| log_yield(s){db.execute_batch(s)}}
         db
       end
       
@@ -77,11 +88,6 @@ module Sequel
         :sqlite
       end
 
-      # Return instance of Sequel::Amalgalite::Dataset with the given options.
-      def dataset(opts = nil)
-        Amalgalite::Dataset.new(self, opts)
-      end
-      
       # Run the given SQL with the given arguments. Returns nil.
       def execute_ddl(sql, opts={})
         _execute(sql, opts){|conn| log_yield(sql){conn.execute_batch(sql)}}
@@ -151,6 +157,8 @@ module Sequel
     # Dataset class for SQLite datasets that use the amalgalite driver.
     class Dataset < Sequel::Dataset
       include ::Sequel::SQLite::DatasetMethods
+
+      Database::DatasetClass = self
       
       # Yield a hash for each row in the dataset.
       def fetch_rows(sql)
@@ -168,8 +176,8 @@ module Sequel
       private
       
       # Quote the string using the adapter instance method.
-      def literal_string(v)
-        db.synchronize{|c| c.quote(v)}
+      def literal_string_append(sql, v)
+        db.synchronize{|c| sql << c.quote(v)}
       end
     end
   end