sequel 2.12.0 → 3.0.0

data/CHANGELOG

@@ -1,3 +1,65 @@
+=== 3.0.0 (2009-05-04)
+
+* Remove dead threads from connection pool if the pool is full and a connection is requested (jeremyevans)
+
+* Add autoincrementing primary key support in the Oracle adapter, using a sequence and trigger (jeremyevans, Mike Golod)
+
+* Make Model#save use the same server it uses for saving as for retrieving the saved record (jeremyevans)
+
+* Add Database#database_type method, for identifying which type of database the object is connecting to (jeremyevans)
+
+* Add ability to reset primary key sequences in the PostgreSQL adapter (jeremyevans)
+
+* Fix parsing of non-simple sequence names (that contain uppercase, spaces, etc.) in the PostgreSQL adapter (jeremyevans)
+
+* Support dumping indexes in the schema_dumper extension (jeremyevans)
+
+* Add index parsing to PostgreSQL, MySQL, SQLite, and JDBC adapters (jeremyevans)
+
+* Correctly quote SQL Array references, and handle qualified identifiers with them (e.g. :table__column.sql_subscript(1)) (jeremyevans)
+
+* Allow dropping an index with a name different than the default name (jeremyevans)
+
+* Allow Dataset#from to remove existing FROM tables when called without an argument, instead of raising an error later (jeremyevans)
+
+* Fix string quoting on Oracle so it doesn't double backslashes (jeremyevans)
+
+* Alias the count function call in Dataset#count, fixes use on MSSQL (akitaonrails, jeremyevans)
+
+* Allow QualifiedIdentifiers to be qualified, to allow :column.qualify(:table).qualify(:schema) (jeremyevans)
+
+* Allow :db_type=>'mssql' option to be respected when using the DBI adapter (akitaonrails)
+
+* Add schema_dumper extension, for dumping schema of tables (jeremyevans)
+
+* Allow generic database types specified as ruby types to take options (jeremyevans)
+
+* Change Dataset#exclude to invert given hash argument, not negate it (jeremyevans)
+
+* Make Dataset#filter and related methods treat multiple arguments more intuitively (jeremyevans)
+
+* Fix full text searching with multiple search terms on MySQL (jeremyevans)
+
+* Fix altering a column name, type, default, or NULL/NOT NULL status on MySQL (jeremyevans)
+
+* Fix index type syntax on MySQL (jeremyevans)
+
+* Add temporary table support, via :temp option to Database#create_table (EppO, jeremyevans)
+
+* Add Amalgalite adapter (jeremyevans)
+
+* Remove Sequel::Metaprogramming#metaattr_accessor and metaattr_reader (jeremyevans)
+
+* Remove Dataset#irregular_function_sql (jeremyevans)
+
+* Add Dataset#full_text_sql to the MySQL adapter (dusty)
+
+* Fix schema type parsing of decimal types on MySQL (jeremyevans)
+
+* Make Dataset#quote_identifier work with SQL::Identifiers (jeremyevans)
+
+* Remove methods and features deprecated in 2.12.0 (jeremyevans)
+
 === 2.12.0 (2009-04-03)
 
 * Deprecate Java::JavaSQL::Timestamp#usec (jeremyevans)

data/README.rdoc

@@ -11,9 +11,9 @@ Sequel is a lightweight database access toolkit for Ruby.
   configurations, and database sharding.
 * Sequel makes it easy to deal with multiple records without having
   to break your teeth on SQL.
-* Sequel currently has adapters for ADO, DataObjects, DB2, DBI,
-  Firebird, Informix, JDBC, MySQL, ODBC, OpenBase, Oracle, PostgreSQL
-  and SQLite3.
+* Sequel currently has adapters for ADO, Amalgalite, DataObjects,
+  DB2, DBI, Firebird, Informix, JDBC, MySQL, ODBC, OpenBase, Oracle,
+  PostgreSQL and SQLite3.
 
 == Resources
 

data/Rakefile

@@ -155,6 +155,13 @@ begin
     t.spec_opts  = spec_opts.call
   end
   
+  desc "Run extention/plugin specs with coverage"
+  Spec::Rake::SpecTask.new("spec_plugin_cov") do |t|
+    t.spec_files = Dir["spec/extensions/*_spec.rb"]
+    t.spec_opts  = spec_opts.call
+    t.rcov, t.rcov_opts = rcov_opts.call
+  end
+  
   desc "Run integration tests"
   Spec::Rake::SpecTask.new("integration") do |t|
     t.spec_files = FileList["spec/integration/*_test.rb"]

data/doc/advanced_associations.rdoc

@@ -603,3 +603,47 @@ name, with no duplicates?
         records.each{|r| r.associations[:songs].uniq!}
       end)
   end
+
+=== Statistics Associations (Sum of Associated Table Column)
+
+In addition to getting associated records, you can use Sequel's association support
+to get aggregate information for columns in associated tables (sums, averages, etc.).
+
+Let's say you have a database with projects and tickets.  A project can have many
+tickets, and each ticket has a number of hours associated with it.  You can use the
+association support to create a Project association that gives the sum of hours for all
+associated tickets.
+
+
+  class Project < Sequel::Model
+    one_to_many :tickets
+    many_to_one :ticket_hours, :read_only=>true, :key=>:id,
+     :dataset=>proc{Ticket.filter(:project_id=>id).select{sum(hours).as(hours)}},
+     :eager_loader=>(proc do |kh, projects, a|
+      projects.each{|p| p.associations[:ticket_hours] = nil}
+      Ticket.filter(:project_id=>kh[:id].keys).
+       group(:project_id).
+       select{[project_id, sum(hours).as(hours)]}.
+       all do |t|
+        p = kh[:id][t.values.delete(:project_id)].first
+        p.associations[:ticket_hours] = t
+       end
+     end)
+    # The association method returns a Ticket object with a single aggregate
+    # sum-of-hours value, but you want it to return an Integer/Float of just the
+    # sum of hours, so you call super and return just the sum-of-hours value.
+    # This works for both lazy loading and eager loading.
+    def ticket_hours
+      if s = super
+        s[:hours]
+      end
+    end
+  end
+  class Ticket < Sequel::Model
+    many_to_one :project
+  end
+
+Note that it is often better to use a sum cache instead of this approach.  You can implement
+a sum cache using before or after save and delete hooks, or using a database trigger
+(the preferred method if you only have to support one database and that database supports
+triggers).

data/doc/release_notes/3.0.0.txt

@@ -0,0 +1,221 @@
+Deprecated Methods/Features Removed
+-----------------------------------
+
+Methods and features that were deprecated in 2.12.0 have been removed
+in 3.0.0.  Many features were moved into plugins or extensions, so in
+many cases you just need to require an extension or use Model.plugin
+and not make any changes to your code.  See the 2.12.0 release notes
+for the list of methods/features deprecated in 2.12.0.
+
+If you are upgrading from a previous 2.x release, please upgrade to
+2.12.0 first, fix your code to remove all deprecation warnings, and
+then upgrade to 3.0.0.
+
+New Adapter
+-----------
+
+* Sequel now has an Amalgalite adapter.  Amalgalite is a ruby
+  extension that embeds SQLite without requiring a separate SQLite
+  installation.  The adapter is functionality complete but
+  significantly slower than the native SQLite adapter.
+
+New Features
+------------
+
+* The JDBC, PostgreSQL, MySQL, and SQLite adapters all now have a
+  Database#indexes method that returns indexes for a given table:
+
+    DB.indexes(:songs)
+    => {:songs_name_index=>{:unique=>true, :columns=>[:name]},
+        :songs_lyricid_index=>{:unique=>false, :columns=>[:lyricid]}}
+
+* A schema_dumper extension was added to Sequel.  It supports dumping
+  the schema of a table (including indexes) as a string that can be
+  evaluated in the context of a Database object to create the table.
+  It also supports dumping all tables in the database as a string
+  containing a Migration subclass that will rebuild the database.
+
+    require 'sequel/extensions/schema_dumper'
+    DB.dump_table_schema(:table)
+    DB.dump_schema_migration
+    DB.dump_schema_migration(:same_db=>true)
+    DB.dump_schema_migration(:indexes=>false)
+    DB.dump_indexes_migration
+
+  The :same_db option causes Sequel to not translate column types
+  to generic column types.  By default, the migration created will
+  use generic types so it will run on other databases.  However, if
+  you only want to support a single database, using the :same_db
+  option will make the migration use the exact database type parsed
+  from the database.
+
+  The :indexes=>false option causes indexes not be included in the
+  migration.  The dump_indexes_migration can be used to create a
+  separate migration with the indexes.  This can be useful if you 
+  plan on loading a lot of data right after creating the tables,
+  since it is faster to add indexes after the data has been added.
+
+* Using options with the generic database types is now supported to
+  a limited extent.  For example, the following code now works:
+
+    DB.create_table(:table) do
+      String :a, :size=>50               # varchar(50)
+      String :b, :text=>true             # text
+      String :c, :fixed=>true, :size=>30 # char(30)
+      Time :ts                           # timestamp
+      Time :t, :only_time=>true          # time
+    end
+
+* Using Dataset#filter and related methods with multiple arguments
+  now works much more intuitively:
+
+    # 2.12.0
+    dataset.filter(:a, :b=>1) # a IS NULL AND (b = 1) IS NULL
+    # 3.0.0
+    dataset.filter(:a, :b=>1) # a AND b = 1
+
+* You can now create temporary tables by passing the :temp=>true
+  option to Database#create_table.
+
+* The Oracle shared adapter now supports emulation of
+  autoincrementing primary keys by creating a sequence and a trigger,
+  similar to how the Firebird adapter works.
+
+* The Database#database_type method was added that returns a symbol
+  specifying the database type being used.  This can be different
+  than Database.adapter_scheme if you are using an adapter like
+  JDBC that allows connecting to multiple different types of
+  databases.
+
+* Database#drop_index and related methods now support an options
+  hash that respects the :name option, so they can now be used to
+  drop an index that doesn't use the default index name.
+
+* The PostgreSQL shared adapter now supports a
+  Database#reset_primary_key_sequence method to reset the
+  primary key sequence for a given table, based on code from
+  ActiveRecord.
+
+* SQL::QualifiedIdentifiers can now be qualified, allowing you to do:
+
+   :column.qualify(:table).qualify(:schema)
+ 
+* Using the :db_type=>'mssql' option with the DBI adapter will now
+  load the MSSQL support.
+
+* The MySQL shared adapter now supports Dataset#full_text_sql, which
+  you can use in queries like the following:
+
+    ds.select(:table.*, ds.full_text_sql(:column, 'value').as(:ft))
+
+Other Improvements
+------------------
+
+* Sequel will now release connections from the connection pool
+  automatically if they are held by a dead thread.  This can happen
+  if you are using MRI 1.8 and you are heavily multithreaded or
+  you call Thread#exit! or similar method explicitly.  Those methods
+  skip the execution of ensure blocks which normally release the
+  connections when the threads exit.
+
+* Model#save will now always use the same server when refreshing data
+  after an insert.  This fixes an issue when Sequel's master/slave
+  database support is used with models.
+
+* SQL Array references are now quoted correctly, so code like this
+  now works:
+
+    :table__column.sql_subscript(1)
+
+* The PostgreSQL shared adapter now handles sequences that need to be
+  quoted correctly (previously these were quoted twice).
+
+* String quoting on Oracle no longer doubles backslashes. 
+
+* Database#count now works correctly when used on MSSQL when using
+  an adapter that doesn't handle unnamed columns.
+
+* Full text searching in the MySQL adapter now works correctly when
+  multiple search terms are used.
+
+* Altering a column's name, type, default, or NULL/NOT NULL status
+  on MySQL now keeps other relevent column information.  For example,
+  if you alter a column's type, it'll keep an existing default. This
+  functionality isn't complete, there may be other column information
+  that is lost.
+
+* Fix creation of an index with a given type on MySQL, since MySQL's
+  documentation lies.
+
+* The schema parser now handles decimal types with size specifiers,
+  fixing use on MySQL.
+
+* Dataset#quote_identifier now works correctly when given an
+  SQL::Identifier.  This allows you to do:
+
+    dataset.select{sum(hours).as(hours)}
+
+Backwards Compatibility
+-----------------------
+
+* Sequel will now use instance_eval on all virtual row blocks without
+  an argument.  This can lead to much nicer code:
+
+    dataset.filter{(number > 10) & (name > 'M')}
+    # WHERE number > 10 AND name > 'M'
+
+  2.12.0 raised a deprecation warning if you used a virtual row block
+  without an argument and you hadn't set
+  Sequel.virtual_row_instance_eval = true.
+
+* Dataset#exclude now inverts the given argument, instead of negating
+  it. This only changes its behavior if it is called with a hash or
+  array of all two pairs that have more than one element.
+
+    # 2.12.0
+    dataset.exclude(:a=>1, :b=>1) # a != 1 AND b != 1
+    # 3.0.0
+    dataset.exclude(:a=>1, :b=>1) # a != 1 OR b != 1
+
+  This was done for consistency, since exclude would only negate a
+  hash if it was given an argument, it would invert the same hash
+  if you used a block:
+
+    # 2.12.0
+    dataset.exclude{{:a=>1, :b=>1}} # a != 1 OR b != 1
+  
+  If you want the previous behavior,
+  change the code to the following:
+
+    dataset.filter({:a=>1, :b=>1}.sql_negate)
+
+* As noted above, the methods/features deprecated in 2.12.0 were
+  removed.
+
+* The private Dataset#select_*_sql methods now only take a single
+  argument, the SQL string being built.
+
+* Dataset#from when called without arguments would previously cause an
+  error to be raised when the SQL string is generated.  Now it causes
+  no FROM clause to be used, similar to how Dataset#select with no
+  arguments causes SELECT * to be used.
+
+* The internals of the generic type support and the schema generators
+  were changed significantly, which could have some fallout in terms
+  of old migrations breaking if they used the generic types and were
+  relying on some undocumented behavior (such as using Integer as a
+  type with the :unsigned option).
+
+* The Firebird adapter no longer translates the text database
+  specific type.  Use the following instead:
+    
+    String :column, :text=>true 
+
+* The MySQL shared adapter used to use the timestamp type for Time,
+  now it uses datetime.  This is because the timestamp type cannot
+  represent everything that the ruby Time class can represent.
+
+* Metaprogramming#metaattr_accessor and #metaattr_reader methods were
+  removed.
+
+* Dataset#irregular_function_sql was removed.

data/lib/sequel/adapters/amalgalite.rb

@@ -0,0 +1,208 @@
+require 'amalgalite'
+Sequel.require 'adapters/shared/sqlite'
+
+module Sequel
+  # Top level module for holding all Amalgalite-related modules and classes
+  # for Sequel.
+  module Amalgalite
+    # Type conversion map class for Sequel's use of Amalgamite
+    class SequelTypeMap < ::Amalgalite::TypeMaps::DefaultMap
+      methods_handling_sql_types.delete('string')
+      methods_handling_sql_types.merge!(
+        'datetime' => %w'datetime timestamp',
+        'time' => %w'time',
+        'float' => ['float', 'double', 'real', 'double precision'],
+        'decimal' => %w'numeric decimal money'
+      )
+      
+      # Return blobs as instances of Sequel::SQL::Blob instead of
+      # Amalgamite::Blob
+      def blob(s)
+        SQL::Blob.new(s)
+      end
+      
+      # Return numeric/decimal types as instances of BigDecimal
+      # instead of Float
+      def decimal(s)
+        BigDecimal.new(s)
+      end
+      
+      # Return datetime types as instances of Sequel.datetime_class
+      def datetime(s)
+        Sequel.string_to_datetime(s)
+      end
+      
+      # Don't raise an error if the value is a string and the declared
+      # type doesn't match a known type, just return the value.
+      def result_value_of(declared_type, value)
+        if value.is_a?(::Amalgalite::Blob)
+          SQL::Blob.new(value.source)
+        elsif value.is_a?(String) && declared_type
+          (meth = self.class.sql_to_method(declared_type.downcase)) ? send(meth, value) : value
+        else
+          super
+        end
+      end
+    end
+    
+    # Database class for SQLite databases used with Sequel and the
+    # amalgalite driver.
+    class Database < Sequel::Database
+      include ::Sequel::SQLite::DatabaseMethods
+      
+      set_adapter_scheme :amalgalite
+      
+      # Mimic the file:// uri, by having 2 preceding slashes specify a relative
+      # path, and 3 preceding slashes specify an absolute path.
+      def self.uri_to_options(uri) # :nodoc:
+        { :database => (uri.host.nil? && uri.path == '/') ? nil : "#{uri.host}#{uri.path}" }
+      end
+      private_class_method :uri_to_options
+      
+      # Connect to the database.  Since SQLite is a file based database,
+      # the only options available are :database (to specify the database
+      # name), and :timeout, to specify how long to wait for the database to
+      # be available if it is locked, given in milliseconds (default is 5000).
+      def connect(server)
+        opts = server_opts(server)
+        opts[:database] = ':memory:' if blank_object?(opts[:database])
+        db = ::Amalgalite::Database.new(opts[:database])
+        db.busy_handler(::Amalgalite::BusyTimeout.new(opts.fetch(:timeout, 5000)/50, 50))
+        db.type_map = SequelTypeMap.new
+        db
+      end
+      
+      # Amalgalite is just the SQLite database without a separate SQLite installation.
+      def database_type
+        :sqlite
+      end
+
+      # Return instance of Sequel::Amalgalite::Dataset with the given options.
+      def dataset(opts = nil)
+        Amalgalite::Dataset.new(self, opts)
+      end
+      
+      # Run the given SQL with the given arguments and reload the schema.  Returns nil.
+      def execute_ddl(sql, opts={})
+        _execute(sql, opts){|conn| conn.execute_batch(sql); conn.reload_schema!}
+        nil
+      end
+      
+      # Run the given SQL with the given arguments and return the number of changed rows.
+      def execute_dui(sql, opts={})
+        _execute(sql, opts){|conn| conn.execute_batch(sql); conn.row_changes}
+      end
+      
+      # Run the given SQL with the given arguments and return the last inserted row id.
+      def execute_insert(sql, opts={})
+        _execute(sql, opts){|conn| conn.execute_batch(sql); conn.last_insert_rowid}
+      end
+      
+      # Run the given SQL with the given arguments and yield each row.
+      def execute(sql, opts={})
+        retried = false
+        _execute(sql, opts) do |conn|
+          conn.prepare(sql) do |stmt|
+            begin
+             stmt.result_meta
+            rescue NoMethodError
+              conn.reload_schema!
+              stmt.result_meta
+            end
+            yield stmt
+          end
+        end
+      end
+      
+      # Run the given SQL with the given arguments and return the first value of the first row.
+      def single_value(sql, opts={})
+        _execute(sql, opts){|conn| conn.first_value_from(sql)}
+      end
+      
+      # Use the native driver transaction method if there isn't already a transaction
+      # in progress on the connection, always yielding a connection inside a transaction
+      # transaction.
+      def transaction(opts={})
+        synchronize(opts[:server]) do |conn|
+          return yield(conn) if conn.in_transaction?
+          begin
+            result = nil
+            log_info('Transaction.begin')
+            conn.transaction{result = yield(conn)}
+            result
+          rescue ::Exception => e
+            log_info('Transaction.rollback')
+            transaction_error(e, ::Amalgalite::Error, ::Amalgalite::SQLite3::Error)
+          ensure
+            log_info('Transaction.commit') unless e
+          end
+        end
+      end
+      
+      private
+      
+      # Log the SQL and yield an available connection.  Rescue
+      # any Amalgalite::Errors and turn them into DatabaseErrors.
+      def _execute(sql, opts)
+        begin
+          log_info(sql)
+          synchronize(opts[:server]){|conn| yield conn}
+        rescue ::Amalgalite::Error, ::Amalgalite::SQLite3::Error => e
+          raise_error(e)
+        end
+      end
+      
+      # The Amagalite adapter does not need the pool to convert exceptions.
+      # Also, force the max connections to 1 if a memory database is being
+      # used, as otherwise each connection gets a separate database.
+      def connection_pool_default_options
+        o = super.merge(:pool_convert_exceptions=>false)
+        # Default to only a single connection if a memory database is used,
+        # because otherwise each connection will get a separate database
+        o[:max_connections] = 1 if @opts[:database] == ':memory:' || blank_object?(@opts[:database])
+        o
+      end
+
+      # Disconnect given connections from the database.
+      def disconnect_connection(c)
+        c.close
+      end
+    end
+    
+    # Dataset class for SQLite datasets that use the amalgalite driver.
+    class Dataset < Sequel::Dataset
+      include ::Sequel::SQLite::DatasetMethods
+      
+      EXPLAIN = 'EXPLAIN %s'.freeze
+      
+      # Return an array of strings specifying a query explanation for the
+      # current dataset.
+      def explain
+        res = []
+        @db.result_set(EXPLAIN % select_sql(opts), nil) {|r| res << r}
+        res
+      end
+      
+      # Yield a hash for each row in the dataset.
+      def fetch_rows(sql)
+        execute(sql) do |stmt|
+          stmt.result_meta
+          @columns = cols = stmt.result_fields.map{|c| output_identifier(c)}
+          col_count = cols.size
+          stmt.each do |result|
+            row = {}
+            col_count.times{|i| row[cols[i]] = result[i]}
+            yield row
+          end
+        end
+      end
+
+      private
+      
+      # Quote the string using the adapter instance method.
+      def literal_string(v)
+        "#{db.synchronize{|c| c.quote(v)}}"
+      end
+    end
+  end
+end