viking-sequel 3.10.0

data/doc/release_notes/3.8.0.txt

@@ -0,0 +1,151 @@
+New Features
+------------
+
+* Dataset#each_server was added, allowing you to run the same query
+  (most likely insert/update/delete) on all shards.  This is useful
+  if you have a sharded database but have lookup tables that should
+  be identical on all shards.  It works by yielding copies of the
+  current dataset that are tied to each server/shard:
+  
+    DB[:table].filter(:id=>1).each_server do |ds|
+      ds.update(:name=>'foo')
+    end
+
+* Database#each_server was added, allowing you to run schema
+  modification methods on all shards.  It works by yielding a
+  new Sequel::Database object for each shard, that will connect to
+  only that shard:
+  
+    DB.each_server do |db|
+      db.create_table(:t){Integer :num}
+    end
+    
+* You can now add and remove servers/shards from the connection
+  pool while Sequel is running:
+  
+    DB.add_servers(:shard1=>{:host=>'s1'}, :shard2=>{:host=>'s2'})
+    DB.remove_servers(:shard1, :shard2)
+    
+* When you attempt to disconnect from a server that has connections
+  currently in use, Sequel will now schedule those connections to
+  be disconnected when they are returned to the pool.  Previously,
+  Sequel disconnected available connections, but ignored connections
+  currently in use, so it wasn't possible to guarantee complete
+  disconnection from the server.  Even with this new feature, you can
+  only guarantee eventual disconnection, since disconnection of
+  connections in use happens asynchronously.
+  
+* Database#disconnect now accepts a :servers option specifying the
+  server(s) from which to disconnect.  This should be a symbol or
+  array of symbols representing servers/shards.  Only those specified
+  will be disconnected:
+  
+    DB.disconnect(:servers=>[:shard1, :shard2])
+
+* A validates_type validation was added to the validation_helpers
+  plugin.  It allows you to check that a given column contains
+  the correct type.  I can be helpful if you are also using the
+  serialization plugin to store serialized ruby objects, by making
+  sure that the objects are of the correct type (e.g. Hash):
+  
+    def validate
+      validates_type(Hash, :options)
+    end
+
+* Sequel::SQL::Expression#== is now supported for all expressions:
+
+    :column.qualify(:table).cast(:type) == \
+      :column.qualify(:table).cast(:type)
+    # => true
+    :column.qualify(:table).cast(:type) == \
+      :other_column.qualify(:table).cast(:type)
+    # => false
+
+* When using the generic File type to create blob columns on
+  MySQL, you can specify the specific database type by using the
+  :size option (with :tiny, :medium, and :long values recognized):
+  
+    DB.create_table(:docs){File :body, :size=>:long} # longblob
+
+* The mysql adapter will now default to using mysqlplus, falling
+  back to use mysql.  mysqlplus is significantly better for threaded
+  code because queries do not block the entire interpreter.
+
+* The JDBC adapter is now able to detect certain types of disconnect
+  errors.
+  
+* ConnectionPool.servers and Database.servers were added, which
+  return an array of symbols specifying the servers/shards in use.
+
+Other Improvements
+------------------
+
+* The single-threaded connection pool now raises
+  DatabaseConnectionErrors if unable to connect, so it now operates
+  more similarly to the default connection pool.
+
+* The single-threaded connection pool now operates more similar
+  to the default connection pool when given a nonexistent server.
+
+* PGErrors are now correctly converted to DatabaseErrors in the
+  postgres adapter when preparing statements or executing prepared
+  statements.
+  
+* DatabaseDisconnectErrors are now raised correctly in the postgres
+  adapter if the connection status is not OK after a query raises an
+  error.
+
+* In the mysql adapter, multiple statements in a single query should
+  now be handled correctly in the all cases, not just when using
+  Dataset#each.  So you can now submit multiple queries in a single
+  string to Database#run.
+
+* Model object creation on Microsoft SQL Server 2000 once again
+  works correctly.  Previously, an optimization was used that was
+  only supported on 2005+.
+
+* Backslashes are no longer doubled inside string literals when
+  connecting to Microsoft SQL Server.
+  
+* The ORDER clause now correctly comes after the HAVING clause on
+  Microsoft SQL Server.
+  
+* Sequel now checks that there is an active transaction before
+  rolling back transactions on Microsoft SQL Server, since
+  there are cases where Microsoft SQL Server will roll back
+  transactions implicitly.
+  
+* Blobs are now handled correctly when connecting to H2.
+
+* 64-bit integers are now handled correctly in JDBC prepared
+  statements.
+
+* In the boolean_readers plugin, correctly handle columns not in
+  the db_schema, and don't raise an error if the model's columns
+  can't be determined.
+  
+* In the identity_map plugin, remove instances from the cache if they
+  are deleted or destroyed.
+
+Backwards Compatibility
+-----------------------
+
+* Dataset::FROM_SELF_KEEP_OPTS was merged into
+  Dataset::NON_SQL_OPTIONS.  While used in different places, they
+  were used for the same purpose, and entries missing from one should
+  have been included in the other.
+
+* The connection pool internals changed substantially.  Now,
+  ConnectionPool #allocated and #available_connections will return
+  nil instead of an array or hash if they are called with a
+  nonexistent server.  These are generally only used internally,
+  though they are part of the public API.  #created_count and #size
+  still return the size of the :default server when called with a
+  nonexistent server, though.
+
+* The meta_eval and metaclass private methods were removed from
+  Sequel::MetaProgramming (only the meta_def public method remains).
+  If you want these methods, use the metaid gem.
+
+* The irregular ox->oxen pluralization rule was removed from the
+  default inflections, as it screws up the more common box->boxes.

data/doc/release_notes/3.9.0.txt

@@ -0,0 +1,233 @@
+New Features
+------------
+
+* The ConnectionPool classes were refactored from 2 separate
+  classes to a 5 class hierarchy, with one main class and 4 
+  subclasses, one for each combination of sharding and threading.
+  
+  The primary reason for this refactoring is to make it so that
+  the user doesn't have to pay a performance penalty for sharding
+  if they aren't using it.  A connection pool that supports sharding
+  is automatically used if the :servers option is used when setting
+  up the database connection.
+  
+  In addition, the default connection pool no longer contains
+  the code to schedule future disconnections of currently allocated
+  connections.  The sharded connection pool must be used if that
+  feature is desired.
+  
+  The unsharded connection pools are about 25-30% faster than the
+  sharded versions.
+
+* An optimistic_locking plugin was added to Sequel::Model.  This
+  plugin implements a simple database-independent locking mechanism
+  to ensure that concurrent updates do not override changes:
+
+    class Person < Sequel::Model
+      plugin :optimistic_locking
+    end
+    p1 = Person[1]
+    p2 = Person[1]
+    # works
+    p1.update(:name=>'Jim')
+    # raises Sequel::Plugins::OptimisticLocking::Error
+    p2.update(:name=>'Bob')
+
+  In order for this plugin to work, you need to make sure that the
+  database table has a lock_version column (or other column you name
+  via the lock_column class level accessor) that defaults to 0.
+
+  The optimistic_locking plugin does not work with the
+  class_table_inheritance plugin.
+
+* Dataset#unused_table_alias was added, which takes a symbol and
+  returns either that symbol or a new symbol which can be used as
+  a table alias when joining a table to the dataset.  The symbol
+  returned is guaranteed to not already be used by the dataset:
+
+    DB[:test].unused_table_alias(:blah) # => :blah
+    DB[:test].unused_table_alias(:test) # => :test_0
+    
+  The use case is when you need to join a table to a dataset, where
+  the table may already be used inside the dataset, and you want
+  to generate a unique alias:
+  
+    ds.join(:table.as(ds.unused_table_alias(:table)), ...)
+    
+* The Sequel::ValidationFailed exception now has an errors accessor
+  which returns the Sequel::Model::Errors instance with the
+  validation errors.  This can be helpful in situations where a
+  generalized rescue is done where the model object reference is
+  not available.
+
+* bin/sequel now works without an argument, which is useful for
+  testing SQL generation (and not much else).
+
+* Support SELECT ... INTO in the MSSQL adapter, using Dataset#into,
+  which takes a table argument.
+  
+* You can now provide your own connection pool class via the
+  :pool_class option when instantiating the database.
+
+Other Improvements
+------------------
+
+* IN/NOT IN constructs with an empty array are now handled properly.
+  
+    DB[:table].filter(:id=>[]) # IN
+    DB[:table].exclude(:id=>[]) # NOT IN
+    
+  Before, the IN construct would mostly work, other than some minor
+  differences in NULL semantics.  However, the NOT IN construct
+  would not work.  Sequel now handles the NOT IN case using an
+  expression that evaluates to true.
+  
+* If using an IN/NOT IN construct with multiple columns and a dataset
+  argument, where multiple column IN/NOT IN support is emulated, a
+  separate query is done to get the records, which is then handled
+  like an array of values.  This means that the following type of
+  query now works on all tested databases:
+  
+    DB[:table1].filter([:id1, :id2]=>DB[:table2].select(:id1, :id2))
+
+* Schemas and aliases are now handled correctly when eager graphing.
+
+* Implicitly qualified symbols are now handled correctly in update
+  statements, useful if you are updating a joined dataset and need
+  to reference a column that appears in multiple tables.
+
+* The active_model plugin has been brought up to date with
+  activemodel 3.0 beta (though it doesn't work on edge).
+  Additionally, the active_model plugin now requires active_model
+  in order to use ActiveModel::Naming.
+
+* In the schema_dumper extension, always include the varchar limit,
+  even if it is 255 columns (the default).  This makes it so that
+  PostgreSQL will use a varchar(255) column instead of a text column
+  when restoring a schema dump of a varchar(255) column from another
+  database.
+  
+* You can now load adapters from outside the Sequel lib directory,
+  now they just need to be in a sequel/adapters directory somewhere
+  in the LOAD_PATH.
+
+* You can now load extensions from outside the Sequel lib directory
+  using Sequel.extension.  External extensions need to be in a
+  sequel/extensions directory somewhere in the LOAD_PATH.
+
+* Using bound variables for limit and offset in prepared statements
+  now works correctly.
+
+* Performance of prepared statements was improved in the native
+  SQLite adapter.
+  
+* The schema_dumper extension now passes the options hash from
+  dump_*_migration to Database#tables.
+
+* In the single_table_inheritance plugin, qualify the sti_key column
+  with the table name, so that subclass datasets can safely be joined
+  to other tables having the same column name.
+
+* In the single_table_inheritance plugin, handle case where the
+  sti_key value is nil or '' specially, so that those cases
+  always return an instance of the main model class.  This fixes
+  issues if constantize(nil) returns Object instead of raising
+  an exception.
+  
+* No longer use Date#to_s for literalization, always use ISO8601
+  format for dates.
+  
+* A couple lambdas which were instance_evaled were changed to procs
+  for ruby 1.9.2 compatibility.
+  
+* MSSQL emulated offset support was simplified to only use one
+  subquery, and made to work correctly on ruby 1.9.
+  
+* Emulate multiple column IN/NOT IN on H2, since it doesn't handle
+  all cases correctly.
+  
+* ODBC timestamps are now handled correctly if the database_timezone
+  is nil.
+  
+* ArgumentErrors raised when running queries in the ODBC adapter are
+  now raised as DatabaseErrors.
+  
+* Attempting to use DISTINCT ON on SQLite now raises an error before
+  sending the query to the database.
+  
+* The options hash passed to the database connection method is no
+  longer modified.  However, there may be additional options
+  present in Database#opts that weren't specified by the options
+  hash passed to the database connection method.
+  
+* Make Dataset#add_graph_aliases handle the case where the dataset
+  has not yet been graphed.
+
+* You can now provide an SQL::Identifier as a 4th argument to
+  Dataset#join_table, and unsupported arguments are caught and an
+  exception is raised.
+  
+* The gem specification has been moved out of the Rakefile, so
+  that the gem can now be built without rake, and works well with
+  gem build and bundler.
+  
+* The Rakefile no longer assumes the current directory is in the
+  $LOAD_PATH, so it should work correctly on ruby 1.9.2.
+
+* All internal uses of require are now thread safe.
+
+* Empty query parameter keys in connection strings are now ignored
+  instead of raising an exception.
+  
+* The specs were changed so that you can run them in parallel.
+  Previously there was a race condition in the migration extension
+  specs.
+
+Backwards Compatibility
+-----------------------
+
+* If you plan on using sharding at any point, you now must pass
+  a :servers option when connecting to the database, even if it is
+  an empty hash.  You can no longer just call Database#add_servers
+  later.
+
+* The connection_proc and disconnection_proc accessors were removed
+  from the connection pools, so you can no longer modify the procs
+  after the connection pool has been instantiated.  You must now
+  provide the connection_proc as the block argument when
+  instantiating the pool, and the disconnection_proc via the
+  :disconnection_proc option.
+
+* In the hash passed to Dataset#update, symbol keys with a double
+  embedded underscore are now considerated as implicit qualifiers,
+  instead of being used verbatim.  If you have a column that includes
+  a double underscore, you now need to wrap it in an SQL::Identifier
+  or use a String instead.
+  
+* The connection pools no longer convert non-StandardError based
+  exceptions to RuntimeErrors.  Previously, all of the common adapters
+  turned this feature off, so there is no change for most users.
+
+* Sequel::ConnectionPool is now considered an abstract class and
+  should not be instantiated directly.  Use ConnectionPool.get_pool
+  to return an instance of the appropriate subclass.
+
+* The Sequel::SingleThreadedPool constant is no longer defined.
+
+* The private Dataset#eager_unique_table_alias method was removed,
+  use the new public Dataset#unused_table_alias method instead, which
+  has a slightly different API.
+
+* The private Dataset#eager_graph_qualify_order method was removed,
+  used Dataset#qualified_expression instead.
+  
+* The private Sequel::Model class methods plugin_gem_location and
+  plugin_gem_location_old have been removed.
+  
+* Gems built with the rake tasks now show up in the root directory
+  instead of the pkg subdirectory, and no tarball package is created.
+
+Other News
+----------
+
+* Sequel now has an official blog at http://sequel.heroku.com.

data/doc/schema.rdoc

@@ -0,0 +1,36 @@
+= Schema Modifications with Sequel
+
+In addition to its support for getting data from databases, Sequel has good support for modifying the schema of databases.  Unfortunately, the documentation from this is spread out in various places in the Sequel RDoc.
+
+The recommended way to set up schema modifications in Sequel is through migrations. Migrations are modifications to the schema that handle modifying the schema in two directions, up and down, with down reversing the changes made by up. Migrations are stored in sequentially numbered files inside a directory (see the Sequel::Migrator documentation for file naming format details).  The migrations can be run on the database using the sequel command line tool (the -m and -M switches).
+
+The format of the individual migration files themselves is explained in the Sequel::Migration documentation.  Each migration file contains a single migration class.  The migration class acts a proxy for the related database (given on the command line if the sequel command line tool is used, or by the db argument to Sequel::Migration#apply if the API is used). The methods that can be used inside Sequel::Migration#up or Sequel::Migration#down are just Sequel::Database instance methods, such as create_table, drop_table, and alter_table.  Most database methods that alter the schema take regular arguments, but create_table and alter_table take a block.  The methods you can use inside the create_table block are documented in Sequel::Schema::Generator, and the methods you can use inside the alter_table block are documented in Sequel::Schema::AlterTableGenerator.
+
+Migrations are not required, you can just call the schema modification methods directly on the database object.  This is often done in test code and examples.  However, it is recommended that you use the migration framework unless the database schema will not be changing in the future, as it provides a way to easily handle modifications to existing database schema.
+
+Sequel has the ability to have database independent migrations using ruby classes as types.  When you use a ruby class as a type, Sequel translates it to the most comparable type in the database you are using.  Here's an example using all supported types:
+
+    DB.create_table(:cats) do
+      primary_key :id, :type=>Integer             # integer
+      String :a                                   # varchar(255)
+      String :a2, :size=>50                       # varchar(50)
+      String :a3, :fixed=>true                    # char(255)
+      String :a4, :fixed=>true, :size=>50         # char(50)
+      String :a5, :text=>true                     # text
+      column :b, File                             # blob
+      Fixnum :c                                   # integer
+      foreign_key :d, :other_table, :type=>Bignum # bigint
+      Float :e                                    # double precision
+      BigDecimal :f                               # numeric
+      BigDecimal :f2, :size=>10                   # numeric(10)
+      BigDecimal :f3, :size=>[10, 2]              # numeric(10, 2)
+      Date :g                                     # date
+      DateTime :h                                 # timestamp
+      Time :i                                     # timestamp
+      Time :i2, :only_time=>true                  # time
+      Numeric :j                                  # numeric
+      TrueClass :k                                # boolean
+      FalseClass :l                               # boolean
+    end
+
+Basically, if you use one of the ruby classes above, it will translate into a database specific type.  If you use a lowercase method, symbol, or string to specify the type, Sequel won't attempt to translate it.

data/doc/sharding.rdoc

@@ -0,0 +1,113 @@
+= Read-Only Slaves/Writable Master and Database Sharding
+
+Sequel has support for read only slave databases
+with a writable master database, as well as database sharding (where you can
+pick a server to use for a given dataset).  Support for both
+features is database independent, and should work for all database adapters
+included with Sequel.
+
+== The :servers Database option
+
+Both features use the :servers Database option.  The :servers option should
+be a hash with symbol keys and values that are either hashes or procs that
+return hashes.  Note that all servers should have the same schema for all
+tables you are accessing, unless you really know what you are doing.
+
+== Master and Slave Database Configurations
+
+=== Single Read-Only Slave, Single Master
+
+To use a single, read-only slave that handles SELECT queries, the following
+is the simplest configuration:
+
+  DB=Sequel.connect('postgres://master_server/database', \
+    :servers=>{:read_only=>{:host=>'slave_server'}})
+
+This will use the slave_server for SELECT queries and master_server for
+other queries.
+
+=== Multiple Read-Only Slaves, Single Master
+
+Let's say you have 4 slave database servers with names slave_server0,
+slave_server1, slave_server2, and slave_server3.
+
+  DB=Sequel.connect('postgres://master_server/database', \
+    :servers=>{:read_only=>proc{|db| {:host=>db.get_slave_host}}})
+  def DB.get_slave_host
+    @current_host ||= -1
+    "slave_server#{(@current_host+=1)%4}"
+  end
+
+This will use one of the slave servers for SELECT queries and use the
+master_server for other queries.  It's also possible to pick a random host
+instead of using the round robin approach presented above, but that can result
+in less optimal resource usage.
+
+=== Multiple Read-Only Slaves, Multiple Masters
+
+This involves the same basic idea as the multiple slaves, single master, but
+it shows that the master database is named :default.  So for 4 masters and
+4 slaves:
+
+  DB=Sequel.connect('postgres://master_server/database', \
+    :servers=>{:read_only=>proc{|db| {:host=>db.get_slave_host}}, \
+      :default=>proc{|db| {:host=>db.get_master_host}}})
+  def DB.get_slave_host
+    @current_slave_host ||= -1
+    "slave_server#{(@current_slave_host+=1)%4}"
+  end
+  def DB.get_master_host
+    @current_master_host ||= -1
+    "master_server#{(@current_master_host+=1)%4}"
+  end
+  
+== Sharding
+
+There is specific support in Sequel for handling master/slave database
+combinations, with the only necessary setup being the database configuration.
+However, since sharding is always going to be implementation dependent, Sequel
+supplies the basic infrastructure, but you have to tell it which server to use
+for each dataset.  Let's assume a simple scenario, a distributed rainbow
+table for SHA-1 hashes, sharding based on the first hex character (for a total
+of 16 shards).  First, you need to configure the database:
+
+  servers = {}
+  (('0'..'9').to_a + ('a'..'f').to_a).each do |hex|
+    servers[hex.to_sym] = {:host=>"hash_host_#{hex}"}
+  end
+  DB=Sequel.connect('postgres://hash_host/hashes', :servers=>servers)
+  
+This configures 17 servers, the 16 shard servers (/hash_host_[0-9a-f]/), and 1
+default server which will be used if no shard is specified ("hash_host").  If
+you want the default server to be one of the shard servers (e.g. hash_host_a),
+it's easiest to do:
+
+  DB=Sequel.connect('postgres://hash_host_a/hashes', :servers=>servers)
+
+That will still set up a second pool of connections for the default server,
+since it considers the default server and shard servers independent.  Note that
+if you always set the shard on a dataset before using it in queries, it will
+not attempt to connect to the default server.  Sequel may use the default
+server in queries it generates itself, such as to get column names or table
+schemas, so you should always have a default server that works.
+
+To set the shard for a given query, you use the Dataset#server method:
+
+  DB[:hashes].server(:a).filter(:hash=>/31337/)
+  
+That will return all matching rows on the hash_host_a shard that have a hash
+column that contains 31337.
+
+Rainbow tables are generally used to find specific hashes, so to save some
+work, you might want to add a method to the dataset that automatically sets
+the shard to use.  This is fairly easy using a Sequel::Model:
+
+  class Rainbow < Sequel::Model(:hashes)
+    def_dataset_method(:plaintext_for_hash) do |hash|
+      raise(ArgumentError, 'Invalid SHA-1 Hash') unless /\A[0-9a-f]{40}\z/.match(hash)
+      row = self.server(hash[0...1].to_sym).first(:hash=>hash)
+      row[:plaintext] if row
+    end
+  end
+  
+  Rainbow.plaintext_for_hash("e580726d31f6e1ad216ffd87279e536d1f74e606")