sequel 4.26.0 → 5.37.0

data/doc/migration.rdoc

@@ -23,7 +23,7 @@ you generally need to run Sequel's migrator with <tt>bin/sequel -m</tt>:
 
   sequel -m path/to/migrations postgres://host/database
 
-Migrations in Sequel use a very simple DSL via the <tt>Sequel.migration</tt>
+Migrations in Sequel use a DSL via the <tt>Sequel.migration</tt>
 method, and inside the DSL, use the <tt>Sequel::Database</tt> schema
 modification methods such as +create_table+ and +alter_table+.
 See the {schema modification guide}[rdoc-ref:doc/schema_modification.rdoc]
@@ -37,7 +37,7 @@ Here is a fairly basic Sequel migration:
     up do
       create_table(:artists) do
         primary_key :id
-        String :name, :null=>false
+        String :name, null: false
       end
     end
 
@@ -49,9 +49,8 @@ Here is a fairly basic Sequel migration:
 This migration has an +up+ block which adds an artist table with an integer primary key named id,
 and a varchar or text column (depending on the database) named +name+ that doesn't accept +NULL+ values.
 Migrations should include both up and +down+ blocks, with the +down+ block reversing
-the change made by up.  However, if you never need to be able to migrate down
-(i.e. you are one of the people that doesn't make mistakes), you can leave out
-the +down+ block.  In this case, the +down+ block just reverses the changes made by up,
+the change made by up.  However, if you never need to be able to migrate down,
+you can leave out the +down+ block.  In this case, the +down+ block just reverses the changes made by up,
 dropping the table.
 
 You can simplify the migration given above by using a reversible migration with a +change+
@@ -61,7 +60,7 @@ block:
     change do
       create_table(:artists) do
         primary_key :id
-        String :name, :null=>false
+        String :name, null: false
       end
     end
   end
@@ -102,7 +101,7 @@ from Sacramento, CA, USA, but now you want to branch out and include artists in
   Sequel.migration do
     up do
       add_column :artists, :location, String
-      from(:artists).update(:location=>'Sacramento')
+      from(:artists).update(location: 'Sacramento')
     end
 
     down do
@@ -117,18 +116,13 @@ just drops the +location+ column from the +artists+ table, reversing the actions
 block.
 
 Note that when updating the +artists+ table in the update, a plain dataset is used, <tt>from(:artists)</tt>.
-This looks a little weird, but you need to be aware that inside an up or +down+ block in a migration,
+This may look a little strange, but you need to be aware that inside an up or +down+ block in a migration,
 self always refers to the <tt>Sequel::Database</tt> object that the migration is being applied to.
 Since <tt>Database#from</tt> creates datasets, using <tt>from(:artists)</tt> inside the +up+ block creates
 a dataset on the database representing all columns in the +artists+ table, and updates it to set the
 +location+ column to <tt>'Sacramento'</tt>.  You should avoid referencing the <tt>Sequel::Database</tt>
 object directly in your migration, and always use self to reference it, otherwise you may run into problems.
 
-It is possible to use model classes inside migrations, as long as they are loaded into the ruby interpreter,
-but it's a bad habit as changes to your model classes can then break old migrations, and this breakage is
-often not caught until much later, such as when a new developer joins the team and wants to run all migrations
-to create their development database.
-
 == The +migration+ extension
 
 The migration code is not technically part of the core of Sequel.  It's not loaded by default as it
@@ -244,15 +238,13 @@ the +TimestampMigrator+.  They both have plusses and minuses:
 * Doesn't allow duplicate migrations
 * Doesn't allow missing migrations by default
 * Just stores the version of the last migration run
-* Good for single developer or small teams with close
-  communication
+* Good for single developer or small teams with close communication
 * Lower risk of undetected conflicting migrations
 * Requires manual merging of simultaneous migrations
 
 === +TimeStampMigrator+
 
-* More complex, use migration versions where the version should
-  represent a timestamp
+* More complex, uses migration versions where the version should represent a timestamp
 * Allows duplicate migrations (since you could have multiple in a given second)
 * Allows missing migrations (since you obviously don't have one every second)
 * Stores the file names of all applied migrations
@@ -360,7 +352,7 @@ you should give it some thought before using it.
 
 == Ignoring missing migrations
 
-In some cases, you may want to allow a migration in the database that does not exist in the filesystem (deploying to an older version of code without running a down migration when deploy auto-migrates, for example). If required, you can pass <tt>:allow_missing_migration_files => true</tt> as an option. This will stop errors from being raised if there are migrations in the database that do not exist in the filesystem.
+In some cases, you may want to allow a migration in the database that does not exist in the filesystem (deploying to an older version of code without running a down migration when deploy auto-migrates, for example). If required, you can pass <tt>allow_missing_migration_files: true</tt> as an option. This will stop errors from being raised if there are migrations in the database that do not exist in the filesystem.
 
 == Modifying existing migrations
 
@@ -369,7 +361,7 @@ Just don't do it.
 In general, you should not modify any migration that has been run on the database and been committed to
 the source control repository, unless the migration contains an error that causes data loss.  As long
 as it is possible to undo the migration without losing data, you should just add another migration
-that undoes the actions of the previous bad migration, and maybe does the correct action afterward.
+that undoes the actions of the previous bad migration, and does the correct action afterward.
 
 The main problem with modifying existing migrations is that you will have to manually modify any
 databases that ran the migration before it was modified.  If you are a single developer, that may be
@@ -409,17 +401,13 @@ artist per album to multiple artists per album:
   Sequel.migration do
     up do
       # Create albums_artists table
-      create_table(:albums_artists) do
-        foreign_key :album_id, :albums
-        foreign_key :artist_id, :artists
-        index [:album_id, :artist_id], :unique=>true
-      end
+      create_join_table(album_id: :albums, artist_id: :artists)
 
       # Insert one row in the albums_artists table
       # for each row in the albums table where there
       # is an associated artist
-      self[:albums_artists].insert([:album_id, :artist_id],
-       self[:albums].select(:id, :artist_id).exclude(:artist_id=>nil))
+      from(:albums_artists).insert([:album_id, :artist_id],
+       from(:albums).select(:id, :artist_id).exclude(artist_id: nil))
 
       # Drop the now unnecesssary column from the albums table
       drop_column :albums, :artist_id
@@ -431,18 +419,13 @@ artist per album to multiple artists per album:
       # If possible, associate each album with one of the artists
       # it was associated with.  This loses information, but
       # there's no way around that.
-      self[:albums_artists].
-       group(:album_id).
-       select{[album_id, max(artist_id).as(artist_id)]}.
-       having{artist_id >  0}.
-       all do |r|
-         self[:artists].
-          filter(:id=>r[:album_id]).
-          update(:artist_id=>r[:artist_id])
-       end
+      from(:albums).update(artist_id: from(:albums_artists).
+        select{max(artist_id)}.
+        where(album_id: Sequel[:albums][:id])
+      )
 
       # Drop the albums_artists table
-      drop_table(:albums_artists)
+      drop_join_table(album_id: :albums, artist_id: :artists)
     end
   end
 
@@ -486,21 +469,16 @@ or equal to that version will be migrated up.
 
 == Running migrations from a Rake task
 
-You can also incorporate migrations into a Rakefile. Here's an example
-using integer migration versions.
+You can also incorporate migrations into a Rakefile:
 
   namespace :db do
     desc "Run migrations"
     task :migrate, [:version] do |t, args|
-      require "sequel"
+      require "sequel/core"
       Sequel.extension :migration
-      db = Sequel.connect(ENV.fetch("DATABASE_URL"))
-      if args[:version]
-        puts "Migrating to version #{args[:version]}"
-        Sequel::Migrator.run(db, "db/migrations", target: args[:version].to_i)
-      else
-        puts "Migrating to latest"
-        Sequel::Migrator.run(db, "db/migrations")
+      version = args[:version].to_i if args[:version]
+      Sequel.connect(ENV.fetch("DATABASE_URL")) do |db|
+        Sequel::Migrator.run(db, "db/migrations", target: version)
       end
     end
   end
@@ -523,6 +501,12 @@ migration takes, you can use the <tt>-E</tt> option to +sequel+ to set up a
 +Database+ logger that logs to +STDOUT+.  You can also log that same output to
 a file using the <tt>-l</tt> option with a log file name.
 
+If you want to include a logger in the rake task above, add a +:logger+ option
+when calling Sequel.connect:
+
+  require "logger"
+  Sequel.connect(ENV.fetch("DATABASE_URL"), logger: Logger.new($stderr))
+
 == Using models in your migrations
 
 Just don't do it.
@@ -611,3 +595,45 @@ or:
 This usage is discouraged in new code, but will continue to be supported indefinitely.
 It is not recommended to convert old-style migration classes to the <tt>Sequel.migration</tt>
 DSL, but it is recommended to use the <tt>Sequel.migration</tt> DSL for all new migrations.
+
+== Database-specific migrations
+
+While not a recommended practice, it is sometimes necessary to have parts of migrations be
+database-specific .  You can use the Sequel::Database#database_type method to check which
+database the migration is being run on, and operate accordingly:
+
+  Sequel.migration do
+    up do
+      if database_type == :mysql
+        run 'MySQL specific code'
+      else
+        run 'Generic SQL code'
+      end
+    end
+  
+    down do
+      if database_type == :mysql
+        run 'MySQL specific code'
+      else
+        run 'Generic SQL code'
+      end
+    end
+  end
+
+== Using Database Extensions in Migrations
+
+If you need to use database extensions in migrations (e.g. +:pg_enum+), you should load the extension in the up or down block as appropriate.
+
+  Sequel.migration do
+    up do
+      extension :pg_enum
+
+      # migration here
+    end
+
+    down do
+      extension :pg_enum
+
+      # migration here
+    end
+  end

data/doc/model_dataset_method_design.rdoc

@@ -0,0 +1,129 @@
+= Model Dataset Method Design Guide
+
+How you design your model dataset methods can significantly affect the flexibility of your API for your model classes, as well as the performance.  The goal of this guide is to provide an example of how to design your model dataset methods for maximum flexibility and performance.
+
+== Flexibility: Use Single Method Per Task
+
+In general, it is recommended that you have a single method per task for maximum flexibility.  For example, let's say you need to retrieve all albums released in a given year, ordered by number of units sold descending, and only care about the id, name and number of units sold.  One way to do this is in your application code (outside the model), you can
+call the dataset methods directly:
+
+  Album.
+    select(:id, :name, :copies_sold).
+    where(release_year: params[:year].to_i).
+    order(Sequel.desc(:copies_sold)).
+    all
+
+One issue with this design is that it ties you to your current database schema, and will make it necessary to change your application code if your schema changes.  In general, it is better to encapsulate your code into a dataset method (or a class method, but a dataset method is more flexible):
+
+  class Album < Sequel::Model
+    dataset_module do
+      def all_albums_released_in_year(year)
+        select(:id, :name, :copies_sold).
+        where(release_year: year).
+        order(Sequel.desc(:copies_sold)).
+        all
+      end
+    end
+  end
+
+Then your application code just needs to call your dataset method:
+
+  Album.all_albums_released_in_year(params[:year].to_i)
+
+The advantage of this approach is that you can change your schema at any point in the future, and you should only need to change your model code, you should never need to change other application code.
+
+== Performance
+
+After designing your dataset methods for flexibility, stop.  Don't worry about performance until you need to worry about performance.  However, assuming you have profiled your application and profiling shows you can benefit from optimizing the above method, you can then consider the performance impact of future design choices.
+
+First, considering that the root cause of the performance issue may not be at the Sequel level, it may be at the database itself.  Use +EXPLAIN+ or the equivalent to analyze the query plan for the query in use, and see if there is something you can do to optimize it, such as adding an appropriate index.
+
+Second, assuming the performance issue is at the Sequel level, you need to understand that one of the best ways to improve performance in most ruby code is to reduce the number of objects allocated.  Here is the above code with comments showing datasets allocated:
+
+  def all_albums_released_in_year(year)
+    select(:id, :name, :copies_sold). # new dataset allocated
+    where(release_year: year).        # new dataset allocated
+    order(Sequel.desc(:copies_sold)). # new dataset allocated
+    all
+  end
+
+Third, you need to understand that Sequel has optimizations specifically designed to reduce the number of objects allocated, by caching intermediate datasets.  Unfortunately, those optimizations do not apply in this case.  The reason for this is that +select+, +where+, and +order+ can potentially receive arbitrary arguments, and enabling caching for them could easily lead to unbounded cache size (denial of service due to memory exhaustion).
+
+To allow intermediate dataset caching to work, you need to signal to Sequel that particular arguments to these methods should be cached, and you can do that by calling methods inside +dataset_module+ blocks such as +select+ and +order+.  These methods will add dataset methods to the model that can cache the returned dataset to optimize performance.  Here is an example using these methods:
+
+  class Album < Sequel::Model
+    dataset_module do
+      select :with_name_and_units, :id, :name, :copies_sold
+      order :by_units_sold, Sequel.desc(:copies_sold)
+
+      def all_albums_released_in_year(year)
+        with_name_and_units.
+        by_units_sold.
+        where(release_year: year).
+        all
+      end
+    end
+  end
+
+Performance aside, this does provide a slightly nicer and more readable internal API, though naming such methods can be problematic.
+
+By calling +select+ and +order+ here, Sequel expects that the created dataset methods may be called more than once on the same dataset, and it knows that the arguments to the underlying +select+ and +order+ methods are fixed, so it can cache the resulting datasets.  Let's comment the above example with dataset allocations:
+
+  def all_albums_released_in_year(year)
+    with_name_and_units.        # cached dataset returned 
+    by_units_sold.              # cached dataset returned
+    where(release_year: year).  # new dataset allocated
+    all
+  end
+
+Note that the order of methods here is important.  If you instead change the method chain to filter the dataset first, then no caching happens:
+
+  def all_albums_released_in_year(year)
+    where(release_year: year).  # new dataset allocated
+    with_name_and_units.        # new dataset allocated
+    by_units_sold.              # new dataset allocated
+    all
+  end
+
+This is because any time a new, uncached dataset is returned by a dataset method, all subsequent methods in the method chain cannot benefit from caching.
+
+Usually, when you are designing methods to process data based on user input, the user input affects the rows selected, and not the columns selected or the order in which the rows are returned.  Sequel is aware of this and has dataset methods that specifically take user input (arguments), interpret them as a filter condition and either:
+
+* Return all matching rows in an array (+where_all+)
+* Iterate over all matching rows (+where_each+)
+* Return first matching row (+first+)
+* Return first column in first matching row, assumes only a single column is selected (+where_single_value+)
+
+After calling these methods on a cached dataset a number of times (currently 3), Sequel will automatically build an optimized loader, cache it, and use it for future loads.  So the above example changes to:
+
+  def all_albums_released_in_year(year)
+    with_name_and_units.           # cached dataset returned
+    by_units_sold.                 # cached dataset returned
+    where_all(release_year: year)  # cached loader used
+  end
+
+This can significantly improve performance, up to 3x for complex method chains that only return a few rows.
+
+So the general advice on designing dataset methods for performance is:
+
+* Use +dataset_module+ methods to create named dataset methods that return cached datasets
+* If any filtering is to be done, have it done last using +where_all+, +where_each+, +first+, or +where_single_value+.
+
+By following this advice, you can significantly increase the performance of your model dataset code.
+
+=== Further Increasing Performance
+
+The best way to further increase performance at the Sequel level is to switch to using prepared statements.  This does require more significant changes to the API.  Here's an example using prepared statements:
+
+  class Album < Sequel::Model
+    ALBUMS_RELEASED_IN_YEAR = select(:id, :name, :copies_sold).
+      where(release_year: :$year).
+      order(Sequel.desc(:copies_sold)).
+      prepare(:all, :all_albums_released_in_year)
+
+    def self.all_albums_released_in_year(year)
+      ALBUMS_RELEASED_IN_YEAR.call(year: year)
+    end
+  end
+
+Note that when using prepared statements, you need to use a class method instead of a dataset method, as the SQL for the prepared statement must be fixed for the class.  This limits the flexibility of the method, since you can no longer call it on arbitrary datasets on the class.

data/doc/model_hooks.rdoc

@@ -4,7 +4,7 @@ This guide is based on http://guides.rubyonrails.org/activerecord_validations_ca
 
 == Overview
 
-Model hooks, also known as model callbacks, are used to specify actions that occur at a given point in a model instance's lifecycle, such as before or after the model object is saved, created, updated, destroyed, or validated.  There are also around hooks for all types, which wrap the before hooks, the behavior, and the after hooks.
+Model hooks are used to specify actions that occur at a given point in a model instance's lifecycle, such as before or after the model object is saved, created, updated, destroyed, or validated.  There are also around hooks for all types, which wrap the before hooks, the behavior, and the after hooks.
 
 == Basic Usage
 
@@ -51,7 +51,7 @@ Sequel calls hooks in the following order when saving an existing object:
 
 Note that all of the hook calls are the same, except that +around_create+, +before_create+ and +after_create+ are used for a new object, and +around_update+, +before_update+ and +after_update+ are used for an existing object.  Note that +around_save+, +before_save+, and +after_save+ are called in both cases.
 
-Also note that the validation hooks are not called if the <tt>:validate => false</tt> option is passed to save.  However, the validation hooks are called if you call <tt>Model#valid?</tt> manually:
+Note that the validation hooks are still called if <tt>validate: false</tt> option is passed to save.  If you call <tt>Model#valid?</tt> manually, then only the validation hooks are called:
 
 * +around_validation+
   * +before_validation+
@@ -67,26 +67,19 @@ Sequel calls hooks in the following order when destroying an existing object:
 
 Note that these hooks are only called when using <tt>Model#destroy</tt>, they are not called if you use <tt>Model#delete</tt>.
 
-== Special Hook-Related Instance Variables
-
-For after_save hooks, a @was_new instance variable is present that indicates whether the record was a new record that was just inserted, or an existing record that was updated.  Sequel marks a record as existing as soon as it inserts the record, so in an after_save or after_create hook, the instance is no longer considered new.  You have to check @was_new to see if the record was inserted.  This exists so that you don't have to have separate after_create and after_update hooks that are mostly the same and only differ slightly depending on whether the record was a new record.
-
-For after_update hooks, a @columns_updated instance variable is present that is a hash of the values used to update the row (keys are column symbols, values are column values).  This should be used by any code that wants to check what columns and values were used during the update.  You can't just check the current values of the instance, since Sequel offers ways to manually specify which columns to use during the save.
-
 == Transaction-related Hooks
 
-There are four other model hooks that <tt>Sequel::Model</tt> supports, all related to transactions.  These are +after_commit+, +after_rollback+, +after_destroy_commit+, and +after_destroy_rollback+.  +after_commit+ is called after the transaction in which you saved the object commits, only if it commits.  +after_rollback+ is called after the transaction in which you saved the object rolls back, if it rolls back.  +after_destroy_commit+ is called after the transaction in which you destroyed the object commits, if it commits.  +after_destroy_rollback+ is called after the transaction in which you destroyed the object rolls back, if it rolls back.
-
-If you aren't using transactions when saving or destroying model objects, and there isn't a currently open transaction, +after_commit+ and +after_destroy_commit+ will be called after +after_save+ and +after_destroy+, respectively, and +after_rollback+ and +after_destroy_rollback+ won't be called at all.
-
-The purpose of these hooks is dealing with external systems that are interacting with the same database.  For example, let's say you have a model that stores a picture, and you have a background job library that makes thumbnails of all of the pictures.  So when a model object is created, you want to add a background job that will create the thumbnail for the picture.  If you used after_save for this and transactions are being used, you are subject to a race condition where the background job library will check the database table for the record before the transaction that saved the record commits, and it won't be able to see the record's data.  Using after_commit, you are guaranteed that the background job library will not get notified of the record until after the transaction commits and the data is viewable.
-
-Note that when using the after_commit or after_rollback hooks, you don't know whether the saved object was newly created or updated.  If you only want to run an action after commit of a newly created record, you need to use the Database's after_commit inside the model's after_create hook:
+Sequel::Model no longer offers transaction hooks for model instances.  However, you can use the database transaction hooks inside model +before_save+ and +after_save+ hooks:
 
   class Album < Sequel::Model
-    def after_create
+    def before_save
+      db.after_rollback{rollback_action}
+      super
+    end
+
+    def after_save
       super
-      db.after_commit{update_external_cache}
+      db.after_commit{commit_action}
     end
   end
 
@@ -96,13 +89,13 @@ Sequel does not provide a simple way to turn off the running of save/create/upda
 
 However, you should note that there are plenty of ways to modify the database without saving a model object.  One example is by using plain datasets, or one of the model's dataset methods:
 
-  Album.where(:name=>'RF').update(:copies_sold=>Sequel.+(:copies_sold, 1))
+  Album.where(name: 'RF').update(copies_sold: Sequel.+(:copies_sold, 1))
   # UPDATE albums SET copies_sold = copies_sold + 1 WHERE name = 'RF'
 
 In this case, the +update+ method is called on the dataset returned by <tt>Album.where</tt>.  Even if there is only a single object with the name RF, this will not call any hooks.  If you want model hooks to be called, you need to make sure to operate on a model object:
 
-  album = Album.first(:name=>'RF')
-  album.update(:copies_sold=>album.copies_sold + 1)
+  album = Album.first(name: 'RF')
+  album.update(copies_sold: album.copies_sold + 1)
   # UPDATE albums SET copies_sold = 2 WHERE id = 1
 
 For the destroy hooks, you need to make sure you call +destroy+ on the object:
@@ -121,11 +114,11 @@ However, skipping hooks is a bad idea in general and should be avoided.  As ment
 
 The +this+ dataset works just like any other dataset, so you can call +update+ on it to modify it:
 
-  album.this.update(:copies_sold=>album.copies_sold + 1)
+  album.this.update(copies_sold: album.copies_sold + 1)
 
 If you want to insert a row into the model's table without running the creation hooks, you can use <tt>Model.insert</tt> instead of <tt>Model.create</tt>:
 
-  Album.insert(:name=>'RF') # does not run hooks
+  Album.insert(name: 'RF') # does not run hooks
 
 == Canceling Actions in Hooks
 
@@ -138,8 +131,6 @@ Sometimes want to cancel an action in a before hook, so the action is not perfor
     end
   end
 
-For backwards compatibility, Sequel also treats the <tt>before_*</tt> hook methods returning +false+ as an indication that the action should be canceled.  This usage is discouraged in new code, but you should be aware of this behavior so that you do not inadvertently return false.
-
 For around hooks, neglecting to call +super+ halts hook processing in the same way as calling +cancel_action+ in a before hook.  It's probably a bad idea to use +cancel_action+ hook processing in after hooks, or after yielding in around hooks, since by then the main processing has already taken place.
 
 By default, Sequel runs hooks other than validation hooks inside a transaction, so if you cancel the action by calling +cancel_action+ in any hook, Sequel will rollback the transaction.  However, note that the implicit use of transactions when saving and destroying model objects is conditional (it depends on the model instance's +use_transactions+ setting and the <tt>:transaction</tt> option passed to save).
@@ -216,7 +207,6 @@ This allows the following general principles to be true:
 
 * before hooks are run in reverse order of inclusion
 * after hooks are run in order of inclusion
-* returning false in any before hook will pass the false value down the hook method chain, halting the hook processing.
 
 So if you define the same before hook in both a model and a plugin that the model uses, the hooks will be called in this order: