sequel 3.22.0 → 3.23.0

data/CHANGELOG

@@ -1,3 +1,31 @@
+=== 3.23.0 (2011-05-02)
+
+* Migrate issue tracker from Google Code to GitHub Issues (jeremyevans)
+
+* Add support for filtering by associations to model datasets (jeremyevans)
+
+* Don't call insert_select when saving a model that doesn't select all columns of the table (jeremyevans)
+
+* Fix bug when using :select=>[] option for a many_to_many association (jeremyevans)
+
+* Add a columns_introspection extension that attempts to skip database queries by introspecting selected columns (jeremyevans)
+
+* When combining old integer migrations and new timestamp migrations, make sure old integer migrations are all applied first (jeremyevans)
+
+* Support dynamic callbacks to customize regular association loading at query time (jeremyevans)
+
+* Support cascading of eager loading with dynamic callbacks for both eager and eager_graph (jeremyevans)
+
+* Make the xml_serializer plugin handle namespaced models by using __ instead of / as a separator (jeremyevans)
+
+* Allow the :eager_grapher association proc to accept a single hash instead of 3 arguments (jfirebaugh)
+
+* Support dynamic callbacks to customize eager loading at query time (jfirebaugh, jeremyevans)
+
+* Fix bug in the identity_map plugin for many_to_one associations when the association reflection hadn't been filled in yet (funny-falcon)
+
+* Add serialization_modification_detection plugin for detecting changes in serialized columns (jeremyevans) (#333)
+
 === 3.22.0 (2011-04-01)
 
 * Add disconnect detection to tinytds adapter, though correct behavior may require an update to tiny_tds (cult_hero)

data/README.rdoc

@@ -20,7 +20,7 @@ toolkit for Ruby.
 * {Website}[http://sequel.rubyforge.org]
 * {Blog}[http://sequel.heroku.com]
 * {Source code}[http://github.com/jeremyevans/sequel]
-* {Bug tracking}[http://code.google.com/p/ruby-sequel/issues/list]
+* {Bug tracking}[http://github.com/jeremyevans/sequel/issues]
 * {Google group}[http://groups.google.com/group/sequel-talk]
 * {RDoc}[http://sequel.rubyforge.org/rdoc]
 
@@ -685,6 +685,20 @@ Associations can be eagerly loaded via +eager+ and the <tt>:eager</tt> associati
 
 In addition to using +eager+, you can also use +eager_graph+, which will use a single query to get the object and all associated objects.  This may be necessary if you want to filter or order the result set based on columns in associated tables.  It works with cascading as well, the syntax is exactly the same.  Note that using eager_graph to eagerly load multiple *_to_many associations will cause the result set to be a cartesian product, so you should be very careful with your filters when using it in that case.
 
+You can dynamically customize the eagerly loaded dataset by using using a proc.  This proc is passed the dataset used for eager loading, and should return a modified copy of that dataset:
+
+  # Eagerly load only replies containing 'foo'
+  Post.eager(:replies=>proc{|ds| ds.filter(text.like('%foo%'))}).all
+
+This also works when using +eager_graph+, in which case the proc is called with dataset to graph into the current dataset:
+
+  Post.eager_graph(:replies=>proc{|ds| ds.filter(text.like('%foo%'))}).all
+
+You can dynamically customize eager loads for both +eager+ and +eager_graph+ while also cascading, by making the value a single entry hash with the proc as a key, and the cascaded associations as the value:
+
+  # Eagerly load only replies containing 'foo', and the person and tags for those replies
+  Post.eager(:replies=>{proc{|ds| ds.filter(text.like('%foo%'))}=>[:person, :tags]}).all
+
 === Extending the underlying dataset
 
 The obvious way to add table-wide logic is to define class methods to the model class definition. That way you can define subsets of the underlying dataset, change the ordering, or perform actions on multiple records:

data/doc/association_basics.rdoc

@@ -293,32 +293,6 @@ Examples:
   @artist.remove_album(@album)
   @artist.remove_all_albums
 
-== Dataset Method
-
-In addition to the above methods, associations also add a instance method
-ending in +_dataset+ that returns a dataset representing the objects in the associated table:
-
-  @album.artist_id
-  # 10
-  @album.artist_dataset
-  # SELECT * FROM artists WHERE (id = 10)
-  
-  @artist.id
-  # 20
-  @artist.albums_dataset
-  # SELECT * FROM albums WHERE (artist_id = 20)
-
-The association dataset is just like any other Sequel dataset, in that
-it can be further filtered, ordered, etc.:
-
-  @artist.albums_dataset.
-   filter(:name.like('A%')).
-   order(:copies_sold).
-   limit(10)
-  # SELECT * FROM albums
-  # WHERE ((artist_id = 20) AND (name LIKE 'A%'))
-  # ORDER BY copies_sold LIMIT 10
-
 == Caching
 
 Associations are cached after being retrieved:
@@ -352,13 +326,112 @@ instance method:
   @album.artists # [<Artist ...>, ...]
   @album.associations[:artists] # [<Artist ...>, ...]
 
-Note that while the association method caches associated objects, if you
-retrieve access through the association dataset directly, the results
-will not be cached:
+== Dataset Method
+
+In addition to the above methods, associations also add a instance method
+ending in +_dataset+ that returns a dataset representing the objects in the associated table:
+
+  @album.artist_id
+  # 10
+  @album.artist_dataset
+  # SELECT * FROM artists WHERE (id = 10)
+  
+  @artist.id
+  # 20
+  @artist.albums_dataset
+  # SELECT * FROM albums WHERE (artist_id = 20)
+
+The association dataset is just like any other Sequel dataset, in that
+it can be further filtered, ordered, etc.:
+
+  @artist.albums_dataset.
+   filter(:name.like('A%')).
+   order(:copies_sold).
+   limit(10)
+  # SELECT * FROM albums
+  # WHERE ((artist_id = 20) AND (name LIKE 'A%'))
+  # ORDER BY copies_sold LIMIT 10
+
+Records retrieved using the +_dataset+ method are not cached in the
+associations cache.
 
   @album.artists_dataset.all # [<Artist ...>, ...]
   @album.associations[:artists] # nil
 
+== Dynamic Association Modification
+
+Similar to the +_dataset+ method, you can provide a block to the association
+method to customize the dataset that will be used to retrieve the records.  So
+you can apply a filter in either of these two ways:
+
+  @artist.albums_dataset.filter(:name.like('A%'))
+  @artist.albums{|ds| ds.filter(:name.like('A%'))}
+
+While they both apply the same filter, using the +_dataset+ method does not
+apply any of the association callbacks or handle association reciprocals (see
+below for details about callbacks and reciprocals).  Using a block instead handles
+all those things, and also caches its results in the associations cache (ignoring
+any previously cached value).
+
+Note that if you are using ruby 1.8.6, you can't pass a block to the association
+method, you have to pass a proc as an argument:
+
+  @artist.albums(proc{|ds| ds.filter(:name.like('A%'))})
+
+== Filtering By Associations
+
+In addition to using the association method to get associated objects, you
+can also use associated objects in filters.  For example, while to get
+all albums for a given artist, you would usually do:
+
+  @artist.albums
+  # or @artist.albums_dataset for a dataset
+
+You can also do the following:
+
+  Album.filter(:artist=>@artist).all
+  # or leave off the .all for a dataset
+
+For filtering by a single association, this isn't very useful.  However, unlike
+using the association method, using a filter allows you to filter by multiple
+associations:
+
+  Album.filter(:artist=>@artist, :publisher=>@publisher)
+
+This will return all albums by that artist and published by that publisher.
+This isn't possible using just the association method approach, though you
+can combine the approaches:
+
+  @artist.albums_dataset.filter(:publisher=>@publisher)
+
+This doesn't just work for +many_to_one+ associations, it also works for
++one_to_one+, +one_to_many+, and +many_to_many+ associations:
+
+  Album.one_to_one :album_info
+  # The album related to that AlbumInfo instance
+  Album.filter(:album_info=>AlbumInfo[2])
+
+  Album.one_to_many :tracks
+  # The album related to that Track instance
+  Album.filter(:tracks=>Track[3])
+
+  Album.many_to_many :tags
+  # All albums related to that Tag instance
+  Album.filter(:tags=>Tag[4])
+
+Note that for +one_to_many+ and +many_to_many+ associations, you still
+use the plural form even though only a single model object is given.
+You cannot use an array of model objects as the value, only a single model
+object.  To use separate model objects for the same association, you can
+use the array form of condition specifiers:
+
+  Album.filter([[:tags, Tag[1]], [:tags, Tag[2]]])
+
+That will return albums associated with both tag 1 and tag 2.
+
+Note that filtering by associations only works correctly for simple
+associations (ones without conditions).
+
 == Name Collisions
 
 Because associations create instance methods, it's possible to override
@@ -1221,22 +1294,30 @@ a JOIN USING or NATURAL JOIN for the graph:
 ==== :eager_grapher
 
 Sets up a custom grapher to use when eager loading the objects via eager_graph.
-This is the eager_graph analogue to the :eager_loader option.
+This is the eager_graph analogue to the :eager_loader option. This isn't generally
+needed, as one of the other eager_graph related association options is usually sufficient.
+
+If specified, should be a proc that accepts one or three three arguments.
+If the proc takes one argument, it will be given a hash with the following keys:
 
-If specified, should be a proc that accepts three three arguments: a dataset,
-an alias to use for the table to graph for this association, and the alias that
-was used for the current table (since you can cascade associations). Should
-return a modified copy of the dataset with the association graphed into it.
+:self :: The dataset that is doing the eager loading
+:table_alias :: An alias to use for the table to graph for this association.
+:implicit_qualifier :: The alias that was used for the current table (since you can cascade associations).
+:callback :: A callback proc used to dynamically modify the dataset to graph into the
+             current dataset, before such graphing is done. This is nil if no callback
+             proc is used.
+
+If the proc takes three arguments, it gets passed the :self, :association_alias,
+and :table_alias values. The 3 argument procs are allowed for backwards
+compatibility, and it is recommended to use the 1 argument proc format
+for new code.
 
   Artist.one_to_many :self_title_albums, :class=>:Album,
-   :eager_grapher=>(proc do |ds, ta, iq|
-    ds.graph(Album, {:artist_id=>:id, :name=>:name},
-      :table_alias=>ta, :implicit_qualifier=>iq)
+   :eager_grapher=>(proc do |eo|
+    eo[:self].graph(Album, {:artist_id=>:id, :name=>:name},
+      :table_alias=>eo[:table_alias], :implicit_qualifier=>eo[:implicit_qualifier])
   end)
 
-This isn't generally needed, as one of the other eager_graph related
-association options is usually sufficient.
-
 ==== :order_eager_graph
 
 Whether to add the order to the dataset's order when graphing via eager_graph.

data/doc/release_notes/3.23.0.txt

@@ -0,0 +1,172 @@
+= New Features
+
+* Sequel now allows dynamic customization for eager loading.
+  Previously, the parameters for eager loading were fixed at
+  association creation time.  Now, they can be modified at query
+  time.  To dynamically modify an eager load, you use a hash with
+  the proc as the value. For example, if you have this code:
+
+    Artist.eager(:albums)
+
+  And you only want to eagerly load albums where the id is greater
+  than or equal to some number provided by the user, you do:
+
+    min = params[:min]
+    Artist.eager(:albums=>proc{|ds| ds.where{id > min}})
+    
+  This also works when eager loading via eager_graph:
+  
+    Artist.eager_graph(:albums=>proc{|ds| ds.where{id > min}})
+  
+  For eager_graph, the dataset is the dataset to graph into the
+  current dataset, and filtering it will result in an SQL query
+  that joins to a subquery.
+  
+  You can also use dynamic customization while cascading to also
+  eagerly load dependent associations, by making the hash value
+  a single entry hash with a proc key and the value being the
+  dependent associations to eagerly load. For example, if you want
+  to eagerly load tracks for those albums:
+
+    Artist.eager(:albums=>{proc{|ds| ds.where{id > min}}=>:tracks})
+
+* Sequel also now allows dynamic customization for regular
+  association loading.  Previously, this was possible by using the
+  association's dataset:
+
+    albums = artist.albums_dataset.filter{id > min}
+
+  However, then there was no handling of caching, callbacks, or
+  reciprocals.  For example:
+
+    albums.each{|album| album.artist}
+    
+  Would issue one query per album to get the artist, because the
+  reciprocal association was not set.  Now you can provide a
+  block to the association method:
+  
+    albums = artist.albums{|ds| ds.filter{id > min}}
+    
+  This block is called with the dataset used to retrieve the
+  associated objects, and should return a modified version of that
+  dataset.
+  
+  Note that ruby 1.8.6 doesn't allow blocks to take block arguments,
+  so you have to pass the block as a separate proc argument to the
+  association method if you are still using 1.8.6.
+
+* Sequel now supports filtering by associations.  This wasn't
+  previously supported as filtering is a dataset level feature and
+  associations are a model level feature, and datasets do not depend
+  on models.  Now, model datasets have the ability to filter by
+  associations.  For example, to get all albums for a given artist,
+  you could do:
+  
+    artist = Artist[1]
+    Album.filter(:artist=>artist)
+    
+  Since the above can also be accomplished with:
+  
+    artist.albums
+    
+  this may not seem like a big improvement, but it allows you to
+  filter on multiple associations simultaneously:
+  
+    Album.filter(:artist=>artist, :publisher=>publisher)
+    
+  For simple many_to_one associations, the above is just a simpler
+  way to do:
+  
+    Album.filter(:artist_id=>artist.id, :publisher_id=>publisher.id)
+    
+  Sequel supports this for all association types, including
+  many_to_many and many_through_many, where a subquery is used, and
+  it also works when composite key associations are used:
+  
+    Album.filter(:artist=>artist, :tags=>tag)
+    
+  This will give you the albums for that artist that are also tagged
+  with that tag.  To provide multiple values for the same
+  association, mostly useful for many_to_many associations, you can
+  either use separate filter calls or specify the conditions as an
+  array:
+  
+    Album.filter(:tags=>tag1).filter(:tags=>tag2)
+    Album.filter([[:tags, tag1], [:tags, tag2]])
+
+* A columns_introspection extension has been added that makes
+  datasets attempt to guess their columns in some cases instead of
+  issuing a database query.  This can improve performance in cases
+  where the columns are needed implicitly, such as graphing.  After
+  loading the extension, you can enable the support for specific
+  datasets by extending them with Sequel::ColumnIntrospection.  To
+  enable introspection for all datasets, use:
+  
+    Sequel::Dataset.introspect_all_columns
+
+* A serialization_modification_detection plugin has been added.
+  Previously, Sequel could not detect modifications made to
+  serialized objects.  It could detect modification if you assigned
+  a new value:
+  
+    model.hash_column = model.hash_column.merge(:foo=>:bar)
+    
+  but not if you just modified the object directly:
+  
+    model.hash_columns[:foo] = :bar
+    
+  With this plugin, such modifications can be detected, at a
+  potentially significant performance cost.
+
+= Other Improvements
+
+* When using a migration directory containing both older integer
+  migrations and newer timestamp migrations, where some integer
+  migrations have not been applied, make sure to apply the remaining
+  integer migrations before the timestamp migrations.  Previously,
+  they could be applied out of order due to a lexicographic sort
+  being used instead of a numeric sort.
+
+* If a model does not select all columns from its table, the
+  insert_select optimization is no longer used.  Previously,
+  creating a new model object for such a model could result in the
+  object containing columns that the model does not select.
+
+* You can now use :select=>[] as an option for many_to_many
+  associations to select all columns from both the associated
+  table and the join table.  Previously, this raised an error and
+  required you do :select=>'*'.lit as a workaround.  The default
+  remains to select all columns in the associated table and none
+  from the join table.
+
+* The xml_serializer plugin now handles namespaced models by
+  using __ instead of / as the namespace separator.  Previously, /
+  was used and caused problems as it is not valid XML.
+  
+* The :eager_grapher association option can now accept a proc that
+  takes a single hash of options instead of a fixed 3 arguments.
+  This is the recommended way going forward of writing custom
+  :eager_graphers, and all of the internal ones have been converted.
+  The previous way of using 3 arguments is still supported.
+  
+* A bug in the identity_map plugin for many_to_one associations
+  without full association reflection information has been fixed.
+
+* Sequel is now using GitHub Issues for issue tracking.  Old issues
+  have been migrated from Google Code.
+
+= Backwards Compatibility
+
+* The filter by associations support breaks backward compatibilty for
+  users who previously added an sql_literal instance method to
+  Sequel::Model.  Usually, that was done to for reasons similar to
+  but inferior than the filter by association support.  The following
+  code can be used as a temporary workaround until you can modify
+  your program to use the new filter by associations support:
+  
+    Sequel::Model::Associations::DatasetMethods.
+      send(:remove_method, :complex_expression_sql)
+
+* The private Sequel::Model#_load_associated_objects method now takes
+  an additional, optional options hash.  Plugins that override that
+  method need to be modified.

data/doc/virtual_rows.rdoc

@@ -40,7 +40,7 @@ available.  For example, you are no longer able to do:
   # WHERE a AND (b OR NOT c)
   
 Because Symbol#&, Symbol#| and Symbol#~ are not defined when the core
-extensions are turned off.  However, with virtual rows allow almost the same
+extensions are turned off.  However, virtual rows allow almost the same
 syntax even without the core extensions:
 
   dataset.filter{a & (b | ~c)}
@@ -87,7 +87,7 @@ and second is the difference between the the use of "a".  In the regular proc,
 you couldn't call c without an explicit receiver in the proc, unless the self of the
 surrounding scope responded to it.  For a, note how ruby calls the method on
 the receiver of the surrounding scope in the regular proc, which returns an integer,
-and does the substraction before Sequel gets access to it.  In the instance evaled
+and does the substitution before Sequel gets access to it.  In the instance evaled
 proc, calling a without a receiver calls the a method on the VirtualRow instance.
 For b, note that it operates the same in both cases, as it is a local variable.
 

data/lib/sequel/extensions/columns_introspection.rb

@@ -0,0 +1,61 @@
+# The columns_introspection extension attempts to introspect the
+# selected columns for a dataset before issuing a query.  If it
+# thinks it can guess correctly at the columns the query will use,
+# it will return the columns without issuing a database query.
+# This method is not fool-proof, it's possible that some databases
+# will use column names that Sequel does not expect.
+#
+# To enable this for a single dataset, extend the dataset with
+# Sequel::ColumnIntrospection.  To enable this for all datasets, run:
+#
+#   Sequel::Dataset.introspect_all_columns
+
+module Sequel
+  module ColumnsIntrospection
+    # Attempt to guess the columns that will be returned
+    # if there are columns selected, in order to skip a database
+    # query to retrieve the columns.  This should work with
+    # Symbols, SQL::Identifiers, SQL::QualifiedIdentifiers, and
+    # SQL::AliasedExpressions.
+    def columns
+      return @columns if @columns
+      return columns_without_introspection unless cols = opts[:select] and !cols.empty?
+      probable_columns = cols.map{|c| probable_column_name(c)}
+      if probable_columns.all?
+        @columns = probable_columns
+      else
+        columns_without_introspection
+      end
+    end
+
+    private
+
+    # Return the probable name of the column, or nil if one
+    # cannot be determined.
+    def probable_column_name(c)
+      case c
+      when Symbol
+        _, c, a = split_symbol(c)
+        (a || c).to_sym
+      when SQL::Identifier
+        c.value.to_sym
+      when SQL::QualifiedIdentifier
+        col = c.column
+        col.is_a?(SQL::Identifier) ? col.value.to_sym : col.to_sym
+      when SQL::AliasedExpression
+        a = c.aliaz
+        a.is_a?(SQL::Identifier) ? a.value.to_sym : a.to_sym
+      end
+    end
+  end
+
+  class Dataset
+    alias columns_without_introspection columns
+
+    # Enable column introspection for every dataset.
+    def self.introspect_all_columns
+      include ColumnsIntrospection
+      remove_method(:columns) if instance_methods(false).map{|x| x.to_s}.include?('columns')
+    end
+  end
+end