datasource 0.1.1 → 0.2.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 892fcf4ff3a4d4988111a50f2b6773df1d102da1
-  data.tar.gz: a9f398491b40f774f767563e4bb1e022b7aee945
+  metadata.gz: 6c4654c0cc97e0cba3fd8ce3d9b99a520d590afa
+  data.tar.gz: a37d72813c8e2d0a6ef6430a380115126f28dfad
 SHA512:
-  metadata.gz: 1595cd7d940f7ec47d5b3cae6c242f71516ddf4ba06c62a04aa5518c563e82ddf7108432940c12107eadaf5f290faeb3f89a1d781dbafffd43a70e80a29eb263
-  data.tar.gz: c851fcf944f8f56e9dab25f36f8371f58fd6cb6719f14a27593649ef847e862a4ff5d92c4292b23484eebd7481142c9f86edfd680165dac47d35e76e2e314bd7
+  metadata.gz: 6d68a095c8b8af62be19f3fbb7048b634b5d0fb5d113f3e81d6e1430959d69bc9f93cce68d02977066ff782361c7915b0a3cab0a9003b85598caa019a5bb5c7e
+  data.tar.gz: 4cf182d351a3bfc21f6dfcd2062eb97f21f36dd39ccf9ef4bfaf6cca418aea7b7fc9105fe613936f95f19a4a32d74714e51b27ebccfc15fa9de9a7b716f25dab

data/README.md

@@ -4,12 +4,16 @@
 - Specify custom SQL snippets for virtual attributes (Query attributes)
 - Write custom preloading logic in a reusable way
 
+** Note: the API of this gem is still unstable and may change a lot between versions! This project uses semantic versioning (until version 1.0.0, minor version changes may include API changes, but patch version will not) **
+
 #### Install
 
-Add to Gemfile
+Requires Ruby 2.0 or higher.
+
+Add to Gemfile (recommended to use github version until API is stable)
 
 ```
-gem 'datasource'
+gem 'datasource', github: 'kundi/datasource'
 ```
 
 ```
@@ -32,18 +36,10 @@ rails g datasource:install
 
 - active_model_serializers
 
-## Simple Mode
-
-Datasource is configured to run in Simple mode by default, which makes it easier
-to start with, but disables some advanced optimizations. See
-[Advanced mode](https://github.com/mrbrdo/datasource/wiki/Advanced-mode) for more
-information after you understand Simple mode.
-
 ### Associations
 
-The most noticable magic effect of using Datasource in Simple mode (Advanced mode
-has other benefits) is that associations will automatically be preloaded using a
-single query.
+The most noticable magic effect of using Datasource is that associations will
+automatically be preloaded using a single query.
 
 ```ruby
 class PostSerializer < ActiveModel::Serializer
@@ -65,7 +61,8 @@ automatically by Datasource.
 
 ### Show action
 
-You will probably want to reuse the same preloading logic in your show action.
+If you use the more advanced features like Loaded, you will probably want to
+reuse the same loading logic in your show action.
 You will need to call `for_serializer` on the scope before you call `find`.
 You can optionally give it the serializer class as an argument.
 
@@ -117,96 +114,120 @@ end
 SELECT users.*, (users.first_name || ' ' || users.last_name) AS full_name FROM users
 ```
 
-Note: If you need data from another table, use a join in a loader (see below).
+Note: If you need data from another table, use a join in a loaded value (see below).
 
-### Loader
+### Standalone Datasource class
 
-You might want to have some more complex preloading logic. In that case you can use a loader.
-A loader will receive ids of the records, and needs to return a hash.
-The key of the hash must be the id of the record for which the value is.
+If you are going to have more complex preloading logic (like using Loaded below),
+then it might be better to put Datasource code into its own class. This is pretty
+easy, just create a directory `app/datasources` (or whatever you like), and create
+a file depending on your model name, for example for a `Post` model, create
+`post_datasource.rb`. The name is important for auto-magic reasons. Example file:
 
-A loader will only be executed if a computed attribute depends on it. See
-[Advanced mode](https://github.com/mrbrdo/datasource/wiki/Advanced-mode) for
-information about computed attributes (but this works the same way in Simple mode).
-A more simple alternative to loader which doesn't require computed attributes is to use
-[Loaded](#loaded).
-If an attribute depends on multiple loaders, pass an array of loaders like
-so `computed :attr, loaders: [:loader1, :loader2]`.
+```ruby
+class PostDatasource < Datasource::From(Post)
+  query(:full_name) { "users.first_name || ' ' || users.last_name" }
+end
+```
 
-Be careful that if your hash does not contain a value for the object ID, the loaded value
-will be nil. However you can use the `default` option for such cases (see below example).
+### Loaded
+
+You might want to have some more complex preloading logic. In that case you can
+use a method to load values for all the records at once (e.g. with a custom query
+or even from a cache). The loading methods are only executed if you use the values,
+otherwise they will be skipped.
+
+First just declare that you want to have a loaded attribute (the parameters will be explained shortly):
 
 ```ruby
-class User < ActiveRecord::Base
-  datasource_module do
-    computed :post_count, loader: :post_counts
-    loader :post_counts, array_to_hash: true, default: 0 do |user_ids|
-      results = Post
-        .where(user_id: user_ids)
-        .group(:user_id)
-        .pluck("user_id, COUNT(id)")
-    end
-  end
+class UserDatasource < Datasource::From(User)
+  loaded :post_count, from: :array, default: 0
 end
+```
 
-class UserSerializer < ActiveModel::Serializer
-  attributes :id, :post_count
+By default, datasource will look for a method named `load_<name>` for loading
+the values, in this case `load_newest_comment`. It needs to be defined in the
+collection block, which has methods to access information about the collection (posts)
+that are being loaded. These methods are `scope`, `models`, `model_ids`,
+`datasource`, `datasource_class` and `params`.
 
-  def post_count
-    # Will automatically give you the value for this user's ID
-    object.loaded_values[:post_counts]
+```ruby
+class UserDatasource < Datasource::From(User)
+  loaded :post_count, from: :array, default: 0
+
+  collection do
+    def load_post_count
+      Post.where(user_id: model_ids)
+      .group(:user_id)
+      .pluck("user_id, COUNT(id)")
+    end
   end
 end
 ```
 
-```sql
-SELECT users.* FROM users
-SELECT user_id, COUNT(id) FROM posts WHERE user_id IN (?)
-```
-
-Datasource provides shortcuts to transform your data into a hash. Here are examples:
+In this case `load_post_count` returns an array of pairs.
+For example: `[[1, 10], [2, 5]]`. Datasource can understand this because of
+`from: :array`. This would result in the following:
 
 ```ruby
-loader :stuff, array_to_hash: true do |ids|
-  [[1, "first"], [2, "second"]]
-  # will be transformed into
-  # { 1 => "first", 2 => "second" }
-end
+post_id_1.post_count # => 10
+post_id_2.post_count # => 5
+# other posts will have the default value or nil if no default value was given
+other_post.post_count # => 0
+```
 
-loader :stuff, group_by: :user_id do |ids|
-  Post.where(user_id: ids)
-  # will be transformed into
-  # { 1 => [#<Post>, #<Post>, ...], 2 => [ ... ], ... }
-end
+Besides `default` and `from: :array`, you can also specify `group_by`, `one`
+and `source`. Source is just the name of the load method.
 
-loader :stuff, group_by: :user_id, one: true do |ids|
-  Post.where(user_id: ids)
-  # will be transformed into
-  # { 1 => #<Post>, 2 => #<Post>, ... }
-end
+The other two are explained in the following example.
 
-loader :stuff, group_by: "user_id", one: true do |ids|
-  # it works the same way on an array of hashes
-  # but be careful about Symbol/String difference
-  [{ "title" => "Something", "user_id" => 10 }]
-  # will be transformed into
-  # { 10 => { "title" => "Something", "user_id" => 10 } }
+```ruby
+class PostDatasource < Datasource::From(Post)
+  loaded :newest_comment, group_by: :post_id, one: true, source: :load_newest_comment
+
+  collection do
+    def load_newest_comment
+      Comment.for_serializer.where(post_id: model_ids)
+        .group("post_id")
+        .having("id = MAX(id)")
+    end
+  end
 end
 ```
 
-### Loaded
+In this case the load method returns an ActiveRecord relation, which for our purposes
+acts the same as an Array (so we could also return an Array if we wanted).
+Using `group_by: :post_id` in the `loaded` call tells datasource to group the
+results in this array by that attribute (or key if it's an array of hashes instead
+of model objects). `one: true` means that we only want a single value instead of
+an array of values (we might want multiple, e.g. `newest_10_comments`).
+So in this case, if we had a Post with id 1, `post.newest_comment` would be a
+Comment from the array that has `post_id` equal to 1.
 
-Loaded is the same as loader, but it automatically creates a computed attribute
-and defines a method with the same name on your model.
+In this case, in the load method, we also used `for_serializer`, which will load
+the `Comment`s according to the `CommentSerializer`.
 
-Here is the previous example with `loaded` instead of `loader`:
+Note that it's perfectly fine (even good) to already have a method with the same
+name in your model.
+If you use that method outside of serializers/datasource, it will work just as
+it should. But when using datasource, it will be overwritten by the datasource
+version. Counts is a good example:
 
 ```ruby
 class User < ActiveRecord::Base
-  datasource_module do
-    loaded :post_count, array_to_hash: true, default: 0 do |user_ids|
-      results = Post
-        .where(user_id: user_ids)
+  has_many :posts
+
+  def post_count
+    posts.count
+  end
+end
+
+class UserDatasource < Datasource::From(User)
+  loaded :post_count, from: :array, default: 0
+
+  collection do
+    def load_post_count
+      Post.where(user_id: model_ids)
         .group(:user_id)
         .pluck("user_id, COUNT(id)")
     end
@@ -214,50 +235,56 @@ class User < ActiveRecord::Base
 end
 
 class UserSerializer < ActiveModel::Serializer
-  attributes :id, :post_count
-  # Note that the User now has a generated post_count method
+  attributes :id, :post_count # <- post_count will be read from load_post_count
 end
+
+User.first.post_count # <- your model method will be called
 ```
 
-When using `loaded`, if you already have the method with this name defined in your
-model, datasource will automatically create a 'wrapper' method that will use the
-loaded value if available (when you are using a serializer/datasource), otherwise
-it will fallback to your original method. This way you can still use the same
-method when you are not using a serializer/datasource. For example:
+### Params
+
+You can also specify params that can be read from collection methods. The params
+can be specified when you call `render`:
 
 ```ruby
-class User < ActiveRecord::Base
-  datasource_module do
-    loaded :post_count, array_to_hash: true, default: 0 do |user_ids|
-      results = Post
-        .where(user_id: user_ids)
-        .group(:user_id)
-        .pluck("user_id, COUNT(id)")
+# controller
+  render json: posts,
+    datasource_params: { include_newest_comments: true }
+
+# datasource
+  loaded :newest_comments, default: []
+
+  collection do
+    def load_newest_comments
+      if params[:include_newest_comments]
+        # ...
+      end
     end
   end
+```
 
-  def post_count
-    posts.count
-  end
-end
-
-class UserSerializer < ActiveModel::Serializer
-  attributes :id, :post_count # <- post_count will be read from loaded_values
-end
+### Debugging and logging
 
-User.first.post_count # <- your method will be called
+Datasource outputs some useful logs that you can use debugging. By default the log level is
+set to warnings only, but you can change it. You can add the following line to your
+`config/initializers/datasource.rb`:
 
+```ruby
+Datasource.logger.level = Logger::INFO
 ```
 
+You can also set it to `DEBUG` for more output. The logger outputs to `stdout` by default. It
+is not recommended to have this enabled in production.
+
 ## Getting Help
 
-If you find a bug, please report an [Issue](https://github.com/mrbrdo/datasource/issues/new).
+If you find a bug, please report an [Issue](https://github.com/kundi/datasource/issues/new).
 
 If you have a question, you can also open an Issue.
 
 ## Contributing
 
-1. Fork it ( https://github.com/mrbrdo/datasource/fork )
+1. Fork it ( https://github.com/kundi/datasource/fork )
 2. Create your feature branch (`git checkout -b my-new-feature`)
 3. Commit your changes (`git commit -am 'Add some feature'`)
 4. Push to the branch (`git push origin my-new-feature`)

data/lib/datasource/adapters/active_record.rb

@@ -5,27 +5,35 @@ module Datasource
   module Adapters
     module ActiveRecord
       module ScopeExtensions
-        def use_datasource_serializer(value)
-          @datasource_serializer = value
-          self
+        def self.extended(mod)
+          mod.instance_exec do
+            @datasource_info ||= { select: [], params: [] }
+          end
         end
 
-        def use_datasource(value)
-          @datasource = value
+        def datasource_set(hash)
+          @datasource_info.merge!(hash)
           self
         end
 
         def datasource_select(*args)
-          @datasource_select = Array(@datasource_select) + args
+          @datasource_info[:select] += args
+          self
+        end
+
+        def datasource_params(*args)
+          @datasource_info[:params] += args
           self
         end
 
         def get_datasource
-          datasource = @datasource.new(self)
-          datasource.select(*Array(@datasource_select))
-          if @datasource_serializer
+          klass = @datasource_info[:datasource_class]
+          datasource = klass.new(self)
+          datasource.select(*@datasource_info[:select])
+          datasource.params(*@datasource_info[:params])
+          if @datasource_info[:serializer_class]
             select = []
-            Datasource::Base.consumer_adapter.to_datasource_select(select, @datasource.orm_klass, @datasource_serializer, nil, datasource.adapter)
+            Datasource::Base.consumer_adapter.to_datasource_select(select, klass.orm_klass, @datasource_info[:serializer_class], nil, datasource.adapter)
 
             datasource.select(*select)
           end
@@ -34,9 +42,12 @@ module Datasource
 
       private
         def exec_queries
-          if @datasource
+          if @datasource_info[:datasource_class]
             datasource = get_datasource
 
+            Datasource.logger.debug { "exec_queries expose_attributes: #{datasource.expose_attributes.inspect}" }
+            Datasource.logger.debug { "exec_queries expose_associations: #{datasource.expose_associations.inspect}" }
+
             @loaded = true
             @records = datasource.results
           else
@@ -49,7 +60,7 @@ module Datasource
         extend ActiveSupport::Concern
 
         included do
-          attr_accessor :loaded_values
+          attr_accessor :_datasource_loaded, :_datasource_instance
         end
 
         def for_serializer(serializer = nil)
@@ -58,7 +69,10 @@ module Datasource
 
           scope = self.class
           .with_datasource(datasource_class)
-          .for_serializer(serializer).where(pk => send(pk))
+          .for_serializer(serializer)
+          .where(pk => send(pk))
+
+          scope = yield(scope) if block_given?
 
           datasource = scope.get_datasource
           if datasource.can_upgrade?(self)
@@ -69,37 +83,47 @@ module Datasource
         end
 
         module ClassMethods
-          def for_serializer(serializer = nil)
-            scope = if all.respond_to?(:use_datasource_serializer)
-              all
-            else
-              all.extending(ScopeExtensions).use_datasource(default_datasource)
-            end
-            scope.use_datasource_serializer(serializer || Datasource::Base.consumer_adapter.get_serializer_for(Adapters::ActiveRecord.scope_to_class(scope)))
+          def for_serializer(serializer_class = nil)
+            scope = scope_with_datasource_ext
+            serializer_class ||=
+              Datasource::Base.consumer_adapter.get_serializer_for(
+                Adapters::ActiveRecord.scope_to_class(scope))
+            scope.datasource_set(serializer_class: serializer_class)
           end
 
-          def with_datasource(datasource = nil)
-            scope = if all.respond_to?(:use_datasource)
-              all
-            else
-              all.extending(ScopeExtensions)
-            end
-            scope.use_datasource(datasource || default_datasource)
+          def with_datasource(datasource_class = nil)
+            scope_with_datasource_ext(datasource_class)
           end
 
           def default_datasource
-            @default_datasource ||= Datasource::From(self)
+            @default_datasource ||= begin
+              "#{name}Datasource".constantize
+            rescue NameError
+              Datasource::From(self)
+            end
           end
 
           def datasource_module(&block)
             default_datasource.instance_exec(&block)
           end
+
+        private
+          def scope_with_datasource_ext(datasource_class = nil)
+            if all.respond_to?(:datasource_set)
+              all
+            else
+              datasource_class ||= default_datasource
+
+              all.extending(ScopeExtensions)
+              .datasource_set(datasource_class: datasource_class)
+            end
+          end
         end
       end
 
     module_function
       def association_reflection(klass, name)
-        if reflection = klass.reflections[name]
+        if reflection = klass.reflect_on_association(name)
           {
             klass: reflection.klass,
             macro: reflection.macro,
@@ -124,6 +148,10 @@ module Datasource
         scope.loaded?
       end
 
+      def scope_to_records(scope)
+        scope.to_a
+      end
+
       def has_attribute?(record, name)
         record.attributes.key?(name.to_s)
       end
@@ -136,22 +164,50 @@ module Datasource
         end
       end
 
-      def load_association(records, name, assoc_select)
+      def association_loaded?(records, name, assoc_select)
+        if records.first.association(name).loaded?
+          all_loaded = records.all? { |record| record.association(name).loaded? }
+          if assoc_select == ["*"]
+            all_loaded
+          elsif all_loaded
+            records.all? do |record|
+              assoc_sample = Array(record.send(name)).first
+              assoc_sample.nil? || assoc_sample._datasource_instance
+            end
+          else
+            false
+          end
+        else
+          false
+        end
+      end
+
+      def load_association(records, name, assoc_select, params)
         return if records.empty?
-        return if records.first.association(name.to_sym).loaded?
+        name = name.to_sym
         klass = records.first.class
-        if reflection = klass.reflections[name.to_sym]
+        if reflection = klass.reflect_on_association(name)
           assoc_class = association_klass(reflection)
           datasource_class = assoc_class.default_datasource
 
           scope = assoc_class.all
           datasource = datasource_class.new(scope)
           assoc_select_attributes = assoc_select.reject { |att| att.kind_of?(Hash) }
-          assoc_select_associations = assoc_select.select { |att| att.kind_of?(Hash) }
-          Datasource::Base.reflection_select(association_reflection(klass, name.to_sym), [], assoc_select_attributes)
+          assoc_select_associations = assoc_select.inject({}) do |hash, att|
+            hash.deep_merge!(att) if att.kind_of?(Hash)
+            hash
+          end
+          Datasource::Base.reflection_select(association_reflection(klass, name), [], assoc_select_attributes)
+          datasource.params(params)
+
+          Datasource.logger.debug { "load_association #{records.first.try!(:class)} #{name}: #{assoc_select_attributes.inspect}" }
           datasource.select(*assoc_select_attributes)
           select_values = datasource.get_select_values
 
+          # TODO: manually load associations, and load them all at once for
+          # nested associations, eg. in following, load all Users in 1 query:
+          # {"user"=>["*"], "players"=>["*"], "picked_players"=>["*",
+          # {:position=>["*"]}], "parent_picked_team"=>["*", {:user=>["*"]}]}
           begin
             ::ActiveRecord::Associations::Preloader
               .new.preload(records, name, assoc_class.select(*select_values))
@@ -161,12 +217,16 @@ module Datasource
           end
 
           assoc_records = records.flat_map { |record| record.send(name) }.compact
-          assoc_select_associations.each do |assocs|
-            assocs.each_pair do |assoc_name, assoc_select|
-              load_association(assoc_records, assoc_name, assoc_select)
+          unless assoc_records.empty?
+            if Datasource.logger.info? && !assoc_select_associations.empty?
+              Datasource.logger.info { "Loading associations " + assoc_select_associations.keys.map(&:to_s).join(", ") + " for #{assoc_records.first.try!(:class)}s" }
             end
+            assoc_select_associations.each_pair do |assoc_name, assoc_select|
+              Datasource.logger.debug { "load_association nested association #{assoc_name}: #{assoc_select.inspect}" }
+              load_association(assoc_records, assoc_name, assoc_select, params)
+            end
+            datasource.results(assoc_records)
           end
-          datasource.results(assoc_records)
         end
       rescue Exception => ex
         if ex.is_a?(SystemStackError) || ex.is_a?(Datasource::RecursionError)
@@ -187,13 +247,18 @@ module Datasource
       end
 
       def upgrade_records(ds, records)
+        Datasource.logger.debug { "Upgrading records #{records.map(&:class).map(&:name).join(', ')}" }
         load_associations(ds, records)
         ds.results(records)
       end
 
       def load_associations(ds, records)
+        if Datasource.logger.info? && !ds.expose_associations.empty?
+          Datasource.logger.info { "Loading associations " + ds.expose_associations.keys.map(&:to_s).join(", ") + " for #{records.first.try!(:class)}s" }
+        end
+        Datasource.logger.debug { "load_associations (#{records.size} #{records.first.try!(:class)}): #{ds.expose_associations.inspect}" }
         ds.expose_associations.each_pair do |assoc_name, assoc_select|
-          load_association(records, assoc_name, assoc_select)
+          load_association(records, assoc_name, assoc_select, ds.params)
         end
       end
 
@@ -207,8 +272,8 @@ module Datasource
         ds.select(*append_select)
 
         scope = select_scope(ds)
-        if scope.respond_to?(:use_datasource)
-          scope = scope.spawn.use_datasource(nil)
+        if scope.respond_to?(:datasource_set)
+          scope = scope.spawn.datasource_set(datasource_class: nil)
         end
         scope.includes_values = []
         scope.to_a.tap do |records|