tanker 1.0.0 → 1.1.0

data/Gemfile

@@ -1,10 +1,15 @@
 source "http://rubygems.org"
 
 gem 'jeweler'
+gem 'json', '>= 1.5.1'
 gem 'will_paginate', '>= 2.3.15'
+gem 'kaminari'
 
-group :test do
-  gem 'rspec', '>= 2.5.0' 
+group :test do 
+  gem 'rspec', '>= 2.5.0'
+  gem 'ruby-debug19'
+  gem 'activerecord', '>= 3.0.7'
+  gem 'sqlite3'
 end
 
 

data/README.rdoc

@@ -1,12 +1,13 @@
 = Tanker - IndexTank integration to your favorite orm
 
-IndexTank is a great search indexing service, this gem tries to make 
-any orm keep in sync with indextank with ease. This gem has a simple 
-but powerful approach to integrating IndexTank to any orm. The gem 
-also supports pagination using the +will_paginate/collection+.
-
-There are very few things needed to integrate Tanker to any orm. 
-Basically the orm instance must respond to +id+ and any attributes 
+IndexTank is a great search indexing service, this gem tries to make
+that any orm stays in sync with IndexTank easily. It has a simple
+yet powerful approach to integrate IndexTank with any ORM and
+also supports pagination using {WillPaginate}[https://github.com/mislav/will_paginate] (default)
+or {Kaminari}[https://github.com/amatsuda/kaminari].
+
+Very little is needed to make it work,
+basically, all you need is that the orm instance respond to +id+, +all+, +find+ and any attributes
 you wish to index.
 
 == Installation
@@ -25,13 +26,25 @@ If you're using Rails, config/initializers/tanker.rb is a good place for this:
 
     YourAppName::Application.config.index_tank_url = 'http://:xxxxxxxxx@xxxxx.api.indextank.com'
 
-If you are not using rails you can put this somewhere before you load your modesl
+If you are not using rails you can put this somewhere before you load your models
 
     Tanker.configuration = {:url => 'http://:xxxxxxxxx@xxxxx.api.indextank.com' }
 
-You would probably want to have a fancier configuration depending on
+You would probably want to have fancier configuration depending on
 your environment. Be sure to copy and paste the correct url provided
-in the IndexTank Dashboard
+by the IndexTank Dashboard
+
+=== Pagination
+
+WillPaginate is used by default and you don't have to add any extra settings to use that. If you
+want to use Kaminari you have to add the following:
+
+    YourAppName::Application.config.index_tank_url = 'http://...'
+    YourAppName::Application.config.tanker_pagination_backend = :kaminari  
+
+or
+
+    Tanker.configuration = {:url => 'http://...', :pagination_backend => :kaminari }
 
 == Example
     
@@ -117,8 +130,39 @@ Search with query variables
 
 Paginate Results
 
-    <% will_paginate(@topics) %>
+    <%= will_paginate @topics %> for WillPaginate 
+    <%= paginate @topics %> for Kaminari
+
+Disable pagination of search results
+
+    @topics = Topic.search_tank('keyword', :paginate => false)
+
+== Fetching attributes and snippets
+
+As of version 1.1 Tanker supports fetching attributes and snippets directly from the Index. If you use either of this search options
+your Model instances will be instanced from the attributes coming from the index not by making another trip to the database.
+
+This can represent a major speed increase for your queries but it also means that you will only have available the attributes
+you indexed in your model. There are edge cases that are not tested so please help us have this feature be super stable and
+have everyone bennefit from the speed this entails.
+
+Example:
+  
+    @very_long_sting = 'Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.'
+    Product.create(:name => @very_long_sting, :href => 'http://google.com' )
+    
+    @products = Product.search_tank('quis exercitation', :snippets => [:name], :fetch => [:href])
+    @products[0].name_snippet 
+    # => 'Ut enim ad minim veniam, <b>quis</b> nostrud <b>exercitation</b> ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla' 
+    @products[0].href
+    # => 'http://google.com'
+    @products[0].any_other_attribute
+    # => nil
+
+Notice that to get the snippeted value that Index Tank sends you get it via the '_snippet' method generated on the fly.
 
+The attributes that are requested to be fetched are set as attributes as new instances of your model objects and no Database Query is executed. NEAT!
+    
 == Geospatial Search Example
 
 IndexTank and the Tanker gem support native geographic calculations. All you need is to define two document variables with your latitude and your longitude
@@ -177,7 +221,85 @@ IndexTank and the Tanker gem support native geographic calculations. All you nee
                       :function => 1,
                       :filter_functions => {2 => [['*', 50]]})
 
+== Extend your index definitions
+
+If you have a bunch of models with a lot of overlapping indexed fields,
+variables, or categories, you might want to abstract those out into a module
+that you can include and then extend in the including classes. Something like:
+
+    module TankerDefaults
+      def self.included(base)
+        base.send(:include, ::Tanker) # include the actual Tanker module
+
+        # provide a default index name
+        base.tankit 'my_index' do
+          # index some common fields
+          indexes :tag_list
+
+          # set some common variables
+          variables do
+            {
+              0 => view_count
+              1 => foo
+            }
+          end
+        end
+      end
+    end
+
+    class SuperModel
+      include TankerDefaults
+
+      # no need to respecify the index if it's the same
+      # (but you can override it)
+      tankit do
+        # `indexes :tag_list` is inherited
+        indexes :name
+        indexes :boyfriend_names do
+          boyfriends.map(&:name)
+        end
+
+        # set some more specific variables
+        variables do
+          {
+            # `0 => view_count` is inherited
+            1 => iq,                 # overwrites "foo"
+            2 => endorsements.count  # adds new variables
+          }
+        end
+      end
+    end
+
+You currently can't remove previously defined stuff, though.
+
+=== Single table inheritance
+
+If you are using tanker with STI models and want the different models to be indexed
+as the same type, you can provide the base model name with an +:as+ option to +tankit+.
+
+    module TankerPetDefaults
+      def self.included(base)
+        base.send(:include, ::Tanker)
+
+        base.tankit 'my_index', :as => 'Pet' do
+          indexes :skills
+        end
+      end
+    end
+
+    class Pet < ActiveRecord::Base
+    end
+
+    class Cat < Pet 
+      include TankerPetDefaults
+    end
+
+    class Monkey < Pet 
+      include TankerPetDefaults
+    end
 
+    # Search using the base model
+    Pet.search('stealth')
 
 == Reindex your data
 
@@ -187,7 +309,7 @@ If you are using rails 3 there are of couple of rake tasks included in your proj
 
 This task deletes all your indexes and recreates empty indexes, if you have not created
 the indexes in the Index Tank interface this task creates them for you provided
-you have enough indexes in your account.
+you have enough available indexes in your account.
 
     rake tanker:reindex
 

data/VERSION

@@ -1 +1 @@
-1.0.0
+1.1.0

data/lib/tanker/configuration.rb

@@ -1,15 +1,14 @@
 module Tanker
-
   module Configuration
-
     def configuration
       @@configuration || raise(NotConfigured, "Please configure Tanker. Set Tanker.configuration = {:url => ''}")
     end
 
     def configuration=(new_configuration)
-      @@configuration = new_configuration
+      # the default pagination backend is WillPaginate
+      @@configuration = new_configuration.tap do |_config|
+        _config.replace({ :pagination_backend => :will_paginate }.merge(_config)) if _config.is_a?(Hash)
+      end
     end
-
   end
-
 end

data/lib/tanker/paginated_array.rb

@@ -0,0 +1,29 @@
+unless defined?(Kaminari)
+  raise(Tanker::BadConfiguration, "Tanker: Please add 'kaminari' to your Gemfile to use kaminari pagination backend")
+end
+
+module Tanker
+  class KaminariPaginatedArray < Array
+    include ::Kaminari::ConfigurationMethods::ClassMethods
+    include ::Kaminari::PageScopeMethods
+
+    attr_reader :limit_value, :offset_value, :total_count
+
+    def initialize(original_array, limit_val = default_per_page, offset_val, total_count)
+      @limit_value, @offset_value, @total_count = limit_val, offset_val, total_count
+      super(original_array)
+    end
+
+    def page(num = 1)
+      self
+    end
+
+    def limit(num)
+      self
+    end
+
+    def current_page
+      offset_value+1
+    end
+  end
+end

data/lib/tanker/railtie.rb

@@ -2,18 +2,27 @@ require 'rails'
 
 module Tanker
   class Railtie < Rails::Railtie
-    config.index_tank_url = ''
-    
+    config.index_tank_url = nil
+    config.tanker_pagination_backend = nil
+
     initializer "tanker.boot" do
-      Tanker.configuration = {:url => config.index_tank_url }
+      setup_tanker_configuration
     end
-    
+
     config.after_initialize do
-      Tanker.configuration = {:url => config.index_tank_url }
+      setup_tanker_configuration
     end
 
     rake_tasks do
       load "tanker/tasks/tanker.rake"
     end
+
+  private
+    def setup_tanker_configuration
+      Tanker.configuration = {}.tap do |_new_conf|
+        _new_conf[:url] = config.index_tank_url if config.index_tank_url
+        _new_conf[:pagination_backend] = config.tanker_pagination_backend if config.tanker_pagination_backend
+      end
+    end
   end
-end
+end

data/lib/tanker/utilities.rb

@@ -11,22 +11,26 @@ module Tanker
 
       def clear_all_indexes
         get_available_indexes.each do |index_name|
-          begin
-            index = Tanker.api.get_index(index_name)
+          clear_index(index_name)
+        end
+      end
 
-            if index.exists?
-              puts "Deleting #{index_name} index"
-              index.delete_index()
-            end
-            puts "Creating #{index_name} index"
-            index.create_index()
-            puts "Waiting for the index to be ready"
-            while not index.running?
-              sleep 0.5
-            end
-          rescue => e
-            puts "There was an error clearing or creating the #{index_name} index: #{e.to_s}"
+      def clear_index(index_name)
+        begin
+          index = Tanker.api.get_index(index_name)
+
+          if index.exists?
+            puts "Deleting #{index_name} index"
+            index.delete_index()
+          end
+          puts "Creating #{index_name} index"
+          index.create_index()
+          puts "Waiting for the index to be ready"
+          while not index.running?
+            sleep 0.5
           end
+        rescue => e
+          puts "There was an error clearing or creating the #{index_name} index: #{e.to_s}"
         end
       end
 

data/lib/tanker.rb

@@ -1,4 +1,3 @@
-
 begin
   require "rubygems"
   require "bundler"
@@ -12,7 +11,6 @@ require 'tanker/configuration'
 require 'tanker/utilities'
 require 'will_paginate/collection'
 
-
 if defined? Rails
   begin
     require 'tanker/railtie'
@@ -23,11 +21,15 @@ end
 module Tanker
 
   class NotConfigured < StandardError; end
+  class BadConfiguration < StandardError; end
   class NoBlockGiven < StandardError; end
+  class NoIndexName < StandardError; end
 
   autoload :Configuration, 'tanker/configuration'
   extend Configuration
 
+  autoload :KaminariPaginatedArray, 'tanker/paginated_array'
+
   class << self
     attr_reader :included_in
 
@@ -36,11 +38,12 @@ module Tanker
     end
 
     def included(klass)
+      configuration # raises error if not defined
+
       @included_in ||= []
       @included_in << klass
       @included_in.uniq!
 
-      configuration # raises error if not defined
       klass.send :include, InstanceMethods
       klass.extend ClassMethods
 
@@ -62,16 +65,16 @@ module Tanker
     def search(models, query, options = {})
       ids      = []
       models   = [models].flatten.uniq
-      page     = (options.delete(:page) || 1).to_i
-      per_page = (options.delete(:per_page) || models.first.per_page).to_i
       index    = models.first.tanker_index
       query    = query.join(' ') if Array === query
+      snippets = options.delete(:snippets)
+      fetch    = options.delete(:fetch)
+      paginate = extract_setup_paginate_options(options, :page => 1, :per_page => models.first.per_page)
 
       if (index_names = models.map(&:tanker_config).map(&:index_name).uniq).size > 1
         raise "You can't search across multiple indexes in one call (#{index_names.inspect})"
       end
 
-
       # move conditions into the query body
       if conditions = options.delete(:conditions)
         conditions.each do |field, value|
@@ -96,55 +99,127 @@ module Tanker
         end
       end
 
-      query = "__any:(#{query.to_s}) __type:(#{models.map(&:name).join(' OR ')})"
-      options = { :start => per_page * (page - 1), :len => per_page }.merge(options)
-      results = index.search(query, options)
-
-      @entries = WillPaginate::Collection.create(page, per_page) do |pager|
-        # inject the result array into the paginated collection:
-        pager.replace instantiate_results(results)
+      # fetch values from index tank or just the type and id to instace results localy
+      options[:fetch]   =  "__type,__id"
+      options[:fetch]   += ",#{fetch.join(',')}" if fetch 
+      options[:snippet] = snippets.join(',') if snippets
+      
+      search_on_fields = models.map{|model| model.tanker_config.indexes.map{|arr| arr[0]}.uniq}.flatten.uniq.join(":(#{query.to_s}) OR ")
+      query = "#{search_on_fields}:(#{query.to_s}) __type:(#{models.map(&:name).map {|name| "\"#{name.split('::').join(' ')}\"" }.join(' OR ')})"
 
-        unless pager.total_entries
-          # the pager didn't manage to guess the total count, do it manually
-          pager.total_entries = results["matches"]
-        end
+      options = { :start => paginate[:per_page] * (paginate[:page] - 1), :len => paginate[:per_page] }.merge(options) if paginate
+      results = index.search(query, options)
+      
+      instantiated_results = if (fetch || snippets)
+        instantiate_results_from_results(results, fetch, snippets)
+      else
+        instantiate_results_from_db(results)
       end
+      paginate === false ? instantiated_results : paginate_results(instantiated_results, paginate, results['matches'])
     end
 
     protected
 
-      def instantiate_results(index_result)
+      def instantiate_results_from_db(index_result) 
         results = index_result['results']
         return [] if results.empty?
 
         id_map = results.inject({}) do |acc, result|
-          model, id = result["docid"].split(" ", 2)
+          model = result["__type"]
+          id = constantize(model).tanker_parse_doc_id(result)
           acc[model] ||= []
-          acc[model] << id.to_i
+          acc[model] << id
           acc
         end
+        
+        id_map.each do |klass, ids|
+          # replace the id list with an eager-loaded list of records for this model
+          id_map[klass] = constantize(klass).find(ids)
+        end
+        # return them in order
+        results.map do |result|
+          model, id = result["__type"], result["__id"]
+          id_map[model].detect {|record| id == record.id.to_s }
+        end
+      end
 
-        if 1 == id_map.size # check for simple case, just one model involved
-          klass = constantize(id_map.keys.first)
-          # eager-load and return just this model's records
-          klass.find(id_map.values.flatten)
-        else # complex case, multiple models involved
-          id_map.each do |klass, ids|
-            # replace the id list with an eager-loaded list of records for this model
-            id_map[klass] = constantize(klass).find(ids)
-          end
-          # return them in order
-          results.map do |result|
-            model, id = result["docid"].split(" ", 2)
-            id_map[model].detect {|record| id.to_i == record.id }
+      def paginate_results(results, pagination_options, total_hits)
+        case Tanker.configuration[:pagination_backend]
+        when :will_paginate
+          WillPaginate::Collection.create(pagination_options[:page],
+                                          pagination_options[:per_page],
+                                          total_hits) { |pager| pager.replace results }
+        when :kaminari
+          Tanker::KaminariPaginatedArray.new(results,
+                                             pagination_options[:per_page],
+                                             pagination_options[:page]-1,
+                                             total_hits)
+        else
+          raise(BadConfiguration, "Unknown pagination backend")
+        end
+      end
+
+      def instantiate_results_from_results(index_result, fetch = false, snippets = false)
+        results = index_result['results']
+        return [] if results.empty?
+        instances = []   
+        id_map = results.inject({}) do |acc, result|
+          model = result["__type"]
+          instance = constantize(model).new()
+          result.each do |key, value|
+            case key
+            when /snippet/
+              # create snippet reader attribute (method)
+              instance.create_snippet_attribute(key, value)
+            when '__id'
+              # assign id attribute to the model
+              instance.id = value
+            when '__type', 'docid'
+              # do nothing
+            else
+              #assign attributes that are fetched if they match attributes in the model
+              if instance.respond_to?("#{key}=".to_sym)
+                instance.send("#{key}=", value)
+              end
+            end
           end
+          instances << instance
         end
+        instances
       end
 
       def constantize(klass_name)
         Object.const_defined?(klass_name) ?
-                  Object.const_get(klass_name) :
-                  Object.const_missing(klass_name)
+        Object.const_get(klass_name) :
+        Object.const_missing(klass_name)
+      end
+
+      # borrowed from Rails' ActiveSupport::Inflector
+      def constantize(camel_cased_word)
+        names = camel_cased_word.split('::')
+        names.shift if names.empty? || names.first.empty?
+
+        constant = Object
+        names.each do |name|
+          constant = constant.const_defined?(name) ? constant.const_get(name) : constant.const_missing(name)
+        end
+        constant
+      end
+
+      def extract_setup_paginate_options(options, defaults)
+        # extract
+        paginate_options = if options[:paginate] or options[:paginate] === false
+          options.delete(:paginate)
+        else
+          { :page => options.delete(:page), :per_page => options.delete(:per_page) }
+        end
+        # setup defaults and ensure we got integer values
+        unless paginate_options === false
+          paginate_options[:page] = defaults[:page] unless paginate_options[:page]
+          paginate_options[:per_page] = defaults[:per_page] unless paginate_options[:per_page]
+          paginate_options.each { |key, value| paginate_options[key] = value.to_i }
+        end
+        paginate_options
       end
   end
 
@@ -154,9 +229,25 @@ module Tanker
 
     attr_accessor :tanker_config
 
-    def tankit(name, &block)
+    def tankit(name = nil, options = {}, &block)
       if block_given?
-        self.tanker_config = ModelConfig.new(name, block)
+        raise(NoIndexName, 'Please provide an index name') if name.nil? && self.tanker_config.nil?
+
+        self.tanker_config ||= ModelConfig.new(name, options, Proc.new)
+        name ||= self.tanker_config.index_name
+
+        self.tanker_config.index_name = name
+
+        config = ModelConfig.new(name, block)
+        config.indexes.each do |key, value|
+          self.tanker_config.indexes << [key, value]
+        end
+
+        unless config.variables.empty?
+          self.tanker_config.variables do
+            instance_exec &config.variables.first
+          end
+        end
       else
         raise(NoBlockGiven, 'Please provide a block')
       end
@@ -176,11 +267,12 @@ module Tanker
       batches = []
       options[:batch_size] ||= 200
       records = options[:scope] ? send(options[:scope]).all : all
-      record_size = records.size
+      record_size = 0
 
       records.each_with_index do |model_instance, idx|
         batch_num = idx / options[:batch_size]
         (batches[batch_num] ||= []) << model_instance
+        record_size += 1
       end
 
       timer = Time.now
@@ -190,14 +282,21 @@ module Tanker
       end
       puts "Indexed #{record_size} #{self} records in #{Time.now - timer} seconds"
     end
+
+    def tanker_parse_doc_id(result)
+      result['docid'].split(' ').last
+    end
   end
 
   class ModelConfig
-    attr_reader :index_name
+    attr_accessor :index_name
+    attr_accessor :options
 
-    def initialize(index_name, block)
+    def initialize(index_name, options = {}, block)
       @index_name = index_name
+      @options    = options
       @indexes    = []
+      @variables  = []
       @functions  = {}
       instance_exec &block
     end
@@ -208,7 +307,7 @@ module Tanker
     end
 
     def variables(&block)
-      @variables = block if block
+      @variables << block if block
       @variables
     end
 
@@ -265,16 +364,27 @@ module Tanker
       end
 
       data[:__any] = data.values.sort_by{|v| v.to_s}.join " . "
-      data[:__type] = self.class.name
+      data[:__type] = type_name
+      data[:__id] = self.id
 
       data
     end
 
+    #dynamically create a snippet read attribute (method)
+    def create_snippet_attribute(key, value)
+      # the method name should something_snippet not snippet_something as the api returns it
+      self.class.send(:define_method,  "#{key.match(/snippet_(\w+)/)[1]}_snippet") do
+        value
+      end
+    end
+
     def tanker_index_options
       options = {}
 
-      if tanker_variables
-        options[:variables] = instance_exec(&tanker_variables)
+      unless tanker_variables.empty?
+        options[:variables] = tanker_variables.inject({}) do |hash, variables|
+          hash.merge(instance_exec(&variables))
+        end
       end
 
       options
@@ -282,7 +392,11 @@ module Tanker
 
     # create a unique index based on the model name and unique id
     def it_doc_id
-      self.class.name + ' ' + self.id.to_s
+      type_name + ' ' + self.id.to_s
+    end
+
+    def type_name
+      tanker_config.options[:as] || self.class.name
     end
   end
 end