tire 0.5.5 → 0.5.6

data/README.markdown

@@ -11,8 +11,7 @@ communicating by JSON over RESTful HTTP, based on [Lucene](http://lucene.apache.
 
 This Readme provides a brief overview of _Tire's_ features. The more detailed documentation is at <http://karmi.github.com/tire/>.
 
-Both of these documents contain a lot of information. Please set aside some time to read them thoroughly, before you blindly dive into „somehow making it work“. Just skimming through it **won't work** for you. For more information, please refer to the [integration test suite](https://github.com/karmi/tire/tree/master/test/integration)
-and [issues](https://github.com/karmi/tire/issues).
+Both of these documents contain a lot of information. Please set aside some time to read them thoroughly, before you blindly dive into „somehow making it work“. Just skimming through it **won't work** for you. For more information, please see the project [Wiki](https://github.com/karmi/tire/wiki/_pages), search the [issues](https://github.com/karmi/tire/issues), and refer to the [integration test suite](https://github.com/karmi/tire/tree/master/test/integration).
 
 Installation
 ------------
@@ -632,8 +631,9 @@ in addition to model instance, use the `each_with_hit` method:
     # One (score: 0.300123)
 ```
 
-Note that _Tire_ search results are fully compatible with [`will_paginate`](https://github.com/mislav/will_paginate),
-so you can pass all the usual parameters to the `search` method in the controller:
+Note that _Tire_ search results are fully compatible with [_WillPaginate_](https://github.com/mislav/will_paginate)
+and [_Kaminari_](https://github.com/amatsuda/kaminari), so you can pass all the usual parameters to the
+`search` method in the controller:
 
 ```ruby
     @articles = Article.search params[:q], :page => (params[:page] || 1)
@@ -648,20 +648,15 @@ OK. Chances are, you have lots of records stored in your database. How will you
 This way, however, all your records are loaded into memory, serialized into JSON,
 and sent down the wire to _Elasticsearch_. Not practical, you say? You're right.
 
-Provided your model implements some sort of _pagination_ — and it probably does —, you can just run:
+When your model is an `ActiveRecord::Base` or `Mongoid::Document` one, or when it implements
+some sort of pagination, you can just run:
 
 ```ruby
     Article.import
 ```
 
-In this case, the `Article.paginate` method is called, and your records are sent to the index
-in chunks of 1000. If that number doesn't suit you, just provide a better one:
-
-```ruby
-    Article.import :per_page => 100
-```
-
-Any other parameters you provide to the `import` method are passed down to the `paginate` method.
+Depending on the setup of your model, either `find_in_batches`, `limit..skip` or pagination is used
+to import your data.
 
 Are we saying you have to fiddle with this thing in a `rails console` or silly Ruby scripts? No.
 Just call the included _Rake_ task on the command line:
@@ -677,13 +672,6 @@ provided by the `mapping` block in your model):
     $ rake environment tire:import CLASS='Article' FORCE=true
 ```
 
-When importing records from multiple tables, you can fight the n+1 queries problem by passing
-`include` parameters to the `paginate` method:
-
-```bash
-    rake environment tire:import PARAMS='{:include => ["comments"]}' CLASS=Article FORCE=true
-```
-
 When you'll spend more time with _Elasticsearch_, you'll notice how
 [index aliases](http://www.elasticsearch.org/guide/reference/api/admin-indices-aliases.html)
 are the best idea since the invention of inverted index.
@@ -695,21 +683,16 @@ You can index your data into a fresh index (and possibly update an alias once ev
 
 Finally, consider the Rake importing task just a convenient starting point. If you're loading
 substantial amounts of data, want better control on which data will be indexed, etc., use the
-lower-level Tire API with eg. `ActiveRecordBase#find_in_batches`:
+lower-level Tire API with eg. `ActiveRecordBase#find_in_batches` directly:
 
 ```ruby
-    Article.where("published_on > ?", Time.parse("2012-10-01")).find_in_batches do |group|
-      Tire.index("articles").import group
+    Article.where("published_on > ?", Time.parse("2012-10-01")).find_in_batches(include: authors) do |batch|
+      Tire.index("articles").import batch
     end
 ```
-
-OK. All this time we have been talking about `ActiveRecord` models, since
-it is a reasonable _Rails_' default for the storage layer.
-
-But what if you use another database such as [MongoDB](http://www.mongodb.org/),
-another object mapping library, such as [Mongoid](http://mongoid.org/) or [MongoMapper](http://mongomapper.com/)?
-
-Well, things stay mostly the same:
+If you're using a different database, such as [MongoDB](http://www.mongodb.org/),
+another object mapping library, such as [Mongoid](http://mongoid.org/) or [MongoMapper](http://mongomapper.com/),
+things stay mostly the same:
 
 ```ruby
     class Article
@@ -779,7 +762,7 @@ and extensions to the core _Tire_ functionality — be sure to check them out.
 Other Clients
 -------------
 
-Check out [other _Elasticsearch_ clients](http://www.elasticsearch.org/guide/appendix/clients.html).
+Check out [other _Elasticsearch_ clients](http://www.elasticsearch.org/guide/clients/).
 
 
 Feedback

data/lib/tire/http/clients/curb.rb

@@ -7,49 +7,53 @@ module Tire
     module Client
 
       class Curb
-        @client = ::Curl::Easy.new
-        @client.resolve_mode = :ipv4
-
-        # @client.verbose = true
+        def self.client
+          Thread.current[:client] ||= begin
+            client = ::Curl::Easy.new
+            client.resolve_mode = :ipv4
+            # client.verbose = true
+            client
+          end
+        end
 
         def self.get(url, data=nil)
-          @client.url = url
+          client.url = url
 
           # FIXME: Curb cannot post bodies with GET requests?
           #        Roy Fielding seems to approve:
           #        <http://tech.groups.yahoo.com/group/rest-discuss/message/9962>
           if data
-            @client.post_body = data
-            @client.http_post
+            client.post_body = data
+            client.http_post
           else
-            @client.http_get
+            client.http_get
           end
-          Response.new @client.body_str, @client.response_code
+          Response.new client.body_str, client.response_code
         end
 
         def self.post(url, data)
-          @client.url = url
-          @client.post_body = data
-          @client.http_post
-          Response.new @client.body_str, @client.response_code
+          client.url = url
+          client.post_body = data
+          client.http_post
+          Response.new client.body_str, client.response_code
         end
 
         def self.put(url, data)
-          @client.url = url
-          @client.http_put data
-          Response.new @client.body_str, @client.response_code
+          client.url = url
+          client.http_put data
+          Response.new client.body_str, client.response_code
         end
 
         def self.delete(url)
-          @client.url = url
-          @client.http_delete
-          Response.new @client.body_str, @client.response_code
+          client.url = url
+          client.http_delete
+          Response.new client.body_str, client.response_code
         end
 
         def self.head(url)
-          @client.url = url
-          @client.http_head
-          Response.new @client.body_str, @client.response_code
+          client.url = url
+          client.http_head
+          Response.new client.body_str, client.response_code
         end
 
       end

data/lib/tire/index.rb

@@ -245,12 +245,13 @@ module Tire
       case
         when method = options.delete(:method)
           options = {:page => 1, :per_page => 1000}.merge options
-          while documents = klass_or_collection.send(method.to_sym, options.merge(:page => options[:page])) \
-                            and documents.to_a.length > 0
+          while (documents = klass_or_collection.send(method.to_sym, options.merge(:page => options[:page]))) \
+                            && documents.to_a.length > 0
 
             documents = yield documents if block_given?
 
             bulk_store documents, options
+            GC.start
             options[:page] += 1
           end
 
@@ -259,8 +260,8 @@ module Tire
           bulk_store documents, options
 
         else
-          raise ArgumentError, "Please pass either an Enumerable compatible class, or a collection object" +
-                               "with a method for fetching records in batches (such as 'paginate')"
+          raise ArgumentError, "Please pass either an Enumerable compatible class, or a collection object " +
+                               "with a method for fetching records in batches (such as 'paginate')."
       end
     end
 

data/lib/tire/model/import.rb

@@ -1,27 +1,114 @@
-
 module Tire
   module Model
 
-    # Provides support for easy importing of large ActiveRecord- and ActiveModel-bound
-    # recordsets into model index.
+    # Provides support for efficient and effective importing of large recordsets into Elasticsearch.
+    #
+    # Tire will use dedicated strategies for fetching records in batches when ActiveRecord or Mongoid models are detected.
+    #
+    # Two dedicated strategies for popular pagination libraries are also provided: WillPaginate and Kaminari.
+    # These could be used in situations where your model is neither ActiveRecord nor Mongoid based.
     #
-    # Relies on pagination support in your model, namely the `paginate` class method.
+    # Note, that it's always possible to use the `Tire::Index#import` method directly.
     #
-    # Please refer to the relevant of the README for more information.
+    # @note See `Tire::Import::Strategy`.
     #
     module Import
 
       module ClassMethods
-
         def import options={}, &block
-          options = { :method => 'paginate' }.update options
-          index   = options[:index] ? Tire::Index.new(options.delete(:index)) : self.index
-          index.import klass, options, &block
+          strategy = Strategy.from_class(klass, options)
+          strategy.import &block
+        end
+      end
+
+      # Importing strategies for common persistence frameworks (ActiveModel, Mongoid), as well as
+      # pagination libraries (WillPaginate, Kaminari).
+      #
+      module Strategy
+        def self.from_class(klass, options={})
+          case
+          when defined?(::ActiveRecord) && klass.ancestors.include?(::ActiveRecord::Base)
+            ActiveRecord.new klass, options
+          when defined?(::Mongoid::Document) && klass.ancestors.include?(::Mongoid::Document)
+            Mongoid.new klass, options
+          when defined?(Kaminari) && klass.respond_to?(:page)
+            Kaminari.new klass, options
+          when defined?(WillPaginate) && klass.respond_to?(:paginate)
+            WillPaginate.new klass, options
+          else
+            Default.new klass, options
+          end
         end
 
+        module Base
+          attr_reader :klass, :options, :index
+          def initialize(klass, options={})
+            @klass   = klass
+            @options = {:per_page => 1000}.update(options)
+            @index   = options[:index] ? Tire::Index.new(options.delete(:index)) : klass.tire.index
+          end
+        end
+
+        class ActiveRecord
+          include Base
+          def import &block
+            klass.find_in_batches(:batch_size => options[:per_page]) do |batch|
+              index.import batch, options, &block
+            end
+            self
+          end
+        end
+
+        class Mongoid
+          include Base
+          def import &block
+            0.step(klass.count, options[:per_page]) do |offset|
+              items = klass.limit(options[:per_page]).skip(offset)
+              index.import items.to_a, options, &block
+            end
+            self
+          end
+        end
+
+        class Kaminari
+          include Base
+          def import &block
+            current = 0
+            page = 1
+            while current < klass.count
+              items = klass.page(page).per(options[:per_page])
+              index.import items, options, &block
+              current = current + items.size
+              page += 1
+            end
+            self
+          end
+        end
+
+        class WillPaginate
+          include Base
+          def import &block
+            current = 0
+            page = 1
+            while current < klass.count
+              items = klass.paginate(:page => page, :per_page => options[:per_page])
+              index.import items, options, &block
+              current += items.size
+              page += 1
+            end
+            self
+          end
+        end
+
+        class Default
+          include Base
+          def import &block
+            index.import klass, options.update(:method => 'paginate'), &block
+            self
+          end
+        end
       end
 
     end
-
   end
 end

data/lib/tire/model/persistence/attributes.rb

@@ -87,13 +87,14 @@ module Tire
           attr_accessor :id
 
           def initialize(attributes={})
-            # Make a copy of objects in the property defaults hash, so default values such as `[]` or `{ foo: [] }` are left intact
+            # Make a copy of objects in the property defaults hash, so default values
+            # such as `[]` or `{ foo: [] }` are preserved.
+            #
             property_defaults = self.class.property_defaults.inject({}) do |hash, item|
               key, value = item
-              hash[key.to_s] = value.class.respond_to?(:new) ? value.clone : value
+              hash[key.to_sym] = value.class.respond_to?(:new) ? value.clone : value
               hash
             end
-
             __update_attributes(property_defaults.merge(attributes))
           end
 

data/lib/tire/search/query.rb

@@ -56,6 +56,10 @@ module Tire
         @value
       end
 
+      def constant_score(&block)
+        @value.update( { :constant_score => ConstantScoreQuery.new(&block).to_hash } ) if block_given?
+      end
+
       def fuzzy(field, value, options={})
         query = { field => { :term => value }.update(options) }
         @value = { :fuzzy => query }
@@ -182,6 +186,29 @@ module Tire
       end
     end
 
+    class ConstantScoreQuery
+      def initialize(&block)
+        @value = {}
+        block.arity < 1 ? self.instance_eval(&block) : block.call(self) if block_given?
+      end
+
+      def filter(type, *options)
+        @value.update(:filter => Filter.new(type, *options).to_hash)
+      end
+
+      def query(&block)
+        @value.update(:query => Query.new(&block).to_hash)
+      end
+
+      def boost(boost)
+        @value.update(:boost => boost)
+      end
+
+      def to_hash
+        @value
+      end
+    end
+
     class DisMaxQuery
       def initialize(options={}, &block)
         @options = options

data/lib/tire/tasks.rb

@@ -1,116 +1,169 @@
 require 'rake'
-require 'benchmark'
+
+module Tire
+  module Tasks
+
+    module Import
+      HRULE = '='*90
+
+      def delete_index(index)
+        puts "[IMPORT] Deleting index '#{index.name}'"
+        index.delete
+      end
+
+      def create_index(index, klass)
+        unless index.exists?
+          mapping = MultiJson.encode(klass.tire.mapping_to_hash, :pretty => Tire::Configuration.pretty)
+          puts "[IMPORT] Creating index '#{index.name}' with mapping:", mapping
+          unless index.create(:mappings => klass.tire.mapping_to_hash, :settings => klass.tire.settings)
+            puts "[ERROR] There has been an error when creating the index -- Elasticsearch returned:",
+                        index.response
+            exit(1)
+          end
+        end
+      end
+
+      def add_pagination_to_klass(klass)
+        if defined?(Kaminari) && klass.respond_to?(:page)
+          klass.instance_eval do
+            def paginate(options = {})
+              page(options[:page]).per(options[:per_page])
+            end
+          end
+        end unless klass.respond_to?(:paginate)
+      end
+
+      def progress_bar(klass, total=nil)
+        @progress_bars ||= {}
+
+        if total
+          @progress_bars[klass.to_s] ||= ANSI::Progressbar.new(klass.to_s, total)
+        else
+          @progress_bars[klass.to_s]
+        end
+      end
+
+      def import_model(index, klass, params)
+        unless progress_bar(klass)
+          puts "[IMPORT] Importing '#{klass.to_s}'"
+        end
+        klass.tire.import(params) do |documents|
+          progress_bar(klass).inc documents.size if progress_bar(klass)
+          documents
+        end
+        progress_bar(klass).finish if progress_bar(klass)
+      end
+
+      extend self
+    end
+
+  end
+end
 
 namespace :tire do
 
-  full_comment_import = <<-DESC.gsub(/    /, '')
-    Import data from your model using paginate: rake environment tire:import CLASS='MyModel'.
+  import_model_desc = <<-DESC.gsub(/    /, '')
+    Import data from your model (pass name as CLASS environment variable).
+
+      $ rake environment tire:import:model CLASS='MyModel'
 
-    Pass params for the `paginate` method:
-      $ rake environment tire:import CLASS='Article' PARAMS='{:page => 1}'
+    Pass params for the `import` method:
+      $ rake environment tire:import:model CLASS='Article' PARAMS='{:page => 1}'
 
     Force rebuilding the index (delete and create):
-      $ rake environment tire:import CLASS='Article' PARAMS='{:page => 1}' FORCE=1
+      $ rake environment tire:import:model CLASS='Article' FORCE=1
 
     Set target index name:
-      $ rake environment tire:import CLASS='Article' INDEX='articles-new'
+      $ rake environment tire:import:model CLASS='Article' INDEX='articles-new'
   DESC
-  desc full_comment_import
-  task :import do |t|
-
-    def elapsed_to_human(elapsed)
-      hour = 60*60
-      day  = hour*24
-
-      case elapsed
-      when 0..59
-        "#{sprintf("%1.2f", elapsed)} seconds"
-      when 60..hour-1
-        "#{(elapsed/60).floor} minutes and #{(elapsed % 60).floor} seconds"
-      when hour..day
-        "#{(elapsed/hour).floor} hours and #{(elapsed/60 % hour).floor} minutes"
-      else
-        "#{(elapsed/hour).round} hours"
+
+  import_all_desc = <<-DESC.gsub(/    /, '')
+    Import all indices from `app/models` (or use DIR environment variable).
+
+      $ rake environment tire:import:all DIR=app/models
+  DESC
+
+  task :import => 'import:model'
+
+  namespace :import do
+    desc import_model_desc
+    task :model do
+      if defined?(Rails)
+        puts "[IMPORT] Rails detected, loading environment..."
+        Rake::Task["environment"].invoke
       end
-    end
 
-    if ENV['CLASS'].to_s == ''
-      puts '='*90, 'USAGE', '='*90, full_comment_import, ""
-      exit(1)
-    end
+      if ENV['CLASS'].to_s == ''
+        puts HRULE, 'USAGE', HRULE, import_model_desc, ""
+        exit(1)
+      end
+
+      klass  = eval(ENV['CLASS'].to_s)
+      params = eval(ENV['PARAMS'].to_s) || {}
+      total  = klass.count rescue nil
+
+      if ENV['INDEX']
+        index = Tire::Index.new(ENV['INDEX'])
+        params[:index] = index.name
+      else
+        index = klass.tire.index
+      end
 
-    klass  = eval(ENV['CLASS'].to_s)
-    params = eval(ENV['PARAMS'].to_s) || {}
+      Tire::Tasks::Import.add_pagination_to_klass(klass)
+      Tire::Tasks::Import.progress_bar(klass, total) if total
 
-    params.update :method => 'paginate'
+      Tire::Tasks::Import.delete_index(index) if ENV['FORCE']
+      Tire::Tasks::Import.create_index(index, klass)
 
-    index = Tire::Index.new( ENV['INDEX'] || klass.tire.index.name )
+      Tire::Tasks::Import.import_model(index, klass, params)
 
-    if ENV['FORCE']
-      puts "[IMPORT] Deleting index '#{index.name}'"
-      index.delete
+      puts '[IMPORT] Done.'
     end
 
-    unless index.exists?
-      mapping = MultiJson.encode(klass.tire.mapping_to_hash, :pretty => Tire::Configuration.pretty)
-      puts "[IMPORT] Creating index '#{index.name}' with mapping:", mapping
-      unless index.create( :mappings => klass.tire.mapping_to_hash, :settings => klass.tire.settings )
-        STDERR.puts "[ERROR] There has been an error when creating the index -- elasticsearch returned:",
-                    index.response
-        exit(1)
+    desc import_all_desc
+    task :all do
+      if defined?(Rails)
+        puts "[IMPORT] Rails detected, loading environment..."
+        Rake::Task["environment"].invoke
       end
-    end
 
-    STDOUT.sync = true
-    puts "[IMPORT] Starting import for the '#{ENV['CLASS']}' class"
-    tty_cols = 80
-    total    = klass.count rescue nil
-    offset   = (total.to_s.size*2)+8
-    done     = 0
-
-    STDOUT.puts '-'*tty_cols
-    elapsed = Benchmark.realtime do
-
-      # Add Kaminari-powered "paginate" method
-      #
-      if defined?(Kaminari) && klass.respond_to?(:page)
-        klass.instance_eval do
-          def paginate(options = {})
-            page(options[:page]).per(options[:per_page])
-          end
-        end
-      end unless klass.respond_to?(:paginate)
+      dir    = ENV['DIR'].to_s != '' ? ENV['DIR'] : Rails.root.join("app/models")
+      params = eval(ENV['PARAMS'].to_s) || {}
 
-      # Import the documents
-      #
-      index.import(klass, params) do |documents|
+      puts "[IMPORT] Loading models from: #{dir}"
+      Dir.glob(File.join("#{dir}/**/*.rb")).each do |path|
+        require path
 
-        if total
-          done += documents.to_a.size
-          # I CAN HAZ PROGREZ BAR LIEK HOMEBRU!
-          percent  = ( (done.to_f / total) * 100 ).to_i
-          glyphs   = ( percent * ( (tty_cols-offset).to_f/100 ) ).to_i
-          STDOUT.print( "#" * glyphs )
-          STDOUT.print( "\r"*tty_cols+"#{done}/#{total} | \e[1m#{percent}%\e[0m " )
-        end
+        model_filename = path[/#{Regexp.escape(dir.to_s)}\/([^\.]+).rb/, 1]
+        klass          = model_filename.classify.constantize
 
-        # Don't forget to return the documents collection back!
-        documents
+        # Skip if the class doesn't have Tire integration
+        next unless klass.respond_to?(:tire)
+
+        total  = klass.count rescue nil
+
+        Tire::Tasks::Import.add_pagination_to_klass(klass)
+        Tire::Tasks::Import.progress_bar(klass, total) if total
+
+        index = klass.tire.index
+        Tire::Tasks::Import.delete_index(index) if ENV['FORCE']
+        Tire::Tasks::Import.create_index(index, klass)
+
+        Tire::Tasks::Import.import_model(index, klass, params)
+        puts
       end
+
+      puts '[Import] Done.'
     end
 
-    puts "", '='*80, "Import finished in #{elapsed_to_human(elapsed)}"
   end
 
   namespace :index do
 
     full_comment_drop = <<-DESC.gsub(/      /, '')
-      Delete indices passed in the INDEX environment variable; separate multiple indices by comma.
+      Delete indices passed in the INDEX/INDICES environment variable; separate multiple indices by comma.
 
-      Pass name of a single index to drop in the INDEX environmnet variable:
         $ rake environment tire:index:drop INDEX=articles
-
-      Pass names of multiple indices to drop in the INDEX or INDICES environmnet variable:
         $ rake environment tire:index:drop INDICES=articles-2011-01,articles-2011-02
 
     DESC
@@ -126,7 +179,7 @@ namespace :tire do
       index_names.each do |name|
         index = Tire::Index.new(name)
         print "* Deleting index \e[1m#{index.name}\e[0m... "
-        puts  index.delete ? "\e[32mOK\e[0m" : "\e[31mFAILED\e[0m  | #{index.response.body}"
+        puts index.delete ? "\e[32mOK\e[0m" : "\e[31mFAILED\e[0m  | #{index.response.body}"
       end
 
       puts ""