escargot 0.0.2

data/.gitignore

@@ -0,0 +1,9 @@
+*.log
+*.db
+
+# Generate gem file
+*.gem
+
+# vim swap files
+*.swp
+*.swo

data/MIT-LICENSE

@@ -0,0 +1,20 @@
+Copyright (c) 2010 [name of plugin creator]
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.markdown

@@ -0,0 +1,376 @@
+
+Escargot
+============
+
+Connects any Rails model with ElasticSearch, supports near real time updates,
+distributed indexing and models that integrate data from many databases.  
+
+Requirements
+============
+Currently only rails 2.3 is supported. You will need ElasticSearch, the 'rubberband' gem
+and (if you want to use the **optional** distributed indexing mode) Redis. 
+
+Usage
+=======
+
+First, [download and start ElasticSearch](http://www.elasticsearch.com/docs/elasticsearch/setup/installation/) (it's really simple). With the default setup of
+of ElasticSearch (listening to localhost and port 9200) no configuration of the plugin is 
+necessary. 
+
+To define an index, simply add a line to your model
+
+    class Post < ActiveRecord::Base
+      elastic_index
+    end
+
+To create the index, execute the rake task that rebuilds all indexes:
+
+    rake escargot:index
+
+Or restrict it to just one model
+    
+    rake "escargot:index[Post]"
+    
+And you are ready to search:
+
+    Post.search "dreams OR nightmares" 
+
+Near Real Time support
+=======
+
+The default behavior is that every time you save or delete a record in an indexed model
+the index will be updated to reflect the changes. You can disable this by 
+
+    class Post < ActiveRecord::Base
+      elastic_index :updates => false
+    end
+
+Please notice that when updates are enabled there may be a slight delay for the changes to appear
+in search results (with the default elasticsearch settings, this delay is just 1 second). If 
+you absolutely need to ensure that the change is made public before returning control to the user,
+ the `:immediate_with_refresh` option provides this assurance.
+
+    class Post < ActiveRecord::Base
+      elastic_index :updates => :immediate_with_refresh
+    end
+
+Enabling `:immediate_with_refresh` is not recommended. A better option is to simply call `Post.refresh_index`
+when you really need the guarantee. 
+
+Choosing the indexed fields
+=======
+
+This plugin doesn't provide a DSL to define what fields you want to be indexed. Instead of that
+it exposes the fact that in ElasticSearch every document is just a JSON string. 
+
+If you define a `indexed_json_document` method in your model this will be used as the JSON 
+representation of the document, otherwise `to_json` will be called instead.
+
+Luckily, ActiveRecord has excellent support for JSON serialization, so it's really easy
+to include associations or custom methods.
+
+     class Post < ActiveRecord::Base
+      elastic_index :updates => false
+      belongs_to :category
+  
+      def indexed_json_document 
+        to_json(:include => :category, :methods => :slug)
+      end
+  
+      def slug
+        title.downcase.gsub(" ", "-")
+      end
+     end
+
+See [ActiveRecord's JSON serialization documentation](http://api.rubyonrails.org/classes/ActiveModel/Serializers/JSON.html)
+
+Search features
+=======
+
+Basic Searching
+----------------
+
+Calling `Model.search` obtains from ElasticSearch the ids of the results matching 
+your query and then queries your database to get the full ActiveRecord objects.
+
+    results = Post.search "dreams OR nightmares" 
+    results.each {|r| puts r.title}
+
+The query is parsed using lucene's [QueryParser syntax](http://lucene.apache.org/java/2_4_0/queryparsersyntax.html).
+You can use boolean operators, restrict your search to a field, etc.
+
+    results = Post.search "prologue:dream OR epilogue:nightmare" 
+
+You can also guide the interpretation of the query, with the options `:default_operator` and `:df` (default_field). These two are equivalent:
+
+    results = Post.search "title:(dreams AND nightmares)"
+    results = Post.search "dreams nightmares" , :default_operator => 'AND', :df => 'title'
+
+Sorting by attributes
+--------
+
+The default order is based on the relevancy of the terms in the document. You can also
+sort by any other field's value.
+
+    Post.search "dreams", :order => :updated_at
+    Post.search "dreams", :order => 'updated_at:desc'
+    Post.search "dreams", :order => ['popularity:desc', 'updated_at:desc']
+
+Sorting by an arbitrary script is possible using the Query DSL. 
+  
+Pagination
+----------
+
+`search` returns a WillPaginate collection and accepts the customary *:per\_page*, and *:page* parameters.
+
+    # controller
+    @posts = Post.search("dreams", :page => params[:page], :per_page => 30)
+  
+    # in the view:
+    will_paginate @posts
+  
+  
+Query DSL
+----------------
+Instead of a string, you can pass a query in ElasticSearch's [Query DSL](http://www.elasticsearch.com/docs/elasticsearch/rest_api/query_dsl/)
+giving you access to the full range of search features. 
+
+    Bird.search(:match_all => true}  
+      
+    Bird.search(:fuzzy => {:name => 'oriale'})
+
+    Bird.search(:custom_score => {:query => {:match_all => true}, :script => "random()"})
+    
+    Bird.search(:dis_max => {
+      :tie_breaker => 0.7,
+      :boost => 1.2,
+      :queries => [:term => {:name => 'oriole'}, :term => {:content => 'oriole'}]
+    })
+
+    Bird.search(:more_like_this => {
+      :like_text => "The orioles are a family of Old World passerine birds"
+    })
+
+ 
+    Bird.search(
+      :filtered => {
+        :query => {
+          :term => {:name => 'oriole'}
+        },
+        :filter => {
+          :term => {:suborder => 'Passeri'}
+        }
+      }
+    )
+  
+Facets
+----------------
+  
+Term facets returning the most popular terms for a field and partial results counts are 
+available through the `facets` class method.
+
+    Post.facets :author_id
+    Post.facets :author_id, :size => 100
+    
+    # restrict the facets to posts that contain 'dream'
+    Post.facets :author_id, :query => "dream"
+    Post.facets [:author_id, :category], :query => "dream"     
+
+This returns a Hash of the form:
+
+    {
+     :author_id => {
+       "1" => 3,
+       "25" => 2
+      }, 
+      :category_id => {
+         12 => 4,
+         42 => 7,
+         47 => 2
+       }
+    }
+
+<!-- You can also combine the standard search results with the facets counts in a single request. 
+
+    results = Post.search_with_facets [:author, :category], :query => "dream"
+    results.each{|post| puts post.title}
+    results.facets -->
+
+You should be aware that this only a very simple subset of the facets feature of ElasticSearch.
+The full feature set (histograms, statistical facets, geo distance facets, etc.) is available
+through the Query DSL. 
+
+  
+Search counts
+----------
+
+Use `search_count` to count the number of matches without getting the results.  
+
+    Post.search_count("dream OR nightmare")
+  
+  
+Index Creation and Type Mapping Options
+=======
+
+Index creation options
+-----------------------
+
+Any value passed in the :index\_options argument will be sent to ElasticSearch as an index
+creation option.
+
+For example, if you want to increase the number of shards for this index:
+ 
+    class Post < ActiveRecord::Base
+      elastic_index :index_options => {:number_of_shards => 10}
+    end
+
+If you want the search to be insensitive to accents and other diacritics:
+
+    class Post < ActiveRecord::Base
+      elastic_index :index_options => {
+          "analysis.analyzer.default.tokenizer" => 'standard',
+          "analysis.analyzer.default.filter" => ["standard", "lowercase", "stop", "asciifolding"]
+      }
+    end
+
+The full list of available options for index creation is documented at
+[http://www.elasticsearch.com/docs/elasticsearch/index_modules/](http://www.elasticsearch.com/docs/elasticsearch/index_modules/)
+
+Mapping options
+----------------
+
+Mapping is the process of defining how a JSON document should be mapped to the Search Engine,
+including its searchable characteristics.
+
+The default (dynamic) mapping provides sane defaults, but defining your own mapping enables
+powerful features such as boosting a field, using a different analyzer for one field, 
+enabling term vectors, etc.
+
+Some examples:
+
+    class Post < ActiveRecord::Base
+      elastic_index :mapping_options => {
+        :properties => {
+          :category => {:type => :string, :index => :not_analyzed}, 
+          :title => {:type => :string, :index => :analyzed, :term_vector => true, :boost => 10.0},
+          :location => {:type => :geo_point}
+        }
+      }
+    end
+
+
+See the [ElasticSearch Documentation](http://www.elasticsearch.com/docs/elasticsearch/mapping/) for mappings.
+
+Distributed indexing
+=======
+You will need distributed indexing when there is a large amount of data to be indexed. In this 
+indexing mode the task of creating an index is divided between a pool of workers that can be 
+as large as  you need. Since ElasticSearch itself provides linear indexing scalability by adding 
+nodes to the cluster, this means that you should, in principle, be able to make your indexing
+time arbitrarily short.  
+
+Currently, the only work queue supported is [Resque](http://github.com/defunkt/resque). To enable distributed indexing you
+should first install Redis and set-up Resque. 
+
+If you're on OS X and use homebrew, installing redis can be done with:
+
+    brew install redis
+    redis-server /usr/local/etc/redis.conf
+
+Install the resque gem:
+
+    $ gem install resque
+  
+Include it on your application:
+
+    require 'resque'
+
+Add this to your Rakefile:
+
+    require 'resque/tasks'
+    namespace :resque do
+      task :setup => :environment
+    end
+  
+And use the resque:work rake task to start a worker:
+
+     $ QUEUE=es_admin,es_nrt,es_batch rake resque:work
+
+Once you have set-up Resque and started a number of workers, you can easily create an index for you model using the distributed model:
+
+    rake "elasticsearch:distributed_index[Post]"
+
+or if you want to re-create all your indexes
+
+    rake elasticsearch:distributed_index
+
+Be aware that due the distributed nature of indexing the new index may be deployed when some workers are still performing their last
+indexing job. 
+
+Setting up a resque work queue also allows you to use the *:update => :enqueue* option
+
+    class Post < ActiveRecord::Base
+      elastic_index :update => :enqueue
+    end
+
+With this setting when a document is updated or deleted the task of updating the index is 
+added to the work queue and will be performed asynchronously by a remote agent. 
+ 
+Index versions
+=======
+In *escargot* indexes are versioned: when you create an index for the
+model Post the actual index created in ElasticSearch will be named something like
+'posts_1287849616.57665' with an alias 'posts' pointing to it. The second time 
+you run the "escargot:index" tasks a new index version will be created and the 
+alias will be updated only when the new index is ready. 
+
+This is useful because it makes the deployment of a new index version atomic. 
+
+When a document is saved and index updates are enabled, both the current index version
+and any version that's in progress will be updated. This ensures that when the new
+index is published it will include the change. 
+
+Searching multiple models
+================
+You can use all the same syntax to search across all indexed models in your application:
+
+    Escargot.search "dreams"
+
+Calling `Escargot.search "dreams"` will return all objects that match, no matter what model they are from, ordered by relevance
+
+If you want to limit global searches to a few specific models, you can do so with the `:classes` option
+
+    Escargot.search "dreams", :classes => [Post, Bird]
+
+Support similar behavior that `Basic Searching` and `Search counts`
+
+Contributing
+================
+Fork on GitHub, create a test & send a pull request. 
+
+Bugs
+================
+Use the [Issue Tracker](http://github.com/angelf/escargot/issues)
+
+ 
+Aknowledgements
+================
+* Some parts of the API plagiarize the excellent Thinking Sphinx plugin, and more will do so in the future.
+* This plugin depends on rubberband for communication with ElasticSearch.
+* Elastic Search rules!
+
+Future Plans
+======
+
+* Search features:
+  * Field conditions and term filters
+  * Single-table inheritance support
+  * (optionally) use the _source field from ES and avoid querying the database 
+
+* Indexing features:
+  * Distributing the task of listing document ids
+  * Index partioning
+  * Support for non-ActiveRecord models
+  * Adding other queue backends
+
+Copyright (c) 2010 Angel Faus & vLex.com, released under the MIT license

data/Rakefile

@@ -0,0 +1,23 @@
+require 'rake'
+require 'rake/testtask'
+require 'rake/rdoctask'
+
+desc 'Default: run unit tests.'
+task :default => :test
+
+desc 'Test the elastic_rails plugin.'
+Rake::TestTask.new(:test) do |t|
+  t.libs << 'lib'
+  t.libs << 'test'
+  t.pattern = 'test/**/*_test.rb'
+  t.verbose = true
+end
+
+desc 'Generate documentation for the elastic_rails plugin.'
+Rake::RDocTask.new(:rdoc) do |rdoc|
+  rdoc.rdoc_dir = 'rdoc'
+  rdoc.title    = 'ElasticRails'
+  rdoc.options << '--line-numbers' << '--inline-source'
+  rdoc.rdoc_files.include('README')
+  rdoc.rdoc_files.include('lib/**/*.rb')
+end

data/escargot.gemspec

@@ -0,0 +1,24 @@
+# -*- encoding: utf-8 -*-
+require File.expand_path("../lib/escargot/version", __FILE__)
+
+Gem::Specification.new do |s|
+  s.name        = "escargot"
+  s.version     = Escargot::VERSION
+  s.platform    = Gem::Platform::RUBY
+  s.authors     = ["Angel Faus"]
+  s.email       = ["angel@vlex.com"]
+  s.homepage    = "http://github.com/angelf/escargot"
+  s.summary     = "ElasticSearch connector for Rails"
+  s.description = "Connects any Rails model with ElasticSearch, supports near real time updates, distributed indexing and models that integrate data from many databases."
+
+  s.required_rubygems_version = ">= 1.3.6"
+  s.rubyforge_project         = "escargot"
+
+  s.add_development_dependency "bundler", ">= 1.0.0"
+  
+  s.add_dependency "rubberband", ">= 0.0.5"
+
+  s.files        = `git ls-files`.split("\n")
+  s.executables  = `git ls-files`.split("\n").map{|f| f =~ /^bin\/(.*)/ ? $1 : nil}.compact
+  s.require_path = 'lib'
+end

data/init.rb

@@ -0,0 +1 @@
+# Include hook code here

data/install.rb

@@ -0,0 +1 @@
+# Install hook code here

data/lib/escargot/activerecord_ex.rb

@@ -0,0 +1,193 @@
+require 'will_paginate/collection'
+
+module Escargot
+  module ActiveRecordExtensions
+
+    def self.included(base)
+      base.send :extend, ClassMethods
+    end
+
+    module ClassMethods
+      attr_accessor :index_name
+      attr_accessor :update_index_policy
+
+      # defines an elastic search index. Valid options:
+      #
+      # :index_name (will default class name using method "underscore")
+      #
+      # :updates, how to to update the contents of the index when a document is changed, valid options are:
+      #
+      #   - false: do not update the index
+      #
+      #   - :immediate: update the index but do not refresh it automatically.
+      #     With the default settings, this means that the change may take up to 1 second
+      #     to be seen by other users.
+      #
+      #     See: http://www.elasticsearch.com/docs/elasticsearch/index_modules/engine/robin/
+      #
+      #     This is the default option.
+      #
+      #   - :immediate_with_refresh: update the index AND ask elasticsearch to refresh it after each
+      #     change. This garantuees that the changes will be seen by other users, but may affect
+      #     performance.
+      #
+      #   - :enqueu: enqueue the document id so that a remote worker will update the index
+      #     This is the recommended options if you have set up a job queue (such as Resque)
+      #
+
+      def elastic_index(options = {})
+        Escargot.register_model(self)
+
+        options.symbolize_keys!
+        send :include, InstanceMethods
+        @index_name = options[:index_name] || self.name.underscore.gsub(/\//,'-')
+        @update_index_policy = options.include?(:updates) ? options[:updates] : :immediate
+        
+        if @update_index_policy
+          after_save :update_index
+          after_destroy :delete_from_index
+        end
+        @index_options = options[:index_options] || {}
+        @mapping = options[:mapping] || false
+      end
+
+      def search(query, options={})
+        Escargot.search(query, options.merge({:index => self.index_name, :type => elastic_search_type}), true)
+      end
+      
+      def search_hits(query, options = {})
+        Escargot.search_hits(query, options.merge({:index => self.index_name, :type => elastic_search_type}), true)
+      end  
+
+      def search_count(query = "*", options = {})
+        Escargot.search_count(query, options.merge({:index => self.index_name, :type => elastic_search_type}), true)
+      end
+
+      def facets(fields_list, options = {})
+        size = options.delete(:size) || 10
+        fields_list = [fields_list] unless fields_list.kind_of?(Array)
+        
+        if !options[:query]
+          options[:query] = {:match_all => true}
+        elsif options[:query].kind_of?(String)
+          options[:query] = {:query_string => {:query => options[:query]}}
+        end
+
+        options[:facets] = {}
+        fields_list.each do |field|
+          options[:facets][field] = {:terms => {:field => field, :size => size}}
+        end
+
+        hits = $elastic_search_client.search(options, {:index => self.index_name, :type => elastic_search_type})
+        out = {}
+        
+        fields_list.each do |field|
+          out[field.to_sym] = {}
+          hits.facets[field.to_s]["terms"].each do |term|
+            out[field.to_sym][term["term"]] = term["count"]
+          end
+        end
+
+        out
+      end
+
+      # explicitly refresh the index, making all operations performed since the last refresh
+      # available for search
+      #
+      # http://www.elasticsearch.com/docs/elasticsearch/rest_api/admin/indices/refresh/
+      def refresh_index(index_version = nil)
+        $elastic_search_client.refresh(index_version || index_name)
+      end
+      
+      # creates a new index version for this model and sets the mapping options for the type
+      def create_index_version
+        index_version = $elastic_search_client.create_index_version(@index_name, @index_options)
+        if @mapping
+          $elastic_search_client.update_mapping(@mapping, :index => index_version, :type => elastic_search_type)
+        end
+        index_version
+      end
+      
+      # deletes all index versions for this model and the alias (if exist)
+      def delete_index
+        # set current version to delete alias later
+        current_version = $elastic_search_client.current_index_version(index_name)
+
+        # deletes any index version and the alias
+        $elastic_search_client.index_versions(index_name).each{|index_version|
+          $elastic_search_client.alias_index(:remove => {index_version => index_name}) if (index_version == current_version)
+          $elastic_search_client.delete_index(index_version)
+        }
+
+        # and delete the index itself if it exists
+        begin
+          $elastic_search_client.delete_index(index_name)
+        rescue ElasticSearch::RequestError
+          # it's ok, this means that the index doesn't exist
+        end
+      end
+      
+      def delete_id_from_index(id, options = {})
+        options[:index] ||= self.index_name
+        options[:type]  ||= elastic_search_type
+        $elastic_search_client.delete(id.to_s, options)
+      end
+      
+      def optimize_index
+        $elastic_search_client.optimize(index_name)
+      end
+      
+      private
+        def elastic_search_type
+          self.name.underscore.singularize.gsub(/\//,'-')
+        end
+
+    end
+
+    module InstanceMethods
+
+      # updates the index using the appropiate policy
+      def update_index
+        if self.class.update_index_policy == :immediate_with_refresh
+          local_index_in_elastic_search(:refresh => true)
+        elsif self.class.update_index_policy == :enqueue
+          Resque.enqueue(DistributedIndexing::ReIndexDocuments, self.class.to_s, [self.id])
+        else
+          local_index_in_elastic_search
+        end
+      end
+
+      # deletes the document from the index using the appropiate policy ("simple" or "distributed")
+      def delete_from_index
+        if self.class.update_index_policy == :immediate_with_refresh
+          self.class.delete_id_from_index(self.id, :refresh => true)
+          # As of Oct 25 2010, :refresh => true is not working
+          self.class.refresh_index()
+        elsif self.class.update_index_policy == :enqueue
+          Resque.enqueue(DistributedIndexing::ReIndexDocuments, self.class.to_s, [self.id])
+        else
+          self.class.delete_id_from_index(self.id)
+        end
+      end
+
+      def local_index_in_elastic_search(options = {})
+        options[:index] ||= self.class.index_name
+        options[:type]  ||= self.class.name.underscore.singularize.gsub(/\//,'-')
+        options[:id]    ||= self.id.to_s
+        
+        $elastic_search_client.index(
+          self.respond_to?(:indexed_json_document) ? self.indexed_json_document : self.to_json,
+          options
+        )
+        
+        ## !!!!! passing :refresh => true should make ES auto-refresh only the affected
+        ## shards but as of Oct 25 2010 with ES 0.12 && rubberband 0.0.2 that's not the case
+        if options[:refresh]
+          self.class.refresh_index(options[:index])
+        end
+          
+      end
+
+    end
+  end
+end