elastictastic 0.5.0 → 0.10.2

data/LICENSE

@@ -1,5 +1,5 @@
 The MIT License (MIT)
-Copyright (c) 2011 Mat Brown
+Copyright (c) 2011-2012 Brewster Inc., Mat Brown
 
 Permission is hereby granted, free of charge, to any person obtaining a copy of
 this software and associated documentation files (the "Software"), to deal in

data/README.md

@@ -35,6 +35,8 @@ the `field` class macro:
 
 ```ruby
 class Post
+  include Elastictastic::Document
+
   field :title
 end
 ```
@@ -82,7 +84,27 @@ field :title,
   }
 ```
 
-### Embedded Objects ###
+### Document Boost ###
+
+Defining a
+[document boost](http://www.elasticsearch.org/guide/reference/mapping/boost-field.html)
+will increase or decrease a document's score in search results based on the
+value of a field in the document. A boost of 1.0 is neutral. To define a boost
+field, use the `boost` class macro:
+
+```ruby
+class Post
+  include Elastictastic::Document
+
+  field :score, :type => 'integer'
+  boost :score
+end
+```
+
+By default, if the boost field is empty, a score of 1.0 will be applied. You can
+override this by passing a `'null_value'` option into the boost method.
+
+### Embedded objects ###
 
 ElasticSearch supports deep nesting of properties by way of
 [object fields](http://www.elasticsearch.org/guide/reference/mapping/object-type.html).
@@ -98,13 +120,13 @@ class Post
 end
 ```
 
-The class that's embedded should include the `Elastictastic::Resource` mixin,
+The class that's embedded should include the `Elastictastic::NestedDocument` mixin,
 which exposes the same configuration DSL as `Elastictastic::Document` but does
 not give the class the functionality of a top-level persistent object:
 
 ```ruby
 class Author
-  include Elastictastic::Resource
+  include Elastictastic::NestedDocument
 
   field :name
   field :email, :index => 'not_analyzed'
@@ -182,13 +204,21 @@ explicit support for this at the moment, although you can use e.g.
 `Post.mapping` to retrieve the mapping structure which you can then merge into
 your template.
 
-### Reserved Attributes ###
+### Reserved attributes ###
 
 All `Elastictastic::Document` models have an `id` and an `index` field, which
 combine to define the full resource locator for the document in ElasticSearch.
 You should not define fields or methods with these names. You may, however, set
 the id explicitly on new (not yet saved) model instances.
 
+### ActiveModel ###
+
+Elastictastic documents include all the usual ActiveModel functionality:
+validations, lifecycle hooks, observers, dirty-tracking, mass-assignment
+security, and the like. If you would like to squeeze a bit of extra performance
+out of the library at the cost of convenience, you can include the
+`Elastictastic::BasicDocument` module instead of `Elastictastic::Document`.
+
 ## Persistence ##
 
 Elastictastic models are persisted the usual way, namely by calling `save`:
@@ -199,7 +229,7 @@ post.title = 'You know, for search.'
 post.save
 ```
 
-To retrieve a document from the data store, use `get`:
+To retrieve a document from the data store, use `find`:
 
 ```ruby
 Post.find('123')
@@ -241,15 +271,136 @@ the `in_index` class method:
 
 ```ruby
 new_post = Post.in_index('my_special_index').new # create in an index
-post = Post.in_index('my_special_index').get('123') # retrieve from an index
+post = Post.in_index('my_special_index').find('123') # retrieve from an index
 ```
 
 To retrieve documents from multiple indices at the same time, pass a hash into
-`get` where the keys are index names and the values are the IDs you wish to
+`find` where the keys are index names and the values are the IDs you wish to
 retrieve from that index:
 
 ```ruby
-Post.get('default' => ['123', '456'], 'my_special_index' => '789')
+Post.find('default' => ['123', '456'], 'my_special_index' => '789')
+```
+
+### Bulk operations ###
+
+If you are writing a large amount of data to ElasticSearch in a single process,
+use of the
+[bulk API](http://www.elasticsearch.org/guide/reference/api/bulk.html)
+is encouraged. To perform bulk operations using Elastictastic, simply wrap your
+operations in a `bulk` block:
+
+```ruby
+Elastictastic.bulk do
+  params[:posts].each do |post_params|
+    post = Post.new(post_params)
+    post.save
+  end
+end
+```
+
+All create, update, and destroy operations inside the block will be executed in
+a single bulk request when the block completes. If you are performing an
+indefinite number of operations in a bulk block, you can pass an `:auto_flush`
+option to flush the bulk buffer after the specified number of operations:
+
+```ruby
+Elastictastic.bulk(:auto_flush => 100) do
+  150.times { Post.new.save! }
+end
+```
+
+The above will perform two bulk requests: the first after the first 100
+operations, and the second when the block completes.
+
+Note that the nature of bulk writes means that any operation inside a bulk block
+is essentially asynchronous: instances are not created, updated, or destroyed
+immediately upon calling `save` or `destroy`, but rather when the bulk block
+exits. You may pass a block to `save` and `destroy` to provide a callback for
+when the instance is actually persisted and its local state updated. Let's say,
+for instance, we wish to expand the example above to pass the IDs of the newly
+created posts to our view layer:
+
+```ruby
+@ids = []
+Elastictastic.bulk do
+  params[:posts].each do |post_params|
+    post = Post.new(post_params)
+    post.save do |e|
+      @ids << post.id
+    end
+  end
+end
+```
+
+If the save was not successful (due to a duplicate ID or a version mismatch,
+for instance), the `e` argument to the block will be passed an exception object;
+if the save was successful, the argument will be nil.
+
+### Concurrent document creation ###
+
+When Elastictastic creates a document with an application-defined ID, it uses
+the `_create` verb in ElasticSearch, ensuring that a document with that ID does
+not already exist. If the document does already exist, an
+`Elastictastic::ServerError::DocumentAlreadyExistsEngineException` will be
+raised. In the case where multiple processes may attempt concurrent creation of
+the same document, you can gracefully handle concurrent creation using the
+`::create_or_update` class method on your document class. This will first
+attempt to create the document; if a document with that ID already exists, it
+will then load the document and modify it using the block passed:
+
+```ruby
+Post.create_or_update('1') do |post|
+	post.title = 'My Post'
+end
+```
+
+In the above case, Elastictastic will first attempt to create a new post with ID
+"1" and title "My Post". If a Post with that ID already exists, it will load it,
+set its title to "My Post", and save it. The update uses the `::update` method
+(see next section) to ensure that concurrent modification doesn't cause data to
+be lost.
+
+### Optimistic locking ###
+
+Elastictastic provides optimistic locking via ElasticSearch's built-in
+[document versioning](http://www.elasticsearch.org/guide/reference/api/index_.html).
+When a document is retrieved from persistence, it carries a version, which is a
+number that increments from 1 on each update. When Elastictastic models are
+updated, the document version that it carried when it was loaded is passed into
+the update operation; if this version does not match ElasticSearch's current
+version for that document, it indicates that another process has modified the
+document concurrently, and an
+`Elastictastic::ServerError::VersionConflictEngineException` is raised. This
+prevents data loss through concurrent conflicting updates.
+
+The easiest way to guard against concurrent modification is to use the
+`::update` class method to make changes to existing documents. Consider the
+following example:
+
+```ruby
+Post.update('12345') do |post|
+  post.title = 'New Title'
+end
+```
+
+In the above, the Post with ID '12345' is loaded from ElasticSearch and yielded
+to the block. When the block completes, the instance is saved back to
+ElasticSearch. If this save results in a version conflict, a new instance is
+loaded from ElasticSearch and the block is run again. The process repeats until
+a successful update.
+
+This method will work inside a bulk operation, but note that if the first update
+generates a version conflict, additional updates will occur in discrete
+requests, not as part of any bulk operation.
+
+If you wish to safely update documents retrieved from a search scope
+(see below), use the `update_each` method:
+
+```ruby
+Post.query { constant_score { filter { term(:blog_id => 1) }}}.update_each do |post|
+  post.title = post.title.upcase
+end
 ```
 
 ## Search ##
@@ -263,7 +414,7 @@ or Mongoid:
 
 ```ruby
 Post.query(:query_string => { :query => 'pizza' }).facets(:cuisine => { :term => { :field => :tags }}).from(10).size(10)
-# Generates { :query => { :query_string => { :query => 'pizza' }}, :facets => { :cuisine => { :term => { :field => :tags }}}, :from => 10, :size => 10 }
+# Generates {"query": {"query_string": {"query": "pizza"}}, "facets": {"cuisine": {"term": {"field": "tags" }}}, "from": 10, "size": 10}
 ```
 
 Elastictastic also has an alternate block-based query builder, if you prefer:
@@ -307,7 +458,7 @@ Post.highlight { fields(:title => {}) }.find_each do |post, hit|
 end
 ```
 
-Search scope also expose a #find_in_batches method, which also yields the raw
+Search scopes also expose a `#find_in_batches` method, which also yields the raw
 hit. The following code gives the same result as the previous example:
 
 ```ruby

data/lib/elastictastic/adapter.rb

@@ -0,0 +1,84 @@
+require 'elastictastic/transport_methods'
+
+module Elastictastic
+
+  class Adapter
+
+    include TransportMethods
+
+    Response = Struct.new(:status, :headers, :body)
+
+    def self.[](str)
+      case str
+      when nil then NetHttpAdapter
+      when /^[a-z_]+$/ then Elastictastic.const_get("#{str.to_s.classify}Adapter")
+      else str.constantize
+      end
+    end
+
+    def initialize(host, options = {})
+      @host = host
+      @request_timeout = options[:request_timeout]
+      @connect_timeout = options[:connect_timeout]
+    end
+
+  end
+
+  class NetHttpAdapter < Adapter
+
+    def initialize(host, options = {})
+      super
+      uri = URI.parse(host)
+      @connection = Net::HTTP.new(uri.host, uri.port)
+      @connection.read_timeout = @request_timeout
+    end
+
+    def request(method, path, body = nil)
+      response =
+        case method
+        when :head then @connection.head(path)
+        when :get then @connection.get(path)
+        when :post then @connection.post(path, body.to_s)
+        when :put then @connection.put(path, body.to_s)
+        when :delete then @connection.delete(path)
+        else raise ArgumentError, "Unsupported method #{method.inspect}"
+        end
+      Response.new(response.code.to_i, response.to_hash, response.body)
+    rescue Errno::ECONNREFUSED, Timeout::Error, SocketError => e
+      raise ConnectionFailed, e
+    end
+
+  end
+
+  class ExconAdapter < Adapter
+
+    def request(method, path, body = nil)
+      response = connection.request(
+        :body => body, :method => method, :path => path
+      )
+      Response.new(response.status, response.headers, response.body)
+    rescue Excon::Errors::Error => e
+      connection.reset
+      raise ConnectionFailed, e
+    end
+
+    private
+
+    def connection
+      @connection ||= Excon.new(@host, connection_params)
+    end
+
+    def connection_params
+      @connection_params ||= {}.tap do |params|
+        if @request_timeout
+          params[:read_timeout] = params[:write_timeout] = @request_timeout
+        end
+        if @connect_timeout
+          params[:connect_timeout] = @connect_timeout
+        end
+      end
+    end
+
+  end
+
+end

data/lib/elastictastic/association.rb

@@ -1,4 +1,10 @@
 module Elastictastic
+  #
+  # Container for information about generic Elastictastic associations --
+  # this might be an embed association or a parent/child association.
+  #
+  # @api private
+  #
   class Association
     attr_reader :name, :options
 

data/lib/elastictastic/basic_document.rb

@@ -0,0 +1,213 @@
+module Elastictastic
+  #
+  # The top-level module mixed in to classes which will be mapped as
+  # ElasticSearch documents. Note that most people will want to use the Document
+  # mixin, which extends BasicDocument with ActiveModel functionality such as
+  # validations, lifecycle hooks, observers, mass-assignment security, etc. The
+  # BasicDocument module is exposed directly for those who wish to avoid the
+  # performance penalty associated with ActiveModel functionality, or those who
+  # wish to only mix in the ActiveModel modules they need.
+  #
+  # Most of the functionality for BasicDocument is provided by submodules; see
+  # below.
+  #
+  # @see Document
+  # @see Scoped
+  # @see Properties
+  # @see Persistence
+  # @see OptimisticLocking
+  # @see ParentChild
+  #
+  module BasicDocument
+    extend ActiveSupport::Concern
+
+    included do
+      extend Scoped
+      include Properties
+      include Persistence
+      include OptimisticLocking
+      include ParentChild
+
+      extend ActiveModel::Naming
+      include ActiveModel::Conversion
+      include ActiveModel::Serializers::JSON
+      include ActiveModel::Serializers::Xml
+
+      self.include_root_in_json = false
+    end
+
+    module ClassMethods
+      #
+      # Retrieve one or more documents by ID.
+      #
+      # @param (see Elastictastic::Scope#find)
+      # @overload (see Elastictastic::Scope#find)
+      #
+
+      #
+      # @method destroy_all
+      #
+      # Destroy all instances of this class in the default index
+      #
+
+      #
+      # @method sync_mapping
+      #
+      # Push the mapping defined in this class to ElasticSearch. Be sure to do
+      # this before saving instances of your class, or after making changes to
+      # the class's mapping (e.g. adding fields)
+      #
+
+      #
+      # @method find_each(batch_options = {}) {|document, hit| ... }
+      #
+      # Iterate over all documents in the default index, retrieving documents
+      # in batches using a cursor, but yielding them one by one.
+      #
+      # @param (see Elastictastic::Scope#find_each)
+      # @option (see Elastictastic::Scope#find_each)
+      # @yield (see Elastictastic::Scope#find_each)
+      # @yieldparam (see Elastictastic::Scope#find_each)
+      # @return (see Elastictastic::Scope#find_each)
+      #
+
+      #
+      # @method find_in_batches(batch_options = {}) {|batch| ... }
+      #
+      # Retrieve all documents in the default index, yielding them in batches.
+      #
+      # @param (see Elastictastic::Scope#find_in_batches)
+      # @option (see Elastictastic::Scope#find_in_batches)
+      # @yield (see Elastictastic::Scope#find_in_batches)
+      # @yieldparam (see Elastictastic::Scope#find_in_batches)
+      # @return (see Elastictastic::Scope#find_in_batches)
+      #
+
+      #
+      # @method first
+      #
+      # @return [Document] The "first" document in the index ("first" is
+      #   undefined).
+      #
+
+      #
+      # @method count
+      #
+      # @return [Fixnum] The number of documents of this type in the default index.
+      #
+
+      #
+      # @method empty?
+      #
+      # @return [TrueClass,FalseClass] True if there are no documents of this
+      #   type in the default index.
+      #
+
+      #
+      # @method any?
+      #
+      # @return [TrueClass,FalseClass] True if there are documents of this type
+      #   in the default index.
+      #
+
+      delegate :find, :destroy_all, :sync_mapping, :inspect, :find_each,
+               :find_in_batches, :first, :count, :empty?, :any?, :all,
+               :query, :filter, :from, :size, :sort, :highlight, :fields,
+               :script_fields, :preference, :facets, :routing,
+               :to => :current_scope
+
+      def mapping
+        mapping_for_type = { 'properties' => properties }
+        mapping_for_type['_boost'] = @_boost if @_boost
+        if @_routing_field
+          mapping_for_type['_routing'] = {
+            'path' => @_routing_field.to_s,
+            'required' => @_routing_required
+          }
+        end
+        { type => mapping_for_type }
+      end
+
+      def type
+        name.underscore
+      end
+
+      def in_index(name_or_index)
+        Scope.new(Elastictastic::Index(name_or_index), self)
+      end
+
+      def scoped(params)
+        current_scope.scoped(params)
+      end
+
+      private
+
+      def default_scope
+        in_index(Index.default)
+      end
+    end
+
+    attr_reader :id
+    attr_accessor :version
+
+    def initialize(attributes = {})
+      self.class.current_scope.initialize_instance(self)
+    end
+
+    def reload
+      params = {}
+      params['routing'] = @_parent_id if @_parent_id
+      self.elasticsearch_hit =
+        Elastictastic.client.get(index, self.class.type, id, params)
+    end
+
+    def elasticsearch_hit=(hit) #:nodoc:
+      @id = hit['_id']
+      @index = Index.new(hit['_index'])
+      @version = hit['_version']
+      persisted!
+
+      doc = {}
+      doc.merge!(hit['_source']) if hit['_source']
+      fields = hit['fields']
+      if fields
+        unflattened_fields =
+          Util.unflatten_hash(fields.reject { |k, v| v.nil? })
+        if unflattened_fields.has_key?('_source')
+          doc.merge!(unflattened_fields.delete('_source'))
+        end
+        doc.merge!(unflattened_fields)
+      end
+      self.elasticsearch_doc=(doc)
+    end
+
+    def id=(id)
+      assert_transient!
+      @id = id
+    end
+
+    def index
+      return @index if defined? @index
+      @index = Index.default
+    end
+
+    def ==(other)
+      index == other.index && id == other.id
+    end
+
+    def attributes
+      { :id => id, :index => index.name }
+    end
+
+    def inspect
+      inspected = "#<#{self.class.name} id: #{id}, index: #{index.name}"
+      attributes.each_pair do |attr, value|
+        inspected << ", #{attr}: #{value.inspect}"
+      end
+      embeds.each_pair do |attr, value|
+        inspected << ", #{attr}: #{value.inspect}"
+      end
+      inspected << ">"
+    end
+  end
+end

data/lib/elastictastic/bulk_persistence_strategy.rb

@@ -2,51 +2,90 @@ require 'stringio'
 
 module Elastictastic
   class BulkPersistenceStrategy
-    def initialize
-      @buffer = StringIO.new
-      @handlers = []
+    DEFAULT_HANDLER = proc { |e| raise(e) if e }
+    Operation = Struct.new(:id, :commands, :handler, :skip)
+
+    def initialize(options)
+      @operations = []
+      @operations_by_id = {}
+      @auto_flush = options.delete(:auto_flush)
     end
 
-    def create(instance, params = {})
+    def create(instance, params = {}, &block)
+      block ||= DEFAULT_HANDLER
       if instance.pending_save?
         raise Elastictastic::OperationNotAllowed,
           "Can't re-save transient document with pending save in bulk operation"
       end
       instance.pending_save!
       add(
+        instance.index,
+        instance.id,
         { 'create' => bulk_identifier(instance) },
         instance.elasticsearch_doc
       ) do |response|
-        instance.id = response['create']['_id']
-        instance.persisted!
+        if response['create']['error']
+          block.call(ServerError[response['create']['error']])
+        else
+          instance.id = response['create']['_id']
+          instance.version = response['create']['_version']
+          instance.persisted!
+          block.call
+        end
       end
     end
 
-    def update(instance)
+    def update(instance, &block)
+      block ||= DEFAULT_HANDLER
       instance.pending_save!
       add(
+        instance.index,
+        instance.id,
         { 'index' => bulk_identifier(instance) },
         instance.elasticsearch_doc
-      )
+      ) do |response|
+        if response['index']['error']
+          block.call(ServerError[response['index']['error']])
+        else
+          instance.version = response['index']['_version']
+          block.call
+        end
+      end
     end
 
-    def destroy(instance)
+    def destroy(instance, &block)
+      block ||= DEFAULT_HANDLER
       instance.pending_destroy!
-      add(:delete => bulk_identifier(instance)) do |response|
-        instance.transient!
+      add(instance.index, instance.id, :delete => bulk_identifier(instance)) do |response|
+        if response['delete']['error']
+          block.call(ServerError[response['delete']['error']])
+        else
+          instance.transient!
+          instance.version = response['delete']['_version']
+          block.call
+        end
       end
     end
 
     def flush
-      return if @buffer.length.zero?
+      return if @operations.empty?
 
       params = {}
       params[:refresh] = true if Elastictastic.config.auto_refresh
-      response = Elastictastic.client.bulk(@buffer.string, params)
+      io = StringIO.new
+      operations = @operations.reject { |operation| operation.skip }
+      @operations.clear
+
+      operations.each do |operation|
+        operation.commands.each do |command|
+          io.puts Elastictastic.json_encode(command)
+        end
+      end
+      response = Elastictastic.client.bulk(io.string, params)
 
       response['items'].each_with_index do |op_response, i|
-        handler = @handlers[i]
-        handler.call(op_response) if handler
+        operation = operations[i]
+        operation.handler.call(op_response) if operation.handler
       end
       response
     end
@@ -56,15 +95,21 @@ module Elastictastic
     def bulk_identifier(instance)
       identifier = { :_index => instance.index.name, :_type => instance.class.type }
       identifier['_id'] = instance.id if instance.id
+      identifier['_version'] = instance.version if instance.version
+      routing = instance.class.route(instance)
+      identifier['_routing'] = routing.to_s if routing
       identifier['parent'] = instance._parent_id if instance._parent_id
       identifier
     end
 
-    def add(*requests, &block)
-      requests.each do |request|
-        @buffer.puts(request.to_json)
+    def add(index, id, *commands, &block)
+      document_id = [index.name, id]
+      if id && @operations_by_id.key?(document_id)
+        @operations_by_id[document_id].skip = true
       end
-      @handlers << block
+      @operations << operation = Operation.new(id, commands, block)
+      @operations_by_id[document_id] = operation
+      flush if @auto_flush && @operations.length >= @auto_flush
     end
   end
 end