es-elasticity 0.3.4 → 0.3.5

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: b02eaa02e7e32429215a8c3156656dcb52dd2f18
-  data.tar.gz: ad74ef063faa4c5d03fb695c95c2b2463c454d69
+  metadata.gz: a99e45f5ef9e625931db6d140fa43abeebb0719e
+  data.tar.gz: 9b8542e68c9c00ce4109c5a67c227dae68804a03
 SHA512:
-  metadata.gz: a0aa8fe83e2d62c2503c5f925c760bd0ce32d9cf4f8388658edcf9de1bac39d9dd845237716c55bca22d22cd0007b331eb0d64779d2526c5c55cbde8e9c243c4
-  data.tar.gz: 4027fa989f33220a9da7b2a08c2cf287cfa117e5a75db6df9571df7719237684d42931bc97fbf60ad840e2dc4005be9b5e3c91803b5e3b5beafbefb2285c2564
+  metadata.gz: ac65a4a90720bb4aa1887dd1f47d33e27ad623a9dfb1f0378d3578608ede383d58b85da0d1a7f5a0eaf36fa803ce52de16a98eca665efded0ab9d61bf69de289
+  data.tar.gz: f75cc2f3ae6c90ed308f9ead04fe8eadde0c3ec052c65cce87a56708a483966285ac0fd0c993595c487ce170b7ddba51dbbe94d0102dd9fa96825466af44d062

data/README.md

@@ -2,16 +2,18 @@
 
 [![Build Status](https://travis-ci.org/doximity/es-elasticity.svg)](https://travis-ci.org/doximity/es-elasticity) [![Test Coverage](https://codeclimate.com/github/doximity/es-elasticity/badges/coverage.svg)](https://codeclimate.com/github/doximity/es-elasticity) [![Code Climate](https://codeclimate.com/github/doximity/es-elasticity/badges/gpa.svg)](https://codeclimate.com/github/doximity/es-elasticity) [![Dependency Status](https://gemnasium.com/doximity/es-elasticity.svg)](https://gemnasium.com/doximity/es-elasticity)
 
-Elasticity provides a higher level abstraction on top of [elasticsearch-ruby](https://github.com/elasticsearch/elasticsearch-ruby) gem.
+Elasticity is a model oriented approach to Elasticsearch. In simple words, a Document is represented by it's own class, similar to what ActiveRecord does for database rows.
 
-Mainly, it provides a model-oriented approach to Elasticsearch, similar to what [ActiveRecord](https://github.com/rails/rails/tree/master/activerecord) provides to relational databases. It leverages [ActiveModel](https://github.com/rails/rails/tree/master/activemodel) to provide a familiar format for Rails developers.
+In Elasticsearch terminology, a document is an entity stored in Elasticsearch and associated to an index. Whenever a search is performed, a collection of documents is returned.
+
+Elasticity maps those documents into objects, providing a rich object representation of a document.
 
 ## Installation
 
 Add this line to your application's Gemfile:
 
 ```ruby
-gem 'elasticity'
+gem 'es-elasticity', require "elasticity"
 ```
 
 And then execute:
@@ -20,55 +22,74 @@ And then execute:
 
 Or install it yourself as:
 
-    $ gem install elasticity
+    $ gem install es-elasticity
+
+## Usage
 
-## Usage overview
+### Document model definition
 
-The first thing to do, is setup a model class for a index and document type that inherits from `Elasticity::Document`.
+The first thing to do, is setup a model representing your documents. The class level represents the index, while the instance level represents each Document stored in the index. This is similar to how ActiveRecord maps tables vs rows.
 
 ```ruby
 class Search::User < Elasticity::Document
-  # All models automatically have the id attribute but you need to define the others.
-  attr_accessor :name, :birthdate
+  configure do |c|
+    # Defines how the index will be named, the final name
+    # will depend on the stragy being used.
+    c.index_base_name = "users"
+
+    # Defines the document type that this class represents.
+    c.document_type = "user"
+
+    # Select which strategy should be used. AliasIndex uses two aliases
+    # in order to support hot remapping of indexes. This is the recommended
+    # strategy.
+    c.strategy = Elasticity::Strategies::AliasIndex
 
-  # Define the index mapping for the index and document type this model represents.
-  self.mappings = {
-    properties: {
-      name: { type: "string" },
-      birthdate: { type: "date" },
+    # Defines the mapping for this index/document_type.
+    c.mapping  = {
+      properties: {
+        name: { type: "string" },
+        birthdate: { type: "date" },
+      }
     }
-  }
+  end
 
   # Defines a search method.
   def self.adults
     date = Date.today - 21.years
 
-    # This is the query that will be submited to ES, same format ES would expect,
-    # translated to a Ruby hash.
+    # This is the query that will be submited to ES, same format ES would 
+    # expect, translated to a Ruby hash.
     body = {
       filter: {
         { range: { birthdate: { gte: date.iso8601 }}},
       },
     }
 
-    # Creates a search object from the body and return it. The returned object is a
-    # lazy evaluated search that behaves like a collection, being automatically
-    # triggered when data is iterated over.
+    # Creates a search object from the body and return it. 
+    # The returned object # is a lazy evaluated search that behaves like a collection, being 
+    # automatically triggered when data is iterated over.
     self.search(body)
   end
 
+  # All models automatically have the id attribute but you need to define the 
+  # other accessors so that they can be set and get properly.
+  attr_accessor :name, :birthdate
+
   # to_document is the only required method that needs to be implemented so an
   # instance of this model can be indexed.
   def to_document
     {
       name: self.name,
-      birthdate: self.birthdate.iso8601,
+      birthdate: self.birthdate.iso8601
     }
   end
 end
 ```
 
-Then instances of that model can be indexed pretty easily by just calling the `update` method.
+### Indexing
+
+An instance of the model is an in-memory representation of a Document. The document can be stored on the index by calling the `update` method.
 
 ```ruby
 # Creates a new document on the index
@@ -80,7 +101,23 @@ u.name = "Jonh Jon"
 u.update
 ```
 
-Getting the results of a search is also pretty straightforward:
+If you need to index a collection of documents, you can use `bulk_index`:
+
+```ruby
+users = [
+  Search::User.new(id: 1, name: "John", birthdate: Date.civil(1985, 10, 31)),
+  Search::User.new(id: 2, name: "Mary", birthdate: Date.civil(1986, 9, 24)),
+]
+
+Search::User.bulk_index(users)
+```
+
+
+### Searching
+
+Class methods have access to the `search` method, which returns a lazy evaluated search. That means that the search will only be performed when the data is necessary, not when the `search` method is called.
+
+The search object implements `Enumerable`, so it can be treated as a collection:
 
 ```ruby
 # Get the search object, which is an instance of `Elasticity::DocumentSearchProxy`.
@@ -91,24 +128,153 @@ adults = User.adults
 adults.each do |user|
   # do something with user
 end
+```
+
+It also has some pretty interesting methods that affects the way the query is performed. Here is a list of available search types:
+
+```ruby
+# Returns an array of document instances, this is the default and what the 
+# enumerable methods will delegate to.
+adults.documents
 
-# Or you can also, map the results back to an ActiveRecord relation.
-# In this case, only the ids will be fetched.
-adults.active_recors(Database::User) # => Array of Database::User instances
+# Returns an array of hashes representing the documents.
+adults.document_hashes
+
+# Performs the search using scan&scroll. It returns a cursor that will lazily
+# fetch all the pages of the search. It can be iterated by batch/page or by 
+# document.
+cursor = adults.scan_documents
+cursor.each_batch { |batch| ... }
+cursor.each { |doc| ... }
+
+# Lastly, a search that maps back to an ActiveRecord::Relation.
+adults = adults.active_record(User)
 ```
 
-## Design Goals
+For more information about the `active_record` method, read [ActiveRecord integration](#activerecord-integration).
+
+### Strategies and Hot-remapping
+
+Strategies define how index creation and index operation happens on the lower level. Basically it define the structure that backs the document model. Currently, there are two strategies available: single-index and alias-index.
+
+The single-index strategy is the most straightforward one. It causes one index to be created and any operation will be performed directly on that index. It's very simple but it has the downside of being a lot harder to update existing mapping since you'll have to drop the index and recreate from scratch.
+
+The alias-index strategy is a bit more complex but it allows for seameless hot remapping. It works by creating an index and two aliases pointing to that index. Any operation is performed on the aliases rather than the index, which allows hot swapping due atomic aliases updates.
+
+Here is what it looks like:
+
+```
+|¯¯¯¯¯¯¯¯¯¯¯¯¯|
+|  MainAlias  |---------|
+|_____________|         |------------> |¯¯¯¯¯¯¯¯¯¯¯¯¯|
+                                       |    Index    |
+|¯¯¯¯¯¯¯¯¯¯¯¯¯|         |------------> |_____________|
+| UpdateAlias |---------|
+|_____________| 
+```
+
+Everytime a search operation is performed, it is performed against the main alias; when an update operation is performed, it is performed against the update alias; and, when a delete operation is performed, it is performed against the indexes pointed by both aliases.
+
+When the mapping needs to change, a hot remapping can be performed by doing the following:
 
-- Provide model specific for Elasticsearch documents instead of an ActiveRecord mixin;
-- proper separation of concerns and de-coupling;
-- lazy search evaluation and easy composition of multi-searches;
-- easy of debug;
-- higher level API that resembles Elasticsearch API;
+1. Create a new index with the new mapping;
+2. change the update alias to point to the new index, and change main alias to point to both indexes; at this point it will look something like this:
+
+  ```
+  |¯¯¯¯¯¯¯¯¯¯¯¯¯|----------------------> |¯¯¯¯¯¯¯¯¯¯¯¯¯|
+  |  MainAlias  |                        |  Old Index  |
+  |_____________|----------|             |_____________|
+                           |                           
+  |¯¯¯¯¯¯¯¯¯¯¯¯¯|          |-----------> |¯¯¯¯¯¯¯¯¯¯¯¯¯|
+  | UpdateAlias |----------------------> |  New Index  |
+  |_____________|                        |_____________|
+  ```
+
+3. iterate over all documents on the old index, copying them to the new index;
+4. change aliases to point only to the new index;
+5. delete the old index.
+
+This is a simplified version, there are other things that happen to ensure consistency and avoid race conditions. For full understanding see `Elasticity::Strategies::AliasIndex#remap`.
+
+### ActiveRecord integration
+
+ActiveRecord integration is mainly a set of conventions rather than implementation, with the exception of one method that allows mapping documents back to a relation. Here is the list of conventions:
+
+* have a class method on the document called `from_active_record` that creates a document object from the active record object;
+* have a class method on the Document for rebuilding the index from the records;
+* have an `after_save` and an `after_destroy` callbacks on the ActiveRecord model;
+
+For example:
+
+  ```ruby
+  class User < ActiveRecord::Base
+    after_save    :update_index_document
+    after_destroy :delete_index_document
+
+    def update_index_document
+      Search::User.from_active_record(self).update
+    end
+
+    def remove_index_document
+      Search::User.delete(self.id)
+    end
+  end
+
+  class Search::User < Elasticity::Document
+    # ... configuration
+
+    def self.from_active_record(ar)
+      new(name: ar.name, birthdate: ar.birthdate)
+    end
+
+    def self.rebuild_index
+      self.recreate_index
+
+      User.find_in_batches do |batch|
+        documents = batch.map { |record| from_active_record(record) }
+        self.bulk_index(documents)
+      end
+    end
+  end
+  ```
+
+This makes the code very clear in intent, easier to see when and how things happen and under the developer control, keeping both parts very decoupled.
+
+The only ActiveRecord specific utility this library have is a way to lazily map a Elasticsearch search to an ActiveRecord relation.
+
+To extend on the previous example, imagine the `Search::User` class also have the following simple search method.
+
+```ruby
+def self.adults
+  date = Date.today - 21.years
+  
+  body = {
+    filter: {
+      { range: { birthdate: { gte: date.iso8601 }}},
+    },
+  }
+
+  self.search(body)
+end
+```
+
+Because the return of that method is a lazy-evaluated search, it allows specific search strategies to be used, one of them being ActiveRecord specific:
+
+```ruby
+adults = Search::User.adults.active_record(User)
+adults.class # => ActiveRecord::Relation
+adults.all   # => [#<User: id: 1, name: "John", birthdate: 1985-10-31>, ...]
+```
+
+Note that the method takes a relation and not a class, so the following is also possible:
+
+```ruby
+Search::User.adults.active_record(User.where(active: true))
+```
 
 ## Roadmap
 
-- [x] Index aliasing support
-  - [x] Index hot swapping support
+- [ ] Make Elasticity::Strategies::AliasIndex the default
 - [ ] Use mapping instead of mappings, we wanna be consistent to ES not to elasticsearch-ruby
 - [ ] Define from_active_record interface
 - [ ] Write more detailed documentation section for:

data/lib/elasticity/search.rb

@@ -91,6 +91,10 @@ module Elasticity
         response["hits"]["total"]
       end
 
+      def aggregations
+        response["aggregations"] ||= {}
+      end
+
       def suggestions
         response["hits"]["suggest"] ||= {}
       end

data/lib/elasticity/version.rb

@@ -1,3 +1,3 @@
 module Elasticity
-  VERSION = "0.3.4"
+  VERSION = "0.3.5"
 end

data/spec/units/search_spec.rb

@@ -13,6 +13,30 @@ RSpec.describe "Search" do
     ]}}
   end
 
+  let :aggregations do
+    {
+      "logins_count" => { "value" => 1495 },
+      "gender" => {
+          "buckets" => [
+              {
+                  "doc_count" => 100,
+                  "key" => "M"
+              },
+              {
+                  "doc_count" => 100,
+                  "key" => "F"
+              }
+          ],
+          "doc_count_error_upper_bound" => 0,
+          "sum_other_doc_count" => 0
+      }
+    }
+  end
+
+  let :full_response_with_aggregations do
+    full_response.merge("aggregations" => aggregations)
+  end
+
   let :ids_response do
     { "hits" => { "total" => 2, "hits" => [
       { "_id" => 1 },
@@ -74,6 +98,13 @@ RSpec.describe "Search" do
       expect(Array(docs)).to eq expected
     end
 
+    it "searches and the index returns aggregations" do
+      expect(client).to receive(:search).with(index: index_name, type: document_type, body: body).and_return(full_response_with_aggregations)
+
+      docs = subject.documents(klass)
+      expect(docs.aggregations).to eq aggregations
+    end
+
     it "searches using scan&scroll" do
       expect(client).to receive(:search).with(index: index_name, type: document_type, body: body, search_type: "scan", size: 100, scroll: "1m").and_return(scan_response)
       expect(client).to receive(:scroll).with(scroll_id: "abc123", scroll: "1m").and_return(scroll_response)

metadata

@@ -1,167 +1,167 @@
 --- !ruby/object:Gem::Specification
 name: es-elasticity
 version: !ruby/object:Gem::Version
-  version: 0.3.4
+  version: 0.3.5
 platform: ruby
 authors:
 - Rodrigo Kochenburger
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2015-03-20 00:00:00.000000000 Z
+date: 2015-03-30 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: bundler
   requirement: !ruby/object:Gem::Requirement
     requirements:
-    - - ~>
+    - - "~>"
       - !ruby/object:Gem::Version
         version: '1.7'
   type: :development
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
-    - - ~>
+    - - "~>"
       - !ruby/object:Gem::Version
         version: '1.7'
 - !ruby/object:Gem::Dependency
   name: rake
   requirement: !ruby/object:Gem::Requirement
     requirements:
-    - - ~>
+    - - "~>"
       - !ruby/object:Gem::Version
         version: '10.0'
   type: :development
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
-    - - ~>
+    - - "~>"
       - !ruby/object:Gem::Version
         version: '10.0'
 - !ruby/object:Gem::Dependency
   name: rspec
   requirement: !ruby/object:Gem::Requirement
     requirements:
-    - - ~>
+    - - "~>"
       - !ruby/object:Gem::Version
         version: 3.1.0
   type: :development
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
-    - - ~>
+    - - "~>"
       - !ruby/object:Gem::Version
         version: 3.1.0
 - !ruby/object:Gem::Dependency
   name: simplecov
   requirement: !ruby/object:Gem::Requirement
     requirements:
-    - - ~>
+    - - "~>"
       - !ruby/object:Gem::Version
         version: 0.7.1
   type: :development
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
-    - - ~>
+    - - "~>"
       - !ruby/object:Gem::Version
         version: 0.7.1
 - !ruby/object:Gem::Dependency
   name: oj
   requirement: !ruby/object:Gem::Requirement
     requirements:
-    - - '>='
+    - - ">="
       - !ruby/object:Gem::Version
         version: '0'
   type: :development
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
-    - - '>='
+    - - ">="
       - !ruby/object:Gem::Version
         version: '0'
 - !ruby/object:Gem::Dependency
   name: pry
   requirement: !ruby/object:Gem::Requirement
     requirements:
-    - - '>='
+    - - ">="
       - !ruby/object:Gem::Version
         version: '0'
   type: :development
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
-    - - '>='
+    - - ">="
       - !ruby/object:Gem::Version
         version: '0'
 - !ruby/object:Gem::Dependency
   name: codeclimate-test-reporter
   requirement: !ruby/object:Gem::Requirement
     requirements:
-    - - '>='
+    - - ">="
       - !ruby/object:Gem::Version
         version: '0'
   type: :development
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
-    - - '>='
+    - - ">="
       - !ruby/object:Gem::Version
         version: '0'
 - !ruby/object:Gem::Dependency
   name: redis
   requirement: !ruby/object:Gem::Requirement
     requirements:
-    - - '>='
+    - - ">="
       - !ruby/object:Gem::Version
         version: '0'
   type: :development
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
-    - - '>='
+    - - ">="
       - !ruby/object:Gem::Version
         version: '0'
 - !ruby/object:Gem::Dependency
   name: activesupport
   requirement: !ruby/object:Gem::Requirement
     requirements:
-    - - ~>
+    - - "~>"
       - !ruby/object:Gem::Version
         version: '4.0'
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
-    - - ~>
+    - - "~>"
       - !ruby/object:Gem::Version
         version: '4.0'
 - !ruby/object:Gem::Dependency
   name: activemodel
   requirement: !ruby/object:Gem::Requirement
     requirements:
-    - - ~>
+    - - "~>"
       - !ruby/object:Gem::Version
         version: '4.0'
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
-    - - ~>
+    - - "~>"
       - !ruby/object:Gem::Version
         version: '4.0'
 - !ruby/object:Gem::Dependency
   name: elasticsearch
   requirement: !ruby/object:Gem::Requirement
     requirements:
-    - - ~>
+    - - "~>"
       - !ruby/object:Gem::Version
         version: 1.0.5
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
-    - - ~>
+    - - "~>"
       - !ruby/object:Gem::Version
         version: 1.0.5
 description: Elasticity provides a higher level abstraction on top of [elasticsearch-ruby](https://github.com/elasticsearch/elasticsearch-ruby)
@@ -172,10 +172,10 @@ executables: []
 extensions: []
 extra_rdoc_files: []
 files:
-- .gitignore
-- .rspec
-- .simplecov
-- .travis.yml
+- ".gitignore"
+- ".rspec"
+- ".simplecov"
+- ".travis.yml"
 - Gemfile
 - LICENSE.txt
 - README.md
@@ -212,17 +212,17 @@ require_paths:
 - lib
 required_ruby_version: !ruby/object:Gem::Requirement
   requirements:
-  - - '>='
+  - - ">="
     - !ruby/object:Gem::Version
       version: '0'
 required_rubygems_version: !ruby/object:Gem::Requirement
   requirements:
-  - - '>='
+  - - ">="
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
 rubyforge_project: 
-rubygems_version: 2.0.0
+rubygems_version: 2.2.2
 signing_key: 
 specification_version: 4
 summary: ActiveModel-based library for working with Elasticsearch
@@ -233,4 +233,3 @@ test_files:
 - spec/units/multi_search_spec.rb
 - spec/units/search_spec.rb
 - spec/units/strategies/single_index_spec.rb
-has_rdoc: