chewy 0.8.4 → 7.3.4

data/README.md

@@ -1,20 +1,15 @@
 [![Gem Version](https://badge.fury.io/rb/chewy.svg)](http://badge.fury.io/rb/chewy)
-[![Build Status](https://travis-ci.org/toptal/chewy.svg)](https://travis-ci.org/toptal/chewy)
+[![GitHub Actions](https://github.com/toptal/chewy/actions/workflows/ruby.yml/badge.svg)](https://github.com/toptal/chewy/actions/workflows/ruby.yml)
 [![Code Climate](https://codeclimate.com/github/toptal/chewy.svg)](https://codeclimate.com/github/toptal/chewy)
 [![Inline docs](http://inch-ci.org/github/toptal/chewy.svg?branch=master)](http://inch-ci.org/github/toptal/chewy)
 
-<p align="right">Sponsored by</p>
-<p align="right"><a href="http://www.toptal.com/"><img src="http://www.toptal.com/assets/public/blocks/logo/big.png" alt="Toptal" width="105" height="34"></a></p>
-
 # Chewy
 
-Chewy is an ODM and wrapper for [the official Elasticsearch client](https://github.com/elasticsearch/elasticsearch-ruby).
+Chewy is an ODM (Object Document Mapper), built on top of [the official Elasticsearch client](https://github.com/elastic/elasticsearch-ruby).
 
 ## Why Chewy?
 
-* Multi-model indices.
-
-  Index classes are independent from ORM/ODM models. Now, implementing e.g. cross-model autocomplete is much easier. You can just define the index and work with it in an object-oriented style. You can define several types for index - one per indexed model.
+In this section we'll cover why you might want to use Chewy instead of the official `elasticsearch-ruby` client gem.
 
 * Every index is observable by all the related models.
 
@@ -28,12 +23,11 @@ Chewy is an ODM and wrapper for [the official Elasticsearch client](https://gith
 
   Chewy has an ActiveRecord-style query DSL. It is chainable, mergeable and lazy, so you can produce queries in the most efficient way. It also has object-oriented query and filter builders.
 
-* Support for ActiveRecord, [Mongoid](https://github.com/mongoid/mongoid) and [Sequel](https://github.com/jeremyevans/sequel).
-
+* Support for ActiveRecord.
 
 ## Installation
 
-Add this line to your application's Gemfile:
+Add this line to your application's `Gemfile`:
 
     gem 'chewy'
 
@@ -45,19 +39,181 @@ Or install it yourself as:
 
     $ gem install chewy
 
-## Usage
+## Compatibility
 
-### Client settings
+### Ruby
+
+Chewy is compatible with MRI 2.6-3.0¹.
+
+> ¹ Ruby 3 is only supported with Rails 6.1
+
+### Elasticsearch compatibility matrix
+
+| Chewy version | Elasticsearch version              |
+| ------------- | ---------------------------------- |
+| 7.2.x         | 7.x                                |
+| 7.1.x         | 7.x                                |
+| 7.0.x         | 6.8, 7.x                           |
+| 6.0.0         | 5.x, 6.x                           |
+| 5.x           | 5.x, limited support for 1.x & 2.x |
+
+**Important:** Chewy doesn't follow SemVer, so you should always
+check the release notes before upgrading. The major version is linked to the
+newest supported Elasticsearch and the minor version bumps may include breaking changes.
+
+See our [migration guide](migration_guide.md) for detailed upgrade instructions between
+various Chewy versions.
+
+### Active Record
+
+5.2, 6.0, 6.1 Active Record versions are supported by all Chewy versions.
+
+## Getting Started
+
+Chewy provides functionality for Elasticsearch index handling, documents import mappings, index update strategies and chainable query DSL.
+
+### Minimal client setting
+
+Create `config/initializers/chewy.rb` with this line:
+
+```ruby
+Chewy.settings = {host: 'localhost:9250'}
+```
+
+And run `rails g chewy:install` to generate `chewy.yml`:
+
+```yaml
+# config/chewy.yml
+# separate environment configs
+test:
+  host: 'localhost:9250'
+  prefix: 'test'
+development:
+  host: 'localhost:9200'
+```
+
+### Elasticsearch
+
+Make sure you have Elasticsearch up and running. You can [install](https://www.elastic.co/guide/en/elasticsearch/reference/current/install-elasticsearch.html) it locally, but the easiest way is to use [Docker](https://www.docker.com/get-started):
+
+```shell
+$ docker run --rm --name elasticsearch -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" elasticsearch:7.11.1
+```
+
+### Index
+
+Create `app/chewy/users_index.rb` with User Index:
+
+```ruby
+class UsersIndex < Chewy::Index
+  settings analysis: {
+    analyzer: {
+      email: {
+        tokenizer: 'keyword',
+        filter: ['lowercase']
+      }
+    }
+  }
 
-There are two ways to configure the Chewy client: the `Chewy.settings` hash and `chewy.yml`
+  index_scope User
+  field :first_name
+  field :last_name
+  field :email, analyzer: 'email'
+end
+```
 
-You can create this file manually or run `rails g chewy:install`.
+### Model
+
+Add User model, table and migrate it:
+
+```shell
+$ bundle exec rails g model User first_name last_name email
+$ bundle exec rails db:migrate
+```
+
+Add `update_index` to app/models/user.rb:
+
+```ruby
+class User < ApplicationRecord
+  update_index('users') { self }
+end
+```
+
+### Example of data request
+
+1. Once a record is created (could be done via the Rails console), it creates User index too:
+
+```
+User.create(
+  first_name: "test1",
+  last_name: "test1",
+  email: 'test1@example.com',
+  # other fields
+)
+# UsersIndex Import (355.3ms) {:index=>1}
+# => #<User id: 1, first_name: "test1", last_name: "test1", email: "test1@example.com", # other fields>
+```
+
+2. A query could be exposed at a given `UsersController`:
+
+```ruby
+def search
+  @users = UsersIndex.query(query_string: { fields: [:first_name, :last_name, :email, ...], query: search_params[:query], default_operator: 'and' })
+  render json: @users.to_json, status: :ok
+end
+
+private
+
+def search_params
+  params.permit(:query, :page, :per)
+end
+```
+
+3. So a request against `http://localhost:3000/users/search?query=test1@example.com` issuing a response like:
+
+```json
+[
+  {
+    "attributes":{
+      "id":"1",
+      "first_name":"test1",
+      "last_name":"test1",
+      "email":"test1@example.com",
+      ...
+      "_score":0.9808291,
+      "_explanation":null
+    },
+    "_data":{
+      "_index":"users",
+      "_type":"_doc",
+      "_id":"1",
+      "_score":0.9808291,
+      "_source":{
+        "first_name":"test1",
+        "last_name":"test1",
+        "email":"test1@example.com",
+        ...
+      }
+    }
+  }
+]
+```
+
+## Usage and configuration
+
+### Client settings
+
+To configure the Chewy client you need to add `chewy.rb` file with `Chewy.settings` hash:
 
 ```ruby
 # config/initializers/chewy.rb
 Chewy.settings = {host: 'localhost:9250'} # do not use environments
 ```
 
+And add `chewy.yml` configuration file.
+
+You can create `chewy.yml` manually or run `rails g chewy:install` to generate it:
+
 ```yaml
 # config/chewy.yml
 # separate environment configs
@@ -83,7 +239,31 @@ Chewy.logger = Logger.new(STDOUT)
 
 See [config.rb](lib/chewy/config.rb) for more details.
 
-### Index definition
+#### AWS Elasticsearch
+
+If you would like to use AWS's Elasticsearch using an IAM user policy, you will need to sign your requests for the `es:*` action by injecting the appropriate headers passing a proc to `transport_options`.
+You'll need an additional gem for Faraday middleware: add `gem 'faraday_middleware-aws-sigv4'` to your Gemfile.
+
+```ruby
+require 'faraday_middleware/aws_sigv4'
+
+Chewy.settings = {
+  host: 'http://my-es-instance-on-aws.us-east-1.es.amazonaws.com:80',
+  port: 80, # 443 for https host
+  transport_options: {
+    headers: { content_type: 'application/json' },
+    proc: -> (f) do
+        f.request :aws_sigv4,
+                  service: 'es',
+                  region: 'us-east-1',
+                  access_key_id: ENV['AWS_ACCESS_KEY'],
+                  secret_access_key: ENV['AWS_SECRET_ACCESS_KEY']
+    end
+  }
+}
+```
+
+#### Index definition
 
 1. Create `/app/chewy/users_index.rb`
 
@@ -93,41 +273,38 @@ See [config.rb](lib/chewy/config.rb) for more details.
   end
   ```
 
-2. Add one or more types mapping
+2. Define index scope (you can omit this part if you don't need to specify a scope (i.e. use PORO objects for import) or options)
 
   ```ruby
   class UsersIndex < Chewy::Index
-    define_type User.active # or just model instead_of scope: define_type User
+    index_scope User.active # or just model instead_of scope: index_scope User
   end
   ```
 
-  Newly-defined index type class is accessible via `UsersIndex.user` or `UsersIndex::User`
-
-3. Add some type mappings
+3. Add some mappings
 
   ```ruby
   class UsersIndex < Chewy::Index
-    define_type User.active.includes(:country, :badges, :projects) do
-      field :first_name, :last_name # multiple fields without additional options
-      field :email, analyzer: 'email' # Elasticsearch-related options
-      field :country, value: ->(user) { user.country.name } # custom value proc
-      field :badges, value: ->(user) { user.badges.map(&:name) } # passing array values to index
-      field :projects do # the same block syntax for multi_field, if `:type` is specified
-        field :title
-        field :description # default data type is `string`
-        # additional top-level objects passed to value proc:
-        field :categories, value: ->(project, user) { project.categories.map(&:name) if user.active? }
-      end
-      field :rating, type: 'integer' # custom data type
-      field :created, type: 'date', include_in_all: false,
-        value: ->{ created_at } # value proc for source object context
+    index_scope User.active.includes(:country, :badges, :projects)
+    field :first_name, :last_name # multiple fields without additional options
+    field :email, analyzer: 'email' # Elasticsearch-related options
+    field :country, value: ->(user) { user.country.name } # custom value proc
+    field :badges, value: ->(user) { user.badges.map(&:name) } # passing array values to index
+    field :projects do # the same block syntax for multi_field, if `:type` is specified
+      field :title
+      field :description # default data type is `text`
+      # additional top-level objects passed to value proc:
+      field :categories, value: ->(project, user) { project.categories.map(&:name) if user.active? }
     end
+    field :rating, type: 'integer' # custom data type
+    field :created, type: 'date', include_in_all: false,
+      value: ->{ created_at } # value proc for source object context
   end
   ```
 
-  [See here for mapping definitions](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping.html).
+  [See here for mapping definitions](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html).
 
-4. Add some index- and type-related settings. Analyzer repositories might be used as well. See `Chewy::Index.settings` docs for details:
+4. Add some index-related settings. Analyzer repositories might be used as well. See `Chewy::Index.settings` docs for details:
 
   ```ruby
   class UsersIndex < Chewy::Index
@@ -140,69 +317,61 @@ See [config.rb](lib/chewy/config.rb) for more details.
       }
     }
 
-    define_type User.active.includes(:country, :badges, :projects) do
-      root date_detection: false do
-        template 'about_translations.*', type: 'string', analyzer: 'standard'
-
-        field :first_name, :last_name
-        field :email, analyzer: 'email'
-        field :country, value: ->(user) { user.country.name }
-        field :badges, value: ->(user) { user.badges.map(&:name) }
-        field :projects do
-          field :title
-          field :description
-        end
-        field :about_translations, type: 'object' # pass object type explicitly if necessary
-        field :rating, type: 'integer'
-        field :created, type: 'date', include_in_all: false,
-          value: ->{ created_at }
+    index_scope User.active.includes(:country, :badges, :projects)
+    root date_detection: false do
+      template 'about_translations.*', type: 'text', analyzer: 'standard'
+
+      field :first_name, :last_name
+      field :email, analyzer: 'email'
+      field :country, value: ->(user) { user.country.name }
+      field :badges, value: ->(user) { user.badges.map(&:name) }
+      field :projects do
+        field :title
+        field :description
       end
+      field :about_translations, type: 'object' # pass object type explicitly if necessary
+      field :rating, type: 'integer'
+      field :created, type: 'date', include_in_all: false,
+        value: ->{ created_at }
     end
   end
   ```
 
-  [See index settings here](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/indices-update-settings.html).
-  [See root object settings here](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-root-object-type.html).
+  [See index settings here](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-update-settings.html).
+  [See root object settings here](https://www.elastic.co/guide/en/elasticsearch/reference/current/dynamic-field-mapping.html).
 
-  See [mapping.rb](lib/chewy/type/mapping.rb) for more details.
+  See [mapping.rb](lib/chewy/index/mapping.rb) for more details.
 
 5. Add model-observing code
 
   ```ruby
   class User < ActiveRecord::Base
-    update_index('users#user') { self } # specifying index, type and back-reference
+    update_index('users') { self } # specifying index and back-reference
                                         # for updating after user save or destroy
   end
 
   class Country < ActiveRecord::Base
     has_many :users
 
-    update_index('users#user') { users } # return single object or collection
+    update_index('users') { users } # return single object or collection
   end
 
   class Project < ActiveRecord::Base
-    update_index('users#user') { user if user.active? } # you can return even `nil` from the back-reference
-  end
-
-  class Badge < ActiveRecord::Base
-    has_and_belongs_to_many :users
-
-    update_index('users') { users } # if index has only one type
-                                    # there is no need to specify updated type
+    update_index('users') { user if user.active? } # you can return even `nil` from the back-reference
   end
 
   class Book < ActiveRecord::Base
-    update_index(->(book) {"books#book_#{book.language}"}) { self } # dynamic index and type with proc.
-                                                                    # For book with language == "en"
-                                                                    # this code will generate `books#book_en`
+    update_index(->(book) {"books_#{book.language}"}) { self } # dynamic index name with proc.
+                                                               # For book with language == "en"
+                                                               # this code will generate `books_en`
   end
   ```
 
   Also, you can use the second argument for method name passing:
 
   ```ruby
-  update_index('users#user', :self)
-  update_index('users#user', :users)
+  update_index('users', :self)
+  update_index('users', :users)
   ```
 
   In the case of a belongs_to association you may need to update both associated objects, previous and current:
@@ -211,47 +380,28 @@ See [config.rb](lib/chewy/config.rb) for more details.
   class City < ActiveRecord::Base
     belongs_to :country
 
-    update_index('cities#city') { self }
-    update_index 'countries#country' do
-      # For the latest active_record changed values are
-      # already in `previous_changes` hash,
-      # but for mongoid you have to use `changes` hash
+    update_index('cities') { self }
+    update_index 'countries' do
       previous_changes['country_id'] || country
     end
   end
   ```
 
-  You can observe Sequel models in the same way as ActiveRecord:
+### Default import options
 
-  ```ruby
-  class User < Sequel::Model
-    update_index('users#user') { self }
-  end
-  ```
-
-  However, to make it work, you must load the chewy plugin into Sequel model:
-
-  ```ruby
-  Sequel::Model.plugin :chewy_observe  # for all models, or...
-  User.plugin :chewy_observe           # just for User
-  ```
-
-### Type default import options
-
-Every type has `default_import_options` configuration to specify, suddenly, default import options:
+Every index has `default_import_options` configuration to specify, suddenly, default import options:
 
 ```ruby
 class ProductsIndex < Chewy::Index
-  define_type Post.includes(:tags) do
-    default_import_options batch_size: 100, bulk_size: 10.megabytes, refresh: false
+  index_scope Post.includes(:tags)
+  default_import_options batch_size: 100, bulk_size: 10.megabytes, refresh: false
 
-    field :name
-    field :tags, value: -> { tags.map(&:name) }
-  end
+  field :name
+  field :tags, value: -> { tags.map(&:name) }
 end
 ```
 
-See [import.rb](lib/chewy/type/import.rb) for available options.
+See [import.rb](lib/chewy/index/import.rb) for available options.
 
 ### Multi (nested) and object field types
 
@@ -269,17 +419,17 @@ This will automatically set the type or root field to `object`. You may also spe
 To define a multi field you have to specify any type except for `object` or `nested` in the root field:
 
 ```ruby
-field :full_name, type: 'string', value: ->{ full_name.strip } do
+field :full_name, type: 'text', value: ->{ full_name.strip } do
   field :ordered, analyzer: 'ordered'
-  field :untouched, index: 'not_analyzed'
+  field :untouched, type: 'keyword'
 end
 ```
 
-The `value:` option for internal fields would no longer be effective.
+The `value:` option for internal fields will no longer be effective.
 
 ### Geo Point fields
 
-You can use [Elasticsearch's geo mapping](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-geo-point-type.html) with the `geo_point` field type, allowing you to query, filter and order by latitude and longitude. You can use the following hash format:
+You can use [Elasticsearch's geo mapping](https://www.elastic.co/guide/en/elasticsearch/reference/current/geo-point.html) with the `geo_point` field type, allowing you to query, filter and order by latitude and longitude. You can use the following hash format:
 
 ```ruby
 field :coordinates, type: 'geo_point', value: ->{ {lat: latitude, lon: longitude} }
@@ -296,20 +446,36 @@ end
 
 See the section on *Script fields* for details on calculating distance in a search.
 
+### Join fields
+
+You can use a [join field](https://www.elastic.co/guide/en/elasticsearch/reference/current/parent-join.html)
+to implement parent-child relationships between documents.
+It [replaces the old `parent_id` based parent-child mapping](https://www.elastic.co/guide/en/elasticsearch/reference/current/removal-of-types.html#parent-child-mapping-types)
+
+To use it, you need to pass `relations` and `join` (with `type` and `id`) options:
+```ruby
+field :hierarchy_link, type: :join, relations: {question: %i[answer comment], answer: :vote, vote: :subvote}, join: {type: :comment_type, id: :commented_id}
+```
+assuming you have `comment_type` and `commented_id` fields in your model.
+
+Note that when you reindex a parent, its children and grandchildren will be reindexed as well.
+This may require additional queries to the primary database and to elastisearch.
+
+Also note that the join field doesn't support crutches (it should be a field directly defined on the model).
+
 ### Crutches™ technology
 
 Assume you are defining your index like this (product has_many categories through product_categories):
 
 ```ruby
 class ProductsIndex < Chewy::Index
-  define_type Product.includes(:categories) do
-    field :name
-    field :category_names, value: ->(product) { product.categories.map(&:name) } # or shorter just -> { categories.map(&:name) }
-  end
+  index_scope Product.includes(:categories)
+  field :name
+  field :category_names, value: ->(product) { product.categories.map(&:name) } # or shorter just -> { categories.map(&:name) }
 end
 ```
 
-Then the Chewy reindexing flow would look like the following pseudo-code (even in Mongoid):
+Then the Chewy reindexing flow will look like the following pseudo-code:
 
 ```ruby
 Product.includes(:categories).find_in_batches(1000) do |batch|
@@ -321,30 +487,27 @@ Product.includes(:categories).find_in_batches(1000) do |batch|
 end
 ```
 
-But in Rails 4.1 and 4.2 you may face a problem with slow associations (take a look at https://github.com/rails/rails/pull/19423). Also, there might be really complicated cases when associations are not applicable.
-
-Then you can replace Rails associations with Chewy Crutches™ technology:
+If you meet complicated cases when associations are not applicable you can replace Rails associations with Chewy Crutches™ technology:
 
 ```ruby
 class ProductsIndex < Chewy::Index
-  define_type Product do
-    crutch :categories do |collection| # collection here is a current batch of products
-      # data is fetched with a lightweight query without objects initialization
-      data = ProductCategory.joins(:category).where(product_id: collection.map(&:id)).pluck(:product_id, 'categories.name')
-      # then we have to convert fetched data to appropriate format
-      # this will return our data in structure like:
-      # {123 => ['sweets', 'juices'], 456 => ['meat']}
-      data.each.with_object({}) { |(id, name), result| (result[id] ||= []).push(name) }
-    end
-
-    field :name
-    # simply use crutch-fetched data as a value:
-    field :category_names, value: ->(product, crutches) { crutches.categories[product.id] }
+  index_scope Product
+  crutch :categories do |collection| # collection here is a current batch of products
+    # data is fetched with a lightweight query without objects initialization
+    data = ProductCategory.joins(:category).where(product_id: collection.map(&:id)).pluck(:product_id, 'categories.name')
+    # then we have to convert fetched data to appropriate format
+    # this will return our data in structure like:
+    # {123 => ['sweets', 'juices'], 456 => ['meat']}
+    data.each.with_object({}) { |(id, name), result| (result[id] ||= []).push(name) }
   end
+
+  field :name
+  # simply use crutch-fetched data as a value:
+  field :category_names, value: ->(product, crutches) { crutches[:categories][product.id] }
 end
 ```
 
-An example flow would look like this:
+An example flow will look like this:
 
 ```ruby
 Product.includes(:categories).find_in_batches(1000) do |batch|
@@ -362,22 +525,21 @@ So Chewy Crutches™ technology is able to increase your indexing performance in
 
 ### Witchcraft™ technology
 
-One more experimental technology to increase import performance. As far as you know, chewy defines value proc for every imported field in mapping, so at the import time each of this procs is executed on imported object to extract result document to import. It would be great for performance to use one huge whole-document-returning proc instead. So basically the idea or Witchcraft™ technology is to compile a single document-returning proc from the type definition.
+One more experimental technology to increase import performance. As far as you know, chewy defines value proc for every imported field in mapping, so at the import time each of these procs is executed on imported object to extract result document to import. It would be great for performance to use one huge whole-document-returning proc instead. So basically the idea or Witchcraft™ technology is to compile a single document-returning proc from the index definition.
 
 ```ruby
-define_type Product do
-  witchcraft!
-
-  field :title
-  field :tags, value: -> { tags.map(&:name) }
-  field :categories do
-    field :name, value: -> (product, category) { category.name }
-    field :type, value: -> (product, category, crutch) { crutch.types[category.name] }
-  end
+index_scope Product
+witchcraft!
+
+field :title
+field :tags, value: -> { tags.map(&:name) }
+field :categories do
+  field :name, value: -> (product, category) { category.name }
+  field :type, value: -> (product, category, crutch) { crutch.types[category.name] }
 end
 ```
 
-The type definition above will be compiled to something close to:
+The index definition above will be compiled to something close to:
 
 ```ruby
 -> (object, crutches) do
@@ -395,19 +557,128 @@ end
 ```
 
 And don't even ask how is it possible, it is a witchcraft.
-Obviously not every type of definition might be compiled, so use reasonable formatting to make `method_source` be able to extract field value proc sources. Also value procs with splat arguments are not supported right now. However, it is quite possible that your type definition will be supported by Witchcraft™ technology out of the box in the most of the cases.
+Obviously not every type of definition might be compiled. There are some restrictions:
+
+1. Use reasonable formatting to make `method_source` be able to extract field value proc sources.
+2. Value procs with splat arguments are not supported right now.
+3. If you are generating fields dynamically use value proc with arguments, argumentless value procs are not supported yet:
+
+  ```ruby
+  [:first_name, :last_name].each do |name|
+    field name, value: -> (o) { o.send(name) }
+  end
+  ```
+
+However, it is quite possible that your index definition will be supported by Witchcraft™ technology out of the box in most of the cases.
+
+### Raw Import
+
+Another way to speed up import time is Raw Imports. This technology is only available in ActiveRecord adapter. Very often, ActiveRecord model instantiation is what consumes most of the CPU and RAM resources. Precious time is wasted on converting, say, timestamps from strings and then serializing them back to strings. Chewy can operate on raw hashes of data directly obtained from the database. All you need is to provide a way to convert that hash to a lightweight object that mimics the behaviour of the normal ActiveRecord object.
+
+```ruby
+class LightweightProduct
+  def initialize(attributes)
+    @attributes = attributes
+  end
+
+  # Depending on the database, `created_at` might
+  # be in different formats. In PostgreSQL, for example,
+  # you might see the following format:
+  #   "2016-03-22 16:23:22"
+  #
+  # Taking into account that Elastic expects something different,
+  # one might do something like the following, just to avoid
+  # unnecessary String -> DateTime -> String conversion.
+  #
+  #   "2016-03-22 16:23:22" -> "2016-03-22T16:23:22Z"
+  def created_at
+    @attributes['created_at'].tr(' ', 'T') << 'Z'
+  end
+end
+
+index_scope Product
+default_import_options raw_import: ->(hash) {
+  LightweightProduct.new(hash)
+}
 
-### Types access
+field :created_at, 'datetime'
+```
+
+Also, you can pass `:raw_import` option to the `import` method explicitly.
+
+### Index creation during import
+
+By default, when you perform import Chewy checks whether an index exists and creates it if it's absent.
+You can turn off this feature to decrease Elasticsearch hits count.
+To do so you need to set `skip_index_creation_on_import` parameter to `false` in your `config/chewy.yml`
+
+### Skip record fields during import
+
+You can use `ignore_blank: true` to skip fields that return `true` for the `.blank?` method:
+
+```ruby
+index_scope Country
+field :id
+field :cities, ignore_blank: true do
+  field :id
+  field :name
+  field :surname, ignore_blank: true
+  field :description
+end
+```
+
+#### Default values for different types
+
+By default `ignore_blank` is false on every type except `geo_point`.
+
+### Journaling
 
-You can access index-defined types with the following API:
+You can record all actions that were made to the separate journal index in ElasticSearch.
+When you create/update/destroy your documents, it will be saved in this special index.
+If you make something with a batch of documents (e.g. during index reset) it will be saved as a one record, including primary keys of each document that was affected.
+Common journal record looks like this:
+
+```json
+{
+  "action": "index",
+  "object_id": [1, 2, 3],
+  "index_name": "...",
+  "created_at": "<timestamp>"
+}
+```
+
+This feature is turned off by default.
+But you can turn it on by setting `journal` setting to `true` in `config/chewy.yml`.
+Also, you can specify journal index name. For example:
+
+```yaml
+# config/chewy.yml
+production:
+  journal: true
+  journal_name: my_super_journal
+```
+
+Also, you can provide this option while you're importing some index:
 
 ```ruby
-UsersIndex::User # => UsersIndex::User
-UsersIndex.type_hash['user'] # => UsersIndex::User
-UsersIndex.types # => [UsersIndex::User]
-UsersIndex.type_names # => ['user']
+CityIndex.import journal: true
 ```
 
+Or as a default import option for an index:
+
+```ruby
+class CityIndex
+  index_scope City
+  default_import_options journal: true
+end
+```
+
+You may be wondering why do you need it? The answer is simple: not to lose the data.
+
+Imagine that you reset your index in a zero-downtime manner (to separate index), and in the meantime somebody keeps updating the data frequently (to old index). So all these actions will be written to the journal index and you'll be able to apply them after index reset using the `Chewy::Journal` interface.
+
+When enabled, journal can grow to enormous size, consider setting up cron job that would clean it occasionally using [`chewy:journal:clean` rake task](#chewyjournal).
+
 ### Index manipulation
 
 ```ruby
@@ -420,24 +691,22 @@ UsersIndex.create! # use bang or non-bang methods
 UsersIndex.purge
 UsersIndex.purge! # deletes then creates index
 
-UsersIndex::User.import # import with 0 arguments process all the data specified in type definition
-                        # literally, User.active.includes(:country, :badges, :projects).find_in_batches
-UsersIndex::User.import User.where('rating > 100') # or import specified users scope
-UsersIndex::User.import User.where('rating > 100').to_a # or import specified users array
-UsersIndex::User.import [1, 2, 42] # pass even ids for import, it will be handled in the most effective way
+UsersIndex.import # import with 0 arguments process all the data specified in index_scope definition
+UsersIndex.import User.where('rating > 100') # or import specified users scope
+UsersIndex.import User.where('rating > 100').to_a # or import specified users array
+UsersIndex.import [1, 2, 42] # pass even ids for import, it will be handled in the most effective way
+UsersIndex.import User.where('rating > 100'), update_fields: [:email] # if update fields are specified - it will update their values only with the `update` bulk action
+UsersIndex.import! # raises an exception in case of any import errors
 
-UsersIndex.import # import every defined type
-UsersIndex.import user: User.where('rating > 100') # import only active users to `user` type.
-  # Other index types, if exists, will be imported with default scope from the type definition.
 UsersIndex.reset! # purges index and imports default data for all types
 ```
 
-If the passed user is `#destroyed?`, or satisfies a `delete_if` type option, or the specified id does not exist in the database, import will perform delete from index action for this object.
+If the passed user is `#destroyed?`, or satisfies a `delete_if` index_scope option, or the specified id does not exist in the database, import will perform delete from index action for this object.
 
 ```ruby
-define_type User, delete_if: :deleted_at
-define_type User, delete_if: -> { deleted_at }
-define_type User, delete_if: ->(user) { user.deleted_at }
+index_scope User, delete_if: :deleted_at
+index_scope User, delete_if: -> { deleted_at }
+index_scope User, delete_if: ->(user) { user.deleted_at }
 ```
 
 See [actions.rb](lib/chewy/index/actions.rb) for more details.
@@ -448,13 +717,12 @@ Assume you've got the following code:
 
 ```ruby
 class City < ActiveRecord::Base
-  update_index 'cities#city', :self
+  update_index 'cities', :self
 end
 
 class CitiesIndex < Chewy::Index
-  define_type City do
-    field :name
-  end
+  index_scope City
+  field :name
 end
 ```
 
@@ -474,26 +742,112 @@ end
 
 Using this strategy delays the index update request until the end of the block. Updated records are aggregated and the index update happens with the bulk API. So this strategy is highly optimized.
 
-#### `:resque`
+#### `:sidekiq`
 
-This does the same thing as `:atomic`, but asynchronously using resque. The default queue name is `chewy`. Patch `Chewy::Strategy::Resque::Worker` for index updates improving.
+This does the same thing as `:atomic`, but asynchronously using sidekiq. Patch `Chewy::Strategy::Sidekiq::Worker` for index updates improving.
 
 ```ruby
-Chewy.strategy(:resque) do
+Chewy.strategy(:sidekiq) do
   City.popular.map(&:do_some_update_action!)
 end
 ```
 
-#### `:sidekiq`
+The default queue name is `chewy`, you can customize it in settings: `sidekiq.queue_name`
+```
+Chewy.settings[:sidekiq] = {queue: :low}
+```
 
-This does the same thing as `:atomic`, but asynchronously using sidekiq. Patch `Chewy::Strategy::Sidekiq::Worker` for index updates improving.
+#### `:lazy_sidekiq`
+
+This does the same thing as `:sidekiq`, but with lazy evaluation. Beware it does not allow you to use any non-persistent record state for indices and conditions because record will be re-fetched from database asynchronously using sidekiq. However for destroying records strategy will fallback to `:sidekiq` because it's not possible to re-fetch deleted records from database.
+
+The purpose of this strategy is to improve the response time of the code that should update indexes, as it does not only defer actual ES calls to a background job but `update_index` callbacks evaluation (for created and updated objects) too. Similar to `:sidekiq`, index update is asynchronous so this strategy cannot be used when data and index synchronization is required.
 
 ```ruby
-Chewy.strategy(:sidekiq) do
+Chewy.strategy(:lazy_sidekiq) do
   City.popular.map(&:do_some_update_action!)
 end
 ```
 
+The default queue name is `chewy`, you can customize it in settings: `sidekiq.queue_name`
+```
+Chewy.settings[:sidekiq] = {queue: :low}
+```
+
+#### `:delayed_sidekiq`
+
+It accumulates ids of records to be reindexed during the latency window in redis and then does the reindexing of all accumulated records at once.
+The strategy is very useful in case of frequently mutated records.
+It supports `update_fields` option, so it will try to select just enough data from the DB
+
+There are three options that can be defined in the index:
+```ruby
+class CitiesIndex...
+  strategy_config delayed_sidekiq: {
+    latency: 3,
+    margin: 2,
+    ttl: 60 * 60 * 24,
+    reindex_wrapper: ->(&reindex) {
+      ActiveRecord::Base.connected_to(role: :reading) { reindex.call }
+    }
+    # latency - will prevent scheduling identical jobs
+    # margin - main purpose is to cover db replication lag by the margin
+    # ttl - a chunk expiration time (in seconds)
+    # reindex_wrapper - lambda that accepts block to wrap that reindex process AR connection block.
+  }
+
+  ...
+end
+```
+
+Also you can define defaults in the `initializers/chewy.rb`
+```ruby
+Chewy.settings = {
+  strategy_config: {
+    delayed_sidekiq: {
+      latency: 3,
+      margin: 2,
+      ttl: 60 * 60 * 24,
+      reindex_wrapper: ->(&reindex) {
+        ActiveRecord::Base.connected_to(role: :reading) { reindex.call }
+      }
+    }
+  }
+}
+
+```
+or in `config/chewy.yml`
+```ruby
+  strategy_config:
+    delayed_sidekiq:
+      latency: 3
+      margin: 2
+      ttl: <%= 60 * 60 * 24 %>
+      # reindex_wrapper setting is not possible here!!! use the initializer instead
+```
+
+You can use the strategy identically to other strategies
+```ruby
+Chewy.strategy(:delayed_sidekiq) do
+  City.popular.map(&:do_some_update_action!)
+end
+```
+
+The default queue name is `chewy`, you can customize it in settings: `sidekiq.queue_name`
+```
+Chewy.settings[:sidekiq] = {queue: :low}
+```
+
+Explicit call of the reindex using `:delayed_sidekiq strategy`
+```ruby
+CitiesIndex.import([1, 2, 3], strategy: :delayed_sidekiq)
+```
+
+Explicit call of the reindex using `:delayed_sidekiq` strategy with `:update_fields` support
+```ruby
+CitiesIndex.import([1, 2, 3], update_fields: [:name], strategy: :delayed_sidekiq)
+```
+
 #### `:active_job`
 
 This does the same thing as `:atomic`, but using ActiveJob. This will inherit the ActiveJob configuration settings including the `active_job.queue_adapter` setting for the environment. Patch `Chewy::Strategy::ActiveJob::Worker` for index updates improving.
@@ -504,6 +858,11 @@ Chewy.strategy(:active_job) do
 end
 ```
 
+The default queue name is `chewy`, you can customize it in settings: `active_job.queue_name`
+```
+Chewy.settings[:active_job] = {queue: :low}
+```
+
 #### `:urgent`
 
 The following strategy is convenient if you are going to update documents in your index one by one.
@@ -514,7 +873,7 @@ Chewy.strategy(:urgent) do
 end
 ```
 
-This code would perform `City.popular.count` requests for ES documents update.
+This code will perform `City.popular.count` requests for ES documents update.
 
 It is convenient for use in e.g. the Rails console with non-block notation:
 
@@ -525,7 +884,9 @@ It is convenient for use in e.g. the Rails console with non-block notation:
 
 #### `:bypass`
 
-The bypass strategy simply silences index updates.
+When the bypass strategy is active the index will not be automatically updated on object save.
+
+For example, on `City.first.save!` the cities index would not be updated.
 
 #### Nesting
 
@@ -579,582 +940,341 @@ RSpec.configure do |config|
 end
 ```
 
-### Index querying
+### Elasticsearch client options
 
-```ruby
-scope = UsersIndex.query(term: {name: 'foo'})
-  .filter(range: {rating: {gte: 100}})
-  .order(created: :desc)
-  .limit(20).offset(100)
-
-scope.to_a # => will produce array of UserIndex::User or other types instances
-scope.map { |user| user.email }
-scope.total_count # => will return total objects count
+All connection options, except the `:prefix`, are passed to the `Elasticseach::Client.new` ([chewy/lib/chewy.rb](https://github.com/toptal/chewy/blob/f5bad9f83c21416ac10590f6f34009c645062e89/lib/chewy.rb#L153-L160)):
 
-scope.per(10).page(3) # supports kaminari pagination
-scope.explain.map { |user| user._explanation }
-scope.only(:id, :email) # returns ids and emails only
+Here's the relevant Elasticsearch documentation on the subject: https://rubydoc.info/gems/elasticsearch-transport#setting-hosts
 
-scope.merge(other_scope) # queries could be merged
-```
+### `ActiveSupport::Notifications` support
 
-Also, queries can be performed on a type individually:
+Chewy has notifying the following events:
 
-```ruby
-UsersIndex::User.filter(term: {name: 'foo'}) # will return UserIndex::User collection only
-```
+#### `search_query.chewy` payload
 
-If you are performing more than one `filter` or `query` in the chain, all the filters and queries will be concatenated in the way specified by
-`filter_mode` and `query_mode` respectively.
+  * `payload[:index]`: requested index class
+  * `payload[:request]`: request hash
 
-The default `filter_mode` is `:and` and the default `query_mode` is `bool`.
+#### `import_objects.chewy` payload
 
-Available filter modes are: `:and`, `:or`, `:must`, `:should` and any minimum_should_match-acceptable value
+  * `payload[:index]`: currently imported index name
+  * `payload[:import]`: imports stats, total imported and deleted objects count:
 
-Available query modes are: `:must`, `:should`, `:dis_max`, any minimum_should_match-acceptable value or float value for dis_max query with tie_breaker specified.
+    ```ruby
+    {index: 30, delete: 5}
+    ```
 
-```ruby
-UsersIndex::User.filter{ name == 'Fred' }.filter{ age < 42 } # will be wrapped with `and` filter
-UsersIndex::User.filter{ name == 'Fred' }.filter{ age < 42 }.filter_mode(:should) # will be wrapped with bool `should` filter
-UsersIndex::User.filter{ name == 'Fred' }.filter{ age < 42 }.filter_mode('75%') # will be wrapped with bool `should` filter with `minimum_should_match: '75%'`
-```
+  * `payload[:errors]`: might not exist. Contains grouped errors with objects ids list:
 
-See [query.rb](lib/chewy/query.rb) for more details.
+    ```ruby
+    {index: {
+      'error 1 text' => ['1', '2', '3'],
+      'error 2 text' => ['4']
+    }, delete: {
+      'delete error text' => ['10', '12']
+    }}
+    ```
 
-### Additional query action.
+### NewRelic integration
 
-You may also perform additional actions on the query scope, such as deleting of all the scope documents:
+To integrate with NewRelic you may use the following example source (config/initializers/chewy.rb):
 
 ```ruby
-UsersIndex.delete_all
-UsersIndex::User.delete_all
-UsersIndex.filter{ age < 42 }.delete_all
-UsersIndex::User.filter{ age < 42 }.delete_all
-```
-
-### Filters query DSL
+require 'new_relic/agent/instrumentation/evented_subscriber'
 
-There is a test version of the filter-creating DSL:
+class ChewySubscriber < NewRelic::Agent::Instrumentation::EventedSubscriber
+  def start(name, id, payload)
+    event = ChewyEvent.new(name, Time.current, nil, id, payload)
+    push_event(event)
+  end
 
-```ruby
-UsersIndex.filter{ name == 'Fred' } # will produce `term` filter.
-UsersIndex.filter{ age <= 42 } # will produce `range` filter.
-```
+  def finish(_name, id, _payload)
+    pop_event(id).finish
+  end
 
-The basis of the DSL is the expression. There are 2 types of expressions:
+  class ChewyEvent < NewRelic::Agent::Instrumentation::Event
+    OPERATIONS = {
+      'import_objects.chewy' => 'import',
+      'search_query.chewy' => 'search',
+      'delete_query.chewy' => 'delete'
+    }.freeze
 
-* Simple function
+    def initialize(*args)
+      super
+      @segment = start_segment
+    end
 
-  ```ruby
-  UsersIndex.filter{ s('doc["num"] > 1') } # script expression
-  UsersIndex.filter{ q(query_string: {query: 'lazy fox'}) } # query expression
-  ```
+    def start_segment
+      segment = NewRelic::Agent::Transaction::DatastoreSegment.new product, operation, collection, host, port
+      if (txn = state.current_transaction)
+        segment.transaction = txn
+      end
+      segment.notice_sql @payload[:request].to_s
+      segment.start
+      segment
+    end
 
-* Field-dependent composite expression
-  Consists of the field name (with or without dot notation), a value, and an action operator between them. The field name might take additional options for passing to the resulting expression.
+    def finish
+      if (txn = state.current_transaction)
+        txn.add_segment @segment
+      end
+      @segment.finish
+    end
 
-  ```ruby
-  UsersIndex.filter{ name == 'Name' } # simple field term filter
-  UsersIndex.filter{ name(:bool) == ['Name1', 'Name2'] } # terms query with `execution: :bool` option passed
-  UsersIndex.filter{ answers.title =~ /regexp/ } # regexp filter for `answers.title` field
-  ```
+    private
 
-You can combine expressions as you wish with the help of combination operators.
+    def state
+      @state ||= NewRelic::Agent::TransactionState.tl_get
+    end
 
-```ruby
-UsersIndex.filter{ (name == 'Name') & (email == 'Email') } # combination produces `and` filter
-UsersIndex.filter{
-  must(
-    should(name =~ 'Fr').should_not(name == 'Fred') & (age == 42), email =~ /gmail\.com/
-  ) | ((roles.admin == true) & name?)
-} # many of the combination possibilities
-```
+    def product
+      'Elasticsearch'
+    end
 
-There is also a special syntax for cache enabling:
+    def operation
+      OPERATIONS[name]
+    end
 
-```ruby
-UsersIndex.filter{ ~name == 'Name' } # you can apply tilde to the field name
-UsersIndex.filter{ ~(name == 'Name') } # or to the whole expression
+    def collection
+      payload.values_at(:type, :index)
+             .reject { |value| value.try(:empty?) }
+             .first
+             .to_s
+    end
 
-# if you are applying cache to the one part of range filter
-# the whole filter will be cached:
-UsersIndex.filter{ ~(age > 42) & (age <= 50) }
+    def host
+      Chewy.client.transport.hosts.first[:host]
+    end
 
-# You can pass cache options as a field option also.
-UsersIndex.filter{ name(cache: true) == 'Name' }
-UsersIndex.filter{ name(cache: false) == 'Name' }
+    def port
+      Chewy.client.transport.hosts.first[:port]
+    end
+  end
+end
 
-# With regexp filter you can pass _cache_key
-UsersIndex.filter{ name(cache: 'name_regexp') =~ /Name/ }
-# Or not
-UsersIndex.filter{ name(cache: true) =~ /Name/ }
+ActiveSupport::Notifications.subscribe(/.chewy$/, ChewySubscriber.new)
 ```
 
-Compliance cheatsheet for filters and DSL expressions:
-
-* Term filter
-
-  ```json
-  {"term": {"name": "Fred"}}
-  {"not": {"term": {"name": "Johny"}}}
-  ```
-
-  ```ruby
-  UsersIndex.filter{ name == 'Fred' }
-  UsersIndex.filter{ name != 'Johny' }
-  ```
-
-* Terms filter
+### Search requests
 
-  ```json
-  {"terms": {"name": ["Fred", "Johny"]}}
-  {"not": {"terms": {"name": ["Fred", "Johny"]}}}
+Quick introduction.
 
-  {"terms": {"name": ["Fred", "Johny"], "execution": "or"}}
+#### Composing requests
 
-  {"terms": {"name": ["Fred", "Johny"], "execution": "and"}}
+The request DSL have the same chainable nature as AR. The main class is `Chewy::Search::Request`.
 
-  {"terms": {"name": ["Fred", "Johny"], "execution": "bool"}}
-
-  {"terms": {"name": ["Fred", "Johny"], "execution": "fielddata"}}
-  ```
-
-  ```ruby
-  UsersIndex.filter{ name == ['Fred', 'Johny'] }
-  UsersIndex.filter{ name != ['Fred', 'Johny'] }
-
-  UsersIndex.filter{ name(:|) == ['Fred', 'Johny'] }
-  UsersIndex.filter{ name(:or) == ['Fred', 'Johny'] }
-  UsersIndex.filter{ name(execution: :or) == ['Fred', 'Johny'] }
-
-  UsersIndex.filter{ name(:&) == ['Fred', 'Johny'] }
-  UsersIndex.filter{ name(:and) == ['Fred', 'Johny'] }
-  UsersIndex.filter{ name(execution: :and) == ['Fred', 'Johny'] }
-
-  UsersIndex.filter{ name(:b) == ['Fred', 'Johny'] }
-  UsersIndex.filter{ name(:bool) == ['Fred', 'Johny'] }
-  UsersIndex.filter{ name(execution: :bool) == ['Fred', 'Johny'] }
-
-  UsersIndex.filter{ name(:f) == ['Fred', 'Johny'] }
-  UsersIndex.filter{ name(:fielddata) == ['Fred', 'Johny'] }
-  UsersIndex.filter{ name(execution: :fielddata) == ['Fred', 'Johny'] }
-  ```
-
-* Regexp filter (== and =~ are equivalent)
-
-  ```json
-  {"regexp": {"name.first": "s.*y"}}
-
-  {"not": {"regexp": {"name.first": "s.*y"}}}
-
-  {"regexp": {"name.first": {"value": "s.*y", "flags": "ANYSTRING|INTERSECTION"}}}
-  ```
-
-  ```ruby
-  UsersIndex.filter{ name.first == /s.*y/ }
-  UsersIndex.filter{ name.first =~ /s.*y/ }
-
-  UsersIndex.filter{ name.first != /s.*y/ }
-  UsersIndex.filter{ name.first !~ /s.*y/ }
-
-  UsersIndex.filter{ name.first(:anystring, :intersection) == /s.*y/ }
-  UsersIndex.filter{ name.first(flags: [:anystring, :intersection]) == /s.*y/ }
-  ```
-
-* Prefix filter
-
-  ```json
-  {"prefix": {"name": "Fre"}}
-  {"not": {"prefix": {"name": "Joh"}}}
-  ```
-
-  ```ruby
-  UsersIndex.filter{ name =~ re' }
-  UsersIndex.filter{ name !~ 'Joh' }
-  ```
-
-* Exists filter
-
-  ```json
-  {"exists": {"field": "name"}}
-  ```
-
-  ```ruby
-  UsersIndex.filter{ name? }
-  UsersIndex.filter{ !!name }
-  UsersIndex.filter{ !!name? }
-  UsersIndex.filter{ name != nil }
-  UsersIndex.filter{ !(name == nil) }
-  ```
-
-* Missing filter
-
-  ```json
-  {"missing": {"field": "name", "existence": true, "null_value": false}}
-  {"missing": {"field": "name", "existence": true, "null_value": true}}
-  {"missing": {"field": "name", "existence": false, "null_value": true}}
-  ```
-
-  ```ruby
-  UsersIndex.filter{ !name }
-  UsersIndex.filter{ !name? }
-  UsersIndex.filter{ name == nil }
-  ```
-
-* Range
-
-  ```json
-  {"range": {"age": {"gt": 42}}}
-  {"range": {"age": {"gte": 42}}}
-  {"range": {"age": {"lt": 42}}}
-  {"range": {"age": {"lte": 42}}}
-
-  {"range": {"age": {"gt": 40, "lt": 50}}}
-  {"range": {"age": {"gte": 40, "lte": 50}}}
-
-  {"range": {"age": {"gt": 40, "lte": 50}}}
-  {"range": {"age": {"gte": 40, "lt": 50}}}
-  ```
-
-  ```ruby
-  UsersIndex.filter{ age > 42 }
-  UsersIndex.filter{ age >= 42 }
-  UsersIndex.filter{ age < 42 }
-  UsersIndex.filter{ age <= 42 }
-
-  UsersIndex.filter{ age == (40..50) }
-  UsersIndex.filter{ (age > 40) & (age < 50) }
-  UsersIndex.filter{ age == [40..50] }
-  UsersIndex.filter{ (age >= 40) & (age <= 50) }
-
-  UsersIndex.filter{ (age > 40) & (age <= 50) }
-  UsersIndex.filter{ (age >= 40) & (age < 50) }
-  ```
-
-* Bool filter
-
-  ```json
-  {"bool": {
-    "must": [{"term": {"name": "Name"}}],
-    "should": [{"term": {"age": 42}}, {"term": {"age": 45}}]
-  }}
-  ```
-
-  ```ruby
-  UsersIndex.filter{ must(name == 'Name').should(age == 42, age == 45) }
-  ```
-
-* And filter
-
-  ```json
-  {"and": [{"term": {"name": "Name"}}, {"range": {"age": {"lt": 42}}}]}
-  ```
+```ruby
+CitiesIndex.query(match: {name: 'London'})
+```
 
-  ```ruby
-  UsersIndex.filter{ (name == 'Name') & (age < 42) }
-  ```
+Main methods of the request DSL are: `query`, `filter` and `post_filter`, it is possible to pass pure query hashes or use `elasticsearch-dsl`.
 
-* Or filter
+```ruby
+CitiesIndex
+  .filter(term: {name: 'Bangkok'})
+  .query(match: {name: 'London'})
+  .query.not(range: {population: {gt: 1_000_000}})
+```
 
-  ```json
-  {"or": [{"term": {"name": "Name"}}, {"range": {"age": {"lt": 42}}}]}
-  ```
+You can query a set of indexes at once:
 
-  ```ruby
-  UsersIndex.filter{ (name == 'Name') | (age < 42) }
-  ```
+```ruby
+CitiesIndex.indices(CountriesIndex).query(match: {name: 'Some'})
+```
 
-  ```json
-  {"not": {"term": {"name": "Name"}}}
-  {"not": {"range": {"age": {"lt": 42}}}}
-  ```
+See https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl.html and https://github.com/elastic/elasticsearch-dsl-ruby for more details.
 
-  ```ruby
-  UsersIndex.filter{ !(name == 'Name') } # or UsersIndex.filter{ name != 'Name' }
-  UsersIndex.filter{ !(age < 42) }
-  ```
+An important part of requests manipulation is merging. There are 4 methods to perform it: `merge`, `and`, `or`, `not`. See [Chewy::Search::QueryProxy](lib/chewy/search/query_proxy.rb) for details. Also, `only` and `except` methods help to remove unneeded parts of the request.
 
-* Match all filter
+Every other request part is covered by a bunch of additional methods, see [Chewy::Search::Request](lib/chewy/search/request.rb) for details:
 
-  ```json
-  {"match_all": {}}
-  ```
+```ruby
+CitiesIndex.limit(10).offset(30).order(:name, {population: {order: :desc}})
+```
 
-  ```ruby
-  UsersIndex.filter{ match_all }
-  ```
+Request DSL also provides additional scope actions, like `delete_all`, `exists?`, `count`, `pluck`, etc.
 
-* Has child filter
+#### Pagination
 
-  ```json
-  {"has_child": {"type": "blog_tag", "query": {"term": {"tag": "something"}}}
-  {"has_child": {"type": "comment", "filter": {"term": {"user": "john"}}}
-  ```
+The request DSL supports pagination with `Kaminari`. An extension is enabled on initialization if `Kaminari` is available. See [Chewy::Search](lib/chewy/search.rb) and [Chewy::Search::Pagination::Kaminari](lib/chewy/search/pagination/kaminari.rb) for details.
 
-  ```ruby
-  UsersIndex.filter{ has_child(:blog_tag).query(term: {tag: 'something'}) }
-  UsersIndex.filter{ has_child(:comment).filter{ user == 'john' } }
-  ```
+#### Named scopes
 
-* Has parent filter
+Chewy supports named scopes functionality. There is no specialized DSL for named scopes definition, it is simply about defining class methods.
 
-  ```json
-  {"has_parent": {"type": "blog", "query": {"term": {"tag": "something"}}}}
-  {"has_parent": {"type": "blog", "filter": {"term": {"text": "bonsai three"}}}}
-  ```
+See [Chewy::Search::Scoping](lib/chewy/search/scoping.rb) for details.
 
-  ```ruby
-  UsersIndex.filter{ has_parent(:blog).query(term: {tag: 'something'}) }
-  UsersIndex.filter{ has_parent(:blog).filter{ text == 'bonsai three' } }
-  ```
+#### Scroll API
 
-See [filters.rb](lib/chewy/query/filters.rb) for more details.
+ElasticSearch scroll API is utilized by a bunch of methods: `scroll_batches`, `scroll_hits`, `scroll_wrappers` and `scroll_objects`.
 
-### Faceting
+See [Chewy::Search::Scrolling](lib/chewy/search/scrolling.rb) for details.
 
-Facets are an optional sidechannel you can request from Elasticsearch describing certain fields of the resulting collection. The most common use for facets is to allow the user to continue filtering specifically within the subset, as opposed to the global index.
+#### Loading objects
 
-For instance, let's request the `country` field as a facet along with our users collection. We can do this with the #facets method like so:
+It is possible to load ORM/ODM source objects with the `objects` method. To provide additional loading options use `load` method:
 
 ```ruby
-UsersIndex.filter{ [...] }.facets({countries: {terms: {field: 'country'}}})
+CitiesIndex.load(scope: -> { active }).to_a # to_a returns `Chewy::Index` wrappers.
+CitiesIndex.load(scope: -> { active }).objects # An array of AR source objects.
 ```
 
-Let's look at what we asked from Elasticsearch. The facets setter method accepts a hash. You can choose custom/semantic key names for this hash for your own convenience (in this case I used the plural version of the actual field), in our case `countries`. The following nested hash tells ES to grab and aggregate values (terms) from the `country` field on our indexed records.
-
-The response will include the `:facets` sidechannel:
-
-```
-< { ... ,"facets":{"countries":{"_type":"terms","missing":?,"total":?,"other":?,"terms":[{"term":"USA","count":?},{"term":"Brazil","count":?}, ...}}
-```
+See [Chewy::Search::Loader](lib/chewy/search/loader.rb) for more details.
 
-### Aggregations
-
-Aggregations are part of the optional sidechannel that can be requested with a query.
-
-You interact with aggregations using the composable #aggregations method (or its alias #aggs)
-
-Let's look at an example.
+In case when it is necessary to iterate through both of the wrappers and objects simultaneously, `object_hash` method helps a lot:
 
 ```ruby
-class UsersIndex < Chewy::Index
-  define_type User do
-    field :name
-    field :rating
-  end
+scope = CitiesIndex.load(scope: -> { active })
+scope.each do |wrapper|
+  scope.object_hash[wrapper]
 end
-
-all_johns = UsersIndex::User.filter { name == 'john' }.aggs({ avg_rating: { avg: { field: 'rating' } } })
-
-avg_johns_rating = all_johns.aggs
-# => {"avg_rating"=>{"value"=>3.5}}
 ```
 
-It is convenient to name aggregations that you intend to reuse regularly. This is achieve with the .aggregation method,
-which is also available under the .agg alias method.
+### Rake tasks
 
-Here's the same example from before
+For a Rails application, some index-maintaining rake tasks are defined.
 
-```ruby
-class UsersIndex < Chewy::Index
-  define_type User do
-    field :name
-    field :rating, type: "long"
-    agg :avg_rating do
-      { avg: { field: 'rating' } }
-    end
-  end
-end
+#### `chewy:reset`
 
-all_johns = UsersIndex::User.filter { name == 'john' }.aggs(:avg_rating)
+Performs zero-downtime reindexing as described [here](https://www.elastic.co/blog/changing-mapping-with-zero-downtime). So the rake task creates a new index with unique suffix and then simply aliases it to the common index name. The previous index is deleted afterwards (see `Chewy::Index.reset!` for more details).
 
-avg_johns_rating = all_johns.aggs
-# => {"avg_rating"=>{"value"=>3.5}}
+```bash
+rake chewy:reset # resets all the existing indices
+rake chewy:reset[users] # resets UsersIndex only
+rake chewy:reset[users,cities] # resets UsersIndex and CitiesIndex
+rake chewy:reset[-users,cities] # resets every index in the application except specified ones
 ```
 
-It is possible to run into collisions between named aggregations. This occurs when there is more than one aggregation
- with the same name. To explicitly reference an aggregation you provide a string to the #aggs method of the form:
- `index_name#document_type.aggregation_name`
+#### `chewy:upgrade`
 
-Consider this example where there are two separate aggregations named `avg_rating`
+Performs reset exactly the same way as `chewy:reset` does, but only when the index specification (setting or mapping) was changed.
 
-```ruby
-class UsersIndex < Chewy::Index
-  define_type User do
-    field :name
-    field :rating, type: "long"
-    agg :avg_rating do
-      { avg: { field: 'rating' } }
-    end
-  end
-  define_type Post do
-    field :title
-    field :body
-    field :comments do
-      field :message
-      field :rating, type: "long"
-    end
-    agg :avg_rating do
-      { avg: { field: 'comments.rating' } }
-    end
-  end
-end
-
-all_docs = UsersIndex.filter {match_all}.aggs("users#user.avg_rating")
-all_docs.aggs
-# => {"users#user.avg_rating"=>{"value"=>3.5}}
-```
+It works only when index specification is locked in `Chewy::Stash::Specification` index. The first run will reset all indexes and lock their specifications.
 
-### Script fields
+See [Chewy::Stash::Specification](lib/chewy/stash.rb) and [Chewy::Index::Specification](lib/chewy/index/specification.rb) for more details.
 
-Script fields allow you to execute Elasticsearch's scripting languages such as groovy and javascript. More about supported languages and what scripting is [here](https://www.elastic.co/guide/en/elasticsearch/reference/0.90/modules-scripting.html). This feature allows you to calculate the distance between geo points, for example. This is how to use the DSL:
 
-```ruby
-UsersIndex.script_fields(
-  distance: {
-    params: {
-      lat: 37.569976,
-      lon: -122.351591
-    },
-    script: "doc['coordinates'].distanceInMiles(lat, lon)"
-  }
-)
+```bash
+rake chewy:upgrade # upgrades all the existing indices
+rake chewy:upgrade[users] # upgrades UsersIndex only
+rake chewy:upgrade[users,cities] # upgrades UsersIndex and CitiesIndex
+rake chewy:upgrade[-users,cities] # upgrades every index in the application except specified ones
 ```
-Here, `coordinates` is a field with type `geo_point`. There will be a `distance` field for the index's model in the search result.
 
-### Script scoring
+#### `chewy:update`
 
-Script scoring is used to score the search results. All scores are added to the search request and combined according to boost mode and score mode. This can be useful if, for example, a score function is computationally expensive and it is sufficient to compute the score on a filtered set of documents. For example, you might want to multiply the score by another numeric field in the doc:
+It doesn't create indexes, it simply imports everything to the existing ones and fails if the index was not created before.
 
-```ruby
-UsersIndex.script_score("_score * doc['my_numeric_field'].value")
+```bash
+rake chewy:update # updates all the existing indices
+rake chewy:update[users] # updates UsersIndex only
+rake chewy:update[users,cities] # updates UsersIndex and CitiesIndex
+rake chewy:update[-users,cities] # updates every index in the application except UsersIndex and CitiesIndex
 ```
 
-### Boost Factor
+#### `chewy:sync`
 
-Boost factors are a way to add a boost to a query where documents match the filter. If you have some users who are experts and some who are regular users, you might want to give the experts a higher score and boost to the top of the search results. You can accomplish this by using the #boost_factor method and adding a boost score of 5 for an expert user:
+Provides a way to synchronize outdated indexes with the source quickly and without doing a full reset. By default field `updated_at` is used to find outdated records, but this could be customized by `outdated_sync_field` as described at [Chewy::Index::Syncer](lib/chewy/index/syncer.rb).
 
-```ruby
-UsersIndex.boost_factor(5, filter: {term: {type: 'Expert'}})
+Arguments are similar to the ones taken by `chewy:update` task.
+
+See [Chewy::Index::Syncer](lib/chewy/index/syncer.rb) for more details.
+
+```bash
+rake chewy:sync # synchronizes all the existing indices
+rake chewy:sync[users] # synchronizes UsersIndex only
+rake chewy:sync[users,cities] # synchronizes UsersIndex and CitiesIndex
+rake chewy:sync[-users,cities] # synchronizes every index in the application except except UsersIndex and CitiesIndex
 ```
 
-### Objects loading
+#### `chewy:deploy`
 
-It is possible to load source objects from the database for every search result:
+This rake task is especially useful during the production deploy. It is a combination of `chewy:upgrade` and `chewy:sync` and the latter is called only for the indexes that were not reset during the first stage.
 
-```ruby
-scope = UsersIndex.filter(range: {rating: {gte: 100}})
+It is not possible to specify any particular indexes for this task as it doesn't make much sense.
 
-scope.load # => scope is marked to return User instances array
-scope.load.query(...) # => since objects are loaded lazily you can complete scope
-scope.load(user: { scope: ->{ includes(:country) }}) # you can also pass loading scopes for each
-                                                     # possibly returned type
-scope.load(user: { scope: User.includes(:country) }) # the second scope passing way.
-scope.load(scope: ->{ includes(:country) }) # and more common scope applied to every loaded object type.
+Right now the approach is that if some data had been updated, but index definition was not changed (no changes satisfying the synchronization algorithm were done), it would be much faster to perform manual partial index update inside data migrations or even manually after the deploy.
 
-scope.only(:id).load # it is optimal to request ids only if you are not planning to use type objects
-```
+Also, there is always full reset alternative with `rake chewy:reset`.
 
-The `preload` method takes the same options as `load` and ORM/ODM objects will be loaded, but the scope will still return an array of Chewy wrappers. To access real objects use the `_object` wrapper method:
+#### `chewy:create_missing_indexes`
 
-```ruby
-UsersIndex.filter(range: {rating: {gte: 100}}).preload(...).query(...).map(&:_object)
-```
+This rake task creates newly defined indexes in ElasticSearch and skips existing ones. Useful for production-like environments.
 
-See [loading.rb](lib/chewy/query/loading.rb) for more details.
+#### Parallelizing rake tasks
 
-### `ActiveSupport::Notifications` support
+Every task described above has its own parallel version. Every parallel rake task takes the number for processes for execution as the first argument and the rest of the arguments are exactly the same as for the non-parallel task version.
 
-Chewy has notifying the following events:
+[https://github.com/grosser/parallel](https://github.com/grosser/parallel) gem is required to use these tasks.
 
-#### `search_query.chewy` payload
+If the number of processes is not specified explicitly - `parallel` gem tries to automatically derive the number of processes to use.
 
-  * `payload[:index]`: requested index class
-  * `payload[:request]`: request hash
+```bash
+rake chewy:parallel:reset
+rake chewy:parallel:upgrade[4]
+rake chewy:parallel:update[4,cities]
+rake chewy:parallel:sync[4,-users]
+rake chewy:parallel:deploy[4] # performs parallel upgrade and parallel sync afterwards
+```
 
-#### `import_objects.chewy` payload
+#### `chewy:journal`
 
-  * `payload[:type]`: currently imported type
-  * `payload[:import]`: imports stats, total imported and deleted objects count:
+This namespace contains two tasks for the journal manipulations: `chewy:journal:apply` and `chewy:journal:clean`. Both are taking time as the first argument (optional for clean) and a list of indexes exactly as the tasks above. Time can be in any format parsable by ActiveSupport.
 
-    ```ruby
-    {index: 30, delete: 5}
-    ```
+```bash
+rake chewy:journal:apply["$(date -v-1H -u +%FT%TZ)"] # apply journaled changes for the past hour
+rake chewy:journal:apply["$(date -v-1H -u +%FT%TZ)",users] # apply journaled changes for the past hour on UsersIndex only
+```
 
-  * `payload[:errors]`: might not exists. Contains grouped errors with objects ids list:
+When the size of the journal becomes very large, the classical way of deletion would be obstructive and resource consuming. Fortunately, Chewy internally uses [delete-by-query](https://www.elastic.co/guide/en/elasticsearch/reference/7.17/docs-delete-by-query.html#docs-delete-by-query-task-api) ES function which supports async execution with batching and [throttling](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-delete-by-query.html#docs-delete-by-query-throttle).
 
-    ```ruby
-    {index: {
-      'error 1 text' => ['1', '2', '3'],
-      'error 2 text' => ['4']
-    }, delete: {
-      'delete error text' => ['10', '12']
-    }}
-    ```
+The available options, which can be set by ENV variables, are listed below:
+* `WAIT_FOR_COMPLETION` - a boolean flag. It controls async execution. It waits by default. When set to `false` (`0`, `f`, `false` or `off` in any case spelling is accepted as `false`), Elasticsearch performs some preflight checks, launches the request, and returns a task reference you can use to cancel the task or get its status.
+* `REQUESTS_PER_SECOND` - float. The throttle for this request in sub-requests per second. No throttling is enforced by default.
+* `SCROLL_SIZE` - integer. The number of documents to be deleted in single sub-request. The default batch size is 1000.
 
-#### NewRelic integration
+```bash
+rake chewy:journal:clean WAIT_FOR_COMPLETION=false REQUESTS_PER_SECOND=10 SCROLL_SIZE=5000
+```
 
-To integrate with NewRelic you may use the following example source (config/initializers/chewy.rb):
+### RSpec integration
 
-```ruby
-ActiveSupport::Notifications.subscribe('import_objects.chewy') do |name, start, finish, id, payload|
-  metric_name = "Database/ElasticSearch/import"
-  duration = (finish - start).to_f
-  logged = "#{payload[:type]} #{payload[:import].to_a.map{ |i| i.join(':') }.join(', ')}"
-
-  self.class.trace_execution_scoped([metric_name]) do
-    NewRelic::Agent.instance.transaction_sampler.notice_sql(logged, nil, duration)
-    NewRelic::Agent.instance.sql_sampler.notice_sql(logged, metric_name, nil, duration)
-    NewRelic::Agent.record_metric(metric_name, duration)
-  end
-end
+Just add `require 'chewy/rspec'` to your spec_helper.rb and you will get additional features:
 
-ActiveSupport::Notifications.subscribe('search_query.chewy') do |name, start, finish, id, payload|
-  metric_name = "Database/ElasticSearch/search"
-  duration = (finish - start).to_f
-  logged = "#{payload[:type].presence || payload[:index]} #{payload[:request]}"
+[update_index](lib/chewy/rspec/update_index.rb) helper
+`mock_elasticsearch_response` helper to mock elasticsearch response
+`mock_elasticsearch_response_sources` helper to mock elasticsearch response sources
+`build_query` matcher to compare request and expected query (returns `true`/`false`)
 
-  self.class.trace_execution_scoped([metric_name]) do
-    NewRelic::Agent.instance.transaction_sampler.notice_sql(logged, nil, duration)
-    NewRelic::Agent.instance.sql_sampler.notice_sql(logged, metric_name, nil, duration)
-    NewRelic::Agent.record_metric(metric_name, duration)
-  end
-end
-```
+To use `mock_elasticsearch_response` and `mock_elasticsearch_response_sources` helpers add `include Chewy::Rspec::Helpers` to your tests.
 
-### Rake tasks
+See [chewy/rspec/](lib/chewy/rspec/) for more details.
 
-Inside the Rails application, some index-maintaining rake tasks are defined.
+### Minitest integration
 
-```bash
-rake chewy:reset # resets all the existing indices, declared in app/chewy
-rake chewy:reset[users] # resets UsersIndex only
+Add `require 'chewy/minitest'` to your test_helper.rb, and then for tests which you'd like indexing test hooks, `include Chewy::Minitest::Helpers`.
 
-rake chewy:update # updates all the existing indices, declared in app/chewy
-rake chewy:update[users] # updates UsersIndex only
-```
+Since you can set `:bypass` strategy for test suites and manually handle import for the index and manually flush test indices using `Chewy.massacre`. This will help reduce unnecessary ES requests
+
+But if you require chewy to index/update model regularly in your test suite then you can specify `:urgent` strategy for documents indexing. Add `Chewy.strategy(:urgent)` to test_helper.rb.
 
-`rake chewy:reset` performs zero-downtime reindexing as described [here](https://www.elastic.co/blog/changing-mapping-with-zero-downtime). So basically rake task creates a new index with uniq suffix and then simply aliases it to the common index name. The previous index is deleted afterwards (see `Chewy::Index.reset!` for more details).
+Also, you can use additional helpers:
 
+`mock_elasticsearch_response` to mock elasticsearch response
+`mock_elasticsearch_response_sources` to mock elasticsearch response sources
+`assert_elasticsearch_query` to compare request and expected query (returns `true`/`false`)
 
-### Rspec integration
+See [chewy/minitest/](lib/chewy/minitest/) for more details.
 
-Just add `require 'chewy/rspec'` to your spec_helper.rb and you will get additional features: See [update_index.rb](lib/chewy/rspec/update_index.rb) for more details.
+### DatabaseCleaner
 
-If you use `DatabaseCleaner` in your tests with [the `transaction` strategy](https://github.com/DatabaseCleaner/database_cleaner#how-to-use), you may run into the problem that `ActiveRecord`'s models are not indexed automatically on save despite the fact that you set the callbacks to do this with the `update_index` method. The issue arises because `chewy` indexes data on `after_commit` run as default, but all `after_commit` callbacks are not run with the `DatabaseCleaner`'s' `transaction` strategy. You can solve this issue by changing the `Chewy.use_after_commit_callbacks` option. Just add the following initializer in your Rails application:
+If you use `DatabaseCleaner` in your tests with [the `transaction` strategy](https://github.com/DatabaseCleaner/database_cleaner#how-to-use), you may run into the problem that `ActiveRecord`'s models are not indexed automatically on save despite the fact that you set the callbacks to do this with the `update_index` method. The issue arises because `chewy` indices data on `after_commit` run as default, but all `after_commit` callbacks are not run with the `DatabaseCleaner`'s' `transaction` strategy. You can solve this issue by changing the `Chewy.use_after_commit_callbacks` option. Just add the following initializer in your Rails application:
 
 ```ruby
 #config/initializers/chewy.rb
 Chewy.use_after_commit_callbacks = !Rails.env.test?
 ```
 
-## TODO a.k.a coming soon:
-
-* Typecasting support
-* Advanced (simplified) query DSL: `UsersIndex.query { email == 'my@gmail.com' }` will produce term query
-* update_all support
-* Maybe, closer ORM/ODM integration, creating index classes implicitly
-
 ## Contributing
 
 1. Fork it (http://github.com/toptal/chewy/fork)
@@ -1164,9 +1284,14 @@ Chewy.use_after_commit_callbacks = !Rails.env.test?
 5. Push to the branch (`git push origin my-new-feature`)
 6. Create new Pull Request
 
-Use the following Rake tasks to control the Elasticsearch cluster while developing.
+Use the following Rake tasks to control the Elasticsearch cluster while developing, if you prefer native Elasticsearch installation over the dockerized one:
 
 ```bash
 rake elasticsearch:start # start Elasticsearch cluster on 9250 port for tests
 rake elasticsearch:stop # stop Elasticsearch
 ```
+
+## Copyright
+
+Copyright (c) 2013-2021 Toptal, LLC. See [LICENSE.txt](LICENSE.txt) for
+further details.