chewy 0.10.1 → 7.0.0

data/CODE_OF_CONDUCT.md

@@ -0,0 +1,14 @@
+# The Chewy Community Code of Conduct
+
+**Note:** We have picked the following code of conduct based on [Ruby's own code of conduct](https://www.ruby-lang.org/en/conduct/).
+
+This document provides a few simple community guidelines for a safe, respectful,
+productive, and collaborative place for any person who is willing to contribute
+to the Chewy community. It applies to all "collaborative spaces", which are
+defined as community communications channels (such as mailing lists, submitted
+patches, commit comments, etc.).
+
+* Participants will be tolerant of opposing views.
+* Participants must ensure that their language and actions are free of personal attacks and disparaging personal remarks.
+* When interpreting the words and actions of others, participants should always assume good intentions.
+* Behaviour which can be reasonably considered harassment will not be tolerated.

data/CONTRIBUTING.md

@@ -0,0 +1,63 @@
+# Contributing
+
+If you discover issues, have ideas for improvements or new features,
+please report them to the [issue tracker][1] of the repository or
+submit a pull request. Please, try to follow these guidelines when you
+do so.
+
+## Issue reporting
+
+* Check that the issue has not already been reported.
+* Check that the issue has not already been fixed in the latest code
+  (a.k.a. `master`).
+* Be clear, concise and precise in your description of the problem.
+* Open an issue with a descriptive title and a summary in grammatically correct,
+  complete sentences.
+* Include the versions of Chewy, Elasticsearch, Ruby, Rails, etc.
+* Include any relevant code to the issue summary.
+
+## Pull requests
+
+* Read [how to properly contribute to open source projects on GitHub][2].
+* Fork the project.
+* Use a topic/feature branch to easily amend a pull request later, if necessary.
+* Write [good commit messages][3].
+* Use the same coding conventions as the rest of the project.
+* Commit and push until you are happy with your contribution.
+* If your change has a corresponding open GitHub issue, prefix the commit message with `[Fix #github-issue-number]`.
+* Make sure to add tests for it. This is important so I don't break it
+  in a future version unintentionally.
+* Add an entry to the [Changelog](CHANGELOG.md).
+* Please try not to mess with the Rakefile, version, or history. If
+  you want to have your own version, or is otherwise necessary, that
+  is fine, but please isolate to its own commit so I can cherry-pick
+  around it.
+* Make sure the test suite is passing and the code you wrote doesn't produce
+  RuboCop offenses.
+* [Squash related commits together][5].
+* Open a [pull request][4] that relates to *only* one subject with a clear title
+  and description in grammatically correct, complete sentences.
+
+## Changelog entry format
+
+Here are a few examples:
+
+```
+* [#753](https://github.com/toptal/chewy/pull/753): Add support for direct_import parameter to skip objects reloading. ([@TikiTDO][], [@dalthon][])
+* [#739](https://github.com/toptal/chewy/pull/739): Remove explicit `main` branch dependencies on `rspec-*` gems after `rspec-mocks` 3.10.2 is released. ([@rabotyaga][])
+```
+
+* Mark it up in [Markdown syntax][6].
+* The entry line should start with `* ` (an asterisk and a space).
+* If the change has a related GitHub issue (e.g. a bug fix for a reported issue), put a link to the issue as `[#123](https://github.com/toptal/chewy/issues/123): `.
+* Describe the brief of the change. The sentence should end with a punctuation.
+* If this is a breaking change, mark it with `**(Breaking)**`.
+* At the end of the entry, add an implicit link to your GitHub user page as `([@username][])`.
+* If this is your first contribution to the project, add a link definition for the implicit link to the bottom of the changelog as `[@username]: https://github.com/username`.
+
+[1]: https://github.com/toptal/chewy/issues
+[2]: https://www.gun.io/blog/how-to-github-fork-branch-and-pull-request
+[3]: https://tbaggery.com/2008/04/19/a-note-about-git-commit-messages.html
+[4]: https://help.github.com/articles/about-pull-requests
+[5]: http://gitready.com/advanced/2009/02/10/squashing-commits-with-rebase.html
+[6]: https://daringfireball.net/projects/markdown/syntax

data/Gemfile

@@ -8,9 +8,11 @@ gem 'activerecord'
 
 gem 'activejob', require: false
 gem 'resque', require: false
-gem 'shoryuken', require: false
 gem 'sidekiq', require: false
 
+gem 'aws-sdk-sqs', require: false
+gem 'shoryuken', require: false
+
 gem 'kaminari-core', require: false
 gem 'will_paginate', require: false
 
@@ -22,3 +24,5 @@ gem 'guard-rspec'
 
 gem 'redcarpet'
 gem 'yard'
+
+gem 'rexml' if RUBY_VERSION >= '3.0.0'

data/LICENSE.txt

@@ -1,4 +1,4 @@
-Copyright (c) 2013 pyromaniac
+Copyright (c) 2013-2021 Toptal, LLC
 
 MIT License
 

data/README.md

@@ -1,14 +1,11 @@
 [![Gem Version](https://badge.fury.io/rb/chewy.svg)](http://badge.fury.io/rb/chewy)
-[![Build Status](https://travis-ci.org/toptal/chewy.svg)](https://travis-ci.org/toptal/chewy)
+[![CircleCI](https://circleci.com/gh/toptal/chewy/tree/master.svg?style=svg)](https://circleci.com/gh/toptal/chewy/tree/master)
 [![Code Climate](https://codeclimate.com/github/toptal/chewy.svg)](https://codeclimate.com/github/toptal/chewy)
 [![Inline docs](http://inch-ci.org/github/toptal/chewy.svg?branch=master)](http://inch-ci.org/github/toptal/chewy)
 
-<p align="right">Sponsored by</p>
-<p align="right"><a href="https://www.toptal.com/"><img src="https://www.toptal.com/assets/public/blocks/logo/big.png" alt="Toptal" width="105" height="34"></a></p>
-
 # Chewy
 
-Chewy is an ODM and wrapper for [the official Elasticsearch client](https://github.com/elastic/elasticsearch-ruby).
+Chewy is an ODM (Object Document Mapper), built on top of the [the official Elasticsearch client](https://github.com/elastic/elasticsearch-ruby).
 
 ## Table of Contents
 
@@ -34,7 +31,7 @@ Chewy is an ODM and wrapper for [the official Elasticsearch client](https://gith
     * [Non-block notation](#non-block-notation)
     * [Designing your own strategies](#designing-your-own-strategies)
   * [Rails application strategies integration](#rails-application-strategies-integration)
-  * [ActiveSupport::Notifications support](#activesupport-notifications-support)
+  * [ActiveSupport::Notifications support](#activesupportnotifications-support)
   * [NewRelic integration](#newrelic-integration)
   * [Search requests](#search-requests)
     * [Composing requests](#composing-requests)
@@ -42,7 +39,6 @@ Chewy is an ODM and wrapper for [the official Elasticsearch client](https://gith
     * [Named scopes](#named-scopes)
     * [Scroll API](#scroll-api)
     * [Loading objects](#loading-objects)
-    * [Legacy DSL incompatibilities](#legacy-dsl-incompatibilities)
   * [Rake tasks](#rake-tasks)
     * [chewy:reset](#chewyreset)
     * [chewy:upgrade](#chewyupgrade)
@@ -51,13 +47,14 @@ Chewy is an ODM and wrapper for [the official Elasticsearch client](https://gith
     * [chewy:deploy](#chewydeploy)
     * [Parallelizing rake tasks](#parallelizing-rake-tasks)
     * [chewy:journal](#chewyjournal)
-  * [Rspec integration](#rspec-integration)
+  * [RSpec integration](#rspec-integration)
   * [Minitest integration](#minitest-integration)
-* [TODO a.k.a coming soon](#todo-aka-coming-soon)
 * [Contributing](#contributing)
 
 ## Why Chewy?
 
+In this section we'll cover why you might want to use Chewy instead of the official `elasticsearch-ruby` client gem.
+
 * Multi-model indices.
 
   Index classes are independent from ORM/ODM models. Now, implementing e.g. cross-model autocomplete is much easier. You can just define the index and work with it in an object-oriented style. You can define several types for index - one per indexed model.
@@ -76,10 +73,9 @@ Chewy is an ODM and wrapper for [the official Elasticsearch client](https://gith
 
 * Support for ActiveRecord, [Mongoid](https://github.com/mongoid/mongoid) and [Sequel](https://github.com/jeremyevans/sequel).
 
-
 ## Installation
 
-Add this line to your application's Gemfile:
+Add this line to your application's `Gemfile`:
 
     gem 'chewy'
 
@@ -91,13 +87,35 @@ Or install it yourself as:
 
     $ gem install chewy
 
+## Compatibility
+
+### Ruby
+
+Chewy is compatible with MRI 2.5-3.0¹.
+
+> ¹ Ruby 3 is only supported with Rails 6.1
+
+### Elasticsearch compatibility matrix
+
+| Chewy version | Elasticsearch version              |
+| ------------- | ---------------------------------- |
+| 7.0.0         | 6.8, 7.x                           |
+| 6.0.0         | 5.x, 6.x                           |
+| 5.x           | 5.x, limited support for 1.x & 2.x |
+
+See [Migration guide](migration_guide.md).
+
 ## Usage
 
 ### Client settings
 
-There are two ways to configure the Chewy client: the `Chewy.settings` hash and `chewy.yml`
+There are two ways to configure the Chewy client:
 
-You can create this file manually or run `rails g chewy:install`.
+* via the hash `Chewy.settings`
+* via the configuration file `chewy.yml`
+
+You can create `chewy.yml` manually or run `rails g chewy:install` to
+generate it.
 
 ```ruby
 # config/initializers/chewy.rb
@@ -129,25 +147,29 @@ Chewy.logger = Logger.new(STDOUT)
 
 See [config.rb](lib/chewy/config.rb) for more details.
 
-#### Aws Elastic Search
-If you would like to use AWS's ElasticSearch using an IAM user policy, you will need to sign your requests for the `es:*` action by injecting the appropriate headers passing a proc to `transport_options`.
+#### AWS Elasticsearch
+
+If you would like to use AWS's Elasticsearch using an IAM user policy, you will need to sign your requests for the `es:*` action by injecting the appropriate headers passing a proc to `transport_options`.
+You'll need an additional gem for Faraday middleware: add `gem 'faraday_middleware-aws-sigv4'` to your Gemfile.
 
 ```ruby
- Chewy.settings = {
-    host: 'http://my-es-instance-on-aws.us-east-1.es.amazonaws.com:80',
-    transport_options: {
-      headers: { content_type: 'application/json' },
-      proc: -> (f) do
-          f.request :aws_signers_v4,
-                    service_name: 'es',
-                    region: 'us-east-1',
-                    credentials: Aws::Credentials.new(
-                      ENV['AWS_ACCESS_KEY'],
-                      ENV['AWS_SECRET_ACCESS_KEY'])
-      end
-    }
+require 'faraday_middleware/aws_sigv4'
+
+Chewy.settings = {
+  host: 'http://my-es-instance-on-aws.us-east-1.es.amazonaws.com:80',
+  port: 80, # 443 for https host
+  transport_options: {
+    headers: { content_type: 'application/json' },
+    proc: -> (f) do
+        f.request :aws_sigv4,
+                  service: 'es',
+                  region: 'us-east-1',
+                  access_key_id: ENV['AWS_ACCESS_KEY'],
+                  secret_access_key: ENV['AWS_SECRET_ACCESS_KEY']
+    end
   }
-  ```
+}
+```
 
 ### Index definition
 
@@ -180,7 +202,7 @@ If you would like to use AWS's ElasticSearch using an IAM user policy, you will
       field :badges, value: ->(user) { user.badges.map(&:name) } # passing array values to index
       field :projects do # the same block syntax for multi_field, if `:type` is specified
         field :title
-        field :description # default data type is `string`
+        field :description # default data type is `text`
         # additional top-level objects passed to value proc:
         field :categories, value: ->(project, user) { project.categories.map(&:name) if user.active? }
       end
@@ -208,7 +230,7 @@ If you would like to use AWS's ElasticSearch using an IAM user policy, you will
 
     define_type User.active.includes(:country, :badges, :projects) do
       root date_detection: false do
-        template 'about_translations.*', type: 'string', analyzer: 'standard'
+        template 'about_translations.*', type: 'text', analyzer: 'standard'
 
         field :first_name, :last_name
         field :email, analyzer: 'email'
@@ -335,9 +357,9 @@ This will automatically set the type or root field to `object`. You may also spe
 To define a multi field you have to specify any type except for `object` or `nested` in the root field:
 
 ```ruby
-field :full_name, type: 'string', value: ->{ full_name.strip } do
+field :full_name, type: 'text', value: ->{ full_name.strip } do
   field :ordered, analyzer: 'ordered'
-  field :untouched, index: 'not_analyzed'
+  field :untouched, type: 'keyword'
 end
 ```
 
@@ -678,6 +700,11 @@ Chewy.strategy(:sidekiq) do
 end
 ```
 
+The default queue name is `chewy`, you can customize it in settings: `sidekiq.queue_name`
+```
+Chewy.settings[:sidekiq] = {queue: :low}
+```
+
 #### `:active_job`
 
 This does the same thing as `:atomic`, but using ActiveJob. This will inherit the ActiveJob configuration settings including the `active_job.queue_adapter` setting for the environment. Patch `Chewy::Strategy::ActiveJob::Worker` for index updates improving.
@@ -688,6 +715,11 @@ Chewy.strategy(:active_job) do
 end
 ```
 
+The default queue name is `chewy`, you can customize it in settings: `active_job.queue_name`
+```
+Chewy.settings[:active_job] = {queue: :low}
+```
+
 #### `:shoryuken`
 
 This does the same thing as `:atomic`, but asynchronously using shoryuken. Patch `Chewy::Strategy::Shoryuken::Worker` for index updates improving.
@@ -807,38 +839,84 @@ Chewy has notifying the following events:
 To integrate with NewRelic you may use the following example source (config/initializers/chewy.rb):
 
 ```ruby
-ActiveSupport::Notifications.subscribe('import_objects.chewy') do |name, start, finish, id, payload|
-  metric_name = "Database/ElasticSearch/import"
-  duration = (finish - start).to_f
-  logged = "#{payload[:type]} #{payload[:import].to_a.map{ |i| i.join(':') }.join(', ')}"
-
-  self.class.trace_execution_scoped([metric_name]) do
-    NewRelic::Agent.instance.transaction_sampler.notice_sql(logged, nil, duration)
-    NewRelic::Agent.instance.sql_sampler.notice_sql(logged, metric_name, nil, duration)
-    NewRelic::Agent.record_metric(metric_name, duration)
+require 'new_relic/agent/instrumentation/evented_subscriber'
+
+class ChewySubscriber < NewRelic::Agent::Instrumentation::EventedSubscriber
+  def start(name, id, payload)
+    event = ChewyEvent.new(name, Time.current, nil, id, payload)
+    push_event(event)
   end
-end
 
-ActiveSupport::Notifications.subscribe('search_query.chewy') do |name, start, finish, id, payload|
-  metric_name = "Database/ElasticSearch/search"
-  duration = (finish - start).to_f
-  logged = "#{payload[:type].presence || payload[:index]} #{payload[:request]}"
+  def finish(_name, id, _payload)
+    pop_event(id).finish
+  end
+
+  class ChewyEvent < NewRelic::Agent::Instrumentation::Event
+    OPERATIONS = {
+      'import_objects.chewy' => 'import',
+      'search_query.chewy' => 'search',
+      'delete_query.chewy' => 'delete'
+    }.freeze
 
-  self.class.trace_execution_scoped([metric_name]) do
-    NewRelic::Agent.instance.transaction_sampler.notice_sql(logged, nil, duration)
-    NewRelic::Agent.instance.sql_sampler.notice_sql(logged, metric_name, nil, duration)
-    NewRelic::Agent.record_metric(metric_name, duration)
+    def initialize(*args)
+      super
+      @segment = start_segment
+    end
+
+    def start_segment
+      segment = NewRelic::Agent::Transaction::DatastoreSegment.new product, operation, collection, host, port
+      if (txn = state.current_transaction)
+        segment.transaction = txn
+      end
+      segment.notice_sql @payload[:request].to_s
+      segment.start
+      segment
+    end
+
+    def finish
+      if (txn = state.current_transaction)
+        txn.add_segment @segment
+      end
+      @segment.finish
+    end
+
+    private
+
+    def state
+      @state ||= NewRelic::Agent::TransactionState.tl_get
+    end
+
+    def product
+      'Elasticsearch'
+    end
+
+    def operation
+      OPERATIONS[name]
+    end
+
+    def collection
+      payload.values_at(:type, :index)
+             .reject { |value| value.try(:empty?) }
+             .first
+             .to_s
+    end
+
+    def host
+      Chewy.client.transport.hosts.first[:host]
+    end
+
+    def port
+      Chewy.client.transport.hosts.first[:port]
+    end
   end
 end
+
+ActiveSupport::Notifications.subscribe(/.chewy$/, ChewySubscriber.new)
 ```
 
 ### Search requests
 
-Long story short: there is a new DSL that supports ES2 and ES5, the previous DSL version (which supports ES1 and ES2) documentation was moved to [LEGACY_DSL.md](LEGACY_DSL.md).
-
-If you want to use it - simply do `Chewy.search_class = Chewy::Query` somewhere before indices are initialized.
-
-The new DSL is enabled by default, here is a quick introduction.
+Quick introduction.
 
 #### Composing requests
 
@@ -906,17 +984,6 @@ scope.each do |wrapper|
 end
 ```
 
-#### Legacy DSL incompatibilities
-
-* Filters advanced block DSL is not supported anymore, `elasticsearch-dsl` is used instead.
-* Things like `query_mode` and `filter_mode` are in past, use advanced DSL to achieve similar behavior. See [Chewy::Search::QueryProxy](lib/chewy/search/query_proxy.rb) for details.
-* `preload` method is no more, the collection returned by scope doesn't depend on loading options, scope always returns `Chewy::Type` wrappers. To get ORM/ODM objects, use `#objects` method.
-* Some of the methods have changed their purpose: `only` was used to filter fields before, now it filters the scope. To filter fields use `source` or `stored_fields`.
-* `types!` method is no more, use `except(:types).types(...)`
-* Named aggregations are not supported, use named scopes instead.
-* A lot of query-level methods were not ported: everything that is related to boost and scoring. Use `query` manipulation to provide them.
-* `Chewy::Type#_object` returns nil always. Use `Chewy::Search::Response#object_hash` instead.
-
 ### Rake tasks
 
 For a Rails application, some index-maintaining rake tasks are defined.
@@ -936,9 +1003,9 @@ rake chewy:reset[-users,places] # resets every index in the application except s
 
 Performs reset exactly the same way as `chewy:reset` does, but only when the index specification (setting or mapping) was changed.
 
-It works only when index specification is locked in `Chewy::Stash` index. The first run will reset all indexes and lock their specifications.
+It works only when index specification is locked in `Chewy::Stash::Specification` index. The first run will reset all indexes and lock their specifications.
 
-See [Chewy::Stash](lib/chewy/stash.rb) and [Chewy::Index::Specification](lib/chewy/index/specification.rb) for more details.
+See [Chewy::Stash::Specification](lib/chewy/stash.rb) and [Chewy::Index::Specification](lib/chewy/index/specification.rb) for more details.
 
 
 ```bash
@@ -963,7 +1030,7 @@ rake chewy:update[-users,places#city] # updates every index in the application e
 
 #### `chewy:sync`
 
-Provides a way to synchronize outdated indexes with the source quickly and without doing a full reset.
+Provides a way to synchronize outdated indexes with the source quickly and without doing a full reset. By default field `updated_at` is used to find outdated records, but this could be customized by `outdated_sync_field` as described at [Chewy::Type::Syncer](lib/chewy/type/syncer.rb).
 
 Arguments are similar to the ones taken by `chewy:update` task. It is possible to specify a particular type or a whole index.
 
@@ -1011,7 +1078,7 @@ rake chewy:journal:apply["$(date -v-1H -u +%FT%TZ)"] # apply journaled changes f
 rake chewy:journal:apply["$(date -v-1H -u +%FT%TZ)",users] # apply journaled changes for the past hour on UsersIndex only
 ```
 
-### Rspec integration
+### RSpec integration
 
 Just add `require 'chewy/rspec'` to your spec_helper.rb and you will get additional features: See [update_index.rb](lib/chewy/rspec/update_index.rb) for more details.
 
@@ -1032,12 +1099,6 @@ If you use `DatabaseCleaner` in your tests with [the `transaction` strategy](htt
 Chewy.use_after_commit_callbacks = !Rails.env.test?
 ```
 
-## TODO a.k.a coming soon:
-
-* Typecasting support
-* update_all support
-* Maybe, closer ORM/ODM integration, creating index classes implicitly
-
 ## Contributing
 
 1. Fork it (http://github.com/toptal/chewy/fork)
@@ -1053,3 +1114,8 @@ Use the following Rake tasks to control the Elasticsearch cluster while developi
 rake elasticsearch:start # start Elasticsearch cluster on 9250 port for tests
 rake elasticsearch:stop # stop Elasticsearch
 ```
+
+## Copyright
+
+Copyright (c) 2013-2021 Toptal, LLC. See [LICENSE.txt](LICENSE.txt) for
+further details.