chewy 7.1.0 → 7.2.0
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/.github/workflows/ruby.yml +58 -0
- data/.rubocop.yml +13 -8
- data/.rubocop_todo.yml +110 -22
- data/CHANGELOG.md +53 -0
- data/Gemfile +0 -7
- data/Guardfile +3 -1
- data/README.md +282 -245
- data/chewy.gemspec +3 -5
- data/gemfiles/rails.5.2.activerecord.gemfile +8 -14
- data/gemfiles/rails.6.0.activerecord.gemfile +8 -14
- data/gemfiles/rails.6.1.activerecord.gemfile +8 -14
- data/lib/chewy.rb +21 -75
- data/lib/chewy/config.rb +40 -40
- data/lib/chewy/errors.rb +0 -12
- data/lib/chewy/fields/base.rb +11 -1
- data/lib/chewy/fields/root.rb +3 -4
- data/lib/chewy/index.rb +46 -87
- data/lib/chewy/index/actions.rb +51 -32
- data/lib/chewy/{type → index}/adapter/active_record.rb +12 -3
- data/lib/chewy/{type → index}/adapter/base.rb +2 -3
- data/lib/chewy/{type → index}/adapter/object.rb +27 -31
- data/lib/chewy/{type → index}/adapter/orm.rb +11 -14
- data/lib/chewy/{type → index}/crutch.rb +5 -5
- data/lib/chewy/{type → index}/import.rb +36 -27
- data/lib/chewy/{type → index}/import/bulk_builder.rb +15 -13
- data/lib/chewy/{type → index}/import/bulk_request.rb +6 -6
- data/lib/chewy/{type → index}/import/journal_builder.rb +10 -10
- data/lib/chewy/{type → index}/import/routine.rb +15 -14
- data/lib/chewy/{type → index}/mapping.rb +26 -31
- data/lib/chewy/{type → index}/observe.rb +9 -19
- data/lib/chewy/index/specification.rb +1 -0
- data/lib/chewy/{type → index}/syncer.rb +60 -57
- data/lib/chewy/{type → index}/witchcraft.rb +11 -7
- data/lib/chewy/{type → index}/wrapper.rb +2 -2
- data/lib/chewy/journal.rb +8 -8
- data/lib/chewy/minitest/helpers.rb +9 -13
- data/lib/chewy/minitest/search_index_receiver.rb +22 -26
- data/lib/chewy/railtie.rb +4 -2
- data/lib/chewy/rake_helper.rb +82 -107
- data/lib/chewy/rspec/update_index.rb +47 -43
- data/lib/chewy/search.rb +4 -17
- data/lib/chewy/search/loader.rb +18 -30
- data/lib/chewy/search/parameters.rb +4 -2
- data/lib/chewy/search/parameters/concerns/query_storage.rb +2 -2
- data/lib/chewy/search/parameters/source.rb +5 -1
- data/lib/chewy/search/query_proxy.rb +9 -2
- data/lib/chewy/search/request.rb +82 -86
- data/lib/chewy/search/response.rb +4 -4
- data/lib/chewy/search/scoping.rb +6 -7
- data/lib/chewy/search/scrolling.rb +11 -11
- data/lib/chewy/stash.rb +14 -22
- data/lib/chewy/strategy.rb +3 -19
- data/lib/chewy/strategy/sidekiq.rb +1 -0
- data/lib/chewy/version.rb +1 -1
- data/lib/generators/chewy/install_generator.rb +1 -1
- data/lib/tasks/chewy.rake +10 -22
- data/migration_guide.md +14 -0
- data/spec/chewy/config_spec.rb +14 -39
- data/spec/chewy/fields/base_spec.rb +412 -148
- data/spec/chewy/fields/root_spec.rb +16 -24
- data/spec/chewy/fields/time_fields_spec.rb +5 -5
- data/spec/chewy/index/actions_spec.rb +270 -24
- data/spec/chewy/{type → index}/adapter/active_record_spec.rb +68 -40
- data/spec/chewy/{type → index}/adapter/object_spec.rb +21 -6
- data/spec/chewy/{type → index}/import/bulk_builder_spec.rb +23 -31
- data/spec/chewy/{type → index}/import/bulk_request_spec.rb +5 -6
- data/spec/chewy/{type → index}/import/journal_builder_spec.rb +9 -15
- data/spec/chewy/{type → index}/import/routine_spec.rb +16 -16
- data/spec/chewy/{type → index}/import_spec.rb +102 -98
- data/spec/chewy/{type → index}/mapping_spec.rb +46 -54
- data/spec/chewy/index/observe_spec.rb +116 -0
- data/spec/chewy/index/settings_spec.rb +3 -1
- data/spec/chewy/index/specification_spec.rb +7 -17
- data/spec/chewy/{type → index}/syncer_spec.rb +14 -19
- data/spec/chewy/{type → index}/witchcraft_spec.rb +20 -22
- data/spec/chewy/index/wrapper_spec.rb +100 -0
- data/spec/chewy/index_spec.rb +59 -102
- data/spec/chewy/journal_spec.rb +9 -22
- data/spec/chewy/minitest/helpers_spec.rb +13 -15
- data/spec/chewy/minitest/search_index_receiver_spec.rb +22 -26
- data/spec/chewy/multi_search_spec.rb +4 -5
- data/spec/chewy/rake_helper_spec.rb +145 -55
- data/spec/chewy/rspec/update_index_spec.rb +74 -71
- data/spec/chewy/search/loader_spec.rb +19 -37
- data/spec/chewy/search/pagination/kaminari_examples.rb +3 -5
- data/spec/chewy/search/pagination/kaminari_spec.rb +1 -1
- data/spec/chewy/search/parameters/indices_spec.rb +2 -8
- data/spec/chewy/search/parameters/order_spec.rb +1 -1
- data/spec/chewy/search/parameters/query_storage_examples.rb +67 -21
- data/spec/chewy/search/parameters/search_after_spec.rb +4 -1
- data/spec/chewy/search/parameters/source_spec.rb +8 -2
- data/spec/chewy/search/parameters_spec.rb +12 -3
- data/spec/chewy/search/query_proxy_spec.rb +68 -17
- data/spec/chewy/search/request_spec.rb +222 -74
- data/spec/chewy/search/response_spec.rb +12 -12
- data/spec/chewy/search/scrolling_spec.rb +7 -9
- data/spec/chewy/search_spec.rb +32 -35
- data/spec/chewy/stash_spec.rb +9 -21
- data/spec/chewy/strategy/active_job_spec.rb +8 -8
- data/spec/chewy/strategy/atomic_spec.rb +9 -10
- data/spec/chewy/strategy/sidekiq_spec.rb +8 -8
- data/spec/chewy/strategy_spec.rb +19 -15
- data/spec/chewy_spec.rb +14 -100
- data/spec/spec_helper.rb +2 -21
- data/spec/support/active_record.rb +15 -5
- metadata +44 -103
- data/.circleci/config.yml +0 -214
- data/Appraisals +0 -81
- data/gemfiles/rails.5.2.mongoid.6.4.gemfile +0 -17
- data/gemfiles/sequel.4.45.gemfile +0 -11
- data/lib/chewy/search/pagination/will_paginate.rb +0 -43
- data/lib/chewy/strategy/resque.rb +0 -27
- data/lib/chewy/strategy/shoryuken.rb +0 -40
- data/lib/chewy/type.rb +0 -120
- data/lib/chewy/type/actions.rb +0 -43
- data/lib/chewy/type/adapter/mongoid.rb +0 -67
- data/lib/chewy/type/adapter/sequel.rb +0 -93
- data/lib/sequel/plugins/chewy_observe.rb +0 -63
- data/spec/chewy/search/pagination/will_paginate_examples.rb +0 -63
- data/spec/chewy/search/pagination/will_paginate_spec.rb +0 -23
- data/spec/chewy/strategy/resque_spec.rb +0 -46
- data/spec/chewy/strategy/shoryuken_spec.rb +0 -70
- data/spec/chewy/type/actions_spec.rb +0 -50
- data/spec/chewy/type/adapter/mongoid_spec.rb +0 -372
- data/spec/chewy/type/adapter/sequel_spec.rb +0 -472
- data/spec/chewy/type/observe_spec.rb +0 -137
- data/spec/chewy/type/wrapper_spec.rb +0 -100
- data/spec/chewy/type_spec.rb +0 -55
- data/spec/support/mongoid.rb +0 -93
- data/spec/support/sequel.rb +0 -80
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: a958ecfd3273f1e8086b3c07aa823f3db8e5d2300a0ca2a86a563e1d6bd2df42
|
4
|
+
data.tar.gz: 8195d2162307dbf368865cc5e5975372e169796aa85a5a6bd8ac404484b99d75
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 7b3286e50679ce8b43c9a595d90502cbd028ac8e94c93e240a3f820de15fbc19052887f1bf973670421bcc8a2e1cfe9dd4d9b2183cb0ccd828726220df005c7c
|
7
|
+
data.tar.gz: 00a24e51c43cd0e191af699949ca3ac1ca738777fd3a22dfe1d97d7d55feb6ebfea29e2577d488ff1f97189e1430266110395afea3efcb949917d07a6c26477f
|
@@ -0,0 +1,58 @@
|
|
1
|
+
name: CI
|
2
|
+
|
3
|
+
on: [push]
|
4
|
+
|
5
|
+
jobs:
|
6
|
+
tests:
|
7
|
+
runs-on: ubuntu-latest
|
8
|
+
strategy:
|
9
|
+
fail-fast: false
|
10
|
+
matrix:
|
11
|
+
ruby: [2.6, 2.7]
|
12
|
+
gemfile: [rails.5.2.activerecord, rails.6.0.activerecord, rails.6.1.activerecord]
|
13
|
+
name: ${{ matrix.ruby }}-${{ matrix.gemfile }}
|
14
|
+
|
15
|
+
env:
|
16
|
+
BUNDLE_GEMFILE: gemfiles/${{ matrix.gemfile }}.gemfile
|
17
|
+
|
18
|
+
steps:
|
19
|
+
- uses: actions/checkout@v2
|
20
|
+
- uses: ruby/setup-ruby@v1
|
21
|
+
with:
|
22
|
+
ruby-version: ${{ matrix.ruby }}
|
23
|
+
bundler-cache: true
|
24
|
+
- name: Run Elasticsearch
|
25
|
+
uses: elastic/elastic-github-actions/elasticsearch@master
|
26
|
+
with:
|
27
|
+
stack-version: 7.10.1
|
28
|
+
port: 9250
|
29
|
+
- name: Tests
|
30
|
+
run: bundle exec rspec
|
31
|
+
|
32
|
+
ruby-3-0-activerecord-6-1:
|
33
|
+
runs-on: ubuntu-latest
|
34
|
+
env:
|
35
|
+
BUNDLE_GEMFILE: gemfiles/rails.6.1.activerecord.gemfile
|
36
|
+
steps:
|
37
|
+
- uses: actions/checkout@v2
|
38
|
+
- uses: ruby/setup-ruby@v1
|
39
|
+
with:
|
40
|
+
ruby-version: '3.0'
|
41
|
+
bundler-cache: true
|
42
|
+
- name: Run Elasticsearch
|
43
|
+
uses: elastic/elastic-github-actions/elasticsearch@master
|
44
|
+
with:
|
45
|
+
stack-version: 7.10.1
|
46
|
+
port: 9250
|
47
|
+
- name: Tests
|
48
|
+
run: bundle exec rspec
|
49
|
+
|
50
|
+
rubocop:
|
51
|
+
runs-on: ubuntu-latest
|
52
|
+
steps:
|
53
|
+
- uses: actions/checkout@v2
|
54
|
+
- uses: ruby/setup-ruby@v1
|
55
|
+
with:
|
56
|
+
ruby-version: 2.7
|
57
|
+
bundler-cache: true
|
58
|
+
- run: bundle exec rubocop --format simple
|
data/.rubocop.yml
CHANGED
@@ -1,24 +1,31 @@
|
|
1
1
|
inherit_from: .rubocop_todo.yml
|
2
2
|
|
3
|
+
AllCops:
|
4
|
+
NewCops: enable
|
5
|
+
TargetRubyVersion: 2.6
|
6
|
+
|
3
7
|
Layout/AccessModifierIndentation:
|
4
8
|
EnforcedStyle: outdent
|
5
9
|
|
6
|
-
Layout/
|
10
|
+
Layout/HashAlignment:
|
7
11
|
EnforcedLastArgumentHashStyle: always_ignore
|
8
12
|
|
9
|
-
Layout/
|
13
|
+
Layout/ParameterAlignment:
|
10
14
|
EnforcedStyle: with_fixed_indentation
|
11
15
|
|
12
16
|
Layout/CaseIndentation:
|
13
17
|
EnforcedStyle: end
|
14
18
|
|
15
|
-
Layout/
|
19
|
+
Layout/EndAlignment:
|
20
|
+
EnforcedStyleAlignWith: variable
|
21
|
+
|
22
|
+
Layout/FirstArrayElementIndentation:
|
16
23
|
EnforcedStyle: consistent
|
17
24
|
|
18
|
-
Layout/
|
25
|
+
Layout/FirstHashElementIndentation:
|
19
26
|
EnforcedStyle: consistent
|
20
27
|
|
21
|
-
Layout/
|
28
|
+
Layout/HeredocIndentation:
|
22
29
|
Enabled: false
|
23
30
|
|
24
31
|
Layout/MultilineMethodCallIndentation:
|
@@ -33,9 +40,6 @@ Layout/SpaceInsideHashLiteralBraces:
|
|
33
40
|
Lint/AmbiguousBlockAssociation:
|
34
41
|
Enabled: false
|
35
42
|
|
36
|
-
Lint/EndAlignment:
|
37
|
-
EnforcedStyleAlignWith: variable
|
38
|
-
|
39
43
|
Style/Alias:
|
40
44
|
EnforcedStyle: prefer_alias_method
|
41
45
|
|
@@ -53,4 +57,5 @@ Metrics/BlockLength:
|
|
53
57
|
|
54
58
|
Metrics/ModuleLength:
|
55
59
|
Exclude:
|
60
|
+
- 'lib/chewy/rake_helper.rb'
|
56
61
|
- '**/*_spec.rb'
|
data/.rubocop_todo.yml
CHANGED
@@ -1,44 +1,132 @@
|
|
1
1
|
# This configuration was generated by
|
2
2
|
# `rubocop --auto-gen-config`
|
3
|
-
# on
|
3
|
+
# on 2021-04-02 12:44:05 UTC using RuboCop version 1.11.0.
|
4
4
|
# The point is for the user to remove these configuration records
|
5
5
|
# one by one as the offenses are removed from the code base.
|
6
6
|
# Note that changes in the inspected code, or installation of new
|
7
7
|
# versions of RuboCop, may require this file to be generated again.
|
8
8
|
|
9
|
-
# Offense count:
|
9
|
+
# Offense count: 1
|
10
|
+
# Configuration parameters: Include.
|
11
|
+
# Include: **/*.gemspec
|
12
|
+
Gemspec/RequiredRubyVersion:
|
13
|
+
Exclude:
|
14
|
+
- 'chewy.gemspec'
|
15
|
+
|
16
|
+
# Offense count: 2
|
17
|
+
# Configuration parameters: AllowedMethods.
|
18
|
+
# AllowedMethods: enums
|
19
|
+
Lint/ConstantDefinitionInBlock:
|
20
|
+
Exclude:
|
21
|
+
- 'lib/chewy/rspec/update_index.rb'
|
22
|
+
- 'spec/chewy/config_spec.rb'
|
23
|
+
|
24
|
+
# Offense count: 10
|
25
|
+
# Configuration parameters: AllowComments, AllowEmptyLambdas.
|
26
|
+
Lint/EmptyBlock:
|
27
|
+
Exclude:
|
28
|
+
- 'spec/chewy/minitest/helpers_spec.rb'
|
29
|
+
- 'spec/chewy/rspec/update_index_spec.rb'
|
30
|
+
- 'spec/chewy/search/scrolling_spec.rb'
|
31
|
+
- 'spec/chewy/strategy/atomic_spec.rb'
|
32
|
+
- 'spec/chewy/strategy_spec.rb'
|
33
|
+
- 'spec/chewy/index/import/bulk_request_spec.rb'
|
34
|
+
- 'spec/chewy/index/witchcraft_spec.rb'
|
35
|
+
- 'spec/chewy_spec.rb'
|
36
|
+
|
37
|
+
# Offense count: 3
|
38
|
+
Lint/MissingSuper:
|
39
|
+
Exclude:
|
40
|
+
- 'lib/chewy/strategy/atomic.rb'
|
41
|
+
- 'lib/chewy/index/adapter/object.rb'
|
42
|
+
- 'lib/chewy/index/adapter/orm.rb'
|
43
|
+
|
44
|
+
# Offense count: 35
|
45
|
+
# Configuration parameters: IgnoredMethods, CountRepeatedAttributes.
|
10
46
|
Metrics/AbcSize:
|
11
|
-
Max:
|
47
|
+
Max: 41
|
12
48
|
|
13
49
|
# Offense count: 4
|
14
|
-
# Configuration parameters: CountComments.
|
50
|
+
# Configuration parameters: CountComments, CountAsOne.
|
15
51
|
Metrics/ClassLength:
|
16
|
-
Max:
|
52
|
+
Max: 267
|
17
53
|
|
18
|
-
# Offense count:
|
54
|
+
# Offense count: 13
|
55
|
+
# Configuration parameters: IgnoredMethods.
|
19
56
|
Metrics/CyclomaticComplexity:
|
20
|
-
Max:
|
21
|
-
|
22
|
-
# Offense count: 1450
|
23
|
-
# Configuration parameters: AllowHeredoc, AllowURI, URISchemes.
|
24
|
-
# URISchemes: http, https
|
25
|
-
Metrics/LineLength:
|
26
|
-
Max: 198
|
57
|
+
Max: 12
|
27
58
|
|
28
|
-
# Offense count:
|
29
|
-
# Configuration parameters: CountComments.
|
59
|
+
# Offense count: 43
|
60
|
+
# Configuration parameters: CountComments, CountAsOne, ExcludedMethods, IgnoredMethods.
|
30
61
|
Metrics/MethodLength:
|
31
|
-
Max:
|
62
|
+
Max: 29
|
32
63
|
|
33
|
-
# Offense count:
|
34
|
-
# Configuration parameters: CountComments.
|
64
|
+
# Offense count: 2
|
65
|
+
# Configuration parameters: CountComments, CountAsOne.
|
35
66
|
Metrics/ModuleLength:
|
36
|
-
Max:
|
67
|
+
Max: 158
|
37
68
|
|
38
|
-
# Offense count:
|
69
|
+
# Offense count: 18
|
70
|
+
# Configuration parameters: IgnoredMethods.
|
39
71
|
Metrics/PerceivedComplexity:
|
40
|
-
Max:
|
72
|
+
Max: 13
|
41
73
|
|
42
|
-
# Offense count:
|
74
|
+
# Offense count: 11
|
75
|
+
# Configuration parameters: EnforcedStyle, CheckMethodNames, CheckSymbols, AllowedIdentifiers.
|
76
|
+
# SupportedStyles: snake_case, normalcase, non_integer
|
77
|
+
# AllowedIdentifiers: capture3, iso8601, rfc1123_date, rfc822, rfc2822, rfc3339
|
78
|
+
Naming/VariableNumber:
|
79
|
+
Exclude:
|
80
|
+
- 'spec/chewy/fields/root_spec.rb'
|
81
|
+
|
82
|
+
# Offense count: 5
|
83
|
+
Style/DocumentDynamicEvalDefinition:
|
84
|
+
Exclude:
|
85
|
+
- 'lib/chewy/index/actions.rb'
|
86
|
+
- 'lib/chewy/repository.rb'
|
87
|
+
- 'lib/chewy/search/pagination/kaminari.rb'
|
88
|
+
- 'lib/chewy/index/crutch.rb'
|
89
|
+
- 'lib/chewy/index/witchcraft.rb'
|
90
|
+
|
91
|
+
# Offense count: 58
|
43
92
|
Style/Documentation:
|
44
93
|
Enabled: false
|
94
|
+
|
95
|
+
# Offense count: 2
|
96
|
+
# Cop supports --auto-correct.
|
97
|
+
Style/EvalWithLocation:
|
98
|
+
Exclude:
|
99
|
+
- 'spec/chewy/index_spec.rb'
|
100
|
+
|
101
|
+
# Offense count: 191
|
102
|
+
# Cop supports --auto-correct.
|
103
|
+
# Configuration parameters: EnforcedStyle.
|
104
|
+
# SupportedStyles: always, always_true, never
|
105
|
+
Style/FrozenStringLiteralComment:
|
106
|
+
Enabled: false
|
107
|
+
|
108
|
+
# Offense count: 2
|
109
|
+
# Configuration parameters: MinBodyLength.
|
110
|
+
Style/GuardClause:
|
111
|
+
Exclude:
|
112
|
+
- 'lib/chewy.rb'
|
113
|
+
- 'spec/support/active_record.rb'
|
114
|
+
|
115
|
+
# Offense count: 10
|
116
|
+
# Cop supports --auto-correct.
|
117
|
+
Style/IfUnlessModifier:
|
118
|
+
Exclude:
|
119
|
+
- 'lib/chewy.rb'
|
120
|
+
- 'lib/chewy/railtie.rb'
|
121
|
+
- 'lib/chewy/rspec/update_index.rb'
|
122
|
+
- 'lib/chewy/search/query_proxy.rb'
|
123
|
+
- 'lib/chewy/index/import.rb'
|
124
|
+
- 'lib/chewy/index/witchcraft.rb'
|
125
|
+
- 'spec/support/active_record.rb'
|
126
|
+
|
127
|
+
# Offense count: 53
|
128
|
+
# Cop supports --auto-correct.
|
129
|
+
# Configuration parameters: AutoCorrect, AllowHeredoc, AllowURI, URISchemes, IgnoreCopDirectives, IgnoredPatterns.
|
130
|
+
# URISchemes: http, https
|
131
|
+
Layout/LineLength:
|
132
|
+
Max: 191
|
data/CHANGELOG.md
CHANGED
@@ -8,6 +8,58 @@
|
|
8
8
|
|
9
9
|
### Bugs Fixed
|
10
10
|
|
11
|
+
## 7.2.0 (2021-04-19)
|
12
|
+
|
13
|
+
### New Features
|
14
|
+
|
15
|
+
* [#778](https://github.com/toptal/chewy/pull/778): Add `ignore_blank` option to `field` method ([@Vitalina-Vakulchyk][]):
|
16
|
+
* `true` by default for the `geo_point` type
|
17
|
+
* `false` by default for other types
|
18
|
+
|
19
|
+
### Changes
|
20
|
+
|
21
|
+
* [#783](https://github.com/toptal/chewy/pull/783): **(Breaking)** Remove `Chewy::Type`, simplify DSL ([@rabotyaga][])
|
22
|
+
* Remove the `Chewy::Type` class
|
23
|
+
* e.g. remove `CitiesIndex::City` / `CitiesIndex.city`
|
24
|
+
* `CitiesIndex::City.import! ...` becomes `CitiesIndex.import! ...`
|
25
|
+
* Simplify index DSL:
|
26
|
+
* `define_type` block -> `index_scope` clause
|
27
|
+
* it can be omitted completely, if you don't need to specify the scope or options, e.g. `name`
|
28
|
+
* Remove type names from string representations:
|
29
|
+
* in `update_index` ActiveRecord helper and RSpec matcher, e.g.
|
30
|
+
* `update_index('cities#city')` -> `update_index('cities')`
|
31
|
+
* `update_index(UsersIndex::User)` -> `update_index(UsersIndex)`
|
32
|
+
* in rake tasks (e.g. `rake chewy:update[cities#city]` -> `rake chewy:update[cities]`)
|
33
|
+
* in rake tasks output (e.g. `Imported CitiesIndex::City in 1s, stats: index 3` -> `Imported CitiesIndex in 1s, stats: index 3`)
|
34
|
+
* Use index name instead of type name in loader additional scope
|
35
|
+
* e.g. `CitiesIndex.filter(...).load(city: {scope: City.where(...)})` -> `CitiesIndex.filter(...).load(cities: {scope: City.where(...)})`
|
36
|
+
* [#692](https://github.com/toptal/chewy/issues/692): Add `.update_mapping` to Index class ([@Vitalina-Vakulchyk][]):
|
37
|
+
* Wrapped Elasticsearch gem `.put_mapping` with `.update_mapping` in Index class
|
38
|
+
* Add `rake chewy:update_mapping` task
|
39
|
+
* [#594](https://github.com/toptal/chewy/issues/594): Add `.reindex` to Index class ([@Vitalina-Vakulchyk][]):
|
40
|
+
* Wrapped Elasticsearch gem `.reindex` with `.reindex` in Index class
|
41
|
+
* Add `rake chewy:reindex` task
|
42
|
+
* [#679](https://github.com/toptal/chewy/issues/679): Wrapped `Elasticsearch::API::Indices::Actions#clear_cache` with `.clear_cache` in Index class ([@Vitalina-Vakulchyk][])
|
43
|
+
* [#495](https://github.com/toptal/chewy/issues/495): Ability to change Rails console strategy with `Chewy.console_strategy` ([@Vitalina-Vakulchyk][])
|
44
|
+
* [#778](https://github.com/toptal/chewy/pull/778): **(Breaking)** Drop support for Ruby 2.5 ([@Vitalina-Vakulchyk][])
|
45
|
+
* [#776](https://github.com/toptal/chewy/pull/776): **(Breaking)** Removal of unnecessary features and integrations ([@Vitalina-Vakulchyk][]):
|
46
|
+
* `aws-sdk-sqs` / `shoryuken`
|
47
|
+
* `mongoid`
|
48
|
+
* `sequel`
|
49
|
+
* `will_paginate`
|
50
|
+
* `resque`
|
51
|
+
* [#769](https://github.com/toptal/chewy/pull/769): **(Breaking)** Removal of deprecated methods and rake tasks ([@Vitalina-Vakulchyk][]):
|
52
|
+
* `Chewy::Index.index_params` is removed, use `Chewy::Index.specification_hash` instead
|
53
|
+
* `Chewy::Index.derivable_index_name` is removed, use `Chewy::Index.derivable_name` instead
|
54
|
+
* `Chewy::Index.default_prefix` is removed, use `Chewy::Index.prefix` instead
|
55
|
+
* `Chewy::Index.build_index_name` is removed, use `Chewy::Index.index_name` instead
|
56
|
+
* `Chewy::RakeHelper.reset_index` is removed, use `Chewy::RakeHelper.reset` instead
|
57
|
+
* `Chewy::RakeHelper.reset_all` is removed, use `Chewy::RakeHelper.reset` instead
|
58
|
+
* `Chewy::RakeHelper.update_index` is removed, use `Chewy::RakeHelper.update` instead
|
59
|
+
* `Chewy::RakeHelper.update_all` is removed, use `Chewy::RakeHelper.update` instead
|
60
|
+
* `rake chewy:apply_changes_from` is removed, use `rake chewy:journal:apply` instead
|
61
|
+
* `rake chewy:clean_journal` is removed, use `rake chewy:journal:clean` instead
|
62
|
+
|
11
63
|
## 7.1.0 (2021-03-03)
|
12
64
|
|
13
65
|
### Changes
|
@@ -607,5 +659,6 @@
|
|
607
659
|
[@taylor-au]: https://github.com/taylor-au
|
608
660
|
[@TikiTDO]: https://github.com/TikiTDO
|
609
661
|
[@undr]: https://github.com/undr
|
662
|
+
[@Vitalina-Vakulchyk]: https://github.com/Vitalina-Vakulchyk
|
610
663
|
[@webgago]: https://github.com/webgago
|
611
664
|
[@yahooguntu]: https://github.com/yahooguntu
|
data/Gemfile
CHANGED
@@ -3,18 +3,11 @@ source 'https://rubygems.org'
|
|
3
3
|
gemspec
|
4
4
|
|
5
5
|
gem 'activerecord'
|
6
|
-
# gem 'mongoid'
|
7
|
-
# gem 'sequel'
|
8
6
|
|
9
7
|
gem 'activejob', require: false
|
10
|
-
gem 'resque', require: false
|
11
8
|
gem 'sidekiq', require: false
|
12
9
|
|
13
|
-
gem 'aws-sdk-sqs', require: false
|
14
|
-
gem 'shoryuken', require: false
|
15
|
-
|
16
10
|
gem 'kaminari-core', require: false
|
17
|
-
gem 'will_paginate', require: false
|
18
11
|
|
19
12
|
gem 'parallel', require: false
|
20
13
|
gem 'ruby-progressbar', require: false
|
data/Guardfile
CHANGED
@@ -9,7 +9,9 @@ guard :rspec, cmd: 'rspec' do
|
|
9
9
|
# Rails example
|
10
10
|
watch(%r{^app/(.+)\.rb$}) { |m| "spec/#{m[1]}_spec.rb" }
|
11
11
|
watch(%r{^app/(.*)(\.erb|\.haml|\.slim)$}) { |m| "spec/#{m[1]}#{m[2]}_spec.rb" }
|
12
|
-
watch(%r{^app/controllers/(.+)_(controller)\.rb$})
|
12
|
+
watch(%r{^app/controllers/(.+)_(controller)\.rb$}) do |m|
|
13
|
+
["spec/routing/#{m[1]}_routing_spec.rb", "spec/#{m[2]}s/#{m[1]}_#{m[2]}_spec.rb", "spec/acceptance/#{m[1]}_spec.rb"]
|
14
|
+
end
|
13
15
|
watch(%r{^spec/support/(.+)\.rb$}) { 'spec' }
|
14
16
|
watch('config/routes.rb') { 'spec/routing' }
|
15
17
|
watch('app/controllers/application_controller.rb') { 'spec/controllers' }
|
data/README.md
CHANGED
@@ -1,5 +1,5 @@
|
|
1
1
|
[![Gem Version](https://badge.fury.io/rb/chewy.svg)](http://badge.fury.io/rb/chewy)
|
2
|
-
[![
|
2
|
+
[![GitHub Actions](https://github.com/toptal/chewy/actions/workflows/ruby.yml/badge.svg)](https://github.com/toptal/chewy/actions/workflows/ruby.yml)
|
3
3
|
[![Code Climate](https://codeclimate.com/github/toptal/chewy.svg)](https://codeclimate.com/github/toptal/chewy)
|
4
4
|
[![Inline docs](http://inch-ci.org/github/toptal/chewy.svg?branch=master)](http://inch-ci.org/github/toptal/chewy)
|
5
5
|
|
@@ -7,49 +7,6 @@
|
|
7
7
|
|
8
8
|
Chewy is an ODM (Object Document Mapper), built on top of the [the official Elasticsearch client](https://github.com/elastic/elasticsearch-ruby).
|
9
9
|
|
10
|
-
## Table of Contents
|
11
|
-
|
12
|
-
* [Why Chewy?](#why-chewy)
|
13
|
-
* [Installation](#installation)
|
14
|
-
* [Usage](#usage)
|
15
|
-
* [Client settings](#client-settings)
|
16
|
-
* [AWS ElasticSearch configuration](#aws-elastic-search)
|
17
|
-
* [Index definition](#index-definition)
|
18
|
-
* [Type default import options](#type-default-import-options)
|
19
|
-
* [Multi (nested) and object field types](#multi-nested-and-object-field-types)
|
20
|
-
* [Geo Point fields](#geo-point-fields)
|
21
|
-
* [Crutches™ technology](#crutches-technology)
|
22
|
-
* [Witchcraft™ technology](#witchcraft-technology)
|
23
|
-
* [Raw Import](#raw-import)
|
24
|
-
* [Index creation during import](#index-creation-during-import)
|
25
|
-
* [Journaling](#journaling)
|
26
|
-
* [Types access](#types-access)
|
27
|
-
* [Index manipulation](#index-manipulation)
|
28
|
-
* [Index update strategies](#index-update-strategies)
|
29
|
-
* [Nesting](#nesting)
|
30
|
-
* [Non-block notation](#non-block-notation)
|
31
|
-
* [Designing your own strategies](#designing-your-own-strategies)
|
32
|
-
* [Rails application strategies integration](#rails-application-strategies-integration)
|
33
|
-
* [ActiveSupport::Notifications support](#activesupportnotifications-support)
|
34
|
-
* [NewRelic integration](#newrelic-integration)
|
35
|
-
* [Search requests](#search-requests)
|
36
|
-
* [Composing requests](#composing-requests)
|
37
|
-
* [Pagination](#pagination)
|
38
|
-
* [Named scopes](#named-scopes)
|
39
|
-
* [Scroll API](#scroll-api)
|
40
|
-
* [Loading objects](#loading-objects)
|
41
|
-
* [Rake tasks](#rake-tasks)
|
42
|
-
* [chewy:reset](#chewyreset)
|
43
|
-
* [chewy:upgrade](#chewyupgrade)
|
44
|
-
* [chewy:update](#chewyupdate)
|
45
|
-
* [chewy:sync](#chewysync)
|
46
|
-
* [chewy:deploy](#chewydeploy)
|
47
|
-
* [Parallelizing rake tasks](#parallelizing-rake-tasks)
|
48
|
-
* [chewy:journal](#chewyjournal)
|
49
|
-
* [RSpec integration](#rspec-integration)
|
50
|
-
* [Minitest integration](#minitest-integration)
|
51
|
-
* [Contributing](#contributing)
|
52
|
-
|
53
10
|
## Why Chewy?
|
54
11
|
|
55
12
|
In this section we'll cover why you might want to use Chewy instead of the official `elasticsearch-ruby` client gem.
|
@@ -66,7 +23,7 @@ In this section we'll cover why you might want to use Chewy instead of the offic
|
|
66
23
|
|
67
24
|
Chewy has an ActiveRecord-style query DSL. It is chainable, mergeable and lazy, so you can produce queries in the most efficient way. It also has object-oriented query and filter builders.
|
68
25
|
|
69
|
-
* Support for ActiveRecord
|
26
|
+
* Support for ActiveRecord.
|
70
27
|
|
71
28
|
## Installation
|
72
29
|
|
@@ -86,7 +43,7 @@ Or install it yourself as:
|
|
86
43
|
|
87
44
|
### Ruby
|
88
45
|
|
89
|
-
Chewy is compatible with MRI 2.
|
46
|
+
Chewy is compatible with MRI 2.6-3.0¹.
|
90
47
|
|
91
48
|
> ¹ Ruby 3 is only supported with Rails 6.1
|
92
49
|
|
@@ -94,6 +51,7 @@ Chewy is compatible with MRI 2.5-3.0¹.
|
|
94
51
|
|
95
52
|
| Chewy version | Elasticsearch version |
|
96
53
|
| ------------- | ---------------------------------- |
|
54
|
+
| 7.2.x | 7.x |
|
97
55
|
| 7.1.x | 7.x |
|
98
56
|
| 7.0.0 | 6.8, 7.x |
|
99
57
|
| 6.0.0 | 5.x, 6.x |
|
@@ -101,23 +59,156 @@ Chewy is compatible with MRI 2.5-3.0¹.
|
|
101
59
|
|
102
60
|
See [Migration guide](migration_guide.md).
|
103
61
|
|
104
|
-
|
62
|
+
### Active Record
|
105
63
|
|
106
|
-
|
64
|
+
5.2, 6.0, 6.1 Active Record versions are supported by all Chewy versions.
|
65
|
+
|
66
|
+
## Getting Started
|
67
|
+
|
68
|
+
Chewy provides functionality for Elasticsearch index handling, documents import mappings, index update strategies and chainable query DSL.
|
69
|
+
|
70
|
+
### Minimal client setting
|
71
|
+
|
72
|
+
Create `config/initializers/chewy.rb` with this line:
|
73
|
+
|
74
|
+
```ruby
|
75
|
+
Chewy.settings = {host: 'localhost:9250'}
|
76
|
+
```
|
77
|
+
|
78
|
+
And run `rails g chewy:install` to generate `chewy.yml`:
|
79
|
+
|
80
|
+
```yaml
|
81
|
+
# config/chewy.yml
|
82
|
+
# separate environment configs
|
83
|
+
test:
|
84
|
+
host: 'localhost:9250'
|
85
|
+
prefix: 'test'
|
86
|
+
development:
|
87
|
+
host: 'localhost:9200'
|
88
|
+
```
|
107
89
|
|
108
|
-
|
90
|
+
### Elasticsearch
|
109
91
|
|
110
|
-
|
111
|
-
|
92
|
+
Make sure you have Elasticsearch up and running. You can [install](https://www.elastic.co/guide/en/elasticsearch/reference/current/install-elasticsearch.html) it locally, but the easiest way is to use [Docker](https://www.docker.com/get-started):
|
93
|
+
|
94
|
+
```shell
|
95
|
+
$ docker run --rm --name elasticsearch -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" elasticsearch:7.11.1
|
96
|
+
```
|
112
97
|
|
113
|
-
|
114
|
-
|
98
|
+
### Index
|
99
|
+
|
100
|
+
Create `app/chewy/user_index.rb` with User Index:
|
101
|
+
|
102
|
+
```ruby
|
103
|
+
class UsersIndex < Chewy::Index
|
104
|
+
settings analysis: {
|
105
|
+
analyzer: {
|
106
|
+
email: {
|
107
|
+
tokenizer: 'keyword',
|
108
|
+
filter: ['lowercase']
|
109
|
+
}
|
110
|
+
}
|
111
|
+
}
|
112
|
+
|
113
|
+
index_scope User
|
114
|
+
field :first_name
|
115
|
+
field :last_name
|
116
|
+
field :email, analyzer: 'email'
|
117
|
+
end
|
118
|
+
```
|
119
|
+
|
120
|
+
### Model
|
121
|
+
|
122
|
+
Add User model, table and migrate it:
|
123
|
+
|
124
|
+
```shell
|
125
|
+
$ bundle exec rails g model User first_name last_name email
|
126
|
+
$ bundle exec rails db:migrate
|
127
|
+
```
|
128
|
+
|
129
|
+
Add `update_index` to app/models/user.rb:
|
130
|
+
|
131
|
+
```ruby
|
132
|
+
class User < ApplicationRecord
|
133
|
+
update_index('users') { self }
|
134
|
+
end
|
135
|
+
```
|
136
|
+
|
137
|
+
### Example of data request
|
138
|
+
|
139
|
+
1. Once a record is created (could be done via the Rails console), it creates User index too:
|
140
|
+
|
141
|
+
```
|
142
|
+
User.create(
|
143
|
+
first_name: "test1",
|
144
|
+
last_name: "test1",
|
145
|
+
email: 'test1@example.com',
|
146
|
+
# other fields
|
147
|
+
)
|
148
|
+
# UsersIndex Import (355.3ms) {:index=>1}
|
149
|
+
# => #<User id: 1, first_name: "test1", last_name: "test1", email: "test1@example.com", # other fields>
|
150
|
+
```
|
151
|
+
|
152
|
+
2. A query could be exposed at a given `UsersController`:
|
153
|
+
|
154
|
+
```ruby
|
155
|
+
def search
|
156
|
+
@users = UsersIndex.query(query_string: { fields: [:first_name, :last_name, :email, ...], query: search_params[:query], default_operator: 'and' })
|
157
|
+
render json: @users.to_json, status: :ok
|
158
|
+
end
|
159
|
+
|
160
|
+
private
|
161
|
+
|
162
|
+
def search_params
|
163
|
+
params.permit(:query, :page, :per)
|
164
|
+
end
|
165
|
+
```
|
166
|
+
|
167
|
+
3. So a request against `http://localhost:3000/users/search?query=test1@example.com` issuing a response like:
|
168
|
+
|
169
|
+
```json
|
170
|
+
[
|
171
|
+
{
|
172
|
+
"attributes":{
|
173
|
+
"id":"1",
|
174
|
+
"first_name":"test1",
|
175
|
+
"last_name":"test1",
|
176
|
+
"email":"test1@example.com",
|
177
|
+
...
|
178
|
+
"_score":0.9808291,
|
179
|
+
"_explanation":null
|
180
|
+
},
|
181
|
+
"_data":{
|
182
|
+
"_index":"users",
|
183
|
+
"_type":"_doc",
|
184
|
+
"_id":"1",
|
185
|
+
"_score":0.9808291,
|
186
|
+
"_source":{
|
187
|
+
"first_name":"test1",
|
188
|
+
"last_name":"test1",
|
189
|
+
"email":"test1@example.com",
|
190
|
+
...
|
191
|
+
}
|
192
|
+
}
|
193
|
+
}
|
194
|
+
]
|
195
|
+
```
|
196
|
+
|
197
|
+
## Usage and configuration
|
198
|
+
|
199
|
+
### Client settings
|
200
|
+
|
201
|
+
To configure the Chewy client you need to add `chewy.rb` file with `Chewy.settings` hash:
|
115
202
|
|
116
203
|
```ruby
|
117
204
|
# config/initializers/chewy.rb
|
118
205
|
Chewy.settings = {host: 'localhost:9250'} # do not use environments
|
119
206
|
```
|
120
207
|
|
208
|
+
And add `chewy.yml` configuration file.
|
209
|
+
|
210
|
+
You can create `chewy.yml` manually or run `rails g chewy:install` to generate it:
|
211
|
+
|
121
212
|
```yaml
|
122
213
|
# config/chewy.yml
|
123
214
|
# separate environment configs
|
@@ -167,7 +258,7 @@ Chewy.settings = {
|
|
167
258
|
}
|
168
259
|
```
|
169
260
|
|
170
|
-
|
261
|
+
#### Index definition
|
171
262
|
|
172
263
|
1. Create `/app/chewy/users_index.rb`
|
173
264
|
|
@@ -177,41 +268,38 @@ Chewy.settings = {
|
|
177
268
|
end
|
178
269
|
```
|
179
270
|
|
180
|
-
2.
|
271
|
+
2. Define index scope (you can omit this part if you don't need to specify a scope (i.e. use PORO objects for import) or options)
|
181
272
|
|
182
273
|
```ruby
|
183
274
|
class UsersIndex < Chewy::Index
|
184
|
-
|
275
|
+
index_scope User.active # or just model instead_of scope: index_scope User
|
185
276
|
end
|
186
277
|
```
|
187
278
|
|
188
|
-
|
189
|
-
|
190
|
-
3. Add some type mappings
|
279
|
+
3. Add some mappings
|
191
280
|
|
192
281
|
```ruby
|
193
282
|
class UsersIndex < Chewy::Index
|
194
|
-
|
195
|
-
|
196
|
-
|
197
|
-
|
198
|
-
|
199
|
-
|
200
|
-
|
201
|
-
|
202
|
-
|
203
|
-
|
204
|
-
end
|
205
|
-
field :rating, type: 'integer' # custom data type
|
206
|
-
field :created, type: 'date', include_in_all: false,
|
207
|
-
value: ->{ created_at } # value proc for source object context
|
283
|
+
index_scope User.active.includes(:country, :badges, :projects)
|
284
|
+
field :first_name, :last_name # multiple fields without additional options
|
285
|
+
field :email, analyzer: 'email' # Elasticsearch-related options
|
286
|
+
field :country, value: ->(user) { user.country.name } # custom value proc
|
287
|
+
field :badges, value: ->(user) { user.badges.map(&:name) } # passing array values to index
|
288
|
+
field :projects do # the same block syntax for multi_field, if `:type` is specified
|
289
|
+
field :title
|
290
|
+
field :description # default data type is `text`
|
291
|
+
# additional top-level objects passed to value proc:
|
292
|
+
field :categories, value: ->(project, user) { project.categories.map(&:name) if user.active? }
|
208
293
|
end
|
294
|
+
field :rating, type: 'integer' # custom data type
|
295
|
+
field :created, type: 'date', include_in_all: false,
|
296
|
+
value: ->{ created_at } # value proc for source object context
|
209
297
|
end
|
210
298
|
```
|
211
299
|
|
212
300
|
[See here for mapping definitions](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html).
|
213
301
|
|
214
|
-
4. Add some index-
|
302
|
+
4. Add some index-related settings. Analyzer repositories might be used as well. See `Chewy::Index.settings` docs for details:
|
215
303
|
|
216
304
|
```ruby
|
217
305
|
class UsersIndex < Chewy::Index
|
@@ -224,23 +312,22 @@ Chewy.settings = {
|
|
224
312
|
}
|
225
313
|
}
|
226
314
|
|
227
|
-
|
228
|
-
|
229
|
-
|
230
|
-
|
231
|
-
|
232
|
-
|
233
|
-
|
234
|
-
|
235
|
-
|
236
|
-
|
237
|
-
|
238
|
-
end
|
239
|
-
field :about_translations, type: 'object' # pass object type explicitly if necessary
|
240
|
-
field :rating, type: 'integer'
|
241
|
-
field :created, type: 'date', include_in_all: false,
|
242
|
-
value: ->{ created_at }
|
315
|
+
index_scope User.active.includes(:country, :badges, :projects)
|
316
|
+
root date_detection: false do
|
317
|
+
template 'about_translations.*', type: 'text', analyzer: 'standard'
|
318
|
+
|
319
|
+
field :first_name, :last_name
|
320
|
+
field :email, analyzer: 'email'
|
321
|
+
field :country, value: ->(user) { user.country.name }
|
322
|
+
field :badges, value: ->(user) { user.badges.map(&:name) }
|
323
|
+
field :projects do
|
324
|
+
field :title
|
325
|
+
field :description
|
243
326
|
end
|
327
|
+
field :about_translations, type: 'object' # pass object type explicitly if necessary
|
328
|
+
field :rating, type: 'integer'
|
329
|
+
field :created, type: 'date', include_in_all: false,
|
330
|
+
value: ->{ created_at }
|
244
331
|
end
|
245
332
|
end
|
246
333
|
```
|
@@ -254,39 +341,32 @@ Chewy.settings = {
|
|
254
341
|
|
255
342
|
```ruby
|
256
343
|
class User < ActiveRecord::Base
|
257
|
-
update_index('users
|
344
|
+
update_index('users') { self } # specifying index and back-reference
|
258
345
|
# for updating after user save or destroy
|
259
346
|
end
|
260
347
|
|
261
348
|
class Country < ActiveRecord::Base
|
262
349
|
has_many :users
|
263
350
|
|
264
|
-
update_index('users
|
351
|
+
update_index('users') { users } # return single object or collection
|
265
352
|
end
|
266
353
|
|
267
354
|
class Project < ActiveRecord::Base
|
268
|
-
update_index('users
|
269
|
-
end
|
270
|
-
|
271
|
-
class Badge < ActiveRecord::Base
|
272
|
-
has_and_belongs_to_many :users
|
273
|
-
|
274
|
-
update_index('users') { users } # if index has only one type
|
275
|
-
# there is no need to specify updated type
|
355
|
+
update_index('users') { user if user.active? } # you can return even `nil` from the back-reference
|
276
356
|
end
|
277
357
|
|
278
358
|
class Book < ActiveRecord::Base
|
279
|
-
update_index(->(book) {"
|
280
|
-
|
281
|
-
|
359
|
+
update_index(->(book) {"books_#{book.language}"}) { self } # dynamic index name with proc.
|
360
|
+
# For book with language == "en"
|
361
|
+
# this code will generate `books_en`
|
282
362
|
end
|
283
363
|
```
|
284
364
|
|
285
365
|
Also, you can use the second argument for method name passing:
|
286
366
|
|
287
367
|
```ruby
|
288
|
-
update_index('users
|
289
|
-
update_index('users
|
368
|
+
update_index('users', :self)
|
369
|
+
update_index('users', :users)
|
290
370
|
```
|
291
371
|
|
292
372
|
In the case of a belongs_to association you may need to update both associated objects, previous and current:
|
@@ -295,47 +375,28 @@ Chewy.settings = {
|
|
295
375
|
class City < ActiveRecord::Base
|
296
376
|
belongs_to :country
|
297
377
|
|
298
|
-
update_index('cities
|
299
|
-
update_index 'countries
|
300
|
-
# For the latest active_record changed values are
|
301
|
-
# already in `previous_changes` hash,
|
302
|
-
# but for mongoid you have to use `changes` hash
|
378
|
+
update_index('cities') { self }
|
379
|
+
update_index 'countries' do
|
303
380
|
previous_changes['country_id'] || country
|
304
381
|
end
|
305
382
|
end
|
306
383
|
```
|
307
384
|
|
308
|
-
|
309
|
-
|
310
|
-
```ruby
|
311
|
-
class User < Sequel::Model
|
312
|
-
update_index('users#user') { self }
|
313
|
-
end
|
314
|
-
```
|
315
|
-
|
316
|
-
However, to make it work, you must load the chewy plugin into Sequel model:
|
385
|
+
### Default import options
|
317
386
|
|
318
|
-
|
319
|
-
Sequel::Model.plugin :chewy_observe # for all models, or...
|
320
|
-
User.plugin :chewy_observe # just for User
|
321
|
-
```
|
322
|
-
|
323
|
-
### Type default import options
|
324
|
-
|
325
|
-
Every type has `default_import_options` configuration to specify, suddenly, default import options:
|
387
|
+
Every index has `default_import_options` configuration to specify, suddenly, default import options:
|
326
388
|
|
327
389
|
```ruby
|
328
390
|
class ProductsIndex < Chewy::Index
|
329
|
-
|
330
|
-
|
391
|
+
index_scope Post.includes(:tags)
|
392
|
+
default_import_options batch_size: 100, bulk_size: 10.megabytes, refresh: false
|
331
393
|
|
332
|
-
|
333
|
-
|
334
|
-
end
|
394
|
+
field :name
|
395
|
+
field :tags, value: -> { tags.map(&:name) }
|
335
396
|
end
|
336
397
|
```
|
337
398
|
|
338
|
-
See [import.rb](lib/chewy/
|
399
|
+
See [import.rb](lib/chewy/index/import.rb) for available options.
|
339
400
|
|
340
401
|
### Multi (nested) and object field types
|
341
402
|
|
@@ -386,14 +447,13 @@ Assume you are defining your index like this (product has_many categories throug
|
|
386
447
|
|
387
448
|
```ruby
|
388
449
|
class ProductsIndex < Chewy::Index
|
389
|
-
|
390
|
-
|
391
|
-
|
392
|
-
end
|
450
|
+
index_scope Product.includes(:categories)
|
451
|
+
field :name
|
452
|
+
field :category_names, value: ->(product) { product.categories.map(&:name) } # or shorter just -> { categories.map(&:name) }
|
393
453
|
end
|
394
454
|
```
|
395
455
|
|
396
|
-
Then the Chewy reindexing flow will look like the following pseudo-code
|
456
|
+
Then the Chewy reindexing flow will look like the following pseudo-code:
|
397
457
|
|
398
458
|
```ruby
|
399
459
|
Product.includes(:categories).find_in_batches(1000) do |batch|
|
@@ -405,26 +465,23 @@ Product.includes(:categories).find_in_batches(1000) do |batch|
|
|
405
465
|
end
|
406
466
|
```
|
407
467
|
|
408
|
-
|
409
|
-
|
410
|
-
Then you can replace Rails associations with Chewy Crutches™ technology:
|
468
|
+
If you meet complicated cases when associations are not applicable you can replace Rails associations with Chewy Crutches™ technology:
|
411
469
|
|
412
470
|
```ruby
|
413
471
|
class ProductsIndex < Chewy::Index
|
414
|
-
|
415
|
-
|
416
|
-
|
417
|
-
|
418
|
-
|
419
|
-
|
420
|
-
|
421
|
-
|
422
|
-
end
|
423
|
-
|
424
|
-
field :name
|
425
|
-
# simply use crutch-fetched data as a value:
|
426
|
-
field :category_names, value: ->(product, crutches) { crutches.categories[product.id] }
|
472
|
+
index_scope Product
|
473
|
+
crutch :categories do |collection| # collection here is a current batch of products
|
474
|
+
# data is fetched with a lightweight query without objects initialization
|
475
|
+
data = ProductCategory.joins(:category).where(product_id: collection.map(&:id)).pluck(:product_id, 'categories.name')
|
476
|
+
# then we have to convert fetched data to appropriate format
|
477
|
+
# this will return our data in structure like:
|
478
|
+
# {123 => ['sweets', 'juices'], 456 => ['meat']}
|
479
|
+
data.each.with_object({}) { |(id, name), result| (result[id] ||= []).push(name) }
|
427
480
|
end
|
481
|
+
|
482
|
+
field :name
|
483
|
+
# simply use crutch-fetched data as a value:
|
484
|
+
field :category_names, value: ->(product, crutches) { crutches.categories[product.id] }
|
428
485
|
end
|
429
486
|
```
|
430
487
|
|
@@ -446,22 +503,21 @@ So Chewy Crutches™ technology is able to increase your indexing performance in
|
|
446
503
|
|
447
504
|
### Witchcraft™ technology
|
448
505
|
|
449
|
-
One more experimental technology to increase import performance. As far as you know, chewy defines value proc for every imported field in mapping, so at the import time each of this procs is executed on imported object to extract result document to import. It would be great for performance to use one huge whole-document-returning proc instead. So basically the idea or Witchcraft™ technology is to compile a single document-returning proc from the
|
506
|
+
One more experimental technology to increase import performance. As far as you know, chewy defines value proc for every imported field in mapping, so at the import time each of this procs is executed on imported object to extract result document to import. It would be great for performance to use one huge whole-document-returning proc instead. So basically the idea or Witchcraft™ technology is to compile a single document-returning proc from the index definition.
|
450
507
|
|
451
508
|
```ruby
|
452
|
-
|
453
|
-
|
454
|
-
|
455
|
-
|
456
|
-
|
457
|
-
|
458
|
-
|
459
|
-
|
460
|
-
end
|
509
|
+
index_scope Product
|
510
|
+
witchcraft!
|
511
|
+
|
512
|
+
field :title
|
513
|
+
field :tags, value: -> { tags.map(&:name) }
|
514
|
+
field :categories do
|
515
|
+
field :name, value: -> (product, category) { category.name }
|
516
|
+
field :type, value: -> (product, category, crutch) { crutch.types[category.name] }
|
461
517
|
end
|
462
518
|
```
|
463
519
|
|
464
|
-
The
|
520
|
+
The index definition above will be compiled to something close to:
|
465
521
|
|
466
522
|
```ruby
|
467
523
|
-> (object, crutches) do
|
@@ -491,7 +547,7 @@ Obviously not every type of definition might be compiled. There are some restric
|
|
491
547
|
end
|
492
548
|
```
|
493
549
|
|
494
|
-
However, it is quite possible that your
|
550
|
+
However, it is quite possible that your index definition will be supported by Witchcraft™ technology out of the box in the most of the cases.
|
495
551
|
|
496
552
|
### Raw Import
|
497
553
|
|
@@ -518,13 +574,12 @@ class LightweightProduct
|
|
518
574
|
end
|
519
575
|
end
|
520
576
|
|
521
|
-
|
522
|
-
|
523
|
-
|
524
|
-
|
577
|
+
index_scope Product
|
578
|
+
default_import_options raw_import: ->(hash) {
|
579
|
+
LightweightProduct.new(hash)
|
580
|
+
}
|
525
581
|
|
526
|
-
|
527
|
-
end
|
582
|
+
field :created_at, 'datetime'
|
528
583
|
```
|
529
584
|
|
530
585
|
Also, you can pass `:raw_import` option to the `import` method explicitly.
|
@@ -535,6 +590,24 @@ By default, when you perform import Chewy checks whether an index exists and cre
|
|
535
590
|
You can turn off this feature to decrease Elasticsearch hits count.
|
536
591
|
To do so you need to set `skip_index_creation_on_import` parameter to `false` in your `config/chewy.yml`
|
537
592
|
|
593
|
+
### Skip record fields during import
|
594
|
+
|
595
|
+
You can use `ignore_blank: true` to skip fields that return `true` for the `.blank?` method:
|
596
|
+
|
597
|
+
```ruby
|
598
|
+
index_scope Country
|
599
|
+
field :id
|
600
|
+
field :cities, ignore_blank: true do
|
601
|
+
field :id
|
602
|
+
field :name
|
603
|
+
field :surname, ignore_blank: true
|
604
|
+
field :description
|
605
|
+
end
|
606
|
+
```
|
607
|
+
|
608
|
+
#### Default values for different types
|
609
|
+
|
610
|
+
By default `ignore_blank` is false on every type except `geo_point`.
|
538
611
|
|
539
612
|
### Journaling
|
540
613
|
|
@@ -548,7 +621,6 @@ Common journal record looks like this:
|
|
548
621
|
"action": "index",
|
549
622
|
"object_id": [1, 2, 3],
|
550
623
|
"index_name": "...",
|
551
|
-
"type_name": "...",
|
552
624
|
"created_at": "<timestamp>"
|
553
625
|
}
|
554
626
|
```
|
@@ -574,9 +646,8 @@ Or as a default import option for an index:
|
|
574
646
|
|
575
647
|
```ruby
|
576
648
|
class CityIndex
|
577
|
-
|
578
|
-
|
579
|
-
end
|
649
|
+
index_scope City
|
650
|
+
default_import_options journal: true
|
580
651
|
end
|
581
652
|
```
|
582
653
|
|
@@ -584,19 +655,6 @@ You may be wondering why do you need it? The answer is simple: not to lose the d
|
|
584
655
|
|
585
656
|
Imagine that you reset your index in a zero-downtime manner (to separate index), and at the meantime somebody keeps updating the data frequently (to old index). So all these actions will be written to the journal index and you'll be able to apply them after index reset using the `Chewy::Journal` interface.
|
586
657
|
|
587
|
-
### Types access
|
588
|
-
|
589
|
-
You can access index-defined type with the following API:
|
590
|
-
|
591
|
-
```ruby
|
592
|
-
UsersIndex::User # => UsersIndex::User
|
593
|
-
UsersIndex.type_hash['user'] # => UsersIndex::User
|
594
|
-
UsersIndex.type('user') # => UsersIndex::User
|
595
|
-
UsersIndex.type('foo') # => raises error UndefinedType("Unknown type in UsersIndex: foo")
|
596
|
-
UsersIndex.types # => [UsersIndex::User]
|
597
|
-
UsersIndex.type_names # => ['user']
|
598
|
-
```
|
599
|
-
|
600
658
|
### Index manipulation
|
601
659
|
|
602
660
|
```ruby
|
@@ -609,24 +667,21 @@ UsersIndex.create! # use bang or non-bang methods
|
|
609
667
|
UsersIndex.purge
|
610
668
|
UsersIndex.purge! # deletes then creates index
|
611
669
|
|
612
|
-
UsersIndex
|
613
|
-
|
614
|
-
UsersIndex
|
615
|
-
UsersIndex
|
616
|
-
UsersIndex
|
617
|
-
UsersIndex::User.import User.where('rating > 100'), update_fields: [:email] # if update fields are specified - it will update their values only with the `update` bulk action
|
670
|
+
UsersIndex.import # import with 0 arguments process all the data specified in index_scope definition
|
671
|
+
UsersIndex.import User.where('rating > 100') # or import specified users scope
|
672
|
+
UsersIndex.import User.where('rating > 100').to_a # or import specified users array
|
673
|
+
UsersIndex.import [1, 2, 42] # pass even ids for import, it will be handled in the most effective way
|
674
|
+
UsersIndex.import User.where('rating > 100'), update_fields: [:email] # if update fields are specified - it will update their values only with the `update` bulk action
|
618
675
|
|
619
|
-
UsersIndex.import # same as UsersIndex::User.import
|
620
|
-
UsersIndex.import user: User.where('rating > 100') # import only specified users
|
621
676
|
UsersIndex.reset! # purges index and imports default data for all types
|
622
677
|
```
|
623
678
|
|
624
|
-
If the passed user is `#destroyed?`, or satisfies a `delete_if`
|
679
|
+
If the passed user is `#destroyed?`, or satisfies a `delete_if` index_scope option, or the specified id does not exist in the database, import will perform delete from index action for this object.
|
625
680
|
|
626
681
|
```ruby
|
627
|
-
|
628
|
-
|
629
|
-
|
682
|
+
index_scope User, delete_if: :deleted_at
|
683
|
+
index_scope User, delete_if: -> { deleted_at }
|
684
|
+
index_scope User, delete_if: ->(user) { user.deleted_at }
|
630
685
|
```
|
631
686
|
|
632
687
|
See [actions.rb](lib/chewy/index/actions.rb) for more details.
|
@@ -637,13 +692,12 @@ Assume you've got the following code:
|
|
637
692
|
|
638
693
|
```ruby
|
639
694
|
class City < ActiveRecord::Base
|
640
|
-
update_index 'cities
|
695
|
+
update_index 'cities', :self
|
641
696
|
end
|
642
697
|
|
643
698
|
class CitiesIndex < Chewy::Index
|
644
|
-
|
645
|
-
|
646
|
-
end
|
699
|
+
index_scope City
|
700
|
+
field :name
|
647
701
|
end
|
648
702
|
```
|
649
703
|
|
@@ -663,16 +717,6 @@ end
|
|
663
717
|
|
664
718
|
Using this strategy delays the index update request until the end of the block. Updated records are aggregated and the index update happens with the bulk API. So this strategy is highly optimized.
|
665
719
|
|
666
|
-
#### `:resque`
|
667
|
-
|
668
|
-
This does the same thing as `:atomic`, but asynchronously using resque. The default queue name is `chewy`. Patch `Chewy::Strategy::Resque::Worker` for index updates improving.
|
669
|
-
|
670
|
-
```ruby
|
671
|
-
Chewy.strategy(:resque) do
|
672
|
-
City.popular.map(&:do_some_update_action!)
|
673
|
-
end
|
674
|
-
```
|
675
|
-
|
676
720
|
#### `:sidekiq`
|
677
721
|
|
678
722
|
This does the same thing as `:atomic`, but asynchronously using sidekiq. Patch `Chewy::Strategy::Sidekiq::Worker` for index updates improving.
|
@@ -703,16 +747,6 @@ The default queue name is `chewy`, you can customize it in settings: `active_job
|
|
703
747
|
Chewy.settings[:active_job] = {queue: :low}
|
704
748
|
```
|
705
749
|
|
706
|
-
#### `:shoryuken`
|
707
|
-
|
708
|
-
This does the same thing as `:atomic`, but asynchronously using shoryuken. Patch `Chewy::Strategy::Shoryuken::Worker` for index updates improving.
|
709
|
-
|
710
|
-
```ruby
|
711
|
-
Chewy.strategy(:shoryuken) do
|
712
|
-
City.popular.map(&:do_some_update_action!)
|
713
|
-
end
|
714
|
-
```
|
715
|
-
|
716
750
|
#### `:urgent`
|
717
751
|
|
718
752
|
The following strategy is convenient if you are going to update documents in your index one by one.
|
@@ -788,6 +822,12 @@ RSpec.configure do |config|
|
|
788
822
|
end
|
789
823
|
```
|
790
824
|
|
825
|
+
### Elasticsearch client options
|
826
|
+
|
827
|
+
All connection options, except the `:prefix`, are passed to the `Elasticseach::Client.new` ([chewy/lib/chewy.rb](https://github.com/toptal/chewy/blob/f5bad9f83c21416ac10590f6f34009c645062e89/lib/chewy.rb#L153-L160)):
|
828
|
+
|
829
|
+
Here's the relevant Elasticsearch documentation on the subject: https://rubydoc.info/gems/elasticsearch-transport#setting-hosts
|
830
|
+
|
791
831
|
### `ActiveSupport::Notifications` support
|
792
832
|
|
793
833
|
Chewy has notifying the following events:
|
@@ -799,7 +839,7 @@ Chewy has notifying the following events:
|
|
799
839
|
|
800
840
|
#### `import_objects.chewy` payload
|
801
841
|
|
802
|
-
* `payload[:
|
842
|
+
* `payload[:index]`: currently imported index name
|
803
843
|
* `payload[:import]`: imports stats, total imported and deleted objects count:
|
804
844
|
|
805
845
|
```ruby
|
@@ -903,14 +943,13 @@ Quick introduction.
|
|
903
943
|
|
904
944
|
#### Composing requests
|
905
945
|
|
906
|
-
The request DSL have the same chainable nature as AR
|
946
|
+
The request DSL have the same chainable nature as AR. The main class is `Chewy::Search::Request`.
|
907
947
|
|
908
948
|
```ruby
|
909
|
-
CitiesIndex.query(match: {name: 'London'})
|
910
|
-
CitiesIndex::City.query(match: {name: 'London'})
|
949
|
+
CitiesIndex.query(match: {name: 'London'})
|
911
950
|
```
|
912
951
|
|
913
|
-
Main methods of the request DSL are: `query`, `filter` and `post_filter`, it is possible to pass pure query hashes or use `elasticsearch-dsl`.
|
952
|
+
Main methods of the request DSL are: `query`, `filter` and `post_filter`, it is possible to pass pure query hashes or use `elasticsearch-dsl`.
|
914
953
|
|
915
954
|
```ruby
|
916
955
|
CitiesIndex
|
@@ -939,7 +978,7 @@ Request DSL also provides additional scope actions, like `delete_all`, `exists?`
|
|
939
978
|
|
940
979
|
#### Pagination
|
941
980
|
|
942
|
-
The request DSL supports pagination with `Kaminari
|
981
|
+
The request DSL supports pagination with `Kaminari`. An extension is enabled on initializtion if `Kaminari` is available. See [Chewy::Search](lib/chewy/search.rb) and [Chewy::Search::Pagination::Kaminari](lib/chewy/search/pagination/kaminari.rb) for details.
|
943
982
|
|
944
983
|
#### Named scopes
|
945
984
|
|
@@ -958,7 +997,7 @@ See [Chewy::Search::Scrolling](lib/chewy/search/scrolling.rb) for details.
|
|
958
997
|
It is possible to load ORM/ODM source objects with the `objects` method. To provide additional loading options use `load` method:
|
959
998
|
|
960
999
|
```ruby
|
961
|
-
CitiesIndex.load(scope: -> { active }).to_a # to_a returns `Chewy::
|
1000
|
+
CitiesIndex.load(scope: -> { active }).to_a # to_a returns `Chewy::Index` wrappers.
|
962
1001
|
CitiesIndex.load(scope: -> { active }).objects # An array of AR source objects.
|
963
1002
|
```
|
964
1003
|
|
@@ -1008,35 +1047,33 @@ rake chewy:upgrade[-users,cities] # upgrades every index in the application exce
|
|
1008
1047
|
|
1009
1048
|
It doesn't create indexes, it simply imports everything to the existing ones and fails if the index was not created before.
|
1010
1049
|
|
1011
|
-
Unlike `reset` or `upgrade` tasks, it is possible to pass type references to update the particular type. In index name is passed without the type specified, it will update all the types defined for this index.
|
1012
|
-
|
1013
1050
|
```bash
|
1014
1051
|
rake chewy:update # updates all the existing indices
|
1015
1052
|
rake chewy:update[users] # updates UsersIndex only
|
1016
|
-
rake chewy:update[users,cities
|
1017
|
-
rake chewy:update[-users,cities
|
1053
|
+
rake chewy:update[users,cities] # updates UsersIndex and CitiesIndex
|
1054
|
+
rake chewy:update[-users,cities] # updates every index in the application except UsersIndex and CitiesIndex
|
1018
1055
|
```
|
1019
1056
|
|
1020
1057
|
#### `chewy:sync`
|
1021
1058
|
|
1022
|
-
Provides a way to synchronize outdated indexes with the source quickly and without doing a full reset. By default field `updated_at` is used to find outdated records, but this could be customized by `outdated_sync_field` as described at [Chewy::
|
1059
|
+
Provides a way to synchronize outdated indexes with the source quickly and without doing a full reset. By default field `updated_at` is used to find outdated records, but this could be customized by `outdated_sync_field` as described at [Chewy::Index::Syncer](lib/chewy/index/syncer.rb).
|
1023
1060
|
|
1024
|
-
Arguments are similar to the ones taken by `chewy:update` task.
|
1061
|
+
Arguments are similar to the ones taken by `chewy:update` task.
|
1025
1062
|
|
1026
|
-
See [Chewy::
|
1063
|
+
See [Chewy::Index::Syncer](lib/chewy/index/syncer.rb) for more details.
|
1027
1064
|
|
1028
1065
|
```bash
|
1029
1066
|
rake chewy:sync # synchronizes all the existing indices
|
1030
1067
|
rake chewy:sync[users] # synchronizes UsersIndex only
|
1031
|
-
rake chewy:sync[users,cities
|
1032
|
-
rake chewy:sync[-users,cities
|
1068
|
+
rake chewy:sync[users,cities] # synchronizes UsersIndex and CitiesIndex
|
1069
|
+
rake chewy:sync[-users,cities] # synchronizes every index in the application except except UsersIndex and CitiesIndex
|
1033
1070
|
```
|
1034
1071
|
|
1035
1072
|
#### `chewy:deploy`
|
1036
1073
|
|
1037
1074
|
This rake task is especially useful during the production deploy. It is a combination of `chewy:upgrade` and `chewy:sync` and the latter is called only for the indexes that were not reset during the first stage.
|
1038
1075
|
|
1039
|
-
It is not possible to specify any particular
|
1076
|
+
It is not possible to specify any particular indexes for this task as it doesn't make much sense.
|
1040
1077
|
|
1041
1078
|
Right now the approach is that if some data had been updated, but index definition was not changed (no changes satisfying the synchronization algorithm were done), it would be much faster to perform manual partial index update inside data migrations or even manually after the deploy.
|
1042
1079
|
|
@@ -1053,14 +1090,14 @@ If the number of processes is not specified explicitly - `parallel` gem tries to
|
|
1053
1090
|
```bash
|
1054
1091
|
rake chewy:parallel:reset
|
1055
1092
|
rake chewy:parallel:upgrade[4]
|
1056
|
-
rake chewy:parallel:update[4,cities
|
1093
|
+
rake chewy:parallel:update[4,cities]
|
1057
1094
|
rake chewy:parallel:sync[4,-users]
|
1058
1095
|
rake chewy:parallel:deploy[4] # performs parallel upgrade and parallel sync afterwards
|
1059
1096
|
```
|
1060
1097
|
|
1061
1098
|
#### `chewy:journal`
|
1062
1099
|
|
1063
|
-
This namespace contains two tasks for the journal manipulations: `chewy:journal:apply` and `chewy:journal:clean`. Both are taking time as the first argument (optional for clean) and a list of indexes
|
1100
|
+
This namespace contains two tasks for the journal manipulations: `chewy:journal:apply` and `chewy:journal:clean`. Both are taking time as the first argument (optional for clean) and a list of indexes exactly as the tasks above. Time can be in any format parsable by ActiveSupport.
|
1064
1101
|
|
1065
1102
|
```bash
|
1066
1103
|
rake chewy:journal:apply["$(date -v-1H -u +%FT%TZ)"] # apply journaled changes for the past hour
|
@@ -1097,7 +1134,7 @@ Chewy.use_after_commit_callbacks = !Rails.env.test?
|
|
1097
1134
|
5. Push to the branch (`git push origin my-new-feature`)
|
1098
1135
|
6. Create new Pull Request
|
1099
1136
|
|
1100
|
-
Use the following Rake tasks to control the Elasticsearch cluster while developing
|
1137
|
+
Use the following Rake tasks to control the Elasticsearch cluster while developing, if you prefer native Elasticsearch installation over the dockerized one:
|
1101
1138
|
|
1102
1139
|
```bash
|
1103
1140
|
rake elasticsearch:start # start Elasticsearch cluster on 9250 port for tests
|