searchkick 4.4.0 → 5.3.1
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/CHANGELOG.md +160 -3
- data/LICENSE.txt +1 -1
- data/README.md +567 -421
- data/lib/searchkick/bulk_reindex_job.rb +12 -8
- data/lib/searchkick/controller_runtime.rb +40 -0
- data/lib/searchkick/index.rb +167 -74
- data/lib/searchkick/index_cache.rb +30 -0
- data/lib/searchkick/index_options.rb +465 -404
- data/lib/searchkick/indexer.rb +15 -8
- data/lib/searchkick/log_subscriber.rb +57 -0
- data/lib/searchkick/middleware.rb +9 -2
- data/lib/searchkick/model.rb +50 -51
- data/lib/searchkick/process_batch_job.rb +9 -25
- data/lib/searchkick/process_queue_job.rb +4 -3
- data/lib/searchkick/query.rb +106 -77
- data/lib/searchkick/record_data.rb +1 -1
- data/lib/searchkick/record_indexer.rb +136 -51
- data/lib/searchkick/reindex_queue.rb +51 -9
- data/lib/searchkick/reindex_v2_job.rb +10 -34
- data/lib/searchkick/relation.rb +247 -0
- data/lib/searchkick/relation_indexer.rb +155 -0
- data/lib/searchkick/results.rb +131 -96
- data/lib/searchkick/version.rb +1 -1
- data/lib/searchkick/where.rb +11 -0
- data/lib/searchkick.rb +202 -96
- data/lib/tasks/searchkick.rake +14 -10
- metadata +18 -85
- data/CONTRIBUTING.md +0 -53
- data/lib/searchkick/bulk_indexer.rb +0 -173
- data/lib/searchkick/logging.rb +0 -246
data/README.md
CHANGED
@@ -20,43 +20,51 @@ Plus:
|
|
20
20
|
- autocomplete
|
21
21
|
- “Did you mean” suggestions
|
22
22
|
- supports many languages
|
23
|
-
- works with
|
23
|
+
- works with Active Record and Mongoid
|
24
24
|
|
25
|
-
|
25
|
+
Check out [Searchjoy](https://github.com/ankane/searchjoy) for analytics and [Autosuggest](https://github.com/ankane/autosuggest) for query suggestions
|
26
26
|
|
27
|
-
:
|
27
|
+
:tangerine: Battle-tested at [Instacart](https://www.instacart.com/opensource)
|
28
28
|
|
29
|
-
[![Build Status](https://
|
29
|
+
[![Build Status](https://github.com/ankane/searchkick/workflows/build/badge.svg?branch=master)](https://github.com/ankane/searchkick/actions)
|
30
30
|
|
31
31
|
## Contents
|
32
32
|
|
33
33
|
- [Getting Started](#getting-started)
|
34
34
|
- [Querying](#querying)
|
35
35
|
- [Indexing](#indexing)
|
36
|
+
- [Intelligent Search](#intelligent-search)
|
36
37
|
- [Instant Search / Autocomplete](#instant-search--autocomplete)
|
37
38
|
- [Aggregations](#aggregations)
|
39
|
+
- [Testing](#testing)
|
38
40
|
- [Deployment](#deployment)
|
39
41
|
- [Performance](#performance)
|
40
|
-
- [
|
42
|
+
- [Advanced Search](#advanced)
|
41
43
|
- [Reference](#reference)
|
42
|
-
- [
|
44
|
+
- [Contributing](#contributing)
|
43
45
|
|
44
46
|
## Getting Started
|
45
47
|
|
46
|
-
|
48
|
+
Install [Elasticsearch](https://www.elastic.co/downloads/elasticsearch) or [OpenSearch](https://opensearch.org/downloads.html). For Homebrew, use:
|
47
49
|
|
48
50
|
```sh
|
49
|
-
brew install elasticsearch
|
50
|
-
brew services start elasticsearch
|
51
|
+
brew install elastic/tap/elasticsearch-full
|
52
|
+
brew services start elasticsearch-full
|
53
|
+
# or
|
54
|
+
brew install opensearch
|
55
|
+
brew services start opensearch
|
51
56
|
```
|
52
57
|
|
53
|
-
Add
|
58
|
+
Add these lines to your application’s Gemfile:
|
54
59
|
|
55
60
|
```ruby
|
56
|
-
gem
|
61
|
+
gem "searchkick"
|
62
|
+
|
63
|
+
gem "elasticsearch" # select one
|
64
|
+
gem "opensearch-ruby" # select one
|
57
65
|
```
|
58
66
|
|
59
|
-
The latest version works with Elasticsearch
|
67
|
+
The latest version works with Elasticsearch 7 and 8 and OpenSearch 1 and 2. For Elasticsearch 6, use version 4.6.3 and [this readme](https://github.com/ankane/searchkick/blob/v4.6.3/README.md).
|
60
68
|
|
61
69
|
Add searchkick to models you want to search.
|
62
70
|
|
@@ -81,14 +89,14 @@ products.each do |product|
|
|
81
89
|
end
|
82
90
|
```
|
83
91
|
|
84
|
-
Searchkick supports the complete [Elasticsearch Search API](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-search.html). As your search becomes more advanced, we recommend you use the [
|
92
|
+
Searchkick supports the complete [Elasticsearch Search API](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-search.html) and [OpenSearch Search API](https://opensearch.org/docs/latest/opensearch/rest-api/search/). As your search becomes more advanced, we recommend you use the [search server DSL](#advanced) for maximum flexibility.
|
85
93
|
|
86
94
|
## Querying
|
87
95
|
|
88
96
|
Query like SQL
|
89
97
|
|
90
98
|
```ruby
|
91
|
-
Product.search
|
99
|
+
Product.search("apples", where: {in_stock: true}, limit: 10, offset: 50)
|
92
100
|
```
|
93
101
|
|
94
102
|
Search specific fields
|
@@ -101,19 +109,20 @@ Where
|
|
101
109
|
|
102
110
|
```ruby
|
103
111
|
where: {
|
104
|
-
expires_at: {gt: Time.now},
|
105
|
-
orders_count: 1..10,
|
106
|
-
aisle_id: [25, 30],
|
107
|
-
store_id: {not: 2},
|
108
|
-
aisle_id: {not: [25, 30]},
|
109
|
-
user_ids: {all: [1, 3]},
|
110
|
-
category: {like: "%frozen%"},
|
111
|
-
category:
|
112
|
-
category:
|
113
|
-
|
112
|
+
expires_at: {gt: Time.now}, # lt, gte, lte also available
|
113
|
+
orders_count: 1..10, # equivalent to {gte: 1, lte: 10}
|
114
|
+
aisle_id: [25, 30], # in
|
115
|
+
store_id: {not: 2}, # not
|
116
|
+
aisle_id: {not: [25, 30]}, # not in
|
117
|
+
user_ids: {all: [1, 3]}, # all elements in array
|
118
|
+
category: {like: "%frozen%"}, # like
|
119
|
+
category: {ilike: "%frozen%"}, # ilike
|
120
|
+
category: /frozen .+/, # regexp
|
121
|
+
category: {prefix: "frozen"}, # prefix
|
122
|
+
store_id: {exists: true}, # exists
|
114
123
|
_or: [{in_stock: true}, {backordered: true}],
|
115
124
|
_and: [{in_stock: true}, {backordered: true}],
|
116
|
-
_not: {store_id: 1}
|
125
|
+
_not: {store_id: 1} # negate a condition
|
117
126
|
}
|
118
127
|
```
|
119
128
|
|
@@ -123,7 +132,7 @@ Order
|
|
123
132
|
order: {_score: :desc} # most relevant first - default
|
124
133
|
```
|
125
134
|
|
126
|
-
[All of these sort options are supported](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-
|
135
|
+
[All of these sort options are supported](https://www.elastic.co/guide/en/elasticsearch/reference/current/sort-search-results.html)
|
127
136
|
|
128
137
|
Limit / offset
|
129
138
|
|
@@ -137,11 +146,11 @@ Select
|
|
137
146
|
select: [:name]
|
138
147
|
```
|
139
148
|
|
140
|
-
[These source filtering options are supported](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-
|
149
|
+
[These source filtering options are supported](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-fields.html#source-filtering)
|
141
150
|
|
142
151
|
### Results
|
143
152
|
|
144
|
-
Searches return a `Searchkick::
|
153
|
+
Searches return a `Searchkick::Relation` object. This responds like an array to most methods.
|
145
154
|
|
146
155
|
```ruby
|
147
156
|
results = Product.search("milk")
|
@@ -150,7 +159,7 @@ results.any?
|
|
150
159
|
results.each { |result| ... }
|
151
160
|
```
|
152
161
|
|
153
|
-
By default, ids are fetched from
|
162
|
+
By default, ids are fetched from the search server and records are fetched from your database. To fetch everything from the search server, use:
|
154
163
|
|
155
164
|
```ruby
|
156
165
|
Product.search("apples", load: false)
|
@@ -168,13 +177,13 @@ Get the time the search took (in milliseconds)
|
|
168
177
|
results.took
|
169
178
|
```
|
170
179
|
|
171
|
-
Get the full response from
|
180
|
+
Get the full response from the search server
|
172
181
|
|
173
182
|
```ruby
|
174
183
|
results.response
|
175
184
|
```
|
176
185
|
|
177
|
-
**Note:** By default, Elasticsearch [
|
186
|
+
**Note:** By default, Elasticsearch and OpenSearch [limit paging](#deep-paging) to the first 10,000 results for performance. This applies to the total count as well.
|
178
187
|
|
179
188
|
### Boosting
|
180
189
|
|
@@ -207,7 +216,7 @@ boost_by_recency: {created_at: {scale: "7d", decay: 0.5}}
|
|
207
216
|
|
208
217
|
You can also boost by:
|
209
218
|
|
210
|
-
- [Conversions](#
|
219
|
+
- [Conversions](#intelligent-search)
|
211
220
|
- [Distance](#boost-by-distance)
|
212
221
|
|
213
222
|
### Get Everything
|
@@ -215,7 +224,7 @@ You can also boost by:
|
|
215
224
|
Use a `*` for the query.
|
216
225
|
|
217
226
|
```ruby
|
218
|
-
Product.search
|
227
|
+
Product.search("*")
|
219
228
|
```
|
220
229
|
|
221
230
|
### Pagination
|
@@ -224,7 +233,7 @@ Plays nicely with kaminari and will_paginate.
|
|
224
233
|
|
225
234
|
```ruby
|
226
235
|
# controller
|
227
|
-
@products = Product.search
|
236
|
+
@products = Product.search("milk", page: params[:page], per_page: 20)
|
228
237
|
```
|
229
238
|
|
230
239
|
View with kaminari
|
@@ -244,13 +253,13 @@ View with will_paginate
|
|
244
253
|
By default, results must match all words in the query.
|
245
254
|
|
246
255
|
```ruby
|
247
|
-
Product.search
|
256
|
+
Product.search("fresh honey") # fresh AND honey
|
248
257
|
```
|
249
258
|
|
250
259
|
To change this, use:
|
251
260
|
|
252
261
|
```ruby
|
253
|
-
Product.search
|
262
|
+
Product.search("fresh honey", operator: "or") # fresh OR honey
|
254
263
|
```
|
255
264
|
|
256
265
|
By default, results must match the entire word - `back` will not match `backpack`. You can change this behavior with:
|
@@ -264,7 +273,7 @@ end
|
|
264
273
|
And to search (after you reindex):
|
265
274
|
|
266
275
|
```ruby
|
267
|
-
Product.search
|
276
|
+
Product.search("back", fields: [:name], match: :word_start)
|
268
277
|
```
|
269
278
|
|
270
279
|
Available options are:
|
@@ -281,12 +290,18 @@ Option | Matches | Example
|
|
281
290
|
|
282
291
|
The default is `:word`. The most matches will happen with `:word_middle`.
|
283
292
|
|
293
|
+
To specify different matching for different fields, use:
|
294
|
+
|
295
|
+
```ruby
|
296
|
+
Product.search(query, fields: [{name: :word_start}, {brand: :word_middle}])
|
297
|
+
```
|
298
|
+
|
284
299
|
### Exact Matches
|
285
300
|
|
286
301
|
To match a field exactly (case-sensitive), use:
|
287
302
|
|
288
303
|
```ruby
|
289
|
-
|
304
|
+
Product.search(query, fields: [{name: :exact}])
|
290
305
|
```
|
291
306
|
|
292
307
|
### Phrase Matches
|
@@ -294,10 +309,12 @@ User.search query, fields: [{email: :exact}, :name]
|
|
294
309
|
To only match the exact order, use:
|
295
310
|
|
296
311
|
```ruby
|
297
|
-
|
312
|
+
Product.search("fresh honey", match: :phrase)
|
298
313
|
```
|
299
314
|
|
300
|
-
### Language
|
315
|
+
### Stemming and Language
|
316
|
+
|
317
|
+
Searchkick stems words by default for better matching. `apple` and `apples` both stem to `appl`, so searches for either term will have the same matches.
|
301
318
|
|
302
319
|
Searchkick defaults to English for stemming. To change this, use:
|
303
320
|
|
@@ -307,19 +324,49 @@ class Product < ApplicationRecord
|
|
307
324
|
end
|
308
325
|
```
|
309
326
|
|
310
|
-
|
311
|
-
|
312
|
-
A few languages require plugins:
|
327
|
+
See the [list of languages](https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-stemmer-tokenfilter.html#analysis-stemmer-tokenfilter-configure-parms). A few languages require plugins:
|
313
328
|
|
314
329
|
- `chinese` - [analysis-ik plugin](https://github.com/medcl/elasticsearch-analysis-ik)
|
315
|
-
- `chinese2` - [analysis-smartcn plugin](https://www.elastic.co/guide/en/elasticsearch/plugins/
|
316
|
-
- `japanese` - [analysis-kuromoji plugin](https://www.elastic.co/guide/en/elasticsearch/plugins/
|
330
|
+
- `chinese2` - [analysis-smartcn plugin](https://www.elastic.co/guide/en/elasticsearch/plugins/current/analysis-smartcn.html)
|
331
|
+
- `japanese` - [analysis-kuromoji plugin](https://www.elastic.co/guide/en/elasticsearch/plugins/current/analysis-kuromoji.html)
|
317
332
|
- `korean` - [analysis-openkoreantext plugin](https://github.com/open-korean-text/elasticsearch-analysis-openkoreantext)
|
318
|
-
- `korean2` - [analysis-nori plugin](https://www.elastic.co/guide/en/elasticsearch/plugins/
|
319
|
-
- `polish` - [analysis-stempel plugin](https://www.elastic.co/guide/en/elasticsearch/plugins/
|
333
|
+
- `korean2` - [analysis-nori plugin](https://www.elastic.co/guide/en/elasticsearch/plugins/current/analysis-nori.html)
|
334
|
+
- `polish` - [analysis-stempel plugin](https://www.elastic.co/guide/en/elasticsearch/plugins/current/analysis-stempel.html)
|
320
335
|
- `ukrainian` - [analysis-ukrainian plugin](https://www.elastic.co/guide/en/elasticsearch/plugins/7.4/analysis-ukrainian.html)
|
321
336
|
- `vietnamese` - [analysis-vietnamese plugin](https://github.com/duydo/elasticsearch-analysis-vietnamese)
|
322
337
|
|
338
|
+
You can also use a Hunspell dictionary for stemming.
|
339
|
+
|
340
|
+
```ruby
|
341
|
+
class Product < ApplicationRecord
|
342
|
+
searchkick stemmer: {type: "hunspell", locale: "en_US"}
|
343
|
+
end
|
344
|
+
```
|
345
|
+
|
346
|
+
Disable stemming with:
|
347
|
+
|
348
|
+
```ruby
|
349
|
+
class Image < ApplicationRecord
|
350
|
+
searchkick stem: false
|
351
|
+
end
|
352
|
+
```
|
353
|
+
|
354
|
+
Exclude certain words from stemming with:
|
355
|
+
|
356
|
+
```ruby
|
357
|
+
class Image < ApplicationRecord
|
358
|
+
searchkick stem_exclusion: ["apples"]
|
359
|
+
end
|
360
|
+
```
|
361
|
+
|
362
|
+
Or change how words are stemmed:
|
363
|
+
|
364
|
+
```ruby
|
365
|
+
class Image < ApplicationRecord
|
366
|
+
searchkick stemmer_override: ["apples => other"]
|
367
|
+
end
|
368
|
+
```
|
369
|
+
|
323
370
|
### Synonyms
|
324
371
|
|
325
372
|
```ruby
|
@@ -333,16 +380,16 @@ Call `Product.reindex` after changing synonyms. Synonyms are applied at search t
|
|
333
380
|
For directional synonyms, use:
|
334
381
|
|
335
382
|
```ruby
|
336
|
-
|
383
|
+
search_synonyms: ["lightbulb => halogenlamp"]
|
337
384
|
```
|
338
385
|
|
339
386
|
### Dynamic Synonyms
|
340
387
|
|
341
388
|
The above approach works well when your synonym list is static, but in practice, this is often not the case. When you analyze search conversions, you often want to add new synonyms without a full reindex.
|
342
389
|
|
343
|
-
#### Elasticsearch 7.3+
|
390
|
+
#### Elasticsearch 7.3+ and OpenSearch
|
344
391
|
|
345
|
-
For Elasticsearch 7.3
|
392
|
+
For Elasticsearch 7.3+ and OpenSearch, we recommend placing synonyms in a file on the search server (in the `config` directory). This allows you to reload synonyms without reindexing.
|
346
393
|
|
347
394
|
```txt
|
348
395
|
pop, soda
|
@@ -352,16 +399,12 @@ burger, hamburger
|
|
352
399
|
Then use:
|
353
400
|
|
354
401
|
```ruby
|
355
|
-
|
356
|
-
|
357
|
-
|
358
|
-
Add [elasticsearch-xpack](https://github.com/elastic/elasticsearch-ruby/tree/master/elasticsearch-xpack) to your Gemfile:
|
359
|
-
|
360
|
-
```ruby
|
361
|
-
gem 'elasticsearch-xpack', '>= 7.8.0.pre'
|
402
|
+
class Product < ApplicationRecord
|
403
|
+
searchkick search_synonyms: "synonyms.txt"
|
404
|
+
end
|
362
405
|
```
|
363
406
|
|
364
|
-
And
|
407
|
+
And reload with:
|
365
408
|
|
366
409
|
```ruby
|
367
410
|
Product.search_index.reload_synonyms
|
@@ -387,7 +430,7 @@ end
|
|
387
430
|
Search with:
|
388
431
|
|
389
432
|
```ruby
|
390
|
-
Product.search
|
433
|
+
Product.search(query, fields: [:name_tagged])
|
391
434
|
```
|
392
435
|
|
393
436
|
### Misspellings
|
@@ -397,13 +440,13 @@ By default, Searchkick handles misspelled queries by returning results with an [
|
|
397
440
|
You can change this with:
|
398
441
|
|
399
442
|
```ruby
|
400
|
-
Product.search
|
443
|
+
Product.search("zucini", misspellings: {edit_distance: 2}) # zucchini
|
401
444
|
```
|
402
445
|
|
403
446
|
To prevent poor precision and improve performance for correctly spelled queries (which should be a majority for most applications), Searchkick can first perform a search without misspellings, and if there are too few results, perform another with them.
|
404
447
|
|
405
448
|
```ruby
|
406
|
-
Product.search
|
449
|
+
Product.search("zuchini", misspellings: {below: 5})
|
407
450
|
```
|
408
451
|
|
409
452
|
If there are fewer than 5 results, a 2nd search is performed with misspellings enabled. The result of this query is returned.
|
@@ -411,13 +454,13 @@ If there are fewer than 5 results, a 2nd search is performed with misspellings e
|
|
411
454
|
Turn off misspellings with:
|
412
455
|
|
413
456
|
```ruby
|
414
|
-
Product.search
|
457
|
+
Product.search("zuchini", misspellings: false) # no zucchini
|
415
458
|
```
|
416
459
|
|
417
460
|
Specify which fields can include misspellings with:
|
418
461
|
|
419
462
|
```ruby
|
420
|
-
Product.search
|
463
|
+
Product.search("zucini", fields: [:name, :color], misspellings: {fields: [:name]})
|
421
464
|
```
|
422
465
|
|
423
466
|
> When doing this, you must also specify fields to search
|
@@ -427,7 +470,7 @@ Product.search "zucini", fields: [:name, :color], misspellings: {fields: [:name]
|
|
427
470
|
If a user searches `butter`, they may also get results for `peanut butter`. To prevent this, use:
|
428
471
|
|
429
472
|
```ruby
|
430
|
-
Product.search
|
473
|
+
Product.search("butter", exclude: ["peanut butter"])
|
431
474
|
```
|
432
475
|
|
433
476
|
You can map queries and terms to exclude with:
|
@@ -438,7 +481,7 @@ exclude_queries = {
|
|
438
481
|
"cream" => ["ice cream", "whipped cream"]
|
439
482
|
}
|
440
483
|
|
441
|
-
Product.search
|
484
|
+
Product.search(query, exclude: exclude_queries[query])
|
442
485
|
```
|
443
486
|
|
444
487
|
You can demote results by boosting by a factor less than one:
|
@@ -454,13 +497,13 @@ Search :ice_cream::cake: and get `ice cream cake`!
|
|
454
497
|
Add this line to your application’s Gemfile:
|
455
498
|
|
456
499
|
```ruby
|
457
|
-
gem
|
500
|
+
gem "gemoji-parser"
|
458
501
|
```
|
459
502
|
|
460
503
|
And use:
|
461
504
|
|
462
505
|
```ruby
|
463
|
-
Product.search
|
506
|
+
Product.search("🍨🍰", emoji: true)
|
464
507
|
```
|
465
508
|
|
466
509
|
## Indexing
|
@@ -489,12 +532,10 @@ class Product < ApplicationRecord
|
|
489
532
|
end
|
490
533
|
```
|
491
534
|
|
492
|
-
By default, all records are indexed. To control which records are indexed, use the `should_index?` method
|
535
|
+
By default, all records are indexed. To control which records are indexed, use the `should_index?` method.
|
493
536
|
|
494
537
|
```ruby
|
495
538
|
class Product < ApplicationRecord
|
496
|
-
scope :search_import, -> { where(active: true) }
|
497
|
-
|
498
539
|
def should_index?
|
499
540
|
active # only index active records
|
500
541
|
end
|
@@ -521,7 +562,7 @@ For large data sets, try [parallel reindexing](#parallel-reindexing).
|
|
521
562
|
|
522
563
|
- app starts
|
523
564
|
|
524
|
-
###
|
565
|
+
### Strategies
|
525
566
|
|
526
567
|
There are four strategies for keeping the index synced with your database.
|
527
568
|
|
@@ -555,11 +596,19 @@ There are four strategies for keeping the index synced with your database.
|
|
555
596
|
end
|
556
597
|
```
|
557
598
|
|
599
|
+
And reindex a record or relation manually.
|
600
|
+
|
601
|
+
```ruby
|
602
|
+
product.reindex
|
603
|
+
# or
|
604
|
+
store.products.reindex(mode: :async)
|
605
|
+
```
|
606
|
+
|
558
607
|
You can also do bulk updates.
|
559
608
|
|
560
609
|
```ruby
|
561
610
|
Searchkick.callbacks(:bulk) do
|
562
|
-
|
611
|
+
Product.find_each(&:update_fields)
|
563
612
|
end
|
564
613
|
```
|
565
614
|
|
@@ -567,10 +616,16 @@ Or temporarily skip updates.
|
|
567
616
|
|
568
617
|
```ruby
|
569
618
|
Searchkick.callbacks(false) do
|
570
|
-
|
619
|
+
Product.find_each(&:update_fields)
|
571
620
|
end
|
572
621
|
```
|
573
622
|
|
623
|
+
Or override the model’s strategy.
|
624
|
+
|
625
|
+
```ruby
|
626
|
+
product.reindex(mode: :async) # :inline or :queue
|
627
|
+
```
|
628
|
+
|
574
629
|
### Associations
|
575
630
|
|
576
631
|
Data is **not** automatically synced when an association is updated. If this is desired, add a callback to reindex:
|
@@ -587,58 +642,155 @@ class Image < ApplicationRecord
|
|
587
642
|
end
|
588
643
|
```
|
589
644
|
|
590
|
-
###
|
645
|
+
### Default Scopes
|
591
646
|
|
592
|
-
|
593
|
-
|
594
|
-
[Searchjoy](https://github.com/ankane/searchjoy) makes it easy.
|
647
|
+
If you have a default scope that filters records, use the `should_index?` method to exclude them from indexing:
|
595
648
|
|
596
649
|
```ruby
|
597
|
-
Product
|
650
|
+
class Product < ApplicationRecord
|
651
|
+
default_scope { where(deleted_at: nil) }
|
652
|
+
|
653
|
+
def should_index?
|
654
|
+
deleted_at.nil?
|
655
|
+
end
|
656
|
+
end
|
598
657
|
```
|
599
658
|
|
600
|
-
|
659
|
+
If you want to index and search filtered records, set:
|
601
660
|
|
602
|
-
|
661
|
+
```ruby
|
662
|
+
class Product < ApplicationRecord
|
663
|
+
searchkick unscope: true
|
664
|
+
end
|
665
|
+
```
|
603
666
|
|
604
|
-
|
605
|
-
- top searches with no results
|
667
|
+
## Intelligent Search
|
606
668
|
|
607
|
-
|
669
|
+
The best starting point to improve your search **by far** is to track searches and conversions. [Searchjoy](https://github.com/ankane/searchjoy) makes it easy.
|
608
670
|
|
609
|
-
|
671
|
+
```ruby
|
672
|
+
Product.search("apple", track: {user_id: current_user.id})
|
673
|
+
```
|
610
674
|
|
611
|
-
|
675
|
+
[See the docs](https://github.com/ankane/searchjoy) for how to install and use. Focus on top searches with a low conversion rate.
|
612
676
|
|
613
|
-
Searchkick
|
677
|
+
Searchkick can then use the conversion data to learn what users are looking for. If a user searches for “ice cream” and adds Ben & Jerry’s Chunky Monkey to the cart (our conversion metric at Instacart), that item gets a little more weight for similar searches. This can make a huge difference on the quality of your search.
|
614
678
|
|
615
|
-
|
679
|
+
Add conversion data with:
|
616
680
|
|
617
681
|
```ruby
|
618
682
|
class Product < ApplicationRecord
|
619
|
-
has_many :
|
683
|
+
has_many :conversions, class_name: "Searchjoy::Conversion", as: :convertable
|
684
|
+
has_many :searches, class_name: "Searchjoy::Search", through: :conversions
|
620
685
|
|
621
686
|
searchkick conversions: [:conversions] # name of field
|
622
687
|
|
623
688
|
def search_data
|
624
689
|
{
|
625
690
|
name: name,
|
626
|
-
conversions: searches.group(:query).
|
691
|
+
conversions: searches.group(:query).distinct.count(:user_id)
|
627
692
|
# {"ice cream" => 234, "chocolate" => 67, "cream" => 2}
|
628
693
|
}
|
629
694
|
end
|
630
695
|
end
|
631
696
|
```
|
632
697
|
|
633
|
-
Reindex and set up a cron job to add new conversions daily.
|
698
|
+
Reindex and set up a cron job to add new conversions daily. For zero downtime deployment, temporarily set `conversions: false` in your search calls until the data is reindexed.
|
699
|
+
|
700
|
+
### Performant Conversions
|
701
|
+
|
702
|
+
A performant way to do conversions is to cache them to prevent N+1 queries. For Postgres, create a migration with:
|
634
703
|
|
635
704
|
```ruby
|
636
|
-
|
705
|
+
add_column :products, :search_conversions, :jsonb
|
637
706
|
```
|
638
707
|
|
639
|
-
|
708
|
+
For MySQL, use `:json`, and for others, use `:text` with a [JSON serializer](https://api.rubyonrails.org/classes/ActiveRecord/AttributeMethods/Serialization/ClassMethods.html).
|
709
|
+
|
710
|
+
Next, update your model. Create a separate method for conversion data so you can use [partial reindexing](#partial-reindexing).
|
711
|
+
|
712
|
+
```ruby
|
713
|
+
class Product < ApplicationRecord
|
714
|
+
searchkick conversions: [:conversions]
|
715
|
+
|
716
|
+
def search_data
|
717
|
+
{
|
718
|
+
name: name,
|
719
|
+
category: category
|
720
|
+
}.merge(conversions_data)
|
721
|
+
end
|
640
722
|
|
641
|
-
|
723
|
+
def conversions_data
|
724
|
+
{
|
725
|
+
conversions: search_conversions || {}
|
726
|
+
}
|
727
|
+
end
|
728
|
+
end
|
729
|
+
```
|
730
|
+
|
731
|
+
Deploy and reindex your data. For zero downtime deployment, temporarily set `conversions: false` in your search calls until the data is reindexed.
|
732
|
+
|
733
|
+
```ruby
|
734
|
+
Product.reindex
|
735
|
+
```
|
736
|
+
|
737
|
+
Then, create a job to update the conversions column and reindex records with new conversions. Here’s one you can use for Searchjoy:
|
738
|
+
|
739
|
+
```ruby
|
740
|
+
class UpdateConversionsJob < ApplicationJob
|
741
|
+
def perform(class_name, since: nil, update: true, reindex: true)
|
742
|
+
model = Searchkick.load_model(class_name)
|
743
|
+
|
744
|
+
# get records that have a recent conversion
|
745
|
+
recently_converted_ids =
|
746
|
+
Searchjoy::Conversion.where(convertable_type: class_name).where(created_at: since..)
|
747
|
+
.order(:convertable_id).distinct.pluck(:convertable_id)
|
748
|
+
|
749
|
+
# split into batches
|
750
|
+
recently_converted_ids.in_groups_of(1000, false) do |ids|
|
751
|
+
if update
|
752
|
+
# fetch conversions
|
753
|
+
conversions =
|
754
|
+
Searchjoy::Conversion.where(convertable_id: ids, convertable_type: class_name)
|
755
|
+
.joins(:search).where.not(searchjoy_searches: {user_id: nil})
|
756
|
+
.group(:convertable_id, :query).distinct.count(:user_id)
|
757
|
+
|
758
|
+
# group by record
|
759
|
+
conversions_by_record = {}
|
760
|
+
conversions.each do |(id, query), count|
|
761
|
+
(conversions_by_record[id] ||= {})[query] = count
|
762
|
+
end
|
763
|
+
|
764
|
+
# update conversions column
|
765
|
+
model.transaction do
|
766
|
+
conversions_by_record.each do |id, conversions|
|
767
|
+
model.where(id: id).update_all(search_conversions: conversions)
|
768
|
+
end
|
769
|
+
end
|
770
|
+
end
|
771
|
+
|
772
|
+
if reindex
|
773
|
+
# reindex conversions data
|
774
|
+
model.where(id: ids).reindex(:conversions_data)
|
775
|
+
end
|
776
|
+
end
|
777
|
+
end
|
778
|
+
end
|
779
|
+
```
|
780
|
+
|
781
|
+
Run the job:
|
782
|
+
|
783
|
+
```ruby
|
784
|
+
UpdateConversionsJob.perform_now("Product")
|
785
|
+
```
|
786
|
+
|
787
|
+
And set it up to run daily.
|
788
|
+
|
789
|
+
```ruby
|
790
|
+
UpdateConversionsJob.perform_later("Product", since: 1.day.ago)
|
791
|
+
```
|
792
|
+
|
793
|
+
## Personalized Results
|
642
794
|
|
643
795
|
Order results differently for each user. For example, show a user’s previously purchased products before other results.
|
644
796
|
|
@@ -656,10 +808,10 @@ end
|
|
656
808
|
Reindex and search with:
|
657
809
|
|
658
810
|
```ruby
|
659
|
-
Product.search
|
811
|
+
Product.search("milk", boost_where: {orderer_ids: current_user.id})
|
660
812
|
```
|
661
813
|
|
662
|
-
|
814
|
+
## Instant Search / Autocomplete
|
663
815
|
|
664
816
|
Autocomplete predicts what a user will type, making the search experience faster and easier.
|
665
817
|
|
@@ -680,7 +832,7 @@ end
|
|
680
832
|
Reindex and search with:
|
681
833
|
|
682
834
|
```ruby
|
683
|
-
Movie.search
|
835
|
+
Movie.search("jurassic pa", fields: [:title], match: :word_start)
|
684
836
|
```
|
685
837
|
|
686
838
|
Typically, you want to use a JavaScript library like [typeahead.js](https://twitter.github.io/typeahead.js/) or [jQuery UI](https://jqueryui.com/autocomplete/).
|
@@ -727,7 +879,7 @@ Then add the search box and JavaScript code to a view.
|
|
727
879
|
</script>
|
728
880
|
```
|
729
881
|
|
730
|
-
|
882
|
+
## Suggestions
|
731
883
|
|
732
884
|
![Suggest](https://gist.github.com/ankane/b6988db2802aca68a589b31e41b44195/raw/40febe948427e5bc53ec4e5dc248822855fef76f/recursion.png)
|
733
885
|
|
@@ -740,51 +892,51 @@ end
|
|
740
892
|
Reindex and search with:
|
741
893
|
|
742
894
|
```ruby
|
743
|
-
products = Product.search
|
895
|
+
products = Product.search("peantu butta", suggest: true)
|
744
896
|
products.suggestions # ["peanut butter"]
|
745
897
|
```
|
746
898
|
|
747
|
-
|
899
|
+
## Aggregations
|
748
900
|
|
749
901
|
[Aggregations](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations.html) provide aggregated search data.
|
750
902
|
|
751
903
|
![Aggregations](https://gist.github.com/ankane/b6988db2802aca68a589b31e41b44195/raw/40febe948427e5bc53ec4e5dc248822855fef76f/facets.png)
|
752
904
|
|
753
905
|
```ruby
|
754
|
-
products = Product.search
|
906
|
+
products = Product.search("chuck taylor", aggs: [:product_type, :gender, :brand])
|
755
907
|
products.aggs
|
756
908
|
```
|
757
909
|
|
758
910
|
By default, `where` conditions apply to aggregations.
|
759
911
|
|
760
912
|
```ruby
|
761
|
-
Product.search
|
913
|
+
Product.search("wingtips", where: {color: "brandy"}, aggs: [:size])
|
762
914
|
# aggregations for brandy wingtips are returned
|
763
915
|
```
|
764
916
|
|
765
917
|
Change this with:
|
766
918
|
|
767
919
|
```ruby
|
768
|
-
Product.search
|
920
|
+
Product.search("wingtips", where: {color: "brandy"}, aggs: [:size], smart_aggs: false)
|
769
921
|
# aggregations for all wingtips are returned
|
770
922
|
```
|
771
923
|
|
772
924
|
Set `where` conditions for each aggregation separately with:
|
773
925
|
|
774
926
|
```ruby
|
775
|
-
Product.search
|
927
|
+
Product.search("wingtips", aggs: {size: {where: {color: "brandy"}}})
|
776
928
|
```
|
777
929
|
|
778
930
|
Limit
|
779
931
|
|
780
932
|
```ruby
|
781
|
-
Product.search
|
933
|
+
Product.search("apples", aggs: {store_id: {limit: 10}})
|
782
934
|
```
|
783
935
|
|
784
936
|
Order
|
785
937
|
|
786
938
|
```ruby
|
787
|
-
Product.search
|
939
|
+
Product.search("wingtips", aggs: {color: {order: {"_key" => "asc"}}}) # alphabetically
|
788
940
|
```
|
789
941
|
|
790
942
|
[All of these options are supported](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-terms-aggregation.html#search-aggregations-bucket-terms-aggregation-order)
|
@@ -793,39 +945,39 @@ Ranges
|
|
793
945
|
|
794
946
|
```ruby
|
795
947
|
price_ranges = [{to: 20}, {from: 20, to: 50}, {from: 50}]
|
796
|
-
Product.search
|
948
|
+
Product.search("*", aggs: {price: {ranges: price_ranges}})
|
797
949
|
```
|
798
950
|
|
799
951
|
Minimum document count
|
800
952
|
|
801
953
|
```ruby
|
802
|
-
Product.search
|
954
|
+
Product.search("apples", aggs: {store_id: {min_doc_count: 2}})
|
803
955
|
```
|
804
956
|
|
805
957
|
Script support
|
806
958
|
|
807
959
|
```ruby
|
808
|
-
Product.search
|
960
|
+
Product.search("*", aggs: {color: {script: {source: "'Color: ' + _value"}}})
|
809
961
|
```
|
810
962
|
|
811
963
|
Date histogram
|
812
964
|
|
813
965
|
```ruby
|
814
|
-
Product.search
|
966
|
+
Product.search("pear", aggs: {products_per_year: {date_histogram: {field: :created_at, interval: :year}}})
|
815
967
|
```
|
816
968
|
|
817
969
|
For other aggregation types, including sub-aggregations, use `body_options`:
|
818
970
|
|
819
971
|
```ruby
|
820
|
-
Product.search
|
972
|
+
Product.search("orange", body_options: {aggs: {price: {histogram: {field: :price, interval: 10}}}})
|
821
973
|
```
|
822
974
|
|
823
|
-
|
975
|
+
## Highlight
|
824
976
|
|
825
977
|
Specify which fields to index with highlighting.
|
826
978
|
|
827
979
|
```ruby
|
828
|
-
class
|
980
|
+
class Band < ApplicationRecord
|
829
981
|
searchkick highlight: [:name]
|
830
982
|
end
|
831
983
|
```
|
@@ -833,7 +985,7 @@ end
|
|
833
985
|
Highlight the search query in the results.
|
834
986
|
|
835
987
|
```ruby
|
836
|
-
bands = Band.search
|
988
|
+
bands = Band.search("cinema", highlight: true)
|
837
989
|
```
|
838
990
|
|
839
991
|
View the highlighted fields with:
|
@@ -847,19 +999,19 @@ end
|
|
847
999
|
To change the tag, use:
|
848
1000
|
|
849
1001
|
```ruby
|
850
|
-
Band.search
|
1002
|
+
Band.search("cinema", highlight: {tag: "<strong>"})
|
851
1003
|
```
|
852
1004
|
|
853
1005
|
To highlight and search different fields, use:
|
854
1006
|
|
855
1007
|
```ruby
|
856
|
-
Band.search
|
1008
|
+
Band.search("cinema", fields: [:name], highlight: {fields: [:description]})
|
857
1009
|
```
|
858
1010
|
|
859
1011
|
By default, the entire field is highlighted. To get small snippets instead, use:
|
860
1012
|
|
861
1013
|
```ruby
|
862
|
-
bands = Band.search
|
1014
|
+
bands = Band.search("cinema", highlight: {fragment_size: 20})
|
863
1015
|
bands.with_highlights(multiple: true).each do |band, highlights|
|
864
1016
|
highlights[:name].join(" and ")
|
865
1017
|
end
|
@@ -868,12 +1020,12 @@ end
|
|
868
1020
|
Additional options can be specified for each field:
|
869
1021
|
|
870
1022
|
```ruby
|
871
|
-
Band.search
|
1023
|
+
Band.search("cinema", fields: [:name], highlight: {fields: {name: {fragment_size: 200}}})
|
872
1024
|
```
|
873
1025
|
|
874
|
-
You can find available highlight options in the [Elasticsearch reference](https://www.elastic.co/guide/en/elasticsearch/reference/current/
|
1026
|
+
You can find available highlight options in the [Elasticsearch reference](https://www.elastic.co/guide/en/elasticsearch/reference/current/highlighting.html).
|
875
1027
|
|
876
|
-
|
1028
|
+
## Similar Items
|
877
1029
|
|
878
1030
|
Find similar items.
|
879
1031
|
|
@@ -882,7 +1034,7 @@ product = Product.first
|
|
882
1034
|
product.similar(fields: [:name], where: {size: "12 oz"})
|
883
1035
|
```
|
884
1036
|
|
885
|
-
|
1037
|
+
## Geospatial Searches
|
886
1038
|
|
887
1039
|
```ruby
|
888
1040
|
class Restaurant < ApplicationRecord
|
@@ -897,13 +1049,13 @@ end
|
|
897
1049
|
Reindex and search with:
|
898
1050
|
|
899
1051
|
```ruby
|
900
|
-
Restaurant.search
|
1052
|
+
Restaurant.search("pizza", where: {location: {near: {lat: 37, lon: -114}, within: "100mi"}}) # or 160km
|
901
1053
|
```
|
902
1054
|
|
903
1055
|
Bounded by a box
|
904
1056
|
|
905
1057
|
```ruby
|
906
|
-
Restaurant.search
|
1058
|
+
Restaurant.search("sushi", where: {location: {top_left: {lat: 38, lon: -123}, bottom_right: {lat: 37, lon: -122}}})
|
907
1059
|
```
|
908
1060
|
|
909
1061
|
**Note:** `top_right` and `bottom_left` also work
|
@@ -911,7 +1063,7 @@ Restaurant.search "sushi", where: {location: {top_left: {lat: 38, lon: -123}, bo
|
|
911
1063
|
Bounded by a polygon
|
912
1064
|
|
913
1065
|
```ruby
|
914
|
-
Restaurant.search
|
1066
|
+
Restaurant.search("dessert", where: {location: {geo_polygon: {points: [{lat: 38, lon: -123}, {lat: 39, lon: -123}, {lat: 37, lon: 122}]}}})
|
915
1067
|
```
|
916
1068
|
|
917
1069
|
### Boost By Distance
|
@@ -919,13 +1071,13 @@ Restaurant.search "dessert", where: {location: {geo_polygon: {points: [{lat: 38,
|
|
919
1071
|
Boost results by distance - closer results are boosted more
|
920
1072
|
|
921
1073
|
```ruby
|
922
|
-
Restaurant.search
|
1074
|
+
Restaurant.search("noodles", boost_by_distance: {location: {origin: {lat: 37, lon: -122}}})
|
923
1075
|
```
|
924
1076
|
|
925
|
-
Also supports [additional options](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-function-score-query.html#
|
1077
|
+
Also supports [additional options](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-function-score-query.html#function-decay)
|
926
1078
|
|
927
1079
|
```ruby
|
928
|
-
Restaurant.search
|
1080
|
+
Restaurant.search("wings", boost_by_distance: {location: {origin: {lat: 37, lon: -122}, function: "linear", scale: "30mi", decay: 0.5}})
|
929
1081
|
```
|
930
1082
|
|
931
1083
|
### Geo Shapes
|
@@ -952,19 +1104,19 @@ See the [Elasticsearch documentation](https://www.elastic.co/guide/en/elasticsea
|
|
952
1104
|
Find shapes intersecting with the query shape
|
953
1105
|
|
954
1106
|
```ruby
|
955
|
-
Restaurant.search
|
1107
|
+
Restaurant.search("soup", where: {bounds: {geo_shape: {type: "polygon", coordinates: [[{lat: 38, lon: -123}, ...]]}}})
|
956
1108
|
```
|
957
1109
|
|
958
1110
|
Falling entirely within the query shape
|
959
1111
|
|
960
1112
|
```ruby
|
961
|
-
Restaurant.search
|
1113
|
+
Restaurant.search("salad", where: {bounds: {geo_shape: {type: "circle", relation: "within", coordinates: {lat: 38, lon: -123}, radius: "1km"}}})
|
962
1114
|
```
|
963
1115
|
|
964
1116
|
Not touching the query shape
|
965
1117
|
|
966
1118
|
```ruby
|
967
|
-
Restaurant.search
|
1119
|
+
Restaurant.search("burger", where: {bounds: {geo_shape: {type: "envelope", relation: "disjoint", coordinates: [{lat: 38, lon: -123}, {lat: 37, lon: -122}]}}})
|
968
1120
|
```
|
969
1121
|
|
970
1122
|
## Inheritance
|
@@ -994,9 +1146,9 @@ Dog.reindex # equivalent, all animals reindexed
|
|
994
1146
|
And to search, use:
|
995
1147
|
|
996
1148
|
```ruby
|
997
|
-
Animal.search
|
998
|
-
Dog.search
|
999
|
-
Animal.search
|
1149
|
+
Animal.search("*") # all animals
|
1150
|
+
Dog.search("*") # just dogs
|
1151
|
+
Animal.search("*", type: [Dog, Cat]) # just cats and dogs
|
1000
1152
|
```
|
1001
1153
|
|
1002
1154
|
**Notes:**
|
@@ -1004,7 +1156,7 @@ Animal.search "*", type: [Dog, Cat] # just cats and dogs
|
|
1004
1156
|
1. The `suggest` option retrieves suggestions from the parent at the moment.
|
1005
1157
|
|
1006
1158
|
```ruby
|
1007
|
-
Dog.search
|
1159
|
+
Dog.search("airbudd", suggest: true) # suggestions for all animals
|
1008
1160
|
```
|
1009
1161
|
2. This relies on a `type` field that is automatically added to the indexed document. Be wary of defining your own `type` field in `search_data`, as it will take precedence.
|
1010
1162
|
|
@@ -1018,13 +1170,13 @@ Product.search("soap", debug: true)
|
|
1018
1170
|
|
1019
1171
|
This prints useful info to `stdout`.
|
1020
1172
|
|
1021
|
-
See how
|
1173
|
+
See how the search server scores your queries with:
|
1022
1174
|
|
1023
1175
|
```ruby
|
1024
1176
|
Product.search("soap", explain: true).response
|
1025
1177
|
```
|
1026
1178
|
|
1027
|
-
See how
|
1179
|
+
See how the search server tokenizes your queries with:
|
1028
1180
|
|
1029
1181
|
```ruby
|
1030
1182
|
Product.search_index.tokens("Dish Washer Soap", analyzer: "searchkick_index")
|
@@ -1047,23 +1199,196 @@ Product.search_index.tokens("dieg", analyzer: "searchkick_word_search")
|
|
1047
1199
|
# ["dieg"] - match!!
|
1048
1200
|
```
|
1049
1201
|
|
1050
|
-
See the [complete list of analyzers](
|
1202
|
+
See the [complete list of analyzers](lib/searchkick/index_options.rb#L36).
|
1203
|
+
|
1204
|
+
## Testing
|
1205
|
+
|
1206
|
+
As you iterate on your search, it’s a good idea to add tests.
|
1207
|
+
|
1208
|
+
For performance, only enable Searchkick callbacks for the tests that need it.
|
1209
|
+
|
1210
|
+
### Parallel Tests
|
1211
|
+
|
1212
|
+
Rails 6 enables parallel tests by default. Add to your `test/test_helper.rb`:
|
1213
|
+
|
1214
|
+
```ruby
|
1215
|
+
class ActiveSupport::TestCase
|
1216
|
+
parallelize_setup do |worker|
|
1217
|
+
Searchkick.index_suffix = worker
|
1218
|
+
|
1219
|
+
# reindex models
|
1220
|
+
Product.reindex
|
1221
|
+
|
1222
|
+
# and disable callbacks
|
1223
|
+
Searchkick.disable_callbacks
|
1224
|
+
end
|
1225
|
+
end
|
1226
|
+
```
|
1227
|
+
|
1228
|
+
And use:
|
1229
|
+
|
1230
|
+
```ruby
|
1231
|
+
class ProductTest < ActiveSupport::TestCase
|
1232
|
+
def setup
|
1233
|
+
Searchkick.enable_callbacks
|
1234
|
+
end
|
1235
|
+
|
1236
|
+
def teardown
|
1237
|
+
Searchkick.disable_callbacks
|
1238
|
+
end
|
1239
|
+
|
1240
|
+
def test_search
|
1241
|
+
Product.create!(name: "Apple")
|
1242
|
+
Product.search_index.refresh
|
1243
|
+
assert_equal ["Apple"], Product.search("apple").map(&:name)
|
1244
|
+
end
|
1245
|
+
end
|
1246
|
+
```
|
1247
|
+
|
1248
|
+
### Minitest
|
1249
|
+
|
1250
|
+
Add to your `test/test_helper.rb`:
|
1251
|
+
|
1252
|
+
```ruby
|
1253
|
+
# reindex models
|
1254
|
+
Product.reindex
|
1255
|
+
|
1256
|
+
# and disable callbacks
|
1257
|
+
Searchkick.disable_callbacks
|
1258
|
+
```
|
1259
|
+
|
1260
|
+
And use:
|
1261
|
+
|
1262
|
+
```ruby
|
1263
|
+
class ProductTest < Minitest::Test
|
1264
|
+
def setup
|
1265
|
+
Searchkick.enable_callbacks
|
1266
|
+
end
|
1267
|
+
|
1268
|
+
def teardown
|
1269
|
+
Searchkick.disable_callbacks
|
1270
|
+
end
|
1271
|
+
|
1272
|
+
def test_search
|
1273
|
+
Product.create!(name: "Apple")
|
1274
|
+
Product.search_index.refresh
|
1275
|
+
assert_equal ["Apple"], Product.search("apple").map(&:name)
|
1276
|
+
end
|
1277
|
+
end
|
1278
|
+
```
|
1279
|
+
|
1280
|
+
### RSpec
|
1281
|
+
|
1282
|
+
Add to your `spec/spec_helper.rb`:
|
1283
|
+
|
1284
|
+
```ruby
|
1285
|
+
RSpec.configure do |config|
|
1286
|
+
config.before(:suite) do
|
1287
|
+
# reindex models
|
1288
|
+
Product.reindex
|
1289
|
+
|
1290
|
+
# and disable callbacks
|
1291
|
+
Searchkick.disable_callbacks
|
1292
|
+
end
|
1293
|
+
|
1294
|
+
config.around(:each, search: true) do |example|
|
1295
|
+
Searchkick.callbacks(nil) do
|
1296
|
+
example.run
|
1297
|
+
end
|
1298
|
+
end
|
1299
|
+
end
|
1300
|
+
```
|
1301
|
+
|
1302
|
+
And use:
|
1303
|
+
|
1304
|
+
```ruby
|
1305
|
+
describe Product, search: true do
|
1306
|
+
it "searches" do
|
1307
|
+
Product.create!(name: "Apple")
|
1308
|
+
Product.search_index.refresh
|
1309
|
+
assert_equal ["Apple"], Product.search("apple").map(&:name)
|
1310
|
+
end
|
1311
|
+
end
|
1312
|
+
```
|
1313
|
+
|
1314
|
+
### Factory Bot
|
1315
|
+
|
1316
|
+
Use a trait and an after `create` hook for each indexed model:
|
1317
|
+
|
1318
|
+
```ruby
|
1319
|
+
FactoryBot.define do
|
1320
|
+
factory :product do
|
1321
|
+
# ...
|
1322
|
+
|
1323
|
+
# Note: This should be the last trait in the list so `reindex` is called
|
1324
|
+
# after all the other callbacks complete.
|
1325
|
+
trait :reindex do
|
1326
|
+
after(:create) do |product, _evaluator|
|
1327
|
+
product.reindex(refresh: true)
|
1328
|
+
end
|
1329
|
+
end
|
1330
|
+
end
|
1331
|
+
end
|
1332
|
+
|
1333
|
+
# use it
|
1334
|
+
FactoryBot.create(:product, :some_trait, :reindex, some_attribute: "foo")
|
1335
|
+
```
|
1336
|
+
|
1337
|
+
### GitHub Actions
|
1338
|
+
|
1339
|
+
Check out [setup-elasticsearch](https://github.com/ankane/setup-elasticsearch) for an easy way to install Elasticsearch:
|
1340
|
+
|
1341
|
+
```yml
|
1342
|
+
- uses: ankane/setup-elasticsearch@v1
|
1343
|
+
```
|
1344
|
+
|
1345
|
+
And [setup-opensearch](https://github.com/ankane/setup-opensearch) for an easy way to install OpenSearch:
|
1346
|
+
|
1347
|
+
```yml
|
1348
|
+
- uses: ankane/setup-opensearch@v1
|
1349
|
+
```
|
1051
1350
|
|
1052
1351
|
## Deployment
|
1053
1352
|
|
1054
|
-
Searchkick uses `ENV["ELASTICSEARCH_URL"]` for
|
1353
|
+
For the search server, Searchkick uses `ENV["ELASTICSEARCH_URL"]` for Elasticsearch and `ENV["OPENSEARCH_URL"]` for OpenSearch. This defaults to `http://localhost:9200`.
|
1354
|
+
|
1355
|
+
- [Elastic Cloud](#elastic-cloud)
|
1356
|
+
- [Heroku](#heroku)
|
1357
|
+
- [Amazon OpenSearch Service](#amazon-opensearch-service)
|
1358
|
+
- [Self-Hosted and Other](#self-hosted-and-other)
|
1359
|
+
|
1360
|
+
### Elastic Cloud
|
1361
|
+
|
1362
|
+
Create an initializer `config/initializers/elasticsearch.rb` with:
|
1363
|
+
|
1364
|
+
```ruby
|
1365
|
+
ENV["ELASTICSEARCH_URL"] = "https://user:password@host:port"
|
1366
|
+
```
|
1367
|
+
|
1368
|
+
Then deploy and reindex:
|
1369
|
+
|
1370
|
+
```sh
|
1371
|
+
rake searchkick:reindex:all
|
1372
|
+
```
|
1055
1373
|
|
1056
1374
|
### Heroku
|
1057
1375
|
|
1058
1376
|
Choose an add-on: [Bonsai](https://elements.heroku.com/addons/bonsai), [SearchBox](https://elements.heroku.com/addons/searchbox), or [Elastic Cloud](https://elements.heroku.com/addons/foundelasticsearch).
|
1059
1377
|
|
1060
|
-
For Bonsai:
|
1378
|
+
For Elasticsearch on Bonsai:
|
1061
1379
|
|
1062
1380
|
```sh
|
1063
1381
|
heroku addons:create bonsai
|
1064
1382
|
heroku config:set ELASTICSEARCH_URL=`heroku config:get BONSAI_URL`
|
1065
1383
|
```
|
1066
1384
|
|
1385
|
+
For OpenSearch on Bonsai:
|
1386
|
+
|
1387
|
+
```sh
|
1388
|
+
heroku addons:create bonsai --engine=opensearch
|
1389
|
+
heroku config:set OPENSEARCH_URL=`heroku config:get BONSAI_URL`
|
1390
|
+
```
|
1391
|
+
|
1067
1392
|
For SearchBox:
|
1068
1393
|
|
1069
1394
|
```sh
|
@@ -1093,21 +1418,21 @@ heroku config:set ELASTICSEARCH_URL=https://elastic:password@12345.us-east-1.aws
|
|
1093
1418
|
Then deploy and reindex:
|
1094
1419
|
|
1095
1420
|
```sh
|
1096
|
-
heroku run rake searchkick:reindex
|
1421
|
+
heroku run rake searchkick:reindex:all
|
1097
1422
|
```
|
1098
1423
|
|
1099
|
-
### Amazon
|
1424
|
+
### Amazon OpenSearch Service
|
1100
1425
|
|
1101
|
-
Create an initializer `config/initializers/
|
1426
|
+
Create an initializer `config/initializers/opensearch.rb` with:
|
1102
1427
|
|
1103
1428
|
```ruby
|
1104
|
-
ENV["
|
1429
|
+
ENV["OPENSEARCH_URL"] = "https://es-domain-1234.us-east-1.es.amazonaws.com:443"
|
1105
1430
|
```
|
1106
1431
|
|
1107
1432
|
To use signed requests, include in your Gemfile:
|
1108
1433
|
|
1109
1434
|
```ruby
|
1110
|
-
gem
|
1435
|
+
gem "faraday_middleware-aws-sigv4"
|
1111
1436
|
```
|
1112
1437
|
|
1113
1438
|
and add to your initializer:
|
@@ -1123,38 +1448,50 @@ Searchkick.aws_credentials = {
|
|
1123
1448
|
Then deploy and reindex:
|
1124
1449
|
|
1125
1450
|
```sh
|
1126
|
-
rake searchkick:reindex
|
1451
|
+
rake searchkick:reindex:all
|
1127
1452
|
```
|
1128
1453
|
|
1129
|
-
### Other
|
1454
|
+
### Self-Hosted and Other
|
1130
1455
|
|
1131
|
-
Create an initializer
|
1456
|
+
Create an initializer with:
|
1132
1457
|
|
1133
1458
|
```ruby
|
1134
1459
|
ENV["ELASTICSEARCH_URL"] = "https://user:password@host:port"
|
1460
|
+
# or
|
1461
|
+
ENV["OPENSEARCH_URL"] = "https://user:password@host:port"
|
1135
1462
|
```
|
1136
1463
|
|
1137
1464
|
Then deploy and reindex:
|
1138
1465
|
|
1139
1466
|
```sh
|
1140
|
-
rake searchkick:reindex
|
1467
|
+
rake searchkick:reindex:all
|
1141
1468
|
```
|
1142
1469
|
|
1143
1470
|
### Data Protection
|
1144
1471
|
|
1145
|
-
We recommend encrypting data at rest and in transit (even inside your own network). This is especially important if you send [personal data](https://en.wikipedia.org/wiki/Personally_identifiable_information) of your users to
|
1472
|
+
We recommend encrypting data at rest and in transit (even inside your own network). This is especially important if you send [personal data](https://en.wikipedia.org/wiki/Personally_identifiable_information) of your users to the search server.
|
1146
1473
|
|
1147
|
-
Bonsai, Elastic Cloud, and Amazon
|
1474
|
+
Bonsai, Elastic Cloud, and Amazon OpenSearch Service all support encryption at rest and HTTPS.
|
1148
1475
|
|
1149
1476
|
### Automatic Failover
|
1150
1477
|
|
1151
|
-
Create an initializer
|
1478
|
+
Create an initializer with multiple hosts:
|
1152
1479
|
|
1153
1480
|
```ruby
|
1154
1481
|
ENV["ELASTICSEARCH_URL"] = "https://user:password@host1,https://user:password@host2"
|
1482
|
+
# or
|
1483
|
+
ENV["OPENSEARCH_URL"] = "https://user:password@host1,https://user:password@host2"
|
1155
1484
|
```
|
1156
1485
|
|
1157
|
-
|
1486
|
+
### Client Options
|
1487
|
+
|
1488
|
+
Create an initializer with:
|
1489
|
+
|
1490
|
+
```ruby
|
1491
|
+
Searchkick.client_options[:reload_connections] = true
|
1492
|
+
```
|
1493
|
+
|
1494
|
+
See the docs for [Elasticsearch](https://www.elastic.co/guide/en/elasticsearch/client/ruby-api/current/advanced-config.html) or [Opensearch](https://rubydoc.info/gems/opensearch-transport#configuration) for a complete list of options.
|
1158
1495
|
|
1159
1496
|
### Lograge
|
1160
1497
|
|
@@ -1177,7 +1514,7 @@ See [Production Rails](https://github.com/ankane/production_rails) for other goo
|
|
1177
1514
|
Significantly increase performance with faster JSON generation. Add [Oj](https://github.com/ohler55/oj) to your Gemfile.
|
1178
1515
|
|
1179
1516
|
```ruby
|
1180
|
-
gem
|
1517
|
+
gem "oj"
|
1181
1518
|
```
|
1182
1519
|
|
1183
1520
|
This speeds up all JSON generation and parsing in your application (automatically!)
|
@@ -1187,7 +1524,7 @@ This speeds up all JSON generation and parsing in your application (automaticall
|
|
1187
1524
|
Significantly increase performance with persistent HTTP connections. Add [Typhoeus](https://github.com/typhoeus/typhoeus) to your Gemfile and it’ll automatically be used.
|
1188
1525
|
|
1189
1526
|
```ruby
|
1190
|
-
gem
|
1527
|
+
gem "typhoeus"
|
1191
1528
|
```
|
1192
1529
|
|
1193
1530
|
To reduce log noise, create an initializer with:
|
@@ -1225,7 +1562,7 @@ end
|
|
1225
1562
|
For large data sets, you can use background jobs to parallelize reindexing.
|
1226
1563
|
|
1227
1564
|
```ruby
|
1228
|
-
Product.reindex(
|
1565
|
+
Product.reindex(mode: :async)
|
1229
1566
|
# {index_name: "products_production_20170111210018065"}
|
1230
1567
|
```
|
1231
1568
|
|
@@ -1250,13 +1587,13 @@ Searchkick.reindex_status(index_name)
|
|
1250
1587
|
You can also have Searchkick wait for reindexing to complete
|
1251
1588
|
|
1252
1589
|
```ruby
|
1253
|
-
Product.reindex(
|
1590
|
+
Product.reindex(mode: :async, wait: true)
|
1254
1591
|
```
|
1255
1592
|
|
1256
1593
|
You can use [ActiveJob::TrafficControl](https://github.com/nickelser/activejob-traffic_control) to control concurrency. Install the gem:
|
1257
1594
|
|
1258
1595
|
```ruby
|
1259
|
-
gem
|
1596
|
+
gem "activejob-traffic_control", ">= 0.1.3"
|
1260
1597
|
```
|
1261
1598
|
|
1262
1599
|
And create an initializer with:
|
@@ -1276,7 +1613,7 @@ This will allow only 3 jobs to run at once.
|
|
1276
1613
|
You can specify a longer refresh interval while reindexing to increase performance.
|
1277
1614
|
|
1278
1615
|
```ruby
|
1279
|
-
Product.reindex(
|
1616
|
+
Product.reindex(mode: :async, refresh_interval: "30s")
|
1280
1617
|
```
|
1281
1618
|
|
1282
1619
|
**Note:** This only makes a noticable difference with parallel reindexing.
|
@@ -1289,7 +1626,7 @@ Product.search_index.promote(index_name, update_refresh_interval: true)
|
|
1289
1626
|
|
1290
1627
|
### Queuing
|
1291
1628
|
|
1292
|
-
Push ids of records needing
|
1629
|
+
Push ids of records needing reindexing to a queue and reindex in bulk for better performance. First, set up Redis in an initializer. We recommend using [connection_pool](https://github.com/mperham/connection_pool).
|
1293
1630
|
|
1294
1631
|
```ruby
|
1295
1632
|
Searchkick.redis = ConnectionPool.new { Redis.new }
|
@@ -1319,7 +1656,7 @@ For more tips, check out [Keeping Elasticsearch in Sync](https://www.elastic.co/
|
|
1319
1656
|
|
1320
1657
|
### Routing
|
1321
1658
|
|
1322
|
-
Searchkick supports [
|
1659
|
+
Searchkick supports [routing](https://www.elastic.co/blog/customizing-your-document-routing), which can significantly speed up searches.
|
1323
1660
|
|
1324
1661
|
```ruby
|
1325
1662
|
class Business < ApplicationRecord
|
@@ -1334,7 +1671,7 @@ end
|
|
1334
1671
|
Reindex and search with:
|
1335
1672
|
|
1336
1673
|
```ruby
|
1337
|
-
Business.search
|
1674
|
+
Business.search("ice cream", routing: params[:city_id])
|
1338
1675
|
```
|
1339
1676
|
|
1340
1677
|
### Partial Reindexing
|
@@ -1345,11 +1682,12 @@ Reindex a subset of attributes to reduce time spent generating search data and c
|
|
1345
1682
|
class Product < ApplicationRecord
|
1346
1683
|
def search_data
|
1347
1684
|
{
|
1348
|
-
name: name
|
1349
|
-
|
1685
|
+
name: name,
|
1686
|
+
category: category
|
1687
|
+
}.merge(prices_data)
|
1350
1688
|
end
|
1351
1689
|
|
1352
|
-
def
|
1690
|
+
def prices_data
|
1353
1691
|
{
|
1354
1692
|
price: price,
|
1355
1693
|
sale_price: sale_price
|
@@ -1361,73 +1699,12 @@ end
|
|
1361
1699
|
And use:
|
1362
1700
|
|
1363
1701
|
```ruby
|
1364
|
-
Product.reindex(:
|
1365
|
-
```
|
1366
|
-
|
1367
|
-
### Performant Conversions
|
1368
|
-
|
1369
|
-
Split out conversions into a separate method so you can use partial reindexing, and cache conversions to prevent N+1 queries. Be sure to use a centralized cache store like Memcached or Redis.
|
1370
|
-
|
1371
|
-
```ruby
|
1372
|
-
class Product < ApplicationRecord
|
1373
|
-
def search_data
|
1374
|
-
{
|
1375
|
-
name: name
|
1376
|
-
}.merge(search_conversions)
|
1377
|
-
end
|
1378
|
-
|
1379
|
-
def search_conversions
|
1380
|
-
{
|
1381
|
-
conversions: Rails.cache.read("search_conversions:#{self.class.name}:#{id}") || {}
|
1382
|
-
}
|
1383
|
-
end
|
1384
|
-
end
|
1385
|
-
```
|
1386
|
-
|
1387
|
-
Create a job to update the cache and reindex records with new conversions.
|
1388
|
-
|
1389
|
-
```ruby
|
1390
|
-
class ReindexConversionsJob < ApplicationJob
|
1391
|
-
def perform(class_name)
|
1392
|
-
# get records that have a recent conversion
|
1393
|
-
recently_converted_ids =
|
1394
|
-
Searchjoy::Search.where("convertable_type = ? AND converted_at > ?", class_name, 1.day.ago)
|
1395
|
-
.order(:convertable_id).uniq.pluck(:convertable_id)
|
1396
|
-
|
1397
|
-
# split into groups
|
1398
|
-
recently_converted_ids.in_groups_of(1000, false) do |ids|
|
1399
|
-
# fetch conversions
|
1400
|
-
conversions =
|
1401
|
-
Searchjoy::Search.where(convertable_id: ids, convertable_type: class_name)
|
1402
|
-
.group(:convertable_id, :query).uniq.count(:user_id)
|
1403
|
-
|
1404
|
-
# group conversions by record
|
1405
|
-
conversions_by_record = {}
|
1406
|
-
conversions.each do |(id, query), count|
|
1407
|
-
(conversions_by_record[id] ||= {})[query] = count
|
1408
|
-
end
|
1409
|
-
|
1410
|
-
# write to cache
|
1411
|
-
conversions_by_record.each do |id, conversions|
|
1412
|
-
Rails.cache.write("search_conversions:#{class_name}:#{id}", conversions)
|
1413
|
-
end
|
1414
|
-
|
1415
|
-
# partial reindex
|
1416
|
-
class_name.constantize.where(id: ids).reindex(:search_conversions)
|
1417
|
-
end
|
1418
|
-
end
|
1419
|
-
end
|
1420
|
-
```
|
1421
|
-
|
1422
|
-
Run the job with:
|
1423
|
-
|
1424
|
-
```ruby
|
1425
|
-
ReindexConversionsJob.perform_later("Product")
|
1702
|
+
Product.reindex(:prices_data)
|
1426
1703
|
```
|
1427
1704
|
|
1428
1705
|
## Advanced
|
1429
1706
|
|
1430
|
-
Searchkick makes it easy to use the Elasticsearch DSL on its own.
|
1707
|
+
Searchkick makes it easy to use the Elasticsearch or OpenSearch DSL on its own.
|
1431
1708
|
|
1432
1709
|
### Advanced Mapping
|
1433
1710
|
|
@@ -1457,7 +1734,7 @@ end
|
|
1457
1734
|
And use the `body` option to search:
|
1458
1735
|
|
1459
1736
|
```ruby
|
1460
|
-
products = Product.search
|
1737
|
+
products = Product.search(body: {query: {match: {name: "milk"}}})
|
1461
1738
|
```
|
1462
1739
|
|
1463
1740
|
View the response with:
|
@@ -1469,21 +1746,21 @@ products.response
|
|
1469
1746
|
To modify the query generated by Searchkick, use:
|
1470
1747
|
|
1471
1748
|
```ruby
|
1472
|
-
products = Product.search
|
1749
|
+
products = Product.search("milk", body_options: {min_score: 1})
|
1473
1750
|
```
|
1474
1751
|
|
1475
1752
|
or
|
1476
1753
|
|
1477
1754
|
```ruby
|
1478
1755
|
products =
|
1479
|
-
Product.search
|
1756
|
+
Product.search("apples") do |body|
|
1480
1757
|
body[:min_score] = 1
|
1481
1758
|
end
|
1482
1759
|
```
|
1483
1760
|
|
1484
|
-
###
|
1761
|
+
### Client
|
1485
1762
|
|
1486
|
-
|
1763
|
+
To access the `Elasticsearch::Client` or `OpenSearch::Client` directly, use:
|
1487
1764
|
|
1488
1765
|
```ruby
|
1489
1766
|
Searchkick.client
|
@@ -1494,8 +1771,8 @@ Searchkick.client
|
|
1494
1771
|
To batch search requests for performance, use:
|
1495
1772
|
|
1496
1773
|
```ruby
|
1497
|
-
products = Product.search("snacks"
|
1498
|
-
coupons = Coupon.search("snacks"
|
1774
|
+
products = Product.search("snacks")
|
1775
|
+
coupons = Coupon.search("snacks")
|
1499
1776
|
Searchkick.multi_search([products, coupons])
|
1500
1777
|
```
|
1501
1778
|
|
@@ -1508,7 +1785,7 @@ Then use `products` and `coupons` as typical results.
|
|
1508
1785
|
Search across multiple models with:
|
1509
1786
|
|
1510
1787
|
```ruby
|
1511
|
-
Searchkick.search
|
1788
|
+
Searchkick.search("milk", models: [Product, Category])
|
1512
1789
|
```
|
1513
1790
|
|
1514
1791
|
Boost specific models with:
|
@@ -1517,9 +1794,13 @@ Boost specific models with:
|
|
1517
1794
|
indices_boost: {Category => 2, Product => 1}
|
1518
1795
|
```
|
1519
1796
|
|
1797
|
+
## Multi-Tenancy
|
1798
|
+
|
1799
|
+
Check out [this great post](https://www.tiagoamaro.com.br/2014/12/11/multi-tenancy-with-searchkick/) on the [Apartment](https://github.com/influitive/apartment) gem. Follow a similar pattern if you use another gem.
|
1800
|
+
|
1520
1801
|
## Scroll API
|
1521
1802
|
|
1522
|
-
Searchkick also supports the [scroll API](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-
|
1803
|
+
Searchkick also supports the [scroll API](https://www.elastic.co/guide/en/elasticsearch/reference/current/paginate-search-results.html#scroll-search-results). Scrolling is not intended for real time user requests, but rather for processing large amounts of data.
|
1523
1804
|
|
1524
1805
|
```ruby
|
1525
1806
|
Product.search("*", scroll: "1m").scroll do |batch|
|
@@ -1530,7 +1811,7 @@ end
|
|
1530
1811
|
You can also scroll batches manually.
|
1531
1812
|
|
1532
1813
|
```ruby
|
1533
|
-
products = Product.search
|
1814
|
+
products = Product.search("*", scroll: "1m")
|
1534
1815
|
while products.any?
|
1535
1816
|
# process batch ...
|
1536
1817
|
|
@@ -1542,7 +1823,7 @@ products.clear_scroll
|
|
1542
1823
|
|
1543
1824
|
## Deep Paging
|
1544
1825
|
|
1545
|
-
By default, Elasticsearch
|
1826
|
+
By default, Elasticsearch and OpenSearch limit paging to the first 10,000 results. [Here’s why](https://www.elastic.co/guide/en/elasticsearch/guide/current/pagination.html). We don’t recommend changing this, but if you really need all results, you can use:
|
1546
1827
|
|
1547
1828
|
```ruby
|
1548
1829
|
class Product < ApplicationRecord
|
@@ -1550,7 +1831,7 @@ class Product < ApplicationRecord
|
|
1550
1831
|
end
|
1551
1832
|
```
|
1552
1833
|
|
1553
|
-
If you just need an accurate total count
|
1834
|
+
If you just need an accurate total count, you can instead use:
|
1554
1835
|
|
1555
1836
|
```ruby
|
1556
1837
|
Product.search("pears", body_options: {track_total_hits: true})
|
@@ -1561,9 +1842,13 @@ Product.search("pears", body_options: {track_total_hits: true})
|
|
1561
1842
|
To query nested data, use dot notation.
|
1562
1843
|
|
1563
1844
|
```ruby
|
1564
|
-
|
1845
|
+
Product.search("san", fields: ["store.city"], where: {"store.zip_code" => 12345})
|
1565
1846
|
```
|
1566
1847
|
|
1848
|
+
## Nearest Neighbors
|
1849
|
+
|
1850
|
+
You can use custom mapping and searching to index vectors and perform k-nearest neighbor search. See the examples for [Elasticsearch](examples/elasticsearch_knn.rb) and [OpenSearch](examples/opensearch_knn.rb).
|
1851
|
+
|
1567
1852
|
## Reference
|
1568
1853
|
|
1569
1854
|
Reindex one record
|
@@ -1647,7 +1932,7 @@ class Product < ApplicationRecord
|
|
1647
1932
|
def search_data
|
1648
1933
|
{
|
1649
1934
|
name: name,
|
1650
|
-
unique_user_conversions: searches.group(:query).
|
1935
|
+
unique_user_conversions: searches.group(:query).distinct.count(:user_id),
|
1651
1936
|
# {"ice cream" => 234, "chocolate" => 67, "cream" => 2}
|
1652
1937
|
total_conversions: searches.group(:query).count
|
1653
1938
|
# {"ice cream" => 412, "chocolate" => 117, "cream" => 6}
|
@@ -1691,7 +1976,7 @@ Searchkick.queue_name = :search_reindex
|
|
1691
1976
|
Eager load associations
|
1692
1977
|
|
1693
1978
|
```ruby
|
1694
|
-
Product.search
|
1979
|
+
Product.search("milk", includes: [:brand, :stores])
|
1695
1980
|
```
|
1696
1981
|
|
1697
1982
|
Eager load different associations by model
|
@@ -1703,7 +1988,7 @@ Searchkick.search("*", models: [Product, Store], model_includes: {Product => [:
|
|
1703
1988
|
Run additional scopes on results
|
1704
1989
|
|
1705
1990
|
```ruby
|
1706
|
-
Product.search
|
1991
|
+
Product.search("milk", scope_results: ->(r) { r.with_attached_images })
|
1707
1992
|
```
|
1708
1993
|
|
1709
1994
|
Specify default fields to search
|
@@ -1723,14 +2008,6 @@ class Product < ApplicationRecord
|
|
1723
2008
|
end
|
1724
2009
|
```
|
1725
2010
|
|
1726
|
-
Turn off stemming
|
1727
|
-
|
1728
|
-
```ruby
|
1729
|
-
class Product < ApplicationRecord
|
1730
|
-
searchkick stem: false
|
1731
|
-
end
|
1732
|
-
```
|
1733
|
-
|
1734
2011
|
Turn on stemming for conversions
|
1735
2012
|
|
1736
2013
|
```ruby
|
@@ -1739,14 +2016,6 @@ class Product < ApplicationRecord
|
|
1739
2016
|
end
|
1740
2017
|
```
|
1741
2018
|
|
1742
|
-
Use a different [similarity algorithm](https://www.elastic.co/guide/en/elasticsearch/reference/current/index-modules-similarity.html) for scoring
|
1743
|
-
|
1744
|
-
```ruby
|
1745
|
-
class Product < ApplicationRecord
|
1746
|
-
searchkick similarity: "classic"
|
1747
|
-
end
|
1748
|
-
```
|
1749
|
-
|
1750
2019
|
Make search case-sensitive
|
1751
2020
|
|
1752
2021
|
```ruby
|
@@ -1781,14 +2050,7 @@ class Product < ApplicationRecord
|
|
1781
2050
|
end
|
1782
2051
|
```
|
1783
2052
|
|
1784
|
-
|
1785
|
-
|
1786
|
-
```ruby
|
1787
|
-
products = Product.search("carrots", execute: false)
|
1788
|
-
products.each { ... } # search not executed until here
|
1789
|
-
```
|
1790
|
-
|
1791
|
-
Add [request parameters](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-uri-request.html), like `search_type` and `query_cache`
|
2053
|
+
Add [request parameters](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-search.html#search-search-api-query-params) like `search_type`
|
1792
2054
|
|
1793
2055
|
```ruby
|
1794
2056
|
Product.search("carrots", request_params: {search_type: "dfs_query_then_fetch"})
|
@@ -1822,203 +2084,85 @@ rake searchkick:reindex:all
|
|
1822
2084
|
Turn on misspellings after a certain number of characters
|
1823
2085
|
|
1824
2086
|
```ruby
|
1825
|
-
Product.search
|
2087
|
+
Product.search("api", misspellings: {prefix_length: 2}) # api, apt, no ahi
|
1826
2088
|
```
|
1827
2089
|
|
1828
|
-
**Note:** With this option, if the query length is the same as `prefix_length`, misspellings are turned off
|
2090
|
+
**Note:** With this option, if the query length is the same as `prefix_length`, misspellings are turned off with Elasticsearch 7 and OpenSearch 1
|
1829
2091
|
|
1830
2092
|
```ruby
|
1831
|
-
Product.search
|
2093
|
+
Product.search("ah", misspellings: {prefix_length: 2}) # ah, no aha
|
1832
2094
|
```
|
1833
2095
|
|
1834
|
-
|
1835
|
-
|
1836
|
-
For performance, only enable Searchkick callbacks for the tests that need it.
|
1837
|
-
|
1838
|
-
### Parallel Tests
|
1839
|
-
|
1840
|
-
Rails 6 enables parallel tests by default. Add to your `test/test_helper.rb`:
|
1841
|
-
|
1842
|
-
```ruby
|
1843
|
-
class ActiveSupport::TestCase
|
1844
|
-
parallelize_setup do |worker|
|
1845
|
-
Searchkick.index_suffix = worker
|
1846
|
-
|
1847
|
-
# reindex models
|
1848
|
-
Product.reindex
|
1849
|
-
|
1850
|
-
# and disable callbacks
|
1851
|
-
Searchkick.disable_callbacks
|
1852
|
-
end
|
1853
|
-
end
|
1854
|
-
```
|
1855
|
-
|
1856
|
-
And use:
|
2096
|
+
BigDecimal values are indexed as floats by default so they can be used for boosting. Convert them to strings to keep full precision.
|
1857
2097
|
|
1858
2098
|
```ruby
|
1859
|
-
class
|
1860
|
-
def
|
1861
|
-
|
1862
|
-
|
1863
|
-
|
1864
|
-
def teardown
|
1865
|
-
Searchkick.disable_callbacks
|
1866
|
-
end
|
1867
|
-
|
1868
|
-
def test_search
|
1869
|
-
Product.create!(name: "Apple")
|
1870
|
-
Product.search_index.refresh
|
1871
|
-
assert_equal ["Apple"], Product.search("apple").map(&:name)
|
2099
|
+
class Product < ApplicationRecord
|
2100
|
+
def search_data
|
2101
|
+
{
|
2102
|
+
units: units.to_s("F")
|
2103
|
+
}
|
1872
2104
|
end
|
1873
2105
|
end
|
1874
2106
|
```
|
1875
2107
|
|
1876
|
-
|
1877
|
-
|
1878
|
-
Add to your `test/test_helper.rb`:
|
1879
|
-
|
1880
|
-
```ruby
|
1881
|
-
# reindex models
|
1882
|
-
Product.reindex
|
2108
|
+
## Gotchas
|
1883
2109
|
|
1884
|
-
|
1885
|
-
Searchkick.disable_callbacks
|
1886
|
-
```
|
2110
|
+
### Consistency
|
1887
2111
|
|
1888
|
-
|
2112
|
+
Elasticsearch and OpenSearch are eventually consistent, meaning it can take up to a second for a change to reflect in search. You can use the `refresh` method to have it show up immediately.
|
1889
2113
|
|
1890
2114
|
```ruby
|
1891
|
-
|
1892
|
-
|
1893
|
-
Searchkick.enable_callbacks
|
1894
|
-
end
|
1895
|
-
|
1896
|
-
def teardown
|
1897
|
-
Searchkick.disable_callbacks
|
1898
|
-
end
|
1899
|
-
|
1900
|
-
def test_search
|
1901
|
-
Product.create!(name: "Apple")
|
1902
|
-
Product.search_index.refresh
|
1903
|
-
assert_equal ["Apple"], Product.search("apple").map(&:name)
|
1904
|
-
end
|
1905
|
-
end
|
2115
|
+
product.save!
|
2116
|
+
Product.search_index.refresh
|
1906
2117
|
```
|
1907
2118
|
|
1908
|
-
###
|
2119
|
+
### Inconsistent Scores
|
1909
2120
|
|
1910
|
-
|
2121
|
+
Due to the distributed nature of Elasticsearch and OpenSearch, you can get incorrect results when the number of documents in the index is low. You can [read more about it here](https://www.elastic.co/blog/understanding-query-then-fetch-vs-dfs-query-then-fetch). To fix this, do:
|
1911
2122
|
|
1912
2123
|
```ruby
|
1913
|
-
|
1914
|
-
|
1915
|
-
# reindex models
|
1916
|
-
Product.reindex
|
1917
|
-
|
1918
|
-
# and disable callbacks
|
1919
|
-
Searchkick.disable_callbacks
|
1920
|
-
end
|
1921
|
-
|
1922
|
-
config.around(:each, search: true) do |example|
|
1923
|
-
Searchkick.callbacks(nil) do
|
1924
|
-
example.run
|
1925
|
-
end
|
1926
|
-
end
|
2124
|
+
class Product < ApplicationRecord
|
2125
|
+
searchkick settings: {number_of_shards: 1}
|
1927
2126
|
end
|
1928
2127
|
```
|
1929
2128
|
|
1930
|
-
|
2129
|
+
For convenience, this is set by default in the test environment.
|
1931
2130
|
|
1932
|
-
|
1933
|
-
describe Product, search: true do
|
1934
|
-
it "searches" do
|
1935
|
-
Product.create!(name: "Apple")
|
1936
|
-
Product.search_index.refresh
|
1937
|
-
assert_equal ["Apple"], Product.search("apple").map(&:name)
|
1938
|
-
end
|
1939
|
-
end
|
1940
|
-
```
|
2131
|
+
## Upgrading
|
1941
2132
|
|
1942
|
-
###
|
2133
|
+
### 5.0
|
1943
2134
|
|
1944
|
-
|
2135
|
+
Searchkick 5 supports both the `elasticsearch` and `opensearch-ruby` gems. Add the one you want to use to your Gemfile:
|
1945
2136
|
|
1946
2137
|
```ruby
|
1947
|
-
|
1948
|
-
|
1949
|
-
|
1950
|
-
|
1951
|
-
# Note: This should be the last trait in the list so `reindex` is called
|
1952
|
-
# after all the other callbacks complete.
|
1953
|
-
trait :reindex do
|
1954
|
-
after(:create) do |product, _evaluator|
|
1955
|
-
product.reindex(refresh: true)
|
1956
|
-
end
|
1957
|
-
end
|
1958
|
-
end
|
1959
|
-
end
|
1960
|
-
|
1961
|
-
# use it
|
1962
|
-
FactoryBot.create(:product, :some_trait, :reindex, some_attribute: "foo")
|
2138
|
+
gem "elasticsearch"
|
2139
|
+
# or
|
2140
|
+
gem "opensearch-ruby"
|
1963
2141
|
```
|
1964
2142
|
|
1965
|
-
|
1966
|
-
|
1967
|
-
Check out [this great post](https://www.tiagoamaro.com.br/2014/12/11/multi-tenancy-with-searchkick/) on the [Apartment](https://github.com/influitive/apartment) gem. Follow a similar pattern if you use another gem.
|
1968
|
-
|
1969
|
-
## Upgrading
|
2143
|
+
If using the deprecated `faraday_middleware-aws-signers-v4` gem, switch to `faraday_middleware-aws-sigv4`.
|
1970
2144
|
|
1971
|
-
|
1972
|
-
|
1973
|
-
## Elasticsearch 6 to 7 Upgrade
|
1974
|
-
|
1975
|
-
1. Install Searchkick 4
|
1976
|
-
2. Upgrade your Elasticsearch cluster
|
1977
|
-
|
1978
|
-
## Elasticsearch 5 to 6 Upgrade
|
1979
|
-
|
1980
|
-
Elasticsearch 6 removes the ability to reindex with the `_all` field. Before you upgrade, we recommend disabling this field manually and specifying default fields on your models.
|
2145
|
+
Also, searches now use lazy loading:
|
1981
2146
|
|
1982
2147
|
```ruby
|
1983
|
-
|
1984
|
-
|
1985
|
-
end
|
1986
|
-
```
|
1987
|
-
|
1988
|
-
If you need search across multiple fields, we recommend creating a similar field in your search data.
|
2148
|
+
# search not executed
|
2149
|
+
Product.search("milk")
|
1989
2150
|
|
1990
|
-
|
1991
|
-
|
1992
|
-
def search_data
|
1993
|
-
{
|
1994
|
-
all: [name, size, quantity].join(" ")
|
1995
|
-
}
|
1996
|
-
end
|
1997
|
-
end
|
2151
|
+
# search executed
|
2152
|
+
Product.search("milk").to_a
|
1998
2153
|
```
|
1999
2154
|
|
2000
|
-
|
2001
|
-
|
2002
|
-
### Consistency
|
2003
|
-
|
2004
|
-
Elasticsearch is eventually consistent, meaning it can take up to a second for a change to reflect in search. You can use the `refresh` method to have it show up immediately.
|
2155
|
+
You can reindex relations in the background:
|
2005
2156
|
|
2006
2157
|
```ruby
|
2007
|
-
|
2008
|
-
|
2158
|
+
store.products.reindex(mode: :async)
|
2159
|
+
# or
|
2160
|
+
store.products.reindex(mode: :queue)
|
2009
2161
|
```
|
2010
2162
|
|
2011
|
-
|
2163
|
+
And there’s a [new option](#default-scopes) for models with default scopes.
|
2012
2164
|
|
2013
|
-
|
2014
|
-
|
2015
|
-
```ruby
|
2016
|
-
class Product < ApplicationRecord
|
2017
|
-
searchkick settings: {number_of_shards: 1}
|
2018
|
-
end
|
2019
|
-
```
|
2020
|
-
|
2021
|
-
For convenience, this is set by default in the test environment.
|
2165
|
+
Check out the [changelog](https://github.com/ankane/searchkick/blob/master/CHANGELOG.md#500-2022-02-21) for the full list of changes.
|
2022
2166
|
|
2023
2167
|
## History
|
2024
2168
|
|
@@ -2045,3 +2189,5 @@ cd searchkick
|
|
2045
2189
|
bundle install
|
2046
2190
|
bundle exec rake test
|
2047
2191
|
```
|
2192
|
+
|
2193
|
+
Feel free to open an issue to get feedback on your idea before spending too much time on it.
|