searchkick 4.6.3 → 5.0.2
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/CHANGELOG.md +35 -1
- data/README.md +187 -124
- data/lib/searchkick/bulk_reindex_job.rb +12 -8
- data/lib/searchkick/controller_runtime.rb +40 -0
- data/lib/searchkick/index.rb +146 -65
- data/lib/searchkick/index_cache.rb +30 -0
- data/lib/searchkick/index_options.rb +17 -67
- data/lib/searchkick/indexer.rb +15 -8
- data/lib/searchkick/log_subscriber.rb +57 -0
- data/lib/searchkick/middleware.rb +1 -1
- data/lib/searchkick/model.rb +48 -49
- data/lib/searchkick/process_batch_job.rb +9 -25
- data/lib/searchkick/process_queue_job.rb +3 -2
- data/lib/searchkick/query.rb +38 -54
- data/lib/searchkick/record_data.rb +1 -1
- data/lib/searchkick/record_indexer.rb +136 -52
- data/lib/searchkick/reindex_queue.rb +26 -3
- data/lib/searchkick/reindex_v2_job.rb +10 -34
- data/lib/searchkick/relation.rb +112 -0
- data/lib/searchkick/relation_indexer.rb +150 -0
- data/lib/searchkick/results.rb +27 -28
- data/lib/searchkick/version.rb +1 -1
- data/lib/searchkick.rb +159 -84
- data/lib/tasks/searchkick.rake +6 -3
- metadata +11 -28
- data/lib/searchkick/bulk_indexer.rb +0 -173
- data/lib/searchkick/logging.rb +0 -246
data/README.md
CHANGED
@@ -20,7 +20,7 @@ Plus:
|
|
20
20
|
- autocomplete
|
21
21
|
- “Did you mean” suggestions
|
22
22
|
- supports many languages
|
23
|
-
- works with Active Record
|
23
|
+
- works with Active Record and Mongoid
|
24
24
|
|
25
25
|
Check out [Searchjoy](https://github.com/ankane/searchjoy) for analytics and [Autosuggest](https://github.com/ankane/autosuggest) for query suggestions
|
26
26
|
|
@@ -39,26 +39,34 @@ Check out [Searchjoy](https://github.com/ankane/searchjoy) for analytics and [Au
|
|
39
39
|
- [Testing](#testing)
|
40
40
|
- [Deployment](#deployment)
|
41
41
|
- [Performance](#performance)
|
42
|
-
- [
|
42
|
+
- [Advanced Search](#advanced)
|
43
43
|
- [Reference](#reference)
|
44
44
|
- [Contributing](#contributing)
|
45
45
|
|
46
|
+
Searchkick 5.0 was recently released! See [how to upgrade](#upgrading)
|
47
|
+
|
46
48
|
## Getting Started
|
47
49
|
|
48
50
|
Install [Elasticsearch](https://www.elastic.co/downloads/elasticsearch) or [OpenSearch](https://opensearch.org/downloads.html). For Homebrew, use:
|
49
51
|
|
50
52
|
```sh
|
51
|
-
brew install elasticsearch
|
52
|
-
brew services start elasticsearch
|
53
|
+
brew install elasticsearch
|
54
|
+
brew services start elasticsearch
|
55
|
+
# or
|
56
|
+
brew install opensearch
|
57
|
+
brew services start opensearch
|
53
58
|
```
|
54
59
|
|
55
|
-
Add
|
60
|
+
Add these lines to your application’s Gemfile:
|
56
61
|
|
57
62
|
```ruby
|
58
|
-
gem
|
63
|
+
gem "searchkick"
|
64
|
+
|
65
|
+
gem "elasticsearch" # select one
|
66
|
+
gem "opensearch-ruby" # select one
|
59
67
|
```
|
60
68
|
|
61
|
-
The latest version works with Elasticsearch
|
69
|
+
The latest version works with Elasticsearch 7 and 8 and OpenSearch 1. For Elasticsearch 6, use version 4.6.3 and [this readme](https://github.com/ankane/searchkick/blob/v4.6.3/README.md).
|
62
70
|
|
63
71
|
Add searchkick to models you want to search.
|
64
72
|
|
@@ -83,14 +91,14 @@ products.each do |product|
|
|
83
91
|
end
|
84
92
|
```
|
85
93
|
|
86
|
-
Searchkick supports the complete [Elasticsearch Search API](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-search.html). As your search becomes more advanced, we recommend you use the [
|
94
|
+
Searchkick supports the complete [Elasticsearch Search API](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-search.html) and [OpenSearch Search API](https://opensearch.org/docs/latest/opensearch/rest-api/search/). As your search becomes more advanced, we recommend you use the [search server DSL](#advanced) for maximum flexibility.
|
87
95
|
|
88
96
|
## Querying
|
89
97
|
|
90
98
|
Query like SQL
|
91
99
|
|
92
100
|
```ruby
|
93
|
-
Product.search
|
101
|
+
Product.search("apples", where: {in_stock: true}, limit: 10, offset: 50)
|
94
102
|
```
|
95
103
|
|
96
104
|
Search specific fields
|
@@ -144,7 +152,7 @@ select: [:name]
|
|
144
152
|
|
145
153
|
### Results
|
146
154
|
|
147
|
-
Searches return a `Searchkick::
|
155
|
+
Searches return a `Searchkick::Relation` object. This responds like an array to most methods.
|
148
156
|
|
149
157
|
```ruby
|
150
158
|
results = Product.search("milk")
|
@@ -153,7 +161,7 @@ results.any?
|
|
153
161
|
results.each { |result| ... }
|
154
162
|
```
|
155
163
|
|
156
|
-
By default, ids are fetched from
|
164
|
+
By default, ids are fetched from the search server and records are fetched from your database. To fetch everything from the search server, use:
|
157
165
|
|
158
166
|
```ruby
|
159
167
|
Product.search("apples", load: false)
|
@@ -171,13 +179,13 @@ Get the time the search took (in milliseconds)
|
|
171
179
|
results.took
|
172
180
|
```
|
173
181
|
|
174
|
-
Get the full response from
|
182
|
+
Get the full response from the search server
|
175
183
|
|
176
184
|
```ruby
|
177
185
|
results.response
|
178
186
|
```
|
179
187
|
|
180
|
-
**Note:** By default, Elasticsearch [
|
188
|
+
**Note:** By default, Elasticsearch and OpenSearch [limit paging](#deep-paging) to the first 10,000 results for performance. This applies to the total count as well.
|
181
189
|
|
182
190
|
### Boosting
|
183
191
|
|
@@ -218,7 +226,7 @@ You can also boost by:
|
|
218
226
|
Use a `*` for the query.
|
219
227
|
|
220
228
|
```ruby
|
221
|
-
Product.search
|
229
|
+
Product.search("*")
|
222
230
|
```
|
223
231
|
|
224
232
|
### Pagination
|
@@ -227,7 +235,7 @@ Plays nicely with kaminari and will_paginate.
|
|
227
235
|
|
228
236
|
```ruby
|
229
237
|
# controller
|
230
|
-
@products = Product.search
|
238
|
+
@products = Product.search("milk", page: params[:page], per_page: 20)
|
231
239
|
```
|
232
240
|
|
233
241
|
View with kaminari
|
@@ -247,13 +255,13 @@ View with will_paginate
|
|
247
255
|
By default, results must match all words in the query.
|
248
256
|
|
249
257
|
```ruby
|
250
|
-
Product.search
|
258
|
+
Product.search("fresh honey") # fresh AND honey
|
251
259
|
```
|
252
260
|
|
253
261
|
To change this, use:
|
254
262
|
|
255
263
|
```ruby
|
256
|
-
Product.search
|
264
|
+
Product.search("fresh honey", operator: "or") # fresh OR honey
|
257
265
|
```
|
258
266
|
|
259
267
|
By default, results must match the entire word - `back` will not match `backpack`. You can change this behavior with:
|
@@ -267,7 +275,7 @@ end
|
|
267
275
|
And to search (after you reindex):
|
268
276
|
|
269
277
|
```ruby
|
270
|
-
Product.search
|
278
|
+
Product.search("back", fields: [:name], match: :word_start)
|
271
279
|
```
|
272
280
|
|
273
281
|
Available options are:
|
@@ -289,7 +297,7 @@ The default is `:word`. The most matches will happen with `:word_middle`.
|
|
289
297
|
To match a field exactly (case-sensitive), use:
|
290
298
|
|
291
299
|
```ruby
|
292
|
-
|
300
|
+
Product.search(query, fields: [{email: :exact}, :name])
|
293
301
|
```
|
294
302
|
|
295
303
|
### Phrase Matches
|
@@ -297,7 +305,7 @@ User.search query, fields: [{email: :exact}, :name]
|
|
297
305
|
To only match the exact order, use:
|
298
306
|
|
299
307
|
```ruby
|
300
|
-
|
308
|
+
Product.search("fresh honey", match: :phrase)
|
301
309
|
```
|
302
310
|
|
303
311
|
### Stemming and Language
|
@@ -375,9 +383,9 @@ search_synonyms: ["lightbulb => halogenlamp"]
|
|
375
383
|
|
376
384
|
The above approach works well when your synonym list is static, but in practice, this is often not the case. When you analyze search conversions, you often want to add new synonyms without a full reindex.
|
377
385
|
|
378
|
-
#### Elasticsearch 7.3+
|
386
|
+
#### Elasticsearch 7.3+ and OpenSearch
|
379
387
|
|
380
|
-
For Elasticsearch 7.3+
|
388
|
+
For Elasticsearch 7.3+ and OpenSearch, we recommend placing synonyms in a file on the search server (in the `config` directory). This allows you to reload synonyms without reindexing.
|
381
389
|
|
382
390
|
```txt
|
383
391
|
pop, soda
|
@@ -418,7 +426,7 @@ end
|
|
418
426
|
Search with:
|
419
427
|
|
420
428
|
```ruby
|
421
|
-
Product.search
|
429
|
+
Product.search(query, fields: [:name_tagged])
|
422
430
|
```
|
423
431
|
|
424
432
|
### Misspellings
|
@@ -428,13 +436,13 @@ By default, Searchkick handles misspelled queries by returning results with an [
|
|
428
436
|
You can change this with:
|
429
437
|
|
430
438
|
```ruby
|
431
|
-
Product.search
|
439
|
+
Product.search("zucini", misspellings: {edit_distance: 2}) # zucchini
|
432
440
|
```
|
433
441
|
|
434
442
|
To prevent poor precision and improve performance for correctly spelled queries (which should be a majority for most applications), Searchkick can first perform a search without misspellings, and if there are too few results, perform another with them.
|
435
443
|
|
436
444
|
```ruby
|
437
|
-
Product.search
|
445
|
+
Product.search("zuchini", misspellings: {below: 5})
|
438
446
|
```
|
439
447
|
|
440
448
|
If there are fewer than 5 results, a 2nd search is performed with misspellings enabled. The result of this query is returned.
|
@@ -442,13 +450,13 @@ If there are fewer than 5 results, a 2nd search is performed with misspellings e
|
|
442
450
|
Turn off misspellings with:
|
443
451
|
|
444
452
|
```ruby
|
445
|
-
Product.search
|
453
|
+
Product.search("zuchini", misspellings: false) # no zucchini
|
446
454
|
```
|
447
455
|
|
448
456
|
Specify which fields can include misspellings with:
|
449
457
|
|
450
458
|
```ruby
|
451
|
-
Product.search
|
459
|
+
Product.search("zucini", fields: [:name, :color], misspellings: {fields: [:name]})
|
452
460
|
```
|
453
461
|
|
454
462
|
> When doing this, you must also specify fields to search
|
@@ -458,7 +466,7 @@ Product.search "zucini", fields: [:name, :color], misspellings: {fields: [:name]
|
|
458
466
|
If a user searches `butter`, they may also get results for `peanut butter`. To prevent this, use:
|
459
467
|
|
460
468
|
```ruby
|
461
|
-
Product.search
|
469
|
+
Product.search("butter", exclude: ["peanut butter"])
|
462
470
|
```
|
463
471
|
|
464
472
|
You can map queries and terms to exclude with:
|
@@ -469,7 +477,7 @@ exclude_queries = {
|
|
469
477
|
"cream" => ["ice cream", "whipped cream"]
|
470
478
|
}
|
471
479
|
|
472
|
-
Product.search
|
480
|
+
Product.search(query, exclude: exclude_queries[query])
|
473
481
|
```
|
474
482
|
|
475
483
|
You can demote results by boosting by a factor less than one:
|
@@ -485,13 +493,13 @@ Search :ice_cream::cake: and get `ice cream cake`!
|
|
485
493
|
Add this line to your application’s Gemfile:
|
486
494
|
|
487
495
|
```ruby
|
488
|
-
gem
|
496
|
+
gem "gemoji-parser"
|
489
497
|
```
|
490
498
|
|
491
499
|
And use:
|
492
500
|
|
493
501
|
```ruby
|
494
|
-
Product.search
|
502
|
+
Product.search("🍨🍰", emoji: true)
|
495
503
|
```
|
496
504
|
|
497
505
|
## Indexing
|
@@ -520,12 +528,10 @@ class Product < ApplicationRecord
|
|
520
528
|
end
|
521
529
|
```
|
522
530
|
|
523
|
-
By default, all records are indexed. To control which records are indexed, use the `should_index?` method
|
531
|
+
By default, all records are indexed. To control which records are indexed, use the `should_index?` method.
|
524
532
|
|
525
533
|
```ruby
|
526
534
|
class Product < ApplicationRecord
|
527
|
-
scope :search_import, -> { where(active: true) }
|
528
|
-
|
529
535
|
def should_index?
|
530
536
|
active # only index active records
|
531
537
|
end
|
@@ -590,7 +596,7 @@ You can also do bulk updates.
|
|
590
596
|
|
591
597
|
```ruby
|
592
598
|
Searchkick.callbacks(:bulk) do
|
593
|
-
|
599
|
+
Product.find_each(&:update_fields)
|
594
600
|
end
|
595
601
|
```
|
596
602
|
|
@@ -598,7 +604,7 @@ Or temporarily skip updates.
|
|
598
604
|
|
599
605
|
```ruby
|
600
606
|
Searchkick.callbacks(false) do
|
601
|
-
|
607
|
+
Product.find_each(&:update_fields)
|
602
608
|
end
|
603
609
|
```
|
604
610
|
|
@@ -618,12 +624,34 @@ class Image < ApplicationRecord
|
|
618
624
|
end
|
619
625
|
```
|
620
626
|
|
627
|
+
### Default Scopes
|
628
|
+
|
629
|
+
If you have a default scope that filters records, use the `should_index?` method to exclude them from indexing:
|
630
|
+
|
631
|
+
```ruby
|
632
|
+
class Product < ApplicationRecord
|
633
|
+
default_scope { where(deleted_at: nil) }
|
634
|
+
|
635
|
+
def should_index?
|
636
|
+
deleted_at.nil?
|
637
|
+
end
|
638
|
+
end
|
639
|
+
```
|
640
|
+
|
641
|
+
If you want to index and search filtered records, set:
|
642
|
+
|
643
|
+
```ruby
|
644
|
+
class Product < ApplicationRecord
|
645
|
+
searchkick unscope: true
|
646
|
+
end
|
647
|
+
```
|
648
|
+
|
621
649
|
## Intelligent Search
|
622
650
|
|
623
651
|
The best starting point to improve your search **by far** is to track searches and conversions. [Searchjoy](https://github.com/ankane/searchjoy) makes it easy.
|
624
652
|
|
625
653
|
```ruby
|
626
|
-
Product.search
|
654
|
+
Product.search("apple", track: {user_id: current_user.id})
|
627
655
|
```
|
628
656
|
|
629
657
|
[See the docs](https://github.com/ankane/searchjoy) for how to install and use.
|
@@ -681,7 +709,7 @@ end
|
|
681
709
|
Reindex and search with:
|
682
710
|
|
683
711
|
```ruby
|
684
|
-
Product.search
|
712
|
+
Product.search("milk", boost_where: {orderer_ids: current_user.id})
|
685
713
|
```
|
686
714
|
|
687
715
|
## Instant Search / Autocomplete
|
@@ -705,7 +733,7 @@ end
|
|
705
733
|
Reindex and search with:
|
706
734
|
|
707
735
|
```ruby
|
708
|
-
Movie.search
|
736
|
+
Movie.search("jurassic pa", fields: [:title], match: :word_start)
|
709
737
|
```
|
710
738
|
|
711
739
|
Typically, you want to use a JavaScript library like [typeahead.js](https://twitter.github.io/typeahead.js/) or [jQuery UI](https://jqueryui.com/autocomplete/).
|
@@ -765,7 +793,7 @@ end
|
|
765
793
|
Reindex and search with:
|
766
794
|
|
767
795
|
```ruby
|
768
|
-
products = Product.search
|
796
|
+
products = Product.search("peantu butta", suggest: true)
|
769
797
|
products.suggestions # ["peanut butter"]
|
770
798
|
```
|
771
799
|
|
@@ -776,40 +804,40 @@ products.suggestions # ["peanut butter"]
|
|
776
804
|
![Aggregations](https://gist.github.com/ankane/b6988db2802aca68a589b31e41b44195/raw/40febe948427e5bc53ec4e5dc248822855fef76f/facets.png)
|
777
805
|
|
778
806
|
```ruby
|
779
|
-
products = Product.search
|
807
|
+
products = Product.search("chuck taylor", aggs: [:product_type, :gender, :brand])
|
780
808
|
products.aggs
|
781
809
|
```
|
782
810
|
|
783
811
|
By default, `where` conditions apply to aggregations.
|
784
812
|
|
785
813
|
```ruby
|
786
|
-
Product.search
|
814
|
+
Product.search("wingtips", where: {color: "brandy"}, aggs: [:size])
|
787
815
|
# aggregations for brandy wingtips are returned
|
788
816
|
```
|
789
817
|
|
790
818
|
Change this with:
|
791
819
|
|
792
820
|
```ruby
|
793
|
-
Product.search
|
821
|
+
Product.search("wingtips", where: {color: "brandy"}, aggs: [:size], smart_aggs: false)
|
794
822
|
# aggregations for all wingtips are returned
|
795
823
|
```
|
796
824
|
|
797
825
|
Set `where` conditions for each aggregation separately with:
|
798
826
|
|
799
827
|
```ruby
|
800
|
-
Product.search
|
828
|
+
Product.search("wingtips", aggs: {size: {where: {color: "brandy"}}})
|
801
829
|
```
|
802
830
|
|
803
831
|
Limit
|
804
832
|
|
805
833
|
```ruby
|
806
|
-
Product.search
|
834
|
+
Product.search("apples", aggs: {store_id: {limit: 10}})
|
807
835
|
```
|
808
836
|
|
809
837
|
Order
|
810
838
|
|
811
839
|
```ruby
|
812
|
-
Product.search
|
840
|
+
Product.search("wingtips", aggs: {color: {order: {"_key" => "asc"}}}) # alphabetically
|
813
841
|
```
|
814
842
|
|
815
843
|
[All of these options are supported](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-terms-aggregation.html#search-aggregations-bucket-terms-aggregation-order)
|
@@ -818,31 +846,31 @@ Ranges
|
|
818
846
|
|
819
847
|
```ruby
|
820
848
|
price_ranges = [{to: 20}, {from: 20, to: 50}, {from: 50}]
|
821
|
-
Product.search
|
849
|
+
Product.search("*", aggs: {price: {ranges: price_ranges}})
|
822
850
|
```
|
823
851
|
|
824
852
|
Minimum document count
|
825
853
|
|
826
854
|
```ruby
|
827
|
-
Product.search
|
855
|
+
Product.search("apples", aggs: {store_id: {min_doc_count: 2}})
|
828
856
|
```
|
829
857
|
|
830
858
|
Script support
|
831
859
|
|
832
860
|
```ruby
|
833
|
-
Product.search
|
861
|
+
Product.search("*", aggs: {color: {script: {source: "'Color: ' + _value"}}})
|
834
862
|
```
|
835
863
|
|
836
864
|
Date histogram
|
837
865
|
|
838
866
|
```ruby
|
839
|
-
Product.search
|
867
|
+
Product.search("pear", aggs: {products_per_year: {date_histogram: {field: :created_at, interval: :year}}})
|
840
868
|
```
|
841
869
|
|
842
870
|
For other aggregation types, including sub-aggregations, use `body_options`:
|
843
871
|
|
844
872
|
```ruby
|
845
|
-
Product.search
|
873
|
+
Product.search("orange", body_options: {aggs: {price: {histogram: {field: :price, interval: 10}}}})
|
846
874
|
```
|
847
875
|
|
848
876
|
## Highlight
|
@@ -850,7 +878,7 @@ Product.search "orange", body_options: {aggs: {price: {histogram: {field: :price
|
|
850
878
|
Specify which fields to index with highlighting.
|
851
879
|
|
852
880
|
```ruby
|
853
|
-
class
|
881
|
+
class Band < ApplicationRecord
|
854
882
|
searchkick highlight: [:name]
|
855
883
|
end
|
856
884
|
```
|
@@ -858,7 +886,7 @@ end
|
|
858
886
|
Highlight the search query in the results.
|
859
887
|
|
860
888
|
```ruby
|
861
|
-
bands = Band.search
|
889
|
+
bands = Band.search("cinema", highlight: true)
|
862
890
|
```
|
863
891
|
|
864
892
|
View the highlighted fields with:
|
@@ -872,19 +900,19 @@ end
|
|
872
900
|
To change the tag, use:
|
873
901
|
|
874
902
|
```ruby
|
875
|
-
Band.search
|
903
|
+
Band.search("cinema", highlight: {tag: "<strong>"})
|
876
904
|
```
|
877
905
|
|
878
906
|
To highlight and search different fields, use:
|
879
907
|
|
880
908
|
```ruby
|
881
|
-
Band.search
|
909
|
+
Band.search("cinema", fields: [:name], highlight: {fields: [:description]})
|
882
910
|
```
|
883
911
|
|
884
912
|
By default, the entire field is highlighted. To get small snippets instead, use:
|
885
913
|
|
886
914
|
```ruby
|
887
|
-
bands = Band.search
|
915
|
+
bands = Band.search("cinema", highlight: {fragment_size: 20})
|
888
916
|
bands.with_highlights(multiple: true).each do |band, highlights|
|
889
917
|
highlights[:name].join(" and ")
|
890
918
|
end
|
@@ -893,7 +921,7 @@ end
|
|
893
921
|
Additional options can be specified for each field:
|
894
922
|
|
895
923
|
```ruby
|
896
|
-
Band.search
|
924
|
+
Band.search("cinema", fields: [:name], highlight: {fields: {name: {fragment_size: 200}}})
|
897
925
|
```
|
898
926
|
|
899
927
|
You can find available highlight options in the [Elasticsearch reference](https://www.elastic.co/guide/en/elasticsearch/reference/current/highlighting.html).
|
@@ -922,13 +950,13 @@ end
|
|
922
950
|
Reindex and search with:
|
923
951
|
|
924
952
|
```ruby
|
925
|
-
Restaurant.search
|
953
|
+
Restaurant.search("pizza", where: {location: {near: {lat: 37, lon: -114}, within: "100mi"}}) # or 160km
|
926
954
|
```
|
927
955
|
|
928
956
|
Bounded by a box
|
929
957
|
|
930
958
|
```ruby
|
931
|
-
Restaurant.search
|
959
|
+
Restaurant.search("sushi", where: {location: {top_left: {lat: 38, lon: -123}, bottom_right: {lat: 37, lon: -122}}})
|
932
960
|
```
|
933
961
|
|
934
962
|
**Note:** `top_right` and `bottom_left` also work
|
@@ -936,7 +964,7 @@ Restaurant.search "sushi", where: {location: {top_left: {lat: 38, lon: -123}, bo
|
|
936
964
|
Bounded by a polygon
|
937
965
|
|
938
966
|
```ruby
|
939
|
-
Restaurant.search
|
967
|
+
Restaurant.search("dessert", where: {location: {geo_polygon: {points: [{lat: 38, lon: -123}, {lat: 39, lon: -123}, {lat: 37, lon: 122}]}}})
|
940
968
|
```
|
941
969
|
|
942
970
|
### Boost By Distance
|
@@ -944,13 +972,13 @@ Restaurant.search "dessert", where: {location: {geo_polygon: {points: [{lat: 38,
|
|
944
972
|
Boost results by distance - closer results are boosted more
|
945
973
|
|
946
974
|
```ruby
|
947
|
-
Restaurant.search
|
975
|
+
Restaurant.search("noodles", boost_by_distance: {location: {origin: {lat: 37, lon: -122}}})
|
948
976
|
```
|
949
977
|
|
950
978
|
Also supports [additional options](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-function-score-query.html#function-decay)
|
951
979
|
|
952
980
|
```ruby
|
953
|
-
Restaurant.search
|
981
|
+
Restaurant.search("wings", boost_by_distance: {location: {origin: {lat: 37, lon: -122}, function: "linear", scale: "30mi", decay: 0.5}})
|
954
982
|
```
|
955
983
|
|
956
984
|
### Geo Shapes
|
@@ -977,19 +1005,19 @@ See the [Elasticsearch documentation](https://www.elastic.co/guide/en/elasticsea
|
|
977
1005
|
Find shapes intersecting with the query shape
|
978
1006
|
|
979
1007
|
```ruby
|
980
|
-
Restaurant.search
|
1008
|
+
Restaurant.search("soup", where: {bounds: {geo_shape: {type: "polygon", coordinates: [[{lat: 38, lon: -123}, ...]]}}})
|
981
1009
|
```
|
982
1010
|
|
983
1011
|
Falling entirely within the query shape
|
984
1012
|
|
985
1013
|
```ruby
|
986
|
-
Restaurant.search
|
1014
|
+
Restaurant.search("salad", where: {bounds: {geo_shape: {type: "circle", relation: "within", coordinates: [{lat: 38, lon: -123}], radius: "1km"}}})
|
987
1015
|
```
|
988
1016
|
|
989
1017
|
Not touching the query shape
|
990
1018
|
|
991
1019
|
```ruby
|
992
|
-
Restaurant.search
|
1020
|
+
Restaurant.search("burger", where: {bounds: {geo_shape: {type: "envelope", relation: "disjoint", coordinates: [{lat: 38, lon: -123}, {lat: 37, lon: -122}]}}})
|
993
1021
|
```
|
994
1022
|
|
995
1023
|
## Inheritance
|
@@ -1019,9 +1047,9 @@ Dog.reindex # equivalent, all animals reindexed
|
|
1019
1047
|
And to search, use:
|
1020
1048
|
|
1021
1049
|
```ruby
|
1022
|
-
Animal.search
|
1023
|
-
Dog.search
|
1024
|
-
Animal.search
|
1050
|
+
Animal.search("*") # all animals
|
1051
|
+
Dog.search("*") # just dogs
|
1052
|
+
Animal.search("*", type: [Dog, Cat]) # just cats and dogs
|
1025
1053
|
```
|
1026
1054
|
|
1027
1055
|
**Notes:**
|
@@ -1029,7 +1057,7 @@ Animal.search "*", type: [Dog, Cat] # just cats and dogs
|
|
1029
1057
|
1. The `suggest` option retrieves suggestions from the parent at the moment.
|
1030
1058
|
|
1031
1059
|
```ruby
|
1032
|
-
Dog.search
|
1060
|
+
Dog.search("airbudd", suggest: true) # suggestions for all animals
|
1033
1061
|
```
|
1034
1062
|
2. This relies on a `type` field that is automatically added to the indexed document. Be wary of defining your own `type` field in `search_data`, as it will take precedence.
|
1035
1063
|
|
@@ -1043,13 +1071,13 @@ Product.search("soap", debug: true)
|
|
1043
1071
|
|
1044
1072
|
This prints useful info to `stdout`.
|
1045
1073
|
|
1046
|
-
See how
|
1074
|
+
See how the search server scores your queries with:
|
1047
1075
|
|
1048
1076
|
```ruby
|
1049
1077
|
Product.search("soap", explain: true).response
|
1050
1078
|
```
|
1051
1079
|
|
1052
|
-
See how
|
1080
|
+
See how the search server tokenizes your queries with:
|
1053
1081
|
|
1054
1082
|
```ruby
|
1055
1083
|
Product.search_index.tokens("Dish Washer Soap", analyzer: "searchkick_index")
|
@@ -1072,7 +1100,7 @@ Product.search_index.tokens("dieg", analyzer: "searchkick_word_search")
|
|
1072
1100
|
# ["dieg"] - match!!
|
1073
1101
|
```
|
1074
1102
|
|
1075
|
-
See the [complete list of analyzers](
|
1103
|
+
See the [complete list of analyzers](lib/searchkick/index_options.rb#L36).
|
1076
1104
|
|
1077
1105
|
## Testing
|
1078
1106
|
|
@@ -1223,7 +1251,7 @@ And [setup-opensearch](https://github.com/ankane/setup-opensearch) for an easy w
|
|
1223
1251
|
|
1224
1252
|
## Deployment
|
1225
1253
|
|
1226
|
-
Searchkick uses `ENV["ELASTICSEARCH_URL"]` for
|
1254
|
+
For the search server, Searchkick uses `ENV["ELASTICSEARCH_URL"]` for Elasticsearch and `ENV["OPENSEARCH_URL"]` for OpenSearch. This defaults to `http://localhost:9200`.
|
1227
1255
|
|
1228
1256
|
- [Elastic Cloud](#elastic-cloud)
|
1229
1257
|
- [Heroku](#heroku)
|
@@ -1248,13 +1276,20 @@ rake searchkick:reindex:all
|
|
1248
1276
|
|
1249
1277
|
Choose an add-on: [Bonsai](https://elements.heroku.com/addons/bonsai), [SearchBox](https://elements.heroku.com/addons/searchbox), or [Elastic Cloud](https://elements.heroku.com/addons/foundelasticsearch).
|
1250
1278
|
|
1251
|
-
For Bonsai:
|
1279
|
+
For Elasticsearch on Bonsai:
|
1252
1280
|
|
1253
1281
|
```sh
|
1254
|
-
heroku addons:create bonsai
|
1282
|
+
heroku addons:create bonsai
|
1255
1283
|
heroku config:set ELASTICSEARCH_URL=`heroku config:get BONSAI_URL`
|
1256
1284
|
```
|
1257
1285
|
|
1286
|
+
For OpenSearch on Bonsai:
|
1287
|
+
|
1288
|
+
```sh
|
1289
|
+
heroku addons:create bonsai --engine=opensearch
|
1290
|
+
heroku config:set OPENSEARCH_URL=`heroku config:get BONSAI_URL`
|
1291
|
+
```
|
1292
|
+
|
1258
1293
|
For SearchBox:
|
1259
1294
|
|
1260
1295
|
```sh
|
@@ -1289,16 +1324,16 @@ heroku run rake searchkick:reindex:all
|
|
1289
1324
|
|
1290
1325
|
### Amazon OpenSearch Service
|
1291
1326
|
|
1292
|
-
Create an initializer `config/initializers/
|
1327
|
+
Create an initializer `config/initializers/opensearch.rb` with:
|
1293
1328
|
|
1294
1329
|
```ruby
|
1295
|
-
ENV["
|
1330
|
+
ENV["OPENSEARCH_URL"] = "https://es-domain-1234.us-east-1.es.amazonaws.com:443"
|
1296
1331
|
```
|
1297
1332
|
|
1298
1333
|
To use signed requests, include in your Gemfile:
|
1299
1334
|
|
1300
1335
|
```ruby
|
1301
|
-
gem
|
1336
|
+
gem "faraday_middleware-aws-sigv4"
|
1302
1337
|
```
|
1303
1338
|
|
1304
1339
|
and add to your initializer:
|
@@ -1319,10 +1354,12 @@ rake searchkick:reindex:all
|
|
1319
1354
|
|
1320
1355
|
### Self-Hosted and Other
|
1321
1356
|
|
1322
|
-
Create an initializer
|
1357
|
+
Create an initializer with:
|
1323
1358
|
|
1324
1359
|
```ruby
|
1325
1360
|
ENV["ELASTICSEARCH_URL"] = "https://user:password@host:port"
|
1361
|
+
# or
|
1362
|
+
ENV["OPENSEARCH_URL"] = "https://user:password@host:port"
|
1326
1363
|
```
|
1327
1364
|
|
1328
1365
|
Then deploy and reindex:
|
@@ -1333,19 +1370,21 @@ rake searchkick:reindex:all
|
|
1333
1370
|
|
1334
1371
|
### Data Protection
|
1335
1372
|
|
1336
|
-
We recommend encrypting data at rest and in transit (even inside your own network). This is especially important if you send [personal data](https://en.wikipedia.org/wiki/Personally_identifiable_information) of your users to
|
1373
|
+
We recommend encrypting data at rest and in transit (even inside your own network). This is especially important if you send [personal data](https://en.wikipedia.org/wiki/Personally_identifiable_information) of your users to the search server.
|
1337
1374
|
|
1338
|
-
Bonsai, Elastic Cloud, and Amazon
|
1375
|
+
Bonsai, Elastic Cloud, and Amazon OpenSearch Service all support encryption at rest and HTTPS.
|
1339
1376
|
|
1340
1377
|
### Automatic Failover
|
1341
1378
|
|
1342
|
-
Create an initializer
|
1379
|
+
Create an initializer with multiple hosts:
|
1343
1380
|
|
1344
1381
|
```ruby
|
1345
1382
|
ENV["ELASTICSEARCH_URL"] = "https://user:password@host1,https://user:password@host2"
|
1383
|
+
# or
|
1384
|
+
ENV["OPENSEARCH_URL"] = "https://user:password@host1,https://user:password@host2"
|
1346
1385
|
```
|
1347
1386
|
|
1348
|
-
See [
|
1387
|
+
See [elastic-transport](https://github.com/elastic/elastic-transport-ruby) or [opensearch-transport](https://github.com/opensearch-project/opensearch-ruby/tree/main/opensearch-transport) for a complete list of options.
|
1349
1388
|
|
1350
1389
|
### Lograge
|
1351
1390
|
|
@@ -1368,7 +1407,7 @@ See [Production Rails](https://github.com/ankane/production_rails) for other goo
|
|
1368
1407
|
Significantly increase performance with faster JSON generation. Add [Oj](https://github.com/ohler55/oj) to your Gemfile.
|
1369
1408
|
|
1370
1409
|
```ruby
|
1371
|
-
gem
|
1410
|
+
gem "oj"
|
1372
1411
|
```
|
1373
1412
|
|
1374
1413
|
This speeds up all JSON generation and parsing in your application (automatically!)
|
@@ -1378,7 +1417,7 @@ This speeds up all JSON generation and parsing in your application (automaticall
|
|
1378
1417
|
Significantly increase performance with persistent HTTP connections. Add [Typhoeus](https://github.com/typhoeus/typhoeus) to your Gemfile and it’ll automatically be used.
|
1379
1418
|
|
1380
1419
|
```ruby
|
1381
|
-
gem
|
1420
|
+
gem "typhoeus"
|
1382
1421
|
```
|
1383
1422
|
|
1384
1423
|
To reduce log noise, create an initializer with:
|
@@ -1416,7 +1455,7 @@ end
|
|
1416
1455
|
For large data sets, you can use background jobs to parallelize reindexing.
|
1417
1456
|
|
1418
1457
|
```ruby
|
1419
|
-
Product.reindex(
|
1458
|
+
Product.reindex(mode: :async)
|
1420
1459
|
# {index_name: "products_production_20170111210018065"}
|
1421
1460
|
```
|
1422
1461
|
|
@@ -1441,13 +1480,13 @@ Searchkick.reindex_status(index_name)
|
|
1441
1480
|
You can also have Searchkick wait for reindexing to complete
|
1442
1481
|
|
1443
1482
|
```ruby
|
1444
|
-
Product.reindex(
|
1483
|
+
Product.reindex(mode: :async, wait: true)
|
1445
1484
|
```
|
1446
1485
|
|
1447
1486
|
You can use [ActiveJob::TrafficControl](https://github.com/nickelser/activejob-traffic_control) to control concurrency. Install the gem:
|
1448
1487
|
|
1449
1488
|
```ruby
|
1450
|
-
gem
|
1489
|
+
gem "activejob-traffic_control", ">= 0.1.3"
|
1451
1490
|
```
|
1452
1491
|
|
1453
1492
|
And create an initializer with:
|
@@ -1467,7 +1506,7 @@ This will allow only 3 jobs to run at once.
|
|
1467
1506
|
You can specify a longer refresh interval while reindexing to increase performance.
|
1468
1507
|
|
1469
1508
|
```ruby
|
1470
|
-
Product.reindex(
|
1509
|
+
Product.reindex(mode: :async, refresh_interval: "30s")
|
1471
1510
|
```
|
1472
1511
|
|
1473
1512
|
**Note:** This only makes a noticable difference with parallel reindexing.
|
@@ -1510,7 +1549,7 @@ For more tips, check out [Keeping Elasticsearch in Sync](https://www.elastic.co/
|
|
1510
1549
|
|
1511
1550
|
### Routing
|
1512
1551
|
|
1513
|
-
Searchkick supports [
|
1552
|
+
Searchkick supports [routing](https://www.elastic.co/blog/customizing-your-document-routing), which can significantly speed up searches.
|
1514
1553
|
|
1515
1554
|
```ruby
|
1516
1555
|
class Business < ApplicationRecord
|
@@ -1525,7 +1564,7 @@ end
|
|
1525
1564
|
Reindex and search with:
|
1526
1565
|
|
1527
1566
|
```ruby
|
1528
|
-
Business.search
|
1567
|
+
Business.search("ice cream", routing: params[:city_id])
|
1529
1568
|
```
|
1530
1569
|
|
1531
1570
|
### Partial Reindexing
|
@@ -1618,7 +1657,7 @@ ReindexConversionsJob.perform_later("Product")
|
|
1618
1657
|
|
1619
1658
|
## Advanced
|
1620
1659
|
|
1621
|
-
Searchkick makes it easy to use the Elasticsearch DSL on its own.
|
1660
|
+
Searchkick makes it easy to use the Elasticsearch or OpenSearch DSL on its own.
|
1622
1661
|
|
1623
1662
|
### Advanced Mapping
|
1624
1663
|
|
@@ -1648,7 +1687,7 @@ end
|
|
1648
1687
|
And use the `body` option to search:
|
1649
1688
|
|
1650
1689
|
```ruby
|
1651
|
-
products = Product.search
|
1690
|
+
products = Product.search(body: {query: {match: {name: "milk"}}})
|
1652
1691
|
```
|
1653
1692
|
|
1654
1693
|
View the response with:
|
@@ -1660,21 +1699,21 @@ products.response
|
|
1660
1699
|
To modify the query generated by Searchkick, use:
|
1661
1700
|
|
1662
1701
|
```ruby
|
1663
|
-
products = Product.search
|
1702
|
+
products = Product.search("milk", body_options: {min_score: 1})
|
1664
1703
|
```
|
1665
1704
|
|
1666
1705
|
or
|
1667
1706
|
|
1668
1707
|
```ruby
|
1669
1708
|
products =
|
1670
|
-
Product.search
|
1709
|
+
Product.search("apples") do |body|
|
1671
1710
|
body[:min_score] = 1
|
1672
1711
|
end
|
1673
1712
|
```
|
1674
1713
|
|
1675
|
-
###
|
1714
|
+
### Client
|
1676
1715
|
|
1677
|
-
|
1716
|
+
To access the `Elasticsearch::Client` or `OpenSearch::Client` directly, use:
|
1678
1717
|
|
1679
1718
|
```ruby
|
1680
1719
|
Searchkick.client
|
@@ -1685,8 +1724,8 @@ Searchkick.client
|
|
1685
1724
|
To batch search requests for performance, use:
|
1686
1725
|
|
1687
1726
|
```ruby
|
1688
|
-
products = Product.search("snacks"
|
1689
|
-
coupons = Coupon.search("snacks"
|
1727
|
+
products = Product.search("snacks")
|
1728
|
+
coupons = Coupon.search("snacks")
|
1690
1729
|
Searchkick.multi_search([products, coupons])
|
1691
1730
|
```
|
1692
1731
|
|
@@ -1699,7 +1738,7 @@ Then use `products` and `coupons` as typical results.
|
|
1699
1738
|
Search across multiple models with:
|
1700
1739
|
|
1701
1740
|
```ruby
|
1702
|
-
Searchkick.search
|
1741
|
+
Searchkick.search("milk", models: [Product, Category])
|
1703
1742
|
```
|
1704
1743
|
|
1705
1744
|
Boost specific models with:
|
@@ -1725,7 +1764,7 @@ end
|
|
1725
1764
|
You can also scroll batches manually.
|
1726
1765
|
|
1727
1766
|
```ruby
|
1728
|
-
products = Product.search
|
1767
|
+
products = Product.search("*", scroll: "1m")
|
1729
1768
|
while products.any?
|
1730
1769
|
# process batch ...
|
1731
1770
|
|
@@ -1737,7 +1776,7 @@ products.clear_scroll
|
|
1737
1776
|
|
1738
1777
|
## Deep Paging
|
1739
1778
|
|
1740
|
-
By default, Elasticsearch
|
1779
|
+
By default, Elasticsearch and OpenSearch limit paging to the first 10,000 results. [Here’s why](https://www.elastic.co/guide/en/elasticsearch/guide/current/pagination.html). We don’t recommend changing this, but if you really need all results, you can use:
|
1741
1780
|
|
1742
1781
|
```ruby
|
1743
1782
|
class Product < ApplicationRecord
|
@@ -1745,7 +1784,7 @@ class Product < ApplicationRecord
|
|
1745
1784
|
end
|
1746
1785
|
```
|
1747
1786
|
|
1748
|
-
If you just need an accurate total count
|
1787
|
+
If you just need an accurate total count, you can instead use:
|
1749
1788
|
|
1750
1789
|
```ruby
|
1751
1790
|
Product.search("pears", body_options: {track_total_hits: true})
|
@@ -1756,7 +1795,7 @@ Product.search("pears", body_options: {track_total_hits: true})
|
|
1756
1795
|
To query nested data, use dot notation.
|
1757
1796
|
|
1758
1797
|
```ruby
|
1759
|
-
|
1798
|
+
Product.search("san", fields: ["store.city"], where: {"store.zip_code" => 12345})
|
1760
1799
|
```
|
1761
1800
|
|
1762
1801
|
## Reference
|
@@ -1886,7 +1925,7 @@ Searchkick.queue_name = :search_reindex
|
|
1886
1925
|
Eager load associations
|
1887
1926
|
|
1888
1927
|
```ruby
|
1889
|
-
Product.search
|
1928
|
+
Product.search("milk", includes: [:brand, :stores])
|
1890
1929
|
```
|
1891
1930
|
|
1892
1931
|
Eager load different associations by model
|
@@ -1898,7 +1937,7 @@ Searchkick.search("*", models: [Product, Store], model_includes: {Product => [:
|
|
1898
1937
|
Run additional scopes on results
|
1899
1938
|
|
1900
1939
|
```ruby
|
1901
|
-
Product.search
|
1940
|
+
Product.search("milk", scope_results: ->(r) { r.with_attached_images })
|
1902
1941
|
```
|
1903
1942
|
|
1904
1943
|
Specify default fields to search
|
@@ -1960,13 +1999,6 @@ class Product < ApplicationRecord
|
|
1960
1999
|
end
|
1961
2000
|
```
|
1962
2001
|
|
1963
|
-
Lazy searching
|
1964
|
-
|
1965
|
-
```ruby
|
1966
|
-
products = Product.search("carrots", execute: false)
|
1967
|
-
products.each { ... } # search not executed until here
|
1968
|
-
```
|
1969
|
-
|
1970
2002
|
Add [request parameters](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-search.html#search-search-api-query-params) like `search_type`
|
1971
2003
|
|
1972
2004
|
```ruby
|
@@ -2001,25 +2033,20 @@ rake searchkick:reindex:all
|
|
2001
2033
|
Turn on misspellings after a certain number of characters
|
2002
2034
|
|
2003
2035
|
```ruby
|
2004
|
-
Product.search
|
2036
|
+
Product.search("api", misspellings: {prefix_length: 2}) # api, apt, no ahi
|
2005
2037
|
```
|
2006
2038
|
|
2007
|
-
**Note:** With this option, if the query length is the same as `prefix_length`, misspellings are turned off
|
2039
|
+
**Note:** With this option, if the query length is the same as `prefix_length`, misspellings are turned off with Elasticsearch 7 and OpenSearch
|
2008
2040
|
|
2009
2041
|
```ruby
|
2010
|
-
Product.search
|
2042
|
+
Product.search("ah", misspellings: {prefix_length: 2}) # ah, no aha
|
2011
2043
|
```
|
2012
2044
|
|
2013
|
-
##
|
2014
|
-
|
2015
|
-
1. Install Searchkick 4
|
2016
|
-
2. Upgrade your Elasticsearch cluster
|
2017
|
-
|
2018
|
-
## Elasticsearch Gotchas
|
2045
|
+
## Gotchas
|
2019
2046
|
|
2020
2047
|
### Consistency
|
2021
2048
|
|
2022
|
-
Elasticsearch
|
2049
|
+
Elasticsearch and OpenSearch are eventually consistent, meaning it can take up to a second for a change to reflect in search. You can use the `refresh` method to have it show up immediately.
|
2023
2050
|
|
2024
2051
|
```ruby
|
2025
2052
|
product.save!
|
@@ -2028,7 +2055,7 @@ Product.search_index.refresh
|
|
2028
2055
|
|
2029
2056
|
### Inconsistent Scores
|
2030
2057
|
|
2031
|
-
Due to the distributed nature of Elasticsearch, you can get incorrect results when the number of documents in the index is low. You can [read more about it here](https://www.elastic.co/blog/understanding-query-then-fetch-vs-dfs-query-then-fetch). To fix this, do:
|
2058
|
+
Due to the distributed nature of Elasticsearch and OpenSearch, you can get incorrect results when the number of documents in the index is low. You can [read more about it here](https://www.elastic.co/blog/understanding-query-then-fetch-vs-dfs-query-then-fetch). To fix this, do:
|
2032
2059
|
|
2033
2060
|
```ruby
|
2034
2061
|
class Product < ApplicationRecord
|
@@ -2038,6 +2065,42 @@ end
|
|
2038
2065
|
|
2039
2066
|
For convenience, this is set by default in the test environment.
|
2040
2067
|
|
2068
|
+
## Upgrading
|
2069
|
+
|
2070
|
+
### 5.0
|
2071
|
+
|
2072
|
+
Searchkick 5 supports both the `elasticsearch` and `opensearch-ruby` gems. Add the one you want to use to your Gemfile:
|
2073
|
+
|
2074
|
+
```ruby
|
2075
|
+
gem "elasticsearch"
|
2076
|
+
# or
|
2077
|
+
gem "opensearch-ruby"
|
2078
|
+
```
|
2079
|
+
|
2080
|
+
If using the deprecated `faraday_middleware-aws-signers-v4` gem, switch to `faraday_middleware-aws-sigv4`.
|
2081
|
+
|
2082
|
+
Also, searches now use lazy loading:
|
2083
|
+
|
2084
|
+
```ruby
|
2085
|
+
# search not executed
|
2086
|
+
Product.search("milk")
|
2087
|
+
|
2088
|
+
# search executed
|
2089
|
+
Product.search("milk").to_a
|
2090
|
+
```
|
2091
|
+
|
2092
|
+
You can reindex relations in the background:
|
2093
|
+
|
2094
|
+
```ruby
|
2095
|
+
store.products.reindex(mode: :async)
|
2096
|
+
# or
|
2097
|
+
store.products.reindex(mode: :queue)
|
2098
|
+
```
|
2099
|
+
|
2100
|
+
And there’s a [new option](#default-scopes) for models with default scopes.
|
2101
|
+
|
2102
|
+
Check out the [changelog](https://github.com/ankane/searchkick/blob/master/CHANGELOG.md#500-2022-02-21) for the full list of changes.
|
2103
|
+
|
2041
2104
|
## History
|
2042
2105
|
|
2043
2106
|
View the [changelog](https://github.com/ankane/searchkick/blob/master/CHANGELOG.md).
|