searchkick_evichat 0.0.2

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (78) hide show
  1. checksums.yaml +7 -0
  2. data/.github/ISSUE_TEMPLATE.md +7 -0
  3. data/.gitignore +22 -0
  4. data/.travis.yml +33 -0
  5. data/CHANGELOG.md +571 -0
  6. data/CONTRIBUTING.md +51 -0
  7. data/Gemfile +16 -0
  8. data/LICENSE.txt +22 -0
  9. data/README.md +1895 -0
  10. data/Rakefile +16 -0
  11. data/benchmark/Gemfile +24 -0
  12. data/benchmark/index.rb +99 -0
  13. data/benchmark/search.rb +48 -0
  14. data/docs/Searchkick-3-Upgrade.md +57 -0
  15. data/lib/searchkick.rb +255 -0
  16. data/lib/searchkick/bulk_indexer.rb +171 -0
  17. data/lib/searchkick/bulk_reindex_job.rb +27 -0
  18. data/lib/searchkick/hash_wrapper.rb +12 -0
  19. data/lib/searchkick/index.rb +331 -0
  20. data/lib/searchkick/index_options.rb +414 -0
  21. data/lib/searchkick/indexer.rb +28 -0
  22. data/lib/searchkick/logging.rb +243 -0
  23. data/lib/searchkick/middleware.rb +12 -0
  24. data/lib/searchkick/model.rb +105 -0
  25. data/lib/searchkick/multi_search.rb +41 -0
  26. data/lib/searchkick/process_batch_job.rb +23 -0
  27. data/lib/searchkick/process_queue_job.rb +23 -0
  28. data/lib/searchkick/query.rb +978 -0
  29. data/lib/searchkick/record_data.rb +128 -0
  30. data/lib/searchkick/record_indexer.rb +59 -0
  31. data/lib/searchkick/reindex_queue.rb +38 -0
  32. data/lib/searchkick/reindex_v2_job.rb +36 -0
  33. data/lib/searchkick/results.rb +235 -0
  34. data/lib/searchkick/tasks.rb +29 -0
  35. data/lib/searchkick/version.rb +3 -0
  36. data/searchkick.gemspec +30 -0
  37. data/test/aggs_test.rb +217 -0
  38. data/test/autocomplete_test.rb +81 -0
  39. data/test/boost_test.rb +207 -0
  40. data/test/callbacks_test.rb +59 -0
  41. data/test/ci/before_install.sh +17 -0
  42. data/test/errors_test.rb +19 -0
  43. data/test/gemfiles/activerecord42.gemfile +7 -0
  44. data/test/gemfiles/activerecord50.gemfile +7 -0
  45. data/test/gemfiles/activerecord51.gemfile +7 -0
  46. data/test/gemfiles/apartment.gemfile +8 -0
  47. data/test/gemfiles/cequel.gemfile +8 -0
  48. data/test/gemfiles/mongoid5.gemfile +7 -0
  49. data/test/gemfiles/mongoid6.gemfile +12 -0
  50. data/test/gemfiles/nobrainer.gemfile +8 -0
  51. data/test/gemfiles/parallel_tests.gemfile +8 -0
  52. data/test/geo_shape_test.rb +171 -0
  53. data/test/highlight_test.rb +94 -0
  54. data/test/index_test.rb +165 -0
  55. data/test/inheritance_test.rb +82 -0
  56. data/test/language_test.rb +79 -0
  57. data/test/marshal_test.rb +13 -0
  58. data/test/match_test.rb +293 -0
  59. data/test/misspellings_test.rb +56 -0
  60. data/test/model_test.rb +40 -0
  61. data/test/multi_search_test.rb +37 -0
  62. data/test/multi_tenancy_test.rb +22 -0
  63. data/test/order_test.rb +40 -0
  64. data/test/pagination_test.rb +96 -0
  65. data/test/partial_reindex_test.rb +65 -0
  66. data/test/query_test.rb +43 -0
  67. data/test/reindex_test.rb +87 -0
  68. data/test/reindex_v2_job_test.rb +27 -0
  69. data/test/routing_test.rb +23 -0
  70. data/test/should_index_test.rb +32 -0
  71. data/test/similar_test.rb +28 -0
  72. data/test/sql_test.rb +190 -0
  73. data/test/suggest_test.rb +100 -0
  74. data/test/support/kaminari.yml +21 -0
  75. data/test/synonyms_test.rb +69 -0
  76. data/test/test_helper.rb +593 -0
  77. data/test/where_test.rb +249 -0
  78. metadata +250 -0
@@ -0,0 +1,51 @@
1
+ # Contributing
2
+
3
+ First, thanks for wanting to contribute. You’re awesome! :heart:
4
+
5
+ ## Questions
6
+
7
+ Use [Stack Overflow](https://stackoverflow.com/) with the tag `searchkick`.
8
+
9
+ ## Feature Requests
10
+
11
+ Create an issue. Start the title with `[Idea]`.
12
+
13
+ ## Issues
14
+
15
+ Think you’ve discovered an issue?
16
+
17
+ 1. Search existing issues to see if it’s been reported.
18
+ 2. Try the `master` branch to make sure it hasn’t been fixed.
19
+
20
+ ```rb
21
+ gem "searchkick", github: "ankane/searchkick"
22
+ ```
23
+
24
+ 3. Try the `debug` option when searching. This can reveal useful info.
25
+
26
+ ```ruby
27
+ Product.search("something", debug: true)
28
+ ```
29
+
30
+ If the above steps don’t help, create an issue.
31
+
32
+ - Recreate the problem by forking [this gist](https://gist.github.com/ankane/f80b0923d9ae2c077f41997f7b704e5c). Include a link to your gist and the output in the issue.
33
+ - For exceptions, include the complete backtrace.
34
+
35
+ ## Pull Requests
36
+
37
+ Fork the project and create a pull request. A few tips:
38
+
39
+ - Keep changes to a minimum. If you have multiple features or fixes, submit multiple pull requests.
40
+ - Follow the existing style. The code should read like it’s written by a single person.
41
+ - Add one or more tests if possible. Make sure existing tests pass with:
42
+
43
+ ```sh
44
+ bundle exec rake test
45
+ ```
46
+
47
+ Feel free to open an issue to get feedback on your idea before spending too much time on it.
48
+
49
+ ---
50
+
51
+ This contributing guide is released under [CCO](https://creativecommons.org/publicdomain/zero/1.0/) (public domain). Use it for your own project without attribution.
data/Gemfile ADDED
@@ -0,0 +1,16 @@
1
+ source "https://rubygems.org"
2
+
3
+ # Specify your gem's dependencies in searchkick.gemspec
4
+ gemspec
5
+
6
+ gem "sqlite3"
7
+ gem "activerecord"
8
+ gem "gemoji-parser"
9
+ gem "typhoeus"
10
+ gem "activejob"
11
+ gem "redis"
12
+ gem "connection_pool"
13
+
14
+ # kaminari
15
+ gem "actionpack"
16
+ gem "kaminari"
@@ -0,0 +1,22 @@
1
+ Copyright (c) 2013-2018 Andrew Kane
2
+
3
+ MIT License
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining
6
+ a copy of this software and associated documentation files (the
7
+ "Software"), to deal in the Software without restriction, including
8
+ without limitation the rights to use, copy, modify, merge, publish,
9
+ distribute, sublicense, and/or sell copies of the Software, and to
10
+ permit persons to whom the Software is furnished to do so, subject to
11
+ the following conditions:
12
+
13
+ The above copyright notice and this permission notice shall be
14
+ included in all copies or substantial portions of the Software.
15
+
16
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
17
+ EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
18
+ MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
19
+ NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
20
+ LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
21
+ OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
22
+ WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
@@ -0,0 +1,1895 @@
1
+ # Searchkick
2
+
3
+ :rocket: Intelligent search made easy
4
+
5
+ **Searchkick learns what your users are looking for.** As more people search, it gets smarter and the results get better. It’s friendly for developers - and magical for your users.
6
+
7
+ Searchkick handles:
8
+
9
+ - stemming - `tomatoes` matches `tomato`
10
+ - special characters - `jalapeno` matches `jalapeño`
11
+ - extra whitespace - `dishwasher` matches `dish washer`
12
+ - misspellings - `zuchini` matches `zucchini`
13
+ - custom synonyms - `qtip` matches `cotton swab`
14
+
15
+ Plus:
16
+
17
+ - query like SQL - no need to learn a new query language
18
+ - reindex without downtime
19
+ - easily personalize results for each user
20
+ - autocomplete
21
+ - “Did you mean” suggestions
22
+ - supports many languages
23
+ - works with ActiveRecord, Mongoid, and NoBrainer
24
+
25
+ :speech_balloon: Get [handcrafted updates](https://chartkick.us7.list-manage.com/subscribe?u=952c861f99eb43084e0a49f98&id=6ea6541e8e&group[0][4]=true) for new features
26
+
27
+ :tangerine: Battle-tested at [Instacart](https://www.instacart.com/opensource)
28
+
29
+ [![Build Status](https://travis-ci.org/ankane/searchkick.svg?branch=master)](https://travis-ci.org/ankane/searchkick)
30
+
31
+ ---
32
+
33
+ Does your startup use Searchkick? Want a free hour of advising? Fill out [this application](https://goo.gl/forms/Or1HQTRb2rgQCNtd2). I’ll reach out to a few companies.
34
+
35
+ ---
36
+
37
+ ## Contents
38
+
39
+ - [Getting Started](#getting-started)
40
+ - [Querying](#querying)
41
+ - [Indexing](#indexing)
42
+ - [Instant Search / Autocomplete](#instant-search--autocomplete)
43
+ - [Aggregations](#aggregations)
44
+ - [Deployment](#deployment)
45
+ - [Performance](#performance)
46
+ - [Elasticsearch DSL](#advanced)
47
+ - [Reference](#reference)
48
+
49
+ **Searchkick 3.0 was recently released!** See [how to upgrade](docs/Searchkick-3-Upgrade.md)
50
+
51
+ Thinking of upgrading from Elasticsearch 5 to 6? [Read this first](#elasticsearch-5-to-6-upgrade)
52
+
53
+ ## Getting Started
54
+
55
+ [Install Elasticsearch](https://www.elastic.co/guide/en/elasticsearch/reference/current/setup.html). For Homebrew, use:
56
+
57
+ ```sh
58
+ brew install elasticsearch
59
+ brew services start elasticsearch
60
+ ```
61
+
62
+ Add this line to your application’s Gemfile:
63
+
64
+ ```ruby
65
+ gem 'searchkick'
66
+ ```
67
+
68
+ The latest version works with Elasticsearch 5 and 6. For Elasticsearch 2, use version 2.5.0 and [this readme](https://github.com/ankane/searchkick/blob/v2.5.0/README.md).
69
+
70
+ Add searchkick to models you want to search.
71
+
72
+ ```ruby
73
+ class Product < ApplicationRecord
74
+ searchkick
75
+ end
76
+ ```
77
+
78
+ Add data to the search index.
79
+
80
+ ```ruby
81
+ Product.reindex
82
+ ```
83
+
84
+ And to query, use:
85
+
86
+ ```ruby
87
+ products = Product.search("apples")
88
+ products.each do |product|
89
+ puts product.name
90
+ end
91
+ ```
92
+
93
+ Searchkick supports the complete [Elasticsearch Search API](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-search.html). As your search becomes more advanced, we recommend you use the [Elasticsearch DSL](#advanced) for maximum flexibility.
94
+
95
+ ## Querying
96
+
97
+ Query like SQL
98
+
99
+ ```ruby
100
+ Product.search "apples", where: {in_stock: true}, limit: 10, offset: 50
101
+ ```
102
+
103
+ Search specific fields
104
+
105
+ ```ruby
106
+ fields: [:name, :brand]
107
+ ```
108
+
109
+ Where
110
+
111
+ ```ruby
112
+ where: {
113
+ expires_at: {gt: Time.now}, # lt, gte, lte also available
114
+ orders_count: 1..10, # equivalent to {gte: 1, lte: 10}
115
+ aisle_id: [25, 30], # in
116
+ store_id: {not: 2}, # not
117
+ aisle_id: {not: [25, 30]}, # not in
118
+ user_ids: {all: [1, 3]}, # all elements in array
119
+ category: /frozen .+/, # regexp
120
+ _or: [{in_stock: true}, {backordered: true}]
121
+ }
122
+ ```
123
+
124
+ Order
125
+
126
+ ```ruby
127
+ order: {_score: :desc} # most relevant first - default
128
+ ```
129
+
130
+ [All of these sort options are supported](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-sort.html)
131
+
132
+ Limit / offset
133
+
134
+ ```ruby
135
+ limit: 20, offset: 40
136
+ ```
137
+
138
+ Select
139
+
140
+ ```ruby
141
+ select: [:name]
142
+ ```
143
+
144
+ [These source filtering options are supported](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-source-filtering.html)
145
+
146
+ ### Results
147
+
148
+ Searches return a `Searchkick::Results` object. This responds like an array to most methods.
149
+
150
+ ```ruby
151
+ results = Product.search("milk")
152
+ results.size
153
+ results.any?
154
+ results.each { |result| ... }
155
+ ```
156
+
157
+ By default, ids are fetched from Elasticsearch and records are fetched from your database. To fetch everything from Elasticsearch, use:
158
+
159
+ ```ruby
160
+ Product.search("apples", load: false)
161
+ ```
162
+
163
+ Get total results
164
+
165
+ ```ruby
166
+ results.total_count
167
+ ```
168
+
169
+ Get the time the search took (in milliseconds)
170
+
171
+ ```ruby
172
+ results.took
173
+ ```
174
+
175
+ Get the full response from Elasticsearch
176
+
177
+ ```ruby
178
+ results.response
179
+ ```
180
+
181
+ ### Boosting
182
+
183
+ Boost important fields
184
+
185
+ ```ruby
186
+ fields: ["title^10", "description"]
187
+ ```
188
+
189
+ Boost by the value of a field (field must be numeric)
190
+
191
+ ```ruby
192
+ boost_by: [:orders_count] # give popular documents a little boost
193
+ boost_by: {orders_count: {factor: 10}} # default factor is 1
194
+ ```
195
+
196
+ Boost matching documents
197
+
198
+ ```ruby
199
+ boost_where: {user_id: 1}
200
+ boost_where: {user_id: {value: 1, factor: 100}} # default factor is 1000
201
+ boost_where: {user_id: [{value: 1, factor: 100}, {value: 2, factor: 200}]}
202
+ ```
203
+
204
+ [Conversions](#keep-getting-better) are also a great way to boost.
205
+
206
+ ### Get Everything
207
+
208
+ Use a `*` for the query.
209
+
210
+ ```ruby
211
+ Product.search "*"
212
+ ```
213
+
214
+ ### Pagination
215
+
216
+ Plays nicely with kaminari and will_paginate.
217
+
218
+ ```ruby
219
+ # controller
220
+ @products = Product.search "milk", page: params[:page], per_page: 20
221
+ ```
222
+
223
+ View with kaminari
224
+
225
+ ```erb
226
+ <%= paginate @products %>
227
+ ```
228
+
229
+ View with will_paginate
230
+
231
+ ```erb
232
+ <%= will_paginate @products %>
233
+ ```
234
+
235
+ ### Partial Matches
236
+
237
+ By default, results must match all words in the query.
238
+
239
+ ```ruby
240
+ Product.search "fresh honey" # fresh AND honey
241
+ ```
242
+
243
+ To change this, use:
244
+
245
+ ```ruby
246
+ Product.search "fresh honey", operator: "or" # fresh OR honey
247
+ ```
248
+
249
+ By default, results must match the entire word - `back` will not match `backpack`. You can change this behavior with:
250
+
251
+ ```ruby
252
+ class Product < ApplicationRecord
253
+ searchkick word_start: [:name]
254
+ end
255
+ ```
256
+
257
+ And to search (after you reindex):
258
+
259
+ ```ruby
260
+ Product.search "back", fields: [:name], match: :word_start
261
+ ```
262
+
263
+ Available options are:
264
+
265
+ ```ruby
266
+ :word # default
267
+ :word_start
268
+ :word_middle
269
+ :word_end
270
+ :text_start
271
+ :text_middle
272
+ :text_end
273
+ ```
274
+
275
+ ### Exact Matches
276
+
277
+ To match a field exactly (case-sensitive), use:
278
+
279
+ ```ruby
280
+ User.search query, fields: [{email: :exact}, :name]
281
+ ```
282
+
283
+ ### Phrase Matches
284
+
285
+ To only match the exact order, use:
286
+
287
+ ```ruby
288
+ User.search "fresh honey", match: :phrase
289
+ ```
290
+
291
+ ### Language
292
+
293
+ Searchkick defaults to English for stemming. To change this, use:
294
+
295
+ ```ruby
296
+ class Product < ApplicationRecord
297
+ searchkick language: "german"
298
+ end
299
+ ```
300
+
301
+ [See the list of stemmers](https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-stemmer-tokenfilter.html)
302
+
303
+ A few languages require plugins:
304
+
305
+ - `chinese` - [analysis-ik plugin](https://github.com/medcl/elasticsearch-analysis-ik)
306
+ - `japanese` - [analysis-kuromoji plugin](https://www.elastic.co/guide/en/elasticsearch/plugins/6.2/analysis-kuromoji.html)
307
+ - `korean` - [analysis-openkoreantext plugin](https://github.com/open-korean-text/elasticsearch-analysis-openkoreantext)
308
+ - `polish` - [analysis-stempel plugin](https://www.elastic.co/guide/en/elasticsearch/plugins/6.2/analysis-stempel.html)
309
+ - `ukrainian` - [analysis-ukrainian plugin](https://www.elastic.co/guide/en/elasticsearch/plugins/6.2/analysis-ukrainian.html)
310
+ - `vietnamese` - [analysis-vietnamese plugin](https://github.com/duydo/elasticsearch-analysis-vietnamese)
311
+
312
+ ### Synonyms
313
+
314
+ ```ruby
315
+ class Product < ApplicationRecord
316
+ searchkick synonyms: [["scallion", "green onion"], ["qtip", "cotton swab"]]
317
+ end
318
+ ```
319
+
320
+ Call `Product.reindex` after changing synonyms.
321
+
322
+ Synonyms cannot be more than two words at the moment.
323
+
324
+ To read synonyms from a file, use:
325
+
326
+ ```ruby
327
+ synonyms: -> { CSV.read("/some/path/synonyms.csv") }
328
+ ```
329
+
330
+ For directional synonyms, use:
331
+
332
+ ```ruby
333
+ synonyms: ["lightbulb => halogenlamp"]
334
+ ```
335
+
336
+ ### Tags and Dynamic Synonyms
337
+
338
+ The above approach works well when your synonym list is static, but in practice, this is often not the case. When you analyze search conversions, you often want to add new synonyms or tags without a full reindex. You can use a library like [ActsAsTaggableOn](https://github.com/mbleigh/acts-as-taggable-on) and do:
339
+
340
+ ```ruby
341
+ class Product < ApplicationRecord
342
+ acts_as_taggable
343
+ scope :search_import, -> { includes(:tags) }
344
+
345
+ def search_data
346
+ {
347
+ name_tagged: "#{name} #{tags.map(&:name).join(" ")}"
348
+ }
349
+ end
350
+ end
351
+ ```
352
+
353
+ Search with:
354
+
355
+ ```ruby
356
+ Product.search query, fields: [:name_tagged]
357
+ ```
358
+
359
+ ### WordNet
360
+
361
+ Prepopulate English synonyms with the [WordNet database](https://en.wikipedia.org/wiki/WordNet).
362
+
363
+ Download [WordNet 3.0](http://wordnetcode.princeton.edu/3.0/WNprolog-3.0.tar.gz) to each Elasticsearch server and move `wn_s.pl` to the `/var/lib` directory.
364
+
365
+ ```sh
366
+ cd /tmp
367
+ curl -o wordnet.tar.gz http://wordnetcode.princeton.edu/3.0/WNprolog-3.0.tar.gz
368
+ tar -zxvf wordnet.tar.gz
369
+ mv prolog/wn_s.pl /var/lib
370
+ ```
371
+
372
+ Tell each model to use it:
373
+
374
+ ```ruby
375
+ class Product < ApplicationRecord
376
+ searchkick wordnet: true
377
+ end
378
+ ```
379
+
380
+ ### Misspellings
381
+
382
+ By default, Searchkick handles misspelled queries by returning results with an [edit distance](https://en.wikipedia.org/wiki/Levenshtein_distance) of one.
383
+
384
+ You can change this with:
385
+
386
+ ```ruby
387
+ Product.search "zucini", misspellings: {edit_distance: 2} # zucchini
388
+ ```
389
+
390
+ To prevent poor precision and improve performance for correctly spelled queries (which should be a majority for most applications), Searchkick can first perform a search without misspellings, and if there are too few results, perform another with them.
391
+
392
+ ```ruby
393
+ Product.search "zuchini", misspellings: {below: 5}
394
+ ```
395
+
396
+ If there are fewer than 5 results, a 2nd search is performed with misspellings enabled. The result of this query is returned.
397
+
398
+ Turn off misspellings with:
399
+
400
+ ```ruby
401
+ Product.search "zuchini", misspellings: false # no zucchini
402
+ ```
403
+
404
+ ### Bad Matches
405
+
406
+ If a user searches `butter`, they may also get results for `peanut butter`. To prevent this, use:
407
+
408
+ ```ruby
409
+ Product.search "butter", exclude: ["peanut butter"]
410
+ ```
411
+
412
+ You can map queries and terms to exclude with:
413
+
414
+ ```ruby
415
+ exclude_queries = {
416
+ "butter" => ["peanut butter"],
417
+ "cream" => ["ice cream", "whipped cream"]
418
+ }
419
+
420
+ Product.search query, exclude: exclude_queries[query]
421
+ ```
422
+
423
+ ### Emoji
424
+
425
+ Search :ice_cream::cake: and get `ice cream cake`!
426
+
427
+ Add this line to your application’s Gemfile:
428
+
429
+ ```ruby
430
+ gem 'gemoji-parser'
431
+ ```
432
+
433
+ And use:
434
+
435
+ ```ruby
436
+ Product.search "🍨🍰", emoji: true
437
+ ```
438
+
439
+ ## Indexing
440
+
441
+ Control what data is indexed with the `search_data` method. Call `Product.reindex` after changing this method.
442
+
443
+ ```ruby
444
+ class Product < ApplicationRecord
445
+ belongs_to :department
446
+
447
+ def search_data
448
+ {
449
+ name: name,
450
+ department_name: department.name,
451
+ on_sale: sale_price.present?
452
+ }
453
+ end
454
+ end
455
+ ```
456
+
457
+ Searchkick uses `find_in_batches` to import documents. To eager load associations, use the `search_import` scope.
458
+
459
+ ```ruby
460
+ class Product < ApplicationRecord
461
+ scope :search_import, -> { includes(:department) }
462
+ end
463
+ ```
464
+
465
+ By default, all records are indexed. To control which records are indexed, use the `should_index?` method together with the `search_import` scope.
466
+
467
+ ```ruby
468
+ class Product < ApplicationRecord
469
+ scope :search_import, -> { where(active: true) }
470
+
471
+ def should_index?
472
+ active # only index active records
473
+ end
474
+ end
475
+ ```
476
+
477
+ If a reindex is interrupted, you can resume it with:
478
+
479
+ ```ruby
480
+ Product.reindex(resume: true)
481
+ ```
482
+
483
+ For large data sets, try [parallel reindexing](#parallel-reindexing).
484
+
485
+ ### To Reindex, or Not to Reindex
486
+
487
+ #### Reindex
488
+
489
+ - when you install or upgrade searchkick
490
+ - change the `search_data` method
491
+ - change the `searchkick` method
492
+
493
+ #### No need to reindex
494
+
495
+ - app starts
496
+
497
+ ### Stay Synced
498
+
499
+ There are four strategies for keeping the index synced with your database.
500
+
501
+ 1. Immediate (default)
502
+
503
+ Anytime a record is inserted, updated, or deleted
504
+
505
+ 2. Asynchronous
506
+
507
+ Use background jobs for better performance
508
+
509
+ ```ruby
510
+ class Product < ApplicationRecord
511
+ searchkick callbacks: :async
512
+ end
513
+ ```
514
+
515
+ Jobs are added to a queue named `searchkick`.
516
+
517
+ 3. Queuing
518
+
519
+ Push ids of records that need updated to a queue and reindex in the background in batches. This is more performant than the asynchronous method, which updates records individually. See [how to set up](#queuing).
520
+
521
+ 4. Manual
522
+
523
+ Turn off automatic syncing
524
+
525
+ ```ruby
526
+ class Product < ApplicationRecord
527
+ searchkick callbacks: false
528
+ end
529
+ ```
530
+
531
+ You can also do bulk updates.
532
+
533
+ ```ruby
534
+ Searchkick.callbacks(:bulk) do
535
+ User.find_each(&:update_fields)
536
+ end
537
+ ```
538
+
539
+ Or temporarily skip updates.
540
+
541
+ ```ruby
542
+ Searchkick.callbacks(false) do
543
+ User.find_each(&:update_fields)
544
+ end
545
+ ```
546
+
547
+ #### Associations
548
+
549
+ Data is **not** automatically synced when an association is updated. If this is desired, add a callback to reindex:
550
+
551
+ ```ruby
552
+ class Image < ApplicationRecord
553
+ belongs_to :product
554
+
555
+ after_commit :reindex_product
556
+
557
+ def reindex_product
558
+ product.reindex
559
+ end
560
+ end
561
+ ```
562
+
563
+ ### Analytics
564
+
565
+ The best starting point to improve your search **by far** is to track searches and conversions.
566
+
567
+ [Searchjoy](https://github.com/ankane/searchjoy) makes it easy.
568
+
569
+ ```ruby
570
+ Product.search "apple", track: {user_id: current_user.id}
571
+ ```
572
+
573
+ [See the docs](https://github.com/ankane/searchjoy) for how to install and use.
574
+
575
+ Focus on:
576
+
577
+ - top searches with low conversions
578
+ - top searches with no results
579
+
580
+ ### Keep Getting Better
581
+
582
+ Searchkick can use conversion data to learn what users are looking for. If a user searches for “ice cream” and adds Ben & Jerry’s Chunky Monkey to the cart (our conversion metric at Instacart), that item gets a little more weight for similar searches.
583
+
584
+ The first step is to define your conversion metric and start tracking conversions. The database works well for low volume, but feel free to use Redis or another datastore.
585
+
586
+ Searchkick automatically treats `apple` and `APPLE` the same.
587
+
588
+ Next, add conversions to the index.
589
+
590
+ ```ruby
591
+ class Product < ApplicationRecord
592
+ has_many :searches, class_name: "Searchjoy::Search", as: :convertable
593
+
594
+ searchkick conversions: [:conversions] # name of field
595
+
596
+ def search_data
597
+ {
598
+ name: name,
599
+ conversions: searches.group(:query).uniq.count(:user_id)
600
+ # {"ice cream" => 234, "chocolate" => 67, "cream" => 2}
601
+ }
602
+ end
603
+ end
604
+ ```
605
+
606
+ Reindex and set up a cron job to add new conversions daily.
607
+
608
+ ```ruby
609
+ rake searchkick:reindex CLASS=Product
610
+ ```
611
+
612
+ **Note:** For a more performant (but more advanced) approach, check out [performant conversions](#performant-conversions).
613
+
614
+ ### Personalized Results
615
+
616
+ Order results differently for each user. For example, show a user’s previously purchased products before other results.
617
+
618
+ ```ruby
619
+ class Product < ApplicationRecord
620
+ def search_data
621
+ {
622
+ name: name,
623
+ orderer_ids: orders.pluck(:user_id) # boost this product for these users
624
+ }
625
+ end
626
+ end
627
+ ```
628
+
629
+ Reindex and search with:
630
+
631
+ ```ruby
632
+ Product.search "milk", boost_where: {orderer_ids: current_user.id}
633
+ ```
634
+
635
+ ### Instant Search / Autocomplete
636
+
637
+ Autocomplete predicts what a user will type, making the search experience faster and easier.
638
+
639
+ ![Autocomplete](https://raw.githubusercontent.com/ankane/searchkick/gh-pages/autocomplete.png)
640
+
641
+ **Note:** To autocomplete on general categories (like `cereal` rather than product names), check out [Autosuggest](https://github.com/ankane/autosuggest).
642
+
643
+ **Note 2:** If you only have a few thousand records, don’t use Searchkick for autocomplete. It’s *much* faster to load all records into JavaScript and autocomplete there (eliminates network requests).
644
+
645
+ First, specify which fields use this feature. This is necessary since autocomplete can increase the index size significantly, but don’t worry - this gives you blazing faster queries.
646
+
647
+ ```ruby
648
+ class Movie < ApplicationRecord
649
+ searchkick word_start: [:title, :director]
650
+ end
651
+ ```
652
+
653
+ Reindex and search with:
654
+
655
+ ```ruby
656
+ Movie.search "jurassic pa", fields: [:title], match: :word_start
657
+ ```
658
+
659
+ Typically, you want to use a JavaScript library like [typeahead.js](https://twitter.github.io/typeahead.js/) or [jQuery UI](https://jqueryui.com/autocomplete/).
660
+
661
+ #### Here’s how to make it work with Rails
662
+
663
+ First, add a route and controller action.
664
+
665
+ ```ruby
666
+ class MoviesController < ApplicationController
667
+ def autocomplete
668
+ render json: Movie.search(params[:query], {
669
+ fields: ["title^5", "director"],
670
+ match: :word_start,
671
+ limit: 10,
672
+ load: false,
673
+ misspellings: {below: 5}
674
+ }).map(&:title)
675
+ end
676
+ end
677
+ ```
678
+
679
+ **Note:** Use `load: false` and `misspellings: {below: n}` (or `misspellings: false`) for best performance.
680
+
681
+ Then add the search box and JavaScript code to a view.
682
+
683
+ ```html
684
+ <input type="text" id="query" name="query" />
685
+
686
+ <script src="jquery.js"></script>
687
+ <script src="typeahead.bundle.js"></script>
688
+ <script>
689
+ var movies = new Bloodhound({
690
+ datumTokenizer: Bloodhound.tokenizers.whitespace,
691
+ queryTokenizer: Bloodhound.tokenizers.whitespace,
692
+ remote: {
693
+ url: '/movies/autocomplete?query=%QUERY',
694
+ wildcard: '%QUERY'
695
+ }
696
+ });
697
+ $('#query').typeahead(null, {
698
+ source: movies
699
+ });
700
+ </script>
701
+ ```
702
+
703
+ ### Suggestions
704
+
705
+ ![Suggest](https://raw.githubusercontent.com/ankane/searchkick/gh-pages/recursion.png)
706
+
707
+ ```ruby
708
+ class Product < ApplicationRecord
709
+ searchkick suggest: [:name] # fields to generate suggestions
710
+ end
711
+ ```
712
+
713
+ Reindex and search with:
714
+
715
+ ```ruby
716
+ products = Product.search "peantu butta", suggest: true
717
+ products.suggestions # ["peanut butter"]
718
+ ```
719
+
720
+ ### Aggregations
721
+
722
+ [Aggregations](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations.html) provide aggregated search data.
723
+
724
+ ![Aggregations](https://raw.githubusercontent.com/ankane/searchkick/gh-pages/facets.png)
725
+
726
+ ```ruby
727
+ products = Product.search "chuck taylor", aggs: [:product_type, :gender, :brand]
728
+ products.aggs
729
+ ```
730
+
731
+ By default, `where` conditions apply to aggregations.
732
+
733
+ ```ruby
734
+ Product.search "wingtips", where: {color: "brandy"}, aggs: [:size]
735
+ # aggregations for brandy wingtips are returned
736
+ ```
737
+
738
+ Change this with:
739
+
740
+ ```ruby
741
+ Product.search "wingtips", where: {color: "brandy"}, aggs: [:size], smart_aggs: false
742
+ # aggregations for all wingtips are returned
743
+ ```
744
+
745
+ Set `where` conditions for each aggregation separately with:
746
+
747
+ ```ruby
748
+ Product.search "wingtips", aggs: {size: {where: {color: "brandy"}}}
749
+ ```
750
+
751
+ Limit
752
+
753
+ ```ruby
754
+ Product.search "apples", aggs: {store_id: {limit: 10}}
755
+ ```
756
+
757
+ Order
758
+
759
+ ```ruby
760
+ Product.search "wingtips", aggs: {color: {order: {"_term" => "asc"}}} # alphabetically
761
+ ```
762
+
763
+ [All of these options are supported](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-terms-aggregation.html#search-aggregations-bucket-terms-aggregation-order)
764
+
765
+ Ranges
766
+
767
+ ```ruby
768
+ price_ranges = [{to: 20}, {from: 20, to: 50}, {from: 50}]
769
+ Product.search "*", aggs: {price: {ranges: price_ranges}}
770
+ ```
771
+
772
+ Minimum document count
773
+
774
+ ```ruby
775
+ Product.search "apples", aggs: {store_id: {min_doc_count: 2}}
776
+ ```
777
+
778
+ Date histogram
779
+
780
+ ```ruby
781
+ Product.search "pear", aggs: {products_per_year: {date_histogram: {field: :created_at, interval: :year}}}
782
+ ```
783
+
784
+ For other aggregation types, including sub-aggregations, use `body_options`:
785
+
786
+ ```ruby
787
+ Product.search "orange", body_options: {aggs: {price: {histogram: {field: :price, interval: 10}}}
788
+ ```
789
+
790
+ ### Highlight
791
+
792
+ Specify which fields to index with highlighting.
793
+
794
+ ```ruby
795
+ class Product < ApplicationRecord
796
+ searchkick highlight: [:name]
797
+ end
798
+ ```
799
+
800
+ Highlight the search query in the results.
801
+
802
+ ```ruby
803
+ bands = Band.search "cinema", highlight: true
804
+ ```
805
+
806
+ View the highlighted fields with:
807
+
808
+ ```ruby
809
+ bands.with_highlights.each do |band, highlights|
810
+ highlights[:name] # "Two Door <em>Cinema</em> Club"
811
+ end
812
+ ```
813
+
814
+ To change the tag, use:
815
+
816
+ ```ruby
817
+ Band.search "cinema", highlight: {tag: "<strong>"}
818
+ ```
819
+
820
+ To highlight and search different fields, use:
821
+
822
+ ```ruby
823
+ Band.search "cinema", fields: [:name], highlight: {fields: [:description]}
824
+ ```
825
+
826
+ By default, the entire field is highlighted. To get small snippets instead, use:
827
+
828
+ ```ruby
829
+ bands = Band.search "cinema", highlight: {fragment_size: 20}
830
+ bands.with_highlights(multiple: true).each do |band, highlights|
831
+ highlights[:name].join(" and ")
832
+ end
833
+ ```
834
+
835
+ Additional options can be specified for each field:
836
+
837
+ ```ruby
838
+ Band.search "cinema", fields: [:name], highlight: {fields: {name: {fragment_size: 200}}}
839
+ ```
840
+
841
+ You can find available highlight options in the [Elasticsearch reference](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-highlighting.html#_highlighted_fragments).
842
+
843
+ ### Similar Items
844
+
845
+ Find similar items.
846
+
847
+ ```ruby
848
+ product = Product.first
849
+ product.similar(fields: [:name], where: {size: "12 oz"})
850
+ ```
851
+
852
+ ### Geospatial Searches
853
+
854
+ ```ruby
855
+ class Restaurant < ApplicationRecord
856
+ searchkick locations: [:location]
857
+
858
+ def search_data
859
+ attributes.merge(location: {lat: latitude, lon: longitude})
860
+ end
861
+ end
862
+ ```
863
+
864
+ Reindex and search with:
865
+
866
+ ```ruby
867
+ Restaurant.search "pizza", where: {location: {near: {lat: 37, lon: -114}, within: "100mi"}} # or 160km
868
+ ```
869
+
870
+ Bounded by a box
871
+
872
+ ```ruby
873
+ Restaurant.search "sushi", where: {location: {top_left: {lat: 38, lon: -123}, bottom_right: {lat: 37, lon: -122}}}
874
+ ```
875
+
876
+ **Note:** `top_right` and `bottom_left` also work
877
+
878
+ Bounded by a polygon
879
+
880
+ ```ruby
881
+ Restaurant.search "dessert", where: {location: {geo_polygon: {points: [{lat: 38, lon: -123}, {lat: 39, lon: -123}, {lat: 37, lon: 122}]}}}
882
+ ```
883
+
884
+ ### Boost By Distance
885
+
886
+ Boost results by distance - closer results are boosted more
887
+
888
+ ```ruby
889
+ Restaurant.search "noodles", boost_by_distance: {location: {origin: {lat: 37, lon: -122}}}
890
+ ```
891
+
892
+ Also supports [additional options](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-function-score-query.html#_decay_functions)
893
+
894
+ ```ruby
895
+ Restaurant.search "wings", boost_by_distance: {location: {origin: {lat: 37, lon: -122}, function: "linear", scale: "30mi", decay: 0.5}}
896
+ ```
897
+
898
+ ### Geo Shapes
899
+
900
+ You can also index and search geo shapes.
901
+
902
+ ```ruby
903
+ class Restaurant < ApplicationRecord
904
+ searchkick geo_shape: {
905
+ bounds: {tree: "geohash", precision: "1km"}
906
+ }
907
+
908
+ def search_data
909
+ attributes.merge(
910
+ bounds: {
911
+ type: "envelope",
912
+ coordinates: [{lat: 4, lon: 1}, {lat: 2, lon: 3}]
913
+ }
914
+ )
915
+ end
916
+ end
917
+ ```
918
+
919
+ See the [Elasticsearch documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current/geo-shape.html) for details.
920
+
921
+ Find shapes intersecting with the query shape
922
+
923
+ ```ruby
924
+ Restaurant.search "soup", where: {bounds: {geo_shape: {type: "polygon", coordinates: [[{lat: 38, lon: -123}, ...]]}}}
925
+ ```
926
+
927
+ Falling entirely within the query shape
928
+
929
+ ```ruby
930
+ Restaurant.search "salad", where: {bounds: {geo_shape: {type: "circle", relation: "within", coordinates: [{lat: 38, lon: -123}], radius: "1km"}}}
931
+ ```
932
+
933
+ Not touching the query shape
934
+
935
+ ```ruby
936
+ Restaurant.search "burger", where: {bounds: {geo_shape: {type: "envelope", relation: "disjoint", coordinates: [{lat: 38, lon: -123}, {lat: 37, lon: -122}]}}}
937
+ ```
938
+
939
+ Containing the query shape
940
+
941
+ ```ruby
942
+ Restaurant.search "fries", where: {bounds: {geo_shape: {type: "envelope", relation: "contains", coordinates: [{lat: 38, lon: -123}, {lat: 37, lon: -122}]}}}
943
+ ```
944
+
945
+ ## Inheritance
946
+
947
+ Searchkick supports single table inheritance.
948
+
949
+ ```ruby
950
+ class Dog < Animal
951
+ end
952
+ ```
953
+
954
+ In your parent model, set:
955
+
956
+ ```ruby
957
+ class Animal < ApplicationRecord
958
+ searchkick inheritance: true
959
+ end
960
+ ```
961
+
962
+ The parent and child model can both reindex.
963
+
964
+ ```ruby
965
+ Animal.reindex
966
+ Dog.reindex # equivalent, all animals reindexed
967
+ ```
968
+
969
+ And to search, use:
970
+
971
+ ```ruby
972
+ Animal.search "*" # all animals
973
+ Dog.search "*" # just dogs
974
+ Animal.search "*", type: [Dog, Cat] # just cats and dogs
975
+ ```
976
+
977
+ **Note:** The `suggest` option retrieves suggestions from the parent at the moment.
978
+
979
+ ```ruby
980
+ Dog.search "airbudd", suggest: true # suggestions for all animals
981
+ ```
982
+
983
+ ## Debugging Queries
984
+
985
+ To help with debugging queries, you can use:
986
+
987
+ ```ruby
988
+ Product.search("soap", debug: true)
989
+ ```
990
+
991
+ This prints useful info to `stdout`.
992
+
993
+ See how Elasticsearch scores your queries with:
994
+
995
+ ```ruby
996
+ Product.search("soap", explain: true).response
997
+ ```
998
+
999
+ See how Elasticsearch tokenizes your queries with:
1000
+
1001
+ ```ruby
1002
+ Product.search_index.tokens("Dish Washer Soap", analyzer: "searchkick_index")
1003
+ # ["dish", "dishwash", "washer", "washersoap", "soap"]
1004
+
1005
+ Product.search_index.tokens("dishwasher soap", analyzer: "searchkick_search")
1006
+ # ["dishwashersoap"] - no match
1007
+
1008
+ Product.search_index.tokens("dishwasher soap", analyzer: "searchkick_search2")
1009
+ # ["dishwash", "soap"] - match!!
1010
+ ```
1011
+
1012
+ Partial matches
1013
+
1014
+ ```ruby
1015
+ Product.search_index.tokens("San Diego", analyzer: "searchkick_word_start_index")
1016
+ # ["s", "sa", "san", "d", "di", "die", "dieg", "diego"]
1017
+
1018
+ Product.search_index.tokens("dieg", analyzer: "searchkick_word_search")
1019
+ # ["dieg"] - match!!
1020
+ ```
1021
+
1022
+ See the [complete list of analyzers](https://github.com/ankane/searchkick/blob/31780ddac7a89eab1e0552a32b403f2040a37931/lib/searchkick/index_options.rb#L32).
1023
+
1024
+ ## Deployment
1025
+
1026
+ Searchkick uses `ENV["ELASTICSEARCH_URL"]` for the Elasticsearch server. This defaults to `http://localhost:9200`.
1027
+
1028
+ ### Heroku
1029
+
1030
+ Choose an add-on: [Bonsai](https://elements.heroku.com/addons/bonsai) or [Elastic Cloud](https://elements.heroku.com/addons/foundelasticsearch). [SearchBox](https://elements.heroku.com/addons/searchbox) does not work at the moment.
1031
+
1032
+ For Bonsai:
1033
+
1034
+ ```sh
1035
+ heroku addons:create bonsai
1036
+ heroku config:set ELASTICSEARCH_URL=`heroku config:get BONSAI_URL`
1037
+ ```
1038
+
1039
+ For Found:
1040
+
1041
+ ```sh
1042
+ heroku addons:create foundelasticsearch
1043
+ heroku addons:open foundelasticsearch
1044
+ ```
1045
+
1046
+ Visit the Shield page and reset your password. You’ll need to add the username and password to your url. Get the existing url with:
1047
+
1048
+ ```sh
1049
+ heroku config:get FOUNDELASTICSEARCH_URL
1050
+ ```
1051
+
1052
+ And add `elastic:password@` right after `https://`:
1053
+
1054
+ ```sh
1055
+ heroku config:set ELASTICSEARCH_URL=https://elastic:password@12345.us-east-1.aws.found.io
1056
+ ```
1057
+
1058
+ Then deploy and reindex:
1059
+
1060
+ ```sh
1061
+ heroku run rake searchkick:reindex CLASS=Product
1062
+ ```
1063
+
1064
+ ### Amazon Elasticsearch Service
1065
+
1066
+ Create an initializer `config/initializers/elasticsearch.rb` with:
1067
+
1068
+ ```ruby
1069
+ ENV["ELASTICSEARCH_URL"] = "https://es-domain-1234.us-east-1.es.amazonaws.com"
1070
+ ```
1071
+
1072
+ To use signed request, include in your Gemfile:
1073
+
1074
+ ```ruby
1075
+ gem 'faraday_middleware-aws-sigv4'
1076
+ ```
1077
+
1078
+ and add to your initializer:
1079
+
1080
+ ```ruby
1081
+ Searchkick.aws_credentials = {
1082
+ access_key_id: ENV["AWS_ACCESS_KEY_ID"],
1083
+ secret_access_key: ENV["AWS_SECRET_ACCESS_KEY"],
1084
+ region: "us-east-1"
1085
+ }
1086
+ ```
1087
+
1088
+ Then deploy and reindex:
1089
+
1090
+ ```sh
1091
+ rake searchkick:reindex CLASS=Product
1092
+ ```
1093
+
1094
+ ### Other
1095
+
1096
+ Create an initializer `config/initializers/elasticsearch.rb` with:
1097
+
1098
+ ```ruby
1099
+ ENV["ELASTICSEARCH_URL"] = "http://username:password@api.searchbox.io"
1100
+ ```
1101
+
1102
+ Then deploy and reindex:
1103
+
1104
+ ```sh
1105
+ rake searchkick:reindex CLASS=Product
1106
+ ```
1107
+
1108
+ ### Automatic Failover
1109
+
1110
+ Create an initializer `config/initializers/elasticsearch.rb` with multiple hosts:
1111
+
1112
+ ```ruby
1113
+ ENV["ELASTICSEARCH_URL"] = "http://localhost:9200,http://localhost:9201"
1114
+ ```
1115
+
1116
+ See [elasticsearch-transport](https://github.com/elastic/elasticsearch-ruby/blob/master/elasticsearch-transport) for a complete list of options.
1117
+
1118
+ ### Lograge
1119
+
1120
+ Add the following to `config/environments/production.rb`:
1121
+
1122
+ ```ruby
1123
+ config.lograge.custom_options = lambda do |event|
1124
+ options = {}
1125
+ options[:search] = event.payload[:searchkick_runtime] if event.payload[:searchkick_runtime].to_f > 0
1126
+ options
1127
+ end
1128
+ ```
1129
+
1130
+ See [Production Rails](https://github.com/ankane/production_rails) for other good practices.
1131
+
1132
+ ## Performance
1133
+
1134
+ ### JSON Generation
1135
+
1136
+ Significantly increase performance with faster JSON generation. Add [Oj](https://github.com/ohler55/oj) to your Gemfile.
1137
+
1138
+ ```ruby
1139
+ gem 'oj'
1140
+ ```
1141
+
1142
+ This speeds up all JSON generation and parsing in your application (automatically!)
1143
+
1144
+ ### Persistent HTTP Connections
1145
+
1146
+ Significantly increase performance with persistent HTTP connections. Add [Typhoeus](https://github.com/typhoeus/typhoeus) to your Gemfile and it’ll automatically be used.
1147
+
1148
+ ```ruby
1149
+ gem 'typhoeus'
1150
+ ```
1151
+
1152
+ To reduce log noise, create an initializer with:
1153
+
1154
+ ```ruby
1155
+ Ethon.logger = Logger.new(nil)
1156
+ ```
1157
+
1158
+ If you run into issues on Windows, check out [this post](https://www.rastating.com/fixing-issues-in-typhoeus-and-httparty-on-windows/).
1159
+
1160
+ ### Searchable Fields
1161
+
1162
+ By default, all string fields are searchable (can be used in `fields` option). Speed up indexing and reduce index size by only making some fields searchable. This disables the `_all` field unless it’s listed.
1163
+
1164
+ ```ruby
1165
+ class Product < ApplicationRecord
1166
+ searchkick searchable: [:name]
1167
+ end
1168
+ ```
1169
+
1170
+ ### Filterable Fields
1171
+
1172
+ By default, all string fields are filterable (can be used in `where` option). Speed up indexing and reduce index size by only making some fields filterable.
1173
+
1174
+ ```ruby
1175
+ class Product < ApplicationRecord
1176
+ searchkick filterable: [:brand]
1177
+ end
1178
+ ```
1179
+
1180
+ **Note:** Non-string fields are always filterable and should not be passed to this option.
1181
+
1182
+ ### Parallel Reindexing
1183
+
1184
+ For large data sets, you can use background jobs to parallelize reindexing.
1185
+
1186
+ ```ruby
1187
+ Product.reindex(async: true)
1188
+ # {index_name: "products_production_20170111210018065"}
1189
+ ```
1190
+
1191
+ Once the jobs complete, promote the new index with:
1192
+
1193
+ ```ruby
1194
+ Product.search_index.promote(index_name)
1195
+ ```
1196
+
1197
+ You can optionally track the status with Redis:
1198
+
1199
+ ```ruby
1200
+ Searchkick.redis = Redis.new
1201
+ ```
1202
+
1203
+ And use:
1204
+
1205
+ ```ruby
1206
+ Searchkick.reindex_status(index_name)
1207
+ ```
1208
+
1209
+ You can also have Searchkick wait for reindexing to complete
1210
+
1211
+ ```ruby
1212
+ Searchkick.reindex(async: {wait: true})
1213
+ ```
1214
+
1215
+ You can use [ActiveJob::TrafficControl](https://github.com/nickelser/activejob-traffic_control) to control concurrency. Install the gem:
1216
+
1217
+ ```ruby
1218
+ gem 'activejob-traffic_control', '>= 0.1.3'
1219
+ ```
1220
+
1221
+ And create an initializer with:
1222
+
1223
+ ```ruby
1224
+ ActiveJob::TrafficControl.client = Searchkick.redis
1225
+
1226
+ class Searchkick::BulkReindexJob
1227
+ concurrency 3
1228
+ end
1229
+ ```
1230
+
1231
+ This will allow only 3 jobs to run at once.
1232
+
1233
+ ### Refresh Interval
1234
+
1235
+ You can specify a longer refresh interval while reindexing to increase performance.
1236
+
1237
+ ```ruby
1238
+ Product.reindex(async: true, refresh_interval: "30s")
1239
+ ```
1240
+
1241
+ **Note:** This only makes a noticable difference with parallel reindexing.
1242
+
1243
+ When promoting, have it restored to the value in your mapping (defaults to `1s`).
1244
+
1245
+ ```ruby
1246
+ Product.search_index.promote(index_name, update_refresh_interval: true)
1247
+ ```
1248
+
1249
+ ### Queuing
1250
+
1251
+ Push ids of records needing reindexed to a queue and reindex in bulk for better performance. First, set up Redis in an initializer. We recommend using [connection_pool](https://github.com/mperham/connection_pool).
1252
+
1253
+ ```ruby
1254
+ Searchkick.redis = ConnectionPool.new { Redis.new }
1255
+ ```
1256
+
1257
+ And ask your models to queue updates.
1258
+
1259
+ ```ruby
1260
+ class Product < ApplicationRecord
1261
+ searchkick callbacks: :queue
1262
+ end
1263
+ ```
1264
+
1265
+ Then, set up a background job to run.
1266
+
1267
+ ```ruby
1268
+ Searchkick::ProcessQueueJob.perform_later(class_name: "Product")
1269
+ ```
1270
+
1271
+ You can check the queue length with:
1272
+
1273
+ ```ruby
1274
+ Product.search_index.reindex_queue.length
1275
+ ```
1276
+
1277
+ For more tips, check out [Keeping Elasticsearch in Sync](https://www.elastic.co/blog/found-keeping-elasticsearch-in-sync).
1278
+
1279
+ ### Routing
1280
+
1281
+ Searchkick supports [Elasticsearch’s routing feature](https://www.elastic.co/blog/customizing-your-document-routing), which can significantly speed up searches.
1282
+
1283
+ ```ruby
1284
+ class Business < ApplicationRecord
1285
+ searchkick routing: true
1286
+
1287
+ def search_routing
1288
+ city_id
1289
+ end
1290
+ end
1291
+ ```
1292
+
1293
+ Reindex and search with:
1294
+
1295
+ ```ruby
1296
+ Business.search "ice cream", routing: params[:city_id]
1297
+ ```
1298
+
1299
+ ### Partial Reindexing
1300
+
1301
+ Reindex a subset of attributes to reduce time spent generating search data and cut down on network traffic.
1302
+
1303
+ ```ruby
1304
+ class Product < ApplicationRecord
1305
+ def search_data
1306
+ {
1307
+ name: name
1308
+ }.merge(search_prices)
1309
+ end
1310
+
1311
+ def search_prices
1312
+ {
1313
+ price: price,
1314
+ sale_price: sale_price
1315
+ }
1316
+ end
1317
+ end
1318
+ ```
1319
+
1320
+ And use:
1321
+
1322
+ ```ruby
1323
+ Product.reindex(:search_prices)
1324
+ ```
1325
+
1326
+ ### Performant Conversions
1327
+
1328
+ Split out conversions into a separate method so you can use partial reindexing, and cache conversions to prevent N+1 queries. Be sure to use a centralized cache store like Memcached or Redis.
1329
+
1330
+ ```ruby
1331
+ class Product < ApplicationRecord
1332
+ def search_data
1333
+ {
1334
+ name: name
1335
+ }.merge(search_conversions)
1336
+ end
1337
+
1338
+ def search_conversions
1339
+ {
1340
+ conversions: Rails.cache.read("search_conversions:#{self.class.name}:#{id}") || {}
1341
+ }
1342
+ end
1343
+ end
1344
+ ```
1345
+
1346
+ Create a job to update the cache and reindex records with new conversions.
1347
+
1348
+ ```ruby
1349
+ class ReindexConversionsJob < ApplicationJob
1350
+ def perform(class_name)
1351
+ # get records that have a recent conversion
1352
+ recently_converted_ids =
1353
+ Searchjoy::Search.where("convertable_type = ? AND converted_at > ?", class_name, 1.day.ago)
1354
+ .order(:convertable_id).uniq.pluck(:convertable_id)
1355
+
1356
+ # split into groups
1357
+ recently_converted_ids.in_groups_of(1000, false) do |ids|
1358
+ # fetch conversions
1359
+ conversions =
1360
+ Searchjoy::Search.where(convertable_id: ids, convertable_type: class_name)
1361
+ .group(:convertable_id, :query).uniq.count(:user_id)
1362
+
1363
+ # group conversions by record
1364
+ conversions_by_record = {}
1365
+ conversions.each do |(id, query), count|
1366
+ (conversions_by_record[id] ||= {})[query] = count
1367
+ end
1368
+
1369
+ # write to cache
1370
+ conversions_by_record.each do |id, conversions|
1371
+ Rails.cache.write("search_conversions:#{class_name}:#{id}", conversions)
1372
+ end
1373
+
1374
+ # partial reindex
1375
+ class_name.constantize.where(id: ids).reindex(:search_conversions)
1376
+ end
1377
+ end
1378
+ end
1379
+ ```
1380
+
1381
+ Run the job with:
1382
+
1383
+ ```ruby
1384
+ ReindexConversionsJob.perform_later("Product")
1385
+ ```
1386
+
1387
+ ## Advanced
1388
+
1389
+ Searchkick makes it easy to use the Elasticsearch DSL on its own.
1390
+
1391
+ ### Advanced Mapping
1392
+
1393
+ Create a custom mapping:
1394
+
1395
+ ```ruby
1396
+ class Product < ApplicationRecord
1397
+ searchkick mappings: {
1398
+ product: {
1399
+ properties: {
1400
+ name: {type: "keyword"}
1401
+ }
1402
+ }
1403
+ }
1404
+ end
1405
+ ```
1406
+ **Note:** If you use a custom mapping, you'll need to use [custom searching](#advanced-search) as well.
1407
+
1408
+ To keep the mappings and settings generated by Searchkick, use:
1409
+
1410
+ ```ruby
1411
+ class Product < ApplicationRecord
1412
+ searchkick merge_mappings: true, mappings: {...}
1413
+ end
1414
+ ```
1415
+
1416
+ ### Advanced Search
1417
+
1418
+ And use the `body` option to search:
1419
+
1420
+ ```ruby
1421
+ products = Product.search body: {match: {name: "milk"}}
1422
+ ```
1423
+
1424
+ **Note:** This replaces the entire body, so other options are ignored.
1425
+
1426
+ View the response with:
1427
+
1428
+ ```ruby
1429
+ products.response
1430
+ ```
1431
+
1432
+ To modify the query generated by Searchkick, use:
1433
+
1434
+ ```ruby
1435
+ products = Product.search "milk", body_options: {min_score: 1}
1436
+ ```
1437
+
1438
+ or
1439
+
1440
+ ```ruby
1441
+ products =
1442
+ Product.search "apples" do |body|
1443
+ body[:min_score] = 1
1444
+ end
1445
+ ```
1446
+
1447
+ ### Elasticsearch Gem
1448
+
1449
+ Searchkick is built on top of the [elasticsearch](https://github.com/elastic/elasticsearch-ruby) gem. To access the client directly, use:
1450
+
1451
+ ```ruby
1452
+ Searchkick.client
1453
+ ```
1454
+
1455
+ ## Multi Search
1456
+
1457
+ To batch search requests for performance, use:
1458
+
1459
+ ```ruby
1460
+ fresh_products = Product.search("fresh", execute: false)
1461
+ frozen_products = Product.search("frozen", execute: false)
1462
+ Searchkick.multi_search([fresh_products, frozen_products])
1463
+ ```
1464
+
1465
+ Then use `fresh_products` and `frozen_products` as typical results.
1466
+
1467
+ **Note:** Errors are not raised as with single requests. Use the `error` method on each query to check for errors. Also, if you use the `below` option for misspellings, misspellings will be disabled.
1468
+
1469
+ ## Multiple Indices
1470
+
1471
+ Search across multiple indices with:
1472
+
1473
+ ```ruby
1474
+ Searchkick.search "milk", index_name: [Product, Category]
1475
+ ```
1476
+
1477
+ Boost specific indices with:
1478
+
1479
+ ```ruby
1480
+ indices_boost: {Category => 2, Product => 1}
1481
+ ```
1482
+
1483
+ ## Nested Data
1484
+
1485
+ To query nested data, use dot notation.
1486
+
1487
+ ```ruby
1488
+ User.search "san", fields: ["address.city"], where: {"address.zip_code" => 12345}
1489
+ ```
1490
+
1491
+ ## Reference
1492
+
1493
+ Reindex one record
1494
+
1495
+ ```ruby
1496
+ product = Product.find(1)
1497
+ product.reindex
1498
+ ```
1499
+
1500
+ Reindex multiple records
1501
+
1502
+ ```ruby
1503
+ Product.where(store_id: 1).reindex
1504
+ ```
1505
+
1506
+ Reindex associations
1507
+
1508
+ ```ruby
1509
+ store.products.reindex
1510
+ ```
1511
+
1512
+ Remove old indices
1513
+
1514
+ ```ruby
1515
+ Product.search_index.clean_indices
1516
+ ```
1517
+
1518
+ Use custom settings
1519
+
1520
+ ```ruby
1521
+ class Product < ApplicationRecord
1522
+ searchkick settings: {number_of_shards: 3}
1523
+ end
1524
+ ```
1525
+
1526
+ Use a different index name
1527
+
1528
+ ```ruby
1529
+ class Product < ApplicationRecord
1530
+ searchkick index_name: "products_v2"
1531
+ end
1532
+ ```
1533
+
1534
+ Use a dynamic index name
1535
+
1536
+ ```ruby
1537
+ class Product < ApplicationRecord
1538
+ searchkick index_name: -> { "#{name.tableize}-#{I18n.locale}" }
1539
+ end
1540
+ ```
1541
+
1542
+ Prefix the index name
1543
+
1544
+ ```ruby
1545
+ class Product < ApplicationRecord
1546
+ searchkick index_prefix: "datakick"
1547
+ end
1548
+ ```
1549
+
1550
+ For all models
1551
+
1552
+ ```ruby
1553
+ Searchkick.index_prefix = "datakick"
1554
+ ```
1555
+
1556
+ Use a different term for boosting by conversions
1557
+
1558
+ ```ruby
1559
+ Product.search("banana", conversions_term: "organic banana")
1560
+ ```
1561
+
1562
+ Multiple conversion fields
1563
+
1564
+ ```ruby
1565
+ class Product < ApplicationRecord
1566
+ has_many :searches, class_name: "Searchjoy::Search"
1567
+
1568
+ # searchkick also supports multiple "conversions" fields
1569
+ searchkick conversions: ["unique_user_conversions", "total_conversions"]
1570
+
1571
+ def search_data
1572
+ {
1573
+ name: name,
1574
+ unique_user_conversions: searches.group(:query).uniq.count(:user_id),
1575
+ # {"ice cream" => 234, "chocolate" => 67, "cream" => 2}
1576
+ total_conversions: searches.group(:query).count
1577
+ # {"ice cream" => 412, "chocolate" => 117, "cream" => 6}
1578
+ }
1579
+ end
1580
+ end
1581
+ ```
1582
+
1583
+ and during query time:
1584
+
1585
+ ```ruby
1586
+ Product.search("banana") # boost by both fields (default)
1587
+ Product.search("banana", conversions: "total_conversions") # only boost by total_conversions
1588
+ Product.search("banana", conversions: false) # no conversion boosting
1589
+ ```
1590
+
1591
+ Change timeout
1592
+
1593
+ ```ruby
1594
+ Searchkick.timeout = 15 # defaults to 10
1595
+ ```
1596
+
1597
+ Set a lower timeout for searches
1598
+
1599
+ ```ruby
1600
+ Searchkick.search_timeout = 3
1601
+ ```
1602
+
1603
+ Change the search method name
1604
+
1605
+ ```ruby
1606
+ Searchkick.search_method_name = :lookup
1607
+ ```
1608
+
1609
+ Change search queue name
1610
+
1611
+ ```ruby
1612
+ Searchkick.queue_name = :search_reindex
1613
+ ```
1614
+
1615
+ Eager load associations
1616
+
1617
+ ```ruby
1618
+ Product.search "milk", includes: [:brand, :stores]
1619
+ ```
1620
+
1621
+ Eager load different associations by model
1622
+
1623
+ ```ruby
1624
+ Searchkick.search("*", index_name: [Product, Store], model_includes: {Product => [:store], Store => [:product]})
1625
+ ```
1626
+
1627
+ Run additional scopes on results
1628
+
1629
+ ```ruby
1630
+ Product.search "milk", scope_results: ->(r) { r.with_attached_images }
1631
+ ```
1632
+
1633
+ Specify default fields to search
1634
+
1635
+ ```ruby
1636
+ class Product < ApplicationRecord
1637
+ searchkick default_fields: [:name]
1638
+ end
1639
+ ```
1640
+
1641
+ Turn off special characters
1642
+
1643
+ ```ruby
1644
+ class Product < ApplicationRecord
1645
+ # A will not match Ä
1646
+ searchkick special_characters: false
1647
+ end
1648
+ ```
1649
+
1650
+ Use a different [similarity algorithm](https://www.elastic.co/guide/en/elasticsearch/reference/current/index-modules-similarity.html) for scoring
1651
+
1652
+ ```ruby
1653
+ class Product < ApplicationRecord
1654
+ searchkick similarity: "classic"
1655
+ end
1656
+ ```
1657
+
1658
+ Change import batch size
1659
+
1660
+ ```ruby
1661
+ class Product < ApplicationRecord
1662
+ searchkick batch_size: 200 # defaults to 1000
1663
+ end
1664
+ ```
1665
+
1666
+ Create index without importing
1667
+
1668
+ ```ruby
1669
+ Product.reindex(import: false)
1670
+ ```
1671
+
1672
+ Lazy searching
1673
+
1674
+ ```ruby
1675
+ products = Product.search("carrots", execute: false)
1676
+ products.each { ... } # search not executed until here
1677
+ ```
1678
+
1679
+ Add [request parameters](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-uri-request.html), like `search_type` and `query_cache`
1680
+
1681
+ ```ruby
1682
+ Product.search("carrots", request_params: {search_type: "dfs_query_then_fetch"})
1683
+ ```
1684
+
1685
+ Reindex conditionally
1686
+
1687
+ ```ruby
1688
+ class Product < ApplicationRecord
1689
+ searchkick callbacks: false
1690
+
1691
+ # add the callbacks manually
1692
+ after_commit :reindex, if: -> (model) { model.previous_changes.key?("name") } # use your own condition
1693
+ end
1694
+ ```
1695
+
1696
+ Reindex all models - Rails only
1697
+
1698
+ ```sh
1699
+ rake searchkick:reindex:all
1700
+ ```
1701
+
1702
+ Turn on misspellings after a certain number of characters
1703
+
1704
+ ```ruby
1705
+ Product.search "api", misspellings: {prefix_length: 2} # api, apt, no ahi
1706
+ ```
1707
+
1708
+ **Note:** With this option, if the query length is the same as `prefix_length`, misspellings are turned off
1709
+
1710
+ ```ruby
1711
+ Product.search "ah", misspellings: {prefix_length: 2} # ah, no aha
1712
+ ```
1713
+
1714
+ ## Testing
1715
+
1716
+ For performance, only enable Searchkick callbacks for the tests that need it.
1717
+
1718
+ ### Minitest
1719
+
1720
+ Add to your `test/test_helper.rb`:
1721
+
1722
+ ```ruby
1723
+ # reindex models
1724
+ Product.reindex
1725
+
1726
+ # and disable callbacks
1727
+ Searchkick.disable_callbacks
1728
+ ```
1729
+
1730
+ And use:
1731
+
1732
+ ```ruby
1733
+ class ProductTest < Minitest::Test
1734
+ def setup
1735
+ Searchkick.enable_callbacks
1736
+ end
1737
+
1738
+ def teardown
1739
+ Searchkick.disable_callbacks
1740
+ end
1741
+
1742
+ def test_search
1743
+ Product.create!(name: "Apple")
1744
+ Product.search_index.refresh
1745
+ assert_equal ["Apple"], Product.search("apple").map(&:name)
1746
+ end
1747
+ end
1748
+ ```
1749
+
1750
+ ### RSpec
1751
+
1752
+ Add to your `spec/spec_helper.rb`:
1753
+
1754
+ ```ruby
1755
+ RSpec.configure do |config|
1756
+ config.before(:suite) do
1757
+ # reindex models
1758
+ Product.reindex
1759
+
1760
+ # and disable callbacks
1761
+ Searchkick.disable_callbacks
1762
+ end
1763
+
1764
+ config.around(:each, search: true) do |example|
1765
+ Searchkick.callbacks(true) do
1766
+ example.run
1767
+ end
1768
+ end
1769
+ end
1770
+ ```
1771
+
1772
+ And use:
1773
+
1774
+ ```ruby
1775
+ describe Product, search: true do
1776
+ it "searches" do
1777
+ Product.create!(name: "Apple")
1778
+ Product.search_index.refresh
1779
+ assert_equal ["Apple"], Product.search("apple").map(&:name)
1780
+ end
1781
+ end
1782
+ ```
1783
+
1784
+ ### Factory Girl
1785
+
1786
+ Use a trait and an after `create` hook for each indexed model:
1787
+
1788
+ ```ruby
1789
+ FactoryGirl.define do
1790
+ factory :product do
1791
+ # ...
1792
+
1793
+ # Note: This should be the last trait in the list so `reindex` is called
1794
+ # after all the other callbacks complete.
1795
+ trait :reindex do
1796
+ after(:create) do |product, _evaluator|
1797
+ product.reindex(refresh: true)
1798
+ end
1799
+ end
1800
+ end
1801
+ end
1802
+
1803
+ # use it
1804
+ FactoryGirl.create(:product, :some_trait, :reindex, some_attribute: "foo")
1805
+ ```
1806
+
1807
+ ### Parallel Tests
1808
+
1809
+ Set:
1810
+
1811
+ ```ruby
1812
+ Searchkick.index_suffix = ENV["TEST_ENV_NUMBER"]
1813
+ ```
1814
+
1815
+ ## Multi-Tenancy
1816
+
1817
+ Check out [this great post](https://www.tiagoamaro.com.br/2014/12/11/multi-tenancy-with-searchkick/) on the [Apartment](https://github.com/influitive/apartment) gem. Follow a similar pattern if you use another gem.
1818
+
1819
+ ## Elasticsearch 5 to 6 Upgrade
1820
+
1821
+ Elasticsearch 6 removes the ability to reindex with the `_all` field. Before you upgrade, we recommend disabling this field manually and specifying default fields on your models.
1822
+
1823
+ ```ruby
1824
+ class Product < ApplicationRecord
1825
+ searchkick _all: false, default_fields: [:name]
1826
+ end
1827
+ ```
1828
+
1829
+ If you need search across multiple fields, we recommend creating a similar field in your search data.
1830
+
1831
+ ```ruby
1832
+ class Product < ApplicationRecord
1833
+ def search_data
1834
+ {
1835
+ all: [name, size, quantity].join(" ")
1836
+ }
1837
+ end
1838
+ end
1839
+ ```
1840
+
1841
+ ## Elasticsearch Gotchas
1842
+
1843
+ ### Consistency
1844
+
1845
+ Elasticsearch is eventually consistent, meaning it can take up to a second for a change to reflect in search. You can use the `refresh` method to have it show up immediately.
1846
+
1847
+ ```ruby
1848
+ product.save!
1849
+ Product.search_index.refresh
1850
+ ```
1851
+
1852
+ ### Inconsistent Scores
1853
+
1854
+ Due to the distributed nature of Elasticsearch, you can get incorrect results when the number of documents in the index is low. You can [read more about it here](https://www.elastic.co/blog/understanding-query-then-fetch-vs-dfs-query-then-fetch). To fix this, do:
1855
+
1856
+ ```ruby
1857
+ class Product < ApplicationRecord
1858
+ searchkick settings: {number_of_shards: 1}
1859
+ end
1860
+ ```
1861
+
1862
+ For convenience, this is set by default in the test environment.
1863
+
1864
+ ## History
1865
+
1866
+ View the [changelog](https://github.com/ankane/searchkick/blob/master/CHANGELOG.md).
1867
+
1868
+ ## Thanks
1869
+
1870
+ Thanks to Karel Minarik for [Elasticsearch Ruby](https://github.com/elasticsearch/elasticsearch-ruby) and [Tire](https://github.com/karmi/retire), Jaroslav Kalistsuk for [zero downtime reindexing](https://gist.github.com/jarosan/3124884), and Alex Leschenko for [Elasticsearch autocomplete](https://github.com/leschenko/elasticsearch_autocomplete).
1871
+
1872
+ ## Roadmap
1873
+
1874
+ - Reindex API
1875
+ - Incorporate human eval
1876
+
1877
+ ## Contributing
1878
+
1879
+ Everyone is encouraged to help improve this project. Here are a few ways you can help:
1880
+
1881
+ - [Report bugs](https://github.com/ankane/searchkick/issues)
1882
+ - Fix bugs and [submit pull requests](https://github.com/ankane/searchkick/pulls)
1883
+ - Write, clarify, or fix documentation
1884
+ - Suggest or add new features
1885
+
1886
+ If you’re looking for ideas, [try here](https://github.com/ankane/searchkick/issues?q=is%3Aissue+is%3Aopen+label%3A%22help+wanted%22).
1887
+
1888
+ To get started with development and testing:
1889
+
1890
+ ```sh
1891
+ git clone https://github.com/ankane/searchkick.git
1892
+ cd searchkick
1893
+ bundle install
1894
+ rake test
1895
+ ```