caoutsearch 0.0.5 → 0.0.7
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/README.md +34 -788
- data/lib/caoutsearch/search/batch/search_after.rb +29 -9
- data/lib/caoutsearch/search/inspect.rb +1 -1
- data/lib/caoutsearch/search/internal_dsl.rb +7 -2
- data/lib/caoutsearch/search/query_builder/contexts.rb +34 -0
- data/lib/caoutsearch/search/query_builder.rb +2 -8
- data/lib/caoutsearch/search/search_methods.rb +10 -5
- data/lib/caoutsearch/testing/mock_requests.rb +42 -16
- data/lib/caoutsearch/version.rb +1 -1
- metadata +214 -3
data/README.md
CHANGED
@@ -1,13 +1,13 @@
|
|
1
1
|
# Caoutsearch [\ˈkawt͡ˈsɝtʃ\\](http://ipa-reader.xyz/?text=ˈkawt͡ˈsɝtʃ)
|
2
2
|
|
3
|
-
[![Gem Version](https://badge.fury.io/rb/caoutsearch.svg)](https://rubygems.org/gems/caoutsearch)
|
4
|
-
[![CI Status](https://github.com/mon-territoire/caoutsearch/actions/workflows/ci.yml/badge.svg)](https://github.com/mon-territoire/caoutsearch/actions/workflows/ci.yml)
|
5
|
-
[![Ruby Style Guide](https://img.shields.io/badge/code_style-standard-brightgreen.svg)](https://github.com/testdouble/standard)
|
6
|
-
[![Maintainability](https://api.codeclimate.com/v1/badges/fbe73db3fd8be9a10e12/maintainability)](https://codeclimate.com/github/mon-territoire/caoutsearch/maintainability)
|
7
|
-
[![Test Coverage](https://api.codeclimate.com/v1/badges/fbe73db3fd8be9a10e12/test_coverage)](https://codeclimate.com/github/mon-territoire/caoutsearch/test_coverage)
|
3
|
+
<span>[![Gem Version](https://badge.fury.io/rb/caoutsearch.svg)](https://rubygems.org/gems/caoutsearch)</span> <span>
|
4
|
+
[![CI Status](https://github.com/mon-territoire/caoutsearch/actions/workflows/ci.yml/badge.svg)](https://github.com/mon-territoire/caoutsearch/actions/workflows/ci.yml)</span> <span>
|
5
|
+
[![Ruby Style Guide](https://img.shields.io/badge/code_style-standard-brightgreen.svg)](https://github.com/testdouble/standard)</span> <span>
|
6
|
+
[![Maintainability](https://api.codeclimate.com/v1/badges/fbe73db3fd8be9a10e12/maintainability)](https://codeclimate.com/github/mon-territoire/caoutsearch/maintainability)</span> <span>
|
7
|
+
[![Test Coverage](https://api.codeclimate.com/v1/badges/fbe73db3fd8be9a10e12/test_coverage)](https://codeclimate.com/github/mon-territoire/caoutsearch/test_coverage)</span>
|
8
8
|
|
9
|
-
[![JRuby](https://github.com/mon-territoire/caoutsearch/actions/workflows/jruby.yml/badge.svg)](https://github.com/mon-territoire/caoutsearch/actions/workflows/jruby.yml)
|
10
|
-
[![Truffle Ruby](https://github.com/mon-territoire/caoutsearch/actions/workflows/truffle_ruby.yml/badge.svg)](https://github.com/mon-territoire/caoutsearch/actions/workflows/truffle_ruby.yml)
|
9
|
+
<span>[![JRuby](https://github.com/mon-territoire/caoutsearch/actions/workflows/jruby.yml/badge.svg)](https://github.com/mon-territoire/caoutsearch/actions/workflows/jruby.yml)</span> <span>
|
10
|
+
[![Truffle Ruby](https://github.com/mon-territoire/caoutsearch/actions/workflows/truffle_ruby.yml/badge.svg)](https://github.com/mon-territoire/caoutsearch/actions/workflows/truffle_ruby.yml)</span>
|
11
11
|
|
12
12
|
**!! Gem under development before public release !!**
|
13
13
|
|
@@ -17,177 +17,45 @@ It provides a simple but powerful DSL to perform complex indexing and searching,
|
|
17
17
|
Caoutsearch only supports Elasticsearch 8.x right now.
|
18
18
|
It is used in production in a robust application, updated and maintained for several years at [Mon Territoire](https://mon-territoire.fr).
|
19
19
|
|
20
|
-
Caoutsearch was inspired by awesome gems such as [elasticsearch-rails](https://github.com/elastic/elasticsearch-rails) or [search_flip](https://github.com/mrkamel/search_flip).
|
21
|
-
|
20
|
+
Caoutsearch was inspired by awesome gems such as [elasticsearch-rails](https://github.com/elastic/elasticsearch-rails) or [search_flip](https://github.com/mrkamel/search_flip).
|
21
|
+
Depending on your search scenarios, they may better suite your needs.
|
22
22
|
|
23
|
-
##
|
23
|
+
## Documentation
|
24
24
|
|
25
|
-
|
26
|
-
- [Configuration](#configuration)
|
27
|
-
- Instrumentation
|
28
|
-
- [Usage](#usage)
|
29
|
-
- [Indice Configuration](#indice-configuration)
|
30
|
-
- Mapping & settings
|
31
|
-
- Text analysis
|
32
|
-
- Versionning
|
33
|
-
- [Index Engine](#index-engine)
|
34
|
-
- Properties
|
35
|
-
- Partial updates
|
36
|
-
- Eager loading
|
37
|
-
- Interdependencies
|
38
|
-
- [Search Engine](#search-engine)
|
39
|
-
- Queries
|
40
|
-
- [Filters](#filters)
|
41
|
-
- Full-text query
|
42
|
-
- Custom filters
|
43
|
-
- Orders
|
44
|
-
- [Aggregations](#aggregations)
|
45
|
-
- [Transform aggregations](#transform-aggregations)
|
46
|
-
- [Responses](#responses)
|
47
|
-
- [Loading records](#loading-records)
|
48
|
-
- [Model integration](#model-integration)
|
49
|
-
- [Add Caoutsearch to your models](#add-caoutsearch-to-your-models)
|
50
|
-
- [Index records](#index-records)
|
51
|
-
- [Index multiple records](#index-multiple-records)
|
52
|
-
- [Index single records](#index-single-records)
|
53
|
-
- [Delete documents](#delete-documents)
|
54
|
-
- [Automatic Callbacks](#automatic-callbacks)
|
55
|
-
- Asynchronous methods
|
56
|
-
- [Search for records](#search-for-records)
|
57
|
-
- [Search API](#search-api)
|
58
|
-
- [Pagination](#pagination)
|
59
|
-
- [Total count](#total-count)
|
60
|
-
- [Iterating results](#iterating-results)
|
61
|
-
- [Testing with Caoutsearch](#testing-with-Caoutsearch)
|
25
|
+
Visit our [offical documentation](https://mon-territoire.github.io/caoutsearch) to understand how to use Caoutsearch.
|
62
26
|
|
63
27
|
## Installation
|
64
28
|
|
29
|
+
Add the gem in your Gemfile:
|
30
|
+
|
65
31
|
```bash
|
66
32
|
bundle add caoutsearch
|
67
33
|
```
|
68
34
|
|
69
|
-
##
|
70
|
-
|
71
|
-
TODO
|
72
|
-
|
73
|
-
## Usage
|
74
|
-
|
75
|
-
### Indice Configuration
|
76
|
-
|
77
|
-
TODO
|
35
|
+
## Overview
|
78
36
|
|
79
|
-
|
37
|
+
Caoutsearch let you create `Index` and `Search` classes to manipulate your data :
|
80
38
|
|
81
|
-
TODO
|
82
|
-
|
83
|
-
### Search Engine
|
84
|
-
|
85
|
-
#### Filters
|
86
|
-
Filters declared in the search engine will define how Caoutsearch will build the queries
|
87
|
-
|
88
|
-
The main use of filters is to expose a field for search, but they can also be used to build more complex queries:
|
89
39
|
```ruby
|
90
|
-
class
|
91
|
-
|
92
|
-
|
40
|
+
class ArticleIndex < Caoutsearch::Index::Base
|
41
|
+
property :title
|
42
|
+
property :published_on
|
43
|
+
property :tags
|
93
44
|
|
94
|
-
|
95
|
-
|
96
|
-
|
97
|
-
# Build a more complex filter by using other filters
|
98
|
-
filter :public, as: :boolean
|
99
|
-
filter :published_on, as: :date
|
100
|
-
filter :active do |value|
|
101
|
-
search_by(published: value, published_on: value)
|
45
|
+
def tags
|
46
|
+
records.tags.public.map(&:to_s)
|
102
47
|
end
|
103
48
|
end
|
104
|
-
```
|
105
|
-
|
106
|
-
Caoutsearch different types of filters to handle different types of data or ways to search them:
|
107
|
-
|
108
|
-
##### Default filter
|
109
|
-
|
110
|
-
##### Boolean filter
|
111
49
|
|
112
|
-
|
50
|
+
ArticleIndex.reindex(:tags)
|
51
|
+
```
|
113
52
|
|
114
|
-
For a date filter defined like this:
|
115
53
|
```ruby
|
116
54
|
class ArticleSearch < Caoutsearch::Search::Base
|
117
|
-
|
118
|
-
|
55
|
+
filter :title, as: :match
|
119
56
|
filter :published_on, as: :date
|
120
|
-
|
121
|
-
```
|
122
|
-
|
123
|
-
You can now search the matching index with the `published_on` criterion:
|
124
|
-
```ruby
|
125
|
-
Article.search(published_on: Date.today)
|
126
|
-
```
|
127
|
-
|
128
|
-
and the following query will be generated to send to elasticsearch:
|
129
|
-
```json
|
130
|
-
{
|
131
|
-
"query": {
|
132
|
-
"bool": {
|
133
|
-
"filter": [
|
134
|
-
{ "range": { "published_on": { "gte": "2022-23-11", "lte": "2022-23-11"}}}
|
135
|
-
]
|
136
|
-
}
|
137
|
-
}
|
138
|
-
}
|
139
|
-
```
|
140
|
-
|
141
|
-
The date filter accepts multiple types of arguments :
|
142
|
-
|
143
|
-
```ruby
|
144
|
-
# Search for articles published on a date:
|
145
|
-
Article.search(published_on: Date.today)
|
146
|
-
|
147
|
-
# Search for articles published before a date:
|
148
|
-
Article.search(published_on: { less_than: "2022-12-25" })
|
149
|
-
Article.search(published_on: { less_than_or_equal: "2022-12-25" })
|
150
|
-
Article.search(published_on: ..Date.new(2022, 12, 25))
|
151
|
-
Article.search(published_on: [[nil, "now-2w/d"]])
|
152
|
-
|
153
|
-
# Search for articles published after a date:
|
154
|
-
Article.search(published_on: { greater_than: "2022-12-25" })
|
155
|
-
Article.search(published_on: { greater_than_or_equal: "2022-12-25" })
|
156
|
-
Article.search(published_on: Date.new(2022, 12, 25)..)
|
157
|
-
Article.search(published_on: [["now-1w/d", nil]])
|
158
|
-
|
159
|
-
# Search for articles published between two dates:
|
160
|
-
Article.search(published_on: { greater_than: "2022-12-25", less_than: "2023-12-25" })
|
161
|
-
Article.search(published_on: Date.new(2022, 12, 25)..Date.new(2023, 12, 25))
|
162
|
-
Article.search(published_on: [["now-1w/d", "now/d"]])
|
163
|
-
```
|
164
|
-
|
165
|
-
Dates of various formats are handled:
|
166
|
-
```ruby
|
167
|
-
"2022-10-11"
|
168
|
-
Date.today
|
169
|
-
Time.zone.now
|
170
|
-
```
|
57
|
+
filter :tags
|
171
58
|
|
172
|
-
We also support elasticsearch's date math
|
173
|
-
```ruby
|
174
|
-
"now-1h"
|
175
|
-
"now+2w/d"
|
176
|
-
```
|
177
|
-
|
178
|
-
##### GeoPoint filter
|
179
|
-
|
180
|
-
##### Match filter
|
181
|
-
|
182
|
-
##### Range filter
|
183
|
-
|
184
|
-
#### Aggregations
|
185
|
-
|
186
|
-
You can define simple to complex aggregations.
|
187
|
-
|
188
|
-
````ruby
|
189
|
-
class ArticleSearch < Caoutsearch::Search::Base
|
190
|
-
has_aggregation :view_count, { sum: { field: :view_count } }
|
191
59
|
has_aggregation :popular_tags, {
|
192
60
|
filter: { term: { published: true } },
|
193
61
|
aggs: {
|
@@ -197,642 +65,14 @@ class ArticleSearch < Caoutsearch::Search::Base
|
|
197
65
|
}
|
198
66
|
}
|
199
67
|
end
|
200
|
-
````
|
201
|
-
|
202
|
-
Then you can request one or more aggregations at the same time or chain the `aggregate` method.
|
203
|
-
The `aggregations` method will trigger a request and returns a [Response::Aggregations](#responses).
|
204
|
-
|
205
|
-
````ruby
|
206
|
-
ArticleSearch.aggregate(:view_count).aggregations
|
207
|
-
# ArticleSearch Search { "body": { "aggs": { "view_count": { "sum": { "field": "view_count" }}}}}
|
208
|
-
# ArticleSearch Search (10ms / took 5ms)
|
209
|
-
=> #<Caoutsearch::Response::Aggregations view_count=#<Caoutsearch::Response::Response value=119652>>
|
210
|
-
|
211
|
-
ArticleSearch.aggregate(:view_count, :popular_tags).aggregations
|
212
|
-
# ArticleSearch Search { "body": { "aggs": { "view_count": {…}, "popular_tags": {…}}}}
|
213
|
-
# ArticleSearch Search (10ms / took 5ms)
|
214
|
-
=> #<Caoutsearch::Response::Aggregations view_count=#<Caoutsearch::Response::Response value=119652> popular_tags=#<Caoutsearch::Response::Response buckets=…>>
|
215
|
-
|
216
|
-
ArticleSearch.aggregate(:view_count).aggregate(:popular_tags).aggregations
|
217
|
-
# ArticleSearch Search { "body": { "aggs": { "view_count": {…}, "popular_tags": {…}}}}
|
218
|
-
# ArticleSearch Search (10ms / took 5ms)
|
219
|
-
=> #<Caoutsearch::Response::Aggregations view_count=#<Caoutsearch::Response::Response value=119652> popular_tags=#<Caoutsearch::Response::Response buckets=…>>
|
220
|
-
````
|
221
|
-
|
222
|
-
You can create powerful aggregations using blocks and pass arguments to them.
|
223
|
-
|
224
|
-
````ruby
|
225
|
-
class ArticleSearch < Caoutsearch::Search::Base
|
226
|
-
has_aggregation :popular_tags_since do |date|
|
227
|
-
raise TypeError unless date.is_a?(Date)
|
228
|
-
|
229
|
-
query.aggregations[:popular_tags_since] = {
|
230
|
-
filter: { range: { publication_date: { gte: date.to_s } } },
|
231
|
-
aggs: {
|
232
|
-
published: {
|
233
|
-
terms: { field: :tags, size: 20 }
|
234
|
-
}
|
235
|
-
}
|
236
|
-
}
|
237
|
-
end
|
238
|
-
end
|
239
|
-
|
240
|
-
ArticleSearch.aggregate(popular_tags_since: 1.day.ago).aggregations
|
241
|
-
# ArticleSearch Search { "body": { "aggs": { "popular_tags_since": {…}}}}
|
242
|
-
# ArticleSearch Search (10ms / took 5ms)
|
243
|
-
=> #<Caoutsearch::Response::Aggregations popular_tags_since=#<Caoutsearch::Response::Response …
|
244
|
-
````
|
245
|
-
|
246
|
-
Only one argument can be passed to an aggregation block.
|
247
|
-
Use an Array or a Hash if you need to pass multiple options.
|
248
|
-
|
249
|
-
````ruby
|
250
|
-
class ArticleSearch < Caoutsearch::Search::Base
|
251
|
-
has_aggregation :popular_tags_since do |options|
|
252
|
-
# …
|
253
|
-
end
|
254
|
-
|
255
|
-
has_aggregation :popular_tags_between do |(first_date, end_date)|
|
256
|
-
# …
|
257
|
-
end
|
258
|
-
end
|
259
|
-
|
260
|
-
ArticleSearch.aggregate(popular_tags_since: { date: 1.day.ago, size: 20 })
|
261
|
-
ArticleSearch.aggregate(popular_tags_between: [date1, date2])
|
262
|
-
````
|
263
|
-
|
264
|
-
Finally, you can create a "catch-all" aggregation to handle cumbersome behaviors:
|
265
|
-
|
266
|
-
````ruby
|
267
|
-
class ArticleSearch < Caoutsearch::Search::Base
|
268
|
-
has_aggregation do |name, options = {}|
|
269
|
-
raise "unxpected_error" unless name.match?(/^view_count_(?<year>\d{4})$/)
|
270
|
-
|
271
|
-
query.aggregations[name] = {
|
272
|
-
filter: { term: { year: $LAST_LATCH_INFO[:year] } },
|
273
|
-
aggs: {
|
274
|
-
filtered: {
|
275
|
-
sum: { field: :view_count }
|
276
|
-
}
|
277
|
-
}
|
278
|
-
}
|
279
|
-
end
|
280
|
-
end
|
281
|
-
|
282
|
-
ArticleSearch.aggregate(:view_count_2020, :view_count_2019).aggregations
|
283
|
-
# ArticleSearch Search { "body": { "aggs": { "view_count_2020": {…}, "view_count_2019": {…}}}}
|
284
|
-
# ArticleSearch Search (10ms / took 5ms)
|
285
|
-
=> #<Caoutsearch::Response::Aggregations view_count_2020=#<Caoutsearch::Response::Response …
|
286
|
-
````
|
287
|
-
|
288
|
-
#### Transform aggregations
|
289
|
-
|
290
|
-
When using [buckets aggregation](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket.html) and/or [pipeline aggregation](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-pipeline.html), the path to the expected values can get complicated and become subject to unexpected changes for a public API.
|
291
|
-
|
292
|
-
````ruby
|
293
|
-
ArticleSearch.aggregate(popular_tags_since: 1.month.ago).aggregations.popular_tags_since.published.buckets.pluck(:key)
|
294
|
-
=> ["Blog", "Tech", …]
|
295
|
-
````
|
296
|
-
|
297
|
-
Instead, you can define transformations to provide simpler access to aggregated data:
|
298
|
-
|
299
|
-
````ruby
|
300
|
-
class ArticleSearch < Caoutsearch::Search::Base
|
301
|
-
has_aggregation :popular_tags_since do |since|
|
302
|
-
# …
|
303
|
-
end
|
304
|
-
|
305
|
-
transform_aggregation :popular_tags_since do |aggs|
|
306
|
-
aggs.dig(:popular_tags_since, :published, :buckets).pluck(:key)
|
307
|
-
end
|
308
|
-
end
|
309
|
-
|
310
|
-
ArticleSearch.aggregate(popular_tags_since: 1.month.ago).aggregations.popular_tags_since
|
311
|
-
=> ["Blog", "Tech", …]
|
312
|
-
````
|
313
|
-
|
314
|
-
You can also use transformations to combine multiple aggregations:
|
315
|
-
|
316
|
-
````ruby
|
317
|
-
class ArticleSearch < Caoutsearch::Search::Base
|
318
|
-
has_aggregation :blog_count, { filter: { term: { category: "blog" } } }
|
319
|
-
has_aggregation :archives_count, { filter: { term: { archived: true } } }
|
320
|
-
|
321
|
-
transform_aggregation :stats, from: %i[blog_count archives_count] do |aggs|
|
322
|
-
{
|
323
|
-
blog_count: aggs.dig(:blog_count, :doc_count),
|
324
|
-
archives_count: aggs.dig(:archives, :doc_count)
|
325
|
-
}
|
326
|
-
end
|
327
|
-
end
|
328
|
-
|
329
|
-
ArticleSearch.aggregate(:stats).aggregations.stats
|
330
|
-
# ArticleSearch Search { "body": { "aggs": { "blog_count": {…}, "archives_count": {…}}}}
|
331
|
-
# ArticleSearch Search (10ms / took 5ms)
|
332
|
-
=> { blog_count: 124, archives_count: 2452 }
|
333
|
-
````
|
334
|
-
|
335
|
-
This is also usefull to unify the API between different search engines:
|
336
|
-
|
337
|
-
````ruby
|
338
|
-
class ArticleSearch < Caoutsearch::Search::Base
|
339
|
-
has_aggregation :popular_tags, {
|
340
|
-
filter: { term: { published: true } },
|
341
|
-
aggs: { published: { terms: { field: :tags, size: 10 } } }
|
342
|
-
}
|
343
|
-
|
344
|
-
transform_aggregation :popular_tags do |aggs|
|
345
|
-
aggs.dig(:popular_tags, :published, :buckets).pluck(:key)
|
346
|
-
end
|
347
|
-
end
|
348
|
-
|
349
|
-
class TagSearch < Caoutsearch::Search::Base
|
350
|
-
has_aggregation :popular_tags, {
|
351
|
-
terms: { field: "label", size: 20, order: { used_count: "desc" } }
|
352
|
-
}
|
353
|
-
|
354
|
-
transform_aggregation :popular_tags do |aggs|
|
355
|
-
aggs.dig(:popular_tags, :buckets).pluck(:key)
|
356
|
-
end
|
357
|
-
end
|
358
|
-
|
359
|
-
ArticleSearch.aggregate(:popular_tags).aggregations.popular_tags
|
360
|
-
=> ["Blog", "Tech", …]
|
361
|
-
|
362
|
-
TagSearch.aggregate(:popular_tags).aggregations.popular_tags
|
363
|
-
=> ["Tech", "Blog", …]
|
364
|
-
````
|
365
|
-
|
366
|
-
Transformations are performed on demand and result is memorized. That means:
|
367
|
-
- the result of transformation is not visible in the [Response::Aggregations](#responses) output.
|
368
|
-
- the block is called only once for the same search instance.
|
369
|
-
|
370
|
-
````ruby
|
371
|
-
class ArticleSearch < Caoutsearch::Search::Base
|
372
|
-
has_aggregation :popular_tags, {…}
|
373
|
-
|
374
|
-
transform_aggregation :popular_tags do |aggs|
|
375
|
-
tags = aggs.dig(:popular_tags, :published, :buckets).pluck(:key)
|
376
|
-
authorized = Tag.where(title: tags, authorize: true).pluck(:title)
|
377
|
-
tags & authorized
|
378
|
-
end
|
379
|
-
end
|
380
|
-
|
381
|
-
article_search = ArticleSearch.aggregate(:popular_tags)
|
382
|
-
=> #<ArticleSearch current_aggregations: [:popular_tags]>
|
383
|
-
|
384
|
-
article_search.aggregations
|
385
|
-
# ArticleSearch Search (10ms / took 5ms)
|
386
|
-
=> #<Caoutsearch::Response::Aggregations popular_tags=#<Caoutsearch::Response::Response doc_count=100 …
|
387
|
-
|
388
|
-
article_search.aggregations.popular_tags
|
389
|
-
# (10.2ms) SELECT "tags"."title" FROM "tags" WHERE "tags"."title" IN …
|
390
|
-
=> ["Blog", "Tech", …]
|
391
|
-
|
392
|
-
article_search.aggregations.popular_tags
|
393
|
-
=> ["Blog", "Tech", …]
|
394
|
-
|
395
|
-
article_search.search("Tech").aggregations.popular_tags
|
396
|
-
# ArticleSearch Search (10ms / took 5ms)
|
397
|
-
# (10.2ms) SELECT "tags"."title" FROM "tags" WHERE "tags"."title" IN …
|
398
|
-
=> ["Blog", "Tech", …]
|
399
|
-
````
|
400
|
-
|
401
|
-
Be careful to avoid using `aggregations.<aggregation_name>` inside a transformation block: it can lead to an infinite recursion.
|
402
|
-
|
403
|
-
````ruby
|
404
|
-
class ArticleSearch < Caoutsearch::Search::Base
|
405
|
-
transform_aggregation :popular_tags do
|
406
|
-
aggregations.popular_tags.buckets.pluck("key")
|
407
|
-
end
|
408
|
-
end
|
409
|
-
|
410
|
-
ArticleSearch.aggregate(:popular_tags).aggregations.popular_tags
|
411
|
-
Traceback (most recent call last):
|
412
|
-
4: from app/searches/article_search.rb:3:in `block in <class:ArticleSearch>'
|
413
|
-
3: from app/searches/article_search.rb:3:in `block in <class:ArticleSearch>'
|
414
|
-
2: from app/searches/article_search.rb:3:in `block in <class:ArticleSearch>'
|
415
|
-
1: from app/searches/article_search.rb:3:in `block in <class:ArticleSearch>'
|
416
|
-
SystemStackError (stack level too deep)
|
417
|
-
````
|
418
|
-
|
419
|
-
Instead, use the argument passed to the block: it's is a shortcut for `response.aggregations` which is a [Response::Reponse](#responses) and not a [Response::Aggregations](#responses).
|
420
|
-
|
421
|
-
````ruby
|
422
|
-
class ArticleSearch < Caoutsearch::Search::Base
|
423
|
-
transform_aggregation :popular_tags do |aggs|
|
424
|
-
aggs.popular_tags.buckets.pluck("key")
|
425
|
-
end
|
426
|
-
end
|
427
|
-
|
428
|
-
ArticleSearch.aggregate(:popular_tags).aggregations.popular_tags
|
429
|
-
=> ["Blog", "Tech", …]
|
430
|
-
````
|
431
|
-
|
432
|
-
One last helpful argument is `track_total_hits` which allows to perform calculations over aggregations using the `total_count` method without sending a second request.
|
433
|
-
Take a look at [Total count](#total-count) to understand why a second request could be performed.
|
434
|
-
|
435
|
-
````ruby
|
436
|
-
class ArticleSearch < Caoutsearch::Search::Base
|
437
|
-
aggregation :tagged, filter: { exist: "tag" }
|
438
|
-
|
439
|
-
transform_aggregation :tagged_rate, from: :tagged, track_total_hits: true do |aggs|
|
440
|
-
count = aggs.dig(:tagged, :doc_count)
|
441
|
-
count.to_f / total_count
|
442
|
-
end
|
443
|
-
|
444
|
-
transform_aggregation :tagged_rate_without_track_total_hits, from: :tagged do |aggs|
|
445
|
-
count = aggs.dig(:tagged, :doc_count)
|
446
|
-
count.to_f / total_count
|
447
|
-
end
|
448
|
-
end
|
449
|
-
|
450
|
-
ArticleSearch.aggregate(:tagged_rate).aggregations.tagged_rate
|
451
|
-
# ArticleSearch Search { "body": { "track_total_hits": true, "aggs": { "blog_count": {…}, "archives_count": {…}}}}
|
452
|
-
# ArticleSearch Search (10ms / took 5ms)
|
453
|
-
=> 0.95
|
454
|
-
|
455
|
-
ArticleSearch.aggregate(:tagged_rate_without_track_total_hits).aggregations.tagged_rate
|
456
|
-
# ArticleSearch Search { "body": { "aggs": { "blog_count": {…}, "archives_count": {…}}}}
|
457
|
-
# ArticleSearch Search (10ms / took 5ms)
|
458
|
-
# ArticleSearch Search { "body": { "track_total_hits": true, "aggs": { "blog_count": {…}, "archives_count":
|
459
|
-
# ArticleSearch Search (10ms / took 5ms)
|
460
|
-
=> 0.95
|
461
|
-
````
|
462
|
-
|
463
|
-
#### Responses
|
464
|
-
|
465
|
-
After the request has been sent by calling a method such as `load`, `response` or `hits`, the results is wrapped in a `Response::Response` class which provides method access to its properties via [Hashie::Mash](http://github.com/intridea/hashie).
|
466
|
-
|
467
|
-
Aggregations and suggestions are wrapped in their own respective subclass of `Response::Response`
|
468
|
-
|
469
|
-
````ruby
|
470
|
-
results.response
|
471
|
-
=> #<Caoutsearch::Response::Response _shards=#<Caoutsearch::Response::Response failed=0 skipped=0 successful=5 total=5> hits=…
|
472
|
-
|
473
|
-
search.hits
|
474
|
-
=> #<Hashie::Array [#<Caoutsearch::Response::Response _id="2"…
|
475
|
-
|
476
|
-
search.aggregations
|
477
|
-
=> #<Caoutsearch::Response::Aggregations view_count=#<Caoutsearch::Response::Response…
|
478
|
-
|
479
|
-
search.suggestions
|
480
|
-
=> #<Caoutsearch::Response::Suggestions tags=#<Caoutsearch::Response::Response…
|
481
|
-
````
|
482
|
-
|
483
|
-
##### Loading records
|
484
|
-
|
485
|
-
When calling `records`, the search engine will try to load records from a model using the same class name without `Search` the suffix:
|
486
|
-
* `ArticleSearch` > `Article`
|
487
|
-
* `Blog::ArticleSearch` > `Blog::Article`
|
488
|
-
|
489
|
-
````ruby
|
490
|
-
ArticleSearch.new.records.first
|
491
|
-
# ArticleSearch Search (10ms / took 5ms)
|
492
|
-
# Article Load (9.6ms) SELECT "articles".* FROM "articles" WHERE "articles"."id" IN (1, …
|
493
|
-
=> #<Article id: 1, …>
|
494
|
-
````
|
495
|
-
|
496
|
-
However, you can define an alternative model to load records. This might be helpful when using [single table inheritance](https://api.rubyonrails.org/classes/ActiveRecord/Inheritance.html).
|
497
|
-
|
498
|
-
````ruby
|
499
|
-
ArticleSearch.new.records(use: BlogArticle).first
|
500
|
-
# ArticleSearch Search (10ms / took 5ms)
|
501
|
-
# BlogArticle Load (9.6ms) SELECT "articles".* FROM "articles" WHERE "articles"."id" IN (1, …
|
502
|
-
=> #<BlogArticle id: 1, …>
|
503
|
-
````
|
504
|
-
|
505
|
-
You can also define an alternative model at class level:
|
506
|
-
|
507
|
-
````ruby
|
508
|
-
class BlogArticleSearch < Caoutsearch::Search::Base
|
509
|
-
self.model_name = "Article"
|
510
68
|
|
511
|
-
|
512
|
-
query.filters << { term: { category: "blog" } }
|
513
|
-
end
|
514
|
-
end
|
515
|
-
|
516
|
-
BlogArticleSearch.new.records.first
|
517
|
-
# BlogArticleSearch Search (10ms / took 5ms)
|
518
|
-
# Article Load (9.6ms) SELECT "articles".* FROM "articles" WHERE "articles"."id" IN (1, …
|
519
|
-
=> #<Article id: 1, …>
|
520
|
-
````
|
521
|
-
|
522
|
-
### Model integration
|
523
|
-
|
524
|
-
#### Add Caoutsearch to your models
|
525
|
-
|
526
|
-
The simplest solution is to add `Caoutsearch::Model` to your model and the link the appropriate `Index` and/or `Search` engines:
|
527
|
-
|
528
|
-
```ruby
|
529
|
-
class Article < ActiveRecord::Base
|
530
|
-
include Caoutsearch::Model
|
531
|
-
|
532
|
-
index_with ArticleIndex
|
533
|
-
search_with ArticleSearch
|
534
|
-
end
|
535
|
-
```
|
536
|
-
|
537
|
-
If you don't need your models to be `Indexable` and `Searchable`, you can include only one of the following two modules:
|
538
|
-
|
539
|
-
````ruby
|
540
|
-
class Article < ActiveRecord::Base
|
541
|
-
include Caoutsearch::Model::Indexable
|
542
|
-
|
543
|
-
index_with ArticleIndex
|
544
|
-
end
|
545
|
-
````
|
546
|
-
or
|
547
|
-
````ruby
|
548
|
-
class Article < ActiveRecord::Base
|
549
|
-
include Caoutsearch::Model::Searchable
|
550
|
-
|
551
|
-
search_with ArticleSearch
|
552
|
-
end
|
553
|
-
````
|
554
|
-
|
555
|
-
The modules can be safely included in the meta model `ApplicationRecord`.
|
556
|
-
Indexing & searching features are not available until you call `index_with` or `search_with`:
|
557
|
-
|
558
|
-
````ruby
|
559
|
-
class ApplicationRecord < ActiveRecord::Base
|
560
|
-
include Caoutsearch::Model
|
561
|
-
end
|
562
|
-
````
|
563
|
-
|
564
|
-
#### Index records
|
565
|
-
|
566
|
-
##### Index multiple records
|
567
|
-
|
568
|
-
Import all your records or a restricted scope of records to Elastcisearch.
|
569
|
-
|
570
|
-
````ruby
|
571
|
-
Article.reindex
|
572
|
-
Article.where(published: true).reindex
|
573
|
-
````
|
574
|
-
|
575
|
-
You can update one or more properties. (see [Indexation Engines](#indexation-engines) to read more about properties):
|
576
|
-
|
577
|
-
````ruby
|
578
|
-
Article.reindex(:category)
|
579
|
-
Article.reindex(%i[category published_on])
|
580
|
-
````
|
581
|
-
|
582
|
-
When `reindex` is called without properties, it'll import the full document to ES.
|
583
|
-
On the contrary, when properties are passed, it'll only update existing documents.
|
584
|
-
You can control this behavior with the `method` argument.
|
585
|
-
|
586
|
-
````ruby
|
587
|
-
Article.where(id: 123).reindex(:category)
|
588
|
-
# ArticleIndex Reindex {"index":"articles","body":[{"update":{"_id":123}},{"doc":{"category":"blog"}}]}
|
589
|
-
# [Error] {"update"=>{"_index"=>"articles", "_id"=>"123", "status"=>404, "error"=>{"type"=>"document_missing_exception", …}}
|
590
|
-
|
591
|
-
Article.where(id: 123).reindex(:category, method: :index)
|
592
|
-
# ArticleIndex Reindex {"index":"articles","body":[{"index":{"_id":123}},{"category":"blog"}]}
|
593
|
-
|
594
|
-
Article.where(id: 123).reindex(method: :update)
|
595
|
-
# ArticleIndex Reindex {"index":"articles","body":[{"update":{"_id":123}},{"doc":{…}}]}
|
596
|
-
````
|
597
|
-
|
598
|
-
##### Index single records
|
599
|
-
|
600
|
-
Import a single record.
|
601
|
-
|
602
|
-
````ruby
|
603
|
-
Article.find(123).update_index
|
604
|
-
````
|
605
|
-
|
606
|
-
You can update one or more properties. (see [Indexation Engines](#indexation-engines) to read more about properties):
|
607
|
-
|
608
|
-
````ruby
|
609
|
-
Article.find(123).update_index(:category)
|
610
|
-
Article.find(123).update_index(%i[category published_on])
|
611
|
-
````
|
612
|
-
|
613
|
-
You can verify if and how documents are indexed.
|
614
|
-
If the document is missing in ES, it'll raise a `Elastic::Transport::Transport::Errors::NotFound`.
|
615
|
-
|
616
|
-
````ruby
|
617
|
-
Article.find(123).indexed_document
|
618
|
-
# Traceback (most recent call last):
|
619
|
-
# 1: from (irb):1
|
620
|
-
# Elastic::Transport::Transport::Errors::NotFound ([404] {"_index":"articles","_id":"123","found":false})
|
621
|
-
|
622
|
-
Article.find(123).update_index
|
623
|
-
Article.find(123).indexed_document
|
624
|
-
=> {"_index"=>"articles", "_id"=>"123", "_version"=>1"found"=>true, "_source"=>{…}}
|
625
|
-
````
|
626
|
-
|
627
|
-
##### Delete documents
|
628
|
-
|
629
|
-
You can delete one or more documents.
|
630
|
-
**Note**: it won't delete records from database, only from the ES indice.
|
631
|
-
|
632
|
-
````ruby
|
633
|
-
Article.delete_indexes
|
634
|
-
Article.where(id: 123).delete_indexed_documents
|
635
|
-
Article.find(123).delete_index
|
636
|
-
````
|
637
|
-
|
638
|
-
If a record is already deleted from the database, you can still delete its document.
|
639
|
-
|
640
|
-
````ruby
|
641
|
-
Article.delete_index(123)
|
642
|
-
````
|
643
|
-
|
644
|
-
##### Automatic Callbacks
|
645
|
-
|
646
|
-
Callbacks are not provided by Caoutsearch but they are very easy to add:
|
647
|
-
|
648
|
-
````ruby
|
649
|
-
class Article < ApplicationRecord
|
650
|
-
index_with ArticleIndex
|
651
|
-
|
652
|
-
after_commit :update_index, on: %i[create update]
|
653
|
-
after_commit :delete_index, on: %i[destroy]
|
654
|
-
end
|
655
|
-
````
|
656
|
-
|
657
|
-
##### Asynchronous methods
|
658
|
-
|
659
|
-
TODO
|
660
|
-
|
661
|
-
#### Search for records
|
662
|
-
|
663
|
-
##### Search API
|
664
|
-
Searching is pretty simple.
|
665
|
-
|
666
|
-
````ruby
|
667
|
-
Article.search("Quick brown fox")
|
668
|
-
=> #<ArticleSearch current_criteria: ["Quick brown fox"]>
|
669
|
-
````
|
670
|
-
|
671
|
-
You can chain criteria and many other parameters:
|
672
|
-
````ruby
|
673
|
-
Article.search("Quick brown fox").search(published: true)
|
674
|
-
=> #<ArticleSearch current_criteria: ["Quick brown fox", {"published"=>true}]>
|
675
|
-
|
676
|
-
Article.search("Quick brown fox").order(:publication_date)
|
677
|
-
=> #<ArticleSearch current_criteria: ["Quick brown fox"], current_order: :publication_date>
|
678
|
-
|
679
|
-
Article.search("Quick brown fox").limit(100).offset(100)
|
680
|
-
=> #<ArticleSearch current_criteria: ["Quick brown fox"], current_limit: 100, current_offset: 100>
|
681
|
-
|
682
|
-
Article.search("Quick brown fox").page(1).per(100)
|
683
|
-
=> #<ArticleSearch current_criteria: ["Quick brown fox"], current_page: 1, current_limit: 100>
|
684
|
-
|
685
|
-
Article.search("Quick brown fox").aggregate(:tags).aggregate(:dates)
|
686
|
-
=> #<ArticleSearch current_criteria: ["Quick brown fox"], current_aggregations: [:tags, :dates]>>
|
687
|
-
````
|
688
|
-
|
689
|
-
##### Pagination
|
690
|
-
|
691
|
-
Search results can be paginated.
|
692
|
-
````ruby
|
693
|
-
search = Article.search("Quick brown fox").page(1).per(100)
|
694
|
-
search.current_page
|
695
|
-
=> 1
|
696
|
-
|
697
|
-
search.total_pages
|
698
|
-
=> 2546
|
699
|
-
|
700
|
-
> search.total_count
|
701
|
-
=> 254514
|
702
|
-
````
|
703
|
-
|
704
|
-
##### Total count
|
705
|
-
|
706
|
-
By default [ES doesn't return the total number of hits](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-your-data.html#track-total-hits). So, when calling `total_count` or `total_pages` a second request might be sent to ES.
|
707
|
-
To avoid a second roundtrip, use `track_total_hits`:
|
708
|
-
|
709
|
-
````ruby
|
710
|
-
search = Article.search("Quick brown fox")
|
711
|
-
search.hits
|
712
|
-
# ArticleSearch Search {…}
|
713
|
-
# ArticleSearch Search (81.8ms / took 16ms)
|
714
|
-
=> […]
|
715
|
-
|
716
|
-
search.total_count
|
717
|
-
# ArticleSearch Search {…, track_total_hits: true }
|
718
|
-
# ArticleSearch Search (135.3ms / took 76ms)
|
719
|
-
=> 276
|
720
|
-
|
721
|
-
search = Article.search("Quick brown fox").track_total_hits
|
722
|
-
search.hits
|
723
|
-
# ArticleSearch Search {…, track_total_hits: true }
|
724
|
-
# ArticleSearch Search (120.2ms / took 56ms)
|
725
|
-
=> […]
|
726
|
-
|
727
|
-
search.total_count
|
728
|
-
=> 276
|
729
|
-
````
|
730
|
-
|
731
|
-
##### Iterating results
|
732
|
-
|
733
|
-
Several methods are provided to loop through a collection or hits or records.
|
734
|
-
These methods are processing batches in the most efficient way: [PIT search_after](https://www.elastic.co/guide/en/elasticsearch/reference/current/paginate-search-results.html#search-after).
|
735
|
-
|
736
|
-
* `find_each_hit` to yield each hit returned by Elasticsearch.
|
737
|
-
* `find_each_record` to yield each record from your database.
|
738
|
-
* `find_hits_in_batches` to yield each batch of hits as returned by Elasticsearch.
|
739
|
-
* `find_records_in_batches` to yield each batch of records from the database.
|
740
|
-
|
741
|
-
Example:
|
742
|
-
|
743
|
-
```ruby
|
744
|
-
Article.search(published: true).find_each_record do |record|
|
745
|
-
record.inspect
|
746
|
-
end
|
747
|
-
```
|
748
|
-
|
749
|
-
The `keep_alive` parameter tells Elasticsearch how long it should keep the point in time alive. Defaults to 1 minute.
|
750
|
-
|
751
|
-
```ruby
|
752
|
-
Article.search(published: true).find_each_record(keep_alive: "2h")
|
753
|
-
```
|
754
|
-
|
755
|
-
To specifies the size of the batch, use `per` chainable method or `batch_size` parameter. Defaults to 1000.
|
756
|
-
|
757
|
-
```ruby
|
758
|
-
Article.search(published: true).find_records_in_batches(batch_size: 500)
|
759
|
-
Article.search(published: true).per(500).find_records_in_batches
|
760
|
-
```
|
761
|
-
|
762
|
-
## Testing with Caoutsearch
|
763
|
-
|
764
|
-
Caoutsearch offers few methods to stub Elasticsearch requests.
|
765
|
-
You first need to add [webmock](https://github.com/bblimke/webmock) to your Gemfile.
|
766
|
-
|
767
|
-
```bash
|
768
|
-
bundle add webmock
|
769
|
-
```
|
770
|
-
|
771
|
-
Then, add `Caoutsearch::Testing::MockRequests` to your test suite.
|
772
|
-
The examples below uses RSpec, but it should be compatible with other test framework.
|
773
|
-
|
774
|
-
```ruby
|
775
|
-
# spec/spec_helper.rb
|
776
|
-
|
777
|
-
require "caoutsearch/testing"
|
778
|
-
|
779
|
-
RSpec.configure do |config|
|
780
|
-
config.include Caoutsearch::Testing::MockRequests
|
781
|
-
end
|
782
|
-
```
|
783
|
-
|
784
|
-
You can then call the following methods:
|
785
|
-
|
786
|
-
```ruby
|
787
|
-
RSpec.describe SomeClass do
|
788
|
-
before do
|
789
|
-
stub_elasticsearch_request(:head, "articles").to_return(status: 200)
|
790
|
-
|
791
|
-
stub_elasticsearch_request(:get, "_cat/indices?format=json&h=index").to_return_json, [
|
792
|
-
{ index: "ca_locals_v14" }
|
793
|
-
])
|
794
|
-
|
795
|
-
stub_elasticsearch_reindex_request("articles")
|
796
|
-
stub_elasticsearch_search_request("articles", [
|
797
|
-
{"_id" => "135", "_source" => {"name" => "Hello World"}},
|
798
|
-
{"_id" => "137", "_source" => {"name" => "Hello World"}}
|
799
|
-
])
|
800
|
-
end
|
801
|
-
|
802
|
-
# ... do your tests...
|
803
|
-
end
|
804
|
-
```
|
805
|
-
|
806
|
-
`stub_elasticsearch_search_request` accepts an array or records:
|
807
|
-
|
808
|
-
```ruby
|
809
|
-
RSpec.describe SomeClass do
|
810
|
-
let(:articles) { create_list(:article, 5) }
|
811
|
-
|
812
|
-
before do
|
813
|
-
stub_elasticsearch_search_request("articles", articles)
|
814
|
-
end
|
815
|
-
|
816
|
-
# ... do your tests...
|
817
|
-
end
|
818
|
-
```
|
819
|
-
|
820
|
-
It allows to shim the total number of hits returned.
|
821
|
-
|
822
|
-
```ruby
|
823
|
-
RSpec.describe SomeClass do
|
824
|
-
before do
|
825
|
-
stub_elasticsearch_search_request("articles", [], total: 250)
|
826
|
-
end
|
827
|
-
|
828
|
-
# ... do your tests...
|
829
|
-
end
|
69
|
+
ArticleSearch.search(published_on: [["now-1y", nil]]).aggregate(:popular_tags)
|
830
70
|
```
|
831
71
|
|
832
72
|
## Contributing
|
833
73
|
|
834
|
-
1. Don't hesitate to submit your feature/idea/fix in [issues](https://github.com/
|
835
|
-
2. Fork the [repository](https://github.com/
|
74
|
+
1. Don't hesitate to submit your feature/idea/fix in [issues](https://github.com/inkstak/activejob-status)
|
75
|
+
2. Fork the [repository](https://github.com/inkstak/activejob-status)
|
836
76
|
3. Create your feature branch
|
837
77
|
4. Ensure RSpec & Rubocop are passing
|
838
78
|
4. Create a pull request
|
@@ -845,6 +85,12 @@ bundle exec rubocop
|
|
845
85
|
bundle exec standardrb
|
846
86
|
```
|
847
87
|
|
88
|
+
To run RSpec against various version of Rails dependencies:
|
89
|
+
```bash
|
90
|
+
bundle exec appraisal install
|
91
|
+
bundle exec appraisal rspec
|
92
|
+
```
|
93
|
+
|
848
94
|
All of them can be run with:
|
849
95
|
|
850
96
|
```bash
|