searchkick 0.0.1 → 0.0.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/.gitignore +1 -0
- data/README.md +163 -11
- data/Rakefile +7 -0
- data/lib/searchkick.rb +5 -45
- data/lib/searchkick/model.rb +80 -0
- data/lib/searchkick/reindex.rb +40 -0
- data/lib/searchkick/search.rb +137 -0
- data/lib/searchkick/version.rb +1 -1
- data/searchkick.gemspec +3 -0
- data/test/searchkick_test.rb +283 -0
- data/test/test_helper.rb +26 -0
- metadata +52 -3
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA1:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: f37d2d366a7f03a6e2757c688385aed25678836c
|
4
|
+
data.tar.gz: e4648bd3692a88ccd7d9669376f8fa89af8e7431
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 071e83eab034feb8784f32791728fee4cb61e3670ef9f36f97ad66961e7c12c299214e44bde1c5aec45f56b92a5d639591e2f9845a3fd2a1c8aee9e3fb14c7a9
|
7
|
+
data.tar.gz: e42658adc41be4c3e2232bfa510f9e2026da8c6a1303b192e5f03692d9c0d15068e3d92252955bba136bea21bfe830674ae84251e7134dc280f642d7b7f9bea6
|
data/.gitignore
CHANGED
data/README.md
CHANGED
@@ -1,36 +1,175 @@
|
|
1
1
|
# Searchkick
|
2
2
|
|
3
|
-
Search made easy
|
3
|
+
:rocket: Search made easy
|
4
|
+
|
5
|
+
Searchkick provides sensible search defaults out of the box. It handles:
|
6
|
+
|
7
|
+
- stemming - `tomatoes` matches `tomato`
|
8
|
+
- special characters - `jalapenos` matches `jalapeños`
|
9
|
+
- extra whitespace - `dishwasher` matches `dish washer`
|
10
|
+
- misspellings - `zuchini` matches `zucchini`
|
11
|
+
- custom synonyms - `qtip` matches `cotton swab`
|
12
|
+
|
13
|
+
Runs on Elasticsearch
|
14
|
+
|
15
|
+
:tangerine: Battle-tested at [Instacart](https://www.instacart.com)
|
4
16
|
|
5
17
|
## Usage
|
6
18
|
|
7
|
-
|
19
|
+
```ruby
|
20
|
+
class Product < ActiveRecord::Base
|
21
|
+
searchkick
|
22
|
+
end
|
23
|
+
```
|
8
24
|
|
9
|
-
|
25
|
+
And to query, use:
|
10
26
|
|
11
27
|
```ruby
|
12
|
-
|
28
|
+
Product.search "2% Milk"
|
13
29
|
```
|
14
30
|
|
15
|
-
|
31
|
+
or only search specific fields:
|
16
32
|
|
17
|
-
|
33
|
+
```ruby
|
34
|
+
Product.search "Butter", fields: [:name, :brand]
|
35
|
+
```
|
18
36
|
|
19
|
-
|
37
|
+
### Query Like SQL
|
20
38
|
|
21
39
|
```ruby
|
22
|
-
|
23
|
-
|
40
|
+
Product.search "2% Milk", where: {in_stock: true}, limit: 10, offset: 50
|
41
|
+
```
|
42
|
+
|
43
|
+
#### Where
|
44
|
+
|
45
|
+
```ruby
|
46
|
+
where: {
|
47
|
+
expires_at: {gt: Time.now}, # lt, gte, lte also available
|
48
|
+
orders_count: 1..10, # equivalent to {gte: 1, lte: 10}
|
49
|
+
aisle_id: [25, 30], # in
|
50
|
+
store_id: {not: 2}, # not
|
51
|
+
aisle_id: {not: [25, 30]}, # not in
|
52
|
+
or: [
|
53
|
+
[{in_stock: true}, {backordered: true}]
|
54
|
+
]
|
55
|
+
}
|
56
|
+
```
|
57
|
+
|
58
|
+
#### Order
|
59
|
+
|
60
|
+
```ruby
|
61
|
+
order: {_score: :desc} # most relevant first - default
|
62
|
+
```
|
63
|
+
|
64
|
+
#### Explain
|
65
|
+
|
66
|
+
```ruby
|
67
|
+
explain: true
|
68
|
+
```
|
69
|
+
|
70
|
+
### Facets
|
71
|
+
|
72
|
+
```ruby
|
73
|
+
Product.search "2% Milk", facets: [:store_id, :aisle_id]
|
74
|
+
```
|
75
|
+
|
76
|
+
Advanced
|
77
|
+
|
78
|
+
```ruby
|
79
|
+
Product.search "2% Milk", facets: {store_id: {where: {in_stock: true}}}
|
80
|
+
```
|
81
|
+
|
82
|
+
### Synonyms
|
83
|
+
|
84
|
+
```ruby
|
85
|
+
class Product < ActiveRecord::Base
|
86
|
+
searchkick synonyms: [["scallion", "green onion"], ["qtip", "cotton swab"]]
|
87
|
+
end
|
88
|
+
```
|
89
|
+
|
90
|
+
You must call `Product.reindex` after changing synonyms.
|
91
|
+
|
92
|
+
### Make Searches Better Over Time
|
93
|
+
|
94
|
+
Improve results with analytics on conversions and give popular documents a little boost.
|
95
|
+
|
96
|
+
First, you must keep track of search conversions. The database works well for low volume, but feel free to use redis or another datastore.
|
97
|
+
|
98
|
+
```ruby
|
99
|
+
class Search < ActiveRecord::Base
|
100
|
+
belongs_to :product
|
101
|
+
# fields: id, query, searched_at, converted_at, product_id
|
102
|
+
end
|
103
|
+
```
|
104
|
+
|
105
|
+
Add the conversions to the index.
|
106
|
+
|
107
|
+
```ruby
|
108
|
+
class Product < ActiveRecord::Base
|
109
|
+
has_many :searches
|
110
|
+
|
111
|
+
searchkick conversions: true
|
112
|
+
|
113
|
+
def to_indexed_json
|
114
|
+
{
|
115
|
+
name: name,
|
116
|
+
conversions: searches.group("query").count.map{|query, count| {query: query, count: count} }, # TODO fix
|
117
|
+
_boost: Math.log(orders_count) # boost more popular products a bit
|
118
|
+
}
|
119
|
+
end
|
120
|
+
end
|
121
|
+
```
|
122
|
+
|
123
|
+
After the reindex is complete (to prevent errors), tell the search method to use conversions.
|
124
|
+
|
125
|
+
```ruby
|
126
|
+
Product.search "Fat Free Milk", conversions: true
|
127
|
+
```
|
128
|
+
|
129
|
+
### Zero Downtime Changes
|
130
|
+
|
131
|
+
```ruby
|
132
|
+
Product.reindex
|
133
|
+
```
|
134
|
+
|
135
|
+
Behind the scenes, this creates a new index `products_20130714181054` and points the `products` alias to the new index when complete - an atomic operation :)
|
136
|
+
|
137
|
+
Searchkick uses `find_in_batches` to import documents. To filter documents or eagar load associations, use the `searchkick_import` scope.
|
138
|
+
|
139
|
+
```ruby
|
140
|
+
class Product < ActiveRecord::Base
|
141
|
+
scope :searchkick_import, where(active: true).includes(:searches)
|
24
142
|
end
|
25
143
|
```
|
26
144
|
|
27
145
|
There is also a rake task.
|
28
146
|
|
29
147
|
```sh
|
30
|
-
rake searchkick:reindex CLASS=
|
148
|
+
rake searchkick:reindex CLASS=Product
|
149
|
+
```
|
150
|
+
|
151
|
+
Thanks to Jaroslav Kalistsuk for the [original implementation](https://gist.github.com/jarosan/3124884).
|
152
|
+
|
153
|
+
### Reference
|
154
|
+
|
155
|
+
Reindex one item
|
156
|
+
|
157
|
+
```ruby
|
158
|
+
product = Product.find(1)
|
159
|
+
product.update_index
|
31
160
|
```
|
32
161
|
|
33
|
-
|
162
|
+
Partial matches (needs better name)
|
163
|
+
|
164
|
+
```ruby
|
165
|
+
Item.search "fresh honey", partial: true # matches organic honey
|
166
|
+
```
|
167
|
+
|
168
|
+
## Elasticsearch Gotchas
|
169
|
+
|
170
|
+
### Inconsistent Scores
|
171
|
+
|
172
|
+
Due to the distributed nature of Elasticsearch, you can get incorrect results when the number of documents in the index is low. You can [read more about it here](http://www.elasticsearch.org/blog/understanding-query-then-fetch-vs-dfs-query-then-fetch/). To fix this, set the search type to `dfs_query_and_fetch`. Alternatively, you can just use one shard with `settings: {number_of_shards: 1}`.
|
34
173
|
|
35
174
|
## Installation
|
36
175
|
|
@@ -46,6 +185,19 @@ And then execute:
|
|
46
185
|
bundle
|
47
186
|
```
|
48
187
|
|
188
|
+
## TODO
|
189
|
+
|
190
|
+
- Autocomplete
|
191
|
+
- Option to turn off fuzzy matching (should this be default?)
|
192
|
+
- Exact phrase matches (in order)
|
193
|
+
- Focus on results format (load: true?)
|
194
|
+
- Test helpers - everyone should test their own search
|
195
|
+
- Built-in synonyms from WordNet
|
196
|
+
- Dashboard w/ real-time analytics?
|
197
|
+
- [Suggest API](http://www.elasticsearch.org/guide/reference/api/search/suggest/) "Did you mean?"
|
198
|
+
- Allow for "exact search" with quotes
|
199
|
+
- Make updates to old and new index while reindexing [possibly with an another alias](http://www.kickstarter.com/backing-and-hacking)
|
200
|
+
|
49
201
|
## Contributing
|
50
202
|
|
51
203
|
1. Fork it
|
data/Rakefile
CHANGED
data/lib/searchkick.rb
CHANGED
@@ -1,49 +1,9 @@
|
|
1
1
|
require "searchkick/version"
|
2
|
+
require "searchkick/reindex"
|
3
|
+
require "searchkick/search"
|
4
|
+
require "searchkick/model"
|
2
5
|
require "searchkick/tasks"
|
3
6
|
require "tire"
|
7
|
+
require "active_record" # TODO only require active_model
|
4
8
|
|
5
|
-
|
6
|
-
module ClassMethods
|
7
|
-
|
8
|
-
# https://gist.github.com/jarosan/3124884
|
9
|
-
def reindex
|
10
|
-
alias_name = klass.tire.index.name
|
11
|
-
new_index = alias_name + "_" + Time.now.strftime("%Y%m%d%H%M%S")
|
12
|
-
|
13
|
-
# Rake::Task["tire:import"].invoke
|
14
|
-
index = Tire::Index.new(new_index)
|
15
|
-
Tire::Tasks::Import.create_index(index, klass)
|
16
|
-
scope = klass.respond_to?(:tire_import) ? klass.tire_import : klass
|
17
|
-
scope.find_in_batches do |batch|
|
18
|
-
index.import batch
|
19
|
-
end
|
20
|
-
|
21
|
-
if a = Tire::Alias.find(alias_name)
|
22
|
-
puts "[IMPORT] Alias found: #{Tire::Alias.find(alias_name).indices.to_ary.join(",")}"
|
23
|
-
old_indices = Tire::Alias.find(alias_name).indices
|
24
|
-
old_indices.each do |index|
|
25
|
-
a.indices.delete index
|
26
|
-
end
|
27
|
-
|
28
|
-
a.indices.add new_index
|
29
|
-
a.save
|
30
|
-
|
31
|
-
old_indices.each do |index|
|
32
|
-
puts "[IMPORT] Deleting index: #{index}"
|
33
|
-
i = Tire::Index.new(index)
|
34
|
-
i.delete if i.exists?
|
35
|
-
end
|
36
|
-
else
|
37
|
-
puts "[IMPORT] No alias found. Deleting index, creating new one, and setting up alias"
|
38
|
-
i = Tire::Index.new(alias_name)
|
39
|
-
i.delete if i.exists?
|
40
|
-
Tire::Alias.create(name: alias_name, indices: [new_index])
|
41
|
-
end
|
42
|
-
|
43
|
-
puts "[IMPORT] Saved alias #{alias_name} pointing to #{new_index}"
|
44
|
-
end
|
45
|
-
|
46
|
-
end
|
47
|
-
end
|
48
|
-
|
49
|
-
Tire::Model::Search::ClassMethodsProxy.send :include, Searchkick::ClassMethods
|
9
|
+
ActiveRecord::Base.send(:extend, Searchkick::Model)
|
@@ -0,0 +1,80 @@
|
|
1
|
+
module Searchkick
|
2
|
+
module Model
|
3
|
+
|
4
|
+
def searchkick(options = {})
|
5
|
+
custom_settings = {
|
6
|
+
analysis: {
|
7
|
+
analyzer: {
|
8
|
+
searchkick_keyword: {
|
9
|
+
type: "custom",
|
10
|
+
tokenizer: "keyword",
|
11
|
+
filter: ["lowercase", "snowball"]
|
12
|
+
},
|
13
|
+
default_index: {
|
14
|
+
type: "custom",
|
15
|
+
tokenizer: "standard",
|
16
|
+
# synonym should come last, after stemming and shingle
|
17
|
+
# shingle must come before snowball
|
18
|
+
filter: ["standard", "lowercase", "asciifolding", "stop", "snowball", "searchkick_index_shingle"]
|
19
|
+
},
|
20
|
+
searchkick_search: {
|
21
|
+
type: "custom",
|
22
|
+
tokenizer: "standard",
|
23
|
+
filter: ["standard", "lowercase", "asciifolding", "stop", "snowball", "searchkick_search_shingle"]
|
24
|
+
},
|
25
|
+
searchkick_search2: {
|
26
|
+
type: "custom",
|
27
|
+
tokenizer: "standard",
|
28
|
+
filter: ["standard", "lowercase", "asciifolding", "stop", "snowball"] #, "searchkick_search_shingle"]
|
29
|
+
}
|
30
|
+
},
|
31
|
+
filter: {
|
32
|
+
searchkick_index_shingle: {
|
33
|
+
type: "shingle",
|
34
|
+
token_separator: ""
|
35
|
+
},
|
36
|
+
# lucky find http://web.archiveorange.com/archive/v/AAfXfQ17f57FcRINsof7
|
37
|
+
searchkick_search_shingle: {
|
38
|
+
type: "shingle",
|
39
|
+
token_separator: "",
|
40
|
+
output_unigrams: false,
|
41
|
+
output_unigrams_if_no_shingles: true
|
42
|
+
}
|
43
|
+
}
|
44
|
+
}
|
45
|
+
}.merge(options[:settings] || {})
|
46
|
+
synonyms = options[:synonyms] || []
|
47
|
+
if synonyms.any?
|
48
|
+
custom_settings[:analysis][:filter][:searchkick_synonym] = {
|
49
|
+
type: "synonym",
|
50
|
+
ignore_case: true,
|
51
|
+
synonyms: synonyms.map{|s| s.join(" => ") } # TODO support more than 2 synonyms on a line
|
52
|
+
}
|
53
|
+
custom_settings[:analysis][:analyzer][:default_index][:filter] << "searchkick_synonym"
|
54
|
+
custom_settings[:analysis][:analyzer][:searchkick_search][:filter].insert(-2, "searchkick_synonym")
|
55
|
+
custom_settings[:analysis][:analyzer][:searchkick_search][:filter] << "searchkick_synonym"
|
56
|
+
custom_settings[:analysis][:analyzer][:searchkick_search2][:filter] << "searchkick_synonym"
|
57
|
+
end
|
58
|
+
|
59
|
+
class_eval do
|
60
|
+
extend Searchkick::Search
|
61
|
+
extend Searchkick::Reindex
|
62
|
+
include Tire::Model::Search
|
63
|
+
include Tire::Model::Callbacks
|
64
|
+
|
65
|
+
tire do
|
66
|
+
settings custom_settings
|
67
|
+
mapping do
|
68
|
+
# indexes field, analyzer: "searchkick"
|
69
|
+
if options[:conversions]
|
70
|
+
indexes :conversions, type: "nested" do
|
71
|
+
indexes :query, analyzer: "searchkick_keyword"
|
72
|
+
indexes :count, type: "integer"
|
73
|
+
end
|
74
|
+
end
|
75
|
+
end
|
76
|
+
end
|
77
|
+
end
|
78
|
+
end
|
79
|
+
end
|
80
|
+
end
|
@@ -0,0 +1,40 @@
|
|
1
|
+
module Searchkick
|
2
|
+
module Reindex
|
3
|
+
|
4
|
+
# https://gist.github.com/jarosan/3124884
|
5
|
+
def reindex
|
6
|
+
alias_name = tire.index.name
|
7
|
+
new_index = alias_name + "_" + Time.now.strftime("%Y%m%d%H%M%S")
|
8
|
+
|
9
|
+
# Rake::Task["tire:import"].invoke
|
10
|
+
index = Tire::Index.new(new_index)
|
11
|
+
Tire::Tasks::Import.create_index(index, self) # TODO remove puts
|
12
|
+
scope = respond_to?(:searchkick_import) ? searchkick_import : self
|
13
|
+
scope.find_in_batches do |batch|
|
14
|
+
index.import batch
|
15
|
+
end
|
16
|
+
|
17
|
+
if a = Tire::Alias.find(alias_name)
|
18
|
+
old_indices = Tire::Alias.find(alias_name).indices
|
19
|
+
old_indices.each do |index|
|
20
|
+
a.indices.delete index
|
21
|
+
end
|
22
|
+
|
23
|
+
a.indices.add new_index
|
24
|
+
a.save
|
25
|
+
|
26
|
+
old_indices.each do |index|
|
27
|
+
i = Tire::Index.new(index)
|
28
|
+
i.delete if i.exists?
|
29
|
+
end
|
30
|
+
else
|
31
|
+
i = Tire::Index.new(alias_name)
|
32
|
+
i.delete if i.exists?
|
33
|
+
Tire::Alias.create(name: alias_name, indices: [new_index])
|
34
|
+
end
|
35
|
+
|
36
|
+
true
|
37
|
+
end
|
38
|
+
|
39
|
+
end
|
40
|
+
end
|
@@ -0,0 +1,137 @@
|
|
1
|
+
module Searchkick
|
2
|
+
# can't check mapping for conversions since the new index may not be built
|
3
|
+
module Search
|
4
|
+
def index_types
|
5
|
+
Hash[ (((Product.index.mapping || {})["product"] || {})["properties"] || {}).map{|k, v| [k, v["type"]] } ].reject{|k, v| k == "conversions" || k[0] == "_" }
|
6
|
+
end
|
7
|
+
|
8
|
+
def search(term, options = {})
|
9
|
+
fields = options[:fields] || ["_all"]
|
10
|
+
operator = options[:partial] ? "or" : "and"
|
11
|
+
tire.search do
|
12
|
+
query do
|
13
|
+
boolean do
|
14
|
+
must do
|
15
|
+
dis_max do
|
16
|
+
query do
|
17
|
+
match fields, term, boost: 10, operator: operator, analyzer: "searchkick_search"
|
18
|
+
end
|
19
|
+
query do
|
20
|
+
match fields, term, boost: 10, operator: operator, analyzer: "searchkick_search2"
|
21
|
+
end
|
22
|
+
query do
|
23
|
+
match fields, term, use_dis_max: false, fuzziness: 0.7, max_expansions: 1, prefix_length: 1, operator: operator, analyzer: "searchkick_search"
|
24
|
+
end
|
25
|
+
query do
|
26
|
+
match fields, term, use_dis_max: false, fuzziness: 0.7, max_expansions: 1, prefix_length: 1, operator: operator, analyzer: "searchkick_search2"
|
27
|
+
end
|
28
|
+
end
|
29
|
+
end
|
30
|
+
if options[:conversions]
|
31
|
+
should do
|
32
|
+
nested path: "conversions", score_mode: "total" do
|
33
|
+
query do
|
34
|
+
custom_score script: "log(doc['count'].value)" do
|
35
|
+
match "query", term
|
36
|
+
end
|
37
|
+
end
|
38
|
+
end
|
39
|
+
end
|
40
|
+
end
|
41
|
+
end
|
42
|
+
end
|
43
|
+
size options[:limit] || 100000 # return all - like sql query
|
44
|
+
from options[:offset] if options[:offset]
|
45
|
+
explain options[:explain] if options[:explain]
|
46
|
+
|
47
|
+
# order
|
48
|
+
if options[:order]
|
49
|
+
sort do
|
50
|
+
options[:order].each do |k, v|
|
51
|
+
by k, v
|
52
|
+
end
|
53
|
+
end
|
54
|
+
end
|
55
|
+
|
56
|
+
# where
|
57
|
+
# TODO expand or
|
58
|
+
|
59
|
+
where_filters =
|
60
|
+
proc do |where|
|
61
|
+
filters = []
|
62
|
+
(where || {}).each do |field, value|
|
63
|
+
if field == :or
|
64
|
+
value.each do |or_clause|
|
65
|
+
filters << {or: or_clause.map{|or_statement| {term: or_statement} }}
|
66
|
+
end
|
67
|
+
else
|
68
|
+
# expand ranges
|
69
|
+
if value.is_a?(Range)
|
70
|
+
value = {gte: value.first, (value.exclude_end? ? :lt : :lte) => value.last}
|
71
|
+
end
|
72
|
+
|
73
|
+
if value.is_a?(Array) # in query
|
74
|
+
filters << {terms: {field => value}}
|
75
|
+
elsif value.is_a?(Hash)
|
76
|
+
value.each do |op, op_value|
|
77
|
+
if op == :not # not equal
|
78
|
+
if op_value.is_a?(Array)
|
79
|
+
filters << {not: {terms: {field => op_value}}}
|
80
|
+
else
|
81
|
+
filters << {not: {term: {field => op_value}}}
|
82
|
+
end
|
83
|
+
else
|
84
|
+
range_query =
|
85
|
+
case op
|
86
|
+
when :gt
|
87
|
+
{from: op_value, include_lower: false}
|
88
|
+
when :gte
|
89
|
+
{from: op_value, include_lower: true}
|
90
|
+
when :lt
|
91
|
+
{to: op_value, include_upper: false}
|
92
|
+
when :lte
|
93
|
+
{to: op_value, include_upper: true}
|
94
|
+
else
|
95
|
+
raise "Unknown where operator"
|
96
|
+
end
|
97
|
+
filters << {range: {field => range_query}}
|
98
|
+
end
|
99
|
+
end
|
100
|
+
else
|
101
|
+
filters << {term: {field => value}}
|
102
|
+
end
|
103
|
+
end
|
104
|
+
end
|
105
|
+
filters
|
106
|
+
end
|
107
|
+
|
108
|
+
where_filters.call(options[:where]).each do |f|
|
109
|
+
type, value = f.first
|
110
|
+
filter type, value
|
111
|
+
end
|
112
|
+
|
113
|
+
# facets
|
114
|
+
if options[:facets]
|
115
|
+
facets = options[:facets] || {}
|
116
|
+
if facets.is_a?(Array) # convert to more advanced syntax
|
117
|
+
facets = Hash[ facets.map{|f| [f, {}] } ]
|
118
|
+
end
|
119
|
+
|
120
|
+
facets.each do |field, facet_options|
|
121
|
+
facet_filters = where_filters.call(facet_options[:where])
|
122
|
+
facet field do
|
123
|
+
terms field
|
124
|
+
if facet_filters.size == 1
|
125
|
+
type, value = facet_filters.first.first
|
126
|
+
facet_filter type, value
|
127
|
+
elsif facet_filters.size > 1
|
128
|
+
facet_filter :and, *facet_filters
|
129
|
+
end
|
130
|
+
end
|
131
|
+
end
|
132
|
+
end
|
133
|
+
|
134
|
+
end
|
135
|
+
end
|
136
|
+
end
|
137
|
+
end
|
data/lib/searchkick/version.rb
CHANGED
data/searchkick.gemspec
CHANGED
@@ -22,4 +22,7 @@ Gem::Specification.new do |spec|
|
|
22
22
|
|
23
23
|
spec.add_development_dependency "bundler", "~> 1.3"
|
24
24
|
spec.add_development_dependency "rake"
|
25
|
+
spec.add_development_dependency "minitest"
|
26
|
+
spec.add_development_dependency "activerecord"
|
27
|
+
spec.add_development_dependency "pg"
|
25
28
|
end
|
@@ -0,0 +1,283 @@
|
|
1
|
+
require "test_helper"
|
2
|
+
|
3
|
+
class Product < ActiveRecord::Base
|
4
|
+
searchkick \
|
5
|
+
synonyms: [
|
6
|
+
["clorox", "bleach"],
|
7
|
+
["scallion", "greenonion"],
|
8
|
+
["saranwrap", "plasticwrap"],
|
9
|
+
["qtip", "cotton swab"],
|
10
|
+
["burger", "hamburger"],
|
11
|
+
["bandaid", "bandag"]
|
12
|
+
],
|
13
|
+
settings: {
|
14
|
+
number_of_shards: 1
|
15
|
+
},
|
16
|
+
conversions: true
|
17
|
+
|
18
|
+
# searchkick do
|
19
|
+
# string :name
|
20
|
+
# boolean :visible
|
21
|
+
# integer :orders_count
|
22
|
+
# end
|
23
|
+
end
|
24
|
+
|
25
|
+
p Product.index_types
|
26
|
+
|
27
|
+
class TestSearchkick < Minitest::Unit::TestCase
|
28
|
+
|
29
|
+
def setup
|
30
|
+
Product.index.delete
|
31
|
+
Product.create_elasticsearch_index
|
32
|
+
end
|
33
|
+
|
34
|
+
def test_reindex
|
35
|
+
assert Product.reindex
|
36
|
+
end
|
37
|
+
|
38
|
+
# exact
|
39
|
+
|
40
|
+
def test_match
|
41
|
+
store_names ["Whole Milk", "Fat Free Milk", "Milk"]
|
42
|
+
assert_search "milk", ["Milk", "Whole Milk", "Fat Free Milk"]
|
43
|
+
end
|
44
|
+
|
45
|
+
def test_case
|
46
|
+
store_names ["Whole Milk", "Fat Free Milk", "Milk"]
|
47
|
+
assert_search "MILK", ["Milk", "Whole Milk", "Fat Free Milk"]
|
48
|
+
end
|
49
|
+
|
50
|
+
def test_cheese_space_in_index
|
51
|
+
store_names ["Pepper Jack Cheese Skewers"]
|
52
|
+
assert_search "pepperjack cheese skewers", ["Pepper Jack Cheese Skewers"]
|
53
|
+
end
|
54
|
+
|
55
|
+
def test_cheese_space_in_query
|
56
|
+
store_names ["Pepperjack Cheese Skewers"]
|
57
|
+
assert_search "pepper jack cheese skewers", ["Pepperjack Cheese Skewers"]
|
58
|
+
end
|
59
|
+
|
60
|
+
def test_middle_token
|
61
|
+
store_names ["Dish Washer Amazing Organic Soap"]
|
62
|
+
assert_search "dish soap", ["Dish Washer Amazing Organic Soap"]
|
63
|
+
end
|
64
|
+
|
65
|
+
def test_percent
|
66
|
+
store_names ["1% Milk", "2% Milk", "Whole Milk"]
|
67
|
+
assert_search "1%", ["1% Milk"]
|
68
|
+
end
|
69
|
+
|
70
|
+
# ascii
|
71
|
+
|
72
|
+
def test_jalapenos
|
73
|
+
store_names ["Jalapeño"]
|
74
|
+
assert_search "jalapeno", ["Jalapeño"]
|
75
|
+
end
|
76
|
+
|
77
|
+
# stemming
|
78
|
+
|
79
|
+
def test_stemming
|
80
|
+
store_names ["Whole Milk", "Fat Free Milk", "Milk"]
|
81
|
+
assert_search "milks", ["Milk", "Whole Milk", "Fat Free Milk"]
|
82
|
+
end
|
83
|
+
|
84
|
+
# fuzzy
|
85
|
+
|
86
|
+
def test_misspelling_sriracha
|
87
|
+
store_names ["Sriracha"]
|
88
|
+
assert_search "siracha", ["Sriracha"]
|
89
|
+
end
|
90
|
+
|
91
|
+
def test_misspelling_tabasco
|
92
|
+
store_names ["Tabasco"]
|
93
|
+
assert_search "tobasco", ["Tabasco"]
|
94
|
+
end
|
95
|
+
|
96
|
+
def test_misspelling_zucchini
|
97
|
+
store_names ["Zucchini"]
|
98
|
+
assert_search "zuchini", ["Zucchini"]
|
99
|
+
end
|
100
|
+
|
101
|
+
def test_misspelling_ziploc
|
102
|
+
store_names ["Ziploc"]
|
103
|
+
assert_search "zip lock", ["Ziploc"]
|
104
|
+
end
|
105
|
+
|
106
|
+
# conversions
|
107
|
+
|
108
|
+
def test_conversions
|
109
|
+
store [
|
110
|
+
{name: "Tomato Sauce", conversions: [{query: "tomato sauce", count: 5}, {query: "tomato", count: 200}]},
|
111
|
+
{name: "Tomato Paste", conversions: []},
|
112
|
+
{name: "Tomatoes", conversions: [{query: "tomato", count: 100}, {query: "tomato sauce", count: 2}]}
|
113
|
+
]
|
114
|
+
assert_search "tomato", ["Tomato Sauce", "Tomatoes", "Tomato Paste"]
|
115
|
+
end
|
116
|
+
|
117
|
+
def test_conversions_stemmed
|
118
|
+
store [
|
119
|
+
{name: "Tomato A", conversions: [{query: "tomato", count: 2}, {query: "tomatos", count: 2}, {query: "Tomatoes", count: 2}]},
|
120
|
+
{name: "Tomato B", conversions: [{query: "tomato", count: 4}]}
|
121
|
+
]
|
122
|
+
assert_search "tomato", ["Tomato A", "Tomato B"]
|
123
|
+
end
|
124
|
+
|
125
|
+
# spaces
|
126
|
+
|
127
|
+
def test_spaces_in_field
|
128
|
+
store_names ["Red Bull"]
|
129
|
+
assert_search "redbull", ["Red Bull"]
|
130
|
+
end
|
131
|
+
|
132
|
+
def test_spaces_in_query
|
133
|
+
store_names ["Dishwasher Soap"]
|
134
|
+
assert_search "dish washer", ["Dishwasher Soap"]
|
135
|
+
end
|
136
|
+
|
137
|
+
def test_spaces_three_words
|
138
|
+
store_names ["Dish Washer Soap", "Dish Washer"]
|
139
|
+
assert_search "dish washer soap", ["Dish Washer Soap"]
|
140
|
+
end
|
141
|
+
|
142
|
+
def test_spaces_stemming
|
143
|
+
store_names ["Almond Milk"]
|
144
|
+
assert_search "almondmilks", ["Almond Milk"]
|
145
|
+
end
|
146
|
+
|
147
|
+
# keywords
|
148
|
+
|
149
|
+
def test_keywords
|
150
|
+
store_names ["Clorox Bleach", "Kroger Bleach", "Saran Wrap", "Kroger Plastic Wrap", "Hamburger Buns", "Band-Aid", "Kroger 12-Pack Bandages"]
|
151
|
+
assert_search "clorox", ["Clorox Bleach", "Kroger Bleach"]
|
152
|
+
assert_search "saran wrap", ["Saran Wrap", "Kroger Plastic Wrap"]
|
153
|
+
assert_search "burger buns", ["Hamburger Buns"]
|
154
|
+
assert_search "bandaids", ["Band-Aid", "Kroger 12-Pack Bandages"]
|
155
|
+
end
|
156
|
+
|
157
|
+
def test_keywords_qtips
|
158
|
+
store_names ["Q Tips", "Kroger Cotton Swabs"]
|
159
|
+
assert_search "q tips", ["Q Tips", "Kroger Cotton Swabs"]
|
160
|
+
end
|
161
|
+
|
162
|
+
def test_keywords_reverse
|
163
|
+
store_names ["Scallions"]
|
164
|
+
assert_search "green onions", ["Scallions"]
|
165
|
+
end
|
166
|
+
|
167
|
+
def test_keywords_exact
|
168
|
+
store_names ["Green Onions", "Yellow Onions"]
|
169
|
+
assert_search "scallion", ["Green Onions"]
|
170
|
+
end
|
171
|
+
|
172
|
+
def test_keywords_stemmed
|
173
|
+
store_names ["Green Onions", "Yellow Onions"]
|
174
|
+
assert_search "scallions", ["Green Onions"]
|
175
|
+
end
|
176
|
+
|
177
|
+
# global boost
|
178
|
+
|
179
|
+
def test_boost
|
180
|
+
store [
|
181
|
+
{name: "Organic Tomato A", _boost: 10},
|
182
|
+
{name: "Tomato B"}
|
183
|
+
]
|
184
|
+
assert_search "tomato", ["Organic Tomato A", "Tomato B"]
|
185
|
+
end
|
186
|
+
|
187
|
+
def test_boost_zero
|
188
|
+
store [
|
189
|
+
{name: "Zero Boost", _boost: 0}
|
190
|
+
]
|
191
|
+
assert_search "zero", ["Zero Boost"]
|
192
|
+
end
|
193
|
+
|
194
|
+
# default to 1
|
195
|
+
def test_boost_null
|
196
|
+
store [
|
197
|
+
{name: "Zero Boost A", _boost: 1.1},
|
198
|
+
{name: "Zero Boost B"},
|
199
|
+
{name: "Zero Boost C", _boost: 0.9},
|
200
|
+
]
|
201
|
+
assert_search "zero", ["Zero Boost A", "Zero Boost B", "Zero Boost C"]
|
202
|
+
end
|
203
|
+
|
204
|
+
# search method
|
205
|
+
|
206
|
+
def test_limit
|
207
|
+
store_names ["Product A", "Product B"]
|
208
|
+
assert_equal 1, Product.search("Product", limit: 1).size
|
209
|
+
end
|
210
|
+
|
211
|
+
def test_offset
|
212
|
+
store_names ["Product A", "Product B"]
|
213
|
+
assert_equal 1, Product.search("Product", offset: 1).size
|
214
|
+
end
|
215
|
+
|
216
|
+
def test_where
|
217
|
+
now = Time.now
|
218
|
+
store [
|
219
|
+
{name: "Product A", store_id: 1, in_stock: true, backordered: true, created_at: now, _boost: 4},
|
220
|
+
{name: "Product B", store_id: 2, in_stock: true, backordered: false, created_at: now - 1, _boost: 3},
|
221
|
+
{name: "Product C", store_id: 3, in_stock: false, backordered: true, created_at: now - 2, _boost: 2},
|
222
|
+
{name: "Product D", store_id: 4, in_stock: false, backordered: false, created_at: now - 3, _boost: 1},
|
223
|
+
]
|
224
|
+
assert_search "product", ["Product A", "Product B"], where: {in_stock: true}
|
225
|
+
# date
|
226
|
+
assert_search "product", ["Product A"], where: {created_at: {gt: now - 1}}
|
227
|
+
assert_search "product", ["Product A", "Product B"], where: {created_at: {gte: now - 1}}
|
228
|
+
assert_search "product", ["Product D"], where: {created_at: {lt: now - 2}}
|
229
|
+
assert_search "product", ["Product C", "Product D"], where: {created_at: {lte: now - 2}}
|
230
|
+
# integer
|
231
|
+
assert_search "product", ["Product A"], where: {store_id: {lt: 2}}
|
232
|
+
assert_search "product", ["Product A", "Product B"], where: {store_id: {lte: 2}}
|
233
|
+
assert_search "product", ["Product D"], where: {store_id: {gt: 3}}
|
234
|
+
assert_search "product", ["Product C", "Product D"], where: {store_id: {gte: 3}}
|
235
|
+
# range
|
236
|
+
assert_search "product", ["Product A", "Product B"], where: {store_id: 1..2}
|
237
|
+
assert_search "product", ["Product A"], where: {store_id: 1...2}
|
238
|
+
assert_search "product", ["Product A", "Product B"], where: {store_id: [1, 2]}
|
239
|
+
assert_search "product", ["Product B", "Product C", "Product D"], where: {store_id: {not: 1}}
|
240
|
+
assert_search "product", ["Product C", "Product D"], where: {store_id: {not: [1, 2]}}
|
241
|
+
assert_search "product", ["Product A", "Product B", "Product C"], where: {or: [[{in_stock: true}, {store_id: 3}]]}
|
242
|
+
end
|
243
|
+
|
244
|
+
def test_order
|
245
|
+
store_names ["Product A", "Product B", "Product C", "Product D"]
|
246
|
+
assert_search "product", ["Product D", "Product C", "Product B", "Product A"], order: {name: :desc}
|
247
|
+
end
|
248
|
+
|
249
|
+
def test_facets
|
250
|
+
store [
|
251
|
+
{name: "Product Show", store_id: 1, in_stock: true, color: "blue"},
|
252
|
+
{name: "Product Hide", store_id: 2, in_stock: false, color: "green"},
|
253
|
+
{name: "Product B", store_id: 2, in_stock: false, color: "red"}
|
254
|
+
]
|
255
|
+
assert_equal 2, Product.search("Product", facets: [:store_id]).facets["store_id"]["terms"].size
|
256
|
+
assert_equal 1, Product.search("Product", facets: {store_id: {where: {in_stock: true}}}).facets["store_id"]["terms"].size
|
257
|
+
assert_equal 1, Product.search("Product", facets: {store_id: {where: {in_stock: true, color: "blue"}}}).facets["store_id"]["terms"].size
|
258
|
+
end
|
259
|
+
|
260
|
+
def test_partial
|
261
|
+
store_names ["Honey"]
|
262
|
+
assert_search "fresh honey", []
|
263
|
+
assert_search "fresh honey", ["Honey"], partial: true
|
264
|
+
end
|
265
|
+
|
266
|
+
protected
|
267
|
+
|
268
|
+
def store(documents)
|
269
|
+
documents.each do |document|
|
270
|
+
Product.index.store ({_type: "product"}).merge(document)
|
271
|
+
end
|
272
|
+
Product.index.refresh
|
273
|
+
end
|
274
|
+
|
275
|
+
def store_names(names)
|
276
|
+
store names.map{|name| {name: name} }
|
277
|
+
end
|
278
|
+
|
279
|
+
def assert_search(term, expected, options = {})
|
280
|
+
assert_equal expected, Product.search(term, options.merge(fields: [:name], conversions: true)).map(&:name)
|
281
|
+
end
|
282
|
+
|
283
|
+
end
|
data/test/test_helper.rb
ADDED
@@ -0,0 +1,26 @@
|
|
1
|
+
require "bundler/setup"
|
2
|
+
Bundler.require(:default)
|
3
|
+
require "minitest/autorun"
|
4
|
+
require "minitest/pride"
|
5
|
+
require "active_record"
|
6
|
+
|
7
|
+
# for debugging
|
8
|
+
# ActiveRecord::Base.logger = Logger.new(STDOUT)
|
9
|
+
|
10
|
+
# rails does this in activerecord/lib/active_record/railtie.rb
|
11
|
+
ActiveRecord::Base.default_timezone = :utc
|
12
|
+
ActiveRecord::Base.time_zone_aware_attributes = true
|
13
|
+
|
14
|
+
# migrations
|
15
|
+
ActiveRecord::Base.establish_connection :adapter => "postgresql", :database => "searchkick_test"
|
16
|
+
|
17
|
+
ActiveRecord::Migration.create_table :products, :force => true do |t|
|
18
|
+
t.string :name
|
19
|
+
t.integer :store_id
|
20
|
+
t.boolean :in_stock
|
21
|
+
t.boolean :backordered
|
22
|
+
t.timestamps
|
23
|
+
end
|
24
|
+
|
25
|
+
File.delete("elasticsearch.log") if File.exists?("elasticsearch.log")
|
26
|
+
Tire.configure { logger "elasticsearch.log", :level => "debug" }
|
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: searchkick
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 0.0.
|
4
|
+
version: 0.0.2
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Andrew Kane
|
8
8
|
autorequire:
|
9
9
|
bindir: bin
|
10
10
|
cert_chain: []
|
11
|
-
date: 2013-07-
|
11
|
+
date: 2013-07-18 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
name: tire
|
@@ -52,6 +52,48 @@ dependencies:
|
|
52
52
|
- - '>='
|
53
53
|
- !ruby/object:Gem::Version
|
54
54
|
version: '0'
|
55
|
+
- !ruby/object:Gem::Dependency
|
56
|
+
name: minitest
|
57
|
+
requirement: !ruby/object:Gem::Requirement
|
58
|
+
requirements:
|
59
|
+
- - '>='
|
60
|
+
- !ruby/object:Gem::Version
|
61
|
+
version: '0'
|
62
|
+
type: :development
|
63
|
+
prerelease: false
|
64
|
+
version_requirements: !ruby/object:Gem::Requirement
|
65
|
+
requirements:
|
66
|
+
- - '>='
|
67
|
+
- !ruby/object:Gem::Version
|
68
|
+
version: '0'
|
69
|
+
- !ruby/object:Gem::Dependency
|
70
|
+
name: activerecord
|
71
|
+
requirement: !ruby/object:Gem::Requirement
|
72
|
+
requirements:
|
73
|
+
- - '>='
|
74
|
+
- !ruby/object:Gem::Version
|
75
|
+
version: '0'
|
76
|
+
type: :development
|
77
|
+
prerelease: false
|
78
|
+
version_requirements: !ruby/object:Gem::Requirement
|
79
|
+
requirements:
|
80
|
+
- - '>='
|
81
|
+
- !ruby/object:Gem::Version
|
82
|
+
version: '0'
|
83
|
+
- !ruby/object:Gem::Dependency
|
84
|
+
name: pg
|
85
|
+
requirement: !ruby/object:Gem::Requirement
|
86
|
+
requirements:
|
87
|
+
- - '>='
|
88
|
+
- !ruby/object:Gem::Version
|
89
|
+
version: '0'
|
90
|
+
type: :development
|
91
|
+
prerelease: false
|
92
|
+
version_requirements: !ruby/object:Gem::Requirement
|
93
|
+
requirements:
|
94
|
+
- - '>='
|
95
|
+
- !ruby/object:Gem::Version
|
96
|
+
version: '0'
|
55
97
|
description: Search made easy
|
56
98
|
email:
|
57
99
|
- andrew@chartkick.com
|
@@ -65,9 +107,14 @@ files:
|
|
65
107
|
- README.md
|
66
108
|
- Rakefile
|
67
109
|
- lib/searchkick.rb
|
110
|
+
- lib/searchkick/model.rb
|
111
|
+
- lib/searchkick/reindex.rb
|
112
|
+
- lib/searchkick/search.rb
|
68
113
|
- lib/searchkick/tasks.rb
|
69
114
|
- lib/searchkick/version.rb
|
70
115
|
- searchkick.gemspec
|
116
|
+
- test/searchkick_test.rb
|
117
|
+
- test/test_helper.rb
|
71
118
|
homepage: https://github.com/ankane/searchkick
|
72
119
|
licenses:
|
73
120
|
- MIT
|
@@ -92,4 +139,6 @@ rubygems_version: 2.0.0
|
|
92
139
|
signing_key:
|
93
140
|
specification_version: 4
|
94
141
|
summary: Search made easy
|
95
|
-
test_files:
|
142
|
+
test_files:
|
143
|
+
- test/searchkick_test.rb
|
144
|
+
- test/test_helper.rb
|