full_search 0.2.0 → 0.3.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/README.md +22 -4
- data/lib/full_search/callbacks.rb +26 -8
- data/lib/full_search/config.rb +0 -1
- data/lib/full_search/distance.rb +33 -0
- data/lib/full_search/dsl.rb +26 -12
- data/lib/full_search/errors.rb +2 -0
- data/lib/full_search/exact_match.rb +7 -13
- data/lib/full_search/highlighter.rb +166 -102
- data/lib/full_search/index.rb +107 -51
- data/lib/full_search/model.rb +2 -1
- data/lib/full_search/multi_search.rb +4 -1
- data/lib/full_search/query_parser.rb +101 -88
- data/lib/full_search/quoting.rb +11 -0
- data/lib/full_search/search.rb +97 -77
- data/lib/full_search/soft_delete.rb +15 -2
- data/lib/full_search/typo.rb +10 -8
- data/lib/full_search/version.rb +1 -1
- data/lib/full_search.rb +11 -1
- data/lib/tasks/full_search.rake +12 -4
- metadata +45 -17
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 50bb1c53fe9f326612a2413a441a37acfe48f25bf4730786de3631c7230b2572
|
|
4
|
+
data.tar.gz: ab53bbd57f8fd9cb464af29d37aa70cd0efb18a2835eff1b501d61a14d0de43f
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: ab98601c786c02808246cf4657ef7c3912e36e0fbc429417c0ff564a3ec1e2575216fda8d64ee2ab22db32c707bcbce0915fcc3d57038059f1dfe05bde48963e
|
|
7
|
+
data.tar.gz: 52564a1ce083d5aa0f6cb624f8f9aa4a0bb0ac13a85216f14a857a5c2c2c744d37fa4370272679c3bd13c76eb6ec732814463cfa143bb714654243968c6c800c
|
data/README.md
CHANGED
|
@@ -1,10 +1,18 @@
|
|
|
1
1
|
# full_search
|
|
2
2
|
|
|
3
|
+
[](https://github.com/bendangelo/full_search/actions/workflows/ci.yml)
|
|
4
|
+
|
|
3
5
|
SQLite FTS5 full-text search for Rails/ActiveRecord. A lightweight, self-contained alternative to `pg_search` for apps already running on SQLite.
|
|
4
6
|
|
|
7
|
+
## Requirements
|
|
8
|
+
|
|
9
|
+
- Ruby 3.1+
|
|
10
|
+
- Rails 8.0+
|
|
11
|
+
- SQLite 3.34+ (if using typo tolerance)
|
|
12
|
+
|
|
5
13
|
## When to use
|
|
6
14
|
|
|
7
|
-
`full_search` is designed for apps with **100,000 records or fewer per table** that want full-text search without running a separate service. If you're on SQLite and need keyword search, phrase matching, typo-tolerant substring queries, or result highlighting, this gem gives you
|
|
15
|
+
`full_search` is designed for apps with **100,000 records or fewer per table** that want full-text search without running a separate service. If you're on SQLite and need keyword search, phrase matching, typo-tolerant substring queries, or result highlighting, this gem gives you full-text search suitable for small to medium SQLite-backed apps with zero infrastructure — no Elasticsearch, no Meilisearch, no Sidekiq queue.
|
|
8
16
|
|
|
9
17
|
## Installation
|
|
10
18
|
|
|
@@ -70,6 +78,10 @@ These two operations are often confused:
|
|
|
70
78
|
- **Rebuild** (`full_search:rebuild` / `Customer.rebuild!`) — drops and recreates the FTS virtual table. Needed when the DSL changes (fields added/removed, tokenizer changed, etc.). The table is re-created from scratch, backfilled, triggers re-installed, and the index is optimized.
|
|
71
79
|
- **Reindex** (`Customer.reindex!` / `FullSearch::Index.reindex_source_fields!`) — updates existing FTS rows with fresh values from computed `source:` fields only. The table structure is untouched. Database triggers cover regular column changes automatically; only Ruby-evaluated `source:` blocks need an explicit reindex.
|
|
72
80
|
|
|
81
|
+
> **Note on rebuild locking:** The `lock_rebuilds` option prevents concurrent rebuilds
|
|
82
|
+
> within the same process/connection. For multi-process or multi-host deployments,
|
|
83
|
+
> run `full_search:rebuild` from a single deployment step.
|
|
84
|
+
|
|
73
85
|
### Auto-rebuild on app load
|
|
74
86
|
|
|
75
87
|
When `auto_rebuild_schema` is enabled (default in the generated initializer), the railtie hooks into Rails `after_initialize` and:
|
|
@@ -132,7 +144,7 @@ If you're not on Solid Queue, most job frameworks support recurring schedules vi
|
|
|
132
144
|
|
|
133
145
|
### Config hash drift detection
|
|
134
146
|
|
|
135
|
-
Every FTS table stores a SHA256 digest of its DSL configuration in the `full_search_index_versions` table. When the DSL changes (e.g., adding a field), the stored hash no longer matches, and `rebuild!` is triggered automatically (if `auto_rebuild_schema` is true) to bring the index in line with the new definition. If `auto_rebuild_schema` is false, queries
|
|
147
|
+
Every FTS table stores a SHA256 digest of its DSL configuration in the `full_search_index_versions` table. When the DSL changes (e.g., adding a field), the stored hash no longer matches, and `rebuild!` is triggered automatically (if `auto_rebuild_schema` is true) to bring the index in line with the new definition. If `auto_rebuild_schema` is false, queries raise `ConfigChangedError` when the hash doesn't match (or log a warning and fall back if configured with `:log_and_fallback`).
|
|
136
148
|
|
|
137
149
|
## Query operators
|
|
138
150
|
|
|
@@ -169,9 +181,15 @@ class Customer < ApplicationRecord
|
|
|
169
181
|
end
|
|
170
182
|
```
|
|
171
183
|
|
|
172
|
-
`typo_tolerance` uses an FTS5 trigram shadow table as a fallback when the primary index returns no results. It is substring
|
|
184
|
+
`typo_tolerance` uses an FTS5 trigram shadow table as a fallback when the primary index returns no results. It is substring/trigram fallback, not edit-distance correction, and requires SQLite >= 3.34.
|
|
173
185
|
|
|
174
186
|
## Known limitations
|
|
175
187
|
|
|
176
188
|
- Queries run with `highlight: true` return an Array of records, not an `ActiveRecord::Relation`. No further chaining (`.where`, `.order`, `.limit`) is possible after highlighting is applied.
|
|
177
|
-
- Per-model only;
|
|
189
|
+
- Per-model only; `multi_search` returns grouped results, not a unified ranked list
|
|
190
|
+
- No built-in multi-model aggregator
|
|
191
|
+
|
|
192
|
+
## Security
|
|
193
|
+
|
|
194
|
+
If you discover a security vulnerability, please email the maintainer directly at
|
|
195
|
+
the address listed in the gemspec. Do not file a public issue.
|
|
@@ -3,6 +3,8 @@
|
|
|
3
3
|
module FullSearch
|
|
4
4
|
module Callbacks
|
|
5
5
|
def self.install!(model)
|
|
6
|
+
return if model.instance_variable_get(:@__full_search_callbacks_installed)
|
|
7
|
+
|
|
6
8
|
dsl = model.full_search_dsl
|
|
7
9
|
|
|
8
10
|
source_fields = dsl.fields.select(&:source)
|
|
@@ -27,6 +29,12 @@ module FullSearch
|
|
|
27
29
|
FullSearch::Callbacks.reindex_dependents!(record, model, field)
|
|
28
30
|
end
|
|
29
31
|
end
|
|
32
|
+
|
|
33
|
+
model.instance_variable_set(:@__full_search_callbacks_installed, true)
|
|
34
|
+
end
|
|
35
|
+
|
|
36
|
+
def self.reset_installed_flag!(model)
|
|
37
|
+
model.instance_variable_set(:@__full_search_callbacks_installed, false)
|
|
30
38
|
end
|
|
31
39
|
|
|
32
40
|
def self.reindex_record!(record)
|
|
@@ -46,22 +54,24 @@ module FullSearch
|
|
|
46
54
|
return unless field&.source
|
|
47
55
|
|
|
48
56
|
value = record.instance_exec(&field.source)
|
|
49
|
-
table = FullSearch::Index.fts_table_name(record.class)
|
|
50
|
-
|
|
51
|
-
|
|
52
|
-
"UPDATE #{table} SET #{field.name} = #{
|
|
57
|
+
table = qt(FullSearch::Index.fts_table_name(record.class))
|
|
58
|
+
conn = ActiveRecord::Base.connection
|
|
59
|
+
conn.execute(
|
|
60
|
+
"UPDATE #{table} SET #{qc(field.name)} = #{q(value.to_s)} WHERE rowid = #{q(record.id)}"
|
|
53
61
|
)
|
|
54
62
|
end
|
|
55
63
|
|
|
56
64
|
def self.remove_record!(record)
|
|
57
|
-
table = FullSearch::Index.fts_table_name(record.class)
|
|
58
|
-
ActiveRecord::Base.connection
|
|
65
|
+
table = qt(FullSearch::Index.fts_table_name(record.class))
|
|
66
|
+
conn = ActiveRecord::Base.connection
|
|
67
|
+
conn.execute("DELETE FROM #{table} WHERE rowid = #{q(record.id)}")
|
|
59
68
|
end
|
|
60
69
|
|
|
61
70
|
def self.reindex_dependents!(parent_record, dependent_model, field)
|
|
62
71
|
fk = association_key(dependent_model, field.reindex_on)
|
|
63
|
-
|
|
64
|
-
|
|
72
|
+
conn = ActiveRecord::Base.connection
|
|
73
|
+
sql = "SELECT id FROM #{qt(dependent_model.table_name)} WHERE #{qc(fk)} = #{q(parent_record.id)}"
|
|
74
|
+
dependent_ids = conn.execute(sql).map { |r| r["id"] }
|
|
65
75
|
|
|
66
76
|
dependent_ids.each do |dep_id|
|
|
67
77
|
if field.async
|
|
@@ -73,6 +83,14 @@ module FullSearch
|
|
|
73
83
|
end
|
|
74
84
|
end
|
|
75
85
|
|
|
86
|
+
class << self
|
|
87
|
+
include FullSearch::Quoting
|
|
88
|
+
end
|
|
89
|
+
|
|
90
|
+
def self.connection
|
|
91
|
+
ActiveRecord::Base.connection
|
|
92
|
+
end
|
|
93
|
+
|
|
76
94
|
def self.associated_class(model, association_name)
|
|
77
95
|
reflection = model.reflect_on_association(association_name.to_sym)
|
|
78
96
|
reflection&.klass
|
data/lib/full_search/config.rb
CHANGED
|
@@ -0,0 +1,33 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module FullSearch
|
|
4
|
+
module Distance
|
|
5
|
+
def self.damerau_levenshtein(a, b)
|
|
6
|
+
a_len = a.length
|
|
7
|
+
b_len = b.length
|
|
8
|
+
return a_len if b_len == 0
|
|
9
|
+
return b_len if a_len == 0
|
|
10
|
+
|
|
11
|
+
d = Array.new(a_len + 1) { Array.new(b_len + 1, 0) }
|
|
12
|
+
(0..a_len).each { |i| d[i][0] = i }
|
|
13
|
+
(0..b_len).each { |j| d[0][j] = j }
|
|
14
|
+
|
|
15
|
+
(1..a_len).each do |i|
|
|
16
|
+
(1..b_len).each do |j|
|
|
17
|
+
cost = (a[i - 1] == b[j - 1]) ? 0 : 1
|
|
18
|
+
d[i][j] = [
|
|
19
|
+
d[i - 1][j] + 1,
|
|
20
|
+
d[i][j - 1] + 1,
|
|
21
|
+
d[i - 1][j - 1] + cost
|
|
22
|
+
].min
|
|
23
|
+
|
|
24
|
+
if i > 1 && j > 1 && a[i - 1] == b[j - 2] && a[i - 2] == b[j - 1]
|
|
25
|
+
d[i][j] = [d[i][j], d[i - 2][j - 2] + 1].min
|
|
26
|
+
end
|
|
27
|
+
end
|
|
28
|
+
end
|
|
29
|
+
|
|
30
|
+
d[a_len][b_len]
|
|
31
|
+
end
|
|
32
|
+
end
|
|
33
|
+
end
|
data/lib/full_search/dsl.rb
CHANGED
|
@@ -2,10 +2,10 @@
|
|
|
2
2
|
|
|
3
3
|
module FullSearch
|
|
4
4
|
class Dsl
|
|
5
|
-
attr_reader :fields, :exact_matches, :filters, :model_class, :
|
|
5
|
+
attr_reader :fields, :exact_matches, :filters, :model_class, :highlight_config, :rank_bys
|
|
6
6
|
|
|
7
|
-
Field = Data.define(:name, :weight, :source, :reindex_on, :async, :as)
|
|
8
|
-
ExactMatch = Data.define(:name, :source)
|
|
7
|
+
Field = Data.define(:name, :weight, :source, :reindex_on, :async, :as, :version)
|
|
8
|
+
ExactMatch = Data.define(:name, :source, :version)
|
|
9
9
|
Filter = Data.define(:name, :required)
|
|
10
10
|
RankBy = Data.define(:column, :direction)
|
|
11
11
|
|
|
@@ -19,35 +19,49 @@ module FullSearch
|
|
|
19
19
|
@soft_delete_column = nil
|
|
20
20
|
end
|
|
21
21
|
|
|
22
|
-
def field(name, weight: 1, source: nil, reindex_on: nil, async: FullSearch.config.default_async_reindex, as: nil)
|
|
22
|
+
def field(name, weight: 1, source: nil, reindex_on: nil, async: FullSearch.config.default_async_reindex, as: nil, version: nil)
|
|
23
23
|
unless valid_name?(name)
|
|
24
24
|
raise InvalidFieldError, "Invalid field name: #{name.inspect}"
|
|
25
25
|
end
|
|
26
26
|
if as && !valid_name?(as)
|
|
27
27
|
raise InvalidFieldError, "Invalid field alias (as): #{as.inspect}"
|
|
28
28
|
end
|
|
29
|
-
|
|
29
|
+
str = name.to_s
|
|
30
|
+
raise InvalidFieldError, "Duplicate field name: #{name.inspect}" if fields.any? { |f| f.name == str }
|
|
31
|
+
raise InvalidFieldError, "Field name conflicts with filter: #{name.inspect}" if filters.any? { |f| f.name == str }
|
|
32
|
+
@fields << Field.new(name: str, weight: weight.to_i, source: source, reindex_on: reindex_on&.to_s, async: async, as: as&.to_s, version: version)
|
|
30
33
|
end
|
|
31
34
|
|
|
32
|
-
def exact_match(name, source: -> { public_send(name) })
|
|
35
|
+
def exact_match(name, source: -> { public_send(name) }, version: nil)
|
|
33
36
|
unless valid_name?(name)
|
|
34
37
|
raise InvalidFieldError, "Invalid exact_match name: #{name.inspect}"
|
|
35
38
|
end
|
|
36
|
-
|
|
39
|
+
str = name.to_s
|
|
40
|
+
raise InvalidFieldError, "Duplicate exact_match name: #{name.inspect}" if exact_matches.any? { |e| e.name == str }
|
|
41
|
+
@exact_matches << ExactMatch.new(name: str, source: source, version: version)
|
|
37
42
|
end
|
|
38
43
|
|
|
39
44
|
def rank_by(column, direction = :desc)
|
|
40
45
|
unless valid_name?(column)
|
|
41
46
|
raise InvalidFieldError, "Invalid rank_by column: #{column.inspect}"
|
|
42
47
|
end
|
|
43
|
-
|
|
48
|
+
dir = direction.to_s.downcase
|
|
49
|
+
unless %w[asc desc].include?(dir)
|
|
50
|
+
raise InvalidFieldError, "Invalid rank_by direction: #{direction.inspect}. Use :asc or :desc."
|
|
51
|
+
end
|
|
52
|
+
str = column.to_s
|
|
53
|
+
raise InvalidFieldError, "Duplicate rank_by column: #{column.inspect}" if rank_bys.any? { |r| r.column == str }
|
|
54
|
+
@rank_bys << RankBy.new(column: str, direction: dir.to_sym)
|
|
44
55
|
end
|
|
45
56
|
|
|
46
57
|
def filter(name, required: false)
|
|
47
58
|
unless valid_name?(name)
|
|
48
59
|
raise InvalidFieldError, "Invalid filter name: #{name.inspect}"
|
|
49
60
|
end
|
|
50
|
-
|
|
61
|
+
str = name.to_s
|
|
62
|
+
raise InvalidFieldError, "Duplicate filter name: #{name.inspect}" if filters.any? { |f| f.name == str }
|
|
63
|
+
raise InvalidFieldError, "Filter name conflicts with field: #{name.inspect}" if fields.any? { |f| f.name == str }
|
|
64
|
+
@filters << Filter.new(name: str, required: required)
|
|
51
65
|
end
|
|
52
66
|
|
|
53
67
|
def soft_delete_column(name = :_no_arg_)
|
|
@@ -67,7 +81,7 @@ module FullSearch
|
|
|
67
81
|
end
|
|
68
82
|
|
|
69
83
|
def highlight(open_tag: "<mark>", close_tag: "</mark>")
|
|
70
|
-
@highlight_config = {
|
|
84
|
+
@highlight_config = {open_tag: open_tag, close_tag: close_tag}
|
|
71
85
|
end
|
|
72
86
|
|
|
73
87
|
def typo_tolerance(enabled = true, min_term_length: nil)
|
|
@@ -91,8 +105,8 @@ module FullSearch
|
|
|
91
105
|
soft_delete_column,
|
|
92
106
|
typo_tolerance?,
|
|
93
107
|
typo_tolerance_min_term_length,
|
|
94
|
-
fields.map { |f| [f.name, f.weight, f.source.nil? ? "column" : f.
|
|
95
|
-
exact_matches.map { |e| [e.name, e.
|
|
108
|
+
fields.map { |f| [f.name, f.weight, f.source.nil? ? "column" : "proc:#{f.version}", f.reindex_on, f.async, f.as] },
|
|
109
|
+
exact_matches.map { |e| [e.name, "proc:#{e.version}"] },
|
|
96
110
|
filters.map { |f| [f.name, f.required] },
|
|
97
111
|
rank_bys.map { |r| [r.column, r.direction] }
|
|
98
112
|
].inspect)
|
data/lib/full_search/errors.rb
CHANGED
|
@@ -3,8 +3,10 @@
|
|
|
3
3
|
module FullSearch
|
|
4
4
|
class Error < StandardError; end
|
|
5
5
|
class MissingRequiredFilterError < Error; end
|
|
6
|
+
class UnknownFilterError < Error; end
|
|
6
7
|
class ConfigChangedError < Error; end
|
|
7
8
|
class InvalidFieldError < Error; end
|
|
8
9
|
class NotConfiguredError < Error; end
|
|
9
10
|
class UnsupportedDatabaseError < Error; end
|
|
11
|
+
class InvalidQueryError < Error; end
|
|
10
12
|
end
|
|
@@ -12,20 +12,14 @@ module FullSearch
|
|
|
12
12
|
relation = model.all
|
|
13
13
|
filters.each { |name, value| relation = relation.where(name => value) }
|
|
14
14
|
|
|
15
|
-
|
|
16
|
-
|
|
17
|
-
|
|
18
|
-
|
|
19
|
-
|
|
15
|
+
ids = []
|
|
16
|
+
relation.find_each do |record|
|
|
17
|
+
dsl.exact_matches.each do |em|
|
|
18
|
+
value = record.instance_exec(&em.source)
|
|
19
|
+
ids << record.id if value.to_s.casecmp?(cleaned)
|
|
20
|
+
end
|
|
20
21
|
end
|
|
21
|
-
|
|
22
|
-
exact_ids.uniq
|
|
23
|
-
end
|
|
24
|
-
|
|
25
|
-
def self.exact_match_value(em, query)
|
|
26
|
-
fake = Object.new
|
|
27
|
-
fake.define_singleton_method(em.name) { query }
|
|
28
|
-
fake.instance_exec(&em.source)
|
|
22
|
+
ids.uniq
|
|
29
23
|
end
|
|
30
24
|
end
|
|
31
25
|
end
|
|
@@ -2,143 +2,207 @@
|
|
|
2
2
|
|
|
3
3
|
module FullSearch
|
|
4
4
|
class Highlighter
|
|
5
|
-
|
|
6
|
-
|
|
7
|
-
|
|
8
|
-
snippets
|
|
5
|
+
class << self
|
|
6
|
+
def apply!(records, model, query)
|
|
7
|
+
snippets = build_snippets(model, query)
|
|
8
|
+
if snippets.values.all?(&:nil?) && records.any?
|
|
9
|
+
snippets = manual_snippets(records, model, query)
|
|
10
|
+
end
|
|
11
|
+
records.each { |record| record.full_search_snippet = snippets[record.id] }
|
|
12
|
+
records
|
|
9
13
|
end
|
|
10
|
-
records.each { |record| record.full_search_snippet = snippets[record.id] }
|
|
11
|
-
records
|
|
12
|
-
end
|
|
13
14
|
|
|
14
|
-
|
|
15
|
-
|
|
16
|
-
|
|
17
|
-
|
|
18
|
-
|
|
19
|
-
|
|
20
|
-
|
|
21
|
-
|
|
22
|
-
|
|
23
|
-
|
|
15
|
+
def apply_fields!(records, model, query)
|
|
16
|
+
fields = build_field_snippets(model, query)
|
|
17
|
+
if fields.values.all?(&:empty?) && records.any?
|
|
18
|
+
fields = manual_field_snippets(records, model, query)
|
|
19
|
+
end
|
|
20
|
+
exact_fields = exact_match_field_snippets(records, model, query)
|
|
21
|
+
records.each do |record|
|
|
22
|
+
merged = fields[record.id] || {}
|
|
23
|
+
exact_fields.fetch(record.id, {}).each { |k, v| merged[k] ||= v }
|
|
24
|
+
record.full_search_highlight_fields = merged
|
|
25
|
+
end
|
|
26
|
+
records
|
|
24
27
|
end
|
|
25
|
-
records
|
|
26
|
-
end
|
|
27
28
|
|
|
28
|
-
|
|
29
|
+
private
|
|
29
30
|
|
|
30
|
-
|
|
31
|
-
|
|
32
|
-
|
|
31
|
+
def build_snippets(model, query)
|
|
32
|
+
rows = highlight_rows(model, query)
|
|
33
|
+
cols = model.full_search_dsl.fields.map(&:name)
|
|
33
34
|
|
|
34
|
-
|
|
35
|
-
|
|
36
|
-
|
|
35
|
+
rows.to_h do |row|
|
|
36
|
+
text = cols.map { |col| row["#{col}_snippet"] }.compact.join(" ").strip
|
|
37
|
+
[row["rowid"], text.presence]
|
|
38
|
+
end
|
|
37
39
|
end
|
|
38
|
-
end
|
|
39
40
|
|
|
40
|
-
|
|
41
|
-
|
|
42
|
-
|
|
43
|
-
|
|
44
|
-
|
|
45
|
-
|
|
46
|
-
|
|
47
|
-
|
|
48
|
-
|
|
49
|
-
|
|
50
|
-
|
|
41
|
+
def build_field_snippets(model, query)
|
|
42
|
+
rows = highlight_rows(model, query)
|
|
43
|
+
dsl = model.full_search_dsl
|
|
44
|
+
fields = dsl.fields
|
|
45
|
+
open_tag = (dsl.highlight_config || {open_tag: "<mark>"})[:open_tag]
|
|
46
|
+
|
|
47
|
+
rows.to_h do |row|
|
|
48
|
+
snippets = fields.each_with_object({}) do |field, hash|
|
|
49
|
+
snippet = row["#{field.name}_snippet"].to_s.strip
|
|
50
|
+
key = field.as || field.name
|
|
51
|
+
hash[key] = snippet if snippet.include?(open_tag)
|
|
52
|
+
end
|
|
53
|
+
[row["rowid"], snippets]
|
|
51
54
|
end
|
|
52
|
-
[row["rowid"], snippets]
|
|
53
55
|
end
|
|
54
|
-
end
|
|
55
56
|
|
|
56
|
-
|
|
57
|
-
|
|
58
|
-
|
|
57
|
+
def compute_max_typos(query, dsl)
|
|
58
|
+
return nil unless dsl.typo_tolerance?
|
|
59
|
+
length = query.length
|
|
60
|
+
min_length = dsl.typo_tolerance_min_term_length.to_i
|
|
61
|
+
return nil if length < min_length
|
|
62
|
+
return 2 if length >= 9
|
|
63
|
+
1
|
|
64
|
+
end
|
|
65
|
+
|
|
66
|
+
def manual_snippets(records, model, query)
|
|
67
|
+
dsl = model.full_search_dsl
|
|
68
|
+
config = dsl.highlight_config || {open_tag: "<mark>", close_tag: "</mark>"}
|
|
59
69
|
cols = dsl.fields.map(&:name)
|
|
70
|
+
max_typos = compute_max_typos(query, dsl)
|
|
60
71
|
|
|
61
72
|
records.to_h do |record|
|
|
62
73
|
text = cols.map { |col| record.full_search_text_for(col).to_s }.join(" ").strip
|
|
63
|
-
highlighted = manual_highlight(text, query, config)
|
|
74
|
+
highlighted = manual_highlight(text, query, config, max_typos: max_typos)
|
|
64
75
|
[record.id, highlighted.presence]
|
|
65
76
|
end
|
|
66
|
-
|
|
77
|
+
end
|
|
78
|
+
|
|
79
|
+
def manual_field_snippets(records, model, query)
|
|
80
|
+
dsl = model.full_search_dsl
|
|
81
|
+
config = dsl.highlight_config || {open_tag: "<mark>", close_tag: "</mark>"}
|
|
82
|
+
fields = dsl.fields
|
|
83
|
+
max_typos = compute_max_typos(query, dsl)
|
|
67
84
|
|
|
68
|
-
|
|
69
|
-
|
|
70
|
-
|
|
71
|
-
|
|
72
|
-
|
|
73
|
-
|
|
74
|
-
|
|
75
|
-
|
|
76
|
-
highlighted = manual_highlight(value, query, config)
|
|
77
|
-
key = field.as || field.name
|
|
78
|
-
hash[key] = highlighted if highlighted.include?(config[:open_tag])
|
|
85
|
+
records.to_h do |record|
|
|
86
|
+
snippets = fields.each_with_object({}) do |field, hash|
|
|
87
|
+
value = record.full_search_text_for(field.name).to_s
|
|
88
|
+
highlighted = manual_highlight(value, query, config, max_typos: max_typos)
|
|
89
|
+
key = field.as || field.name
|
|
90
|
+
hash[key] = highlighted if highlighted.include?(config[:open_tag])
|
|
91
|
+
end
|
|
92
|
+
[record.id, snippets]
|
|
79
93
|
end
|
|
80
|
-
[record.id, snippets]
|
|
81
94
|
end
|
|
82
|
-
end
|
|
83
95
|
|
|
84
|
-
|
|
85
|
-
|
|
86
|
-
|
|
96
|
+
def exact_match_field_snippets(records, model, query)
|
|
97
|
+
dsl = model.full_search_dsl
|
|
98
|
+
return {} if dsl.exact_matches.empty?
|
|
87
99
|
|
|
88
|
-
|
|
100
|
+
config = dsl.highlight_config || {open_tag: "<mark>", close_tag: "</mark>"}
|
|
101
|
+
max_typos = compute_max_typos(query, dsl)
|
|
89
102
|
|
|
90
|
-
|
|
91
|
-
|
|
92
|
-
|
|
93
|
-
|
|
94
|
-
|
|
103
|
+
records.to_h do |record|
|
|
104
|
+
snippets = dsl.exact_matches.each_with_object({}) do |em, hash|
|
|
105
|
+
value = exact_match_field_value(em, record)
|
|
106
|
+
highlighted = manual_highlight(value.to_s, query, config, max_typos: max_typos)
|
|
107
|
+
hash[em.name.to_s] = highlighted if highlighted.include?(config[:open_tag])
|
|
108
|
+
end
|
|
109
|
+
[record.id, snippets]
|
|
95
110
|
end
|
|
96
|
-
[record.id, snippets]
|
|
97
111
|
end
|
|
98
|
-
end
|
|
99
112
|
|
|
100
|
-
|
|
101
|
-
|
|
102
|
-
|
|
103
|
-
|
|
104
|
-
|
|
113
|
+
def exact_match_field_value(em, record)
|
|
114
|
+
if em.source
|
|
115
|
+
record.instance_exec(&em.source)
|
|
116
|
+
else
|
|
117
|
+
record.public_send(em.name)
|
|
118
|
+
end
|
|
105
119
|
end
|
|
106
|
-
end
|
|
107
120
|
|
|
108
|
-
|
|
109
|
-
|
|
110
|
-
|
|
111
|
-
|
|
121
|
+
def manual_highlight(text, query, config, max_typos: nil)
|
|
122
|
+
return text if text.empty? || query.empty?
|
|
123
|
+
|
|
124
|
+
open_tag = config[:open_tag]
|
|
125
|
+
close_tag = config[:close_tag]
|
|
126
|
+
escaped_query = Regexp.escape(query)
|
|
112
127
|
|
|
113
|
-
|
|
114
|
-
|
|
115
|
-
|
|
116
|
-
|
|
117
|
-
|
|
128
|
+
if text.match?(/#{escaped_query}/i)
|
|
129
|
+
return text.gsub(/#{escaped_query}/i, "#{open_tag}\\0#{close_tag}")
|
|
130
|
+
end
|
|
131
|
+
|
|
132
|
+
best = best_fuzzy_match(text, query, max_typos: max_typos)
|
|
133
|
+
if best
|
|
134
|
+
start_pos, end_pos = best
|
|
135
|
+
return "#{text[0...start_pos]}#{open_tag}#{text[start_pos...end_pos]}#{close_tag}#{text[end_pos..]}"
|
|
136
|
+
end
|
|
118
137
|
|
|
119
|
-
|
|
120
|
-
|
|
121
|
-
return [] if cols.empty?
|
|
138
|
+
text
|
|
139
|
+
end
|
|
122
140
|
|
|
123
|
-
|
|
124
|
-
|
|
125
|
-
|
|
141
|
+
def best_fuzzy_match(text, query, max_typos: nil)
|
|
142
|
+
query_len = query.length
|
|
143
|
+
text_len = text.length
|
|
144
|
+
return nil if query_len == 0 || text_len == 0
|
|
126
145
|
|
|
127
|
-
|
|
128
|
-
|
|
129
|
-
FROM #{table}
|
|
130
|
-
WHERE #{table} MATCH #{quote(match_expr)}
|
|
131
|
-
SQL
|
|
146
|
+
max_typos ||= max_allowed_typos(query_len)
|
|
147
|
+
return nil if max_typos < 0
|
|
132
148
|
|
|
133
|
-
|
|
134
|
-
|
|
149
|
+
best_score = max_typos + 1
|
|
150
|
+
best_range = nil
|
|
135
151
|
|
|
136
|
-
|
|
137
|
-
|
|
138
|
-
|
|
152
|
+
min_window = [1, query_len - max_typos].max
|
|
153
|
+
max_window = [text_len, query_len + max_typos].min
|
|
154
|
+
return nil if min_window > max_window
|
|
155
|
+
|
|
156
|
+
(min_window..max_window).each do |window_len|
|
|
157
|
+
(0..(text_len - window_len)).each do |start|
|
|
158
|
+
substr = text[start, window_len]
|
|
159
|
+
distance = Distance.damerau_levenshtein(query.downcase, substr.downcase)
|
|
160
|
+
next if distance > max_typos
|
|
161
|
+
|
|
162
|
+
if best_range.nil? || distance < best_score || (distance == best_score && window_len > (best_range[1] - best_range[0]))
|
|
163
|
+
best_score = distance
|
|
164
|
+
best_range = [start, start + window_len]
|
|
165
|
+
end
|
|
166
|
+
end
|
|
167
|
+
end
|
|
168
|
+
|
|
169
|
+
best_range
|
|
170
|
+
end
|
|
139
171
|
|
|
140
|
-
|
|
141
|
-
|
|
172
|
+
def max_allowed_typos(length)
|
|
173
|
+
return -1 if length < 3
|
|
174
|
+
return 2 if length >= 9
|
|
175
|
+
1
|
|
176
|
+
end
|
|
177
|
+
|
|
178
|
+
def highlight_rows(model, query)
|
|
179
|
+
dsl = model.full_search_dsl
|
|
180
|
+
config = dsl.highlight_config || {open_tag: "<mark>", close_tag: "</mark>"}
|
|
181
|
+
match_expr = QueryParser.to_match_expression(QueryParser.parse(query))
|
|
182
|
+
return [] if match_expr.empty?
|
|
183
|
+
|
|
184
|
+
table = qt(FullSearch::Index.fts_table_name(model))
|
|
185
|
+
cols = dsl.fields.map(&:name)
|
|
186
|
+
return [] if cols.empty?
|
|
187
|
+
|
|
188
|
+
highlight_parts = cols.each_with_index.map do |col, idx|
|
|
189
|
+
"highlight(#{table}, #{idx}, #{q(config[:open_tag])}, #{q(config[:close_tag])}) AS #{qc("#{col}_snippet")}"
|
|
190
|
+
end.join(", ")
|
|
191
|
+
|
|
192
|
+
sql = <<~SQL
|
|
193
|
+
SELECT rowid, #{highlight_parts}
|
|
194
|
+
FROM #{table}
|
|
195
|
+
WHERE #{table} MATCH #{q(match_expr)}
|
|
196
|
+
SQL
|
|
197
|
+
|
|
198
|
+
connection.execute(sql)
|
|
199
|
+
end
|
|
200
|
+
|
|
201
|
+
include Quoting
|
|
202
|
+
|
|
203
|
+
def connection
|
|
204
|
+
ActiveRecord::Base.connection
|
|
205
|
+
end
|
|
142
206
|
end
|
|
143
207
|
end
|
|
144
208
|
end
|