full_search 0.2.0 → 0.3.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 339422f4765fd1cd96ef55e9d026d782e10b08c2c6d1636d6e2e13454cf12b51
4
- data.tar.gz: fc5b3a6105d9359162aebb6d14dddf6763e0e0e301351157a1c1c278386d3355
3
+ metadata.gz: 866994a5c7f55cea4237fb2edcbf59f1f7c8c32772c6830df737bcbea4a6f1c6
4
+ data.tar.gz: 5a11ccf1c48abd1050288bbb7107b8ade82132e009ac23ed5803ed153675f535
5
5
  SHA512:
6
- metadata.gz: 38a981537301ec24ec8ee6d38d1174e771b5dbdb3ebc4fa3e0bea857c60d3a440156e5cf9b1ae43be068c616f65381f4866d6294aa9aa4dab5a3e662cb3bfe44
7
- data.tar.gz: 1c1af3ef0948200256f76013f66da00e1764379dbbc07ac8d90264e1a55b9e416218e8569675179736b59821b288aa9ce6fe24fccea48079f60847f127129451
6
+ metadata.gz: 35eb2afacd021899e68eb808b65a6f323db3d66770940471aa94d9c3238e41c409c678ac79d82dc609dcafaf3117d937895fa8effc9a8e665790a8625ef1f231
7
+ data.tar.gz: 5f1f25fe9e2c268e4f6b95c988ddb7958c34649fcf53aa941d0b19d1b605a5b4345864bb7206b597fc509b9802a110db6c3900a40fe1ab3d3ce1468db28aef85
data/README.md CHANGED
@@ -1,10 +1,18 @@
1
1
  # full_search
2
2
 
3
+ [![CI](https://github.com/bendangelo/full_search/actions/workflows/ci.yml/badge.svg)](https://github.com/bendangelo/full_search/actions/workflows/ci.yml)
4
+
3
5
  SQLite FTS5 full-text search for Rails/ActiveRecord. A lightweight, self-contained alternative to `pg_search` for apps already running on SQLite.
4
6
 
7
+ ## Requirements
8
+
9
+ - Ruby 3.1+
10
+ - Rails 8.0+
11
+ - SQLite 3.34+ (if using typo tolerance)
12
+
5
13
  ## When to use
6
14
 
7
- `full_search` is designed for apps with **100,000 records or fewer per table** that want full-text search without running a separate service. If you're on SQLite and need keyword search, phrase matching, typo-tolerant substring queries, or result highlighting, this gem gives you production-quality search with zero infrastructure — no Elasticsearch, no Meilisearch, no Sidekiq queue.
15
+ `full_search` is designed for apps with **100,000 records or fewer per table** that want full-text search without running a separate service. If you're on SQLite and need keyword search, phrase matching, typo-tolerant substring queries, or result highlighting, this gem gives you full-text search suitable for small to medium SQLite-backed apps with zero infrastructure — no Elasticsearch, no Meilisearch, no Sidekiq queue.
8
16
 
9
17
  ## Installation
10
18
 
@@ -70,6 +78,10 @@ These two operations are often confused:
70
78
  - **Rebuild** (`full_search:rebuild` / `Customer.rebuild!`) — drops and recreates the FTS virtual table. Needed when the DSL changes (fields added/removed, tokenizer changed, etc.). The table is re-created from scratch, backfilled, triggers re-installed, and the index is optimized.
71
79
  - **Reindex** (`Customer.reindex!` / `FullSearch::Index.reindex_source_fields!`) — updates existing FTS rows with fresh values from computed `source:` fields only. The table structure is untouched. Database triggers cover regular column changes automatically; only Ruby-evaluated `source:` blocks need an explicit reindex.
72
80
 
81
+ > **Note on rebuild locking:** The `lock_rebuilds` option prevents concurrent rebuilds
82
+ > within the same process/connection. For multi-process or multi-host deployments,
83
+ > run `full_search:rebuild` from a single deployment step.
84
+
73
85
  ### Auto-rebuild on app load
74
86
 
75
87
  When `auto_rebuild_schema` is enabled (default in the generated initializer), the railtie hooks into Rails `after_initialize` and:
@@ -132,7 +144,7 @@ If you're not on Solid Queue, most job frameworks support recurring schedules vi
132
144
 
133
145
  ### Config hash drift detection
134
146
 
135
- Every FTS table stores a SHA256 digest of its DSL configuration in the `full_search_index_versions` table. When the DSL changes (e.g., adding a field), the stored hash no longer matches, and `rebuild!` is triggered automatically (if `auto_rebuild_schema` is true) to bring the index in line with the new definition. If `auto_rebuild_schema` is false, queries still run but raise `ConfigChangedError` when the hash doesn't match.
147
+ Every FTS table stores a SHA256 digest of its DSL configuration in the `full_search_index_versions` table. When the DSL changes (e.g., adding a field), the stored hash no longer matches, and `rebuild!` is triggered automatically (if `auto_rebuild_schema` is true) to bring the index in line with the new definition. If `auto_rebuild_schema` is false, queries raise `ConfigChangedError` when the hash doesn't match (or log a warning and fall back if configured with `:log_and_fallback`).
136
148
 
137
149
  ## Query operators
138
150
 
@@ -169,9 +181,15 @@ class Customer < ApplicationRecord
169
181
  end
170
182
  ```
171
183
 
172
- `typo_tolerance` uses an FTS5 trigram shadow table as a fallback when the primary index returns no results. It is substring matching, not edit-distance correction, and requires SQLite >= 3.34.
184
+ `typo_tolerance` uses an FTS5 trigram shadow table as a fallback when the primary index returns no results. It is substring/trigram fallback, not edit-distance correction, and requires SQLite >= 3.34.
173
185
 
174
186
  ## Known limitations
175
187
 
176
188
  - Queries run with `highlight: true` return an Array of records, not an `ActiveRecord::Relation`. No further chaining (`.where`, `.order`, `.limit`) is possible after highlighting is applied.
177
- - Per-model only; no built-in multi-model aggregator
189
+ - Per-model only; `multi_search` returns grouped results, not a unified ranked list
190
+ - No built-in multi-model aggregator
191
+
192
+ ## Security
193
+
194
+ If you discover a security vulnerability, please email the maintainer directly at
195
+ the address listed in the gemspec. Do not file a public issue.
@@ -3,6 +3,8 @@
3
3
  module FullSearch
4
4
  module Callbacks
5
5
  def self.install!(model)
6
+ return if model.instance_variable_get(:@__full_search_callbacks_installed)
7
+
6
8
  dsl = model.full_search_dsl
7
9
 
8
10
  source_fields = dsl.fields.select(&:source)
@@ -27,6 +29,12 @@ module FullSearch
27
29
  FullSearch::Callbacks.reindex_dependents!(record, model, field)
28
30
  end
29
31
  end
32
+
33
+ model.instance_variable_set(:@__full_search_callbacks_installed, true)
34
+ end
35
+
36
+ def self.reset_installed_flag!(model)
37
+ model.instance_variable_set(:@__full_search_callbacks_installed, false)
30
38
  end
31
39
 
32
40
  def self.reindex_record!(record)
@@ -46,22 +54,24 @@ module FullSearch
46
54
  return unless field&.source
47
55
 
48
56
  value = record.instance_exec(&field.source)
49
- table = FullSearch::Index.fts_table_name(record.class)
50
- quoted_value = ActiveRecord::Base.connection.quote(value.to_s)
51
- ActiveRecord::Base.connection.execute(
52
- "UPDATE #{table} SET #{field.name} = #{quoted_value} WHERE rowid = #{record.id}"
57
+ table = qt(FullSearch::Index.fts_table_name(record.class))
58
+ conn = ActiveRecord::Base.connection
59
+ conn.execute(
60
+ "UPDATE #{table} SET #{qc(field.name)} = #{q(value.to_s)} WHERE rowid = #{q(record.id)}"
53
61
  )
54
62
  end
55
63
 
56
64
  def self.remove_record!(record)
57
- table = FullSearch::Index.fts_table_name(record.class)
58
- ActiveRecord::Base.connection.execute("DELETE FROM #{table} WHERE rowid = #{record.id}")
65
+ table = qt(FullSearch::Index.fts_table_name(record.class))
66
+ conn = ActiveRecord::Base.connection
67
+ conn.execute("DELETE FROM #{table} WHERE rowid = #{q(record.id)}")
59
68
  end
60
69
 
61
70
  def self.reindex_dependents!(parent_record, dependent_model, field)
62
71
  fk = association_key(dependent_model, field.reindex_on)
63
- sql = "SELECT id FROM #{dependent_model.table_name} WHERE #{fk} = #{parent_record.id}"
64
- dependent_ids = ActiveRecord::Base.connection.execute(sql).map { |r| r["id"] }
72
+ conn = ActiveRecord::Base.connection
73
+ sql = "SELECT id FROM #{qt(dependent_model.table_name)} WHERE #{qc(fk)} = #{q(parent_record.id)}"
74
+ dependent_ids = conn.execute(sql).map { |r| r["id"] }
65
75
 
66
76
  dependent_ids.each do |dep_id|
67
77
  if field.async
@@ -73,6 +83,14 @@ module FullSearch
73
83
  end
74
84
  end
75
85
 
86
+ class << self
87
+ include FullSearch::Quoting
88
+ end
89
+
90
+ def self.connection
91
+ ActiveRecord::Base.connection
92
+ end
93
+
76
94
  def self.associated_class(model, association_name)
77
95
  reflection = model.reflect_on_association(association_name.to_sym)
78
96
  reflection&.klass
@@ -11,7 +11,6 @@ module FullSearch
11
11
  @default_async_reindex = true
12
12
  @default_tokenizer = "unicode61"
13
13
  end
14
-
15
14
  end
16
15
 
17
16
  class << self
@@ -0,0 +1,33 @@
1
+ # frozen_string_literal: true
2
+
3
+ module FullSearch
4
+ module Distance
5
+ def self.damerau_levenshtein(a, b)
6
+ a_len = a.length
7
+ b_len = b.length
8
+ return a_len if b_len == 0
9
+ return b_len if a_len == 0
10
+
11
+ d = Array.new(a_len + 1) { Array.new(b_len + 1, 0) }
12
+ (0..a_len).each { |i| d[i][0] = i }
13
+ (0..b_len).each { |j| d[0][j] = j }
14
+
15
+ (1..a_len).each do |i|
16
+ (1..b_len).each do |j|
17
+ cost = (a[i - 1] == b[j - 1]) ? 0 : 1
18
+ d[i][j] = [
19
+ d[i - 1][j] + 1,
20
+ d[i][j - 1] + 1,
21
+ d[i - 1][j - 1] + cost
22
+ ].min
23
+
24
+ if i > 1 && j > 1 && a[i - 1] == b[j - 2] && a[i - 2] == b[j - 1]
25
+ d[i][j] = [d[i][j], d[i - 2][j - 2] + 1].min
26
+ end
27
+ end
28
+ end
29
+
30
+ d[a_len][b_len]
31
+ end
32
+ end
33
+ end
@@ -2,10 +2,10 @@
2
2
 
3
3
  module FullSearch
4
4
  class Dsl
5
- attr_reader :fields, :exact_matches, :filters, :model_class, :tokenize, :highlight_config, :rank_bys
5
+ attr_reader :fields, :exact_matches, :filters, :model_class, :highlight_config, :rank_bys
6
6
 
7
- Field = Data.define(:name, :weight, :source, :reindex_on, :async, :as)
8
- ExactMatch = Data.define(:name, :source)
7
+ Field = Data.define(:name, :weight, :source, :reindex_on, :async, :as, :version)
8
+ ExactMatch = Data.define(:name, :source, :version)
9
9
  Filter = Data.define(:name, :required)
10
10
  RankBy = Data.define(:column, :direction)
11
11
 
@@ -19,35 +19,49 @@ module FullSearch
19
19
  @soft_delete_column = nil
20
20
  end
21
21
 
22
- def field(name, weight: 1, source: nil, reindex_on: nil, async: FullSearch.config.default_async_reindex, as: nil)
22
+ def field(name, weight: 1, source: nil, reindex_on: nil, async: FullSearch.config.default_async_reindex, as: nil, version: nil)
23
23
  unless valid_name?(name)
24
24
  raise InvalidFieldError, "Invalid field name: #{name.inspect}"
25
25
  end
26
26
  if as && !valid_name?(as)
27
27
  raise InvalidFieldError, "Invalid field alias (as): #{as.inspect}"
28
28
  end
29
- @fields << Field.new(name: name.to_s, weight: weight.to_i, source: source, reindex_on: reindex_on&.to_s, async: async, as: as&.to_s)
29
+ str = name.to_s
30
+ raise InvalidFieldError, "Duplicate field name: #{name.inspect}" if fields.any? { |f| f.name == str }
31
+ raise InvalidFieldError, "Field name conflicts with filter: #{name.inspect}" if filters.any? { |f| f.name == str }
32
+ @fields << Field.new(name: str, weight: weight.to_i, source: source, reindex_on: reindex_on&.to_s, async: async, as: as&.to_s, version: version)
30
33
  end
31
34
 
32
- def exact_match(name, source: -> { public_send(name) })
35
+ def exact_match(name, source: -> { public_send(name) }, version: nil)
33
36
  unless valid_name?(name)
34
37
  raise InvalidFieldError, "Invalid exact_match name: #{name.inspect}"
35
38
  end
36
- @exact_matches << ExactMatch.new(name: name.to_s, source: source)
39
+ str = name.to_s
40
+ raise InvalidFieldError, "Duplicate exact_match name: #{name.inspect}" if exact_matches.any? { |e| e.name == str }
41
+ @exact_matches << ExactMatch.new(name: str, source: source, version: version)
37
42
  end
38
43
 
39
44
  def rank_by(column, direction = :desc)
40
45
  unless valid_name?(column)
41
46
  raise InvalidFieldError, "Invalid rank_by column: #{column.inspect}"
42
47
  end
43
- @rank_bys << RankBy.new(column: column.to_s, direction: direction.to_sym)
48
+ dir = direction.to_s.downcase
49
+ unless %w[asc desc].include?(dir)
50
+ raise InvalidFieldError, "Invalid rank_by direction: #{direction.inspect}. Use :asc or :desc."
51
+ end
52
+ str = column.to_s
53
+ raise InvalidFieldError, "Duplicate rank_by column: #{column.inspect}" if rank_bys.any? { |r| r.column == str }
54
+ @rank_bys << RankBy.new(column: str, direction: dir.to_sym)
44
55
  end
45
56
 
46
57
  def filter(name, required: false)
47
58
  unless valid_name?(name)
48
59
  raise InvalidFieldError, "Invalid filter name: #{name.inspect}"
49
60
  end
50
- @filters << Filter.new(name: name.to_s, required: required)
61
+ str = name.to_s
62
+ raise InvalidFieldError, "Duplicate filter name: #{name.inspect}" if filters.any? { |f| f.name == str }
63
+ raise InvalidFieldError, "Filter name conflicts with field: #{name.inspect}" if fields.any? { |f| f.name == str }
64
+ @filters << Filter.new(name: str, required: required)
51
65
  end
52
66
 
53
67
  def soft_delete_column(name = :_no_arg_)
@@ -67,7 +81,7 @@ module FullSearch
67
81
  end
68
82
 
69
83
  def highlight(open_tag: "<mark>", close_tag: "</mark>")
70
- @highlight_config = { open_tag: open_tag, close_tag: close_tag }
84
+ @highlight_config = {open_tag: open_tag, close_tag: close_tag}
71
85
  end
72
86
 
73
87
  def typo_tolerance(enabled = true, min_term_length: nil)
@@ -91,8 +105,8 @@ module FullSearch
91
105
  soft_delete_column,
92
106
  typo_tolerance?,
93
107
  typo_tolerance_min_term_length,
94
- fields.map { |f| [f.name, f.weight, f.source.nil? ? "column" : f.source.source_location&.join(":"), f.reindex_on, f.async, f.as] },
95
- exact_matches.map { |e| [e.name, e.source&.source_location&.join(":")] },
108
+ fields.map { |f| [f.name, f.weight, f.source.nil? ? "column" : "proc:#{f.version}", f.reindex_on, f.async, f.as] },
109
+ exact_matches.map { |e| [e.name, "proc:#{e.version}"] },
96
110
  filters.map { |f| [f.name, f.required] },
97
111
  rank_bys.map { |r| [r.column, r.direction] }
98
112
  ].inspect)
@@ -3,8 +3,10 @@
3
3
  module FullSearch
4
4
  class Error < StandardError; end
5
5
  class MissingRequiredFilterError < Error; end
6
+ class UnknownFilterError < Error; end
6
7
  class ConfigChangedError < Error; end
7
8
  class InvalidFieldError < Error; end
8
9
  class NotConfiguredError < Error; end
9
10
  class UnsupportedDatabaseError < Error; end
11
+ class InvalidQueryError < Error; end
10
12
  end
@@ -12,20 +12,14 @@ module FullSearch
12
12
  relation = model.all
13
13
  filters.each { |name, value| relation = relation.where(name => value) }
14
14
 
15
- exact_ids = dsl.exact_matches.flat_map do |em|
16
- value = exact_match_value(em, cleaned)
17
- next [] if value.nil? || value.to_s.empty?
18
-
19
- relation.where(em.name => value).pluck(:id)
15
+ ids = []
16
+ relation.find_each do |record|
17
+ dsl.exact_matches.each do |em|
18
+ value = record.instance_exec(&em.source)
19
+ ids << record.id if value.to_s.casecmp?(cleaned)
20
+ end
20
21
  end
21
-
22
- exact_ids.uniq
23
- end
24
-
25
- def self.exact_match_value(em, query)
26
- fake = Object.new
27
- fake.define_singleton_method(em.name) { query }
28
- fake.instance_exec(&em.source)
22
+ ids.uniq
29
23
  end
30
24
  end
31
25
  end
@@ -2,143 +2,207 @@
2
2
 
3
3
  module FullSearch
4
4
  class Highlighter
5
- def self.apply!(records, model, query)
6
- snippets = build_snippets(model, query)
7
- if snippets.values.all?(&:nil?) && records.any?
8
- snippets = manual_snippets(records, model, query)
5
+ class << self
6
+ def apply!(records, model, query)
7
+ snippets = build_snippets(model, query)
8
+ if snippets.values.all?(&:nil?) && records.any?
9
+ snippets = manual_snippets(records, model, query)
10
+ end
11
+ records.each { |record| record.full_search_snippet = snippets[record.id] }
12
+ records
9
13
  end
10
- records.each { |record| record.full_search_snippet = snippets[record.id] }
11
- records
12
- end
13
14
 
14
- def self.apply_fields!(records, model, query)
15
- fields = build_field_snippets(model, query)
16
- if fields.empty? && records.any?
17
- fields = manual_field_snippets(records, model, query)
18
- end
19
- exact_fields = exact_match_field_snippets(records, model, query)
20
- records.each do |record|
21
- merged = fields[record.id] || {}
22
- exact_fields.fetch(record.id, {}).each { |k, v| merged[k] ||= v }
23
- record.full_search_highlight_fields = merged
15
+ def apply_fields!(records, model, query)
16
+ fields = build_field_snippets(model, query)
17
+ if fields.values.all?(&:empty?) && records.any?
18
+ fields = manual_field_snippets(records, model, query)
19
+ end
20
+ exact_fields = exact_match_field_snippets(records, model, query)
21
+ records.each do |record|
22
+ merged = fields[record.id] || {}
23
+ exact_fields.fetch(record.id, {}).each { |k, v| merged[k] ||= v }
24
+ record.full_search_highlight_fields = merged
25
+ end
26
+ records
24
27
  end
25
- records
26
- end
27
28
 
28
- private
29
+ private
29
30
 
30
- def self.build_snippets(model, query)
31
- rows = highlight_rows(model, query)
32
- cols = model.full_search_dsl.fields.map(&:name)
31
+ def build_snippets(model, query)
32
+ rows = highlight_rows(model, query)
33
+ cols = model.full_search_dsl.fields.map(&:name)
33
34
 
34
- rows.to_h do |row|
35
- text = cols.map { |col| row["#{col}_snippet"] }.compact.join(" ").strip
36
- [row["rowid"], text.presence]
35
+ rows.to_h do |row|
36
+ text = cols.map { |col| row["#{col}_snippet"] }.compact.join(" ").strip
37
+ [row["rowid"], text.presence]
38
+ end
37
39
  end
38
- end
39
40
 
40
- def self.build_field_snippets(model, query)
41
- rows = highlight_rows(model, query)
42
- dsl = model.full_search_dsl
43
- fields = dsl.fields
44
- open_tag = (dsl.highlight_config || { open_tag: "<mark>" })[:open_tag]
45
-
46
- rows.to_h do |row|
47
- snippets = fields.each_with_object({}) do |field, hash|
48
- snippet = row["#{field.name}_snippet"].to_s.strip
49
- key = field.as || field.name
50
- hash[key] = snippet if snippet.include?(open_tag)
41
+ def build_field_snippets(model, query)
42
+ rows = highlight_rows(model, query)
43
+ dsl = model.full_search_dsl
44
+ fields = dsl.fields
45
+ open_tag = (dsl.highlight_config || {open_tag: "<mark>"})[:open_tag]
46
+
47
+ rows.to_h do |row|
48
+ snippets = fields.each_with_object({}) do |field, hash|
49
+ snippet = row["#{field.name}_snippet"].to_s.strip
50
+ key = field.as || field.name
51
+ hash[key] = snippet if snippet.include?(open_tag)
52
+ end
53
+ [row["rowid"], snippets]
51
54
  end
52
- [row["rowid"], snippets]
53
55
  end
54
- end
55
56
 
56
- def self.manual_snippets(records, model, query)
57
- dsl = model.full_search_dsl
58
- config = dsl.highlight_config || { open_tag: "<mark>", close_tag: "</mark>" }
57
+ def compute_max_typos(query, dsl)
58
+ return nil unless dsl.typo_tolerance?
59
+ length = query.length
60
+ min_length = dsl.typo_tolerance_min_term_length.to_i
61
+ return nil if length < min_length
62
+ return 2 if length >= 9
63
+ 1
64
+ end
65
+
66
+ def manual_snippets(records, model, query)
67
+ dsl = model.full_search_dsl
68
+ config = dsl.highlight_config || {open_tag: "<mark>", close_tag: "</mark>"}
59
69
  cols = dsl.fields.map(&:name)
70
+ max_typos = compute_max_typos(query, dsl)
60
71
 
61
72
  records.to_h do |record|
62
73
  text = cols.map { |col| record.full_search_text_for(col).to_s }.join(" ").strip
63
- highlighted = manual_highlight(text, query, config)
74
+ highlighted = manual_highlight(text, query, config, max_typos: max_typos)
64
75
  [record.id, highlighted.presence]
65
76
  end
66
- end
77
+ end
78
+
79
+ def manual_field_snippets(records, model, query)
80
+ dsl = model.full_search_dsl
81
+ config = dsl.highlight_config || {open_tag: "<mark>", close_tag: "</mark>"}
82
+ fields = dsl.fields
83
+ max_typos = compute_max_typos(query, dsl)
67
84
 
68
- def self.manual_field_snippets(records, model, query)
69
- dsl = model.full_search_dsl
70
- config = dsl.highlight_config || { open_tag: "<mark>", close_tag: "</mark>" }
71
- fields = dsl.fields
72
-
73
- records.to_h do |record|
74
- snippets = fields.each_with_object({}) do |field, hash|
75
- value = record.full_search_text_for(field.name).to_s
76
- highlighted = manual_highlight(value, query, config)
77
- key = field.as || field.name
78
- hash[key] = highlighted if highlighted.include?(config[:open_tag])
85
+ records.to_h do |record|
86
+ snippets = fields.each_with_object({}) do |field, hash|
87
+ value = record.full_search_text_for(field.name).to_s
88
+ highlighted = manual_highlight(value, query, config, max_typos: max_typos)
89
+ key = field.as || field.name
90
+ hash[key] = highlighted if highlighted.include?(config[:open_tag])
91
+ end
92
+ [record.id, snippets]
79
93
  end
80
- [record.id, snippets]
81
94
  end
82
- end
83
95
 
84
- def self.exact_match_field_snippets(records, model, query)
85
- dsl = model.full_search_dsl
86
- return {} if dsl.exact_matches.empty?
96
+ def exact_match_field_snippets(records, model, query)
97
+ dsl = model.full_search_dsl
98
+ return {} if dsl.exact_matches.empty?
87
99
 
88
- config = dsl.highlight_config || { open_tag: "<mark>", close_tag: "</mark>" }
100
+ config = dsl.highlight_config || {open_tag: "<mark>", close_tag: "</mark>"}
101
+ max_typos = compute_max_typos(query, dsl)
89
102
 
90
- records.to_h do |record|
91
- snippets = dsl.exact_matches.each_with_object({}) do |em, hash|
92
- value = exact_match_field_value(em, record)
93
- highlighted = manual_highlight(value.to_s, query, config)
94
- hash[em.name.to_s] = highlighted if highlighted.include?(config[:open_tag])
103
+ records.to_h do |record|
104
+ snippets = dsl.exact_matches.each_with_object({}) do |em, hash|
105
+ value = exact_match_field_value(em, record)
106
+ highlighted = manual_highlight(value.to_s, query, config, max_typos: max_typos)
107
+ hash[em.name.to_s] = highlighted if highlighted.include?(config[:open_tag])
108
+ end
109
+ [record.id, snippets]
95
110
  end
96
- [record.id, snippets]
97
111
  end
98
- end
99
112
 
100
- def self.exact_match_field_value(em, record)
101
- if em.source
102
- record.instance_exec(&em.source)
103
- else
104
- record.public_send(em.name)
113
+ def exact_match_field_value(em, record)
114
+ if em.source
115
+ record.instance_exec(&em.source)
116
+ else
117
+ record.public_send(em.name)
118
+ end
105
119
  end
106
- end
107
120
 
108
- def self.manual_highlight(text, query, config)
109
- return text if text.empty? || query.empty?
110
- text.gsub(/#{Regexp.escape(query)}/i, "#{config[:open_tag]}\\0#{config[:close_tag]}")
111
- end
121
+ def manual_highlight(text, query, config, max_typos: nil)
122
+ return text if text.empty? || query.empty?
123
+
124
+ open_tag = config[:open_tag]
125
+ close_tag = config[:close_tag]
126
+ escaped_query = Regexp.escape(query)
112
127
 
113
- def self.highlight_rows(model, query)
114
- dsl = model.full_search_dsl
115
- config = dsl.highlight_config || { open_tag: "<mark>", close_tag: "</mark>" }
116
- match_expr = QueryParser.to_match_expression(QueryParser.parse(query))
117
- return [] if match_expr.empty?
128
+ if text.match?(/#{escaped_query}/i)
129
+ return text.gsub(/#{escaped_query}/i, "#{open_tag}\\0#{close_tag}")
130
+ end
131
+
132
+ best = best_fuzzy_match(text, query, max_typos: max_typos)
133
+ if best
134
+ start_pos, end_pos = best
135
+ return "#{text[0...start_pos]}#{open_tag}#{text[start_pos...end_pos]}#{close_tag}#{text[end_pos..]}"
136
+ end
118
137
 
119
- table = FullSearch::Index.fts_table_name(model)
120
- cols = dsl.fields.map(&:name)
121
- return [] if cols.empty?
138
+ text
139
+ end
122
140
 
123
- highlight_parts = cols.each_with_index.map do |col, idx|
124
- "highlight(#{table}, #{idx}, #{quote(config[:open_tag])}, #{quote(config[:close_tag])}) AS #{col}_snippet"
125
- end.join(", ")
141
+ def best_fuzzy_match(text, query, max_typos: nil)
142
+ query_len = query.length
143
+ text_len = text.length
144
+ return nil if query_len == 0 || text_len == 0
126
145
 
127
- sql = <<~SQL
128
- SELECT rowid, #{highlight_parts}
129
- FROM #{table}
130
- WHERE #{table} MATCH #{quote(match_expr)}
131
- SQL
146
+ max_typos ||= max_allowed_typos(query_len)
147
+ return nil if max_typos < 0
132
148
 
133
- connection.execute(sql)
134
- end
149
+ best_score = max_typos + 1
150
+ best_range = nil
135
151
 
136
- def self.connection
137
- ActiveRecord::Base.connection
138
- end
152
+ min_window = [1, query_len - max_typos].max
153
+ max_window = [text_len, query_len + max_typos].min
154
+ return nil if min_window > max_window
155
+
156
+ (min_window..max_window).each do |window_len|
157
+ (0..(text_len - window_len)).each do |start|
158
+ substr = text[start, window_len]
159
+ distance = Distance.damerau_levenshtein(query.downcase, substr.downcase)
160
+ next if distance > max_typos
161
+
162
+ if best_range.nil? || distance < best_score || (distance == best_score && window_len > (best_range[1] - best_range[0]))
163
+ best_score = distance
164
+ best_range = [start, start + window_len]
165
+ end
166
+ end
167
+ end
168
+
169
+ best_range
170
+ end
139
171
 
140
- def self.quote(value)
141
- connection.quote(value)
172
+ def max_allowed_typos(length)
173
+ return -1 if length < 3
174
+ return 2 if length >= 9
175
+ 1
176
+ end
177
+
178
+ def highlight_rows(model, query)
179
+ dsl = model.full_search_dsl
180
+ config = dsl.highlight_config || {open_tag: "<mark>", close_tag: "</mark>"}
181
+ match_expr = QueryParser.to_match_expression(QueryParser.parse(query))
182
+ return [] if match_expr.empty?
183
+
184
+ table = qt(FullSearch::Index.fts_table_name(model))
185
+ cols = dsl.fields.map(&:name)
186
+ return [] if cols.empty?
187
+
188
+ highlight_parts = cols.each_with_index.map do |col, idx|
189
+ "highlight(#{table}, #{idx}, #{q(config[:open_tag])}, #{q(config[:close_tag])}) AS #{qc("#{col}_snippet")}"
190
+ end.join(", ")
191
+
192
+ sql = <<~SQL
193
+ SELECT rowid, #{highlight_parts}
194
+ FROM #{table}
195
+ WHERE #{table} MATCH #{q(match_expr)}
196
+ SQL
197
+
198
+ connection.execute(sql)
199
+ end
200
+
201
+ include Quoting
202
+
203
+ def connection
204
+ ActiveRecord::Base.connection
205
+ end
142
206
  end
143
207
  end
144
208
  end