full_search 0.1.2 → 0.2.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 025c498951cfbd1e531bb07c3994af9b195bf0411bfe65b65ff96aacedbc157a
-  data.tar.gz: f1c0d8eb49892d94f567ab162e8c94576b8f574848077d95d5942236d51b215a
+  metadata.gz: 339422f4765fd1cd96ef55e9d026d782e10b08c2c6d1636d6e2e13454cf12b51
+  data.tar.gz: fc5b3a6105d9359162aebb6d14dddf6763e0e0e301351157a1c1c278386d3355
 SHA512:
-  metadata.gz: d3ea481bc2b665046432f415a6e892b4735b1aedee46716ac271b51cb873c490b002280000b02a92ba3369b8e3c6a27b98d8da18b876b4269a6a516cc51262b4
-  data.tar.gz: 93e2ed7136f1b20d7bb7025891d29c015b61db0d427c527eb4345e6bfd94ec0968562aba8528590737d89cbad2988c4d029ad114f3f18e6467d5a19f3ff8c4b2
+  metadata.gz: 38a981537301ec24ec8ee6d38d1174e771b5dbdb3ebc4fa3e0bea857c60d3a440156e5cf9b1ae43be068c616f65381f4866d6294aa9aa4dab5a3e662cb3bfe44
+  data.tar.gz: 1c1af3ef0948200256f76013f66da00e1764379dbbc07ac8d90264e1a55b9e416218e8569675179736b59821b288aa9ce6fe24fccea48079f60847f127129451

data/README.md

@@ -2,6 +2,10 @@
 
 SQLite FTS5 full-text search for Rails/ActiveRecord. A lightweight, self-contained alternative to `pg_search` for apps already running on SQLite.
 
+## When to use
+
+`full_search` is designed for apps with **100,000 records or fewer per table** that want full-text search without running a separate service. If you're on SQLite and need keyword search, phrase matching, typo-tolerant substring queries, or result highlighting, this gem gives you production-quality search with zero infrastructure — no Elasticsearch, no Meilisearch, no Sidekiq queue.
+
 ## Installation
 
 Add to your Gemfile:
@@ -30,12 +34,12 @@ end
 ```
 
 ```ruby
-Customer.full_search("sam", filters: { account_id: 1 }).page(params[:page])
+Customer.search("sam", filters: { account_id: 1 }).page(params[:page])
 ```
 
 ## Features
 
-- Declarative `full_search` DSL
+- Declarative `full_search` DSL with `search` / `full_search` query methods
 - SQLite FTS5 backed
 - Required-filter enforcement for multi-tenant apps
 - Exact-match queries for encrypted identifiers
@@ -44,14 +48,98 @@ Customer.full_search("sam", filters: { account_id: 1 }).page(params[:page])
 - Phrase, exclusion, and OR query operators
 - FTS5 `highlight()` support for result snippets
 - Opt-in trigram typo/substring fallback (requires SQLite >= 3.34)
-- Rake tasks: `full_search:rebuild`, `full_search:optimize`, `full_search:status`
+
+### Per-model operations
+
+Once a model declares `full_search`, you can call these class methods:
+
+| Method | What it does |
+|--------|-------------|
+| `Customer.rebuild!` | Force-drop and recreate the model's FTS table, backfill from the source table, and reinstall triggers. |
+| `Customer.reindex!` | Re-evaluates computed `source:` fields for every record and updates the FTS table in-place. Table structure is untouched. |
+| `Customer.optimize!` | Runs FTS5 `optimize` on the model's index to merge b-tree segments. |
+
+## Index management
+
+FTS indexes are SQLite virtual tables (`customers_fts`, `vehicles_fts`, etc.) that mirror your model tables. They stay in sync via database triggers on `INSERT` / `UPDATE` / `DELETE`.
+
+### Rebuild vs reindex
+
+These two operations are often confused:
+
+- **Rebuild** (`full_search:rebuild` / `Customer.rebuild!`) — drops and recreates the FTS virtual table. Needed when the DSL changes (fields added/removed, tokenizer changed, etc.). The table is re-created from scratch, backfilled, triggers re-installed, and the index is optimized.
+- **Reindex** (`Customer.reindex!` / `FullSearch::Index.reindex_source_fields!`) — updates existing FTS rows with fresh values from computed `source:` fields only. The table structure is untouched. Database triggers cover regular column changes automatically; only Ruby-evaluated `source:` blocks need an explicit reindex.
+
+### Auto-rebuild on app load
+
+When `auto_rebuild_schema` is enabled (default in the generated initializer), the railtie hooks into Rails `after_initialize` and:
+
+1. Creates missing FTS tables for every model using `full_search`
+2. Compares each table's stored config hash against the current DSL
+3. Rebuilds the index automatically when the DSL changes (new fields, different tokenizer, etc.)
+
+```ruby
+# config/initializers/full_search.rb
+FullSearch.configure do |config|
+  config.auto_rebuild_schema = true
+end
+```
+
+### In production
+
+Auto-rebuild runs on every Rails process boot (web, worker, console). For zero-downtime deploys where old processes still serve traffic, or if you prefer explicit control, set `auto_rebuild_schema` to `false` and run the rebuild task manually:
+
+```bash
+# Rebuild only indexes whose DSL has changed (fast, safe for production)
+bin/rails full_search:rebuild
+
+# Target specific models by table name
+bin/rails 'full_search:rebuild[customers,vehicles]'
+```
+
+The rebuild task checks each model's stored config hash against the current DSL and only drops/recreates indexes when they differ. For a full forced reset (drops and recreates all FTS tables regardless):
+
+```bash
+bin/rails full_search:reset
+```
+
+### Rake tasks
+
+| Task | Description |
+|------|-------------|
+| `full_search:rebuild` | Drops and recreates the FTS virtual table only when the DSL config hash has changed (safe for production). Pass model names to target specific tables. |
+| `full_search:reset` | Force a full rebuild — drops and recreates all FTS tables regardless of config hash. Use when data may be out of sync. |
+| `full_search:optimize` | Run FTS5 [`optimize`](https://www.sqlite.org/fts5.html#the_optimize_command) to merge b-tree segments. Useful after bulk updates. |
+| `full_search:status` | Show each model's index status (`ok` / `stale`) and count of empty sourced fields. |
+
+## Background jobs
+
+### Scheduled optimization
+
+FTS5 b-tree segments accumulate over time as rows are inserted, updated, and deleted. Running `optimize` periodically merges these segments, keeping queries fast. For most apps, once a day during a low-traffic window is sufficient. Apps with heavy write volume may benefit from every few hours.
+
+The gem ships `FullSearch::OptimizeJob` ready to use. Schedule it with Solid Queue's `recurring.yml`:
+
+```yaml
+# config/recurring.yml
+full_search_optimize:
+  class: FullSearch::OptimizeJob
+  schedule: daily at 4am
+  description: "Merge FTS5 b-tree segments for full_search indexes"
+```
+
+If you're not on Solid Queue, most job frameworks support recurring schedules via gems like `sidekiq-cron` or `whenever`.
+
+### Config hash drift detection
+
+Every FTS table stores a SHA256 digest of its DSL configuration in the `full_search_index_versions` table. When the DSL changes (e.g., adding a field), the stored hash no longer matches, and `rebuild!` is triggered automatically (if `auto_rebuild_schema` is true) to bring the index in line with the new definition. If `auto_rebuild_schema` is false, queries still run but raise `ConfigChangedError` when the hash doesn't match.
 
 ## Query operators
 
 ```ruby
-Customer.full_search('"Sam Smith"', filters: { account_id: 1 })      # phrase
-Customer.full_search("honda -civic", filters: { account_id: 1 })     # exclusion
-Customer.full_search("honda OR toyota", filters: { account_id: 1 })  # OR
+Customer.search('"Sam Smith"', filters: { account_id: 1 })      # phrase
+Customer.search("honda -civic", filters: { account_id: 1 })     # exclusion
+Customer.search("honda OR toyota", filters: { account_id: 1 })  # OR
 ```
 
 ## Highlighting
@@ -66,7 +154,7 @@ end
 ```
 
 ```ruby
-Customer.full_search("sam", filters: { account_id: 1 }, highlight: true)
+Customer.search("sam", filters: { account_id: 1 }, highlight: true)
 # each result has #full_search_snippet
 ```
 

data/lib/full_search/config.rb

@@ -2,15 +2,16 @@
 
 module FullSearch
   class Config
-    attr_accessor :auto_manage_schema, :stale_query_behavior, :lock_rebuilds, :default_async_reindex, :default_tokenizer
+    attr_accessor :auto_rebuild_schema, :stale_query_behavior, :lock_rebuilds, :default_async_reindex, :default_tokenizer
 
     def initialize
-      @auto_manage_schema = false
+      @auto_rebuild_schema = false
       @stale_query_behavior = :raise
       @lock_rebuilds = true
       @default_async_reindex = true
       @default_tokenizer = "unicode61"
     end
+
   end
 
   class << self

data/lib/full_search/dsl.rb

@@ -4,7 +4,7 @@ module FullSearch
   class Dsl
     attr_reader :fields, :exact_matches, :filters, :model_class, :tokenize, :highlight_config, :rank_bys
 
-    Field = Data.define(:name, :weight, :source, :reindex_on, :async)
+    Field = Data.define(:name, :weight, :source, :reindex_on, :async, :as)
     ExactMatch = Data.define(:name, :source)
     Filter = Data.define(:name, :required)
     RankBy = Data.define(:column, :direction)
@@ -19,11 +19,14 @@ module FullSearch
       @soft_delete_column = nil
     end
 
-    def field(name, weight: 1, source: nil, reindex_on: nil, async: FullSearch.config.default_async_reindex)
+    def field(name, weight: 1, source: nil, reindex_on: nil, async: FullSearch.config.default_async_reindex, as: nil)
       unless valid_name?(name)
         raise InvalidFieldError, "Invalid field name: #{name.inspect}"
       end
-      @fields << Field.new(name: name.to_s, weight: weight.to_i, source: source, reindex_on: reindex_on&.to_s, async: async)
+      if as && !valid_name?(as)
+        raise InvalidFieldError, "Invalid field alias (as): #{as.inspect}"
+      end
+      @fields << Field.new(name: name.to_s, weight: weight.to_i, source: source, reindex_on: reindex_on&.to_s, async: async, as: as&.to_s)
     end
 
     def exact_match(name, source: -> { public_send(name) })
@@ -88,8 +91,8 @@ module FullSearch
         soft_delete_column,
         typo_tolerance?,
         typo_tolerance_min_term_length,
-        fields.map { |f| [f.name, f.weight, f.source.nil? ? "column" : "source", f.reindex_on, f.async] },
-        exact_matches.map { |e| [e.name] },
+        fields.map { |f| [f.name, f.weight, f.source.nil? ? "column" : f.source.source_location&.join(":"), f.reindex_on, f.async, f.as] },
+        exact_matches.map { |e| [e.name, e.source&.source_location&.join(":")] },
         filters.map { |f| [f.name, f.required] },
         rank_bys.map { |r| [r.column, r.direction] }
       ].inspect)

data/lib/full_search/exact_match.rb

@@ -12,11 +12,20 @@ module FullSearch
       relation = model.all
       filters.each { |name, value| relation = relation.where(name => value) }
 
-      conditions = dsl.exact_matches.map do |em|
-        relation.model.arel_table[em.name].eq(cleaned)
+      exact_ids = dsl.exact_matches.flat_map do |em|
+        value = exact_match_value(em, cleaned)
+        next [] if value.nil? || value.to_s.empty?
+
+        relation.where(em.name => value).pluck(:id)
       end
 
-      relation.where(conditions.reduce(:or)).pluck(:id)
+      exact_ids.uniq
+    end
+
+    def self.exact_match_value(em, query)
+      fake = Object.new
+      fake.define_singleton_method(em.name) { query }
+      fake.instance_exec(&em.source)
     end
   end
 end

data/lib/full_search/highlighter.rb

@@ -4,13 +4,24 @@ module FullSearch
   class Highlighter
     def self.apply!(records, model, query)
       snippets = build_snippets(model, query)
+      if snippets.values.all?(&:nil?) && records.any?
+        snippets = manual_snippets(records, model, query)
+      end
       records.each { |record| record.full_search_snippet = snippets[record.id] }
       records
     end
 
     def self.apply_fields!(records, model, query)
       fields = build_field_snippets(model, query)
-      records.each { |record| record.full_search_highlight_fields = fields[record.id] || {} }
+      if fields.empty? && records.any?
+        fields = manual_field_snippets(records, model, query)
+      end
+      exact_fields = exact_match_field_snippets(records, model, query)
+      records.each do |record|
+        merged = fields[record.id] || {}
+        exact_fields.fetch(record.id, {}).each { |k, v| merged[k] ||= v }
+        record.full_search_highlight_fields = merged
+      end
       records
     end
 
@@ -29,18 +40,76 @@ module FullSearch
     def self.build_field_snippets(model, query)
       rows = highlight_rows(model, query)
       dsl = model.full_search_dsl
-      cols = dsl.fields.map(&:name)
+      fields = dsl.fields
       open_tag = (dsl.highlight_config || { open_tag: "<mark>" })[:open_tag]
 
       rows.to_h do |row|
-        snippets = cols.each_with_object({}) do |col, hash|
-          snippet = row["#{col}_snippet"].to_s.strip
-          hash[col.to_s] = snippet if snippet.include?(open_tag)
+        snippets = fields.each_with_object({}) do |field, hash|
+          snippet = row["#{field.name}_snippet"].to_s.strip
+          key = field.as || field.name
+          hash[key] = snippet if snippet.include?(open_tag)
         end
         [row["rowid"], snippets]
       end
     end
 
+    def self.manual_snippets(records, model, query)
+      dsl = model.full_search_dsl
+      config = dsl.highlight_config || { open_tag: "<mark>", close_tag: "</mark>" }
+        cols = dsl.fields.map(&:name)
+
+        records.to_h do |record|
+          text = cols.map { |col| record.full_search_text_for(col).to_s }.join(" ").strip
+          highlighted = manual_highlight(text, query, config)
+          [record.id, highlighted.presence]
+        end
+    end
+
+    def self.manual_field_snippets(records, model, query)
+      dsl = model.full_search_dsl
+      config = dsl.highlight_config || { open_tag: "<mark>", close_tag: "</mark>" }
+      fields = dsl.fields
+
+      records.to_h do |record|
+        snippets = fields.each_with_object({}) do |field, hash|
+          value = record.full_search_text_for(field.name).to_s
+          highlighted = manual_highlight(value, query, config)
+          key = field.as || field.name
+          hash[key] = highlighted if highlighted.include?(config[:open_tag])
+        end
+        [record.id, snippets]
+      end
+    end
+
+    def self.exact_match_field_snippets(records, model, query)
+      dsl = model.full_search_dsl
+      return {} if dsl.exact_matches.empty?
+
+      config = dsl.highlight_config || { open_tag: "<mark>", close_tag: "</mark>" }
+
+      records.to_h do |record|
+        snippets = dsl.exact_matches.each_with_object({}) do |em, hash|
+          value = exact_match_field_value(em, record)
+          highlighted = manual_highlight(value.to_s, query, config)
+          hash[em.name.to_s] = highlighted if highlighted.include?(config[:open_tag])
+        end
+        [record.id, snippets]
+      end
+    end
+
+    def self.exact_match_field_value(em, record)
+      if em.source
+        record.instance_exec(&em.source)
+      else
+        record.public_send(em.name)
+      end
+    end
+
+    def self.manual_highlight(text, query, config)
+      return text if text.empty? || query.empty?
+      text.gsub(/#{Regexp.escape(query)}/i, "#{config[:open_tag]}\\0#{config[:close_tag]}")
+    end
+
     def self.highlight_rows(model, query)
       dsl = model.full_search_dsl
       config = dsl.highlight_config || { open_tag: "<mark>", close_tag: "</mark>" }

data/lib/full_search/index.rb

@@ -14,17 +14,26 @@ module FullSearch
 
         create_metadata_table!
 
+        table_was_created = false
         unless table_exists?(model)
           conn.execute(create_virtual_table_sql(model))
+          table_was_created = true
         end
 
         if dsl.typo_tolerance? && !trigram_table_exists?(model)
           FullSearch::Typo.warn_unsupported! unless FullSearch::Typo.supported?
           conn.execute(create_trigram_virtual_table_sql(model))
+          table_was_created = true
+        end
+
+        if table_was_created
+          conn.execute(backfill_sql(model))
+          conn.execute(backfill_trigram_sql(model)) if dsl.typo_tolerance?
+          reindex_source_fields!(model) if dsl.fields.any?(&:source)
+          store_config_hash!(model)
         end
 
         ensure_triggers!(model) if model_table_exists?(model)
-        store_config_hash!(model)
       end
 
       def rebuild!(model)
@@ -47,16 +56,41 @@ module FullSearch
           end
           conn.execute(backfill_sql(model))
           conn.execute(backfill_trigram_sql(model)) if dsl.typo_tolerance?
+          reindex_source_fields!(model) if dsl.fields.any?(&:source)
           create_triggers!(model)
           optimize!(model)
           store_config_hash!(model, rebuilt_at: Time.current)
         end
       end
 
+      def rebuild_if_needed!(model)
+        return false unless FullSearch::Index.sqlite?
+
+        dsl = model.full_search_dsl
+        return false unless dsl
+
+        create_metadata_table!
+        ensure_table!(model)
+
+        stored = stored_config_hash(model)
+        if stored && stored == dsl.config_hash
+          false
+        else
+          rebuild!(model)
+          true
+        end
+      end
+
       def optimize!(model)
         connection.execute("INSERT INTO #{fts_table_name(model)}(#{fts_table_name(model)}) VALUES('optimize');")
       end
 
+      def reindex_source_fields!(model)
+        model.find_each do |record|
+          FullSearch::Callbacks.reindex_record!(record)
+        end
+      end
+
       def drop!(model)
         drop_triggers!(model)
         connection.execute("DROP TABLE IF EXISTS #{fts_table_name(model)};")

data/lib/full_search/model.rb

@@ -5,7 +5,7 @@ module FullSearch
     extend ActiveSupport::Concern
 
     class_methods do
-      def full_search(query_or_options = nil, filters: {}, include_soft_deleted: false, limit: nil, offset: nil, highlight: false, highlight_fields: false, &block)
+      def full_search(query_or_options = nil, filters: {}, include_soft_deleted: false, limit: nil, offset: nil, highlight: false, highlight_fields: false, matching_strategy: nil, &block)
         if block_given? || query_or_options.is_a?(Hash)
           @full_search_dsl ||= FullSearch::Dsl.new(self)
           @full_search_dsl.tokenize(query_or_options[:tokenize]) if query_or_options.is_a?(Hash) && query_or_options.key?(:tokenize)
@@ -16,7 +16,7 @@ module FullSearch
           FullSearch.models << self unless FullSearch.models.include?(self)
           @full_search_dsl
         else
-          FullSearch::Search.new(self, query_or_options, filters: filters, include_soft_deleted: include_soft_deleted, limit: limit, offset: offset, highlight: highlight, highlight_fields: highlight_fields).relation
+          FullSearch::Search.new(self, query_or_options, filters: filters, include_soft_deleted: include_soft_deleted, limit: limit, offset: offset, highlight: highlight, highlight_fields: highlight_fields, matching_strategy: matching_strategy).relation
         end
       end
 
@@ -27,6 +27,24 @@ module FullSearch
       def full_search_ids(query, filters: {}, include_soft_deleted: false, limit: 1000)
         full_search(query, filters: filters, include_soft_deleted: include_soft_deleted, limit: limit).pluck(:id)
       end
+
+      def rebuild!
+        FullSearch::Index.rebuild!(self)
+      end
+
+      def optimize!
+        FullSearch::Index.optimize!(self)
+      end
+
+      def reindex!
+        FullSearch::Index.reindex_source_fields!(self)
+      end
+    end
+
+    included do |base|
+      unless base.respond_to?(:search)
+        base.singleton_class.alias_method :search, :full_search
+      end
     end
 
     module InstanceMethods
@@ -34,7 +52,7 @@ module FullSearch
 
       def full_search_text_for(field_name)
         dsl = self.class.full_search_dsl
-        field = dsl.fields.find { |f| f.name == field_name.to_s }
+        field = dsl.fields.find { |f| f.name == field_name.to_s || f.as == field_name.to_s }
         return nil unless field
 
         field.source ? instance_exec(&field.source) : public_send(field.name)

data/lib/full_search/optimize_job.rb

@@ -0,0 +1,13 @@
+# frozen_string_literal: true
+
+require "active_job"
+
+module FullSearch
+  class OptimizeJob < ActiveJob::Base
+    queue_as :low
+
+    def perform
+      FullSearch.optimize!
+    end
+  end
+end

data/lib/full_search/railtie.rb

@@ -10,16 +10,10 @@ module FullSearch
 
     initializer "full_search.ensure_tables" do
       config.after_initialize do
-        next unless FullSearch.config.auto_manage_schema
+        next unless FullSearch.config.auto_rebuild_schema
         ActiveSupport.on_load(:active_record) do
           FullSearch.models.each do |model|
-            FullSearch::Index.ensure_table!(model)
-            next unless FullSearch.config.auto_manage_schema == true
-
-            stored = FullSearch::Index.stored_config_hash(model)
-            if stored && stored != model.full_search_dsl.config_hash
-              FullSearch::Index.rebuild!(model)
-            end
+            FullSearch::Index.rebuild_if_needed!(model)
           end
         end
       end

data/lib/full_search/search.rb

@@ -2,9 +2,9 @@
 
 module FullSearch
   class Search
-    attr_reader :model, :query, :filters, :include_soft_deleted, :limit, :offset, :highlight, :highlight_fields
+    attr_reader :model, :query, :filters, :include_soft_deleted, :limit, :offset, :highlight, :highlight_fields, :matching_strategy
 
-    def initialize(model, query, filters:, include_soft_deleted:, limit:, offset:, highlight: false, highlight_fields: false)
+    def initialize(model, query, filters:, include_soft_deleted:, limit:, offset:, highlight: false, highlight_fields: false, matching_strategy: nil)
       @model = model
       @query = query.to_s.strip
       @filters = filters
@@ -13,6 +13,7 @@ module FullSearch
       @offset = offset
       @highlight = highlight
       @highlight_fields = highlight_fields
+      @matching_strategy = matching_strategy
     end
 
     def relation
@@ -21,9 +22,10 @@ module FullSearch
       parsed = QueryParser.parse(query)
       exact_ids = ExactMatch.ids_for(model, query, filters)
       primary_ids = fts_match_ids(parsed)
-      fallback_ids = dsl.typo_tolerance? ? trigram_match_ids(parsed, primary_ids) : []
+      fallback_ids = dsl.typo_tolerance? && matching_strategy != "all" ? trigram_match_ids(parsed, primary_ids) : []
+      fuzzy_ids = dsl.typo_tolerance? && matching_strategy != "all" && primary_ids.empty? && fallback_ids.empty? ? fuzzy_match_ids(parsed) : []
 
-      all_ids = (exact_ids + primary_ids + fallback_ids).uniq
+      all_ids = (exact_ids + primary_ids + fallback_ids + fuzzy_ids).uniq
       return model.none if all_ids.empty?
 
       rel = model.where(id: all_ids)
@@ -105,17 +107,76 @@ module FullSearch
     end
 
     def like_prefix_ids(term)
+      column_fields = dsl.fields.select { |f| f.source.nil? }
+      source_fields = dsl.fields.select { |f| f.source }
+
+      ids = []
+
+      if column_fields.any?
+        like_conditions = column_fields.map do |field|
+          "#{connection.quote_table_name(model.table_name)}.#{connection.quote_column_name(field.name)} LIKE #{connection.quote("#{term}%")}"
+        end.join(" OR ")
+
+        sql = <<~SQL
+          SELECT #{model.table_name}.id
+          FROM #{model.table_name}
+          WHERE (#{like_conditions})
+        SQL
+
+        filter_conditions = filters.map do |name, value|
+          "AND #{model.table_name}.#{name} = #{connection.quote(value)}"
+        end.join(" ")
+
+        ids = connection.execute("#{sql} #{filter_conditions}").map { |r| r["id"] }
+        return ids if ids.any?
+      end
+
+      if source_fields.any?
+        fts_table = FullSearch::Index.fts_table_name(model)
+        like_conditions = source_fields.map do |field|
+          "#{fts_table}.#{field.name} LIKE #{connection.quote("#{term}%")}"
+        end.join(" OR ")
+
+        sql = <<~SQL
+          SELECT #{model.table_name}.id
+          FROM #{fts_table}
+          JOIN #{model.table_name} ON #{model.table_name}.id = #{fts_table}.rowid
+          WHERE (#{like_conditions})
+        SQL
+
+        filter_conditions = filters.map do |name, value|
+          "AND #{fts_table}.#{name} = #{connection.quote(value)}"
+        end.join(" ")
+
+        ids = connection.execute("#{sql} #{filter_conditions}").map { |r| r["id"] }
+      end
+
+      ids
+    end
+
+    def fuzzy_match_ids(parsed)
+      term = parsed.last rescue nil
+      return [] if term.nil?
+
+      term_str = term.is_a?(Array) ? extract_last_term_string(term) : term.to_s
+      return [] if term_str.empty?
+
+      max_typos = max_allowed_typos(term_str.length)
+      return [] if max_typos < 0
+
       column_fields = dsl.fields.select { |f| f.source.nil? }
       return [] if column_fields.empty?
 
-      like_conditions = column_fields.map do |field|
-        "#{connection.quote_table_name(model.table_name)}.#{connection.quote_column_name(field.name)} LIKE #{connection.quote("#{term}%")}"
+      register_levenshtein!
+
+      conditions = column_fields.map do |field|
+        "levenshtein(LOWER(#{connection.quote_table_name(model.table_name)}.#{connection.quote_column_name(field.name)}), #{connection.quote(term_str.downcase)}) <= #{max_typos}"
       end.join(" OR ")
 
       sql = <<~SQL
         SELECT #{model.table_name}.id
         FROM #{model.table_name}
-        WHERE (#{like_conditions})
+        WHERE (#{conditions})
       SQL
 
       filter_conditions = filters.map do |name, value|
@@ -125,6 +186,56 @@ module FullSearch
       connection.execute("#{sql} #{filter_conditions}").map { |r| r["id"] }
     end
 
+    def max_allowed_typos(length)
+      min_length = dsl.typo_tolerance_min_term_length.to_i
+      return -1 if length < min_length
+      return 2 if length >= 9
+      1
+    end
+
+    def extract_last_term_string(terms)
+      last = terms.last
+      last.is_a?(Array) ? last.last.to_s : last.to_s
+    end
+
+    def register_levenshtein!
+      return if @levenshtein_registered
+
+      raw = connection.raw_connection
+      raw.create_function("levenshtein", 2) do |func, s1, s2|
+        func.result = damerau_levenshtein(s1.to_s, s2.to_s)
+      end
+      @levenshtein_registered = true
+    end
+
+    def damerau_levenshtein(a, b)
+      a_len = a.length
+      b_len = b.length
+      return a_len if b_len == 0
+      return b_len if a_len == 0
+
+      d = Array.new(a_len + 1) { Array.new(b_len + 1, 0) }
+      (0..a_len).each { |i| d[i][0] = i }
+      (0..b_len).each { |j| d[0][j] = j }
+
+      (1..a_len).each do |i|
+        (1..b_len).each do |j|
+          cost = a[i - 1] == b[j - 1] ? 0 : 1
+          d[i][j] = [
+            d[i - 1][j] + 1,
+            d[i][j - 1] + 1,
+            d[i - 1][j - 1] + cost
+          ].min
+
+          if i > 1 && j > 1 && a[i - 1] == b[j - 2] && a[i - 2] == b[j - 1]
+            d[i][j] = [d[i][j], d[i - 2][j - 2] + 1].min
+          end
+        end
+      end
+
+      d[a_len][b_len]
+    end
+
     def apply_ranking(rel, all_ids, exact_ids)
       return rel if all_ids.empty?
 

data/lib/full_search/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module FullSearch
-  VERSION = "0.1.2"
+  VERSION = "0.2.0"
 end

data/lib/full_search.rb

@@ -19,6 +19,7 @@ require "full_search/search"
 require "full_search/soft_delete"
 require "full_search/callbacks"
 require "full_search/reindex_job"
+require "full_search/optimize_job"
 require "full_search/test_helpers"
 require "full_search/multi_search"
 
@@ -31,6 +32,10 @@ module FullSearch
     def models
       @models ||= []
     end
+
+    def optimize!
+      models.each { |model| Index.optimize!(model) }
+    end
   end
 end
 

data/lib/generators/full_search/install/templates/full_search.rb

@@ -1,7 +1,7 @@
 # frozen_string_literal: true
 
 FullSearch.configure do |config|
-  config.auto_manage_schema = Rails.env.local?
+  config.auto_rebuild_schema = Rails.env.local?
 
   config.stale_query_behavior = Rails.env.production? ? :log_and_fallback : :raise
 

data/lib/tasks/full_search.rake

@@ -1,30 +1,45 @@
 # frozen_string_literal: true
 
+def resolve_full_search_models(args)
+  if args[:models]
+    args[:models].split(",").map do |name|
+      klass = name.singularize.camelize.constantize rescue nil
+      klass || FullSearch.models.find { |m| m.table_name == name }
+    end.compact
+  else
+    FullSearch.models
+  end
+end
+
 namespace :full_search do
-  desc "Rebuild full_search indexes"
+  desc "Rebuild indexes when DSL has changed (checks config hash; safe for production)"
   task :rebuild, [:models] => :environment do |_t, args|
     Rails.application.eager_load!
 
-    models = if args[:models]
-               args[:models].split(",").map do |name|
-                 klass = name.singularize.camelize.constantize rescue nil
-                 klass || FullSearch.models.find { |m| m.table_name == name }
-               end.compact
-             else
-               FullSearch.models
-             end
+    resolve_full_search_models(args).each do |model|
+      if FullSearch::Index.rebuild_if_needed!(model)
+        puts "Rebuilt #{FullSearch::Index.fts_table_name(model)}"
+      else
+        puts "#{FullSearch::Index.fts_table_name(model)} is current"
+      end
+    end
+  end
 
-    models.each do |model|
+  desc "Force reset indexes (drops and recreates all FTS tables)"
+  task :reset, [:models] => :environment do |_t, args|
+    Rails.application.eager_load!
+
+    resolve_full_search_models(args).each do |model|
       FullSearch::Index.rebuild!(model)
-      puts "Rebuilt #{FullSearch::Index.fts_table_name(model)}"
+      puts "Reset #{FullSearch::Index.fts_table_name(model)}"
     end
   end
 
   desc "Optimize full_search indexes"
   task optimize: :environment do
     Rails.application.eager_load!
+    FullSearch.optimize!
     FullSearch.models.each do |model|
-      FullSearch::Index.optimize!(model)
       puts "Optimized #{FullSearch::Index.fts_table_name(model)}"
     end
   end
@@ -36,7 +51,20 @@ namespace :full_search do
       stored = FullSearch::Index.stored_config_hash(model)
       current = model.full_search_dsl.config_hash
       status = stored == current ? "ok" : "stale"
-      puts "#{model.table_name}: #{status} (stored=#{stored}, current=#{current})"
+      source_fields = model.full_search_dsl.fields.select(&:source).map(&:name)
+
+      drift_info = if source_fields.any?
+        empty_count = ActiveRecord::Base.connection.execute(<<~SQL).first["c"]
+          SELECT COUNT(*) AS c
+          FROM #{FullSearch::Index.fts_table_name(model)}
+          WHERE #{source_fields.map { |c| "#{c} = ''" }.join(' OR ')}
+        SQL
+        " | empty source fields: #{empty_count}"
+      else
+        ""
+      end
+
+      puts "#{model.table_name}: #{status}#{drift_info}"
     end
   end
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: full_search
 version: !ruby/object:Gem::Version
-  version: 0.1.2
+  version: 0.2.0
 platform: ruby
 authors:
 - Ben D'Angelo
@@ -93,6 +93,20 @@ dependencies:
     - - "~>"
       - !ruby/object:Gem::Version
         version: '13.0'
+- !ruby/object:Gem::Dependency
+  name: irb
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
 description: Declarative full-text search for Rails apps backed by SQLite FTS5
 executables: []
 extensions: []
@@ -109,6 +123,7 @@ files:
 - lib/full_search/index.rb
 - lib/full_search/model.rb
 - lib/full_search/multi_search.rb
+- lib/full_search/optimize_job.rb
 - lib/full_search/query_parser.rb
 - lib/full_search/railtie.rb
 - lib/full_search/reindex_job.rb