fuzzy_search 0.3 → 0.4

data/README.md

@@ -2,7 +2,7 @@
 
 Search through your models while tolerating slight mis-spellings. If you have a Person in your database named O'Reilly, you want your users to be able to find it even if they type "OReilly" or "O'Rielly".
 
-This gem is not as powerful as dedicated search tools like Solr, but it's much quicker and easier to set up. It uses your regular database for indexing, rather than an external service that has to be maintained separately.
+This gem is not as powerful as dedicated search tools like Solr, but it's much easier to set up and more appropriate for searching small strings, such names of people or products. It uses your regular database, rather than an external service that has to be maintained separately.
 
 Currently only Rails 2 is supported. I welcome any contributions that resolve this!
 
@@ -10,14 +10,9 @@ Currently only Rails 2 is supported. I welcome any contributions that resolve th
 
 Add `fuzzy_search` to your Rails project's Gemfile, and do the usual `bundle install` dance.
 
-Then, run the generator and migrate to create the search table:
-
-    $ ./script/generate fuzzy_search_setup
-    $ rake db:migrate
-
 ## Example
 
-To allow a model to be searched, specify which columns are to be indexed:
+To allow a model to be searched, specify which columns are to be searched on:
 
 ```ruby
 class Person < ActiveRecord::Base
@@ -26,24 +21,37 @@ class Person < ActiveRecord::Base
     # ...
 end
 ```
-Now, the gem will update the index whenever a Person is saved. To index all the existing records in a model, do this:
+And then create a search table, and the initial trigrams, for that model:
 
-```ruby
-Person.rebuild_fuzzy_search_index!
-```
+    $ ./script/generate fuzzy_search_table Person
+    $ rake db:migrate
 
-The fuzzy_search method returns arrays:
+The fuzzy_search method returns search results:
 
 ```ruby
 people = Person.fuzzy_search "OReilly"
 ```
-
-Fuzzy find works on scopes too, including named_scopes and on-the-fly scopes:
+It works thru scopes too, including named_scopes and on-the-fly scopes:
 
 ```ruby
 people = Person.scoped({:conditions => ["state='active'"]}).fuzzy_search("OReilly")
 ```
 
+If you have a very large data set but are typically searching for items
+within a scoped subset of that data, you can get a significant performance
+boost for those searches by having FuzzySearch include the scope-defining
+field (which currently must be an integer) in the search table:
+
+```ruby
+class Person < ActiveRecord::Base
+    # ...
+    fuzzy_searchable_on :first_name, :last_name, :subset_on => :zipcode
+    # ...
+end
+
+bev_hills_people = Person.fuzzy_search("OReilly", :subset => {:zipcode => 90210})
+```
+
 ## Licence and credits
 
 This gem is based on the rails-fuzzy-search plugin by iulianu

data/TODO

@@ -0,0 +1,37 @@
+* Have the test app's fuzzy search table migrations generated
+automatically by the test suite, instead of having to manually
+update them every time I change the template.
+
+* MyISAM: Consider setting concurrent_insert to 2 on MySQL to make sure we don't
+have nasty locking scenarios when there are holes in the middle
+of the trigrams table. See:
+http://dev.mysql.com/doc/refman/5.0/en/server-system-variables.html#sysvar_concurrent_insert
+
+Or, as an alternate way of doing defragmentation that doesn't block the server
+for ages like OPTIMIZE, perhaps can just rebuild trigrams for the most-recently
+inserted trigram rows until fragmentation no longer in place? In concurrent_insert
+mode 1, this should cause it to reallocate those trigram rows backwards into the hole.
+
+* Watch out for interactions between fuzzy_search_limit and scopes.
+If the first 25 results from the search don't fit the scope, the
+user will end up with an empty result set.
+
+* If it doesn't slow things up too much, prefer short matches when given
+short query strings, i.e. "ama" should rank "Amad" higher than
+"Amalamadingdongwitcherydoo".  Can do this by using the total
+number of trigrams for the given record (or maybe a given word?)
+as a secondary order field.
+
+* Phonetic coding (i.e metaphone). Needs to be all-or-nothing
+for any given AR model, otherwise we'd have to try searching
+on both coded and raw versions of each query string to match
+against both coded and non-coded properties, which would
+throw away the optimization (although it would still allow
+for phonetic near-matches, which is cool).
+
+Maybe can make this a user-adjustable thing by bringing
+back the 'specify a normalize method' feature.
+
+* Maybe allow a [trigram,rec_id] to appear multiple times
+under different subsets? Just have to make sure my scoring
+rule counts only unique trigram hits.

data/gen_fake_data.rb

@@ -0,0 +1,26 @@
+require 'rubygems'
+require 'faker'
+require 'active_record'
+require 'active_support'
+require 'lib/split_trigrams.rb'
+
+$KCODE = 'utf-8'
+
+people_f = open("preloaded_people.csv", "w")
+trigrams_f = open("preloaded_person_fuzzy_search_trigrams.csv", "w")
+
+srand(1234)
+
+1_000_000.times do |i|
+  idx = i+1
+  first_name, last_name = Faker::Name.first_name, Faker::Name.last_name
+  fav_num = rand(100)+1
+  people_f.puts([idx, first_name, last_name, 'Model airplanes', fav_num].join(","))
+
+  trigrams = FuzzySearch::split_trigrams([first_name, last_name])
+  trigrams.each do |tri|
+    trigrams_f.puts([fav_num, tri, idx].join(","))
+  end
+end
+
+[people_f, trigrams_f].each(&:close)

data/lib/fuzzy_model_extensions.rb

@@ -1,111 +1,71 @@
+require 'set'
+
 module FuzzySearch
-  module ModelExtensions
+  module FuzzyModelExtensions
     def self.included(base)
-      base.extend ClassMethods
-
-      base.write_inheritable_attribute :fuzzy_search_properties, []
-      base.class_inheritable_reader :fuzzy_search_properties
+      {
+        :fuzzy_search_properties => [],
+        :fuzzy_search_limit => 25,
+        :fuzzy_search_subset_property => nil
+      }.each do |key, value|
+        base.write_inheritable_attribute key, value
+        base.class_inheritable_reader key
+      end
 
-      base.write_inheritable_attribute :fuzzy_search_threshold, 5
-      base.class_inheritable_reader :fuzzy_search_threshold
+      base.extend ClassMethods
     end
 
     module ClassMethods
       def fuzzy_searchable_on(*properties)
         # TODO: Complain if fuzzy_searchable_on is called more than once
+        # TODO: Complain if no properties were given
+        options = properties.last.is_a?(Hash) ? properties.pop : {}
         write_inheritable_attribute :fuzzy_search_properties, properties
-        has_many :fuzzy_search_trigrams, :as => :rec, :dependent => :destroy
-        after_save :update_fuzzy_search_trigrams!
-        named_scope :fuzzy_search_scope, lambda { |words| generate_fuzzy_search_scope_params(words) }
-        extend WordNormalizerClassMethod unless respond_to? :normalize
-        include InstanceMethods
-      end
-
-      def fuzzy_search(words)
-        # TODO: If fuzzy_search_scope doesn't exist, provide a useful error
-        fuzzy_search_scope(words).all
-      end
-
-      def rebuild_fuzzy_search_index!
-        FuzzySearchTrigram.delete_all(:rec_type => self.class.name)
-        all.each do |rec|
-          rec.update_fuzzy_search_trigrams!
+        if options[:subset_on]
+          write_inheritable_attribute :fuzzy_search_subset_property, options[:subset_on]
+          options.delete(:subset_on)
         end
-      end
-
-      private
 
-      def generate_fuzzy_search_scope_params(words)
-        no_results = {:conditions => "0 = 1"}
-        return no_results unless words != nil
-        words = words.strip.to_s.split(/[\s\-]+/) unless words.instance_of? Array
-        return no_results unless words.size > 0
-
-        trigrams = []
-        words.each do |w|
-          word = ' ' + normalize(w) + ' '
-          word_as_chars = word.mb_chars
-          trigrams << (0..word_as_chars.length-3).collect {|idx| word_as_chars[idx,3].to_s}
+        unless options.empty?
+          # TODO Test me
+          raise "Invalid options: #{options.keys.join(",")}"
         end
-        trigrams = trigrams.flatten.uniq
-
-        # Transform the list of columns in the searchable entity into 
-        # a SQL fragment like:
-        # "table_name.id, table_name.field1, table_name.field2, ..."
-        entity_fields = columns.map {|col| table_name + "." + col.name}.join(", ")
 
-        # The SQL expression for calculating fuzzy_score
-        # Has to be used multiple times because some databases (i.e. Postgres) do not support HAVING on named SELECT fields
-        # TODO: See if we can't get the count(*) out of here, that's a non-trivial operation in some databases
-        fuzzy_score_expr = "(((count(*)*100.0)/#{trigrams.size}) + " +
-          "((count(*)*100.0)/(SELECT count(*) FROM fuzzy_search_trigrams WHERE rec_id = #{table_name}.#{primary_key} AND rec_type = '#{name}')))/2.0"
-
-        # TODO: Optimize this query. In a large trigram table, this is going to go through a lot of dead ends.
-        # Maybe I need to just bite the bullet and learn how to do procedures? That would break cross-database compatibility, though...
-        return {
-          :select => "#{fuzzy_score_expr} AS fuzzy_score, #{entity_fields}",
-          :joins => ["LEFT OUTER JOIN fuzzy_search_trigrams ON fuzzy_search_trigrams.rec_id = #{table_name}.#{primary_key}"],
-          :conditions => ["fuzzy_search_trigrams.token IN (?) AND rec_type = '#{name}'", trigrams],
-          :group => "#{table_name}.#{primary_key}",
-          :order => "fuzzy_score DESC",
-          :having => "#{fuzzy_score_expr} >= #{fuzzy_search_threshold}"
+        named_scope :fuzzy_search_scope, lambda { |words|
+          fuzzy_search_scope_with_opts(words, {})
+        }
+        named_scope :fuzzy_search_scope_with_opts, lambda { |words, opts|
+          self::FuzzySearchTrigram.params_for_search(words, opts)
         }
+        extend FuzzySearchClassMethods
+        include InstanceMethods
+        after_save :update_fuzzy_search_trigrams!
+        after_destroy :delete_fuzzy_search_trigrams!
+
+        const_set(:FuzzySearchTrigram, Class.new(ActiveRecord::Base))
+        self::FuzzySearchTrigram.extend TrigramModelExtensions
+        self::FuzzySearchTrigram.set_target_class self
+        self::FuzzySearchTrigram.table_name = "#{name.underscore}_fuzzy_search_trigrams"
       end
     end
 
-    module WordNormalizerClassMethod
-      def normalize(word)
-        word.mb_chars.normalize(:kd).gsub(/[^\x00-\x7F]/n,'').downcase.to_s
+    module FuzzySearchClassMethods
+      def fuzzy_search(words, opts = {})
+        fuzzy_search_scope_with_opts(words, opts)
+      end
+
+      def rebuild_fuzzy_search_index!
+        self::FuzzySearchTrigram.rebuild_index
       end
     end
 
     module InstanceMethods
       def update_fuzzy_search_trigrams!
-        FuzzySearchTrigram.delete_all(:rec_id => self.id, :rec_type => self.class.name)
-
-        # to avoid double entries
-        tokens = []
-        self.class.fuzzy_search_properties.each do |prop|
-          prop_value = send(prop)
-          next if prop_value.nil?
-          # split the property into words (which are separated by whitespaces)
-          # and generate the trigrams for each word
-          prop_value.to_s.split(/[\s\-]+/).each do |p|
-            # put a space in front and at the end to emphasize the endings
-            word = ' ' + self.class.normalize(p) + ' '
-            word_as_chars = word.mb_chars
-            (0..word_as_chars.length - 3).each do |idx|
-              token = word_as_chars[idx, 3].to_s
-              tokens << token unless tokens.member?(token)
-            end
-          end
-        end
+        self.class::FuzzySearchTrigram.update_trigrams(self)
+      end
 
-        FuzzySearchTrigram.import(
-          [:token, :rec_id, :rec_type],
-          tokens.map{|t| [t, self.id, self.class.name]},
-          :validate => false
-        )
+      def delete_fuzzy_search_trigrams!
+        self.class::FuzzySearchTrigram.delete_trigrams(self)
       end
     end
   end

data/lib/fuzzy_search.rb

@@ -11,7 +11,8 @@ module ActiveRecord # :nodoc:
 end
 
 require 'fuzzy_model_extensions'
-require 'fuzzy_search_trigram'
 require 'fuzzy_search_ver'
+require 'split_trigrams'
+require 'trigram_model_extensions'
 
-ActiveRecord::Base.send(:include, FuzzySearch::ModelExtensions)
+ActiveRecord::Base.send(:include, FuzzySearch::FuzzyModelExtensions)

data/lib/fuzzy_search_ver.rb

@@ -1,3 +1,3 @@
 module FuzzySearch
-  VERSION = "0.3"
+  VERSION = "0.4"
 end

data/lib/split_trigrams.rb

@@ -0,0 +1,16 @@
+module FuzzySearch
+  def self.split_trigrams(s)
+    s = s.join(" ") if s.is_a?(Array)
+    return [] unless s and s.respond_to?(:to_s)
+    words = s.to_s.strip.split(/[\s\-]+/)
+    trigrams = Set.new
+    words.each do |w|
+      chars = w.mb_chars.normalize(:kd).gsub(/[^\x00-\x7F]/n,'').downcase.strip
+      chars = " " + chars + " "
+      (0..chars.length-3).each do |idx|
+        trigrams << chars[idx,3].to_s
+      end
+    end
+    return trigrams.to_a
+  end
+end

data/lib/trigram_model_extensions.rb

@@ -0,0 +1,102 @@
+module FuzzySearch
+  module TrigramModelExtensions
+    def set_target_class(cls)
+      write_inheritable_attribute :target_class, cls
+      class_inheritable_reader :target_class
+    end
+
+    def rebuild_index
+      reset_column_information
+      delete_all
+      target_class.find_each do |rec|
+        # Maybe can make this more efficient by updating trigrams for
+        # batches of records...
+        update_trigrams(rec)
+      end
+    end
+
+    def params_for_search(search_term, opts = {})
+      trigrams = FuzzySearch::split_trigrams(search_term)
+      # No results for empty search string
+      return {:conditions => "0 = 1"} unless trigrams and !trigrams.empty?
+
+      subset = nil
+      if opts[:subset]
+        if (
+        opts[:subset].size == 1 &&
+        opts[:subset].keys.first == target_class.fuzzy_search_subset_property
+        )
+          subset = opts[:subset].values.first.to_i
+          opts.delete(:subset)
+        else
+          # TODO Test me
+          raise "Invalid subset argument #{opts[:subset]}"
+        end
+      end
+
+      unless opts.empty?
+        # TODO Test me
+        raise "Invalid options: #{opts.keys.join(",")}"
+      end
+
+      # Retrieve the IDs of the matching items
+      search_result = connection.select_rows(
+        "SELECT rec_id, count(*) FROM #{i(table_name)} " +
+        (connection.adapter_name.downcase == 'mysql' ?
+          "IGNORE INDEX (index_#{table_name}_on_rec_id) " : ""
+        ) +
+        "WHERE token IN (#{trigrams.map{|t| v(t)}.join(',')}) " +
+        (subset ? "AND subset = #{subset} " : "") +
+        "GROUP by rec_id " +
+        "ORDER BY count(*) DESC " +
+        "LIMIT #{target_class.send(:fuzzy_search_limit)}"
+      )
+      return {:conditions => "0 = 1"} if search_result.empty?
+
+      # Perform a join between the target table and a fake table of matching ids
+      static_sql_union = search_result.map{|rec_id, count|
+        "SELECT #{v(rec_id)} AS id, #{count} AS score"
+      }.join(" UNION ");
+      primary_key_expr = "#{i(target_class.table_name)}.#{i(target_class.primary_key)}"
+      return {
+        :joins => "INNER JOIN (#{static_sql_union}) AS fuzzy_search_results ON " +
+                  "fuzzy_search_results.id = #{primary_key_expr}",
+        :order => "fuzzy_search_results.score DESC"
+      }
+    end
+
+    def update_trigrams(rec)
+      delete_trigrams(rec)
+
+      values = target_class.fuzzy_search_properties.map{|p| rec.send(p)}
+      values = values.select{|p| p and p.respond_to?(:to_s)}
+      trigrams = FuzzySearch::split_trigrams(values)
+
+      subset_prop = target_class.fuzzy_search_subset_property
+      subset = subset_prop ? rec.send(subset_prop) : 0
+
+      # Ar-extensions import, much much faster than individual creates
+      import(
+        [:subset, :token, :rec_id],
+        trigrams.map{|t| [subset, t, rec.id]},
+        :validate => false
+      )
+    end
+
+    def delete_trigrams(rec)
+      delete_all(:rec_id => rec.id)
+    end
+
+    private
+
+    # Quote SQL identifier (i.e. table or column name)
+    def i(s)
+      connection.quote_column_name(s)
+    end
+
+    # Quote SQL value (i.e. a string or number)
+    def v(s)
+      connection.quote(s)
+    end
+  end
+end

data/rails_generators/fuzzy_search_table/fuzzy_search_table_generator.rb

@@ -0,0 +1,21 @@
+class FuzzySearchTableGenerator < Rails::Generator::NamedBase
+  attr_reader :target_model_name
+  attr_reader :table_name
+  attr_reader :migration_filename
+  attr_reader :migration_name
+
+  def initialize(runtime_args, runtime_options = {})
+    super
+    @target_model_name = name.classify
+    @table_name = "#{name.underscore}_fuzzy_search_trigrams"
+    @migration_filename = "create_#{name.underscore}_fuzzy_search_table"
+    @migration_name = migration_filename.classify
+  end
+
+  def manifest
+    record do |m|
+      m.migration_template "create_fuzzy_search_table.rb", "db/migrate",
+        :migration_file_name => migration_filename
+    end
+  end
+end

data/rails_generators/fuzzy_search_table/templates/create_fuzzy_search_table.rb

@@ -0,0 +1,30 @@
+class <%= migration_name %> < ActiveRecord::Migration
+  def self.up
+    is_mysql = ActiveRecord::Base.connection.adapter_name.downcase == 'mysql'
+    table = '<%= table_name %>'.to_sym
+
+    create_table table, :id => false do |t|
+      t.column :subset, :integer, :limit => 4, :null => false
+      if is_mysql
+        t.column :token, "binary(3)", :null => false
+      else
+        t.column :token, :binary, :limit => 3, :null => false
+      end
+      t.column :rec_id, :integer, :null => false
+    end
+
+    if is_mysql
+      ActiveRecord::Base.connection.execute(
+        "ALTER TABLE #{table.to_s} ENGINE = MyISAM"
+      )
+    end
+    add_index table, [:token, :subset, :rec_id], :name => "full_cover"
+    add_index table, [:rec_id]
+
+    <%= target_model_name %>.rebuild_fuzzy_search_index!
+  end
+
+  def self.down
+    drop_table '<%= table_name %>'.to_sym
+  end
+end

data/test/app_root/app/models/person.rb

@@ -1,3 +1,3 @@
 class Person < ActiveRecord::Base
-  fuzzy_searchable_on :first_name, :last_name
+  fuzzy_searchable_on :first_name, :last_name, :subset_on => :favorite_number
 end

data/test/app_root/config/database.yml

@@ -1,6 +1,11 @@
+#test:
+#  adapter: sqlite3
+#  database: ":memory:"
+#  verbosity: quiet
+#  pool: 5
+#  timeout: 5000
+
 test:
-  adapter: sqlite3
-  database: ":memory:"
-  verbosity: quiet
-  pool: 5
-  timeout: 5000
+  adapter: mysql
+  database: "fuzzy"
+  username: root

data/test/app_root/db/migrate/{20100529235049_create_tables.rb → 2_create_tables.rb}

@@ -8,6 +8,7 @@ class CreateTables < ActiveRecord::Migration
       t.string :first_name
       t.string :last_name
       t.string :hobby
+      t.integer :favorite_number
     end
   end
   

data/test/app_root/db/migrate/3_create_person_fuzzy_search_table.rb

@@ -0,0 +1,30 @@
+class CreatePersonFuzzySearchTable < ActiveRecord::Migration
+  def self.up
+    is_mysql = ActiveRecord::Base.connection.adapter_name.downcase == 'mysql'
+    table = 'person_fuzzy_search_trigrams'.to_sym
+
+    create_table table, :id => false do |t|
+      t.column :subset, :integer, :limit => 4, :null => false
+      if is_mysql
+        t.column :token, "binary(3)", :null => false
+      else
+        t.column :token, :binary, :limit => 3, :null => false
+      end
+      t.column :rec_id, :integer, :null => false
+    end
+
+    if is_mysql
+      ActiveRecord::Base.connection.execute(
+        "ALTER TABLE #{table.to_s} ENGINE = MyISAM"
+      )
+    end
+    add_index table, [:token, :subset, :rec_id], :name => "full_cover"
+    add_index table, [:rec_id]
+
+    Person.rebuild_fuzzy_search_index!
+  end
+
+  def self.down
+    drop_table 'person_fuzzy_search_trigrams'.to_sym
+  end
+end

data/test/app_root/db/migrate/4_create_email_fuzzy_search_table.rb

@@ -0,0 +1,30 @@
+class CreateEmailFuzzySearchTable < ActiveRecord::Migration
+  def self.up
+    is_mysql = ActiveRecord::Base.connection.adapter_name.downcase == 'mysql'
+    table = 'email_fuzzy_search_trigrams'.to_sym
+
+    create_table table, :id => false do |t|
+      t.column :subset, :integer, :limit => 4, :null => false
+      if is_mysql
+        t.column :token, "binary(3)", :null => false
+      else
+        t.column :token, :binary, :limit => 3, :null => false
+      end
+      t.column :rec_id, :integer, :null => false
+    end
+
+    if is_mysql
+      ActiveRecord::Base.connection.execute(
+        "ALTER TABLE #{table.to_s} ENGINE = MyISAM"
+      )
+    end
+    add_index table, [:token, :subset, :rec_id], :name => "full_cover"
+    add_index table, [:rec_id]
+
+    Email.rebuild_fuzzy_search_index!
+  end
+
+  def self.down
+    drop_table 'email_fuzzy_search_trigrams'.to_sym
+  end
+end

data/test/factories.rb

@@ -5,6 +5,7 @@ FactoryGirl.define do
     first_name 'John'
     last_name 'Doe'
     hobby 'Flying kites'
+    sequence(:favorite_number) {|n| (n%3) + 1 }
   end
 
   factory :email do

data/test/test_helper.rb

@@ -18,6 +18,7 @@ end
 
 require 'rubygems'
 require 'minitest/autorun'
+require 'minitest/benchmark' if ENV["BENCH"]
 require 'redgreen'
 require 'pp'
 
@@ -61,9 +62,21 @@ module MiniTest
 end
 
 require 'factories'
+require 'faker'
 MiniTest::Unit::TestCase.send(:include, Factory::Syntax::Methods)
 
 MiniTest::Unit::TestCase.add_setup_hook do
   ActiveRecord::Migration.verbose = false
   ActiveRecord::Migrator.migrate("#{Rails.root}/db/migrate") # Migrations in the test app
 end
+
+# From http://stackoverflow.com/questions/1090801/1091106
+ActiveRecord::ConnectionAdapters::AbstractAdapter.class_eval do
+  attr_reader :last_query
+
+  def log_with_last_query(sql, name, &block)
+    @last_query = [sql, name]
+    log_without_last_query(sql, name, &block)
+  end
+  alias_method_chain :log, :last_query
+end

data/test/unit/fuzzy_search_test.rb

@@ -2,7 +2,7 @@ require File.expand_path(File.dirname(__FILE__) + '/../test_helper')
 
 describe "fuzzy_search" do
   before do 
-    create(:person, :last_name => "meier", :first_name => "kristian")
+    @kris = create(:person, :last_name => "meier", :first_name => "kristian")
     create(:person, :last_name => "meyer", :first_name => "christian", :hobby => "Bicycling")
     create(:person, :last_name => "mayr", :first_name => "Chris")
     create(:person, :last_name => "maier", :first_name => "christoph", :hobby => "Bicycling")
@@ -18,12 +18,13 @@ describe "fuzzy_search" do
   after do
     Person.delete_all
     Email.delete_all
-    FuzzySearchTrigram.delete_all
+    Person::FuzzySearchTrigram.delete_all
+    Email::FuzzySearchTrigram.delete_all
   end
 
   it "can search for records with similar strings to a query" do
-    assert_equal 3, Person.fuzzy_search("meyr").size
-    assert_equal 1, Person.fuzzy_search("myr").size
+    refute_empty Person.fuzzy_search("maier")
+    refute_empty Person.fuzzy_search("ather")
   end
   
   it "can search on multiple columns" do
@@ -32,14 +33,10 @@ describe "fuzzy_search" do
     assert_equal "meier", result[0].last_name
   end
 
-  it "sorts results by their fuzzy match score" do
+  it "sorts results with the best match first" do
     result = Person.fuzzy_search("kristian meier")
-    assert_equal 100, result[0].fuzzy_score
-    prior = 100
-    (1..3).each do |idx|
-      assert result[idx].fuzzy_score <= prior
-      prior = result[idx].fuzzy_score
-    end
+    assert_equal @kris, result.first
+    assert result.size > 1
   end
 
   it "returns an empty result set when given an empty query string" do
@@ -48,9 +45,16 @@ describe "fuzzy_search" do
   end
 
   it "updates the search index automatically when a new record is saved" do
-    assert_empty Person.fuzzy_search("Dave")
+    assert_empty Person.fuzzy_search("David")
     create(:person, :first_name => "David", :last_name => "Simon")
-    refute_empty Person.fuzzy_search("Dave")
+    refute_empty Person.fuzzy_search("David")
+  end
+
+  it "only updates the appropriate trigram table" do
+    original_state = Email::FuzzySearchTrigram.all.to_a
+    create(:person, :first_name => "David", :last_name => "Simon")
+    new_state = Email::FuzzySearchTrigram.all.to_a
+    assert_equal original_state, new_state
   end
 
   it "updates the search index automatically when a record is updated" do
@@ -80,12 +84,8 @@ describe "fuzzy_search" do
     refute_empty Email.fuzzy_search("oscar")
   end
 
-  it "can normalize strings" do
-    assert_equal("aaaaaa", Person.normalize("ÀÁÂÃÄÅ"))
-  end
-
   it "normalizes strings before searching on them" do
-    assert_equal 1, Person.fuzzy_search("Müll").size
+    assert_equal 1, Person.fuzzy_search("Müell").size
     assert_equal 1, Email.fuzzy_search("öscar").size
   end
 
@@ -95,12 +95,21 @@ describe "fuzzy_search" do
 
   it "can search through a scope" do
     scope = Person.scoped({:conditions => {:hobby => "Bicycling"}})
-    assert_equal 4, Person.fuzzy_search("chris").size
-    assert_equal 2, scope.fuzzy_search("chris").size
+    full = Person.fuzzy_search("chris")
+    subset = scope.fuzzy_search("chris")
+    assert full.size > subset.size
+    assert subset.size > 0
+  end
+
+  it "can use the subset field to narrow the search range" do
+    full = Person.fuzzy_search("chris")
+    subset = Person.fuzzy_search("chris", :subset => {:favorite_number => 2})
+    assert full.size > subset.size
+    assert subset.size > 0
   end
 
   it "can rebuild the search index from scratch" do
-    FuzzySearchTrigram.delete_all
+    Person::FuzzySearchTrigram.delete_all
     assert_empty Person.fuzzy_search("chris")
     Person.rebuild_fuzzy_search_index!
     refute_empty Person.fuzzy_search("chris")

data/test/unit/performance_test.rb

@@ -0,0 +1,51 @@
+require File.expand_path(File.dirname(__FILE__) + '/../test_helper')
+
+describe "fuzzy_search" do
+  if ENV["BENCH"]
+    before do
+      Person.set_table_name "preloaded_people"
+      Person::FuzzySearchTrigram.set_table_name "preloaded_person_fuzzy_search_trigrams"
+
+      # Force Person to reload its fuzzy type id from preloaded_types
+      Person.send(:write_inheritable_attribute, :fuzzy_search_cached_type_id, nil)
+    end
+
+    after do
+      Person.set_table_name "people"
+      Person::FuzzySearchTrigram.set_table_name "person_fuzzy_search_trigrams"
+    end
+
+    bench_range do
+      [1, 10, 50]
+    end
+
+    bench_performance_linear "regular queries", 0.9 do |n|
+      c = 0
+      n.times do
+        result = Person.scoped(:limit => 20).fuzzy_search(Faker::Name.last_name)
+        c += result.size
+      end
+      if n >= 30
+        rpq = c/(n.to_f)
+        if rpq < 2
+          raise "Sanity check failure, average results per query: #{rpq}"
+        end
+      end
+    end
+
+    bench_performance_linear "subset queries", 0.9 do |n|
+      c = 0
+      n.times do
+        result = Person.scoped(:limit => 20).fuzzy_search(Faker::Name.last_name,
+          :subset => {:favorite_number => rand(100)+1})
+        c += result.size
+      end
+      if n >= 30
+        rpq = c/(n.to_f)
+        if rpq < 0.1
+          raise "Sanity check failure, average results per query: #{rpq}"
+        end
+      end
+    end
+  end
+end

data/test/unit/split_trigrams_test.rb

@@ -0,0 +1,18 @@
+require File.expand_path(File.dirname(__FILE__) + '/../test_helper')
+require 'set'
+
+describe "fuzzy search trigram splitter" do
+  it "can split a string into trigrams with emphasized edges" do
+    t = [" da", "dav", "avi", "vid", "id "]
+    assert_equal Set.new(t), Set.new(FuzzySearch::split_trigrams("david"))
+  end
+
+  it "can normalize strings" do
+    assert_equal([" a "], FuzzySearch::split_trigrams("À"))
+  end
+
+  it "can handle an array of strings" do
+    t = [" x ", " y ", " zi", "zig", "ig "]
+    assert_equal Set.new(t), Set.new(FuzzySearch::split_trigrams(["x", "y", "zig"]))
+  end
+end

metadata

@@ -1,12 +1,12 @@
 --- !ruby/object:Gem::Specification 
 name: fuzzy_search
 version: !ruby/object:Gem::Version 
-  hash: 13
+  hash: 3
   prerelease: 
   segments: 
   - 0
-  - 3
-  version: "0.3"
+  - 4
+  version: "0.4"
 platform: ruby
 authors: 
 - Kristian Meier
@@ -15,7 +15,7 @@ autorequire:
 bindir: bin
 cert_chain: []
 
-date: 2011-10-13 00:00:00 Z
+date: 2011-10-27 00:00:00 Z
 dependencies: 
 - !ruby/object:Gem::Dependency 
   name: ar-extensions
@@ -45,12 +45,15 @@ files:
 - MIT-LICENSE
 - README.md
 - Rakefile
+- TODO
+- gen_fake_data.rb
 - lib/fuzzy_model_extensions.rb
 - lib/fuzzy_search.rb
-- lib/fuzzy_search_trigram.rb
 - lib/fuzzy_search_ver.rb
-- rails_generators/fuzzy_search_setup/fuzzy_search_setup_generator.rb
-- rails_generators/fuzzy_search_setup/templates/create_fuzzy_search_trigrams.rb
+- lib/split_trigrams.rb
+- lib/trigram_model_extensions.rb
+- rails_generators/fuzzy_search_table/fuzzy_search_table_generator.rb
+- rails_generators/fuzzy_search_table/templates/create_fuzzy_search_table.rb
 - test/app_root/app/models/email.rb
 - test/app_root/app/models/person.rb
 - test/app_root/config/boot.rb
@@ -58,13 +61,16 @@ files:
 - test/app_root/config/environment.rb
 - test/app_root/config/environments/test.rb
 - test/app_root/config/routes.rb
-- test/app_root/db/migrate/20100529235049_create_tables.rb
-- test/app_root/db/migrate/20111013132330_create_fuzzy_search_trigrams.rb
+- test/app_root/db/migrate/2_create_tables.rb
+- test/app_root/db/migrate/3_create_person_fuzzy_search_table.rb
+- test/app_root/db/migrate/4_create_email_fuzzy_search_table.rb
 - test/app_root/vendor/plugins/fuzzy_search/init.rb
 - test/factories.rb
 - test/test.watchr
 - test/test_helper.rb
 - test/unit/fuzzy_search_test.rb
+- test/unit/performance_test.rb
+- test/unit/split_trigrams_test.rb
 homepage: http://github.com/DavidMikeSimon/fuzzy_search
 licenses: []
 

data/lib/fuzzy_search_trigram.rb

@@ -1,2 +0,0 @@
-class FuzzySearchTrigram < ActiveRecord::Base
-end

data/rails_generators/fuzzy_search_setup/fuzzy_search_setup_generator.rb

@@ -1,8 +0,0 @@
-class FuzzySearchSetupGenerator < Rails::Generator::Base
-  def manifest
-    record do |m|
-      m.migration_template "create_fuzzy_search_trigrams.rb", "db/migrate",
-        :migration_file_name => "create_fuzzy_search_trigrams"
-    end
-  end
-end

data/rails_generators/fuzzy_search_setup/templates/create_fuzzy_search_trigrams.rb

@@ -1,15 +0,0 @@
-class CreateFuzzySearchTrigrams < ActiveRecord::Migration
-  def self.up
-    create_table :fuzzy_search_trigrams, :id => false do |t|
-      t.column :token, :string, :limit => 3
-      t.column :rec_type, :string
-      t.column :rec_id, :integer
-    end
-
-    add_index :fuzzy_search_trigrams, [:rec_type, :token]
-  end
-
-  def self.down
-    drop_table :fuzzy_search_trigrams
-  end
-end

data/test/app_root/db/migrate/20111013132330_create_fuzzy_search_trigrams.rb

@@ -1,15 +0,0 @@
-class CreateFuzzySearchTrigrams < ActiveRecord::Migration
-  def self.up
-    create_table :fuzzy_search_trigrams, :id => false do |t|
-      t.column :token, :string, :limit => 3
-      t.column :rec_type, :string
-      t.column :rec_id, :integer
-    end
-
-    add_index :fuzzy_search_trigrams, [:rec_type, :token]
-  end
-
-  def self.down
-    drop_table :fuzzy_search_trigrams
-  end
-end