fuzzy_match 1.2.2 → 1.3.0

data/README.markdown

@@ -13,6 +13,8 @@ Replaces [`loose_tight_dictionary`](https://github.com/seamusabshere/loose_tight
     >> matcher.find('Shamus')
     => "seamus"
 
+See also the blog post [Fuzzy match in Ruby](http://numbers.brighterplanet.com/2012/01/18/fuzzy-match-in-ruby/).
+
 ## Default matching (string similarity)
 
 At the core, and even if you configure nothing else, string similarity (calculated by "pair distance" aka Dice's) is used to compare records.
@@ -45,12 +47,6 @@ Group records together.
 
 Setting a blocking of `/Airbus/` ensures that strings containing "Airbus" will only be scored against to other strings containing "Airbus". A better blocking in this case would probably be `/airbus/i`.
 
-### Normalizers (formerly called tighteners)
-
-Strip strings down to the essentials.
-
-Adding a normalizer like `/(boeing).*(7\d\d)/i` will cause "BOEING COMPANY 747" and "boeing747" to be normalized to "BOEING 747" and "boeing 747", respectively. Since things are generally downcased before they are compared, these would be an exact match.
-
 ### Identities
 
 Prevent impossible matches.
@@ -59,10 +55,16 @@ Adding an identity like `/(f)-?(\d50)/i` ensures that "Ford F-150" and "Ford F-2
 
 ### Stop words
 
-Ignore common and/or meaningless words.
+Ignore common and/or meaningless words. Applied before normalizers.
 
 Adding a stop word like `THE` ensures that it is not taken into account when comparing "THE CAT", "THE DAT", and "THE CATT"
 
+### Normalizers (formerly called tighteners)
+
+Strip strings down to the essentials. Applied after stop words.
+
+Adding a normalizer like `/(boeing).*(7\d\d)/i` will cause "BOEING COMPANY 747" and "boeing747" to be normalized to "BOEING 747" and "boeing 747", respectively. Since things are generally downcased before they are compared, these would be an exact match.
+
 ## Find options
 
 * `read`: how to interpret each record in the 'haystack', either a Proc or a symbol
@@ -127,9 +129,13 @@ The admittedly imperfect metaphor is "look for a needle in a haystack"
 * needle: the search term
 * haystack: the records you are searching (<b>your result will be an object from here</b>)
 
-## Credits (and how to make things faster)
+## Using amatch to make it faster
+
+You can optionally use [`amatch`](http://flori.github.com/amatch/) by [Florian Frank](https://github.com/flori) (thanks Flori!) to make string similarity calculations in a C extension.
 
-If you add the [`amatch`](http://flori.github.com/amatch/) gem to your Gemfile, it will use that, which is much faster (but [segfaults have been seen in the wild](https://github.com/flori/amatch/issues/3)). Thanks [Flori](https://github.com/flori)!
+    require 'fuzzy_match'
+    require 'amatch' # note that you have to require this... fuzzy_match won't require it for you
+    FuzzyMatch.engine = :amatch
 
 Otherwise, pure ruby versions of the string similarity algorithms derived from the [answer to a StackOverflow question](http://stackoverflow.com/questions/653157/a-better-similarity-ranking-algorithm-for-variable-length-strings) and [the text gem](https://github.com/threedaymonk/text/blob/master/lib/text/levenshtein.rb) are used. Thanks [marzagao](http://stackoverflow.com/users/10997/marzagao) and [threedaymonk](https://github.com/threedaymonk)!
 

data/examples/bts_aircraft/test_bts_aircraft.rb

@@ -1,7 +1,5 @@
 require File.expand_path('../../../test/helper.rb', __FILE__)
 
-require 'shoulda'
-
 # How to iteratively develop a dictionary.
 
 # ruby ./examples/bts_aircraft/test_bts_aircraft.rb
@@ -70,36 +68,36 @@ FINAL_OPTIONS = {
   :blockings => BLOCKINGS
 }
 
-class TestBtsAircraft < Test::Unit::TestCase
-  should "understand records by using the haystack reader" do
+class TestBtsAircraft < MiniTest::Spec
+  it "understand records by using the haystack reader" do
     d = FuzzyMatch.new HAYSTACK, FINAL_OPTIONS
-    assert d.haystack.map { |record| record.to_str }.include?('boeing boeing 707-100')
+    d.haystack.map { |record| record.to_str }.must_include 'boeing boeing 707-100'
   end
 
-  should "find an easy match" do
+  it "find an easy match" do
     d = FuzzyMatch.new HAYSTACK, FINAL_OPTIONS
     record = d.find('boeing 707(100)')
-    assert_equal HAYSTACK_RECORD_CLASS, record.class
-    assert_equal HAYSTACK_READER.call(record), 'boeing boeing 707-100'
+    record.class.must_equal HAYSTACK_RECORD_CLASS
+    HAYSTACK_READER.call(record).must_equal 'boeing boeing 707-100'
   end
   
   POSITIVES.each do |row|
     needle = row['needle']
     correct_record = row['haystack']
-    should %{find #{correct_record.blank? ? 'nothing' : correct_record} when looking for #{needle}} do
+    it %{find #{correct_record.blank? ? 'nothing' : correct_record} when looking for #{needle}} do
       d = FuzzyMatch.new HAYSTACK, FINAL_OPTIONS
       record = d.find(needle.downcase)
-      assert_equal correct_record.downcase, HAYSTACK_READER.call(record)
+      HAYSTACK_READER.call(record).must_equal correct_record.downcase
     end
   end
   
   NEGATIVES.each do |row|
     needle = row['needle']
     incorrect_record = row['haystack']
-    should %{not find #{incorrect_record} when looking for #{needle}} do
+    it %{not find #{incorrect_record} when looking for #{needle}} do
       d = FuzzyMatch.new HAYSTACK, FINAL_OPTIONS
       record = d.find(needle.downcase)
-      assert(incorrect_record.downcase != HAYSTACK_READER.call(record))
+      HAYSTACK_READER.call(record)).wont_equal incorrect_record.downcase
     end
   end
 end

data/fuzzy_match.gemspec

@@ -19,10 +19,9 @@ Gem::Specification.new do |s|
   s.executables   = `git ls-files -- bin/*`.split("\n").map{ |f| File.basename(f) }
   s.require_paths = ["lib"]
 
-  s.add_development_dependency "shoulda"
-  s.add_development_dependency "remote_table"
+  s.add_development_dependency "minitest"
   s.add_development_dependency 'activerecord', '>=3'
-  s.add_development_dependency 'mysql'
+  s.add_development_dependency 'mysql2'
   s.add_development_dependency 'cohort_scope'
   s.add_development_dependency 'weighted_average'
   s.add_development_dependency 'rake'

data/lib/fuzzy_match.rb

@@ -20,6 +20,29 @@ end
 
 # See the README for more information.
 class FuzzyMatch
+  class << self
+    def engine
+      @@engine ||= :pure_ruby
+    end
+    
+    def engine=(alt_engine)
+      @@engine = alt_engine
+    end
+    
+    def score_class
+      case engine
+      when :pure_ruby
+        Score::PureRuby
+      when :amatch
+        Score::Amatch
+      else
+        raise ::ArgumentError, "[fuzzy_match] #{engine.inspect} is not a recognized engine."
+      end
+    end
+  end
+  
+  DEFAULT_ENGINE = :pure_ruby
+  
   DEFAULT_OPTIONS = {
     :first_blocking_decides => false,
     :must_match_blocking => false,

data/lib/fuzzy_match/cached_result.rb

@@ -1,6 +1,10 @@
 class FuzzyMatch
   class CachedResult < ::ActiveRecord::Base
-    set_table_name :fuzzy_match_cached_results
+    if ::ActiveRecord::VERSION::STRING >= '3.2'
+      self.table_name = :fuzzy_match_cached_results
+    else
+      set_table_name :fuzzy_match_cached_results
+    end
     
     def self.create_table
       connection.create_table :fuzzy_match_cached_results do |t|
@@ -17,7 +21,7 @@ class FuzzyMatch
     
     def self.setup(from_scratch = false)
       connection.drop_table :fuzzy_match_cached_results if from_scratch and table_exists?
-      create_table unless table_exists?
+      create_table
     end
     
     module ActiveRecordBaseExtension

data/lib/fuzzy_match/score.rb

@@ -1,7 +1,8 @@
+require 'fuzzy_match/score/pure_ruby'
+require 'fuzzy_match/score/amatch'
+
 class FuzzyMatch
   class Score
-    extend ::ActiveSupport::Memoizable
-    
     attr_reader :str1
     attr_reader :str2
 
@@ -10,10 +11,6 @@ class FuzzyMatch
       @str2 = str2.downcase
     end
 
-    def inspect
-      %{#<FuzzyMatch::Score: str1=#{str1.inspect} str2=#{str2.inspect} dices_coefficient_similar=#{dices_coefficient_similar} levenshtein_similar=#{levenshtein_similar}>}
-    end
-
     def <=>(other)
       by_dices_coefficient = (dices_coefficient_similar <=> other.dices_coefficient_similar)
       if by_dices_coefficient == 0
@@ -22,107 +19,5 @@ class FuzzyMatch
         by_dices_coefficient
       end
     end
-
-    if defined?(::Amatch)
-
-      def dices_coefficient_similar
-        if str1 == str2
-          return 1.0
-        elsif str1.length == 1 and str2.length == 1
-          return 0.0
-        end
-        str1.pair_distance_similar str2
-      end
-      memoize :dices_coefficient_similar
-
-      def levenshtein_similar
-        str1.levenshtein_similar str2
-      end
-      memoize :levenshtein_similar
-
-    else
-
-      SPACE = ' '
-      # http://stackoverflow.com/questions/653157/a-better-similarity-ranking-algorithm-for-variable-length-strings
-      def dices_coefficient_similar
-        if str1 == str2
-          return 1.0
-        elsif str1.length == 1 and str2.length == 1
-          return 0.0
-        end
-        pairs1 = (0..str1.length-2).map do |i|
-          str1[i,2]
-        end.reject do |pair|
-          pair.include? SPACE
-        end
-        pairs2 = (0..str2.length-2).map do |i|
-          str2[i,2]
-        end.reject do |pair|
-          pair.include? SPACE
-        end
-        union = pairs1.size + pairs2.size
-        intersection = 0
-        pairs1.each do |p1|
-          0.upto(pairs2.size-1) do |i|
-            if p1 == pairs2[i]
-              intersection += 1
-              pairs2.slice!(i)
-              break
-            end
-          end
-        end
-        (2.0 * intersection) / union
-      end
-      memoize :dices_coefficient_similar
-
-      # this seems like it would slow things down
-      def utf8?
-        (defined?(::Encoding) ? str1.encoding.to_s : $KCODE).downcase.start_with?('u')
-      end
-      memoize :utf8?
-
-      # extracted/adapted from the text gem version 1.0.2
-      # normalization added for utf-8 strings
-      # lib/text/levenshtein.rb
-      def levenshtein_similar
-        if utf8?
-          unpack_rule = 'U*'
-        else
-          unpack_rule = 'C*'
-        end
-        s = str1.unpack(unpack_rule)
-        t = str2.unpack(unpack_rule)
-        n = s.length
-        m = t.length
-        if n == 0 or m == 0
-          return 0.0
-        end
-        d = (0..m).to_a
-        x = nil
-        (0...n).each do |i|
-          e = i+1
-          (0...m).each do |j|
-            cost = (s[i] == t[j]) ? 0 : 1
-            x = [
-              d[j+1] + 1, # insertion
-              e + 1,      # deletion
-              d[j] + cost # substitution
-            ].min
-            d[j] = e
-            e = x
-          end
-          d[m] = x
-        end
-        # normalization logic from https://github.com/flori/amatch/blob/master/ext/amatch_ext.c#L301
-        # if (b_len > a_len) {
-        #     result = rb_float_new(1.0 - ((double) v[p][b_len]) / b_len);
-        # } else {
-        #     result = rb_float_new(1.0 - ((double) v[p][b_len]) / a_len);
-        # }
-        1.0 - x.to_f / [n, m].max
-      end
-      memoize :levenshtein_similar
-
-    end
   end
-end
+end

data/lib/fuzzy_match/score/amatch.rb

@@ -0,0 +1,25 @@
+class FuzzyMatch
+  class Score
+    # be sure to `require 'amatch'` before you use this class
+    class Amatch < Score
+
+      def inspect
+        %{#<FuzzyMatch::Score::Amatch: str1=#{str1.inspect} str2=#{str2.inspect} dices_coefficient_similar=#{dices_coefficient_similar} levenshtein_similar=#{levenshtein_similar}>}
+      end
+
+      def dices_coefficient_similar
+        @dices_coefficient_similar ||= if str1 == str2
+          1.0
+        elsif str1.length == 1 and str2.length == 1
+          0.0
+        else
+          str1.pair_distance_similar str2
+        end
+      end
+
+      def levenshtein_similar
+        @levenshtein_similar ||= str1.levenshtein_similar str2
+      end
+    end
+  end
+end

data/lib/fuzzy_match/score/pure_ruby.rb

@@ -0,0 +1,103 @@
+class FuzzyMatch
+  class Score
+    class PureRuby < Score
+
+      SPACE = ' '
+
+      def inspect
+        %{#<FuzzyMatch::Score::PureRuby: str1=#{str1.inspect} str2=#{str2.inspect} dices_coefficient_similar=#{dices_coefficient_similar} levenshtein_similar=#{levenshtein_similar}>}
+      end
+
+      # http://stackoverflow.com/questions/653157/a-better-similarity-ranking-algorithm-for-variable-length-strings
+      def dices_coefficient_similar
+        return @dices_coefficient_similar if @dices_coefficient_similar.is_a?(::Float)
+      
+        if str1 == str2
+          @dices_coefficient_similar = 1.0
+          return @dices_coefficient_similar
+        elsif str1.length == 1 and str2.length == 1
+          @dices_coefficient_similar = 0.0
+          return @dices_coefficient_similar
+        end
+      
+        pairs1 = (0..str1.length-2).map do |i|
+          str1[i,2]
+        end.reject do |pair|
+          pair.include? SPACE
+        end
+        pairs2 = (0..str2.length-2).map do |i|
+          str2[i,2]
+        end.reject do |pair|
+          pair.include? SPACE
+        end
+        union = pairs1.size + pairs2.size
+        intersection = 0
+        pairs1.each do |p1|
+          0.upto(pairs2.size-1) do |i|
+            if p1 == pairs2[i]
+              intersection += 1
+              pairs2.slice!(i)
+              break
+            end
+          end
+        end
+        @dices_coefficient_similar = (2.0 * intersection) / union
+      end
+
+      # extracted/adapted from the text gem version 1.0.2
+      # normalization added for utf-8 strings
+      # lib/text/levenshtein.rb
+      def levenshtein_similar
+        return @levenshtein_similar if @levenshtein_similar.is_a?(::Float)
+      
+        if utf8?
+          unpack_rule = 'U*'
+        else
+          unpack_rule = 'C*'
+        end
+        s = str1.unpack(unpack_rule)
+        t = str2.unpack(unpack_rule)
+        n = s.length
+        m = t.length
+      
+        if n == 0 or m == 0
+          @levenshtein_similar = 0.0
+          return @levenshtein_similar
+        end
+      
+        d = (0..m).to_a
+        x = nil
+        (0...n).each do |i|
+          e = i+1
+          (0...m).each do |j|
+            cost = (s[i] == t[j]) ? 0 : 1
+            x = [
+              d[j+1] + 1, # insertion
+              e + 1,      # deletion
+              d[j] + cost # substitution
+            ].min
+            d[j] = e
+            e = x
+          end
+          d[m] = x
+        end
+        # normalization logic from https://github.com/flori/amatch/blob/master/ext/amatch_ext.c#L301
+        # if (b_len > a_len) {
+        #     result = rb_float_new(1.0 - ((double) v[p][b_len]) / b_len);
+        # } else {
+        #     result = rb_float_new(1.0 - ((double) v[p][b_len]) / a_len);
+        # }
+        @levenshtein_similar = 1.0 - x.to_f / [n, m].max
+      end
+
+      private
+    
+      def utf8?
+        return @utf8_query[0] if @utf8_query.is_a?(::Array) # ActiveSupport::Memoizable is deprecated in 3.2, how annoying
+        utf8_query = (defined?(::Encoding) ? str1.encoding.to_s : $KCODE).downcase.start_with?('u')
+        @utf8_query = [utf8_query]
+        utf8_query
+      end
+    end
+  end
+end

data/lib/fuzzy_match/similarity.rb

@@ -23,7 +23,7 @@ class FuzzyMatch
     end
     
     def best_score
-      @best_score ||= Score.new best_wrapper1_variant, best_wrapper2_variant
+      @best_score ||= FuzzyMatch.score_class.new best_wrapper1_variant, best_wrapper2_variant
     end
     
     def best_wrapper1_variant
@@ -39,8 +39,8 @@ class FuzzyMatch
         wrapper1_variant1, wrapper2_variant1 = tuple1
         wrapper1_variant2, wrapper2_variant2 = tuple2
 
-        score1 = Score.new wrapper1_variant1, wrapper2_variant1
-        score2 = Score.new wrapper1_variant2, wrapper2_variant2
+        score1 = FuzzyMatch.score_class.new wrapper1_variant1, wrapper2_variant1
+        score2 = FuzzyMatch.score_class.new wrapper1_variant2, wrapper2_variant2
         
         score1 <=> score2
       end[-1]

data/lib/fuzzy_match/version.rb

@@ -1,3 +1,3 @@
 class FuzzyMatch
-  VERSION = '1.2.2'
+  VERSION = '1.3.0'
 end

data/test/helper.rb

@@ -1,15 +1,10 @@
 require 'rubygems'
 require 'bundler'
 Bundler.setup
-require 'test/unit'
+require 'minitest/spec'
+require 'minitest/autorun'
 require 'stringio'
-require 'remote_table'
-if ENV['AMATCH'] == 'true'
-  require 'amatch'
-end
+
 $LOAD_PATH.unshift(File.dirname(__FILE__))
 $LOAD_PATH.unshift(File.join(File.dirname(__FILE__), '..', 'lib'))
 require 'fuzzy_match'
-
-class Test::Unit::TestCase
-end

data/test/test_amatch.rb

@@ -0,0 +1,16 @@
+unless RUBY_PLATFORM == 'java'
+  require 'helper'
+  require 'test_fuzzy_match'
+  require 'amatch'
+  
+  class TestAmatch < TestFuzzyMatch
+    before do
+      $testing_amatch = true
+      FuzzyMatch.engine = :amatch
+    end
+    after do
+      $testing_amatch = false
+      FuzzyMatch.engine = nil
+    end
+  end
+end

data/test/test_blocking.rb

@@ -1,28 +1,28 @@
 require 'helper'
 
-class TestBlocking < Test::Unit::TestCase
-  def test_001_match_one
+class TestBlocking < MiniTest::Spec
+  it %{matches a single string argument} do
     b = FuzzyMatch::Blocking.new %r{apple}
-    assert_equal true, b.match?('2 apples')
+    b.match?('2 apples').must_equal true
   end
-  
-  def test_002_join_both
-    b = FuzzyMatch::Blocking.new %r{apple}
-    assert_equal true, b.join?('apple', '2 apples')    
+
+  it %{embraces case insensitivity} do
+    b = FuzzyMatch::Blocking.new %r{apple}i
+    b.match?('2 Apples').must_equal true
   end
   
-  def test_002_doesnt_join_both
+  it %{joins two string arguments} do
     b = FuzzyMatch::Blocking.new %r{apple}
-    assert_equal false, b.join?('orange', '2 apples')
+    b.join?('apple', '2 apples').must_equal true
   end
   
-  def test_003_no_information
+  it %{fails to join two string arguments} do
     b = FuzzyMatch::Blocking.new %r{apple}
-    assert_equal nil, b.join?('orange', 'orange')
+    b.join?('orange', '2 apples').must_equal false
   end
   
-  def test_004_accepts_case_insensitivity
-    b = FuzzyMatch::Blocking.new %r{apple}i
-    assert_equal true, b.match?('2 Apples')
+  it %{returns nil instead of false when it has no information} do
+    b = FuzzyMatch::Blocking.new %r{apple}
+    b.join?('orange', 'orange').must_be_nil
   end
 end

data/test/test_cache.rb

@@ -6,7 +6,7 @@ require 'cohort_scope'
 require 'weighted_average'
 
 ActiveRecord::Base.establish_connection(
-  'adapter' => 'mysql',
+  'adapter' => 'mysql2',
   'database' => 'fuzzy_match_test',
   'username' => 'root',
   'password' => 'password'
@@ -99,32 +99,32 @@ FlightSegment.find_each do |fs|
   fs.cache_aircraft!
 end
 
-class TestCache < Test::Unit::TestCase
-  def test_002_one_degree_of_separation
+class TestCache < MiniTest::Spec
+  it %{joins aircraft to flight segments} do
     aircraft = Aircraft.find('B742')
-    assert_equal 2, aircraft.flight_segments.count
+    aircraft.flight_segments.count.must_equal 2
   end
   
-  def test_003_standard_sql_calculations
+  it %{allow simple SQL operations} do
     aircraft = Aircraft.find('B742')
-    assert_equal 110, aircraft.flight_segments.sum(:passengers)
+    aircraft.flight_segments.sum(:passengers).must_equal 110
   end
   
-  def test_004_weighted_average
+  it %{works with weighted_average} do
     aircraft = Aircraft.find('B742')
-    assert_equal 5.45455, aircraft.flight_segments.weighted_average(:seats, :weighted_by => :passengers)
+    aircraft.flight_segments.weighted_average(:seats, :weighted_by => :passengers).must_equal 5.45455
   end
   
-  def test_005_right_way_to_do_cohorts
+  it %{works with cohort_scope (albeit rather clumsily)} do
     aircraft = Aircraft.find('B742')
-    assert_equal 2, FlightSegment.big_cohort(:aircraft_description => aircraft.flight_segments_foreign_keys).count
+    FlightSegment.big_cohort(:aircraft_description => aircraft.flight_segments_foreign_keys).count.must_equal 2
   end
   
-  def test_006_you_can_get_aircraft_from_flight_segments
-    fs = FlightSegment.first
-    # you need to add an aircraft_description column
-    assert_raises(ActiveRecord::StatementInvalid) do
-      assert_equal 2, fs.aircraft.count
-    end
-  end
+  # def test_006_you_can_get_aircraft_from_flight_segments
+  #   fs = FlightSegment.first
+  #   # you need to add an aircraft_description column
+  #   lambda do
+  #     fs.aircraft.count.must_equal 2
+  #   end.must_raise ActiveRecord::StatementInvalid
+  # end
 end

data/test/test_fuzzy_match.rb

@@ -1,181 +1,180 @@
 # -*- encoding: utf-8 -*-
 require 'helper'
 
-class TestFuzzyMatch < Test::Unit::TestCase
-  def test_001_find
+class TestFuzzyMatch < MiniTest::Spec
+  it %{identify the best match based on string similarity} do
     d = FuzzyMatch.new %w{ RATZ CATZ }
-    assert_equal 'RATZ', d.find('RITZ')
-    assert_equal 'RATZ', d.find('RíTZ')
+    d.find('RITZ').must_equal 'RATZ'
+    d.find('RíTZ').must_equal 'RATZ'
 
     d = FuzzyMatch.new [ 'X' ]
-    assert_equal 'X', d.find('X')
-    assert_equal nil, d.find('A')
+    d.find('X').must_equal 'X'
+    d.find('A').must_be_nil
   end
 
-  def test_002_dont_gather_last_result_by_default
+  it %{not gather metadata about the last result by default} do
     d = FuzzyMatch.new %w{ NISSAN HONDA }
     d.find('MISSAM')
-    assert_raises(::RuntimeError, /gather_last_result/) do
+    lambda do
       d.last_result
-    end
+    end.must_raise ::RuntimeError, /gather_last_result/
   end
 
-  def test_003_last_result
+  it %{optionally gather metadata about the last result} do
     d = FuzzyMatch.new %w{ NISSAN HONDA }
     d.find 'MISSAM', :gather_last_result => true
-    assert_equal 0.6, d.last_result.score
-    assert_equal 'NISSAN', d.last_result.winner
+    d.last_result.score.must_equal 0.6
+    d.last_result.winner.must_equal 'NISSAN'
   end
 
-  def test_005_correct_with_normalizer
+  it %{use NORMALIZERS} do
     d = FuzzyMatch.new ['BOEING 737-100/200', 'BOEING 737-900']
-    assert_equal 'BOEING 737-900', d.find('BOEING 737100 number 900') # false positive without normalizer
+    d.find('BOEING 737100 number 900').must_equal 'BOEING 737-900'
 
     normalizers = [
       %r{(7\d)(7|0)-?(\d{1,3})} # tighten 737-100/200 => 737100, which will cause it to win over 737-900
     ]
     d = FuzzyMatch.new ['BOEING 737-100/200', 'BOEING 737-900'], :normalizers => normalizers
-    assert_equal 'BOEING 737-100/200', d.find('BOEING 737100 number 900')
+    d.find('BOEING 737100 number 900').must_equal 'BOEING 737-100/200'
   end
 
-  def test_008_false_positive_without_identity
+  it %{use IDENTITIES} do
+    # false positive without identity
     d = FuzzyMatch.new %w{ foo bar }
-    assert_equal 'bar', d.find('baz')
-  end
+    d.find('baz').must_equal 'bar'
 
-  def test_008_identify_false_positive
     d = FuzzyMatch.new %w{ foo bar }, :identities => [ /ba(.)/ ]
-    assert_equal nil, d.find('baz')
+    d.find('baz').must_be_nil
   end
 
   # TODO this is not very helpful
-  def test_009_blocking
+  it %{use BLOCKINGS} do
     d = FuzzyMatch.new [ 'X' ], :blockings => [ /X/, /Y/ ]
-    assert_equal 'X', d.find('X')
-    assert_equal nil, d.find('A')
+    d.find('X').must_equal 'X'
+    d.find('A').must_be_nil
   end
 
   # TODO this is not very helpful
-  def test_0095_must_match_blocking
+  it %{optionally only attempt matches with records that fit into a blocking} do
     d = FuzzyMatch.new [ 'X' ], :blockings => [ /X/, /Y/ ], :must_match_blocking => true
-    assert_equal 'X', d.find('X')
-    assert_equal nil, d.find('A')
+    d.find('X').must_equal 'X'
+    d.find('A').must_be_nil
 
     d = FuzzyMatch.new [ 'X' ], :blockings => [ /X/, /Y/ ]
-    assert_equal 'X', d.find('X', :must_match_blocking => true)
-    assert_equal nil, d.find('A', :must_match_blocking => true)
+    d.find('X', :must_match_blocking => true).must_equal 'X'
+    d.find('A', :must_match_blocking => true).must_be_nil
   end
 
-  def test_011_free_does_nothing
+  it %{receive the deprecated FuzzyMatch#free method without complaint} do
     d = FuzzyMatch.new %w{ A B }
-    assert_nothing_raised do
-      d.free
-      d.find 'A'
-    end
+    d.free
+    d.find('A').wont_be_nil
   end
 
-  def test_012_find_all
+  it %{return all records in sorted order} do
     d = FuzzyMatch.new [ 'X', 'X22', 'Y', 'Y4' ], :blockings => [ /X/, /Y/ ], :must_match_blocking => true
-    assert_equal ['X', 'X22' ], d.find_all('X')
-    assert_equal [], d.find_all('A')
+    d.find_all('X').must_equal ['X', 'X22' ]
+    d.find_all('A').must_equal []
   end
 
-  def test_013_first_blocking_decides
+  it %{optionally force the first blocking to decide} do
     d = FuzzyMatch.new [ 'Boeing 747', 'Boeing 747SR', 'Boeing ER6' ], :blockings => [ /(boeing \d{3})/i, /boeing/i ]
-    assert_equal [ 'Boeing 747', 'Boeing 747SR', 'Boeing ER6' ], d.find_all('Boeing 747')
+    d.find_all('Boeing 747').must_equal [ 'Boeing 747', 'Boeing 747SR', 'Boeing ER6' ]
 
     d = FuzzyMatch.new [ 'Boeing 747', 'Boeing 747SR', 'Boeing ER6' ], :blockings => [ /(boeing \d{3})/i, /boeing/i ], :first_blocking_decides => true
-    assert_equal [ 'Boeing 747', 'Boeing 747SR' ], d.find_all('Boeing 747')
+    d.find_all('Boeing 747').must_equal [ 'Boeing 747', 'Boeing 747SR' ]
 
     # first_blocking_decides refers to the needle
     d = FuzzyMatch.new [ 'Boeing 747', 'Boeing 747SR', 'Boeing ER6' ], :blockings => [ /(boeing \d{3})/i, /boeing/i ], :first_blocking_decides => true
-    assert_equal ["Boeing ER6", "Boeing 747", "Boeing 747SR"], d.find_all('Boeing ER6')
+    d.find_all('Boeing ER6').must_equal ["Boeing ER6", "Boeing 747", "Boeing 747SR"]
 
     d = FuzzyMatch.new [ 'Boeing 747', 'Boeing 747SR', 'Boeing ER6' ], :blockings => [ /(boeing \d{3})/i, /boeing (7|E)/i, /boeing/i ], :first_blocking_decides => true
-    assert_equal [ 'Boeing ER6' ], d.find_all('Boeing ER6')
+    d.find_all('Boeing ER6').must_equal [ 'Boeing ER6' ]
 
     # or equivalently with an identity
     d = FuzzyMatch.new [ 'Boeing 747', 'Boeing 747SR', 'Boeing ER6' ], :blockings => [ /(boeing \d{3})/i, /boeing/i ], :first_blocking_decides => true, :identities => [ /boeing (7|E)/i ]
-    assert_equal [ 'Boeing ER6' ], d.find_all('Boeing ER6')
-  end
-
-  MyStruct = Struct.new(:one, :two)
-  def test_014_symbol_read_sends_method
-    ab = MyStruct.new('a', 'b')
-    ba = MyStruct.new('b', 'a')
-    haystack = [ab, ba]
-    by_first = FuzzyMatch.new haystack, :read => :one
-    by_last = FuzzyMatch.new haystack, :read => :two
-    assert_equal :one, by_first.read
-    assert_equal :two, by_last.read
-    assert_equal ab, by_first.find('a')
-    assert_equal ab, by_last.find('b')
-    assert_equal ba, by_first.find('b')
-    assert_equal ba, by_last.find('a')
-  end
-
-  def test_015_symbol_read_reads_array
-    ab = ['a', 'b']
-    ba = ['b', 'a']
-    haystack = [ab, ba]
-    by_first = FuzzyMatch.new haystack, :read => 0
-    by_last = FuzzyMatch.new haystack, :read => 1
-    assert_equal ab, by_first.find('a')
-    assert_equal ab, by_last.find('b')
-    assert_equal ba, by_first.find('b')
-    assert_equal ba, by_last.find('a')
-  end
-
-  def test_016_symbol_read_reads_hash
-    ab = { :one => 'a', :two => 'b' }
-    ba = { :one => 'b', :two => 'a' }
-    haystack = [ab, ba]
-    by_first = FuzzyMatch.new haystack, :read => :one
-    by_last = FuzzyMatch.new haystack, :read => :two
-    assert_equal ab, by_first.find('a')
-    assert_equal ab, by_last.find('b')
-    assert_equal ba, by_first.find('b')
-    assert_equal ba, by_last.find('a')
-  end
-
-  def test_017_understands_haystack_reader_option
-    ab = ['a', 'b']
-    ba = ['b', 'a']
-    haystack = [ab, ba]
-    by_first = FuzzyMatch.new haystack, :haystack_reader => 0
-    assert_equal ab, by_first.find('a')
-    assert_equal ba, by_first.find('b')
-  end
-
-  def test_018_no_result_if_best_score_is_zero
-    assert_equal nil, FuzzyMatch.new(['a']).find('b')
-  end
-
-  def test_019_must_match_at_least_one_word
+    d.find_all('Boeing ER6').must_equal [ 'Boeing ER6' ]
+  end
+
+  describe "the :read option" do
+    it %{interpret a Numeric as an array index} do
+      ab = ['a', 'b']
+      ba = ['b', 'a']
+      haystack = [ab, ba]
+      by_first = FuzzyMatch.new haystack, :read => 0
+      by_last = FuzzyMatch.new haystack, :read => 1
+      by_first.find('a').must_equal ab
+      by_last.find('b').must_equal ab
+      by_first.find('b').must_equal ba
+      by_last.find('a').must_equal ba
+    end
+
+    it %{interpret a Symbol, etc. as hash key} do
+      ab = { :one => 'a', :two => 'b' }
+      ba = { :one => 'b', :two => 'a' }
+      haystack = [ab, ba]
+      by_first = FuzzyMatch.new haystack, :read => :one
+      by_last = FuzzyMatch.new haystack, :read => :two
+      by_first.find('a').must_equal ab
+      by_last.find('b').must_equal ab
+      by_first.find('b').must_equal ba
+      by_last.find('a').must_equal ba
+    end
+
+    MyStruct = Struct.new(:one, :two)
+    it %{interpret a Symbol as a method id (if the object responds to it)} do
+      ab = MyStruct.new('a', 'b')
+      ba = MyStruct.new('b', 'a')
+      haystack = [ab, ba]
+      by_first = FuzzyMatch.new haystack, :read => :one
+      by_last = FuzzyMatch.new haystack, :read => :two
+      by_first.read.must_equal :one
+      by_last.read.must_equal :two
+      by_first.find('a').must_equal ab
+      by_last.find('b').must_equal ab
+      by_first.find('b').must_equal ba
+      by_last.find('a').must_equal ba
+    end
+
+    it %{treat the deprecrated :haystack_reader option as an alias} do
+      ab = ['a', 'b']
+      ba = ['b', 'a']
+      haystack = [ab, ba]
+      by_first = FuzzyMatch.new haystack, :haystack_reader => 0
+      by_first.find('a').must_equal ab
+      by_first.find('b').must_equal ba
+    end
+  end
+
+  it %{not return any result if the maximum score is zero} do
+    FuzzyMatch.new(['a']).find('b').must_be_nil
+  end
+
+  it %{optionally require that the matching record share at least one word with the needle} do
     d = FuzzyMatch.new %w{ RATZ CATZ }, :must_match_at_least_one_word => true
-    assert_equal nil, d.find('RITZ')
+    d.find('RITZ').must_be_nil
 
     d = FuzzyMatch.new ["Foo's Bar"], :must_match_at_least_one_word => true
-    assert_equal "Foo's Bar", d.find("Foo's")
-    assert_equal nil, d.find("'s")
-    assert_equal nil, d.find("Foo")
+    d.find("Foo's").must_equal "Foo's Bar"
+    d.find("'s").must_be_nil
+    d.find("Foo").must_be_nil
     
     d = FuzzyMatch.new ["Bolivia, Plurinational State of"], :must_match_at_least_one_word => true
-    assert_equal "Bolivia, Plurinational State of", d.find("Bolivia")
+    d.find("Bolivia").must_equal "Bolivia, Plurinational State of"
   end
 
-  def test_020_stop_words
+  it %{use STOP WORDS} do
     d = FuzzyMatch.new [ 'A HOTEL', 'B HTL' ]
-    assert_equal 'B HTL', d.find('A HTL', :must_match_at_least_one_word => true)
+    d.find('A HTL', :must_match_at_least_one_word => true).must_equal 'B HTL'
 
     d = FuzzyMatch.new [ 'A HOTEL', 'B HTL' ], :must_match_at_least_one_word => true
-    assert_equal 'B HTL', d.find('A HTL')
+    d.find('A HTL').must_equal 'B HTL'
 
     d = FuzzyMatch.new [ 'A HOTEL', 'B HTL' ], :must_match_at_least_one_word => true, :stop_words => [ %r{HO?TE?L} ]
-    assert_equal 'A HOTEL', d.find('A HTL')
+    d.find('A HTL').must_equal 'A HOTEL'
   end
 
-  def test_021_explain_prints_to_stdout
+  it %{print a basic explanation to stdout} do
     require 'stringio'
     capture = StringIO.new
     begin
@@ -187,17 +186,24 @@ class TestFuzzyMatch < Test::Unit::TestCase
       $stdout = old_stdout
     end
     capture.rewind
-    assert capture.read.include?('CATZ')
+    capture.read.must_include 'CATZ'
   end
 
-  def test_022_compare_words_with_words
+  it %{not be fooled by substrings (but rather compare whole words to whole words)} do
     d = FuzzyMatch.new [ 'PENINSULA HOTELS' ], :must_match_at_least_one_word => true
-    assert_equal nil, d.find('DOLCE LA HULPE BXL FI')
+    d.find('DOLCE LA HULPE BXL FI').must_be_nil
   end
 
-  def test_023_must_match_at_least_one_word_is_case_insensitive
+  it %{not be case-sensitive when checking for sharing of words} do
     d = FuzzyMatch.new [ 'A', 'B' ]
-    assert_equal 'A', d.find('a', :must_match_at_least_one_word => true)
+    d.find('a', :must_match_at_least_one_word => true).must_equal 'A'
   end
 
+  it %{defaults to a pure-ruby engine, but also has amatch} do
+    if defined?($testing_amatch) and $testing_amatch
+      FuzzyMatch.engine.must_equal :amatch
+    else
+      FuzzyMatch.engine.must_equal :pure_ruby
+    end
+  end
 end

data/test/test_fuzzy_match_convoluted.rb.disabled

@@ -4,7 +4,7 @@ require 'shoulda'
 
 $log = false
 
-class TestFuzzyMatchConvoluted < Test::Unit::TestCase
+class TestFuzzyMatchConvoluted < MiniTest::Spec
   def setup
     clear_ltd
     

data/test/test_identity.rb

@@ -1,38 +1,36 @@
 require 'helper'
 
-class TestIdentity < Test::Unit::TestCase
-  def test_001_identical
+class TestIdentity < MiniTest::Spec
+  it %{determines whether two records COULD be identical} do
     i = FuzzyMatch::Identity.new %r{(A)[ ]*(\d)}
-    assert_equal true, i.identical?('A1', 'A     1foobar')
+    i.identical?('A1', 'A     1foobar').must_equal true
   end
   
-  def test_002_certainly_different
+  it %{determines that two records MUST NOT be identical} do
     i = FuzzyMatch::Identity.new %r{(A)[ ]*(\d)}
-    assert_equal false, i.identical?('A1', 'A     2foobar')    
+    i.identical?('A1', 'A     2foobar').must_equal false
   end
   
-  def test_003_no_information_ie_possible_identical
+  it %{returns nil indicating no information} do
     i = FuzzyMatch::Identity.new %r{(A)[ ]*(\d)}
-    assert_equal nil, i.identical?('B1', 'A     2foobar')    
+    i.identical?('B1', 'A     2foobar').must_equal nil
   end
 
-  def test_004_regexp
+  it %{can be initialized with a regexp} do
     i = FuzzyMatch::Identity.new %r{\A\\?/(.*)etc/mysql\$$}
-    assert_equal %r{\A\\?/(.*)etc/mysql\$$}, i.regexp
+    i.regexp.must_equal %r{\A\\?/(.*)etc/mysql\$$}
   end
   
-  def test_005_regexp_from_string
+  it %{can be initialized from a string (via to_regexp gem)} do
     i = FuzzyMatch::Identity.new '%r{\A\\\?/(.*)etc/mysql\$$}'
-    assert_equal %r{\A\\?/(.*)etc/mysql\$$}, i.regexp
-  end
-  
-  def test_006_regexp_from_string_using_slash_delim
+    i.regexp.must_equal %r{\A\\?/(.*)etc/mysql\$$}
+
     i = FuzzyMatch::Identity.new '/\A\\\?\/(.*)etc\/mysql\$$/'
-    assert_equal %r{\A\\?/(.*)etc/mysql\$$}, i.regexp
+    i.regexp.must_equal %r{\A\\?/(.*)etc/mysql\$$}
   end
   
-  def test_007_accepts_case_insensitivity
+  it %{embraces case insensitivity} do
     i = FuzzyMatch::Identity.new %r{(A)[ ]*(\d)}i
-    assert_equal true, i.identical?('A1', 'a     1foobar')
+    i.identical?('A1', 'a     1foobar').must_equal true
   end
 end

data/test/test_normalizer.rb

@@ -1,10 +1,10 @@
 require 'helper'
 
-class TestNormalizer < Test::Unit::TestCase
-  def test_001_apply
+class TestNormalizer < MiniTest::Spec
+  it %{applies itself to a string argument} do
     t = FuzzyMatch::Normalizer.new %r{(Ford )[ ]*(F)[\- ]*(\d\d\d)}i
-    assert_equal 'Ford F350', t.apply('Ford F-350')
-    assert_equal 'Ford F150', t.apply('Ford F150')
-    assert_equal 'Ford F350', t.apply('Ford F 350')
+    t.apply('Ford F-350').must_equal 'Ford F350'
+    t.apply('Ford F150').must_equal 'Ford F150'
+    t.apply('Ford F 350').must_equal 'Ford F350'
   end
 end

data/test/test_wrapper.rb

@@ -1,26 +1,26 @@
 require 'helper'
 
-class TestWrapper < Test::Unit::TestCase
-  def test_001_apostrophe_s_is_not_a_word
+class TestWrapper < MiniTest::Spec
+  it %{does not treat "'s" as a word} do
     assert_split ["foo's", "bar"], "Foo's Bar"
   end
   
-  def test_002_bolivia_comma_is_just_bolivia
+  it %{treats "bolivia," as just "bolivia"} do
     assert_split ["bolivia", "plurinational", "state"], "Bolivia, Plurinational State"
   end
   
-  def test_003_hyphenated_words_are_not_split_up
+  it %{does not split up hyphenated words} do
     assert_split ['north-west'], "north-west"
   end
   
-  def test_004_as_expected
+  it %{splits up words as expected} do
     assert_split ['the', 'quick', "fox's", 'mouth', 'is', 'always', 'full'], "the quick fox's mouth -- is always full."
   end
   
   private
   
   def assert_split(ary, str)
-    assert_equal ary, FuzzyMatch::Wrapper.new(null_fuzzy_match, str, true).words
+    FuzzyMatch::Wrapper.new(null_fuzzy_match, str, true).words.must_equal ary
   end
   
   def null_fuzzy_match

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: fuzzy_match
 version: !ruby/object:Gem::Version
-  version: 1.2.2
+  version: 1.3.0
   prerelease: 
 platform: ruby
 authors:
@@ -9,11 +9,11 @@ authors:
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2012-01-27 00:00:00.000000000Z
+date: 2012-02-07 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
-  name: shoulda
-  requirement: &2170570400 !ruby/object:Gem::Requirement
+  name: minitest
+  requirement: &2152246040 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ! '>='
@@ -21,21 +21,10 @@ dependencies:
         version: '0'
   type: :development
   prerelease: false
-  version_requirements: *2170570400
-- !ruby/object:Gem::Dependency
-  name: remote_table
-  requirement: &2170569000 !ruby/object:Gem::Requirement
-    none: false
-    requirements:
-    - - ! '>='
-      - !ruby/object:Gem::Version
-        version: '0'
-  type: :development
-  prerelease: false
-  version_requirements: *2170569000
+  version_requirements: *2152246040
 - !ruby/object:Gem::Dependency
   name: activerecord
-  requirement: &2170568220 !ruby/object:Gem::Requirement
+  requirement: &2152244820 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ! '>='
@@ -43,10 +32,10 @@ dependencies:
         version: '3'
   type: :development
   prerelease: false
-  version_requirements: *2170568220
+  version_requirements: *2152244820
 - !ruby/object:Gem::Dependency
-  name: mysql
-  requirement: &2170567580 !ruby/object:Gem::Requirement
+  name: mysql2
+  requirement: &2152260680 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ! '>='
@@ -54,10 +43,10 @@ dependencies:
         version: '0'
   type: :development
   prerelease: false
-  version_requirements: *2170567580
+  version_requirements: *2152260680
 - !ruby/object:Gem::Dependency
   name: cohort_scope
-  requirement: &2170566940 !ruby/object:Gem::Requirement
+  requirement: &2152260120 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ! '>='
@@ -65,10 +54,10 @@ dependencies:
         version: '0'
   type: :development
   prerelease: false
-  version_requirements: *2170566940
+  version_requirements: *2152260120
 - !ruby/object:Gem::Dependency
   name: weighted_average
-  requirement: &2170566060 !ruby/object:Gem::Requirement
+  requirement: &2152259560 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ! '>='
@@ -76,10 +65,10 @@ dependencies:
         version: '0'
   type: :development
   prerelease: false
-  version_requirements: *2170566060
+  version_requirements: *2152259560
 - !ruby/object:Gem::Dependency
   name: rake
-  requirement: &2170558600 !ruby/object:Gem::Requirement
+  requirement: &2152258980 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ! '>='
@@ -87,10 +76,10 @@ dependencies:
         version: '0'
   type: :development
   prerelease: false
-  version_requirements: *2170558600
+  version_requirements: *2152258980
 - !ruby/object:Gem::Dependency
   name: yard
-  requirement: &2170556920 !ruby/object:Gem::Requirement
+  requirement: &2152258500 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ! '>='
@@ -98,10 +87,10 @@ dependencies:
         version: '0'
   type: :development
   prerelease: false
-  version_requirements: *2170556920
+  version_requirements: *2152258500
 - !ruby/object:Gem::Dependency
   name: amatch
-  requirement: &2170555740 !ruby/object:Gem::Requirement
+  requirement: &2152258020 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ! '>='
@@ -109,10 +98,10 @@ dependencies:
         version: '0'
   type: :development
   prerelease: false
-  version_requirements: *2170555740
+  version_requirements: *2152258020
 - !ruby/object:Gem::Dependency
   name: activesupport
-  requirement: &2170555200 !ruby/object:Gem::Requirement
+  requirement: &2152257320 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ! '>='
@@ -120,10 +109,10 @@ dependencies:
         version: '3'
   type: :runtime
   prerelease: false
-  version_requirements: *2170555200
+  version_requirements: *2152257320
 - !ruby/object:Gem::Dependency
   name: to_regexp
-  requirement: &2170554500 !ruby/object:Gem::Requirement
+  requirement: &2152256720 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ! '>='
@@ -131,7 +120,7 @@ dependencies:
         version: 0.0.3
   type: :runtime
   prerelease: false
-  version_requirements: *2170554500
+  version_requirements: *2152256720
 description: Find a needle in a haystack using string similarity and (optionally)
   regexp rules. Replaces loose_tight_dictionary.
 email:
@@ -173,11 +162,14 @@ files:
 - lib/fuzzy_match/normalizer.rb
 - lib/fuzzy_match/result.rb
 - lib/fuzzy_match/score.rb
+- lib/fuzzy_match/score/amatch.rb
+- lib/fuzzy_match/score/pure_ruby.rb
 - lib/fuzzy_match/similarity.rb
 - lib/fuzzy_match/stop_word.rb
 - lib/fuzzy_match/version.rb
 - lib/fuzzy_match/wrapper.rb
 - test/helper.rb
+- test/test_amatch.rb
 - test/test_blocking.rb
 - test/test_cache.rb
 - test/test_fuzzy_match.rb
@@ -205,13 +197,14 @@ required_rubygems_version: !ruby/object:Gem::Requirement
       version: '0'
 requirements: []
 rubyforge_project: fuzzy_match
-rubygems_version: 1.8.10
+rubygems_version: 1.8.15
 signing_key: 
 specification_version: 3
 summary: Find a needle in a haystack using string similarity and (optionally) regexp
   rules. Replaces loose_tight_dictionary.
 test_files:
 - test/helper.rb
+- test/test_amatch.rb
 - test/test_blocking.rb
 - test/test_cache.rb
 - test/test_fuzzy_match.rb