fuzzy_match 1.2.1 → 1.2.2

data/README.markdown

@@ -1,35 +1,44 @@
 # fuzzy_match
 
-Find a needle in a haystack based on string similarity (using the Pair Distance algorithm and Levenshtein distance) and regular expressions.
+Find a needle in a haystack based on string similarity and regular expression rules.
 
 Replaces [`loose_tight_dictionary`](https://github.com/seamusabshere/loose_tight_dictionary) because that was a confusing name.
 
 ## Quickstart
 
     >> require 'fuzzy_match'
-    => true 
-    >> FuzzyMatch.new(['seamus', 'andy', 'ben']).find('Shamus')
+    => true
+    >> matcher = FuzzyMatch.new(['seamus', 'andy', 'ben'])
+    => #<FuzzyMatch: [...]>
+    >> matcher.find('Shamus')
     => "seamus"
 
 ## Default matching (string similarity)
 
-If you configure nothing else, string similarity matching is used. That's why we call it fuzzy matching.
+At the core, and even if you configure nothing else, string similarity (calculated by "pair distance" aka Dice's) is used to compare records.
 
-The algorithm is [Dice's Coefficient](http://en.wikipedia.org/wiki/Dice's_coefficient) (aka Pair Distance) because it seemed to work better than Jaro Winkler, etc.
+You can tell `FuzzyMatch` what field or method to use via the `:read` option... for example, let's say you want to match a `Country` object like `#<Country name:"Uruguay" iso_3166_code:"UY">`
 
-## Rules (regular expressions)
+    >> matcher = FuzzyMatch.new(Country.all, :read => :name)  # Country#name will be called when comparing
+    => #<FuzzyMatch: [...]>
+    >> matcher.find('youruguay')
+    => #<Country name:"Uruguay" iso_3166_code:"UY">            # the matcher returns a Country object
 
-You can improve the default matchings with rules, which are generally regular expressions.
+## Optional rules (regular expressions)
 
-    >> require 'fuzzy_match'
-    => true 
-    >> matcher = FuzzyMatch.new(['Ford F-150', 'Ford F-250', 'GMC 1500', 'GMC 2500'], :blockings => [ /ford/i, /gmc/i ], :normalizers => [ /K(\d500)/i ], :identities => [ /(f)-?(\d\d\d)/i ])
+You can improve the default matchings with rules. There are 4 different kinds of rules. Each rule is a regular expression. Depending on the kind of rule, the results of running the regular expression are used for a particular purpose.
+
+We suggest that you **first try without any rules** and only define them to improve matching, prevent false positives, etc.
+
+    >> matcher = FuzzyMatch.new(['Ford F-150', 'Ford F-250', 'GMC 1500', 'GMC 2500'], :blockings => [ /ford/i, /gmc/i ], :normalizers => [ /K(\d500)/i ], :identities => [ /(f)-?(\d50)/i ])
     => #<FuzzyMatch: [...]> 
     >> matcher.find('fordf250')
     => "Ford F-250" 
     >> matcher.find('gmc truck k1500')
     => "GMC 1500" 
 
+For identities and normalizers (see below), **only the captures are used.** For example, `/(f)-?(\d50)/i` captures the "F" and the "250" but ignores the dash. So place your parentheses carefully! Blockings work the same way, except that if you don't have any captures, a simple match will pass.
+
 ### Blockings
 
 Group records together.
@@ -40,13 +49,13 @@ Setting a blocking of `/Airbus/` ensures that strings containing "Airbus" will o
 
 Strip strings down to the essentials.
 
-Adding a normalizer like `/(boeing).*(7\d\d)/i` will cause "BOEING COMPANY 747" and "boeing747" to be scored as if they were "BOEING 747" and "boeing 747", respectively. See also "Case sensitivity" below.
+Adding a normalizer like `/(boeing).*(7\d\d)/i` will cause "BOEING COMPANY 747" and "boeing747" to be normalized to "BOEING 747" and "boeing 747", respectively. Since things are generally downcased before they are compared, these would be an exact match.
 
 ### Identities
 
 Prevent impossible matches.
 
-Adding an identity like `/(F)\-?(\d50)/` ensures that "Ford F-150" and "Ford F-250" never match.
+Adding an identity like `/(f)-?(\d50)/i` ensures that "Ford F-150" and "Ford F-250" never match.
 
 ### Stop words
 
@@ -58,37 +67,48 @@ Adding a stop word like `THE` ensures that it is not taken into account when com
 
 * `read`: how to interpret each record in the 'haystack', either a Proc or a symbol
 * `must_match_blocking`: don't return a match unless the needle fits into one of the blockings you specified
-* `must_match_at_least_one_word`: don't return a match unless the needle shares at least one word with the match
+* `must_match_at_least_one_word`: don't return a match unless the needle shares at least one word with the match. Note that "Foo's" is treated like one word (so that it won't match "'s") and "Bolivia," is treated as just "bolivia"
 * `first_blocking_decides`: force records into the first blocking they match, rather than choosing a blocking that will give them a higher score
 * `gather_last_result`: enable `last_result`
 
-### `:read`
-
-So, what if your needle is a string like `youruguay` and your haystack is full of `Country` objects like `<Country name:"Uruguay">`?
-
-    >> FuzzyMatch.new(Country.all, :read => :name).find('youruguay')
-    => <Country name:"Uruguay">
-
 ## Case sensitivity
 
 String similarity is case-insensitive. Everything is downcased before scoring. This is a change from previous versions.
 
 Be careful when trying to use case-sensitivity in your rules; in general, things are downcased before comparing.
 
-## Dice's coefficient edge case
+## String similarity algorithm
+
+The algorithm is [Dice's Coefficient](http://en.wikipedia.org/wiki/Dice's_coefficient) (aka Pair Distance) because it seemed to work better than Longest Substring, Hamming, Jaro Winkler, Levenshtein (although see edge case below) etc.
+
+Here's a great explanation copied from [the wikipedia entry](http://en.wikipedia.org/wiki/Dice%27s_coefficient):
+
+    to calculate the similarity between:
+
+        night
+        nacht
+
+    We would find the set of bigrams in each word:
+
+        {ni,ig,gh,ht}
+        {na,ac,ch,ht}
+
+    Each set has four elements, and the intersection of these two sets has only one element: ht.
+
+    Inserting these numbers into the formula, we calculate, s = (2 · 1) / (4 + 4) = 0.25.
+
+### Edge case: when Dice's fails, use Levenshtein
 
 In edge cases where Dice's finds that two strings are equally similar to a third string, then Levenshtein distance is used. For example, pair distance considers "RATZ" and "CATZ" to be equally similar to "RITZ" so we invoke Levenshtein.
 
-    >> require 'amatch'
-    => true 
     >> 'RITZ'.pair_distance_similar 'RATZ'
     => 0.3333333333333333 
-    >> 'RITZ'.pair_distance_similar 'CATZ'  # <-- pair distance can't tell the difference, so we fall back to levenshtein...
-    => 0.3333333333333333 
+    >> 'RITZ'.pair_distance_similar 'CATZ'
+    => 0.3333333333333333                   # pair distance can't tell the difference, so we fall back to levenshtein...
     >> 'RITZ'.levenshtein_similar 'RATZ'
     => 0.75 
-    >> 'RITZ'.levenshtein_similar 'CATZ'    # <-- which properly shows that RATZ should win
-    => 0.5 
+    >> 'RITZ'.levenshtein_similar 'CATZ'
+    => 0.5                                  # which properly shows that RATZ should win
 
 ## Production use
 

data/Rakefile

@@ -11,8 +11,4 @@ end
 task :default => :test
 
 require 'yard'
-require File.expand_path('../lib/fuzzy_match/version.rb', __FILE__)
-YARD::Rake::YardocTask.new do |t|
-  t.files   = ['lib/**/*.rb', 'README.markdown']   # optional
-  # t.options = ['--any', '--extra', '--opts'] # optional
-end
+YARD::Rake::YardocTask.new

data/fuzzy_match.gemspec

@@ -26,7 +26,9 @@ Gem::Specification.new do |s|
   s.add_development_dependency 'cohort_scope'
   s.add_development_dependency 'weighted_average'
   s.add_development_dependency 'rake'
-  # s.add_development_dependency 'amatch'
+  s.add_development_dependency 'yard'
+  s.add_development_dependency 'amatch'
+
   s.add_runtime_dependency 'activesupport', '>=3'
   s.add_runtime_dependency 'to_regexp', '>=0.0.3'
 end

data/lib/fuzzy_match.rb

@@ -262,7 +262,7 @@ EOS
     last_result.explain
   end
 
-  # DEPRECATED - doesn't do anything
+  # DEPRECATED
   def free
   end
 end

data/lib/fuzzy_match/normalizer.rb

@@ -22,7 +22,7 @@ class FuzzyMatch
     end
     
     def inspect
-      "#<Normalizer regexp=#{regexp.inspect}>"
+      "#<FuzzyMatch::Normalizer regexp=#{regexp.inspect}>"
     end
   end
 end

data/lib/fuzzy_match/score.rb

@@ -1,22 +1,19 @@
-begin
-  require 'amatch'
-rescue ::LoadError
-  # using native ruby similarity scoring
-end
-
 class FuzzyMatch
   class Score
-    attr_reader :str1, :str2
+    extend ::ActiveSupport::Memoizable
+    
+    attr_reader :str1
+    attr_reader :str2
 
     def initialize(str1, str2)
       @str1 = str1.downcase
       @str2 = str2.downcase
     end
-    
+
     def inspect
-      %{#<Score: dices_coefficient=#{dices_coefficient_similar} levenshtein=#{levenshtein_similar}>}
+      %{#<FuzzyMatch::Score: str1=#{str1.inspect} str2=#{str2.inspect} dices_coefficient_similar=#{dices_coefficient_similar} levenshtein_similar=#{levenshtein_similar}>}
     end
-    
+
     def <=>(other)
       by_dices_coefficient = (dices_coefficient_similar <=> other.dices_coefficient_similar)
       if by_dices_coefficient == 0
@@ -25,23 +22,26 @@ class FuzzyMatch
         by_dices_coefficient
       end
     end
-        
-    def utf8?
-      (defined?(::Encoding) ? str1.encoding.to_s : $KCODE).downcase.start_with?('u')
-    end
-    
+
     if defined?(::Amatch)
-      
+
       def dices_coefficient_similar
+        if str1 == str2
+          return 1.0
+        elsif str1.length == 1 and str2.length == 1
+          return 0.0
+        end
         str1.pair_distance_similar str2
       end
-      
+      memoize :dices_coefficient_similar
+
       def levenshtein_similar
         str1.levenshtein_similar str2
       end
-      
+      memoize :levenshtein_similar
+
     else
-      
+
       SPACE = ' '
       # http://stackoverflow.com/questions/653157/a-better-similarity-ranking-algorithm-for-variable-length-strings
       def dices_coefficient_similar
@@ -60,19 +60,26 @@ class FuzzyMatch
         end.reject do |pair|
           pair.include? SPACE
         end
-        union = pairs1.size + pairs2.size 
-        intersection = 0 
-        pairs1.each do |p1| 
-          0.upto(pairs2.size-1) do |i| 
-            if p1 == pairs2[i] 
-              intersection += 1 
-              pairs2.slice!(i) 
-              break 
-            end 
-          end 
-        end 
+        union = pairs1.size + pairs2.size
+        intersection = 0
+        pairs1.each do |p1|
+          0.upto(pairs2.size-1) do |i|
+            if p1 == pairs2[i]
+              intersection += 1
+              pairs2.slice!(i)
+              break
+            end
+          end
+        end
         (2.0 * intersection) / union
       end
+      memoize :dices_coefficient_similar
+
+      # this seems like it would slow things down
+      def utf8?
+        (defined?(::Encoding) ? str1.encoding.to_s : $KCODE).downcase.start_with?('u')
+      end
+      memoize :utf8?
 
       # extracted/adapted from the text gem version 1.0.2
       # normalization added for utf-8 strings
@@ -114,12 +121,8 @@ class FuzzyMatch
         # }
         1.0 - x.to_f / [n, m].max
       end
-      
+      memoize :levenshtein_similar
+
     end
-    
-    extend ::ActiveSupport::Memoizable
-    memoize :dices_coefficient_similar
-    memoize :levenshtein_similar
-    memoize :utf8?
   end
 end

data/lib/fuzzy_match/similarity.rb

@@ -47,7 +47,7 @@ class FuzzyMatch
     end
     
     def inspect
-      %{#<Similarity "#{wrapper2.render}"=>"#{best_wrapper2_variant}" versus "#{wrapper1.render}"=>"#{best_wrapper1_variant}" original_weight=#{"%0.5f" % original_weight} best_score=#{best_score.inspect}>}
+      %{#<FuzzyMatch::Similarity #{wrapper2.render.inspect}=>#{best_wrapper2_variant.inspect} versus #{wrapper1.render.inspect}=>#{best_wrapper1_variant.inspect} original_weight=#{"%0.5f" % original_weight} best_score=#{best_score.inspect}>}
     end
   end
 end

data/lib/fuzzy_match/stop_word.rb

@@ -13,7 +13,7 @@ class FuzzyMatch
     end
     
     def inspect
-      "#<StopWord regexp=#{regexp.inspect}>"
+      "#<FuzzyMatch::StopWord regexp=#{regexp.inspect}>"
     end
   end
 end

data/lib/fuzzy_match/version.rb

@@ -1,3 +1,3 @@
 class FuzzyMatch
-  VERSION = '1.2.1'
+  VERSION = '1.2.2'
 end

data/lib/fuzzy_match/wrapper.rb

@@ -13,7 +13,7 @@ class FuzzyMatch
     end
 
     def inspect
-      "#<Wrapper render=#{render} variants=#{variants.length}>"
+      "#<FuzzyMatch::Wrapper render=#{render.inspect} variants=#{variants.length}>"
     end
     
     def read
@@ -47,8 +47,10 @@ class FuzzyMatch
 
     alias :to_str :render
 
-    # "Foo's Bar" should be treated as [ "Foo's", "Bar" ], so we don't use traditional regexp word boundaries (\b)
-    WORD_BOUNDARY = %r{\s+}
+    # "Foo's" is one word
+    # "North-west" is just one word
+    # "Bolivia," is just Bolivia
+    WORD_BOUNDARY = %r{\W*(?:\s+|$)}
     def words
       @words ||= render.downcase.split(WORD_BOUNDARY)
     end

data/test/helper.rb

@@ -4,6 +4,9 @@ Bundler.setup
 require 'test/unit'
 require 'stringio'
 require 'remote_table'
+if ENV['AMATCH'] == 'true'
+  require 'amatch'
+end
 $LOAD_PATH.unshift(File.dirname(__FILE__))
 $LOAD_PATH.unshift(File.join(File.dirname(__FILE__), '..', 'lib'))
 require 'fuzzy_match'

data/test/test_fuzzy_match.rb

@@ -156,8 +156,12 @@ class TestFuzzyMatch < Test::Unit::TestCase
     assert_equal nil, d.find('RITZ')
 
     d = FuzzyMatch.new ["Foo's Bar"], :must_match_at_least_one_word => true
-    assert_equal nil, d.find("Jacob's")
     assert_equal "Foo's Bar", d.find("Foo's")
+    assert_equal nil, d.find("'s")
+    assert_equal nil, d.find("Foo")
+    
+    d = FuzzyMatch.new ["Bolivia, Plurinational State of"], :must_match_at_least_one_word => true
+    assert_equal "Bolivia, Plurinational State of", d.find("Bolivia")
   end
 
   def test_020_stop_words

data/test/test_wrapper.rb

@@ -0,0 +1,29 @@
+require 'helper'
+
+class TestWrapper < Test::Unit::TestCase
+  def test_001_apostrophe_s_is_not_a_word
+    assert_split ["foo's", "bar"], "Foo's Bar"
+  end
+  
+  def test_002_bolivia_comma_is_just_bolivia
+    assert_split ["bolivia", "plurinational", "state"], "Bolivia, Plurinational State"
+  end
+  
+  def test_003_hyphenated_words_are_not_split_up
+    assert_split ['north-west'], "north-west"
+  end
+  
+  def test_004_as_expected
+    assert_split ['the', 'quick', "fox's", 'mouth', 'is', 'always', 'full'], "the quick fox's mouth -- is always full."
+  end
+  
+  private
+  
+  def assert_split(ary, str)
+    assert_equal ary, FuzzyMatch::Wrapper.new(null_fuzzy_match, str, true).words
+  end
+  
+  def null_fuzzy_match
+    FuzzyMatch.new []
+  end
+end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: fuzzy_match
 version: !ruby/object:Gem::Version
-  version: 1.2.1
+  version: 1.2.2
   prerelease: 
 platform: ruby
 authors:
@@ -9,11 +9,11 @@ authors:
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2012-01-18 00:00:00.000000000Z
+date: 2012-01-27 00:00:00.000000000Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: shoulda
-  requirement: &2177380220 !ruby/object:Gem::Requirement
+  requirement: &2170570400 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ! '>='
@@ -21,10 +21,10 @@ dependencies:
         version: '0'
   type: :development
   prerelease: false
-  version_requirements: *2177380220
+  version_requirements: *2170570400
 - !ruby/object:Gem::Dependency
   name: remote_table
-  requirement: &2177379700 !ruby/object:Gem::Requirement
+  requirement: &2170569000 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ! '>='
@@ -32,10 +32,10 @@ dependencies:
         version: '0'
   type: :development
   prerelease: false
-  version_requirements: *2177379700
+  version_requirements: *2170569000
 - !ruby/object:Gem::Dependency
   name: activerecord
-  requirement: &2177379100 !ruby/object:Gem::Requirement
+  requirement: &2170568220 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ! '>='
@@ -43,10 +43,10 @@ dependencies:
         version: '3'
   type: :development
   prerelease: false
-  version_requirements: *2177379100
+  version_requirements: *2170568220
 - !ruby/object:Gem::Dependency
   name: mysql
-  requirement: &2177378440 !ruby/object:Gem::Requirement
+  requirement: &2170567580 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ! '>='
@@ -54,10 +54,10 @@ dependencies:
         version: '0'
   type: :development
   prerelease: false
-  version_requirements: *2177378440
+  version_requirements: *2170567580
 - !ruby/object:Gem::Dependency
   name: cohort_scope
-  requirement: &2177377600 !ruby/object:Gem::Requirement
+  requirement: &2170566940 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ! '>='
@@ -65,10 +65,10 @@ dependencies:
         version: '0'
   type: :development
   prerelease: false
-  version_requirements: *2177377600
+  version_requirements: *2170566940
 - !ruby/object:Gem::Dependency
   name: weighted_average
-  requirement: &2177377020 !ruby/object:Gem::Requirement
+  requirement: &2170566060 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ! '>='
@@ -76,10 +76,10 @@ dependencies:
         version: '0'
   type: :development
   prerelease: false
-  version_requirements: *2177377020
+  version_requirements: *2170566060
 - !ruby/object:Gem::Dependency
   name: rake
-  requirement: &2177376420 !ruby/object:Gem::Requirement
+  requirement: &2170558600 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ! '>='
@@ -87,10 +87,32 @@ dependencies:
         version: '0'
   type: :development
   prerelease: false
-  version_requirements: *2177376420
+  version_requirements: *2170558600
+- !ruby/object:Gem::Dependency
+  name: yard
+  requirement: &2170556920 !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: *2170556920
+- !ruby/object:Gem::Dependency
+  name: amatch
+  requirement: &2170555740 !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: *2170555740
 - !ruby/object:Gem::Dependency
   name: activesupport
-  requirement: &2177375240 !ruby/object:Gem::Requirement
+  requirement: &2170555200 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ! '>='
@@ -98,10 +120,10 @@ dependencies:
         version: '3'
   type: :runtime
   prerelease: false
-  version_requirements: *2177375240
+  version_requirements: *2170555200
 - !ruby/object:Gem::Dependency
   name: to_regexp
-  requirement: &2177374500 !ruby/object:Gem::Requirement
+  requirement: &2170554500 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ! '>='
@@ -109,7 +131,7 @@ dependencies:
         version: 0.0.3
   type: :runtime
   prerelease: false
-  version_requirements: *2177374500
+  version_requirements: *2170554500
 description: Find a needle in a haystack using string similarity and (optionally)
   regexp rules. Replaces loose_tight_dictionary.
 email:
@@ -162,6 +184,7 @@ files:
 - test/test_fuzzy_match_convoluted.rb.disabled
 - test/test_identity.rb
 - test/test_normalizer.rb
+- test/test_wrapper.rb
 homepage: https://github.com/seamusabshere/fuzzy_match
 licenses: []
 post_install_message: 
@@ -195,4 +218,5 @@ test_files:
 - test/test_fuzzy_match_convoluted.rb.disabled
 - test/test_identity.rb
 - test/test_normalizer.rb
+- test/test_wrapper.rb
 has_rdoc: