jaro_winkler 1.2.8 → 1.3.0.beta

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 13eaf9c47df66ba5c9883611ee7a1c8468cc9e7a
-  data.tar.gz: 7c43db047cfb1aaade4c0d6f2c907ed19f8a79ab
+  metadata.gz: eb7b3cd6c1b5f5a137879bd1e3bd8237a85fa081
+  data.tar.gz: 0d9d7fa4893a23b36cb04de1bf5a4c37f739e400
 SHA512:
-  metadata.gz: 57421be340741b44879c3104689363a7ca2897014a6e9cd0c3fcaa524b29abf1a5db370cab72e4de0299f26ca2e14357ce24f7b8b7cafce97ca1527c27c46798
-  data.tar.gz: 8db803506546b4a99dd7e267bc561c90664282b4c8dc253f22695a0a8cc55df64e559be3c7e68cb26e05fd4ca08626481a82c82128f8616e0a2b378b29012413
+  metadata.gz: fb8b721fa6c615f2ac972be2469342a3e4f3d53b428be34248340021aedaafeee5f302fc393e132cd4fca4be58d1de40e9ae4b814b4635f37afea559c7ee762b
+  data.tar.gz: 4b5cf2339c8b07f309ae0dec2b7e678b96b17c4ae1f7a62e2a71a6f0e373008af2f45640cc1948961578e0c96f93ec63633fcff2bc9a1e341ec16003a376ef03

data/README.md

@@ -1,6 +1,6 @@
 # About
 
-It's a implementation of [Jaro-Winkler distance](http://en.wikipedia.org/wiki/Jaro%E2%80%93Winkler_distance) algorithm, it uses C extension and will fallback to pure Ruby version in JRuby. Both implementation supports UTF-8 string.
+It's an implementation of [Jaro-Winkler distance](http://en.wikipedia.org/wiki/Jaro%E2%80%93Winkler_distance) algorithm, it uses C extension and will fallback to pure Ruby version in JRuby. Both of them supports UTF-8 string.
 
 # Installation
 
@@ -24,8 +24,6 @@ JaroWinkler.c_distance "MARTHA", "MARHTA" # C extension
 JaroWinkler.r_distance "MARTHA", "MARHTA" # Pure Ruby
 ```
 
-**Both implementations support UTF-8 string.**
-
 ## Options
 
 Name        | Type    | Default | Note
@@ -33,35 +31,47 @@ Name        | Type    | Default | Note
 ignore_case | boolean | false   | All lower case characters are converted to upper case prior to the comparison.
 weight      | number  | 0.1     | A constant scaling factor for how much the score is adjusted upwards for having common prefixes.
 threshold   | number  | 0.7     | The prefix bonus is only added when the compared strings have a Jaro distance above the threshold.
+adj_table   | boolean | false   | The option is used to give partial credit for characters that may be errors due to known phonetic or character recognition errors. A typical example is to match the letter "O" with the number "0".
+
+## Default Adjusting Table
+
+```
+['A', 'E'], ['A', 'I'], ['A', 'O'], ['A', 'U'], ['B', 'V'], ['E', 'I'], ['E', 'O'], ['E', 'U'], ['I', 'O'], ['I', 'U'],
+['O', 'U'], ['I', 'Y'], ['E', 'Y'], ['C', 'G'], ['E', 'F'], ['W', 'U'], ['W', 'V'], ['X', 'K'], ['S', 'Z'], ['X', 'S'],
+['Q', 'C'], ['U', 'V'], ['M', 'N'], ['L', 'I'], ['Q', 'O'], ['P', 'R'], ['I', 'J'], ['2', 'Z'], ['5', 'S'], ['8', 'B'],
+['1', 'I'], ['1', 'L'], ['0', 'O'], ['0', 'Q'], ['C', 'K'], ['G', 'J'], ['E', ' '], ['Y', ' '], ['S', ' ']
+```
 
 # Why This?
 
-There is also another gem named [fuzzy-string-match](https://github.com/kiyoka/fuzzy-string-match), it uses the same algorithm and both provides C and Ruby implementation.
+There is also another similar gem named [fuzzy-string-match](https://github.com/kiyoka/fuzzy-string-match) which both provides C and Ruby version as well.
 
-I reinvent this wheel because of the naming in `fuzzy-string-match` such as `getDistance` breaks convention, and some weird code like `a1 = s1.split( // )` (`s1.chars` could be better), furthermore, it's bugged (see table below).
+I reinvent this wheel because of the naming in `fuzzy-string-match` such as `getDistance` breaks convention, and some weird code like `a1 = s1.split( // )` (`s1.chars` could be better), furthermore, it's bugged (see tables below).
 
 # Compare with other gems
 
-             | jaro_winkler | fuzzystringmatch | hotwater | amatch
------------- | ------------ | ---------------- | -------- | ------
-UTF-8 Suport | Yes          | Pure Ruby only   |          |
-Native       | Yes          | Yes              | Yes      | Yes
-Pure Ruby    | Yes          | Yes              |          |
-Speed        | Medium       | Fast             | Medium   | Low
-Bug Found    |              | Yes              |          | Yes
+                | jaro_winkler | fuzzystringmatch | hotwater    | amatch
+--------------- | ------------ | ---------------- | --------    | ------
+UTF-8 Suport    | **Yes**      | Pure Ruby only   | No          | No
+Windows Support | **Yes**      |                  | No          | **Yes**
+Adjusting Table | **Yes**      | No               | No          | No
+Native          | **Yes**      | **Yes**          | **Yes**     | **Yes**
+Pure Ruby       | **Yes**      | **Yes**          | No          | No
+Speed           | Medium       | Fast             | Medium      | Slow
+Bug Found       | **Not Yet**  | Yes              | **Not Yet** | Yes
 
 For `Bug Found`, I made a rake task to build the table below, the source code is in `Rakefile`:
 
-str_1      | str_2      | origin       | jaro_winkler | fuzzystringmatch | hotwater | amatch
----        | ---        | ---          | ---          | ---              | ---      | ---
-"henka"    | "henkan"   | 0.9667       | 0.9667       | **0.9722**           | 0.9667   | **0.9444**
-"al"       | "al"       | 1.0          | 1.0          | 1.0              | 1.0      | 1.0
-"martha"   | "marhta"   | 0.9611       | 0.9611       | 0.9611           | 0.9611   | **0.9444**
-"jones"    | "johnson"  | 0.8324       | 0.8324       | 0.8324           | 0.8324   | **0.7905**
-"abcvwxyz" | "cabvwxyz" | 0.9583       | 0.9583       | 0.9583           | 0.9583   | 0.9583
-"dwayne"   | "duane"    | 0.84         | 0.84         | 0.84             | 0.84     | **0.8222**
-"dixon"    | "dicksonx" | 0.8133       | 0.8133       | 0.8133           | 0.8133   | **0.7667**
-"fvie"     | "ten"      | 0.0          | 0.0          | 0.0              | 0.0      | 0.0
+str_1      | str_2      | origin | jaro_winkler | fuzzystringmatch | hotwater | amatch
+---        | ---        | ---    | ---          | ---              | ---      | ---
+"henka"    | "henkan"   | 0.9667 | 0.9667       | **0.9722**       | 0.9667   | **0.9444**
+"al"       | "al"       | 1.0    | 1.0          | 1.0              | 1.0      | 1.0
+"martha"   | "marhta"   | 0.9611 | 0.9611       | 0.9611           | 0.9611   | **0.9444**
+"jones"    | "johnson"  | 0.8324 | 0.8324       | 0.8324           | 0.8324   | **0.7905**
+"abcvwxyz" | "cabvwxyz" | 0.9583 | 0.9583       | 0.9583           | 0.9583   | 0.9583
+"dwayne"   | "duane"    | 0.84   | 0.84         | 0.84             | 0.84     | **0.8222**
+"dixon"    | "dicksonx" | 0.8133 | 0.8133       | 0.8133           | 0.8133   | **0.7667**
+"fvie"     | "ten"      | 0.0    | 0.0          | 0.0              | 0.0      | 0.0
 
 - The origin result is from the [original C implementation by the author of the algorithm](http://web.archive.org/web/20100227020019/http://www.census.gov/geo/msb/stand/strcmp.c).
 - Test data are borrowed from [fuzzy-string-match's rspec file](https://github.com/kiyoka/fuzzy-string-match/blob/master/test/basic_pure_spec.rb).
@@ -94,5 +104,5 @@ amatch           | 0.960000 | 0.010000 | 0.970000 | (  0.964803)
 
 # Todo
 
-- Make it faster
-- Adjusting word table (Reference to original C implementation.)
+- Custom adjusting word table.
+- If the adjusting table is ASCII encoded, use dense matrix instread of sparse matrix to speed up.

data/Rakefile

@@ -5,15 +5,20 @@ Rake::ExtensionTask.new("jaro_winkler") do |ext|
   ext.lib_dir = "lib/jaro_winkler"
 end
 
-task :benchmark do
+desc 'type can be "native" or "pure"'
+task :benchmark, :type do |t, args|
+  args.with_defaults(type: :all)
   ROOT_PATH = File.expand_path('..', __FILE__)
   LIB_PATH = File.join(ROOT_PATH, 'lib')
   BENCHMARK_PATH = File.join(ROOT_PATH, 'benchmark')
-  Dir[File.join(BENCHMARK_PATH, '*.rb')].each do |path|
+
+  files = File.join(BENCHMARK_PATH, args[:type] == :all ? '*.rb' : "#{args[:type]}.rb")
+  Dir[files].each do |path|
     output_path = File.join(BENCHMARK_PATH, File.basename(path, '*.rb').sub('.rb', '.txt'))
-    cmd = "RUBYLIB=#{LIB_PATH} ruby #{path} > #{output_path}"
+    cmd = "RUBYLIB=#{LIB_PATH} ruby #{path}"
     puts cmd
-    system(cmd)
+    output = `#{cmd}`
+    File.write(output_path, output)
   end
 end
 

data/ext/jaro_winkler/distance.c

@@ -3,93 +3,161 @@
 #include <ctype.h>
 #include "distance.h"
 
-Option* option_new(){
-  Option *opt = calloc(1, sizeof(Option));
-  opt->ignore_case = 0;
-  opt->weight = 0.1;
-  opt->threshold = 0.7;
-  return opt;
-}
+typedef struct{
+  unsigned long long code;
+  unsigned int byte_length;
+} UnicodeHash;
 
-static int char_bytes_num(char first_char){
-  unsigned char c = first_char;
-  if(c >= 252) return 6;      // 1111110x
-  else if(c >= 248) return 5; // 111110xx
-  else if(c >= 240) return 4; // 11110xxx
-  else if(c >= 224) return 3; // 1110xxxx
-  else if(c >= 192) return 2; // 110xxxxx
-  else return 1;
-}
+typedef struct{
+  unsigned long long *ary;
+  int length;
+} Codepoints;
 
-static unsigned long long* codepoints(const char *str, int byte_len, int *ret_len){
-  unsigned long long *ret = calloc(byte_len, sizeof(long long));
-  int count = 0;
-  for(int i = 0; i < byte_len;){
-    int bytes_num = char_bytes_num(str[i]);
-    memcpy(&ret[count], &str[i], bytes_num);
-    count++;
-    i += bytes_num;
-  }
-  *ret_len = count;
-  return ret;
+typedef struct{
+  unsigned long long x, y;
+} Coord;
+
+typedef struct{
+  Coord *coords;
+  int length;
+} Matrix;
+
+static const char *DEFAULT_ADJ_TABLE[] = {
+  "A","E", "A","I", "A","O", "A","U", "B","V", "E","I", "E","O", "E","U", "I","O", "I","U", "O","U",
+  "I","Y", "E","Y", "C","G", "E","F", "W","U", "W","V", "X","K", "S","Z", "X","S", "Q","C", "U","V",
+  "M","N", "L","I", "Q","O", "P","R", "I","J", "2","Z", "5","S", "8","B", "1","I", "1","L", "0","O",
+  "0","Q", "C","K", "G","J", "E"," ", "Y"," ", "S"," "
+};
+static Matrix DEFAULT_MATRIX;
+
+static UnicodeHash unicode_hash_new(const char *str);
+static Codepoints codepoints_new(const char *str, int byte_len);
+static Matrix matrix_new(const char **adj_table, int length);
+static char matrix_find(Matrix matrix, unsigned long long code_1, unsigned long long code_2);
+
+Option option_new(){
+  Option opt;
+  opt.ignore_case = 0;
+  opt.weight = 0.1;
+  opt.threshold = 0.7;
+  return opt;
 }
 
-double c_distance(char *s1, int byte_len1, char *s2, int byte_len2, Option *opt){
+double c_distance(char *s1, int s1_byte_len, char *s2, int s2_byte_len, Option opt){
   // set default option if NULL passed
   int free_opt_flag = 0;
-  if(!opt){ free_opt_flag = 1; opt = option_new(); }
 
-  int ary_1_len, ary_2_len;
-  unsigned long long *ary_1 = codepoints(s1, byte_len1, &ary_1_len), *ary_2 = codepoints(s2, byte_len2, &ary_2_len);
+  Codepoints code_ary_1 = codepoints_new(s1, s1_byte_len);
+  Codepoints code_ary_2 = codepoints_new(s2, s2_byte_len);
 
-  if(opt->ignore_case){
-    for(int i = 0; i < ary_1_len; ++i) if(ary_1[i] < 256 && islower(ary_1[i])) ary_1[i] -= 32;
-    for(int i = 0; i < ary_2_len; ++i) if(ary_2[i] < 256 && islower(ary_2[i])) ary_2[i] -= 32;
+  if(opt.ignore_case){
+    for(int i = 0; i < code_ary_1.length; ++i) if(code_ary_1.ary[i] < 256 && islower(code_ary_1.ary[i])) code_ary_1.ary[i] -= 32;
+    for(int i = 0; i < code_ary_2.length; ++i) if(code_ary_2.ary[i] < 256 && islower(code_ary_2.ary[i])) code_ary_2.ary[i] -= 32;
   }
 
   // Guarantee the order
-  if(ary_1_len > ary_2_len){
-    unsigned long long *tmp = ary_1; ary_1 = ary_2; ary_2 = tmp;
-    int tmp2 = ary_1_len; ary_1_len = ary_2_len; ary_2_len = tmp2;
+  if(code_ary_1.length > code_ary_2.length){
+    unsigned long long *tmp = code_ary_1.ary; code_ary_1.ary = code_ary_2.ary; code_ary_2.ary = tmp;
+    int tmp2 = code_ary_1.length; code_ary_1.length = code_ary_2.length; code_ary_2.length = tmp2;
   }
-  int window_size = ary_2_len / 2 - 1;
+
+  // Adjusting table
+  static char first_time = 1;
+  if(opt.adj_table){
+    if(first_time) DEFAULT_MATRIX = matrix_new(DEFAULT_ADJ_TABLE, sizeof(DEFAULT_ADJ_TABLE) / 8);
+    first_time = 0;
+  }
+
+  // Compute jaro distance
+  int window_size = code_ary_2.length / 2 - 1;
   if(window_size < 0) window_size = 0;
   double matches     = 0.0;
+  double sim_matches = 0.0;
   int transpositions = 0;
   int previous_index = -1;
-  int max_index      = ary_2_len - 1;
-  for(int i = 0; i < ary_1_len; i++){
+  int max_index      = code_ary_2.length - 1;
+  for(int i = 0; i < code_ary_1.length; i++){
     int left  = i - window_size;
     int right = i + window_size;
     if(left  < 0) left = 0;
     if(right > max_index) right = max_index;
-    char matched = 0, found = 0;
+    char matched     = 0;
+    char found       = 0;
+    char sim_matched = 0;
     for(int j = left; j <= right; j++){
-      if(ary_1[i] == ary_2[j]){
+      if(code_ary_1.ary[i] == code_ary_2.ary[j]){
         matched = 1;
         if(!found && j > previous_index){
           previous_index = j;
           found = 1;
         }
-      } // if(ary_1[i] == ary_2[j]){
+      }else if(opt.adj_table && matrix_find(DEFAULT_MATRIX, code_ary_1.ary[i], code_ary_2.ary[j])) sim_matched = 1;
     } // for(int j = left; j <= right; j++){
     if(matched){
       matches++;
       if(!found) transpositions++;
-    }
-  } // for(int i = 0; i < ary_1_len; i++){
+    }else if(sim_matched) sim_matches += 3;
+  } // for(int i = 0; i < code_ary_1.length; i++){
+
   // Don't divide transpositions by 2 since it's been counted directly by above code.
-  double jaro_distance =  matches == 0 ? 0 : (matches / ary_1_len + matches / ary_2_len + (matches - transpositions) / matches) / 3.0;
+  double similarity = matches;
+  if(opt.adj_table) similarity += sim_matches / 10;
+  double jaro_distance = matches == 0 ? 0 : (similarity / code_ary_1.length + similarity / code_ary_2.length + (matches - transpositions) / matches) / 3.0;
 
   // calculate jaro-winkler distance
-  double threshold = opt->threshold, weight = opt->weight;
+  double threshold = opt.threshold, weight = opt.weight;
   int prefix = 0;
-  int max_length = ary_1_len > 4 ? 4 : ary_1_len;
+  int max_length = code_ary_1.length > 4 ? 4 : code_ary_1.length;
   for(int i = 0; i < max_length; ++i){
-    if(ary_1[i] == ary_2[i]) prefix++;
+    if(code_ary_1.ary[i] == code_ary_2.ary[i]) prefix++;
     else break;
   }
-  free(ary_1); free(ary_2);
-  if(free_opt_flag) free(opt);
+  free(code_ary_1.ary); free(code_ary_2.ary);
   return jaro_distance < threshold ? jaro_distance : jaro_distance + ((prefix * weight) * (1 - jaro_distance));
+}
+
+static UnicodeHash unicode_hash_new(const char *str){
+  UnicodeHash ret = {};
+  unsigned char first_char = str[0];
+  if(first_char >= 252) ret.byte_length = 6;      // 1111110x
+  else if(first_char >= 248) ret.byte_length = 5; // 111110xx
+  else if(first_char >= 240) ret.byte_length = 4; // 11110xxx
+  else if(first_char >= 224) ret.byte_length = 3; // 1110xxxx
+  else if(first_char >= 192) ret.byte_length = 2; // 110xxxxx
+  else ret.byte_length = 1;
+  memcpy(&ret.code, str, ret.byte_length);
+  return ret;
+}
+
+static Codepoints codepoints_new(const char *str, int byte_len){
+  Codepoints ret = {0};
+  ret.ary = calloc(byte_len, sizeof(long long));
+  int count = 0;
+  for(int i = 0; i < byte_len;){
+    UnicodeHash hash = unicode_hash_new(str + i);
+    ret.ary[count] = hash.code;
+    count++;
+    i += hash.byte_length;
+  }
+  ret.length += count;
+  return ret;
+}
+
+static Matrix matrix_new(const char **adj_table, int length){
+  Matrix ret;
+  ret.coords = calloc(length, sizeof(Coord));
+  ret.length = length;
+  for(int i = 0; i < length; i += 2){
+    UnicodeHash h1 = unicode_hash_new(adj_table[i]);
+    UnicodeHash h2 = unicode_hash_new(adj_table[i+1]);
+    ret.coords[i].x = ret.coords[i+1].y = h1.code;
+    ret.coords[i].y = ret.coords[i+1].x = h2.code;
+  }
+  return ret;
+}
+
+static char matrix_find(Matrix matrix, unsigned long long code_1, unsigned long long code_2){
+  for (int i = 0; i < matrix.length; i++)
+    if(matrix.coords[i].x == code_1 && matrix.coords[i].y == code_2) return 1;
+  return 0;
 }

data/ext/jaro_winkler/distance.h

@@ -2,11 +2,11 @@
 #define DISTANCE_H 1
 
 typedef struct{
-  char ignore_case;
   double weight, threshold;
+  char ignore_case, adj_table;
 } Option;
 
-double c_distance(char *s1, int byte_len1, char *s2, int byte_len2, Option *opt);
-Option* option_new();
+double      c_distance(char *s1, int s1_byte_len, char *s2, int s2_byte_len, Option opt);
+Option      option_new();
 
-#endif /* DISTANCE_H */
+#endif /* DISTANCE_H */

data/ext/jaro_winkler/jaro_winkler.c

@@ -11,17 +11,18 @@ void Init_jaro_winkler(void){
 VALUE rb_distance(int argc, VALUE *argv, VALUE self){
   VALUE s1, s2, opt;
   rb_scan_args(argc, argv, "2:", &s1, &s2, &opt);
-  Option *c_opt = option_new();
+  Option c_opt = option_new();
   if(TYPE(opt) == T_HASH){
-    VALUE weight = rb_hash_aref(opt, ID2SYM(rb_intern("weight")));
-    VALUE threshold = rb_hash_aref(opt, ID2SYM(rb_intern("threshold")));
+    VALUE weight      = rb_hash_aref(opt, ID2SYM(rb_intern("weight")));
+    VALUE threshold   = rb_hash_aref(opt, ID2SYM(rb_intern("threshold")));
     VALUE ignore_case = rb_hash_aref(opt, ID2SYM(rb_intern("ignore_case")));
-    if(!NIL_P(weight)) c_opt->weight = NUM2DBL(weight);
-    if(c_opt->weight > 0.25) rb_raise(rb_eRuntimeError, "Scaling factor should not exceed 0.25, otherwise the distance can become larger than 1.");
-    if(!NIL_P(threshold)) c_opt->threshold = NUM2DBL(threshold);
-    if(!NIL_P(ignore_case)) c_opt->ignore_case = (TYPE(ignore_case) == T_FALSE || NIL_P(ignore_case)) ? 0 : 1;
+    VALUE adj_table   = rb_hash_aref(opt, ID2SYM(rb_intern("adj_table")));
+    if(!NIL_P(weight)) c_opt.weight = NUM2DBL(weight);
+    if(c_opt.weight > 0.25) rb_raise(rb_eRuntimeError, "Scaling factor should not exceed 0.25, otherwise the distance can become larger than 1.");
+    if(!NIL_P(threshold)) c_opt.threshold     = NUM2DBL(threshold);
+    if(!NIL_P(ignore_case)) c_opt.ignore_case = (TYPE(ignore_case)  == T_FALSE || NIL_P(ignore_case)) ? 0 : 1;
+    if(!NIL_P(adj_table)) c_opt.adj_table     = (TYPE(adj_table)    == T_FALSE || NIL_P(adj_table)) ? 0 : 1;
   }
   VALUE ret = rb_float_new(c_distance(StringValuePtr(s1), RSTRING_LEN(s1), StringValuePtr(s2), RSTRING_LEN(s2), c_opt));
-  free(c_opt);
   return ret;
 }

data/lib/jaro_winkler.rb

@@ -1,8 +1,10 @@
 require 'jaro_winkler/fallback'
+require 'jaro_winkler/adjusting_table'
 require 'jaro_winkler/jaro_winkler.so' unless JaroWinkler.fallback?
 module JaroWinkler
   module_function
-  def jaro_distance s1, s2
+  def jaro_distance s1, s2, options = {}
+    options = {adj_table: false}.merge options
     length1, length2 = s1.length, s2.length
     # Guarantee the length order
     if s1.length > s2.length
@@ -12,6 +14,7 @@ module JaroWinkler
     window_size    = (length2 / 2) - 1
     window_size    = 0 if window_size < 0
     matches        = 0.0
+    sim_matches    = 0.0
     transpositions = 0
     previous_index = -1
     max_index      = length2 - 1
@@ -21,6 +24,7 @@ module JaroWinkler
       left      = 0         if left  < 0
       right     = max_index if right > max_index
       matched   = false
+      sim_matched = false
       found     = false
       s2[left..right].chars.each_with_index do |c2, j|
         if c1 == c2
@@ -30,15 +34,21 @@ module JaroWinkler
             previous_index = s2_index
             found          = true
           end
+        elsif options[:adj_table] && DEFAULT_ADJ_TABLE[c1][c2]
+          sim_matched = true
         end
       end
       if matched
         matches += 1
         transpositions += 1 unless found
+      elsif sim_matched # not matched but similarly matched
+        sim_matches += 3
       end
     end
     # Don't divide transpositions by 2 since it's been counted directly by above code.
-    matches == 0 ? 0 : (matches / length1 + matches / length2 + (matches - transpositions) / matches) / 3.0
+    similarity = matches
+    similarity += sim_matches / 10 if options[:adj_table]
+    matches == 0 ? 0 : (similarity / length1 + similarity / length2 + (matches - transpositions) / matches) / 3.0
   end
 
   def r_distance s1, s2, options = {}
@@ -46,7 +56,7 @@ module JaroWinkler
     weight, threshold, ignore_case = options[:weight], options[:threshold], options[:ignore_case]
     raise 'Scaling factor should not exceed 0.25, otherwise the distance can become larger than 1' if weight > 0.25
     s1, s2     = s1.upcase, s2.upcase if ignore_case
-    distance   = jaro_distance(s1, s2)
+    distance   = jaro_distance(s1, s2, options)
     prefix     = 0
     max_length = [4, s1.length, s2.length].min
     s1[0, max_length].chars.each_with_index do |c1, i|

data/lib/jaro_winkler/adjusting_table.rb

@@ -0,0 +1,10 @@
+module JaroWinkler
+  DEFAULT_ADJ_TABLE = Hash.new({})
+  [
+    ['A', 'E'], ['A', 'I'], ['A', 'O'], ['A', 'U'], ['B', 'V'], ['E', 'I'], ['E', 'O'], ['E', 'U'], ['I', 'O'],
+    ['I', 'U'], ['O', 'U'], ['I', 'Y'], ['E', 'Y'], ['C', 'G'], ['E', 'F'], ['W', 'U'], ['W', 'V'], ['X', 'K'],
+    ['S', 'Z'], ['X', 'S'], ['Q', 'C'], ['U', 'V'], ['M', 'N'], ['L', 'I'], ['Q', 'O'], ['P', 'R'], ['I', 'J'],
+    ['2', 'Z'], ['5', 'S'], ['8', 'B'], ['1', 'I'], ['1', 'L'], ['0', 'O'], ['0', 'Q'], ['C', 'K'], ['G', 'J'],
+    ['E', ' '], ['Y', ' '], ['S', ' ']
+  ].each{ |s1, s2| DEFAULT_ADJ_TABLE[s1][s2] = DEFAULT_ADJ_TABLE[s2][s1] = true }
+end

data/lib/jaro_winkler/version.rb

@@ -1,3 +1,3 @@
 module JaroWinkler
-  VERSION = "1.2.8"
+  VERSION = "1.3.0.beta"
 end

data/spec/jaro_winkler_spec.rb

@@ -1,62 +1,84 @@
 # encoding: utf-8
 require 'jaro_winkler'
 include JaroWinkler
-describe JaroWinkler do
-  before(:all) do
-    @ary = [
-      ['henka', 'henkan', 0.9667],
-      ['al', 'al', 1.0],
-      ['martha', 'marhta', 0.9611],
-      ['jones', 'johnson', 0.8323],
-      ['abcvwxyz', 'cabvwxyz', 0.9583],
-      ['dwayne', 'duane', 0.8400],
-      ['dixon', 'dicksonx', 0.8133],
-      ['fvie', 'ten', 0.0],
-      ['tony', 'tony', 1.0],
-      ['tonytonyjan', 'tonytonyjan', 1.0],
-      ['x', 'x', 1.0],
-      ['', '', 0.0],
-      ['tony', '', 0.0],
-      ['', 'tony', 0.0],
-      ['tonytonyjan', 'tony', 0.8727],
-      ['tony', 'tonytonyjan', 0.8727]
-    ]
-  end
 
+shared_examples 'common' do |strategy|
   it 'works' do
-    @ary.each do |s1, s2, ans|
-      expect(r_distance(s1, s2)).to be_within(0.0001).of(ans)
-      expect(c_distance(s1, s2)).to be_within(0.0001).of(ans)
+    ary = [
+      ['henka'       , 'henkan'      , 0.9667] ,
+      ['al'          , 'al'          , 1.0]    ,
+      ['martha'      , 'marhta'      , 0.9611] ,
+      ['jones'       , 'johnson'     , 0.8323] ,
+      ['abcvwxyz'    , 'cabvwxyz'    , 0.9583] ,
+      ['dwayne'      , 'duane'       , 0.8400] ,
+      ['dixon'       , 'dicksonx'    , 0.8133] ,
+      ['fvie'        , 'ten'         , 0.0]    ,
+      ['tony'        , 'tony'        , 1.0]    ,
+      ['tonytonyjan' , 'tonytonyjan' , 1.0]    ,
+      ['x'           , 'x'           , 1.0]    ,
+      [''            , ''            , 0.0]    ,
+      ['tony'        , ''            , 0.0]    ,
+      [''            , 'tony'        , 0.0]    ,
+      ['tonytonyjan' , 'tony'        , 0.8727] ,
+      ['tony'        , 'tonytonyjan' , 0.8727]
+    ]
+    ary.each do |s1, s2, ans|
+      expect(send(strategy, s1, s2)).to be_within(0.0001).of(ans)
     end
   end
 
   it 'works with UTF-8' do
-    expect(c_distance('變形金剛4:絕跡重生', '變形金剛4: 絕跡重生')).to be_within(0.0001).of(0.9818)
-    expect(c_distance('連勝文', '連勝丼')).to be_within(0.0001).of(0.8222)
-    expect(c_distance('馬英九', '馬英丸')).to be_within(0.0001).of(0.8222)
+    ary = [
+      ['變形金剛4:絕跡重生' , '變形金剛4: 絕跡重生' , 0.9818] ,
+      ['連勝文'             , '連勝丼'              , 0.8222] ,
+      ['馬英九'             , '馬英丸'              , 0.8222] ,
+      ['良い'               , 'いい'                , 0.6666] ,
+    ]
+    ary.each do |s1, s2, ans|
+      expect(send(strategy, s1, s2)).to be_within(0.0001).of(ans)
+    end
   end
 
   it 'sets ignore_case' do
-    params = 'MARTHA', 'marhta', {ignore_case: true}
-    expect(r_distance(*params)).to be_within(0.0001).of(0.9611)
-    expect(c_distance(*params)).to be_within(0.0001).of(0.9611)
+    expect(send(strategy, 'MARTHA', 'marhta', ignore_case: true)).to be_within(0.0001).of(0.9611)
   end
 
   it 'sets weight' do
-    params = 'MARTHA', 'MARHTA', {weight: 0.2}
-    expect(r_distance(*params)).to be_within(0.0001).of(0.9778)
-    expect(c_distance(*params)).to be_within(0.0001).of(0.9778)
+    expect(send(strategy, 'MARTHA', 'MARHTA', weight: 0.2)).to be_within(0.0001).of(0.9778)
   end
 
   it 'sets threshold' do
-    params = 'MARTHA', 'MARHTA', {threshold: 0.99}
-    expect(r_distance(*params)).to be_within(0.0001).of(0.9445)
-    expect(c_distance(*params)).to be_within(0.0001).of(0.9445)
+    expect(send(strategy, 'MARTHA', 'MARHTA', threshold: 0.99)).to be_within(0.0001).of(0.9445)
   end
 
-  it 'throws exception when weight exceeding 0.25' do
-    params = 'MARTHA', 'MARHTA', {weight: 0.26}
-    expect{ r_distance(*params) }.to raise_error
-    expect{ c_distance(*params) }.to raise_error
+
+  it 'works with adjusting table' do
+    ary = [
+      ['HENKA'       , 'HENKAN'      , 0.9667] , # m=5, t=0, s=0
+      ['AL'          , 'AL'          , 1.0   ],  # m=2, t=0, s=0
+      ['MARTHA'      , 'MARHTA'      , 0.9611] , # m=6, t=1, s=0
+      ['JONES'       , 'JOHNSON'     , 0.8598] , # m=4, t=0, s=3
+      ['ABCVWXYZ'    , 'CABVWXYZ'    , 0.9583] , # m=8, t=1, s=0
+      ['DWAYNE'      , 'DUANE'       , 0.8730] , # m=4, t=0, s=3
+      ['DIXON'       , 'DICKSONX'    , 0.8393] , # m=4, t=0, s=3
+      ['FVIE'        , 'TEN'         , 0.0   ]
+    ]
+    ary.each do |s1, s2, ans|
+      expect(send(strategy, s1, s2, adj_table: true)).to be_within(0.0001).of(ans)
+    end
+  end
+
+  context 'with weight exceeding 0.25' do
+    it 'throws exception' do
+      expect{ send(strategy, 'MARTHA', 'MARHTA', weight: 0.26) }.to raise_error
+    end
   end
+end
+
+describe 'Pure Ruby' do
+  include_examples 'common', :r_distance
+end
+
+describe 'C extention' do
+  include_examples 'common', :c_distance
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: jaro_winkler
 version: !ruby/object:Gem::Version
-  version: 1.2.8
+  version: 1.3.0.beta
 platform: ruby
 authors:
 - Jian Weihang
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2014-09-08 00:00:00.000000000 Z
+date: 2014-09-10 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: bundler
@@ -77,9 +77,9 @@ files:
 - ext/jaro_winkler/extconf.rb
 - ext/jaro_winkler/jaro_winkler.c
 - ext/jaro_winkler/jaro_winkler.h
-- ext/jaro_winkler/strcmp95.c
 - jaro_winkler.gemspec
 - lib/jaro_winkler.rb
+- lib/jaro_winkler/adjusting_table.rb
 - lib/jaro_winkler/fallback.rb
 - lib/jaro_winkler/version.rb
 - spec/jaro_winkler_spec.rb
@@ -99,9 +99,9 @@ required_ruby_version: !ruby/object:Gem::Requirement
       version: '0'
 required_rubygems_version: !ruby/object:Gem::Requirement
   requirements:
-  - - ">="
+  - - ">"
     - !ruby/object:Gem::Version
-      version: '0'
+      version: 1.3.1
 requirements: []
 rubyforge_project: 
 rubygems_version: 2.4.1

data/ext/jaro_winkler/strcmp95.c

@@ -1,199 +0,0 @@
-
-/* strcmp95.c   Version 2                 */
-
-/* The strcmp95 function returns a double precision value from 0.0 (total
-   disagreement) to 1.0 (character-by-character agreement).  The returned 
-   value is a measure of the similarity of the two strings.                   */
-
-/* Date of Release:  Jan. 26, 1994                */
-/* Modified: April 24, 1994  Corrected the processing of the single length
-             character strings.
-   Authors:  This function was written using the logic from code written by
-             Bill Winkler, George McLaughlin and Matt Jaro with modifications
-             by Maureen Lynch. 
-   Comment:  This is the official string comparator to be used for matching 
-             during the 1995 Test Census.                                     */
-
-#include <ctype.h>
-#include <string.h>
-
-#define NOTNUM(c) ((c>57) || (c<48))
-#define INRANGE(c)      ((c>0)  && (c<91))
-#define MAX_VAR_SIZE 61
-#define NULL60 "                                                            "
-
-double  strcmp95(char *ying, char *yang, long y_length, int *ind_c)
-
-{
-/* Arguments:
-
-   ying and yang are pointers to the 2 strings to be compared.  The strings
-   need not be NUL-terminated strings because the length is passed.
-
-   y_length is the length of the strings. 
-
-   ind_c is an array that is used to define whether certain options should be 
-   activated.  A nonzero value indicates the option is deactivated.  
-   The options are:
-     ind_c[0] Increase the probability of a match when the number of matched
-              characters is large.  This option allows for a little more
-              tolerance when the strings are large.  It is not an appropriate
-              test when comparing fixed length fields such as phone and
-              social security numbers.
-     ind_c[1] All lower case characters are converted to upper case prior
-              to the comparison.  Disabling this feature means that the lower 
-              case string "code" will not be recognized as the same as the 
-              upper case string "CODE".  Also, the adjustment for similar 
-              characters section only applies to uppercase characters.
-
-   The suggested values are all zeros for character strings such as names.    */
-
-static  int pass=1/* TODO: make it an option parameter */, adjwt[91][91];
-static  char  sp[39][2] =
- {'A','E',  'A','I',  'A','O',  'A','U',  'B','V',  'E','I',  'E','O',  'E','U',
-  'I','O',  'I','U',  'O','U',  'I','Y',  'E','Y',  'C','G',  'E','F',
-  'W','U',  'W','V',  'X','K',  'S','Z',  'X','S',  'Q','C',  'U','V',
-  'M','N',  'L','I',  'Q','O',  'P','R',  'I','J',  '2','Z',  '5','S',
-  '8','B',  '1','I',  '1','L',  '0','O',  '0','Q',  'C','K',  'G','J',
-  'E',' ',  'Y',' ',  'S',' '};
-
-char    ying_hold[MAX_VAR_SIZE],
-        yang_hold[MAX_VAR_SIZE],
-        ying_flag[MAX_VAR_SIZE],
-        yang_flag[MAX_VAR_SIZE];
-
-double  weight, Num_sim;
-
-long    minv,   search_range,   lowlim,    ying_length,
-        hilim,  N_trans,        Num_com,   yang_length;
-
-int yl1,  yi_st,  N_simi;
-
-register        int     i,      j,      k;
-
-/* Initialize the adjwt array on the first call to the function only.
-   The adjwt array is used to give partial credit for characters that 
-   may be errors due to known phonetic or character recognition errors.
-   A typical example is to match the letter "O" with the number "0"           */
-if (!pass) {
-  pass++;
-  for (i=0; i<91; i++) for (j=0; j<91; j++) adjwt[i][j] = 0;
-  for (i=0; i<36; i++) {
-    adjwt[sp[i][0]][sp[i][1]] = 3;
-    adjwt[sp[i][1]][sp[i][0]] = 3;
-} }
-
-/* If either string is blank - return - added in Version 2                    */
-if (!strncmp(ying,NULL60,y_length)) return(0.0);                 
-if (!strncmp(yang,NULL60,y_length)) return(0.0);                 
-
-/* Identify the strings to be compared by stripping off all leading and 
-   trailing spaces.                   */
-k = y_length - 1;
-for(j = 0;((ying[j]==' ') && (j < k));j++);
-for(i = k;((ying[i]==' ') && (i > 0));i--);
-ying_length = i + 1 - j;
-yi_st = j;
-
-for(j = 0;((yang[j]==' ') && (j < k));j++);
-for(i = k;((yang[i]==' ') && (i > 0));i--);
-yang_length = i + 1 - j;
-
-ying_hold[0]=yang_hold[0]=0;
-strncat(ying_hold,&ying[yi_st],ying_length);
-strncat(yang_hold,&yang[j],yang_length);
-
-if (ying_length > yang_length) {
-  search_range = ying_length;
-  minv = yang_length;
-  }
- else {
-  search_range = yang_length;
-  minv = ying_length;
-  }
-
-/* If either string is blank - return                                         */
-/* if (!minv) return(0.0);                   removed in version 2             */
-
-/* Blank out the flags                    */
-ying_flag[0] = yang_flag[0] = 0;
-strncat(ying_flag,NULL60,search_range);
-strncat(yang_flag,NULL60,search_range);
-search_range = (search_range/2) - 1;
-if (search_range < 0) search_range = 0;   /* added in version 2               */
-
-/* Convert all lower case characters to upper case.                           */
-if (!ind_c[1]) {
-  for (i = 0;i < ying_length;i++) if (islower(ying_hold[i])) ying_hold[i] -= 32;
-  for (j = 0;j < yang_length;j++) if (islower(yang_hold[j])) yang_hold[j] -= 32;
-}
-
-/* Looking only within the search range, count and flag the matched pairs.    */
-Num_com = 0; 
-yl1 = yang_length - 1;
-for (i = 0;i < ying_length;i++) {
-  lowlim = (i >= search_range) ? i - search_range : 0;
-  hilim = ((i + search_range) <= yl1) ? (i + search_range) : yl1;
-  for (j = lowlim;j <= hilim;j++)  {
-    if ((yang_flag[j] != '1') && (yang_hold[j] == ying_hold[i])) {
-        yang_flag[j] = '1';
-        ying_flag[i] = '1';
-        Num_com++;
-        break;
-} } }
-
-/* If no characters in common - return                                        */
-if (!Num_com) return(0.0);                          
-
-/* Count the number of transpositions                                         */
-k = N_trans = 0;
-for (i = 0;i < ying_length;i++) {
-  if (ying_flag[i] == '1') {
-    for (j = k;j < yang_length;j++) {
-        if (yang_flag[j] == '1') { 
-         k = j + 1;
-         break;
-    } }
-    if (ying_hold[i] != yang_hold[j]) N_trans++;
-} }
-N_trans = N_trans / 2;
-
-/* adjust for similarities in nonmatched characters                           */
-N_simi = 0;
-if (minv > Num_com) {   
-  for (i = 0;i < ying_length;i++) {
-    if (ying_flag[i] == ' ' && INRANGE(ying_hold[i])) { 
-      for (j = 0;j < yang_length;j++) {
-        if (yang_flag[j] == ' ' && INRANGE(yang_hold[j])) {
-          if (adjwt[ying_hold[i]][yang_hold[j]] > 0) {
-            N_simi += adjwt[ying_hold[i]][yang_hold[j]];
-            yang_flag[j] = '2';
-            break;
-} } } } } }
-Num_sim = ((double) N_simi)/10.0 + Num_com;
-
-/* Main weight computation.                 */
-weight= Num_sim / ((double) ying_length) + Num_sim / ((double) yang_length)
-   + ((double) (Num_com - N_trans)) / ((double) Num_com);
-weight = weight / 3.0;
-
-/* Continue to boost the weight if the strings are similar                    */
-if (weight > 0.7) {
-
-  /* Adjust for having up to the first 4 characters in common                 */
-  j = (minv >= 4) ? 4 : minv;
-  for (i=0;((i<j)&&(ying_hold[i]==yang_hold[i])&&(NOTNUM(ying_hold[i])));i++); 
-  if (i) weight += i * 0.1 * (1.0 - weight);
-
-  /* Optionally adjust for long strings.                                      */
-  /* After agreeing beginning chars, at least two more must agree and 
-       the agreeing characters must be > .5 of remaining characters.          */
-  if ((!ind_c[0]) && (minv>4) && (Num_com>i+1) && (2*Num_com>=minv+i)) 
-  if (NOTNUM(ying_hold[0])) 
-    weight += (double) (1.0-weight) *
-   ((double) (Num_com-i-1) / ((double) (ying_length+yang_length-i*2+2)));
- }
-
-return(weight);
-
-} /* strcmp95 */