dhash-vips 0.0.4.1 → 0.0.5.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: a2e3a2b86e4e98ad5e145893959ac6297f4aa660
-  data.tar.gz: 5ada76a3f6567dde373947c8ea02d706e706cff5
+  metadata.gz: 0263e2d1f269153107a10986ba97d2221f1191d6
+  data.tar.gz: 4e1365a3ab0507e3e14cf8b06ae1f2f7d84482f1
 SHA512:
-  metadata.gz: cf8c95f0106d35145e3389a818fd6817538c38301ea9585eb71259d24d4d3791d079cc462765d0b18913d0f3783953ee210a4c26cb8477e9224747c4fe33d262
-  data.tar.gz: 4eba5b36480988463f9e98df5bc1883870006bfab754fbdde3db0e9c3ae3c389f25cde1b8f0129335773738f8c9721f4472f9504308dd2bdb32c16feaa4b0e02
+  metadata.gz: c00ea6ebead50175bfaa74bf0814a394c1b06159d238716d2edb7e2718587ee0a6c5497ec55823806193b5f46fa1a6ac1b43ba7e277ea468dd2e0ba3c5ea97a7
+  data.tar.gz: 57d32bfbf327226ff8ba0771f6346d1ccdadbe8b61dcfaa4aa81795976bd3146fce877f9532d6d9fc1a798ba2ffe3f63a1a940aee263f3702a9b48a4c835a6e6

data/Gemfile

@@ -1,4 +1,3 @@
 source 'https://rubygems.org'
 
-# Specify your gem's dependencies in dhash.gemspec
 gemspec

data/README.md

@@ -2,29 +2,27 @@
 
 # dHash and IDHash gem powered by ruby-vips
 
-The "dHash" is an algorithm of hashing that can be used for measuring the similarity of two images.
+The "dHash" is an algorithm of fingerprinting that can be used to measure the similarity of two images.
 
 You may read about it in "Kind of Like That" blog post (21 January 2013): http://www.hackerfactor.com/blog/index.php?/archives/529-Kind-of-Like-That.html  
-The original idea is that you split the image into 64 segments and so there are 64 bits -- each tells if the one segment is brighter or darker than the neighbor one. Then the [Hamming distance](https://en.wikipedia.org/wiki/Hamming_distance) between hashes is the opposite of images similarity.
+The original idea is that you split the image into 64 segments and so there are 64 bits -- each tells if the one segment is brighter or darker than the neighbor one. Then the [Hamming distance](https://en.wikipedia.org/wiki/Hamming_distance) between fingerprints is the opposite of images similarity.
 
-There are several implementations on Github already but they all depend on ImageMagick. My implementation takes an advantage of libvips (the `ruby-vips` gem) -- it also uses the `.conv` method and in result converts image to an array of grayscale bytes almost 10 times faster:
+There were several implementations on Github already but they all depend on ImageMagick. My implementation takes an advantage of libvips (the `ruby-vips` gem) -- it also uses the `.conv` method and in result converts image to an array of grayscale bytes almost 10 times faster:
 ```
-$ rake compare_speed
-
-load and calculate:
-                         user     system      total        real
-Dhash               12.610000   0.850000  13.460000 ( 13.726590)
-DHashVips::DHash     1.280000   0.250000   1.530000 (  1.435285)
-DHashVips::IDHash    1.240000   0.160000   1.400000 (  1.315536)
-
-distance (1000 times):
-                         user     system      total        real
-Dhash                2.500000   0.050000   2.550000 (  2.579340)
-DHashVips::DHash     2.350000   0.020000   2.370000 (  2.401252)
-DHashVips::IDHash    5.190000   0.040000   5.230000 (  5.279742)
+load and calculate the fingerprint:
+                          user     system      total        real
+Dhash                12.400000   0.820000  13.220000 ( 13.329952)
+DHashVips::DHash      1.330000   0.230000   1.560000 (  1.509826)
+DHashVips::IDHash     1.060000   0.090000   1.150000 (  1.100332)
+
+measure the distance (1000 times):
+                                   user     system      total        real
+Dhash hamming                  3.140000   0.020000   3.160000 (  3.179392)
+DHashVips::DHash hamming       3.040000   0.020000   3.060000 (  3.095190)
+DHashVips::IDHash distance     6.720000   0.030000   6.750000 (  6.790900)
 ```
 
-Here the `Dhash` is a https://github.com/maccman/dhash that I used earlier in my projects.  
+Here the `Dhash` is [another gem](https://github.com/maccman/dhash) that I used earlier in my projects.  
 The `DHashVips::DHash` is a port of it that uses vips. I would like to tell you that you can replace the `dhash` with `dhash-vips` gem right now but it appeared to have a barely noticeable issue. There is a lot of magic behind the libvips speed and resizing -- you may not notice it with unarmed eyes but when two neighbor segments are enough similar by luminosity the difference can change the sign. So I found two identical images that were just of different colorspace and size (photo by Jordan Voth):  
 ![](https://storage.googleapis.com/dhash-vips.nakilon.pro/dhash_issue_example.png)  
 but the distance between their hashes appeared to be equal to 5 while `dhash` gem reported 0.
@@ -33,36 +31,40 @@ This is why `DHashVips::IDHash` appeared.
 
 ## IDHash (the Important Difference Hash)
 
-It has improvements over the dHash that made hashing less sensitive to the resizing algorithm and effectively made the pair of images mentioned above to have a distance of 0 again. Three improvements are:  
-* The "Importance" is an array of extra 64 bits that tells the comparing function which half of 64 bits is important (when the difference between neighbors was enough significant) and which is not. So not every bit in a hash is being compared but only half of them.
-* It subtracts not only horizontally but also vertically -- that adds 128 more bits.
+It has improvements over the dHash that made fingerprinting less sensitive to the resizing algorithm and effectively made the pair of images mentioned above to have a distance of 0 again. Three improvements are:  
+* The "Importance" is an array of extra 64 bits that tells the comparing function which half of 64 bits is important (when the difference between neighbors was enough significant) and which is not. So not every bit in a fingerprint is being compared but only half of them.  
+* It subtracts not only horizontally but also vertically -- that adds 128 more bits.  
 * Instead of resizing to 9x8 it resizes to 8x8 and puts the image on a torus so it subtracts the left column from the right one and the top from bottom.
 
-For example, here are two photos (by Brian Lauer):  
-![](https://storage.googleapis.com/dhash-vips.nakilon.pro/idhash_example_in.png)  
-and visualization of IDHash (`rake compare_images -- image1.jpg image2.jpg`):  
-![](https://storage.googleapis.com/dhash-vips.nakilon.pro/idhash_example_out.png)  
-
-Here in each of 64 cells, there are two circles that color the difference between that cell and the neighbor one. If the difference is low the Importance bit is set to zero and the circle is invisible. So there are 128 pairs of corresponding circles and when you take one, if at least one circle is visible and is of different color the line is to be drawn. Here you see 15 lines and so the distance between hashes will be equal to 15 (that is pretty low and can be interpreted as "images look similar"). Also, you see here that floor on this photo matters -- classic dHash won't see that it's darker than wall because it's comparing only horizontal neighbors and if one photo had no floor the distance function won't notice that. Also, it sees the Important difference between the very right and left columns because the wall has a slow but visible gradient.
-
-You could see in hash calculation benchmark earlier that these improvements didn't make it slower than dHash because most of the time is spent on image resizing. The calculation of distance is what became two times slower:
+You could see in fingerprint calculation benchmark earlier that these improvements didn't make it slower than dHash because most of the time is spent on image resizing. The calculation of distance is what became two times slower:
 ```ruby
-((a | b) & (a >> 128 ^ b >> 128)).to_s(2).count "1"
+((a | b) & ((a ^ b) >> 128)).to_s(2).count "1"
 ```
 vs
 ```ruby
 (a ^ b).to_s(2).count "1"
 ```
 
-Remaining problems:  
+### Example
+
+Here are two photos (by Brian Lauer):  
+![](https://storage.googleapis.com/dhash-vips.nakilon.pro/idhash_example_in.png)  
+and visualization of IDHash (`rake compare_images -- image1.jpg image2.jpg`):  
+![](https://storage.googleapis.com/dhash-vips.nakilon.pro/idhash_example_out.png)  
+
+Here in each of 64 cells, there are two circles that color the difference between that cell and the neighbor one. If the difference is low the Importance bit is set to zero and the circle is invisible. So there are 128 pairs of corresponding circles and when you take one, if at least one circle is visible and is of different color the line is to be drawn. Here you see 15 lines and so the distance between fingerprints will be equal to 15 (that is pretty low and can be interpreted as "images look similar"). Also, you see here that floor on this photo matters -- classic dHash won't see that it's darker than wall because it's comparing only horizontal neighbors and if one photo had no floor the distance function won't notice that. Also, it sees the Important difference between the very right and left columns because the wall has a slow but visible gradient.
+
+### Remaining problems
+
 * Neither dHash nor IDHash can't automatically detect very shifted crops and rotated images but you can make a wrapper that would call the comparison function iteratively.  
 * These algorithms are color blind because of converting an image to grayscale. If you take a photo of something in your yard the sun will create lights and shadows, but if you compare photos of something green painted on a blue wall there is a possibility the machine would see nothing painted at all. The `dhash` gem had such image in specs and that made them pretty useless (this was supposed to be a face):  
 ![](https://storage.googleapis.com/dhash-vips.nakilon.pro/colorblind.png)  
-* If you have a pile of 1000000 images to compare them with each other that would take a month or two. To improve the process you in case of DHashVips::DHash that uses Hamming distance you may want to read these:  
+* If you have a pile of 1000000 images comparing them with each other would take a month or two. To improve the process in case of dHash that uses Hamming distance you may want to read these threads on Stackexchange network:  
   * [How to find the closest pairs of a string of binary bins in Ruby without O^2 issues?](https://stackoverflow.com/q/8734034/322020)  
   * [Find all pairs of values that are close under Hamming distance](https://cstheory.stackexchange.com/q/18516/27420)  
   * [Finding the closest pair between two sets of points on the hypercube](https://cstheory.stackexchange.com/q/16322/27420)  
   * [Would PCA work for boolean data types?](https://stats.stackexchange.com/q/159705/1125)  
+  * [Using pHash to search agaist a huge image database, what is the best approach?](https://stackoverflow.com/q/18257641/322020)  
 
 ## Installation
 
@@ -95,8 +97,6 @@ else
 end
 ```
 
-These `10` and `20` numbers are found empirically and just work enough well for 8-byte hashes.
-
 ### IDHash:
 
 ```ruby
@@ -115,7 +115,74 @@ else
 end
 ```
 
-Note that `DHash#calculate` accepts `hash_size` optional parameter that sets hash size in bytes, but `IDHash` size is currently hardcoded and can't be adjusted.
+These `10` and `20` numbers are found empirically and just work enough well for 8-byte hashes.  
+To find out these tresholds we can run a rake task with hardcoded test cases:
+```
+$ rake compare_matrices
+
+Dhash 
+Absolutely the same image: 0..0
+Complex B/W and the same but colorful: 0
+Similar images: 13..16
+Different images: 9..41
+
+DHashVips::DHash 
+Absolutely the same image: 0..0
+Complex B/W and the same but colorful: 5
+Similar images: 17..18
+Different images: 14..39
+
+DHashVips::IDHash 
+Absolutely the same image: 0..0
+Complex B/W and the same but colorful: 0
+Similar images: 15..23
+Different images: 19..64
+
+DHashVips::IDHash 4
+Absolutely the same image: 0..0
+Complex B/W and the same but colorful: 0
+Similar images: 71..108
+Different images: 102..211
+
+```
+
+### Notes
+
+* Methods were renamed from `#calculate` to `#fingerprint` and from `#hamming` to `#distance`.  
+* The `DHash#calculate` accepts `hash_size` optional parameter that is 8 by default. The `IDHash#fingerprint`'s optional parameter is called `power` and works in a bit different way: 3 means 8 and 4 means 16 -- other sizes are not supported because they don't seem to be useful (higher fingerprint resolution makes it vulnerable to image shifts and croppings). Because IDHash's fingerprint is more complex than DHash's one it's not that straight forward to compare them so under the hood the `#distance` methods have to check the size of fingerprint -- this trade-off costs 30-40% of speed that can be eliminated by using `#distance3` method that assumes fingerprint to be of power=3. So the full benchmark is this one:
+
+```
+$ rake compare_speed
+
+load and calculate the fingerprint:
+                          user     system      total        real
+Dhash                12.400000   0.820000  13.220000 ( 13.329952)
+DHashVips::DHash      1.330000   0.230000   1.560000 (  1.509826)
+DHashVips::IDHash     1.060000   0.090000   1.150000 (  1.100332)
+DHashVips::IDHash 4   1.030000   0.080000   1.110000 (  1.089148)
+
+measure the distance (1000 times):
+                                   user     system      total        real
+Dhash hamming                  3.140000   0.020000   3.160000 (  3.179392)
+DHashVips::DHash hamming       3.040000   0.020000   3.060000 (  3.095190)
+DHashVips::IDHash distance     8.170000   0.040000   8.210000 (  8.279950)
+DHashVips::IDHash distance3    6.720000   0.030000   6.750000 (  6.790900)
+DHashVips::IDHash distance 4  24.430000   0.130000  24.560000 ( 24.652625)
+```
+
+Also note that to make `#distance` able to assume the fingerprint resolution from the size of Integer that represents it, the change in its structure was needed (left half of bits was swapped with right one), so fingerprints between versions 0.0.4 and 0.0.5 became incompatible, but you probably can convert them manually. I know, incompatibilities suck but if we put the version or structure information inside fingerprint it will became slow to (de)serialize and store.
+
+## Troubleshooting
+
+El Captain and rbenv may cause environment issues that would make you do things like:
+```
+./ruby `rbenv which rake` compare_matrixes
+```
+instead of just
+```
+rake compare_matrixes
+```
+For more information on that: https://github.com/jcupitt/ruby-vips/issues/141
 
 ## Credits
 

data/Rakefile

@@ -50,99 +50,144 @@ task :compare_kernels do |_|
   end
 end
 
-
+# ./ruby `rbenv which rake` compare_matrixes
 desc "Compare the quality of Dhash, DHashVips::DHash and DHashVips::IDHash -- run it only after `rake test`"
-task :compare_matrixes do |_|
+task :compare_matrices do |_|
   require "dhash"
   require_relative "lib/dhash-vips"
   require "mll"
-  [[Dhash, :hamming], [DHashVips::DHash, :hamming], [DHashVips::IDHash, :distance]].each do |m, dm|
+  [
+    [Dhash, :calculate, :hamming],
+    [DHashVips::DHash, :calculate, :hamming],
+    [DHashVips::IDHash, :fingerprint, :distance],
+    [DHashVips::IDHash, :fingerprint, :distance, 4],
+  ].each do |m, calc, dm, power|
+    puts "\n#{m} #{power}"
     hashes = %w{
       71662d4d4029a3b41d47d5baf681ab9a.jpg
       ad8a37f872956666c3077a3e9e737984.jpg
+
+      6d97739b4a08f965dc9239dd24382e96.jpg
+      1b1d4bde376084011d027bba1c047a4b.jpg
+
       1d468d064d2e26b5b5de9a0241ef2d4b.jpg
       92d90b8977f813af803c78107e7f698e.jpg
+
       309666c7b45ecbf8f13e85a0bd6b0a4c.jpg
       3f9f3db06db20d1d9f8188cd753f6ef4.jpg
       df0a3b93e9412536ee8a11255f974141.jpg
       679634ff89a31279a39f03e278bc9a01.jpg
-    }.map{ |filename| m.calculate "images/#{filename}" }
+    }.map{ |filename| m.public_send calc, "images/#{filename}", *power }
     table = MLL::table[m.method(dm), [hashes], [hashes]]
-    array = Array.new(4){ [] }
+    # require "pp"
+    # pp table
+    array = Array.new(5){ [] }
     hashes.size.times.to_a.repeated_combination(2) do |i, j|
-      array[i == j ? 0 : (j - i).abs == 1 && (i + j - 1) % 4 == 0 ? [i, j] == [0, 1] ? 1 : 2 : 3].push table[i][j]
+      array[i == j ? 0 : (j - i).abs == 1 && (i + j - 1) % 4 == 0 ? [i, j] == [0, 1] ? 1 : [i, j] == [2, 3] ? 2 : 3 : 4].push table[i][j]
     end
-    p array.map &:sort
+    # p array.map &:sort
+    puts "Absolutely the same image: #{array[0].minmax.join ".."}"
+    puts "Complex B/W and the same but colorful: #{array[1][0]}"
+    puts "Similar images: #{array[3].minmax.join ".."}"
+    puts "Different images: #{[*array[2], *array[4]].minmax.join ".."}"
   end
 end
 
 # ruby -c Rakefile && rm -f ab.png && rake compare_images -- fc762fa286489d8afc80adc8cdcb125e.jpg 9c2c240ec02356472fb532f404d28dde.jpg 2>/dev/null && ql ab.png
+# rm -f ab.png && ./ruby `rbenv which rake` compare_images -- 6d97739b4a08f965dc9239dd24382e96.jpg 1b1d4bde376084011d027bba1c047a4b.jpg 2>/dev/null && ql ab.png
 desc "Visualizes the IDHash difference measurement between two images"
 task :compare_images do |_|
-  abort "there should be two image filenames passed as arguments" unless ARGV.size == 3
+  abort "there should be two image filenames passed as arguments (and optionally the `power`)" unless (4..5) === ARGV.size
+  abort "the optional argument should be either 3 or 4" unless [3, 4].include?(power = (ARGV[4] || 3).to_i)
+  task ARGV.last do ; end
   require_relative "lib/dhash-vips"
-  ha, hb = ARGV.drop(1).map &DHashVips::IDHash.method(:calculate)
+  ha, hb = ARGV[2, 2].map{ |filename| DHashVips::IDHash.fingerprint(filename, power) }
   puts "distance: #{DHashVips::IDHash.distance ha, hb}"
+  size = 2 ** power
+  shift = 2 * size * size
+  ai = ha >> shift
+  ad = ha - (ai << shift)
+  bi = hb >> shift
+  bd = hb - (bi << shift)
+
+  a, b = ARGV[2, 2].map do |filename|
+    image = Vips::Image.new_from_file filename
+    image = image.resize(size.fdiv(image.width), vscale: size.fdiv(image.height)).colourspace("b-w").
+                  resize(100, vscale: 100, kernel: :nearest).colourspace("srgb")
+  end
+  fail unless a.width == b.width && a.height == b.height
 
-  require "delegate"
-  class ImageMutable < SimpleDelegator
-    %i{ draw_line draw_circle }.each do |m|
-      define_method "#{m}!" do |*args|
-        __setobj__ self.send m, *args
-        self
+  _127 = shift - 1
+  _63 = size * size - 1
+  n = 0
+  width = a.width
+  height = a.height
+
+  Vips::Operation.class_eval do
+    old_initialize = instance_method :initialize
+    define_method :initialize do |value|
+      old_initialize.bind(self).(value).tap do
+        self.instance_variable_set "@operation_name", value
       end
     end
+    old_set = instance_method :set
+    define_method :set do |*args|
+      args[1].instance_variable_set "@operation_name", self.instance_variable_get("@operation_name") if args.first == "image"
+      old_set.bind(self).(*args)
+    end
   end
-  a, b = ARGV.drop(1).map do |filename|
-    image = Vips::Image.new_from_file filename
-    ImageMutable.new image.resize(8.fdiv(image.width), vscale: 8.fdiv(image.height)).colourspace("b-w").
-                           resize(100, vscale: 100, kernel: :nearest).colourspace("srgb")
+  Vips::Image.class_eval do
+    def copy
+      return self if caller.first.end_with?("/gems/ruby-vips-2.0.9/lib/vips/operation.rb:148:in `set'") &&
+                     %w{ draw_line draw_circle }.include?(instance_variable_get "@operation_name")
+      method_missing :copy
+    end
   end
 
-  ad = ha >> 128
-  ai = ha - (ad << 128)
-  bd = hb >> 128
-  bi = hb - (bd << 128)
-
-  n = 0
-  [[a, ad, ai], [b, bd, bi]].each do |image, xd, xi|
-    127.downto(0).each do |i|
-      if i > 63
-        y, x = (127 - i).divmod 8
+  require "get_process_mem"
+  a, b = [[a, ad, ai], [b, bd, bi]].map do |image, xd, xi|
+    _127.downto(0).each_with_index do |i, ii|
+      mem = GetProcessMem.new(Process.pid).mb
+      abort ">1000mb of memory consumed" if 1000 < mem
+      if i > _63
+        y, x = (_127 - i).divmod size
       else
-        x, y = (63 - i).divmod 8
+        x, y = (_63 - i).divmod size
       end
-      x = (image.width  * (x + 0.5) / 8).round
-      y = (image.height * (y + 0.5) / 8).round
-      if i > 63
+      x = (width  * (x + 0.5) / size).round
+      y = (height * (y + 0.5) / size).round
+      if i > _63
         (x-2..x+2).map do |x| [
-          [x,  y                                        , x, (y + image.height / 16 - 1) % image.height],
-          [x, (y + image.height / 16 + 1) % image.height, x, (y + image.height /  8    ) % image.height],
+          [x,  y                                  , x, (y + height / size / 2 - 1) % height],
+          [x, (y + height / size / 2 + 1) % height, x, (y + height / size        ) % height],
         ] end
       else
         (y-2..y+2).map do |y| [
-          [ x                                      , y, (x + image.width / 16 - 1) % image.width, y],
-          [(x + image.width / 16 + 1) % image.width, y, (x + image.width /  8    ) % image.width, y],
+          [ x                                , y, (x + width / size / 2 - 1) % width, y],
+          [(x + width / size / 2 + 1) % width, y, (x + width / size        ) % width, y],
         ] end
       end.each do |coords1, coords2|
         n += 1
-        image.draw_line! (1 - xd[i]) * 255, *coords1
-        image.draw_line!      xd[i]  * 255, *coords2
+        image = image.draw_line (1 - xd[i]) * 255, *coords1
+        image = image.draw_line      xd[i]  * 255, *coords2
       end if ai[i] + bi[i] > 0 && ad[i] != bd[i]
-      cx, cy = if i > 63
-        [x, y + 20]
+      cx, cy = if i > _63
+        [x, y + 30]
       else
-        [x + 20, y]
+        [x + 30, y]
       end
-      image.draw_circle!      xd[i]  * 255, cx, cy, 11, fill: true if xi[i] > 0
-      image.draw_circle! (1 - xd[i]) * 255, cx, cy, 10, fill: true if xi[i] > 0
+      image = image.draw_circle      xd[i]  * 255, cx, cy, 11, fill: true if xi[i] > 0
+      image = image.draw_circle (1 - xd[i]) * 255, cx, cy, 10, fill: true if xi[i] > 0
     end
+    image
   end
   puts "distance: #{n / 10}"
+  puts "(above should be equal if raketask works correcly)"
 
-  a.join(b.__getobj__, :horizontal, shim: 10).write_to_file "ab.png"
+  a.join(b, :horizontal, shim: 15).write_to_file "ab.png"
 end
 
+# ./ruby `rbenv which rake` compare_speed
 desc "Benchmarks Dhash, DHashVips::DHash and DHashVips::IDHash"
 task :compare_speed do
   require "dhash"
@@ -169,21 +214,33 @@ task :compare_speed do
   end
 
   require "benchmark"
-  puts "load and calculate:"
+  puts "load and calculate the fingerprint:"
   hashes = []
-  Benchmark.bm 18 do |bm|
-    [Dhash, DHashVips::DHash, DHashVips::IDHash].each do |m|
-      bm.report m do
-        hashes.push filenames.map &m.method(:calculate)
+  Benchmark.bm 19 do |bm|
+    [
+      [Dhash, :calculate],
+      [DHashVips::DHash, :calculate],
+      [DHashVips::IDHash, :fingerprint],
+      [DHashVips::IDHash, :fingerprint, 4],
+    ].each do |m, calc, power|
+      bm.report "#{m} #{power}" do
+        hashes.push filenames.map{ |filename| m.send calc, filename, *power }
       end
     end
   end
-  puts "distance (1000 times):"
-  Benchmark.bm 18 do |bm|
-    [[Dhash, :hamming], [DHashVips::DHash, :hamming], [DHashVips::IDHash, :distance]].zip(hashes) do |(m, dm), hs|
-      bm.report m do
+  hashes[-1, 1] = hashes[-2, 2]     # for `distance` and `distance3` we use the same hashes
+  puts "\nmeasure the distance (1000 times):"
+  Benchmark.bm 28 do |bm|
+    [
+      [Dhash, :hamming],
+      [DHashVips::DHash, :hamming],
+      [DHashVips::IDHash, :distance],
+      [DHashVips::IDHash, :distance3],
+      [DHashVips::IDHash, :distance, 4],
+    ].zip(hashes) do |(m, dm, power), hs|
+      bm.report "#{m} #{dm} #{power}" do
         hs.product hs do |h1, h2|
-          1000.times{ m.send dm, h1, h2 }
+          1000.times{ m.public_send dm, h1, h2 }
         end
       end
     end

data/dhash-vips.gemspec

@@ -1,6 +1,6 @@
 Gem::Specification.new do |spec|
   spec.name          = "dhash-vips"
-  spec.version       = (require_relative "lib/dhash-vips/version"; DHashVips::VERSION)
+  spec.version       = "0.0.5.0"
   spec.author        = "Victor Maslov"
   spec.email         = "nakilon@gmail.com"
   spec.summary       = "dHash and IDHash powered by Vips"
@@ -16,4 +16,5 @@ Gem::Specification.new do |spec|
   spec.add_development_dependency "rake"
   spec.add_development_dependency "rspec-core"
   spec.add_development_dependency "dhash"
+  spec.add_development_dependency "get_process_mem"
 end

data/lib/dhash-vips.rb

@@ -1,4 +1,3 @@
-require_relative "dhash-vips/version"
 require "vips"
 
 module DHashVips
@@ -30,9 +29,19 @@ module DHashVips
   module IDHash
     extend self
 
+    def distance3 a, b
+      return ((a ^ b) & (a | b) >> 128).to_s(2).count "1"
+    end
     def distance a, b
-      # TODO: the hash_size is hardcoded here
-      ((a | b) & (a >> 128 ^ b >> 128)).to_s(2).count "1"
+      size_a, size_b = [a, b].map do |x|
+        case x.size
+        when            32 ; 8
+        when 128, 124, 120 ; 16
+        else          ; fail "invalid size of fingerprint; #{x.size}"
+        end
+      end
+      fail "fingerprints were taken with different `power` param: #{size_a} and #{size_b}" if size_a != size_b
+      ((a ^ b) & (a | b) >> 2 * size_a * size_a).to_s(2).count "1"
     end
 
     @@median = lambda do |array|
@@ -55,10 +64,10 @@ module DHashVips
     fail unless 1 == @@median[[1, 1, 1]]
     fail unless 1 == @@median[[1, 1]]
 
-    def calculate file
-      hash_size = 8
+    def fingerprint file, power = 3
+      size = 2 ** power
       image = Vips::Image.new_from_file file
-      image = image.resize(hash_size.fdiv(image.width), vscale: hash_size.fdiv(image.height)).colourspace("b-w")
+      image = image.resize(size.fdiv(image.width), vscale: size.fdiv(image.height)).colourspace("b-w")
 
       array = image.to_a.map &:flatten
       d1, i1, d2, i2 = [array, array.transpose].flat_map do |a|
@@ -69,7 +78,7 @@ module DHashVips
           d.map{ |c| c.abs >= m ? 1 : 0 }.join.to_i(2),
         ]
       end
-      (((((d1 << hash_size * hash_size) + d2) << hash_size * hash_size) + i1) << hash_size * hash_size) + i2
+      (((((i1 << size * size) + i2) << size * size) + d1) << size * size) + d2
     end
 
   end

data/spec/_spec.rb

@@ -2,8 +2,10 @@ require "dhash-vips"
 
 require "pp"
 
+# TODO tests about `fingerprint(4)`
+
 [
-  [DHashVips::DHash, :hamming, 17, 18, 22, 39],
+  [DHashVips::DHash, :hamming, :calculate, 17, 18, 22, 39],
     # [[0, 17, 29, 27, 22, 29, 30, 29],
     #  [17, 0, 30, 26, 33, 36, 37, 36],
     #  [29, 30, 0, 18, 39, 30, 39, 36],
@@ -12,7 +14,7 @@ require "pp"
     #  [29, 36, 30, 30, 17, 0, 33, 30],
     #  [30, 37, 39, 35, 28, 33, 0, 5],
     #  [29, 36, 36, 34, 23, 30, 5, 0]]
-  [DHashVips::IDHash, :distance, 15, 23, 28, 64],
+  [DHashVips::IDHash, :distance, :fingerprint, 15, 23, 28, 64],
     # [[0, 16, 30, 32, 46, 58, 43, 43],
     #  [16, 0, 28, 28, 47, 59, 46, 47],
     #  [30, 28, 0, 15, 53, 49, 53, 52],
@@ -21,7 +23,7 @@ require "pp"
     #  [58, 59, 49, 53, 23, 0, 44, 44],
     #  [43, 46, 53, 61, 43, 44, 0, 0],
     #  [43, 47, 52, 64, 45, 44, 0, 0]]
-].each do |lib, dm, min_similar, max_similar, min_not_similar, max_not_similar|
+].each do |lib, dm, calc, min_similar, max_similar, min_not_similar, max_not_similar|
 
 describe lib do
 
@@ -39,10 +41,13 @@ describe lib do
       df0a3b93e9412536ee8a11255f974141.jpg
       679634ff89a31279a39f03e278bc9a01.jpg
     }   # these images a consecutive pairs of slightly (but enough for nice asserts) silimar images
+      # 6d97739b4a08f965dc9239dd24382e96.jpg
+      # 1b1d4bde376084011d027bba1c047a4b.jpg
+        # while these two are tend to be false positive match by idhash
     bw1, bw2 = %w{
       71662d4d4029a3b41d47d5baf681ab9a.jpg
       ad8a37f872956666c3077a3e9e737984.jpg
-    }   # these is the same photo but of different size and colorspace
+    }   # these are the same photo but of different size and colorspace
 
     example.metadata[:extra_failure_lines] = []
     FileUtils.mkdir_p dir = "images"
@@ -59,12 +64,12 @@ describe lib do
       end
     end
 
-    hashes = [*images, bw1, bw2].map &described_class.method(:calculate)
+    hashes = [*images, bw1, bw2].map &described_class.method(calc)
     table = MLL::table[described_class.method(dm), [hashes], [hashes]]
 
     # require "pp"
     # pp table
-    # abort
+    # next
 
     aggregate_failures do
       hashes.size.times.to_a.repeated_combination(2) do |i, j|

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: dhash-vips
 version: !ruby/object:Gem::Version
-  version: 0.0.4.1
+  version: 0.0.5.0
 platform: ruby
 authors:
 - Victor Maslov
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2017-10-21 00:00:00.000000000 Z
+date: 2018-02-11 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: ruby-vips
@@ -66,6 +66,20 @@ dependencies:
     - - '>='
       - !ruby/object:Gem::Version
         version: '0'
+- !ruby/object:Gem::Dependency
+  name: get_process_mem
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
 description: 
 email: nakilon@gmail.com
 executables: []
@@ -79,7 +93,6 @@ files:
 - Rakefile
 - dhash-vips.gemspec
 - lib/dhash-vips.rb
-- lib/dhash-vips/version.rb
 - spec/_spec.rb
 homepage: https://github.com/nakilon/dhash-vips
 licenses:

data/lib/dhash-vips/version.rb

@@ -1,3 +0,0 @@
-module DHashVips
-  VERSION = "0.0.4.1"
-end