winnow 0.0.1

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: 09e56c471afeb1bf113f07da677460d4c6b4eb9f
+  data.tar.gz: e778e453f5f50fa49d305b90b97813aaccce4cdd
+SHA512:
+  metadata.gz: 941aaca687e3350ce2bafc6c6e7a9d2209bc36ad23078ed659d308cec42f25435195e561ebfa9f3f8e824d21da437a53abc983821440b16deddd04f39cf765b9
+  data.tar.gz: 0e0075ad638eb46ab271ae5d0c0c896db3c3598200af5e89493eab28b117f9660a2e624a09b7080baf64976dfbd18d7e745ab6d2a2ce3e98d9e484c1e4ab700e

data/.gitignore

@@ -0,0 +1,17 @@
+*.gem
+*.rbc
+.bundle
+.config
+.yardoc
+Gemfile.lock
+InstalledFiles
+_yardoc
+coverage
+doc/
+lib/bundler/man
+pkg
+rdoc
+spec/reports
+test/tmp
+test/version_tmp
+tmp

data/Gemfile

@@ -0,0 +1,2 @@
+source 'https://rubygems.org'
+gemspec

data/LICENSE.txt

@@ -0,0 +1,22 @@
+Copyright (c) 2014 Ulysse Carion
+
+MIT License
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.md

@@ -0,0 +1,150 @@
+# Winnow
+
+A tiny Ruby library for document fingerprinting.
+
+## What is document fingerprinting?
+
+Document fingerprinting converts a document (e.g. a book, a piece of code, or
+any other string) into a much smaller number of hashes called *fingerprints*. If
+two documents share any fingerprints, then this means there is an exact
+substring match between the two documents.
+
+Document fingerprinting has many applications, but the most obvious one is for
+plagiarism detection. By taking fingerprints of two documents, you can detect if
+parts of one document were copied from another.
+
+This library implements a fingerprinting algorithm called *winnowing*, described
+by Saul Schleimer, Daniel S. Wilkerson, and Alex Aiken in a paper titled
+[*Winnowing: Local Algorithms for Document Fingerprinting*][swa_paper].
+
+## Usage
+
+The `Fingerprinter` class takes care of fingerprinting documents. To create a
+fingerprint, you need to provide two parameters, called the *noise threshold*
+and the *guarantee threshold*. When comparing two documents' fingerprints, no
+match shorter than the noise threshold will be detected, but any match at least
+as long as the guarantee threshold is guaranteed to be found.
+
+The proper values for your noise and guarantee thresholds varies by context.
+Experiment with the data you're looking at until you're happy with the results.
+
+Creating a fingerprinter is easy:
+
+```ruby
+fingerprinter = Winnow::Fingerprinter.new(noise_threshold: 10, guarantee_threshold: 18)
+```
+
+Then, use `#fingerprints` get the fingerprints. Optionally, pass `:source`
+(default is `nil`) so that Winnow can later report which document a match is
+from.
+
+```ruby
+document = File.new('hamlet.txt')
+fingerprints = fingerprinter.fingerprints(document.read, source: document)
+```
+
+`#fingerprints` just returns a plain-old Ruby `Hash`. The keys of the hash are
+generated from substrings of the document being fingerprinted. Finding shared
+substrings between documents is as simple as seeing if they share any of the
+keys in their `#fingerprints` hash.
+
+To keep things easier for you, Winnow comes with a `Matcher` class that will
+find matches between two documents.
+
+Here's an example that puts everything together:
+
+```ruby
+require 'winnow'
+
+str_a = <<-EOF
+'Twas brillig, and the slithy toves
+Did gyre and gimble in the wabe;
+This is copied.
+All mimsy were the borogoves,
+And the mome raths outgrabe.
+EOF
+
+str_b = <<-EOF
+"Beware the Jabberwock, my son!
+The jaws that bite, the claws that catch!
+Beware the Jubjub bird, and shun
+The frumious -- This is copied. -- Bandersnatch!"
+EOF
+
+fprinter = Winnow::Fingerprinter.new(
+  guarantee_threshold: 13, noise_threshold: 9)
+
+f1 = fprinter.fingerprints(str_a, source: "Stanza 1")
+f2 = fprinter.fingerprints(str_b, source: "Stanza 2")
+
+matches = Winnow::Matcher.find_matches(f1, f2)
+
+# Because 'This is copied' is longer than the guarantee threshold, there might
+# be a couple of matches found here. For the sake of brevity, let's only look at
+# the first match found.
+match = matches.first
+
+# It's possible for the same key to appear in a document multiple times (e.g. if
+# 'This is copied' appears more than once). Winnow::Matcher will return all
+# matches from the same key in array.
+#
+# In this case, we know there's only one match (because 'This is copied' appears
+# only once in each document), so let's only look at the first one.
+match_a = match.matches_from_a.first
+match_b = match.matches_from_b.first
+
+p match_a.index, match_b.index # 71, 125
+
+match_context_a = str_a[match_a.index - 10 .. match_a.index + 20]
+match_context_b = str_b[match_b.index - 10 .. match_b.index + 20]
+
+# Match from Stanza 1: "e wabe;\nThis is copied.\nAll mim"
+puts "Match from #{match_a.source}: #{match_context_a.inspect}"
+
+# Match from Stanza 2: "ious -- This is copied. -- Band"
+puts "Match from #{match_b.source}: #{match_context_b.inspect}"
+```
+
+You may find that `Matcher` doesn't handle your exact use-case. That's not a
+problem. [The built-in matcher.rb file](lib/winnow/matcher.rb)
+is only about 10 lines of code, so you could easily make your own.
+
+## :boom: :bomb: A major caveat with `String#hash` :bomb: :boom:
+
+In order to avoid [algorithmic complexity attacks][wiki_aca], the value returned
+from Ruby's `String#hash` method [changes every time you restart the
+interpreter][hash_stackoverflow]:
+
+```sh
+$ irb
+2.0.0p353 :001 > "hello".hash
+ => 482951767139383391
+2.0.0p353 :002 > exit
+
+$ irb
+2.0.0p353 :001 > "hello".hash
+ => 3216751850140847920
+2.0.0p353 :002 > exit
+```
+
+(This is the case even if you're using JRuby.)
+
+This means that although the winnowing algorithm *should* allow you to
+precalculate a document's fingerprints and store them somewhere, doing so in
+Ruby will not work unless you're careful to make sure you never restart your
+Ruby runtime.
+
+### A workaround
+
+Winnow looks for the presence of a `String#consistent_hash` method. If it finds
+one, it'll call that rather than call `String#hash`. You can therefore describe
+your own hash function if you want to precalculate fingerprint data.
+
+I've put together a super-simple (but effective) gem called
+[consistent_hash][consistent_hash] that implements exactly this. It's about a
+dozen lines of MRI C code and it'll probably work for you as well.
+
+[swa_paper]: http://theory.stanford.edu/~aiken/publications/papers/sigmod03.pdf
+[wiki_aca]: http://en.wikipedia.org/wiki/Algorithmic_complexity_attack
+[hash_stackoverflow]: http://stackoverflow.com/questions/23331725/why-are-ruby-hash-methods-randomized
+[consistent_hash]: https://github.com/ucarion/consistent_hash

data/Rakefile

@@ -0,0 +1,6 @@
+require 'bundler/gem_tasks'
+require 'rspec/core/rake_task'
+
+RSpec::Core::RakeTask.new(:spec)
+
+task default: :spec

data/lib/winnow.rb

@@ -0,0 +1,12 @@
+require 'winnow/version'
+require 'winnow/preprocessor'
+require 'winnow/fingerprinter'
+require 'winnow/matcher'
+
+module Winnow
+  class Location < Struct.new(:source, :index)
+  end
+
+  class MatchDatum < Struct.new(:matches_from_a, :matches_from_b)
+  end
+end

data/lib/winnow/fingerprinter.rb

@@ -0,0 +1,60 @@
+module Winnow
+  class Fingerprinter
+    attr_reader :guarantee, :noise, :preprocessor
+    alias_method :guarantee_threshold, :guarantee
+    alias_method :noise_threshold, :noise
+
+    def initialize(params)
+      @guarantee = params[:guarantee_threshold] || params[:t]
+      @noise = params[:noise_threshold] || params[:k]
+      @preprocessor = params[:preprocessor] || Preprocessors::Plaintext.new
+    end
+
+    def fingerprints(str, params = {})
+      source = params[:source]
+
+      fingerprints = {}
+
+      windows(str, source).each do |window|
+        least_fingerprint = window.min_by { |fingerprint| fingerprint[:value] }
+        value = least_fingerprint[:value]
+        location = least_fingerprint[:location]
+
+        (fingerprints[value] ||= []) << location
+      end
+
+      fingerprints
+    end
+
+    private
+
+    def windows(str, source)
+      k_grams(str, source).each_cons(window_size)
+    end
+
+    def window_size
+      guarantee - noise + 1
+    end
+
+    def k_grams(str, source)
+      tokens(str).each_cons(noise).map do |tokens_k_gram|
+        value = hash(tokens_k_gram.map { |(char)| char }.join)
+        location = Location.new(source, tokens_k_gram.first[1])
+
+        {value: value, location: location}
+      end
+    end
+
+    def tokens(str)
+      preprocessor.preprocess(str)
+    end
+
+    def hash(str)
+      if str.respond_to?(:consistent_hash)
+        str.consistent_hash
+      else
+        str.hash
+      end
+    end
+  end
+end

data/lib/winnow/matcher.rb

@@ -0,0 +1,16 @@
+module Winnow
+  class Matcher
+    class << self
+      def find_matches(fingerprints_a, fingerprints_b, params = {})
+        whitelist = params[:whitelist] || []
+
+        matched_values = fingerprints_a.keys & fingerprints_b.keys - whitelist
+
+        matched_values.map do |value|
+          matches_a, matches_b = fingerprints_a[value], fingerprints_b[value]
+          MatchDatum.new(matches_a, matches_b)
+        end
+      end
+    end
+  end
+end

data/lib/winnow/preprocessor.rb

@@ -0,0 +1,10 @@
+module Winnow
+  class Preprocessor
+    def preprocess
+      raise NotImplementedError
+    end
+  end
+end
+
+require 'winnow/preprocessors/plaintext'
+require 'winnow/preprocessors/source_code'

data/lib/winnow/preprocessors/plaintext.rb

@@ -0,0 +1,9 @@
+module Winnow
+  module Preprocessors
+    class Plaintext < Preprocessor
+      def preprocess(str)
+        str.chars.each_with_index.to_a
+      end
+    end
+  end
+end

data/lib/winnow/preprocessors/source_code.rb

@@ -0,0 +1,41 @@
+require 'rouge'
+
+module Winnow
+  module Preprocessors
+    class SourceCode < Preprocessor
+      attr_reader :lexer
+
+      def initialize(language)
+        @lexer = Rouge::Lexer.find(language)
+      end
+
+      def preprocess(str)
+        current_index = 0
+        processed_chars = []
+
+        lexer.lex(str).to_a.each do |token|
+          type, chunk = token
+
+          processed_chunk = case
+          when type <= Rouge::Token::Tokens::Name
+            'x'
+          when type <= Rouge::Token::Tokens::Comment
+            ''
+          when type <= Rouge::Token::Tokens::Text
+            ''
+          else
+            chunk
+          end
+
+          processed_chars += processed_chunk.chars.map do |c|
+            [c, current_index]
+          end
+
+          current_index += chunk.length
+        end
+
+        processed_chars
+      end
+    end
+  end
+end

data/lib/winnow/version.rb

@@ -0,0 +1,3 @@
+module Winnow
+  VERSION = "0.0.1"
+end

data/spec/fingerprinter_spec.rb

@@ -0,0 +1,56 @@
+require 'spec_helper'
+
+describe Winnow::Fingerprinter do
+  describe '#initialize' do
+    it 'accepts :guarantee_threshold, :noise_threshold' do
+      fprinter = Winnow::Fingerprinter.new(
+        guarantee_threshold: 0, noise_threshold: 1)
+      expect(fprinter.guarantee_threshold).to eq 0
+      expect(fprinter.noise_threshold).to eq 1
+    end
+
+    it 'accepts :t and :k' do
+      fprinter = Winnow::Fingerprinter.new(t: 0, k: 1)
+      expect(fprinter.guarantee_threshold).to eq 0
+      expect(fprinter.noise_threshold).to eq 1
+    end
+  end
+
+  describe '#fingerprints' do
+    it 'hashes strings to get keys' do
+      # if t = k = 1, then each character will become a fingerprint
+      fprinter = Winnow::Fingerprinter.new(t: 1, k: 1)
+      fprints = fprinter.fingerprints("abcdefg")
+
+      hashes = ('a'..'g').map(&:hash)
+
+      expect(fprints.keys).to eq hashes
+    end
+
+    it 'chooses the smallest hash per window' do
+      # window size = t - k + 1 = 2 ; for a two-char string, the sole
+      # fingerprint should just be from the char with the smallest hash value
+      fprinter = Winnow::Fingerprinter.new(t: 2, k: 1)
+      fprints = fprinter.fingerprints("ab")
+
+      expect(fprints.keys.length).to eq 1
+      expect(fprints.keys.first).to eq ["a", "b"].map(&:hash).min
+    end
+
+    it 'correctly reports the location of a fingerprint' do
+      fprinter = Winnow::Fingerprinter.new(t: 1, k: 1)
+      fprints = fprinter.fingerprints("a\nb\ncde\nfg", source: "example")
+
+      fprint_d = fprints['d'.hash].first
+
+      expect(fprint_d.index).to eq 5
+      expect(fprint_d.source).to eq "example"
+    end
+
+    it 'uses #consistent_hash when possible' do
+      String.any_instance.should_receive(:consistent_hash)
+
+      Winnow::Fingerprinter.new(t: 1, k: 1).fingerprints("a")
+    end
+  end
+end

data/spec/matcher_spec.rb

@@ -0,0 +1,61 @@
+require 'spec_helper'
+
+describe Winnow::Matcher do
+  describe '#find_matches' do
+    def make_locations(*indices)
+      indices.map { |n| Winnow::Location.new(nil, n) }
+    end
+
+    let(:fprint1) do
+      {
+        0 => make_locations(0),
+        1 => make_locations(1, 2),
+      }
+    end
+
+    let(:fprint2) do
+      {
+        0 => make_locations(3),
+        1 => make_locations(4),
+        3 => make_locations(5)
+      }
+    end
+
+    let(:matches) { Winnow::Matcher.find_matches(fprint1, fprint2) }
+
+    def match_with_loc(index, matches = matches)
+      matches.find do |data|
+        data.matches_from_a.find { |loc| loc.index == index } ||
+          data.matches_from_b.find { |loc| loc.index == index }
+      end
+    end
+
+    it 'returns an array of match data' do
+      expect(matches).to be_an(Array)
+      expect(matches.first).to be_a(Winnow::MatchDatum)
+    end
+
+    it 'reports a match when values are equal' do
+      match = match_with_loc(0)
+      matchloc_b = match.matches_from_b.first
+      expect(matchloc_b.index).to eq 3
+    end
+
+    it 'reports nothing when there is no match' do
+      match = match_with_loc(5)
+      expect(match).to be_nil
+    end
+
+    it 'reports all matches when multi matches' do
+      match = match_with_loc(1)
+      expect(match.matches_from_a.length).to eq 2
+      expect(match.matches_from_b.length).to eq 1
+    end
+
+    it 'ignores whitelisted values' do
+      matches = Winnow::Matcher.find_matches(fprint1, fprint2, whitelist: [0])
+
+      expect(match_with_loc(0, matches)).to be_nil
+    end
+  end
+end

data/spec/preprocessor_spec.rb

@@ -0,0 +1,49 @@
+require 'spec_helper'
+
+describe Winnow::Preprocessors::Plaintext do
+  subject { Winnow::Preprocessors::Plaintext.new }
+
+  it 'converts a string to an array of chars and indices' do
+    str = "abcde"
+    char_indices = [['a', 0], ['b', 1], ['c', 2], ['d', 3], ['e', 4]]
+
+    expect(subject.preprocess(str)).to eq char_indices
+  end
+end
+
+describe Winnow::Preprocessors::SourceCode do
+  subject { Winnow::Preprocessors::SourceCode.new(:java) }
+
+  it 'simplifies a string, but remembers correct locations' do
+    str = "i = 5"
+    result = [['x', 0], ['=', 2], ['5', 4]]
+
+    expect(subject.preprocess(str)).to eq result
+  end
+
+  it 'groups up the indices of a single token' do
+    str = "int i"
+    result = [['i', 0], ['n', 0], ['t', 0], ['x', 4]]
+
+    expect(subject.preprocess(str)).to eq result
+  end
+
+  def reconstruct_string(processed)
+    processed.map { |(char, _)| char }.join
+  end
+
+  it 'removes whitespace' do
+    str = '3;    '
+    expect(reconstruct_string(subject.preprocess(str))).to eq '3;'
+  end
+
+  it 'removes variable names' do
+    str = 'class MyClass { int myInteger = 5 }'
+    expect(reconstruct_string(subject.preprocess(str))).to eq 'classx{intx=5}'
+  end
+
+  it 'removes comments' do
+    str = 'fooBar();//this is a comment'
+    expect(reconstruct_string(subject.preprocess(str))).to eq 'x();'
+  end
+end

data/spec/spec_helper.rb

@@ -0,0 +1,8 @@
+require 'bundler/setup'
+Bundler.setup
+
+require 'winnow'
+
+RSpec.configure do |config|
+  config.color_enabled = true
+end

data/winnow.gemspec

@@ -0,0 +1,27 @@
+# coding: utf-8
+lib = File.expand_path('../lib', __FILE__)
+$LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
+require 'winnow/version'
+
+Gem::Specification.new do |spec|
+  spec.name          = "winnow"
+  spec.version       = Winnow::VERSION
+  spec.authors       = ["Ulysse Carion"]
+  spec.email         = ["ulyssecarion@gmail.com"]
+  spec.description   = %q{A tiny Ruby library that implements Winnowing,
+                          an algorithm for document fingerprinting.}
+  spec.summary       = %q{Simple document fingerprinting and plagiarism detection.}
+  spec.homepage      = "https://github.com/ucarion/winnow"
+  spec.license       = "MIT"
+
+  spec.files         = `git ls-files`.split($/)
+  spec.executables   = spec.files.grep(%r{^bin/}) { |f| File.basename(f) }
+  spec.test_files    = spec.files.grep(%r{^(test|spec|features)/})
+  spec.require_paths = ["lib"]
+
+  spec.add_dependency 'rouge', '~> 1.3'
+
+  spec.add_development_dependency "bundler", "~> 1.3"
+  spec.add_development_dependency "rake"
+  spec.add_development_dependency "rspec", "~> 2.14"
+end

metadata

@@ -0,0 +1,123 @@
+--- !ruby/object:Gem::Specification
+name: winnow
+version: !ruby/object:Gem::Version
+  version: 0.0.1
+platform: ruby
+authors:
+- Ulysse Carion
+autorequire: 
+bindir: bin
+cert_chain: []
+date: 2014-05-08 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: rouge
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: '1.3'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: '1.3'
+- !ruby/object:Gem::Dependency
+  name: bundler
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: '1.3'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: '1.3'
+- !ruby/object:Gem::Dependency
+  name: rake
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: rspec
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: '2.14'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: '2.14'
+description: |-
+  A tiny Ruby library that implements Winnowing,
+                            an algorithm for document fingerprinting.
+email:
+- ulyssecarion@gmail.com
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- .gitignore
+- Gemfile
+- LICENSE.txt
+- README.md
+- Rakefile
+- lib/winnow.rb
+- lib/winnow/fingerprinter.rb
+- lib/winnow/matcher.rb
+- lib/winnow/preprocessor.rb
+- lib/winnow/preprocessors/plaintext.rb
+- lib/winnow/preprocessors/source_code.rb
+- lib/winnow/version.rb
+- spec/fingerprinter_spec.rb
+- spec/matcher_spec.rb
+- spec/preprocessor_spec.rb
+- spec/spec_helper.rb
+- winnow.gemspec
+homepage: https://github.com/ucarion/winnow
+licenses:
+- MIT
+metadata: {}
+post_install_message: 
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - '>='
+    - !ruby/object:Gem::Version
+      version: '0'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - '>='
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubyforge_project: 
+rubygems_version: 2.1.11
+signing_key: 
+specification_version: 4
+summary: Simple document fingerprinting and plagiarism detection.
+test_files:
+- spec/fingerprinter_spec.rb
+- spec/matcher_spec.rb
+- spec/preprocessor_spec.rb
+- spec/spec_helper.rb