wilderpeople 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: 4591ba4d3f4654cae88164e3106ef022b0dfbc32
+  data.tar.gz: d364ee06d64951745386aba6b7fadbc3783cd4e5
+SHA512:
+  metadata.gz: 756050f37ba7b4ed278e80276b977d006ed74fd7ef8cfc1c29819ffed3c5c9623d1b03b085d9b53711872508365c2d547aad1379b86532485556df6eb18a2fc9
+  data.tar.gz: 4068c8206a34f4f0bac26d7eb7f7c34663ed090a8cd2f18510aefc15965ae54425c252c9053a84644b5acd228cee282187f5a3e59d072776cd25e1a0fc8549d9

data/.gitignore

@@ -0,0 +1,9 @@
+/.bundle/
+/.yardoc
+/Gemfile.lock
+/_yardoc/
+/coverage/
+/doc/
+/pkg/
+/spec/reports/
+/tmp/

data/.travis.yml

@@ -0,0 +1,5 @@
+sudo: false
+language: ruby
+rvm:
+  - 2.0.0
+before_install: gem install bundler -v 1.13.6

data/Gemfile

@@ -0,0 +1,4 @@
+source 'https://rubygems.org'
+
+# Specify your gem's dependencies in wilderpeople.gemspec
+gemspec

data/LICENSE.txt

@@ -0,0 +1,21 @@
+The MIT License (MIT)
+
+Copyright (c) 2017 Rob Nichols and Warwickshire County Council
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

data/README.md

@@ -0,0 +1,173 @@
+# Wilderpeople
+
+This tool was specifically design to assist in the identification of people
+based on sets of matching data. However, it could also be used to retrieve
+other types of data from an array.
+
+The tool assumes you have an array of hashes, where each hash describes an
+unique item or person. It then allows you to pass in a "matching dataset" and
+in return it will return the hash that best matches that data, **if there
+is only one best match.**
+
+## Installation
+
+Add this line to your application's Gemfile:
+
+```ruby
+gem 'wilderpeople'
+```
+
+And then execute:
+
+    $ bundle
+
+Or install it yourself as:
+
+    $ gem install wilderpeople
+
+## Usage
+
+Given the following dataset:
+
+```ruby
+data = [
+  {surname: 'Bloggs', forename: 'Fred', gender: 'Male'},
+  {surname: 'Bloggs', forename: 'Winifred', gender: 'Female'},
+  {surname: 'Bloggs', forename: 'Jane', gender: 'Female'}
+]
+```
+
+We have someone called Fred Bloggs, and we wish to find the hash that matches
+them.
+
+To do that we first need to identify the criteria on which to base the match.
+As we just have a forename and surname, let's start with:
+
+```ruby
+config = {
+  must: {
+    surname: :exact,
+    forename: :exact
+  }
+}
+```
+
+That is, to get a match the hash in the dataset must exactly match both the
+surname and the forename.
+
+With that information, we can now perform a search:
+
+```ruby
+search = Wilderpeople::Search.new(data: data, config: config)
+person = search.find surname: 'Bloggs', forename: 'Fred'
+person == {surname: 'Bloggs', forename: 'Fred', gender: 'Male'}
+```
+
+Success .... however, it turns out that the Winifred Bloggs we're after is
+Frederick Blogg's sister, who also goes by the name 'Fred'.
+
+So we could try changing the criteria in the config to look for a female Bloggs:
+
+```ruby
+config = {
+  must: {
+    surname: :exact,
+    gender: :exact
+  }
+}
+search = Wilderpeople::Search.new(data: data, config: config)
+person = search.find surname: 'Bloggs', gender: 'Female'
+person == nil
+```
+
+Unfortunately both Winifred Bloggs, and Jane Bloggs match that criteria, so the
+search is unable to find an unique match, so returns `nil`.
+
+So how do we identify the correct hash. The solution is to use fuzzy logic.
+
+```ruby
+config = {
+  must: {
+    surname: :exact,
+    forename: :hypocorism,
+    gender: :exact
+  }
+}
+search = Wilderpeople::Search.new(data: data, config: config)
+person = search.find surname: 'Bloggs', forename: 'Fred',  gender: 'Female'
+person == {surname: 'Bloggs', forename: 'Winifred', gender: 'Female'}
+```
+
+### Criteria configuration
+
+The `config` is a hash of one or two parts. The parts being:
+
+* **must** The rules that must be matched
+* **can** The rules that may be matched
+
+Each part is itself a hash where the keys match the data keys, and the values
+are the matchers to be used to compare the data in those keys.
+
+```ruby
+config = { must: { surname: :exact } }
+```
+
+With this configuration, the `exact` matcher will be used to compare the
+`surname` of each hash in the data.
+
+### Matchers
+
+The matchers are defined by the protected methods in the
+[Wilderpeople::Matcher class](https://github.com/reggieb/wilderpeople/blob/master/lib/wilderpeople/matcher.rb).
+Please read the comments in this class to find out how each matcher operates.
+
+### Can matching
+
+If possible the match will be attempted using only the `must` criteria, but
+if more that one match is returned, the system can then use the `can` criteria
+to try and find a unique match.
+
+So in the example above, Winifred could have been found using:
+
+```ruby
+config = {
+  must: {
+    surname: :exact,
+    gender: :exact
+  },
+  can: {
+    forename: :hypocorism
+  }
+}
+search = Wilderpeople::Search.new(data: data, config: config)
+person = search.find surname: 'Bloggs', forename: 'Fred',  gender: 'Female'
+person == {surname: 'Bloggs', forename: 'Winifred', gender: 'Female'}
+```
+
+With this configuration the `hypocorism` matcher would only be called if the
+search was unable to find a unique record by `surname` and `gender`. That is,
+if we then went on to get Frederick's hash using the same configuration:
+
+```ruby
+person = search.find surname: 'Bloggs', forename: 'Fred',  gender: 'Male'
+```
+
+A unique record would be identified without having to do the `hypocorism` match.
+
+## Development
+
+After checking out the repo, run `bin/setup` to install dependencies. Then, run
+`rake` to run the tests. You can also run `bin/console` for an interactive
+prompt that will allow you to experiment.
+
+## Contributing
+
+Bug reports and pull requests are welcome on GitHub at
+https://github.com/reggieb/wilderpeople.
+
+
+## License
+
+The gem is available as open source under the terms of the
+[MIT License](http://opensource.org/licenses/MIT).
+

data/Rakefile

@@ -0,0 +1,10 @@
+require "bundler/gem_tasks"
+require "rake/testtask"
+
+Rake::TestTask.new(:test) do |t|
+  t.libs << "test"
+  t.libs << "lib"
+  t.test_files = FileList['test/**/*_test.rb']
+end
+
+task :default => :test

data/bin/console

@@ -0,0 +1,14 @@
+#!/usr/bin/env ruby
+
+require "bundler/setup"
+require "wilderpeople"
+
+# You can add fixtures and/or initialization code here to make experimenting
+# with your gem easier. You can also use a different console, if you like.
+
+# (If you use this, don't forget to add pry to your Gemfile!)
+# require "pry"
+# Pry.start
+
+require "irb"
+IRB.start

data/bin/setup

@@ -0,0 +1,8 @@
+#!/usr/bin/env bash
+set -euo pipefail
+IFS=$'\n\t'
+set -vx
+
+bundle install
+
+# Do any other automated setup that you need to do here

data/lib/wilderpeople/matcher.rb

@@ -0,0 +1,153 @@
+require 'hypocorism'
+require 'date'
+require 'levenshtein'
+module Wilderpeople
+  class Matcher
+
+    class << self
+      attr_writer :levenshtein_threshold
+
+      def levenshtein_threshold
+        @levenshtein_threshold ||= 0.3
+      end
+
+      # Using `method_missing` so that instead of having to do:
+      #   Matcher.new(a, b).exact
+      # We can do
+      #   Matcher.exact(a, b)
+      def method_missing(method, *args, &block)
+        if protected_instance_methods.include?(method)
+          by(method, args[0], args[1])
+        else
+          super
+        end
+      end
+
+      # Another way to use matcher is:
+      #   Matcher.by :exact, a, b
+      def by(method, a, b)
+        raise "Method must be defined" unless method
+        new(a, b).send(method)
+      end
+    end
+
+    attr_reader :a, :b, :dates
+
+    # Passing arguments into initialize so that prep can be done just once
+    # and before main test.
+    # Also means that I don't have to worry about the result of one match
+    # poluting the next match
+    def initialize(a, b)
+      @a, @b = a.clone, b.clone
+      prep
+    end
+
+    protected # note that method_missing looks for protected_instance_methods
+
+    # `exact` used in other methods so needs to work with either stored `a` and `b`
+    # or items passed into it
+    def exact(x = a, y = b)
+      x == y
+    end
+
+    # All the right letters but not necessarily in the right order
+    def transposed
+      return true if exact
+      exact *[a,b].collect{|x| x.chars.sort}
+    end
+
+    # Designed for matching street names, so don't have to worry about
+    # Road/Rb and Street/St.
+    # Note that if matching one word, just that word is matched
+    def exact_except_last_word
+      return true if exact
+      words = [a,b].collect(&:split)
+      exact *words.collect{|w| w.size == 1 ? w : w[0..-2]}
+    end
+
+    # Match the first letter only
+    def first_letter
+      return true if exact
+      exact *[a,b].collect{|x| x[0]}
+    end
+
+    # Match 'Foo bar' with 'Foobar'
+    def exact_no_space
+      return true if exact
+      exact *[a,b].collect{|x| x.gsub(/\s/, '')}
+    end
+
+    # Match English first names with alternative forms
+    # So 'Robert' matches 'Rob'
+    def hypocorism
+      Hypocorism.match(a,b)
+    end
+
+    # Match dates
+    def date
+      return true if exact
+      @dates = [a, b].collect{|x| date_parse(x)}
+      exact(*dates)
+    rescue ArgumentError # Error raised when entry won't parse
+      false
+    end
+
+    # Matches dates, but also handles day and month being swapped.
+    # So 3/5/2001 matches 5/3/2001
+    def fuzzy_date
+      return true if date
+      return false unless dates
+      return false if dates[1].day > 12
+      exact(dates[0], swap_day_month(@dates[1]))
+    end
+
+    # User Levenshtein distance to compare similar strings.
+    # The Levenshtein distance is a string metric for measuring the difference
+    # between two sequences. Informally, the Levenshtein distance between two
+    # words is the minimum number of single-character edits required to change
+    # one word into the other.
+    # See https://github.com/tliff/levenshtein
+    def fuzzy(threshold = self.class.levenshtein_threshold)
+      exact
+      return false if a.empty? || b.empty?
+      !!Levenshtein.normalized_distance(a,b, threshold)
+    end
+
+    private
+
+    def prep
+      [a,b].each do |attr|
+        case attr
+        when String
+          attr.downcase!
+          attr.strip! if attr
+        when Array
+          attr.collect{|x| prep(x)}
+        else
+          attr
+        end
+      end
+    end
+
+    def swap_day_month(date)
+      Date.new(date.year, date.day, date.month)
+    end
+
+    def date_parse(string)
+      Date.parse(string)
+    rescue ArgumentError
+      try_american_format(string)
+    end
+
+    def try_american_format(string)
+      Date.strptime string, "%m/%d/%Y"
+    rescue ArgumentError
+      # Need to catch instance where system is using American format
+      try_proper_format(string)
+    end
+
+    def try_proper_format(string)
+      Date.strptime string, "%d/%m/%Y"
+    end
+  end
+end

data/lib/wilderpeople/search.rb

@@ -0,0 +1,64 @@
+require 'active_support/core_ext/hash/indifferent_access'
+module Wilderpeople
+  class Search
+    attr_reader :data, :config, :result, :args
+    def initialize(data: [], config: {})
+      @data = data.collect(&:with_indifferent_access)
+      @config = config
+    end
+
+    def find(args)
+      @result = data
+      @args = args
+      select_must || select_can
+    end
+
+    private
+
+    def select_must
+      return if result.empty?
+      return unless must_rules
+      @result = result.select do |datum|
+        must_rules.all? do |key, matcher_method|
+          return false unless datum[key] && args[key]
+          Matcher.by matcher_method, datum[key], args[key]
+        end
+      end
+      result.first if result.size == 1
+    end
+
+    def must_rules
+      config[:must]
+    end
+
+    def select_can
+      return if result.empty?
+      return unless can_rules
+      # Get all of the matches for each of the can rules.
+      matches = can_rules.collect do |key, matcher_method|
+        result.select do |datum|
+          [matcher_method, datum[key], args[key]]
+          Matcher.by matcher_method, datum[key], args[key]
+        end
+      end
+      # Then determine if one datum appears in more matches
+      # than any other, and if so return that one.
+      occurrences = find_occurrences(matches)
+      count_of_commonest = occurrences.values.max
+      if occurrences.values.count(count_of_commonest) == 1
+        occurrences.rassoc(count_of_commonest)[0]
+      end
+    end
+
+    def can_rules
+      config[:can]
+    end
+
+    # Returns a hash with each item as key, and the count of occurrences as value
+    # See: http://jerodsanto.net/2013/10/ruby-quick-tip-easily-count-occurrences-of-array-elements/
+    def find_occurrences(array)
+      array.flatten.each_with_object(Hash.new(0)){ |item,count| count[item] += 1 }
+    end
+
+  end
+end

data/lib/wilderpeople/version.rb

@@ -0,0 +1,3 @@
+module Wilderpeople
+  VERSION = "0.1.0"
+end

data/lib/wilderpeople.rb

@@ -0,0 +1,7 @@
+require 'wilderpeople/version'
+require 'wilderpeople/matcher'
+require 'wilderpeople/search'
+
+module Wilderpeople
+
+end

data/wilderpeople.gemspec

@@ -0,0 +1,32 @@
+# coding: utf-8
+lib = File.expand_path('../lib', __FILE__)
+$LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
+require 'wilderpeople/version'
+
+Gem::Specification.new do |spec|
+  spec.name          = "wilderpeople"
+  spec.version       = Wilderpeople::VERSION
+  spec.authors       = ["Rob Nichols"]
+  spec.email         = ["robnichols@warwickshire.gov.uk"]
+
+  spec.summary       = %q{A tool for fuzzy matching people data}
+  spec.description   = %q{Allow people data from one source to be compared with anothe source}
+  spec.homepage      = "https://github.com/reggieb/wilderpeople"
+  spec.license       = "MIT"
+
+  spec.files         = `git ls-files -z`.split("\x0").reject do |f|
+    f.match(%r{^(test|spec|features)/})
+  end
+  spec.bindir        = "exe"
+  spec.executables   = spec.files.grep(%r{^exe/}) { |f| File.basename(f) }
+  spec.require_paths = ["lib"]
+
+  spec.add_dependency 'hypocorism', "~> 0.0.2"
+  spec.add_dependency 'levenshtein'
+  spec.add_dependency 'activesupport'
+
+  spec.add_development_dependency "bundler", "~> 1.13"
+  spec.add_development_dependency "rake", "~> 10.0"
+  spec.add_development_dependency "minitest", "~> 5.0"
+  spec.add_development_dependency "pry"
+end

metadata

@@ -0,0 +1,155 @@
+--- !ruby/object:Gem::Specification
+name: wilderpeople
+version: !ruby/object:Gem::Version
+  version: 0.1.0
+platform: ruby
+authors:
+- Rob Nichols
+autorequire: 
+bindir: exe
+cert_chain: []
+date: 2017-03-09 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: hypocorism
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: 0.0.2
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: 0.0.2
+- !ruby/object:Gem::Dependency
+  name: levenshtein
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: activesupport
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: bundler
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: '1.13'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: '1.13'
+- !ruby/object:Gem::Dependency
+  name: rake
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: '10.0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: '10.0'
+- !ruby/object:Gem::Dependency
+  name: minitest
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: '5.0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: '5.0'
+- !ruby/object:Gem::Dependency
+  name: pry
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+description: Allow people data from one source to be compared with anothe source
+email:
+- robnichols@warwickshire.gov.uk
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- .gitignore
+- .travis.yml
+- Gemfile
+- LICENSE.txt
+- README.md
+- Rakefile
+- bin/console
+- bin/setup
+- lib/wilderpeople.rb
+- lib/wilderpeople/matcher.rb
+- lib/wilderpeople/search.rb
+- lib/wilderpeople/version.rb
+- wilderpeople.gemspec
+homepage: https://github.com/reggieb/wilderpeople
+licenses:
+- MIT
+metadata: {}
+post_install_message: 
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - '>='
+    - !ruby/object:Gem::Version
+      version: '0'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - '>='
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubyforge_project: 
+rubygems_version: 2.4.8
+signing_key: 
+specification_version: 4
+summary: A tool for fuzzy matching people data
+test_files: []