lycopodium 0.0.2 → 0.0.3

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: 4bc1c70f9f05818e7ba034a54541e4b00a73c467
+  data.tar.gz: 45aa5ecad0f8ea72fbd3e52dd3c955b009d1b5b7
+SHA512:
+  metadata.gz: 980b7eb2f0a907186731a62f81903fb3e9eee5985190c9aed11bb61355437e120acfd8f3169145eff44bad2f3e14a2e7e8efdbc8cc725dc0850d06e5a6217078
+  data.tar.gz: a36b387c15a1ce568f07a8606947178be56380384540ff90a055ac0d97a3c928bc9703a9688ac1a5e2d7a720121d3f24346609c0d24f3b29fc5ca4898dde68e4

data/.travis.yml

@@ -1,7 +1,6 @@
 language: ruby
 rvm:
-  - 1.8.7
   - 1.9.2
   - 1.9.3
   - 2.0.0
-  - ree
+  - 2.1.0

data/Gemfile

@@ -1,4 +1,4 @@
-source "http://rubygems.org"
+source "https://rubygems.org/"
 
 # Specify your gem's dependencies in the gemspec
 gemspec

data/LICENSE

@@ -1,4 +1,4 @@
-Copyright (c) 2013 Open North Inc.
+Copyright (c) 2013 James McKinney
 
 Permission is hereby granted, free of charge, to any person obtaining
 a copy of this software and associated documentation files (the

data/README.md

@@ -1,13 +1,37 @@
 # Lycopodium Finds Fingerprints
 
-[![Build Status](https://secure.travis-ci.org/opennorth/lycopodium.png)](http://travis-ci.org/opennorth/lycopodium)
-[![Dependency Status](https://gemnasium.com/opennorth/lycopodium.png)](https://gemnasium.com/opennorth/lycopodium)
-[![Coverage Status](https://coveralls.io/repos/opennorth/lycopodium/badge.png?branch=master)](https://coveralls.io/r/opennorth/lycopodium)
-[![Code Climate](https://codeclimate.com/github/opennorth/lycopodium.png)](https://codeclimate.com/github/opennorth/lycopodium)
+[![Gem Version](https://badge.fury.io/rb/lycopodium.svg)](https://badge.fury.io/rb/lycopodium)
+[![Build Status](https://secure.travis-ci.org/jpmckinney/lycopodium.png)](https://travis-ci.org/jpmckinney/lycopodium)
+[![Dependency Status](https://gemnasium.com/jpmckinney/lycopodium.png)](https://gemnasium.com/jpmckinney/lycopodium)
+[![Coverage Status](https://coveralls.io/repos/jpmckinney/lycopodium/badge.png)](https://coveralls.io/r/jpmckinney/lycopodium)
+[![Code Climate](https://codeclimate.com/github/jpmckinney/lycopodium.png)](https://codeclimate.com/github/jpmckinney/lycopodium)
 
-Test what transformations you can make to a set of values without creating collisions.
+Lycopodium does two things:
 
-> Historically, Lycopodium powder, the spores of Lycopodium and related plants, was used as a fingerprint powder. – [Wikipedia](http://en.wikipedia.org/wiki/Fingerprint_powder#Composition)
+1. Test what transformations you can make to a set of values without creating collisions.
+1. Find [unique key](http://en.wikipedia.org/wiki/Unique_key) constraints in a data table.
+
+> Historically, Lycopodium powder, the spores of Lycopodium and related plants, was used as a fingerprint powder. – [Wikipedia](https://en.wikipedia.org/wiki/Fingerprint_powder#Composition)
+
+## What it tries to solve
+
+### Find a key collision method
+
+Let's say you have an authoritative list of names: for example, a list of organization names from a [company register](https://www.ic.gc.ca/app/scr/cc/CorporationsCanada/fdrlCrpSrch.html?locale=en_CA). You want to match a messy list of names – for example, a list of government contractors published by a city – against this authoritative list.
+
+For context, [Open Refine](http://openrefine.org/) offers [two methods to solve this problem](https://github.com/OpenRefine/OpenRefine/wiki/Clustering-In-Depth):
+
+* Key collision methods group names that transform into the same fingerprint; transformations include lowercasing letters, removing whitespace and punctuation, sorting words, etc.
+
+* Nearest neighbor methods group names that are close to each other, using distance functions like [Levenshtein distance](https://en.wikipedia.org/wiki/Levenshtein_distance) and [Prediction by Partial Matching](https://en.wikipedia.org/wiki/Prediction_by_Partial_Matching).
+
+Key collision methods tend to be fast and strict, whereas nearest neighbor methods are more likely to produce false positives, especially when dealing with short strings.
+
+If you want fast and strict reconciliation, Lycopodium lets you figure out what transformations can be applied to an authoritative list of names without creating collisions between names. Those transformations can then be safely applied to the names on the messy list to match against the authoritative list.
+
+### Find a unique key in a data table
+
+Let's say you have a data table: for example, the City of Toronto publishes [voting records grouped by city councillor](http://app.toronto.ca/tmmis/getAdminReport.do?function=prepareMemberVoteReport). You want to instead group the voting records by motion being voted on. However, the data table doesn't contain one, single column identifing the motion. You instead need to identify the combination of columns that identify the motion. In other words, you are looking for the data table's [unique key](http://en.wikipedia.org/wiki/Unique_key). Lycopodium does this.
 
 ## Usage
 
@@ -15,7 +39,23 @@ Test what transformations you can make to a set of values without creating colli
 require 'lycopodium'
 ```
 
-First, write a method that transforms a value, for example:
+### Find a unique key in a data table
+
+```ruby
+table = [
+  ['foo', 'bar', 'baz'],
+  ['foo', 'bar', 'bzz'],
+  ['foo', 'zzz', 'bzz'],
+]
+Lycopodium.unique_key(table)
+# => [1, 2]
+```
+
+The values of the second and third columns – taken together - are unique for each row in the table. In other words, you can uniquely identify a row by taking the values of its second and third columns.
+
+### Find a key collision method
+
+Write a method that transforms a value, for example:
 
 ```ruby
 meth1 = ->(string) do
@@ -23,10 +63,10 @@ meth1 = ->(string) do
 end
 ```
 
-Then, initialize a `Lycopodium` instance with a set of values and the transformation method:
+Then, initialize a `Lycopodium` instance with a set of values and the method:
 
 ```ruby
-set = Lycopodium.new(["foo", "f o o"], meth1)
+set = Lycopodium.new(["foo", "f o o", " bar "], meth1)
 ```
 
 Lastly, test whether the method creates collisions between the members of the set:
@@ -47,21 +87,47 @@ meth2 = ->(string) do
 end
 ```
 
-It will return the mapping from original to transformed string:
+It will return the mapping from original to transformed string (hence `value_to_fingerprint`):
 
-    {"foo" => "FOO", "f o o" => "F O O"}
+```ruby
+set.function = meth2
+set.value_to_fingerprint
+# => {"foo"=>"FOO", "f o o"=>"F O O", "bar"=>" BAR "}
+```
 
 We thus learn that whitespace disambiguates between members of the set, but letter case does not.
 
-To remove all members of the set that collide after transformation, run:
+If you can't find a suitable method, you can remove all values that collide after transformation:
 
 ```ruby
+set.function = meth1
 set_without_collisions = set.reject_collisions
+# => [" bar "]
+set_without_collisions.value_to_fingerprint
+# => {" bar "=>"bar"}
 ```
 
 A `Lycopodium` instance otherwise behaves as an array.
 
-## Method definition
+### Use the key collision method 
+
+You can now apply the method to other values…
+
+```ruby
+messy = "\tbar\n"
+fingerprint = meth1.call(messy)
+# => "bar"
+```
+
+… and match against your original values:
+
+```
+fingerprint_to_value = set_without_collisions.value_to_fingerprint.invert
+fingerprint_to_value.fetch(fingerprint)
+# => " bar "
+```
+
+### Method definition
 
 Besides the `->` syntax above, you can define the same method as:
 
@@ -96,8 +162,10 @@ end
 meth = Object.method(:func)
 ```
 
-## Bugs? Questions?
+## Related projects
 
-This project's main repository is on GitHub: [http://github.com/opennorth/lycopodium](http://github.com/opennorth/lycopodium), where your contributions, forks, bug reports, feature requests, and feedback are greatly welcomed.
+* [Nomenklatura](http://nomenklatura.okfnlabs.org/) is a web service to maintain a canonical list of entities and to match messy input against it, either via the user interface or via Open Refine reconciliation.
+* [dedupe](https://github.com/open-city/dedupe) is a Python library to determine when two records are about the same thing.
+* [name-cleaver](https://github.com/sunlightlabs/name-cleaver) is a Python library to parse and standardize the names of people and organizations.
 
-Copyright (c) 2013 Open North Inc., released under the MIT license
+Copyright (c) 2013 James McKinney, released under the MIT license

data/lib/lycopodium.rb

@@ -3,29 +3,53 @@ require "set"
 class Lycopodium < Array
   class Error < StandardError; end
   class Collision < Error; end
+  class RaggedRow < Error; end
 
   attr_accessor :function
 
+  def self.unique_key(data)
+    columns_size = data.first.size
+    data.each do |row|
+      unless row.size == columns_size
+        raise RaggedRow, row.inspect
+      end
+    end
+
+    columns = (0...columns_size).to_a
+    1.upto(columns_size) do |k|
+      columns.combination(k) do |combination|
+        if unique_key?(data, combination)
+          return combination
+        end
+      end
+    end
+    nil
+  end
+
   # @param [Array] set a set of values
   # @param [Proc] function a method that transforms a value
   def initialize(set, function = lambda{|value| value})
     replace(set)
-    self.function = function
+    @function = function
   end
 
+  # Removes all members of the set that collide after transformation.
+  #
   # @return [Array] the members of the set without collisions
   def reject_collisions
     hashes, collisions = hashes_and_collisions
 
-    items = hashes.reject do |_,hash|
-      collisions.include?(hash)
-    end.map do |item,_|
-      item
+    items = []
+    hashes.each do |item,hash|
+      unless collisions.include?(hash)
+        items << item
+      end
     end
-
     self.class.new(items, function)
   end
 
+  # Returns a mapping from the original to the transformed value.
+  #
   # @return [Hash] a mapping from the original to the transformed value
   # @raise [Collision] if the method creates collisions between members of the set
   def value_to_fingerprint
@@ -49,6 +73,17 @@ class Lycopodium < Array
 
 private
 
+  def self.unique_key?(data, combination)
+    set = Set.new
+    data.each_with_index do |row,index|
+      set.add(row.values_at(*combination))
+      if set.size <= index
+        return false
+      end
+    end
+    true
+  end
+
   def hashes_and_collisions
     collisions = Set.new
 

data/lib/lycopodium/version.rb

@@ -1,3 +1,3 @@
-module Lycopodium
-  VERSION = "0.0.2"
+class Lycopodium < Array
+  VERSION = "0.0.3"
 end

data/lycopodium.gemspec

@@ -5,10 +5,10 @@ Gem::Specification.new do |s|
   s.name        = "lycopodium"
   s.version     = Lycopodium::VERSION
   s.platform    = Gem::Platform::RUBY
-  s.authors     = ["Open North"]
-  s.email       = ["info@opennorth.ca"]
-  s.homepage    = "http://github.com/opennorth/lycopodium"
-  s.summary     = %q{Test what transformations you can make to a set of unique strings without creating collisions}
+  s.authors     = ["James McKinney"]
+  s.homepage    = "https://github.com/jpmckinney/lycopodium"
+  s.summary     = %q{Test what transformations you can make to a set of values without creating collisions}
+  s.license     = 'MIT'
 
   s.files         = `git ls-files`.split("\n")
   s.test_files    = `git ls-files -- {test,spec,features}/*`.split("\n")
@@ -17,4 +17,5 @@ Gem::Specification.new do |s|
 
   s.add_development_dependency('rspec', '~> 2.10')
   s.add_development_dependency('rake')
+  s.add_development_dependency('coveralls')
 end

data/spec/lycopodium_spec.rb

@@ -0,0 +1,64 @@
+require File.expand_path(File.dirname(__FILE__) + '/spec_helper')
+
+describe Lycopodium do
+  describe '#initialize' do
+    it 'should use an identify function by default' do
+      set = Lycopodium.new([])
+      set.function.call(1).should == 1
+    end
+
+    it 'should accept a function as an argument' do
+      set = Lycopodium.new([], lambda{|value| value * 2})
+      set.function.call(1).should == 2
+    end
+  end
+
+  let :collisions do
+    Lycopodium.new(['foo', 'f o o', 'bar'], lambda{|string| string.gsub(' ', '')})
+  end
+
+  let :no_collisions do
+    Lycopodium.new(['foo', 'f o o', 'bar'], lambda{|string| string.upcase})
+  end
+
+  describe '#unique_key' do
+    it 'should return a unique key if a unique key is found' do
+      Lycopodium.unique_key([
+        ['foo', 'bar', 'baz'],
+        ['foo', 'bar', 'bzz'],
+        ['foo', 'zzz', 'bzz'],
+      ]).should == [1, 2]
+    end
+
+    it 'should return nil if no unique key is found' do
+      Lycopodium.unique_key([
+        ['foo', 'bar'],
+        ['foo', 'bar'],
+        ['foo', 'bar'],
+      ]).should == nil
+    end
+
+    it 'should raise an error if ragged rows' do
+      expect{Lycopodium.unique_key([
+        ['foo'],
+        ['foo', 'bar'],
+      ])}.to raise_error(Lycopodium::RaggedRow, %(["foo", "bar"]))
+    end
+  end
+
+  describe '#reject_collisions' do
+    it 'should remove all members of the set that collide after transformation' do
+      collisions.reject_collisions.should == ['bar']
+    end
+  end
+
+  describe '#value_to_fingerprint' do
+    it 'should return a mapping from the original to the transformed value' do
+      no_collisions.value_to_fingerprint.should == {'foo' => 'FOO', 'f o o' => 'F O O', 'bar' => 'BAR'}
+    end
+
+    it 'should raise an error if the method creates collisions between members of the set' do
+      expect{collisions.value_to_fingerprint}.to raise_error(Lycopodium::Collision, %("foo", "f o o" => "foo"))
+    end
+  end
+end

data/spec/spec_helper.rb

@@ -1,3 +1,7 @@
 require 'rubygems'
+
+require 'coveralls'
+Coveralls.wear!
+
 require 'rspec'
 require File.dirname(__FILE__) + '/../lib/lycopodium'

metadata

@@ -1,58 +1,66 @@
 --- !ruby/object:Gem::Specification
 name: lycopodium
 version: !ruby/object:Gem::Version
-  version: 0.0.2
-  prerelease: 
+  version: 0.0.3
 platform: ruby
 authors:
-- Open North
+- James McKinney
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2013-08-16 00:00:00.000000000 Z
+date: 2015-02-03 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: rspec
   requirement: !ruby/object:Gem::Requirement
-    none: false
     requirements:
-    - - ~>
+    - - "~>"
       - !ruby/object:Gem::Version
         version: '2.10'
   type: :development
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
-    none: false
     requirements:
-    - - ~>
+    - - "~>"
       - !ruby/object:Gem::Version
         version: '2.10'
 - !ruby/object:Gem::Dependency
   name: rake
   requirement: !ruby/object:Gem::Requirement
-    none: false
     requirements:
-    - - ! '>='
+    - - ">="
       - !ruby/object:Gem::Version
         version: '0'
   type: :development
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
-    none: false
     requirements:
-    - - ! '>='
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: coveralls
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
       - !ruby/object:Gem::Version
         version: '0'
 description: 
-email:
-- info@opennorth.ca
+email: 
 executables: []
 extensions: []
 extra_rdoc_files: []
 files:
-- .gitignore
-- .travis.yml
-- .yardopts
+- ".gitignore"
+- ".travis.yml"
+- ".yardopts"
 - Gemfile
 - LICENSE
 - README.md
@@ -61,37 +69,33 @@ files:
 - lib/lycopodium.rb
 - lib/lycopodium/version.rb
 - lycopodium.gemspec
+- spec/lycopodium_spec.rb
 - spec/spec_helper.rb
-homepage: http://github.com/opennorth/lycopodium
-licenses: []
+homepage: https://github.com/jpmckinney/lycopodium
+licenses:
+- MIT
+metadata: {}
 post_install_message: 
 rdoc_options: []
 require_paths:
 - lib
 required_ruby_version: !ruby/object:Gem::Requirement
-  none: false
   requirements:
-  - - ! '>='
+  - - ">="
     - !ruby/object:Gem::Version
       version: '0'
-      segments:
-      - 0
-      hash: 750622676358250438
 required_rubygems_version: !ruby/object:Gem::Requirement
-  none: false
   requirements:
-  - - ! '>='
+  - - ">="
     - !ruby/object:Gem::Version
       version: '0'
-      segments:
-      - 0
-      hash: 750622676358250438
 requirements: []
 rubyforge_project: 
-rubygems_version: 1.8.25
+rubygems_version: 2.2.2
 signing_key: 
-specification_version: 3
-summary: Test what transformations you can make to a set of unique strings without
-  creating collisions
+specification_version: 4
+summary: Test what transformations you can make to a set of values without creating
+  collisions
 test_files:
+- spec/lycopodium_spec.rb
 - spec/spec_helper.rb