linkage 0.0.8 → 0.1.0.pre

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: ad6a9ee6a6add94a342e3d02d49d8a4bfeb9122b
+  data.tar.gz: a989e8e810602dfcd4da596fcc1154a922dedccd
+SHA512:
+  metadata.gz: 0d78da4904100679826cf76ae07f7daf984c12e2a762aceb0ad2d1387c5a5778403a782ff104ce48c1fd3ec5588c728fe2b8ea7a79a896da153a8d91c7e4cb14
+  data.tar.gz: 6797e5598f47413022d18c6732cbf0613efaccba886cda56eb390e3344d131ccfd977b5992c5a2c4557cfbc4f93bf747ddffe849770be5860e793f27fe3e2286

data/.gitignore

@@ -8,3 +8,4 @@ test.rb
 results.db
 .rbenv-version
 bin
+Gemfile.lock

data/.yardopts

@@ -0,0 +1 @@
+-m markdown

data/Gemfile

@@ -1,21 +1,3 @@
-source 'http://rubygems.org'
+source 'https://rubygems.org'
 
 gemspec
-
-group :development do
-  gem 'bundler'
-  gem 'test-unit'
-  gem 'mocha'
-  gem 'yard'
-  gem 'rake'
-  gem 'versionomy'
-  gem 'sqlite3', :platforms => :ruby
-  gem 'mysql2', :platforms => :ruby
-  gem 'jdbc-sqlite3', :platforms => :jruby
-  gem 'jdbc-mysql', :platforms => :jruby
-  gem 'rdiscount'
-  gem 'guard-test'
-  gem 'guard-yard', :platforms => :ruby_19
-  gem 'rb-inotify', '~> 0.8.8'
-  gem 'debugger'
-end

data/Gemfile-java

@@ -0,0 +1,3 @@
+source 'https://rubygems.org'
+
+gemspec

data/README.markdown

@@ -1,52 +1,106 @@
-# linkage
+# Linkage
 
-Linkage is a library for record linkage between one or two database tables.
+Linkage is a Ruby library for record linkage between one or two database tables.
+
+## What is record linkage?
+
+In an ideal world, records that reference the same entity can be easily
+identified. Unfortunately, this isn't always the case. Sometimes there are no
+good identifiers in the datasets that you're interested in (ID, social security
+number, etc). In such cases, it is necessary to use other means to determine
+which records refer to which entity, and this process is known as **record
+linkage**.
+
+## Prerequisites
+
+In order to use Linkage, the records you want to link must be in a database.
+Linkage has the ability to perform record linkage across different kinds of
+databases, so it's okay if your records are not all in the same place.
+
+Since Linkage uses [Sequel](http://sequel.jeremyevans.net/) to communicate with
+databases, any database that Sequel supports will work. See [Connecting to a
+database](http://sequel.jeremyevans.net/documentation.html) on the Sequel
+website for more information about what databases are supported.
 
 ## Usage
 
-Linkage uses Sequel to talk to databases, so any database that Sequel can
-talk to, Linkage can talk to. You just give Linkage the Sequel-style URI
-and the database table name:
+To perform a record linkage, Linkage needs information about the following:
+datasets, result set, and comparators. A dataset refers to a table in a
+database. A result set is a place to put score and match information that
+Linkage generates.  Comparators describe how records are compared.
+
+A dataset is created via the `Linkage::Dataset` class, along with a connection URI
+and a table name:
 
-    ds = Linkage::Dataset.new('mysql://example.com/database_name', 'table_name')
+```ruby
+ds = Linkage::Dataset.new('mysql://example.com/database_name', 'table_name')
+```
 
-To describe a linkage, you use the `Dataset#link_with` method.
+Result sets have different options depending on what storage medium you're
+using (CSV or database). For CSVs, you could use:
 
-    parents = Linkage::Dataset.new('postgres://example.com/foo', 'parents')
-    children = Linkage::Dataset.new('mysql://some-other-host.net/bar', 'children')
-    config = parents.link_with(children) do
-      lhs[:first_name].must == rhs[:parent_first_name]
-      lhs[:last_name].must == rhs[:parent_last_name]
-      lhs[:last_name].must_not == "Smith"  # exclude parents with the last
-                                           # name "Smith"
+```ruby
+result_set = Linkage::ResultSet['csv'].new('~/my_results')
+```
 
-      save_results_in('sqlite://results.db') # see below
-    end
+In this case, scores and matches will be saved in CSV files in the `my_results`
+directory in your home folder.
 
-Note that the datasets don't have to be in the same database, or even on
-the same machine.
+To describe a linkage, you can use the `Dataset#link_with` method. This creates
+a linkage configuration that you can use to describe how you want the records in
+each dataset to be compared. For example:
+
+```ruby
+demo = Linkage::Dataset.new('postgres://example.com/foo', 'demographics')
+visits = Linkage::Dataset.new('mysql://some-other-host.net/bar', 'visits')
+result_set = Linkage::ResultSet['csv'].new('~/my_results')
+config = demo.link_with(visits, result_set) do |config|
+  config.compare([:first_name, :last_name], [:first_name, :last_name], :equal)
+end
+```
+
+This linkage would match records from a demographics table to records in a table
+with information about doctor visits by using first name and last name.
+
+The `compare` method creates a `Compare` comparator. This is the simplest
+comparator in Linkage, and it just compares fields with the operator you specify
+(`:equal`, `:less_than`, `:greater_than`, etc). When a comparator compares
+two records, it gives the pair of records a score between 0 and 1. In the case
+of the example above, records that have the same first name and last name get a
+score of 1, and records that don't get a score of 0 (or sometimes, they aren't
+scored and assumed to have a score of 0).
+
+Other comparators are `Strcompare` for approximate string matching and
+`Within` for matching numbers within a range.
 
 To run a linkage, use a Runner with the resulting configuration from
 `Dataset#link_with`:
 
-    runner = Linkage::SingleThreadedRunner.new(config)
-    runner.execute
+```ruby
+runner = Linkage::Runner.new(config)
+runner.execute
+```
+
+After running a linkage, there will be a list of matches in a CSV file or
+database, depending on how you configured your result set.
+
+The default way linkage determines if two records match is by comparing the
+average score to a threshold value (which is 0.5 by default). You can configure
+the threshold value like so: `config.threshold = 0.9`.
 
-The runner saves results in a database that you specify in the configuration
-(via the `save_results_in` method). It stores its results in two database
-tables: `groups` and `groups_records`. The `groups` table contains all of the
-unique combinations of values in your datasets, and `groups_records` maps
-records to groups.
+## Other examples
 
-You can also link a dataset to itself:
+Linking a dataset to itself:
 
-    births = Linkage::Dataset.new('postgres://example.com/hospital_data', 'births')
-    config = births.link_with(births) do
-      lhs[:mother_first_name].must == rhs[:mother_first_name]
-      lhs[:mother_last_name].must == rhs[:mother_last_name]
-    end
-    runner = Linkage::SingleThreadedRunner.new(config, 'sqlite://results.db')
-    runner.execute
+```ruby
+births = Linkage::Dataset.new('postgres://example.com/hospital_data', 'births')
+result_set = Linkage::ResultSet['csv'].new('~/my_birth_results')
+config = births.link_with(births, result_set) do |config|
+  config.compare([:mother_first_name, :mother_last_name], [:mother_first_name, :mother_last_name], :equal)
+end
+runner = Linkage::Runner.new(config)
+runner.execute
+```
 
 The above example would find birth records that have mothers with the same
 name.
@@ -62,6 +116,6 @@ name.
 
 ## Copyright
 
-Copyright (c) 2011 Vanderbilt University. See LICENSE.txt for
+Copyright (c) 2011-2014 Vanderbilt University. See LICENSE.txt for
 further details.
 

data/Rakefile

@@ -1,16 +1,4 @@
-# encoding: utf-8
-
-require 'rubygems'
-require 'bundler'
-begin
-  Bundler.setup(:default, :development)
-rescue Bundler::BundlerError => e
-  $stderr.puts e.message
-  $stderr.puts "Run `bundle install` to install missing gems"
-  exit e.status_code
-end
-require 'rake'
-require "bundler/gem_tasks"
+require 'bundler/gem_tasks'
 
 require 'rake/testtask'
 Rake::TestTask.new(:test) do |test|
@@ -18,9 +6,22 @@ Rake::TestTask.new(:test) do |test|
   test.pattern = 'test/**/test_*.rb'
   test.verbose = true
 end
-
 task :default => :test
 
+namespace :test do
+  Rake::TestTask.new(:unit) do |test|
+    test.libs << 'lib' << 'test'
+    test.pattern = 'test/unit/**/test_*.rb'
+    test.verbose = true
+  end
+
+  Rake::TestTask.new(:integration) do |test|
+    test.libs << 'lib' << 'test'
+    test.pattern = 'test/integration/**/test_*.rb'
+    test.verbose = true
+  end
+end
+
 require 'yard'
 YARD::Rake::YardocTask.new do |t|
   t.files = ['lib/**/*.rb']
@@ -28,7 +29,7 @@ end
 
 # Yoinked from https://github.com/rails/rails/blob/master/railties/lib/rails/tasks/annotations.rake
 namespace :notes do
-  ["OPTIMIZE", "FIXME", "TODO"].each do |annotation|
+  ['OPTIMIZE', 'FIXME', 'TODO'].each do |annotation|
     desc "Enumerate all #{annotation} annotations"
     task annotation.downcase.intern do
       SourceAnnotationExtractor.enumerate annotation

data/TODO

@@ -0,0 +1,4 @@
+Features
+- add matcher algorithms
+- change comparator_id to more than just an index; need to get comparator ids
+  from result set instead of configuration

data/lib/linkage/comparator.rb

@@ -1,172 +1,167 @@
 module Linkage
-  # @abstract Abstract class to represent record comparators.
+  # {Comparator} is the superclass for comparators in Linkage. Comparators are
+  # used to compare two records and compute scores based on how closely the two
+  # records relate.
+  #
+  # Each comparator should inherit from {Comparator} and declare itself as
+  # simple or advanced by overriding {#type} (the default is simple). Simple
+  # comparators must define the {#score} method that uses data from two records
+  # and returns a number (`Integer` or `Float`) between 0 and 1 (inclusive).
+  # Advanced comparators must define both {#score_dataset} and {#score_datasets}
+  # that use one or two {Dataset}s respectively to create scores.
+  #
+  # Each comparator can be registered via the {.register} function. This allows
+  # {Configuration} a way to find a comparator by name via
+  # {Configuration#method_missing}. For example, `config.compare(...)` creates a
+  # new {Comparators::Compare} object, since that comparator is registered under
+  # the name `"compare"`.
+  #
+  # See documentation for the methods below for more information.
+  #
+  # @abstract
   class Comparator
-    # Register a new comparator.
-    #
-    # @param [Class] klass Comparator subclass
-    def self.register(klass)
-      name = nil
-      begin
-        name = klass.comparator_name
-      rescue NotImplementedError
-        raise ArgumentError, "comparator_name class method must be defined"
-      end
-
-      if !klass.instance_methods(false).include?(:score)
-        raise ArgumentError, "class must define the score method"
-      end
-
-      begin
-        if klass.parameters.length > 0
-          @comparators ||= {}
-          @comparators[name] = klass
-        else
-          raise ArgumentError, "class must have at least one parameter"
+    include Observable
+
+    class << self
+      # Register a new comparator. Subclasses must define at least {#score} for
+      # simple comparators, or {#score_dataset} and {#score_datasets} for
+      # advanced comparators. Otherwise, an `ArgumentError` will be raised when
+      # you try to call {.register}. The `name` parameter is used in
+      # {Configuration#method_missing} as an easy way for users to select
+      # comparators for their linkage.
+      #
+      # @param [String] name Comparator name used in {.klass_for}
+      # @param [Class] klass Comparator subclass
+      def register(name, klass)
+        methods = klass.instance_methods(false)
+        if !methods.include?(:score) && (!methods.include?(:score_datasets) || !methods.include?(:score_dataset))
+          raise ArgumentError, "class must define either #score or both #score_datasets and #score_dataset methods"
         end
-      rescue NotImplementedError
-        raise ArgumentError, "parameters class method must be defined"
+
+        @comparators ||= {}
+        @comparators[name] = klass
       end
 
-      begin
-        range = klass.score_range
-        if !range.is_a?(Range) || !range.first.is_a?(Numeric) ||
-              !range.last.is_a?(Numeric)
-          raise ArgumentError, "score_range must be a Range of two numbers"
-        end
-      rescue NotImplementedError
-        raise ArgumentError, "score_range class method must be defined"
+      # Return a registered Comparator subclass or `nil` if it doesn't exist.
+      #
+      # @param [String] name of registered comparator
+      # @return [Class, nil]
+      def klass_for(name)
+        @comparators ? @comparators[name] : nil
       end
+      alias :[] :klass_for
     end
 
-    def self.[](name)
-      @comparators ? @comparators[name] : nil
+    # Return the type of this comparator. When {#type} returns `:simple`,
+    # {#score_and_notify} is called by {Runner#score_records} with each pair of
+    # records in order to create scores. When {#type} returns `:advanced`,
+    # either {#score_dataset} or {#score_datasets} is called by
+    # {Runner#score_records}. In advanced mode, it is left up to the
+    # {Comparator} subclass to determine which records to compare and how to
+    # compare them.
+    #
+    # @return [Symbol] either `:simple` or `:advanced`
+    def type
+      @type || :simple
     end
 
-    # @abstract Override this to return the name of the comparator.
-    # @return [String]
-    def self.comparator_name
+    # Override this to return the score of the linkage strength of two records.
+    # This method is used to score records by {Runner#score_records} when
+    # {#type} returns `:simple`.
+    #
+    # @abstract
+    # @param [Hash] record_1 data from first record
+    # @param [Hash] record_2 data from second record
+    # @return [Numeric] value between 0 and 1 (inclusive)
+    def score(record_1, record_2)
       raise NotImplementedError
     end
 
-    # @abstract Override this to require a specific number of arguments of a
-    #   certain class. To require two parameters of either String or Integer,
-    #   do something like this:
+    # Override this to score the linkage strength of records in two datasets.
+    # This method is used to score records by {Runner#score_records} when
+    # {#type} returns `:advanced` and {Configuration} is setup to link two
+    # datasets together.
     #
-    #     @@parameters = [[String, Integer], [String, Integer]]
-    #     def self.parameters
-    #       @@parameters
-    #     end
+    # Since each {Dataset} delegates to a
+    # {http://sequel.jeremyevans.net/rdoc/classes/Sequel/Dataset.html `Sequel::Dataset`},
+    # you can use any
+    # {http://sequel.jeremyevans.net/rdoc/classes/Sequel/Dataset.html `Sequel::Dataset`}
+    # methods that you wish in order to select records to compare.
     #
-    #   At least one argument must be defined.
-    # @return [Array]
-    def self.parameters
-      raise NotImplementedError
-    end
-
-    # @abstract Override this to return a Range of the possible scores for the
-    #   comparator.
-    # @return [Range]
-    def self.score_range
+    # To record scores, subclasses must call
+    # {http://ruby-doc.org/stdlib/libdoc/observer/rdoc/Observable.html `Observable#notify_observers`}
+    # like so:
+    #
+    # ```ruby
+    # changed
+    # notify_observers(self, record_1, record_2, score)
+    # ```
+    #
+    # This works by notifying any observers, typically {ScoreRecorder}, that a
+    # new score has been generated. {ScoreRecorder#update} then calls
+    # {ScoreSet#add_score} with comparator ID, the primary key of each record
+    # and the score.
+    #
+    # @abstract
+    # @param [Linkage::Dataset] dataset_1
+    # @param [Linkage::Dataset] dataset_2
+    # @see http://ruby-doc.org/stdlib/libdoc/observer/rdoc/Observable.html Observable
+    # @see http://sequel.jeremyevans.net/rdoc/classes/Sequel/Dataset.html Sequel::Dataset
+    def score_datasets(dataset_1, dataset_2)
       raise NotImplementedError
     end
 
-    attr_reader :args, :lhs_args, :rhs_args
-
-    # Create a new Comparator object.
-    # @param [Linkage::MetaObject, Hash] args Comparator arguments
-    def initialize(*args)
-      @args = args
-      @lhs_args = []
-      @rhs_args = []
-      @options = args.last.is_a?(Hash) ? args.pop : {}
-      process_args
-    end
-
-    # @abstract Override this to return the score of the linkage strength of
-    #   two records.
-    # @return [Numeric]
-    def score(record_1, record_2)
+    # Override this to score the linkage strength of records in one dataset.
+    # This method is used to score records by {Runner#score_records} when
+    # {#type} returns `:advanced` and {Configuration} is setup to link a
+    # dataset to itself.
+    #
+    # Since a {Dataset} delegates to a
+    # {http://sequel.jeremyevans.net/rdoc/classes/Sequel/Dataset.html `Sequel::Dataset`},
+    # you can use any
+    # {http://sequel.jeremyevans.net/rdoc/classes/Sequel/Dataset.html `Sequel::Dataset`}
+    # methods that you wish in order to select records to compare.
+    #
+    # To record scores, subclasses must call
+    # {http://ruby-doc.org/stdlib/libdoc/observer/rdoc/Observable.html `Observable#notify_observers`}
+    # like so:
+    #
+    # ```ruby
+    # changed
+    # notify_observers(self, record_1, record_2, score)
+    # ```
+    #
+    # This works by notifying any observers, typically {ScoreRecorder}, that a
+    # new score has been generated.  {ScoreRecorder#update} then calls
+    # {ScoreSet#add_score} with comparator ID, the primary key of each record
+    # and the score.
+    #
+    # @abstract
+    # @param [Linkage::Dataset] dataset
+    # @see http://ruby-doc.org/stdlib/libdoc/observer/rdoc/Observable.html Observable
+    # @see http://sequel.jeremyevans.net/rdoc/classes/Sequel/Dataset.html Sequel::Dataset
+    def score_dataset(dataset)
       raise NotImplementedError
     end
 
-    private
-
-    def process_args
-      parameters = self.class.parameters
-      if parameters.length != @args.length
-        raise ArgumentError, "wrong number of arguments (#{@args.length} for #{parameters.length})"
-      end
-
-      first_side = nil
-      second_side = nil
-      @args.each_with_index do |arg, i|
-        type = arg.ruby_type[:type]
-
-        parameter_types = parameters[i]
-        if parameter_types.last.is_a?(Hash)
-          parameter_options = parameter_types[-1]
-          parameter_types = parameter_types[0..-2]
-        else
-          parameter_options = {}
-        end
-
-        if parameter_types[0] != :any && !parameter_types.include?(type)
-          raise TypeError, "expected type #{parameters[i].join(" or ")}, got #{type}"
-        end
-
-        if parameter_options.has_key?(:values) && arg.raw? && !parameter_options[:values].include?(arg.object)
-          raise ArgumentError, "argument #{i + 1} (#{arg.object.inspect}) was not one of the expected values: #{parameter_options[:values].inspect}"
-        end
-
-        if parameter_options.has_key?(:same_type_as)
-          arg_index = parameter_options[:same_type_as]
-          other_type = @args[arg_index].ruby_type[:type]
-          if type != other_type
-            raise TypeError, "argument #{i + 1} (#{type}) was expected to have the same type as argument #{arg_index + 1} (#{other_type})"
-          end
-        end
-
-        if parameter_options.has_key?(:static) &&
-              parameter_options[:static] != arg.static?
-          raise TypeError, "argument #{i + 1} was expected to #{arg.static? ? "not be" : "be"} static"
-        end
-
-        if !arg.static?
-          if first_side.nil?
-            first_side = arg.side
-          elsif arg.side != first_side && second_side.nil?
-            second_side = arg.side
-          end
-
-          valid_side = true
-          case parameter_options[:side]
-          when :first
-            if arg.side != first_side
-              valid_side = false
-            end
-          when :second
-            if second_side.nil? || arg.side != second_side
-              valid_side = false
-            end
-          end
-
-          if !valid_side
-            raise TypeError, "argument #{i + 1} was expected to have a different side value"
-          end
-
-          case arg.side
-          when :lhs
-            @lhs_args << arg
-          when :rhs
-            @rhs_args << arg
-          end
-        end
-      end
+    # Calls {#score} with two hashes of record data. The result is then used to
+    # notify any observers (typically {ScoreRecorder}).
+    #
+    # This method is used by {Runner#score_records} when {#type} returns
+    # `:simple`. Subclasses should override {#score} to implement the scoring
+    # algorithm.
+    #
+    # @param [Hash] record_1 data from first record
+    # @param [Hash] record_2 data from second record
+    def score_and_notify(record_1, record_2)
+      value = score(record_1, record_2)
+      changed
+      notify_observers(self, record_1, record_2, value)
     end
   end
 end
 
 path = File.expand_path(File.join(File.dirname(__FILE__), "comparators"))
-require File.join(path, "binary")
 require File.join(path, "compare")
 require File.join(path, "within")
+require File.join(path, "strcompare")