ofac 1.1.11

data/.document

@@ -0,0 +1,5 @@
+README.rdoc
+lib/**/*.rb
+bin/*
+features/**/*.feature
+LICENSE

data/.gitignore

@@ -0,0 +1,6 @@
+*.sw?
+.DS_Store
+coverage
+rdoc
+pkg
+nbproject

data/History.txt

@@ -0,0 +1,72 @@
+== 0.1.0 2009-05-7
+
+* 1 major enhancement:
+  * Table creation and data load task complete
+
+== 1.0.0 2009-05-11
+
+* 1 major enhancement:
+  * Initail release
+
+== 1.1.0 2009-05-12
+
+* 1 minor enhancement:
+  * Modified the match alogorithm to reduct the score if there is not an address or city match if the data is in the database.
+
+== 1.1.2 2009-05-13
+
+* 2 minor changes:
+  * Changed the sql in the initial search to do a like instead of a soundex.  For short names, the soundex returned almost the entire table
+    making the process take too long.
+  * Also changed the sql to only return individuals, also for the sake of performance.
+
+== 1.1.3 2009-05-15
+
+* 1 bug fix:
+  * fixed a bug that threw an error if a space was passed in for the name.
+
+== 1.1.4 2009-06-02
+
+* 1 minor enhancement:
+  * Improved performance by ignoring initial in names when searching the database for possible hits.
+
+== 1.1.5 2009-06-03
+
+* 1 bug fix:
+  * fixed a bug that threw an error if only single character initials are passed in for the name.
+
+== 1.1.6 2009-06-29
+
+* 1 minor enhancement:
+  * Allow passing of first and last name seperately...to improve performance.
+
+== 1.1.7 2009-06-30
+
+* 1 bug fix:
+  * fixed error when passing a nil value into the new :first_name or :last_name hash values when initializing the Ofac object.
+
+== 1.1.8 2009-06-30
+
+* 1 bug fix:
+  * Refactored the select on OfacSdn to use the AR connection instead of building sql and using the raw connection.  Fixes a bug
+    introducted in 1.1.6 where quotes in the name raised an error.
+
+== 1.1.9 2009-07-24
+
+* 1 minor enhancement:
+  * Added a method, db_hit? for when your more concerned with speed than accuracy.  db_hit? will retun true if there is an exact name match
+    in the ofac_sdn database.  This method ignores address and city and does not produce a score.
+    Usage: Ofac.new({:name => 'Oscar Hernandez', :city => 'Clearwater', :address => '123 somewhere ln'}).db_hit?
+
+== 1.1.10 2009-07-28
+
+* 1 minor enhancement:
+  * Modified the select in OfacSdn to use the city to pull records from the database.  This is another compromise to improve performance,
+    but still get a score.  The db_hit? method is still faster, but this will still calculate a score for accuracy.
+
+== 1.1.11 2009-07-29
+
+* 1 minor enhancement:
+  * Removed the changes from 1.1.10.  1.1.10 produced a score of 0 when a higher score was more accurate.
+    Changed the scoring algorithm to not include partial matches on sounds like.  This code has little impact on score, and
+    has a very high overhead on performance.

data/LICENSE

@@ -0,0 +1,20 @@
+Copyright (c) 2009 Kevin Tyll
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/PostInstall.txt

@@ -0,0 +1,11 @@
+For more information on ofac, see http://kevintyll.github.com/ofac/
+
+* To create the necessary db migration, from the command line, run:
+    script/generate ofac_migration
+* Require the gem in your environment.rb file in the Rails::Initializer block:
+    config.gem 'kevintyll-ofac', :lib => 'ofac'
+* To load your table with the current OFAC data, from the command line, run:
+    rake ofac:update_data
+
+    * The OFAC data is not updated with any regularity, but you can sign up for email notifications when the data changes at
+        http://www.treas.gov/offices/enforcement/ofac/sdn/index.shtml.

data/README.rdoc

@@ -0,0 +1,123 @@
+= ofac
+
+* http://kevintyll.github.com/ofac
+* http://www.drexel-labs.com
+
+* http://www.treas.gov/offices/enforcement/ofac/sdn/index.shtml
+
+== DESCRIPTION:
+
+ofac is a ruby gem that tries to find a match of a person's name and address against the
+Office of Foreign Assets Control's Specially Designated Nationals list...the so called
+terrorist watch list.
+
+This gem, like the ssn_validator gem, started as a need for the company I work for, Clarity Services Inc.
+We decided once again to create a gem out of it and share it with the community.  Much
+thanks goes to the management at Clarity Services Inc. for allowing this code to be open sourced.  Thanks
+also to Larry Berland at Clarity Services Inc.  The matching logic in the ofac_match.rb file was derived from
+his work.
+
+== FEATURES:
+
+Creates a score, 1 - 100, based on how well the name, address and city match the data on the SDN list.  Since
+we have to match on strings, the likely hood of an exact match are virtually nil.  So we've created an
+algorithm that creates a score.  The better the match, the higher the score.  A score of 100 would be
+a perfect match.
+
+The score is calculated by adding up the weightings of each part that is matched. So
+if only name is matched, then the max score is the weight for <tt>:name</tt> which is 60
+
+It's possible to get partial matches, which will add partial weight to the score.  If there
+is not a match on the element as it is passed in, then each word element gets broken down
+and matches are tried on each partial element.  The weighting is distrubuted equally for
+each partial that is matched.
+
+If exact matches are not made, then a sounds like match is attempted.  Any match made by sounds like
+is given 75% of it's weight to the score.
+Example:
+
+If you are trying to match the name Kevin Tyll and there is a record for Smith, Kevin in the database, then
+we will try to match both Kevin and Tyll separately, with each element Smith and Kevin.  Since only Kevin
+will find a match, and there were 2 elements in the searched name, the score will be added by half the weighting
+for <tt>:name</tt>.  So since the weight for <tt>:name</tt> is 60, then we will add 30 to the score.
+
+If you are trying to match the name Kevin Gregory Tyll and there is a record for Tyll, Kevin in the database, then
+we will try to match Kevin and Gregory and Tyll separately, with each element Tyll and Kevin.  Since both Kevin
+and Tyll will find a match, and there were 3 elements in the searched name, the score will be added by 2/3 the weighting
+for <tt>:name</tt>.  So since the weight for <tt>:name</tt> is 60, then we will add 40 to the score.
+
+If you are trying to match the name Kevin Tyll and there is a record for Kevin Gregory Tyll in the database, then
+we will try to match Kevin and Tyll separately, with each element Tyll and Kevin and Gregory.  Since both Kevin
+and Tyll will find a match, and there were 2 elements in the searched name, the score will be added by 2/2 the weighting
+for <tt>:name</tt>.  So since the weight for <tt>:name</tt> is 60, then we will add 60 to the score.
+
+If you are trying to match the name Kevin Tyll, and there is a record for Teel, Kevin in the database, then an exact match
+will be found for Kevin, and a sounds like match will be made for Tyll.  Since there were 2 elements in the searched name,
+and the weight for <tt>:name</tt> is 60, then each element is worth 30.  Since Kevin was an exact match, it will add 30, and
+since Tyll was a sounds like match, it will add 30 * .75.  So the <tt>:name</tt> portion of the search will be worth 53.
+
+If data is in the database for city and or address, and you pass data in for these elements, the score will be reduced by 10%
+of the weight if there is no match or sounds like match.  So if you get a match on name, you've already got a score of 60.  So
+if you don't pass in an address or city, or if you do, but there is no city or address info in the database, then your final score
+will be 60.  But if you do pass in a city, say Tampa, and the city in the Database is New York, then we will deduct 10% of the
+weight (30 * .1) = 3 from the score since 30 is the weight for <tt>:city</tt>.  So the final score will be 57.
+
+If were searching for New York, and the database had New Deli, then there would be a match on New, but not on Deli.
+Since there were 2 elements in the searched city, each hit is worth 15.  So the match on New would add 15, but the non-match
+on York would subtract (15 * .1) = 1.5 from the score.  So the score would be (60 + 15 - 1.5) = 74, due to rounding.
+
+Only <tt>:city</tt> and <tt>:address</tt> subtract from the score, No match on name simply returns 0.
+
+Matches for name are made for both the name and any aliases in the OFAC database.
+
+Matches for <tt>:city</tt> and <tt>:address</tt> will only be added to the score if there is first a match on <tt>:name</tt>.
+
+We consider a score of 60 to be reasonable as a hit.
+
+== SYNOPSIS:
+Accepts a hash with the identity's demographic information
+
+   Ofac.new({:name => 'Oscar Hernandez', :city => 'Clearwater', :address => '123 somewhere ln'})
+
+<tt>:name</tt> is required to get a score.  If <tt>:name</tt> is missing, an error will not be thrown, but a score of 0 will be returned.
+
+The more information provided, the higher the score could be.  A score of 100 would mean all fields
+were passed in, and all fields were 100% matches.  If only the name is passed in without an address,
+it will be impossible to get a score of 100, even if the name matches perfectly.
+
+Acceptable hash keys and their weighting in score calculation:
+
+* <tt>:name</tt> (weighting = 60%) (required) This can be a person, business, or marine vessel
+* <tt>:address</tt> (weighting = 10%)
+* <tt>:city</tt> (weighting = 30%)
+
+* Instantiate the object with the identity's name, street address, and city.
+    ofac = Ofac.new(:name => 'Kevin Tyll', :city => 'Clearwater', :address => '123 Somewhere Ln.')
+
+* Then get the score
+    ofac.score => return the score 1 - 100
+
+* You can also get the list of all the partial matches with the score of each record.
+    ofac.possible_hits => returns an array of hashes.
+
+== REQUIREMENTS:
+
+* Rails 2.0.0 or greater
+
+== INSTALL:
+
+* To install the gem:
+    sudo gem install kevintyll-ofac
+* To create the necessary db migration, from the command line, run:
+    script/generate ofac_migration
+* Require the gem in your environment.rb file in the Rails::Initializer block:
+    config.gem 'kevintyll-ofac', :lib => 'ofac'
+* To load your table with the current OFAC data, from the command line, run:
+    rake ofac:update_data
+
+    * The OFAC data is not updated with any regularity, but you can sign up for email notifications when the data changes at
+        http://www.treas.gov/offices/enforcement/ofac/sdn/index.shtml.
+
+== Copyright
+
+Copyright (c) 2009 Kevin Tyll. See LICENSE for details.

data/Rakefile

@@ -0,0 +1,60 @@
+require 'rubygems'
+require 'rake'
+
+begin
+  require 'jeweler'
+  Jeweler::Tasks.new do |gem|
+    gem.name = "ofac"
+    gem.summary = %Q{Attempts to find a hit on the Office of Foreign Assets Control's Specially Designated Nationals list.}
+    gem.description = %Q{Attempts to find a hit on the Office of Foreign Assets Control's Specially Designated Nationals list.}
+    gem.email = "kevintyll@gmail.com"
+    gem.homepage = "http://github.com/kevintyll/ofac"
+    gem.authors = ["Kevin Tyll"]
+    gem.post_install_message = File.readlines("PostInstall.txt").join("")
+    # gem is a Gem::Specification... see http://www.rubygems.org/read/chapter/20 for additional settings
+  end
+
+  Jeweler::GemcutterTasks.new
+
+rescue LoadError
+  puts "Jeweler not available. Install it with: sudo gem install technicalpickles-jeweler -s http://gems.github.com"
+end
+
+require 'rake/testtask'
+Rake::TestTask.new(:test) do |test|
+  test.libs << 'lib' << 'test'
+  test.pattern = 'test/**/*_test.rb'
+  test.verbose = true
+end
+
+begin
+  require 'rcov/rcovtask'
+  Rcov::RcovTask.new do |test|
+    test.libs << 'test'
+    test.pattern = 'test/**/*_test.rb'
+    test.verbose = true
+  end
+rescue LoadError
+  task :rcov do
+    abort "RCov is not available. In order to run rcov, you must: sudo gem install spicycode-rcov"
+  end
+end
+
+
+task :default => :test
+
+require 'rake/rdoctask'
+Rake::RDocTask.new do |rdoc|
+  if File.exist?('VERSION.yml')
+    config = YAML.load(File.read('VERSION.yml'))
+    version = "#{config[:major]}.#{config[:minor]}.#{config[:patch]}"
+  else
+    version = ""
+  end
+
+  rdoc.rdoc_dir = 'rdoc'
+  rdoc.title = "ofac #{version}"
+  rdoc.rdoc_files.include('README*')
+  rdoc.rdoc_files.include('lib/**/*.rb')
+end
+

data/VERSION.yml

@@ -0,0 +1,4 @@
+--- 
+:minor: 1
+:patch: 11
+:major: 1

data/generators/ofac_migration/ofac_migration_generator.rb

@@ -0,0 +1,12 @@
+class OfacMigrationGenerator < Rails::Generator::Base
+  def manifest 
+    record do |m|
+      #m.directory File.join('db')
+      m.migration_template 'migration.rb', 'db/migrate' 
+    end 
+  end
+  
+  def file_name
+    "create_ofac_sdn_table"
+  end
+end

data/generators/ofac_migration/templates/migration.rb

@@ -0,0 +1,31 @@
+class CreateOfacSdnTable < ActiveRecord::Migration
+  
+  def self.up
+    create_table :ofac_sdns do |t|
+      t.text      :name
+      t.string    :sdn_type
+      t.string    :program
+      t.string    :title
+      t.string    :vessel_call_sign
+      t.string    :vessel_type
+      t.string    :vessel_tonnage
+      t.string    :gross_registered_tonnage
+      t.string    :vessel_flag
+      t.string    :vessel_owner
+      t.text      :remarks
+      t.text      :address
+      t.string    :city
+      t.string    :country
+      t.string    :address_remarks
+      t.string    :alternate_identity_type
+      t.text      :alternate_identity_name
+      t.string    :alternate_identity_remarks
+      t.timestamps
+    end
+    add_index :ofac_sdns, :sdn_type
+  end
+  
+  def self.down
+    drop_table :ofac_sdns
+  end
+end

data/lib/ofac.rb

@@ -0,0 +1,9 @@
+require 'rake'
+require 'ofac/ruby_string_extensions'
+require 'ofac/ofac_match'
+require 'ofac/models/ofac_sdn'
+require 'ofac/models/ofac_sdn_loader'
+require 'ofac/models/ofac'
+
+# Load rake file
+import "#{File.dirname(__FILE__)}/tasks/ofac.rake"

data/lib/ofac/models/ofac.rb

@@ -0,0 +1,177 @@
+class Ofac
+
+  
+  # Accepts a hash with the identity's demographic information
+  #
+  #   Ofac.new({:name => 'Oscar Hernandez', :city => 'Clearwater', :address => '123 somewhere ln'})
+  #
+  # <tt>:name</tt> is required to get a score.  If <tt>:name</tt> is missing, an error will not be thrown, but a score of 0 will be returned.
+  #
+  # You can pass a string in for the full name:
+  #   Ofac.new(:name => 'Victor De La Garza')
+  #
+  # Or you can specify the last and first names
+  #   Ofac.new(:name => {:first_name => 'Victor', :last_name => 'De La Garza'})
+  #
+  # The first method will build a larger list of names for ruby to parse through and more likely to find similar names.
+  # The second method is quicker.
+  #
+  # The more information provided, the higher the score could be.  A score of 100 would mean all fields
+  # were passed in, and all fields were 100% matches.  If only the name is passed in without an address,
+  # it will be impossible to get a score of 100, even if the name matches perfectly.
+  #
+  # Acceptable hash keys and their weighting in score calculation:
+  #
+  # * <tt>:name</tt> (weighting = 60%) (required) This can be a person, business, or marine vessel
+  # * <tt>:address</tt> (weighting = 10%)
+  # * <tt>:city</tt> (weighting = 30%)
+  def initialize(identity)
+    @identity = identity
+  end
+
+  # Creates a score, 1 - 100, based on how well the name and address match the data on the
+  # SDN (Specially Designated Nationals) list.
+  #
+  # The score is calculated by adding up the weightings of each part that is matched. So
+  # if only name is matched, then the max score is the weight for <tt>:name</tt> which is 60
+  #
+  # It's possible to get partial matches, which will add partial weight to the score.  If there
+  # is not a match on the element as it is passed in, then each word element gets broken down
+  # and matches are tried on each partial element.  The weighting is distrubuted equally for
+  # each partial that is matched.
+  #
+  # If exact matches are not made, then a sounds like match is attempted.  Any match made by sounds like
+  # is given 75% of it's weight to the score.
+  #
+  # Example:
+  #
+  # If you are trying to match the name Kevin Tyll and there is a record for Smith, Kevin in the database, then
+  # we will try to match both Kevin and Tyll separately, with each element Smith and Kevin.  Since only Kevin
+  # will find a match, and there were 2 elements in the searched name, the score will be added by half the weighting
+  # for <tt>:name</tt>.  So since the weight for <tt>:name</tt> is 60, then we will add 30 to the score.
+  #
+  # If you are trying to match the name Kevin Gregory Tyll and there is a record for Tyll, Kevin in the database, then
+  # we will try to match Kevin and Gregory and Tyll separately, with each element Tyll and Kevin.  Since both Kevin
+  # and Tyll will find a match, and there were 3 elements in the searched name, the score will be added by 2/3 the weighting
+  # for <tt>:name</tt>.  So since the weight for <tt>:name</tt> is 60, then we will add 40 to the score.
+  #
+  # If you are trying to match the name Kevin Tyll and there is a record for Kevin Gregory Tyll in the database, then
+  # we will try to match Kevin and Tyll separately, with each element Tyll and Kevin and Gregory.  Since both Kevin
+  # and Tyll will find a match, and there were 2 elements in the searched name, the score will be added by 2/2 the weighting
+  # for <tt>:name</tt>.  So since the weight for <tt>:name</tt> is 60, then we will add 60 to the score.
+  #
+  # If you are trying to match the name Kevin Tyll, and there is a record for Teel, Kevin in the database, then an exact match
+  # will be found for Kevin, and a sounds like match will be made for Tyll.  Since there were 2 elements in the searched name,
+  # and the weight for <tt>:name</tt> is 60, then each element is worth 30.  Since Kevin was an exact match, it will add 30, and
+  # since Tyll was a sounds like match, it will add 30 * .75.  So the <tt>:name</tt> portion of the search will be worth 53.
+  #
+  # If data is in the database for city and or address, and you pass data in for these elements, the score will be reduced by 10%
+  # of the weight if there is no match or sounds like match.  So if you get a match on name, you've already got a score of 60.  So
+  # if you don't pass in an address or city, or if you do, but there is no city or address info in the database, then your final score
+  # will be 60.  But if you do pass in a city, say Tampa, and the city in the Database is New York, then we will deduct 10% of the
+  # weight (30 * .1) = 3 from the score since 30 is the weight for <tt>:city</tt>.  So the final score will be 57.
+  #
+  # If were searching for New York, and the database had New Deli, then there would be a match on New, but not on Deli.
+  # Since there were 2 elements in the searched city, each hit is worth 15.  So the match on New would add 15, but the non-match
+  # on York would subtract (15 * .1) = 1.5 from the score.  So the score would be (60 + 15 - 1.5) = 74, due to rounding.
+  #
+  # Only <tt>:city</tt> and <tt>:address</tt> subtract from the score, No match on name simply returns 0.
+  #
+  # Matches for name are made for both the name and any aliases in the OFAC database.
+  #
+  # Matches for <tt>:city</tt> and <tt>:address</tt> will only be added to the score if there is first a match on <tt>:name</tt>.
+  #
+  # We consider a score of 60 to be reasonable as a hit.
+  def score
+    @score || calculate_score
+  end
+
+  def db_hit?
+    unless @identity[:name].to_s.blank?
+
+      #first get a list from the database of possible matches by name
+      #this query is pretty liberal, we just want to get a list of possible
+      #matches from the database that we can run through our ruby matching algorithm
+      possible_sdns = []
+      name_array = process_name
+
+      name_array.delete_if{|n| n.strip.size < 2}
+      unless name_array.empty?
+        sql_name_partial = name_array.collect {|partial_name| ["name like ?", "%#{partial_name}%"]}
+        sql_alt_name_partial = name_array.collect {|partial_name| ["alternate_identity_name like ?", "%#{partial_name}%"]}
+        
+        name_conditions = sql_name_partial.transpose
+        name_values = name_conditions.second
+        name_conditions = [name_conditions.first.join(' and ')]
+        alt_name_conditions = sql_alt_name_partial.transpose
+        alt_name_values = alt_name_conditions.second
+        alt_name_conditions = [alt_name_conditions.first.join(' and ')]
+        conditions = ["(#{name_conditions}) or (#{alt_name_conditions})"] + name_values + alt_name_values
+
+        possible_sdns = OfacSdn.find_all_by_sdn_type('individual',:select => 'name, alternate_identity_name, address, city', :conditions => conditions)
+        
+      end
+    end
+    !possible_sdns.empty?
+  end
+
+  # Returns an array of hashes of records in the OFAC data that found partial matches with that record's score.
+  # 
+  #     Ofac.new({:name => 'Oscar Hernandez', :city => 'Clearwater', :address => '123 somewhere ln'}).possible_hits
+  #returns
+  #     [{:address=>"123 Somewhere Ln", :score=>100, :name=>"HERNANDEZ, Oscar|GUAMATUR, S.A.", :city=>"Clearwater"}, {:address=>"123 Somewhere Ln", :score=>100, :name=>"HERNANDEZ, Oscar|Alternate Name", :city=>"Clearwater"}]
+  #
+  def possible_hits
+    @possible_hits || retrieve_possible_hits
+  end
+
+  private
+
+  def retrieve_possible_hits
+    score
+    @possible_hits
+  end
+
+  def calculate_score
+    unless @identity[:name].to_s.blank?
+      
+      #first get a list from the database of possible matches by name
+      #this query is pretty liberal, we just want to get a list of possible
+      #matches from the database that we can run through our ruby matching algorithm
+
+      name_array = process_name
+
+      name_array.delete_if{|n| n.strip.size < 2}
+      unless name_array.empty?
+        sql_name_partial = name_array.collect {|partial_name| ["name like ?", "%#{partial_name}%"]}
+        sql_alt_name_partial = name_array.collect {|partial_name| ["alternate_identity_name like ?", "%#{partial_name}%"]}
+        conditions = sql_name_partial + sql_alt_name_partial
+        conditions = conditions.transpose
+        conditions = [conditions.first.join(' or ')] + conditions.second
+
+        possible_sdns = OfacSdn.find_all_by_sdn_type('individual',:select => 'name, alternate_identity_name, address, city', :conditions => conditions)
+        possible_sdns = possible_sdns.collect {|sdn|{:name => "#{sdn['name']}|#{sdn['alternate_identity_name']}", :city => sdn['city'], :address => sdn['address']}}
+     
+        match = OfacMatch.new({:name => {:weight => 60, :token => "#{name_array.join(', ')}"},
+            :address => {:weight => 10, :token => @identity[:address]},
+            :city => {:weight => 30, :token => @identity[:city]}})
+
+        score = match.score(possible_sdns)
+        @possible_hits = match.possible_hits
+      end
+    end
+    @score = score || 0
+    return @score
+  end
+
+  def process_name
+    #you can pass in a full name, or specify the first and last name
+    if @identity[:name].kind_of?(Hash)
+      name_array = [@identity[:name][:last_name],@identity[:name][:first_name]].compact
+    else
+      partial_name = @identity[:name].gsub(/\W/,'|')
+      name_array = partial_name.split('|').reverse
+    end
+  end
+
+end