kevintyll-ofac 1.0.0

data/History.txt

@@ -0,0 +1,9 @@
+== 0.1.0 2009-05-7
+
+* 1 major enhancement:
+  * Table creation and data load task complete
+
+== 1.0.0 2009-05-11
+
+* 1 major enhancement:
+  * Initail release

data/LICENSE

@@ -0,0 +1,20 @@
+Copyright (c) 2009 Kevin Tyll
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/PostInstall.txt

@@ -0,0 +1,11 @@
+For more information on ofac, see http://kevintyll.github.com/ofac/
+
+* To create the necessary db migration, from the command line, run:
+    script/generate ofac_migration
+* Require the gem in your environment.rb file in the Rails::Initializer block:
+    config.gem 'kevintyll-ofac', :lib => 'ofac'
+* To load your table with the current OFAC data, from the command line, run:
+    rake ofac:update_data
+
+    * The OFAC data is not updated with any regularity, but you can sign up for email notifications when the data changes at
+        http://www.treas.gov/offices/enforcement/ofac/sdn/index.shtml.

data/README.rdoc

@@ -0,0 +1,109 @@
+= ofac
+
+* http://kevintyll.github.com/ofac
+* http://www.drexel-labs.com
+
+* http://www.treas.gov/offices/enforcement/ofac/sdn/index.shtml
+
+== DESCRIPTION:
+
+ofac is a ruby gem that tries to find a match of a person's name and address against the
+Office of Foreign Assets Control's Specially Designated Nationals list...the so called
+terrorist watch list.
+
+This gem, like the ssn_validator gem, started as a need for the company I work for, Clarity Services Inc.
+We decided once again to create a gem out of it and share it with the community.  Much
+thanks goes to the management at Clarity Services Inc. for allowing this code to be open sourced.  Thanks
+also to Larry Berland at Clarity Services Inc.  The matching logic in the ofac_match.rb file was derived from
+his work.
+
+== FEATURES:
+
+Creates a score, 1 - 100, based on how well the name, address and city match the data on the SDN list.  Since
+we have to match on strings, the likely hood of an exact match are virtually nil.  So we've created an
+algorithm that creates a score.  The better the match, the higher the score.  A score of 100 would be
+a perfect match.
+
+The score is calculated by adding up the weightings of each part that is matched. So
+if only name is matched, then the max score is the weight for <tt>:name</tt> which is 60
+
+It's possible to get partial matches, which will add partial weight to the score.  If there
+is not a match on the element as it is passed in, then each word element gets broken down
+and matches are tried on each partial element.  The weighting is distrubuted equally for
+each partial that is matched.
+
+If exact matches are not made, then a sounds like match is attempted.  Any match made by sounds like
+is given 75% of it's weight to the score.
+Example:
+
+If you are trying to match the name Kevin Tyll and there is a record for Smith, Kevin in the database, then
+we will try to match both Kevin and Tyll separately, with each element Smith and Kevin.  Since only Kevin
+will find a match, and there were 2 elements in the searched name, the score will be added by half the weighting
+for <tt>:name</tt>.  So since the weight for <tt>:name</tt> is 60, then we will add 30 to the score.
+
+If you are trying to match the name Kevin Gregory Tyll and there is a record for Tyll, Kevin in the database, then
+we will try to match Kevin and Gregory and Tyll separately, with each element Tyll and Kevin.  Since both Kevin
+and Tyll will find a match, and there were 3 elements in the searched name, the score will be added by 2/3 the weighting
+for <tt>:name</tt>.  So since the weight for <tt>:name</tt> is 60, then we will add 40 to the score.
+
+If you are trying to match the name Kevin Tyll and there is a record for Kevin Gregory Tyll in the database, then
+we will try to match Kevin and Tyll separately, with each element Tyll and Kevin and Gregory.  Since both Kevin
+and Tyll will find a match, and there were 2 elements in the searched name, the score will be added by 2/2 the weighting
+for <tt>:name</tt>.  So since the weight for <tt>:name</tt> is 60, then we will add 60 to the score.
+
+If you are trying to match the name Kevin Tyll, and there is a record for Teel, Kevin in the database, then an exact match
+will be found for Kevin, and a sounds like match will be made for Tyll.  Since there were 2 elements in hte searched name,
+and the weight for <tt>:name</tt> is 60, then each element is worth 30.  Since Kevin was an exact match, it will add 30, and
+since Tyll was a sounds like match, it will add 30 * .75.  So the <tt>:name</tt> portion of the search will be worth 53.
+
+Matches for name are made for both the name and any aliases in the OFAC database.
+
+Matches for <tt>:city</tt> and <tt>:address</tt> will only be added to the score if there is first a match on <tt>:name</tt>.
+
+== SYNOPSIS:
+Accepts a hash with the identity's demographic information
+
+   Ofac.new({:name => 'Oscar Hernandez', :city => 'Clearwater', :address => '123 somewhere ln'})
+
+<tt>:name</tt> is required to get a score.  If <tt>:name</tt> is missing, an error will not be thrown, but a score of 0 will be returned.
+
+The more information provided, the higher the score could be.  A score of 100 would mean all fields
+were passed in, and all fields were 100% matches.  If only the name is passed in without an address,
+it will be impossible to get a score of 100, even if the name matches perfectly.
+
+Acceptable hash keys and their weighting in score calculation:
+
+* <tt>:name</tt> (weighting = 60%) (required) This can be a person, business, or marine vessel
+* <tt>:address</tt> (weighting = 10%)
+* <tt>:city</tt> (weighting = 30%)
+
+* Instantiate the object with the identity's name, street address, and city.
+    ofac = Ofac.new(:name => 'Kevin Tyll', :city => 'Clearwater', :address => '123 Somewhere Ln.')
+
+* Then get the score
+    ofac.score => return the score 1 - 100
+
+* You can also get the list of all the partial matches with the score of each record.
+    ofac.possible_hits => returns an array of hashes.
+
+== REQUIREMENTS:
+
+* Rails 2.0.0 or greater
+
+== INSTALL:
+
+* To install the gem:
+    sudo gem install kevintyll-ofac
+* To create the necessary db migration, from the command line, run:
+    script/generate ofac_migration
+* Require the gem in your environment.rb file in the Rails::Initializer block:
+    config.gem 'kevintyll-ofac', :lib => 'ofac'
+* To load your table with the current OFAC data, from the command line, run:
+    rake ofac:update_data
+
+    * The OFAC data is not updated with any regularity, but you can sign up for email notifications when the data changes at
+        http://www.treas.gov/offices/enforcement/ofac/sdn/index.shtml.
+
+== Copyright
+
+Copyright (c) 2009 Kevin Tyll. See LICENSE for details.

data/Rakefile

@@ -0,0 +1,57 @@
+require 'rubygems'
+require 'rake'
+
+begin
+  require 'jeweler'
+  Jeweler::Tasks.new do |gem|
+    gem.name = "ofac"
+    gem.summary = %Q{Attempts to find a hit on the Office of Foreign Assets Control's Specially Designated Nationals list.}
+    gem.description = %Q{Attempts to find a hit on the Office of Foreign Assets Control's Specially Designated Nationals list.}
+    gem.email = "kevintyll@gmail.com"
+    gem.homepage = "http://github.com/kevintyll/ofac"
+    gem.authors = ["Kevin Tyll"]
+    gem.post_install_message = File.readlines("PostInstall.txt").join("")
+    # gem is a Gem::Specification... see http://www.rubygems.org/read/chapter/20 for additional settings
+  end
+rescue LoadError
+  puts "Jeweler not available. Install it with: sudo gem install technicalpickles-jeweler -s http://gems.github.com"
+end
+
+require 'rake/testtask'
+Rake::TestTask.new(:test) do |test|
+  test.libs << 'lib' << 'test'
+  test.pattern = 'test/**/*_test.rb'
+  test.verbose = true
+end
+
+begin
+  require 'rcov/rcovtask'
+  Rcov::RcovTask.new do |test|
+    test.libs << 'test'
+    test.pattern = 'test/**/*_test.rb'
+    test.verbose = true
+  end
+rescue LoadError
+  task :rcov do
+    abort "RCov is not available. In order to run rcov, you must: sudo gem install spicycode-rcov"
+  end
+end
+
+
+task :default => :test
+
+require 'rake/rdoctask'
+Rake::RDocTask.new do |rdoc|
+  if File.exist?('VERSION.yml')
+    config = YAML.load(File.read('VERSION.yml'))
+    version = "#{config[:major]}.#{config[:minor]}.#{config[:patch]}"
+  else
+    version = ""
+  end
+
+  rdoc.rdoc_dir = 'rdoc'
+  rdoc.title = "ofac #{version}"
+  rdoc.rdoc_files.include('README*')
+  rdoc.rdoc_files.include('lib/**/*.rb')
+end
+

data/VERSION.yml

@@ -0,0 +1,4 @@
+--- 
+:minor: 0
+:patch: 0
+:major: 1

data/generators/ofac_migration/ofac_migration_generator.rb

@@ -0,0 +1,12 @@
+class OfacMigrationGenerator < Rails::Generator::Base
+  def manifest 
+    record do |m|
+      #m.directory File.join('db')
+      m.migration_template 'migration.rb', 'db/migrate' 
+    end 
+  end
+  
+  def file_name
+    "create_ofac_sdn_table"
+  end
+end

data/generators/ofac_migration/templates/migration.rb

@@ -0,0 +1,30 @@
+class CreateOfacSdnTable < ActiveRecord::Migration
+  
+  def self.up
+    create_table :ofac_sdns do |t|
+      t.text      :name
+      t.string    :sdn_type
+      t.string    :program
+      t.string    :title
+      t.string    :vessel_call_sign
+      t.string    :vessel_type
+      t.string    :vessel_tonnage
+      t.string    :gross_registered_tonnage
+      t.string    :vessel_flag
+      t.string    :vessel_owner
+      t.text      :remarks
+      t.text      :address
+      t.string    :city
+      t.string    :country
+      t.string    :address_remarks
+      t.string    :alternate_identity_type
+      t.text      :alternate_identity_name
+      t.string    :alternate_identity_remarks
+      t.timestamps
+    end
+  end
+  
+  def self.down
+    drop_table :ofac_sdns
+  end
+end

data/lib/ofac.rb

@@ -0,0 +1,9 @@
+require 'rake'
+require 'ofac/ruby_string_extensions'
+require 'ofac/ofac_match'
+require 'ofac/models/ofac_sdn'
+require 'ofac/models/ofac_sdn_loader'
+require 'ofac/models/ofac'
+
+# Load rake file
+import "#{File.dirname(__FILE__)}/tasks/ofac.rake"

data/lib/ofac/models/ofac.rb

@@ -0,0 +1,119 @@
+require 'activerecord'
+require 'active_record/connection_adapters/mysql_adapter'
+
+class Ofac
+
+  
+  # Accepts a hash with the identity's demographic information
+  #
+  #   Ofac.new({:name => 'Oscar Hernandez', :city => 'Clearwater', :address => '123 somewhere ln'})
+  #
+  # <tt>:name</tt> is required to get a score.  If <tt>:name</tt> is missing, an error will not be thrown, but a score of 0 will be returned.
+  #
+  # The more information provided, the higher the score could be.  A score of 100 would mean all fields
+  # were passed in, and all fields were 100% matches.  If only the name is passed in without an address,
+  # it will be impossible to get a score of 100, even if the name matches perfectly.
+  #
+  # Acceptable hash keys and their weighting in score calculation:
+  #
+  # * <tt>:name</tt> (weighting = 60%) (required) This can be a person, business, or marine vessel
+  # * <tt>:address</tt> (weighting = 10%)
+  # * <tt>:city</tt> (weighting = 30%)
+  def initialize(identity)
+    @identity = identity
+  end
+
+  # Creates a score, 1 - 100, based on how well the name and address match the data on the
+  # SDN (Specially Designated Nationals) list.
+  #
+  # The score is calculated by adding up the weightings of each part that is matched. So
+  # if only name is matched, then the max score is the weight for <tt>:name</tt> which is 60
+  #
+  # It's possible to get partial matches, which will add partial weight to the score.  If there
+  # is not a match on the element as it is passed in, then each word element gets broken down
+  # and matches are tried on each partial element.  The weighting is distrubuted equally for
+  # each partial that is matched.
+  #
+  # If exact matches are not made, then a sounds like match is attempted.  Any match made by sounds like
+  # is given 75% of it's weight to the score.
+  #
+  # Example:
+  #
+  # If you are trying to match the name Kevin Tyll and there is a record for Smith, Kevin in the database, then
+  # we will try to match both Kevin and Tyll separately, with each element Smith and Kevin.  Since only Kevin
+  # will find a match, and there were 2 elements in the searched name, the score will be added by half the weighting
+  # for <tt>:name</tt>.  So since the weight for <tt>:name</tt> is 60, then we will add 30 to the score.
+  #
+  # If you are trying to match the name Kevin Gregory Tyll and there is a record for Tyll, Kevin in the database, then
+  # we will try to match Kevin and Gregory and Tyll separately, with each element Tyll and Kevin.  Since both Kevin
+  # and Tyll will find a match, and there were 3 elements in the searched name, the score will be added by 2/3 the weighting
+  # for <tt>:name</tt>.  So since the weight for <tt>:name</tt> is 60, then we will add 40 to the score.
+  #
+  # If you are trying to match the name Kevin Tyll and there is a record for Kevin Gregory Tyll in the database, then
+  # we will try to match Kevin and Tyll separately, with each element Tyll and Kevin and Gregory.  Since both Kevin
+  # and Tyll will find a match, and there were 2 elements in the searched name, the score will be added by 2/2 the weighting
+  # for <tt>:name</tt>.  So since the weight for <tt>:name</tt> is 60, then we will add 60 to the score.
+  #
+  # If you are trying to match the name Kevin Tyll, and there is a record for Teel, Kevin in the database, then an exact match
+  # will be found for Kevin, and a sounds like match will be made for Tyll.  Since there were 2 elements in hte searched name,
+  # and the weight for <tt>:name</tt> is 60, then each element is worth 30.  Since Kevin was an exact match, it will add 30, and
+  # since Tyll was a sounds like match, it will add 30 * .75.  So the <tt>:name</tt> portion of the search will be worth 53.
+  #
+  # Matches for name are made for both the name and any aliases in the OFAC database.
+  #
+  # Matches for <tt>:city</tt> and <tt>:address</tt> will only be added to the score if there is first a match on <tt>:name</tt>.
+  def score
+    @score || calculate_score
+  end
+
+  # Returns an array of hashes of records in the OFAC data that found partial matches with that record's score.
+  # 
+  #     Ofac.new({:name => 'Oscar Hernandez', :city => 'Clearwater', :address => '123 somewhere ln'}).possible_hits
+  #returns
+  #     [{:address=>"123 Somewhere Ln", :score=>100, :name=>"HERNANDEZ, Oscar|GUAMATUR, S.A.", :city=>"Clearwater"}, {:address=>"123 Somewhere Ln", :score=>100, :name=>"HERNANDEZ, Oscar|Alternate Name", :city=>"Clearwater"}]
+  #
+  def possible_hits
+    @possible_hits || retrieve_possible_hits
+  end
+
+  private
+
+  def retrieve_possible_hits
+    score
+    @possible_hits
+  end
+
+  def calculate_score
+    unless @identity[:name].to_s == ''
+      if OfacSdn.connection.kind_of?(ActiveRecord::ConnectionAdapters::MysqlAdapter)
+        #first get a list from the database of possible matches by name
+        #this query is pretty liberal, we just want to get a list of possible
+        #matches from the database that we can run through our ruby matching algorithm
+        partial_name = @identity[:name].gsub!(/\W/,'|')
+        name_array = partial_name.split('|')
+        name_array.delete('')
+        sql_name_partial = name_array.collect {|partial_name| "INSTR(SUBSTR(SOUNDEX(concat('O',name)), 2), REPLACE(SUBSTR(SOUNDEX('O#{partial_name}'), 2), '0', '')) > 0"}.join(' and ')
+        sql_alt_name_partial = name_array.collect {|partial_name| "INSTR(SUBSTR(SOUNDEX(concat('O',alternate_identity_name)), 2), REPLACE(SUBSTR(SOUNDEX('O#{partial_name}'), 2), '0', '')) > 0"}.join(' and ')
+        ##this sql for getting "accurate sounds like" functionality comes from:
+        #http://jgeewax.wordpress.com/2006/07/21/efficient-sounds-like-searches-in-mysql/
+        possible_sdns = OfacSdn.connection.select_all("select concat(name,'|', alternate_identity_name) name, address, city
+                                from ofac_sdns
+                                where name is not null
+                                and (((#{sql_name_partial}))
+                                or ((#{sql_alt_name_partial})))")
+      else
+        possible_sdns = OfacSdn.find(:all, :select => 'name, alternate_identity_name, address, city').collect{|sdn| {:name => "#{sdn.name}|#{sdn.alternate_identity_name}", :address => sdn.address, :city => sdn.city}}
+      end
+      
+      match = OfacMatch.new({:name => {:weight => 60, :token => "#{@identity[:name]}"},
+                                  :address => {:weight => 10, :token => @identity[:address]},
+                                  :city => {:weight => 30, :token => @identity[:city]}})
+
+      score = match.score(possible_sdns)
+      @possible_hits = match.possible_hits
+    end
+    @score = score || 0
+    return @score
+  end
+
+end

data/lib/ofac/models/ofac_sdn.rb

@@ -0,0 +1,5 @@
+require 'activerecord'
+
+class OfacSdn < ActiveRecord::Base
+
+end

data/lib/ofac/models/ofac_sdn_loader.rb

@@ -0,0 +1,305 @@
+require 'net/http'
+require 'activerecord'
+require 'active_record/connection_adapters/mysql_adapter'
+
+class OfacSdnLoader
+
+
+  #Loads the most recent file from http://www.treas.gov/offices/enforcement/ofac/sdn/delimit/index.shtml
+  def self.load_current_sdn_file
+    puts "Reloading OFAC sdn data"
+    puts "Downloading OFAC data from http://www.treas.gov/offices/enforcement/ofac/sdn"
+    #get the 3 data files
+    sdn = Tempfile.new('sdn')
+    sdn.write(Net::HTTP.get(URI.parse('http://www.treas.gov/offices/enforcement/ofac/sdn/delimit/sdn.pip')))
+    sdn.rewind
+    address = Tempfile.new('sdn')
+    address.write(Net::HTTP.get(URI.parse('http://www.treas.gov/offices/enforcement/ofac/sdn/delimit/add.pip')))
+    address.rewind
+    alt = Tempfile.new('sdn')
+    alt.write(Net::HTTP.get(URI.parse('http://www.treas.gov/offices/enforcement/ofac/sdn/delimit/alt.pip')))
+    alt.rewind
+    
+    if OfacSdn.connection.kind_of?(ActiveRecord::ConnectionAdapters::MysqlAdapter)
+      puts "Converting file to csv format for Mysql import.  This could take several minutes."
+
+      csv_file = convert_to_flattened_csv(sdn, address, alt)
+
+      bulk_mysql_update(csv_file)
+    else
+      active_record_file_load(sdn, address, alt)
+    end
+
+    sdn.close
+    @address.close
+    @alt.close
+  end
+
+
+  private
+
+  #convert the file's null value to an empty string
+  #and removes " chars.
+  def self.clean_file_string(line)
+    line.gsub!(/-0-(\s)?/,'')
+    line.gsub!(/\n/,'')
+    line.gsub(/\"/,'')
+  end
+
+  #split the line into an array
+  def self.convert_line_to_array(line)
+    clean_file_string(line).split('|') unless line.nil?
+  end
+
+  #return an 2 arrays of the records matching the sdn primary key
+  #1 array of address records and one array of alt records
+  def self.foreign_key_records(sdn_id)
+    address_records = []
+    alt_records = []
+
+    #the first element in each array is the primary and foreign keys
+    #we are denormalizing the data
+    if @current_address_hash && @current_address_hash[:id] == sdn_id
+      address_records << @current_address_hash
+      loop do
+        @current_address_hash = address_text_to_hash(@address.gets)
+        if @current_address_hash && @current_address_hash[:id] == sdn_id
+          address_records << @current_address_hash
+        else
+          break
+        end
+      end
+    end
+
+    if @current_alt_hash && @current_alt_hash[:id] == sdn_id
+      alt_records << @current_alt_hash
+      loop do
+        @current_alt_hash = alt_text_to_hash(@alt.gets)
+        if @current_alt_hash && @current_alt_hash[:id] == sdn_id
+          alt_records << @current_alt_hash
+        else
+          break
+        end
+      end
+    end
+    return address_records, alt_records
+  end
+
+  def self.sdn_text_to_hash(line)
+    unless line.nil?
+      value_array = convert_line_to_array(line)
+      {:id => value_array[0],
+        :name => value_array[1],
+        :sdn_type => value_array[2],
+        :program => value_array[3],
+        :title => value_array[4],
+        :vessel_call_sign => value_array[5],
+        :vessel_type => value_array[6],
+        :vessel_tonnage => value_array[7],
+        :gross_registered_tonnage => value_array[8],
+        :vessel_flag => value_array[9],
+        :vessel_owner => value_array[10],
+        :remarks => value_array[11]
+      }
+    end
+  end
+
+  def self.address_text_to_hash(line)
+    unless line.nil?
+      value_array = convert_line_to_array(line)
+      {:id => value_array[0],
+       :address => value_array[2],
+       :city => value_array[3],
+       :country => value_array[4],
+       :address_remarks => value_array[5]
+      }
+    end
+  end
+
+  def self.alt_text_to_hash(line)
+    unless line.nil?
+      value_array = convert_line_to_array(line)
+      {:id => value_array[0],
+       :alternate_identity_type => value_array[2],
+       :alternate_identity_name => value_array[3],
+       :alternate_identity_remarks => value_array[4]
+      }
+    end
+  end
+
+  def self.convert_hash_to_mysql_import_string(record_hash)
+    # empty field for id to be generated by mysql.
+    new_line = "``|" +
+      #    :name
+    "`#{record_hash[:name]}`|" +
+      #    :sdn_type
+    "`#{record_hash[:sdn_type]}`|" +
+      #    :program
+    "`#{record_hash[:program]}`|" +
+      #    :title
+    "`#{record_hash[:title]}`|" +
+      #    :vessel_call_sign
+    "`#{record_hash[:vessel_call_sign]}`|" +
+      #    :vessel_type
+    "`#{record_hash[:vessel_type]}`|" +
+      #    :vessel_tonnage
+    "`#{record_hash[:vessel_tonnage]}`|" +
+      #    :gross_registered_tonnage
+    "`#{record_hash[:gross_registered_tonnage]}`|" +
+      #    :vessel_flag
+    "`#{record_hash[:vessel_flag]}`|" +
+      #    :vessel_owner
+    "`#{record_hash[:vessel_owner]}`|" +
+      #    :remarks
+    "`#{record_hash[:remarks]}`|" +
+    #    :address
+    "`#{record_hash[:address]}`|" +
+    #    :city
+    "`#{record_hash[:city]}`|" +
+    #    :country
+    "`#{record_hash[:country]}`|" +
+    #    :address_remarks
+    "`#{record_hash[:address_remarks]}`|" +
+    #    :alternate_identity_type
+    "`#{record_hash[:alternate_identity_type]}`|" +
+    #    :alternate_identity_name
+    "`#{record_hash[:alternate_identity_name]}`|" +
+    #    :alternate_identity_remarks
+    "`#{record_hash[:alternate_identity_remarks]}`|" +
+    #:created_at
+    "`#{Time.now.to_s(:db)}`|" +
+      # updated_at
+    "`#{Time.now.to_s(:db)}`" + "\n"
+
+    new_line
+  end
+
+  def self.convert_to_flattened_csv(sdn_file, address_file, alt_file)
+    @address = address_file
+    @alt = alt_file
+
+    csv_file = Tempfile.new("ofac")    # create temp file for converted csv format.
+    #get the first line from the address and alt files
+    @current_address_hash = address_text_to_hash(@address.gets)
+    @current_alt_hash = alt_text_to_hash(@alt.gets)
+
+    start = Time.now
+    
+    sdn_file.each_with_index do |line, i|
+      
+      #initialize the address and alt atributes to empty strings
+      address_attributes = address_text_to_hash("|||||")
+      alt_attributes = alt_text_to_hash("||||")
+
+      sdn_attributes = sdn_text_to_hash(line)
+
+      #get the foreign key records for this sdn
+      address_records, alt_records = foreign_key_records(sdn_attributes[:id])
+
+      if address_records.empty?
+        #no matching address records, so initialized blank values will be used.
+        if alt_records.empty?
+          #no matching address records, so initialized blank values will be used.
+          csv_file.syswrite(convert_hash_to_mysql_import_string(sdn_attributes.merge(address_attributes).merge(alt_attributes)))
+        else
+          alt_records.each do |alt|
+            csv_file.syswrite(convert_hash_to_mysql_import_string(sdn_attributes.merge(address_attributes).merge(alt)))
+          end
+        end
+      else
+        address_records.each do |address|
+          if alt_records.empty?
+            #no matching address records, so initialized blank values will be used.
+            csv_file.syswrite(convert_hash_to_mysql_import_string(sdn_attributes.merge(address).merge(alt_attributes)))
+          else
+            alt_records.each do |alt|
+              csv_file.syswrite(convert_hash_to_mysql_import_string(sdn_attributes.merge(address).merge(alt)))
+            end
+          end
+        end
+      end
+      puts "#{i} records processed." if (i % 1000 == 0) && (i > 0)
+    end
+    puts "File conversion ran for #{(Time.now - start) / 60} minutes."
+    return csv_file
+  end
+
+  def self.active_record_file_load(sdn_file, address_file, alt_file)
+    @address = address_file
+    @alt = alt_file
+
+    #OFAC data is a complete list, so we have to dump and load
+    OfacSdn.delete_all
+
+    #get the first line from the address and alt files
+    @current_address_hash = address_text_to_hash(@address.gets)
+    @current_alt_hash = alt_text_to_hash(@alt.gets)
+    attributes = {}
+    sdn_file.each_with_index do |line, i|
+
+      #initialize the address and alt atributes to empty strings
+      address_attributes = address_text_to_hash("|||||")
+      alt_attributes = alt_text_to_hash("||||")
+
+      sdn_attributes = sdn_text_to_hash(line)
+     
+      #get the foreign key records for this sdn
+      address_records, alt_records = foreign_key_records(sdn_attributes[:id])
+
+      if address_records.empty?
+        #no matching address records, so initialized blank values will be used.
+        if alt_records.empty?
+          #no matching address records, so initialized blank values will be used.
+          attributes = sdn_attributes.merge(address_attributes).merge(alt_attributes)
+          attributes.delete(:id)
+          OfacSdn.create(attributes)
+        else
+          alt_records.each do |alt|
+            attributes = sdn_attributes.merge(address_attributes).merge(alt)
+            attributes.delete(:id)
+            OfacSdn.create(attributes)
+          end
+        end
+      else
+        address_records.each do |address|
+          if alt_records.empty?
+            #no matching address records, so initialized blank values will be used.
+            attributes = sdn_attributes.merge(address).merge(alt_attributes)
+            attributes.delete(:id)
+            OfacSdn.create(attributes)
+          else
+            alt_records.each do |alt|
+              attributes = sdn_attributes.merge(address).merge(alt)
+              attributes.delete(:id)
+              OfacSdn.create(attributes)
+            end
+          end
+        end
+      end
+      
+      puts "#{i} records processed." if (i % 5000 == 0) && (i > 0)
+    end
+  end
+
+  # For mysql, use:
+  # LOAD DATA LOCAL INFILE 'ssdm1.csv' INTO TABLE death_master_files FIELDS TERMINATED BY '|' ENCLOSED BY "`" LINES TERMINATED BY '\n';
+  # This is a much faster way of loading large amounts of data into mysql.  For information on the LOAD DATA command
+  # see http://dev.mysql.com/doc/refman/5.1/en/load-data.html
+  def self.bulk_mysql_update(csv_file)
+    puts "Deleting all records in ofac_sdn..."
+
+    #OFAC data is a complete list, so we have to dump and load
+    OfacSdn.delete_all
+
+    puts "Importing into Mysql..."
+    
+    mysql_command = <<-TEXT
+    LOAD DATA LOCAL INFILE '#{csv_file.path}' REPLACE INTO TABLE ofac_sdns FIELDS TERMINATED BY '|' ENCLOSED BY "`" LINES TERMINATED BY '\n';
+    TEXT
+
+    OfacSdn.connection.execute(mysql_command)
+    puts "Mysql import complete."
+
+  end
+
+end

data/lib/ofac/ofac_match.rb

@@ -0,0 +1,132 @@
+class OfacMatch
+
+  attr_reader :possible_hits
+
+  #Intialize a Match object with a record hash of fields you want to match on.
+  #Each key in the hash, also has a data hash value for the weight, token, and type.
+  #
+  #    match = Ofac::Match.new({:name => {:weight => 10, :token => 'Kevin Tyll'},
+  #        :city   => {:weight => 40, :token => 'Clearwater',     },
+  #        :address => {:weight => 40, :token => '1234 Park St.',    },
+  #        :zip    => {:weight => 10, :token => '33759', :type => :number}})
+  #
+  # data hash keys:
+  #   * <tt>data[:weight]</tt> - value to apply to the score if there is a match (Default is 100/number of key in the record hash)
+  #   * <tt>data[:token]</tt> - string to match
+  #   * <tt>data[:match]</tt> - set from records hash
+  #   * <tt>data[:score]</tt> - output field
+  #   * <tt>data[:type]</tt> - the type of match that should be performed (valid values are +:sound+ | +:number+) (Default is +:sound+)
+  def initialize(stats={})
+    @possible_hits = []
+    @stats = stats.dup
+    weight = 100
+    weight = 100 / @stats.length if @stats.length > 0
+    @stats.each_value do |data|
+      data[:weight] ||= weight
+      data[:match]  ||= ''
+      data[:type]   ||= :sound
+      data[:score]  ||= 0
+      data[:token]  = data[:token].to_s.upcase
+    end
+  end
+
+  # match_records is an array of hashes.
+  #
+  # The hash keys must match the record hash keys set when initialized.
+  #
+  # score will return the highest score of all the records that
+  # are sent in match_records.
+  def score(match_records)
+    score_results = Array.new
+    unless match_records.empty?
+      #place the match_records information
+      #into our @stats hash
+      match_records.each do |match|
+        match.each do |key, value|
+          @stats[key.to_sym][:match] = value.to_s.upcase
+        end
+        record_score = calculate_record
+        score_results.push(record_score)
+        @possible_hits << match.merge(:score => record_score) if record_score > 0
+      end
+      score = score_results.max #take max score
+    end
+    @possible_hits.uniq!
+    score ||= 0
+  end
+
+  private
+
+
+  # calculate the score for this record
+  # comparing the token to the match fields in the @stats hash
+  # and storing the score into the record
+  def calculate_record
+    score = 0
+    unless @stats.nil?
+      #need to make sure we check the name first, since city and address don't
+      #get added to the score unless there is a name match
+      [:name,:city,:address].each do |field|
+        data = @stats[field]
+        if (data[:token].blank?)
+          value = 0 #token is blank can't be sure of a match if nothing to match against
+        else
+          if (data[:match].blank?)
+            value = 0 #token has value match is blank
+          else
+            #token and match both have values
+            if (data[:type] == :number)
+              value = data[:token] == data[:match] ? 1 : 0
+            else
+              #first see if there is an exact match
+              value = data[:token] == data[:match] ? 1 : 0
+
+              unless value > 0
+                #do a sounds like with the data as given to see if we get a match
+                #if match on sounds_like, only give .75 of the weight.
+                value = data[:token].ofac_sounds_like(data[:match],false) ? 0.75 : 0
+              end
+                
+              #if no match, then break the data down and see if we can find matches on the
+              #individual words
+              unless value > 0
+                token_data = data[:token].gsub(/\W/,'|')
+                token_array = token_data.split('|')
+                token_array.delete('')
+
+                match_data = data[:match].gsub(/\W/,'|')
+                match_array = match_data.split('|')
+                match_array.delete('')
+
+                value = 0
+                partial_weight = 1/token_array.length.to_f
+                token_array.each do |partial_token|
+                  #first see if we get an exact match of the partial
+                  if match_array.include?(partial_token)
+                    value += partial_weight
+                  else
+                    #otherwise, see if the partial sounds like any part of the OFAC record
+                    match_array.each do |partial_match|
+                      if partial_match.ofac_sounds_like(partial_token,false)
+                        #give partial value for every part of token that is matched.
+                        value += partial_weight * 0.75
+                        break
+                      end
+                    end
+                  end
+                end
+              end
+            end
+          end
+        end
+        data[:score] = data[:weight] * value
+        score += data[:score]
+        break if field == :name && data[:score] == 0
+      end
+
+    end
+    score.round
+  end
+
+end
+

data/lib/ofac/ruby_string_extensions.rb

@@ -0,0 +1,22 @@
+class String
+  
+  Ofac_SoundexChars = 'BPFVCSKGJQXZDTLMNR'
+  Ofac_SoundexNums  = '111122222222334556'
+  Ofac_SoundexCharsEx = '^' + Ofac_SoundexChars
+  Ofac_SoundexCharsDel = '^A-Z'
+
+  # desc: http://en.wikipedia.org/wiki/Soundex
+  def ofac_soundex(census = true)
+    str = upcase.delete(Ofac_SoundexCharsDel).squeeze
+
+    str[0 .. 0] + str[1 .. -1].
+      delete(Ofac_SoundexCharsEx).
+      tr(Ofac_SoundexChars, Ofac_SoundexNums)[0 .. (census ? 2 : -1)].
+      ljust(3, '0') rescue ''
+  end
+
+  def ofac_sounds_like(other, census = true)
+    ofac_soundex(census) == other.ofac_soundex(census)
+  end
+
+end

data/lib/tasks/ofac.rake

@@ -0,0 +1,8 @@
+
+namespace :ofac do
+  desc "Loads the current file from http://www.treas.gov/offices/enforcement/ofac/sdn/delimit/index.shtml."
+  task :update_data => :environment do
+    OfacSdnLoader.load_current_sdn_file
+  end
+
+end

data/test/files/test_address_data_load.pip

@@ -0,0 +1,10 @@
+10|7|-0- |-0- |"Panama"|-0-
+15|12|-0- |-0- |"Panama"|-0-
+22|14|"123 Somewhere Ln"|"Clearwater"|"United States"|-0-
+39|27|-0- |"Managua"|"Nicaragua"|-0-
+39|29|"Bal Harbour Shopping Center, Via Italia"|"Panama City"|"Panama"|-0-
+41|41|"Avenida de Concha, Espina 8, E-28036"|"Madrid"|"Spain"|-0-
+41|102|-0- |-0- |-0- |-0-
+66|111|-0- |"Milan"|"Italy"|-0-
+66|117|-0- |-0- |"Panama"|-0-
+66|125|"1840 West 49th Street"|"Hialeah, FL"|"United States"|-0-

data/test/files/test_alt_data_load.pip

@@ -0,0 +1,10 @@
+15|14|"aka"|"VIAJES GUAMA TOURS"|-0-
+22|15|"aka"|"HERNANDEZ, Oscar Grouch"|-0-
+22|16|"aka"|"Alternate Name"|-0-
+25|57|"aka"|"AVIA IMPORT"|-0-
+36|219|"aka"|"BNC"|-0-
+36|220|"aka"|"NATIONAL BANK OF CUBA"|-0-
+36|221|"aka"|"BNC"|-0-
+41|222|"aka"|"NATIONAL BANK OF CUBA"|-0-
+66|223|"aka"|"BNC"|-0-
+66|224|"aka"|"NATIONAL BANK OF CUBA"|-0-

data/test/files/test_sdn_data_load.pip

@@ -0,0 +1,9 @@
+10|"ABASTECEDORA NAVAL Y INDUSTRIAL, S.A."|-0- |"CUBA"|-0- |-0- |-0- |-0- |-0- |-0- |-0- |-0-
+15|"ABDELNUR| Nury de Jesus"|"individual"|"CUBA"|-0- |-0- |-0- |-0- |-0- |-0- |-0- |-0-
+22|"HERNANDEZ, Oscar"|"individual"|"CUBA"|-0- |-0- |"Unknown vessel type"|-0- |-0- |-0- |"Acechilly Navigation Co., Malta"|-0-
+24|"LOPEZ MENDEZ, Luis Eduardo"|"individual"|"CUBA"|-0- |-0- |"Unknown vessel type"|-0- |-0- |-0- |"Acefrosty Shipping Co., Malta"|-0-
+25|"ACEFROSTY SHIPPING CO., LTD."|-0- |"CUBA"|-0- |-0- |-0- |-0- |-0- |-0- |-0- |-0-
+36|"AEROCARIBBEAN AIRLINES"|-0- |"CUBA"|-0- |-0- |-0- |-0- |-0- |-0- |-0- |-0-
+39|"AEROTAXI EJECUTIVO, S.A."|-0- |"CUBA"|-0- |-0- |-0- |-0- |-0- |-0- |-0- |-0-
+41|"AGENCIA DE VIAJES GUAMA"|-0- |"CUBA"|-0- |-0- |-0- |-0- |-0- |-0- |-0- |-0-
+66|"AGUIAR, Raul"|"individual"|"CUBA"|"Director, Banco Nacional de Cuba"|-0- |-0- |-0- |-0- |-0- |-0- |"; Director, Banco Nacional de Cuba."

data/test/files/valid_flattened_file.csv

@@ -0,0 +1,19 @@
+``|`ABASTECEDORA NAVAL Y INDUSTRIAL, S.A.`|``|`CUBA`|``|``|``|``|``|``|``|``|``|``|`Panama`|``|``|``|``|`2009-05-06 15:55:24`|`2009-05-06 15:55:24`
+``|`ABDELNUR`|` Nury de Jesus`|`individual`|`CUBA`|``|``|``|``|``|``|``|``|``|`Panama`|``|`aka`|`VIAJES GUAMA TOURS`|``|`2009-05-06 15:55:24`|`2009-05-06 15:55:24`
+``|`HERNANDEZ, Oscar`|`individual`|`CUBA`|``|``|`Unknown vessel type`|``|``|``|`Acechilly Navigation Co., Malta`|``|`123 Somewhere Ln`|`Clearwater`|`United States`|``|`aka`|`HERNANDEZ, Oscar Grouch`|``|`2009-05-06 15:55:24`|`2009-05-06 15:55:24`
+``|`HERNANDEZ, Oscar`|`individual`|`CUBA`|``|``|`Unknown vessel type`|``|``|``|`Acechilly Navigation Co., Malta`|``|`123 Somewhere Ln`|`Clearwater`|`United States`|``|`aka`|`Alternate Name`|``|`2009-05-06 15:55:24`|`2009-05-06 15:55:24`
+``|`LOPEZ MENDEZ, Luis Eduardo`|`individual`|`CUBA`|``|``|`Unknown vessel type`|``|``|``|`Acefrosty Shipping Co., Malta`|``|``|``|``|``|``|``|``|`2009-05-06 15:55:24`|`2009-05-06 15:55:24`
+``|`ACEFROSTY SHIPPING CO., LTD.`|``|`CUBA`|``|``|``|``|``|``|``|``|``|``|``|``|`aka`|`AVIA IMPORT`|``|`2009-05-06 15:55:24`|`2009-05-06 15:55:24`
+``|`AEROCARIBBEAN AIRLINES`|``|`CUBA`|``|``|``|``|``|``|``|``|``|``|``|``|`aka`|`BNC`|``|`2009-05-06 15:55:24`|`2009-05-06 15:55:24`
+``|`AEROCARIBBEAN AIRLINES`|``|`CUBA`|``|``|``|``|``|``|``|``|``|``|``|``|`aka`|`NATIONAL BANK OF CUBA`|``|`2009-05-06 15:55:24`|`2009-05-06 15:55:24`
+``|`AEROCARIBBEAN AIRLINES`|``|`CUBA`|``|``|``|``|``|``|``|``|``|``|``|``|`aka`|`BNC`|``|`2009-05-06 15:55:24`|`2009-05-06 15:55:24`
+``|`AEROTAXI EJECUTIVO, S.A.`|``|`CUBA`|``|``|``|``|``|``|``|``|``|`Managua`|`Nicaragua`|``|``|``|``|`2009-05-06 15:55:24`|`2009-05-06 15:55:24`
+``|`AEROTAXI EJECUTIVO, S.A.`|``|`CUBA`|``|``|``|``|``|``|``|``|`Bal Harbour Shopping Center, Via Italia`|`Panama City`|`Panama`|``|``|``|``|`2009-05-06 15:55:24`|`2009-05-06 15:55:24`
+``|`AGENCIA DE VIAJES GUAMA`|``|`CUBA`|``|``|``|``|``|``|``|``|`Avenida de Concha, Espina 8, E-28036`|`Madrid`|`Spain`|``|`aka`|`NATIONAL BANK OF CUBA`|``|`2009-05-06 15:55:24`|`2009-05-06 15:55:24`
+``|`AGENCIA DE VIAJES GUAMA`|``|`CUBA`|``|``|``|``|``|``|``|``|``|``|``|``|`aka`|`NATIONAL BANK OF CUBA`|``|`2009-05-06 15:55:24`|`2009-05-06 15:55:24`
+``|`AGUIAR, Raul`|`individual`|`CUBA`|`Director, Banco Nacional de Cuba`|``|``|``|``|``|``|`; Director, Banco Nacional de Cuba.`|``|`Milan`|`Italy`|``|`aka`|`BNC`|``|`2009-05-06 15:55:24`|`2009-05-06 15:55:24`
+``|`AGUIAR, Raul`|`individual`|`CUBA`|`Director, Banco Nacional de Cuba`|``|``|``|``|``|``|`; Director, Banco Nacional de Cuba.`|``|`Milan`|`Italy`|``|`aka`|`NATIONAL BANK OF CUBA`|``|`2009-05-06 15:55:24`|`2009-05-06 15:55:24`
+``|`AGUIAR, Raul`|`individual`|`CUBA`|`Director, Banco Nacional de Cuba`|``|``|``|``|``|``|`; Director, Banco Nacional de Cuba.`|``|``|`Panama`|``|`aka`|`BNC`|``|`2009-05-06 15:55:24`|`2009-05-06 15:55:24`
+``|`AGUIAR, Raul`|`individual`|`CUBA`|`Director, Banco Nacional de Cuba`|``|``|``|``|``|``|`; Director, Banco Nacional de Cuba.`|``|``|`Panama`|``|`aka`|`NATIONAL BANK OF CUBA`|``|`2009-05-06 15:55:24`|`2009-05-06 15:55:24`
+``|`AGUIAR, Raul`|`individual`|`CUBA`|`Director, Banco Nacional de Cuba`|``|``|``|``|``|``|`; Director, Banco Nacional de Cuba.`|`1840 West 49th Street`|`Hialeah, FL`|`United States`|``|`aka`|`BNC`|``|`2009-05-06 15:55:24`|`2009-05-06 15:55:24`
+``|`AGUIAR, Raul`|`individual`|`CUBA`|`Director, Banco Nacional de Cuba`|``|``|``|``|``|``|`; Director, Banco Nacional de Cuba.`|`1840 West 49th Street`|`Hialeah, FL`|`United States`|``|`aka`|`NATIONAL BANK OF CUBA`|``|`2009-05-06 15:55:24`|`2009-05-06 15:55:24`

data/test/mocks/test/ofac_sdn_loader.rb

@@ -0,0 +1,20 @@
+require 'ofac/models/ofac_sdn_loader'
+
+class OfacSdnLoader
+
+  def self.load_current_sdn_file
+    sdn = File.new(File.dirname(__FILE__) + '/../../files/test_sdn_data_load.pip')
+    address = File.new(File.dirname(__FILE__) + '/../../files/test_address_data_load.pip')
+    alt = File.new(File.dirname(__FILE__) + '/../../files/test_alt_data_load.pip')
+    active_record_file_load(sdn, address, alt)
+    sdn.close
+    address.close
+    alt.close
+  end
+
+  #Gives access to the private convert_to_flattened_csv method
+  def self.create_csv_file(sdn, address, alt)
+    convert_to_flattened_csv(sdn, address, alt)
+  end
+
+end

data/test/ofac_sdn_loader_test.rb

@@ -0,0 +1,40 @@
+require 'test_helper'
+
+class OfacSdnLoaderTest < Test::Unit::TestCase
+
+  context '' do
+    setup do setup_ofac_sdn_table end
+
+    should "load table from files multiple times and always have the same record count" do
+      assert_equal(0,OfacSdn.count)
+      OfacSdnLoader.load_current_sdn_file #this method is mocked to load test files instead of the live files from the web.
+      assert_equal(19, OfacSdn.count)
+      OfacSdnLoader.load_current_sdn_file
+      assert_equal(19, OfacSdn.count)
+    end
+
+    should "create flattened_csv_file_for_mysql_import" do
+      #since, I'm using sqlight3 for it's in memory db, I can't test the mysql load
+      #but I can test the csv file creation.
+      sdn = File.new(File.dirname(__FILE__) + '/files/test_sdn_data_load.pip')
+      address = File.new(File.dirname(__FILE__) + '/files/test_address_data_load.pip')
+      alt = File.new(File.dirname(__FILE__) + '/files/test_alt_data_load.pip')
+
+      csv = OfacSdnLoader.create_csv_file(sdn, address, alt) #this method was created in the mock only to call the private convert_to_flattened_csv method
+      correctly_formatted_csv = File.open(File.dirname(__FILE__) + '/files/valid_flattened_file.csv')
+
+      csv.rewind
+      generated_file = csv.readlines
+      #compare the values of each csv line, with the correctly formated "control file"
+      correctly_formatted_csv.each_with_index do |line,i|
+        csv_line = generated_file[i]
+        correctly_formatted_record_array = line.split('|')
+        csv_record_array = csv_line.split('|')
+        (0..18).each do |i| #skip indices 19 and 20, they are the created_at and updated_at fields, they will never match.
+          assert_equal correctly_formatted_record_array[i], csv_record_array[i]
+        end
+      end
+    end
+
+  end
+end

data/test/ofac_test.rb

@@ -0,0 +1,76 @@
+require 'test_helper'
+
+class OfacTest < Test::Unit::TestCase
+
+  context '' do
+    setup do
+      setup_ofac_sdn_table
+      OfacSdnLoader.load_current_sdn_file #this method is mocked to load test files instead of the live files from the web.
+    end
+    
+    should "give a score of 0 if no name is given" do
+      assert_equal 0, Ofac.new({:address => '123 somewhere'}).score
+    end
+
+    should "give a score of 0 if there is no name match" do
+      assert_equal 0, Ofac.new({:name => 'Kevin'}).score
+    end
+
+    should "give a score of 0 if there is no name match but there is an address and city match" do
+      assert_equal 0, Ofac.new({:name => 'Kevin', :address => '123 somewhere ln', :city => 'Clearwater'}).score
+    end
+
+    should "give a score of 60 if there is a name match" do
+      assert_equal 60, Ofac.new({:name => 'Oscar Hernandez'}).score
+      assert_equal 60, Ofac.new({:name => 'Oscar Hernandez', :city => 'no match', :address => 'no match'}).score
+      assert_equal 60, Ofac.new({:name => 'Oscar Hernandez', :city => 'Las Vegas', :address => 'no match'}).score
+      assert_equal 60, Ofac.new({:name => 'Luis Lopez', :city => 'Las Vegas', :address => 'no match'}).score
+    end
+
+    should "give a score of 60 if there is a name match on alternate identity name" do
+      assert_equal 60, Ofac.new({:name => 'Alternate Name'}).score
+    end
+
+    should "give a partial score if there is a partial name match" do
+      assert_equal 40, Ofac.new({:name => 'Oscar middlename Hernandez'}).score
+      assert_equal 30, Ofac.new({:name => 'Oscar WrongLastName'}).score
+      assert_equal 70, Ofac.new({:name => 'Oscar middlename Hernandez',:city => 'Clearwater'}).score
+    end
+
+    should "give a score of 90 if there is a name and city match" do
+      assert_equal 90, Ofac.new({:name => 'Oscar Hernandez', :city => 'Clearwater', :address => 'no match'}).score
+    end
+
+    should "give a score of 100 if there is a name and city and address match" do
+      assert_equal 100, Ofac.new({:name => 'Oscar Hernandez', :city => 'Clearwater', :address => '123 somewhere ln'}).score
+    end
+
+    should "give partial scores for sounds like matches" do
+
+      #32456 summer lane sounds like 32456 Somewhere ln so is adds 75% of the address weight to the score, or 8.
+      assert_equal 98, Ofac.new({:name => 'Oscar Hernandez', :city => 'Clearwater', :address => '32456 summer lane'}).score
+
+      #summer sounds like somewhere, and all numbers sound alike, so 2 of the 3 address elements match by sound.
+      #Each element is worth 10\3 or 3.33.  Exact matches add 2.33 each, and the sounds like adds 2.33 * .75 or 2.5
+      #because sounds like matches only add 75% of it's weight.
+      #2.5 + 2.5 = 5
+      assert_equal 95, Ofac.new({:name => 'Oscar Hernandez', :city => 'Clearwater', :address => '12358 summer blvd'}).score
+
+
+      #Louis sounds like Luis, and Lopez is an exact match:
+      #:name has a weight of 60, so each element is worth 30.  A sounds like match is worth 30 * .75
+      assert_equal 53, Ofac.new({:name => 'Louis Lopez', :city => 'Las Vegas', :address => 'no match'}).score
+    end
+
+    should "return an array of possible hits" do
+      #it should matter which order you call score or possible hits.
+      sdn = Ofac.new({:name => 'Oscar Hernandez', :city => 'Clearwater', :address => '123 somewhere ln'})
+      assert sdn.score > 0
+      assert !sdn.possible_hits.empty?
+
+      sdn = Ofac.new({:name => 'Oscar Hernandez', :city => 'Clearwater', :address => '123 somewhere ln'})
+      assert !sdn.possible_hits.empty?
+      assert sdn.score > 0
+    end
+  end
+end

data/test/test_helper.rb

@@ -0,0 +1,48 @@
+require 'rubygems'
+require 'test/unit'
+require 'shoulda'
+require 'mocks/test/ofac_sdn_loader'
+
+$LOAD_PATH.unshift(File.join(File.dirname(__FILE__), '..', 'lib'))
+$LOAD_PATH.unshift(File.dirname(__FILE__))
+require 'ofac'
+
+ActiveRecord::Base.establish_connection :adapter => 'sqlite3', :database => ':memory:'
+
+class Test::Unit::TestCase
+  def setup_ofac_sdn_table
+    ActiveRecord::Base.connection.tables.each { |table| ActiveRecord::Base.connection.drop_table(table) }
+    create_ofac_sdn_table
+  end
+
+  private
+
+  def create_ofac_sdn_table
+    silence_stream(STDOUT) do
+      ActiveRecord::Schema.define(:version => 1) do
+        create_table :ofac_sdns do |t|
+          t.text      :name
+          t.string    :sdn_type
+          t.string    :program
+          t.string    :title
+          t.string    :vessel_call_sign
+          t.string    :vessel_type
+          t.string    :vessel_tonnage
+          t.string    :gross_registered_tonnage
+          t.string    :vessel_flag
+          t.string    :vessel_owner
+          t.text      :remarks
+          t.text      :address
+          t.string    :city
+          t.string    :country
+          t.string    :address_remarks
+          t.string    :alternate_identity_type
+          t.text      :alternate_identity_name
+          t.string    :alternate_identity_remarks
+          t.timestamps
+        end
+      end
+    end
+  end
+
+end

metadata

@@ -0,0 +1,90 @@
+--- !ruby/object:Gem::Specification 
+name: kevintyll-ofac
+version: !ruby/object:Gem::Version 
+  version: 1.0.0
+platform: ruby
+authors: 
+- Kevin Tyll
+autorequire: 
+bindir: bin
+cert_chain: []
+
+date: 2009-05-11 00:00:00 -07:00
+default_executable: 
+dependencies: []
+
+description: Attempts to find a hit on the Office of Foreign Assets Control's Specially Designated Nationals list.
+email: kevintyll@gmail.com
+executables: []
+
+extensions: []
+
+extra_rdoc_files: 
+- LICENSE
+- README.rdoc
+files: 
+- History.txt
+- LICENSE
+- PostInstall.txt
+- README.rdoc
+- Rakefile
+- VERSION.yml
+- generators/ofac_migration/ofac_migration_generator.rb
+- generators/ofac_migration/templates/migration.rb
+- lib/ofac.rb
+- lib/ofac/models/ofac.rb
+- lib/ofac/models/ofac_sdn.rb
+- lib/ofac/models/ofac_sdn_loader.rb
+- lib/ofac/ofac_match.rb
+- lib/ofac/ruby_string_extensions.rb
+- lib/tasks/ofac.rake
+- test/files/test_address_data_load.pip
+- test/files/test_alt_data_load.pip
+- test/files/test_sdn_data_load.pip
+- test/files/valid_flattened_file.csv
+- test/mocks/test/ofac_sdn_loader.rb
+- test/ofac_sdn_loader_test.rb
+- test/ofac_test.rb
+- test/test_helper.rb
+has_rdoc: true
+homepage: http://github.com/kevintyll/ofac
+post_install_message: |-
+  For more information on ofac, see http://kevintyll.github.com/ofac/
+  
+  * To create the necessary db migration, from the command line, run:
+      script/generate ofac_migration
+  * Require the gem in your environment.rb file in the Rails::Initializer block:
+      config.gem 'kevintyll-ofac', :lib => 'ofac'
+  * To load your table with the current OFAC data, from the command line, run:
+      rake ofac:update_data
+  
+      * The OFAC data is not updated with any regularity, but you can sign up for email notifications when the data changes at
+          http://www.treas.gov/offices/enforcement/ofac/sdn/index.shtml.
+rdoc_options: 
+- --charset=UTF-8
+require_paths: 
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement 
+  requirements: 
+  - - ">="
+    - !ruby/object:Gem::Version 
+      version: "0"
+  version: 
+required_rubygems_version: !ruby/object:Gem::Requirement 
+  requirements: 
+  - - ">="
+    - !ruby/object:Gem::Version 
+      version: "0"
+  version: 
+requirements: []
+
+rubyforge_project: 
+rubygems_version: 1.2.0
+signing_key: 
+specification_version: 2
+summary: Attempts to find a hit on the Office of Foreign Assets Control's Specially Designated Nationals list.
+test_files: 
+- test/mocks/test/ofac_sdn_loader.rb
+- test/ofac_sdn_loader_test.rb
+- test/ofac_test.rb
+- test/test_helper.rb