kevintyll-ssn_validator 0.1.2 → 1.0.0

data/History.txt

@@ -3,8 +3,16 @@
 * 1 major enhancement:
   * Initial release
 
-== 0.1.2 2009-15-14
+== 0.1.2 2009-04-15
 
 * 2 minor enhancements:
   * Added rdoc files
   * Updated documentation
+
+== 1.0.0 2009-04-22
+
+* 1 major enhancement:
+  * Added the death master file to determine if the ssn belongs to the deceased.
+
+* 1 minor enhancement:
+  * Fixed bug where an error was thrown if the ssn area has not been assigned yet.

data/Manifest.txt

@@ -5,8 +5,14 @@ README.rdoc
 Rakefile
 generators/ssn_validator_migration/templates/migration.rb
 generators/ssn_validator_migration/ssn_validator_migration_generator.rb
+generators/death_master_file_migration/templates/migration.rb
+generators/death_master_file_migration/death_master_file_migration_generator.rb
+lib/ssn_validator/ntis.rb
 lib/ssn_validator/models/ssn_high_group_code.rb
+lib/ssn_validator/models/ssn_high_group_code_loader.rb
 lib/ssn_validator/models/ssn_validator.rb
+lib/ssn_validator/models/death_master_file.rb
+lib/ssn_validator/models/death_master_file_loader.rb
 lib/ssn_validator.rb
 lib/tasks/ssn_validator.rake
 script/console
@@ -14,4 +20,8 @@ script/destroy
 script/generate
 test/test_helper.rb
 test/test_ssn_validator.rb
-test/test_ssn_high_group_code.rb
+test/test_ssn_high_group_code_loader.rb
+test/test_death_master_file_loader.rb
+test/mocks/test/death_master_file_loader.rb
+test/files/test_dmf_initial_load.txt
+test/files/test_dmf_update_load.txt

data/PostInstall.txt

@@ -1,9 +1,42 @@
 
 For more information on ssn_validator, see http://kevintyll.github.com/ssn_validator/
 
-* To create the necessary db migration, from the command line, run:  script/generate ssn_validator_migration
-* Require the gem in your environment.rb file in the Rails::Initializer block:  config.gem 'ssn_validator'
-* To load your table with the current SSN data, from the command line, run:  rake ssn_validator:update_data
-* The SSN data is updated monthly, so you'll want to run this rake task monthly to keep your validations accurate.
+* To create the necessary db migration, from the command line, run:
+    script/generate ssn_validator_migration
+* Require the gem in your environment.rb file in the Rails::Initializer block:
+    config.gem 'kevintyll-ssn_validator', :lib => 'ssn_validator'
+* To load your table with the current SSN data, from the command line, run:
+    rake ssn_validator:update_data
+
+    * The SSN data is updated monthly, so you'll want to run this rake task monthly to keep your validations accurate.
+
+* If you've purchased the death master file data, create the death_master_files migration:
+    script/generate death_master_file_migration
+* To load the dmf files you receive from ntis:
+    rake ssn_validator:death_master_file:load_file PATH='path/to/file' AS_OF='2009-03-01'
+
+    * You'll need to pass in the full path to where the file is on disk.  You'll also need
+      to pass in the date this file's data are as of in the format yyyy-mm-dd.
+
+    * This task must be used to load the initial file you receive from ntis on CD.  It can optionally be used
+      to load the monthly update files you download from the website.  If you manually download the update files,
+      you do not need to add your user name and password to the environment.rb file.  For a more automated approach
+      to loading the update files, add your user name and password to the environment.rb file and use the 2nd rake task.
+
+* To load the monthly updates from the ntis website:
+    * Add your user_name and password to the environment.rb file
+
+        SsnValidator::Ntis.user_name = 'REPLACE WITH YOUR dmf.ntis.gov USER NAME'
+        SsnValidator::Ntis.password = 'REPLACE WITH YOUR dmf.ntis.gov PASSWORD'
+
+    * Run the rake task:
+
+        rake ssn_validator:death_master_file:update_data
+
+        * This rake task will determine the most recent file that has been loaded, and loads all subsequent files in order
+        from the dmf.ntis.gov website.
+
+        * The death master file data is updated monthly, so you'll want to run this rake task monthly to keep your validations accurate.
+
 
 

data/README.rdoc

@@ -13,7 +13,7 @@ has ever been issued or not.
 
 ssn_validator started as a need for the company I work for, Clarity Services Inc.  Incredibly, we
 couldn't find an existing gem or service that provided what we needed.  Since we were going to have
-to role our own solution, we decided to create a gem out of it and share it with the community.  Much
+to roll our own solution, we decided to create a gem out of it and share it with the community.  Much
 thanks goes to the management at Clarity Services Inc. for allowing this code to be open sourced.
 
 
@@ -21,13 +21,13 @@ thanks goes to the management at Clarity Services Inc. for allowing this code to
 
 * What it can do:
     Validates the likelyhood that an SSN has been issued to someone.
+    Checks the Death Master File if the SSN belongs to a dead person.  This will require you to purchase the dmf data from https://dmf.ntis.gov
 
 * What it cannot do:
     Validate that an SSN actually belongs to a particular person.
 
 * What it's planned to do:
     Determine when an SSN was issued...if i can find the historical data.  This can be used to further validate an SSN by comparing it to a Date of Birth.
-    Check the Death Master File if the SSN belongs to a dead person.  This will require you to purchase the dmf data from https://dmf.ntis.gov 
 
 == SYNOPSIS:
 
@@ -36,6 +36,14 @@ thanks goes to the management at Clarity Services Inc. for allowing this code to
 
 * Then check if it's valid
     ssn.valid?
+    ssn.death_master_file_hit?
+
+    ssn.valid? only checks to see if the number itself is valid.  Check the death_master_file_hit? method
+    as well to verify the ssn does not belong the the deceased.
+
+    The death_master_file_hit? method will only return true of false if you have purchased the dmf data and
+    loaded it into your database.  There is a generator and rake task to facilitate this, but you'll have
+    to purchase the data yourself at https://dmf.ntis.gov.
 
 * You can check the errors array to see why it's not valid.
     ssn.errors
@@ -51,10 +59,39 @@ thanks goes to the management at Clarity Services Inc. for allowing this code to
 * To create the necessary db migration, from the command line, run:
     script/generate ssn_validator_migration
 * Require the gem in your environment.rb file in the Rails::Initializer block:
-    config.gem 'ssn_validator'
+    config.gem 'kevintyll-ssn_validator', :lib => 'ssn_validator'
 * To load your table with the current SSN data, from the command line, run:
     rake ssn_validator:update_data
-* The SSN data is updated monthly, so you'll want to run this rake task monthly to keep your validations accurate.
+
+    * The SSN data is updated monthly, so you'll want to run this rake task monthly to keep your validations accurate.
+
+* If you've purchased the death master file data, create the death_master_files migration:
+    script/generate death_master_file_migration
+* To load the dmf files you receive from ntis:
+    rake ssn_validator:death_master_file:load_file PATH='path/to/file' AS_OF='2009-03-01'
+
+    * You'll need to pass in the full path to where the file is on disk.  You'll also need
+      to pass in the date this file's data are as of in the format yyyy-mm-dd.
+
+    * This task must be used to load the initial file you receive from ntis on CD.  It can optionally be used
+      to load the monthly update files you download from the website.  If you manually download the update files,
+      you do not need to add your user name and password to the environment.rb file.  For a more automated approach
+      to loading the update files, add your user name and password to the environment.rb file and use the 2nd rake task.
+
+* To load the monthly updates from the ntis website:
+    * Add your user_name and password to the environment.rb file
+
+        SsnValidator::Ntis.user_name = 'REPLACE WITH YOUR dmf.ntis.gov USER NAME'
+        SsnValidator::Ntis.password = 'REPLACE WITH YOUR dmf.ntis.gov PASSWORD'
+
+    * Run the rake task:
+
+        rake ssn_validator:death_master_file:update_data
+
+        * This rake task will determine the most recent file that has been loaded, and loads all subsequent files in order
+        from the dmf.ntis.gov website.
+
+        * The death master file data is updated monthly, so you'll want to run this rake task monthly to keep your validations accurate.
 
 == LICENSE:
 

data/Rakefile

@@ -13,8 +13,8 @@ $hoe = Hoe.new('ssn_validator', SsnValidator::VERSION) do |p|
   p.extra_dev_deps = [
     ['newgem', ">= #{::Newgem::VERSION}"]
   ]
-  
-  p.clean_globs |= %w[**/.DS_Store tmp *.log]
+
+  p.clean_globs |= %w[**/.DS_Store tmp *.log nbproject pkg]
   path = (p.rubyforge_name == p.name) ? p.rubyforge_name : "\#{p.rubyforge_name}/\#{p.name}"
   p.remote_rdoc_dir = File.join(path.gsub(/^#{p.rubyforge_name}\/?/,''), 'rdoc')
   p.rsync_args = '-av --delete --ignore-errors'

data/generators/death_master_file_migration/death_master_file_migration_generator.rb

@@ -0,0 +1,12 @@
+class DeathMasterFileMigrationGenerator < Rails::Generator::Base
+  def manifest 
+    record do |m|
+      #m.directory File.join('db')
+      m.migration_template 'migration.rb', 'db/migrate' 
+    end 
+  end
+  
+  def file_name
+    "create_death_master_file_table"
+  end
+end

data/generators/death_master_file_migration/templates/migration.rb

@@ -0,0 +1,26 @@
+class CreateDeathMasterFileTable < ActiveRecord::Migration
+  
+  def self.up
+    create_table :death_master_files do |t|
+      t.string :social_security_number
+      t.string  :last_name
+      t.string  :name_suffix
+      t.string :first_name
+      t.string :middle_name
+      t.string :verify_proof_code
+      t.date  :date_of_death
+      t.date  :date_of_birth
+      t.string  :state_of_residence
+      t.string :last_known_zip_residence
+      t.string :last_known_zip_payment
+      t.time_stamps
+      t.date :as_of
+    end
+    add_index :death_master_files, :social_security_number, :unique => true
+    add_index :death_master_files, :as_of
+  end
+
+  def self.down
+    drop_table :death_master_files
+  end
+end

data/lib/ssn_validator/models/death_master_file.rb

@@ -0,0 +1,5 @@
+require 'activerecord'
+
+class DeathMasterFile < ActiveRecord::Base
+ 
+end

data/lib/ssn_validator/models/death_master_file_loader.rb

@@ -0,0 +1,235 @@
+require 'net/http'
+require 'net/https'
+require 'activerecord'
+require 'active_record/connection_adapters/mysql_adapter'
+require 'ssn_validator/ntis'
+
+class DeathMasterFileLoader
+
+  # path_or_url is the full path to the file to load on disk, or the url of an update file.
+  # as_of is a string in the formatt YYYY-MM-DD for which the file data is accurate.
+  def initialize(path_or_url,file_as_of)
+    @file_path_or_url = path_or_url
+    @file_as_of = file_as_of
+    valid?
+  end
+
+  def valid?
+    raise(ArgumentError, "path_or_url not specified") unless @file_path_or_url
+    raise(ArgumentError, "as_of not specified") unless @file_as_of
+    max_as_of = DeathMasterFile.maximum(:as_of)
+    raise(ArgumentError, "A more recent file has already been processed.  DB as_of date #{max_as_of}") if  max_as_of && (max_as_of >= @file_as_of.to_date)
+   
+    if File.exists?(@file_path_or_url)
+      @download_file = File.open(@file_path_or_url)
+    elsif URI.parse(@file_path_or_url).kind_of?(URI::HTTP)
+      @download_file = get_file_from_web
+    else
+      raise(Errno::ENOENT, @file_path_or_url)
+    end
+  end
+
+  def load_file
+
+    if DeathMasterFile.connection.kind_of?(ActiveRecord::ConnectionAdapters::MysqlAdapter)
+      puts "Converting file to csv format for Mysql import.  This could take several minutes."
+
+      csv_file = convert_file_to_csv
+
+      bulk_mysql_update(csv_file)
+    else
+      active_record_file_load
+    end
+
+  end
+
+  def get_file_from_web
+    uri = URI.parse(@file_path_or_url)
+    
+    request = Net::HTTP::Get.new(uri.request_uri)
+    request.basic_auth(SsnValidator::Ntis.user_name,SsnValidator::Ntis.password)
+
+    http = Net::HTTP.new(uri.host, uri.port)
+    http.use_ssl = (uri.port == 443)
+    http.verify_mode = OpenSSL::SSL::VERIFY_NONE
+    
+    response = http.request(request)
+    
+    raise(ArgumentError, "Invalid URL: #{@file_path_or_url}") if response.kind_of?(Net::HTTPNotFound)
+    raise(ArgumentError, "Authorization Required: Invalid username or password.  Set the variables SsnValidator::Ntis.user_name and SsnValidator::Ntis.password in your environment.rb file.") if response.kind_of?(Net::HTTPUnauthorized)
+    
+    return response.body
+  end
+
+  #Loads all the update files from dmf.ntis.gov.
+  #It starts with the last file loaded, and loads each
+  #missing file in sequence up to the current file.
+  def self.load_update_files_from_web
+    max_as_of = DeathMasterFile.maximum(:as_of)
+    run_file_date = max_as_of.beginning_of_month.next_month
+    last_file_date = Date.today.beginning_of_month
+    while run_file_date <= last_file_date
+      url = "https://dmf.ntis.gov/dmldata/monthly/MA#{run_file_date.strftime("%y%m%d")}"
+      puts "Loading file #{url}"
+      DeathMasterFileLoader.new(url,run_file_date.strftime("%Y-%m-%d")).load_file
+      run_file_date += 1.month
+    end
+  end
+
+  private
+
+  # Processes 28 million rows in 23 minutes. Input file 2.6GB output: 2.9GB.
+  # Used to convert a packed fixed-length file into csv for mysql import.
+  def convert_file_to_csv
+    #packed_file = File.open(file_path)
+    csv_file = Tempfile.new("dmf")    # create temp file for converted csv formmat.
+
+
+    start = Time.now
+    timenow = start.to_s(:db)
+
+    @delete_ssns = []
+
+    @download_file.each_with_index do |line,i|
+      action = record_action(line)
+      attributes_hash = text_to_hash(line)
+      if  action == 'D'
+        #keep track of all the records to delete.  We'll delete at the end all at once.
+        @delete_ssns << attributes_hash[:social_security_number]
+      else
+        # empty field for id to be generated by mysql.
+        newline = "''," +
+          # social_security_number
+        "'#{attributes_hash[:social_security_number]}'," +
+          # last_name
+        "'#{attributes_hash[:last_name]}'," +
+          # name_suffix
+        "'#{attributes_hash[:name_suffix]}'," +
+          # first_name
+        "'#{attributes_hash[:first_name]}'," +
+          # middle_name
+        "'#{attributes_hash[:middle_name]}'," +
+          # verify_proof_code
+        "'#{attributes_hash[:verify_proof_code]}'," +
+          # date_of_death - need YYYY-MM-DD.
+        "'#{attributes_hash[:date_of_death]}'," +
+          # date_of_birth - need YYYY-MM-DD.
+        "'#{attributes_hash[:date_of_birth]}'," +
+          # state_of_residence - must be code between 01 and 65 or else nil.
+        "'#{attributes_hash[:state_of_residence]}'," +
+          # last_known_zip_residence
+        "'#{attributes_hash[:last_known_zip_residence]}'," +
+          # last_known_zip_payment
+        "'#{attributes_hash[:last_known_zip_payment]}'," +
+          # created_at
+        "'#{timenow}'," +
+          # updated_at
+        "'#{timenow}'," +
+          # as_of
+        "'#{attributes_hash[:as_of]}'" +"\n"
+
+        csv_file.puts newline
+        puts "#{i} records processed." if (i % 25000 == 0) && (i > 0)
+      end
+    end
+    puts "File conversion ran for #{(Time.now - start) / 60} minutes."
+    return csv_file
+  end
+
+  #Uses active record to load the data.
+  #The benefit is it will work on any database.
+  #The downside is it's really slow.
+  def active_record_file_load
+    puts 'Importing file into database. This could take many minutes.'
+
+    @download_file.each_with_index do |line,i|
+      action = record_action(line)
+      attributes_hash = text_to_hash(line)
+      if  action == 'D'
+        DeathMasterFile.destroy_all(['social_security_number = ?', attributes_hash[:social_security_number]])
+      else
+
+        # empty field for id to be generated by mysql.
+#        record_hash = {
+#          :as_of => @file_as_of.to_date.to_s(:db),
+#          :social_security_number => parse_record(line,:social_security_number),
+#          :last_name => parse_record(line,:last_name),
+#          :name_suffix => parse_record(line,:name_suffix),
+#          :first_name => parse_record(line,:first_name),
+#          :middle_name => parse_record(line,:middle_name),
+#          :verify_proof_code => parse_record(line,:verify_proof_code),
+#          :date_of_death => parse_record(line,:date_of_death),
+#          :date_of_birth => parse_record(line,:date_of_birth),
+#          # - must be code between 01 and 65 or else nil.
+#          :state_of_residence=> parse_record(line,:state_of_residence=),
+#          :last_known_zip_residence => parse_record(line,:last_known_zip_residence),
+#          :last_known_zip_payment => parse_record(line,:last_known_zip_payment)
+#        }
+
+        case action
+        when '',nil,' '
+          #the initial file leaves this field blank
+          DeathMasterFile.create(attributes_hash)
+        else
+          dmf = DeathMasterFile.find_by_social_security_number(attributes_hash[:social_security_number])
+          if dmf
+            #a record already exists, update this record
+            dmf.update_attributes(attributes_hash)
+          else
+            #create a new record
+            DeathMasterFile.create(attributes_hash)
+          end
+        end
+      end
+
+      puts "#{i} records processed." if (i % 2500 == 0) && (i > 0)
+
+    end
+
+    puts "Import complete."
+  end
+
+  # For mysql, use:
+  # LOAD DATA LOCAL INFILE 'ssdm1.csv' INTO TABLE death_master_files FIELDS TERMINATED BY ',' ENCLOSED BY "'" LINES TERMINATED BY '\n';
+  # This is a much faster way of loading large amounts of data into mysql.  For information on the LOAD DATA command
+  # see http://dev.mysql.com/doc/refman/5.1/en/load-data.html
+  def bulk_mysql_update(csv_file)
+    puts "Importing into Mysql..."
+
+    #delete all the 'D' records
+    DeathMasterFile.destroy_all(:social_security_number => @delete_ssns)
+
+    #This will insert new records, and replace records with existing ssns.
+    #This only works because there is a unique index on social_security_number.
+    mysql_command = <<TEXT
+LOAD DATA LOCAL INFILE '#{csv_file.path}' REPLACE INTO TABLE death_master_files FIELDS TERMINATED BY ',' ENCLOSED BY "'" LINES TERMINATED BY '\n';
+TEXT
+
+    DeathMasterFile.connection.execute(mysql_command)
+    puts "Mysql import complete."
+
+  end
+
+  def record_action(line)
+    line[0,1].to_s.strip
+  end
+
+  def text_to_hash(line)
+
+    {:as_of => @file_as_of.to_date.to_s(:db),
+     :social_security_number => line[1,9].to_s.strip,
+     :last_name => line[10,20].to_s.strip,
+     :name_suffix => line[30,4].to_s.strip,
+     :first_name => line[34,15].to_s.strip,
+     :middle_name => line[49,15].to_s.strip,
+     :verify_proof_code => line[64,1].to_s.strip,
+     :date_of_death => (Date.strptime(line[65,8].to_s.strip,'%m%d%Y') rescue nil),
+     :date_of_birth => (Date.strptime(line[73,8].to_s.strip,'%m%d%Y') rescue nil),
+      # - must be code between 01 and 65 or else nil.
+     :state_of_residence => (line[81,2].to_s.strip.between?('01', '65') ? line[81,2].to_s.strip : nil),
+     :last_known_zip_residence => line[83,5].to_s.strip,
+     :last_known_zip_payment => line[88,5].to_s.strip}
+  rescue Exception => e
+    puts '@@@@@@@@@ Error = ' + e.message + ':   ' + line.inspect
+  end
+end

data/lib/ssn_validator/models/ssn_high_group_code.rb

@@ -1,81 +1,5 @@
 require 'activerecord'
-require 'net/http'
-class SsnHighGroupCode < ActiveRecord::Base
-
-#  def self.load_historical_high_group_codes_file
-#    ['Jan','Feb','Mar','Apr','May','June','July','Aug','Sept','Oct','Nov','Dec'].each do |month|
-#      (1..10).each do |day|
-#        string_day = day.to_s
-#        string_day.insert(0,'0') if day < 10
-#        current_year = Date.today.year
-#        #(1932..current_year).each do |year|
-#        [2003].each do |year|
-#          string_year = year.to_s.last(2)
-#          ['','corrected'].each do |mod|
-#            ['ssns','ssnvs'].each do |url_mod|
-#              file_name = "HG#{month}#{string_day}#{string_year}#{mod}.txt"
-#              text = Net::HTTP.get(URI.parse("http://www.socialsecurity.gov/employer/#{url_mod}/#{file_name}"))
-#              puts file_name.inspect
-#              puts '@@@@@@@@@ found file_name = ' + file_name.inspect unless text.include? 'File Not Found'
-#            end
-#          end
-#
-#        end
-#      end
-#
-#    end
-#  end
-
-  #Loads the most recent file from http://www.socialsecurity.gov/employer/ssns/highgroup.txt
-  def self.load_current_high_group_codes_file
-    text = Net::HTTP.get(URI.parse('http://www.socialsecurity.gov/employer/ssns/highgroup.txt'))
-    create_records(parse_text(text),extract_as_of_date(text))
-  end
-
-
-  private
 
-  def self.already_loaded?(file_as_of_date)
-    self.find_by_as_of(file_as_of_date)
-  end
-
-  def self.create_records(area_groups,file_as_of)
-    if already_loaded?(file_as_of)
-      "File as of #{file_as_of} has already been loaded."
-    else
-      area_groups.each do |area_group|
-        self.create(area_group.merge!(:as_of => file_as_of.to_s(:db)))
-      end
-      "File as of #{file_as_of} loaded."
-    end
-  end
-
-  #extract the date from the file in the format mm/dd/yy
-  def self.extract_as_of_date(text)
-    as_of_start_index = text =~ /\d\d\/\d\d\/\d\d/
-    ::Date.new(*::Date._parse($&,true).values_at(:year, :mon, :mday)) unless as_of_start_index.nil?
-  end
-
-  #The formatting of the file is a little bit messy. Sometimes tabs are
-  #used as delimiters and sometimes spaces are used as delimiters. Also, the asterisks indicating recent changes are not
-  #necessary for our purposes
-  #Returns an array of hashes.
-  def self.parse_text(text)
-    text.gsub!('*',' ')
-    text.gsub!(/\t/, ' ')
-    text_array = text.split(/\n/).compact
-    area_groups = []
-    text_array.each do |t|
-      t.gsub!(/\s+/,' ')
-      next if t =~ /[[:alpha:]]/ #skip over the header lines
+class SsnHighGroupCode < ActiveRecord::Base
 
-      if t =~ /\d\d\d \d\d/ #we want the lines with area group pairs
-        t.gsub(/\d\d\d \d\d/) do |s|
-          area_group = s.split(' ')
-          area_groups << {:area => area_group.first, :group => area_group.last}
-        end
-      end
-    end
-    return area_groups
-  end
 end

data/lib/ssn_validator/models/ssn_high_group_code_loader.rb

@@ -0,0 +1,80 @@
+
+require 'net/http'
+class SsnHighGroupCodeLoader
+  #  def self.load_historical_high_group_codes_file
+#    ['Jan','Feb','Mar','Apr','May','June','July','Aug','Sept','Oct','Nov','Dec'].each do |month|
+#      (1..10).each do |day|
+#        string_day = day.to_s
+#        string_day.insert(0,'0') if day < 10
+#        current_year = Date.today.year
+#        #(1932..current_year).each do |year|
+#        [2003].each do |year|
+#          string_year = year.to_s.last(2)
+#          ['','corrected'].each do |mod|
+#            ['ssns','ssnvs'].each do |url_mod|
+#              file_name = "HG#{month}#{string_day}#{string_year}#{mod}.txt"
+#              text = Net::HTTP.get(URI.parse("http://www.socialsecurity.gov/employer/#{url_mod}/#{file_name}"))
+#              puts file_name.inspect
+#              puts '@@@@@@@@@ found file_name = ' + file_name.inspect unless text.include? 'File Not Found'
+#            end
+#          end
+#
+#        end
+#      end
+#
+#    end
+#  end
+
+  #Loads the most recent file from http://www.socialsecurity.gov/employer/ssns/highgroup.txt
+  def self.load_current_high_group_codes_file
+    text = Net::HTTP.get(URI.parse('http://www.socialsecurity.gov/employer/ssns/highgroup.txt'))
+    create_records(parse_text(text),extract_as_of_date(text))
+  end
+
+
+  private
+
+  def self.already_loaded?(file_as_of_date)
+    SsnHighGroupCode.find_by_as_of(file_as_of_date)
+  end
+
+  def self.create_records(area_groups,file_as_of)
+    if already_loaded?(file_as_of)
+      puts "File as of #{file_as_of} has already been loaded."
+    else
+      area_groups.each do |area_group|
+        SsnHighGroupCode.create(area_group.merge!(:as_of => file_as_of.to_s(:db)))
+      end
+      puts "File as of #{file_as_of} loaded."
+    end
+  end
+
+  #extract the date from the file in the format mm/dd/yy
+  def self.extract_as_of_date(text)
+    as_of_start_index = text =~ /\d\d\/\d\d\/\d\d/
+    Date.strptime($&,'%m/%d/%y') unless as_of_start_index.nil?
+  end
+
+  #The formatting of the file is a little bit messy. Sometimes tabs are
+  #used as delimiters and sometimes spaces are used as delimiters. Also, the asterisks indicating recent changes are not
+  #necessary for our purposes
+  #Returns an array of hashes.
+  def self.parse_text(text)
+    text.gsub!('*',' ')
+    text.gsub!(/\t/, ' ')
+    text_array = text.split(/\n/).compact
+    area_groups = []
+    text_array.each do |t|
+      t.gsub!(/\s+/,' ')
+      next if t =~ /[[:alpha:]]/ #skip over the header lines
+
+      if t =~ /\d\d\d \d\d/ #we want the lines with area group pairs
+        t.gsub(/\d\d\d \d\d/) do |s|
+          area_group = s.split(' ')
+          area_groups << {:area => area_group.first, :group => area_group.last}
+        end
+      end
+    end
+    return area_groups
+  end
+end

data/lib/ssn_validator/models/ssn_validator.rb

@@ -1,7 +1,7 @@
 module SsnValidator
   class Ssn
 
-    attr_reader :ssn,:area,:group,:serial_number
+    attr_reader :ssn,:area,:group,:serial_number,:as_of
     attr_reader :errors
 
 
@@ -42,6 +42,7 @@ module SsnValidator
         if @ssn_high_group_code.nil?
           @errors << "Area '#{@area}' has not been assigned."
         else
+          @as_of = @ssn_high_group_code.as_of
 
           define_group_ranks
           
@@ -60,6 +61,11 @@ module SsnValidator
       @errors.empty?
     end
 
+    #Determines if the passed in ssn belongs to the deceased.
+    def death_master_file_hit?
+      DeathMasterFile.find_by_social_security_number(@ssn)
+    end
+
 
     private
 

data/lib/ssn_validator/ntis.rb

@@ -0,0 +1,9 @@
+module SsnValidator
+  class Ntis
+    @@user_name = 'REPLACE WITH YOUR dmf.ntis.gov USER NAME'
+    cattr_accessor :user_name
+    
+    @@password = 'REPLACE WITH YOUR dmf.ntis.gov PASSWORD'
+    cattr_accessor :password
+  end
+end

data/lib/ssn_validator.rb

@@ -2,12 +2,15 @@ $:.unshift(File.dirname(__FILE__)) unless
   $:.include?(File.dirname(__FILE__)) || $:.include?(File.expand_path(File.dirname(__FILE__)))
   
 require 'ssn_validator/models/ssn_high_group_code'
+require 'ssn_validator/models/ssn_high_group_code_loader'
 require 'ssn_validator/models/ssn_validator'
+require 'ssn_validator/models/death_master_file'
+require 'ssn_validator/models/death_master_file_loader'
 require 'rake'
 
 # Load rake file
 import "#{File.dirname(__FILE__)}/tasks/ssn_validator.rake"
 
 module SsnValidator
-  VERSION = '0.1.2'
+  VERSION = '1.0.0'
 end

data/lib/tasks/ssn_validator.rake

@@ -2,7 +2,23 @@
 namespace :ssn_validator do
   desc "Loads the current file from http://www.socialsecurity.gov/employer/ssns/highgroup.txt if it hasn't already been loaded."
   task :update_data => :environment do
-    puts SsnHighGroupCode.load_current_high_group_codes_file
+    SsnHighGroupCode.load_current_high_group_codes_file
   end
-  
+
+  namespace :death_master_file do
+    desc "Loads a death master file.  Specify the path of the file: PATH=path.  Specify the date of the data: AS_OF=YYYY-MM-DD.  Optimized for Mysql databases."
+    task :load_file => :environment do
+      if ENV["PATH"] && ENV["AS_OF"]
+        DeathMasterFileLoader.new(ENV["PATH"],ENV["AS_OF"]).load_file
+      else
+        puts "You must specify the PATH and AS_OF variables:  rake ssn_validator:death_master_file:load_file PATH='path/to/file' AS_OF='2009-03-01'"
+      end
+    end
+
+    desc "Determines the most recent file that has been loaded, and loads all subsequent files in order from the dmf.ntis.gov website.  Optimized for Mysql databases."
+    task :update_data => :environment do
+      DeathMasterFileLoader.load_update_files_from_web
+    end
+  end
+
 end

data/test/files/test_dmf_initial_load.txt

@@ -0,0 +1,5 @@
+ 772781978NARANJO                 ANTHONY        GABRIEL        P030320090130200910
+ 772783025MANCO ESPICHAN          JUAN                          P030220091101191010
+ 772786197OWENS                   ISAIAH                        P0202200901102009FO
+ 772789442ASTACIO                 JUANJOSE       TOMAS          P021320090129200910
+ 772789569BROWN                   ALANNAH        SYEISSIE NICOL P030720090125200910

data/test/files/test_dmf_update_load.txt

@@ -0,0 +1,5 @@
+D772781978NARANJO                 ANTHONY        GABRIEL        P030320090130200910
+A772783025REPLACED                JUAN                          P030220091101191010
+A772786198NEW                     ISAIAH                        P0202200901102009FO
+D772789442CHANGED                 JUANJOSE       TOMAS          P021320090129200910
+C772789569CHANGED                 ALANNAH        SYEISSIE NICOL P030720090125200910

data/test/mocks/test/death_master_file_loader.rb

@@ -0,0 +1,35 @@
+require 'ssn_validator/models/death_master_file_loader'
+
+class DeathMasterFileLoader
+
+  def get_file_from_web
+    case @file_path_or_url
+    when /MA\d\d\d\d\d\d/ #these are the valid urls I want to mock a response to.
+      first_upload = Date.today.beginning_of_month - 2.months #based on the test, we know we are loading the last 3 months
+      if @file_path_or_url =~ /MA#{first_upload.strftime("%y%m%d")}/
+        return ['A772783123UPDATED                 JUAN                          P030220091101191010']
+      elsif @file_path_or_url =~ /MA#{(first_upload + 1.month).strftime("%y%m%d")}/
+        return ['A772783456UPDATED                 JUAN                          P030220091101191010']
+      elsif @file_path_or_url =~ /MA#{(first_upload + 2.months).strftime("%y%m%d")}/
+        return ['A772783789UPDATED                 JUAN                          P030220091101191010']
+      end
+    else
+      uri = URI.parse(@file_path_or_url)
+
+      request = Net::HTTP::Get.new(uri.request_uri)
+      request.basic_auth(SsnValidator::Ntis.user_name,SsnValidator::Ntis.password)
+
+      http = Net::HTTP.new(uri.host, uri.port)
+      http.use_ssl = (uri.port == 443)
+      http.verify_mode = OpenSSL::SSL::VERIFY_NONE
+      
+      response = http.request(request)
+      
+      raise(ArgumentError, "Invalid URL: #{@file_path_or_url}") if response.kind_of?(Net::HTTPNotFound)
+      raise(ArgumentError, "Authorization Required: Invalid username or password.  Set the variables SsnValidator::Ntis.user_name and SsnValidator::Ntis.password in your environment.rb file.") if response.kind_of?(Net::HTTPUnauthorized)
+
+      return response.body
+    end
+  end
+
+end

data/test/test_death_master_file_loader.rb

@@ -0,0 +1,85 @@
+require File.dirname(__FILE__) + '/test_helper.rb'
+
+class TestDeathMasterFileLoader < Test::Unit::TestCase
+
+  def setup
+    setup_death_master_file_table
+  end
+
+  def test_validations_for_load_file
+    exception = assert_raise ArgumentError do
+      DeathMasterFileLoader.new(nil,nil)
+    end
+    assert_equal "path_or_url not specified", exception.message
+
+    exception = assert_raise ArgumentError do
+      DeathMasterFileLoader.new('some/path.txt',nil)
+    end
+    assert_equal "as_of not specified", exception.message
+
+    exception = assert_raise Errno::ENOENT do
+      DeathMasterFileLoader.new('some/path.txt','2009-01-01')
+    end
+    assert_equal "No such file or directory - some/path.txt", exception.message
+
+    exception = assert_raise URI::InvalidURIError do
+      DeathMasterFileLoader.new('https://dmf.bad_url.gov/dmldata/monthly/bad_url','2009-01-01')
+    end
+    assert_equal "the scheme https does not accept registry part: dmf.bad_url.gov (or bad hostname?)", exception.message
+
+    exception = assert_raise ArgumentError do
+      DeathMasterFileLoader.new('https://dmf.ntis.gov/bad_url','2009-01-01')
+    end
+    assert_equal "Invalid URL: https://dmf.ntis.gov/bad_url", exception.message
+
+    exception = assert_raise ArgumentError do
+      DeathMasterFileLoader.new('https://dmf.ntis.gov/dmldata/monthly/bad_url','2009-01-01')
+    end
+    assert_equal "Authorization Required: Invalid username or password.  Set the variables SsnValidator::Ntis.user_name and SsnValidator::Ntis.password in your environment.rb file.", exception.message
+  end
+
+  def test_should_load_table_from_file
+    assert_equal(0,DeathMasterFile.count)
+    DeathMasterFileLoader.new(File.dirname(__FILE__) + '/files/test_dmf_initial_load.txt','2009-01-01').load_file
+    assert DeathMasterFile.count == 5
+
+    #load update file
+    #update file adds 1 record, deletes 2 records, and changes 2 records
+    DeathMasterFileLoader.new(File.dirname(__FILE__) + '/files/test_dmf_update_load.txt','2009-02-01').load_file
+    assert DeathMasterFile.count == 4
+    assert 1, DeathMasterFile.count(:conditions => {:last_name => 'NEW'})
+    assert 2, DeathMasterFile.count(:conditions => {:last_name => 'CHANGED'})
+  end
+
+  def test_should_not_load_table_if_as_of_already_in_table_for_load_file
+    DeathMasterFileLoader.new(File.dirname(__FILE__) + '/files/test_dmf_initial_load.txt','2009-01-01').load_file
+    assert DeathMasterFile.count > 0
+    record_count = DeathMasterFile.count
+    
+    exception = assert_raise ArgumentError do
+      DeathMasterFileLoader.new(File.dirname(__FILE__) + '/files/test_dmf_update_load.txt','2008-01-01').load_file
+    end
+    assert_equal "A more recent file has already been processed.  DB as_of date 2009-01-01", exception.message
+
+    assert_equal(record_count, DeathMasterFile.count)
+  end
+
+  def test_should_load_all_unloaded_updates
+    #initial load was 3 months ago
+    initial_run = Date.today.beginning_of_month - 3.months
+    DeathMasterFileLoader.new(File.dirname(__FILE__) + '/files/test_dmf_initial_load.txt',initial_run.strftime('%Y-%m-%d')).load_file
+    assert DeathMasterFile.count == 5
+
+    #this will load the last 3 months of files
+    #this has been mocked, not really hitting the web site for this
+    DeathMasterFileLoader.load_update_files_from_web
+    assert DeathMasterFile.count == 8
+    assert DeathMasterFile.find_by_as_of(initial_run.to_s(:db))
+    assert DeathMasterFile.find_by_as_of((initial_run += 1.month).to_s(:db))
+    assert DeathMasterFile.find_by_as_of((initial_run += 1.month).to_s(:db))
+    assert DeathMasterFile.find_by_as_of((initial_run += 1.month).to_s(:db))
+
+
+  end
+
+end

data/test/test_helper.rb

@@ -1,9 +1,23 @@
 require 'stringio'
 require 'test/unit'
+require 'mocks/test/death_master_file_loader'
 require File.dirname(__FILE__) + '/../lib/ssn_validator'
 
 ActiveRecord::Base.establish_connection :adapter => 'sqlite3', :database => ':memory:'
 
+
+def setup_high_group_codes_table
+	ActiveRecord::Base.connection.tables.each { |table| ActiveRecord::Base.connection.drop_table(table) }
+  create_ssn_high_group_codes_table
+end
+
+def setup_death_master_file_table
+	ActiveRecord::Base.connection.tables.each { |table| ActiveRecord::Base.connection.drop_table(table) }
+  create_death_master_file_table
+end
+
+private
+
 def create_ssn_high_group_codes_table
   silence_stream(STDOUT) do
     ActiveRecord::Schema.define(:version => 1) do
@@ -20,7 +34,26 @@ def create_ssn_high_group_codes_table
   end
 end
 
-def setup_high_group_codes_table
-	ActiveRecord::Base.connection.tables.each { |table| ActiveRecord::Base.connection.drop_table(table) }
-  create_ssn_high_group_codes_table
+def create_death_master_file_table
+  silence_stream(STDOUT) do
+    ActiveRecord::Schema.define(:version => 1) do
+      create_table :death_master_files do |t|
+        t.string :social_security_number
+        t.string  :last_name
+        t.string  :name_suffix
+        t.string :first_name
+        t.string :middle_name
+        t.string :verify_proof_code
+        t.date  :date_of_death
+        t.date  :date_of_birth
+        t.string  :state_of_residence
+        t.string :last_known_zip_residence
+        t.string :last_known_zip_payment
+        t.timestamps
+        t.date :as_of
+      end
+      add_index :death_master_files, :social_security_number, :unique => true
+      add_index :death_master_files, :as_of
+    end
+  end
 end

data/test/{test_ssn_high_group_code.rb → test_ssn_high_group_code_loader.rb}

@@ -1,6 +1,6 @@
 require File.dirname(__FILE__) + '/test_helper.rb'
 
-class TestSsnHighGroupCode < Test::Unit::TestCase
+class TestSsnHighGroupCodeLoader < Test::Unit::TestCase
 
   def setup
     setup_high_group_codes_table
@@ -8,15 +8,15 @@ class TestSsnHighGroupCode < Test::Unit::TestCase
   
   def test_should_load_table_with_current_file
     assert_equal(0,SsnHighGroupCode.count)
-    SsnHighGroupCode.load_current_high_group_codes_file
+    SsnHighGroupCodeLoader.load_current_high_group_codes_file
     assert SsnHighGroupCode.count > 0
   end
 
   def test_should_not_load_table_if_as_of_already_in_table
-    SsnHighGroupCode.load_current_high_group_codes_file
+    SsnHighGroupCodeLoader.load_current_high_group_codes_file
     assert SsnHighGroupCode.count > 0
     record_count = SsnHighGroupCode.count
-    SsnHighGroupCode.load_current_high_group_codes_file
+    SsnHighGroupCodeLoader.load_current_high_group_codes_file
     assert_equal(record_count, SsnHighGroupCode.count)
   end
 end

data/test/test_ssn_validator.rb

@@ -8,13 +8,14 @@ class TestSsnValidator < Test::Unit::TestCase
   NONDIGIT_SSNS = %w(078051a20 078F51120 78051#20 051,20 078051m20)
   INVALID_ZEROS_SSNS = %w(166-00-1234 073-96-0000)
   GROUPS_NOT_ASSIGNED_TO_AREA_SSNS = %w(752991234 755971234 762991254)
+  AREA_NOT_ASSIGNED_SSNS = %w(666991234)
 
   VALID_SSNS = %w(001021234 161-84-9876 223981111)
   VALID_INTEGER_SSNS = [115941234, 161849876, 223981111]
   
   def setup
     setup_high_group_codes_table
-    SsnHighGroupCode.load_current_high_group_codes_file
+    SsnHighGroupCodeLoader.load_current_high_group_codes_file
   end
   
   def test_ssn_validations
@@ -48,6 +49,12 @@ class TestSsnValidator < Test::Unit::TestCase
       assert validator.errors.include?('Invalid group or serial number.'), "Errors: #{validator.errors}"
     end
 
+    AREA_NOT_ASSIGNED_SSNS.each do |ssn|
+      validator = SsnValidator::Ssn.new(ssn)
+      assert !validator.valid?
+      assert validator.errors.include?("Area '#{validator.area}' has not been assigned."), "Errors: #{validator.errors}"
+    end
+
     GROUPS_NOT_ASSIGNED_TO_AREA_SSNS.each do |ssn|
       validator = SsnValidator::Ssn.new(ssn)
       assert !validator.valid?
@@ -62,4 +69,15 @@ class TestSsnValidator < Test::Unit::TestCase
 
   end
 
+  def test_death_master_file_hit
+    create_death_master_file_table
+    DeathMasterFileLoader.new(File.dirname(__FILE__) + '/files/test_dmf_initial_load.txt','2009-01-01').load_file
+    
+    validator = SsnValidator::Ssn.new('772781978')#ssn from file
+    assert validator.death_master_file_hit?
+
+    validator = SsnValidator::Ssn.new('666781978')#ssn not in file
+    assert !validator.death_master_file_hit?
+  end
+
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification 
 name: kevintyll-ssn_validator
 version: !ruby/object:Gem::Version 
-  version: 0.1.2
+  version: 1.0.0
 platform: ruby
 authors: 
 - Kevin Tyll
@@ -11,18 +11,9 @@ cert_chain: []
 
 date: 2009-04-13 00:00:00 -07:00
 default_executable: 
-dependencies: 
-- !ruby/object:Gem::Dependency 
-  name: activerecord
-  type: :runtime
-  version_requirement: 
-  version_requirements: !ruby/object:Gem::Requirement 
-    requirements: 
-    - - ">"
-      - !ruby/object:Gem::Version 
-        version: 2.0.0
-    version: 
-description: 
+dependencies: []
+
+description: Validates whether an SSN has likely been issued or not.
 email: kevintyll@gmail.com
 executables: []
 
@@ -38,8 +29,14 @@ files:
 - Rakefile
 - generators/ssn_validator_migration/templates/migration.rb
 - generators/ssn_validator_migration/ssn_validator_migration_generator.rb
+- generators/death_master_file_migration/templates/migration.rb
+- generators/death_master_file_migration/death_master_file_migration_generator.rb
+- lib/ssn_validator/ntis.rb
 - lib/ssn_validator/models/ssn_high_group_code.rb
+- lib/ssn_validator/models/ssn_high_group_code_loader.rb
 - lib/ssn_validator/models/ssn_validator.rb
+- lib/ssn_validator/models/death_master_file.rb
+- lib/ssn_validator/models/death_master_file_loader.rb
 - lib/ssn_validator.rb
 - lib/tasks/ssn_validator.rake
 - script/console
@@ -47,10 +44,14 @@ files:
 - script/generate
 - test/test_helper.rb
 - test/test_ssn_validator.rb
-- test/test_ssn_high_group_code.rb
+- test/test_ssn_high_group_code_loader.rb
+- test/test_death_master_file_loader.rb
+- test/mocks/test/death_master_file_loader.rb
+- test/files/test_dmf_initial_load.txt
+- test/files/test_dmf_update_load.txt
 has_rdoc: true
-homepage: 
-post_install_message: 
+homepage: http://kevintyll.git.com/ssn_validator
+post_install_message: Read the PostInstall.txt file for instructions on how to get the necessary tables created and data loaded.
 rdoc_options: []
 
 require_paths: 
@@ -77,4 +78,8 @@ summary: Validates whether an SSN has likely been issued or not.
 test_files: 
 - test/test_helper.rb
 - test/test_ssn_validator.rb
-- test/test_ssn_high_group_code.rb
+- test/test_ssn_high_group_code_loader.rb
+- test/test_death_master_file_loader.rb
+- test/mocks/test/death_master_file_loader.rb
+- test/files/test_dmf_initial_load.txt
+- test/files/test_dmf_update_load.txt