ssn_validator 1.0.4

data/.gitignore

@@ -0,0 +1,4 @@
+*.DS_Store
+*.svn
+pkg
+nbproject

data/History.txt

@@ -0,0 +1,41 @@
+== 1.0.4 2010-02-05
+
+* 1 bug fixes:
+  * The ssa website changed it's process for the most recent month.  Modified the rake task
+    to determine the most recent month loaded and bring it up to date.
+
+== 1.0.3 2009-04-24
+
+* 2 minor enhancement:
+  * Added method #death_master_file_record to SsnValidator::Ssn to return the actual dmf record if the SSN exists
+  * Changed the method #death_master_file_hit? to simply return true or false.
+
+== 1.0.2 2009-04-24
+
+* 1 minor enhancement:
+  * Using jeweler gem to create gemspec and rdoc files
+
+== 1.0.1 2009-04-23
+
+* 2 bug fixes:
+  * Fixed dmf mysql load error when single quote is in a data file record.
+  * Fixed ssn_validator:update_data rake task that broke on the last build.
+
+== 1.0.0 2009-04-22
+
+* 1 major enhancement:
+  * Added the death master file to determine if the ssn belongs to the deceased.
+
+* 1 bug fix:
+  * Fixed bug where an error was thrown if the ssn area has not been assigned yet.
+
+== 0.1.2 2009-04-15
+
+* 2 minor enhancements:
+  * Added rdoc files
+  * Updated documentation
+
+== 0.1.0 2009-04-14
+
+* 1 major enhancement:
+  * Initial release

data/LICENSE

@@ -0,0 +1,20 @@
+Copyright (c) 2009 Kevin Tyll
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/PostInstall.txt

@@ -0,0 +1,29 @@
+
+For more information on ssn_validator, see http://kevintyll.github.com/ssn_validator/
+
+* To create the necessary db migration, from the command line, run:
+    script/generate ssn_validator_migration
+* Require the gem in your environment.rb file in the Rails::Initializer block:
+    config.gem 'kevintyll-ssn_validator', :lib => 'ssn_validator'
+* To load your table with the current SSN data, from the command line, run:
+    rake ssn_validator:update_data
+    * The SSN data is updated monthly, so you'll want to run this rake task monthly to keep your validations accurate.
+* If you've purchased the death master file data, create the death_master_files migration:
+    script/generate death_master_file_migration
+* To load the dmf files you receive from ntis:
+    rake ssn_validator:death_master_file:load_file PATH='path/to/file' AS_OF='2009-03-01'
+    * You'll need to pass in the full path to where the file is on disk.  You'll also need
+      to pass in the date this file's data are as of in the format yyyy-mm-dd.
+    * This task must be used to load the initial file you receive from ntis on CD.  It can optionally be used
+      to load the monthly update files you download from the website.  If you manually download the update files,
+      you do not need to add your user name and password to the environment.rb file.  For a more automated approach
+      to loading the update files, add your user name and password to the environment.rb file and use the 2nd rake task.
+* To load the monthly updates from the ntis website:
+    * Add your user_name and password to the environment.rb file
+        SsnValidator::Ntis.user_name = 'REPLACE WITH YOUR dmf.ntis.gov USER NAME'
+        SsnValidator::Ntis.password = 'REPLACE WITH YOUR dmf.ntis.gov PASSWORD'
+    * Run the rake task:
+        rake ssn_validator:death_master_file:update_data
+        * This rake task will determine the most recent file that has been loaded, and loads all subsequent files in order
+        from the dmf.ntis.gov website.
+        * The death master file data is updated monthly, so you'll want to run this rake task monthly to keep your validations accurate.

data/README.rdoc

@@ -0,0 +1,98 @@
+= ssn_validator
+
+* http://kevintyll.github.com/ssn_validator
+* http://www.drexel-labs.com
+
+== DESCRIPTION:
+
+ssn_validator is a ruby gem that validates whether an SSN has likely been issued or not.
+What exactly does that mean "likely" been issued?  We can't tell for sure if an SSN belongs
+to a particular person, but knowing the "rules"[http://www.socialsecurity.gov/employer/ssnweb.htm]
+on how the social security administration assigns numbers, we can determine whether a number
+has ever been issued or not.
+
+ssn_validator started as a need for the company I work for, Clarity Services Inc.  Incredibly, we
+couldn't find an existing gem or service that provided what we needed.  Since we were going to have
+to roll our own solution, we decided to create a gem out of it and share it with the community.  Much
+thanks goes to the management at Clarity Services Inc. for allowing this code to be open sourced.
+
+
+== FEATURES/PROBLEMS:
+
+* What it can do:
+    Validates the likelyhood that an SSN has been issued to someone.
+    Checks the Death Master File if the SSN belongs to a dead person.  This will require you to purchase the dmf data from https://dmf.ntis.gov
+
+* What it cannot do:
+    Validate that an SSN actually belongs to a particular person.
+
+* What it's planned to do:
+    Determine when an SSN was issued...if i can find the historical data.  This can be used to further validate an SSN by comparing it to a Date of Birth.
+
+== SYNOPSIS:
+
+* Just instantiate the object with an SSN.
+    ssn = SsnValidator::Ssn.new('123-45-6789')
+
+* Then check if it's valid
+    ssn.valid?
+    ssn.death_master_file_hit?
+
+    ssn.valid? only checks to see if the number itself is valid.  Check the death_master_file_hit? method
+    as well to verify the ssn does not belong the the deceased.
+
+    The death_master_file_hit? method will only return true of false if you have purchased the dmf data and
+    loaded it into your database.  There is a generator and rake task to facilitate this, but you'll have
+    to purchase the data yourself at https://dmf.ntis.gov.
+
+* You can check the errors array to see why it's not valid.
+    ssn.errors
+
+== REQUIREMENTS:
+
+* Rails 2.0.0 or greater
+
+== INSTALL:
+
+* To install the gem:
+    sudo gem install kevintyll-ssn_validator
+* To create the necessary db migration, from the command line, run:
+    script/generate ssn_validator_migration
+* Require the gem in your environment.rb file in the Rails::Initializer block:
+    config.gem 'kevintyll-ssn_validator', :lib => 'ssn_validator'
+* To load your table with the current SSN data, from the command line, run:
+    rake ssn_validator:update_data
+
+    * The SSN data is updated monthly, so you'll want to run this rake task monthly to keep your validations accurate.
+
+* If you've purchased the death master file data, create the death_master_files migration:
+    script/generate death_master_file_migration
+* To load the dmf files you receive from ntis:
+    rake ssn_validator:death_master_file:load_file PATH='path/to/file' AS_OF='2009-03-01'
+
+    * You'll need to pass in the full path to where the file is on disk.  You'll also need
+      to pass in the date this file's data are as of in the format yyyy-mm-dd.
+
+    * This task must be used to load the initial file you receive from ntis on CD.  It can optionally be used
+      to load the monthly update files you download from the website.  If you manually download the update files,
+      you do not need to add your user name and password to the environment.rb file.  For a more automated approach
+      to loading the update files, add your user name and password to the environment.rb file and use the 2nd rake task.
+
+* To load the monthly updates from the ntis website:
+    * Add your user_name and password to the environment.rb file
+
+        SsnValidator::Ntis.user_name = 'REPLACE WITH YOUR dmf.ntis.gov USER NAME'
+        SsnValidator::Ntis.password = 'REPLACE WITH YOUR dmf.ntis.gov PASSWORD'
+
+    * Run the rake task:
+
+        rake ssn_validator:death_master_file:update_data
+
+        * This rake task will determine the most recent file that has been loaded, and loads all subsequent files in order
+        from the dmf.ntis.gov website.
+
+        * The death master file data is updated monthly, so you'll want to run this rake task monthly to keep your validations accurate.
+
+== Copyright
+
+Copyright (c) 2009 Kevin Tyll. See LICENSE for details.

data/Rakefile

@@ -0,0 +1,60 @@
+require 'rubygems'
+require 'rake'
+
+begin
+  require 'jeweler'
+  Jeweler::Tasks.new do |gemspec|
+    gemspec.name = "ssn_validator"
+    gemspec.author = "Kevin Tyll"
+    gemspec.email = "kevintyll@gmail.com"
+    gemspec.homepage = %q{http://kevintyll.git.com/ssn_validator}
+    gemspec.summary = "Validates whether an SSN has likely been issued or not."
+    gemspec.description = "Validates whether an SSN has likely been issued or not."
+    gemspec.post_install_message = File.readlines("PostInstall.txt").join("")
+  end
+
+  Jeweler::GemcutterTasks.new
+  
+rescue LoadError
+  puts "Jeweler not available. Install it with: sudo gem install technicalpickles-jeweler -s http://gems.github.com"
+end
+
+require 'rake/testtask'
+Rake::TestTask.new(:test) do |test|
+  test.libs  << 'test'
+  test.pattern = 'test/test_*.rb'
+  test.verbose = true
+end
+
+begin
+  require 'rcov/rcovtask'
+  Rcov::RcovTask.new do |test|
+    test.libs << 'test'
+    test.pattern = 'test/**/*_test.rb'
+    test.verbose = true
+  end
+rescue LoadError
+  task :rcov do
+    abort "RCov is not available. In order to run rcov, you must: sudo gem install spicycode-rcov"
+  end
+end
+
+
+task :default => :test
+
+require 'rake/rdoctask'
+Rake::RDocTask.new do |rdoc|
+  if File.exist?('VERSION.yml')
+    config = YAML.load(File.read('VERSION.yml'))
+    version = "#{config[:major]}.#{config[:minor]}.#{config[:patch]}"
+  else
+    version = ""
+  end
+
+  rdoc.rdoc_dir = 'rdoc'
+  rdoc.title = "ssn_validator #{version}"
+  rdoc.rdoc_files.include('README*')
+  rdoc.rdoc_files.include('LICENSE*')
+  rdoc.rdoc_files.include('lib/**/*.rb')
+end
+

data/VERSION.yml

@@ -0,0 +1,5 @@
+--- 
+:major: 1
+:minor: 0
+:patch: 4
+:build: 

data/generators/death_master_file_migration/death_master_file_migration_generator.rb

@@ -0,0 +1,12 @@
+class DeathMasterFileMigrationGenerator < Rails::Generator::Base
+  def manifest 
+    record do |m|
+      #m.directory File.join('db')
+      m.migration_template 'migration.rb', 'db/migrate' 
+    end 
+  end
+  
+  def file_name
+    "create_death_master_file_table"
+  end
+end

data/generators/death_master_file_migration/templates/migration.rb

@@ -0,0 +1,26 @@
+class CreateDeathMasterFileTable < ActiveRecord::Migration
+  
+  def self.up
+    create_table :death_master_files do |t|
+      t.string :social_security_number
+      t.string  :last_name
+      t.string  :name_suffix
+      t.string :first_name
+      t.string :middle_name
+      t.string :verify_proof_code
+      t.date  :date_of_death
+      t.date  :date_of_birth
+      t.string  :state_of_residence
+      t.string :last_known_zip_residence
+      t.string :last_known_zip_payment
+      t.time_stamps
+      t.date :as_of
+    end
+    add_index :death_master_files, :social_security_number, :unique => true
+    add_index :death_master_files, :as_of
+  end
+
+  def self.down
+    drop_table :death_master_files
+  end
+end

data/generators/ssn_validator_migration/ssn_validator_migration_generator.rb

@@ -0,0 +1,12 @@
+class SsnValidatorMigrationGenerator < Rails::Generator::Base
+  def manifest 
+    record do |m|
+      #m.directory File.join('db')
+      m.migration_template 'migration.rb', 'db/migrate' 
+    end 
+  end
+  
+  def file_name
+    "create_ssn_validator_tables"
+  end
+end

data/generators/ssn_validator_migration/templates/migration.rb

@@ -0,0 +1,18 @@
+class CreateSsnValidatorTables < ActiveRecord::Migration
+  
+  def self.up
+    create_table :ssn_high_group_codes do |t|
+      t.date      :as_of
+      t.string    :area
+      t.string    :group
+      t.timestamps
+    end
+    
+    add_index :ssn_high_group_codes, [:area]
+    add_index :ssn_high_group_codes, [:area, :as_of]
+  end
+  
+  def self.down
+    drop_table :ssn_high_group_codes
+  end
+end

data/lib/ssn_validator.rb

@@ -0,0 +1,12 @@
+$:.unshift(File.dirname(__FILE__)) unless
+  $:.include?(File.dirname(__FILE__)) || $:.include?(File.expand_path(File.dirname(__FILE__)))
+  
+require 'ssn_validator/models/ssn_high_group_code'
+require 'ssn_validator/models/ssn_high_group_code_loader'
+require 'ssn_validator/models/ssn_validator'
+require 'ssn_validator/models/death_master_file'
+require 'ssn_validator/models/death_master_file_loader'
+require 'rake'
+
+# Load rake file
+import "#{File.dirname(__FILE__)}/tasks/ssn_validator.rake"

data/lib/ssn_validator/models/death_master_file.rb

@@ -0,0 +1,5 @@
+require 'activerecord'
+
+class DeathMasterFile < ActiveRecord::Base
+ 
+end

data/lib/ssn_validator/models/death_master_file_loader.rb

@@ -0,0 +1,235 @@
+require 'net/http'
+require 'net/https'
+require 'activerecord'
+require 'active_record/connection_adapters/mysql_adapter'
+require 'ssn_validator/ntis'
+
+class DeathMasterFileLoader
+
+  # path_or_url is the full path to the file to load on disk, or the url of an update file.
+  # as_of is a string in the formatt YYYY-MM-DD for which the file data is accurate.
+  def initialize(path_or_url,file_as_of)
+    @file_path_or_url = path_or_url
+    @file_as_of = file_as_of
+    valid?
+  end
+
+  def valid?
+    raise(ArgumentError, "path_or_url not specified") unless @file_path_or_url
+    raise(ArgumentError, "as_of not specified") unless @file_as_of
+    max_as_of = DeathMasterFile.maximum(:as_of)
+    raise(ArgumentError, "A more recent file has already been processed.  DB as_of date #{max_as_of}") if  max_as_of && (max_as_of >= @file_as_of.to_date)
+
+    if File.exists?(@file_path_or_url)
+      @download_file = File.open(@file_path_or_url)
+    elsif URI.parse(@file_path_or_url).kind_of?(URI::HTTP)
+      @download_file = get_file_from_web
+    else
+      raise(Errno::ENOENT, @file_path_or_url)
+    end
+  end
+
+  def load_file
+
+    if DeathMasterFile.connection.kind_of?(ActiveRecord::ConnectionAdapters::MysqlAdapter)
+      puts "Converting file to csv format for Mysql import.  This could take several minutes."
+
+      csv_file = convert_file_to_csv
+
+      bulk_mysql_update(csv_file)
+    else
+      active_record_file_load
+    end
+
+  end
+
+  def get_file_from_web
+    uri = URI.parse(@file_path_or_url)
+
+    request = Net::HTTP::Get.new(uri.request_uri)
+    request.basic_auth(SsnValidator::Ntis.user_name,SsnValidator::Ntis.password)
+
+    http = Net::HTTP.new(uri.host, uri.port)
+    http.use_ssl = (uri.port == 443)
+    http.verify_mode = OpenSSL::SSL::VERIFY_NONE
+
+    response = http.request(request)
+
+    raise(ArgumentError, "Invalid URL: #{@file_path_or_url}") if response.kind_of?(Net::HTTPNotFound)
+    raise(ArgumentError, "Authorization Required: Invalid username or password.  Set the variables SsnValidator::Ntis.user_name and SsnValidator::Ntis.password in your environment.rb file.") if response.kind_of?(Net::HTTPUnauthorized)
+
+    return response.body
+  end
+
+  #Loads all the update files from dmf.ntis.gov.
+  #It starts with the last file loaded, and loads each
+  #missing file in sequence up to the current file.
+  def self.load_update_files_from_web
+    max_as_of = DeathMasterFile.maximum(:as_of)
+    run_file_date = max_as_of.beginning_of_month.next_month
+    last_file_date = Date.today.beginning_of_month
+    while run_file_date <= last_file_date
+      url = "https://dmf.ntis.gov/dmldata/monthly/MA#{run_file_date.strftime("%y%m%d")}"
+      puts "Loading file #{url}"
+      DeathMasterFileLoader.new(url,run_file_date.strftime("%Y-%m-%d")).load_file
+      run_file_date += 1.month
+    end
+  end
+
+  private
+
+  # Processes 28 million rows in 23 minutes. Input file 2.6GB output: 2.9GB.
+  # Used to convert a packed fixed-length file into csv for mysql import.
+  def convert_file_to_csv
+
+    csv_file = Tempfile.new("dmf")    # create temp file for converted csv formmat.
+
+
+    start = Time.now
+    timenow = start.to_s(:db)
+
+    @delete_ssns = []
+
+    @download_file.each_with_index do |line,i|
+      action = record_action(line)
+      attributes_hash = text_to_hash(line)
+      if  action == 'D'
+        #keep track of all the records to delete.  We'll delete at the end all at once.
+        @delete_ssns << attributes_hash[:social_security_number]
+      else
+        # empty field for id to be generated by mysql.
+        newline = "``," +
+          # social_security_number
+        "`#{attributes_hash[:social_security_number]}`," +
+          # last_name
+        "`#{attributes_hash[:last_name]}`," +
+          # name_suffix
+        "`#{attributes_hash[:name_suffix]}`," +
+          # first_name
+        "`#{attributes_hash[:first_name]}`," +
+          # middle_name
+        "`#{attributes_hash[:middle_name]}`," +
+          # verify_proof_code
+        "`#{attributes_hash[:verify_proof_code]}`," +
+          # date_of_death - need YYYY-MM-DD.
+        "`#{attributes_hash[:date_of_death]}`," +
+          # date_of_birth - need YYYY-MM-DD.
+        "`#{attributes_hash[:date_of_birth]}`," +
+          # state_of_residence - must be code between 01 and 65 or else nil.
+        "`#{attributes_hash[:state_of_residence]}`," +
+          # last_known_zip_residence
+        "`#{attributes_hash[:last_known_zip_residence]}`," +
+          # last_known_zip_payment
+        "`#{attributes_hash[:last_known_zip_payment]}`," +
+          # created_at
+        "`#{timenow}`," +
+          # updated_at
+        "`#{timenow}`," +
+          # as_of
+        "`#{attributes_hash[:as_of]}`" +"\n"
+
+        csv_file.syswrite newline
+        puts "#{i} records processed." if (i % 25000 == 0) && (i > 0)
+      end
+    end
+    puts "File conversion ran for #{(Time.now - start) / 60} minutes."
+    return csv_file
+  end
+
+  #Uses active record to load the data.
+  #The benefit is it will work on any database.
+  #The downside is it's really slow.
+  def active_record_file_load
+    puts 'Importing file into database. This could take many minutes.'
+
+    @download_file.each_with_index do |line,i|
+      action = record_action(line)
+      attributes_hash = text_to_hash(line)
+      if  action == 'D'
+        DeathMasterFile.destroy_all(['social_security_number = ?', attributes_hash[:social_security_number]])
+      else
+
+        # empty field for id to be generated by mysql.
+#        record_hash = {
+#          :as_of => @file_as_of.to_date.to_s(:db),
+#          :social_security_number => parse_record(line,:social_security_number),
+#          :last_name => parse_record(line,:last_name),
+#          :name_suffix => parse_record(line,:name_suffix),
+#          :first_name => parse_record(line,:first_name),
+#          :middle_name => parse_record(line,:middle_name),
+#          :verify_proof_code => parse_record(line,:verify_proof_code),
+#          :date_of_death => parse_record(line,:date_of_death),
+#          :date_of_birth => parse_record(line,:date_of_birth),
+#          # - must be code between 01 and 65 or else nil.
+#          :state_of_residence=> parse_record(line,:state_of_residence=),
+#          :last_known_zip_residence => parse_record(line,:last_known_zip_residence),
+#          :last_known_zip_payment => parse_record(line,:last_known_zip_payment)
+#        }
+
+        case action
+        when '',nil,' '
+          #the initial file leaves this field blank
+          DeathMasterFile.create(attributes_hash)
+        else
+          dmf = DeathMasterFile.find_by_social_security_number(attributes_hash[:social_security_number])
+          if dmf
+            #a record already exists, update this record
+            dmf.update_attributes(attributes_hash)
+          else
+            #create a new record
+            DeathMasterFile.create(attributes_hash)
+          end
+        end
+      end
+
+      puts "#{i} records processed." if (i % 2500 == 0) && (i > 0)
+
+    end
+
+    puts "Import complete."
+  end
+
+  # For mysql, use:
+  # LOAD DATA LOCAL INFILE 'ssdm1.csv' INTO TABLE death_master_files FIELDS TERMINATED BY ',' ENCLOSED BY "'" LINES TERMINATED BY '\n';
+  # This is a much faster way of loading large amounts of data into mysql.  For information on the LOAD DATA command
+  # see http://dev.mysql.com/doc/refman/5.1/en/load-data.html
+  def bulk_mysql_update(csv_file)
+    puts "Importing into Mysql..."
+
+    #delete all the 'D' records
+    DeathMasterFile.delete_all(:social_security_number => @delete_ssns)
+
+    #This will insert new records, and replace records with existing ssns.
+    #This only works because there is a unique index on social_security_number.
+    mysql_command = <<-TEXT
+    LOAD DATA LOCAL INFILE '#{csv_file.path}' REPLACE INTO TABLE death_master_files FIELDS TERMINATED BY ',' ENCLOSED BY "`" LINES TERMINATED BY '\n';
+    TEXT
+
+    DeathMasterFile.connection.execute(mysql_command)
+    puts "Mysql import complete."
+
+  end
+
+  def record_action(line)
+    line[0,1].to_s.strip
+  end
+
+  def text_to_hash(line)
+
+    {:as_of => @file_as_of.to_date.to_s(:db),
+     :social_security_number => line[1,9].to_s.strip,
+     :last_name => line[10,20].to_s.strip,
+     :name_suffix => line[30,4].to_s.strip,
+     :first_name => line[34,15].to_s.strip,
+     :middle_name => line[49,15].to_s.strip,
+     :verify_proof_code => line[64,1].to_s.strip,
+     :date_of_death => (Date.strptime(line[65,8].to_s.strip,'%m%d%Y') rescue nil),
+     :date_of_birth => (Date.strptime(line[73,8].to_s.strip,'%m%d%Y') rescue nil),
+      # - must be code between 01 and 65 or else nil.
+     :state_of_residence => (line[81,2].to_s.strip.between?('01', '65') ? line[81,2].to_s.strip : nil),
+     :last_known_zip_residence => line[83,5].to_s.strip,
+     :last_known_zip_payment => line[88,5].to_s.strip}
+  rescue Exception => e
+    puts '@@@@@@@@@ Error = ' + e.message + ':   ' + line.inspect
+  end
+end