idata 0.1.33 → 0.2.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: efc9c2bc3a3ee9b79d83c531abbcbc185360dda4
-  data.tar.gz: 0fbca044baf3f5a454f5e55e3ff3b40fd8b480f8
+  metadata.gz: 26fd9bafa90c2af8eef61231c2b070c6a395bbcb
+  data.tar.gz: b1952edd955ea4eeb72d31f8c640f053d5515a17
 SHA512:
-  metadata.gz: 59e4d3862a80d105fcc6bfa7973dd18c766fa93a921f4f3c9784398ed71187d2f259d05d9375d15e5dc723bdfe9e037bac861dcd677bd7a70a06b4a058cc3152
-  data.tar.gz: a685200d759507941fc6800615cf3fc7191e055baecbf56bf4cc8c2f9bce929fc30cd18df76797d75b46a0c2ffa35dcb7cfc1c3a84800f455967cfaea1313693
+  metadata.gz: 79b8f98bad03bcfccea93cf2cb074271391def428288784059842435e4e6334928f1d5806c50707816a61621e38e4761cae761443c634bcf2ba0baa5187c3e61
+  data.tar.gz: 2513beafcacb28bc7d745e93ab6dd09e0f3b246c25bcee2d81b5bee92b66778b26feab7f43887790a284dfbc8e6906c3860d71082f530a78f1ff82e598085df7

data/README.md

@@ -1,5 +1,5 @@
 # Overview
-We provide some useful utilities for validating data in a PostgreSQL data table
+We provide some utilities for validating data in a PostgreSQL data table.
 These utilities can be used as simple terminal commands and can be installed by:
 
     gem install idata
@@ -13,7 +13,7 @@ idata comes along with the following commands:
 * imerge
 * isanitize
 
-Run a command with --help switch for the details
+Run a command with `--help` switch for the details
 
 Prequisites:
 * PostgreSQL 9.0 or above
@@ -23,34 +23,34 @@ Prequisites:
 # Usage
 Suppose we have an `items` table, and we want to validate its records against certain criteria like:
 
-* `item_id` must not be null
-* `item_title` must not be null
-* The composite `[item_id, item_title]` must be unique
-* One `item_id` corresponds to only ONE `item_title` (in other words, there must not be two items with different titles but with the same `item_id`)
+* `vendor_code` must not be null
+* `vendor_name` must not be null
+* The composite `[vendor_code, vendor_name]` must be unique
+* One `vendor_code` corresponds to only ONE `vendor_name` (in other words, there must not be two items with different `vendor_name` but with the same `vendor_code`)
 and vice-versa
 * `vendor_code` must reference the `code` column in the `vendors` table
 
 Then the validation command could be:
 ```
-      ivalidate --host=localhost --user=postgres --database=mydb --table=items --log-to=validation_errors \
-                --not-null="vendor_id" \
+      ivalidate --host=localhost --user=postgres --database=mydb --table=items
+                --log-to=validation_errors \
+                --not-null="vendor_code" \
                 --not-null="vendor_name" \
-                --unique="vendor_id,vendor_name" \
-                --consistent-by="vendor_id|vendor_name" \
-                --consistent-by="vendor_id|vendor_name" \
+                --unique="vendor_code,vendor_name" \
+                --consistent-by="vendor_code|vendor_name" \
+                --consistent-by="vendor_name|vendor_code" \
                 --cross-reference="vendor_code|vendors.code"
 ```
 Validation results for every single record are logged to an additional column named `validation_errors`
-of the `items` table, as specified by the `--log-to` switch
-
-As you can see, most common checks can be performed using the supported switches:
+of the `items` table, as specified by the `--log-to` switch. As you can see, most common checks can be performed using the supported switches:
 ```
     --not-null
     --unique
     --consistent-by
     --cross-reference
 ```
-For more generic check, we support some other switches:
+# Custom Validation
+For more customized checks, we support some other switches.
 
 The `--match="field/pattern/"` switch tells the program to check if value of a `field` matches the provided `pattern` (which is a regular expression).
 For example:
@@ -61,22 +61,20 @@ For example:
     # Check if value of status is either 'A' or 'I' (any other value is not allowed)
     ivalidate --match="status/^(A|I)$/"
 ```
-In case you need even more customized validation other than the supported ones (match, unique, not-null, cross-reference...)
-then `--query` switch may be helpful. For example:
+In case you need even more customized validation other than the supported ones (`match`, `unique`, `not-null`, `cross-reference`...)
+then the `--query` switch may come in handy. For example:
 ```
-    ivalidate --query="string_to_date(start_date) >= '01/02/2014' -- invalid date"
+    ivalidate --query="start_date >= string_to_date('01/02/2014') -- invalid date"
 ``` 
 You can also use `--rquery` which is the reversed counterpart of `--query`
-For example, the following two checks are equivalent:
+For example, the following two checks are equivalent, mark any record whose `start_date < '01/02/2014'` as "invalid date":
 ```
-    ivalidate --query="string_to_date(start_date) >= '01/02/2014' -- invalid date"
-    ivalidate --rquery="string_to_date(start_date) < '01/02/2014' -- invalid date"
+    ivalidate --query="start_date >= string_to_date('01/02/2014') -- invalid date"
+    ivalidate --rquery="start_date < string_to_date('01/02/2014') -- invalid date"
 ``` 
-(mark any record whose `start_date < '01/02/2014'` as "invalid date")
 
 Note: run `ivalidate --help` to see the full list of supported switches
 
-
 # Put it all together
 You can put several `ivalidate` commands (for several data tables) in one single bash/sh file.
 Besides `ivalidate`, we also support some other utilities to:
@@ -84,6 +82,6 @@ Besides `ivalidate`, we also support some other utilities to:
 + Modify data tables
 + Generate summary reports
 
-For a full example, see our `sample.sh`
+See our `sample.sh` for a comprehensive example
 
 

data/README2.md

@@ -0,0 +1,37 @@
+### Overview
+File dùng cho validation criteria có cấu trúc chung như sau:
+```yaml
+table:
+- field: field1, field2, etc.
+  validations:
+    - rule:
+      code:
+      error:
+      impact:
+      solution:
+```
+
+##### Explain:
++ Một `table` có một hoặc nhiều `field`
++ Một `field` có một section `validations` chứa một hoặc nhiều rules
++ Mỗi rule chứa `rule` (required) và `code`, `error`, `solution`, `impact`, `priority` tương ứng (optional)
++ Rule: viết theo format covered bên dưới
++ Các field khác free text trên cùng 1 line, dùng \n làm dấu cách dòng
++ Giá trị của field có thể là một hoặc tập gồm nhiều field name, separated by comma
+
+## Writing rules
+Supported rules include:
+
+| Rule | Description | Example |
+| ---- | ----------- | ------- |
+| `not null` | Giá trị của field tương ứng không được rỗng |  |
+| `unique` | Giá trị của field tương ứng phải unique trong table |  |
+| `matches "/regexp/"` | Giá trị của field phải thỏa format định nghĩa bởi `regexp` | |
+| `not matches "/regexp/"` | Reverse counterpart của `matches` | |
+| `consistent by "ref"` | Giá trị của field tương ứng phải consistent với `ref` | |
+| `cross references "table.field"` | Giá trị của field phải reference tới một field khác `table.field` | |
+| `custom query "query"` | Dùng custom SQL `query` (trong trường hợp business phức tạp không thể biểu diễn bằng các rule khác) | |
+| `reverse query "query"` | Reverse counterpart của `custom query` | |
+
+## Others
+TBD

data/bin/ivalidate2

@@ -0,0 +1,426 @@
+#!/usr/bin/env ruby
+# DATA VALIDATOR
+#
+# @author Nghi Pham
+# @date April 2014
+#
+# Data validation includes:
+# * Uniqueness
+# * Integrity (cross reference)
+# * Data type: numeric, text, enum, etc.
+# * Data format: text size, text values, enum, inclusion, exclusion, etc.
+#
+# Issue ruby load.rb --help for guideline/examples
+#
+require 'optparse'
+require 'active_record'
+require 'rubygems'
+require 'logger'
+
+$options = {}
+parser = OptionParser.new("", 24) do |opts|
+  opts.banner = "\nProgram: Data Validator\nAuthor: MCKI\n\n"
+  
+  opts.on("-c", "--config FILE", "Configuration file") do |v|
+    $options[:config] = v
+  end
+  
+  opts.on("-h", "--host HOST", "PostgreSQL host") do |v|
+    $options[:host] = v
+  end
+
+  opts.on("-d", "--database DATABASE", "PostgreSQL database") do |v|
+    $options[:database] = v
+  end
+
+  opts.on("-u", "--username USER", "PostgreSQL username") do |v|
+    $options[:username] = v
+  end
+
+  opts.on("-p", "--password PASSWORD", "PostgreSQL password") do |v|
+    $options[:password] = v
+  end
+
+  opts.on("-l", "--listen PORT", "PostgreSQL listen port (default to 5432)") do |v|
+    $options[:listen] = v
+  end
+
+  opts.on_tail('--help', 'Displays this help') do
+		puts opts, "", help
+    exit
+	end
+end
+
+def help
+  return ''
+end
+
+begin
+  parser.parse!
+rescue SystemExit => ex
+  exit
+end
+
+# Load parameters from ENVIRONMENT if exist
+$options[:host] ||= ENV['HOST']
+$options[:username] ||= ENV['USERNAME']
+$options[:password] ||= ENV['PASSWORD']
+$options[:listen] ||= ENV['LISTEN']
+$options[:database] ||= ENV['DATABASE']
+
+# validate parameters
+if $options[:config].nil?
+  puts "\nPlease specify config file: -c\n\n"
+  exit
+end
+
+if $options[:host].nil?
+  puts "\nPlease specify host name: -h\n\n"
+  exit
+end
+
+if $options[:database].nil?
+  puts "\nPlease specify PostgreSQL database name: -d\n\n"
+  exit
+end
+
+if $options[:username].nil?
+  puts "\nPlease specify PostgreSQL username: -u\n\n"
+  exit
+end
+
+# Default value
+$options[:listen] ||= 5432
+
+# Database dump
+ActiveRecord::Base.establish_connection(
+  'adapter' => 'postgresql',
+  'host' => $options[:host],
+  'database' => $options[:database],
+  'username' => $options[:username],
+  'password' => $options[:password],
+  'port' => $options[:listen],
+  'timeout' => 15000
+)
+
+class String
+  def not_null_sql
+    a = self.split(/\s*,\s*/)
+    sql = a.map{|s|
+      "#{s} IS NOT NULL AND length(trim(#{s}::text)) <> 0"
+    }.join(" AND ")
+    
+    "(#{sql})"
+  end
+  
+  def lower
+    a = self.split(/\s*,\s*/)
+    sql = a.map{|s|
+      "lower(#{s})"
+    }.join(",")
+    
+    sql
+  end
+
+  def black;          "\033[30m#{self}\033[0m" end
+  def red;            "\033[31m#{self}\033[0m" end
+  def green;          "\033[32m#{self}\033[0m" end
+  def brown;          "\033[33m#{self}\033[0m" end
+  def blue;           "\033[34m#{self}\033[0m" end
+  def magenta;        "\033[35m#{self}\033[0m" end
+  def cyan;           "\033[36m#{self}\033[0m" end
+  def gray;           "\033[37m#{self}\033[0m" end
+end
+
+class Logger
+  alias_method :_old_info, :info
+  alias_method :_old_warn, :warn
+  alias_method :_old_error, :error
+
+
+  def error(msg)
+    _old_error(msg.red)
+  end
+
+  def info(msg)
+    _old_info(msg.green)
+  end
+
+  def warn(msg)
+    _old_warn(msg.brown)
+  end
+end
+
+module IData
+  class Validator
+    SUPPORTED_RULES_REGEXP = /^\s*(not null|cross references|matches|not matches|custom query|reverse query|unique|consistent by)\s*/
+    DEFAULT_ERROR_FIELD = 'validation_errors'
+    META_TABLE = 'validation_meta'
+    SUMMARY_TABLE = 'summary'
+
+    def initialize(file)
+      @config = YAML.load_file(file)
+      @logger = Logger.new(STDOUT)
+      @rules = []
+      @logger.formatter = proc do |severity, datetime, progname, msg|
+        "#{severity}: #{datetime} - #{msg}\n"
+      end
+      @config.each do |table, fields|
+        unless table_exists?(table)
+          @logger.warn "Table #{table} does not exist!"
+        end
+        @logger.info "Validating table #{table}"
+        fields.each do |field|
+          field['validations'].each do |rule|
+            p rule
+            type, args = parse_rule(rule['rule'])
+            options = rule.merge('table' => table, 'field' => field['field'], 'type' => type, 'args' => args)
+            options['code'] = Digest::SHA1.hexdigest([table, field['field'], rule['rule']].join(""))
+            @rules << options
+          end
+        end
+      end
+    end
+
+    def validate!
+      # reset the meta table
+      execute("DROP TABLE IF EXISTS #{META_TABLE}")
+
+      # validate
+      @logger.info "Validation started!"
+      @rules.each {|r| validate(r) }
+
+      # create meta table
+      create_table_from_array(@rules, table_name: META_TABLE, drop_table: true, extra_fields: ['impact', 'solution', 'count', 'percentage'])
+
+      # Done!
+      @logger.info "Validation done!"
+    end
+
+    def create_table_from_array(entries, options = {})
+      raise "Please specify :table_name" unless options[:table_name]
+
+      extra_fields = options[:extra_fields] || []
+      columns = entries.inject([]) {|x, i| x += i.keys } + extra_fields
+      columns.uniq!
+
+      if options[:drop_table] 
+        execute "DROP TABLE IF EXISTS #{options[:table_name]}"
+      end
+
+      execute "CREATE TABLE IF NOT EXISTS #{options[:table_name]} ( #{columns.map{|c| quote_col_name(c.to_s) + ' VARCHAR' }.join(', ')} )"
+
+      insert_sql = entries.map { |r|
+        "INSERT INTO #{options[:table_name]}(#{columns.map{|c| quote_col_name(c.to_s)}.join(', ')}) VALUES(#{ columns.map{|c| quote(r[c])}.join(',') });"
+      }.join("")
+
+      execute insert_sql
+    end
+    
+    def validate(options)
+      unless table_exists?(options['table'])
+        return
+      end
+
+      add_error_field(options)
+      case options['type']
+      
+      when 'not null'
+        validate_not_null(options)
+      when 'custom query'
+        validate_custom_query(options)
+      when 'reverse query'
+        validate_reverse_query(options)
+      when 'custom query reversed'
+        validate_custom_query(options)
+      when 'matches'
+        validate_match(options)
+      when 'cross references'
+        validate_cross_reference(options)
+      when 'consistent by'
+        validate_consistent_by(options)
+      when 'unique'
+        validate_unique(options)
+      else
+        raise "Rule not recognized"
+      end
+    rescue Exception => ex
+      @logger.warn ex.message.split(/[\n]/).first.strip
+    end
+
+    def report!
+      sql = @rules.map {|r|
+        "(SELECT unnest(string_to_array(#{DEFAULT_ERROR_FIELD}, ' || ')) as code, count(*), round((count(*) * 100)::numeric / (SELECT count(*) FROM #{r['table']}), 2)::varchar || '%' AS percentage FROM #{r['table']} GROUP BY code)"
+      }
+      
+      execute("
+        UPDATE #{META_TABLE} meta
+        SET count = stat.count,
+            percentage = stat.percentage
+        FROM (#{sql.join(" UNION ")}) stat
+        WHERE meta.code = stat.code"
+      )
+    end
+    
+    private
+    def add_error_field(options)
+      error_field = options['log_to'] || DEFAULT_ERROR_FIELD
+      execute("ALTER TABLE #{options['table']} ADD COLUMN #{error_field} VARCHAR DEFAULT '';")
+    rescue Exception => ex
+      # @todo
+    end
+
+    def parse_rule(rule)
+      # @todo
+      type = rule[SUPPORTED_RULES_REGEXP]
+      if type.nil?
+        @logger.error "Invalid rule: #{rule}"
+        exit(0)
+      end
+
+      type.strip!
+      args = rule.gsub(SUPPORTED_RULES_REGEXP, '').gsub(/(^\s*["']|["']\s*$)/, "")
+      return type, args
+    end
+
+    def validate_not_null(options)
+      @logger.info "Validating data presence: #{options['table']}.[#{options['field']}]"
+      options['error'] ||= "[#{options['field']}] is null"
+      execute <<-eos
+        #{ update_sql(options) }
+        WHERE #{options['field']} IS NULL OR length(trim(#{options['field']})) = 0;
+      eos
+    end
+    
+    def validate_custom_query(options)
+      @logger.info "Validating with custom query: #{options['args'][0..50]}#{(options['args'].size > 50) ? '...' : ''}"
+      options['error'] ||= "Unknown"
+      execute <<-eos
+        #{ update_sql(options) }
+        WHERE NOT (#{options['args']})
+      eos
+    end
+    
+    def validate_reverse_query(options)
+      @logger.info "Validating with custom query: #{options['args'][0..50]}#{(options['args'].size > 50) ? '...' : ''}"
+      options['error'] ||= "Unknown"
+      execute <<-eos
+        #{ update_sql(options) }
+        WHERE (#{options['args']})
+      eos
+    end
+    
+    def validate_consistent_by(options)
+      @logger.info "Validating integrity: #{options['table']}.[#{options['field']}] #{options['rule']}"
+      options['error'] ||= "Same [#{options['field']}] but different [#{options['args']}]"
+
+      f1_case = f1 = options['field']
+      f2_case = f2 = options['args']
+
+      if options['case_insensitive']
+        f1_case = f1_case.lower
+        f2_case = f2_case.lower
+      end
+
+      execute <<-eos
+        #{ update_sql(options) }
+        WHERE id IN (
+          SELECT unnest(array_agg(id)) FROM #{options['table']}
+          WHERE #{f1.not_null_sql} AND #{f2.not_null_sql}
+          GROUP BY #{f2_case}
+          HAVING COUNT(distinct #{f1_case}) > 1
+        );
+      eos
+    end
+    
+    def validate_unique(options)
+      @logger.info "Validating uniqueness: #{options['table']}.[#{options['field']}]"
+      options['error'] ||= "[#{options['field']}] is not unique"
+
+      if options['case_insensitive']
+        f_lower = options['field'].lower
+      else
+        f_lower = options['field']
+      end
+      
+      execute <<-eos
+        #{ update_sql(options) }
+        WHERE id IN (
+          SELECT unnest(array_agg(id)) FROM #{options['table']} GROUP BY #{f_lower}
+          HAVING count(*) > 1
+        ) AND #{options['field'].not_null_sql};
+      eos
+    end
+    
+    def validate_cross_reference(options)
+      @logger.info "Validating reference: #{options['table']}.[#{options['field']}] #{options['rule']}"
+
+      options['error'] ||= "[#{options['field']}] does not reference [#{options['args']}]"
+
+      field = options['field']
+      ref_table, ref_field = options['args'].split(/[\.]/)
+
+      if options['args'].split(/[\.]/).size != 2
+        raise "Invalid rule"
+        exit(0)
+      end
+      
+      if options['case_insensitive']
+        join_condition = "on lower(origin.#{field}) = lower(target.#{ref_field})"
+      else
+        join_condition = "on origin.#{field}::text = target.#{ref_field}::text"
+      end
+
+      # @todo: poor performance here, think of a better SQL!!!
+      execute <<-eos
+        #{ update_sql(options) }
+        WHERE #{field} IN (
+          SELECT origin.#{field} from #{options['table']} origin LEFT JOIN #{ref_table} target
+          #{join_condition}
+          where target.#{ref_field} is null
+        ) AND #{field} IS NOT NULL AND length(trim(#{field})) <> 0;
+      eos
+    end
+    
+    def validate_match(options)
+      @logger.info "Validating regexp: #{options['table']}.[#{options['field']}] #{options['rule']}"
+      options['error'] ||= "[#{options['field']}] does not match #{options['args']}"
+      execute <<-eos
+        #{ update_sql(options) }
+        WHERE #{options['field']} IS NOT NULL AND length(trim(#{options['field']})) <> 0 AND #{options['field']} !~ '#{options['args']}';
+      eos
+    end
+    
+    def update_sql(options)
+      log_to = options['log_to'] || DEFAULT_ERROR_FIELD
+      sql = "UPDATE #{options['table']} SET #{log_to} = array_to_string(string_to_array(#{log_to}, ' || ') || string_to_array(#{quote(options['code'])}, ' || '), ' || ')"
+    end
+    
+    def execute(sql)
+      ActiveRecord::Base.connection.execute(sql)
+    end
+    
+    def quote(str = "")
+      ActiveRecord::Base.connection.quote(str)
+    end
+
+    def quote_col_name(str = "")
+      ActiveRecord::Base.connection.quote_column_name(str)
+    end
+
+    def table_exists?(table)
+      results = execute "SELECT * FROM pg_tables WHERE schemaname='public' AND tablename = #{quote(table)};"
+      return !results.first.nil?
+    end
+
+    def drop_table(table_name)
+      execute "DROP TABLE IF EXISTS #{table_name}"
+    end
+  end
+end
+
+
+x = IData::Validator.new $options[:config]
+x.validate!
+x.report!
+