iron-import 0.7.0 → 0.8.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 6931785ff3b1cb03394e6a8be0943337c589ac90
-  data.tar.gz: 82a7b1f123134eea5388672e8aed578590cd2daa
+  metadata.gz: 87cd90d663132748c61dfaa450449136f0cc00c4
+  data.tar.gz: e350419e3bdc6afb98b6a84b4fef8c9a4f1cd4be
 SHA512:
-  metadata.gz: d5d689572ef2680fce8273cc22c05ba76b2dc9a8a56f07c890d64ba6f1b4d4aba78f6bffea8f2b2b2eb9bdf057115bcf6911241f1febe567c7caee3a9eea6011
-  data.tar.gz: 4fafd479220b82ed7fd983b77b1c54d73ba993442d29c4b19a37c087e94feb35c25517549966fff2426be333491f71f59e23c20650c02a3ca5f6b34e4e6f4e91
+  metadata.gz: 1646f83be5af42715b1f71f9090aa0bf323bf9495d97d4abbb5543f85cb98f006fe9bd407823810c343e3dbc81a477da1445f399bcad9eca79108fd2869719f5
+  data.tar.gz: ba01474f1f4eebf7eaf2a23bc12824d0d57c00de61226ede66e1f1c9bd554d57161b8d811aeb2793cedc42f5a4b0bf8e9d34961c98f755aee5f09cfef1b81847

data/History.txt

@@ -1,10 +1,31 @@
+== 0.8.0 / 2017-06-29
+* Breaking Change: change signature of Importer#add_error to support new features
+* Breaking Change: Importer.missing_headers will be [] instead of nil on all headers found
+* Breaking Change: remove deprecated method Column#required!
+* Add Importer#rows to directly access rows post-import
+* Add Column#optional! to enable non-mandatory columns during header detection
+* Add Column#present? and Column#missing? to test for presence after import attempt
+* Add Importer#validate_columns to allow detecting invalid column combinations
+* Add Importer#validate_rows to allow whole-row validation
+* Add Importer#virtual_column and Column#calculate to enable virtual columns
+* Track actual header text found for columns
+* Update Column#to_s to use actual header text when present
+* Revamp error tracking significantly to provide better reporting ability
+* Improve error tracking to include row/column when knowable
+* Improve error tracking to include problem values when knowable
+* Add Column#error_values to return unique raw error values by column
+* Add Column#error_values? which will be true when there is at least one error value for the column
+* Add Row#error_map to return map of column key => raw value for each row
+* Change Column#parse block processing to allow explicit #add_error call
+* Change Column#validate block processing to allow explicit #add_error call and to add an implicit error on returned false
+
 == 0.7.0 / 2017-02-16
 
 * Breaking Change: Removed multi-sheet support - use multiple importers instead
-* Breaking Change: Removed warnings as they were not being used
-* Breaking Change: Removed Column#required! due to bugginess and overlap with Column#validate
-* Add Importer#scope to allow narrowing the search to one or more sheets/tables when importing
+* Breaking Change: Removed warnings as they were confusing and not being used
+* Deprecate Column#required! due to bugginess and overlap with Column#validate
 * Add new HtmlReader support to handle parsing HTML <table> rows
+* Add Importer#scope to allow narrowing the search to one or more sheets/tables when importing
 * Modify Importer#import to support block mode combining #import and #process
 * Add Importer#import_string for handling explicit CSV/HTML/Custom text
 * Add Importer#on_error(&block) to allow inline conditional error handling

data/README.rdoc

@@ -4,11 +4,17 @@ Written by Rob Morris @ Irongaze Consulting LLC (http://irongaze.com)
 
 == DESCRIPTION
 
-Simple, reliable tabular data import.
+Simple, versatile, reliable tabular data import.
 
 This gem provides a set of classes to support automating import of tabular data from 
-CSV, HTML, XLS and XLSX files, or custom formats via a simple block reader.  Provides help 
-in defining columns, auto-detecting column order, pre-parsing data, and error tracking.
+CSV, HTML, XLS and XLSX files.  Key features include defining columns, auto-detecting column order, 
+pre-parsing data, validating data, filtering rows, and robust error tracking.
+
+IMPORTANT NOTE: this gem is in flux as we work to define the best possible abstraction
+for the task.  Breaking changes will be noted by increases in the minor version,
+ie 0.5.0 and 0.5.1 will be compatible, but 0.6.0 will not (i.e. we follow semantic versioning).
+
+== WHO IS THIS FOR?
 
 The Roo/Spreadsheet gems do a great job of providing general purpose spreadsheet reading.
 However, using them with unreliable user submitted data requires a lot of error checking,
@@ -17,21 +23,36 @@ businesses, where Excel files are the lingua franca for all kinds of uses.  This
 attempts to extract years of experience building one-off importers into a simple library 
 for rapid import coding.
 
+In addition, it's quite common for the same data to be transmitted in varying formats -
+Excel files, HTML files, CSV files, custom text streams...  Use iron-import to have a single
+tool-set for processing any of these types of data, often without changing a line of code.
+
 This is NOT a general-purpose tool for reading spreadsheets.  If you want access to 
 cell styling, reading underlying formulas, etc., you will be better served building
-a custom importer based on Roo.  But if you're looking to take an uploaded CSV file,
+a custom importer based on Roo.  But if you're looking to take a customer-uploaded CSV file,
 validate and coerce values, then write each row to a database, all the while tracking
 any errors encountered... well, this is the library for you!
 
-IMPORTANT NOTE: this gem is in flux as we work to define the best possible abstraction
-for the task.  Breaking changes will be noted by increases in the minor version,
-ie 0.5.0 and 0.5.1 will be compatible, but 0.6.0 will not (i.e. we follow semantic versioning).
+== KEY FEATURES
+
+- Simple yet robust data import and error handling using elegant builder syntax
+- Import data from file, stream or string data sources
+- Import XLS, XLSX, CSV and HTML tabular data
+- Import custom tabular data via passed block
+- Automatic column order and start row detection
+- Support for optional columns and dynamic column sets
+- Basic data coercion supporting string, int, float, date and cents types
+- Custom data coercion via passed block
+- Custom data validation via passed block
+- Row filtering using custom block
+- Automatically track and report errors with fine-grained context
+- Prefer capturing errors over raising exceptions for more robust imports
 
 == SAMPLE USAGE
 
     # Define our importer, with three columns.  The importer will look for a row containing
     # "name"/"product", "description" and "price" (case insensitively) and automatically determine column
-    # order and starting row of the data.
+    # order and the starting row of the data.
     importer = Importer.build do
       column :name do
         # Column order and start row are auto-detected
@@ -42,9 +63,9 @@ ie 0.5.0 and 0.5.1 will be compatible, but 0.6.0 will not (i.e. we follow semant
         parse do |raw_val|
           raw_val.to_s.strip
         end
-        # And per-row validation
+        # And custom validation
         validate do |parsed_val|
-          raise "Invalid description" unless parsed_val.length > 5
+          add_error('Invalid description') unless parsed_val.length > 5
         end
       end
       column :price do
@@ -52,13 +73,13 @@ ie 0.5.0 and 0.5.1 will be compatible, but 0.6.0 will not (i.e. we follow semant
         type :cents
       end
       
-      # Need to skip rows?  Use a filter!
-      filter do |row|
+      # Need to skip rows?  Use a filter!  Return true to include a row when processing
+      filter_rows do |row|
         row[:price] != 0 && row[:name] != 'Sample'
       end
     end
     
-    # Import the provided file row-by-row (if importing succeeds), automatically
+    # Import the provided file or stream row-by-row (if importing succeeds), automatically
     # using the proper library to read CSV data.  This same code would work
     # with XLS or XLSX files with no changes to the code.
     importer.import('/tmp/source.csv') do |row|
@@ -67,9 +88,52 @@ ie 0.5.0 and 0.5.1 will be compatible, but 0.6.0 will not (i.e. we follow semant
 
     # Check for errors and do the right thing:
     importer.on_error do
-      puts "Error: " + error_summary
+      if missing_headers.any?
+        # Can't find required column header(s)
+        puts "Unable to locate columns: #{missing_headers}"
+        
+      elsif columns.any?(&:error_values?)
+        # Invalid or unexpected values in one or more columns
+        columns.select(&:error_values?).each do |col|
+          puts "Invalid values for #{col}: #{col.error_values}"
+        end
+        
+      else
+        # General errors, dump report
+        puts "Error(s) on import: " + error_summary
+      end
     end
 
+    # You can chain the build/import/on-error blocks for a cleaner flow:
+    Importer.build do
+      column :one
+      column :two
+    end.import(params[:uploaded_file]) do |row|
+      SomeModel.create(row)
+    end.on_error do
+      raise "Errors found: " + error_summary
+    end
+    
+== IMPORT EXECUTION ORDER
+
+It can be tricky to keep track of what happens in Importer#import, so here's a quick cheat-sheet:
+
+- Determine the **format** of stream/file to import
+- Determine **import scope** (sheet/table/whatever) using Importer#scope settings, if any
+- **Find column headers + start row**
+- Validate presence of **required columns**
+- **Validate column set** using Importer#validate_columns
+- Run each row:
+  - **Parse** each column's value using Column#parse or Column#type
+  - **Filter the row** using Importer#filter_rows on parsed values to reject unwanted rows
+  - **Calculate virtual columns** using Column#calculate
+  - **Validate each parsed value** using Column#validate
+  - **Validate entire row** using Importer#validate_rows
+
+Generally, the import will stop when an error occurs, save on row processing, where each row will
+be run until an error for that row is found.  The goal is to accumulate actionable info for
+presentation to the end user who is uploading the file.
+
 == REQUIREMENTS
 
 Depends on the iron-extensions and iron-dsl gems for CSV and custom import formats.

data/Version.txt

@@ -1 +1 @@
-0.7.0
+0.8.0

data/lib/iron/import/column.rb

@@ -8,13 +8,17 @@ class Importer
   #
   #   Importer.build do
   #     column :key do
+  #       # Mark this column as optional, i.e. if the header isn't found, the import will
+  #       # work without error and the imported row will simply not contain this column's data.
+  #       optional!
+  #
   #       # Set a fixed position - may be a column number or a letter-based
   #       # column description, ie 'A' == 1.  In most cases, you can leave
   #       # this defaulted to nil, which will mean "look for the proper header"
   #       position 'C'
   #
   #       # Specify a regex to locate the header for this column, defaults to 
-  #       # finding a string containing the key.
+  #       # finding a string containing the key, ignored if position is set.
   #       header /(price|cost)/i
   #
   #       # Tells the data parser what type of data this column contains, one
@@ -24,7 +28,8 @@ class Importer
   #       # Instead of a type, you can set an explicit parse block.  Be aware
   #       # that different source types may give you different raw values for what
   #       # seems like the "same" source value, for example an Excel source file
-  #       # will give you a float value for all numeric types, even "integers".
+  #       # will give you a float value for all numeric types, even "integers", while
+  #       # CSV and HTML values are always strings.
   #       parse do |raw_value|
   #         val = raw_value.to_i + 1000
   #         # NOTE: we're in a block, so don't do this:
@@ -35,9 +40,20 @@ class Importer
   #      
   #       # You can also add a custom validator to check the value and add
   #       # an error if it's not within a given range, or whatever.  To fail validation,
-  #       # simply raise the error you wish recorded.
-  #       validate do |parsed_value|
-  #         raise "Out of range" unless (parsed_value > 0 && parsed_value < 5000)
+  #       # return false, raise an exception, or use #add_error
+  #       validate do |parsed_value, row|
+  #         add_error "Out of range" unless (parsed_value > 0 && parsed_value < 5000)
+  #       end
+  #
+  #       # Mark a column as _virtual_, meaning it won't be looked for in the source
+  #       # file/stream, and instead will be calculated using #calculate.  When set,
+  #       # causes importer to ignore position/header/type/parse settings.
+  #       virtual!
+  #
+  #       # When #virtual! is set, gets called to calculate each row's value for this
+  #       # column using the row's parsed values.
+  #       calculate do |row|
+  #         row[:some_col] + 5
   #       end
   #     end
   #   end
@@ -46,10 +62,14 @@ class Importer
 
     # Holds load-time data
     class Data
-      attr_accessor :index
+      attr_accessor :index, :header_text, :errors
+      
+      def initialize
+        @errors = []
+      end
       
       def pos
-        @index ? Column::index_to_pos(@index) : 'Unknown'
+        @index ? Column::index_to_pos(@index) : 'Not Found'
       end
     end
 
@@ -59,7 +79,8 @@ class Importer
 
     # Configuration
     dsl_accessor :header, :position, :type
-    dsl_accessor :parse, :validate
+    dsl_accessor :parse, :validate, :calculate
+    dsl_flag :optional, :virtual
    
     def self.pos_to_index(pos)
       raise 'Invalid column position: ' + pos.inspect unless pos.is_a?(String) && pos.match(/\A[a-z]{1,3}\z/i)
@@ -73,6 +94,7 @@ class Importer
       total - 1
     end
     
+    # Convert a numeric index to an Excel-like column position, e.g. 3 => 'C'
     def self.index_to_pos(index)
       val = index.to_i
       raise 'Invalid column index: ' + index.inspect if (!index.is_a?(Fixnum) || index.to_i < 0)
@@ -95,6 +117,12 @@ class Importer
       # Save off our info
       @key = key
       @importer = importer
+
+      # Are we optional?
+      @optional = options_hash.delete(:optional) { false }
+      
+      # Are we virtual?
+      @virtual = options_hash.delete(:virtual) { false }
       
       # Return it as a string, by default
       @type = options_hash.delete(:type) { :string }
@@ -105,8 +133,14 @@ class Importer
       # By default, don't parse incoming data, just pass it through
       @parse = options_hash.delete(:parse)
       
+      # Custom validation, anyone?
+      @validate = options_hash.delete(:validate)
+      
+      # Custom validation, anyone?
+      @calculate = options_hash.delete(:calculate)
+      
       # Default matcher, looks for the presence of the column key as text anywhere
-      # in the header string, ignoring case and using underscores as spaces, ie
+      # in the header string, ignoring case and treating underscores as spaces, ie
       # :order_id => /\A\s*order id\s*\z/i
       @header = options_hash.delete(:header) {
         Regexp.new('\A\s*' + key.to_s.gsub('_', ' ') + '\s*\z', Regexp::IGNORECASE)
@@ -126,22 +160,10 @@ class Importer
       @data = Data.new
     end
 
-    # DEPRECATED - duplicates functionality better provided by #validate, e.g.
-    #
-    #   validate do |val|
-    #     raise 'Missing required value for column foo' if val.nil?
-    #   end
-    def required!
-      Kernel.warn "[DEPRECATION] Importer::Column#required! is deprecated.  Please use #validate instead."
-      col = self.key
-      validate do |val|
-        raise "Missing required value for column :#{col}"
-      end
-    end
-    
     # When true, our header definition or index match the passed text or column index.
-    def match_header?(text, index)
-      return true if index == self.fixed_index
+    def match_header?(text, test_index)
+      return false if virtual?
+      return true if test_index == self.fixed_index
       if @header.is_a?(Regexp)
         return !@header.match(text).nil?
       else
@@ -149,34 +171,11 @@ class Importer
       end
     end
     
-    # Applies any custom parser defined to process the given value, capturing
-    # errors as needed
-    def parse_value(row, val)
-      return val if @parse.nil?
-      begin 
-        @parse.call(val)
-      rescue Exception => e
-        @importer.add_error(row, "Error parsing #{self}: #{e}")
-        nil
-      end
-    end
-    
-    # Applies any validation to a parsed value
-    def validate_value(row, val)
-      return true unless @validate
-      begin 
-        @validate.call(val)
-        true
-      rescue Exception => e
-        @importer.add_error(row, "Validation error in #{self}: #{e}")
-        false
-      end
-    end
-    
     # Returns the fixed index of this column based on the set position.
     # In other words, a position of 2 would return an index of 1 (as
     # indicies are 0-based), where a position of 'C' would return 2.
     def fixed_index
+      return nil if virtual?
       return nil unless @position
       if @position.is_a?(Fixnum)
         @position - 1
@@ -185,9 +184,98 @@ class Importer
       end
     end
     
+    # Applies any custom parser defined to process the given value, capturing
+    # errors as needed
+    def parse_value(row, raw_val)
+      return raw_val if @parse.nil?
+
+      res = nil
+      had_error = Error.with_context(@importer, row, self, raw_val) do
+        res = DslProxy.exec(@importer, raw_val, &@parse)
+      end
+      had_error ? nil : res
+    end
+    
+    def calculate_value(row)
+      return nil if @calculate.nil?
+      res = nil
+      had_error = Error.with_context(@importer, row, self, nil) do
+        res = DslProxy.exec(@importer, row, &@calculate)
+      end
+      had_error ? nil : res
+    end
+    
+    # Applies any validation to a parsed value
+    def validate_value(row, parsed_val)
+      return true unless @validate
+
+      valid = false
+      had_error = Error.with_context(@importer, row, self, parsed_val) do
+        valid = DslProxy.exec(@importer, parsed_val, row, &@validate)
+      end
+      if had_error
+        return false
+      elsif valid.is_a?(FalseClass)
+        @importer.add_error("Invalid value: #{parsed_val.inspect}", :row => row, :column => self, :value => parsed_val)
+        return false
+      else
+        return true
+      end
+    end
+    
+    # Index of the column in the most recent import, if found, or
+    # nil if not present.
+    def index
+      @data.index
+    end
+    
+    # When true, column was found in the last import, eg:
+    #
+    #   importer.process do |row|
+    #     puts "Size: #{row[:size]}" if column(:size).present?
+    #   end
+    def present?
+      !@data.index.nil?
+    end
+    
+    # Sugar, simply the opposite of #present?
+    def missing?
+      !present?
+    end
+    
+    def parses?
+      !@parse.nil?
+    end
+    
+    def validates?
+      !@validate.nil?
+    end
+    
+    def calculates?
+      !@calculate.nil?
+    end
+    
+    def errors
+      @data.errors
+    end
+    
+    def error_values
+      errors.collect(&:value).uniq
+    end
+
+    def error_values?
+      error_values.any?
+    end
+    
     # Pretty name for ourselves
     def to_s
-      'Column ' + @data.pos
+      if !virtual? && @data.header_text.blank?
+        "Column #{@data.pos}"
+      else
+        name = virtual? ? key.to_s : @data.header_text
+        name = name.gsub(/(^[a-z]|\s[a-z])/) {|m| m.capitalize } 
+        "#{name} Column"
+      end
     end
     
     # Extracts the imported values for this column and returns them in an array.

data/lib/iron/import/csv_reader.rb

@@ -30,9 +30,9 @@ class Importer
       end
     end
    
-    # Normally, we'd check the scopes and return the proper data, but for CSV files, 
-    # there's only one scope...
     def load_raw(scopes, &block)
+      # Normally, we'd check the scopes and return the proper data, but for CSV files, 
+      # there's only one scope...
       block.call(@raw_rows)
     end
     

data/lib/iron/import/data_reader.rb

@@ -182,8 +182,14 @@ class Importer
       !@importer.has_errors?
     end
     
-    # Load up the sheets in the correct mode
+    # Load up the sheet in the correct mode
     def load_each(mode, source, scopes, &block)
+      # Handle some common error cases centrally
+      if mode == :file && !File.exist?(source)
+        add_error("File not found: #{source}")
+        return
+      end
+      
       # Let our derived classes open the file, etc. as they need
       if init_source(mode, source)
         # Once the source is set, run through each defined sheet, pass it to
@@ -209,12 +215,11 @@ class Importer
     # handle edge cases like converting '5.00' to 5 when in integer mode, etc.  If you find your inputs aren't
     # being parsed correctly, add a custom #parse block on your Column definition.
     def parse_value(val, type)
-      return nil if val.nil? || val.to_s == ''
+      return nil if val.nil? || val.to_s.strip == ''
       
       case type
       when :string then
         val = val.to_s.strip
-        val.blank? ? nil : val
         
       when :integer, :int then 
         if val.class < Numeric

data/lib/iron/import/error.rb

@@ -1,14 +1,65 @@
 class Importer
   
   class Error
-     
-    attr_reader :row, :text
     
-    def initialize(context, text)
-      if context.is_a?(Importer::Row)
-        @row = context
+    attr_reader :row, :column, :value, :text
+    
+    # Block wrapper to set error context for any errors generated within the block
+    def self.with_context(importer, row, column, val)
+      # Set new context
+      old_row = @context_row
+      @context_row = row
+      old_col = @context_column
+      @context_column = column
+      old_val = @context_value
+      @context_value = val
+      old_err = @error_occurred
+      @error_occurred = false
+      
+      # Run the block, catch raised exceptions as errors
+      begin
+        yield
+      rescue RuntimeError => e
+        # Old-style way of registering errors was to just raise 'foo'
+        importer.add_error(e.to_s)
       end
+      had_error = @error_occurred
+      
+      # Reset to old context
+      @context_row = old_row
+      @context_column = old_col
+      @context_value = old_val
+      @error_occurred = old_err
+      
+      return had_error
+    end
+    
+    def self.context_row
+      @context_row
+    end
+    
+    def self.context_column
+      @context_column
+    end
+    
+    def self.context_value
+      @context_value
+    end
+    
+    def self.error_occurred!
+      @error_occurred = true
+    end
+    
+    def initialize(text, context = {})
       @text = text.to_s
+      @row = context[:row] || Error.context_row
+      @column = context[:column] || Error.context_column
+      @value = context[:value] || Error.context_value
+      
+      @row.errors << self if @row
+      @column.errors << self if @column
+      
+      Error.error_occurred!
     end
     
     def summary
@@ -39,7 +90,7 @@ class Importer
     end
 
     # Returns true if this error is for the given context, where
-    # context can be a Row, Sheet or Importer instance.
+    # context can be a Row or Importer instance.
     def for_context?(context)
       case context
       when Row

data/lib/iron/import/excel_reader.rb

@@ -25,7 +25,7 @@ class Importer
         false
       end
     rescue Exception => e
-      add_error("Error reading file #{source}: #{e}")
+      add_error("Error reading #{source}: #{e}")
       false
     end
     

data/lib/iron/import/html_reader.rb

@@ -13,12 +13,7 @@ class Importer
       if mode == :stream
         @html = Nokogiri::HTML(source)
       elsif mode == :file
-        if File.exist?(source)
-          @html = File.open(source) {|f| Nokogiri::HTML(f) }
-        else
-          add_error("File not found: #{source}")
-          return false
-        end
+        @html = File.open(source) {|f| Nokogiri::HTML(f) }
       else
         add_error("Unsupported HTML mode: #{mode}")
         return false

data/lib/iron/import/importer.rb

@@ -21,21 +21,51 @@
 # A more realistic and complex example follows:
 #
 #   Importer.build do
-#     # Define our columns and settings
+#     # Define our columns and their settings
 #     column :order_number do
-#       header /order (num.*|id)/i
+#       optional!
+#       header /order (\#|num.*|id)/i
 #       type :int
 #     end
+#     column :po_number do
+#       optional!
+#       type :string
+#       validate do |num|
+#         num.match(/[a-z0-9]{12}/i)
+#       end
+#     end
 #     column :date do
 #       type :date
 #     end
 #     column :amount do
 #       type :cents
 #     end
+#     virtual_column :tax do
+#       calculate do |row|
+#         row[:amount] * 0.05
+#       end
+#     end
 #  
+#     # When you have optional columns, you can validate that you have enough of them
+#     # using a custom block returning true if the found columns are good enough to 
+#     # continue.
+#     validate_columns do |cols|
+#       # Require either an order # or a PO # column
+#       keys = cols.collect(&:key)
+#       keys.include?(:order_number) || keys.include?(:po_number)
+#     end
+#
 #     # Filter out any rows missing an order number
 #     filter do |row|
-#       !row[:order_number].nil?
+#       !row[:order_number].nil? || !row[:po_number].nil?
+#     end
+#
+#     # Use row-level validation to validate using
+#     # any or all column values for that row, to allow complex validation
+#     # scenarios that depend on the full context.
+#     validate_rows do |row|
+#       # Ensure PO Numbers are only valid starting in 2017
+#       add_error 'Invalid order - PO Num from before 2017' unless (row[:date] > Date.parse('2017-01-01') || row[:po_number].nil?)
 #     end
 #
 #   end.import('/path/to/file.csv', format: :csv) do |row|
@@ -52,17 +82,16 @@ class Importer
 
   # Inner class for holding load-time data that gets reset on each load call
   class Data
-    attr_accessor :start_row, :rows
+    attr_accessor :start_row, :rows, :errors
     def initialize
       @start_row = nil
       @rows = []
+      @errors = []
     end
   end
 
   # Array of defined columns
   attr_reader :columns
-  # Array of error messages collected during an import/process run
-  attr_accessor :errors
   # Custom reader, if one has been defined using #on_file or #on_stream
   attr_reader :custom_reader
   # Set to the format selected during past import
@@ -81,6 +110,8 @@ class Importer
   # Set to a block/lambda taking a parsed but unvalidated row as a hash,
   # return true to keep, false to skip.
   dsl_accessor :filter
+  # Alias for #filter
+  def filter_rows(*args, &block); filter(*args, &block); end
   # Source file/stream encoding, assumes UTF-8 if none specified
   dsl_accessor :encoding
 
@@ -113,14 +144,15 @@ class Importer
   #     headerless!
   #
   #     # Manually set the start row for data, defaults to nil
-  #     # indicating that the data rows start immediatly following the header.
+  #     # indicating that the data rows start immediatly following the header, or
+  #     # at the first row if #headerless!.
   #     start_row 4
   #
   #     # Define a filter that will skip unneeded rows.  The filter command takes
   #     # a block that receives the parsed (but not validated!) row data as an 
   #     # associative hash of :col_key => <parsed value>, and returns 
   #     # true to keep the row or false to exclude it.
-  #     filter do |row|
+  #     filter_rows do |row|
   #       row[:id].to_i > 5000
   #     end
   #
@@ -171,6 +203,7 @@ class Importer
   # Use whichever you prefer!
   def column(key, options_hash = {}, &block)
     # Find existing column with key to allow re-opening an existing definition
+    key = key.to_sym
     col = @columns.detect {|c| c.key == key }
     unless col
       # if none found, add a new one
@@ -184,6 +217,11 @@ class Importer
     col
   end
   
+  def virtual_column(key, options_hash = {}, &block)
+    options_hash[:virtual] = true
+    column(key, options_hash, &block)
+  end
+  
   # Limit the search scope for a single format (:xls, :xlsx, :html, :custom)
   # to the given value or values - the meaning and format of scopes is determined
   # by that format's data reader.
@@ -213,7 +251,7 @@ class Importer
   # a block accepting a file path, and returning an array of arrays (rows of
   # raw column values).  Use #add_error(msg) to add a reading error.
   # 
-  # Adding a custom stream parser will change the importer's default
+  # Adding a custom file/stream parser will change the importer's default
   # format to :custom, though you can override it when calling #import as 
   # usual.
   #
@@ -268,7 +306,7 @@ class Importer
   #   encoding: source encoding override, defaults to guessing based on input
   #   
   # Generally, you should be able to throw a path or stream at it and it should work.  The
-  # options exist to allow overriding in cases where the automation heuristics
+  # options exist to allow overriding in cases where the automated heuristics
   # have failed and the input type is known by the caller.
   #
   # If you're trying to import from a raw string, use Importer#import_string instead.
@@ -362,7 +400,7 @@ class Importer
   # Use this form of import for the common case of having a raw CSV or HTML string.
   def import_string(string, options = {}, &block)
     # Get a format here if needed
-    if options[:format].nil?
+    if options[:format].nil? || options[:format] == :auto
       if @custom_reader
         format = :custom
       else
@@ -378,8 +416,8 @@ class Importer
   # Call with a block accepting a single Importer::Row with contents that
   # look like :column_key => <parsed value>.  Any filtered rows
   # will not be present.  If you want to register an error, simply 
-  # raise "some text" and it will be added to the importer's error
-  # list for display to the user, logging, or whatever.
+  # raise "some text" or call #add_error and it will be added to the importer's 
+  # error list for display to the user, logging, or whatever.
   def process
     @data.rows.each do |row|
       begin
@@ -390,20 +428,46 @@ class Importer
     end
   end
   
+  # Call with a block to process error handling tasks.  Block will only execute
+  # if an error (read, validate, exception, etc.) has occurred during the 
+  # just-completed #import.
+  #
+  # Your block can access the #error_summary or the #errors array to do whatever
+  # logging, reporting etc. is desired.
   def on_error(&block)
     raise 'Invalid block passed to Importer#on_error: block may accept 0, 1 or 2 arguments' if block.arity > 2
     
     if has_errors?
       case block.arity
       when 0 then DslProxy.exec(self, &block)
-      when 1 then DslProxy.exec(self, @errors, &block)
-      when 2 then DslProxy.exec(self, @errors, error_summary, &block)
+      when 1 then DslProxy.exec(self, errors, &block)
+      when 2 then DslProxy.exec(self, errors, error_summary, &block)
       end
     end
     
     self
   end
   
+  # Call with a block accepting an array of Column objects and returning
+  # true if the columns in the array should constitute a valid header row.  Intended
+  # for use with optional columns to define multiple supported column sets, or
+  # conditionally required secondary columns.  Columns will be passed in in the
+  # order detected, so you can use ordering to help determine which columns are
+  # required if that helps.
+  def validate_columns(&block)
+    raise 'Invalid block passed to Importer#validate_columns: block should accept a single argument' if block.arity != 1
+    @column_validator = block
+  end
+  
+  # Call with a block accepting a single Row instance.  Just like Column#validate, you
+  # can fail by returning false, calling #add_error(msg) or by raising an exception.
+  # The intent of this method of validation is to allow using the full row context to
+  # validate 
+  def validate_rows(&block)
+    raise 'Invalid block passed to Importer#validate_columns: block should accept a single Row argument' if block.arity != 1
+    @row_validator = block
+  end
+  
   # Process the raw values for the first rows in a sheet,
   # and attempt to build a map of the column layout, and
   # detect the first row of real data
@@ -419,7 +483,7 @@ class Importer
         next_index += 1
       end
       @data.start_row = @start_row || 1
-      @missing_headers = nil
+      @missing_headers = []
       return true
       
     else
@@ -430,21 +494,42 @@ class Importer
         next unless row
         
         # Set up for this iteration
-        remaining = @columns.dup
+        remaining = @columns.select {|c| !c.virtual? }
     
         # Step through this row's raw values, and look for a matching column for all columns
         row.each_with_index do |val, i|
-          col = remaining.detect {|c| c.match_header?(val.to_s, i) }
+          val = val.to_s
+          col = remaining.detect {|c| c.match_header?(val, i) }
           if col
             remaining -= [col]
             col.data.index = i
+            col.data.header_text = val
           end
         end
+        # Reset remaining cols
+        remaining.each do |col| 
+          col.data.index = nil
+          col.data.header_text = nil
+        end
         
-        if remaining.empty?
+        # Have we found them all, or at least a valid sub-set?
+        header_found = remaining.empty?
+        unless header_found
+          if remaining.all?(&:optional?)
+            if @column_validator
+              # Run custom column validator
+              cols = found_columns
+              header_found = @column_validator.call(cols)
+            else
+              # No validator... do we have any found columns at all???
+              header_found = @columns.any?(&:present?)
+            end
+          end
+        end
+        if header_found
           # Found all columns, have a map, update our start row to be the next line and return!
           @data.start_row = @start_row || i+2
-          @missing_headers = nil
+          @missing_headers = []
           return true
         else
           missing = remaining if (missing.nil? || missing.count > remaining.count)
@@ -452,7 +537,7 @@ class Importer
       end
       
       # If we get here, we're hosed
-      @missing_headers = missing.collect(&:key) if @missing_headers.nil? || @missing_headers.count > missing.count
+      @missing_headers = missing.collect(&:key) if @missing_headers.empty? || @missing_headers.count > missing.count
       false
     end
   end
@@ -467,46 +552,81 @@ class Importer
     # Parse out the values
     values = {}
     @columns.each do |col|
-      index = col.data.index
-      raw_val = raw_data[index]
-      if col.parse
-        # Use custom parser if this row has one
-        val = col.parse_value(row, raw_val)
-      else
-        # Otherwise use our standard parser
-        val = @reader.parse_value(raw_val, col.type)
+      if col.present? && !col.virtual?
+        index = col.data.index
+        raw_val = raw_data[index]
+        if col.parses?
+          # Use custom parser if this row has one
+          val = col.parse_value(row, raw_val)
+        else
+          # Otherwise use our standard parser
+          val = @reader.parse_value(raw_val, col.type)
+        end
+        values[col.key] = val
       end
-      values[col.key] = val
     end
 
-    # Set the values and filter if needed
+    # Set the values
     row.set_values(values)
-    return nil if @filter && !@filter.call(row)
+    
+    if !row.has_errors?
+      # Filter if needed
+      return nil if @filter && !@filter.call(row)
 
-    # Row is desired, now validate values
-    @columns.each do |col|
-      val = values[col.key]
-      col.validate_value(row, val)
-    end
+      # Calculate virtual columns' values
+      @columns.each do |col|
+        if col.virtual?
+          row.values[col.key] = col.calculate_value(row)
+        end
+      end
+
+      # Validate values if any column has a custom validator
+      @columns.each do |col|
+        if col.present? && col.validates?
+          val = values[col.key]
+          col.validate_value(row, val)
+        end
+      end
     
+      # If we have a row validator, call it on the full row
+      if @row_validator && !row.has_errors?
+        valid = false
+        had_error = Error.with_context(@importer, row, nil, nil) do
+          valid = DslProxy.exec(self, row, &@row_validator)
+        end
+        if !had_error && valid.is_a?(FalseClass)
+          add_error("Invalid row: #{row.to_hash.inspect}", :row => row)
+        end
+      end
+    end
+
     # We is good
     @data.rows << row
     row
   end
+  
+  def rows
+    @data.rows
+  end
+  
+  def found_columns
+    @columns.select(&:present?).sort_by(&:index)
+  end
+
+  # Array of error messages collected during an import/process run
+  def errors
+    @data.errors
+  end
 
   # When true, one or more errors have been recorded during this import/process
   # cycle.
   def has_errors?
-    @errors.any?
+    @data.errors.any?
   end
   
   # Add an error to our error list.  Will result in a failed import.
-  def add_error(context, msg = nil)
-    if context.is_a?(String) && msg.nil?
-      msg = context
-      context = nil
-    end
-    @errors << Error.new(context, msg)
+  def add_error(msg, context = {})
+    @data.errors << Error.new(msg, context)
   end
   
   # Returns a human-readable summary of the errors present on the importer, or
@@ -517,7 +637,7 @@ class Importer
 
     # Group by error text - we often get the same error dozens of times
     list = {}
-    @errors.each do |err|
+    @data.errors.each do |err|
       errs = list[err.text] || []
       errs << err
       list[err.text] = errs
@@ -544,8 +664,7 @@ class Importer
   protected
   
   def reset
-    @errors = []
-    @missing_headers = nil
+    @missing_headers = []
     @format = nil
     @reader = nil
     @data = Data.new

data/lib/iron/import/row.rb

@@ -2,12 +2,14 @@ class Importer
   
   class Row
 
-    attr_reader :line, :values
+    attr_reader :line, :values, :errors
     
     def initialize(importer, line, value_hash = nil)
       @importer = importer
       @line = line
       set_values(value_hash)
+      
+      @errors = []
     end
     
     def set_values(value_hash)
@@ -58,6 +60,19 @@ class Importer
       @importer.add_error(self, msg)
     end
     
+    def has_errors?
+      @errors && @errors.count > 0
+    end
+    
+    # Return a map of column key to Error, intended for use in error reporting.
+    def error_map
+      map = {}
+      @errors.each do |err|
+        map[err.column.key] = err
+      end
+      map
+    end
+    
   end
   
 end

data/spec/importer/column_spec.rb

@@ -112,4 +112,29 @@ describe Importer::Column do
     @importer.has_errors?.should be_true
   end
   
+  it 'should record optionalness' do
+    @col.optional?.should be_false
+    @col.optional!
+    @col.optional?.should be_true
+  end
+  
+  it 'should know if it is present in the headers' do
+    @col.present?.should be_false
+    @col.data.index = 2
+    @col.present?.should be_true
+  end
+  
+  it 'should use the header text as its name if present' do
+    @col.data.index = 2
+    @col.to_s.should == 'Column C'
+    @col.data.header_text = 'Invoice #'
+    @col.to_s.should == 'Invoice # Column'
+  end
+  
+  it 'should support virtual operation' do
+    @col.virtual!
+    @col.virtual?.should be_true
+    @col.to_s.should == 'Test Column'
+  end
+  
 end

data/spec/importer/error_spec.rb

@@ -0,0 +1,34 @@
+describe Importer::Error do
+
+  before do
+    @importer = Importer.new
+    @row = Importer::Row.new(@importer, 5)
+    @col = Importer::Column.new(@importer, :test)
+  end
+
+  it 'should capture context' do
+    val = 'foo'
+    err = nil
+    Importer::Error.with_context(@importer, @row, @col, val) do
+      err = Importer::Error.new('hi')
+    end
+    err.row.should == @row
+    err.column.should == @col
+    err.value.should == val
+  end
+
+  it 'should return error status for #with_context' do
+    # Block runs fine, no error
+    had_err = Importer::Error.with_context(@importer, @row, @col, 'bob') do
+      false
+    end
+    had_err.should be_false
+
+    # Create a new error, we should get a true
+    had_err = Importer::Error.with_context(@importer, @row, @col, 'bob') do
+      Importer::Error.new('hi')
+    end
+    had_err.should be_true
+  end
+
+end

data/spec/importer/importer_spec.rb

@@ -23,6 +23,21 @@ describe Importer do
     importer.scopes.should == { :xls => [1, 'Sheet 2'], :html => ['table.funny'] }
   end
   
+  it 'should calculate virtual columns' do
+    importer = Importer.build do
+      column :num, :type => :int
+      virtual_column :summary do
+        calculate do |row|
+          "Value = #{row[:num]}"
+        end
+      end
+    end
+    
+    importer.import_string("num\n1\n2")
+    importer.error_summary.should be_nil
+    importer.column(:summary).to_a.should == ['Value = 1', 'Value = 2']
+  end
+  
   it 'should find headers automatically' do
     # Define a few sample columns
     importer = Importer.new
@@ -57,6 +72,59 @@ describe Importer do
     importer.missing_headers.should == [:alpha]
   end
 
+  it 'should succeed when missing optional columns' do
+    # Define a few sample columns
+    importer = Importer.new
+    importer.column(:alpha).optional!
+    importer.column(:beta)
+    importer.column(:gamma)
+    # Some dummy data
+    rows = [
+      ['Bob', 'Beta', 'Gamma', 'Epsilon']
+    ]
+
+    # Parse it!
+    importer.find_header(rows).should be_true
+    importer.missing_headers.should be_empty
+  end
+  
+  it 'should support row-based validation' do
+    importer = Importer.build do
+      column :a, :type => :int
+      column :b, :type => :int
+      
+      validate_rows do |row|
+        row[:a] + row[:b] == 5
+      end
+    end
+    
+    importer.import_string("a,b\n1,4\n6,-1\n7,0\n1,1")
+    importer.errors.count.should == 2
+  end
+  
+  it 'should support column order/presence validation' do
+    # Build an importer with optional columns
+    importer = Importer.new
+    importer.column(:alpha).optional!
+    importer.column(:beta).optional!
+    importer.column(:gamma)
+    # Set up a column validator
+    importer.validate_columns do |cols|
+      cols = cols.collect(&:key)
+      cols.sort == [:alpha, :gamma] || cols.sort == [:beta, :gamma]
+    end
+
+    # Missing required column
+    importer.find_header([['Alpha', 'Beta', 'Epsilon']]).should be_false
+    # Missing both optional
+    importer.find_header([['Bob', 'Gamma', 'Epsilon']]).should be_false
+    # Required + single optional
+    importer.find_header([['Bob', 'Gamma', 'Alpha']]).should be_true
+    importer.find_header([['Bob', 'Gamma', 'Beta']]).should be_true
+    # Required + both optional
+    importer.find_header([['Alpha', 'Gamma', 'Beta']]).should be_true
+  end
+
   it 'should capture errors' do
     importer = Importer.build do
       column :foo
@@ -108,7 +176,7 @@ describe Importer do
   it 'should import a string' do
     sum = 0
     csv = "one,two\n1,2"
-    Importer.build do
+    importer = Importer.build do
       column :one
       column :two
     end.import_string(csv, :format => :csv) do |rows|
@@ -117,6 +185,7 @@ describe Importer do
       sum = rows[:one].to_i + rows[:two].to_i
     end
     # Just make sure we ran correctly
+    importer.column(:one).to_s.should == 'One Column'
     sum.should == 3
   end
   
@@ -131,5 +200,54 @@ describe Importer do
     importer.import_string("<div><table><tr><td>one</td></tr></table></div>")
     importer.format.should == :html
   end
+
+  it 'should capture errors with context' do
+    sum = 0
+    csv = "one,two,three\n1,2,X\n1,,3"
+    importer = Importer.build do
+      column :one
+      column :two do
+        validate do |val|
+          val.to_i == 2
+        end
+      end
+      column :three do
+        validate do |val|
+          add_error('Invalid value') unless val.to_i > 0
+        end
+      end
+    end
+    importer.import_string(csv)
+    
+    # Just make sure we ran correctly
+    importer.errors.count.should == 2
+    importer.column(:two).errors.count.should == 1
+    importer.column(:three).errors.count.should == 1
+    importer.column(:three).error_values.should == ['X']
+    map = importer.rows.first.error_map
+    map[:two].should be_nil
+    map[:three].should be_a(Importer::Error)
+  end
+  
+  it 'should import properly when optional columns are missing' do
+    csv = "one,two\n1,2\n1,"
+    importer = Importer.build do
+      column :one
+      column :two do
+        validate do |val|
+          val.to_i == 2
+        end
+      end
+      column :three do
+        optional!
+        validate do |val|
+          add_error('Invalid value') unless val.to_i > 0
+        end
+      end
+    end
+    importer.import_string(csv)
+    
+    importer.found_columns.count.should == 2
+  end
   
 end

data/spec/importer/row_spec.rb

@@ -33,6 +33,11 @@ describe Importer::Row do
     @row.should be_empty
   end
   
+  it 'should return nil on missing data' do
+    @row.set_values(:a => 1, :b => 2)
+    @row[:c].should be_nil
+  end
+  
   it 'should not change when to_hash values are changed' do
     @row.set_values(:a => 1, :b => 2)
     hash = @row.to_hash

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: iron-import
 version: !ruby/object:Gem::Version
-  version: 0.7.0
+  version: 0.8.0
 platform: ruby
 authors:
 - Rob Morris
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2017-02-16 00:00:00.000000000 Z
+date: 2017-06-29 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: iron-extensions
@@ -117,6 +117,7 @@ files:
 - spec/importer/csv_reader_spec.rb
 - spec/importer/custom_reader_spec.rb
 - spec/importer/data_reader_spec.rb
+- spec/importer/error_spec.rb
 - spec/importer/html_reader_spec.rb
 - spec/importer/importer_spec.rb
 - spec/importer/row_spec.rb
@@ -156,5 +157,5 @@ rubyforge_project:
 rubygems_version: 2.4.3
 signing_key: 
 specification_version: 4
-summary: CSV, HTML, XLS, and XLSX import automation support
+summary: CSV, HTML, XLS, and XLSX import processing support
 test_files: []