smarter_csv 1.1.5 → 1.12.1

data/docs/row_col_sep.md

@@ -0,0 +1,104 @@
+
+### Contents
+
+  * [Introduction](./_introduction.md)
+  * [The Basic API](./basic_api.md)
+  * [Batch Processing](././batch_processing.md)
+  * [Configuration Options](./options.md)
+  * [**Row and Column Separators**](./row_col_sep.md)
+  * [Header Transformations](./header_transformations.md)
+  * [Header Validations](./header_validations.md)
+  * [Data Transformations](./data_transformations.md)
+  * [Value Converters](./value_converters.md)
+    
+--------------    
+
+# Row and Column Separators
+
+## Automatic Detection
+
+Convenient defaults allow automatic detection of the column and row separators: `row_sep: :auto`, `col_sep: :auto`. This makes it easier to process any CSV files without having to examine the line endings or column separators, e.g. when users upload CSV files to your service and you have no control over the incoming files.
+
+You can change the setting `:auto_row_sep_chars` to only analyze the first N characters of the file (default is 500 characters); `nil` or `0` will check the whole file). Of course you can also set the `:row_sep` manually.
+
+
+## Column Separator `col_sep`
+
+The automatic detection of column separators considers: `,`, `\t`, `;`, `:`, `|`.
+
+Some CSV files may contain an unusual column separqator, which could even be a control character.
+
+## Row Separator `row_sep`
+
+The automatic detection of row separators considers: `\n`, `\r\n`, `\r`.
+
+Some CSV files may contain an unusual row separqator, which could even be a control character.
+
+
+## Custom / Non-Standard CSV Formats
+
+Besides custom values for `col_sep`, `row_sep`, some other customizations of CSV files are:
+*  the presence of a number of leading lines before the header or data section start.
+*  the presence of comment lines, e.g. lines starting with `#`
+
+To explore these special cases, please use the following examples.
+
+### Example 1: reading an iTunes DB dump
+
+This data format uses CTRL-A as the column separator, and CTRL-B as the record separator. It also has comment lines that start with a `#` character. This also maps the header `name` to `genre`, and ignores the column `export_date`.
+
+```ruby
+    filename = '/tmp/itunes_db_dump'   
+    options = {
+      :col_sep => "\cA", :row_sep => "\cB\n", :comment_regexp => /^#/,
+      :chunk_size => 100 , :key_mapping => {export_date: nil, name: :genre},
+    }
+    n = SmarterCSV.process(filename, options) do |chunk|
+      SidekiqWorkerClass.process_async(chunk) # pass an array of hashes to Sidekiq workers for parallel processing
+    end
+    => returns number of chunks
+```
+
+### Example 2: Reading a CSV-File with custom col_sep, row_sep
+In this example we have an unusual CSV file with `|` as the row separator, and `#` as the column separator.
+This unusual format needs explicit options `col_sep` and `row_sep`.
+
+```ruby
+    filename = '/tmp/input_file.txt'
+    recordsA = SmarterCSV.process(filename, {col_sep: "#", row_sep: "|"})
+
+    => returns an array of hashes
+```
+
+### Example 3:
+In this example, we use `skip_lines: 3` to skip and ignore the first 3 lines in the input
+
+
+```ruby
+    filename = '/tmp/input_file.txt'
+    recordsA = SmarterCSV.process(filename, {skip_lines: 3})
+
+    => returns an array of hashes
+```
+  
+
+### Example 4: reading an iTunes DB dump
+
+In this example, we use `comment_regexp` to filter out and ignore any lines starting with `#`
+
+
+```ruby
+    # Consider a file with CRTL-A as col_separator, and with CTRL-B\n as record_separator (hello iTunes!)
+    filename = '/tmp/strange_db_dump'   
+    options = {
+      :col_sep => "\cA", :row_sep => "\cB\n", :comment_regexp => /^#/,
+      :chunk_size => 100 , :key_mapping => {:export_date => nil, :name => :genre},
+    }
+    n = SmarterCSV.process(filename, options) do |chunk|
+      SidekiqWorkerClass.process_async(chunk) # pass an array of hashes to Sidekiq workers for parallel processing
+    end
+    => returns number of chunks
+```
+
+----------------
+PREVIOUS: [Configuration Options](./options.md) | NEXT: [Header Transformations](./header_transformations.md)

data/docs/value_converters.md

@@ -0,0 +1,68 @@
+
+### Contents
+
+  * [Introduction](./_introduction.md)
+  * [The Basic API](./basic_api.md)
+  * [Batch Processing](././batch_processing.md)
+  * [Configuration Options](./options.md)
+  * [Row and Column Separators](./row_col_sep.md)
+  * [Header Transformations](./header_transformations.md)
+  * [Header Validations](./header_validations.md)
+  * [Data Transformations](./data_transformations.md)
+  * [**Value Converters**](./value_converters.md)
+    
+--------------  
+
+# Using Value Converters
+
+Value Converters allow you to do custom transformations specific rows, to help you massage the data so it fits the expectations of your down-stream process, such as creating a DB record.
+
+If you use `key_mappings` and `value_converters`, make sure that the value converters references the keys based on the final mapped name, not the original name in the CSV file.
+
+```ruby
+    $ cat spec/fixtures/with_dates.csv
+    first,last,date,price
+    Ben,Miller,10/30/1998,$44.50
+    Tom,Turner,2/1/2011,$15.99
+    Ken,Smith,01/09/2013,$199.99
+
+    $ irb
+    > require 'smarter_csv'
+    > require 'date'
+
+    # define a custom converter class, which implements self.convert(value)
+    class DateConverter
+      def self.convert(value)
+        Date.strptime( value, '%m/%d/%Y') # parses custom date format into Date instance
+      end
+    end
+
+    class DollarConverter
+      def self.convert(value)
+        value.sub('$','').to_f # strips the dollar sign and creates a Float value
+      end
+    end
+
+    require 'money'
+    class MoneyConverter
+      def self.convert(value)
+        # depending on locale you might want to also remove the indicator for thousands, e.g. comma 
+        Money.from_amount(value.gsub(/[\s\$]/,'').to_f) # creates a Money instance (based on cents)
+      end
+    end
+
+    options = {:value_converters => {:date => DateConverter, :price => DollarConverter}}
+    data = SmarterCSV.process("spec/fixtures/with_dates.csv", options)
+    first_record = data.first
+    first_record[:date]
+      => #<Date: 1998-10-30 ((2451117j,0s,0n),+0s,2299161j)>
+    first_record[:date].class
+      => Date
+    first_record[:price]
+      => 44.50
+    first_record[:price].class
+      => Float
+```
+
+--------------------
+PREVIOUS: [Data Transformations](./data_transformations.md) | UP: [README](../README.md)

data/ext/smarter_csv/extconf.rb

@@ -0,0 +1,14 @@
+# frozen_string_literal: true
+
+require 'mkmf'
+require "rbconfig"
+
+if RbConfig::MAKEFILE_CONFIG["CFLAGS"].include?("-g -O3")
+  fixed_CFLAGS = RbConfig::MAKEFILE_CONFIG["CFLAGS"].sub("-g -O3", "-O3 $(cflags)")
+  puts("Fix CFLAGS: #{RbConfig::MAKEFILE_CONFIG["CFLAGS"]} -> #{fixed_CFLAGS}")
+  RbConfig::MAKEFILE_CONFIG["CFLAGS"] = fixed_CFLAGS
+end
+
+CONFIG["optflags"] = "-O3"
+
+create_makefile('smarter_csv/smarter_csv')

data/ext/smarter_csv/smarter_csv.c

@@ -0,0 +1,97 @@
+#include "ruby.h"
+#include "ruby/encoding.h"
+#include <stdio.h>
+#include <stdbool.h>
+
+#ifndef bool
+  #define bool int
+  #define false ((bool)0)
+  #define true  ((bool)1)
+#endif
+
+/*
+   max_size: pass nil if no limit is specified
+ */
+static VALUE rb_parse_csv_line(VALUE self, VALUE line, VALUE col_sep, VALUE quote_char, VALUE max_size) {
+  if (RB_TYPE_P(line, T_NIL) == 1) {
+    return rb_ary_new();
+  }
+
+  if (RB_TYPE_P(line, T_STRING) != 1) {
+    rb_raise(rb_eTypeError, "ERROR in SmarterCSV.parse_line: line has to be a string or nil");
+  }
+
+  rb_encoding *encoding = rb_enc_get(line); /* get the encoding from the input line */
+  char *startP = RSTRING_PTR(line); /* may not be null terminated */
+  long line_len = RSTRING_LEN(line);
+  char *endP = startP + line_len ; /* points behind the string */
+  char *p = startP;
+
+  char *col_sepP = RSTRING_PTR(col_sep);
+  long col_sep_len = RSTRING_LEN(col_sep);
+
+  char *quoteP = RSTRING_PTR(quote_char);
+  long quote_count = 0;
+
+  bool col_sep_found = true;
+
+  VALUE elements = rb_ary_new();
+  VALUE field;
+  long i;
+
+  char prev_char = '\0'; // Store the previous character for comparison against an escape character
+  long backslash_count = 0; // to count consecutive backslash characters
+
+  while (p < endP) {
+    /* does the remaining string start with col_sep ? */
+    col_sep_found = true;
+    for(i=0; (i < col_sep_len) && (p+i < endP) ; i++) {
+      col_sep_found = col_sep_found && (*(p+i) == *(col_sepP+i));
+    }
+    /* if col_sep was found and we have even quotes */
+    if (col_sep_found && (quote_count % 2 == 0)) {
+      /* if max_size != nil && lements.size >= header_size */
+      if ((max_size != Qnil) && RARRAY_LEN(elements) >= NUM2INT(max_size)) {
+        break;
+      } else {
+        /* push that field with original encoding onto the results */
+        field = rb_enc_str_new(startP, p - startP, encoding);
+        rb_ary_push(elements, field);
+
+        p += col_sep_len;
+        startP = p;
+      }
+    } else {
+      if (*p == '\\') {
+        backslash_count++;
+      } else {
+        if (*p == *quoteP && (backslash_count % 2 == 0)) {
+          quote_count++;
+        }
+        backslash_count = 0; // no more consecutive backslash characters
+      }
+      p++;
+    }
+
+    prev_char = *(p - 1); // Update the previous character
+  } /* while */
+
+  /* check if the last part of the line needs to be processed */
+  if ((max_size == Qnil) || RARRAY_LEN(elements) < NUM2INT(max_size)) {
+    /* copy the remaining line as a field with original encoding onto the results */
+    field = rb_enc_str_new(startP, endP - startP, encoding);
+    rb_ary_push(elements, field);
+  }
+
+  return elements;
+}
+
+VALUE SmarterCSV = Qnil;
+VALUE Parser = Qnil;
+
+void Init_smarter_csv(void) {
+  SmarterCSV = rb_define_module("SmarterCSV");
+  Parser = rb_define_module_under(SmarterCSV, "Parser");
+
+  rb_define_module_function(Parser, "parse_csv_line_c", rb_parse_csv_line, 4);
+}

data/lib/smarter_csv/auto_detection.rb

@@ -0,0 +1,78 @@
+# frozen_string_literal: true
+
+module SmarterCSV
+  module AutoDetection
+    protected
+
+    # If file has headers, then guesses column separator from headers.
+    # Otherwise guesses column separator from contents.
+    # Raises exception if none is found.
+    def guess_column_separator(filehandle, options)
+      skip_lines(filehandle, options)
+
+      delimiters = [',', "\t", ';', ':', '|']
+
+      line = nil
+      has_header = options[:headers_in_file]
+      candidates = Hash.new(0)
+      count = has_header ? 1 : 5
+      count.times do
+        line = readline_with_counts(filehandle, options)
+        delimiters.each do |d|
+          escaped_quote = Regexp.escape(options[:quote_char])
+
+          # Count only non-quoted occurrences of the delimiter
+          non_quoted_text = line.split(/#{escaped_quote}[^#{escaped_quote}]*#{escaped_quote}/).join
+
+          candidates[d] += non_quoted_text.scan(d).count
+        end
+      rescue EOFError # short files
+        break
+      end
+      rewind(filehandle)
+
+      if candidates.values.max == 0
+        # if the header only contains
+        return ',' if line.chomp(options[:row_sep]) =~ /^\w+$/
+
+        raise SmarterCSV::NoColSepDetected
+      end
+
+      candidates.key(candidates.values.max)
+    end
+
+    # limitation: this currently reads the whole file in before making a decision
+    def guess_line_ending(filehandle, options)
+      counts = {"\n" => 0, "\r" => 0, "\r\n" => 0}
+      quoted_char = false
+
+      # count how many of the pre-defined line-endings we find
+      # ignoring those contained within quote characters
+      last_char = nil
+      lines = 0
+      filehandle.each_char do |c|
+        quoted_char = !quoted_char if c == options[:quote_char]
+        next if quoted_char
+
+        if last_char == "\r"
+          if c == "\n"
+            counts["\r\n"] += 1
+          else
+            counts["\r"] += 1 # \r are counted after they appeared
+          end
+        elsif c == "\n"
+          counts["\n"] += 1
+        end
+        last_char = c
+        lines += 1
+        break if options[:auto_row_sep_chars] && options[:auto_row_sep_chars] > 0 && lines >= options[:auto_row_sep_chars]
+      end
+      rewind(filehandle)
+
+      counts["\r"] += 1 if last_char == "\r"
+      # find the most frequent key/value pair:
+      most_frequent_key, _count = counts.max_by{|_, v| v}
+      most_frequent_key
+    end
+  end
+end

data/lib/smarter_csv/errors.rb

@@ -0,0 +1,16 @@
+# frozen_string_literal: true
+
+module SmarterCSV
+  class Error < StandardError; end # new code should rescue this instead
+  # Reader:
+  class SmarterCSVException < Error; end # for backwards compatibility
+  class HeaderSizeMismatch < SmarterCSVException; end
+  class IncorrectOption < SmarterCSVException; end
+  class ValidationError < SmarterCSVException; end
+  class DuplicateHeaders < SmarterCSVException; end
+  class MissingKeys < SmarterCSVException; end # previously known as MissingHeaders
+  class NoColSepDetected < SmarterCSVException; end
+  class KeyMappingError < SmarterCSVException; end
+  # Writer:
+  class InvalidInputData < SmarterCSVException; end
+end

data/lib/smarter_csv/file_io.rb

@@ -0,0 +1,50 @@
+# frozen_string_literal: true
+
+module SmarterCSV
+  module FileIO
+    protected
+
+    def readline_with_counts(filehandle, options)
+      line = filehandle.readline(options[:row_sep])
+      @file_line_count += 1
+      @csv_line_count += 1
+      line = remove_bom(line) if @csv_line_count == 1
+      line
+    end
+
+    def skip_lines(filehandle, options)
+      options[:skip_lines].to_i.times do
+        readline_with_counts(filehandle, options)
+      end
+    end
+
+    def rewind(filehandle)
+      @file_line_count = 0
+      @csv_line_count = 0
+      filehandle.rewind
+    end
+
+    private
+
+    UTF_32_BOM = %w[0 0 fe ff].freeze
+    UTF_32LE_BOM = %w[ff fe 0 0].freeze
+    UTF_8_BOM = %w[ef bb bf].freeze
+    UTF_16_BOM = %w[fe ff].freeze
+    UTF_16LE_BOM = %w[ff fe].freeze
+
+    def remove_bom(str)
+      str_as_hex = str.bytes.map{|x| x.to_s(16)}
+      # if string does not start with one of the bytes, there is no BOM
+      return str unless %w[ef fe ff 0].include?(str_as_hex[0])
+
+      return str.byteslice(4..-1) if [UTF_32_BOM, UTF_32LE_BOM].include?(str_as_hex[0..3])
+      return str.byteslice(3..-1) if str_as_hex[0..2] == UTF_8_BOM
+      return str.byteslice(2..-1) if [UTF_16_BOM, UTF_16LE_BOM].include?(str_as_hex[0..1])
+
+      # :nocov:
+      puts "SmarterCSV found unhandled BOM! #{str.chars[0..7].inspect}"
+      str
+      # :nocov:
+    end
+  end
+end

data/lib/smarter_csv/hash_transformations.rb

@@ -0,0 +1,91 @@
+# frozen_string_literal: true
+
+module SmarterCSV
+  module HashTransformations
+    def hash_transformations(hash, options)
+      # there may be unmapped keys, or keys purposedly mapped to nil or an empty key..
+      # make sure we delete any key/value pairs from the hash, which the user wanted to delete:
+      remove_empty_values = options[:remove_empty_values] == true
+      remove_zero_values = options[:remove_zero_values]
+      remove_values_matching = options[:remove_values_matching]
+      convert_to_numeric = options[:convert_values_to_numeric]
+      value_converters = options[:value_converters]
+
+      hash.each_with_object({}) do |(k, v), new_hash|
+        next if k.nil? || k == '' || k == :""
+        next if remove_empty_values && (has_rails ? v.blank? : blank?(v))
+        next if remove_zero_values && v.is_a?(String) && v =~ /^(0+|0+\.0+)$/ # values are Strings
+        next if remove_values_matching && v =~ remove_values_matching
+
+        # deal with the :only / :except options to :convert_values_to_numeric
+        if convert_to_numeric && !limit_execution_for_only_or_except(options, :convert_values_to_numeric, k)
+          if v =~ /^[+-]?\d+\.\d+$/
+            v = v.to_f
+          elsif v =~ /^[+-]?\d+$/
+            v = v.to_i
+          end
+        end
+
+        converter = value_converters[k] if value_converters
+        v = converter.convert(v) if converter
+
+        new_hash[k] = v
+      end
+    end
+
+    # def hash_transformations(hash, options)
+    #   # there may be unmapped keys, or keys purposedly mapped to nil or an empty key..
+    #   # make sure we delete any key/value pairs from the hash, which the user wanted to delete:
+    #   hash.delete(nil)
+    #   hash.delete('')
+    #   hash.delete(:"")
+
+    #   if options[:remove_empty_values] == true
+    #     hash.delete_if{|_k, v| has_rails ? v.blank? : blank?(v)}
+    #   end
+
+    #   hash.delete_if{|_k, v| !v.nil? && v =~ /^(0+|0+\.0+)$/} if options[:remove_zero_values] # values are Strings
+    #   hash.delete_if{|_k, v| v =~ options[:remove_values_matching]} if options[:remove_values_matching]
+
+    #   if options[:convert_values_to_numeric]
+    #     hash.each do |k, v|
+    #       # deal with the :only / :except options to :convert_values_to_numeric
+    #       next if limit_execution_for_only_or_except(options, :convert_values_to_numeric, k)
+
+    #       # convert if it's a numeric value:
+    #       case v
+    #       when /^[+-]?\d+\.\d+$/
+    #         hash[k] = v.to_f
+    #       when /^[+-]?\d+$/
+    #         hash[k] = v.to_i
+    #       end
+    #     end
+    #   end
+
+    #   if options[:value_converters]
+    #     hash.each do |k, v|
+    #       converter = options[:value_converters][k]
+    #       next unless converter
+
+    #       hash[k] = converter.convert(v)
+    #     end
+    #   end
+
+    #   hash
+    # end
+
+    protected
+
+    # acts as a road-block to limit processing when iterating over all k/v pairs of a CSV-hash:
+    def limit_execution_for_only_or_except(options, option_name, key)
+      if options[option_name].is_a?(Hash)
+        if options[option_name].has_key?(:except)
+          return true if Array(options[option_name][:except]).include?(key)
+        elsif options[option_name].has_key?(:only)
+          return true unless Array(options[option_name][:only]).include?(key)
+        end
+      end
+      false
+    end
+  end
+end

data/lib/smarter_csv/header_transformations.rb

@@ -0,0 +1,63 @@
+# frozen_string_literal: true
+
+module SmarterCSV
+  module HeaderTransformations
+    # transform the headers that were in the file:
+    def header_transformations(header_array, options)
+      header_array.map!{|x| x.gsub(%r/#{options[:quote_char]}/, '')}
+      header_array.map!{|x| x.strip} if options[:strip_whitespace]
+
+      unless options[:keep_original_headers]
+        header_array.map!{|x| x.gsub(/\s+|-+/, '_')}
+        header_array.map!{|x| x.downcase} if options[:downcase_header]
+      end
+
+      # detect duplicate headers and disambiguate
+      header_array = disambiguate_headers(header_array, options) if options[:duplicate_header_suffix]
+      # symbolize headers
+      header_array = header_array.map{|x| x.to_sym } unless options[:strings_as_keys] || options[:keep_original_headers]
+      # doesn't make sense to re-map when we have user_provided_headers
+      header_array = remap_headers(header_array, options) if options[:key_mapping]
+
+      header_array
+    end
+
+    def disambiguate_headers(headers, options)
+      counts = Hash.new(0)
+      headers.map do |header|
+        counts[header] += 1
+        counts[header] > 1 ? "#{header}#{options[:duplicate_header_suffix]}#{counts[header]}" : header
+      end
+    end
+
+    # do some key mapping on the keys in the file header
+    # if you want to completely delete a key, then map it to nil or to ''
+    def remap_headers(headers, options)
+      key_mapping = options[:key_mapping]
+      if key_mapping.empty? || !key_mapping.is_a?(Hash) || key_mapping.keys.empty?
+        raise(SmarterCSV::IncorrectOption, "ERROR: incorrect format for key_mapping! Expecting hash with from -> to mappings")
+      end
+
+      key_mapping = options[:key_mapping]
+      # if silence_missing_keys are not set, raise error if missing header
+      missing_keys = key_mapping.keys - headers
+      # if the user passes a list of speciffic mapped keys that are optional
+      missing_keys -= options[:silence_missing_keys] if options[:silence_missing_keys].is_a?(Array)
+
+      unless missing_keys.empty? || options[:silence_missing_keys] == true
+        raise SmarterCSV::KeyMappingError, "ERROR: can not map headers: #{missing_keys.join(', ')}"
+      end
+
+      headers.map! do |header|
+        if key_mapping.has_key?(header)
+          key_mapping[header].nil? ? nil : key_mapping[header]
+        elsif options[:remove_unmapped_keys]
+          nil
+        else
+          header
+        end
+      end
+      headers
+    end
+  end
+end

data/lib/smarter_csv/header_validations.rb

@@ -0,0 +1,34 @@
+# frozen_string_literal: true
+
+module SmarterCSV
+  module HeaderValidations
+    def header_validations(headers, options)
+      check_duplicate_headers(headers, options)
+      check_required_headers(headers, options)
+    end
+
+    def check_duplicate_headers(headers, _options)
+      header_counts = Hash.new(0)
+      headers.each { |header| header_counts[header] += 1 unless header.nil? }
+
+      duplicates = header_counts.select { |_, count| count > 1 }
+
+      unless duplicates.empty?
+        raise(SmarterCSV::DuplicateHeaders, "Duplicate Headers in CSV: #{duplicates.inspect}")
+      end
+    end
+
+    require 'set'
+
+    def check_required_headers(headers, options)
+      if options[:required_keys] && options[:required_keys].is_a?(Array)
+        headers_set = headers.to_set
+        missing_keys = options[:required_keys].select { |k| !headers_set.include?(k) }
+
+        unless missing_keys.empty?
+          raise SmarterCSV::MissingKeys, "ERROR: missing attributes: #{missing_keys.join(',')}. Check `reader.headers` for original headers."
+        end
+      end
+    end
+  end
+end

data/lib/smarter_csv/headers.rb

@@ -0,0 +1,68 @@
+# frozen_string_literal: true
+
+module SmarterCSV
+  module Headers
+    def process_headers(filehandle, options)
+      @raw_header = nil # header as it appears in the file
+      @headers = nil # the processed headers
+      header_array = []
+      file_header_size = nil
+
+      # if headers_in_file, get the headers -> We get the number of columns, even when user provided headers
+      if options[:headers_in_file] # extract the header line
+        # process the header line in the CSV file..
+        # the first line of a CSV file contains the header .. it might be commented out, so we need to read it anyhow
+        header_line = @raw_header = readline_with_counts(filehandle, options)
+        header_line = preprocess_header_line(header_line, options)
+
+        file_header_array, file_header_size = parse(header_line, options)
+
+        file_header_array = header_transformations(file_header_array, options)
+
+      else
+        unless options[:user_provided_headers]
+          raise SmarterCSV::IncorrectOption, "ERROR: If :headers_in_file is set to false, you have to provide :user_provided_headers"
+        end
+      end
+
+      if options[:user_provided_headers]
+        unless options[:user_provided_headers].is_a?(Array) && !options[:user_provided_headers].empty?
+          raise(SmarterCSV::IncorrectOption, "ERROR: incorrect format for user_provided_headers! Expecting array with headers.")
+        end
+
+        # use user-provided headers
+        user_header_array = options[:user_provided_headers]
+        # user_provided_headers: their count should match the headers_in_file if any
+        if defined?(file_header_size) && !file_header_size.nil?
+          if user_header_array.size != file_header_size
+            raise SmarterCSV::HeaderSizeMismatch, "ERROR: :user_provided_headers defines #{user_header_array.size} headers !=  CSV-file has #{file_header_size} headers"
+          else
+            # we could print out the mapping of file_header_array to header_array here
+          end
+        end
+
+        header_array = user_header_array
+      else
+        header_array = file_header_array
+      end
+
+      [header_array, header_array.size]
+    end
+
+    private
+
+    def preprocess_header_line(header_line, options)
+      header_line = enforce_utf8_encoding(header_line, options)
+      header_line = remove_comments_from_header(header_line, options)
+      header_line = header_line.chomp(options[:row_sep])
+      header_line.gsub!(options[:strip_chars_from_headers], '') if options[:strip_chars_from_headers]
+      header_line
+    end
+
+    def remove_comments_from_header(header, options)
+      return header unless options[:comment_regexp]
+
+      header.sub(options[:comment_regexp], '')
+    end
+  end
+end