smarter_csv 1.0.1 → 1.0.2

data/README.md

@@ -125,25 +125,38 @@ or an Array of Arrays, which contain Hashes, or processes Chunks of Hashes via a
 The options and the block are optional.
 
 `SmarterCSV.process` supports the following options:
- * :col_sep : column separator , which defaults to ','
- * :row_sep : row separator or record separator , defaults to system's $/ , which defaults to "\n"
- * :quote_char : quotation character , defaults to '"'
- * :comment_regexp : regular expression which matches comment lines , defaults to /^#/ (see NOTE about the CSV header)
- * :chunk_size : if set, determines the desired chunk-size (defaults to nil, no chunk processing)
- * :key_mapping : a hash which maps headers from the CSV file to keys in the result hash (default: nil)
- * :downcase_header : downcase all column headers (default: true)
- * :strings_as_keys : use strings instead of symbols as the keys in the result hashes (default: false)
- * :remove_empty_values : remove values which have nil or empty strings as values (default: true)
- * :remove_zero_values  : remove values which have a numeric value equal to zero / 0 (default: false)
- * :remove_values_matching : removes key/value pairs if value matches given regular expressions (default: nil) , e.g. /^\$0\.0+$/ to match $0.00 , or /^#VALUE!$/ to match errors in Excel spreadsheets
- * :convert_values_to_numeric : converts strings containing Integers or Floats to the appropriate class (default: true)
- * :remove_empty_hashes : remove / ignore any hashes which don't have any key/value pairs (default: true)
+
+      Option                     | Default  |  Explanation
+    -----------------------------+----------+--------------------------------------------------------------
+     :col_sep                    |   ','    | column separator
+     :row_sep                    | $/ ,"\n" | row separator or record separator , defaults to system's $/ , which defaults to "\n"
+     :quote_char                 |   '"'    | quotation character
+     :comment_regexp             |   /^#/   | regular expression which matches comment lines (see NOTE about the CSV header)
+     :chunk_size                 |   nil    | if set, determines the desired chunk-size (defaults to nil, no chunk processing)
+     :key_mapping                |   nil    | a hash which maps headers from the CSV file to keys in the result hash
+     :downcase_header            |   true   | downcase all column headers
+     :strings_as_keys            |   false  | use strings instead of symbols as the keys in the result hashes
+     :remove_empty_values        |   true   | remove values which have nil or empty strings as values
+     :remove_zero_values         |   true   | remove values which have a numeric value equal to zero / 0
+     :remove_values_matching     |   nil    | removes key/value pairs if value matches given regular expressions. e.g.:
+                                 |          | /^\$0\.0+$/ to match $0.00 , or /^#VALUE!$/ to match errors in Excel spreadsheets
+     :convert_values_to_numeric  |   true   | converts strings containing Integers or Floats to the appropriate class
+     :remove_empty_hashes        |   true   | remove / ignore any hashes which don't have any key/value pairs
+     :user_provided_headers      |   nil    | user provided Array of header strings or symbols, to define
+                                 |          | what headers should be used, overriding any in-file headers. (dangerous)
+                                 |          | You can not combine the :user_provided_headers and :key_mapping options
+     :headers_in_file            |   true   | Whether or not the file contains headers as the first line. 
+                                 |          | Important if the file does not contain headers, 
+                                 |          | otherwise you would lose the first line of data.
+
 
 #### NOTES about CSV Headers:
  * as this method parses CSV files, it is assumed that the first line of any file will contain a valid header
  * the first line with the CSV header may or may not be commented out according to the :comment_regexp
  * any occurences of :comment_regexp or :row_sep will be stripped from the first line with the CSV header
  * any of the keys in the header line will be downcased, spaces replaced by underscore, and converted to Ruby symbols before being used as keys in the returned Hashes
+ * you can not combine the :user_provided_headers and :key_mapping options
+ * if the incorrect number of headers are provided via :user_provided_headers, exception SmarterCSV::HeaderSizeMismatch is raised
 
 #### NOTES on Key Mapping:
  * keys in the header line of the file can be re-mapped to a chosen set of symbols, so the resulting Hashes can be better used internally in your application (e.g. when directly creating MongoDB entries with them)
@@ -178,8 +191,15 @@ Or install it yourself as:
     $ gem install smarter_csv
 
 
+
 ## Changes
 
+#### 1.0.1 (2012-08-02)
+
+ * added more options for dealing with headers:
+    * :user_provided_headers ,user provided Array with header strings or symbols, to precisely define what the headers should be, overriding any in-file headers (default: nil)
+    * :headers_in_file , if the file contains headers as the first line (default: true)
+
 #### 1.0.1 (2012-07-30)
 
  * added the following options: 

data/lib/smarter_csv/smarter_csv.rb

@@ -1,8 +1,12 @@
 module SmarterCSV
+
+  class HeaderSizeMismatch < Exception
+  end
+
   def SmarterCSV.process(filename, options={}, &block)
     default_options = {:col_sep => ',' , :row_sep => $/ , :quote_char => '"',
       :remove_empty_values => true, :remove_zero_values => false , :remove_values_matching => nil , :remove_empty_hashes => true ,
-      :convert_values_to_numeric => true, :strip_chars_from_headers => nil ,
+      :convert_values_to_numeric => true, :strip_chars_from_headers => nil , :user_provided_headers => nil , :headers_in_file => true,
       :comment_regexp => /^#/, :chunk_size => nil , :key_mapping_hash => nil , :downcase_header => true, :strings_as_keys => false 
     }
     options = default_options.merge(options)
@@ -12,20 +16,39 @@ module SmarterCSV
     begin
       $/ = options[:row_sep]
       f = File.open(filename, "r")
-      
-      # process the header line in the CSV file..
-      # the first line of a CSV file contains the header .. it might be commented out, so we need to read it anyhow
-      headerA = f.readline.sub(options[:comment_regexp],'').chomp(options[:row_sep])
-      headerA = headerA.gsub(options[:strip_chars_from_headers], '') if options[:strip_chars_from_headers]
-      headerA = headerA.split(options[:col_sep]).map{|x| x.gsub(%r/options[:quote_char]/,'').gsub(/\s+/,'_')}
-      headerA.map!{|x| x.downcase }   if options[:downcase_header]
+
+      if options[:headers_in_file]        # extract the header line
+        # process the header line in the CSV file..
+        # the first line of a CSV file contains the header .. it might be commented out, so we need to read it anyhow
+        header = f.readline.sub(options[:comment_regexp],'').chomp(options[:row_sep])
+        header = header.gsub(options[:strip_chars_from_headers], '') if options[:strip_chars_from_headers]
+        file_headerA = header.split(options[:col_sep]).map{|x| x.gsub(%r/options[:quote_char]/,'').gsub(/\s+/,'_')}
+        file_headerA.map!{|x| x.downcase }   if options[:downcase_header]
+        file_header_size = file_headerA.size
+      end
+      if options[:user_provided_headers] && options[:user_provided_headers].class == Array && ! options[:user_provided_headers].empty?
+        # use user-provided headers 
+        headerA = options[:user_provided_headers]
+        if defined?(file_header_size)
+          if headerA.size != file_header_size
+            raise SmarterCSV::HeaderSizeMismatch , "ERROR [smarter_csv]: :user_provided_headers defines #{headerA.size} headers !=  CSV-file #{filename} has #{file_header_size} headers" 
+          else
+            # we could print out the mapping of file_headerA to headerA here
+          end
+        end
+      else
+        headerA = file_headerA
+      end
       headerA.map!{|x| x.to_sym } unless options[:strings_as_keys]
-      key_mappingH = options[:key_mapping]
       
-      # do some key mapping on the keys in the file header
-      #   if you want to completely delete a key, then map it to nil or to ''
-      if ! key_mappingH.nil? && key_mappingH.class == Hash && key_mappingH.keys.size > 0
-        headerA.map!{|x| key_mappingH.has_key?(x) ? (key_mappingH[x].nil? ? nil : key_mappingH[x].to_sym) : x}
+      unless options[:user_provided_headers] # wouldn't make sense to re-map user provided headers 
+        key_mappingH = options[:key_mapping]
+      
+        # do some key mapping on the keys in the file header
+        #   if you want to completely delete a key, then map it to nil or to ''
+        if ! key_mappingH.nil? && key_mappingH.class == Hash && key_mappingH.keys.size > 0
+          headerA.map!{|x| key_mappingH.has_key?(x) ? (key_mappingH[x].nil? ? nil : key_mappingH[x].to_sym) : x}
+        end
       end
 
       # in case we use chunking.. we'll need to set it up..

data/lib/smarter_csv/version.rb

@@ -1,3 +1,3 @@
 module SmarterCSV
-  VERSION = "1.0.1"
+  VERSION = "1.0.2"
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: smarter_csv
 version: !ruby/object:Gem::Version
-  version: 1.0.1
+  version: 1.0.2
   prerelease: 
 platform: ruby
 authors:
@@ -11,7 +11,7 @@ authors:
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2012-07-30 00:00:00.000000000 Z
+date: 2012-08-02 00:00:00.000000000 Z
 dependencies: []
 description: Ruby Gem for smarter importing of CSV Files as Array(s) of Hashes, with
   optional features for processing large files in parallel, embedded comments, unusual